2024-03-23T11:18:27.114 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T11:18:27.114 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rishabh@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239 -- /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239/orig.config.yaml 2024-03-23T11:18:27.168 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T11:18:27.250 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239 branch: rishabh-yuri-RunCephCmd-base-reef-2 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7618239' kernel: flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: rishabh-yuri-RunCephCmd-base-reef-2 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b630d6cd702af63dfefcf906294a092ddce1c609 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b630d6cd702af63dfefcf906294a092ddce1c609 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-yuri-RunCephCmd-base-reef-2 sha1: b630d6cd702af63dfefcf906294a092ddce1c609 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1956 sha1: b630d6cd702af63dfefcf906294a092ddce1c609 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: rishabh-yuri-RunCephCmd-base-reef-2 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b630d6cd702af63dfefcf906294a092ddce1c609 targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZ/qoO9C16Im0WCrliXXl/Y2P2XOEe+HAy+M748RPelNmAHfXFhkVNM7bBHuNW7JJ1GoeUNXTomtEQMb0/j7u4= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5j5A6YBMOOvWGAY3htlUVPNCZy+Qx5eeo7L3ouY7ywjmsNbxbhHOsxbHaOhwejKBEuJ2x7V4KNcQrCPKQuWY0= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-23_09:38:22 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-23T11:18:27.250 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa; will attempt to use it 2024-03-23T11:18:27.251 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks 2024-03-23T11:18:27.251 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T11:18:27.253 INFO:teuthology.task.internal:Checking packages... 2024-03-23T11:18:27.275 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b630d6cd702af63dfefcf906294a092ddce1c609' 2024-03-23T11:18:27.275 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:18:27.275 INFO:teuthology.packaging:ref: None 2024-03-23T11:18:27.275 INFO:teuthology.packaging:tag: None 2024-03-23T11:18:27.275 INFO:teuthology.packaging:branch: rishabh-yuri-RunCephCmd-base-reef-2 2024-03-23T11:18:27.275 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:18:27.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rishabh-yuri-RunCephCmd-base-reef-2 2024-03-23T11:18:27.492 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-820.gb630d6cd 2024-03-23T11:18:27.494 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T11:18:27.499 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T11:18:27.499 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T11:18:27.506 INFO:teuthology.task.internal:Saving configuration 2024-03-23T11:18:27.526 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T11:18:27.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T11:18:27.558 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:12:55.021359', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZ/qoO9C16Im0WCrliXXl/Y2P2XOEe+HAy+M748RPelNmAHfXFhkVNM7bBHuNW7JJ1GoeUNXTomtEQMb0/j7u4='} 2024-03-23T11:18:27.581 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:12:55.022832', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5j5A6YBMOOvWGAY3htlUVPNCZy+Qx5eeo7L3ouY7ywjmsNbxbhHOsxbHaOhwejKBEuJ2x7V4KNcQrCPKQuWY0='} 2024-03-23T11:18:27.582 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T11:18:27.589 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-23T11:18:27.589 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-23T11:18:27.589 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T11:18:27.660 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feccc546160>, signals=[15]) 2024-03-23T11:18:27.660 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T11:18:27.667 INFO:teuthology.task.internal:Opening connections... 2024-03-23T11:18:27.667 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:18:27.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:18:27.749 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-03-23T11:18:27.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:18:27.823 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T11:18:27.830 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-03-23T11:18:27.855 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-03-23T11:18:27.855 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="8" 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="8" 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el8" 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-23T11:18:27.916 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-03-23T11:18:27.917 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-23T11:18:27.917 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-03-23T11:18:27.917 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T11:18:27.917 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-23T11:18:27.917 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T11:18:27.917 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-03-23T11:18:27.937 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-03-23T11:18:27.959 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-03-23T11:18:27.960 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-03-23T11:18:28.020 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-03-23T11:18:28.020 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8" 2024-03-23T11:18:28.020 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-23T11:18:28.021 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T11:18:28.021 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-03-23T11:18:28.045 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T11:18:28.057 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T11:18:28.065 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T11:18:28.065 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-03-23T11:18:28.067 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-03-23T11:18:28.087 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T11:18:28.095 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T11:18:28.095 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-03-23T11:18:28.128 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-03-23T11:18:28.173 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T11:18:28.228 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T11:18:28.244 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-23T11:18:28.244 INFO:teuthology.task.kernel:config {'host.a': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'host.b': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, timeout 300 2024-03-23T11:18:28.244 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T11:18:28.245 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T11:18:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:18:28.265 DEBUG:teuthology.task.kernel:sha1 for host.a is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:18:28.265 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T11:18:28.266 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-23T11:18:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:18:28.267 DEBUG:teuthology.task.kernel:sha1 for host.b is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:18:28.267 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T11:18:28.267 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-23T11:18:28.327 INFO:teuthology.orchestra.run.smithi063.stdout:4.18.0-489.el8.x86_64 2024-03-23T11:18:28.327 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:18:28.327 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T11:18:28.328 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:18:28.332 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-489.el8.x86_64 2024-03-23T11:18:28.332 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:18:28.332 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T11:18:28.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:18:28.478 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum upgrade -y linux-firmware 2024-03-23T11:18:28.552 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum upgrade -y linux-firmware 2024-03-23T11:18:32.799 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - AppStream 17 MB/s | 28 MB 00:01 2024-03-23T11:18:32.943 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:01 2024-03-23T11:18:38.442 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-03-23T11:18:39.202 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-03-23T11:18:40.441 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2024-03-23T11:18:40.959 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.5 kB 00:00 2024-03-23T11:18:41.227 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-03-23T11:18:41.269 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - Extras 51 kB/s | 18 kB 00:00 2024-03-23T11:18:41.944 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 7.5 kB 00:00 2024-03-23T11:18:42.196 INFO:teuthology.orchestra.run.smithi063.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-03-23T11:18:42.669 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-03-23T11:18:46.478 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 436 kB/s | 24 kB 00:00 2024-03-23T11:18:49.583 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:18:49.584 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:18:49.585 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 1 Package 2024-03-23T11:18:49.585 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:18:49.585 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 355 M 2024-03-23T11:18:49.585 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:19:01.628 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 858 kB/s | 16 MB 00:19 2024-03-23T11:19:05.653 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 436 kB/s | 24 kB 00:00 2024-03-23T11:19:08.965 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:19:08.966 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:19:08.966 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-23T11:19:08.966 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 1 Package 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 355 M 2024-03-23T11:19:08.967 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:19:23.051 INFO:teuthology.orchestra.run.smithi111.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 11 MB/s | 355 MB 00:33 2024-03-23T11:19:23.051 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:19:23.051 INFO:teuthology.orchestra.run.smithi111.stdout:Total 11 MB/s | 355 MB 00:33 2024-03-23T11:19:25.151 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:19:25.220 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:19:25.220 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:19:27.407 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:19:27.430 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:19:43.593 INFO:teuthology.orchestra.run.smithi063.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 10 MB/s | 355 MB 00:34 2024-03-23T11:19:43.593 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:19:43.593 INFO:teuthology.orchestra.run.smithi063.stdout:Total 10 MB/s | 355 MB 00:34 2024-03-23T11:19:45.797 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:19:45.856 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:19:45.856 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:19:48.151 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:19:48.176 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:19:50.125 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:19:50.439 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-23T11:19:50.676 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-23T11:19:50.918 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-23T11:19:50.918 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-23T11:19:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-23T11:19:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:19:51.365 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-03-23T11:19:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-03-23T11:19:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:19:51.365 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:19:51.563 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on host.b... 2024-03-23T11:19:51.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:19:51.706 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/8/flavors/default/x86_64/ 2024-03-23T11:19:51.707 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-23T11:19:51.709 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi111.front.sepia.ceph.com to complete... 2024-03-23T11:20:05.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23 11:20:05 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/8/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [401601496/401601496] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T11:20:05.870 INFO:teuthology.orchestra.run.smithi111.stderr:FINISHED --2024-03-23 11:20:05-- 2024-03-23T11:20:05.870 INFO:teuthology.orchestra.run.smithi111.stderr:Total wall clock time: 14s 2024-03-23T11:20:05.870 INFO:teuthology.orchestra.run.smithi111.stderr:Downloaded: 1 files, 383M in 14s (27.8 MB/s) 2024-03-23T11:20:05.873 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on host.b... 2024-03-23T11:20:05.873 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T11:20:07.599 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying... ######################################## 2024-03-23T11:20:07.642 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing... ######################################## 2024-03-23T11:20:07.751 INFO:teuthology.orchestra.run.smithi111.stdout:Updating / installing... 2024-03-23T11:20:12.271 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:20:12.718 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-23T11:20:12.971 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-23T11:20:13.219 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-23T11:20:13.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-23T11:20:13.735 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-23T11:20:13.735 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:20:13.735 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-03-23T11:20:13.735 INFO:teuthology.orchestra.run.smithi063.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-03-23T11:20:13.735 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:20:13.735 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:20:13.935 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on host.a... 2024-03-23T11:20:13.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:20:14.281 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/8/flavors/default/x86_64/ 2024-03-23T11:20:14.281 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-23T11:20:14.283 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi063.front.sepia.ceph.com to complete... 2024-03-23T11:20:28.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23 11:20:28 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/8/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [401601496/401601496] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T11:20:28.506 INFO:teuthology.orchestra.run.smithi063.stderr:FINISHED --2024-03-23 11:20:28-- 2024-03-23T11:20:28.506 INFO:teuthology.orchestra.run.smithi063.stderr:Total wall clock time: 14s 2024-03-23T11:20:28.506 INFO:teuthology.orchestra.run.smithi063.stderr:Downloaded: 1 files, 383M in 14s (27.5 MB/s) 2024-03-23T11:20:28.508 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on host.a... 2024-03-23T11:20:28.508 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T11:20:30.354 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying... ######################################## 2024-03-23T11:20:30.401 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing... ######################################## 2024-03-23T11:20:30.518 INFO:teuthology.orchestra.run.smithi063.stdout:Updating / installing... 2024-03-23T11:20:48.195 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T11:20:49.587 INFO:teuthology.orchestra.run.smithi111.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T11:21:09.723 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T11:21:09.723 DEBUG:teuthology.orchestra.run.smithi111:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T11:21:09.783 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T11:21:09.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T11:21:09.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T11:21:09.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T11:21:09.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T11:21:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T11:21:09.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T11:21:09.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T11:21:09.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T11:21:09.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T11:21:09.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T11:21:09.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T11:21:09.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T11:21:09.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T11:21:09.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T11:21:09.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T11:21:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T11:21:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T11:21:09.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T11:21:09.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T11:21:09.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T11:21:09.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T11:21:09.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T11:21:09.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T11:21:09.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T11:21:09.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T11:21:09.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T11:21:09.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T11:21:09.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T11:21:09.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T11:21:09.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T11:21:09.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T11:21:09.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T11:21:09.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T11:21:09.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T11:21:09.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T11:21:09.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T11:21:09.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T11:21:09.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T11:21:09.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T11:21:09.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T11:21:09.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T11:21:09.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T11:21:09.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T11:21:09.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T11:21:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T11:21:09.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T11:21:09.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T11:21:09.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T11:21:09.832 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T11:21:09.833 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T11:21:09.834 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T11:21:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T11:21:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T11:21:09.837 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T11:21:09.838 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T11:21:09.839 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T11:21:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T11:21:09.841 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T11:21:09.842 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T11:21:09.843 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T11:21:09.844 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T11:21:09.845 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T11:21:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T11:21:09.847 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T11:21:09.848 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T11:21:09.849 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T11:21:09.850 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T11:21:09.851 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T11:21:09.852 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T11:21:09.853 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T11:21:09.854 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T11:21:09.855 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T11:21:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T11:21:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T11:21:09.859 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T11:21:09.860 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T11:21:09.861 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T11:21:09.862 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T11:21:09.863 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T11:21:09.864 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T11:21:09.865 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T11:21:09.866 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T11:21:09.868 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T11:21:09.869 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T11:21:09.872 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T11:21:09.873 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T11:21:09.874 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T11:21:09.875 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T11:21:09.876 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T11:21:09.877 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T11:21:09.878 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T11:21:09.879 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T11:21:09.880 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T11:21:09.881 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T11:21:09.882 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T11:21:09.883 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T11:21:09.884 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T11:21:09.885 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T11:21:09.886 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T11:21:09.887 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T11:21:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T11:21:09.889 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T11:21:09.890 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T11:21:09.891 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T11:21:09.892 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T11:21:09.893 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T11:21:09.894 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T11:21:09.895 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T11:21:09.896 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T11:21:09.897 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T11:21:09.898 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T11:21:09.899 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T11:21:09.900 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T11:21:09.901 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T11:21:09.902 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T11:21:09.903 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T11:21:09.904 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T11:21:09.905 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T11:21:09.906 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T11:21:09.907 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T11:21:09.908 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T11:21:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T11:21:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T11:21:09.911 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T11:21:09.912 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T11:21:09.913 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T11:21:09.914 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T11:21:09.915 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:21:09.919 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:09.920 DEBUG:teuthology.orchestra.run.smithi111:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T11:21:09.944 INFO:teuthology.orchestra.run.smithi111.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:fi 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:for file in vmlinuz System.map config; do 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout: fi 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:done 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:fi 2024-03-23T11:21:09.946 INFO:teuthology.orchestra.run.smithi111.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T11:21:09.946 INFO:teuthology.orchestra.run.smithi111.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T11:21:09.946 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:09.946 INFO:teuthology.orchestra.run.smithi111.stdout:fi 2024-03-23T11:21:09.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-03-23T11:21:10.022 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-03-23T11:21:10.022 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-03-23T11:21:10.022 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.02 2024-03-23T11:21:10.022 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 129.el8 2024-03-23T11:21:10.022 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-03-23T11:21:10.022 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Group : System Environment/Base 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 9265586 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Relocations : (not relocatable) 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : CentOS BuildSystem 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:21:10.023 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T11:21:10.024 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T11:21:10.024 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T11:21:10.024 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-03-23T11:21:10.024 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:21:10.024 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T11:21:10.025 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T11:21:10.025 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:10.025 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T11:21:13.551 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T11:21:15.266 INFO:teuthology.orchestra.run.smithi063.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T11:21:16.197 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-03-23T11:21:18.826 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-03-23T11:21:18.828 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-03-23T11:21:18.856 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.MU5LYh2T2z 2024-03-23T11:21:18.856 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MU5LYh2T2z 2024-03-23T11:21:18.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.MU5LYh2T2z 2024-03-23T11:21:19.090 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.MU5LYh2T2z is 6KB 2024-03-23T11:21:19.139 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.MU5LYh2T2z 2024-03-23T11:21:19.156 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T11:21:19.227 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-23T11:21:19.227 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-23T11:21:19.228 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-23T11:21:19.228 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T11:21:19.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:19.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-03-23T11:21:39.779 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T11:21:39.779 DEBUG:teuthology.orchestra.run.smithi063:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T11:21:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T11:21:39.842 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T11:21:39.843 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T11:21:39.844 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T11:21:39.845 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T11:21:39.846 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T11:21:39.847 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T11:21:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T11:21:39.849 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T11:21:39.850 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T11:21:39.851 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T11:21:39.852 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T11:21:39.853 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T11:21:39.854 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T11:21:39.855 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T11:21:39.856 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T11:21:39.857 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T11:21:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T11:21:39.859 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T11:21:39.860 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T11:21:39.861 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T11:21:39.862 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T11:21:39.863 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T11:21:39.864 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T11:21:39.865 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T11:21:39.866 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T11:21:39.867 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T11:21:39.868 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T11:21:39.869 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T11:21:39.870 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T11:21:39.871 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T11:21:39.872 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T11:21:39.873 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T11:21:39.874 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T11:21:39.875 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T11:21:39.876 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T11:21:39.877 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T11:21:39.878 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T11:21:39.879 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T11:21:39.880 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T11:21:39.881 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T11:21:39.882 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T11:21:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T11:21:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T11:21:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T11:21:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T11:21:39.887 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T11:21:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T11:21:39.889 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T11:21:39.890 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T11:21:39.891 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T11:21:39.892 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T11:21:39.893 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T11:21:39.894 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T11:21:39.895 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T11:21:39.896 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T11:21:39.897 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T11:21:39.898 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T11:21:39.899 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T11:21:39.900 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T11:21:39.901 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T11:21:39.902 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T11:21:39.903 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T11:21:39.904 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T11:21:39.905 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T11:21:39.906 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T11:21:39.907 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T11:21:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T11:21:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T11:21:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T11:21:39.911 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T11:21:39.912 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T11:21:39.913 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T11:21:39.914 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T11:21:39.915 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T11:21:39.916 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T11:21:39.917 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T11:21:39.918 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T11:21:39.919 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T11:21:39.920 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T11:21:39.921 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T11:21:39.922 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T11:21:39.923 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T11:21:39.924 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T11:21:39.925 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T11:21:39.926 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T11:21:39.927 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T11:21:39.928 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T11:21:39.929 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T11:21:39.930 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T11:21:39.931 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T11:21:39.932 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T11:21:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T11:21:39.934 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T11:21:39.935 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T11:21:39.936 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T11:21:39.937 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T11:21:39.938 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T11:21:39.939 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T11:21:39.940 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T11:21:39.941 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T11:21:39.942 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T11:21:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T11:21:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T11:21:39.945 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T11:21:39.946 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T11:21:39.947 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T11:21:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T11:21:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T11:21:39.950 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T11:21:39.951 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T11:21:39.952 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T11:21:39.953 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T11:21:39.954 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T11:21:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T11:21:39.956 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T11:21:39.957 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T11:21:39.958 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T11:21:39.959 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T11:21:39.960 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T11:21:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T11:21:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T11:21:39.963 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T11:21:39.964 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T11:21:39.965 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T11:21:39.966 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T11:21:39.967 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T11:21:39.968 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T11:21:39.969 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T11:21:39.970 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:21:39.973 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:39.973 DEBUG:teuthology.orchestra.run.smithi063:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:fi 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:for file in vmlinuz System.map config; do 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout: fi 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:done 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:21:40.003 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:40.004 INFO:teuthology.orchestra.run.smithi063.stdout:fi 2024-03-23T11:21:40.004 INFO:teuthology.orchestra.run.smithi063.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T11:21:40.004 INFO:teuthology.orchestra.run.smithi063.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T11:21:40.004 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:40.004 INFO:teuthology.orchestra.run.smithi063.stdout:fi 2024-03-23T11:21:40.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-03-23T11:21:40.086 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-03-23T11:21:40.086 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-03-23T11:21:40.086 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.02 2024-03-23T11:21:40.086 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 129.el8 2024-03-23T11:21:40.086 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-03-23T11:21:40.086 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Group : System Environment/Base 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 9265586 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Relocations : (not relocatable) 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : CentOS BuildSystem 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:21:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T11:21:40.088 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T11:21:40.088 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T11:21:40.088 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-03-23T11:21:40.088 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:21:40.088 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T11:21:40.088 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T11:21:40.089 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:40.089 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T11:21:42.446 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-03-23T11:21:44.622 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-03-23T11:21:44.624 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-23T11:21:44.651 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.AmPutgl3E2 2024-03-23T11:21:44.651 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AmPutgl3E2 2024-03-23T11:21:44.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.AmPutgl3E2 2024-03-23T11:21:44.881 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.AmPutgl3E2 is 6KB 2024-03-23T11:21:44.904 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.AmPutgl3E2 2024-03-23T11:21:44.922 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T11:21:44.996 INFO:teuthology.orchestra.run.smithi063.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-23T11:21:44.996 INFO:teuthology.orchestra.run.smithi063.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-23T11:21:44.996 INFO:teuthology.orchestra.run.smithi063.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-23T11:21:44.996 INFO:teuthology.orchestra.run.smithi063.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T11:21:44.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g5ec3e6e88b2e 2024-03-23T11:21:45.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-03-23T11:21:49.390 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-23T11:21:49.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-23T11:21:49.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:22:07.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-23T11:22:15.185 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-23T11:22:15.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-23T11:22:15.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-23T11:22:16.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:22:19.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-23T11:22:31.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-23T11:22:31.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:22:33.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-23T11:22:34.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-23T11:22:42.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-23T11:22:42.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:22:45.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-23T11:22:49.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-23T11:22:49.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:22:57.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-23T11:22:57.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:23:16.130 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-23T11:23:21.889 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-23T11:23:22.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-23T11:23:22.421 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T11:23:22.422 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-23T11:23:22.465 INFO:teuthology.orchestra.run.smithi111.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T11:23:22.465 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T11:23:22.465 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T11:23:22.465 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T11:23:23.466 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-23T11:23:23.466 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T11:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-03-23T11:23:23.597 DEBUG:teuthology.parallel:result is None 2024-03-23T11:23:31.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-23T11:23:31.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:23:31.538 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-23T11:23:31.988 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-23T11:23:31.988 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T11:23:31.988 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-23T11:23:32.031 INFO:teuthology.orchestra.run.smithi063.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T11:23:32.031 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T11:23:32.031 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T11:23:32.031 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T11:23:33.032 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-23T11:23:33.032 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T11:23:33.222 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-03-23T11:23:33.250 DEBUG:teuthology.parallel:result is None 2024-03-23T11:23:33.250 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T11:23:33.261 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T11:23:33.261 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T11:23:33.265 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T11:23:33.314 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T11:23:33.320 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T11:23:33.327 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T11:23:33.327 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T11:23:33.349 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T11:23:33.418 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T11:23:33.425 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T11:23:33.425 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T11:23:33.428 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T11:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:23:33.509 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:23:33.534 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:23:33.537 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T11:23:33.545 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T11:23:33.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T11:23:33.551 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T11:23:33.618 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T11:23:33.627 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T11:23:33.627 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T11:23:33.631 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T11:23:33.675 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:23:33.787 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:23:33.858 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:23:33.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:23:33.997 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:23:33.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T11:23:34.066 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:23:34.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:23:34.215 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:23:34.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:23:34.354 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:23:34.354 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T11:23:34.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-03-23T11:23:34.424 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-03-23T11:23:34.545 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T11:23:34.600 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T11:23:34.938 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T11:23:34.946 INFO:teuthology.task.internal:Starting timer... 2024-03-23T11:23:34.946 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T11:23:34.958 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T11:23:34.966 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-23T11:23:34.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-03-23T11:23:35.111 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-03-23T11:23:35.114 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-03-23T11:23:35.222 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-03-23T11:23:35.225 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T11:23:35.225 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-03-23T11:23:35.292 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-03-23T11:23:35.293 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-03-23T11:23:35.349 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-03-23T11:23:35.349 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive'} 2024-03-23T11:23:35.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T11:23:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:23:35.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T11:23:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:23:35.494 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T11:23:35.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-03-23T11:23:35.560 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-03-23T11:23:35.629 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T11:23:35.639 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T11:23:35.655 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T11:23:35.656 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi111.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-23T11:30:56.555 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com')] 2024-03-23T11:30:56.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-23T11:30:56.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:30:56.648 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-23T11:30:56.766 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-23T11:30:56.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-23T11:30:56.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:30:56.851 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-23T11:30:56.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-23T11:30:56.963 INFO:teuthology.run_tasks:Running task clock... 2024-03-23T11:30:56.973 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-23T11:30:56.973 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T11:30:56.973 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T11:30:56.977 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T11:30:56.977 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T11:30:57.047 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T11:30:57.074 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T11:30:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T11:30:57.105 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T11:30:57.169 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-03-23T11:30:57.193 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-03-23T11:30:57.195 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-03-23T11:30:57.219 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-03-23T11:30:57.227 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T11:30:57.249 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T11:30:57.259 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T11:30:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T11:30:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:bash: ntpq: command not found 2024-03-23T11:30:57.369 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T11:30:57.369 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-23T11:30:57.369 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.369 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.369 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.369 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.371 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2024-03-23T11:30:57.379 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T11:30:57.379 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-23T11:30:57.379 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.379 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.379 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.380 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:30:57.381 INFO:teuthology.run_tasks:Running task pexec... 2024-03-23T11:30:57.390 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-23T11:30:57.390 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:30:57.390 DEBUG:teuthology.orchestra.run.smithi063:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-23T11:30:57.391 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi111.front.sepia.ceph.com 2024-03-23T11:30:57.391 DEBUG:teuthology.orchestra.run.smithi111:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-23T11:30:58.314 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:44 ago on Sat 23 Mar 2024 11:29:14 AM UTC. 2024-03-23T11:30:58.358 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:35 ago on Sat 23 Mar 2024 11:29:23 AM UTC. 2024-03-23T11:31:00.485 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:31:00.486 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:00.486 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-23T11:31:00.486 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:00.486 INFO:teuthology.orchestra.run.smithi111.stdout:Resetting modules: 2024-03-23T11:31:00.486 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools 2024-03-23T11:31:00.486 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:00.487 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:31:00.487 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:00.487 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:00.707 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:31:00.708 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:00.708 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-23T11:31:00.708 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:00.708 INFO:teuthology.orchestra.run.smithi063.stdout:Resetting modules: 2024-03-23T11:31:00.708 INFO:teuthology.orchestra.run.smithi063.stdout: container-tools 2024-03-23T11:31:00.709 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:00.709 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:31:00.709 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:00.709 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:00.761 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:31:00.985 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:31:01.746 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:47 ago on Sat 23 Mar 2024 11:29:14 AM UTC. 2024-03-23T11:31:02.041 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:38 ago on Sat 23 Mar 2024 11:29:23 AM UTC. 2024-03-23T11:31:03.906 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout:Installing group/module packages: 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-23T11:31:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout:Installing module profiles: 2024-03-23T11:31:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools/common 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling module streams: 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools rhel8 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:03.947 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 27 M 2024-03-23T11:31:03.948 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 91 M 2024-03-23T11:31:03.948 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:31:04.371 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:31:04.414 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout:Installing group/module packages: 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-23T11:31:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Installing module profiles: 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: container-tools/common 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling module streams: 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: container-tools rhel8 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-03-23T11:31:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:04.417 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 27 M 2024-03-23T11:31:04.417 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 91 M 2024-03-23T11:31:04.417 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:31:04.819 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 848 kB/s | 689 kB 00:00 2024-03-23T11:31:04.857 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.2 MB/s | 1.0 MB 00:00 2024-03-23T11:31:04.917 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 395 kB/s | 38 kB 00:00 2024-03-23T11:31:04.957 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.5 MB/s | 258 kB 00:00 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.6 MB/s | 180 kB 00:00 2024-03-23T11:31:05.158 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 271 kB/s | 25 kB 00:00 2024-03-23T11:31:05.213 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.1 MB/s | 689 kB 00:00 2024-03-23T11:31:05.264 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.5 MB/s | 1.0 MB 00:00 2024-03-23T11:31:05.291 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): buildah-1.33.6-2.module_el8+976+1136073 7.5 MB/s | 9.6 MB 00:01 2024-03-23T11:31:05.306 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 412 kB/s | 38 kB 00:00 2024-03-23T11:31:05.316 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 592 kB/s | 94 kB 00:00 2024-03-23T11:31:05.364 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.5 MB/s | 258 kB 00:00 2024-03-23T11:31:05.392 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 8.6 MB/s | 4.1 MB 00:00 2024-03-23T11:31:05.456 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.9 MB/s | 180 kB 00:00 2024-03-23T11:31:05.492 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 527 kB/s | 53 kB 00:00 2024-03-23T11:31:05.557 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 248 kB/s | 25 kB 00:00 2024-03-23T11:31:05.657 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 943 kB/s | 94 kB 00:00 2024-03-23T11:31:05.669 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 233 kB/s | 41 kB 00:00 2024-03-23T11:31:05.774 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): buildah-1.33.6-2.module_el8+976+1136073 8.3 MB/s | 9.6 MB 00:01 2024-03-23T11:31:05.908 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 6.8 MB/s | 4.1 MB 00:00 2024-03-23T11:31:05.999 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 576 kB/s | 53 kB 00:00 2024-03-23T11:31:06.013 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 3.6 MB/s | 2.5 MB 00:00 2024-03-23T11:31:06.050 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 9.2 MB/s | 2.5 MB 00:00 2024-03-23T11:31:06.091 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 448 kB/s | 41 kB 00:00 2024-03-23T11:31:06.268 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 9.0 MB/s | 8.8 MB 00:00 2024-03-23T11:31:06.269 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:31:06.269 INFO:teuthology.orchestra.run.smithi111.stdout:Total 12 MB/s | 27 MB 00:02 2024-03-23T11:31:06.642 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 8.9 MB/s | 8.8 MB 00:00 2024-03-23T11:31:06.642 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:31:06.643 INFO:teuthology.orchestra.run.smithi063.stdout:Total 12 MB/s | 27 MB 00:02 2024-03-23T11:31:06.669 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:31:06.694 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:31:06.694 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:31:06.972 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:31:06.976 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:31:07.068 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:31:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:31:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:31:07.680 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:31:08.079 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:31:08.698 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-23T11:31:08.912 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-23T11:31:09.072 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-23T11:31:09.127 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-23T11:31:09.267 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-23T11:31:09.308 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-23T11:31:09.458 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-23T11:31:09.528 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-23T11:31:09.708 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-23T11:31:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-23T11:31:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-23T11:31:10.580 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-23T11:31:11.190 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-23T11:31:11.587 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-23T11:31:11.660 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-23T11:31:11.834 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-23T11:31:11.896 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-23T11:31:12.122 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-23T11:31:12.232 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-23T11:31:12.472 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-23T11:31:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-23T11:31:12.712 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-23T11:31:12.771 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-23T11:31:12.894 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-23T11:31:12.993 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-23T11:31:13.171 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-23T11:31:13.538 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-23T11:31:13.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-23T11:31:13.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-23T11:31:13.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-23T11:31:13.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-23T11:31:13.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-23T11:31:13.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-23T11:31:13.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-23T11:31:13.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-23T11:31:13.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-23T11:31:13.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-23T11:31:13.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-23T11:31:13.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-23T11:31:13.858 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-23T11:31:13.859 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-23T11:31:13.859 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-23T11:31:13.859 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-23T11:31:13.859 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-23T11:31:13.859 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-23T11:31:14.113 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-23T11:31:14.114 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-23T11:31:14.114 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-23T11:31:14.114 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-23T11:31:14.114 INFO:teuthology.orchestra.run.smithi111.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-23T11:31:14.114 INFO:teuthology.orchestra.run.smithi111.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-23T11:31:14.114 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:14.114 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:31:14.338 DEBUG:teuthology.parallel:result is None 2024-03-23T11:31:14.364 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-23T11:31:14.365 INFO:teuthology.orchestra.run.smithi063.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-23T11:31:14.366 INFO:teuthology.orchestra.run.smithi063.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-23T11:31:14.366 INFO:teuthology.orchestra.run.smithi063.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-23T11:31:14.366 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:14.366 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:31:14.597 DEBUG:teuthology.parallel:result is None 2024-03-23T11:31:14.597 INFO:teuthology.run_tasks:Running task install... 2024-03-23T11:31:14.617 DEBUG:teuthology.task.install:project ceph 2024-03-23T11:31:14.617 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'}} 2024-03-23T11:31:14.617 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'} 2024-03-23T11:31:14.618 INFO:teuthology.task.install:Using flavor: default 2024-03-23T11:31:14.629 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-23T11:31:14.630 INFO:teuthology.task.install:extra packages: [] 2024-03-23T11:31:14.630 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609', 'tag': None, 'wait_for_package': False} 2024-03-23T11:31:14.631 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:31:14.631 INFO:teuthology.packaging:ref: None 2024-03-23T11:31:14.631 INFO:teuthology.packaging:tag: None 2024-03-23T11:31:14.631 INFO:teuthology.packaging:branch: quincy 2024-03-23T11:31:14.631 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:31:14.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-23T11:31:14.634 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609', 'tag': None, 'wait_for_package': False} 2024-03-23T11:31:14.634 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:31:14.634 INFO:teuthology.packaging:ref: None 2024-03-23T11:31:14.634 INFO:teuthology.packaging:tag: None 2024-03-23T11:31:14.634 INFO:teuthology.packaging:branch: quincy 2024-03-23T11:31:14.634 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:31:14.635 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-23T11:31:14.888 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/ 2024-03-23T11:31:14.889 INFO:teuthology.task.install.rpm:Package version is 17.2.7-726.g9c4e0c34 2024-03-23T11:31:14.925 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/ 2024-03-23T11:31:14.926 INFO:teuthology.task.install.rpm:Package version is 17.2.7-726.g9c4e0c34 2024-03-23T11:31:15.059 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T11:31:15.059 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:31:15.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T11:31:15.061 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T11:31:15.061 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:31:15.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T11:31:15.128 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-23T11:31:15.128 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:31:15.128 INFO:teuthology.packaging:ref: None 2024-03-23T11:31:15.128 INFO:teuthology.packaging:tag: None 2024-03-23T11:31:15.128 INFO:teuthology.packaging:branch: quincy 2024-03-23T11:31:15.128 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:31:15.128 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T11:31:15.129 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-23T11:31:15.129 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:31:15.130 INFO:teuthology.packaging:ref: None 2024-03-23T11:31:15.130 INFO:teuthology.packaging:tag: None 2024-03-23T11:31:15.130 INFO:teuthology.packaging:branch: quincy 2024-03-23T11:31:15.130 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:31:15.130 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T11:31:15.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T11:31:15.237 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T11:31:15.328 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T11:31:15.329 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T11:31:15.394 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-03-23T11:31:15.395 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-03-23T11:31:15.397 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-03-23T11:31:15.398 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-03-23T11:31:15.760 INFO:teuthology.orchestra.run.smithi111.stdout:76 files removed 2024-03-23T11:31:15.765 INFO:teuthology.orchestra.run.smithi063.stdout:76 files removed 2024-03-23T11:31:15.797 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2024-03-23T11:31:15.802 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-03-23T11:31:17.096 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 103 kB/s | 78 kB 00:00 2024-03-23T11:31:17.153 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 100 kB/s | 78 kB 00:00 2024-03-23T11:31:17.739 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 15 kB/s | 9.1 kB 00:00 2024-03-23T11:31:17.787 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 15 kB/s | 9.1 kB 00:00 2024-03-23T11:31:18.398 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2024-03-23T11:31:18.438 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2024-03-23T11:31:20.289 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - AppStream 15 MB/s | 28 MB 00:01 2024-03-23T11:31:21.263 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - AppStream 9.9 MB/s | 28 MB 00:02 2024-03-23T11:31:26.415 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-03-23T11:31:27.638 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-03-23T11:31:28.523 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-03-23T11:31:29.075 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 7.5 kB 00:00 2024-03-23T11:31:29.797 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2024-03-23T11:31:30.317 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 7.5 kB 00:00 2024-03-23T11:31:41.034 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 8 - PowerTools 473 kB/s | 4.9 MB 00:10 2024-03-23T11:31:42.402 INFO:teuthology.orchestra.run.smithi063.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-03-23T11:31:43.619 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-03-23T11:31:45.592 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - PowerTools 306 kB/s | 4.9 MB 00:16 2024-03-23T11:31:46.751 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-03-23T11:31:47.552 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2024-03-23T11:31:48.353 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 452 kB/s | 24 kB 00:00 2024-03-23T11:31:52.019 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 484 kB/s | 24 kB 00:00 2024-03-23T11:31:52.305 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:31:52.308 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:52.308 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 17 M 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.5 M 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.6 M 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 5.2 M 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 23 M 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 24 k 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-23T11:31:52.309 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 798 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 506 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 5.6 M 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 45 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 117 k 2024-03-23T11:31:52.310 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 190 k 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 370 k 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 358 k 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 106 k 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout:Install 21 Packages 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-03-23T11:31:52.311 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:31:52.312 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 64 M 2024-03-23T11:31:52.312 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:31:53.213 INFO:teuthology.orchestra.run.smithi063.stdout:(1/23): ceph-base-17.2.7-726.g9c4e0c34.el8.x86_ 6.8 MB/s | 5.2 MB 00:00 2024-03-23T11:31:53.288 INFO:teuthology.orchestra.run.smithi063.stdout:(2/23): ceph-selinux-17.2.7-726.g9c4e0c34.el8.x 326 kB/s | 24 kB 00:00 2024-03-23T11:31:53.421 INFO:teuthology.orchestra.run.smithi063.stdout:(3/23): libcephfs2-17.2.7-726.g9c4e0c34.el8.x86 5.9 MB/s | 798 kB 00:00 2024-03-23T11:31:53.522 INFO:teuthology.orchestra.run.smithi063.stdout:(4/23): libradosstriper1-17.2.7-726.g9c4e0c34.e 4.9 MB/s | 506 kB 00:00 2024-03-23T11:31:54.222 INFO:teuthology.orchestra.run.smithi063.stdout:(5/23): librgw2-17.2.7-726.g9c4e0c34.el8.x86_64 8.0 MB/s | 5.6 MB 00:00 2024-03-23T11:31:54.314 INFO:teuthology.orchestra.run.smithi063.stdout:(6/23): python3-ceph-argparse-17.2.7-726.g9c4e0 488 kB/s | 45 kB 00:00 2024-03-23T11:31:54.548 INFO:teuthology.orchestra.run.smithi063.stdout:(7/23): ceph-common-17.2.7-726.g9c4e0c34.el8.x8 11 MB/s | 23 MB 00:02 2024-03-23T11:31:54.573 INFO:teuthology.orchestra.run.smithi063.stdout:(8/23): python3-ceph-common-17.2.7-726.g9c4e0c3 452 kB/s | 117 kB 00:00 2024-03-23T11:31:55.760 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:31:55.762 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:55.762 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 17 M 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.5 M 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.6 M 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 5.2 M 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 23 M 2024-03-23T11:31:55.763 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 24 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 798 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 506 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 5.6 M 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-23T11:31:55.764 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 45 k 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 117 k 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 190 k 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 370 k 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 358 k 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 106 k 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout:Install 21 Packages 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-03-23T11:31:55.765 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:31:55.766 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 64 M 2024-03-23T11:31:55.766 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:31:56.688 INFO:teuthology.orchestra.run.smithi111.stdout:(1/23): ceph-base-17.2.7-726.g9c4e0c34.el8.x86_ 6.8 MB/s | 5.2 MB 00:00 2024-03-23T11:31:56.745 INFO:teuthology.orchestra.run.smithi111.stdout:(2/23): ceph-selinux-17.2.7-726.g9c4e0c34.el8.x 428 kB/s | 24 kB 00:00 2024-03-23T11:31:56.854 INFO:teuthology.orchestra.run.smithi111.stdout:(3/23): libcephfs2-17.2.7-726.g9c4e0c34.el8.x86 7.2 MB/s | 798 kB 00:00 2024-03-23T11:31:56.921 INFO:teuthology.orchestra.run.smithi111.stdout:(4/23): libradosstriper1-17.2.7-726.g9c4e0c34.e 7.4 MB/s | 506 kB 00:00 2024-03-23T11:31:57.354 INFO:teuthology.orchestra.run.smithi111.stdout:(5/23): librgw2-17.2.7-726.g9c4e0c34.el8.x86_64 13 MB/s | 5.6 MB 00:00 2024-03-23T11:31:57.413 INFO:teuthology.orchestra.run.smithi111.stdout:(6/23): python3-ceph-argparse-17.2.7-726.g9c4e0 770 kB/s | 45 kB 00:00 2024-03-23T11:31:57.480 INFO:teuthology.orchestra.run.smithi111.stdout:(7/23): python3-ceph-common-17.2.7-726.g9c4e0c3 1.7 MB/s | 117 kB 00:00 2024-03-23T11:31:57.539 INFO:teuthology.orchestra.run.smithi111.stdout:(8/23): python3-cephfs-17.2.7-726.g9c4e0c34.el8 3.2 MB/s | 190 kB 00:00 2024-03-23T11:31:57.622 INFO:teuthology.orchestra.run.smithi111.stdout:(9/23): python3-rados-17.2.7-726.g9c4e0c34.el8. 4.3 MB/s | 370 kB 00:00 2024-03-23T11:31:57.697 INFO:teuthology.orchestra.run.smithi111.stdout:(10/23): python3-rbd-17.2.7-726.g9c4e0c34.el8.x 4.7 MB/s | 358 kB 00:00 2024-03-23T11:31:57.764 INFO:teuthology.orchestra.run.smithi111.stdout:(11/23): python3-rgw-17.2.7-726.g9c4e0c34.el8.x 1.6 MB/s | 106 kB 00:00 2024-03-23T11:31:58.340 INFO:teuthology.orchestra.run.smithi111.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.1 MB/s | 677 kB 00:00 2024-03-23T11:31:58.440 INFO:teuthology.orchestra.run.smithi111.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2024-03-23T11:31:58.540 INFO:teuthology.orchestra.run.smithi111.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 439 kB/s | 44 kB 00:00 2024-03-23T11:31:58.616 INFO:teuthology.orchestra.run.smithi111.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-03-23T11:31:59.216 INFO:teuthology.orchestra.run.smithi111.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2024-03-23T11:31:59.250 INFO:teuthology.orchestra.run.smithi111.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.7 MB/s | 59 kB 00:00 2024-03-23T11:31:59.275 INFO:teuthology.orchestra.run.smithi111.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.9 MB/s | 75 kB 00:00 2024-03-23T11:31:59.350 INFO:teuthology.orchestra.run.smithi111.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 23 MB/s | 1.7 MB 00:00 2024-03-23T11:32:01.140 INFO:teuthology.orchestra.run.smithi063.stdout:(9/23): python3-cephfs-17.2.7-726.g9c4e0c34.el8 29 kB/s | 190 kB 00:06 2024-03-23T11:32:01.164 INFO:teuthology.orchestra.run.smithi063.stdout:(10/23): python3-rados-17.2.7-726.g9c4e0c34.el8 56 kB/s | 370 kB 00:06 2024-03-23T11:32:01.273 INFO:teuthology.orchestra.run.smithi063.stdout:(11/23): python3-rbd-17.2.7-726.g9c4e0c34.el8.x 2.6 MB/s | 358 kB 00:00 2024-03-23T11:32:01.298 INFO:teuthology.orchestra.run.smithi063.stdout:(12/23): python3-rgw-17.2.7-726.g9c4e0c34.el8.x 797 kB/s | 106 kB 00:00 2024-03-23T11:32:01.467 INFO:teuthology.orchestra.run.smithi111.stdout:(20/23): librados2-17.2.7-726.g9c4e0c34.el8.x86 1.6 MB/s | 3.5 MB 00:02 2024-03-23T11:32:01.824 INFO:teuthology.orchestra.run.smithi063.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 495 kB/s | 259 kB 00:00 2024-03-23T11:32:01.850 INFO:teuthology.orchestra.run.smithi111.stdout:(21/23): librbd1-17.2.7-726.g9c4e0c34.el8.x86_6 9.4 MB/s | 3.6 MB 00:00 2024-03-23T11:32:01.991 INFO:teuthology.orchestra.run.smithi063.stdout:(14/23): ceph-radosgw-17.2.7-726.g9c4e0c34.el8. 1.8 MB/s | 17 MB 00:09 2024-03-23T11:32:02.016 INFO:teuthology.orchestra.run.smithi063.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 229 kB/s | 44 kB 00:00 2024-03-23T11:32:02.074 INFO:teuthology.orchestra.run.smithi063.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-03-23T11:32:02.250 INFO:teuthology.orchestra.run.smithi063.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 693 kB/s | 677 kB 00:00 2024-03-23T11:32:02.267 INFO:teuthology.orchestra.run.smithi111.stdout:(22/23): ceph-radosgw-17.2.7-726.g9c4e0c34.el8. 2.7 MB/s | 17 MB 00:06 2024-03-23T11:32:02.467 INFO:teuthology.orchestra.run.smithi063.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 150 kB/s | 59 kB 00:00 2024-03-23T11:32:02.625 INFO:teuthology.orchestra.run.smithi063.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 200 kB/s | 75 kB 00:00 2024-03-23T11:32:02.651 INFO:teuthology.orchestra.run.smithi063.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 483 kB/s | 306 kB 00:00 2024-03-23T11:32:02.709 INFO:teuthology.orchestra.run.smithi111.stdout:(23/23): ceph-common-17.2.7-726.g9c4e0c34.el8.x 3.5 MB/s | 23 MB 00:06 2024-03-23T11:32:02.713 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:32:02.713 INFO:teuthology.orchestra.run.smithi111.stdout:Total 9.2 MB/s | 64 MB 00:06 2024-03-23T11:32:02.804 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:32:02.855 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:32:02.855 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:32:02.934 INFO:teuthology.orchestra.run.smithi063.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 3.7 MB/s | 1.7 MB 00:00 2024-03-23T11:32:03.259 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:32:03.262 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:32:03.350 INFO:teuthology.orchestra.run.smithi063.stdout:(22/23): librbd1-17.2.7-726.g9c4e0c34.el8.x86_6 5.1 MB/s | 3.6 MB 00:00 2024-03-23T11:32:03.535 INFO:teuthology.orchestra.run.smithi063.stdout:(23/23): librados2-17.2.7-726.g9c4e0c34.el8.x86 3.8 MB/s | 3.5 MB 00:00 2024-03-23T11:32:03.537 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:32:03.537 INFO:teuthology.orchestra.run.smithi063.stdout:Total 5.7 MB/s | 64 MB 00:11 2024-03-23T11:32:03.634 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:32:03.690 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:32:03.690 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:32:04.088 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:32:04.092 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:32:04.305 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:32:04.510 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-23T11:32:04.736 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-23T11:32:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:32:05.164 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-23T11:32:05.330 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-23T11:32:05.369 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/25 2024-03-23T11:32:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/25 2024-03-23T11:32:05.539 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-23T11:32:05.749 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 4/25 2024-03-23T11:32:05.944 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/25 2024-03-23T11:32:05.994 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-23T11:32:06.180 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/25 2024-03-23T11:32:06.298 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/25 2024-03-23T11:32:06.383 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/25 2024-03-23T11:32:06.556 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 4/25 2024-03-23T11:32:06.586 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/25 2024-03-23T11:32:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/25 2024-03-23T11:32:06.826 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/25 2024-03-23T11:32:07.048 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-23T11:32:07.189 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/25 2024-03-23T11:32:07.242 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-23T11:32:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/25 2024-03-23T11:32:07.629 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/25 2024-03-23T11:32:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-23T11:32:07.945 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-23T11:32:08.007 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-23T11:32:08.360 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 10/25 2024-03-23T11:32:08.446 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 10/25 2024-03-23T11:32:08.589 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el8.x 11/25 2024-03-23T11:32:08.734 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-23T11:32:08.794 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 12/25 2024-03-23T11:32:08.888 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 10/25 2024-03-23T11:32:08.971 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 10/25 2024-03-23T11:32:08.998 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 13/25 2024-03-23T11:32:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el8.x 11/25 2024-03-23T11:32:09.216 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 14/25 2024-03-23T11:32:09.303 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 12/25 2024-03-23T11:32:09.361 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 15/25 2024-03-23T11:32:09.437 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 15/25 2024-03-23T11:32:09.528 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 13/25 2024-03-23T11:32:09.716 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-23T11:32:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 14/25 2024-03-23T11:32:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 15/25 2024-03-23T11:32:09.920 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-23T11:32:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 15/25 2024-03-23T11:32:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-23T11:32:10.204 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-23T11:32:10.336 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86 19/25 2024-03-23T11:32:10.382 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-23T11:32:10.575 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-23T11:32:10.789 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86 19/25 2024-03-23T11:32:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 20/25 2024-03-23T11:32:13.452 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 20/25 2024-03-23T11:32:13.905 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 20/25 2024-03-23T11:32:14.088 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 20/25 2024-03-23T11:32:14.121 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 20/25 2024-03-23T11:32:14.413 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 21/25 2024-03-23T11:32:14.610 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 21/25 2024-03-23T11:32:14.793 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 20/25 2024-03-23T11:32:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 22/25 2024-03-23T11:32:15.030 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 21/25 2024-03-23T11:32:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 21/25 2024-03-23T11:32:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 22/25 2024-03-23T11:32:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 22/25 2024-03-23T11:32:35.861 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-03-23T11:32:35.861 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-03-23T11:32:35.861 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2024-03-23T11:32:35.861 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /run 2024-03-23T11:32:35.861 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-03-23T11:32:35.861 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-03-23T11:32:35.861 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-03-23T11:32:35.862 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-03-23T11:32:35.862 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2024-03-23T11:32:35.862 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 22/25 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /dev 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /run 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /dev 2024-03-23T11:32:37.220 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:32:37.710 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 23/25 2024-03-23T11:32:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 23/25 2024-03-23T11:32:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-23T11:32:38.069 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-23T11:32:38.069 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-23T11:32:39.058 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 4/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 7/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el8.x 8/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86 9/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 10/25 2024-03-23T11:32:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 11/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 12/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 13/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 22/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-23T11:32:39.060 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 24/25 2024-03-23T11:32:39.127 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 23/25 2024-03-23T11:32:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 23/25 2024-03-23T11:32:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-23T11:32:39.581 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-23T11:32:39.582 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.737 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.738 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:39.739 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:32:40.105 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2024-03-23T11:32:40.564 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 4/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 7/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el8.x 8/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86 9/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 10/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 11/25 2024-03-23T11:32:40.565 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 12/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 13/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 22/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-23T11:32:40.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 24/25 2024-03-23T11:32:41.028 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:50 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.482 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.483 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.484 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:41.484 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-23T11:32:41.484 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:32:41.484 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:32:41.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-03-23T11:32:42.688 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 52 M 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 179 k 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-23T11:32:42.690 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-23T11:32:42.691 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:42.691 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:32:42.691 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:32:42.691 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-23T11:32:42.691 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:42.691 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 53 M 2024-03-23T11:32:42.691 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 234 M 2024-03-23T11:32:42.692 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:32:42.820 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:54 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:32:43.352 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 433 kB/s | 203 kB 00:00 2024-03-23T11:32:43.452 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-03-23T11:32:43.552 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.1 MB/s | 323 kB 00:00 2024-03-23T11:32:43.653 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 706 kB/s | 71 kB 00:00 2024-03-23T11:32:43.677 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): libcephsqlite-17.2.7-726.g9c4e0c34.el8.x 225 kB/s | 179 kB 00:00 2024-03-23T11:32:44.627 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:32:44.628 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:32:44.628 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:32:44.628 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 52 M 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 179 k 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-03-23T11:32:44.629 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:32:44.630 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 53 M 2024-03-23T11:32:44.630 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 234 M 2024-03-23T11:32:44.630 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:32:45.128 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 458 kB/s | 203 kB 00:00 2024-03-23T11:32:45.203 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): libcephsqlite-17.2.7-726.g9c4e0c34.el8.x 345 kB/s | 179 kB 00:00 2024-03-23T11:32:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-03-23T11:32:45.387 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 771 kB/s | 71 kB 00:00 2024-03-23T11:32:45.754 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 587 kB/s | 323 kB 00:00 2024-03-23T11:32:47.961 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-test-17.2.7-726.g9c4e0c34.el8.x86_6 10 MB/s | 52 MB 00:05 2024-03-23T11:32:47.963 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:32:47.963 INFO:teuthology.orchestra.run.smithi111.stdout:Total 10 MB/s | 53 MB 00:05 2024-03-23T11:32:48.053 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:32:48.078 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:32:48.078 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:32:48.396 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:32:48.400 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:32:48.938 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:32:49.071 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): ceph-test-17.2.7-726.g9c4e0c34.el8.x86_6 12 MB/s | 52 MB 00:04 2024-03-23T11:32:49.073 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:32:49.073 INFO:teuthology.orchestra.run.smithi063.stdout:Total 12 MB/s | 53 MB 00:04 2024-03-23T11:32:49.140 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-23T11:32:49.165 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:32:49.191 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:32:49.191 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:32:49.325 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-23T11:32:49.487 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-23T11:32:49.526 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:32:49.530 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:32:49.585 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-23T11:32:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-23T11:32:49.979 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/6 2024-03-23T11:32:50.048 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:32:50.260 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-23T11:32:50.431 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-23T11:32:50.574 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-23T11:32:50.671 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-23T11:32:50.852 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-23T11:32:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/6 2024-03-23T11:32:55.293 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/6 2024-03-23T11:32:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/6 2024-03-23T11:32:55.904 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/6 2024-03-23T11:32:55.904 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/6 2024-03-23T11:32:55.904 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/6 2024-03-23T11:32:55.904 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-23T11:32:55.904 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-23T11:32:55.905 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-23T11:32:56.377 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-23T11:32:56.377 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:56.377 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:32:56.377 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:56.377 INFO:teuthology.orchestra.run.smithi111.stdout: jq-1.6-8.el8.x86_64 2024-03-23T11:32:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-23T11:32:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-23T11:32:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-23T11:32:56.378 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:56.378 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:32:56.561 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2024-03-23T11:32:56.759 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/6 2024-03-23T11:32:56.965 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/6 2024-03-23T11:32:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 6/6 2024-03-23T11:32:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/6 2024-03-23T11:32:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/6 2024-03-23T11:32:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-23T11:32:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-23T11:32:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-23T11:32:57.512 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:06 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:32:57.808 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: jq-1.6-8.el8.x86_64 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:32:57.809 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:32:58.001 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-03-23T11:32:58.967 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:10 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:32:59.192 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 6.4 k 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 2.3 M 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 1.6 M 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 240 k 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.9 M 2024-03-23T11:32:59.196 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 17 M 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-23T11:32:59.197 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-23T11:32:59.198 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-23T11:32:59.199 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:59.199 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:32:59.199 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:32:59.199 INFO:teuthology.orchestra.run.smithi111.stdout:Install 32 Packages 2024-03-23T11:32:59.199 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:32:59.199 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 28 M 2024-03-23T11:32:59.200 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 110 M 2024-03-23T11:32:59.200 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:32:59.964 INFO:teuthology.orchestra.run.smithi111.stdout:(1/32): ceph-17.2.7-726.g9c4e0c34.el8.x86_64.rp 32 kB/s | 6.4 kB 00:00 2024-03-23T11:33:00.106 INFO:teuthology.orchestra.run.smithi111.stdout:(2/32): ceph-mds-17.2.7-726.g9c4e0c34.el8.x86_6 6.8 MB/s | 2.3 MB 00:00 2024-03-23T11:33:00.473 INFO:teuthology.orchestra.run.smithi111.stdout:(3/32): ceph-mgr-17.2.7-726.g9c4e0c34.el8.x86_6 2.2 MB/s | 1.6 MB 00:00 2024-03-23T11:33:00.573 INFO:teuthology.orchestra.run.smithi111.stdout:(4/32): ceph-mgr-modules-core-17.2.7-726.g9c4e0 2.3 MB/s | 240 kB 00:00 2024-03-23T11:33:00.779 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 6.4 k 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 2.3 M 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 1.6 M 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 240 k 2024-03-23T11:33:00.783 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.9 M 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 17 M 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-23T11:33:00.784 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:00.785 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:33:00.786 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:00.786 INFO:teuthology.orchestra.run.smithi063.stdout:Install 32 Packages 2024-03-23T11:33:00.786 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:00.786 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 28 M 2024-03-23T11:33:00.786 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 110 M 2024-03-23T11:33:00.786 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:33:00.957 INFO:teuthology.orchestra.run.smithi111.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 410 kB/s | 157 kB 00:00 2024-03-23T11:33:01.057 INFO:teuthology.orchestra.run.smithi111.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 388 kB/s | 39 kB 00:00 2024-03-23T11:33:01.124 INFO:teuthology.orchestra.run.smithi111.stdout:(7/32): ceph-mon-17.2.7-726.g9c4e0c34.el8.x86_6 3.4 MB/s | 3.9 MB 00:01 2024-03-23T11:33:01.149 INFO:teuthology.orchestra.run.smithi111.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.1 MB/s | 103 kB 00:00 2024-03-23T11:33:01.237 INFO:teuthology.orchestra.run.smithi063.stdout:(1/32): ceph-17.2.7-726.g9c4e0c34.el8.x86_64.rp 31 kB/s | 6.4 kB 00:00 2024-03-23T11:33:01.341 INFO:teuthology.orchestra.run.smithi111.stdout:(9/32): ceph-osd-17.2.7-726.g9c4e0c34.el8.x86_6 14 MB/s | 17 MB 00:01 2024-03-23T11:33:01.383 INFO:teuthology.orchestra.run.smithi111.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.9 MB/s | 457 kB 00:00 2024-03-23T11:33:01.413 INFO:teuthology.orchestra.run.smithi063.stdout:(2/32): ceph-mds-17.2.7-726.g9c4e0c34.el8.x86_6 6.1 MB/s | 2.3 MB 00:00 2024-03-23T11:33:01.455 INFO:teuthology.orchestra.run.smithi063.stdout:(3/32): ceph-mgr-17.2.7-726.g9c4e0c34.el8.x86_6 3.7 MB/s | 1.6 MB 00:00 2024-03-23T11:33:01.475 INFO:teuthology.orchestra.run.smithi111.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.8 MB/s | 249 kB 00:00 2024-03-23T11:33:01.534 INFO:teuthology.orchestra.run.smithi111.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 4.2 MB/s | 641 kB 00:00 2024-03-23T11:33:01.538 INFO:teuthology.orchestra.run.smithi063.stdout:(4/32): ceph-mgr-modules-core-17.2.7-726.g9c4e0 2.8 MB/s | 240 kB 00:00 2024-03-23T11:33:01.559 INFO:teuthology.orchestra.run.smithi111.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-03-23T11:33:01.584 INFO:teuthology.orchestra.run.smithi111.stdout:(14/32): python3-pytz-2017.2-11.el8.noarch.rpm 124 kB/s | 57 kB 00:00 2024-03-23T11:33:01.768 INFO:teuthology.orchestra.run.smithi111.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 186 kB/s | 44 kB 00:00 2024-03-23T11:33:01.869 INFO:teuthology.orchestra.run.smithi111.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 599 kB/s | 185 kB 00:00 2024-03-23T11:33:01.893 INFO:teuthology.orchestra.run.smithi111.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 559 kB/s | 173 kB 00:00 2024-03-23T11:33:01.944 INFO:teuthology.orchestra.run.smithi111.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2024-03-23T11:33:01.969 INFO:teuthology.orchestra.run.smithi111.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-03-23T11:33:01.994 INFO:teuthology.orchestra.run.smithi111.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 174 kB/s | 18 kB 00:00 2024-03-23T11:33:02.020 INFO:teuthology.orchestra.run.smithi111.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2024-03-23T11:33:02.020 INFO:teuthology.orchestra.run.smithi063.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 326 kB/s | 157 kB 00:00 2024-03-23T11:33:02.045 INFO:teuthology.orchestra.run.smithi111.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 779 kB/s | 59 kB 00:00 2024-03-23T11:33:02.087 INFO:teuthology.orchestra.run.smithi111.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-03-23T11:33:02.112 INFO:teuthology.orchestra.run.smithi111.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-03-23T11:33:02.137 INFO:teuthology.orchestra.run.smithi111.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-03-23T11:33:02.163 INFO:teuthology.orchestra.run.smithi111.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 316 kB/s | 24 kB 00:00 2024-03-23T11:33:02.204 INFO:teuthology.orchestra.run.smithi111.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 440 kB/s | 29 kB 00:00 2024-03-23T11:33:02.206 INFO:teuthology.orchestra.run.smithi063.stdout:(6/32): ceph-osd-17.2.7-726.g9c4e0c34.el8.x86_6 21 MB/s | 17 MB 00:00 2024-03-23T11:33:02.222 INFO:teuthology.orchestra.run.smithi063.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 192 kB/s | 39 kB 00:00 2024-03-23T11:33:02.230 INFO:teuthology.orchestra.run.smithi111.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 300 kB/s | 27 kB 00:00 2024-03-23T11:33:02.255 INFO:teuthology.orchestra.run.smithi111.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-03-23T11:33:02.305 INFO:teuthology.orchestra.run.smithi111.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-03-23T11:33:02.323 INFO:teuthology.orchestra.run.smithi063.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 569 kB/s | 57 kB 00:00 2024-03-23T11:33:02.330 INFO:teuthology.orchestra.run.smithi111.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-03-23T11:33:02.355 INFO:teuthology.orchestra.run.smithi111.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.6 MB/s | 251 kB 00:00 2024-03-23T11:33:02.357 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:02.357 INFO:teuthology.orchestra.run.smithi111.stdout:Total 9.0 MB/s | 28 MB 00:03 2024-03-23T11:33:02.523 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:33:02.564 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:33:02.564 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:33:02.615 INFO:teuthology.orchestra.run.smithi063.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 1.5 MB/s | 457 kB 00:00 2024-03-23T11:33:02.640 INFO:teuthology.orchestra.run.smithi063.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 238 kB/s | 103 kB 00:00 2024-03-23T11:33:02.724 INFO:teuthology.orchestra.run.smithi063.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2024-03-23T11:33:02.766 INFO:teuthology.orchestra.run.smithi063.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 2.6 MB/s | 109 kB 00:00 2024-03-23T11:33:02.792 INFO:teuthology.orchestra.run.smithi063.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 4.2 MB/s | 641 kB 00:00 2024-03-23T11:33:02.899 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:33:02.903 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:33:02.958 INFO:teuthology.orchestra.run.smithi063.stdout:(14/32): ceph-mon-17.2.7-726.g9c4e0c34.el8.x86_ 2.3 MB/s | 3.9 MB 00:01 2024-03-23T11:33:03.175 INFO:teuthology.orchestra.run.smithi063.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 107 kB/s | 44 kB 00:00 2024-03-23T11:33:03.334 INFO:teuthology.orchestra.run.smithi063.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 342 kB/s | 185 kB 00:00 2024-03-23T11:33:03.434 INFO:teuthology.orchestra.run.smithi063.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-03-23T11:33:03.484 INFO:teuthology.orchestra.run.smithi063.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 329 kB/s | 173 kB 00:00 2024-03-23T11:33:03.534 INFO:teuthology.orchestra.run.smithi063.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-03-23T11:33:03.560 INFO:teuthology.orchestra.run.smithi063.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 140 kB/s | 18 kB 00:00 2024-03-23T11:33:03.593 INFO:teuthology.orchestra.run.smithi063.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2024-03-23T11:33:03.636 INFO:teuthology.orchestra.run.smithi063.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 585 kB/s | 59 kB 00:00 2024-03-23T11:33:03.694 INFO:teuthology.orchestra.run.smithi063.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2024-03-23T11:33:03.736 INFO:teuthology.orchestra.run.smithi063.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2024-03-23T11:33:03.761 INFO:teuthology.orchestra.run.smithi063.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-03-23T11:33:03.811 INFO:teuthology.orchestra.run.smithi063.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2024-03-23T11:33:03.837 INFO:teuthology.orchestra.run.smithi063.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-03-23T11:33:03.862 INFO:teuthology.orchestra.run.smithi063.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-03-23T11:33:03.970 INFO:teuthology.orchestra.run.smithi063.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 787 kB/s | 85 kB 00:00 2024-03-23T11:33:03.996 INFO:teuthology.orchestra.run.smithi063.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-03-23T11:33:04.021 INFO:teuthology.orchestra.run.smithi063.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.2 MB/s | 255 kB 00:00 2024-03-23T11:33:04.045 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:33:04.071 INFO:teuthology.orchestra.run.smithi063.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-03-23T11:33:04.075 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:04.075 INFO:teuthology.orchestra.run.smithi063.stdout:Total 8.6 MB/s | 28 MB 00:03 2024-03-23T11:33:04.253 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:33:04.292 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-23T11:33:04.298 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:33:04.298 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:33:04.576 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-23T11:33:04.641 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:33:04.644 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:33:04.807 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-23T11:33:05.059 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-23T11:33:05.223 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-23T11:33:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-23T11:33:05.622 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-23T11:33:05.762 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:33:05.766 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-23T11:33:05.980 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-23T11:33:06.018 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-23T11:33:06.250 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-23T11:33:06.291 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-23T11:33:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-23T11:33:06.550 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-23T11:33:06.729 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-23T11:33:06.789 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-23T11:33:06.986 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-23T11:33:07.154 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-23T11:33:07.163 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-23T11:33:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-23T11:33:07.423 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-23T11:33:07.497 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-23T11:33:07.602 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-23T11:33:07.725 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-23T11:33:07.835 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-23T11:33:07.984 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-23T11:33:08.087 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-23T11:33:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-23T11:33:08.407 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-23T11:33:08.514 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-23T11:33:08.600 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-23T11:33:08.805 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-23T11:33:08.971 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-23T11:33:08.990 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-23T11:33:09.247 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-23T11:33:09.281 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-23T11:33:09.466 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-23T11:33:09.704 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-23T11:33:09.909 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-23T11:33:10.237 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-23T11:33:10.337 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-23T11:33:10.424 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-23T11:33:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-23T11:33:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-23T11:33:10.820 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-23T11:33:10.984 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-23T11:33:11.333 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-23T11:33:11.570 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-23T11:33:11.717 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el8.n 27/32 2024-03-23T11:33:11.931 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-23T11:33:11.979 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 28/32 2024-03-23T11:33:12.199 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-23T11:33:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-23T11:33:12.930 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-23T11:33:13.337 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el8.n 27/32 2024-03-23T11:33:13.531 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 28/32 2024-03-23T11:33:13.573 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 28/32 2024-03-23T11:33:13.787 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 29/32 2024-03-23T11:33:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 29/32 2024-03-23T11:33:14.500 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 30/32 2024-03-23T11:33:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 30/32 2024-03-23T11:33:14.959 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 31/32 2024-03-23T11:33:14.999 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 31/32 2024-03-23T11:33:15.125 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 32/32 2024-03-23T11:33:15.241 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 28/32 2024-03-23T11:33:15.252 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 32/32 2024-03-23T11:33:15.252 INFO:teuthology.orchestra.run.smithi111.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-23T11:33:15.252 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 29/32 2024-03-23T11:33:15.784 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 4/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el8.n 6/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-23T11:33:15.785 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-23T11:33:15.786 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-23T11:33:15.787 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-23T11:33:15.787 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-23T11:33:15.787 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-23T11:33:15.787 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-23T11:33:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 29/32 2024-03-23T11:33:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 30/32 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:16.640 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-23T11:33:16.641 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-2.6-1.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:16.642 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:33:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 30/32 2024-03-23T11:33:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 31/32 2024-03-23T11:33:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 31/32 2024-03-23T11:33:16.960 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2024-03-23T11:33:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 32/32 2024-03-23T11:33:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 32/32 2024-03-23T11:33:17.120 INFO:teuthology.orchestra.run.smithi063.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-23T11:33:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:17.671 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/32 2024-03-23T11:33:17.671 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/32 2024-03-23T11:33:17.671 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 3/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 4/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 5/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el8.n 6/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-23T11:33:17.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-23T11:33:17.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-23T11:33:17.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-23T11:33:17.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-23T11:33:17.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-23T11:33:17.887 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:26 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:18.598 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-2.6-1.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-23T11:33:18.599 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:18.600 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:33:18.922 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-03-23T11:33:19.486 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:33:19.538 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:33:19.539 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:33:19.540 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:33:19.655 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2024-03-23T11:33:19.895 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:31 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:33:20.576 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:29 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:33:21.668 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:33:21.725 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:33:21.726 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:33:21.727 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:33:21.863 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-03-23T11:33:22.239 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:33:22.240 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:22.240 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:33:22.240 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:22.240 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:33:22.240 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 82 k 2024-03-23T11:33:22.240 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:22.240 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:33:22.241 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:22.241 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-23T11:33:22.241 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:22.241 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 82 k 2024-03-23T11:33:22.241 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 372 k 2024-03-23T11:33:22.241 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:33:22.825 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:34 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:33:22.864 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-17.2.7-726.g9c4e0c34.el8.noarch.rpm 132 kB/s | 82 kB 00:00 2024-03-23T11:33:22.864 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:22.865 INFO:teuthology.orchestra.run.smithi111.stdout:Total 131 kB/s | 82 kB 00:00 2024-03-23T11:33:22.865 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:33:22.876 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:33:22.876 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:33:22.879 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:33:22.883 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:33:22.931 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:33:23.521 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:24.038 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:24.484 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:24.485 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:24.485 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:33:24.485 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:24.485 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:24.485 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:33:24.632 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:33:24.633 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:33:24.633 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:24.633 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:33:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 82 k 2024-03-23T11:33:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:24.634 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:33:24.634 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:24.634 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-23T11:33:24.634 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:24.634 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 82 k 2024-03-23T11:33:24.634 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 372 k 2024-03-23T11:33:24.634 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:33:24.738 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2024-03-23T11:33:25.186 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-17.2.7-726.g9c4e0c34.el8.noarch.rpm 149 kB/s | 82 kB 00:00 2024-03-23T11:33:25.186 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:25.186 INFO:teuthology.orchestra.run.smithi063.stdout:Total 148 kB/s | 82 kB 00:00 2024-03-23T11:33:25.187 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:33:25.198 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:33:25.198 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:33:25.201 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:33:25.205 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:33:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:33:25.666 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:34 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:33:25.855 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:26.034 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:26.345 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:27.341 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:33:27.341 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:27.341 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-23T11:33:27.341 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:27.342 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:33:27.342 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 162 k 2024-03-23T11:33:27.342 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:27.342 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:33:27.342 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:27.342 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-23T11:33:27.342 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:27.343 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 162 k 2024-03-23T11:33:27.343 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 494 k 2024-03-23T11:33:27.343 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:33:27.994 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-17.2.7-726.g9c4e0c3 248 kB/s | 162 kB 00:00 2024-03-23T11:33:27.995 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:27.995 INFO:teuthology.orchestra.run.smithi111.stdout:Total 248 kB/s | 162 kB 00:00 2024-03-23T11:33:27.995 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:33:28.009 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:33:28.009 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:33:28.065 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:33:28.068 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:33:28.187 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/1 2024-03-23T11:33:28.187 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:28.187 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:33:28.187 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:28.187 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:28.187 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:33:28.201 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:33:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-23T11:33:28.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-03-23T11:33:29.056 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-23T11:33:29.418 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:41 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:33:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-23T11:33:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:29.493 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:33:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:29.493 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:33:29.705 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2024-03-23T11:33:30.632 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:39 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:33:31.218 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:33:31.218 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:31.218 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-23T11:33:31.218 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:31.218 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:33:31.219 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 162 k 2024-03-23T11:33:31.219 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:31.219 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:33:31.219 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:31.219 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-23T11:33:31.219 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:31.219 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 162 k 2024-03-23T11:33:31.220 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 494 k 2024-03-23T11:33:31.220 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:33:31.799 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-17.2.7-726.g9c4e0c3 279 kB/s | 162 kB 00:00 2024-03-23T11:33:31.799 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:31.800 INFO:teuthology.orchestra.run.smithi063.stdout:Total 279 kB/s | 162 kB 00:00 2024-03-23T11:33:31.800 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:33:31.814 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:33:31.815 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:33:31.872 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:33:31.876 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:33:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:33:32.153 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-23T11:33:32.375 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:33:32.428 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:33:32.429 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:33:32.429 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:33:32.535 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2024-03-23T11:33:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-23T11:33:33.288 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-23T11:33:33.288 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:33.288 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:33:33.288 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:33:33.289 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:33.289 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:33:33.444 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:42 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:33:33.537 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-03-23T11:33:34.493 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:46 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:33:35.093 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:33:35.094 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:35.094 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:33:35.094 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:35.094 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 1.7 M 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 23 k 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 14 k 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-23T11:33:35.095 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:35.096 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 2.0 M 2024-03-23T11:33:35.096 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 8.0 M 2024-03-23T11:33:35.096 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:33:35.713 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): ceph-prometheus-alerts-17.2.7-726.g9c4e0 29 kB/s | 14 kB 00:00 2024-03-23T11:33:35.738 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): ceph-grafana-dashboards-17.2.7-726.g9c4e 44 kB/s | 23 kB 00:00 2024-03-23T11:33:35.831 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 364 kB/s | 43 kB 00:00 2024-03-23T11:33:35.906 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 202 kB/s | 34 kB 00:00 2024-03-23T11:33:36.006 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-03-23T11:33:36.281 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-mgr-dashboard-17.2.7-726.g9c4e0c34. 1.6 MB/s | 1.7 MB 00:01 2024-03-23T11:33:36.285 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:36.285 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-23T11:33:36.291 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:33:36.307 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:33:36.320 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:33:36.320 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:33:36.347 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:33:36.349 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:33:36.349 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:33:36.425 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:33:36.429 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:33:36.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-03-23T11:33:36.730 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:33:36.959 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-03-23T11:33:37.211 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-03-23T11:33:37.407 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-03-23T11:33:37.436 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:49 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:33:37.554 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el8.no 4/6 2024-03-23T11:33:37.936 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el8.n 5/6 2024-03-23T11:33:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 6/6 2024-03-23T11:33:38.526 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 6/6 2024-03-23T11:33:38.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el8.n 1/6 2024-03-23T11:33:38.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 2/6 2024-03-23T11:33:38.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el8.no 3/6 2024-03-23T11:33:38.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-03-23T11:33:38.527 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-03-23T11:33:38.974 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-03-23T11:33:38.974 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:38.975 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:33:39.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-23T11:33:39.336 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 1.7 M 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 23 k 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 14 k 2024-03-23T11:33:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-23T11:33:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-23T11:33:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-23T11:33:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:39.338 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:33:39.338 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:33:39.338 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-03-23T11:33:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:39.339 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.0 M 2024-03-23T11:33:39.339 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 8.0 M 2024-03-23T11:33:39.339 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:33:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): ceph-prometheus-alerts-17.2.7-726.g9c4e0 40 kB/s | 14 kB 00:00 2024-03-23T11:33:40.002 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 734 kB/s | 43 kB 00:00 2024-03-23T11:33:40.028 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): ceph-grafana-dashboards-17.2.7-726.g9c4e 52 kB/s | 23 kB 00:00 2024-03-23T11:33:40.054 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:49 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:33:40.278 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 122 kB/s | 34 kB 00:00 2024-03-23T11:33:40.404 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 524 kB/s | 196 kB 00:00 2024-03-23T11:33:40.503 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): ceph-mgr-dashboard-17.2.7-726.g9c4e0c34. 1.8 MB/s | 1.7 MB 00:00 2024-03-23T11:33:40.507 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:40.507 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-23T11:33:40.530 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:33:40.543 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:33:40.543 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:33:40.648 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:33:40.652 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:33:40.933 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:33:41.157 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-03-23T11:33:41.434 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-03-23T11:33:41.604 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-03-23T11:33:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el8.no 4/6 2024-03-23T11:33:41.738 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 7.4 M 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-23T11:33:41.740 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-23T11:33:41.741 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:41.741 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:33:41.741 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-23T11:33:41.741 INFO:teuthology.orchestra.run.smithi111.stdout:Install 8 Packages 2024-03-23T11:33:41.741 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:41.742 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 36 M 2024-03-23T11:33:41.742 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 210 M 2024-03-23T11:33:41.742 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:33:42.100 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el8.n 5/6 2024-03-23T11:33:42.284 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 6/6 2024-03-23T11:33:42.548 INFO:teuthology.orchestra.run.smithi111.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.7-726 13 MB/s | 7.4 MB 00:00 2024-03-23T11:33:42.641 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 6/6 2024-03-23T11:33:42.641 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el8.n 1/6 2024-03-23T11:33:42.641 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 2/6 2024-03-23T11:33:42.641 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el8.no 3/6 2024-03-23T11:33:42.641 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-03-23T11:33:42.642 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-03-23T11:33:42.823 INFO:teuthology.orchestra.run.smithi111.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.6 MB/s | 4.7 MB 00:00 2024-03-23T11:33:42.910 INFO:teuthology.orchestra.run.smithi111.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.3 MB/s | 4.9 MB 00:00 2024-03-23T11:33:42.932 INFO:teuthology.orchestra.run.smithi111.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-03-23T11:33:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:43.090 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:33:43.116 INFO:teuthology.orchestra.run.smithi111.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 645 kB 00:00 2024-03-23T11:33:43.174 INFO:teuthology.orchestra.run.smithi111.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2024-03-23T11:33:43.319 INFO:teuthology.orchestra.run.smithi111.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.9 MB/s | 3.7 MB 00:00 2024-03-23T11:33:43.433 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-23T11:33:43.858 INFO:teuthology.orchestra.run.smithi111.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 15 MB/s | 14 MB 00:00 2024-03-23T11:33:43.858 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:43.859 INFO:teuthology.orchestra.run.smithi111.stdout:Total 17 MB/s | 36 MB 00:02 2024-03-23T11:33:44.169 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:33:44.192 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:33:44.192 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:33:44.397 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:56 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:33:44.630 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:33:44.634 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:33:45.216 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:33:45.378 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-23T11:33:45.525 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-23T11:33:45.711 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-23T11:33:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:33:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-23T11:33:46.206 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:33:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-23T11:33:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 7.4 M 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-23T11:33:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:Install 8 Packages 2024-03-23T11:33:46.208 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 36 M 2024-03-23T11:33:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 210 M 2024-03-23T11:33:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:33:46.365 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-23T11:33:46.536 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-03-23T11:33:47.204 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-03-23T11:33:47.303 INFO:teuthology.orchestra.run.smithi063.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.7-726 11 MB/s | 7.4 MB 00:00 2024-03-23T11:33:47.678 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-23T11:33:48.178 INFO:teuthology.orchestra.run.smithi063.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.3 MB/s | 3.7 MB 00:00 2024-03-23T11:33:48.490 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-23T11:33:48.844 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-03-23T11:33:49.078 INFO:teuthology.orchestra.run.smithi063.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 250 kB/s | 225 kB 00:00 2024-03-23T11:33:49.111 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-23T11:33:49.737 INFO:teuthology.orchestra.run.smithi063.stdout:(4/8): python3-scipy-1.0.0-21.module_el8+762+77 22 MB/s | 14 MB 00:00 2024-03-23T11:33:49.862 INFO:teuthology.orchestra.run.smithi063.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 5.1 MB/s | 645 kB 00:00 2024-03-23T11:33:49.904 INFO:teuthology.orchestra.run.smithi063.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2024-03-23T11:33:49.971 INFO:teuthology.orchestra.run.smithi063.stdout:(7/8): openblas-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.7 MB 00:03 2024-03-23T11:33:50.047 INFO:teuthology.orchestra.run.smithi063.stdout:(8/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.9 MB 00:03 2024-03-23T11:33:50.047 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:50.047 INFO:teuthology.orchestra.run.smithi063.stdout:Total 9.4 MB/s | 36 MB 00:03 2024-03-23T11:33:50.367 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:33:50.391 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:33:50.391 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:33:50.826 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:33:50.830 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:33:51.120 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-23T11:33:51.447 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:33:51.583 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-23T11:33:51.740 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-23T11:33:51.875 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-23T11:33:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-03-23T11:33:52.556 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-23T11:33:52.702 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34 8/8 2024-03-23T11:33:52.742 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-03-23T11:33:53.108 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34 8/8 2024-03-23T11:33:53.109 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34 1/8 2024-03-23T11:33:53.109 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-03-23T11:33:53.109 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-03-23T11:33:53.109 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-03-23T11:33:53.109 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-03-23T11:33:53.109 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-03-23T11:33:53.109 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-03-23T11:33:53.445 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-23T11:33:53.560 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-23T11:33:53.561 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-23T11:33:53.561 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-23T11:33:53.561 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:53.561 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:33:53.591 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-23T11:33:53.722 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2024-03-23T11:33:54.478 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-23T11:33:54.635 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:03 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:33:54.840 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-03-23T11:33:55.108 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-23T11:33:56.317 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:33:56.319 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:56.319 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:33:56.319 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 47 k 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-23T11:33:56.320 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-03-23T11:33:56.321 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:33:56.322 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.9 M 2024-03-23T11:33:56.322 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 25 M 2024-03-23T11:33:56.322 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:33:56.897 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): ceph-mgr-rook-17.2.7-726.g9c4e0c34.el8. 228 kB/s | 47 kB 00:00 2024-03-23T11:33:56.914 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 89 kB/s | 20 kB 00:00 2024-03-23T11:33:56.939 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 106 kB/s | 27 kB 00:00 2024-03-23T11:33:57.048 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2024-03-23T11:33:57.073 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2024-03-23T11:33:57.123 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 558 kB/s | 126 kB 00:00 2024-03-23T11:33:57.149 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 468 kB/s | 110 kB 00:00 2024-03-23T11:33:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-23T11:33:57.324 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 81 kB/s | 16 kB 00:00 2024-03-23T11:33:57.349 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 119 kB/s | 33 kB 00:00 2024-03-23T11:33:57.433 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 291 kB/s | 81 kB 00:00 2024-03-23T11:33:57.467 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 549 kB/s | 64 kB 00:00 2024-03-23T11:33:57.525 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 670 kB/s | 61 kB 00:00 2024-03-23T11:33:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2024-03-23T11:33:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:33:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-03-23T11:33:57.824 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:33:57.844 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:33:57.844 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:33:57.953 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:33:57.957 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:33:58.515 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:33:58.716 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-03-23T11:33:58.737 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-03-23T11:33:58.907 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34 8/8 2024-03-23T11:33:58.932 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-03-23T11:33:59.183 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-03-23T11:33:59.331 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34 8/8 2024-03-23T11:33:59.331 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34 1/8 2024-03-23T11:33:59.332 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-03-23T11:33:59.332 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-03-23T11:33:59.332 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-03-23T11:33:59.332 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-03-23T11:33:59.332 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-03-23T11:33:59.332 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-03-23T11:33:59.410 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-03-23T11:33:59.610 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-23T11:33:59.802 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-23T11:33:59.803 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-23T11:33:59.803 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-23T11:33:59.803 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-23T11:33:59.803 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:33:59.803 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:33:59.840 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-03-23T11:33:59.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-03-23T11:34:00.129 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-03-23T11:34:00.376 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-03-23T11:34:00.936 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:12 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:01.635 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-03-23T11:34:01.900 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-03-23T11:34:02.085 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-03-23T11:34:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-03-23T11:34:02.509 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 13/13 2024-03-23T11:34:02.736 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 47 k 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-23T11:34:02.738 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-03-23T11:34:02.739 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:02.740 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 1.9 M 2024-03-23T11:34:02.740 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 25 M 2024-03-23T11:34:02.740 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:34:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 13/13 2024-03-23T11:34:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 1/13 2024-03-23T11:34:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-03-23T11:34:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-03-23T11:34:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-03-23T11:34:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-03-23T11:34:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-03-23T11:34:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-03-23T11:34:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-03-23T11:34:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-03-23T11:34:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-03-23T11:34:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-03-23T11:34:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-03-23T11:34:03.386 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-23T11:34:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-23T11:34:03.388 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-23T11:34:03.388 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-23T11:34:03.388 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:03.388 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:03.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2024-03-23T11:34:04.453 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:13 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:04.617 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): ceph-mgr-rook-17.2.7-726.g9c4e0c34.el8. 236 kB/s | 47 kB 00:00 2024-03-23T11:34:04.642 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 118 kB/s | 27 kB 00:00 2024-03-23T11:34:04.668 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 80 kB/s | 20 kB 00:00 2024-03-23T11:34:04.751 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.8 MB/s | 155 kB 00:00 2024-03-23T11:34:04.812 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 703 kB/s | 43 kB 00:00 2024-03-23T11:34:04.935 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 376 kB/s | 110 kB 00:00 2024-03-23T11:34:04.994 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 182 kB/s | 33 kB 00:00 2024-03-23T11:34:05.028 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 308 kB/s | 126 kB 00:00 2024-03-23T11:34:05.095 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 852 kB/s | 81 kB 00:00 2024-03-23T11:34:05.145 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-03-23T11:34:05.203 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-03-23T11:34:05.287 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): python3-certifi-2018.10.15-7.el8.noarc 46 kB/s | 16 kB 00:00 2024-03-23T11:34:05.387 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2024-03-23T11:34:05.391 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:05.391 INFO:teuthology.orchestra.run.smithi063.stdout:Total 746 kB/s | 1.9 MB 00:02 2024-03-23T11:34:05.523 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:34:05.544 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:34:05.544 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:34:05.661 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:34:05.665 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:34:06.123 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 123 k 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-23T11:34:06.124 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-23T11:34:06.125 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-23T11:34:06.125 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:06.125 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:34:06.125 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:06.125 INFO:teuthology.orchestra.run.smithi111.stdout:Install 5 Packages 2024-03-23T11:34:06.125 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:06.125 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 6.0 M 2024-03-23T11:34:06.126 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-03-23T11:34:06.126 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:34:06.273 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:34:06.494 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-03-23T11:34:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-03-23T11:34:06.881 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-03-23T11:34:07.123 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-03-23T11:34:07.333 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-03-23T11:34:07.572 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-03-23T11:34:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-03-23T11:34:08.100 INFO:teuthology.orchestra.run.smithi111.stdout:(1/5): ceph-mgr-cephadm-17.2.7-726.g9c4e0c34.el 287 kB/s | 123 kB 00:00 2024-03-23T11:34:08.116 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-03-23T11:34:08.208 INFO:teuthology.orchestra.run.smithi111.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.0 MB/s | 578 kB 00:00 2024-03-23T11:34:08.458 INFO:teuthology.orchestra.run.smithi111.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 6.1 MB/s | 4.8 MB 00:00 2024-03-23T11:34:08.559 INFO:teuthology.orchestra.run.smithi111.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 172 kB/s | 60 kB 00:00 2024-03-23T11:34:08.733 INFO:teuthology.orchestra.run.smithi111.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 751 kB/s | 476 kB 00:00 2024-03-23T11:34:08.735 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:08.735 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.3 MB/s | 6.0 MB 00:02 2024-03-23T11:34:08.814 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:34:08.829 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:34:08.829 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:34:08.955 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:34:08.958 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:34:09.246 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:34:09.399 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-03-23T11:34:09.579 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-03-23T11:34:09.681 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-03-23T11:34:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-03-23T11:34:10.103 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-03-23T11:34:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 13/13 2024-03-23T11:34:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-03-23T11:34:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 13/13 2024-03-23T11:34:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 1/13 2024-03-23T11:34:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-03-23T11:34:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-03-23T11:34:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-03-23T11:34:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-03-23T11:34:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-03-23T11:34:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-03-23T11:34:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-03-23T11:34:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-03-23T11:34:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-03-23T11:34:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-03-23T11:34:10.636 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-03-23T11:34:10.984 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-03-23T11:34:11.154 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-03-23T11:34:11.154 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:11.155 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:11.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-03-23T11:34:11.324 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-03-23T11:34:11.532 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 5/5 2024-03-23T11:34:11.874 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 5/5 2024-03-23T11:34:11.874 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/5 2024-03-23T11:34:11.874 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-03-23T11:34:11.874 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-03-23T11:34:11.874 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-03-23T11:34:12.274 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:24 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:12.302 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-03-23T11:34:12.302 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:12.302 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:34:12.302 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:34:12.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-23T11:34:12.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-23T11:34:12.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-23T11:34:12.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-23T11:34:12.303 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:12.303 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:12.451 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2024-03-23T11:34:13.366 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:22 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:14.069 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:14.070 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:14.070 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:34:14.070 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:14.070 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:34:14.070 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:17.2.7-726.g9c4e0c34.el8 ceph-noarch 123 k 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout:Install 5 Packages 2024-03-23T11:34:14.071 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:14.072 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 6.0 M 2024-03-23T11:34:14.072 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-03-23T11:34:14.072 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:34:14.636 INFO:teuthology.orchestra.run.smithi063.stdout:(1/5): ceph-mgr-cephadm-17.2.7-726.g9c4e0c34.el 381 kB/s | 123 kB 00:00 2024-03-23T11:34:14.761 INFO:teuthology.orchestra.run.smithi063.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 3.7 MB/s | 476 kB 00:00 2024-03-23T11:34:14.786 INFO:teuthology.orchestra.run.smithi063.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 2.4 MB/s | 60 kB 00:00 2024-03-23T11:34:14.911 INFO:teuthology.orchestra.run.smithi063.stdout:(4/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 966 kB/s | 578 kB 00:00 2024-03-23T11:34:15.027 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:15.027 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:15.027 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 876 k 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-23T11:34:15.028 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:15.029 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 959 k 2024-03-23T11:34:15.029 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 3.1 M 2024-03-23T11:34:15.029 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:34:15.145 INFO:teuthology.orchestra.run.smithi063.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.8 MB/s | 4.8 MB 00:00 2024-03-23T11:34:15.146 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:15.146 INFO:teuthology.orchestra.run.smithi063.stdout:Total 5.6 MB/s | 6.0 MB 00:01 2024-03-23T11:34:15.223 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:34:15.238 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:34:15.238 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:34:15.333 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 782 kB/s | 83 kB 00:00 2024-03-23T11:34:15.368 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:34:15.372 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:34:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:34:15.991 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-fuse-17.2.7-726.g9c4e0c34.el8.x86_6 1.1 MB/s | 876 kB 00:00 2024-03-23T11:34:15.992 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:15.992 INFO:teuthology.orchestra.run.smithi111.stdout:Total 995 kB/s | 959 kB 00:00 2024-03-23T11:34:16.015 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:34:16.041 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:34:16.041 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:34:16.060 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-03-23T11:34:16.151 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:34:16.154 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:34:16.364 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:34:16.648 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-23T11:34:16.848 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/2 2024-03-23T11:34:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-03-23T11:34:17.434 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/2 2024-03-23T11:34:17.434 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/2 2024-03-23T11:34:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-03-23T11:34:17.831 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-03-23T11:34:17.874 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-23T11:34:17.874 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:17.874 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:34:17.874 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-23T11:34:17.875 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:17.875 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:18.013 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 5/5 2024-03-23T11:34:18.106 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2024-03-23T11:34:18.369 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 5/5 2024-03-23T11:34:18.369 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 1/5 2024-03-23T11:34:18.369 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-03-23T11:34:18.369 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-03-23T11:34:18.369 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el8.noarch 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:18.840 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:19.017 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-03-23T11:34:19.051 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:28 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:19.973 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:31 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:20.744 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 130 k 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:20.745 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-23T11:34:20.746 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:20.746 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 130 k 2024-03-23T11:34:20.746 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 464 k 2024-03-23T11:34:20.746 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:34:20.947 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-17.2.7-726.g9c4e0c34.el8.x86_64. 648 kB/s | 130 kB 00:00 2024-03-23T11:34:20.947 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:20.948 INFO:teuthology.orchestra.run.smithi111.stdout:Total 644 kB/s | 130 kB 00:00 2024-03-23T11:34:20.948 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:34:20.955 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:34:20.955 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:34:21.012 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:34:21.015 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:34:21.156 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:34:21.456 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:21.778 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 876 k 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-03-23T11:34:21.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:21.780 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 959 k 2024-03-23T11:34:21.780 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 3.1 M 2024-03-23T11:34:21.780 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:34:21.839 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:21.902 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2024-03-23T11:34:22.152 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-17.2.7-726.g9c4e0c34.el8.x86_6 2.7 MB/s | 876 kB 00:00 2024-03-23T11:34:22.153 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:22.153 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.5 MB/s | 959 kB 00:00 2024-03-23T11:34:22.175 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:34:22.201 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:34:22.201 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:34:22.280 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:22.280 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:22.280 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:34:22.280 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:34:22.281 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:22.281 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:22.316 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:34:22.320 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:34:22.512 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2024-03-23T11:34:22.527 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:34:22.810 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-23T11:34:22.996 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/2 2024-03-23T11:34:23.443 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:32 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:23.630 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2/2 2024-03-23T11:34:23.630 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/2 2024-03-23T11:34:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-23T11:34:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:24.073 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:34:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-23T11:34:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:24.073 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:24.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-03-23T11:34:25.116 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:25.168 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:25.169 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:34:25.169 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:25.290 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:37 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:25.293 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2024-03-23T11:34:26.204 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:35 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:27.104 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:27.105 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:27.105 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-23T11:34:27.105 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:27.105 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:34:27.106 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 130 k 2024-03-23T11:34:27.106 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:27.106 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:34:27.106 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:27.106 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-23T11:34:27.106 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:27.106 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 130 k 2024-03-23T11:34:27.107 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 464 k 2024-03-23T11:34:27.107 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:34:27.291 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-17.2.7-726.g9c4e0c34.el8.x86_64. 705 kB/s | 130 kB 00:00 2024-03-23T11:34:27.292 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:27.292 INFO:teuthology.orchestra.run.smithi063.stdout:Total 699 kB/s | 130 kB 00:00 2024-03-23T11:34:27.292 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:34:27.300 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:34:27.301 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:34:27.361 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:34:27.364 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:34:27.492 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:34:27.719 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:27.870 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:27.870 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:27.870 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-23T11:34:27.870 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:27.870 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:34:27.871 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 25 k 2024-03-23T11:34:27.871 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:27.871 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:34:27.871 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:27.871 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-23T11:34:27.871 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:27.871 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 25 k 2024-03-23T11:34:27.872 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 109 k 2024-03-23T11:34:27.872 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:34:28.037 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-17.2.7-726.g9c4e0c34.el8.x86_64 151 kB/s | 25 kB 00:00 2024-03-23T11:34:28.037 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:28.037 INFO:teuthology.orchestra.run.smithi111.stdout:Total 150 kB/s | 25 kB 00:00 2024-03-23T11:34:28.038 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:34:28.042 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:34:28.042 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:34:28.046 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:34:28.049 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:34:28.105 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:28.107 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:34:28.328 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:28.572 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:28.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:34:28.572 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:34:28.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:28.573 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:28.641 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:28.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-03-23T11:34:29.048 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:29.048 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:29.048 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:34:29.049 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:34:29.049 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:29.049 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:29.219 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2024-03-23T11:34:29.787 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:41 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:30.131 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:39 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:31.529 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:31.585 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:31.586 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:34:31.586 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:31.745 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:31.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-03-23T11:34:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:34:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:31.920 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2024-03-23T11:34:32.715 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:44 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:32.833 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:41 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:34.450 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:34.506 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:34.508 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:34:34.508 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:34.543 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:34.543 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 25 k 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-23T11:34:34.544 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:34.545 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 25 k 2024-03-23T11:34:34.545 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 109 k 2024-03-23T11:34:34.545 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:34:34.765 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-17.2.7-726.g9c4e0c34.el8.x86_64 113 kB/s | 25 kB 00:00 2024-03-23T11:34:34.765 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:34.765 INFO:teuthology.orchestra.run.smithi063.stdout:Total 112 kB/s | 25 kB 00:00 2024-03-23T11:34:34.766 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:34:34.771 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:34:34.771 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:34:34.775 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:34:34.779 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:34:34.832 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:34:34.884 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2024-03-23T11:34:35.035 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:35.361 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:35.731 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:35.731 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:35.731 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:34:35.731 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:34:35.731 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:35.732 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:35.802 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:44 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:35.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-03-23T11:34:36.865 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:48 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:37.446 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:37.499 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:37.500 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:34:37.500 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:37.620 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2024-03-23T11:34:38.542 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:47 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:38.676 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:38.734 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:38.736 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:34:38.736 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:38.869 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-03-23T11:34:39.826 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:51 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:40.151 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:40.203 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:40.205 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:34:40.205 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:40.330 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2024-03-23T11:34:41.237 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:50 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:41.822 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:41.878 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:41.880 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:34:41.880 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:42.014 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-03-23T11:34:42.878 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:42.931 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:42.932 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:34:42.932 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:42.980 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:54 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:43.050 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2024-03-23T11:34:43.969 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:52 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:44.754 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:44.811 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:44.812 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:34:44.812 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:44.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-03-23T11:34:45.726 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:45.778 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:45.779 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-23T11:34:45.779 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:45.901 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2024-03-23T11:34:45.925 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:57 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:46.816 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:55 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:47.708 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:47.709 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:34:47.710 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:47.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-03-23T11:34:48.491 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 90 k 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-23T11:34:48.492 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:48.493 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 90 k 2024-03-23T11:34:48.493 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 268 k 2024-03-23T11:34:48.493 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:34:48.792 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-17.2.7-726.g9c4e0c34.el8.x86_64.rpm 303 kB/s | 90 kB 00:00 2024-03-23T11:34:48.793 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:48.793 INFO:teuthology.orchestra.run.smithi111.stdout:Total 301 kB/s | 90 kB 00:00 2024-03-23T11:34:48.793 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:34:48.805 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:34:48.805 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:34:48.811 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:00 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:48.862 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:34:48.865 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:34:48.977 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:34:49.233 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:49.638 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:50.279 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:50.279 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:50.279 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:34:50.279 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:34:50.279 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:50.279 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:50.524 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2024-03-23T11:34:50.547 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:50.603 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:50.604 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:34:50.605 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:50.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-03-23T11:34:51.463 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:00 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:51.697 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:03 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:53.112 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.5 M 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-23T11:34:53.113 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:53.114 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.5 M 2024-03-23T11:34:53.114 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 16 M 2024-03-23T11:34:53.114 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:34:53.442 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 is already installed. 2024-03-23T11:34:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-23T11:34:53.499 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:53.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-03-23T11:34:53.678 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-17.2.7-726.g9c4e0c34.el8.x86_64.rpm 6.1 MB/s | 3.5 MB 00:00 2024-03-23T11:34:53.679 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:53.679 INFO:teuthology.orchestra.run.smithi111.stdout:Total 6.1 MB/s | 3.5 MB 00:00 2024-03-23T11:34:53.680 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:34:53.694 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:34:53.694 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:34:53.764 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:34:53.767 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:34:54.256 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:34:54.476 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:54.584 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:06 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:55.124 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:55.570 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:34:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:34:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:55.571 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:34:55.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2024-03-23T11:34:56.374 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:34:56.375 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 90 k 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 90 k 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 268 k 2024-03-23T11:34:56.376 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:34:56.621 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-17.2.7-726.g9c4e0c34.el8.x86_64.rpm 371 kB/s | 90 kB 00:00 2024-03-23T11:34:56.621 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:56.621 INFO:teuthology.orchestra.run.smithi063.stdout:Total 369 kB/s | 90 kB 00:00 2024-03-23T11:34:56.622 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:34:56.635 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:34:56.635 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:34:56.694 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:34:56.698 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:34:56.698 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:05 ago on Sat 23 Mar 2024 11:31:51 AM UTC. 2024-03-23T11:34:56.814 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:34:57.007 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:57.393 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:57.824 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:57.824 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:57.824 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:34:57.824 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:34:57.824 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:34:57.824 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:34:58.076 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-03-23T11:34:58.352 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 176 k 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-23T11:34:58.353 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:34:58.354 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 176 k 2024-03-23T11:34:58.354 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 556 k 2024-03-23T11:34:58.354 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-23T11:34:59.023 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-17.2.7-726.g9c4e0c34.el8.x86_64.rpm 262 kB/s | 176 kB 00:00 2024-03-23T11:34:59.024 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:34:59.024 INFO:teuthology.orchestra.run.smithi111.stdout:Total 262 kB/s | 176 kB 00:00 2024-03-23T11:34:59.025 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-23T11:34:59.039 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-23T11:34:59.039 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-23T11:34:59.044 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:11 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:34:59.102 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-23T11:34:59.105 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-23T11:34:59.239 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-23T11:34:59.457 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:34:59.845 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:35:00.296 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-23T11:35:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:35:00.296 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:35:00.297 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-23T11:35:00.539 DEBUG:teuthology.parallel:result is None 2024-03-23T11:35:00.833 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:35:00.834 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:35:00.834 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-23T11:35:00.834 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:35:00.834 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 3.5 M 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.5 M 2024-03-23T11:35:00.835 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 16 M 2024-03-23T11:35:00.836 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:35:02.215 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-17.2.7-726.g9c4e0c34.el8.x86_64.rpm 2.5 MB/s | 3.5 MB 00:01 2024-03-23T11:35:02.216 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:35:02.217 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.5 MB/s | 3.5 MB 00:01 2024-03-23T11:35:02.217 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:35:02.234 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:35:02.234 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:35:02.312 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:35:02.316 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:35:02.839 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:35:02.999 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:04.193 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:35:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:35:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:04.193 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:35:04.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-03-23T11:35:05.395 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:17 ago on Sat 23 Mar 2024 11:31:48 AM UTC. 2024-03-23T11:35:07.192 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:17.2.7-726.g9c4e0c34.el8 ceph 176 k 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-23T11:35:07.193 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:07.194 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 176 k 2024-03-23T11:35:07.194 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 556 k 2024-03-23T11:35:07.194 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-23T11:35:07.526 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-17.2.7-726.g9c4e0c34.el8.x86_64.rpm 529 kB/s | 176 kB 00:00 2024-03-23T11:35:07.527 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:35:07.527 INFO:teuthology.orchestra.run.smithi063.stdout:Total 527 kB/s | 176 kB 00:00 2024-03-23T11:35:07.527 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-23T11:35:07.542 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-23T11:35:07.542 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-23T11:35:07.603 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-23T11:35:07.607 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-23T11:35:07.732 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-23T11:35:07.914 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:08.736 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 1/1 2024-03-23T11:35:08.736 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:08.736 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-23T11:35:08.736 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:17.2.7-726.g9c4e0c34.el8.x86_64 2024-03-23T11:35:08.736 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:08.736 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-23T11:35:09.093 DEBUG:teuthology.parallel:result is None 2024-03-23T11:35:09.094 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:35:09.094 INFO:teuthology.packaging:ref: None 2024-03-23T11:35:09.094 INFO:teuthology.packaging:tag: None 2024-03-23T11:35:09.094 INFO:teuthology.packaging:branch: quincy 2024-03-23T11:35:09.094 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:35:09.094 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-23T11:35:09.456 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T11:35:09.516 INFO:teuthology.orchestra.run.smithi063.stdout:17.2.7-726.g9c4e0c34.el8 2024-03-23T11:35:09.517 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-726.g9c4e0c34.el8 2024-03-23T11:35:09.517 INFO:teuthology.task.install:The correct ceph version 17.2.7-726.g9c4e0c34 is installed. 2024-03-23T11:35:09.519 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:35:09.519 INFO:teuthology.packaging:ref: None 2024-03-23T11:35:09.519 INFO:teuthology.packaging:tag: None 2024-03-23T11:35:09.519 INFO:teuthology.packaging:branch: quincy 2024-03-23T11:35:09.519 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:35:09.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-23T11:35:09.887 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T11:35:09.945 INFO:teuthology.orchestra.run.smithi111.stdout:17.2.7-726.g9c4e0c34.el8 2024-03-23T11:35:09.945 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-726.g9c4e0c34.el8 2024-03-23T11:35:09.946 INFO:teuthology.task.install:The correct ceph version 17.2.7-726.g9c4e0c34 is installed. 2024-03-23T11:35:09.947 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-23T11:35:09.948 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:35:09.948 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T11:35:10.019 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:35:10.019 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T11:35:10.089 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-23T11:35:10.090 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:35:10.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T11:35:10.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T11:35:10.264 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:35:10.264 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T11:35:10.330 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T11:35:10.436 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-23T11:35:10.436 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:35:10.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T11:35:10.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T11:35:10.609 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:35:10.609 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T11:35:10.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T11:35:10.779 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-23T11:35:10.779 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:35:10.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T11:35:10.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T11:35:10.952 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:35:10.952 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T11:35:11.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T11:35:11.123 INFO:teuthology.run_tasks:Running task print... 2024-03-23T11:35:11.132 INFO:teuthology.task.print:**** done install task... 2024-03-23T11:35:11.132 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-23T11:35:11.231 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'} 2024-03-23T11:35:11.232 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-23T11:35:11.232 INFO:tasks.cephadm:Cluster fsid is 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:35:11.232 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-23T11:35:11.232 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-23T11:35:11.233 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi063': '172.21.15.63', 'mon.smithi111': '172.21.15.111'} 2024-03-23T11:35:11.233 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-23T11:35:11.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-03-23T11:35:11.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-03-23T11:35:11.368 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-23T11:35:11.368 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-23T11:35:11.543 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Mar 23 11:35 /home/ubuntu/cephtest/cephadm 2024-03-23T11:35:11.543 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-23T11:35:11.731 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Mar 23 11:35 /home/ubuntu/cephtest/cephadm 2024-03-23T11:35:11.732 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-23T11:35:11.778 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-23T11:35:11.835 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-23T11:35:11.835 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-23T11:35:11.839 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-23T11:35:13.246 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-23T11:35:13.248 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-23T11:35:40.799 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-23T11:35:40.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-23T11:35:40.800 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de", 2024-03-23T11:35:40.800 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-03-23T11:35:40.800 INFO:teuthology.orchestra.run.smithi111.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232" 2024-03-23T11:35:40.800 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-03-23T11:35:40.800 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-23T11:35:44.285 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:35:44.286 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-23T11:35:44.286 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de", 2024-03-23T11:35:44.286 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-03-23T11:35:44.286 INFO:teuthology.orchestra.run.smithi063.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232" 2024-03-23T11:35:44.286 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-03-23T11:35:44.286 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:35:44.311 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-03-23T11:35:44.381 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-03-23T11:35:44.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-03-23T11:35:44.530 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-03-23T11:35:44.605 INFO:tasks.cephadm:Writing seed config... 2024-03-23T11:35:44.606 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-23T11:35:44.606 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-23T11:35:44.606 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-23T11:35:44.607 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-23T11:35:44.608 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-23T11:35:44.609 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-23T11:35:44.609 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-23T11:35:44.609 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-23T11:35:44.609 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-23T11:35:44.609 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-23T11:35:44.609 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-23T11:35:44.609 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:35:44.610 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-23T11:35:44.653 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6880ffb6-e909-11ee-95cd-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-23T11:35:44.653 DEBUG:teuthology.orchestra.run.smithi063:mon.smithi063> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063.service 2024-03-23T11:35:44.697 INFO:tasks.cephadm:Bootstrapping... 2024-03-23T11:35:44.697 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.63 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-23T11:35:44.763 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:-- Logs begin at Sat 2024-03-23 11:23:13 UTC. -- 2024-03-23T11:35:45.002 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:35:45.002 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '6880ffb6-e909-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.63', '--skip-admin-label'] 2024-03-23T11:35:45.051 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-23T11:35:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-23T11:35:45.052 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying podman|docker is present... 2024-03-23T11:35:45.097 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-23T11:35:45.097 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying lvm2 is present... 2024-03-23T11:35:45.097 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying time synchronization is in place... 2024-03-23T11:35:45.112 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-23T11:35:45.112 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-23T11:35:45.124 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-23T11:35:45.124 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-03-23T11:35:45.137 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-03-23T11:35:45.149 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-03-23T11:35:45.150 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-03-23T11:35:45.150 INFO:teuthology.orchestra.run.smithi063.stdout:Repeating the final host check... 2024-03-23T11:35:45.194 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-23T11:35:45.194 INFO:teuthology.orchestra.run.smithi063.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-23T11:35:45.194 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl is present 2024-03-23T11:35:45.194 INFO:teuthology.orchestra.run.smithi063.stdout:lvcreate is present 2024-03-23T11:35:45.207 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-23T11:35:45.207 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-23T11:35:45.219 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-23T11:35:45.220 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-03-23T11:35:45.233 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-03-23T11:35:45.245 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-03-23T11:35:45.245 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-03-23T11:35:45.245 INFO:teuthology.orchestra.run.smithi063.stdout:Host looks OK 2024-03-23T11:35:45.245 INFO:teuthology.orchestra.run.smithi063.stdout:Cluster fsid: 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:35:45.246 INFO:teuthology.orchestra.run.smithi063.stdout:Acquiring lock 140168710640528 on /run/cephadm/6880ffb6-e909-11ee-95cd-87774f69a715.lock 2024-03-23T11:35:45.246 INFO:teuthology.orchestra.run.smithi063.stdout:Lock 140168710640528 acquired on /run/cephadm/6880ffb6-e909-11ee-95cd-87774f69a715.lock 2024-03-23T11:35:45.246 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying IP 172.21.15.63 port 3300 ... 2024-03-23T11:35:45.247 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying IP 172.21.15.63 port 6789 ... 2024-03-23T11:35:45.247 INFO:teuthology.orchestra.run.smithi063.stdout:Base mon IP(s) is [172.21.15.63:3300, 172.21.15.63:6789], mon addrv is [v2:172.21.15.63:3300,v1:172.21.15.63:6789] 2024-03-23T11:35:45.255 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.63 metric 100 2024-03-23T11:35:45.255 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.63 metric 100 2024-03-23T11:35:45.262 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-23T11:35:45.262 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-23T11:35:45.262 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-03-23T11:35:45.268 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-23T11:35:45.268 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-23T11:35:45.268 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-23T11:35:45.269 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-23T11:35:45.269 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8d7/64 scope link 2024-03-23T11:35:45.269 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-23T11:35:45.269 INFO:teuthology.orchestra.run.smithi063.stdout:Mon IP `172.21.15.63` is in CIDR network `172.21.0.0/20` 2024-03-23T11:35:45.269 INFO:teuthology.orchestra.run.smithi063.stdout:Mon IP `172.21.15.63` is in CIDR network `172.21.0.0/20` 2024-03-23T11:35:45.270 INFO:teuthology.orchestra.run.smithi063.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-23T11:35:45.270 INFO:teuthology.orchestra.run.smithi063.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-23T11:35:45.271 INFO:teuthology.orchestra.run.smithi063.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-23T11:35:45.762 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de 2024-03-23T11:35:45.763 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-23T11:35:45.763 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Getting image source signatures 2024-03-23T11:35:45.763 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:601ec453fe24458667695dda21742a4f3ea719c31ad8aa8ec48bf585cc42a7d7 2024-03-23T11:35:45.763 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-23T11:35:45.763 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying config sha256:837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de 2024-03-23T11:35:45.763 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-23T11:35:46.834 INFO:teuthology.orchestra.run.smithi063.stdout:ceph: stdout ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable) 2024-03-23T11:35:46.835 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph version: ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable) 2024-03-23T11:35:46.835 INFO:teuthology.orchestra.run.smithi063.stdout:Extracting ceph user uid/gid from container image... 2024-03-23T11:35:47.870 INFO:teuthology.orchestra.run.smithi063.stdout:stat: stdout 167 167 2024-03-23T11:35:47.870 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial keys... 2024-03-23T11:35:48.960 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCUvv5l0IuNERAA7z1QZ8a2H9zOHDCttkW+mg== 2024-03-23T11:35:50.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCVvv5lUVXJLBAAZf2MXlYRv77zYhjp6GPB4g== 2024-03-23T11:35:51.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCWvv5lFuaYLBAAL/UNxQkL2V0LPBWn3/FWIg== 2024-03-23T11:35:51.334 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial monmap... 2024-03-23T11:35:52.303 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-23T11:35:52.303 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-23T11:35:52.303 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:35:52.303 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-23T11:35:52.303 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool for smithi063 [v2:172.21.15.63:3300,v1:172.21.15.63:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-23T11:35:52.303 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = octopus 2024-03-23T11:35:52.304 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: set fsid to 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:35:52.304 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-23T11:35:52.304 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:35:52.304 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mon... 2024-03-23T11:35:53.285 INFO:teuthology.orchestra.run.smithi063.stdout:create mon.smithi063 on 2024-03-23T11:35:53.494 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-23T11:35:53.683 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-23T11:35:53.893 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6880ffb6-e909-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-6880ffb6-e909-11ee-95cd-87774f69a715.target. 2024-03-23T11:35:53.894 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6880ffb6-e909-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-6880ffb6-e909-11ee-95cd-87774f69a715.target. 2024-03-23T11:35:54.197 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063 2024-03-23T11:35:54.197 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063.service: Unit ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063.service not loaded. 2024-03-23T11:35:54.420 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6880ffb6-e909-11ee-95cd-87774f69a715.target.wants/ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063.service → /etc/systemd/system/ceph-6880ffb6-e909-11ee-95cd-87774f69a715@.service. 2024-03-23T11:35:54.461 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:54 smithi063 systemd[1]: Starting Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:35:55.820 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T11:35:55.821 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-03-23T11:35:55.837 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T11:35:55.837 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-03-23T11:35:55.837 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-03-23T11:35:55.837 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-03-23T11:35:55.838 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon to start... 2024-03-23T11:35:55.838 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon... 2024-03-23T11:35:56.080 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:55 smithi063 ceph-mon[34706]: mkfs 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:35:56.080 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:55 smithi063 ceph-mon[34706]: mon.smithi063 is new leader, mons smithi063 in quorum (ranks 0) 2024-03-23T11:35:56.080 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:55 smithi063 bash[34668]: d49f9ca465563685cd127e4d60dec8e0ac5bb08b7c556c8221ca722221b4860d 2024-03-23T11:35:56.080 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:55 smithi063 systemd[1]: Started Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:35:57.006 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout cluster: 2024-03-23T11:35:57.014 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout id: 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:35:57.014 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-23T11:35:57.014 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:35:57.014 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout services: 2024-03-23T11:35:57.014 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi063 (age 0.715365s) 2024-03-23T11:35:57.014 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout data: 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pgs: 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.389+0000 7efdbe000700 1 Processor -- start 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.389+0000 7efdbe000700 1 -- start start 2024-03-23T11:35:57.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.389+0000 7efdbe000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.389+0000 7efdbe000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb8079160 con 0x7efdb8078d50 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.390+0000 7efdb7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.390+0000 7efdb7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:32798/0 (socket says 172.21.15.63:32798) 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.390+0000 7efdb7400700 1 -- 172.21.15.63:0/3600425639 learned_addr learned my addr 172.21.15.63:0/3600425639 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.390+0000 7efdb7400700 1 -- 172.21.15.63:0/3600425639 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb80777a0 con 0x7efdb8078d50 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.391+0000 7efdb7400700 1 --2- 172.21.15.63:0/3600425639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8077260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efda80098e0 tx=0x7efda8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf5f057d91532db server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.391+0000 7efdb6000700 1 -- 172.21.15.63:0/3600425639 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda800b860 con 0x7efdb8078d50 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.391+0000 7efdb6000700 1 -- 172.21.15.63:0/3600425639 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda800b9c0 con 0x7efdb8078d50 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.391+0000 7efdb6000700 1 -- 172.21.15.63:0/3600425639 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda800bc90 con 0x7efdb8078d50 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.391+0000 7efdbe000700 1 -- 172.21.15.63:0/3600425639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 msgr2=0x7efdb8077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.391+0000 7efdbe000700 1 --2- 172.21.15.63:0/3600425639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8077260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efda80098e0 tx=0x7efda8009250 comp rx=0 tx=0).stop 2024-03-23T11:35:57.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.392+0000 7efdbe000700 1 -- 172.21.15.63:0/3600425639 shutdown_connections 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.392+0000 7efdbe000700 1 --2- 172.21.15.63:0/3600425639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8077260 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.392+0000 7efdbe000700 1 -- 172.21.15.63:0/3600425639 >> 172.21.15.63:0/3600425639 conn(0x7efdb80fdd00 msgr2=0x7efdb8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.392+0000 7efdbe000700 1 -- 172.21.15.63:0/3600425639 shutdown_connections 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.392+0000 7efdbe000700 1 -- 172.21.15.63:0/3600425639 wait complete. 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.393+0000 7efdbe000700 1 Processor -- start 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.393+0000 7efdbe000700 1 -- start start 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.393+0000 7efdbe000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8112e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.393+0000 7efdbe000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb8113360 con 0x7efdb8078d50 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.394+0000 7efdb7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8112e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.394+0000 7efdb7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8112e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:32806/0 (socket says 172.21.15.63:32806) 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.394+0000 7efdb7400700 1 -- 172.21.15.63:0/1945981467 learned_addr learned my addr 172.21.15.63:0/1945981467 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:35:57.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.394+0000 7efdb7400700 1 -- 172.21.15.63:0/1945981467 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda80094f0 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.394+0000 7efdb7400700 1 --2- 172.21.15.63:0/1945981467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8112e20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efda8009290 tx=0x7efda8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.394+0000 7efdaf400700 1 -- 172.21.15.63:0/1945981467 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda801c5c0 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.394+0000 7efdaf400700 1 -- 172.21.15.63:0/1945981467 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda801c740 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.395+0000 7efdaf400700 1 -- 172.21.15.63:0/1945981467 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda801ca50 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.395+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb8113560 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.395+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb8113f90 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.395+0000 7efdaf400700 1 -- 172.21.15.63:0/1945981467 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7efda801a450 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.395+0000 7efdaf400700 1 -- 172.21.15.63:0/1945981467 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efda802e790 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.396+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb805f5f0 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.399+0000 7efdaf400700 1 -- 172.21.15.63:0/1945981467 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7efda801ce30 con 0x7efdb8078d50 2024-03-23T11:35:57.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.474+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7efdb8114b00 con 0x7efdb8078d50 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.475+0000 7efdaf400700 1 -- 172.21.15.63:0/1945981467 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7efda801a890 con 0x7efdb8078d50 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.475+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 msgr2=0x7efdb8112e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.475+0000 7efdbe000700 1 --2- 172.21.15.63:0/1945981467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8112e20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efda8009290 tx=0x7efda8016000 comp rx=0 tx=0).stop 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.475+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 shutdown_connections 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.475+0000 7efdbe000700 1 --2- 172.21.15.63:0/1945981467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdb8078d50 0x7efdb8112e20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.475+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 >> 172.21.15.63:0/1945981467 conn(0x7efdb80fdd00 msgr2=0x7efdb8109b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.476+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 shutdown_connections 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:56.476+0000 7efdbe000700 1 -- 172.21.15.63:0/1945981467 wait complete. 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:mon is available 2024-03-23T11:35:57.019 INFO:teuthology.orchestra.run.smithi063.stdout:Assimilating anything we can from ceph.conf... 2024-03-23T11:35:57.020 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:56 smithi063 ceph-mon[34706]: mon.smithi063 is new leader, mons smithi063 in quorum (ranks 0) 2024-03-23T11:35:57.020 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:56 smithi063 ceph-mon[34706]: monmap e1: 1 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:35:57.020 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:56 smithi063 ceph-mon[34706]: fsmap 2024-03-23T11:35:57.020 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:56 smithi063 ceph-mon[34706]: osdmap e1: 0 total, 0 up, 0 in 2024-03-23T11:35:57.020 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:56 smithi063 ceph-mon[34706]: mgrmap e1: no daemons active 2024-03-23T11:35:57.020 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:56 smithi063 ceph-mon[34706]: from='client.? 172.21.15.63:0/1945981467' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-23T11:35:57.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:57 smithi063 ceph-mon[34706]: from='client.? 172.21.15.63:0/2240444279' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-23T11:35:57.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:57 smithi063 ceph-mon[34706]: from='client.? 172.21.15.63:0/2240444279' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-23T11:35:58.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:35:58.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.63:3300,v1:172.21.15.63:6789] 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-23T11:35:58.143 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.535+0000 7f587ac00700 1 Processor -- start 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.535+0000 7f587ac00700 1 -- start start 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.536+0000 7f587ac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874077330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.536+0000 7f587ac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874079190 con 0x7f5874078d80 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.536+0000 7f5873e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874077330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.536+0000 7f5873e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874077330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:32818/0 (socket says 172.21.15.63:32818) 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.536+0000 7f5873e00700 1 -- 172.21.15.63:0/4189408398 learned_addr learned my addr 172.21.15.63:0/4189408398 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.536+0000 7f5873e00700 1 -- 172.21.15.63:0/4189408398 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874077870 con 0x7f5874078d80 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f5873e00700 1 --2- 172.21.15.63:0/4189408398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874077330 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f58640098e0 tx=0x7f5864009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff160e8205d0d104 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f5872a00700 1 -- 172.21.15.63:0/4189408398 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f586400b860 con 0x7f5874078d80 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f5872a00700 1 -- 172.21.15.63:0/4189408398 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586400b9c0 con 0x7f5874078d80 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f5872a00700 1 -- 172.21.15.63:0/4189408398 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f586400bc90 con 0x7f5874078d80 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f587ac00700 1 -- 172.21.15.63:0/4189408398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 msgr2=0x7f5874077330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:35:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f587ac00700 1 --2- 172.21.15.63:0/4189408398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874077330 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f58640098e0 tx=0x7f5864009250 comp rx=0 tx=0).stop 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f587ac00700 1 -- 172.21.15.63:0/4189408398 shutdown_connections 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f587ac00700 1 --2- 172.21.15.63:0/4189408398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874077330 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.537+0000 7f587ac00700 1 -- 172.21.15.63:0/4189408398 >> 172.21.15.63:0/4189408398 conn(0x7f58740fdd40 msgr2=0x7f5874100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.538+0000 7f587ac00700 1 -- 172.21.15.63:0/4189408398 shutdown_connections 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.538+0000 7f587ac00700 1 -- 172.21.15.63:0/4189408398 wait complete. 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.538+0000 7f587ac00700 1 Processor -- start 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.539+0000 7f587ac00700 1 -- start start 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.539+0000 7f587ac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.539+0000 7f587ac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874113330 con 0x7f5874078d80 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.539+0000 7f5873e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.539+0000 7f5873e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:32830/0 (socket says 172.21.15.63:32830) 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.539+0000 7f5873e00700 1 -- 172.21.15.63:0/2240444279 learned_addr learned my addr 172.21.15.63:0/2240444279 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.539+0000 7f5873e00700 1 -- 172.21.15.63:0/2240444279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58640094f0 con 0x7f5874078d80 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f5873e00700 1 --2- 172.21.15.63:0/2240444279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874112df0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5864009290 tx=0x7f5864016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:35:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f586401c5c0 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586401c740 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f586401ca50 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5874113530 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5874113f60 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f586401a450 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.540+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f586402e790 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.541+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587405f5f0 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.544+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f586401ce30 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.617+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f587410cab0 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.621+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f586401b3e0 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.621+0000 7f586be00700 1 -- 172.21.15.63:0/2240444279 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f586402e950 con 0x7f5874078d80 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.622+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 msgr2=0x7f5874112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.622+0000 7f587ac00700 1 --2- 172.21.15.63:0/2240444279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874112df0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5864009290 tx=0x7f5864016000 comp rx=0 tx=0).stop 2024-03-23T11:35:58.146 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.622+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 shutdown_connections 2024-03-23T11:35:58.147 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.622+0000 7f587ac00700 1 --2- 172.21.15.63:0/2240444279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874078d80 0x7f5874112df0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:35:58.147 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.622+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 >> 172.21.15.63:0/2240444279 conn(0x7f58740fdd40 msgr2=0x7f5874109c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:35:58.147 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.622+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 shutdown_connections 2024-03-23T11:35:58.147 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:57.622+0000 7f587ac00700 1 -- 172.21.15.63:0/2240444279 wait complete. 2024-03-23T11:35:58.147 INFO:teuthology.orchestra.run.smithi063.stdout:Generating new minimal ceph.conf... 2024-03-23T11:35:59.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:58 smithi063 ceph-mon[34706]: from='client.? 172.21.15.63:0/2157642071' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.690+0000 7f3401a00700 1 Processor -- start 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.691+0000 7f3401a00700 1 -- start start 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.691+0000 7f3401a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.691+0000 7f3401a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc102ed0 con 0x7f33fc102580 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.691+0000 7f33faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.691+0000 7f33faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:32846/0 (socket says 172.21.15.63:32846) 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.691+0000 7f33faa00700 1 -- 172.21.15.63:0/1006915141 learned_addr learned my addr 172.21.15.63:0/1006915141 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.692+0000 7f33faa00700 1 -- 172.21.15.63:0/1006915141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc103010 con 0x7f33fc102580 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.692+0000 7f33faa00700 1 --2- 172.21.15.63:0/1006915141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc102990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0098e0 tx=0x7f33ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8719ce6a42183ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.692+0000 7f33f9600700 1 -- 172.21.15.63:0/1006915141 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec00b860 con 0x7f33fc102580 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.692+0000 7f33f9600700 1 -- 172.21.15.63:0/1006915141 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f33ec00be30 con 0x7f33fc102580 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.692+0000 7f33f9600700 1 -- 172.21.15.63:0/1006915141 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec010380 con 0x7f33fc102580 2024-03-23T11:35:59.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.693+0000 7f3401a00700 1 -- 172.21.15.63:0/1006915141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 msgr2=0x7f33fc102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.693+0000 7f3401a00700 1 --2- 172.21.15.63:0/1006915141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc102990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0098e0 tx=0x7f33ec009250 comp rx=0 tx=0).stop 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.693+0000 7f3401a00700 1 -- 172.21.15.63:0/1006915141 shutdown_connections 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.693+0000 7f3401a00700 1 --2- 172.21.15.63:0/1006915141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc102990 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.693+0000 7f3401a00700 1 -- 172.21.15.63:0/1006915141 >> 172.21.15.63:0/1006915141 conn(0x7f33fc0fdd00 msgr2=0x7f33fc100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.693+0000 7f3401a00700 1 -- 172.21.15.63:0/1006915141 shutdown_connections 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.693+0000 7f3401a00700 1 -- 172.21.15.63:0/1006915141 wait complete. 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.694+0000 7f3401a00700 1 Processor -- start 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.694+0000 7f3401a00700 1 -- start start 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.694+0000 7f3401a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.694+0000 7f3401a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc113a80 con 0x7f33fc102580 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:32858/0 (socket says 172.21.15.63:32858) 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33faa00700 1 -- 172.21.15.63:0/2157642071 learned_addr learned my addr 172.21.15.63:0/2157642071 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:35:59.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33faa00700 1 -- 172.21.15.63:0/2157642071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec0094f0 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33faa00700 1 --2- 172.21.15.63:0/2157642071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc113540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f33ec000c00 tx=0x7f33ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33f2a00700 1 -- 172.21.15.63:0/2157642071 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec010a80 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33f2a00700 1 -- 172.21.15.63:0/2157642071 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f33ec01a430 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f33f2a00700 1 -- 172.21.15.63:0/2157642071 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec0245b0 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.695+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc113c80 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.696+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc1147f0 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.696+0000 7f33f2a00700 1 -- 172.21.15.63:0/2157642071 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f33ec024a20 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.696+0000 7f33f2a00700 1 -- 172.21.15.63:0/2157642071 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f33ec02eb00 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.697+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc05f5f0 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.699+0000 7f33f2a00700 1 -- 172.21.15.63:0/2157642071 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f33ec01a960 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.773+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f33fc104390 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.773+0000 7f33f2a00700 1 -- 172.21.15.63:0/2157642071 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f33ec02ecc0 con 0x7f33fc102580 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.774+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 msgr2=0x7f33fc113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.774+0000 7f3401a00700 1 --2- 172.21.15.63:0/2157642071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc113540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f33ec000c00 tx=0x7f33ec009250 comp rx=0 tx=0).stop 2024-03-23T11:35:59.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.774+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 shutdown_connections 2024-03-23T11:35:59.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.774+0000 7f3401a00700 1 --2- 172.21.15.63:0/2157642071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc102580 0x7f33fc113540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:35:59.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.775+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 >> 172.21.15.63:0/2157642071 conn(0x7f33fc0fdd00 msgr2=0x7f33fc0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:35:59.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.775+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 shutdown_connections 2024-03-23T11:35:59.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:35:58.775+0000 7f3401a00700 1 -- 172.21.15.63:0/2157642071 wait complete. 2024-03-23T11:35:59.358 INFO:teuthology.orchestra.run.smithi063.stdout:Restarting the monitor... 2024-03-23T11:35:59.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:59 smithi063 systemd[1]: Stopping Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:35:59.920 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:59 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[34683]: 2024-03-23T11:35:59.675+0000 7f87cd800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi063 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:35:59.920 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:35:59 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[34683]: 2024-03-23T11:35:59.675+0000 7f87cd800700 -1 mon.smithi063@0(leader) e1 *** Got Signal Terminated *** 2024-03-23T11:36:00.249 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:00 smithi063 bash[35032]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063 2024-03-23T11:36:00.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:00 smithi063 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063.service: Succeeded. 2024-03-23T11:36:00.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:00 smithi063 systemd[1]: Stopped Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:36:00.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:00 smithi063 systemd[1]: Starting Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:36:01.006 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: set uid:gid to 167:167 (ceph:ceph) 2024-03-23T11:36:01.069 INFO:teuthology.orchestra.run.smithi063.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-23T11:36:01.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable), process ceph-mon, pid 7 2024-03-23T11:36:01.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: pidfile_write: ignore empty --pid-file 2024-03-23T11:36:01.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: load: jerasure load: lrc 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: RocksDB version: 6.15.5 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Compile date Mar 21 2024 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: DB SUMMARY 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: DB Session ID: 3XU085YE8GHYOMEL05OG 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: CURRENT file: CURRENT 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: IDENTITY file: IDENTITY 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi063/store.db dir, Total Num: 1, files: 000008.sst 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi063/store.db: 000010.log size: 96760 ; 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.error_if_exists: 0 2024-03-23T11:36:01.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.create_if_missing: 0 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.paranoid_checks: 1 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.env: 0x55f0286e3a80 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.fs: Posix File System 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.info_log: 0x55f029ebfc00 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_file_opening_threads: 16 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.statistics: (nil) 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.use_fsync: 0 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_log_file_size: 0 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-23T11:36:01.389 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.keep_log_file_num: 1000 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.recycle_log_file_num: 0 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.allow_fallocate: 1 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.allow_mmap_reads: 0 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.allow_mmap_writes: 0 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.use_direct_reads: 0 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.create_missing_column_families: 0 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.db_log_dir: 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi063/store.db 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-23T11:36:01.390 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.advise_random_on_open: 1 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.db_write_buffer_size: 0 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.write_buffer_manager: 0x55f02ac34ab0 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-23T11:36:01.392 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.rate_limiter: (nil) 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.wal_recovery_mode: 2 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.enable_thread_tracking: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.enable_pipelined_write: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.unordered_write: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.row_cache: None 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.wal_filter: None 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.allow_ingest_behind: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.preserve_deletes: 0 2024-03-23T11:36:01.393 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.two_write_queues: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.manual_wal_flush: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.atomic_flush: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.log_readahead_size: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.best_efforts_recovery: 0 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-23T11:36:01.394 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.allow_data_in_errors: 0 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.db_host_id: __hostname__ 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_background_jobs: 2 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_background_compactions: -1 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_subcompactions: 1 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_total_wal_size: 0 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_open_files: -1 2024-03-23T11:36:01.395 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bytes_per_sync: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_readahead_size: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_background_flushes: -1 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Compression algorithms supported: 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kZSTD supported: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kXpressCompression supported: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kLZ4HCCompression supported: 1 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kZlibCompression supported: 1 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kSnappyCompression supported: 1 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kLZ4Compression supported: 1 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: kBZip2Compression supported: 0 2024-03-23T11:36:01.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi063/store.db/MANIFEST-000009 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.merge_operator: 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_filter: None 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_filter_factory: None 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.sst_partitioner_factory: None 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f029f14f58) 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: cache_index_and_filter_blocks: 1 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: pin_top_level_index_and_filter: 1 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_type: 0 2024-03-23T11:36:01.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: data_block_index_type: 0 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_shortening: 1 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: hash_index_allow_collision: 1 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: checksum: 1 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: no_block_cache: 0 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache: 0x55f029f47090 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_name: BinnedLRUCache 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_options: 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: capacity : 536870912 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: num_shard_bits : 4 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: strict_capacity_limit : 0 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: high_pri_pool_ratio: 0.000 2024-03-23T11:36:01.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_compressed: (nil) 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: persistent_cache: (nil) 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_size: 4096 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_size_deviation: 10 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_restart_interval: 16 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_block_restart_interval: 1 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: metadata_block_size: 4096 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: partition_filters: 0 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: use_delta_encoding: 1 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: whole_key_filtering: 1 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: verify_compression: 0 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: read_amp_bytes_per_bit: 0 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: format_version: 4 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: enable_index_compression: 1 2024-03-23T11:36:01.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_align: 0 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.write_buffer_size: 33554432 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_write_buffer_number: 2 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression: NoCompression 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression: Disabled 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.prefix_extractor: nullptr 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.num_levels: 7 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-23T11:36:01.400 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression_opts.level: 32767 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression_opts.strategy: 0 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compression_opts.enabled: false 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.target_file_size_base: 67108864 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-23T11:36:01.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.arena_block_size: 4194304 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-23T11:36:01.402 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.disable_auto_compactions: 0 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.table_properties_collectors: 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.inplace_update_support: 0 2024-03-23T11:36:01.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.bloom_locality: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.max_successive_merges: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.paranoid_file_checks: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.force_consistency_checks: 1 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.report_bg_io_stats: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.ttl: 2592000 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.enable_blob_files: false 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.min_blob_size: 0 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.blob_file_size: 268435456 2024-03-23T11:36:01.404 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi063/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711193761027479, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711193761029000, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92129, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90224, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82963, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711193761, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "79d1dda2-776a-437f-a2c3-60ff8d040140", "db_session_id": "3XU085YE8GHYOMEL05OG"}} 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711193761029741, "job": 1, "event": "recovery_finished"} 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi063/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55f029edb180 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: DB pointer 0x55f029fb0000 2024-03-23T11:36:01.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** DB Stats ** 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T11:36:01.406 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.3 0.00 0.00 1 0.001 0 0 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.3 0.00 0.00 1 0.001 0 0 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.3 0.00 0.00 1 0.001 0 0 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 87.3 0.00 0.00 1 0.001 0 0 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative compaction: 0.00 GB write, 13.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval compaction: 0.00 GB write, 13.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:36:01.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.3 0.00 0.00 1 0.001 0 0 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.3 0.00 0.00 1 0.001 0 0 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 87.3 0.00 0.00 1 0.001 0 0 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T11:36:01.408 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative compaction: 0.00 GB write, 13.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: starting mon.smithi063 rank 0 at public addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] at bind addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi063 fsid 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???) e1 preinit fsid 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???).mds e1 new map 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???).mds e1 print_map 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: e1 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:36:01.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: legacy client fscid: -1 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: No filesystems configured 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 bash[35152]: 16d320c90a943ddf92422012abd0475160e8835a3729466b97270e47fbf2caa0 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 systemd[1]: Started Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mon.smithi063 is new leader, mons smithi063 in quorum (ranks 0) 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: monmap e1: 1 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:36:01.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: fsmap 2024-03-23T11:36:01.411 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: osdmap e1: 0 total, 0 up, 0 in 2024-03-23T11:36:01.411 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:01 smithi063 ceph-mon[35189]: mgrmap e1: no daemons active 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.711+0000 7f62b9200700 1 Processor -- start 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.711+0000 7f62b9200700 1 -- start start 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.711+0000 7f62b9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.711+0000 7f62b9200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4106490 con 0x7f62b4105ab0 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.712+0000 7f62b3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.712+0000 7f62b3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4105ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51388/0 (socket says 172.21.15.63:51388) 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.712+0000 7f62b3400700 1 -- 172.21.15.63:0/34850767 learned_addr learned my addr 172.21.15.63:0/34850767 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.712+0000 7f62b3400700 1 -- 172.21.15.63:0/34850767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b41065d0 con 0x7f62b4105ab0 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.713+0000 7f62b3400700 1 --2- 172.21.15.63:0/34850767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4105ec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f62a40098e0 tx=0x7f62a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dacc9c73f342cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:02.269 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.713+0000 7f62b2000700 1 -- 172.21.15.63:0/34850767 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a400b860 con 0x7f62b4105ab0 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.713+0000 7f62b2000700 1 -- 172.21.15.63:0/34850767 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f62a400be30 con 0x7f62b4105ab0 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.713+0000 7f62b2000700 1 -- 172.21.15.63:0/34850767 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a4010380 con 0x7f62b4105ab0 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.714+0000 7f62b9200700 1 -- 172.21.15.63:0/34850767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 msgr2=0x7f62b4105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.714+0000 7f62b9200700 1 --2- 172.21.15.63:0/34850767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4105ec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f62a40098e0 tx=0x7f62a4009250 comp rx=0 tx=0).stop 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.714+0000 7f62b9200700 1 -- 172.21.15.63:0/34850767 shutdown_connections 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.714+0000 7f62b9200700 1 --2- 172.21.15.63:0/34850767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4105ec0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.714+0000 7f62b9200700 1 -- 172.21.15.63:0/34850767 >> 172.21.15.63:0/34850767 conn(0x7f62b4077d60 msgr2=0x7f62b4078180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.714+0000 7f62b9200700 1 -- 172.21.15.63:0/34850767 shutdown_connections 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.714+0000 7f62b9200700 1 -- 172.21.15.63:0/34850767 wait complete. 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.715+0000 7f62b9200700 1 Processor -- start 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.715+0000 7f62b9200700 1 -- start start 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.715+0000 7f62b9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.715+0000 7f62b9200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4113940 con 0x7f62b4105ab0 2024-03-23T11:36:02.270 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.716+0000 7f62b3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.716+0000 7f62b3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51396/0 (socket says 172.21.15.63:51396) 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.716+0000 7f62b3400700 1 -- 172.21.15.63:0/584488592 learned_addr learned my addr 172.21.15.63:0/584488592 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.716+0000 7f62b3400700 1 -- 172.21.15.63:0/584488592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a40094f0 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.716+0000 7f62b3400700 1 --2- 172.21.15.63:0/584488592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4113400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f62a4000c00 tx=0x7f62a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.717+0000 7f62ab400700 1 -- 172.21.15.63:0/584488592 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a4010960 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.717+0000 7f62ab400700 1 -- 172.21.15.63:0/584488592 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f62a4024440 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.717+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b4113b40 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.717+0000 7f62ab400700 1 -- 172.21.15.63:0/584488592 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a401a5f0 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.717+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b41146b0 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.717+0000 7f62ab400700 1 -- 172.21.15.63:0/584488592 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f62a401aaa0 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.717+0000 7f62ab400700 1 -- 172.21.15.63:0/584488592 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f62a402da40 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.718+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b405f5f0 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.721+0000 7f62ab400700 1 -- 172.21.15.63:0/584488592 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f62a4024950 con 0x7f62b4105ab0 2024-03-23T11:36:02.271 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.793+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f62b4105130 con 0x7f62b4105ab0 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.795+0000 7f62ab400700 1 -- 172.21.15.63:0/584488592 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f62a40245b0 con 0x7f62b4105ab0 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.796+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 msgr2=0x7f62b4113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.796+0000 7f62b9200700 1 --2- 172.21.15.63:0/584488592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4113400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f62a4000c00 tx=0x7f62a4009250 comp rx=0 tx=0).stop 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.796+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 shutdown_connections 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.796+0000 7f62b9200700 1 --2- 172.21.15.63:0/584488592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62b4105ab0 0x7f62b4113400 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.796+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 >> 172.21.15.63:0/584488592 conn(0x7f62b4077d60 msgr2=0x7f62b4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.796+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 shutdown_connections 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:01.797+0000 7f62b9200700 1 -- 172.21.15.63:0/584488592 wait complete. 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mgr... 2024-03-23T11:36:02.272 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying port 9283 ... 2024-03-23T11:36:02.545 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mgr.smithi063.gehkcr 2024-03-23T11:36:02.546 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mgr.smithi063.gehkcr.service: Unit ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mgr.smithi063.gehkcr.service not loaded. 2024-03-23T11:36:02.737 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6880ffb6-e909-11ee-95cd-87774f69a715.target.wants/ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mgr.smithi063.gehkcr.service → /etc/systemd/system/ceph-6880ffb6-e909-11ee-95cd-87774f69a715@.service. 2024-03-23T11:36:02.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:02 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/584488592' entity='client.admin' 2024-03-23T11:36:03.407 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T11:36:03.407 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-03-23T11:36:03.426 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T11:36:03.426 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-03-23T11:36:03.426 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-03-23T11:36:03.426 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-03-23T11:36:03.446 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T11:36:03.446 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-03-23T11:36:03.465 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T11:36:03.465 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-03-23T11:36:03.465 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-03-23T11:36:03.466 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-23T11:36:03.466 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr to start... 2024-03-23T11:36:03.466 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr... 2024-03-23T11:36:04.442 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:04 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3688479987' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-23T11:36:04.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "6880ffb6-e909-11ee-95cd-87774f69a715", 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-03-23T11:36:04.735 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "smithi063" 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-23T11:36:04.736 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-23T11:36:04.743 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-23T11:36:04.743 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-23T11:36:04.743 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-23T11:36:04.743 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:04.743 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-23T11:36:04.744 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-03-23T11:35:55.762408+0000", 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T11:36:04.745 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.062+0000 7fe895400700 1 Processor -- start 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.063+0000 7fe895400700 1 -- start start 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.063+0000 7fe895400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe8901051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.063+0000 7fe895400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8901056f0 con 0x7fe890076c80 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.064+0000 7fe88e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe8901051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.064+0000 7fe88e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe8901051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51434/0 (socket says 172.21.15.63:51434) 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.064+0000 7fe88e800700 1 -- 172.21.15.63:0/4146682469 learned_addr learned my addr 172.21.15.63:0/4146682469 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.064+0000 7fe88e800700 1 -- 172.21.15.63:0/4146682469 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890105830 con 0x7fe890076c80 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.064+0000 7fe88e800700 1 --2- 172.21.15.63:0/4146682469 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe8901051b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8800098e0 tx=0x7fe880009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e086a4ad4d33d973 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe88d400700 1 -- 172.21.15.63:0/4146682469 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe88000b860 con 0x7fe890076c80 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe88d400700 1 -- 172.21.15.63:0/4146682469 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe88000be30 con 0x7fe890076c80 2024-03-23T11:36:04.746 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe88d400700 1 -- 172.21.15.63:0/4146682469 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe880010380 con 0x7fe890076c80 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe895400700 1 -- 172.21.15.63:0/4146682469 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 msgr2=0x7fe8901051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe895400700 1 --2- 172.21.15.63:0/4146682469 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe8901051b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8800098e0 tx=0x7fe880009250 comp rx=0 tx=0).stop 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe895400700 1 -- 172.21.15.63:0/4146682469 shutdown_connections 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe895400700 1 --2- 172.21.15.63:0/4146682469 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe8901051b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe895400700 1 -- 172.21.15.63:0/4146682469 >> 172.21.15.63:0/4146682469 conn(0x7fe8900fcc60 msgr2=0x7fe8900ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.065+0000 7fe895400700 1 -- 172.21.15.63:0/4146682469 shutdown_connections 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.066+0000 7fe895400700 1 -- 172.21.15.63:0/4146682469 wait complete. 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.066+0000 7fe895400700 1 Processor -- start 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.066+0000 7fe895400700 1 -- start start 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.067+0000 7fe895400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe890102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.067+0000 7fe895400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe890103040 con 0x7fe890076c80 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.067+0000 7fe88e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe890102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.067+0000 7fe88e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe890102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51438/0 (socket says 172.21.15.63:51438) 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.067+0000 7fe88e800700 1 -- 172.21.15.63:0/3688479987 learned_addr learned my addr 172.21.15.63:0/3688479987 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:04.747 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.067+0000 7fe88e800700 1 -- 172.21.15.63:0/3688479987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8800094f0 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.068+0000 7fe88e800700 1 --2- 172.21.15.63:0/3688479987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe890102b00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe880006b00 tx=0x7fe880009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.068+0000 7fe886a00700 1 -- 172.21.15.63:0/3688479987 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe880010a80 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.068+0000 7fe886a00700 1 -- 172.21.15.63:0/3688479987 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe88001a430 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.068+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8901010b0 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.068+0000 7fe886a00700 1 -- 172.21.15.63:0/3688479987 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8800245b0 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.068+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8901015a0 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.068+0000 7fe886a00700 1 -- 172.21.15.63:0/3688479987 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe880024a60 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.069+0000 7fe886a00700 1 -- 172.21.15.63:0/3688479987 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe88002eaa0 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.069+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89004bed0 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.072+0000 7fe886a00700 1 -- 172.21.15.63:0/3688479987 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe88002ed00 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.148+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe890106750 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.149+0000 7fe886a00700 1 -- 172.21.15.63:0/3688479987 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fe880010be0 con 0x7fe890076c80 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.150+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 msgr2=0x7fe890102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.150+0000 7fe895400700 1 --2- 172.21.15.63:0/3688479987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe890102b00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe880006b00 tx=0x7fe880009250 comp rx=0 tx=0).stop 2024-03-23T11:36:04.748 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.150+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 shutdown_connections 2024-03-23T11:36:04.749 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.150+0000 7fe895400700 1 --2- 172.21.15.63:0/3688479987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe890076c80 0x7fe890102b00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:04.749 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.150+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 >> 172.21.15.63:0/3688479987 conn(0x7fe8900fcc60 msgr2=0x7fe8900fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:04.749 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.150+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 shutdown_connections 2024-03-23T11:36:04.749 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:04.150+0000 7fe895400700 1 -- 172.21.15.63:0/3688479987 wait complete. 2024-03-23T11:36:04.749 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (1/15)... 2024-03-23T11:36:12.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: Activating manager daemon smithi063.gehkcr 2024-03-23T11:36:12.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: mgrmap e2: smithi063.gehkcr(active, starting, since 0.00444239s) 2024-03-23T11:36:12.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:36:12.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:36:12.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:36:12.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:36:12.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:36:12.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:36:12.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:36:12.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:36:12.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:12.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:12.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:11 smithi063 ceph-mon[35189]: from='mgr.14100 172.21.15.63:0/3351075180' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:14.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:13 smithi063 ceph-mon[35189]: mgrmap e3: smithi063.gehkcr(active, since 1.00942s) 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "6880ffb6-e909-11ee-95cd-87774f69a715", 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-23T11:36:15.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "smithi063" 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-23T11:36:15.055 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-23T11:36:15.056 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:15.056 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-23T11:36:15.056 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:15.056 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-23T11:36:15.056 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-23T11:36:15.056 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-23T11:36:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-23T11:36:15.061 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-03-23T11:35:55.762408+0000", 2024-03-23T11:36:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-23T11:36:15.063 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:14 smithi063 ceph-mon[35189]: mgrmap e4: smithi063.gehkcr(active, since 2s) 2024-03-23T11:36:15.063 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:14 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3917206961' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-23T11:36:15.068 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-03-23T11:36:15.068 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-23T11:36:15.068 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-03-23T11:36:15.068 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.171+0000 7fee77e00700 1 Processor -- start 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.171+0000 7fee77e00700 1 -- start start 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.171+0000 7fee77e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.171+0000 7fee77e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee70102ed0 con 0x7fee70102580 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.171+0000 7fee75400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.172+0000 7fee75400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45540/0 (socket says 172.21.15.63:45540) 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.172+0000 7fee75400700 1 -- 172.21.15.63:0/3225672266 learned_addr learned my addr 172.21.15.63:0/3225672266 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.172+0000 7fee75400700 1 -- 172.21.15.63:0/3225672266 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee70103010 con 0x7fee70102580 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.172+0000 7fee75400700 1 --2- 172.21.15.63:0/3225672266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70102990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee640098e0 tx=0x7fee64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e10bfe52322124 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.172+0000 7fee6fe00700 1 -- 172.21.15.63:0/3225672266 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee6400b860 con 0x7fee70102580 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.172+0000 7fee6fe00700 1 -- 172.21.15.63:0/3225672266 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fee6400be30 con 0x7fee70102580 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee6fe00700 1 -- 172.21.15.63:0/3225672266 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee64010380 con 0x7fee70102580 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee77e00700 1 -- 172.21.15.63:0/3225672266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 msgr2=0x7fee70102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee77e00700 1 --2- 172.21.15.63:0/3225672266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70102990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee640098e0 tx=0x7fee64009250 comp rx=0 tx=0).stop 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee77e00700 1 -- 172.21.15.63:0/3225672266 shutdown_connections 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee77e00700 1 --2- 172.21.15.63:0/3225672266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70102990 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee77e00700 1 -- 172.21.15.63:0/3225672266 >> 172.21.15.63:0/3225672266 conn(0x7fee700fdd00 msgr2=0x7fee70100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee77e00700 1 -- 172.21.15.63:0/3225672266 shutdown_connections 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.173+0000 7fee77e00700 1 -- 172.21.15.63:0/3225672266 wait complete. 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.174+0000 7fee77e00700 1 Processor -- start 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.174+0000 7fee77e00700 1 -- start start 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.175+0000 7fee77e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70111340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.175+0000 7fee77e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee70111880 con 0x7fee70102580 2024-03-23T11:36:15.070 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.175+0000 7fee75400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70111340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.175+0000 7fee75400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70111340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45542/0 (socket says 172.21.15.63:45542) 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.175+0000 7fee75400700 1 -- 172.21.15.63:0/3917206961 learned_addr learned my addr 172.21.15.63:0/3917206961 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.175+0000 7fee75400700 1 -- 172.21.15.63:0/3917206961 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640094f0 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.175+0000 7fee75400700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70111340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee64006b00 tx=0x7fee64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.176+0000 7fee6d600700 1 -- 172.21.15.63:0/3917206961 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee640109b0 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.176+0000 7fee6d600700 1 -- 172.21.15.63:0/3917206961 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fee64024440 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.176+0000 7fee6d600700 1 -- 172.21.15.63:0/3917206961 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee6401a5f0 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.176+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee70111a80 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.176+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee701125f0 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.177+0000 7fee6d600700 1 -- 172.21.15.63:0/3917206961 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fee64024990 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.177+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7005f5f0 con 0x7fee70102580 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.177+0000 7fee6d600700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fee58036790 0x7fee58038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:15.071 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.177+0000 7fee6d600700 1 -- 172.21.15.63:0/3917206961 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fee6404a6f0 con 0x7fee70102580 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.178+0000 7fee74a00700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fee58036790 0x7fee58038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.178+0000 7fee74a00700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fee58036790 0x7fee58038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009990 tx=0x7fee5c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.183+0000 7fee6d600700 1 -- 172.21.15.63:0/3917206961 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fee64006d70 con 0x7fee70102580 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.469+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fee7004bed0 con 0x7fee70102580 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.470+0000 7fee6d600700 1 -- 172.21.15.63:0/3917206961 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fee640233a0 con 0x7fee70102580 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fee58036790 msgr2=0x7fee58038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fee58036790 0x7fee58038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009990 tx=0x7fee5c006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 msgr2=0x7fee70111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70111340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee64006b00 tx=0x7fee64009250 comp rx=0 tx=0).stop 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 shutdown_connections 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fee58036790 0x7fee58038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 --2- 172.21.15.63:0/3917206961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee70102580 0x7fee70111340 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:15.072 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 >> 172.21.15.63:0/3917206961 conn(0x7fee700fdd00 msgr2=0x7fee700fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:15.073 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 shutdown_connections 2024-03-23T11:36:15.073 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:14.472+0000 7fee77e00700 1 -- 172.21.15.63:0/3917206961 wait complete. 2024-03-23T11:36:15.073 INFO:teuthology.orchestra.run.smithi063.stdout:mgr is available 2024-03-23T11:36:16.102 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:15 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/431872367' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-23T11:36:16.415 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-03-23T11:36:16.416 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.597+0000 7fb619000700 1 Processor -- start 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.597+0000 7fb619000700 1 -- start start 2024-03-23T11:36:16.417 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.597+0000 7fb619000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:16.418 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.597+0000 7fb619000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614102d20 con 0x7fb6141023d0 2024-03-23T11:36:16.418 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.597+0000 7fb613400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:16.418 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.597+0000 7fb613400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141027e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45548/0 (socket says 172.21.15.63:45548) 2024-03-23T11:36:16.418 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.597+0000 7fb613400700 1 -- 172.21.15.63:0/1483452498 learned_addr learned my addr 172.21.15.63:0/1483452498 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:16.418 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.598+0000 7fb613400700 1 -- 172.21.15.63:0/1483452498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb614102e60 con 0x7fb6141023d0 2024-03-23T11:36:16.418 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.598+0000 7fb613400700 1 --2- 172.21.15.63:0/1483452498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141027e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb6040098e0 tx=0x7fb604009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68aa1d0b4e0ab9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:16.418 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.598+0000 7fb612000700 1 -- 172.21.15.63:0/1483452498 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60400b860 con 0x7fb6141023d0 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.598+0000 7fb612000700 1 -- 172.21.15.63:0/1483452498 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb60400be30 con 0x7fb6141023d0 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.598+0000 7fb612000700 1 -- 172.21.15.63:0/1483452498 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb604010380 con 0x7fb6141023d0 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.599+0000 7fb619000700 1 -- 172.21.15.63:0/1483452498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 msgr2=0x7fb6141027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.599+0000 7fb619000700 1 --2- 172.21.15.63:0/1483452498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141027e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb6040098e0 tx=0x7fb604009250 comp rx=0 tx=0).stop 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.599+0000 7fb619000700 1 -- 172.21.15.63:0/1483452498 shutdown_connections 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.599+0000 7fb619000700 1 --2- 172.21.15.63:0/1483452498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141027e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.599+0000 7fb619000700 1 -- 172.21.15.63:0/1483452498 >> 172.21.15.63:0/1483452498 conn(0x7fb6140fdb90 msgr2=0x7fb6140fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.599+0000 7fb619000700 1 -- 172.21.15.63:0/1483452498 shutdown_connections 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.599+0000 7fb619000700 1 -- 172.21.15.63:0/1483452498 wait complete. 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.600+0000 7fb619000700 1 Processor -- start 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.600+0000 7fb619000700 1 -- start start 2024-03-23T11:36:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.600+0000 7fb619000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.600+0000 7fb619000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614113900 con 0x7fb6141023d0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb613400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb613400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45556/0 (socket says 172.21.15.63:45556) 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb613400700 1 -- 172.21.15.63:0/431872367 learned_addr learned my addr 172.21.15.63:0/431872367 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb613400700 1 -- 172.21.15.63:0/431872367 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6040094f0 con 0x7fb6141023d0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb613400700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141133c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb604000c00 tx=0x7fb604009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb60b400700 1 -- 172.21.15.63:0/431872367 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb604010960 con 0x7fb6141023d0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb60b400700 1 -- 172.21.15.63:0/431872367 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb604024440 con 0x7fb6141023d0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb614113b00 con 0x7fb6141023d0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.601+0000 7fb60b400700 1 -- 172.21.15.63:0/431872367 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60401a5f0 con 0x7fb6141023d0 2024-03-23T11:36:16.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.602+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb614114670 con 0x7fb6141023d0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.602+0000 7fb60b400700 1 -- 172.21.15.63:0/431872367 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fb604024950 con 0x7fb6141023d0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.603+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61405f5f0 con 0x7fb6141023d0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.603+0000 7fb60b400700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fb5f8036790 0x7fb5f8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.603+0000 7fb60b400700 1 -- 172.21.15.63:0/431872367 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb60404a580 con 0x7fb6141023d0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.603+0000 7fb612a00700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fb5f8036790 0x7fb5f8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.604+0000 7fb612a00700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fb5f8036790 0x7fb5f8038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb600009990 tx=0x7fb600006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.608+0000 7fb60b400700 1 -- 172.21.15.63:0/431872367 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb6040245b0 con 0x7fb6141023d0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.811+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb61404bed0 con 0x7fb6141023d0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.812+0000 7fb60b400700 1 -- 172.21.15.63:0/431872367 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fb6040233a0 con 0x7fb6141023d0 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fb5f8036790 msgr2=0x7fb5f8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:16.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fb5f8036790 0x7fb5f8038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb600009990 tx=0x7fb600006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 msgr2=0x7fb6141133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141133c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb604000c00 tx=0x7fb604009250 comp rx=0 tx=0).stop 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 shutdown_connections 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fb5f8036790 0x7fb5f8038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 --2- 172.21.15.63:0/431872367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6141023d0 0x7fb6141133c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 >> 172.21.15.63:0/431872367 conn(0x7fb6140fdb90 msgr2=0x7fb6140fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.815+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 shutdown_connections 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:15.816+0000 7fb619000700 1 -- 172.21.15.63:0/431872367 wait complete. 2024-03-23T11:36:16.422 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling cephadm module... 2024-03-23T11:36:17.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:17 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3622923785' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-23T11:36:17.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.964+0000 7f0301600700 1 Processor -- start 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.964+0000 7f0301600700 1 -- start start 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.965+0000 7f0301600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.965+0000 7f0301600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc079170 con 0x7f02fc078d60 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.965+0000 7f02faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.965+0000 7f02faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45560/0 (socket says 172.21.15.63:45560) 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.965+0000 7f02faa00700 1 -- 172.21.15.63:0/4084613402 learned_addr learned my addr 172.21.15.63:0/4084613402 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.965+0000 7f02faa00700 1 -- 172.21.15.63:0/4084613402 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc0777a0 con 0x7f02fc078d60 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.966+0000 7f02faa00700 1 --2- 172.21.15.63:0/4084613402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc077260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0098e0 tx=0x7f02ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9d115a5b4bc56b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.966+0000 7f02f9600700 1 -- 172.21.15.63:0/4084613402 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec00b860 con 0x7f02fc078d60 2024-03-23T11:36:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.966+0000 7f02f9600700 1 -- 172.21.15.63:0/4084613402 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f02ec00be30 con 0x7f02fc078d60 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.966+0000 7f02f9600700 1 -- 172.21.15.63:0/4084613402 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec010380 con 0x7f02fc078d60 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.966+0000 7f0301600700 1 -- 172.21.15.63:0/4084613402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 msgr2=0x7f02fc077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.966+0000 7f0301600700 1 --2- 172.21.15.63:0/4084613402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc077260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0098e0 tx=0x7f02ec009250 comp rx=0 tx=0).stop 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.967+0000 7f0301600700 1 -- 172.21.15.63:0/4084613402 shutdown_connections 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.967+0000 7f0301600700 1 --2- 172.21.15.63:0/4084613402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc077260 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.967+0000 7f0301600700 1 -- 172.21.15.63:0/4084613402 >> 172.21.15.63:0/4084613402 conn(0x7f02fc0fdcd0 msgr2=0x7f02fc100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.967+0000 7f0301600700 1 -- 172.21.15.63:0/4084613402 shutdown_connections 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.967+0000 7f0301600700 1 -- 172.21.15.63:0/4084613402 wait complete. 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.968+0000 7f0301600700 1 Processor -- start 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.968+0000 7f0301600700 1 -- start start 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.968+0000 7f0301600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.968+0000 7f0301600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc113b70 con 0x7f02fc078d60 2024-03-23T11:36:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45570/0 (socket says 172.21.15.63:45570) 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02faa00700 1 -- 172.21.15.63:0/3622923785 learned_addr learned my addr 172.21.15.63:0/3622923785 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02faa00700 1 -- 172.21.15.63:0/3622923785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec0094f0 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02faa00700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc113630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f02ec006b00 tx=0x7f02ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec010750 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc113d70 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f02ec010d20 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.969+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02ec01acf0 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.970+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc1148e0 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.971+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f02ec01a5a0 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.970+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc05f5f0 con 0x7f02fc078d60 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.971+0000 7f02f2a00700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f02e0036740 0x7f02e0038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.971+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f02ec01e070 con 0x7f02fc078d60 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.971+0000 7f02fa000700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f02e0036740 0x7f02e0038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.972+0000 7f02fa000700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f02e0036740 0x7f02e0038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009990 tx=0x7f02e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:16.976+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f02ec029340 con 0x7f02fc078d60 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.234+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f02fc04bed0 con 0x7f02fc078d60 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.288+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f02ec02b960 con 0x7f02fc078d60 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.290+0000 7f02f2a00700 1 -- 172.21.15.63:0/3622923785 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f02ec029800 con 0x7f02fc078d60 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f02e0036740 msgr2=0x7f02e0038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f02e0036740 0x7f02e0038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009990 tx=0x7f02e8006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 msgr2=0x7f02fc113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc113630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f02ec006b00 tx=0x7f02ec009250 comp rx=0 tx=0).stop 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 shutdown_connections 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f02e0036740 0x7f02e0038bf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 --2- 172.21.15.63:0/3622923785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02fc078d60 0x7f02fc113630 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:17.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 >> 172.21.15.63:0/3622923785 conn(0x7f02fc0fdcd0 msgr2=0x7f02fc0fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:17.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 shutdown_connections 2024-03-23T11:36:17.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:17.292+0000 7f0301600700 1 -- 172.21.15.63:0/3622923785 wait complete. 2024-03-23T11:36:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:18 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3622923785' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-23T11:36:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:18 smithi063 ceph-mon[35189]: mgrmap e5: smithi063.gehkcr(active, since 5s) 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "smithi063.gehkcr", 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.445+0000 7fef48e00700 1 Processor -- start 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.445+0000 7fef48e00700 1 -- start start 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.445+0000 7fef48e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.445+0000 7fef48e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef441056f0 con 0x7fef44076c80 2024-03-23T11:36:19.368 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.446+0000 7fef41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.446+0000 7fef41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45596/0 (socket says 172.21.15.63:45596) 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.446+0000 7fef41e00700 1 -- 172.21.15.63:0/1172210108 learned_addr learned my addr 172.21.15.63:0/1172210108 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.446+0000 7fef41e00700 1 -- 172.21.15.63:0/1172210108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44105830 con 0x7fef44076c80 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef41e00700 1 --2- 172.21.15.63:0/1172210108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441051b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fef340098e0 tx=0x7fef34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999fbcd8a5d20ada server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef40a00700 1 -- 172.21.15.63:0/1172210108 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3400b860 con 0x7fef44076c80 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef40a00700 1 -- 172.21.15.63:0/1172210108 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fef3400be30 con 0x7fef44076c80 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef40a00700 1 -- 172.21.15.63:0/1172210108 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef34010380 con 0x7fef44076c80 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef48e00700 1 -- 172.21.15.63:0/1172210108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 msgr2=0x7fef441051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef48e00700 1 --2- 172.21.15.63:0/1172210108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441051b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fef340098e0 tx=0x7fef34009250 comp rx=0 tx=0).stop 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef48e00700 1 -- 172.21.15.63:0/1172210108 shutdown_connections 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef48e00700 1 --2- 172.21.15.63:0/1172210108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441051b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.447+0000 7fef48e00700 1 -- 172.21.15.63:0/1172210108 >> 172.21.15.63:0/1172210108 conn(0x7fef440fcc60 msgr2=0x7fef440ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.448+0000 7fef48e00700 1 -- 172.21.15.63:0/1172210108 shutdown_connections 2024-03-23T11:36:19.369 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.448+0000 7fef48e00700 1 -- 172.21.15.63:0/1172210108 wait complete. 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.448+0000 7fef48e00700 1 Processor -- start 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.449+0000 7fef48e00700 1 -- start start 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.449+0000 7fef48e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.449+0000 7fef48e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef44113b20 con 0x7fef44076c80 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.449+0000 7fef41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.449+0000 7fef41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45606/0 (socket says 172.21.15.63:45606) 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.449+0000 7fef41e00700 1 -- 172.21.15.63:0/2295227797 learned_addr learned my addr 172.21.15.63:0/2295227797 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.450+0000 7fef41e00700 1 -- 172.21.15.63:0/2295227797 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef340094f0 con 0x7fef44076c80 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.450+0000 7fef41e00700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441135e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fef34006b00 tx=0x7fef34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.450+0000 7fef3a000700 1 -- 172.21.15.63:0/2295227797 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef34010a80 con 0x7fef44076c80 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.450+0000 7fef3a000700 1 -- 172.21.15.63:0/2295227797 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fef3401a430 con 0x7fef44076c80 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.450+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef44113d20 con 0x7fef44076c80 2024-03-23T11:36:19.370 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.450+0000 7fef3a000700 1 -- 172.21.15.63:0/2295227797 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef340245b0 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.450+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef44114840 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.452+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4405f5f0 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.452+0000 7fef3a000700 1 -- 172.21.15.63:0/2295227797 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fef34010be0 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.452+0000 7fef3a000700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 0x7fef28038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.452+0000 7fef3a000700 1 -- 172.21.15.63:0/2295227797 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fef3404acc0 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.452+0000 7fef41400700 1 -- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 msgr2=0x7fef28038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1052733058 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.452+0000 7fef41400700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 0x7fef28038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.457+0000 7fef3a000700 1 -- 172.21.15.63:0/2295227797 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fef3401a5a0 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.653+0000 7fef41400700 1 -- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 msgr2=0x7fef28038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1052733058 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.653+0000 7fef41400700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 0x7fef28038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.715+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fef44114b80 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.716+0000 7fef3a000700 1 -- 172.21.15.63:0/2295227797 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fef34023b80 con 0x7fef44076c80 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 msgr2=0x7fef28038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:36:19.371 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 0x7fef28038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 msgr2=0x7fef441135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441135e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fef34006b00 tx=0x7fef34009250 comp rx=0 tx=0).stop 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 shutdown_connections 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7fef280367e0 0x7fef28038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 --2- 172.21.15.63:0/2295227797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef44076c80 0x7fef441135e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 >> 172.21.15.63:0/2295227797 conn(0x7fef440fcc60 msgr2=0x7fef440fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 shutdown_connections 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:18.718+0000 7fef48e00700 1 -- 172.21.15.63:0/2295227797 wait complete. 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-03-23T11:36:19.372 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 5... 2024-03-23T11:36:19.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:19 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2295227797' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-23T11:36:26.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: Active manager daemon smithi063.gehkcr restarted 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: Activating manager daemon smithi063.gehkcr 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: osdmap e2: 0 total, 0 up, 0 in 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: mgrmap e6: smithi063.gehkcr(active, starting, since 0.00386023s) 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:26.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:25 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:36:27.171 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:26 smithi063 ceph-mon[35189]: Found migration_current of "None". Setting to last migration. 2024-03-23T11:36:27.171 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:26 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:36:27.171 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:26 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:36:27.171 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:26 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:36:27.171 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:26 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:36:27.171 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:26 smithi063 ceph-mon[35189]: mgrmap e7: smithi063.gehkcr(active, since 1.00706s) 2024-03-23T11:36:27.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.959+0000 7f9a0d400700 1 Processor -- start 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.960+0000 7f9a0d400700 1 -- start start 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.960+0000 7f9a0d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a08077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.960+0000 7f9a0d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a08079160 con 0x7f9a08078d50 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.960+0000 7f9a06800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a08077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.960+0000 7f9a06800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a08077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45610/0 (socket says 172.21.15.63:45610) 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.960+0000 7f9a06800700 1 -- 172.21.15.63:0/374890044 learned_addr learned my addr 172.21.15.63:0/374890044 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.961+0000 7f9a06800700 1 -- 172.21.15.63:0/374890044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a080777a0 con 0x7f9a08078d50 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.961+0000 7f9a06800700 1 --2- 172.21.15.63:0/374890044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a08077260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f99f80098e0 tx=0x7f99f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3472862343bfa9dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.961+0000 7f9a05400700 1 -- 172.21.15.63:0/374890044 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f800b860 con 0x7f9a08078d50 2024-03-23T11:36:27.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.961+0000 7f9a05400700 1 -- 172.21.15.63:0/374890044 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f99f800be30 con 0x7f9a08078d50 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.961+0000 7f9a05400700 1 -- 172.21.15.63:0/374890044 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f8010380 con 0x7f9a08078d50 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.962+0000 7f9a0d400700 1 -- 172.21.15.63:0/374890044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 msgr2=0x7f9a08077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.962+0000 7f9a0d400700 1 --2- 172.21.15.63:0/374890044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a08077260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f99f80098e0 tx=0x7f99f8009250 comp rx=0 tx=0).stop 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.962+0000 7f9a0d400700 1 -- 172.21.15.63:0/374890044 shutdown_connections 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.962+0000 7f9a0d400700 1 --2- 172.21.15.63:0/374890044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a08077260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.962+0000 7f9a0d400700 1 -- 172.21.15.63:0/374890044 >> 172.21.15.63:0/374890044 conn(0x7f9a080fdd00 msgr2=0x7f9a08100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.962+0000 7f9a0d400700 1 -- 172.21.15.63:0/374890044 shutdown_connections 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.962+0000 7f9a0d400700 1 -- 172.21.15.63:0/374890044 wait complete. 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.963+0000 7f9a0d400700 1 Processor -- start 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.963+0000 7f9a0d400700 1 -- start start 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.964+0000 7f9a0d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a081135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.964+0000 7f9a0d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a08113ae0 con 0x7f9a08078d50 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.964+0000 7f9a06800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a081135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:27.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.964+0000 7f9a06800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a081135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45614/0 (socket says 172.21.15.63:45614) 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.964+0000 7f9a06800700 1 -- 172.21.15.63:0/2041608960 learned_addr learned my addr 172.21.15.63:0/2041608960 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.964+0000 7f9a06800700 1 -- 172.21.15.63:0/2041608960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80094f0 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.964+0000 7f9a06800700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a081135a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f99f8006fd0 tx=0x7f99f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.965+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f8010750 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.965+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f99f8010d20 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.965+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f801ada0 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.965+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a08113ce0 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.965+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a08114850 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.966+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f99f80108b0 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.966+0000 7f99fea00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.966+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 --> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99ec0393f0 con 0x7f99ec036830 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.966+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f99f8049cc0 con 0x7f9a08078d50 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.967+0000 7f9a05e00700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 msgr2=0x7f99ec038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1052733058 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:19.967+0000 7f9a05e00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:36:27.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:20.168+0000 7f9a05e00700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 msgr2=0x7f99ec038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1052733058 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:20.168+0000 7f9a05e00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:20.569+0000 7f9a05e00700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 msgr2=0x7f99ec038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1052733058 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:20.569+0000 7f9a05e00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:21.370+0000 7f9a05e00700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 msgr2=0x7f99ec038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1052733058 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:21.370+0000 7f9a05e00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:22.971+0000 7f9a05e00700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 msgr2=0x7f99ec038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1052733058 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:22.972+0000 7f9a05e00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:25.878+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f99f802a260 con 0x7f9a08078d50 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:25.879+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 msgr2=0x7f99ec038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:25.879+0000 7f99fea00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.881+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f99f800ed40 con 0x7f9a08078d50 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.881+0000 7f99fea00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f99ec039fc0 0x7f99ec03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.881+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99ec0393f0 con 0x7f99ec039fc0 2024-03-23T11:36:27.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.882+0000 7f9a05e00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f99ec039fc0 0x7f99ec03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.883+0000 7f9a05e00700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f99ec039fc0 0x7f99ec03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f99f40039d0 tx=0x7f99f4009350 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.883+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f99ec0393f0 con 0x7f99ec039fc0 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.890+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9a08114170 con 0x7f99ec039fc0 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.890+0000 7f99fea00700 1 -- 172.21.15.63:0/2041608960 <== mgr.14116 v2:172.21.15.63:6800/1704436637 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9a08114170 con 0x7f99ec039fc0 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f99ec039fc0 msgr2=0x7f99ec03c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f99ec039fc0 0x7f99ec03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f99f40039d0 tx=0x7f99f4009350 comp rx=0 tx=0).stop 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 msgr2=0x7f9a081135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a081135a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f99f8006fd0 tx=0x7f99f8009250 comp rx=0 tx=0).stop 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 shutdown_connections 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f99ec039fc0 0x7f99ec03c3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a08078d50 0x7f9a081135a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 --2- 172.21.15.63:0/2041608960 >> [v2:172.21.15.63:6800/1052733058,v1:172.21.15.63:6801/1052733058] conn(0x7f99ec036830 0x7f99ec038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 >> 172.21.15.63:0/2041608960 conn(0x7f9a080fdd00 msgr2=0x7f9a080fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 shutdown_connections 2024-03-23T11:36:27.479 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:26.891+0000 7f9a0d400700 1 -- 172.21.15.63:0/2041608960 wait complete. 2024-03-23T11:36:27.480 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 5 is available 2024-03-23T11:36:27.480 INFO:teuthology.orchestra.run.smithi063.stdout:Setting orchestrator backend to cephadm... 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: [23/Mar/2024:11:36:27] ENGINE Bus STARTING 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: [23/Mar/2024:11:36:27] ENGINE Serving on https://172.21.15.63:7150 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: [23/Mar/2024:11:36:27] ENGINE Bus STARTED 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:36:28.610 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:28.611 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:28 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:36:28.871 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.039+0000 7fa39ea00700 1 Processor -- start 2024-03-23T11:36:28.871 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.039+0000 7fa39ea00700 1 -- start start 2024-03-23T11:36:28.871 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.040+0000 7fa39ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa3980772a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:28.871 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.040+0000 7fa39ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980791b0 con 0x7fa398078da0 2024-03-23T11:36:28.871 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.040+0000 7fa397e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa3980772a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:28.871 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.040+0000 7fa397e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa3980772a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54614/0 (socket says 172.21.15.63:54614) 2024-03-23T11:36:28.871 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.040+0000 7fa397e00700 1 -- 172.21.15.63:0/3556485937 learned_addr learned my addr 172.21.15.63:0/3556485937 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.041+0000 7fa397e00700 1 -- 172.21.15.63:0/3556485937 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3980777e0 con 0x7fa398078da0 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.041+0000 7fa397e00700 1 --2- 172.21.15.63:0/3556485937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa3980772a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa3880098e0 tx=0x7fa388009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff718e50f2dc701 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.041+0000 7fa396a00700 1 -- 172.21.15.63:0/3556485937 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa38800b860 con 0x7fa398078da0 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.041+0000 7fa396a00700 1 -- 172.21.15.63:0/3556485937 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa38800be30 con 0x7fa398078da0 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.041+0000 7fa396a00700 1 -- 172.21.15.63:0/3556485937 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa388010380 con 0x7fa398078da0 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.042+0000 7fa39ea00700 1 -- 172.21.15.63:0/3556485937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 msgr2=0x7fa3980772a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.042+0000 7fa39ea00700 1 --2- 172.21.15.63:0/3556485937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa3980772a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa3880098e0 tx=0x7fa388009250 comp rx=0 tx=0).stop 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.042+0000 7fa39ea00700 1 -- 172.21.15.63:0/3556485937 shutdown_connections 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.042+0000 7fa39ea00700 1 --2- 172.21.15.63:0/3556485937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa3980772a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.042+0000 7fa39ea00700 1 -- 172.21.15.63:0/3556485937 >> 172.21.15.63:0/3556485937 conn(0x7fa3980fdcd0 msgr2=0x7fa398100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.042+0000 7fa39ea00700 1 -- 172.21.15.63:0/3556485937 shutdown_connections 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.042+0000 7fa39ea00700 1 -- 172.21.15.63:0/3556485937 wait complete. 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.043+0000 7fa39ea00700 1 Processor -- start 2024-03-23T11:36:28.872 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.043+0000 7fa39ea00700 1 -- start start 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.043+0000 7fa39ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa398113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.043+0000 7fa39ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa398113b60 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.043+0000 7fa397e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa398113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.043+0000 7fa397e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa398113620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54622/0 (socket says 172.21.15.63:54622) 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.043+0000 7fa397e00700 1 -- 172.21.15.63:0/90567296 learned_addr learned my addr 172.21.15.63:0/90567296 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.044+0000 7fa397e00700 1 -- 172.21.15.63:0/90567296 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880094f0 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.044+0000 7fa397e00700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa398113620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa38800b440 tx=0x7fa388009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.044+0000 7fa38fe00700 1 -- 172.21.15.63:0/90567296 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa388010960 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.044+0000 7fa38fe00700 1 -- 172.21.15.63:0/90567296 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa38801a430 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.044+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa398113d60 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.044+0000 7fa38fe00700 1 -- 172.21.15.63:0/90567296 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3880245b0 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.044+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3981148d0 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.045+0000 7fa38fe00700 1 -- 172.21.15.63:0/90567296 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fa38801a9a0 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.045+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39805f5f0 con 0x7fa398078da0 2024-03-23T11:36:28.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.046+0000 7fa38fe00700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa37c0366b0 0x7fa37c038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.046+0000 7fa38fe00700 1 -- 172.21.15.63:0/90567296 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa38804a5d0 con 0x7fa398078da0 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.046+0000 7fa397400700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa37c0366b0 0x7fa37c038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.046+0000 7fa397400700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa37c0366b0 0x7fa37c038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa384009990 tx=0x7fa384006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.051+0000 7fa38fe00700 1 -- 172.21.15.63:0/90567296 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa38802c340 con 0x7fa398078da0 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.271+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3980ff100 con 0x7fa37c0366b0 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.282+0000 7fa38fe00700 1 -- 172.21.15.63:0/90567296 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa3980ff100 con 0x7fa37c0366b0 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.284+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa37c0366b0 msgr2=0x7fa37c038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.284+0000 7fa39ea00700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa37c0366b0 0x7fa37c038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa384009990 tx=0x7fa384006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.284+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 msgr2=0x7fa398113620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.284+0000 7fa39ea00700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa398113620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa38800b440 tx=0x7fa388009250 comp rx=0 tx=0).stop 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.285+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 shutdown_connections 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.285+0000 7fa39ea00700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa37c0366b0 0x7fa37c038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:28.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.285+0000 7fa39ea00700 1 --2- 172.21.15.63:0/90567296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa398078da0 0x7fa398113620 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:28.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.285+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 >> 172.21.15.63:0/90567296 conn(0x7fa3980fdcd0 msgr2=0x7fa3980fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:28.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.285+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 shutdown_connections 2024-03-23T11:36:28.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:28.285+0000 7fa39ea00700 1 -- 172.21.15.63:0/90567296 wait complete. 2024-03-23T11:36:29.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:29 smithi063 ceph-mon[35189]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:29.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:29 smithi063 ceph-mon[35189]: mgrmap e8: smithi063.gehkcr(active, since 2s) 2024-03-23T11:36:30.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-23T11:36:30.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.445+0000 7f4b06000700 1 Processor -- start 2024-03-23T11:36:30.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.445+0000 7f4b06000700 1 -- start start 2024-03-23T11:36:30.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.445+0000 7f4b06000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b00077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:30.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.445+0000 7f4b06000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b00079160 con 0x7f4b00078d50 2024-03-23T11:36:30.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.446+0000 7f4aff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b00077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:30.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.446+0000 7f4aff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b00077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54638/0 (socket says 172.21.15.63:54638) 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.446+0000 7f4aff400700 1 -- 172.21.15.63:0/1680320409 learned_addr learned my addr 172.21.15.63:0/1680320409 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.446+0000 7f4aff400700 1 -- 172.21.15.63:0/1680320409 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b000777a0 con 0x7f4b00078d50 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.446+0000 7f4aff400700 1 --2- 172.21.15.63:0/1680320409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b00077260 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4af00098e0 tx=0x7f4af0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37c6f6c8eff8029 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4afe000700 1 -- 172.21.15.63:0/1680320409 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af000b860 con 0x7f4b00078d50 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4afe000700 1 -- 172.21.15.63:0/1680320409 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4af000be30 con 0x7f4b00078d50 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4afe000700 1 -- 172.21.15.63:0/1680320409 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af0010380 con 0x7f4b00078d50 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4b06000700 1 -- 172.21.15.63:0/1680320409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 msgr2=0x7f4b00077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4b06000700 1 --2- 172.21.15.63:0/1680320409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b00077260 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4af00098e0 tx=0x7f4af0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4b06000700 1 -- 172.21.15.63:0/1680320409 shutdown_connections 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4b06000700 1 --2- 172.21.15.63:0/1680320409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b00077260 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4b06000700 1 -- 172.21.15.63:0/1680320409 >> 172.21.15.63:0/1680320409 conn(0x7f4b000fdd00 msgr2=0x7f4b00100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:30.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4b06000700 1 -- 172.21.15.63:0/1680320409 shutdown_connections 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.447+0000 7f4b06000700 1 -- 172.21.15.63:0/1680320409 wait complete. 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.448+0000 7f4b06000700 1 Processor -- start 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.448+0000 7f4b06000700 1 -- start start 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.449+0000 7f4b06000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b001135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.449+0000 7f4b06000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b00113ae0 con 0x7f4b00078d50 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.449+0000 7f4aff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b001135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.449+0000 7f4aff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b001135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54650/0 (socket says 172.21.15.63:54650) 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.449+0000 7f4aff400700 1 -- 172.21.15.63:0/3220587844 learned_addr learned my addr 172.21.15.63:0/3220587844 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.449+0000 7f4aff400700 1 -- 172.21.15.63:0/3220587844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af00094f0 con 0x7f4b00078d50 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.449+0000 7f4aff400700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b001135a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4af000b440 tx=0x7f4af0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.450+0000 7f4af7400700 1 -- 172.21.15.63:0/3220587844 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af00107a0 con 0x7f4b00078d50 2024-03-23T11:36:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.450+0000 7f4af7400700 1 -- 172.21.15.63:0/3220587844 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4af0010d70 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.450+0000 7f4af7400700 1 -- 172.21.15.63:0/3220587844 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4af001adc0 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.450+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b00113ce0 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.450+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b00114850 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.451+0000 7f4af7400700 1 -- 172.21.15.63:0/3220587844 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4af0010900 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.451+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0005f5f0 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.451+0000 7f4af7400700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f4ae40367e0 0x7f4ae4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.451+0000 7f4af7400700 1 -- 172.21.15.63:0/3220587844 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4af0049f20 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.452+0000 7f4afea00700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f4ae40367e0 0x7f4ae4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.452+0000 7f4afea00700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f4ae40367e0 0x7f4ae4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009990 tx=0x7f4aec006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.457+0000 7f4af7400700 1 -- 172.21.15.63:0/3220587844 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4af00239a0 con 0x7f4b00078d50 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.662+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b000fef50 con 0x7f4ae40367e0 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.663+0000 7f4af7400700 1 -- 172.21.15.63:0/3220587844 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4b000fef50 con 0x7f4ae40367e0 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f4ae40367e0 msgr2=0x7f4ae4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:30.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f4ae40367e0 0x7f4ae4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009990 tx=0x7f4aec006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 msgr2=0x7f4b001135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b001135a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4af000b440 tx=0x7f4af0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 shutdown_connections 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f4ae40367e0 0x7f4ae4038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 --2- 172.21.15.63:0/3220587844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b00078d50 0x7f4b001135a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 >> 172.21.15.63:0/3220587844 conn(0x7f4b000fdd00 msgr2=0x7f4b000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 shutdown_connections 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:29.665+0000 7f4b06000700 1 -- 172.21.15.63:0/3220587844 wait complete. 2024-03-23T11:36:30.189 INFO:teuthology.orchestra.run.smithi063.stdout:Generating ssh key... 2024-03-23T11:36:30.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:30 smithi063 ceph-mon[35189]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:31.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:31 smithi063 ceph-mon[35189]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:31.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:31 smithi063 ceph-mon[35189]: Generating ssh key... 2024-03-23T11:36:32.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.717+0000 7f19a0c00700 1 Processor -- start 2024-03-23T11:36:32.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.717+0000 7f19a0c00700 1 -- start start 2024-03-23T11:36:32.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.718+0000 7f19a0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.718+0000 7f19a0c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c079160 con 0x7f199c078d50 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.718+0000 7f1999e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.718+0000 7f1999e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54654/0 (socket says 172.21.15.63:54654) 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.718+0000 7f1999e00700 1 -- 172.21.15.63:0/3231601371 learned_addr learned my addr 172.21.15.63:0/3231601371 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.718+0000 7f1999e00700 1 -- 172.21.15.63:0/3231601371 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c0777a0 con 0x7f199c078d50 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.719+0000 7f1999e00700 1 --2- 172.21.15.63:0/3231601371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c077260 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f198c0098e0 tx=0x7f198c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae4ff9b950131ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.719+0000 7f1999400700 1 -- 172.21.15.63:0/3231601371 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c00b860 con 0x7f199c078d50 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.719+0000 7f1999400700 1 -- 172.21.15.63:0/3231601371 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f198c00be30 con 0x7f199c078d50 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.719+0000 7f1999400700 1 -- 172.21.15.63:0/3231601371 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c010380 con 0x7f199c078d50 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.719+0000 7f19a0c00700 1 -- 172.21.15.63:0/3231601371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 msgr2=0x7f199c077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.719+0000 7f19a0c00700 1 --2- 172.21.15.63:0/3231601371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c077260 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f198c0098e0 tx=0x7f198c009250 comp rx=0 tx=0).stop 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.720+0000 7f19a0c00700 1 -- 172.21.15.63:0/3231601371 shutdown_connections 2024-03-23T11:36:32.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.720+0000 7f19a0c00700 1 --2- 172.21.15.63:0/3231601371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c077260 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.720+0000 7f19a0c00700 1 -- 172.21.15.63:0/3231601371 >> 172.21.15.63:0/3231601371 conn(0x7f199c0fdd00 msgr2=0x7f199c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.720+0000 7f19a0c00700 1 -- 172.21.15.63:0/3231601371 shutdown_connections 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.720+0000 7f19a0c00700 1 -- 172.21.15.63:0/3231601371 wait complete. 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.720+0000 7f19a0c00700 1 Processor -- start 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.721+0000 7f19a0c00700 1 -- start start 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.721+0000 7f19a0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c1135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.721+0000 7f19a0c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c113ae0 con 0x7f199c078d50 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.721+0000 7f1999e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c1135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.721+0000 7f1999e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c1135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54656/0 (socket says 172.21.15.63:54656) 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.721+0000 7f1999e00700 1 -- 172.21.15.63:0/3975268948 learned_addr learned my addr 172.21.15.63:0/3975268948 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.721+0000 7f1999e00700 1 -- 172.21.15.63:0/3975268948 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0094f0 con 0x7f199c078d50 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.722+0000 7f1999e00700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c1135a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f198c006fd0 tx=0x7f198c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.722+0000 7f1992000700 1 -- 172.21.15.63:0/3975268948 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c010750 con 0x7f199c078d50 2024-03-23T11:36:32.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.722+0000 7f1992000700 1 -- 172.21.15.63:0/3975268948 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f198c010d20 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.722+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199c113ce0 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.722+0000 7f1992000700 1 -- 172.21.15.63:0/3975268948 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c01ada0 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.722+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199c114850 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.723+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199c05f5f0 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.723+0000 7f1992000700 1 -- 172.21.15.63:0/3975268948 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f198c0108b0 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.724+0000 7f1992000700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1980036790 0x7f1980038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.724+0000 7f1992000700 1 -- 172.21.15.63:0/3975268948 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f198c049d90 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.724+0000 7f1993e00700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1980036790 0x7f1980038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.725+0000 7f1993e00700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1980036790 0x7f1980038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1984009990 tx=0x7f1984006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.729+0000 7f1992000700 1 -- 172.21.15.63:0/3975268948 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f198c022720 con 0x7f199c078d50 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:30.935+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f199c0fef50 con 0x7f1980036790 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.585+0000 7f1992000700 1 -- 172.21.15.63:0/3975268948 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f199c0fef50 con 0x7f1980036790 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1980036790 msgr2=0x7f1980038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:32.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1980036790 0x7f1980038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1984009990 tx=0x7f1984006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 msgr2=0x7f199c1135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c1135a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f198c006fd0 tx=0x7f198c009250 comp rx=0 tx=0).stop 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 shutdown_connections 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1980036790 0x7f1980038c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 --2- 172.21.15.63:0/3975268948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f199c078d50 0x7f199c1135a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.587+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 >> 172.21.15.63:0/3975268948 conn(0x7f199c0fdd00 msgr2=0x7f199c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.588+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 shutdown_connections 2024-03-23T11:36:32.198 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:31.588+0000 7f19a0c00700 1 -- 172.21.15.63:0/3975268948 wait complete. 2024-03-23T11:36:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:32 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:32 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:33.505 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg8yA4o4dJe6NMe5k5uLNKKi7oHhaDUJUnfH9PcCLOiIoRZDmdEXQ6jAeNfazlFdFbhFLRG4bzm06ovDDs2HAPb31gIgLDAsJshc34p1Ao4SmjBs87HlmuzJkATXFRc3WR1KnKUWsbyE+bZOIw5qWIxPP4EfYgQ6CAQ3W9wdEjmhdbLtrSTdAuUsk+kKs34Vh5B72bVhXggFiaxcqGsybXiAGOvrRS7LiASO5s04d/uJfFTKRTVqooMktZ2RQ1rHR6WWtOWNzSKTVOYgyzBhkjMAqARO4AqHf7Cot/lF0KRVjdoKtyQJKLxzSzR79s2piO6f/lA7lGznO3lUgJxKouuuA8tTWa3jfYc81H+6ZFUB9ki0W/4J+0HMcm1LXrpJnBxLjxj1e39gxsaIrNPOpMLZ4loYBrdNN8DwKQzZC6k+jNClWI46AkVpcFOBq5FvprX+1FsnIYU/BkrD54RUeEVjRw9ymluLSrzJ4xTcW6Lx6Ugy+1rt7mG0g8bZbsJR0= ceph-6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.739+0000 7fe6e0200700 1 Processor -- start 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.739+0000 7fe6e0200700 1 -- start start 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.739+0000 7fe6e0200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d8077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.739+0000 7fe6e0200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8079160 con 0x7fe6d8078d50 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.739+0000 7fe6dd800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d8077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.740+0000 7fe6dd800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d8077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55336/0 (socket says 172.21.15.63:55336) 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.740+0000 7fe6dd800700 1 -- 172.21.15.63:0/4205519438 learned_addr learned my addr 172.21.15.63:0/4205519438 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.740+0000 7fe6dd800700 1 -- 172.21.15.63:0/4205519438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80777a0 con 0x7fe6d8078d50 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.740+0000 7fe6dd800700 1 --2- 172.21.15.63:0/4205519438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d8077260 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0098e0 tx=0x7fe6cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83fe6e3d5a1e208 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.740+0000 7fe6d7e00700 1 -- 172.21.15.63:0/4205519438 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6cc00b860 con 0x7fe6d8078d50 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.740+0000 7fe6d7e00700 1 -- 172.21.15.63:0/4205519438 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe6cc00be30 con 0x7fe6d8078d50 2024-03-23T11:36:33.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.740+0000 7fe6d7e00700 1 -- 172.21.15.63:0/4205519438 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6cc010380 con 0x7fe6d8078d50 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.741+0000 7fe6e0200700 1 -- 172.21.15.63:0/4205519438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 msgr2=0x7fe6d8077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.741+0000 7fe6e0200700 1 --2- 172.21.15.63:0/4205519438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d8077260 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0098e0 tx=0x7fe6cc009250 comp rx=0 tx=0).stop 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.741+0000 7fe6e0200700 1 -- 172.21.15.63:0/4205519438 shutdown_connections 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.741+0000 7fe6e0200700 1 --2- 172.21.15.63:0/4205519438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d8077260 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.741+0000 7fe6e0200700 1 -- 172.21.15.63:0/4205519438 >> 172.21.15.63:0/4205519438 conn(0x7fe6d80fdd00 msgr2=0x7fe6d8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.741+0000 7fe6e0200700 1 -- 172.21.15.63:0/4205519438 shutdown_connections 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.741+0000 7fe6e0200700 1 -- 172.21.15.63:0/4205519438 wait complete. 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.742+0000 7fe6e0200700 1 Processor -- start 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.742+0000 7fe6e0200700 1 -- start start 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.743+0000 7fe6e0200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d81135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.743+0000 7fe6e0200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8113ae0 con 0x7fe6d8078d50 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.743+0000 7fe6dd800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d81135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.743+0000 7fe6dd800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d81135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55340/0 (socket says 172.21.15.63:55340) 2024-03-23T11:36:33.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.743+0000 7fe6dd800700 1 -- 172.21.15.63:0/3445209261 learned_addr learned my addr 172.21.15.63:0/3445209261 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.743+0000 7fe6dd800700 1 -- 172.21.15.63:0/3445209261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc0094f0 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.743+0000 7fe6dd800700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d81135a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc006fd0 tx=0x7fe6cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.744+0000 7fe6d5600700 1 -- 172.21.15.63:0/3445209261 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6cc010750 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.744+0000 7fe6d5600700 1 -- 172.21.15.63:0/3445209261 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe6cc010d20 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.744+0000 7fe6d5600700 1 -- 172.21.15.63:0/3445209261 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6cc01ada0 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.744+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d8113ce0 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.744+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d8114850 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.745+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d805f5f0 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.745+0000 7fe6d5600700 1 -- 172.21.15.63:0/3445209261 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe6cc0108b0 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.745+0000 7fe6d5600700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fe6c00367e0 0x7fe6c0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.746+0000 7fe6d5600700 1 -- 172.21.15.63:0/3445209261 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe6cc049df0 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.746+0000 7fe6dce00700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fe6c00367e0 0x7fe6c0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.746+0000 7fe6dce00700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fe6c00367e0 0x7fe6c0038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009990 tx=0x7fe6c4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.751+0000 7fe6d5600700 1 -- 172.21.15.63:0/3445209261 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe6cc022720 con 0x7fe6d8078d50 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.957+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6d80fef50 con 0x7fe6c00367e0 2024-03-23T11:36:33.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.958+0000 7fe6d5600700 1 -- 172.21.15.63:0/3445209261 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe6d80fef50 con 0x7fe6c00367e0 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fe6c00367e0 msgr2=0x7fe6c0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fe6c00367e0 0x7fe6c0038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009990 tx=0x7fe6c4006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 msgr2=0x7fe6d81135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d81135a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc006fd0 tx=0x7fe6cc009250 comp rx=0 tx=0).stop 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 shutdown_connections 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fe6c00367e0 0x7fe6c0038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 --2- 172.21.15.63:0/3445209261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d8078d50 0x7fe6d81135a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 >> 172.21.15.63:0/3445209261 conn(0x7fe6d80fdd00 msgr2=0x7fe6d80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.960+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 shutdown_connections 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:32.961+0000 7fe6e0200700 1 -- 172.21.15.63:0/3445209261 wait complete. 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:Adding key to root@localhost authorized_keys... 2024-03-23T11:36:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:Adding host smithi063... 2024-03-23T11:36:33.759 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:33 smithi063 ceph-mon[35189]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:34.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:34 smithi063 ceph-mon[35189]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "addr": "172.21.15.63", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:35.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:35 smithi063 ceph-mon[35189]: Deploying cephadm binary to smithi063 2024-03-23T11:36:37.554 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Added host 'smithi063' with addr '172.21.15.63' 2024-03-23T11:36:37.554 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.090+0000 7fc112a00700 1 Processor -- start 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.090+0000 7fc112a00700 1 -- start start 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.090+0000 7fc112a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c0772a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.091+0000 7fc112a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c0791b0 con 0x7fc10c078da0 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.091+0000 7fc10be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c0772a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.091+0000 7fc10be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c0772a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55356/0 (socket says 172.21.15.63:55356) 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.091+0000 7fc10be00700 1 -- 172.21.15.63:0/3201511993 learned_addr learned my addr 172.21.15.63:0/3201511993 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.092+0000 7fc10be00700 1 -- 172.21.15.63:0/3201511993 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c0777e0 con 0x7fc10c078da0 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.092+0000 7fc10be00700 1 --2- 172.21.15.63:0/3201511993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c0772a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0098e0 tx=0x7fc0fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ccb38e790f2f3f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.092+0000 7fc10aa00700 1 -- 172.21.15.63:0/3201511993 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc00b860 con 0x7fc10c078da0 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.092+0000 7fc10aa00700 1 -- 172.21.15.63:0/3201511993 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc0fc00be30 con 0x7fc10c078da0 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.092+0000 7fc10aa00700 1 -- 172.21.15.63:0/3201511993 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc010380 con 0x7fc10c078da0 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.093+0000 7fc112a00700 1 -- 172.21.15.63:0/3201511993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 msgr2=0x7fc10c0772a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.093+0000 7fc112a00700 1 --2- 172.21.15.63:0/3201511993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c0772a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0098e0 tx=0x7fc0fc009250 comp rx=0 tx=0).stop 2024-03-23T11:36:37.555 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.093+0000 7fc112a00700 1 -- 172.21.15.63:0/3201511993 shutdown_connections 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.093+0000 7fc112a00700 1 --2- 172.21.15.63:0/3201511993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c0772a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.093+0000 7fc112a00700 1 -- 172.21.15.63:0/3201511993 >> 172.21.15.63:0/3201511993 conn(0x7fc10c0fdcd0 msgr2=0x7fc10c100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.093+0000 7fc112a00700 1 -- 172.21.15.63:0/3201511993 shutdown_connections 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.093+0000 7fc112a00700 1 -- 172.21.15.63:0/3201511993 wait complete. 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.094+0000 7fc112a00700 1 Processor -- start 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.094+0000 7fc112a00700 1 -- start start 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.094+0000 7fc112a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c1179d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.094+0000 7fc112a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c117f10 con 0x7fc10c078da0 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.094+0000 7fc10be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c1179d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.094+0000 7fc10be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c1179d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55372/0 (socket says 172.21.15.63:55372) 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.094+0000 7fc10be00700 1 -- 172.21.15.63:0/2278543477 learned_addr learned my addr 172.21.15.63:0/2278543477 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.095+0000 7fc10be00700 1 -- 172.21.15.63:0/2278543477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc0094f0 con 0x7fc10c078da0 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.095+0000 7fc10be00700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c1179d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00b440 tx=0x7fc0fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.095+0000 7fc103e00700 1 -- 172.21.15.63:0/2278543477 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc010960 con 0x7fc10c078da0 2024-03-23T11:36:37.556 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.095+0000 7fc103e00700 1 -- 172.21.15.63:0/2278543477 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc0fc01a430 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.095+0000 7fc103e00700 1 -- 172.21.15.63:0/2278543477 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0fc0245b0 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.095+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c118110 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.095+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c118c80 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.096+0000 7fc103e00700 1 -- 172.21.15.63:0/2278543477 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc0fc01a9a0 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.096+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10c05f5f0 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.097+0000 7fc103e00700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fc0f0036830 0x7fc0f0038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.097+0000 7fc103e00700 1 -- 172.21.15.63:0/2278543477 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc0fc04a7e0 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.097+0000 7fc10b400700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fc0f0036830 0x7fc0f0038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.098+0000 7fc10b400700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fc0f0036830 0x7fc0f0038ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009990 tx=0x7fc0f8006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.102+0000 7fc103e00700 1 -- 172.21.15.63:0/2278543477 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc0fc02c340 con 0x7fc10c078da0 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:34.313+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi063", "addr": "172.21.15.63", "target": ["mon-mgr", ""]}) v1 -- 0x7fc10c119860 con 0x7fc0f0036830 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.850+0000 7fc103e00700 1 -- 172.21.15.63:0/2278543477 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc10c119860 con 0x7fc0f0036830 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fc0f0036830 msgr2=0x7fc0f0038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fc0f0036830 0x7fc0f0038ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009990 tx=0x7fc0f8006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:37.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 msgr2=0x7fc10c1179d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c1179d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00b440 tx=0x7fc0fc009250 comp rx=0 tx=0).stop 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 shutdown_connections 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fc0f0036830 0x7fc0f0038ce0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 --2- 172.21.15.63:0/2278543477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc10c078da0 0x7fc10c1179d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 >> 172.21.15.63:0/2278543477 conn(0x7fc10c0fdcd0 msgr2=0x7fc10c10eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 shutdown_connections 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:36.853+0000 7fc112a00700 1 -- 172.21.15.63:0/2278543477 wait complete. 2024-03-23T11:36:37.558 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying mon service with default placement... 2024-03-23T11:36:38.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:37 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:38.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:37 smithi063 ceph-mon[35189]: Added host smithi063 2024-03-23T11:36:38.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:37 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:36:39.225 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-23T11:36:39.225 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.196+0000 7f1b75000700 1 Processor -- start 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.196+0000 7f1b75000700 1 -- start start 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.197+0000 7f1b75000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b70077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.197+0000 7f1b75000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b70079160 con 0x7f1b70078d50 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.197+0000 7f1b6de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b70077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.197+0000 7f1b6de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b70077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55388/0 (socket says 172.21.15.63:55388) 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.197+0000 7f1b6de00700 1 -- 172.21.15.63:0/593264040 learned_addr learned my addr 172.21.15.63:0/593264040 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.198+0000 7f1b6de00700 1 -- 172.21.15.63:0/593264040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b700777a0 con 0x7f1b70078d50 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.198+0000 7f1b6de00700 1 --2- 172.21.15.63:0/593264040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b70077260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b600098e0 tx=0x7f1b60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fdde7fd8403873 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.198+0000 7f1b6ca00700 1 -- 172.21.15.63:0/593264040 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b6000b860 con 0x7f1b70078d50 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.198+0000 7f1b6ca00700 1 -- 172.21.15.63:0/593264040 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1b6000be30 con 0x7f1b70078d50 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.198+0000 7f1b6ca00700 1 -- 172.21.15.63:0/593264040 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b60010380 con 0x7f1b70078d50 2024-03-23T11:36:39.226 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.199+0000 7f1b75000700 1 -- 172.21.15.63:0/593264040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 msgr2=0x7f1b70077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.199+0000 7f1b75000700 1 --2- 172.21.15.63:0/593264040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b70077260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b600098e0 tx=0x7f1b60009250 comp rx=0 tx=0).stop 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.199+0000 7f1b75000700 1 -- 172.21.15.63:0/593264040 shutdown_connections 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.199+0000 7f1b75000700 1 --2- 172.21.15.63:0/593264040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b70077260 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.199+0000 7f1b75000700 1 -- 172.21.15.63:0/593264040 >> 172.21.15.63:0/593264040 conn(0x7f1b700fdd00 msgr2=0x7f1b70100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.199+0000 7f1b75000700 1 -- 172.21.15.63:0/593264040 shutdown_connections 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.199+0000 7f1b75000700 1 -- 172.21.15.63:0/593264040 wait complete. 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.200+0000 7f1b75000700 1 Processor -- start 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.200+0000 7f1b75000700 1 -- start start 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.201+0000 7f1b75000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b701135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.201+0000 7f1b75000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b70113ae0 con 0x7f1b70078d50 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.201+0000 7f1b6de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b701135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.201+0000 7f1b6de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b701135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55404/0 (socket says 172.21.15.63:55404) 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.201+0000 7f1b6de00700 1 -- 172.21.15.63:0/1148632866 learned_addr learned my addr 172.21.15.63:0/1148632866 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.201+0000 7f1b6de00700 1 -- 172.21.15.63:0/1148632866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b600094f0 con 0x7f1b70078d50 2024-03-23T11:36:39.227 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.202+0000 7f1b6de00700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b701135a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000b440 tx=0x7f1b60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.202+0000 7f1b66000700 1 -- 172.21.15.63:0/1148632866 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b600107a0 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.202+0000 7f1b66000700 1 -- 172.21.15.63:0/1148632866 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1b60010d70 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.202+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b70113ce0 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.202+0000 7f1b66000700 1 -- 172.21.15.63:0/1148632866 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b6001adc0 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.202+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b70114850 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.203+0000 7f1b66000700 1 -- 172.21.15.63:0/1148632866 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1b60010900 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.203+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7005f5f0 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.204+0000 7f1b66000700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1b540367e0 0x7f1b54038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.204+0000 7f1b66000700 1 -- 172.21.15.63:0/1148632866 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1b60049f30 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.204+0000 7f1b6d400700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1b540367e0 0x7f1b54038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.204+0000 7f1b6d400700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1b540367e0 0x7f1b54038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009990 tx=0x7f1b5c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.209+0000 7f1b66000700 1 -- 172.21.15.63:0/1148632866 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1b60006d70 con 0x7f1b70078d50 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.444+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b700fef50 con 0x7f1b540367e0 2024-03-23T11:36:39.228 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.449+0000 7f1b66000700 1 -- 172.21.15.63:0/1148632866 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1b700fef50 con 0x7f1b540367e0 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1b540367e0 msgr2=0x7f1b54038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1b540367e0 0x7f1b54038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009990 tx=0x7f1b5c006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 msgr2=0x7f1b701135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b701135a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000b440 tx=0x7f1b60009250 comp rx=0 tx=0).stop 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 shutdown_connections 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1b540367e0 0x7f1b54038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 --2- 172.21.15.63:0/1148632866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b70078d50 0x7f1b701135a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.451+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 >> 172.21.15.63:0/1148632866 conn(0x7f1b700fdd00 msgr2=0x7f1b700fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.452+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 shutdown_connections 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:38.452+0000 7f1b75000700 1 -- 172.21.15.63:0/1148632866 wait complete. 2024-03-23T11:36:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying mgr service with default placement... 2024-03-23T11:36:39.497 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:39 smithi063 ceph-mon[35189]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:39.498 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:39 smithi063 ceph-mon[35189]: Saving service mon spec with placement count:5 2024-03-23T11:36:39.498 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:39 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:40.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-23T11:36:40.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.877+0000 7fa34e600700 1 Processor -- start 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.877+0000 7fa34e600700 1 -- start start 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.877+0000 7fa34e600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa348077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.877+0000 7fa34e600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa348079160 con 0x7fa348078d50 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.878+0000 7fa347400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa348077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.878+0000 7fa347400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa348077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55418/0 (socket says 172.21.15.63:55418) 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.878+0000 7fa347400700 1 -- 172.21.15.63:0/1836871129 learned_addr learned my addr 172.21.15.63:0/1836871129 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.878+0000 7fa347400700 1 -- 172.21.15.63:0/1836871129 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480777a0 con 0x7fa348078d50 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.878+0000 7fa347400700 1 --2- 172.21.15.63:0/1836871129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa348077260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa3380098e0 tx=0x7fa338009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc29e919d6687db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa346000700 1 -- 172.21.15.63:0/1836871129 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa33800b860 con 0x7fa348078d50 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa346000700 1 -- 172.21.15.63:0/1836871129 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa33800be30 con 0x7fa348078d50 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa346000700 1 -- 172.21.15.63:0/1836871129 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa338010380 con 0x7fa348078d50 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa34e600700 1 -- 172.21.15.63:0/1836871129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 msgr2=0x7fa348077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa34e600700 1 --2- 172.21.15.63:0/1836871129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa348077260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa3380098e0 tx=0x7fa338009250 comp rx=0 tx=0).stop 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa34e600700 1 -- 172.21.15.63:0/1836871129 shutdown_connections 2024-03-23T11:36:40.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa34e600700 1 --2- 172.21.15.63:0/1836871129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa348077260 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa34e600700 1 -- 172.21.15.63:0/1836871129 >> 172.21.15.63:0/1836871129 conn(0x7fa3480fdd00 msgr2=0x7fa348100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa34e600700 1 -- 172.21.15.63:0/1836871129 shutdown_connections 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.879+0000 7fa34e600700 1 -- 172.21.15.63:0/1836871129 wait complete. 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.880+0000 7fa34e600700 1 Processor -- start 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.880+0000 7fa34e600700 1 -- start start 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.881+0000 7fa34e600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa3481135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.881+0000 7fa34e600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa348113ae0 con 0x7fa348078d50 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.881+0000 7fa347400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa3481135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.881+0000 7fa347400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa3481135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55428/0 (socket says 172.21.15.63:55428) 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.881+0000 7fa347400700 1 -- 172.21.15.63:0/1509136026 learned_addr learned my addr 172.21.15.63:0/1509136026 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.881+0000 7fa347400700 1 -- 172.21.15.63:0/1509136026 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3380094f0 con 0x7fa348078d50 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.882+0000 7fa347400700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa3481135a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa338006fd0 tx=0x7fa338009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.882+0000 7fa33f400700 1 -- 172.21.15.63:0/1509136026 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa338010750 con 0x7fa348078d50 2024-03-23T11:36:40.706 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.882+0000 7fa33f400700 1 -- 172.21.15.63:0/1509136026 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa338010d20 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.882+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa348113ce0 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.882+0000 7fa33f400700 1 -- 172.21.15.63:0/1509136026 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa33801ada0 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.882+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa348114850 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.883+0000 7fa33f400700 1 -- 172.21.15.63:0/1509136026 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa3380108b0 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.883+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34805f5f0 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.883+0000 7fa33f400700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa32c0367e0 0x7fa32c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.883+0000 7fa33f400700 1 -- 172.21.15.63:0/1509136026 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa338049d70 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.884+0000 7fa346a00700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa32c0367e0 0x7fa32c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.884+0000 7fa346a00700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa32c0367e0 0x7fa32c038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa334009990 tx=0x7fa334006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:39.888+0000 7fa33f400700 1 -- 172.21.15.63:0/1509136026 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa338010e90 con 0x7fa348078d50 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.105+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3480fef50 con 0x7fa32c0367e0 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.109+0000 7fa33f400700 1 -- 172.21.15.63:0/1509136026 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa3480fef50 con 0x7fa32c0367e0 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa32c0367e0 msgr2=0x7fa32c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:40.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa32c0367e0 0x7fa32c038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa334009990 tx=0x7fa334006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 msgr2=0x7fa3481135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa3481135a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa338006fd0 tx=0x7fa338009250 comp rx=0 tx=0).stop 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 shutdown_connections 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa32c0367e0 0x7fa32c038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 --2- 172.21.15.63:0/1509136026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa348078d50 0x7fa3481135a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.112+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 >> 172.21.15.63:0/1509136026 conn(0x7fa3480fdd00 msgr2=0x7fa3480fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.114+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 shutdown_connections 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:40.114+0000 7fa34e600700 1 -- 172.21.15.63:0/1509136026 wait complete. 2024-03-23T11:36:40.708 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying crash service with default placement... 2024-03-23T11:36:41.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:41 smithi063 ceph-mon[35189]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:41.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:41 smithi063 ceph-mon[35189]: Saving service mgr spec with placement count:2 2024-03-23T11:36:41.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:41 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:41.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:41 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:41.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:41 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.285+0000 7f25d5000700 1 Processor -- start 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.285+0000 7f25d5000700 1 -- start start 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.285+0000 7f25d5000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d0077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.285+0000 7f25d5000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0079050 con 0x7f25d0078c40 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.286+0000 7f25cde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d0077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.286+0000 7f25cde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d0077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55442/0 (socket says 172.21.15.63:55442) 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.286+0000 7f25cde00700 1 -- 172.21.15.63:0/2560280305 learned_addr learned my addr 172.21.15.63:0/2560280305 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.286+0000 7f25cde00700 1 -- 172.21.15.63:0/2560280305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0077780 con 0x7f25d0078c40 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.287+0000 7f25cde00700 1 --2- 172.21.15.63:0/2560280305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d0077240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f25c00098e0 tx=0x7f25c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67377e243b989486 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.287+0000 7f25cca00700 1 -- 172.21.15.63:0/2560280305 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c000b860 con 0x7f25d0078c40 2024-03-23T11:36:42.276 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.287+0000 7f25cca00700 1 -- 172.21.15.63:0/2560280305 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25c000be70 con 0x7f25d0078c40 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.287+0000 7f25cca00700 1 -- 172.21.15.63:0/2560280305 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c0010380 con 0x7f25d0078c40 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.288+0000 7f25d5000700 1 -- 172.21.15.63:0/2560280305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 msgr2=0x7f25d0077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.288+0000 7f25d5000700 1 --2- 172.21.15.63:0/2560280305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d0077240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f25c00098e0 tx=0x7f25c0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.288+0000 7f25d5000700 1 -- 172.21.15.63:0/2560280305 shutdown_connections 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.288+0000 7f25d5000700 1 --2- 172.21.15.63:0/2560280305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d0077240 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.288+0000 7f25d5000700 1 -- 172.21.15.63:0/2560280305 >> 172.21.15.63:0/2560280305 conn(0x7f25d00fddd0 msgr2=0x7f25d01001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.288+0000 7f25d5000700 1 -- 172.21.15.63:0/2560280305 shutdown_connections 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.288+0000 7f25d5000700 1 -- 172.21.15.63:0/2560280305 wait complete. 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.289+0000 7f25d5000700 1 Processor -- start 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.289+0000 7f25d5000700 1 -- start start 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.290+0000 7f25d5000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d01135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.290+0000 7f25d5000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0113b20 con 0x7f25d0078c40 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.290+0000 7f25cde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d01135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:42.277 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.290+0000 7f25cde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d01135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55452/0 (socket says 172.21.15.63:55452) 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.290+0000 7f25cde00700 1 -- 172.21.15.63:0/457563745 learned_addr learned my addr 172.21.15.63:0/457563745 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.290+0000 7f25cde00700 1 -- 172.21.15.63:0/457563745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00094f0 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.291+0000 7f25cde00700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d01135e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f25c0011ca0 tx=0x7f25c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.291+0000 7f25c6000700 1 -- 172.21.15.63:0/457563745 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c00105f0 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.291+0000 7f25c6000700 1 -- 172.21.15.63:0/457563745 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25c0010c00 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.291+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d0113d20 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.291+0000 7f25c6000700 1 -- 172.21.15.63:0/457563745 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c001aca0 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.291+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d0114890 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.292+0000 7f25c6000700 1 -- 172.21.15.63:0/457563745 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f25c0025400 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.292+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d005f5f0 con 0x7f25d0078c40 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.293+0000 7f25c6000700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f25b40367e0 0x7f25b4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:42.278 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.293+0000 7f25c6000700 1 -- 172.21.15.63:0/457563745 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f25c004a230 con 0x7f25d0078c40 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.293+0000 7f25cd400700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f25b40367e0 0x7f25b4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.294+0000 7f25cd400700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f25b40367e0 0x7f25b4038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f25bc009990 tx=0x7f25bc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.298+0000 7f25c6000700 1 -- 172.21.15.63:0/457563745 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f25c002c8f0 con 0x7f25d0078c40 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.538+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f25d00ff050 con 0x7f25b40367e0 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.544+0000 7f25c6000700 1 -- 172.21.15.63:0/457563745 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f25d00ff050 con 0x7f25b40367e0 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.546+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f25b40367e0 msgr2=0x7f25b4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.546+0000 7f25d5000700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f25b40367e0 0x7f25b4038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f25bc009990 tx=0x7f25bc006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.546+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 msgr2=0x7f25d01135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.546+0000 7f25d5000700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d01135e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f25c0011ca0 tx=0x7f25c0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.547+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 shutdown_connections 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.547+0000 7f25d5000700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f25b40367e0 0x7f25b4038c90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.547+0000 7f25d5000700 1 --2- 172.21.15.63:0/457563745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25d0078c40 0x7f25d01135e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.547+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 >> 172.21.15.63:0/457563745 conn(0x7f25d00fddd0 msgr2=0x7f25d00fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.547+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 shutdown_connections 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:41.547+0000 7f25d5000700 1 -- 172.21.15.63:0/457563745 wait complete. 2024-03-23T11:36:42.279 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying prometheus service with default placement... 2024-03-23T11:36:42.784 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:42 smithi063 ceph-mon[35189]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:42.785 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:42 smithi063 ceph-mon[35189]: Saving service crash spec with placement * 2024-03-23T11:36:42.785 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:42 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:43.953 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-23T11:36:43.953 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.102+0000 7fa487a00700 1 Processor -- start 2024-03-23T11:36:43.953 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.102+0000 7fa487a00700 1 -- start start 2024-03-23T11:36:43.953 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.102+0000 7fa487a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:43.953 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.102+0000 7fa487a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa480102ed0 con 0x7fa480102580 2024-03-23T11:36:43.953 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.103+0000 7fa485000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.103+0000 7fa485000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36586/0 (socket says 172.21.15.63:36586) 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.103+0000 7fa485000700 1 -- 172.21.15.63:0/1094193112 learned_addr learned my addr 172.21.15.63:0/1094193112 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.103+0000 7fa485000700 1 -- 172.21.15.63:0/1094193112 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa480103010 con 0x7fa480102580 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.103+0000 7fa485000700 1 --2- 172.21.15.63:0/1094193112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480102990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4740098e0 tx=0x7fa474009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b24875ee9d718bb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa47f400700 1 -- 172.21.15.63:0/1094193112 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa47400b860 con 0x7fa480102580 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa47f400700 1 -- 172.21.15.63:0/1094193112 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa47400be70 con 0x7fa480102580 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa47f400700 1 -- 172.21.15.63:0/1094193112 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa474010380 con 0x7fa480102580 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa487a00700 1 -- 172.21.15.63:0/1094193112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 msgr2=0x7fa480102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa487a00700 1 --2- 172.21.15.63:0/1094193112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480102990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4740098e0 tx=0x7fa474009250 comp rx=0 tx=0).stop 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa487a00700 1 -- 172.21.15.63:0/1094193112 shutdown_connections 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa487a00700 1 --2- 172.21.15.63:0/1094193112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480102990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.104+0000 7fa487a00700 1 -- 172.21.15.63:0/1094193112 >> 172.21.15.63:0/1094193112 conn(0x7fa4800fdd00 msgr2=0x7fa480100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:43.954 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.105+0000 7fa487a00700 1 -- 172.21.15.63:0/1094193112 shutdown_connections 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.105+0000 7fa487a00700 1 -- 172.21.15.63:0/1094193112 wait complete. 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.106+0000 7fa487a00700 1 Processor -- start 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.106+0000 7fa487a00700 1 -- start start 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.106+0000 7fa487a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.106+0000 7fa487a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4801118c0 con 0x7fa480102580 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.107+0000 7fa485000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.107+0000 7fa485000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36602/0 (socket says 172.21.15.63:36602) 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.107+0000 7fa485000700 1 -- 172.21.15.63:0/388137832 learned_addr learned my addr 172.21.15.63:0/388137832 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.107+0000 7fa485000700 1 -- 172.21.15.63:0/388137832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4740094f0 con 0x7fa480102580 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.107+0000 7fa485000700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480111380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa47400c010 tx=0x7fa474009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.108+0000 7fa47cc00700 1 -- 172.21.15.63:0/388137832 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa474010960 con 0x7fa480102580 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.108+0000 7fa47cc00700 1 -- 172.21.15.63:0/388137832 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa47401a430 con 0x7fa480102580 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.108+0000 7fa47cc00700 1 -- 172.21.15.63:0/388137832 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa474019580 con 0x7fa480102580 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.108+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa480111ac0 con 0x7fa480102580 2024-03-23T11:36:43.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.108+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa480112630 con 0x7fa480102580 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.109+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48005f5f0 con 0x7fa480102580 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.110+0000 7fa47cc00700 1 -- 172.21.15.63:0/388137832 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa47401a980 con 0x7fa480102580 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.110+0000 7fa47cc00700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4680367e0 0x7fa468038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.110+0000 7fa47cc00700 1 -- 172.21.15.63:0/388137832 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa47404a620 con 0x7fa480102580 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.110+0000 7fa47fe00700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4680367e0 0x7fa468038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.111+0000 7fa47fe00700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4680367e0 0x7fa468038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009990 tx=0x7fa46c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.115+0000 7fa47cc00700 1 -- 172.21.15.63:0/388137832 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa47402bb30 con 0x7fa480102580 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.341+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4800fef40 con 0x7fa4680367e0 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.347+0000 7fa47cc00700 1 -- 172.21.15.63:0/388137832 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa4800fef40 con 0x7fa4680367e0 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4680367e0 msgr2=0x7fa468038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4680367e0 0x7fa468038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009990 tx=0x7fa46c006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 msgr2=0x7fa480111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480111380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa47400c010 tx=0x7fa474009250 comp rx=0 tx=0).stop 2024-03-23T11:36:43.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 shutdown_connections 2024-03-23T11:36:43.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4680367e0 0x7fa468038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:43.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 --2- 172.21.15.63:0/388137832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa480102580 0x7fa480111380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:43.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 >> 172.21.15.63:0/388137832 conn(0x7fa4800fdd00 msgr2=0x7fa4800fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:43.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 shutdown_connections 2024-03-23T11:36:43.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:43.349+0000 7fa487a00700 1 -- 172.21.15.63:0/388137832 wait complete. 2024-03-23T11:36:43.957 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying grafana service with default placement... 2024-03-23T11:36:43.957 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:43 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:43.957 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:43 smithi063 ceph-mon[35189]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:43.957 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:43 smithi063 ceph-mon[35189]: Saving service prometheus spec with placement count:1 2024-03-23T11:36:43.957 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:43 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:45.551 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:45 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:45.551 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:45 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:46.132 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.565+0000 7f9cce600700 1 Processor -- start 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.566+0000 7f9cce600700 1 -- start start 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.566+0000 7f9cce600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.566+0000 7f9cce600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc8102ed0 con 0x7f9cc8102580 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.566+0000 7f9cc7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.566+0000 7f9cc7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36612/0 (socket says 172.21.15.63:36612) 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.566+0000 7f9cc7400700 1 -- 172.21.15.63:0/2751697112 learned_addr learned my addr 172.21.15.63:0/2751697112 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.567+0000 7f9cc7400700 1 -- 172.21.15.63:0/2751697112 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc8103010 con 0x7f9cc8102580 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.567+0000 7f9cc7400700 1 --2- 172.21.15.63:0/2751697112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8102990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80098e0 tx=0x7f9cb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3049b5cc2d56a5cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.567+0000 7f9cc6000700 1 -- 172.21.15.63:0/2751697112 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb800b860 con 0x7f9cc8102580 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.567+0000 7f9cc6000700 1 -- 172.21.15.63:0/2751697112 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cb800be70 con 0x7f9cc8102580 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.567+0000 7f9cc6000700 1 -- 172.21.15.63:0/2751697112 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb8010380 con 0x7f9cc8102580 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.568+0000 7f9cce600700 1 -- 172.21.15.63:0/2751697112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 msgr2=0x7f9cc8102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.568+0000 7f9cce600700 1 --2- 172.21.15.63:0/2751697112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8102990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80098e0 tx=0x7f9cb8009250 comp rx=0 tx=0).stop 2024-03-23T11:36:46.133 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.568+0000 7f9cce600700 1 -- 172.21.15.63:0/2751697112 shutdown_connections 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.568+0000 7f9cce600700 1 --2- 172.21.15.63:0/2751697112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8102990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.568+0000 7f9cce600700 1 -- 172.21.15.63:0/2751697112 >> 172.21.15.63:0/2751697112 conn(0x7f9cc80fdd00 msgr2=0x7f9cc8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.568+0000 7f9cce600700 1 -- 172.21.15.63:0/2751697112 shutdown_connections 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.568+0000 7f9cce600700 1 -- 172.21.15.63:0/2751697112 wait complete. 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.569+0000 7f9cce600700 1 Processor -- start 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.569+0000 7f9cce600700 1 -- start start 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.569+0000 7f9cce600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.569+0000 7f9cce600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc8113ac0 con 0x7f9cc8102580 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.569+0000 7f9cc7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:46.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cc7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8113580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36628/0 (socket says 172.21.15.63:36628) 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cc7400700 1 -- 172.21.15.63:0/3498436555 learned_addr learned my addr 172.21.15.63:0/3498436555 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cc7400700 1 -- 172.21.15.63:0/3498436555 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80094f0 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cc7400700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8113580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c010 tx=0x7f9cb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cbf400700 1 -- 172.21.15.63:0/3498436555 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb8010960 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cbf400700 1 -- 172.21.15.63:0/3498436555 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cb801a430 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cbf400700 1 -- 172.21.15.63:0/3498436555 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb8019580 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.570+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc8113cc0 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.571+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc8114830 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.572+0000 7f9cbf400700 1 -- 172.21.15.63:0/3498436555 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9cb8010ac0 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.572+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc805f5f0 con 0x7f9cc8102580 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.572+0000 7f9cbf400700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f9cac036830 0x7f9cac038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:46.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.572+0000 7f9cbf400700 1 -- 172.21.15.63:0/3498436555 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9cb804a620 con 0x7f9cc8102580 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.572+0000 7f9cc6a00700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f9cac036830 0x7f9cac038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.573+0000 7f9cc6a00700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f9cac036830 0x7f9cac038ce0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009990 tx=0x7f9cb4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.577+0000 7f9cbf400700 1 -- 172.21.15.63:0/3498436555 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9cb802bcd0 con 0x7f9cc8102580 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.821+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cc80fef40 con 0x7f9cac036830 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.826+0000 7f9cbf400700 1 -- 172.21.15.63:0/3498436555 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f9cc80fef40 con 0x7f9cac036830 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f9cac036830 msgr2=0x7f9cac038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f9cac036830 0x7f9cac038ce0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009990 tx=0x7f9cb4006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 msgr2=0x7f9cc8113580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8113580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c010 tx=0x7f9cb8009250 comp rx=0 tx=0).stop 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 shutdown_connections 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f9cac036830 0x7f9cac038ce0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 --2- 172.21.15.63:0/3498436555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cc8102580 0x7f9cc8113580 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:46.136 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 >> 172.21.15.63:0/3498436555 conn(0x7f9cc80fdd00 msgr2=0x7f9cc80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:46.137 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 shutdown_connections 2024-03-23T11:36:46.137 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:44.828+0000 7f9cce600700 1 -- 172.21.15.63:0/3498436555 wait complete. 2024-03-23T11:36:46.137 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying node-exporter service with default placement... 2024-03-23T11:36:46.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:46 smithi063 ceph-mon[35189]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:46.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:46 smithi063 ceph-mon[35189]: Saving service grafana spec with placement count:1 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.682+0000 7f141c600700 1 Processor -- start 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.682+0000 7f141c600700 1 -- start start 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.682+0000 7f141c600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f1414077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.682+0000 7f141c600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1414079160 con 0x7f1414078d50 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.683+0000 7f1419c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f1414077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.683+0000 7f1419c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f1414077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36630/0 (socket says 172.21.15.63:36630) 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.683+0000 7f1419c00700 1 -- 172.21.15.63:0/468229339 learned_addr learned my addr 172.21.15.63:0/468229339 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.683+0000 7f1419c00700 1 -- 172.21.15.63:0/468229339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14140777a0 con 0x7f1414078d50 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.684+0000 7f1419c00700 1 --2- 172.21.15.63:0/468229339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f1414077260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f14080098e0 tx=0x7f1408009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f4a5ce8194531a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:47.487 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.684+0000 7f1413e00700 1 -- 172.21.15.63:0/468229339 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f140800b860 con 0x7f1414078d50 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.684+0000 7f1413e00700 1 -- 172.21.15.63:0/468229339 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f140800be70 con 0x7f1414078d50 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.684+0000 7f1413e00700 1 -- 172.21.15.63:0/468229339 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1408010380 con 0x7f1414078d50 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.684+0000 7f141c600700 1 -- 172.21.15.63:0/468229339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 msgr2=0x7f1414077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.684+0000 7f141c600700 1 --2- 172.21.15.63:0/468229339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f1414077260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f14080098e0 tx=0x7f1408009250 comp rx=0 tx=0).stop 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.685+0000 7f141c600700 1 -- 172.21.15.63:0/468229339 shutdown_connections 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.685+0000 7f141c600700 1 --2- 172.21.15.63:0/468229339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f1414077260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.685+0000 7f141c600700 1 -- 172.21.15.63:0/468229339 >> 172.21.15.63:0/468229339 conn(0x7f14140fdd00 msgr2=0x7f1414100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.685+0000 7f141c600700 1 -- 172.21.15.63:0/468229339 shutdown_connections 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.685+0000 7f141c600700 1 -- 172.21.15.63:0/468229339 wait complete. 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.686+0000 7f141c600700 1 Processor -- start 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.686+0000 7f141c600700 1 -- start start 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.686+0000 7f141c600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f14141135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.687+0000 7f141c600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1414113ae0 con 0x7f1414078d50 2024-03-23T11:36:47.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.687+0000 7f1419c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f14141135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.687+0000 7f1419c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f14141135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36646/0 (socket says 172.21.15.63:36646) 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.687+0000 7f1419c00700 1 -- 172.21.15.63:0/1574950014 learned_addr learned my addr 172.21.15.63:0/1574950014 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.687+0000 7f1419c00700 1 -- 172.21.15.63:0/1574950014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14080094f0 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.687+0000 7f1419c00700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f14141135a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1408006d40 tx=0x7f1408009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.688+0000 7f1411600700 1 -- 172.21.15.63:0/1574950014 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1408010960 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.688+0000 7f1411600700 1 -- 172.21.15.63:0/1574950014 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f140801a430 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.688+0000 7f1411600700 1 -- 172.21.15.63:0/1574950014 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14080245b0 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.688+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1414113ce0 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.688+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1414114850 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.689+0000 7f1411600700 1 -- 172.21.15.63:0/1574950014 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f140801a9f0 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.689+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141405f5f0 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.689+0000 7f1411600700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f13fc036790 0x7f13fc038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.689+0000 7f1411600700 1 -- 172.21.15.63:0/1574950014 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f140804a3b0 con 0x7f1414078d50 2024-03-23T11:36:47.489 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.689+0000 7f1419200700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f13fc036790 0x7f13fc038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.690+0000 7f1419200700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f13fc036790 0x7f13fc038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1400009990 tx=0x7f1400006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.694+0000 7f1411600700 1 -- 172.21.15.63:0/1574950014 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f140802b8e0 con 0x7f1414078d50 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.910+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f14140fef50 con 0x7f13fc036790 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.915+0000 7f1411600700 1 -- 172.21.15.63:0/1574950014 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f14140fef50 con 0x7f13fc036790 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f13fc036790 msgr2=0x7f13fc038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f13fc036790 0x7f13fc038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1400009990 tx=0x7f1400006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 msgr2=0x7f14141135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f14141135a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1408006d40 tx=0x7f1408009250 comp rx=0 tx=0).stop 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 shutdown_connections 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f13fc036790 0x7f13fc038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 --2- 172.21.15.63:0/1574950014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414078d50 0x7f14141135a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.917+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 >> 172.21.15.63:0/1574950014 conn(0x7f14140fdd00 msgr2=0x7f14140fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.918+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 shutdown_connections 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:46.918+0000 7f141c600700 1 -- 172.21.15.63:0/1574950014 wait complete. 2024-03-23T11:36:47.490 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying alertmanager service with default placement... 2024-03-23T11:36:47.491 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:47 smithi063 ceph-mon[35189]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:36:47.491 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:47 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:48.549 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:48 smithi063 ceph-mon[35189]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:48.549 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:48 smithi063 ceph-mon[35189]: Saving service node-exporter spec with placement * 2024-03-23T11:36:48.792 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-23T11:36:48.792 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.045+0000 7f79b3e00700 1 Processor -- start 2024-03-23T11:36:48.792 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.045+0000 7f79b3e00700 1 -- start start 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.045+0000 7f79b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.045+0000 7f79b3e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4079140 con 0x7f79b4078d30 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.045+0000 7f79b2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.046+0000 7f79b2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36650/0 (socket says 172.21.15.63:36650) 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.046+0000 7f79b2a00700 1 -- 172.21.15.63:0/1453166431 learned_addr learned my addr 172.21.15.63:0/1453166431 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.046+0000 7f79b2a00700 1 -- 172.21.15.63:0/1453166431 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b4077780 con 0x7f79b4078d30 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.046+0000 7f79b2a00700 1 --2- 172.21.15.63:0/1453166431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4077240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f79a40098e0 tx=0x7f79a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a10e552ae3d25fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.046+0000 7f79b1600700 1 -- 172.21.15.63:0/1453166431 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79a400b860 con 0x7f79b4078d30 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.046+0000 7f79b1600700 1 -- 172.21.15.63:0/1453166431 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79a400be70 con 0x7f79b4078d30 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b1600700 1 -- 172.21.15.63:0/1453166431 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79a4010380 con 0x7f79b4078d30 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b3e00700 1 -- 172.21.15.63:0/1453166431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 msgr2=0x7f79b4077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b3e00700 1 --2- 172.21.15.63:0/1453166431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4077240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f79a40098e0 tx=0x7f79a4009250 comp rx=0 tx=0).stop 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b3e00700 1 -- 172.21.15.63:0/1453166431 shutdown_connections 2024-03-23T11:36:48.793 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b3e00700 1 --2- 172.21.15.63:0/1453166431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4077240 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b3e00700 1 -- 172.21.15.63:0/1453166431 >> 172.21.15.63:0/1453166431 conn(0x7f79b40fdb90 msgr2=0x7f79b40fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b3e00700 1 -- 172.21.15.63:0/1453166431 shutdown_connections 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.047+0000 7f79b3e00700 1 -- 172.21.15.63:0/1453166431 wait complete. 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.048+0000 7f79b3e00700 1 Processor -- start 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.048+0000 7f79b3e00700 1 -- start start 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.048+0000 7f79b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4113460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.048+0000 7f79b3e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b41139a0 con 0x7f79b4078d30 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.049+0000 7f79b2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4113460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.049+0000 7f79b2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4113460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36662/0 (socket says 172.21.15.63:36662) 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.049+0000 7f79b2a00700 1 -- 172.21.15.63:0/1070120363 learned_addr learned my addr 172.21.15.63:0/1070120363 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.049+0000 7f79b2a00700 1 -- 172.21.15.63:0/1070120363 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a40094f0 con 0x7f79b4078d30 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.049+0000 7f79b2a00700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4113460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f79a4011b60 tx=0x7f79a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:48.794 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.050+0000 7f79aaa00700 1 -- 172.21.15.63:0/1070120363 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79a4010800 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.050+0000 7f79aaa00700 1 -- 172.21.15.63:0/1070120363 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79a4010e10 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.050+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b4113ba0 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.050+0000 7f79aaa00700 1 -- 172.21.15.63:0/1070120363 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79a401ae60 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.050+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b4114710 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.051+0000 7f79aaa00700 1 -- 172.21.15.63:0/1070120363 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f79a401a430 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.051+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b405f5f0 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.051+0000 7f79aaa00700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f79980367e0 0x7f7998038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.051+0000 7f79aaa00700 1 -- 172.21.15.63:0/1070120363 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f79a404a5f0 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.052+0000 7f79b2000700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f79980367e0 0x7f7998038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.052+0000 7f79b2000700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f79980367e0 0x7f7998038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009990 tx=0x7f79a0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.056+0000 7f79aaa00700 1 -- 172.21.15.63:0/1070120363 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f79a402b950 con 0x7f79b4078d30 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.273+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f79b40fee70 con 0x7f79980367e0 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.278+0000 7f79aaa00700 1 -- 172.21.15.63:0/1070120363 <== mgr.14116 v2:172.21.15.63:6800/1704436637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f79b40fee70 con 0x7f79980367e0 2024-03-23T11:36:48.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.280+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f79980367e0 msgr2=0x7f7998038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.280+0000 7f79b3e00700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f79980367e0 0x7f7998038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009990 tx=0x7f79a0006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.280+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 msgr2=0x7f79b4113460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.280+0000 7f79b3e00700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4113460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f79a4011b60 tx=0x7f79a4009250 comp rx=0 tx=0).stop 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.281+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 shutdown_connections 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.281+0000 7f79b3e00700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f79980367e0 0x7f7998038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.281+0000 7f79b3e00700 1 --2- 172.21.15.63:0/1070120363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b4078d30 0x7f79b4113460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.281+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 >> 172.21.15.63:0/1070120363 conn(0x7f79b40fdb90 msgr2=0x7f79b40fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.281+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 shutdown_connections 2024-03-23T11:36:48.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:48.281+0000 7f79b3e00700 1 -- 172.21.15.63:0/1070120363 wait complete. 2024-03-23T11:36:49.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:49 smithi063 ceph-mon[35189]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:36:49.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:49 smithi063 ceph-mon[35189]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:36:49.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:49 smithi063 ceph-mon[35189]: Saving service alertmanager spec with placement count:1 2024-03-23T11:36:49.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:49 smithi063 ceph-mon[35189]: from='mgr.14116 172.21.15.63:0/2128276075' entity='mgr.smithi063.gehkcr' 2024-03-23T11:36:50.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.388+0000 7f20e8200700 1 Processor -- start 2024-03-23T11:36:50.193 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.389+0000 7f20e8200700 1 -- start start 2024-03-23T11:36:50.193 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.389+0000 7f20e8200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:50.193 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.389+0000 7f20e8200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0102ed0 con 0x7f20e0102580 2024-03-23T11:36:50.193 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.389+0000 7f20e5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:50.193 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.389+0000 7f20e5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36674/0 (socket says 172.21.15.63:36674) 2024-03-23T11:36:50.193 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.389+0000 7f20e5800700 1 -- 172.21.15.63:0/4171703464 learned_addr learned my addr 172.21.15.63:0/4171703464 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.390+0000 7f20e5800700 1 -- 172.21.15.63:0/4171703464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0103010 con 0x7f20e0102580 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.390+0000 7f20e5800700 1 --2- 172.21.15.63:0/4171703464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0102990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f20d40098e0 tx=0x7f20d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510e95f264f3c957 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.390+0000 7f20dfe00700 1 -- 172.21.15.63:0/4171703464 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d400b860 con 0x7f20e0102580 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.390+0000 7f20dfe00700 1 -- 172.21.15.63:0/4171703464 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20d400be70 con 0x7f20e0102580 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.390+0000 7f20dfe00700 1 -- 172.21.15.63:0/4171703464 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d4010380 con 0x7f20e0102580 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.391+0000 7f20e8200700 1 -- 172.21.15.63:0/4171703464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 msgr2=0x7f20e0102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.391+0000 7f20e8200700 1 --2- 172.21.15.63:0/4171703464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0102990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f20d40098e0 tx=0x7f20d4009250 comp rx=0 tx=0).stop 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.391+0000 7f20e8200700 1 -- 172.21.15.63:0/4171703464 shutdown_connections 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.391+0000 7f20e8200700 1 --2- 172.21.15.63:0/4171703464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0102990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.391+0000 7f20e8200700 1 -- 172.21.15.63:0/4171703464 >> 172.21.15.63:0/4171703464 conn(0x7f20e00fdd00 msgr2=0x7f20e0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.391+0000 7f20e8200700 1 -- 172.21.15.63:0/4171703464 shutdown_connections 2024-03-23T11:36:50.194 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.391+0000 7f20e8200700 1 -- 172.21.15.63:0/4171703464 wait complete. 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.392+0000 7f20e8200700 1 Processor -- start 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.392+0000 7f20e8200700 1 -- start start 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.392+0000 7f20e8200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0113590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.392+0000 7f20e8200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0113ad0 con 0x7f20e0102580 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.392+0000 7f20e5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0113590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.392+0000 7f20e5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0113590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36684/0 (socket says 172.21.15.63:36684) 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.392+0000 7f20e5800700 1 -- 172.21.15.63:0/2137680194 learned_addr learned my addr 172.21.15.63:0/2137680194 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.393+0000 7f20e5800700 1 -- 172.21.15.63:0/2137680194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d40094f0 con 0x7f20e0102580 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.393+0000 7f20e5800700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0113590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20d400c010 tx=0x7f20d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.393+0000 7f20dd600700 1 -- 172.21.15.63:0/2137680194 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d4010960 con 0x7f20e0102580 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.393+0000 7f20dd600700 1 -- 172.21.15.63:0/2137680194 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20d401a430 con 0x7f20e0102580 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.393+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e0113cd0 con 0x7f20e0102580 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.393+0000 7f20dd600700 1 -- 172.21.15.63:0/2137680194 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d4019580 con 0x7f20e0102580 2024-03-23T11:36:50.195 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.393+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e0114840 con 0x7f20e0102580 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.394+0000 7f20dd600700 1 -- 172.21.15.63:0/2137680194 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f20d401a980 con 0x7f20e0102580 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.394+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e005f5f0 con 0x7f20e0102580 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.395+0000 7f20dd600700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f20c8036790 0x7f20c8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.395+0000 7f20dd600700 1 -- 172.21.15.63:0/2137680194 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f20d404a690 con 0x7f20e0102580 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.395+0000 7f20e4e00700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f20c8036790 0x7f20c8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.396+0000 7f20e4e00700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f20c8036790 0x7f20c8038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009990 tx=0x7f20cc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.400+0000 7f20dd600700 1 -- 172.21.15.63:0/2137680194 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f20d402bb30 con 0x7f20e0102580 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.606+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f20e004bed0 con 0x7f20e0102580 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.610+0000 7f20dd600700 1 -- 172.21.15.63:0/2137680194 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f20d402b300 con 0x7f20e0102580 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f20c8036790 msgr2=0x7f20c8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f20c8036790 0x7f20c8038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009990 tx=0x7f20cc006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 msgr2=0x7f20e0113590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0113590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20d400c010 tx=0x7f20d4009250 comp rx=0 tx=0).stop 2024-03-23T11:36:50.196 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 shutdown_connections 2024-03-23T11:36:50.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f20c8036790 0x7f20c8038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:50.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 --2- 172.21.15.63:0/2137680194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102580 0x7f20e0113590 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:50.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 >> 172.21.15.63:0/2137680194 conn(0x7f20e00fdd00 msgr2=0x7f20e00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:50.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.612+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 shutdown_connections 2024-03-23T11:36:50.197 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:49.613+0000 7f20e8200700 1 -- 172.21.15.63:0/2137680194 wait complete. 2024-03-23T11:36:50.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:50 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2137680194' entity='client.admin' 2024-03-23T11:36:50.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:50 smithi063 ceph-mon[35189]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.775+0000 7f1a3e400700 1 Processor -- start 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.776+0000 7f1a3e400700 1 -- start start 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.776+0000 7f1a3e400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38102980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.776+0000 7f1a3e400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38102ec0 con 0x7f1a38102570 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.776+0000 7f1a37400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.776+0000 7f1a37400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38102980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36696/0 (socket says 172.21.15.63:36696) 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.776+0000 7f1a37400700 1 -- 172.21.15.63:0/2451270054 learned_addr learned my addr 172.21.15.63:0/2451270054 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:51.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.777+0000 7f1a37400700 1 -- 172.21.15.63:0/2451270054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a38103000 con 0x7f1a38102570 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.777+0000 7f1a37400700 1 --2- 172.21.15.63:0/2451270054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38102980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1a280098e0 tx=0x7f1a28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa1eaf3ce0a8c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.777+0000 7f1a36000700 1 -- 172.21.15.63:0/2451270054 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a2800b860 con 0x7f1a38102570 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.777+0000 7f1a36000700 1 -- 172.21.15.63:0/2451270054 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a2800be70 con 0x7f1a38102570 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.777+0000 7f1a36000700 1 -- 172.21.15.63:0/2451270054 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a28010380 con 0x7f1a38102570 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.778+0000 7f1a3e400700 1 -- 172.21.15.63:0/2451270054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 msgr2=0x7f1a38102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.778+0000 7f1a3e400700 1 --2- 172.21.15.63:0/2451270054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38102980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1a280098e0 tx=0x7f1a28009250 comp rx=0 tx=0).stop 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.778+0000 7f1a3e400700 1 -- 172.21.15.63:0/2451270054 shutdown_connections 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.778+0000 7f1a3e400700 1 --2- 172.21.15.63:0/2451270054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38102980 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.778+0000 7f1a3e400700 1 -- 172.21.15.63:0/2451270054 >> 172.21.15.63:0/2451270054 conn(0x7f1a380fdcd0 msgr2=0x7f1a38100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.778+0000 7f1a3e400700 1 -- 172.21.15.63:0/2451270054 shutdown_connections 2024-03-23T11:36:51.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.778+0000 7f1a3e400700 1 -- 172.21.15.63:0/2451270054 wait complete. 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.779+0000 7f1a3e400700 1 Processor -- start 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.779+0000 7f1a3e400700 1 -- start start 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.779+0000 7f1a3e400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38111400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.779+0000 7f1a3e400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38111940 con 0x7f1a38102570 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.779+0000 7f1a37400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38111400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.779+0000 7f1a37400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38111400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36702/0 (socket says 172.21.15.63:36702) 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.779+0000 7f1a37400700 1 -- 172.21.15.63:0/2971372785 learned_addr learned my addr 172.21.15.63:0/2971372785 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.780+0000 7f1a37400700 1 -- 172.21.15.63:0/2971372785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280094f0 con 0x7f1a38102570 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.780+0000 7f1a37400700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38111400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1a280090b0 tx=0x7f1a28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.780+0000 7f1a2f400700 1 -- 172.21.15.63:0/2971372785 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a28010800 con 0x7f1a38102570 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.780+0000 7f1a2f400700 1 -- 172.21.15.63:0/2971372785 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a28010e10 con 0x7f1a38102570 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.780+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a38111b40 con 0x7f1a38102570 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.780+0000 7f1a2f400700 1 -- 172.21.15.63:0/2971372785 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a2801ae60 con 0x7f1a38102570 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.780+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a381126b0 con 0x7f1a38102570 2024-03-23T11:36:51.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.781+0000 7f1a2f400700 1 -- 172.21.15.63:0/2971372785 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1a28010960 con 0x7f1a38102570 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.781+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3805f5f0 con 0x7f1a38102570 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.782+0000 7f1a2f400700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1a1c0367e0 0x7f1a1c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.782+0000 7f1a2f400700 1 -- 172.21.15.63:0/2971372785 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1a2804a6d0 con 0x7f1a38102570 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.782+0000 7f1a36a00700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1a1c0367e0 0x7f1a1c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.783+0000 7f1a36a00700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1a1c0367e0 0x7f1a1c038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009990 tx=0x7f1a24006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.787+0000 7f1a2f400700 1 -- 172.21.15.63:0/2971372785 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1a28018530 con 0x7f1a38102570 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.992+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f1a3804bed0 con 0x7f1a38102570 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.996+0000 7f1a2f400700 1 -- 172.21.15.63:0/2971372785 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f1a28018350 con 0x7f1a38102570 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1a1c0367e0 msgr2=0x7f1a1c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1a1c0367e0 0x7f1a1c038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009990 tx=0x7f1a24006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 msgr2=0x7f1a38111400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38111400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1a280090b0 tx=0x7f1a28009250 comp rx=0 tx=0).stop 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 shutdown_connections 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7f1a1c0367e0 0x7f1a1c038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:51.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 --2- 172.21.15.63:0/2971372785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38102570 0x7f1a38111400 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:51.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 >> 172.21.15.63:0/2971372785 conn(0x7f1a380fdcd0 msgr2=0x7f1a380fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:51.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 shutdown_connections 2024-03-23T11:36:51.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:50.999+0000 7f1a3e400700 1 -- 172.21.15.63:0/2971372785 wait complete. 2024-03-23T11:36:51.619 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling the dashboard module... 2024-03-23T11:36:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:52 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2971372785' entity='client.admin' 2024-03-23T11:36:53.293 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:53 smithi063 ceph-mon[35189]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:36:53.293 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:53 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1111322630' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.205+0000 7fd8c5a00700 1 Processor -- start 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.205+0000 7fd8c5a00700 1 -- start start 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.205+0000 7fd8c5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0102940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.205+0000 7fd8c5a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c0102e80 con 0x7fd8c0102530 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.206+0000 7fd8bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.206+0000 7fd8bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41776/0 (socket says 172.21.15.63:41776) 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.206+0000 7fd8bea00700 1 -- 172.21.15.63:0/894497617 learned_addr learned my addr 172.21.15.63:0/894497617 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.206+0000 7fd8bea00700 1 -- 172.21.15.63:0/894497617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c0102fc0 con 0x7fd8c0102530 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.206+0000 7fd8bea00700 1 --2- 172.21.15.63:0/894497617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0102940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00098e0 tx=0x7fd8b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d51563498a170a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8bd600700 1 -- 172.21.15.63:0/894497617 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b000b860 con 0x7fd8c0102530 2024-03-23T11:36:53.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8bd600700 1 -- 172.21.15.63:0/894497617 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8b000be70 con 0x7fd8c0102530 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8bd600700 1 -- 172.21.15.63:0/894497617 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b0010380 con 0x7fd8c0102530 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8c5a00700 1 -- 172.21.15.63:0/894497617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 msgr2=0x7fd8c0102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8c5a00700 1 --2- 172.21.15.63:0/894497617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0102940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00098e0 tx=0x7fd8b0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8c5a00700 1 -- 172.21.15.63:0/894497617 shutdown_connections 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8c5a00700 1 --2- 172.21.15.63:0/894497617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0102940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8c5a00700 1 -- 172.21.15.63:0/894497617 >> 172.21.15.63:0/894497617 conn(0x7fd8c00fdcd0 msgr2=0x7fd8c0100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8c5a00700 1 -- 172.21.15.63:0/894497617 shutdown_connections 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.207+0000 7fd8c5a00700 1 -- 172.21.15.63:0/894497617 wait complete. 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.208+0000 7fd8c5a00700 1 Processor -- start 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.209+0000 7fd8c5a00700 1 -- start start 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.209+0000 7fd8c5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0111460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.209+0000 7fd8c5a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c01119a0 con 0x7fd8c0102530 2024-03-23T11:36:53.613 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.209+0000 7fd8bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0111460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.209+0000 7fd8bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0111460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41788/0 (socket says 172.21.15.63:41788) 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8bea00700 1 -- 172.21.15.63:0/1111322630 learned_addr learned my addr 172.21.15.63:0/1111322630 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8bea00700 1 -- 172.21.15.63:0/1111322630 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00094f0 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8bea00700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0111460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00094c0 tx=0x7fd8b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b0010640 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8b0010c50 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c0111ba0 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b001ad30 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.210+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c0112710 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.211+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd8b0010dc0 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.211+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c005f5f0 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.212+0000 7fd8b6a00700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fd8a4036790 0x7fd8a4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.212+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd8b004a310 con 0x7fd8c0102530 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.212+0000 7fd8be000700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fd8a4036790 0x7fd8a4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:53.614 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.213+0000 7fd8be000700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fd8a4036790 0x7fd8a4038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009990 tx=0x7fd8ac006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.217+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd8b0018940 con 0x7fd8c0102530 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:52.473+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fd8c004bed0 con 0x7fd8c0102530 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.007+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fd8b0025400 con 0x7fd8c0102530 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.009+0000 7fd8b6a00700 1 -- 172.21.15.63:0/1111322630 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fd8b0016cc0 con 0x7fd8c0102530 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fd8a4036790 msgr2=0x7fd8a4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fd8a4036790 0x7fd8a4038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009990 tx=0x7fd8ac006e40 comp rx=0 tx=0).stop 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 msgr2=0x7fd8c0111460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0111460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00094c0 tx=0x7fd8b0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 shutdown_connections 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fd8a4036790 0x7fd8a4038c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 --2- 172.21.15.63:0/1111322630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8c0102530 0x7fd8c0111460 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 >> 172.21.15.63:0/1111322630 conn(0x7fd8c00fdcd0 msgr2=0x7fd8c00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 shutdown_connections 2024-03-23T11:36:53.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:53.012+0000 7fd8c5a00700 1 -- 172.21.15.63:0/1111322630 wait complete. 2024-03-23T11:36:54.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:54 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1111322630' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-23T11:36:54.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:54 smithi063 ceph-mon[35189]: mgrmap e9: smithi063.gehkcr(active, since 27s) 2024-03-23T11:36:55.087 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-03-23T11:36:55.087 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-23T11:36:55.087 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-23T11:36:55.087 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "smithi063.gehkcr", 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.248+0000 7fa4e3600700 1 Processor -- start 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.249+0000 7fa4e3600700 1 -- start start 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.249+0000 7fa4e3600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.249+0000 7fa4e3600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc079160 con 0x7fa4dc078d50 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.249+0000 7fa4e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.249+0000 7fa4e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41798/0 (socket says 172.21.15.63:41798) 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.249+0000 7fa4e0c00700 1 -- 172.21.15.63:0/1166435289 learned_addr learned my addr 172.21.15.63:0/1166435289 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.250+0000 7fa4e0c00700 1 -- 172.21.15.63:0/1166435289 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc0777a0 con 0x7fa4dc078d50 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.250+0000 7fa4e0c00700 1 --2- 172.21.15.63:0/1166435289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc077260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00098e0 tx=0x7fa4d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bdd774fc2386d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.250+0000 7fa4db400700 1 -- 172.21.15.63:0/1166435289 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d000b860 con 0x7fa4dc078d50 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.250+0000 7fa4db400700 1 -- 172.21.15.63:0/1166435289 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4d000be70 con 0x7fa4dc078d50 2024-03-23T11:36:55.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.250+0000 7fa4db400700 1 -- 172.21.15.63:0/1166435289 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d0010380 con 0x7fa4dc078d50 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.251+0000 7fa4e3600700 1 -- 172.21.15.63:0/1166435289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 msgr2=0x7fa4dc077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.251+0000 7fa4e3600700 1 --2- 172.21.15.63:0/1166435289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc077260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00098e0 tx=0x7fa4d0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.251+0000 7fa4e3600700 1 -- 172.21.15.63:0/1166435289 shutdown_connections 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.251+0000 7fa4e3600700 1 --2- 172.21.15.63:0/1166435289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc077260 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.251+0000 7fa4e3600700 1 -- 172.21.15.63:0/1166435289 >> 172.21.15.63:0/1166435289 conn(0x7fa4dc0fdd00 msgr2=0x7fa4dc100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.251+0000 7fa4e3600700 1 -- 172.21.15.63:0/1166435289 shutdown_connections 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.251+0000 7fa4e3600700 1 -- 172.21.15.63:0/1166435289 wait complete. 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.252+0000 7fa4e3600700 1 Processor -- start 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.252+0000 7fa4e3600700 1 -- start start 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.252+0000 7fa4e3600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc1135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.252+0000 7fa4e3600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc113ae0 con 0x7fa4dc078d50 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.253+0000 7fa4e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc1135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.253+0000 7fa4e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc1135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41802/0 (socket says 172.21.15.63:41802) 2024-03-23T11:36:55.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.253+0000 7fa4e0c00700 1 -- 172.21.15.63:0/2260963468 learned_addr learned my addr 172.21.15.63:0/2260963468 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.253+0000 7fa4e0c00700 1 -- 172.21.15.63:0/2260963468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d00094f0 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.253+0000 7fa4e0c00700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc1135a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0006d40 tx=0x7fa4d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.253+0000 7fa4d8c00700 1 -- 172.21.15.63:0/2260963468 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d0010960 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.254+0000 7fa4d8c00700 1 -- 172.21.15.63:0/2260963468 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4d001a430 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.254+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc113ce0 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.254+0000 7fa4d8c00700 1 -- 172.21.15.63:0/2260963468 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d00245b0 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.254+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc114850 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.255+0000 7fa4d8c00700 1 -- 172.21.15.63:0/2260963468 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa4d001a9f0 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.255+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc05f5f0 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.255+0000 7fa4d8c00700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 0x7fa4c4038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.256+0000 7fa4d8c00700 1 -- 172.21.15.63:0/2260963468 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa4d004a3d0 con 0x7fa4dc078d50 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.256+0000 7fa4dbe00700 1 -- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 msgr2=0x7fa4c4038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1704436637 2024-03-23T11:36:55.090 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.256+0000 7fa4dbe00700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 0x7fa4c4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.260+0000 7fa4d8c00700 1 -- 172.21.15.63:0/2260963468 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa4d002b8c0 con 0x7fa4dc078d50 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.457+0000 7fa4dbe00700 1 -- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 msgr2=0x7fa4c4038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1704436637 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.457+0000 7fa4dbe00700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 0x7fa4c4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.520+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa4dc114b90 con 0x7fa4dc078d50 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.521+0000 7fa4d8c00700 1 -- 172.21.15.63:0/2260963468 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa4d002b2f0 con 0x7fa4dc078d50 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 msgr2=0x7fa4c4038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 0x7fa4c4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 msgr2=0x7fa4dc1135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc1135a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0006d40 tx=0x7fa4d0009250 comp rx=0 tx=0).stop 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 shutdown_connections 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7fa4c4036830 0x7fa4c4038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 --2- 172.21.15.63:0/2260963468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4dc078d50 0x7fa4dc1135a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.523+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 >> 172.21.15.63:0/2260963468 conn(0x7fa4dc0fdd00 msgr2=0x7fa4dc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:36:55.091 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.524+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 shutdown_connections 2024-03-23T11:36:55.092 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:54.524+0000 7fa4e3600700 1 -- 172.21.15.63:0/2260963468 wait complete. 2024-03-23T11:36:55.092 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-03-23T11:36:55.092 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 9... 2024-03-23T11:36:55.092 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:36:55 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2260963468' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-23T11:37:01.805 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: Active manager daemon smithi063.gehkcr restarted 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: Activating manager daemon smithi063.gehkcr 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: osdmap e3: 0 total, 0 up, 0 in 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: mgrmap e10: smithi063.gehkcr(active, starting, since 0.0041009s) 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:37:01.806 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:01 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:37:03.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:02 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:37:03.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:02 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:37:03.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:02 smithi063 ceph-mon[35189]: [23/Mar/2024:11:37:02] ENGINE Bus STARTING 2024-03-23T11:37:03.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:02 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:03.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:02 smithi063 ceph-mon[35189]: mgrmap e11: smithi063.gehkcr(active, since 1.007s) 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.746+0000 7efc8b200700 1 Processor -- start 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.746+0000 7efc8b200700 1 -- start start 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.746+0000 7efc8b200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc84077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.746+0000 7efc8b200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84079160 con 0x7efc84078d50 2024-03-23T11:37:03.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.747+0000 7efc83e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc84077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:03.677 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.747+0000 7efc83e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc84077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41816/0 (socket says 172.21.15.63:41816) 2024-03-23T11:37:03.677 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.747+0000 7efc83e00700 1 -- 172.21.15.63:0/1003197495 learned_addr learned my addr 172.21.15.63:0/1003197495 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:03.677 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.747+0000 7efc83e00700 1 -- 172.21.15.63:0/1003197495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc840777a0 con 0x7efc84078d50 2024-03-23T11:37:03.677 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.748+0000 7efc83e00700 1 --2- 172.21.15.63:0/1003197495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc84077260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efc740098e0 tx=0x7efc74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611fd576dd71039f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:03.683 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.748+0000 7efc82a00700 1 -- 172.21.15.63:0/1003197495 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc7400b860 con 0x7efc84078d50 2024-03-23T11:37:03.683 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.748+0000 7efc82a00700 1 -- 172.21.15.63:0/1003197495 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc7400be70 con 0x7efc84078d50 2024-03-23T11:37:03.683 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.748+0000 7efc82a00700 1 -- 172.21.15.63:0/1003197495 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc74010380 con 0x7efc84078d50 2024-03-23T11:37:03.683 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.748+0000 7efc8b200700 1 -- 172.21.15.63:0/1003197495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 msgr2=0x7efc84077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:03.683 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.748+0000 7efc8b200700 1 --2- 172.21.15.63:0/1003197495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc84077260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efc740098e0 tx=0x7efc74009250 comp rx=0 tx=0).stop 2024-03-23T11:37:03.683 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.749+0000 7efc8b200700 1 -- 172.21.15.63:0/1003197495 shutdown_connections 2024-03-23T11:37:03.683 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.749+0000 7efc8b200700 1 --2- 172.21.15.63:0/1003197495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc84077260 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.749+0000 7efc8b200700 1 -- 172.21.15.63:0/1003197495 >> 172.21.15.63:0/1003197495 conn(0x7efc840fdd00 msgr2=0x7efc84100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.749+0000 7efc8b200700 1 -- 172.21.15.63:0/1003197495 shutdown_connections 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.749+0000 7efc8b200700 1 -- 172.21.15.63:0/1003197495 wait complete. 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.750+0000 7efc8b200700 1 Processor -- start 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.750+0000 7efc8b200700 1 -- start start 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.750+0000 7efc8b200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc841135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.750+0000 7efc8b200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84113ae0 con 0x7efc84078d50 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.750+0000 7efc83e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc841135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.750+0000 7efc83e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc841135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41822/0 (socket says 172.21.15.63:41822) 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.750+0000 7efc83e00700 1 -- 172.21.15.63:0/3461434516 learned_addr learned my addr 172.21.15.63:0/3461434516 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.751+0000 7efc83e00700 1 -- 172.21.15.63:0/3461434516 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc740094f0 con 0x7efc84078d50 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.751+0000 7efc83e00700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc841135a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efc74006d40 tx=0x7efc74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.751+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc74010960 con 0x7efc84078d50 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.751+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc7401a430 con 0x7efc84078d50 2024-03-23T11:37:03.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.751+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc740245b0 con 0x7efc84078d50 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.751+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc84113ce0 con 0x7efc84078d50 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.751+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc84114850 con 0x7efc84078d50 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.752+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7efc7401a9f0 con 0x7efc84078d50 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.753+0000 7efc7be00700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.753+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 --> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc8405f5f0 con 0x7efc68036830 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.753+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7efc7404a3d0 con 0x7efc84078d50 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.753+0000 7efc83400700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 msgr2=0x7efc68038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1704436637 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.753+0000 7efc83400700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.954+0000 7efc83400700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 msgr2=0x7efc68038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1704436637 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:55.954+0000 7efc83400700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:56.355+0000 7efc83400700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 msgr2=0x7efc68038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1704436637 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:56.355+0000 7efc83400700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:57.156+0000 7efc83400700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 msgr2=0x7efc68038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1704436637 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:57.156+0000 7efc83400700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T11:37:03.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:58.757+0000 7efc83400700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 msgr2=0x7efc68038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1704436637 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:36:58.757+0000 7efc83400700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:01.695+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7efc7402d3f0 con 0x7efc84078d50 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:01.695+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 msgr2=0x7efc68038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:01.695+0000 7efc7be00700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.697+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7efc74010ac0 con 0x7efc84078d50 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.698+0000 7efc7be00700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7efc68039ff0 0x7efc6803c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.698+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc8405f5f0 con 0x7efc68039ff0 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.698+0000 7efc83400700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7efc68039ff0 0x7efc6803c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.699+0000 7efc83400700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7efc68039ff0 0x7efc6803c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efc700039d0 tx=0x7efc70009350 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.700+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mgr.14158 v2:172.21.15.63:6800/213424997 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7efc8405f5f0 con 0x7efc68039ff0 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.707+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7efc84114170 con 0x7efc68039ff0 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.707+0000 7efc7be00700 1 -- 172.21.15.63:0/3461434516 <== mgr.14158 v2:172.21.15.63:6800/213424997 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7efc84114170 con 0x7efc68039ff0 2024-03-23T11:37:03.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7efc68039ff0 msgr2=0x7efc6803c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7efc68039ff0 0x7efc6803c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efc700039d0 tx=0x7efc70009350 comp rx=0 tx=0).stop 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 msgr2=0x7efc841135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc841135a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efc74006d40 tx=0x7efc74009250 comp rx=0 tx=0).stop 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 shutdown_connections 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7efc68039ff0 0x7efc6803c3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efc84078d50 0x7efc841135a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 --2- 172.21.15.63:0/3461434516 >> [v2:172.21.15.63:6800/1704436637,v1:172.21.15.63:6801/1704436637] conn(0x7efc68036830 0x7efc68038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 >> 172.21.15.63:0/3461434516 conn(0x7efc840fdd00 msgr2=0x7efc840fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 shutdown_connections 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:02.708+0000 7efc8b200700 1 -- 172.21.15.63:0/3461434516 wait complete. 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 9 is available 2024-03-23T11:37:03.687 INFO:teuthology.orchestra.run.smithi063.stdout:Generating a dashboard self-signed certificate... 2024-03-23T11:37:03.961 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:03 smithi063 ceph-mon[35189]: [23/Mar/2024:11:37:02] ENGINE Serving on https://172.21.15.63:7150 2024-03-23T11:37:03.961 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:03 smithi063 ceph-mon[35189]: [23/Mar/2024:11:37:02] ENGINE Bus STARTED 2024-03-23T11:37:03.961 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:03 smithi063 ceph-mon[35189]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-23T11:37:03.961 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:03 smithi063 ceph-mon[35189]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-23T11:37:03.962 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:03 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:05.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-23T11:37:05.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.244+0000 7f8e1f600700 1 Processor -- start 2024-03-23T11:37:05.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.244+0000 7f8e1f600700 1 -- start start 2024-03-23T11:37:05.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.244+0000 7f8e1f600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e18077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:05.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.244+0000 7f8e1f600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18079160 con 0x7f8e18078d50 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.245+0000 7f8e1cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e18077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.245+0000 7f8e1cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e18077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44858/0 (socket says 172.21.15.63:44858) 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.245+0000 7f8e1cc00700 1 -- 172.21.15.63:0/447408473 learned_addr learned my addr 172.21.15.63:0/447408473 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.246+0000 7f8e1cc00700 1 -- 172.21.15.63:0/447408473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180777a0 con 0x7f8e18078d50 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.246+0000 7f8e1cc00700 1 --2- 172.21.15.63:0/447408473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e18077260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0098e0 tx=0x7f8e0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66cad4f5819ce126 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.246+0000 7f8e17400700 1 -- 172.21.15.63:0/447408473 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0c00b860 con 0x7f8e18078d50 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.246+0000 7f8e17400700 1 -- 172.21.15.63:0/447408473 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e0c00be70 con 0x7f8e18078d50 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.246+0000 7f8e17400700 1 -- 172.21.15.63:0/447408473 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0c010380 con 0x7f8e18078d50 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.247+0000 7f8e1f600700 1 -- 172.21.15.63:0/447408473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 msgr2=0x7f8e18077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.247+0000 7f8e1f600700 1 --2- 172.21.15.63:0/447408473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e18077260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0098e0 tx=0x7f8e0c009250 comp rx=0 tx=0).stop 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.248+0000 7f8e1f600700 1 -- 172.21.15.63:0/447408473 shutdown_connections 2024-03-23T11:37:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.248+0000 7f8e1f600700 1 --2- 172.21.15.63:0/447408473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e18077260 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.248+0000 7f8e1f600700 1 -- 172.21.15.63:0/447408473 >> 172.21.15.63:0/447408473 conn(0x7f8e180fdd00 msgr2=0x7f8e18100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.248+0000 7f8e1f600700 1 -- 172.21.15.63:0/447408473 shutdown_connections 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.248+0000 7f8e1f600700 1 -- 172.21.15.63:0/447408473 wait complete. 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.249+0000 7f8e1f600700 1 Processor -- start 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.249+0000 7f8e1f600700 1 -- start start 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.249+0000 7f8e1f600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e181135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.249+0000 7f8e1f600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18113ae0 con 0x7f8e18078d50 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.250+0000 7f8e1cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e181135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.250+0000 7f8e1cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e181135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44866/0 (socket says 172.21.15.63:44866) 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.250+0000 7f8e1cc00700 1 -- 172.21.15.63:0/4205639877 learned_addr learned my addr 172.21.15.63:0/4205639877 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.250+0000 7f8e1cc00700 1 -- 172.21.15.63:0/4205639877 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c0094f0 con 0x7f8e18078d50 2024-03-23T11:37:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.250+0000 7f8e1cc00700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e181135a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c006d40 tx=0x7f8e0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.251+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0c010960 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.251+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e0c01a430 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.251+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e18113ce0 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.251+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e0c0245b0 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.251+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e18114850 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.252+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1805f5f0 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.252+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f8e0c02d3f0 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.253+0000 7f8e14c00700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8e000366b0 0x7f8e00038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.253+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8e0c04a410 con 0x7f8e18078d50 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.253+0000 7f8e17e00700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8e000366b0 0x7f8e00038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.254+0000 7f8e17e00700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8e000366b0 0x7f8e00038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009990 tx=0x7f8e04006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.260+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8e0c018b80 con 0x7f8e18078d50 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.463+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f8e0c01a9f0 con 0x7f8e18078d50 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.490+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e180fef50 con 0x7f8e000366b0 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.578+0000 7f8e14c00700 1 -- 172.21.15.63:0/4205639877 <== mgr.14158 v2:172.21.15.63:6800/213424997 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8e180fef50 con 0x7f8e000366b0 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8e000366b0 msgr2=0x7f8e00038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8e000366b0 0x7f8e00038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009990 tx=0x7f8e04006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 msgr2=0x7f8e181135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e181135a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c006d40 tx=0x7f8e0c009250 comp rx=0 tx=0).stop 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 shutdown_connections 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8e000366b0 0x7f8e00038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 --2- 172.21.15.63:0/4205639877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e18078d50 0x7f8e181135a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.580+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 >> 172.21.15.63:0/4205639877 conn(0x7f8e180fdd00 msgr2=0x7f8e180fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:05.223 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.581+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 shutdown_connections 2024-03-23T11:37:05.224 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:04.581+0000 7f8e1f600700 1 -- 172.21.15.63:0/4205639877 wait complete. 2024-03-23T11:37:05.224 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial admin user... 2024-03-23T11:37:05.504 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:05 smithi063 ceph-mon[35189]: mgrmap e12: smithi063.gehkcr(active, since 2s) 2024-03-23T11:37:05.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:05 smithi063 ceph-mon[35189]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:37:05.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:05 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:05.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:05 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:05.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:05 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:06.600 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:06 smithi063 ceph-mon[35189]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:37:06.600 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:06 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$uvTrFh9RAf118BTlzRB/bOASEms3Z.OA2ia9l/9U3k4v37pS/aus.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711193826, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.792+0000 7f8df4600700 1 Processor -- start 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.792+0000 7f8df4600700 1 -- start start 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.793+0000 7f8df4600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.793+0000 7f8df4600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec079160 con 0x7f8dec078d50 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.793+0000 7f8df1c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.793+0000 7f8df1c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44872/0 (socket says 172.21.15.63:44872) 2024-03-23T11:37:07.557 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.793+0000 7f8df1c00700 1 -- 172.21.15.63:0/2054643093 learned_addr learned my addr 172.21.15.63:0/2054643093 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.794+0000 7f8df1c00700 1 -- 172.21.15.63:0/2054643093 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec0777a0 con 0x7f8dec078d50 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.794+0000 7f8df1c00700 1 --2- 172.21.15.63:0/2054643093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec077260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8de00098e0 tx=0x7f8de0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69012a73f311557c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.794+0000 7f8debe00700 1 -- 172.21.15.63:0/2054643093 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de000b860 con 0x7f8dec078d50 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.794+0000 7f8debe00700 1 -- 172.21.15.63:0/2054643093 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8de000be70 con 0x7f8dec078d50 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.794+0000 7f8debe00700 1 -- 172.21.15.63:0/2054643093 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de0010380 con 0x7f8dec078d50 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 -- 172.21.15.63:0/2054643093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 msgr2=0x7f8dec077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 --2- 172.21.15.63:0/2054643093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec077260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8de00098e0 tx=0x7f8de0009250 comp rx=0 tx=0).stop 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 -- 172.21.15.63:0/2054643093 shutdown_connections 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 --2- 172.21.15.63:0/2054643093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec077260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 -- 172.21.15.63:0/2054643093 >> 172.21.15.63:0/2054643093 conn(0x7f8dec0fdd00 msgr2=0x7f8dec100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 -- 172.21.15.63:0/2054643093 shutdown_connections 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 -- 172.21.15.63:0/2054643093 wait complete. 2024-03-23T11:37:07.558 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 Processor -- start 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 -- start start 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec1135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.796+0000 7f8df4600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec113af0 con 0x7f8dec078d50 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8df1c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec1135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8df1c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec1135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44888/0 (socket says 172.21.15.63:44888) 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8df1c00700 1 -- 172.21.15.63:0/1130562737 learned_addr learned my addr 172.21.15.63:0/1130562737 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8df1c00700 1 -- 172.21.15.63:0/1130562737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00094f0 con 0x7f8dec078d50 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8df1c00700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec1135b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006d40 tx=0x7f8de0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8de9600700 1 -- 172.21.15.63:0/1130562737 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de0010960 con 0x7f8dec078d50 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8de9600700 1 -- 172.21.15.63:0/1130562737 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8de001a430 con 0x7f8dec078d50 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8de9600700 1 -- 172.21.15.63:0/1130562737 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de00245b0 con 0x7f8dec078d50 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.797+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dec113cf0 con 0x7f8dec078d50 2024-03-23T11:37:07.559 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.799+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dec114860 con 0x7f8dec078d50 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.799+0000 7f8de9600700 1 -- 172.21.15.63:0/1130562737 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f8de001a9f0 con 0x7f8dec078d50 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.799+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dec05f5f0 con 0x7f8dec078d50 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.799+0000 7f8de9600700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8dd4036830 0x7f8dd4038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.799+0000 7f8de9600700 1 -- 172.21.15.63:0/1130562737 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8de004a3f0 con 0x7f8dec078d50 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.799+0000 7f8df1200700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8dd4036830 0x7f8dd4038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.801+0000 7f8df1200700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8dd4036830 0x7f8dd4038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8009990 tx=0x7f8dd8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:05.805+0000 7f8de9600700 1 -- 172.21.15.63:0/1130562737 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8de002b8c0 con 0x7f8dec078d50 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.034+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8dec05f410 con 0x7f8dd4036830 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.283+0000 7f8de9600700 1 -- 172.21.15.63:0/1130562737 <== mgr.14158 v2:172.21.15.63:6800/213424997 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f8dec05f410 con 0x7f8dd4036830 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8dd4036830 msgr2=0x7f8dd4038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8dd4036830 0x7f8dd4038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8009990 tx=0x7f8dd8006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 msgr2=0x7f8dec1135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec1135b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006d40 tx=0x7f8de0009250 comp rx=0 tx=0).stop 2024-03-23T11:37:07.561 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 shutdown_connections 2024-03-23T11:37:07.561 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f8dd4036830 0x7f8dd4038ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:07.561 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 --2- 172.21.15.63:0/1130562737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dec078d50 0x7f8dec1135b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:07.561 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 >> 172.21.15.63:0/1130562737 conn(0x7f8dec0fdd00 msgr2=0x7f8dec0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:07.561 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 shutdown_connections 2024-03-23T11:37:07.561 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:06.285+0000 7f8df4600700 1 -- 172.21.15.63:0/1130562737 wait complete. 2024-03-23T11:37:07.561 INFO:teuthology.orchestra.run.smithi063.stdout:Fetching dashboard port number... 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 8443 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.143+0000 7f4be9000700 1 Processor -- start 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.143+0000 7f4be9000700 1 -- start start 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.143+0000 7f4be9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.143+0000 7f4be9000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be4079170 con 0x7f4be4078d60 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.143+0000 7f4be1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.144+0000 7f4be1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44896/0 (socket says 172.21.15.63:44896) 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.144+0000 7f4be1e00700 1 -- 172.21.15.63:0/2891527315 learned_addr learned my addr 172.21.15.63:0/2891527315 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:08.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.144+0000 7f4be1e00700 1 -- 172.21.15.63:0/2891527315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be40777a0 con 0x7f4be4078d60 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.144+0000 7f4be1e00700 1 --2- 172.21.15.63:0/2891527315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4077260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40098e0 tx=0x7f4bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b07dd24b670ed9c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.144+0000 7f4be0a00700 1 -- 172.21.15.63:0/2891527315 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd400b860 con 0x7f4be4078d60 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be0a00700 1 -- 172.21.15.63:0/2891527315 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bd400be70 con 0x7f4be4078d60 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be0a00700 1 -- 172.21.15.63:0/2891527315 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd4010380 con 0x7f4be4078d60 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be9000700 1 -- 172.21.15.63:0/2891527315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 msgr2=0x7f4be4077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be9000700 1 --2- 172.21.15.63:0/2891527315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4077260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40098e0 tx=0x7f4bd4009250 comp rx=0 tx=0).stop 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be9000700 1 -- 172.21.15.63:0/2891527315 shutdown_connections 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be9000700 1 --2- 172.21.15.63:0/2891527315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4077260 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be9000700 1 -- 172.21.15.63:0/2891527315 >> 172.21.15.63:0/2891527315 conn(0x7f4be40fdcd0 msgr2=0x7f4be4100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be9000700 1 -- 172.21.15.63:0/2891527315 shutdown_connections 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.145+0000 7f4be9000700 1 -- 172.21.15.63:0/2891527315 wait complete. 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.146+0000 7f4be9000700 1 Processor -- start 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.146+0000 7f4be9000700 1 -- start start 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.147+0000 7f4be9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.147+0000 7f4be9000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be4113b70 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.147+0000 7f4be1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.147+0000 7f4be1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44912/0 (socket says 172.21.15.63:44912) 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.147+0000 7f4be1e00700 1 -- 172.21.15.63:0/423219808 learned_addr learned my addr 172.21.15.63:0/423219808 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.147+0000 7f4be1e00700 1 -- 172.21.15.63:0/423219808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd40094f0 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.147+0000 7f4be1e00700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4113630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4011b60 tx=0x7f4bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.148+0000 7f4bda000700 1 -- 172.21.15.63:0/423219808 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd4010800 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.148+0000 7f4bda000700 1 -- 172.21.15.63:0/423219808 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bd4010e10 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.148+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be4113d70 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.148+0000 7f4bda000700 1 -- 172.21.15.63:0/423219808 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd401ae60 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.148+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be41148e0 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.149+0000 7f4bda000700 1 -- 172.21.15.63:0/423219808 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f4bd401a430 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.149+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be405f5f0 con 0x7f4be4078d60 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.149+0000 7f4bda000700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f4bc8036880 0x7f4bc8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.149+0000 7f4bda000700 1 -- 172.21.15.63:0/423219808 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4bd404a8b0 con 0x7f4be4078d60 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.150+0000 7f4be1400700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f4bc8036880 0x7f4bc8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.150+0000 7f4be1400700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f4bc8036880 0x7f4bc8038d30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009990 tx=0x7f4bd0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.155+0000 7f4bda000700 1 -- 172.21.15.63:0/423219808 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4bd402b950 con 0x7f4be4078d60 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.360+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4be404bed0 con 0x7f4be4078d60 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.360+0000 7f4bda000700 1 -- 172.21.15.63:0/423219808 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4bd402b300 con 0x7f4be4078d60 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.362+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f4bc8036880 msgr2=0x7f4bc8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.362+0000 7f4be9000700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f4bc8036880 0x7f4bc8038d30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009990 tx=0x7f4bd0006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.362+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 msgr2=0x7f4be4113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.362+0000 7f4be9000700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4113630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4011b60 tx=0x7f4bd4009250 comp rx=0 tx=0).stop 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.362+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 shutdown_connections 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.362+0000 7f4be9000700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f4bc8036880 0x7f4bc8038d30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.362+0000 7f4be9000700 1 --2- 172.21.15.63:0/423219808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4be4078d60 0x7f4be4113630 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.363+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 >> 172.21.15.63:0/423219808 conn(0x7f4be40fdcd0 msgr2=0x7f4be40fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.363+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 shutdown_connections 2024-03-23T11:37:08.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:08.363+0000 7f4be9000700 1 -- 172.21.15.63:0/423219808 wait complete. 2024-03-23T11:37:08.923 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:08 smithi063 ceph-mon[35189]: mgrmap e13: smithi063.gehkcr(active, since 6s) 2024-03-23T11:37:08.923 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:08 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/423219808' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-23T11:37:08.935 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-23T11:37:08.935 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-03-23T11:37:08.952 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-23T11:37:08.952 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-03-23T11:37:08.952 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-03-23T11:37:08.952 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-23T11:37:08.955 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph Dashboard is now available at: 2024-03-23T11:37:08.955 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:08.955 INFO:teuthology.orchestra.run.smithi063.stdout: URL: https://smithi063.front.sepia.ceph.com:8443/ 2024-03-23T11:37:08.955 INFO:teuthology.orchestra.run.smithi063.stdout: User: admin 2024-03-23T11:37:08.955 INFO:teuthology.orchestra.run.smithi063.stdout: Password: jiflcw8d6i 2024-03-23T11:37:08.955 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:08.956 INFO:teuthology.orchestra.run.smithi063.stdout:Saving cluster configuration to /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config directory 2024-03-23T11:37:08.956 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling autotune for osd_memory_target 2024-03-23T11:37:10.299 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.505+0000 7fcc95000700 1 Processor -- start 2024-03-23T11:37:10.299 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.505+0000 7fcc95000700 1 -- start start 2024-03-23T11:37:10.299 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.505+0000 7fcc95000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90102940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:10.299 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.505+0000 7fcc95000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90102e80 con 0x7fcc90102530 2024-03-23T11:37:10.299 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.506+0000 7fcc8de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:10.299 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.506+0000 7fcc8de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44922/0 (socket says 172.21.15.63:44922) 2024-03-23T11:37:10.299 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.506+0000 7fcc8de00700 1 -- 172.21.15.63:0/479955307 learned_addr learned my addr 172.21.15.63:0/479955307 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.506+0000 7fcc8de00700 1 -- 172.21.15.63:0/479955307 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc90102fc0 con 0x7fcc90102530 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.506+0000 7fcc8de00700 1 --2- 172.21.15.63:0/479955307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90102940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcc800098e0 tx=0x7fcc80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83dc09e7b6e2734 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc8ca00700 1 -- 172.21.15.63:0/479955307 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8000b860 con 0x7fcc90102530 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc8ca00700 1 -- 172.21.15.63:0/479955307 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc8000be70 con 0x7fcc90102530 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc8ca00700 1 -- 172.21.15.63:0/479955307 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc80010380 con 0x7fcc90102530 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc95000700 1 -- 172.21.15.63:0/479955307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 msgr2=0x7fcc90102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc95000700 1 --2- 172.21.15.63:0/479955307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90102940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcc800098e0 tx=0x7fcc80009250 comp rx=0 tx=0).stop 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc95000700 1 -- 172.21.15.63:0/479955307 shutdown_connections 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc95000700 1 --2- 172.21.15.63:0/479955307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90102940 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc95000700 1 -- 172.21.15.63:0/479955307 >> 172.21.15.63:0/479955307 conn(0x7fcc900fdcd0 msgr2=0x7fcc90100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc95000700 1 -- 172.21.15.63:0/479955307 shutdown_connections 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.507+0000 7fcc95000700 1 -- 172.21.15.63:0/479955307 wait complete. 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.508+0000 7fcc95000700 1 Processor -- start 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.508+0000 7fcc95000700 1 -- start start 2024-03-23T11:37:10.300 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.509+0000 7fcc95000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90111460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.509+0000 7fcc95000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc901119a0 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.509+0000 7fcc8de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90111460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.509+0000 7fcc8de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90111460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44924/0 (socket says 172.21.15.63:44924) 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.509+0000 7fcc8de00700 1 -- 172.21.15.63:0/3402694697 learned_addr learned my addr 172.21.15.63:0/3402694697 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.509+0000 7fcc8de00700 1 -- 172.21.15.63:0/3402694697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800094f0 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.509+0000 7fcc8de00700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90111460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc80000c00 tx=0x7fcc80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.510+0000 7fcc86000700 1 -- 172.21.15.63:0/3402694697 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc800105f0 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.510+0000 7fcc86000700 1 -- 172.21.15.63:0/3402694697 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc80010c00 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.510+0000 7fcc86000700 1 -- 172.21.15.63:0/3402694697 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8001ac70 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.510+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc90111ba0 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.510+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc90112710 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.511+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9005f5f0 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.511+0000 7fcc86000700 1 -- 172.21.15.63:0/3402694697 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fcc80010d70 con 0x7fcc90102530 2024-03-23T11:37:10.301 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.511+0000 7fcc86000700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fcc740368d0 0x7fcc74038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.511+0000 7fcc86000700 1 -- 172.21.15.63:0/3402694697 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcc8004a520 con 0x7fcc90102530 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.512+0000 7fcc8d400700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fcc740368d0 0x7fcc74038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.512+0000 7fcc8d400700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fcc740368d0 0x7fcc74038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009990 tx=0x7fcc7c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.516+0000 7fcc86000700 1 -- 172.21.15.63:0/3402694697 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcc80018530 con 0x7fcc90102530 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.723+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fcc9004bed0 con 0x7fcc90102530 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.723+0000 7fcc86000700 1 -- 172.21.15.63:0/3402694697 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fcc80020030 con 0x7fcc90102530 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.725+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fcc740368d0 msgr2=0x7fcc74038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.725+0000 7fcc95000700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fcc740368d0 0x7fcc74038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009990 tx=0x7fcc7c006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.725+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 msgr2=0x7fcc90111460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.726+0000 7fcc95000700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90111460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc80000c00 tx=0x7fcc80009250 comp rx=0 tx=0).stop 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.726+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 shutdown_connections 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.726+0000 7fcc95000700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fcc740368d0 0x7fcc74038d80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.726+0000 7fcc95000700 1 --2- 172.21.15.63:0/3402694697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc90102530 0x7fcc90111460 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:10.303 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.726+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 >> 172.21.15.63:0/3402694697 conn(0x7fcc900fdcd0 msgr2=0x7fcc900fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:10.303 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.726+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 shutdown_connections 2024-03-23T11:37:10.303 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:09.726+0000 7fcc95000700 1 -- 172.21.15.63:0/3402694697 wait complete. 2024-03-23T11:37:11.713 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.867+0000 7ff520a00700 1 Processor -- start 2024-03-23T11:37:11.713 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.867+0000 7ff520a00700 1 -- start start 2024-03-23T11:37:11.713 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.867+0000 7ff520a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c102940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.867+0000 7ff520a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c102e80 con 0x7ff51c102530 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.868+0000 7ff519e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.868+0000 7ff519e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44928/0 (socket says 172.21.15.63:44928) 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.868+0000 7ff519e00700 1 -- 172.21.15.63:0/2470407745 learned_addr learned my addr 172.21.15.63:0/2470407745 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.868+0000 7ff519e00700 1 -- 172.21.15.63:0/2470407745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c102fc0 con 0x7ff51c102530 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.868+0000 7ff519e00700 1 --2- 172.21.15.63:0/2470407745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c102940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0098e0 tx=0x7ff50c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b11c937cd51762 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.868+0000 7ff518a00700 1 -- 172.21.15.63:0/2470407745 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c00b860 con 0x7ff51c102530 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff518a00700 1 -- 172.21.15.63:0/2470407745 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff50c00be70 con 0x7ff51c102530 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff518a00700 1 -- 172.21.15.63:0/2470407745 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c010380 con 0x7ff51c102530 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff520a00700 1 -- 172.21.15.63:0/2470407745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 msgr2=0x7ff51c102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff520a00700 1 --2- 172.21.15.63:0/2470407745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c102940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0098e0 tx=0x7ff50c009250 comp rx=0 tx=0).stop 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff520a00700 1 -- 172.21.15.63:0/2470407745 shutdown_connections 2024-03-23T11:37:11.714 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff520a00700 1 --2- 172.21.15.63:0/2470407745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c102940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff520a00700 1 -- 172.21.15.63:0/2470407745 >> 172.21.15.63:0/2470407745 conn(0x7ff51c0fdcd0 msgr2=0x7ff51c100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff520a00700 1 -- 172.21.15.63:0/2470407745 shutdown_connections 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.869+0000 7ff520a00700 1 -- 172.21.15.63:0/2470407745 wait complete. 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.870+0000 7ff520a00700 1 Processor -- start 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.870+0000 7ff520a00700 1 -- start start 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.871+0000 7ff520a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.871+0000 7ff520a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c113bb0 con 0x7ff51c102530 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.871+0000 7ff519e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.871+0000 7ff519e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44936/0 (socket says 172.21.15.63:44936) 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.871+0000 7ff519e00700 1 -- 172.21.15.63:0/1294116263 learned_addr learned my addr 172.21.15.63:0/1294116263 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.871+0000 7ff519e00700 1 -- 172.21.15.63:0/1294116263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c0094f0 con 0x7ff51c102530 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.871+0000 7ff519e00700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c113670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0094c0 tx=0x7ff50c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.872+0000 7ff512000700 1 -- 172.21.15.63:0/1294116263 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c010640 con 0x7ff51c102530 2024-03-23T11:37:11.715 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.872+0000 7ff512000700 1 -- 172.21.15.63:0/1294116263 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff50c010c50 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.872+0000 7ff512000700 1 -- 172.21.15.63:0/1294116263 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c01ad30 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.872+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c113db0 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.872+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c114920 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.873+0000 7ff512000700 1 -- 172.21.15.63:0/1294116263 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ff50c010dc0 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.873+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c05f5f0 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.873+0000 7ff512000700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7ff5000368d0 0x7ff500038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.873+0000 7ff512000700 1 -- 172.21.15.63:0/1294116263 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff50c04a460 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.873+0000 7ff519400700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7ff5000368d0 0x7ff500038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.874+0000 7ff519400700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7ff5000368d0 0x7ff500038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff508009990 tx=0x7ff508006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:10.878+0000 7ff512000700 1 -- 172.21.15.63:0/1294116263 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff50c018920 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.190+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7ff51c04bed0 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.193+0000 7ff512000700 1 -- 172.21.15.63:0/1294116263 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7ff50c020030 con 0x7ff51c102530 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7ff5000368d0 msgr2=0x7ff500038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:11.716 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7ff5000368d0 0x7ff500038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff508009990 tx=0x7ff508006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 msgr2=0x7ff51c113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c113670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0094c0 tx=0x7ff50c009250 comp rx=0 tx=0).stop 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 shutdown_connections 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7ff5000368d0 0x7ff500038d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 --2- 172.21.15.63:0/1294116263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff51c102530 0x7ff51c113670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 >> 172.21.15.63:0/1294116263 conn(0x7ff51c0fdcd0 msgr2=0x7ff51c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.195+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 shutdown_connections 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-03-23T11:37:11.196+0000 7ff520a00700 1 -- 172.21.15.63:0/1294116263 wait complete. 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout:Or, if you are only running a single cluster on this host: 2024-03-23T11:37:11.717 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: ceph telemetry on 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout:For more information see: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:11.718 INFO:teuthology.orchestra.run.smithi063.stdout:Bootstrap complete. 2024-03-23T11:37:11.768 INFO:tasks.cephadm:Fetching config... 2024-03-23T11:37:11.768 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:37:11.769 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-23T11:37:11.818 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-23T11:37:11.818 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:37:11.818 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-23T11:37:11.903 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-23T11:37:11.904 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:37:11.904 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/keyring of=/dev/stdout 2024-03-23T11:37:12.012 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-23T11:37:12.012 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:37:12.012 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-23T11:37:12.102 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-23T11:37:12.102 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg8yA4o4dJe6NMe5k5uLNKKi7oHhaDUJUnfH9PcCLOiIoRZDmdEXQ6jAeNfazlFdFbhFLRG4bzm06ovDDs2HAPb31gIgLDAsJshc34p1Ao4SmjBs87HlmuzJkATXFRc3WR1KnKUWsbyE+bZOIw5qWIxPP4EfYgQ6CAQ3W9wdEjmhdbLtrSTdAuUsk+kKs34Vh5B72bVhXggFiaxcqGsybXiAGOvrRS7LiASO5s04d/uJfFTKRTVqooMktZ2RQ1rHR6WWtOWNzSKTVOYgyzBhkjMAqARO4AqHf7Cot/lF0KRVjdoKtyQJKLxzSzR79s2piO6f/lA7lGznO3lUgJxKouuuA8tTWa3jfYc81H+6ZFUB9ki0W/4J+0HMcm1LXrpJnBxLjxj1e39gxsaIrNPOpMLZ4loYBrdNN8DwKQzZC6k+jNClWI46AkVpcFOBq5FvprX+1FsnIYU/BkrD54RUeEVjRw9ymluLSrzJ4xTcW6Lx6Ugy+1rt7mG0g8bZbsJR0= ceph-6880ffb6-e909-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-23T11:37:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg8yA4o4dJe6NMe5k5uLNKKi7oHhaDUJUnfH9PcCLOiIoRZDmdEXQ6jAeNfazlFdFbhFLRG4bzm06ovDDs2HAPb31gIgLDAsJshc34p1Ao4SmjBs87HlmuzJkATXFRc3WR1KnKUWsbyE+bZOIw5qWIxPP4EfYgQ6CAQ3W9wdEjmhdbLtrSTdAuUsk+kKs34Vh5B72bVhXggFiaxcqGsybXiAGOvrRS7LiASO5s04d/uJfFTKRTVqooMktZ2RQ1rHR6WWtOWNzSKTVOYgyzBhkjMAqARO4AqHf7Cot/lF0KRVjdoKtyQJKLxzSzR79s2piO6f/lA7lGznO3lUgJxKouuuA8tTWa3jfYc81H+6ZFUB9ki0W/4J+0HMcm1LXrpJnBxLjxj1e39gxsaIrNPOpMLZ4loYBrdNN8DwKQzZC6k+jNClWI46AkVpcFOBq5FvprX+1FsnIYU/BkrD54RUeEVjRw9ymluLSrzJ4xTcW6Lx6Ugy+1rt7mG0g8bZbsJR0= ceph-6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:37:12.226 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg8yA4o4dJe6NMe5k5uLNKKi7oHhaDUJUnfH9PcCLOiIoRZDmdEXQ6jAeNfazlFdFbhFLRG4bzm06ovDDs2HAPb31gIgLDAsJshc34p1Ao4SmjBs87HlmuzJkATXFRc3WR1KnKUWsbyE+bZOIw5qWIxPP4EfYgQ6CAQ3W9wdEjmhdbLtrSTdAuUsk+kKs34Vh5B72bVhXggFiaxcqGsybXiAGOvrRS7LiASO5s04d/uJfFTKRTVqooMktZ2RQ1rHR6WWtOWNzSKTVOYgyzBhkjMAqARO4AqHf7Cot/lF0KRVjdoKtyQJKLxzSzR79s2piO6f/lA7lGznO3lUgJxKouuuA8tTWa3jfYc81H+6ZFUB9ki0W/4J+0HMcm1LXrpJnBxLjxj1e39gxsaIrNPOpMLZ4loYBrdNN8DwKQzZC6k+jNClWI46AkVpcFOBq5FvprX+1FsnIYU/BkrD54RUeEVjRw9ymluLSrzJ4xTcW6Lx6Ugy+1rt7mG0g8bZbsJR0= ceph-6880ffb6-e909-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-23T11:37:12.323 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg8yA4o4dJe6NMe5k5uLNKKi7oHhaDUJUnfH9PcCLOiIoRZDmdEXQ6jAeNfazlFdFbhFLRG4bzm06ovDDs2HAPb31gIgLDAsJshc34p1Ao4SmjBs87HlmuzJkATXFRc3WR1KnKUWsbyE+bZOIw5qWIxPP4EfYgQ6CAQ3W9wdEjmhdbLtrSTdAuUsk+kKs34Vh5B72bVhXggFiaxcqGsybXiAGOvrRS7LiASO5s04d/uJfFTKRTVqooMktZ2RQ1rHR6WWtOWNzSKTVOYgyzBhkjMAqARO4AqHf7Cot/lF0KRVjdoKtyQJKLxzSzR79s2piO6f/lA7lGznO3lUgJxKouuuA8tTWa3jfYc81H+6ZFUB9ki0W/4J+0HMcm1LXrpJnBxLjxj1e39gxsaIrNPOpMLZ4loYBrdNN8DwKQzZC6k+jNClWI46AkVpcFOBq5FvprX+1FsnIYU/BkrD54RUeEVjRw9ymluLSrzJ4xTcW6Lx6Ugy+1rt7mG0g8bZbsJR0= ceph-6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:37:12.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-23T11:37:12.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:12 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1294116263' entity='client.admin' 2024-03-23T11:37:12.774 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.771+0000 7f5937e00700 1 -- 172.21.15.63:0/1280862446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 msgr2=0x7f59300fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.771+0000 7f5937e00700 1 --2- 172.21.15.63:0/1280862446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f59300fce80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59240048c0 tx=0x7f5924009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.771+0000 7f5937e00700 1 -- 172.21.15.63:0/1280862446 shutdown_connections 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.771+0000 7f5937e00700 1 --2- 172.21.15.63:0/1280862446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f59300fce80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.771+0000 7f5937e00700 1 -- 172.21.15.63:0/1280862446 >> 172.21.15.63:0/1280862446 conn(0x7f59300f81d0 msgr2=0x7f59300fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.771+0000 7f5937e00700 1 -- 172.21.15.63:0/1280862446 shutdown_connections 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.771+0000 7f5937e00700 1 -- 172.21.15.63:0/1280862446 wait complete. 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.772+0000 7f5937e00700 1 Processor -- start 2024-03-23T11:37:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.772+0000 7f5937e00700 1 -- start start 2024-03-23T11:37:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.772+0000 7f5937e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f593010f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.772+0000 7f5937e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593010f720 con 0x7f59300fca70 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f5935400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f593010f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f5935400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f593010f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38274/0 (socket says 172.21.15.63:38274) 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f5935400700 1 -- 172.21.15.63:0/4145786732 learned_addr learned my addr 172.21.15.63:0/4145786732 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f5935400700 1 -- 172.21.15.63:0/4145786732 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924005040 con 0x7f59300fca70 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f5935400700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f593010f1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5924000c00 tx=0x7f5924009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f592d600700 1 -- 172.21.15.63:0/4145786732 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5924009620 con 0x7f59300fca70 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f592d600700 1 -- 172.21.15.63:0/4145786732 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5924009780 con 0x7f59300fca70 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593010f920 con 0x7f59300fca70 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.773+0000 7f592d600700 1 -- 172.21.15.63:0/4145786732 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f592401c5b0 con 0x7f59300fca70 2024-03-23T11:37:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.774+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5930110490 con 0x7f59300fca70 2024-03-23T11:37:14.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.774+0000 7f592d600700 1 -- 172.21.15.63:0/4145786732 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f5924013420 con 0x7f59300fca70 2024-03-23T11:37:14.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.775+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5930102e40 con 0x7f59300fca70 2024-03-23T11:37:14.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.775+0000 7f592d600700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f5918036920 0x7f5918038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:14.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.775+0000 7f592d600700 1 -- 172.21.15.63:0/4145786732 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5924049ad0 con 0x7f59300fca70 2024-03-23T11:37:14.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.775+0000 7f5934a00700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f5918036920 0x7f5918038dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:14.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.776+0000 7f5934a00700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f5918036920 0x7f5918038dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f591c009990 tx=0x7f591c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:14.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.780+0000 7f592d600700 1 -- 172.21.15.63:0/4145786732 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5924011920 con 0x7f59300fca70 2024-03-23T11:37:14.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.988+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f593005f5f0 con 0x7f59300fca70 2024-03-23T11:37:14.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.991+0000 7f592d600700 1 -- 172.21.15.63:0/4145786732 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f59240116f0 con 0x7f59300fca70 2024-03-23T11:37:14.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f5918036920 msgr2=0x7f5918038dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:14.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f5918036920 0x7f5918038dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f591c009990 tx=0x7f591c006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:14.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 msgr2=0x7f593010f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:14.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f593010f1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5924000c00 tx=0x7f5924009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:14.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 shutdown_connections 2024-03-23T11:37:14.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f5918036920 0x7f5918038dd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:14.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 --2- 172.21.15.63:0/4145786732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59300fca70 0x7f593010f1e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:14.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 >> 172.21.15.63:0/4145786732 conn(0x7f59300f81d0 msgr2=0x7f59300fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:14.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 shutdown_connections 2024-03-23T11:37:14.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:14.994+0000 7f5937e00700 1 -- 172.21.15.63:0/4145786732 wait complete. 2024-03-23T11:37:15.649 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-23T11:37:15.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-23T11:37:16.232 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:37:16.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:15 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4145786732' entity='client.admin' 2024-03-23T11:37:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.590+0000 7f1c14200700 1 -- 172.21.15.63:0/3041480611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 msgr2=0x7f1c0c0703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.590+0000 7f1c14200700 1 --2- 172.21.15.63:0/3041480611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c0703b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c000048c0 tx=0x7f1c00009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.590+0000 7f1c14200700 1 -- 172.21.15.63:0/3041480611 shutdown_connections 2024-03-23T11:37:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.590+0000 7f1c14200700 1 --2- 172.21.15.63:0/3041480611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c0703b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.590+0000 7f1c14200700 1 -- 172.21.15.63:0/3041480611 >> 172.21.15.63:0/3041480611 conn(0x7f1c0c0f81c0 msgr2=0x7f1c0c0fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.591+0000 7f1c14200700 1 -- 172.21.15.63:0/3041480611 shutdown_connections 2024-03-23T11:37:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.591+0000 7f1c14200700 1 -- 172.21.15.63:0/3041480611 wait complete. 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.591+0000 7f1c14200700 1 Processor -- start 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.592+0000 7f1c14200700 1 -- start start 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.592+0000 7f1c14200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c10f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.592+0000 7f1c14200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c10f780 con 0x7f1c0c071ea0 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.592+0000 7f1c11800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c10f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.592+0000 7f1c11800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c10f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38280/0 (socket says 172.21.15.63:38280) 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.592+0000 7f1c11800700 1 -- 172.21.15.63:0/1006199819 learned_addr learned my addr 172.21.15.63:0/1006199819 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.593+0000 7f1c11800700 1 -- 172.21.15.63:0/1006199819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c00005040 con 0x7f1c0c071ea0 2024-03-23T11:37:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.593+0000 7f1c11800700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c10f240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000b860 tx=0x7f1c00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.593+0000 7f1c09600700 1 -- 172.21.15.63:0/1006199819 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c00009580 con 0x7f1c0c071ea0 2024-03-23T11:37:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.593+0000 7f1c09600700 1 -- 172.21.15.63:0/1006199819 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c000096e0 con 0x7f1c0c071ea0 2024-03-23T11:37:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.593+0000 7f1c09600700 1 -- 172.21.15.63:0/1006199819 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c0001c5b0 con 0x7f1c0c071ea0 2024-03-23T11:37:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.593+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0c10f980 con 0x7f1c0c071ea0 2024-03-23T11:37:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.593+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c1104f0 con 0x7f1c0c071ea0 2024-03-23T11:37:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.594+0000 7f1c09600700 1 -- 172.21.15.63:0/1006199819 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f1c00013420 con 0x7f1c0c071ea0 2024-03-23T11:37:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.594+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c105020 con 0x7f1c0c071ea0 2024-03-23T11:37:18.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.595+0000 7f1c09600700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f1bf4036880 0x7f1bf4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:18.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.595+0000 7f1c09600700 1 -- 172.21.15.63:0/1006199819 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1c00049a10 con 0x7f1c0c071ea0 2024-03-23T11:37:18.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.595+0000 7f1c10e00700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f1bf4036880 0x7f1bf4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:18.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.596+0000 7f1c10e00700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f1bf4036880 0x7f1bf4038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009990 tx=0x7f1bf8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:18.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.601+0000 7f1c09600700 1 -- 172.21.15.63:0/1006199819 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1c00013960 con 0x7f1c0c071ea0 2024-03-23T11:37:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.827+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c0c1107f0 con 0x7f1bf4036880 2024-03-23T11:37:18.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.830+0000 7f1c09600700 1 -- 172.21.15.63:0/1006199819 <== mgr.14158 v2:172.21.15.63:6800/213424997 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1c0c1107f0 con 0x7f1bf4036880 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f1bf4036880 msgr2=0x7f1bf4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f1bf4036880 0x7f1bf4038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009990 tx=0x7f1bf8006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 msgr2=0x7f1c0c10f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c10f240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000b860 tx=0x7f1c00009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 shutdown_connections 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f1bf4036880 0x7f1bf4038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 --2- 172.21.15.63:0/1006199819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c0c071ea0 0x7f1c0c10f240 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 >> 172.21.15.63:0/1006199819 conn(0x7f1c0c0f81c0 msgr2=0x7f1c0c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 shutdown_connections 2024-03-23T11:37:18.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:18.833+0000 7f1c14200700 1 -- 172.21.15.63:0/1006199819 wait complete. 2024-03-23T11:37:19.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:19.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:19.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:37:19.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:19.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:37:19.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-23T11:37:19.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:37:19.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: Deploying daemon crash.smithi063 on smithi063 2024-03-23T11:37:19.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:37:19.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:19 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:20.339 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi111 2024-03-23T11:37:20.339 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:37:20.339 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.conf 2024-03-23T11:37:20.388 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:37:20.388 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:37:20.476 INFO:tasks.cephadm:Adding host smithi111 to orchestrator... 2024-03-23T11:37:20.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch host add smithi111 2024-03-23T11:37:21.286 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:37:22.456 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:22 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:22.457 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:22 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:22.457 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:22 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:22.457 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:22 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:22.457 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:22 smithi063 ceph-mon[35189]: Deploying daemon node-exporter.smithi063 on smithi063 2024-03-23T11:37:22.457 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:22 smithi063 ceph-mon[35189]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:22.457 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:22 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:23.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.118+0000 7f94a7c00700 1 -- 172.21.15.63:0/2820371784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 msgr2=0x7f94a00703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:23.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.118+0000 7f94a7c00700 1 --2- 172.21.15.63:0/2820371784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a00703b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f94940048c0 tx=0x7f9494009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:23.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.118+0000 7f94a7c00700 1 -- 172.21.15.63:0/2820371784 shutdown_connections 2024-03-23T11:37:23.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.118+0000 7f94a7c00700 1 --2- 172.21.15.63:0/2820371784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a00703b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:23.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.118+0000 7f94a7c00700 1 -- 172.21.15.63:0/2820371784 >> 172.21.15.63:0/2820371784 conn(0x7f94a00f81d0 msgr2=0x7f94a00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:23.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.118+0000 7f94a7c00700 1 -- 172.21.15.63:0/2820371784 shutdown_connections 2024-03-23T11:37:23.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.118+0000 7f94a7c00700 1 -- 172.21.15.63:0/2820371784 wait complete. 2024-03-23T11:37:23.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.119+0000 7f94a7c00700 1 Processor -- start 2024-03-23T11:37:23.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.119+0000 7f94a7c00700 1 -- start start 2024-03-23T11:37:23.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.120+0000 7f94a7c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a0113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:23.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.120+0000 7f94a7c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0113b60 con 0x7f94a0071ea0 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.120+0000 7f94a5200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a0113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.120+0000 7f94a5200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a0113620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54310/0 (socket says 172.21.15.63:54310) 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.120+0000 7f94a5200700 1 -- 172.21.15.63:0/2630712201 learned_addr learned my addr 172.21.15.63:0/2630712201 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.120+0000 7f94a5200700 1 -- 172.21.15.63:0/2630712201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9494005040 con 0x7f94a0071ea0 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.121+0000 7f94a5200700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a0113620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f949400b860 tx=0x7f9494009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.121+0000 7f949cc00700 1 -- 172.21.15.63:0/2630712201 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9494009580 con 0x7f94a0071ea0 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.121+0000 7f949cc00700 1 -- 172.21.15.63:0/2630712201 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94940096e0 con 0x7f94a0071ea0 2024-03-23T11:37:23.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.121+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a0113d60 con 0x7f94a0071ea0 2024-03-23T11:37:23.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.121+0000 7f949cc00700 1 -- 172.21.15.63:0/2630712201 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f949401c5b0 con 0x7f94a0071ea0 2024-03-23T11:37:23.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.121+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a01148d0 con 0x7f94a0071ea0 2024-03-23T11:37:23.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.122+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a01094c0 con 0x7f94a0071ea0 2024-03-23T11:37:23.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.122+0000 7f949cc00700 1 -- 172.21.15.63:0/2630712201 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9494013420 con 0x7f94a0071ea0 2024-03-23T11:37:23.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.123+0000 7f949cc00700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94880368d0 0x7f9488038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:23.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.123+0000 7f949cc00700 1 -- 172.21.15.63:0/2630712201 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9494049a90 con 0x7f94a0071ea0 2024-03-23T11:37:23.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.123+0000 7f949fe00700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94880368d0 0x7f9488038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:23.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.124+0000 7f949fe00700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94880368d0 0x7f9488038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f948c009990 tx=0x7f948c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:23.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.128+0000 7f949cc00700 1 -- 172.21.15.63:0/2630712201 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9494013960 con 0x7f94a0071ea0 2024-03-23T11:37:23.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:23.342+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}) v1 -- 0x7f94a005f690 con 0x7f94880368d0 2024-03-23T11:37:25.010 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:24 smithi063 ceph-mon[35189]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:37:25.011 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:24 smithi063 ceph-mon[35189]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:25.921 INFO:teuthology.orchestra.run.smithi063.stdout:Added host 'smithi111' with addr '172.21.15.111' 2024-03-23T11:37:25.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.932+0000 7f949cc00700 1 -- 172.21.15.63:0/2630712201 <== mgr.14158 v2:172.21.15.63:6800/213424997 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f94a005f690 con 0x7f94880368d0 2024-03-23T11:37:25.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.934+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94880368d0 msgr2=0x7f9488038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.934+0000 7f94a7c00700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94880368d0 0x7f9488038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f948c009990 tx=0x7f948c006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 msgr2=0x7f94a0113620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a0113620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f949400b860 tx=0x7f9494009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 shutdown_connections 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94880368d0 0x7f9488038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 --2- 172.21.15.63:0/2630712201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94a0071ea0 0x7f94a0113620 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 >> 172.21.15.63:0/2630712201 conn(0x7f94a00f81d0 msgr2=0x7f94a00f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 shutdown_connections 2024-03-23T11:37:25.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:25.935+0000 7f94a7c00700 1 -- 172.21.15.63:0/2630712201 wait complete. 2024-03-23T11:37:26.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:25 smithi063 ceph-mon[35189]: Deploying cephadm binary to smithi111 2024-03-23T11:37:26.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-23T11:37:27.016 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:37:27.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:27.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:27.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: Added host smithi111 2024-03-23T11:37:27.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:27.266 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:27.266 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:27.266 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:27.266 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: Deploying daemon alertmanager.smithi063 on smithi063 2024-03-23T11:37:27.266 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:26 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:29.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.332+0000 7f94fb600700 1 -- 172.21.15.63:0/1590704045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 msgr2=0x7f94f40fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:29.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.332+0000 7f94fb600700 1 --2- 172.21.15.63:0/1590704045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f40fce80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f94e80048c0 tx=0x7f94e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:29.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.332+0000 7f94fb600700 1 -- 172.21.15.63:0/1590704045 shutdown_connections 2024-03-23T11:37:29.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.332+0000 7f94fb600700 1 --2- 172.21.15.63:0/1590704045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f40fce80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.332+0000 7f94fb600700 1 -- 172.21.15.63:0/1590704045 >> 172.21.15.63:0/1590704045 conn(0x7f94f40f81d0 msgr2=0x7f94f40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.332+0000 7f94fb600700 1 -- 172.21.15.63:0/1590704045 shutdown_connections 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.332+0000 7f94fb600700 1 -- 172.21.15.63:0/1590704045 wait complete. 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.334+0000 7f94fb600700 1 Processor -- start 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.334+0000 7f94fb600700 1 -- start start 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.334+0000 7f94fb600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.334+0000 7f94fb600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f410d510 con 0x7f94f40fca70 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.334+0000 7f94f8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94f8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54330/0 (socket says 172.21.15.63:54330) 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94f8c00700 1 -- 172.21.15.63:0/1896473767 learned_addr learned my addr 172.21.15.63:0/1896473767 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94f8c00700 1 -- 172.21.15.63:0/1896473767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8005040 con 0x7f94f40fca70 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94f8c00700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f410cfd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f94e8000c00 tx=0x7f94e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94f0c00700 1 -- 172.21.15.63:0/1896473767 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94e8009620 con 0x7f94f40fca70 2024-03-23T11:37:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94f0c00700 1 -- 172.21.15.63:0/1896473767 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94e8009780 con 0x7f94f40fca70 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94f0c00700 1 -- 172.21.15.63:0/1896473767 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94e801c5b0 con 0x7f94f40fca70 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f410d710 con 0x7f94f40fca70 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.335+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f410e280 con 0x7f94f40fca70 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.337+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f4102e40 con 0x7f94f40fca70 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.337+0000 7f94f0c00700 1 -- 172.21.15.63:0/1896473767 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f94e8013a60 con 0x7f94f40fca70 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.337+0000 7f94f0c00700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94dc0368d0 0x7f94dc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.337+0000 7f94f0c00700 1 -- 172.21.15.63:0/1896473767 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f94e8049ae0 con 0x7f94f40fca70 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.337+0000 7f94f3e00700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94dc0368d0 0x7f94dc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.338+0000 7f94f3e00700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94dc0368d0 0x7f94dc038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009990 tx=0x7f94e0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.342+0000 7f94f0c00700 1 -- 172.21.15.63:0/1896473767 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f94e8025400 con 0x7f94f40fca70 2024-03-23T11:37:29.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:28 smithi063 ceph-mon[35189]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:29.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.556+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f94f410e4c0 con 0x7f94dc0368d0 2024-03-23T11:37:29.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.558+0000 7f94f0c00700 1 -- 172.21.15.63:0/1896473767 <== mgr.14158 v2:172.21.15.63:6800/213424997 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f94f410e4c0 con 0x7f94dc0368d0 2024-03-23T11:37:29.545 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:37:29.546 INFO:teuthology.orchestra.run.smithi063.stdout:[{"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}] 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94dc0368d0 msgr2=0x7f94dc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94dc0368d0 0x7f94dc038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009990 tx=0x7f94e0006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 msgr2=0x7f94f410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f410cfd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f94e8000c00 tx=0x7f94e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 shutdown_connections 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f94dc0368d0 0x7f94dc038d80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 --2- 172.21.15.63:0/1896473767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94f40fca70 0x7f94f410cfd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 >> 172.21.15.63:0/1896473767 conn(0x7f94f40f81d0 msgr2=0x7f94f40f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 shutdown_connections 2024-03-23T11:37:29.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:29.560+0000 7f94fb600700 1 -- 172.21.15.63:0/1896473767 wait complete. 2024-03-23T11:37:31.130 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-23T11:37:31.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-23T11:37:31.196 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:30 smithi063 ceph-mon[35189]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T11:37:31.196 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:30 smithi063 ceph-mon[35189]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:32.571 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:37:33.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:32 smithi063 ceph-mon[35189]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:35.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:34 smithi063 ceph-mon[35189]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:36.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.298+0000 7f69dae00700 1 -- 172.21.15.63:0/1466703082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 msgr2=0x7f69d4070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:36.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.298+0000 7f69dae00700 1 --2- 172.21.15.63:0/1466703082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d4070350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f69c40048c0 tx=0x7f69c4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:36.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.298+0000 7f69dae00700 1 -- 172.21.15.63:0/1466703082 shutdown_connections 2024-03-23T11:37:36.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.298+0000 7f69dae00700 1 --2- 172.21.15.63:0/1466703082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d4070350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:36.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.298+0000 7f69dae00700 1 -- 172.21.15.63:0/1466703082 >> 172.21.15.63:0/1466703082 conn(0x7f69d40f81c0 msgr2=0x7f69d40fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:36.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.298+0000 7f69dae00700 1 -- 172.21.15.63:0/1466703082 shutdown_connections 2024-03-23T11:37:36.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.298+0000 7f69dae00700 1 -- 172.21.15.63:0/1466703082 wait complete. 2024-03-23T11:37:36.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.299+0000 7f69dae00700 1 Processor -- start 2024-03-23T11:37:36.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.299+0000 7f69dae00700 1 -- start start 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.300+0000 7f69dae00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d410f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.300+0000 7f69dae00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d410f760 con 0x7f69d4071e40 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.300+0000 7f69d3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d410f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.300+0000 7f69d3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d410f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45816/0 (socket says 172.21.15.63:45816) 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.300+0000 7f69d3e00700 1 -- 172.21.15.63:0/2803381660 learned_addr learned my addr 172.21.15.63:0/2803381660 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.300+0000 7f69d3e00700 1 -- 172.21.15.63:0/2803381660 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c4005040 con 0x7f69d4071e40 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.301+0000 7f69d3e00700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d410f220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f69c400b860 tx=0x7f69c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.301+0000 7f69cbe00700 1 -- 172.21.15.63:0/2803381660 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69c4009580 con 0x7f69d4071e40 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.301+0000 7f69cbe00700 1 -- 172.21.15.63:0/2803381660 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69c40096e0 con 0x7f69d4071e40 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.301+0000 7f69cbe00700 1 -- 172.21.15.63:0/2803381660 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69c401c5b0 con 0x7f69d4071e40 2024-03-23T11:37:36.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.301+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d410f960 con 0x7f69d4071e40 2024-03-23T11:37:36.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.301+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d41104d0 con 0x7f69d4071e40 2024-03-23T11:37:36.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.302+0000 7f69cbe00700 1 -- 172.21.15.63:0/2803381660 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f69c401c710 con 0x7f69d4071e40 2024-03-23T11:37:36.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.302+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d4105040 con 0x7f69d4071e40 2024-03-23T11:37:36.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.303+0000 7f69cbe00700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f69b80368d0 0x7f69b8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:36.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.303+0000 7f69cbe00700 1 -- 172.21.15.63:0/2803381660 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f69c4049ba0 con 0x7f69d4071e40 2024-03-23T11:37:36.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.303+0000 7f69d3400700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f69b80368d0 0x7f69b8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:36.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.303+0000 7f69d3400700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f69b80368d0 0x7f69b8038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009990 tx=0x7f69c0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:36.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.308+0000 7f69cbe00700 1 -- 172.21.15.63:0/2803381660 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f69c4025400 con 0x7f69d4071e40 2024-03-23T11:37:36.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.534+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f69d405f5f0 con 0x7f69d4071e40 2024-03-23T11:37:36.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.940+0000 7f69cbe00700 1 -- 172.21.15.63:0/2803381660 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f69c4011370 con 0x7f69d4071e40 2024-03-23T11:37:36.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.942+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f69b80368d0 msgr2=0x7f69b8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.942+0000 7f69dae00700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f69b80368d0 0x7f69b8038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009990 tx=0x7f69c0006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.942+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 msgr2=0x7f69d410f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.942+0000 7f69dae00700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d410f220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f69c400b860 tx=0x7f69c4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.943+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 shutdown_connections 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.943+0000 7f69dae00700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f69b80368d0 0x7f69b8038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.943+0000 7f69dae00700 1 --2- 172.21.15.63:0/2803381660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69d4071e40 0x7f69d410f220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.943+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 >> 172.21.15.63:0/2803381660 conn(0x7f69d40f81c0 msgr2=0x7f69d40f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.943+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 shutdown_connections 2024-03-23T11:37:36.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:37:36.943+0000 7f69dae00700 1 -- 172.21.15.63:0/2803381660 wait complete. 2024-03-23T11:37:36.932 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-03-23T11:37:37.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:36 smithi063 ceph-mon[35189]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:37.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:36 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2803381660' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-23T11:37:37.709 INFO:tasks.cephadm:Adding mon.smithi063 on smithi063 2024-03-23T11:37:37.709 INFO:tasks.cephadm:Adding mon.smithi111 on smithi111 2024-03-23T11:37:37.709 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch apply mon '2;smithi063:172.21.15.63=smithi063;smithi111:172.21.15.111=smithi111' 2024-03-23T11:37:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:38.200 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2803381660' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-23T11:37:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: osdmap e4: 0 total, 0 up, 0 in 2024-03-23T11:37:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:38.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:38.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-23T11:37:38.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:37 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:39.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:38 smithi063 ceph-mon[35189]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-23T11:37:39.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:38 smithi063 ceph-mon[35189]: Deploying daemon grafana.smithi063 on smithi063 2024-03-23T11:37:39.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:38 smithi063 ceph-mon[35189]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:40.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.017+0000 7f0cde600700 1 -- 172.21.15.111:0/249036762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 msgr2=0x7f0cd80fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:40.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.017+0000 7f0cde600700 1 --2- 172.21.15.111:0/249036762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd80fce80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80048c0 tx=0x7f0cc8009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:40.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.017+0000 7f0cde600700 1 -- 172.21.15.111:0/249036762 shutdown_connections 2024-03-23T11:37:40.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.017+0000 7f0cde600700 1 --2- 172.21.15.111:0/249036762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd80fce80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:40.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.017+0000 7f0cde600700 1 -- 172.21.15.111:0/249036762 >> 172.21.15.111:0/249036762 conn(0x7f0cd80f81d0 msgr2=0x7f0cd80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:40.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.017+0000 7f0cde600700 1 -- 172.21.15.111:0/249036762 shutdown_connections 2024-03-23T11:37:40.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.017+0000 7f0cde600700 1 -- 172.21.15.111:0/249036762 wait complete. 2024-03-23T11:37:40.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.018+0000 7f0cde600700 1 Processor -- start 2024-03-23T11:37:40.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.018+0000 7f0cde600700 1 -- start start 2024-03-23T11:37:40.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.018+0000 7f0cde600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd8071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:40.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.018+0000 7f0cde600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80722a0 con 0x7f0cd80fca70 2024-03-23T11:37:40.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.019+0000 7f0cd7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd8071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:40.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.019+0000 7f0cd7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd8071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:38180/0 (socket says 172.21.15.111:38180) 2024-03-23T11:37:40.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.019+0000 7f0cd7400700 1 -- 172.21.15.111:0/72611769 learned_addr learned my addr 172.21.15.111:0/72611769 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:37:40.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.019+0000 7f0cd7400700 1 -- 172.21.15.111:0/72611769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8005040 con 0x7f0cd80fca70 2024-03-23T11:37:40.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.019+0000 7f0cd7400700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd8071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8000c00 tx=0x7f0cc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:40.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.020+0000 7f0ccf400700 1 -- 172.21.15.111:0/72611769 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc8009620 con 0x7f0cd80fca70 2024-03-23T11:37:40.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.020+0000 7f0ccf400700 1 -- 172.21.15.111:0/72611769 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cc8009780 con 0x7f0cd80fca70 2024-03-23T11:37:40.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.020+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd8070370 con 0x7f0cd80fca70 2024-03-23T11:37:40.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.020+0000 7f0ccf400700 1 -- 172.21.15.111:0/72611769 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc801c5b0 con 0x7f0cd80fca70 2024-03-23T11:37:40.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.020+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd8070860 con 0x7f0cd80fca70 2024-03-23T11:37:40.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.021+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd805f5f0 con 0x7f0cd80fca70 2024-03-23T11:37:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.021+0000 7f0ccf400700 1 -- 172.21.15.111:0/72611769 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f0cc8013420 con 0x7f0cd80fca70 2024-03-23T11:37:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.021+0000 7f0ccf400700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f0cbc0368d0 0x7f0cbc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.021+0000 7f0ccf400700 1 -- 172.21.15.111:0/72611769 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0cc8049ad0 con 0x7f0cd80fca70 2024-03-23T11:37:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.022+0000 7f0cd6a00700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f0cbc0368d0 0x7f0cbc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.022+0000 7f0cd6a00700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f0cbc0368d0 0x7f0cbc038d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009990 tx=0x7f0cc4006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:40.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.026+0000 7f0ccf400700 1 -- 172.21.15.111:0/72611769 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0cc8025400 con 0x7f0cd80fca70 2024-03-23T11:37:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.236+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 --> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi063:172.21.15.63=smithi063;smithi111:172.21.15.111=smithi111", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cd8106cb0 con 0x7f0cbc0368d0 2024-03-23T11:37:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.243+0000 7f0ccf400700 1 -- 172.21.15.111:0/72611769 <== mgr.14158 v2:172.21.15.63:6800/213424997 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0cd8106cb0 con 0x7f0cbc0368d0 2024-03-23T11:37:40.231 INFO:teuthology.orchestra.run.smithi111.stdout:Scheduled mon update... 2024-03-23T11:37:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f0cbc0368d0 msgr2=0x7f0cbc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f0cbc0368d0 0x7f0cbc038d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009990 tx=0x7f0cc4006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 msgr2=0x7f0cd8071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd8071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8000c00 tx=0x7f0cc8009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 shutdown_connections 2024-03-23T11:37:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f0cbc0368d0 0x7f0cbc038d80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 --2- 172.21.15.111:0/72611769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd80fca70 0x7f0cd8071d60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 >> 172.21.15.111:0/72611769 conn(0x7f0cd80f81d0 msgr2=0x7f0cd80f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 shutdown_connections 2024-03-23T11:37:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:40.245+0000 7f0cde600700 1 -- 172.21.15.111:0/72611769 wait complete. 2024-03-23T11:37:40.874 DEBUG:teuthology.orchestra.run.smithi111:mon.smithi111> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi111.service 2024-03-23T11:37:40.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:37:40.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:37:40.948 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:-- Logs begin at Sat 2024-03-23 11:22:58 UTC. -- 2024-03-23T11:37:41.314 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:41.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:40 smithi063 ceph-mon[35189]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:41.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:40 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:41 smithi063 ceph-mon[35189]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi063:172.21.15.63=smithi063;smithi111:172.21.15.111=smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:37:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:41 smithi063 ceph-mon[35189]: Saving service mon spec with placement smithi063:172.21.15.63=smithi063;smithi111:172.21.15.111=smithi111;count:2 2024-03-23T11:37:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:41 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:43.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.067+0000 7f18e8400700 1 -- 172.21.15.111:0/4095201439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 msgr2=0x7f18e00fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:43.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.067+0000 7f18e8400700 1 --2- 172.21.15.111:0/4095201439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e00fce80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18d40048c0 tx=0x7f18d4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:43.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.067+0000 7f18e8400700 1 -- 172.21.15.111:0/4095201439 shutdown_connections 2024-03-23T11:37:43.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.067+0000 7f18e8400700 1 --2- 172.21.15.111:0/4095201439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e00fce80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:43.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.067+0000 7f18e8400700 1 -- 172.21.15.111:0/4095201439 >> 172.21.15.111:0/4095201439 conn(0x7f18e00f81d0 msgr2=0x7f18e00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:43.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.067+0000 7f18e8400700 1 -- 172.21.15.111:0/4095201439 shutdown_connections 2024-03-23T11:37:43.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.067+0000 7f18e8400700 1 -- 172.21.15.111:0/4095201439 wait complete. 2024-03-23T11:37:43.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.068+0000 7f18e8400700 1 Processor -- start 2024-03-23T11:37:43.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.068+0000 7f18e8400700 1 -- start start 2024-03-23T11:37:43.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.068+0000 7f18e8400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e010f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:43.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.068+0000 7f18e8400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e010f720 con 0x7f18e00fca70 2024-03-23T11:37:43.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.068+0000 7f18e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e010f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:43.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.068+0000 7f18e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e010f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:38194/0 (socket says 172.21.15.111:38194) 2024-03-23T11:37:43.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.068+0000 7f18e5a00700 1 -- 172.21.15.111:0/3178107132 learned_addr learned my addr 172.21.15.111:0/3178107132 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:37:43.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.069+0000 7f18e5a00700 1 -- 172.21.15.111:0/3178107132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4005040 con 0x7f18e00fca70 2024-03-23T11:37:43.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.069+0000 7f18e5a00700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e010f1e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18d4000c00 tx=0x7f18d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:43.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.069+0000 7f18dd600700 1 -- 172.21.15.111:0/3178107132 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18d4009620 con 0x7f18e00fca70 2024-03-23T11:37:43.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.069+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e010f920 con 0x7f18e00fca70 2024-03-23T11:37:43.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.069+0000 7f18dd600700 1 -- 172.21.15.111:0/3178107132 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18d4009780 con 0x7f18e00fca70 2024-03-23T11:37:43.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.069+0000 7f18dd600700 1 -- 172.21.15.111:0/3178107132 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18d401c5b0 con 0x7f18e00fca70 2024-03-23T11:37:43.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.069+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e0110490 con 0x7f18e00fca70 2024-03-23T11:37:43.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.071+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e0102e40 con 0x7f18e00fca70 2024-03-23T11:37:43.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.071+0000 7f18dd600700 1 -- 172.21.15.111:0/3178107132 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f18d4013a60 con 0x7f18e00fca70 2024-03-23T11:37:43.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.071+0000 7f18dd600700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f18c8036880 0x7f18c8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:43.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.071+0000 7f18dd600700 1 -- 172.21.15.111:0/3178107132 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f18d4049af0 con 0x7f18e00fca70 2024-03-23T11:37:43.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.071+0000 7f18e5000700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f18c8036880 0x7f18c8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.072+0000 7f18e5000700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f18c8036880 0x7f18c8038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009990 tx=0x7f18cc006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:43.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.079+0000 7f18dd600700 1 -- 172.21.15.111:0/3178107132 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f18d4025400 con 0x7f18e00fca70 2024-03-23T11:37:43.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.350+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18e01106d0 con 0x7f18e00fca70 2024-03-23T11:37:43.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.350+0000 7f18dd600700 1 -- 172.21.15.111:0/3178107132 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f18d4011b60 con 0x7f18e00fca70 2024-03-23T11:37:43.339 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:37:43.339 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f18c8036880 msgr2=0x7f18c8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f18c8036880 0x7f18c8038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009990 tx=0x7f18cc006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 msgr2=0x7f18e010f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e010f1e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18d4000c00 tx=0x7f18d4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 shutdown_connections 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f18c8036880 0x7f18c8038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 --2- 172.21.15.111:0/3178107132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18e00fca70 0x7f18e010f1e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 >> 172.21.15.111:0/3178107132 conn(0x7f18e00f81d0 msgr2=0x7f18e00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:43.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 shutdown_connections 2024-03-23T11:37:43.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:43.353+0000 7f18e8400700 1 -- 172.21.15.111:0/3178107132 wait complete. 2024-03-23T11:37:43.342 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:37:43.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:42 smithi063 ceph-mon[35189]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:44.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:43 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/3178107132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:37:44.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:37:44.916 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:37:45.322 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:45.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:44 smithi063 ceph-mon[35189]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:45.408 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:47.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.102+0000 7f7ce9e00700 1 -- 172.21.15.111:0/88393830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 msgr2=0x7f7ce40fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:47.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.102+0000 7f7ce9e00700 1 --2- 172.21.15.111:0/88393830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce40fce80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40048c0 tx=0x7f7cd4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:47.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.102+0000 7f7ce9e00700 1 -- 172.21.15.111:0/88393830 shutdown_connections 2024-03-23T11:37:47.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.102+0000 7f7ce9e00700 1 --2- 172.21.15.111:0/88393830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce40fce80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:47.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.102+0000 7f7ce9e00700 1 -- 172.21.15.111:0/88393830 >> 172.21.15.111:0/88393830 conn(0x7f7ce40f81d0 msgr2=0x7f7ce40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:47.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.103+0000 7f7ce9e00700 1 -- 172.21.15.111:0/88393830 shutdown_connections 2024-03-23T11:37:47.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.103+0000 7f7ce9e00700 1 -- 172.21.15.111:0/88393830 wait complete. 2024-03-23T11:37:47.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.103+0000 7f7ce9e00700 1 Processor -- start 2024-03-23T11:37:47.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.103+0000 7f7ce9e00700 1 -- start start 2024-03-23T11:37:47.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.104+0000 7f7ce9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce410f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:47.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.104+0000 7f7ce9e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce410f720 con 0x7f7ce40fca70 2024-03-23T11:37:47.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.104+0000 7f7ce2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce410f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:47.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.104+0000 7f7ce2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce410f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:38212/0 (socket says 172.21.15.111:38212) 2024-03-23T11:37:47.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.104+0000 7f7ce2a00700 1 -- 172.21.15.111:0/3429005457 learned_addr learned my addr 172.21.15.111:0/3429005457 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:37:47.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.104+0000 7f7ce2a00700 1 -- 172.21.15.111:0/3429005457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4005040 con 0x7f7ce40fca70 2024-03-23T11:37:47.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.105+0000 7f7ce2a00700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce410f1e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4000c00 tx=0x7f7cd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:47.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.105+0000 7f7cdaa00700 1 -- 172.21.15.111:0/3429005457 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd4009620 con 0x7f7ce40fca70 2024-03-23T11:37:47.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.105+0000 7f7cdaa00700 1 -- 172.21.15.111:0/3429005457 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cd4009780 con 0x7f7ce40fca70 2024-03-23T11:37:47.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.105+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce410f920 con 0x7f7ce40fca70 2024-03-23T11:37:47.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.105+0000 7f7cdaa00700 1 -- 172.21.15.111:0/3429005457 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd401c5b0 con 0x7f7ce40fca70 2024-03-23T11:37:47.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.105+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce4110490 con 0x7f7ce40fca70 2024-03-23T11:37:47.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.106+0000 7f7cdaa00700 1 -- 172.21.15.111:0/3429005457 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f7cd4013a60 con 0x7f7ce40fca70 2024-03-23T11:37:47.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.106+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce4102e40 con 0x7f7ce40fca70 2024-03-23T11:37:47.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.107+0000 7f7cdaa00700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f7cc8036880 0x7f7cc8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:47.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.107+0000 7f7cdaa00700 1 -- 172.21.15.111:0/3429005457 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7cd4049af0 con 0x7f7ce40fca70 2024-03-23T11:37:47.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.107+0000 7f7ce2000700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f7cc8036880 0x7f7cc8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:47.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.108+0000 7f7ce2000700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f7cc8036880 0x7f7cc8038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009990 tx=0x7f7cd0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:47.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.116+0000 7f7cdaa00700 1 -- 172.21.15.111:0/3429005457 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7cd4025400 con 0x7f7ce40fca70 2024-03-23T11:37:47.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.388+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ce41106d0 con 0x7f7ce40fca70 2024-03-23T11:37:47.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.388+0000 7f7cdaa00700 1 -- 172.21.15.111:0/3429005457 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7cd4011b60 con 0x7f7ce40fca70 2024-03-23T11:37:47.376 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:37:47.377 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f7cc8036880 msgr2=0x7f7cc8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f7cc8036880 0x7f7cc8038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009990 tx=0x7f7cd0006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 msgr2=0x7f7ce410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce410f1e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4000c00 tx=0x7f7cd4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 shutdown_connections 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f7cc8036880 0x7f7cc8038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 --2- 172.21.15.111:0/3429005457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ce40fca70 0x7f7ce410f1e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 >> 172.21.15.111:0/3429005457 conn(0x7f7ce40f81d0 msgr2=0x7f7ce40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.390+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 shutdown_connections 2024-03-23T11:37:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:47.391+0000 7f7ce9e00700 1 -- 172.21.15.111:0/3429005457 wait complete. 2024-03-23T11:37:47.380 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:37:47.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:46 smithi063 ceph-mon[35189]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:48.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:47 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/3429005457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:37:49.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:37:49.033 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:37:49.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:48 smithi063 ceph-mon[35189]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:49.440 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:51.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.218+0000 7f81c8e00700 1 -- 172.21.15.111:0/1250930065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 msgr2=0x7f81c40fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:51.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.218+0000 7f81c8e00700 1 --2- 172.21.15.111:0/1250930065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c40fce80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81b40048c0 tx=0x7f81b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.218+0000 7f81c8e00700 1 -- 172.21.15.111:0/1250930065 shutdown_connections 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.218+0000 7f81c8e00700 1 --2- 172.21.15.111:0/1250930065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c40fce80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.218+0000 7f81c8e00700 1 -- 172.21.15.111:0/1250930065 >> 172.21.15.111:0/1250930065 conn(0x7f81c40f81d0 msgr2=0x7f81c40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.218+0000 7f81c8e00700 1 -- 172.21.15.111:0/1250930065 shutdown_connections 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.218+0000 7f81c8e00700 1 -- 172.21.15.111:0/1250930065 wait complete. 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.219+0000 7f81c8e00700 1 Processor -- start 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.219+0000 7f81c8e00700 1 -- start start 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.220+0000 7f81c8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.220+0000 7f81c8e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c40722a0 con 0x7f81c40fca70 2024-03-23T11:37:51.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.220+0000 7f81c1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:51.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.220+0000 7f81c1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c4071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:38162/0 (socket says 172.21.15.111:38162) 2024-03-23T11:37:51.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.220+0000 7f81c1e00700 1 -- 172.21.15.111:0/3995051450 learned_addr learned my addr 172.21.15.111:0/3995051450 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:37:51.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.220+0000 7f81c1e00700 1 -- 172.21.15.111:0/3995051450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b4005040 con 0x7f81c40fca70 2024-03-23T11:37:51.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.221+0000 7f81c1e00700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c4071d60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f81b4000c00 tx=0x7f81b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:51.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.221+0000 7f81ba000700 1 -- 172.21.15.111:0/3995051450 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81b4009620 con 0x7f81c40fca70 2024-03-23T11:37:51.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.221+0000 7f81ba000700 1 -- 172.21.15.111:0/3995051450 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81b4009780 con 0x7f81c40fca70 2024-03-23T11:37:51.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.221+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c4070370 con 0x7f81c40fca70 2024-03-23T11:37:51.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.221+0000 7f81ba000700 1 -- 172.21.15.111:0/3995051450 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81b401c5b0 con 0x7f81c40fca70 2024-03-23T11:37:51.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.221+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c4070860 con 0x7f81c40fca70 2024-03-23T11:37:51.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.222+0000 7f81ba000700 1 -- 172.21.15.111:0/3995051450 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f81b4013420 con 0x7f81c40fca70 2024-03-23T11:37:51.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.222+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c405f5f0 con 0x7f81c40fca70 2024-03-23T11:37:51.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.222+0000 7f81ba000700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f81a8036880 0x7f81a8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:51.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.222+0000 7f81ba000700 1 -- 172.21.15.111:0/3995051450 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f81b4049af0 con 0x7f81c40fca70 2024-03-23T11:37:51.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.223+0000 7f81c1400700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f81a8036880 0x7f81a8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:51.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.223+0000 7f81c1400700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f81a8036880 0x7f81a8038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009990 tx=0x7f81b0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:51.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.228+0000 7f81ba000700 1 -- 172.21.15.111:0/3995051450 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f81b4025400 con 0x7f81c40fca70 2024-03-23T11:37:51.276 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:50 smithi063 ceph-mon[35189]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:51.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.503+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81c4106b90 con 0x7f81c40fca70 2024-03-23T11:37:51.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.503+0000 7f81ba000700 1 -- 172.21.15.111:0/3995051450 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f81b40116f0 con 0x7f81c40fca70 2024-03-23T11:37:51.492 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:37:51.492 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f81a8036880 msgr2=0x7f81a8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f81a8036880 0x7f81a8038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009990 tx=0x7f81b0006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 msgr2=0x7f81c4071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c4071d60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f81b4000c00 tx=0x7f81b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 shutdown_connections 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f81a8036880 0x7f81a8038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 --2- 172.21.15.111:0/3995051450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81c40fca70 0x7f81c4071d60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:51.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 >> 172.21.15.111:0/3995051450 conn(0x7f81c40f81d0 msgr2=0x7f81c40f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:51.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 shutdown_connections 2024-03-23T11:37:51.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:51.506+0000 7f81c8e00700 1 -- 172.21.15.111:0/3995051450 wait complete. 2024-03-23T11:37:51.497 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:37:52.103 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:51 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/3995051450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:37:53.112 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:37:53.113 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:37:53.320 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:52 smithi063 ceph-mon[35189]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:53.513 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:53.600 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: Deploying daemon prometheus.smithi063 on smithi063 2024-03-23T11:37:54.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:54 smithi063 ceph-mon[35189]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:55.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.382+0000 7fbe4ec00700 1 -- 172.21.15.111:0/60378061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 msgr2=0x7fbe480fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.382+0000 7fbe4ec00700 1 --2- 172.21.15.111:0/60378061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe480fce80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe380048c0 tx=0x7fbe38009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.382+0000 7fbe4ec00700 1 -- 172.21.15.111:0/60378061 shutdown_connections 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.382+0000 7fbe4ec00700 1 --2- 172.21.15.111:0/60378061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe480fce80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.382+0000 7fbe4ec00700 1 -- 172.21.15.111:0/60378061 >> 172.21.15.111:0/60378061 conn(0x7fbe480f81d0 msgr2=0x7fbe480fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.382+0000 7fbe4ec00700 1 -- 172.21.15.111:0/60378061 shutdown_connections 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.382+0000 7fbe4ec00700 1 -- 172.21.15.111:0/60378061 wait complete. 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.383+0000 7fbe4ec00700 1 Processor -- start 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.383+0000 7fbe4ec00700 1 -- start start 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.384+0000 7fbe4ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe4810f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:55.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.384+0000 7fbe4ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4810f720 con 0x7fbe480fca70 2024-03-23T11:37:55.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.384+0000 7fbe47e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe4810f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:55.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.384+0000 7fbe47e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe4810f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:38192/0 (socket says 172.21.15.111:38192) 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.384+0000 7fbe47e00700 1 -- 172.21.15.111:0/1410771906 learned_addr learned my addr 172.21.15.111:0/1410771906 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.384+0000 7fbe47e00700 1 -- 172.21.15.111:0/1410771906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38005040 con 0x7fbe480fca70 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.385+0000 7fbe47e00700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe4810f1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000c00 tx=0x7fbe38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.385+0000 7fbe3fe00700 1 -- 172.21.15.111:0/1410771906 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe38009620 con 0x7fbe480fca70 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.385+0000 7fbe3fe00700 1 -- 172.21.15.111:0/1410771906 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe38009780 con 0x7fbe480fca70 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.385+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4810f920 con 0x7fbe480fca70 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.385+0000 7fbe3fe00700 1 -- 172.21.15.111:0/1410771906 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe3801c5b0 con 0x7fbe480fca70 2024-03-23T11:37:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.385+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe48110490 con 0x7fbe480fca70 2024-03-23T11:37:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.386+0000 7fbe3fe00700 1 -- 172.21.15.111:0/1410771906 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fbe38013a60 con 0x7fbe480fca70 2024-03-23T11:37:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.386+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe48102e40 con 0x7fbe480fca70 2024-03-23T11:37:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.386+0000 7fbe3fe00700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fbe2c036880 0x7fbe2c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.386+0000 7fbe3fe00700 1 -- 172.21.15.111:0/1410771906 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbe38049af0 con 0x7fbe480fca70 2024-03-23T11:37:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.387+0000 7fbe47400700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fbe2c036880 0x7fbe2c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.387+0000 7fbe47400700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fbe2c036880 0x7fbe2c038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009990 tx=0x7fbe34006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:55.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.392+0000 7fbe3fe00700 1 -- 172.21.15.111:0/1410771906 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbe38025400 con 0x7fbe480fca70 2024-03-23T11:37:55.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.660+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe481106d0 con 0x7fbe480fca70 2024-03-23T11:37:55.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.661+0000 7fbe3fe00700 1 -- 172.21.15.111:0/1410771906 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbe38011b60 con 0x7fbe480fca70 2024-03-23T11:37:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:37:55.649 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:37:55.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fbe2c036880 msgr2=0x7fbe2c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:55.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fbe2c036880 0x7fbe2c038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009990 tx=0x7fbe34006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 msgr2=0x7fbe4810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe4810f1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000c00 tx=0x7fbe38009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 shutdown_connections 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fbe2c036880 0x7fbe2c038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 --2- 172.21.15.111:0/1410771906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe480fca70 0x7fbe4810f1e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 >> 172.21.15.111:0/1410771906 conn(0x7fbe480f81d0 msgr2=0x7fbe480fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 shutdown_connections 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:55.664+0000 7fbe4ec00700 1 -- 172.21.15.111:0/1410771906 wait complete. 2024-03-23T11:37:55.654 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:37:56.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:55 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/1410771906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:37:57.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:56 smithi063 ceph-mon[35189]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:57.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:37:57.291 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:37:57.696 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:57.785 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:37:58.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:57 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:37:59.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:37:58 smithi063 ceph-mon[35189]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:37:59.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.509+0000 7f300a600700 1 -- 172.21.15.111:0/681957736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 msgr2=0x7f30040fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:59.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.509+0000 7f300a600700 1 --2- 172.21.15.111:0/681957736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f30040fce80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40048c0 tx=0x7f2ff4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:59.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.509+0000 7f300a600700 1 -- 172.21.15.111:0/681957736 shutdown_connections 2024-03-23T11:37:59.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.509+0000 7f300a600700 1 --2- 172.21.15.111:0/681957736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f30040fce80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:59.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.509+0000 7f300a600700 1 -- 172.21.15.111:0/681957736 >> 172.21.15.111:0/681957736 conn(0x7f30040f81d0 msgr2=0x7f30040fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:59.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.509+0000 7f300a600700 1 -- 172.21.15.111:0/681957736 shutdown_connections 2024-03-23T11:37:59.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.509+0000 7f300a600700 1 -- 172.21.15.111:0/681957736 wait complete. 2024-03-23T11:37:59.498 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.510+0000 7f300a600700 1 Processor -- start 2024-03-23T11:37:59.498 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.511+0000 7f300a600700 1 -- start start 2024-03-23T11:37:59.498 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.511+0000 7f300a600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f3004071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:59.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.511+0000 7f300a600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30040722a0 con 0x7f30040fca70 2024-03-23T11:37:59.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.511+0000 7f3003400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f3004071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:59.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f3003400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f3004071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:35760/0 (socket says 172.21.15.111:35760) 2024-03-23T11:37:59.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f3003400700 1 -- 172.21.15.111:0/1249483513 learned_addr learned my addr 172.21.15.111:0/1249483513 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:37:59.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f3003400700 1 -- 172.21.15.111:0/1249483513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4005040 con 0x7f30040fca70 2024-03-23T11:37:59.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f3003400700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f3004071d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4000c00 tx=0x7f2ff4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:59.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f2ffb400700 1 -- 172.21.15.111:0/1249483513 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff4009620 con 0x7f30040fca70 2024-03-23T11:37:59.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3004070370 con 0x7f30040fca70 2024-03-23T11:37:59.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f2ffb400700 1 -- 172.21.15.111:0/1249483513 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ff4009780 con 0x7f30040fca70 2024-03-23T11:37:59.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.512+0000 7f2ffb400700 1 -- 172.21.15.111:0/1249483513 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff401c5b0 con 0x7f30040fca70 2024-03-23T11:37:59.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.513+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3004070860 con 0x7f30040fca70 2024-03-23T11:37:59.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.514+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300405f5f0 con 0x7f30040fca70 2024-03-23T11:37:59.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.514+0000 7f2ffb400700 1 -- 172.21.15.111:0/1249483513 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f2ff4013a60 con 0x7f30040fca70 2024-03-23T11:37:59.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.514+0000 7f2ffb400700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f2fe80368d0 0x7f2fe8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:37:59.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.514+0000 7f2ffb400700 1 -- 172.21.15.111:0/1249483513 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2ff4049a20 con 0x7f30040fca70 2024-03-23T11:37:59.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.515+0000 7f3002a00700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f2fe80368d0 0x7f2fe8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:37:59.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.515+0000 7f3002a00700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f2fe80368d0 0x7f2fe8038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009990 tx=0x7f2ff0006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:37:59.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.520+0000 7f2ffb400700 1 -- 172.21.15.111:0/1249483513 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2ff4025400 con 0x7f30040fca70 2024-03-23T11:37:59.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.796+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3004106b90 con 0x7f30040fca70 2024-03-23T11:37:59.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.796+0000 7f2ffb400700 1 -- 172.21.15.111:0/1249483513 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2ff4011b60 con 0x7f30040fca70 2024-03-23T11:37:59.784 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:37:59.784 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:37:59.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f2fe80368d0 msgr2=0x7f2fe8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:59.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f2fe80368d0 0x7f2fe8038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009990 tx=0x7f2ff0006e40 comp rx=0 tx=0).stop 2024-03-23T11:37:59.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 msgr2=0x7f3004071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:37:59.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f3004071d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4000c00 tx=0x7f2ff4009e30 comp rx=0 tx=0).stop 2024-03-23T11:37:59.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 shutdown_connections 2024-03-23T11:37:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f2fe80368d0 0x7f2fe8038d80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 --2- 172.21.15.111:0/1249483513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30040fca70 0x7f3004071d60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:37:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 >> 172.21.15.111:0/1249483513 conn(0x7f30040f81d0 msgr2=0x7f30040f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:37:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 shutdown_connections 2024-03-23T11:37:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:37:59.799+0000 7f300a600700 1 -- 172.21.15.111:0/1249483513 wait complete. 2024-03-23T11:37:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:01.007 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:00 smithi063 ceph-mon[35189]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:01.007 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:00 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/1249483513' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:01.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:01.376 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:01.779 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:01.866 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:03.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:02 smithi063 ceph-mon[35189]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:03.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.520+0000 7f35ff200700 1 -- 172.21.15.111:0/3039950223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 msgr2=0x7f35f80fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:03.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.520+0000 7f35ff200700 1 --2- 172.21.15.111:0/3039950223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f80fce80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f35e80048c0 tx=0x7f35e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:03.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.521+0000 7f35ff200700 1 -- 172.21.15.111:0/3039950223 shutdown_connections 2024-03-23T11:38:03.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.521+0000 7f35ff200700 1 --2- 172.21.15.111:0/3039950223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f80fce80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:03.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.521+0000 7f35ff200700 1 -- 172.21.15.111:0/3039950223 >> 172.21.15.111:0/3039950223 conn(0x7f35f80f81d0 msgr2=0x7f35f80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:03.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.521+0000 7f35ff200700 1 -- 172.21.15.111:0/3039950223 shutdown_connections 2024-03-23T11:38:03.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.521+0000 7f35ff200700 1 -- 172.21.15.111:0/3039950223 wait complete. 2024-03-23T11:38:03.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.521+0000 7f35ff200700 1 Processor -- start 2024-03-23T11:38:03.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.522+0000 7f35ff200700 1 -- start start 2024-03-23T11:38:03.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.522+0000 7f35ff200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f810f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:03.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.522+0000 7f35ff200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f810f720 con 0x7f35f80fca70 2024-03-23T11:38:03.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.522+0000 7f35f7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f810f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:03.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.522+0000 7f35f7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f810f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:35776/0 (socket says 172.21.15.111:35776) 2024-03-23T11:38:03.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.522+0000 7f35f7e00700 1 -- 172.21.15.111:0/3392843893 learned_addr learned my addr 172.21.15.111:0/3392843893 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:03.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.523+0000 7f35f7e00700 1 -- 172.21.15.111:0/3392843893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8005040 con 0x7f35f80fca70 2024-03-23T11:38:03.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.523+0000 7f35f7e00700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f810f1e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009300 tx=0x7f35e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:03.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.523+0000 7f35efe00700 1 -- 172.21.15.111:0/3392843893 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e8009600 con 0x7f35f80fca70 2024-03-23T11:38:03.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.523+0000 7f35efe00700 1 -- 172.21.15.111:0/3392843893 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35e8009760 con 0x7f35f80fca70 2024-03-23T11:38:03.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.523+0000 7f35efe00700 1 -- 172.21.15.111:0/3392843893 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e801c5b0 con 0x7f35f80fca70 2024-03-23T11:38:03.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.523+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f810f920 con 0x7f35f80fca70 2024-03-23T11:38:03.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.523+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f8110490 con 0x7f35f80fca70 2024-03-23T11:38:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.524+0000 7f35efe00700 1 -- 172.21.15.111:0/3392843893 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f35e8013420 con 0x7f35f80fca70 2024-03-23T11:38:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.524+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f8102e40 con 0x7f35f80fca70 2024-03-23T11:38:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.525+0000 7f35efe00700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f35dc0368d0 0x7f35dc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.525+0000 7f35efe00700 1 -- 172.21.15.111:0/3392843893 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f35e8049ad0 con 0x7f35f80fca70 2024-03-23T11:38:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.525+0000 7f35f7400700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f35dc0368d0 0x7f35dc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.526+0000 7f35f7400700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f35dc0368d0 0x7f35dc038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009990 tx=0x7f35e4006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:03.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.530+0000 7f35efe00700 1 -- 172.21.15.111:0/3392843893 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f35e8013960 con 0x7f35f80fca70 2024-03-23T11:38:03.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.799+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35f81106d0 con 0x7f35f80fca70 2024-03-23T11:38:03.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.799+0000 7f35efe00700 1 -- 172.21.15.111:0/3392843893 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f35e8011370 con 0x7f35f80fca70 2024-03-23T11:38:03.787 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:03.787 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f35dc0368d0 msgr2=0x7f35dc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f35dc0368d0 0x7f35dc038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009990 tx=0x7f35e4006e40 comp rx=0 tx=0).stop 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 msgr2=0x7f35f810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f810f1e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009300 tx=0x7f35e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 shutdown_connections 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f35dc0368d0 0x7f35dc038d80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 --2- 172.21.15.111:0/3392843893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35f80fca70 0x7f35f810f1e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:03.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 >> 172.21.15.111:0/3392843893 conn(0x7f35f80f81d0 msgr2=0x7f35f80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:03.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 shutdown_connections 2024-03-23T11:38:03.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:03.802+0000 7f35ff200700 1 -- 172.21.15.111:0/3392843893 wait complete. 2024-03-23T11:38:03.790 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:05.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:04 smithi063 ceph-mon[35189]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:05.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:04 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/3392843893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:05.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:05.448 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:05.849 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:05.936 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:06.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:06 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:06.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:06 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:06.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:06 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:06.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:06 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-23T11:38:07.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:07 smithi063 ceph-mon[35189]: from='mgr.14158 172.21.15.63:0/1712463402' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-23T11:38:07.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:07 smithi063 ceph-mon[35189]: mgrmap e14: smithi063.gehkcr(active, since 64s) 2024-03-23T11:38:07.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.578+0000 7f9eb9000700 1 -- 172.21.15.111:0/1094670980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 msgr2=0x7f9eb40fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.578+0000 7f9eb9000700 1 --2- 172.21.15.111:0/1094670980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb40fce80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40048c0 tx=0x7f9ea4009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.578+0000 7f9eb9000700 1 -- 172.21.15.111:0/1094670980 shutdown_connections 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.578+0000 7f9eb9000700 1 --2- 172.21.15.111:0/1094670980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb40fce80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.578+0000 7f9eb9000700 1 -- 172.21.15.111:0/1094670980 >> 172.21.15.111:0/1094670980 conn(0x7f9eb40f81d0 msgr2=0x7f9eb40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.578+0000 7f9eb9000700 1 -- 172.21.15.111:0/1094670980 shutdown_connections 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.578+0000 7f9eb9000700 1 -- 172.21.15.111:0/1094670980 wait complete. 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.579+0000 7f9eb9000700 1 Processor -- start 2024-03-23T11:38:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.579+0000 7f9eb9000700 1 -- start start 2024-03-23T11:38:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.579+0000 7f9eb9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.579+0000 7f9eb9000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb410d510 con 0x7f9eb40fca70 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.579+0000 7f9eb1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.579+0000 7f9eb1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:35780/0 (socket says 172.21.15.111:35780) 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.579+0000 7f9eb1e00700 1 -- 172.21.15.111:0/2065832592 learned_addr learned my addr 172.21.15.111:0/2065832592 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.580+0000 7f9eb1e00700 1 -- 172.21.15.111:0/2065832592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea4005040 con 0x7f9eb40fca70 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.580+0000 7f9eb1e00700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb410cfd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009300 tx=0x7f9ea4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.580+0000 7f9eaa000700 1 -- 172.21.15.111:0/2065832592 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ea4009600 con 0x7f9eb40fca70 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.580+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb410d710 con 0x7f9eb40fca70 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.580+0000 7f9eaa000700 1 -- 172.21.15.111:0/2065832592 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ea4009760 con 0x7f9eb40fca70 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.580+0000 7f9eaa000700 1 -- 172.21.15.111:0/2065832592 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ea401c5b0 con 0x7f9eb40fca70 2024-03-23T11:38:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.581+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb410e280 con 0x7f9eb40fca70 2024-03-23T11:38:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.581+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb4102e40 con 0x7f9eb40fca70 2024-03-23T11:38:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.582+0000 7f9eaa000700 1 -- 172.21.15.111:0/2065832592 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f9ea4013960 con 0x7f9eb40fca70 2024-03-23T11:38:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.582+0000 7f9eaa000700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 0x7f9e98038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.582+0000 7f9eaa000700 1 -- 172.21.15.111:0/2065832592 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9ea4049b70 con 0x7f9eb40fca70 2024-03-23T11:38:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.582+0000 7f9eb1400700 1 -- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 msgr2=0x7f9e98038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/213424997 2024-03-23T11:38:07.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.582+0000 7f9eb1400700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 0x7f9e98038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:38:07.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.588+0000 7f9eaa000700 1 -- 172.21.15.111:0/2065832592 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9ea4011550 con 0x7f9eb40fca70 2024-03-23T11:38:07.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.783+0000 7f9eb1400700 1 -- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 msgr2=0x7f9e98038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/213424997 2024-03-23T11:38:07.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.783+0000 7f9eb1400700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 0x7f9e98038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:38:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.862+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9eb410ec90 con 0x7f9eb40fca70 2024-03-23T11:38:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.863+0000 7f9eaa000700 1 -- 172.21.15.111:0/2065832592 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9ea4011370 con 0x7f9eb40fca70 2024-03-23T11:38:07.850 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:07.850 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 msgr2=0x7f9e98038dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 0x7f9e98038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 msgr2=0x7f9eb410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb410cfd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009300 tx=0x7f9ea4009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 shutdown_connections 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7f9e98036920 0x7f9e98038dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 --2- 172.21.15.111:0/2065832592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb40fca70 0x7f9eb410cfd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 >> 172.21.15.111:0/2065832592 conn(0x7f9eb40f81d0 msgr2=0x7f9eb40f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 shutdown_connections 2024-03-23T11:38:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:07.865+0000 7f9eb9000700 1 -- 172.21.15.111:0/2065832592 wait complete. 2024-03-23T11:38:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:08.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:08 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/2065832592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:09.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:09.475 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:09.876 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:09.963 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:11.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.678+0000 7fdad3200700 1 -- 172.21.15.111:0/1217108145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 msgr2=0x7fdad40703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:11.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.678+0000 7fdad3200700 1 --2- 172.21.15.111:0/1217108145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad40703b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdac40048c0 tx=0x7fdac4009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:11.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.678+0000 7fdad3200700 1 -- 172.21.15.111:0/1217108145 shutdown_connections 2024-03-23T11:38:11.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.678+0000 7fdad3200700 1 --2- 172.21.15.111:0/1217108145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad40703b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:11.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.678+0000 7fdad3200700 1 -- 172.21.15.111:0/1217108145 >> 172.21.15.111:0/1217108145 conn(0x7fdad40f8020 msgr2=0x7fdad40fa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:11.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.678+0000 7fdad3200700 1 -- 172.21.15.111:0/1217108145 shutdown_connections 2024-03-23T11:38:11.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.678+0000 7fdad3200700 1 -- 172.21.15.111:0/1217108145 wait complete. 2024-03-23T11:38:11.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.680+0000 7fdad3200700 1 Processor -- start 2024-03-23T11:38:11.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.680+0000 7fdad3200700 1 -- start start 2024-03-23T11:38:11.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.680+0000 7fdad3200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad41133f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:11.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.680+0000 7fdad3200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad4113930 con 0x7fdad4071ea0 2024-03-23T11:38:11.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.681+0000 7fdad1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad41133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:11.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.681+0000 7fdad1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad41133f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:45900/0 (socket says 172.21.15.111:45900) 2024-03-23T11:38:11.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.681+0000 7fdad1e00700 1 -- 172.21.15.111:0/2729231842 learned_addr learned my addr 172.21.15.111:0/2729231842 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:11.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.681+0000 7fdad1e00700 1 -- 172.21.15.111:0/2729231842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4005040 con 0x7fdad4071ea0 2024-03-23T11:38:11.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.681+0000 7fdad1e00700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad41133f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdac400b860 tx=0x7fdac4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:11.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.681+0000 7fdaca000700 1 -- 172.21.15.111:0/2729231842 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac4009580 con 0x7fdad4071ea0 2024-03-23T11:38:11.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.682+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad4113b30 con 0x7fdad4071ea0 2024-03-23T11:38:11.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.682+0000 7fdaca000700 1 -- 172.21.15.111:0/2729231842 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdac40096e0 con 0x7fdad4071ea0 2024-03-23T11:38:11.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.682+0000 7fdaca000700 1 -- 172.21.15.111:0/2729231842 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac401c5b0 con 0x7fdad4071ea0 2024-03-23T11:38:11.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.682+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad41146a0 con 0x7fdad4071ea0 2024-03-23T11:38:11.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.683+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad41092d0 con 0x7fdad4071ea0 2024-03-23T11:38:11.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.683+0000 7fdaca000700 1 -- 172.21.15.111:0/2729231842 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7fdac4013420 con 0x7fdad4071ea0 2024-03-23T11:38:11.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.683+0000 7fdaca000700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 0x7fdab8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:11.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.683+0000 7fdaca000700 1 -- 172.21.15.111:0/2729231842 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdac4049bb0 con 0x7fdad4071ea0 2024-03-23T11:38:11.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.683+0000 7fdacbe00700 1 -- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 msgr2=0x7fdab8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/213424997 2024-03-23T11:38:11.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.683+0000 7fdacbe00700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 0x7fdab8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:38:11.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.689+0000 7fdaca000700 1 -- 172.21.15.111:0/2729231842 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdac4011550 con 0x7fdad4071ea0 2024-03-23T11:38:11.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.884+0000 7fdacbe00700 1 -- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 msgr2=0x7fdab8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/213424997 2024-03-23T11:38:11.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.884+0000 7fdacbe00700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 0x7fdab8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:38:11.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.956+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdad41150b0 con 0x7fdad4071ea0 2024-03-23T11:38:11.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.957+0000 7fdaca000700 1 -- 172.21.15.111:0/2729231842 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fdac4011370 con 0x7fdad4071ea0 2024-03-23T11:38:11.944 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:11.944 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.959+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 msgr2=0x7fdab8038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.959+0000 7fdad3200700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 0x7fdab8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.959+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 msgr2=0x7fdad41133f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.959+0000 7fdad3200700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad41133f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdac400b860 tx=0x7fdac4009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.960+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 shutdown_connections 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.960+0000 7fdad3200700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:6800/213424997,v1:172.21.15.63:6801/213424997] conn(0x7fdab80368d0 0x7fdab8038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.960+0000 7fdad3200700 1 --2- 172.21.15.111:0/2729231842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdad4071ea0 0x7fdad41133f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.960+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 >> 172.21.15.111:0/2729231842 conn(0x7fdad40f8020 msgr2=0x7fdad40f8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.960+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 shutdown_connections 2024-03-23T11:38:11.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:11.960+0000 7fdad3200700 1 -- 172.21.15.111:0/2729231842 wait complete. 2024-03-23T11:38:11.948 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:12.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:12 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/2729231842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:13.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:13.581 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:13.978 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:14.065 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:14.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: Active manager daemon smithi063.gehkcr restarted 2024-03-23T11:38:14.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: Activating manager daemon smithi063.gehkcr 2024-03-23T11:38:14.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: osdmap e5: 0 total, 0 up, 0 in 2024-03-23T11:38:14.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: mgrmap e15: smithi063.gehkcr(active, starting, since 0.00429406s) 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:38:14.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:38:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.882+0000 7f27ffa00700 1 -- 172.21.15.111:0/4011134814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 msgr2=0x7f27f80703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.882+0000 7f27ffa00700 1 --2- 172.21.15.111:0/4011134814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f80703b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0048c0 tx=0x7f27ec009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.883+0000 7f27ffa00700 1 -- 172.21.15.111:0/4011134814 shutdown_connections 2024-03-23T11:38:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.883+0000 7f27ffa00700 1 --2- 172.21.15.111:0/4011134814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f80703b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.883+0000 7f27ffa00700 1 -- 172.21.15.111:0/4011134814 >> 172.21.15.111:0/4011134814 conn(0x7f27f80f81d0 msgr2=0x7f27f80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:15.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.883+0000 7f27ffa00700 1 -- 172.21.15.111:0/4011134814 shutdown_connections 2024-03-23T11:38:15.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.883+0000 7f27ffa00700 1 -- 172.21.15.111:0/4011134814 wait complete. 2024-03-23T11:38:15.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.883+0000 7f27ffa00700 1 Processor -- start 2024-03-23T11:38:15.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.884+0000 7f27ffa00700 1 -- start start 2024-03-23T11:38:15.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.884+0000 7f27ffa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f810f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:15.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.884+0000 7f27ffa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f810f780 con 0x7f27f8071ea0 2024-03-23T11:38:15.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.884+0000 7f27fd000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f810f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:15.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.884+0000 7f27fd000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f810f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:45918/0 (socket says 172.21.15.111:45918) 2024-03-23T11:38:15.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.884+0000 7f27fd000700 1 -- 172.21.15.111:0/2648639148 learned_addr learned my addr 172.21.15.111:0/2648639148 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:15.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.885+0000 7f27fd000700 1 -- 172.21.15.111:0/2648639148 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec005040 con 0x7f27f8071ea0 2024-03-23T11:38:15.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.885+0000 7f27fd000700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f810f240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b860 tx=0x7f27ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:15.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.885+0000 7f27f4c00700 1 -- 172.21.15.111:0/2648639148 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27ec009580 con 0x7f27f8071ea0 2024-03-23T11:38:15.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.885+0000 7f27f4c00700 1 -- 172.21.15.111:0/2648639148 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27ec0096e0 con 0x7f27f8071ea0 2024-03-23T11:38:15.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.885+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f810f980 con 0x7f27f8071ea0 2024-03-23T11:38:15.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.885+0000 7f27f4c00700 1 -- 172.21.15.111:0/2648639148 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27ec01c5b0 con 0x7f27f8071ea0 2024-03-23T11:38:15.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.885+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f81104f0 con 0x7f27f8071ea0 2024-03-23T11:38:15.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.886+0000 7f27f4c00700 1 -- 172.21.15.111:0/2648639148 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f27ec013420 con 0x7f27f8071ea0 2024-03-23T11:38:15.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.886+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f8105050 con 0x7f27f8071ea0 2024-03-23T11:38:15.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.887+0000 7f27f4c00700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f27e0036710 0x7f27e0038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:15.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.887+0000 7f27f4c00700 1 -- 172.21.15.111:0/2648639148 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f27ec049920 con 0x7f27f8071ea0 2024-03-23T11:38:15.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.887+0000 7f27f7e00700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f27e0036710 0x7f27e0038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:15.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.888+0000 7f27f7e00700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f27e0036710 0x7f27e0038bc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009990 tx=0x7f27e4006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:15.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:15.892+0000 7f27f4c00700 1 -- 172.21.15.111:0/2648639148 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f27ec025400 con 0x7f27f8071ea0 2024-03-23T11:38:16.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:16.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:38:16.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:38:16.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:38:16.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:16.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:15 smithi063 ceph-mon[35189]: mgrmap e16: smithi063.gehkcr(active, since 1.00655s) 2024-03-23T11:38:16.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.199+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27f805f5f0 con 0x7f27f8071ea0 2024-03-23T11:38:16.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.200+0000 7f27f4c00700 1 -- 172.21.15.111:0/2648639148 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f27ec011370 con 0x7f27f8071ea0 2024-03-23T11:38:16.187 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:16.187 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f27e0036710 msgr2=0x7f27e0038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f27e0036710 0x7f27e0038bc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009990 tx=0x7f27e4006e40 comp rx=0 tx=0).stop 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 msgr2=0x7f27f810f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f810f240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b860 tx=0x7f27ec009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 shutdown_connections 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f27e0036710 0x7f27e0038bc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 --2- 172.21.15.111:0/2648639148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27f8071ea0 0x7f27f810f240 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 >> 172.21.15.111:0/2648639148 conn(0x7f27f80f81d0 msgr2=0x7f27f80f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 shutdown_connections 2024-03-23T11:38:16.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:16.203+0000 7f27ffa00700 1 -- 172.21.15.111:0/2648639148 wait complete. 2024-03-23T11:38:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:17.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:16 smithi063 ceph-mon[35189]: [23/Mar/2024:11:38:15] ENGINE Bus STARTING 2024-03-23T11:38:17.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:16 smithi063 ceph-mon[35189]: [23/Mar/2024:11:38:15] ENGINE Serving on https://172.21.15.63:7150 2024-03-23T11:38:17.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:16 smithi063 ceph-mon[35189]: [23/Mar/2024:11:38:15] ENGINE Bus STARTED 2024-03-23T11:38:17.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:17.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:16 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/2648639148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:17.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:17.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:17.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:18 smithi063 ceph-mon[35189]: mgrmap e17: smithi063.gehkcr(active, since 2s) 2024-03-23T11:38:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:18 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:19.213 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:21.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.364+0000 7f9e8d000700 1 -- 172.21.15.111:0/4094312726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 msgr2=0x7f9e880fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:21.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.364+0000 7f9e8d000700 1 --2- 172.21.15.111:0/4094312726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e880fce80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9e780048c0 tx=0x7f9e78009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:21.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.364+0000 7f9e8d000700 1 -- 172.21.15.111:0/4094312726 shutdown_connections 2024-03-23T11:38:21.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.364+0000 7f9e8d000700 1 --2- 172.21.15.111:0/4094312726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e880fce80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:21.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.364+0000 7f9e8d000700 1 -- 172.21.15.111:0/4094312726 >> 172.21.15.111:0/4094312726 conn(0x7f9e880f81d0 msgr2=0x7f9e880fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:21.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.364+0000 7f9e8d000700 1 -- 172.21.15.111:0/4094312726 shutdown_connections 2024-03-23T11:38:21.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.364+0000 7f9e8d000700 1 -- 172.21.15.111:0/4094312726 wait complete. 2024-03-23T11:38:21.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.365+0000 7f9e8d000700 1 Processor -- start 2024-03-23T11:38:21.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.365+0000 7f9e8d000700 1 -- start start 2024-03-23T11:38:21.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.365+0000 7f9e8d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e8810f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:21.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.365+0000 7f9e8d000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8810f720 con 0x7f9e880fca70 2024-03-23T11:38:21.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.366+0000 7f9e85e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e8810f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:21.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.366+0000 7f9e85e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e8810f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:56442/0 (socket says 172.21.15.111:56442) 2024-03-23T11:38:21.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.366+0000 7f9e85e00700 1 -- 172.21.15.111:0/493145636 learned_addr learned my addr 172.21.15.111:0/493145636 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:21.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.366+0000 7f9e85e00700 1 -- 172.21.15.111:0/493145636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e78005040 con 0x7f9e880fca70 2024-03-23T11:38:21.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.366+0000 7f9e85e00700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e8810f1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9e78000c00 tx=0x7f9e78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:21.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.367+0000 7f9e7e000700 1 -- 172.21.15.111:0/493145636 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e78009620 con 0x7f9e880fca70 2024-03-23T11:38:21.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.367+0000 7f9e7e000700 1 -- 172.21.15.111:0/493145636 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e78009780 con 0x7f9e880fca70 2024-03-23T11:38:21.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.367+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8810f920 con 0x7f9e880fca70 2024-03-23T11:38:21.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.367+0000 7f9e7e000700 1 -- 172.21.15.111:0/493145636 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e7801c5b0 con 0x7f9e880fca70 2024-03-23T11:38:21.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.367+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e88110490 con 0x7f9e880fca70 2024-03-23T11:38:21.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.368+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e88102e40 con 0x7f9e880fca70 2024-03-23T11:38:21.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.368+0000 7f9e7e000700 1 -- 172.21.15.111:0/493145636 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f9e78013a60 con 0x7f9e880fca70 2024-03-23T11:38:21.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.369+0000 7f9e7e000700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e6c036970 0x7f9e6c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:21.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.369+0000 7f9e7e000700 1 -- 172.21.15.111:0/493145636 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9e78049d70 con 0x7f9e880fca70 2024-03-23T11:38:21.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.369+0000 7f9e85400700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e6c036970 0x7f9e6c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:21.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.370+0000 7f9e85400700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e6c036970 0x7f9e6c038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009990 tx=0x7f9e74006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:21.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.374+0000 7f9e7e000700 1 -- 172.21.15.111:0/493145636 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9e78023300 con 0x7f9e880fca70 2024-03-23T11:38:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.649+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e881106d0 con 0x7f9e880fca70 2024-03-23T11:38:21.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.653+0000 7f9e7e000700 1 -- 172.21.15.111:0/493145636 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9e78011b60 con 0x7f9e880fca70 2024-03-23T11:38:21.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:21.640 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:21.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e6c036970 msgr2=0x7f9e6c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e6c036970 0x7f9e6c038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009990 tx=0x7f9e74006e40 comp rx=0 tx=0).stop 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 msgr2=0x7f9e8810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e8810f1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9e78000c00 tx=0x7f9e78009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 shutdown_connections 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e6c036970 0x7f9e6c038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 --2- 172.21.15.111:0/493145636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e880fca70 0x7f9e8810f1e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 >> 172.21.15.111:0/493145636 conn(0x7f9e880f81d0 msgr2=0x7f9e880fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.655+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 shutdown_connections 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:21.656+0000 7f9e8d000700 1 -- 172.21.15.111:0/493145636 wait complete. 2024-03-23T11:38:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:21.951 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:21 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/493145636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:23.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:23.524 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:23.569 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:23.569 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:23.569 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:23.569 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:23.569 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:38:23.931 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:24.018 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-23T11:38:25.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:25.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.826+0000 7f5fa7400700 1 -- 172.21.15.111:0/2452227610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 msgr2=0x7f5fa00fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.826+0000 7f5fa7400700 1 --2- 172.21.15.111:0/2452227610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa00fce80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f940048c0 tx=0x7f5f94009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.826+0000 7f5fa7400700 1 -- 172.21.15.111:0/2452227610 shutdown_connections 2024-03-23T11:38:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.826+0000 7f5fa7400700 1 --2- 172.21.15.111:0/2452227610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa00fce80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.826+0000 7f5fa7400700 1 -- 172.21.15.111:0/2452227610 >> 172.21.15.111:0/2452227610 conn(0x7f5fa00f81d0 msgr2=0x7f5fa00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.827+0000 7f5fa7400700 1 -- 172.21.15.111:0/2452227610 shutdown_connections 2024-03-23T11:38:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.827+0000 7f5fa7400700 1 -- 172.21.15.111:0/2452227610 wait complete. 2024-03-23T11:38:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.827+0000 7f5fa7400700 1 Processor -- start 2024-03-23T11:38:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.828+0000 7f5fa7400700 1 -- start start 2024-03-23T11:38:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.828+0000 7f5fa7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa010f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.828+0000 7f5fa7400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa010f720 con 0x7f5fa00fca70 2024-03-23T11:38:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.828+0000 7f5fa4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa010f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.828+0000 7f5fa4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa010f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:56452/0 (socket says 172.21.15.111:56452) 2024-03-23T11:38:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.828+0000 7f5fa4a00700 1 -- 172.21.15.111:0/81492153 learned_addr learned my addr 172.21.15.111:0/81492153 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.829+0000 7f5fa4a00700 1 -- 172.21.15.111:0/81492153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f94005040 con 0x7f5fa00fca70 2024-03-23T11:38:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.829+0000 7f5fa4a00700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa010f1e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.829+0000 7f5f9cc00700 1 -- 172.21.15.111:0/81492153 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f94009620 con 0x7f5fa00fca70 2024-03-23T11:38:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.829+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa010f920 con 0x7f5fa00fca70 2024-03-23T11:38:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.829+0000 7f5f9cc00700 1 -- 172.21.15.111:0/81492153 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f94009780 con 0x7f5fa00fca70 2024-03-23T11:38:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.829+0000 7f5f9cc00700 1 -- 172.21.15.111:0/81492153 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f9401c5b0 con 0x7f5fa00fca70 2024-03-23T11:38:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.829+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa0110490 con 0x7f5fa00fca70 2024-03-23T11:38:25.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.830+0000 7f5f9cc00700 1 -- 172.21.15.111:0/81492153 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f5f94013420 con 0x7f5fa00fca70 2024-03-23T11:38:25.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.830+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa0102e40 con 0x7f5fa00fca70 2024-03-23T11:38:25.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.831+0000 7f5f9cc00700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5f88036970 0x7f5f88038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:25.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.831+0000 7f5f9cc00700 1 -- 172.21.15.111:0/81492153 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f5f94049d80 con 0x7f5fa00fca70 2024-03-23T11:38:25.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.831+0000 7f5f9fe00700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5f88036970 0x7f5f88038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:25.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.832+0000 7f5f9fe00700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5f88036970 0x7f5f88038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c009990 tx=0x7f5f8c006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:25.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:25.837+0000 7f5f9cc00700 1 -- 172.21.15.111:0/81492153 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5f940119c0 con 0x7f5fa00fca70 2024-03-23T11:38:26.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.136+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fa0102e40 con 0x7f5fa00fca70 2024-03-23T11:38:26.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.136+0000 7f5f9cc00700 1 -- 172.21.15.111:0/81492153 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5f94011370 con 0x7f5fa00fca70 2024-03-23T11:38:26.125 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:26.125 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:26.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5f88036970 msgr2=0x7f5f88038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:26.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5f88036970 0x7f5f88038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c009990 tx=0x7f5f8c006e40 comp rx=0 tx=0).stop 2024-03-23T11:38:26.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 msgr2=0x7f5fa010f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:26.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa010f1e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000c00 tx=0x7f5f94009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:26.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 shutdown_connections 2024-03-23T11:38:26.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5f88036970 0x7f5f88038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:26.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 --2- 172.21.15.111:0/81492153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00fca70 0x7f5fa010f1e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:26.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 >> 172.21.15.111:0/81492153 conn(0x7f5fa00f81d0 msgr2=0x7f5fa00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:26.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 shutdown_connections 2024-03-23T11:38:26.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:26.139+0000 7f5fa7400700 1 -- 172.21.15.111:0/81492153 wait complete. 2024-03-23T11:38:26.128 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:26 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/81492153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:27.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:27 smithi063 ceph-mon[35189]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:38:27.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:27 smithi063 ceph-mon[35189]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:38:27.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:27 smithi063 ceph-mon[35189]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:38:27.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:27 smithi063 ceph-mon[35189]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:38:27.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:27.882 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:28.756 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:28.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:28 smithi063 ceph-mon[35189]: Deploying daemon crash.smithi111 on smithi111 2024-03-23T11:38:30.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:29 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.462+0000 7f9e6a200700 1 -- 172.21.15.111:0/1868866433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 msgr2=0x7f9e64070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.462+0000 7f9e6a200700 1 --2- 172.21.15.111:0/1868866433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64070730 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9e540048c0 tx=0x7f9e54009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.462+0000 7f9e6a200700 1 -- 172.21.15.111:0/1868866433 shutdown_connections 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.462+0000 7f9e6a200700 1 --2- 172.21.15.111:0/1868866433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64070730 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.462+0000 7f9e6a200700 1 -- 172.21.15.111:0/1868866433 >> 172.21.15.111:0/1868866433 conn(0x7f9e640f85e0 msgr2=0x7f9e640fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.463+0000 7f9e6a200700 1 -- 172.21.15.111:0/1868866433 shutdown_connections 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.463+0000 7f9e6a200700 1 -- 172.21.15.111:0/1868866433 wait complete. 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.463+0000 7f9e6a200700 1 Processor -- start 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.463+0000 7f9e6a200700 1 -- start start 2024-03-23T11:38:31.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.464+0000 7f9e6a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64115220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.464+0000 7f9e6a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64115760 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.464+0000 7f9e63400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64115220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.464+0000 7f9e63400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64115220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:59960/0 (socket says 172.21.15.111:59960) 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.464+0000 7f9e63400700 1 -- 172.21.15.111:0/1530759739 learned_addr learned my addr 172.21.15.111:0/1530759739 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.465+0000 7f9e63400700 1 -- 172.21.15.111:0/1530759739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54005040 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.465+0000 7f9e63400700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64115220 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e54007e30 tx=0x7f9e54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.465+0000 7f9e5b400700 1 -- 172.21.15.111:0/1530759739 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e540093c0 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.465+0000 7f9e5b400700 1 -- 172.21.15.111:0/1530759739 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e54013420 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.465+0000 7f9e5b400700 1 -- 172.21.15.111:0/1530759739 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e5401c5b0 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.465+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e64115960 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.465+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e64115dd0 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.466+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e64071aa0 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.467+0000 7f9e5b400700 1 -- 172.21.15.111:0/1530759739 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f9e54013970 con 0x7f9e64070360 2024-03-23T11:38:31.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.467+0000 7f9e5b400700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e480369c0 0x7f9e48038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:31.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.467+0000 7f9e5b400700 1 -- 172.21.15.111:0/1530759739 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9e54049ab0 con 0x7f9e64070360 2024-03-23T11:38:31.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.467+0000 7f9e62a00700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e480369c0 0x7f9e48038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:31.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.468+0000 7f9e62a00700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e480369c0 0x7f9e48038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009990 tx=0x7f9e50006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:31.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.474+0000 7f9e5b400700 1 -- 172.21.15.111:0/1530759739 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9e54011960 con 0x7f9e64070360 2024-03-23T11:38:31.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.767+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e6404bed0 con 0x7f9e64070360 2024-03-23T11:38:31.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.768+0000 7f9e5b400700 1 -- 172.21.15.111:0/1530759739 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9e54011370 con 0x7f9e64070360 2024-03-23T11:38:31.756 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:31.756 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:31.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e480369c0 msgr2=0x7f9e48038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:31.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e480369c0 0x7f9e48038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009990 tx=0x7f9e50006e40 comp rx=0 tx=0).stop 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 msgr2=0x7f9e64115220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64115220 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e54007e30 tx=0x7f9e54009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 shutdown_connections 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e480369c0 0x7f9e48038e70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 --2- 172.21.15.111:0/1530759739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e64070360 0x7f9e64115220 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.770+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 >> 172.21.15.111:0/1530759739 conn(0x7f9e640f85e0 msgr2=0x7f9e64105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.771+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 shutdown_connections 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:31.771+0000 7f9e6a200700 1 -- 172.21.15.111:0/1530759739 wait complete. 2024-03-23T11:38:31.759 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:32 smithi063 ceph-mon[35189]: Deploying daemon node-exporter.smithi111 on smithi111 2024-03-23T11:38:32.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:32 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/1530759739' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:33.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:33.333 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:33.738 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:36.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:35 smithi063 ceph-mon[35189]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:36.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:35 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:36.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.478+0000 7f7baf800700 1 -- 172.21.15.111:0/3029963665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 msgr2=0x7f7ba80706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:36.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.478+0000 7f7baf800700 1 --2- 172.21.15.111:0/3029963665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba80706b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0048c0 tx=0x7f7b9c009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:36.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.479+0000 7f7baf800700 1 -- 172.21.15.111:0/3029963665 shutdown_connections 2024-03-23T11:38:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.479+0000 7f7baf800700 1 --2- 172.21.15.111:0/3029963665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba80706b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.479+0000 7f7baf800700 1 -- 172.21.15.111:0/3029963665 >> 172.21.15.111:0/3029963665 conn(0x7f7ba80f85e0 msgr2=0x7f7ba80fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.479+0000 7f7baf800700 1 -- 172.21.15.111:0/3029963665 shutdown_connections 2024-03-23T11:38:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.479+0000 7f7baf800700 1 -- 172.21.15.111:0/3029963665 wait complete. 2024-03-23T11:38:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.479+0000 7f7baf800700 1 Processor -- start 2024-03-23T11:38:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.480+0000 7f7baf800700 1 -- start start 2024-03-23T11:38:36.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.480+0000 7f7baf800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba81152d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:36.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.480+0000 7f7baf800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8115810 con 0x7f7ba80702e0 2024-03-23T11:38:36.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.480+0000 7f7bace00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba81152d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.480+0000 7f7bace00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba81152d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:59980/0 (socket says 172.21.15.111:59980) 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.480+0000 7f7bace00700 1 -- 172.21.15.111:0/860844078 learned_addr learned my addr 172.21.15.111:0/860844078 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.481+0000 7f7bace00700 1 -- 172.21.15.111:0/860844078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c005040 con 0x7f7ba80702e0 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.481+0000 7f7bace00700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba81152d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c007e30 tx=0x7f7b9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.481+0000 7f7ba4c00700 1 -- 172.21.15.111:0/860844078 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b9c0093c0 con 0x7f7ba80702e0 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.481+0000 7f7ba4c00700 1 -- 172.21.15.111:0/860844078 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b9c013420 con 0x7f7ba80702e0 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.481+0000 7f7ba4c00700 1 -- 172.21.15.111:0/860844078 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b9c01c5b0 con 0x7f7ba80702e0 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.481+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba8115a10 con 0x7f7ba80702e0 2024-03-23T11:38:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.481+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba8115e80 con 0x7f7ba80702e0 2024-03-23T11:38:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.482+0000 7f7ba4c00700 1 -- 172.21.15.111:0/860844078 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f7b9c013970 con 0x7f7ba80702e0 2024-03-23T11:38:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.482+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba8071a20 con 0x7f7ba80702e0 2024-03-23T11:38:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.483+0000 7f7ba4c00700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7b90036970 0x7f7b90038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.483+0000 7f7ba4c00700 1 -- 172.21.15.111:0/860844078 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7b9c049b10 con 0x7f7ba80702e0 2024-03-23T11:38:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.483+0000 7f7ba7e00700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7b90036970 0x7f7b90038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:36.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.484+0000 7f7ba7e00700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7b90036970 0x7f7b90038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009990 tx=0x7f7b94006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:36.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.489+0000 7f7ba4c00700 1 -- 172.21.15.111:0/860844078 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7b9c011960 con 0x7f7ba80702e0 2024-03-23T11:38:36.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.770+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ba805f5f0 con 0x7f7ba80702e0 2024-03-23T11:38:36.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.770+0000 7f7ba4c00700 1 -- 172.21.15.111:0/860844078 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7b9c01b390 con 0x7f7ba80702e0 2024-03-23T11:38:36.758 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:36.758 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7b90036970 msgr2=0x7f7b90038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7b90036970 0x7f7b90038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009990 tx=0x7f7b94006e40 comp rx=0 tx=0).stop 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 msgr2=0x7f7ba81152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba81152d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c007e30 tx=0x7f7b9c009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 shutdown_connections 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7b90036970 0x7f7b90038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 --2- 172.21.15.111:0/860844078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ba80702e0 0x7f7ba81152d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 >> 172.21.15.111:0/860844078 conn(0x7f7ba80f85e0 msgr2=0x7f7ba8105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 shutdown_connections 2024-03-23T11:38:36.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:36.773+0000 7f7baf800700 1 -- 172.21.15.111:0/860844078 wait complete. 2024-03-23T11:38:36.762 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:37.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:36 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/860844078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:38:38.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-23T11:38:38.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:38:38.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:38.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:37 smithi063 ceph-mon[35189]: Deploying daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:38:38.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:38.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:38.906 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:40.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:39 smithi063 ceph-mon[35189]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:40.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:39 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:41.111 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.123+0000 7f929d400700 1 -- 172.21.15.111:0/4236159562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 msgr2=0x7f929009ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:41.111 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.123+0000 7f929d400700 1 --2- 172.21.15.111:0/4236159562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f929009ae00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f92840048c0 tx=0x7f9284009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.124+0000 7f929d400700 1 -- 172.21.15.111:0/4236159562 shutdown_connections 2024-03-23T11:38:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.124+0000 7f929d400700 1 --2- 172.21.15.111:0/4236159562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f929009ae00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.124+0000 7f929d400700 1 -- 172.21.15.111:0/4236159562 >> 172.21.15.111:0/4236159562 conn(0x7f929000b740 msgr2=0x7f929000bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.124+0000 7f929d400700 1 -- 172.21.15.111:0/4236159562 shutdown_connections 2024-03-23T11:38:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.125+0000 7f929d400700 1 -- 172.21.15.111:0/4236159562 wait complete. 2024-03-23T11:38:41.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.127+0000 7f929d400700 1 Processor -- start 2024-03-23T11:38:41.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.131+0000 7f929d400700 1 -- start start 2024-03-23T11:38:41.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.132+0000 7f929d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f92900b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:41.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.132+0000 7f929d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92900b18d0 con 0x7f929009aa30 2024-03-23T11:38:41.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.132+0000 7f9297e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f92900b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:41.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.132+0000 7f9297e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f92900b1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.111:38534/0 (socket says 172.21.15.111:38534) 2024-03-23T11:38:41.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.132+0000 7f9297e00700 1 -- 172.21.15.111:0/2964603327 learned_addr learned my addr 172.21.15.111:0/2964603327 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:41.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.133+0000 7f9297e00700 1 -- 172.21.15.111:0/2964603327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9284005040 con 0x7f929009aa30 2024-03-23T11:38:41.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.133+0000 7f9297e00700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f92900b1390 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9284009040 tx=0x7f9284009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:41.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.133+0000 7f928be00700 1 -- 172.21.15.111:0/2964603327 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9284009220 con 0x7f929009aa30 2024-03-23T11:38:41.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.133+0000 7f928be00700 1 -- 172.21.15.111:0/2964603327 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9284009830 con 0x7f929009aa30 2024-03-23T11:38:41.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.133+0000 7f928be00700 1 -- 172.21.15.111:0/2964603327 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f928401c3f0 con 0x7f929009aa30 2024-03-23T11:38:41.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.133+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92900b1ad0 con 0x7f929009aa30 2024-03-23T11:38:41.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.134+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92900b1fc0 con 0x7f929009aa30 2024-03-23T11:38:41.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.135+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9290096130 con 0x7f929009aa30 2024-03-23T11:38:41.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.135+0000 7f928be00700 1 -- 172.21.15.111:0/2964603327 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f9284009380 con 0x7f929009aa30 2024-03-23T11:38:41.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.136+0000 7f928be00700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f92800369c0 0x7f9280038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:41.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.136+0000 7f928be00700 1 -- 172.21.15.111:0/2964603327 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9284049ad0 con 0x7f929009aa30 2024-03-23T11:38:41.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.137+0000 7f9297400700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f92800369c0 0x7f9280038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:41.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.139+0000 7f9297400700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f92800369c0 0x7f9280038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f928c009990 tx=0x7f928c006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:41.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.143+0000 7f928be00700 1 -- 172.21.15.111:0/2964603327 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f92840119c0 con 0x7f929009aa30 2024-03-23T11:38:41.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.437+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92900050c0 con 0x7f929009aa30 2024-03-23T11:38:41.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.438+0000 7f928be00700 1 -- 172.21.15.111:0/2964603327 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9284011370 con 0x7f929009aa30 2024-03-23T11:38:41.426 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:41.426 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:35:51.785116Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f92800369c0 msgr2=0x7f9280038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f92800369c0 0x7f9280038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f928c009990 tx=0x7f928c006e40 comp rx=0 tx=0).stop 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 msgr2=0x7f92900b1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f92900b1390 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9284009040 tx=0x7f9284009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 shutdown_connections 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f92800369c0 0x7f9280038e70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 --2- 172.21.15.111:0/2964603327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f929009aa30 0x7f92900b1390 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 >> 172.21.15.111:0/2964603327 conn(0x7f929000b740 msgr2=0x7f9290093a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 shutdown_connections 2024-03-23T11:38:41.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:41.441+0000 7f929d400700 1 -- 172.21.15.111:0/2964603327 wait complete. 2024-03-23T11:38:41.430 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:42.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: Deploying daemon mon.smithi111 on smithi111 2024-03-23T11:38:42.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:41 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/2964603327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:43.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-23T11:38:43.089 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-23T11:38:44.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:43 smithi063 ceph-mon[35189]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:44.472 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:38:45.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:38:45.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:46.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:45 smithi063 ceph-mon[35189]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:46.612 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:46 smithi111 ceph-mon[44005]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:46.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:46 smithi111 ceph-mon[44005]: mon.smithi111@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-23T11:38:46.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:46 smithi111 bash[43924]: 02d10f77a67061e43227c330564ac050e521a340af5450bda4f3d3181c049f26 2024-03-23T11:38:46.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:46 smithi111 systemd[1]: Started Ceph mon.smithi111 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:38:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:38:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: mon.smithi063 calling monitor election 2024-03-23T11:38:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: mon.smithi111 calling monitor election 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: mon.smithi063 is new leader, mons smithi063,smithi111 in quorum (ranks 0,1) 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: monmap e2: 2 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: fsmap 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: osdmap e5: 0 total, 0 up, 0 in 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: mgrmap e17: smithi063.gehkcr(active, since 36s) 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: overall HEALTH_OK 2024-03-23T11:38:52.137 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: mon.smithi063 calling monitor election 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: mon.smithi111 calling monitor election 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:52.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: mon.smithi063 is new leader, mons smithi063,smithi111 in quorum (ranks 0,1) 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: monmap e2: 2 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: fsmap 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: osdmap e5: 0 total, 0 up, 0 in 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: mgrmap e17: smithi063.gehkcr(active, since 36s) 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: overall HEALTH_OK 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.157 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:38:52.158 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/637457977' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:38:52.158 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.158 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.158 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.158 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:52.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.762+0000 7f6895800700 1 -- 172.21.15.111:0/316109007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 msgr2=0x7f6874008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:52.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.762+0000 7f6895800700 1 --2- 172.21.15.111:0/316109007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 0x7f6874008650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f68900713a0 tx=0x7f6880002040 comp rx=0 tx=0).stop 2024-03-23T11:38:52.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.762+0000 7f6895800700 1 -- 172.21.15.111:0/316109007 shutdown_connections 2024-03-23T11:38:52.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.762+0000 7f6895800700 1 --2- 172.21.15.111:0/316109007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 0x7f6874008650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:52.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.762+0000 7f6895800700 1 -- 172.21.15.111:0/316109007 >> 172.21.15.111:0/316109007 conn(0x7f68900f86f0 msgr2=0x7f68900fab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:52.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.763+0000 7f6895800700 1 -- 172.21.15.111:0/316109007 shutdown_connections 2024-03-23T11:38:52.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.763+0000 7f6895800700 1 -- 172.21.15.111:0/316109007 wait complete. 2024-03-23T11:38:52.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.763+0000 7f6895800700 1 Processor -- start 2024-03-23T11:38:52.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.764+0000 7f6895800700 1 -- start start 2024-03-23T11:38:52.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.764+0000 7f6895800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 0x7f6890115530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:52.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.764+0000 7f6895800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 0x7f689010f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:52.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.764+0000 7f6895800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6890116030 con 0x7f68740061f0 2024-03-23T11:38:52.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.764+0000 7f6895800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689010fa40 con 0x7f6890115a70 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.764+0000 7f688e000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 0x7f689010f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688e000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 0x7f689010f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40920/0 (socket says 172.21.15.111:40920) 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688e000700 1 -- 172.21.15.111:0/3230916004 learned_addr learned my addr 172.21.15.111:0/3230916004 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688ea00700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 0x7f6890115530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688e000700 1 -- 172.21.15.111:0/3230916004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 msgr2=0x7f689010f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688e000700 1 -- 172.21.15.111:0/3230916004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 msgr2=0x7f689010f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688e000700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 0x7f689010f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-23T11:38:52.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688e000700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 0x7f689010f500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688ea00700 1 -- 172.21.15.111:0/3230916004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 msgr2=0x7f689010f500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688ea00700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 0x7f689010f500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688ea00700 1 -- 172.21.15.111:0/3230916004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6880005040 con 0x7f68740061f0 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.765+0000 7f688ea00700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 0x7f6890115530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f688001f000 tx=0x7f6880002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.766+0000 7f6886a00700 1 -- 172.21.15.111:0/3230916004 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6880004290 con 0x7f68740061f0 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.766+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689010fcc0 con 0x7f68740061f0 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.766+0000 7f6886a00700 1 -- 172.21.15.111:0/3230916004 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6880017410 con 0x7f68740061f0 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.766+0000 7f6886a00700 1 -- 172.21.15.111:0/3230916004 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6880022a40 con 0x7f68740061f0 2024-03-23T11:38:52.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.766+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68901101a0 con 0x7f68740061f0 2024-03-23T11:38:52.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.767+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6890109360 con 0x7f68740061f0 2024-03-23T11:38:52.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.768+0000 7f6886a00700 1 -- 172.21.15.111:0/3230916004 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f6880022cb0 con 0x7f68740061f0 2024-03-23T11:38:52.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.768+0000 7f6886a00700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6870069360 0x7f687006b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:52.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.768+0000 7f6886a00700 1 -- 172.21.15.111:0/3230916004 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f68800885c0 con 0x7f68740061f0 2024-03-23T11:38:52.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.768+0000 7f688e000700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6870069360 0x7f687006b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:52.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.770+0000 7f688e000700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6870069360 0x7f687006b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f687c006fd0 tx=0x7f687c008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:52.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:52.774+0000 7f6886a00700 1 -- 172.21.15.111:0/3230916004 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6880059340 con 0x7f68740061f0 2024-03-23T11:38:53.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.125+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f689005f620 con 0x7f68740061f0 2024-03-23T11:38:53.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.126+0000 7f6886a00700 1 -- 172.21.15.111:0/3230916004 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f6880058ed0 con 0x7f68740061f0 2024-03-23T11:38:53.114 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-23T11:38:53.114 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":2,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","modified":"2024-03-23T11:38:46.630378Z","created":"2024-03-23T11:35:51.785116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6870069360 msgr2=0x7f687006b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6870069360 0x7f687006b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f687c006fd0 tx=0x7f687c008040 comp rx=0 tx=0).stop 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 msgr2=0x7f6890115530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 0x7f6890115530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f688001f000 tx=0x7f6880002040 comp rx=0 tx=0).stop 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 shutdown_connections 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6870069360 0x7f687006b810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68740061f0 0x7f6890115530 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 --2- 172.21.15.111:0/3230916004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6890115a70 0x7f689010f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 >> 172.21.15.111:0/3230916004 conn(0x7f68900f86f0 msgr2=0x7f68901046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 shutdown_connections 2024-03-23T11:38:53.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:38:53.129+0000 7f6895800700 1 -- 172.21.15.111:0/3230916004 wait complete. 2024-03-23T11:38:53.118 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 2 2024-03-23T11:38:53.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:52 smithi063 ceph-mon[35189]: mgrmap e18: smithi063.gehkcr(active, since 36s), standbys: smithi111.nnblsk 2024-03-23T11:38:53.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:52 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:38:53.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:52 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:38:53.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:52 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:53.137 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:52 smithi111 ceph-mon[44005]: mgrmap e18: smithi063.gehkcr(active, since 36s), standbys: smithi111.nnblsk 2024-03-23T11:38:53.137 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:52 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:38:53.138 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:52 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:38:53.138 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:52 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:38:53.741 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:53 smithi111 ceph-mon[44005]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:53.742 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:53 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/3230916004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:53.766 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-23T11:38:53.766 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-23T11:38:54.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:53 smithi063 ceph-mon[35189]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:54.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:53 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/3230916004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-23T11:38:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:55.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:55.853 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:55 smithi063 ceph-mon[35189]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:55.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:55.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:55 smithi111 ceph-mon[44005]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:38:56.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.155+0000 7f9ecda00700 1 -- 172.21.15.63:0/2665365586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec80fdae0 msgr2=0x7f9ec80fdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:56.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.155+0000 7f9ecda00700 1 --2- 172.21.15.63:0/2665365586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec80fdae0 0x7f9ec80fdeb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80048c0 tx=0x7f9eb8009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:56.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.155+0000 7f9ecda00700 1 -- 172.21.15.63:0/2665365586 shutdown_connections 2024-03-23T11:38:56.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.155+0000 7f9ecda00700 1 --2- 172.21.15.63:0/2665365586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec80fdae0 0x7f9ec80fdeb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:56.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.155+0000 7f9ecda00700 1 -- 172.21.15.63:0/2665365586 >> 172.21.15.63:0/2665365586 conn(0x7f9ec8071800 msgr2=0x7f9ec8071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:56.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.155+0000 7f9ecda00700 1 -- 172.21.15.63:0/2665365586 shutdown_connections 2024-03-23T11:38:56.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.155+0000 7f9ecda00700 1 -- 172.21.15.63:0/2665365586 wait complete. 2024-03-23T11:38:56.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ecda00700 1 Processor -- start 2024-03-23T11:38:56.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ecda00700 1 -- start start 2024-03-23T11:38:56.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ecda00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 0x7f9ec81153d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:56.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ecda00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec8115910 0x7f9ec810f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ecda00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8115f20 con 0x7f9ec8115910 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ecda00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec810f870 con 0x7f9ec80fdae0 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ec6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec8115910 0x7f9ec810f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ec6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec8115910 0x7f9ec810f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55110/0 (socket says 172.21.15.63:55110) 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.156+0000 7f9ec6000700 1 -- 172.21.15.63:0/2336132127 learned_addr learned my addr 172.21.15.63:0/2336132127 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6a00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 0x7f9ec81153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6000700 1 -- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 msgr2=0x7f9ec81153d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6a00700 1 -- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 msgr2=0x7f9ec81153d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6a00700 1 -- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 msgr2=0x7f9ec81153d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6a00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 0x7f9ec81153d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6a00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 0x7f9ec81153d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6000700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 0x7f9ec81153d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:56.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6000700 1 -- 172.21.15.63:0/2336132127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb8005040 con 0x7f9ec8115910 2024-03-23T11:38:56.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ec6000700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec8115910 0x7f9ec810f330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400b790 tx=0x7f9eb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:56.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.157+0000 7f9ebea00700 1 -- 172.21.15.63:0/2336132127 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb4010870 con 0x7f9ec8115910 2024-03-23T11:38:56.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.158+0000 7f9ebea00700 1 -- 172.21.15.63:0/2336132127 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9eb4010e80 con 0x7f9ec8115910 2024-03-23T11:38:56.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.158+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec810fb50 con 0x7f9ec8115910 2024-03-23T11:38:56.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.158+0000 7f9ebea00700 1 -- 172.21.15.63:0/2336132127 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb400d3b0 con 0x7f9ec8115910 2024-03-23T11:38:56.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.158+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec8110090 con 0x7f9ec8115910 2024-03-23T11:38:56.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.159+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec80658f0 con 0x7f9ec8115910 2024-03-23T11:38:56.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.160+0000 7f9ebea00700 1 -- 172.21.15.63:0/2336132127 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f9eb400f3e0 con 0x7f9ec8115910 2024-03-23T11:38:56.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.161+0000 7f9ebea00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eac069360 0x7f9eac06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:38:56.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.161+0000 7f9ebea00700 1 -- 172.21.15.63:0/2336132127 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9eb40866b0 con 0x7f9ec8115910 2024-03-23T11:38:56.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.161+0000 7f9ec6a00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eac069360 0x7f9eac06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:38:56.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.162+0000 7f9ec6a00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eac069360 0x7f9eac06b810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800b3b0 tx=0x7f9eb8009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:38:56.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.166+0000 7f9ebea00700 1 -- 172.21.15.63:0/2336132127 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9eb40574b0 con 0x7f9ec8115910 2024-03-23T11:38:56.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.377+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9ec805f5f0 con 0x7f9ec8115910 2024-03-23T11:38:56.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.377+0000 7f9ebea00700 1 -- 172.21.15.63:0/2336132127 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f9eb4014100 con 0x7f9ec8115910 2024-03-23T11:38:56.364 INFO:teuthology.orchestra.run.smithi063.stdout:# minimal ceph.conf for 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:38:56.364 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-03-23T11:38:56.364 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:38:56.364 INFO:teuthology.orchestra.run.smithi063.stdout: mon_host = [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eac069360 msgr2=0x7f9eac06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eac069360 0x7f9eac06b810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800b3b0 tx=0x7f9eb8009e30 comp rx=0 tx=0).stop 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec8115910 msgr2=0x7f9ec810f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec8115910 0x7f9ec810f330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400b790 tx=0x7f9eb4009250 comp rx=0 tx=0).stop 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 shutdown_connections 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eac069360 0x7f9eac06b810 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec80fdae0 0x7f9ec81153d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 --2- 172.21.15.63:0/2336132127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ec8115910 0x7f9ec810f330 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 >> 172.21.15.63:0/2336132127 conn(0x7f9ec8071800 msgr2=0x7f9ec80fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 shutdown_connections 2024-03-23T11:38:56.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:38:56.380+0000 7f9ecda00700 1 -- 172.21.15.63:0/2336132127 wait complete. 2024-03-23T11:38:57.033 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.033 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.033 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.033 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.033 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: Reconfiguring mon.smithi063 (unknown last config time)... 2024-03-23T11:38:57.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:38:57.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:38:57.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:57.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: Reconfiguring daemon mon.smithi063 on smithi063 2024-03-23T11:38:57.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:56 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2336132127' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:57.056 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-23T11:38:57.056 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:38:57.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-03-23T11:38:57.128 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:38:57.128 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: Reconfiguring mon.smithi063 (unknown last config time)... 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: Reconfiguring daemon mon.smithi063 on smithi063 2024-03-23T11:38:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:56 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2336132127' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:57.237 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:38:57.238 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-03-23T11:38:57.314 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:38:57.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:38:57.423 INFO:tasks.cephadm:Deploying OSDs... 2024-03-23T11:38:57.423 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:38:57.423 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T11:38:57.468 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T11:38:57.468 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-03-23T11:38:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T11:38:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:57.554 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 448 Links: 1 2024-03-23T11:38:57.554 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:57.555 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:57.555 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-23 11:38:04.116579462 +0000 2024-03-23T11:38:57.555 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-23 11:30:19.942518585 +0000 2024-03-23T11:38:57.555 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-23 11:30:19.942518585 +0000 2024-03-23T11:38:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-23 11:30:19.912518866 +0000 2024-03-23T11:38:57.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T11:38:57.622 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-23T11:38:57.623 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-23T11:38:57.623 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000317776 s, 1.6 MB/s 2024-03-23T11:38:57.625 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T11:38:57.672 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 454 Links: 1 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-23 11:38:04.116579462 +0000 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-23 11:30:20.426514055 +0000 2024-03-23T11:38:57.717 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-23 11:30:20.426514055 +0000 2024-03-23T11:38:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-23 11:30:20.399514308 +0000 2024-03-23T11:38:57.718 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T11:38:57.821 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-23T11:38:57.821 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-23T11:38:57.821 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000273453 s, 1.9 MB/s 2024-03-23T11:38:57.823 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T11:38:57.872 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-03-23T11:38:57.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:57 smithi063 ceph-mon[35189]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:57.916 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T11:38:57.916 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:57.916 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 460 Links: 1 2024-03-23T11:38:57.916 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:57.916 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:57.916 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-23 11:38:04.116579462 +0000 2024-03-23T11:38:57.917 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-23 11:30:20.910509525 +0000 2024-03-23T11:38:57.917 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-23 11:30:20.910509525 +0000 2024-03-23T11:38:57.917 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-23 11:30:20.884509769 +0000 2024-03-23T11:38:57.917 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T11:38:58.027 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-23T11:38:58.027 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-23T11:38:58.027 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.00646325 s, 79.2 kB/s 2024-03-23T11:38:58.030 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T11:38:58.079 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-03-23T11:38:58.123 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T11:38:58.123 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:58.123 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 466 Links: 1 2024-03-23T11:38:58.123 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:58.124 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:58.124 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-23 11:38:04.117579452 +0000 2024-03-23T11:38:58.124 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-23 11:30:21.395504986 +0000 2024-03-23T11:38:58.124 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-23 11:30:21.395504986 +0000 2024-03-23T11:38:58.124 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-23 11:30:21.368505239 +0000 2024-03-23T11:38:58.124 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T11:38:58.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:57 smithi111 ceph-mon[44005]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:58.226 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-23T11:38:58.226 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-23T11:38:58.227 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000298839 s, 1.7 MB/s 2024-03-23T11:38:58.229 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T11:38:58.275 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:38:58.275 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T11:38:58.324 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T11:38:58.325 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-03-23T11:38:58.371 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T11:38:58.371 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:58.372 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 448 Links: 1 2024-03-23T11:38:58.372 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:58.372 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:58.372 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-23 11:38:44.901603065 +0000 2024-03-23T11:38:58.372 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-23 11:30:11.556267797 +0000 2024-03-23T11:38:58.372 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-23 11:30:11.556267797 +0000 2024-03-23T11:38:58.372 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-23 11:30:11.531268227 +0000 2024-03-23T11:38:58.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T11:38:58.476 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-23T11:38:58.476 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-23T11:38:58.476 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000325165 s, 1.6 MB/s 2024-03-23T11:38:58.479 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T11:38:58.525 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-03-23T11:38:58.569 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T11:38:58.570 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:58.570 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 454 Links: 1 2024-03-23T11:38:58.570 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:58.570 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:58.570 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-23 11:38:44.901603065 +0000 2024-03-23T11:38:58.570 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-23 11:30:12.020259818 +0000 2024-03-23T11:38:58.571 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-23 11:30:12.020259818 +0000 2024-03-23T11:38:58.571 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-23 11:30:11.994260265 +0000 2024-03-23T11:38:58.571 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T11:38:58.673 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-23T11:38:58.673 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-23T11:38:58.673 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000288787 s, 1.8 MB/s 2024-03-23T11:38:58.675 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T11:38:58.721 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-03-23T11:38:58.765 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T11:38:58.765 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:58.766 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 460 Links: 1 2024-03-23T11:38:58.766 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:58.766 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:58.766 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-23 11:38:44.902603048 +0000 2024-03-23T11:38:58.766 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-23 11:30:12.494251668 +0000 2024-03-23T11:38:58.766 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-23 11:30:12.494251668 +0000 2024-03-23T11:38:58.766 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-23 11:30:12.469252097 +0000 2024-03-23T11:38:58.766 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T11:38:58.868 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-23T11:38:58.868 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-23T11:38:58.868 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000301627 s, 1.7 MB/s 2024-03-23T11:38:58.870 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T11:38:58.916 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-03-23T11:38:58.959 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 466 Links: 1 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-23 11:38:44.902603048 +0000 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-23 11:30:12.972243448 +0000 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-23 11:30:12.972243448 +0000 2024-03-23T11:38:58.960 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-23 11:30:12.946243895 +0000 2024-03-23T11:38:58.960 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T11:38:59.062 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-23T11:38:59.062 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-23T11:38:59.062 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000302171 s, 1.7 MB/s 2024-03-23T11:38:59.064 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T11:38:59.111 INFO:tasks.cephadm:Deploying osd.0 on smithi063 with /dev/vg_nvme/lv_4... 2024-03-23T11:38:59.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: Reconfiguring mgr.smithi063.gehkcr (unknown last config time)... 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: Reconfiguring daemon mgr.smithi063.gehkcr on smithi063 2024-03-23T11:38:59.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:38:59 smithi111 ceph-mon[44005]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:38:59.701 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:59.701 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:38:59.701 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: Reconfiguring mgr.smithi063.gehkcr (unknown last config time)... 2024-03-23T11:38:59.701 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:38:59.701 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:38:59.701 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:38:59.701 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: Reconfiguring daemon mgr.smithi063.gehkcr on smithi063 2024-03-23T11:38:59.702 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:38:59 smithi063 ceph-mon[35189]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:00.071 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:39:00.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:00.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:00.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:00.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:00 smithi063 ceph-mon[35189]: Reconfiguring crash.smithi063 (monmap changed)... 2024-03-23T11:39:00.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:39:00.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:00.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:00 smithi063 ceph-mon[35189]: Reconfiguring daemon crash.smithi063 on smithi063 2024-03-23T11:39:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:00 smithi111 ceph-mon[44005]: Reconfiguring crash.smithi063 (monmap changed)... 2024-03-23T11:39:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:39:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:00 smithi111 ceph-mon[44005]: Reconfiguring daemon crash.smithi063 on smithi063 2024-03-23T11:39:01.611 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:01 smithi063 ceph-mon[35189]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:01.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:01 smithi111 ceph-mon[44005]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:03.556 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-23T11:39:03.556 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-23T11:39:03.556 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-23T11:39:03.556 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-23T11:39:03.557 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.050034 s, 210 MB/s 2024-03-23T11:39:03.557 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-23T11:39:03.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:03 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:03.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:03 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:03.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:03 smithi063 ceph-mon[35189]: Reconfiguring alertmanager.smithi063 (dependencies changed)... 2024-03-23T11:39:03.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:03 smithi063 ceph-mon[35189]: Reconfiguring daemon alertmanager.smithi063 on smithi063 2024-03-23T11:39:03.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:03 smithi063 ceph-mon[35189]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:03.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:03 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:03.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:03 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:03.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:03 smithi111 ceph-mon[44005]: Reconfiguring alertmanager.smithi063 (dependencies changed)... 2024-03-23T11:39:03.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:03 smithi111 ceph-mon[44005]: Reconfiguring daemon alertmanager.smithi063 on smithi063 2024-03-23T11:39:03.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:03 smithi111 ceph-mon[44005]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:04.305 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_4 2024-03-23T11:39:06.101 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:05 smithi063 ceph-mon[35189]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:06.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:05 smithi111 ceph-mon[44005]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:06.495 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:39:08.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:07 smithi063 ceph-mon[35189]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:08.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:07 smithi111 ceph-mon[44005]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:09.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 -- 172.21.15.63:0/3019239048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 msgr2=0x7f06800ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:39:09.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 --2- 172.21.15.63:0/3019239048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f06800ffa20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0674004940 tx=0x7f0674009e30 comp rx=0 tx=0).stop 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 -- 172.21.15.63:0/3019239048 shutdown_connections 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 --2- 172.21.15.63:0/3019239048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06800fff60 0x7f0680102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 --2- 172.21.15.63:0/3019239048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f06800ffa20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 -- 172.21.15.63:0/3019239048 >> 172.21.15.63:0/3019239048 conn(0x7f06800f7430 msgr2=0x7f06800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 -- 172.21.15.63:0/3019239048 shutdown_connections 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.248+0000 7f0687600700 1 -- 172.21.15.63:0/3019239048 wait complete. 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.249+0000 7f0687600700 1 Processor -- start 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.249+0000 7f0687600700 1 -- start start 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.249+0000 7f0687600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f068010d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:39:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.249+0000 7f0687600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06800fff60 0x7f068010dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.249+0000 7f0687600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068010e550 con 0x7f0680066280 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.249+0000 7f0687600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068010ed40 con 0x7f06800fff60 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f0684c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f068010d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f0684c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f068010d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36178/0 (socket says 172.21.15.63:36178) 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f0684c00700 1 -- 172.21.15.63:0/1879027863 learned_addr learned my addr 172.21.15.63:0/1879027863 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f067fe00700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06800fff60 0x7f068010dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f0684c00700 1 -- 172.21.15.63:0/1879027863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06800fff60 msgr2=0x7f068010dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:39:09.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f0684c00700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06800fff60 0x7f068010dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f0684c00700 1 -- 172.21.15.63:0/1879027863 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0674005040 con 0x7f0680066280 2024-03-23T11:39:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f0684c00700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f068010d980 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f067400b3b0 tx=0x7f0674009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.250+0000 7f067cc00700 1 -- 172.21.15.63:0/1879027863 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f067400e040 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.251+0000 7f067cc00700 1 -- 172.21.15.63:0/1879027863 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0674004290 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.251+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068010efc0 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.251+0000 7f067cc00700 1 -- 172.21.15.63:0/1879027863 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f067401c3f0 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.251+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068010f580 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.252+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068005f5f0 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.253+0000 7f067cc00700 1 -- 172.21.15.63:0/1879027863 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f067400bc50 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.253+0000 7f067cc00700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f0668069210 0x7f066806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.253+0000 7f067cc00700 1 -- 172.21.15.63:0/1879027863 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0674090920 con 0x7f0680066280 2024-03-23T11:39:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.254+0000 7f067fe00700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f0668069210 0x7f066806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:39:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.254+0000 7f067fe00700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f0668069210 0x7f066806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f066c006fd0 tx=0x7f066c008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:39:09.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.258+0000 7f067cc00700 1 -- 172.21.15.63:0/1879027863 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f067408d060 con 0x7f0680066280 2024-03-23T11:39:09.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:09.492+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f068010f8d0 con 0x7f0668069210 2024-03-23T11:39:10.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:09 smithi063 ceph-mon[35189]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:10.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:39:10.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:39:10.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:10.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:09 smithi111 ceph-mon[44005]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:10.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:39:10.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:39:10.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:11.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:11 smithi063 ceph-mon[35189]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:39:11.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:11 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:11.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:11 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:11.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:11 smithi063 ceph-mon[35189]: Reconfiguring grafana.smithi063 (dependencies changed)... 2024-03-23T11:39:11.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:11 smithi063 ceph-mon[35189]: Reconfiguring daemon grafana.smithi063 on smithi063 2024-03-23T11:39:11.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:11 smithi111 ceph-mon[44005]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:39:11.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:11 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:11.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:11 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:11.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:11 smithi111 ceph-mon[44005]: Reconfiguring grafana.smithi063 (dependencies changed)... 2024-03-23T11:39:11.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:11 smithi111 ceph-mon[44005]: Reconfiguring daemon grafana.smithi063 on smithi063 2024-03-23T11:39:12.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:12 smithi063 ceph-mon[35189]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:12.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:12 smithi111 ceph-mon[44005]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:14.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:14 smithi063 ceph-mon[35189]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:14.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:14 smithi111 ceph-mon[44005]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:15.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:15.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:16.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:16 smithi063 ceph-mon[35189]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:16.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:16.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:16.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:16 smithi111 ceph-mon[44005]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:16.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:16.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:17.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:17 smithi063 ceph-mon[35189]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-03-23T11:39:17.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:17 smithi063 ceph-mon[35189]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-03-23T11:39:17.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:17 smithi111 ceph-mon[44005]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-03-23T11:39:17.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:17 smithi111 ceph-mon[44005]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-03-23T11:39:18.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:18 smithi111 ceph-mon[44005]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:18.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:18 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1301472960' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e4f29eb-375c-4f37-a8d8-f82b40650ccb"}]: dispatch 2024-03-23T11:39:18.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:18 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1301472960' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e4f29eb-375c-4f37-a8d8-f82b40650ccb"}]': finished 2024-03-23T11:39:18.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:18 smithi111 ceph-mon[44005]: osdmap e6: 1 total, 0 up, 1 in 2024-03-23T11:39:18.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:18 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:18.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:18 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1112079437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:39:18.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:18 smithi063 ceph-mon[35189]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:18 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1301472960' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e4f29eb-375c-4f37-a8d8-f82b40650ccb"}]: dispatch 2024-03-23T11:39:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:18 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1301472960' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e4f29eb-375c-4f37-a8d8-f82b40650ccb"}]': finished 2024-03-23T11:39:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:18 smithi063 ceph-mon[35189]: osdmap e6: 1 total, 0 up, 1 in 2024-03-23T11:39:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:18 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:18.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:18 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1112079437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:39:20.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:20 smithi063 ceph-mon[35189]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:20 smithi111 ceph-mon[44005]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:22.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:22 smithi063 ceph-mon[35189]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:22 smithi111 ceph-mon[44005]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:24 smithi111 ceph-mon[44005]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:24.437 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:24 smithi063 ceph-mon[35189]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:26.280 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:26 smithi063 ceph-mon[35189]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:26.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:26 smithi111 ceph-mon[44005]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:28.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:28 smithi063 ceph-mon[35189]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:28.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:28.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:28.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:39:28.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:28.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:28 smithi111 ceph-mon[44005]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:28.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:28.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:28.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:39:28.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:29.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:29 smithi063 ceph-mon[35189]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-23T11:39:29.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:29 smithi063 ceph-mon[35189]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-23T11:39:29.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:29 smithi111 ceph-mon[44005]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-23T11:39:29.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:29 smithi111 ceph-mon[44005]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:30.447 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:30.484 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:30.485 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:31.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:31 smithi111 ceph-mon[44005]: Deploying daemon osd.0 on smithi063 2024-03-23T11:39:31.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:31 smithi111 ceph-mon[44005]: Reconfiguring mgr.smithi111.nnblsk (monmap changed)... 2024-03-23T11:39:31.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:31 smithi111 ceph-mon[44005]: Reconfiguring daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:39:31.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:31 smithi063 ceph-mon[35189]: Deploying daemon osd.0 on smithi063 2024-03-23T11:39:31.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:31 smithi063 ceph-mon[35189]: Reconfiguring mgr.smithi111.nnblsk (monmap changed)... 2024-03-23T11:39:31.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:31 smithi063 ceph-mon[35189]: Reconfiguring daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:39:32.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:32 smithi111 ceph-mon[44005]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:32.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:32 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:32.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:32 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:32.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:32 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:39:32.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:32 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:39:32.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:32 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:32.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:32 smithi063 ceph-mon[35189]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:32.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:32.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:32.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:39:32.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:39:32.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:32 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:33.184 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: Reconfiguring mon.smithi111 (monmap changed)... 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:39:33.470 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-03-23T11:39:33.471 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.471 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:39:33.471 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-03-23T11:39:33.471 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.471 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:39:33.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: Reconfiguring mon.smithi111 (monmap changed)... 2024-03-23T11:39:33.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:33.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:39:34.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:34 smithi111 ceph-mon[44005]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:34.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:34 smithi111 ceph-mon[44005]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:39:34.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:34 smithi111 ceph-mon[44005]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-03-23T11:39:34.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:34 smithi111 ceph-mon[44005]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:39:34.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:34 smithi111 ceph-mon[44005]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-03-23T11:39:34.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:34 smithi111 ceph-mon[44005]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:39:34.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:34 smithi111 ceph-mon[44005]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-03-23T11:39:34.446 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:34 smithi063 ceph-mon[35189]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:34.446 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:34 smithi063 ceph-mon[35189]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:39:34.446 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:34 smithi063 ceph-mon[35189]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-03-23T11:39:34.446 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:34 smithi063 ceph-mon[35189]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:39:34.446 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:34 smithi063 ceph-mon[35189]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-03-23T11:39:34.447 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:34 smithi063 ceph-mon[35189]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:39:34.447 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:34 smithi063 ceph-mon[35189]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-03-23T11:39:36.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:36 smithi063 ceph-mon[35189]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:36.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:36 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:36.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:36 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:36.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:36 smithi111 ceph-mon[44005]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:36.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:36 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:36.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:36 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:38.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:38 smithi063 ceph-mon[35189]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:38.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:38 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:38.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:38 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:38.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:38 smithi111 ceph-mon[44005]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:38.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:38 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:38.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:38 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:40.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:40 smithi063 ceph-mon[35189]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:40.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:40 smithi111 ceph-mon[44005]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:41.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:41.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:41.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:41 smithi063 ceph-mon[35189]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:41.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:41 smithi063 ceph-mon[35189]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-23T11:39:41.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:41.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:39:41.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:41 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:41.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:41 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:41.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:41 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:41.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:41 smithi111 ceph-mon[44005]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:41.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:41 smithi111 ceph-mon[44005]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-23T11:39:41.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:41 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:41.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:41 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:39:41.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:41 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:42.321 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 0 on host 'smithi063' 2024-03-23T11:39:42.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.333+0000 7f067cc00700 1 -- 172.21.15.63:0/1879027863 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f068010f8d0 con 0x7f0668069210 2024-03-23T11:39:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f0668069210 msgr2=0x7f066806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:39:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f0668069210 0x7f066806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f066c006fd0 tx=0x7f066c008040 comp rx=0 tx=0).stop 2024-03-23T11:39:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 msgr2=0x7f068010d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:39:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f068010d980 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f067400b3b0 tx=0x7f0674009e30 comp rx=0 tx=0).stop 2024-03-23T11:39:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 shutdown_connections 2024-03-23T11:39:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f0668069210 0x7f066806b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0680066280 0x7f068010d980 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 --2- 172.21.15.63:0/1879027863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06800fff60 0x7f068010dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 >> 172.21.15.63:0/1879027863 conn(0x7f06800f7430 msgr2=0x7f06800f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:39:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 shutdown_connections 2024-03-23T11:39:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:42.336+0000 7f0687600700 1 -- 172.21.15.63:0/1879027863 wait complete. 2024-03-23T11:39:42.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:42 smithi063 ceph-mon[35189]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-23T11:39:42.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:42 smithi063 ceph-mon[35189]: osdmap e7: 1 total, 0 up, 1 in 2024-03-23T11:39:42.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:42 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:42.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:42 smithi063 ceph-mon[35189]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:39:42.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:42 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:39:42.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:42 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:42.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:42 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:42.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:42 smithi111 ceph-mon[44005]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-23T11:39:42.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:42 smithi111 ceph-mon[44005]: osdmap e7: 1 total, 0 up, 1 in 2024-03-23T11:39:42.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:42 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:42.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:42 smithi111 ceph-mon[44005]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:39:42.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:42 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:39:42.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:42 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:42.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:42 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:43.073 DEBUG:teuthology.orchestra.run.smithi063:osd.0> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.0.service 2024-03-23T11:39:43.078 INFO:tasks.cephadm:Deploying osd.1 on smithi063 with /dev/vg_nvme/lv_3... 2024-03-23T11:39:43.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-23T11:39:43.150 INFO:journalctl@ceph.osd.0.smithi063.stdout:-- Logs begin at Sat 2024-03-23 11:23:13 UTC. -- 2024-03-23T11:39:43.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:43 smithi111 ceph-mon[44005]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-23T11:39:43.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:43 smithi111 ceph-mon[44005]: osdmap e8: 1 total, 0 up, 1 in 2024-03-23T11:39:43.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:43 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:43.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:43 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:43.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:43 smithi111 ceph-mon[44005]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:43.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:43 smithi063 ceph-mon[35189]: from='osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-23T11:39:43.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:43 smithi063 ceph-mon[35189]: osdmap e8: 1 total, 0 up, 1 in 2024-03-23T11:39:43.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:43 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:43.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:43 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:43.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:43 smithi063 ceph-mon[35189]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-23T11:39:43.901 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:39:44.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: purged_snaps scrub starts 2024-03-23T11:39:44.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: purged_snaps scrub ok 2024-03-23T11:39:44.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:44.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] boot 2024-03-23T11:39:44.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: osdmap e9: 1 total, 1 up, 1 in 2024-03-23T11:39:44.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:44.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:44.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:44.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: purged_snaps scrub starts 2024-03-23T11:39:44.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: purged_snaps scrub ok 2024-03-23T11:39:44.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:44.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: osd.0 [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] boot 2024-03-23T11:39:44.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: osdmap e9: 1 total, 1 up, 1 in 2024-03-23T11:39:44.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:39:44.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:44.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:44 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:45.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:45 smithi063 ceph-mon[35189]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:45.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:45.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:45 smithi063 ceph-mon[35189]: osdmap e10: 1 total, 1 up, 1 in 2024-03-23T11:39:45.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:45 smithi111 ceph-mon[44005]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:45.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:39:45.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:45 smithi111 ceph-mon[44005]: osdmap e10: 1 total, 1 up, 1 in 2024-03-23T11:39:46.835 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-23T11:39:46.835 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-23T11:39:46.836 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-23T11:39:46.836 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-23T11:39:46.836 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0564468 s, 186 MB/s 2024-03-23T11:39:46.836 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-23T11:39:48.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:47 smithi063 ceph-mon[35189]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:48.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_3 2024-03-23T11:39:48.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:47 smithi111 ceph-mon[44005]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:48.842 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:39:50.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:49 smithi063 ceph-mon[35189]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:50.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:49 smithi111 ceph-mon[44005]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:51.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.172+0000 7f3faa000700 1 -- 172.21.15.63:0/3083972624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 msgr2=0x7f3fa4070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:39:51.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.172+0000 7f3faa000700 1 --2- 172.21.15.63:0/3083972624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4070e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3f90004940 tx=0x7f3f90009e30 comp rx=0 tx=0).stop 2024-03-23T11:39:51.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.172+0000 7f3faa000700 1 -- 172.21.15.63:0/3083972624 shutdown_connections 2024-03-23T11:39:51.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.172+0000 7f3faa000700 1 --2- 172.21.15.63:0/3083972624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4070e10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:51.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.172+0000 7f3faa000700 1 --2- 172.21.15.63:0/3083972624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa4071f50 0x7f3fa4070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:51.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.172+0000 7f3faa000700 1 -- 172.21.15.63:0/3083972624 >> 172.21.15.63:0/3083972624 conn(0x7f3fa40f8560 msgr2=0x7f3fa40fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:39:51.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.173+0000 7f3faa000700 1 -- 172.21.15.63:0/3083972624 shutdown_connections 2024-03-23T11:39:51.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.173+0000 7f3faa000700 1 -- 172.21.15.63:0/3083972624 wait complete. 2024-03-23T11:39:51.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.173+0000 7f3faa000700 1 Processor -- start 2024-03-23T11:39:51.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3faa000700 1 -- start start 2024-03-23T11:39:51.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3faa000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:39:51.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3faa000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa4071f50 0x7f3fa41144b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:39:51.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3faa000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa4114b40 con 0x7f3fa40709a0 2024-03-23T11:39:51.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3faa000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa4115330 con 0x7f3fa4071f50 2024-03-23T11:39:51.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3fa3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3fa3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59802/0 (socket says 172.21.15.63:59802) 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3fa3400700 1 -- 172.21.15.63:0/1475259891 learned_addr learned my addr 172.21.15.63:0/1475259891 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.174+0000 7f3fa2a00700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa4071f50 0x7f3fa41144b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3fa3400700 1 -- 172.21.15.63:0/1475259891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa4071f50 msgr2=0x7f3fa41144b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3fa3400700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa4071f50 0x7f3fa41144b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3fa3400700 1 -- 172.21.15.63:0/1475259891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f90005040 con 0x7f3fa40709a0 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3fa3400700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4113f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400b790 tx=0x7f3f94009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3f9b400700 1 -- 172.21.15.63:0/1475259891 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9400f6f0 con 0x7f3fa40709a0 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa4115610 con 0x7f3fa40709a0 2024-03-23T11:39:51.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3f9b400700 1 -- 172.21.15.63:0/1475259891 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f9400fd00 con 0x7f3fa40709a0 2024-03-23T11:39:51.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3f9b400700 1 -- 172.21.15.63:0/1475259891 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f94011e90 con 0x7f3fa40709a0 2024-03-23T11:39:51.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.175+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa4115bd0 con 0x7f3fa40709a0 2024-03-23T11:39:51.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.177+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa405f5f0 con 0x7f3fa40709a0 2024-03-23T11:39:51.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.177+0000 7f3f9b400700 1 -- 172.21.15.63:0/1475259891 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f3f9400f850 con 0x7f3fa40709a0 2024-03-23T11:39:51.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.178+0000 7f3f9b400700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f88069210 0x7f3f8806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:39:51.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.178+0000 7f3f9b400700 1 -- 172.21.15.63:0/1475259891 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f3f94086150 con 0x7f3fa40709a0 2024-03-23T11:39:51.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.178+0000 7f3fa2a00700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f88069210 0x7f3f8806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:39:51.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.179+0000 7f3fa2a00700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f88069210 0x7f3f8806b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3f90004f90 tx=0x7f3f90009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:39:51.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.183+0000 7f3f9b400700 1 -- 172.21.15.63:0/1475259891 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3f94056cf0 con 0x7f3fa40709a0 2024-03-23T11:39:51.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:39:51.415+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fa4115f10 con 0x7f3f88069210 2024-03-23T11:39:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:51 smithi063 ceph-mon[35189]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:39:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:39:52.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:52.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:51 smithi111 ceph-mon[44005]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:52.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:52.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:52.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:39:52.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:39:52.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:53.278 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:52 smithi063 ceph-mon[35189]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:39:53.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:52 smithi111 ceph-mon[44005]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:39:54.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:53 smithi063 ceph-mon[35189]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:54.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:53 smithi111 ceph-mon[44005]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:56.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:56.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:39:56.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:56.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:39:56.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:39:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:39:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:39:56.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:39:56.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:56.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:39:57.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:56 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi063 2024-03-23T11:39:57.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:56 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi063 to 13046M 2024-03-23T11:39:57.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:56 smithi063 ceph-mon[35189]: Detected new or changed devices on smithi063 2024-03-23T11:39:57.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:56 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi063 to 13046M 2024-03-23T11:39:58.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:57 smithi063 ceph-mon[35189]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:58.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:57 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/101737774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e3ddf8b-e2c3-4188-b495-ac2743805f33"}]: dispatch 2024-03-23T11:39:58.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:57 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/101737774' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e3ddf8b-e2c3-4188-b495-ac2743805f33"}]': finished 2024-03-23T11:39:58.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:57 smithi063 ceph-mon[35189]: osdmap e11: 2 total, 1 up, 2 in 2024-03-23T11:39:58.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:39:58.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:57 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/769725442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:39:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:57 smithi111 ceph-mon[44005]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:57 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/101737774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e3ddf8b-e2c3-4188-b495-ac2743805f33"}]: dispatch 2024-03-23T11:39:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:57 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/101737774' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e3ddf8b-e2c3-4188-b495-ac2743805f33"}]': finished 2024-03-23T11:39:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:57 smithi111 ceph-mon[44005]: osdmap e11: 2 total, 1 up, 2 in 2024-03-23T11:39:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:39:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:57 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/769725442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:39:59.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:59.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:59.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:39:59 smithi063 ceph-mon[35189]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:39:59.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:59.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:39:59.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:39:59 smithi111 ceph-mon[44005]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:00.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:00.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:00 smithi063 ceph-mon[35189]: overall HEALTH_OK 2024-03-23T11:40:00.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:00.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:00 smithi111 ceph-mon[44005]: overall HEALTH_OK 2024-03-23T11:40:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:02 smithi063 ceph-mon[35189]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:40:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:02 smithi111 ceph-mon[44005]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:40:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:04.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:04 smithi063 ceph-mon[35189]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:04.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:04 smithi111 ceph-mon[44005]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:06.314 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:06 smithi063 ceph-mon[35189]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:06.315 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:06 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T11:40:06.315 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:06 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:06 smithi111 ceph-mon[44005]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:06 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T11:40:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:06 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:07.171 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:07 smithi063 ceph-mon[35189]: Deploying daemon osd.1 on smithi063 2024-03-23T11:40:07.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:07 smithi111 ceph-mon[44005]: Deploying daemon osd.1 on smithi063 2024-03-23T11:40:08.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:08 smithi111 ceph-mon[44005]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:08.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:08 smithi063 ceph-mon[35189]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:10.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:10 smithi063 ceph-mon[35189]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:10.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:10 smithi111 ceph-mon[44005]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:12.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:12 smithi063 ceph-mon[35189]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:12.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:12 smithi111 ceph-mon[44005]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:13.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:13 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:13.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:13 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:13.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:13 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:40:13.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:13 smithi063 ceph-mon[35189]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:13.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:13 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:13.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:13 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:13.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:13 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:40:13.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:13 smithi111 ceph-mon[44005]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:15.151 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:15.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:16.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:15 smithi063 ceph-mon[35189]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:16.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:15 smithi111 ceph-mon[44005]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:17.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.138+0000 7f3f9b400700 1 -- 172.21.15.63:0/1475259891 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3fa4115f10 con 0x7f3f88069210 2024-03-23T11:40:17.125 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 1 on host 'smithi063' 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f88069210 msgr2=0x7f3f8806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f88069210 0x7f3f8806b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3f90004f90 tx=0x7f3f90009e30 comp rx=0 tx=0).stop 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 msgr2=0x7f3fa4113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4113f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400b790 tx=0x7f3f94009220 comp rx=0 tx=0).stop 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 shutdown_connections 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f88069210 0x7f3f8806b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa40709a0 0x7f3fa4113f70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 --2- 172.21.15.63:0/1475259891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa4071f50 0x7f3fa41144b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.140+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 >> 172.21.15.63:0/1475259891 conn(0x7f3fa40f8560 msgr2=0x7f3fa40f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.141+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 shutdown_connections 2024-03-23T11:40:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:17.141+0000 7f3faa000700 1 -- 172.21.15.63:0/1475259891 wait complete. 2024-03-23T11:40:17.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:16 smithi063 ceph-mon[35189]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T11:40:17.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:16 smithi111 ceph-mon[44005]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T11:40:17.836 DEBUG:teuthology.orchestra.run.smithi063:osd.1> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.1.service 2024-03-23T11:40:17.840 INFO:tasks.cephadm:Deploying osd.2 on smithi063 with /dev/vg_nvme/lv_2... 2024-03-23T11:40:17.840 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-23T11:40:17.913 INFO:journalctl@ceph.osd.1.smithi063.stdout:-- Logs begin at Sat 2024-03-23 11:23:13 UTC. -- 2024-03-23T11:40:18.135 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[68510]: 2024-03-23T11:40:17.928+0000 7fde4ae00700 -1 osd.1 0 waiting for initial osdmap 2024-03-23T11:40:18.136 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[68510]: 2024-03-23T11:40:17.934+0000 7fde46800700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:40:18.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-mon[35189]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:18.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-mon[35189]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-23T11:40:18.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-mon[35189]: osdmap e12: 2 total, 1 up, 2 in 2024-03-23T11:40:18.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:18.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-mon[35189]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:40:18.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:18.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:17 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:18.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:17 smithi111 ceph-mon[44005]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:18.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:17 smithi111 ceph-mon[44005]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-23T11:40:18.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:17 smithi111 ceph-mon[44005]: osdmap e12: 2 total, 1 up, 2 in 2024-03-23T11:40:18.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:17 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:18.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:17 smithi111 ceph-mon[44005]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:40:18.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:17 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:18.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:17 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:18.676 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:40:19.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:18 smithi063 ceph-mon[35189]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-23T11:40:19.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:18 smithi063 ceph-mon[35189]: osdmap e13: 2 total, 1 up, 2 in 2024-03-23T11:40:19.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:18 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:19.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:18 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:19.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:18 smithi111 ceph-mon[44005]: from='osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-23T11:40:19.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:18 smithi111 ceph-mon[44005]: osdmap e13: 2 total, 1 up, 2 in 2024-03-23T11:40:19.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:18 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:19.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:18 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:20.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:19 smithi063 ceph-mon[35189]: purged_snaps scrub starts 2024-03-23T11:40:20.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:19 smithi063 ceph-mon[35189]: purged_snaps scrub ok 2024-03-23T11:40:20.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:19 smithi063 ceph-mon[35189]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:20.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:19 smithi063 ceph-mon[35189]: osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] boot 2024-03-23T11:40:20.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:19 smithi063 ceph-mon[35189]: osdmap e14: 2 total, 2 up, 2 in 2024-03-23T11:40:20.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:19 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:19 smithi111 ceph-mon[44005]: purged_snaps scrub starts 2024-03-23T11:40:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:19 smithi111 ceph-mon[44005]: purged_snaps scrub ok 2024-03-23T11:40:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:19 smithi111 ceph-mon[44005]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-23T11:40:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:19 smithi111 ceph-mon[44005]: osd.1 [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] boot 2024-03-23T11:40:20.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:19 smithi111 ceph-mon[44005]: osdmap e14: 2 total, 2 up, 2 in 2024-03-23T11:40:20.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:19 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:40:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:20 smithi063 ceph-mon[35189]: osdmap e15: 2 total, 2 up, 2 in 2024-03-23T11:40:21.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:20 smithi111 ceph-mon[44005]: osdmap e15: 2 total, 2 up, 2 in 2024-03-23T11:40:21.725 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-23T11:40:21.726 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-23T11:40:21.726 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-23T11:40:21.726 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-23T11:40:21.726 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.056493 s, 186 MB/s 2024-03-23T11:40:21.726 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-23T11:40:22.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:21 smithi063 ceph-mon[35189]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:22.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:21 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:22.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:21 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:21 smithi111 ceph-mon[44005]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:21 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:21 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:22.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_2 2024-03-23T11:40:23.849 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:40:24.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:23 smithi063 ceph-mon[35189]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:23 smithi111 ceph-mon[44005]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.937+0000 7f9ed4200700 1 -- 172.21.15.63:0/2069851585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 msgr2=0x7f9ecc1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.937+0000 7f9ed4200700 1 --2- 172.21.15.63:0/2069851585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc1023a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8004940 tx=0x7f9eb8009e30 comp rx=0 tx=0).stop 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.937+0000 7f9ed4200700 1 -- 172.21.15.63:0/2069851585 shutdown_connections 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.937+0000 7f9ed4200700 1 --2- 172.21.15.63:0/2069851585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc1023a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.937+0000 7f9ed4200700 1 --2- 172.21.15.63:0/2069851585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc066220 0x7f9ecc0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.937+0000 7f9ed4200700 1 -- 172.21.15.63:0/2069851585 >> 172.21.15.63:0/2069851585 conn(0x7f9ecc0f7430 msgr2=0x7f9ecc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.938+0000 7f9ed4200700 1 -- 172.21.15.63:0/2069851585 shutdown_connections 2024-03-23T11:40:25.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.938+0000 7f9ed4200700 1 -- 172.21.15.63:0/2069851585 wait complete. 2024-03-23T11:40:25.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.939+0000 7f9ed4200700 1 Processor -- start 2024-03-23T11:40:25.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.939+0000 7f9ed4200700 1 -- start start 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.939+0000 7f9ed4200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc066220 0x7f9ecc10d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.939+0000 7f9ed4200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc10de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.939+0000 7f9ed4200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc10e420 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.939+0000 7f9ed4200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc10ec70 con 0x7f9ecc066220 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed0e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc10de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed0e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc10de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43996/0 (socket says 172.21.15.63:43996) 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed0e00700 1 -- 172.21.15.63:0/4122696227 learned_addr learned my addr 172.21.15.63:0/4122696227 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:40:25.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed1800700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc066220 0x7f9ecc10d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:40:25.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed0e00700 1 -- 172.21.15.63:0/4122696227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc066220 msgr2=0x7f9ecc10d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:25.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed0e00700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc066220 0x7f9ecc10d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:25.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed0e00700 1 -- 172.21.15.63:0/4122696227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb8005040 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.940+0000 7f9ed0e00700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc10de20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800b0d0 tx=0x7f9eb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:40:25.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.941+0000 7f9ec9600700 1 -- 172.21.15.63:0/4122696227 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb800e070 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.941+0000 7f9ec9600700 1 -- 172.21.15.63:0/4122696227 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9eb801d460 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.941+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc10eef0 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.941+0000 7f9ec9600700 1 -- 172.21.15.63:0/4122696227 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb800b650 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.941+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc10f430 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.942+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc05f5f0 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.943+0000 7f9ec9600700 1 -- 172.21.15.63:0/4122696227 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f9eb8013a50 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.944+0000 7f9ec9600700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eb4069210 0x7f9eb406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:40:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.944+0000 7f9ec9600700 1 -- 172.21.15.63:0/4122696227 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f9eb8087450 con 0x7f9ecc0fffc0 2024-03-23T11:40:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.944+0000 7f9ed1800700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eb4069210 0x7f9eb406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:40:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.945+0000 7f9ed1800700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eb4069210 0x7f9eb406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0004600 tx=0x7f9ec000b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:40:25.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:25.949+0000 7f9ec9600700 1 -- 172.21.15.63:0/4122696227 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9eb8057f70 con 0x7f9ecc0fffc0 2024-03-23T11:40:26.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:26.166+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ecc10f760 con 0x7f9eb4069210 2024-03-23T11:40:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:25 smithi063 ceph-mon[35189]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:26.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:25 smithi111 ceph-mon[44005]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:27.284 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:26 smithi063 ceph-mon[35189]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:40:27.284 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:40:27.284 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:40:27.284 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:26 smithi111 ceph-mon[44005]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:40:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:40:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:40:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:28.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:28.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:40:28.362 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:28.362 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.362 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.362 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:40:28.362 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:40:28.363 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.363 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:28.363 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:40:28.363 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:28.363 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:40:29.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:28 smithi063 ceph-mon[35189]: Detected new or changed devices on smithi063 2024-03-23T11:40:29.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:28 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi063 to 6523M 2024-03-23T11:40:29.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:28 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi063 2024-03-23T11:40:29.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:28 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi063 to 6523M 2024-03-23T11:40:30.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:29 smithi063 ceph-mon[35189]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:30.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:29 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:30.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:29 smithi111 ceph-mon[44005]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:30.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:29 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:31.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:30 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2052767443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc4babe-dff9-462c-9af7-75e30b8095af"}]: dispatch 2024-03-23T11:40:31.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:30 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2052767443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edc4babe-dff9-462c-9af7-75e30b8095af"}]': finished 2024-03-23T11:40:31.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:30 smithi063 ceph-mon[35189]: osdmap e16: 3 total, 2 up, 3 in 2024-03-23T11:40:31.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:31.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:30 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2052767443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc4babe-dff9-462c-9af7-75e30b8095af"}]: dispatch 2024-03-23T11:40:31.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:30 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2052767443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edc4babe-dff9-462c-9af7-75e30b8095af"}]': finished 2024-03-23T11:40:31.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:30 smithi111 ceph-mon[44005]: osdmap e16: 3 total, 2 up, 3 in 2024-03-23T11:40:31.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:32.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:31 smithi063 ceph-mon[35189]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:32.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:31 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2396069685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:40:32.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:31 smithi111 ceph-mon[44005]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:32.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:31 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2396069685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:40:34.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:33 smithi063 ceph-mon[35189]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:34.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:34.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:34.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:34.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:40:34.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:33 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:34.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:33 smithi111 ceph-mon[44005]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:34.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:34.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:34.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:34.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:40:34.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:33 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:36.230 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:35 smithi063 ceph-mon[35189]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:36.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:35 smithi111 ceph-mon[44005]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:38.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:37 smithi111 ceph-mon[44005]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:38.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:37 smithi063 ceph-mon[35189]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:39.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:38 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T11:40:39.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:38 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:39.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:38 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T11:40:39.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:38 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:40.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:39 smithi063 ceph-mon[35189]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:40.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:39 smithi063 ceph-mon[35189]: Deploying daemon osd.2 on smithi063 2024-03-23T11:40:40.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:39 smithi111 ceph-mon[44005]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:40.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:39 smithi111 ceph-mon[44005]: Deploying daemon osd.2 on smithi063 2024-03-23T11:40:42.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:41 smithi063 ceph-mon[35189]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:42.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:41 smithi111 ceph-mon[44005]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:44.267 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:43 smithi063 ceph-mon[35189]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:44.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:43 smithi111 ceph-mon[44005]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:45.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:45.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:40:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:46 smithi063 ceph-mon[35189]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:46 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:46 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:46.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:46 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:40:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:46 smithi111 ceph-mon[44005]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:46 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:46 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:46 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:40:48.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:48 smithi063 ceph-mon[35189]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:48.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:48 smithi111 ceph-mon[44005]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:50 smithi063 ceph-mon[35189]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:50 smithi063 ceph-mon[35189]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T11:40:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:50 smithi111 ceph-mon[44005]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:50 smithi111 ceph-mon[44005]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T11:40:50.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.582+0000 7f9ec9600700 1 -- 172.21.15.63:0/4122696227 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9ecc10f760 con 0x7f9eb4069210 2024-03-23T11:40:50.569 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 2 on host 'smithi063' 2024-03-23T11:40:50.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eb4069210 msgr2=0x7f9eb406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:50.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eb4069210 0x7f9eb406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0004600 tx=0x7f9ec000b040 comp rx=0 tx=0).stop 2024-03-23T11:40:50.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 msgr2=0x7f9ecc10de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:50.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc10de20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800b0d0 tx=0x7f9eb8009e30 comp rx=0 tx=0).stop 2024-03-23T11:40:50.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 shutdown_connections 2024-03-23T11:40:50.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9eb4069210 0x7f9eb406b6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:50.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc066220 0x7f9ecc10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:50.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 --2- 172.21.15.63:0/4122696227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ecc0fffc0 0x7f9ecc10de20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:50.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 >> 172.21.15.63:0/4122696227 conn(0x7f9ecc0f7430 msgr2=0x7f9ecc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:40:50.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 shutdown_connections 2024-03-23T11:40:50.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:40:50.585+0000 7f9ed4200700 1 -- 172.21.15.63:0/4122696227 wait complete. 2024-03-23T11:40:51.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:51 smithi063 ceph-mon[35189]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-23T11:40:51.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:51 smithi063 ceph-mon[35189]: osdmap e17: 3 total, 2 up, 3 in 2024-03-23T11:40:51.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:51.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:51 smithi063 ceph-mon[35189]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:40:51.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:51.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:51.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:51 smithi111 ceph-mon[44005]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-23T11:40:51.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:51 smithi111 ceph-mon[44005]: osdmap e17: 3 total, 2 up, 3 in 2024-03-23T11:40:51.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:51.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:51 smithi111 ceph-mon[44005]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:40:51.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:51.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:51.712 DEBUG:teuthology.orchestra.run.smithi063:osd.2> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.2.service 2024-03-23T11:40:51.716 INFO:tasks.cephadm:Deploying osd.3 on smithi111 with /dev/vg_nvme/lv_4... 2024-03-23T11:40:51.716 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-23T11:40:51.787 INFO:journalctl@ceph.osd.2.smithi063.stdout:-- Logs begin at Sat 2024-03-23 11:23:13 UTC. -- 2024-03-23T11:40:52.134 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:40:52.183 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:52 smithi111 ceph-mon[44005]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:52.183 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:52 smithi111 ceph-mon[44005]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-23T11:40:52.183 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:52 smithi111 ceph-mon[44005]: osdmap e18: 3 total, 2 up, 3 in 2024-03-23T11:40:52.183 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:52 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:52.183 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:52 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:52.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:52 smithi063 ceph-mon[35189]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:52.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:52 smithi063 ceph-mon[35189]: from='osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-23T11:40:52.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:52 smithi063 ceph-mon[35189]: osdmap e18: 3 total, 2 up, 3 in 2024-03-23T11:40:52.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:52 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:52.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:52 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:53.607 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:53 smithi111 ceph-mon[44005]: purged_snaps scrub starts 2024-03-23T11:40:53.607 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:53 smithi111 ceph-mon[44005]: purged_snaps scrub ok 2024-03-23T11:40:53.607 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:53 smithi111 ceph-mon[44005]: osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] boot 2024-03-23T11:40:53.607 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:53 smithi111 ceph-mon[44005]: osdmap e19: 3 total, 3 up, 3 in 2024-03-23T11:40:53.607 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:53 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:53.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:53 smithi063 ceph-mon[35189]: purged_snaps scrub starts 2024-03-23T11:40:53.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:53 smithi063 ceph-mon[35189]: purged_snaps scrub ok 2024-03-23T11:40:53.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:53 smithi063 ceph-mon[35189]: osd.2 [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] boot 2024-03-23T11:40:53.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:53 smithi063 ceph-mon[35189]: osdmap e19: 3 total, 3 up, 3 in 2024-03-23T11:40:53.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:53 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:40:54.483 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-23T11:40:54.483 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-23T11:40:54.483 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-03-23T11:40:54.483 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-03-23T11:40:54.483 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0507091 s, 207 MB/s 2024-03-23T11:40:54.483 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-03-23T11:40:54.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:54 smithi111 ceph-mon[44005]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:54.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-23T11:40:54.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-23T11:40:54.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:54 smithi111 ceph-mon[44005]: osdmap e20: 3 total, 3 up, 3 in 2024-03-23T11:40:54.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:54 smithi063 ceph-mon[35189]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:54 smithi063 ceph-mon[35189]: osdmap e20: 3 total, 3 up, 3 in 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:54.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:55.154 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-03-23T11:40:55.563 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:40:55.793 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-23T11:40:55.793 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:55 smithi111 ceph-mon[44005]: osdmap e21: 3 total, 3 up, 3 in 2024-03-23T11:40:55.793 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:55 smithi111 ceph-mon[44005]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:40:55.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-23T11:40:55.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:55 smithi063 ceph-mon[35189]: osdmap e21: 3 total, 3 up, 3 in 2024-03-23T11:40:55.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:55 smithi063 ceph-mon[35189]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:40:56.385 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79625]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T11:40:56.385 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79625]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:40:56.385 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79625]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:40:56.386 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79625]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:40:56.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:56 smithi111 ceph-mon[44005]: osdmap e22: 3 total, 3 up, 3 in 2024-03-23T11:40:56.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:56 smithi063 ceph-mon[35189]: osdmap e22: 3 total, 3 up, 3 in 2024-03-23T11:40:56.885 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79639]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T11:40:56.885 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79639]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:40:56.885 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79639]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:40:56.886 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:40:56 smithi063 sudo[79639]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:40:57.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 -- 172.21.15.111:0/5158121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fce70 msgr2=0x7ff7380fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:57.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 --2- 172.21.15.111:0/5158121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fce70 0x7ff7380fd280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff728004880 tx=0x7ff728009e30 comp rx=0 tx=0).stop 2024-03-23T11:40:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 -- 172.21.15.111:0/5158121 shutdown_connections 2024-03-23T11:40:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 --2- 172.21.15.111:0/5158121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7380fdfe0 0x7ff7380fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 --2- 172.21.15.111:0/5158121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fce70 0x7ff7380fd280 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 -- 172.21.15.111:0/5158121 >> 172.21.15.111:0/5158121 conn(0x7ff7380f8550 msgr2=0x7ff7380fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:40:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 -- 172.21.15.111:0/5158121 shutdown_connections 2024-03-23T11:40:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.378+0000 7ff73d400700 1 -- 172.21.15.111:0/5158121 wait complete. 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.379+0000 7ff73d400700 1 Processor -- start 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.379+0000 7ff73d400700 1 -- start start 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.379+0000 7ff73d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7380fce70 0x7ff73810fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.379+0000 7ff73d400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fdfe0 0x7ff73810ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.379+0000 7ff73d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381105b0 con 0x7ff7380fce70 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.379+0000 7ff73d400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff738110720 con 0x7ff7380fdfe0 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff735e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fdfe0 0x7ff73810ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff736800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7380fce70 0x7ff73810fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff735e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fdfe0 0x7ff73810ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53784/0 (socket says 172.21.15.111:53784) 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff735e00700 1 -- 172.21.15.111:0/3005255923 learned_addr learned my addr 172.21.15.111:0/3005255923 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff735e00700 1 -- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7380fce70 msgr2=0x7ff73810fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:40:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff735e00700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7380fce70 0x7ff73810fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff735e00700 1 -- 172.21.15.111:0/3005255923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff728005040 con 0x7ff7380fdfe0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff735e00700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fdfe0 0x7ff73810ffb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff7240041d0 tx=0x7ff72400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.380+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff724007cc0 con 0x7ff7380fdfe0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.381+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff72400f040 con 0x7ff7380fdfe0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.381+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff738115120 con 0x7ff7380fdfe0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.381+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72400c990 con 0x7ff7380fdfe0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.381+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7381156e0 con 0x7ff7380fdfe0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.382+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff738101970 con 0x7ff7380fdfe0 2024-03-23T11:40:57.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.383+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7ff72400caf0 con 0x7ff7380fdfe0 2024-03-23T11:40:57.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.383+0000 7ff72ea00700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff71c069210 0x7ff71c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:40:57.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.383+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7ff7240861f0 con 0x7ff7380fdfe0 2024-03-23T11:40:57.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.384+0000 7ff736800700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff71c069210 0x7ff71c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:40:57.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.384+0000 7ff736800700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff71c069210 0x7ff71c06b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff728004fa0 tx=0x7ff728009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:40:57.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.389+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff72404e190 con 0x7ff7380fdfe0 2024-03-23T11:40:57.385 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79688]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T11:40:57.385 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79688]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:40:57.385 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79688]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:40:57.385 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79688]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:40:57.579 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.593+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff73802b410 con 0x7ff71c069210 2024-03-23T11:40:57.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:40:57.626+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff7240532a0 con 0x7ff7380fdfe0 2024-03-23T11:40:57.885 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T11:40:57.885 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:40:57.885 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:40:57.885 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:40:57 smithi063 sudo[79710]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:40:57.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:57.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:57 smithi063 ceph-mon[35189]: pgmap v91: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:40:57.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:40:57.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:40:57.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:57.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:40:57.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:57 smithi111 ceph-mon[44005]: pgmap v91: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:40:57.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:40:57.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:40:57.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:40:58.611 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[81677]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T11:40:58.611 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[81677]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:40:58.611 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[81677]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:40:58.611 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[81677]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:40:58.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:58 smithi111 ceph-mon[44005]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:40:58.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:40:58 smithi111 ceph-mon[44005]: mgrmap e19: smithi063.gehkcr(active, since 2m), standbys: smithi111.nnblsk 2024-03-23T11:40:58.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:58 smithi063 ceph-mon[35189]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:40:58.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:58 smithi063 ceph-mon[35189]: mgrmap e19: smithi063.gehkcr(active, since 2m), standbys: smithi111.nnblsk 2024-03-23T11:40:58.885 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[83508]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T11:40:58.885 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[83508]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:40:58.885 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[83508]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:40:58.885 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:40:58 smithi063 sudo[83508]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:40:59.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83520]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T11:40:59.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:40:59.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83520]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:40:59.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83520]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:41:00.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83523]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T11:41:00.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83523]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:41:00.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83523]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:41:00.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:40:59 smithi063 sudo[83523]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:41:00.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:00 smithi111 sudo[47999]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-23T11:41:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:00 smithi111 sudo[47999]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:41:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:00 smithi111 sudo[47999]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:41:00.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:00 smithi111 sudo[47999]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:41:01.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: pgmap v92: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:01.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T11:41:01.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T11:41:01.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:41:01.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:41:01.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:01.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:01.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:41:01.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:41:01.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 sudo[48002]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-23T11:41:01.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 sudo[48002]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-23T11:41:01.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 sudo[48002]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-23T11:41:01.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 sudo[48002]: pam_unix(sudo:session): session closed for user root 2024-03-23T11:41:01.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: pgmap v92: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:01.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T11:41:01.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T11:41:01.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:41:01.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:41:01.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:01.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:01.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:41:01.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:41:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: Detected new or changed devices on smithi063 2024-03-23T11:41:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T11:41:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T11:41:02.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi063 to 4348M 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/469868766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bc94d72-a814-447a-8112-9d50058a2759"}]: dispatch 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bc94d72-a814-447a-8112-9d50058a2759"}]: dispatch 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bc94d72-a814-447a-8112-9d50058a2759"}]': finished 2024-03-23T11:41:02.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: osdmap e23: 4 total, 3 up, 4 in 2024-03-23T11:41:02.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:02.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:02.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:02.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:02.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:02 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/3519718391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi063 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:02.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi063 to 4348M 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/469868766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bc94d72-a814-447a-8112-9d50058a2759"}]: dispatch 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bc94d72-a814-447a-8112-9d50058a2759"}]: dispatch 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bc94d72-a814-447a-8112-9d50058a2759"}]': finished 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: osdmap e23: 4 total, 3 up, 4 in 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:02.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:02.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:02.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:02.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:02 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/3519718391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:41:04.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:04 smithi063 ceph-mon[35189]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:04.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:04 smithi111 ceph-mon[44005]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:06.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:06 smithi063 ceph-mon[35189]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:06 smithi111 ceph-mon[44005]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:08.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:08 smithi111 ceph-mon[44005]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-23T11:41:08.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:08 smithi063 ceph-mon[35189]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-23T11:41:10.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:10 smithi111 ceph-mon[44005]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-23T11:41:10.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:10 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T11:41:10.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:10 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:10.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:10 smithi111 ceph-mon[44005]: Deploying daemon osd.3 on smithi111 2024-03-23T11:41:10.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:10 smithi063 ceph-mon[35189]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-23T11:41:10.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:10 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T11:41:10.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:10 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:10.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:10 smithi063 ceph-mon[35189]: Deploying daemon osd.3 on smithi111 2024-03-23T11:41:12.251 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:12 smithi111 ceph-mon[44005]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-23T11:41:12.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:12 smithi063 ceph-mon[35189]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-23T11:41:14.371 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:14 smithi111 ceph-mon[44005]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 68 KiB/s wr, 4 op/s 2024-03-23T11:41:14.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:14 smithi063 ceph-mon[35189]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 68 KiB/s wr, 4 op/s 2024-03-23T11:41:15.323 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:15.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:16.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:16 smithi111 ceph-mon[44005]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-23T11:41:16.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:16.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:16.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:16.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:16 smithi063 ceph-mon[35189]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-23T11:41:16.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:16.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:16.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:18.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:18 smithi063 ceph-mon[35189]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-23T11:41:18.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:18 smithi111 ceph-mon[44005]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-23T11:41:19.759 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 3 on host 'smithi111' 2024-03-23T11:41:19.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.771+0000 7ff72ea00700 1 -- 172.21.15.111:0/3005255923 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff73802b410 con 0x7ff71c069210 2024-03-23T11:41:19.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff71c069210 msgr2=0x7ff71c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:19.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff71c069210 0x7ff71c06b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff728004fa0 tx=0x7ff728009e30 comp rx=0 tx=0).stop 2024-03-23T11:41:19.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fdfe0 msgr2=0x7ff73810ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:19.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fdfe0 0x7ff73810ffb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff7240041d0 tx=0x7ff72400b040 comp rx=0 tx=0).stop 2024-03-23T11:41:19.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 shutdown_connections 2024-03-23T11:41:19.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff71c069210 0x7ff71c06b6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:19.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7380fce70 0x7ff73810fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:19.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 --2- 172.21.15.111:0/3005255923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7380fdfe0 0x7ff73810ffb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:19.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 >> 172.21.15.111:0/3005255923 conn(0x7ff7380f8550 msgr2=0x7ff7380f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:41:19.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 shutdown_connections 2024-03-23T11:41:19.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:19.774+0000 7ff73d400700 1 -- 172.21.15.111:0/3005255923 wait complete. 2024-03-23T11:41:20.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:20.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:20.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:20.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T11:41:20.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:20.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:20.521 DEBUG:teuthology.orchestra.run.smithi111:osd.3> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.3.service 2024-03-23T11:41:20.525 INFO:tasks.cephadm:Deploying osd.4 on smithi111 with /dev/vg_nvme/lv_3... 2024-03-23T11:41:20.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-23T11:41:20.604 INFO:journalctl@ceph.osd.3.smithi111.stdout:-- Logs begin at Sat 2024-03-23 11:22:58 UTC. -- 2024-03-23T11:41:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:21 smithi063 ceph-mon[35189]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-23T11:41:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:21 smithi063 ceph-mon[35189]: osdmap e24: 4 total, 3 up, 4 in 2024-03-23T11:41:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:21 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:21 smithi063 ceph-mon[35189]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:41:21.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:21 smithi111 ceph-mon[44005]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-23T11:41:21.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:21 smithi111 ceph-mon[44005]: osdmap e24: 4 total, 3 up, 4 in 2024-03-23T11:41:21.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:21 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:21.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:21 smithi111 ceph-mon[44005]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:41:21.403 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:41:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[50390]: 2024-03-23T11:41:21.098+0000 7fd189a00700 -1 osd.3 0 waiting for initial osdmap 2024-03-23T11:41:21.403 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:41:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[50390]: 2024-03-23T11:41:21.106+0000 7fd184000700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:41:21.714 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:41:22.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:22 smithi111 ceph-mon[44005]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:22.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:22 smithi111 ceph-mon[44005]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-23T11:41:22.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:22 smithi111 ceph-mon[44005]: osdmap e25: 4 total, 3 up, 4 in 2024-03-23T11:41:22.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:22 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:22.273 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:22 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:22.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:22 smithi063 ceph-mon[35189]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-23T11:41:22.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:22 smithi063 ceph-mon[35189]: from='osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-23T11:41:22.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:22 smithi063 ceph-mon[35189]: osdmap e25: 4 total, 3 up, 4 in 2024-03-23T11:41:22.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:22 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:22.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:22 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:23 smithi063 ceph-mon[35189]: purged_snaps scrub starts 2024-03-23T11:41:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:23 smithi063 ceph-mon[35189]: purged_snaps scrub ok 2024-03-23T11:41:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:23 smithi063 ceph-mon[35189]: osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] boot 2024-03-23T11:41:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:23 smithi063 ceph-mon[35189]: osdmap e26: 4 total, 4 up, 4 in 2024-03-23T11:41:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:23.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:23 smithi111 ceph-mon[44005]: purged_snaps scrub starts 2024-03-23T11:41:23.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:23 smithi111 ceph-mon[44005]: purged_snaps scrub ok 2024-03-23T11:41:23.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:23 smithi111 ceph-mon[44005]: osd.3 [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] boot 2024-03-23T11:41:23.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:23 smithi111 ceph-mon[44005]: osdmap e26: 4 total, 4 up, 4 in 2024-03-23T11:41:23.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:41:24.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:24 smithi063 ceph-mon[35189]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:24.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:24 smithi063 ceph-mon[35189]: osdmap e27: 4 total, 4 up, 4 in 2024-03-23T11:41:24.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:24 smithi063 ceph-mon[35189]: osdmap e28: 4 total, 4 up, 4 in 2024-03-23T11:41:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:24 smithi111 ceph-mon[44005]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:24 smithi111 ceph-mon[44005]: osdmap e27: 4 total, 4 up, 4 in 2024-03-23T11:41:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:24 smithi111 ceph-mon[44005]: osdmap e28: 4 total, 4 up, 4 in 2024-03-23T11:41:24.454 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-23T11:41:24.454 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-23T11:41:24.454 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-03-23T11:41:24.454 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-03-23T11:41:24.454 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0295328 s, 355 MB/s 2024-03-23T11:41:24.455 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-03-23T11:41:25.373 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-03-23T11:41:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi111 2024-03-23T11:41:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi111 to 16118M 2024-03-23T11:41:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:25.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:25.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:25 smithi111 ceph-mon[44005]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:25.829 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: Detected new or changed devices on smithi111 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi111 to 16118M 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:25.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:25 smithi063 ceph-mon[35189]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:26.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:26 smithi063 ceph-mon[35189]: osdmap e29: 4 total, 4 up, 4 in 2024-03-23T11:41:26.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:26 smithi111 ceph-mon[44005]: osdmap e29: 4 total, 4 up, 4 in 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 -- 172.21.15.111:0/3228770743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fcee0 msgr2=0x7f43340fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 --2- 172.21.15.111:0/3228770743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fcee0 0x7f43340fd310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4328004970 tx=0x7f4328009e30 comp rx=0 tx=0).stop 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 -- 172.21.15.111:0/3228770743 shutdown_connections 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 --2- 172.21.15.111:0/3228770743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43340fe0d0 0x7f43340fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 --2- 172.21.15.111:0/3228770743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fcee0 0x7f43340fd310 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 -- 172.21.15.111:0/3228770743 >> 172.21.15.111:0/3228770743 conn(0x7f43340f8560 msgr2=0x7f43340fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 -- 172.21.15.111:0/3228770743 shutdown_connections 2024-03-23T11:41:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.649+0000 7f433c000700 1 -- 172.21.15.111:0/3228770743 wait complete. 2024-03-23T11:41:27.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.650+0000 7f433c000700 1 Processor -- start 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.650+0000 7f433c000700 1 -- start start 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f433c000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43340fcee0 0x7f433410fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f433c000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fe0d0 0x7f4334110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f433c000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4334110660 con 0x7f43340fcee0 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f433c000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4334110eb0 con 0x7f43340fe0d0 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f4338c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fe0d0 0x7f4334110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f4338c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fe0d0 0x7f4334110060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35106/0 (socket says 172.21.15.111:35106) 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f4338c00700 1 -- 172.21.15.111:0/101363763 learned_addr learned my addr 172.21.15.111:0/101363763 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f4339600700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43340fcee0 0x7f433410fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f4338c00700 1 -- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43340fcee0 msgr2=0x7f433410fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f4338c00700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43340fcee0 0x7f433410fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:27.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.651+0000 7f4338c00700 1 -- 172.21.15.111:0/101363763 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4328005040 con 0x7f43340fe0d0 2024-03-23T11:41:27.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.652+0000 7f4338c00700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fe0d0 0x7f4334110060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4320002a50 tx=0x7f43200046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:41:27.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.652+0000 7f4331600700 1 -- 172.21.15.111:0/101363763 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4320007cc0 con 0x7f43340fe0d0 2024-03-23T11:41:27.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.652+0000 7f4331600700 1 -- 172.21.15.111:0/101363763 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f432000f040 con 0x7f43340fe0d0 2024-03-23T11:41:27.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.652+0000 7f4331600700 1 -- 172.21.15.111:0/101363763 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f432000d9c0 con 0x7f43340fe0d0 2024-03-23T11:41:27.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.652+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4334111190 con 0x7f43340fe0d0 2024-03-23T11:41:27.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.652+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43341116d0 con 0x7f43340fe0d0 2024-03-23T11:41:27.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.653+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4334101a60 con 0x7f43340fe0d0 2024-03-23T11:41:27.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.654+0000 7f4331600700 1 -- 172.21.15.111:0/101363763 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f432000db20 con 0x7f43340fe0d0 2024-03-23T11:41:27.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.655+0000 7f4331600700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f431c0692b0 0x7f431c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:41:27.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.655+0000 7f4331600700 1 -- 172.21.15.111:0/101363763 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f43200867e0 con 0x7f43340fe0d0 2024-03-23T11:41:27.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.655+0000 7f4339600700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f431c0692b0 0x7f431c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:41:27.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.656+0000 7f4339600700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f431c0692b0 0x7f431c06b760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4328009fd0 tx=0x7f4328009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:41:27.646 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.660+0000 7f4331600700 1 -- 172.21.15.111:0/101363763 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4320056ba0 con 0x7f43340fe0d0 2024-03-23T11:41:27.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:27 smithi111 ceph-mon[44005]: osdmap e30: 4 total, 4 up, 4 in 2024-03-23T11:41:27.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:27 smithi111 ceph-mon[44005]: pgmap v114: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-23T11:41:27.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:27.859+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4334111940 con 0x7f431c0692b0 2024-03-23T11:41:27.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:27 smithi063 ceph-mon[35189]: osdmap e30: 4 total, 4 up, 4 in 2024-03-23T11:41:27.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:27 smithi063 ceph-mon[35189]: pgmap v114: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-23T11:41:28.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:28 smithi063 ceph-mon[35189]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-23T11:41:28.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:28 smithi063 ceph-mon[35189]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:41:28.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:41:28.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:41:28.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:28.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:28 smithi111 ceph-mon[44005]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-23T11:41:28.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:28 smithi111 ceph-mon[44005]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:41:28.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:41:28.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:41:28.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:29.794 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:29 smithi111 ceph-mon[44005]: pgmap v115: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-23T11:41:29.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:29 smithi063 ceph-mon[35189]: pgmap v115: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-23T11:41:31.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:31.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:31.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:31.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:32.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-23T11:41:32.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-23T11:41:32.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: Cluster is now healthy 2024-03-23T11:41:32.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/2722950005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67"}]: dispatch 2024-03-23T11:41:32.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67"}]: dispatch 2024-03-23T11:41:32.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67"}]': finished 2024-03-23T11:41:32.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: osdmap e31: 5 total, 4 up, 5 in 2024-03-23T11:41:32.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:32.136 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:31 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/1002370618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: Cluster is now healthy 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/2722950005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67"}]: dispatch 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67"}]: dispatch 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67"}]': finished 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: osdmap e31: 5 total, 4 up, 5 in 2024-03-23T11:41:32.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:32.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:31 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/1002370618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:41:34.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:33 smithi063 ceph-mon[35189]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-23T11:41:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:33 smithi111 ceph-mon[44005]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-23T11:41:36.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:35 smithi111 ceph-mon[44005]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-23T11:41:36.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:35 smithi063 ceph-mon[35189]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-23T11:41:38.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:37 smithi111 ceph-mon[44005]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T11:41:38.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:37 smithi063 ceph-mon[35189]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T11:41:39.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:38 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T11:41:39.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:38 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:39.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:38 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T11:41:39.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:38 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:39.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:39 smithi111 ceph-mon[44005]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T11:41:39.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:39 smithi111 ceph-mon[44005]: Deploying daemon osd.4 on smithi111 2024-03-23T11:41:40.196 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:39 smithi063 ceph-mon[35189]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-23T11:41:40.196 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:39 smithi063 ceph-mon[35189]: Deploying daemon osd.4 on smithi111 2024-03-23T11:41:42.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:41 smithi111 ceph-mon[44005]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:42.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:41 smithi063 ceph-mon[35189]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:44.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:43 smithi111 ceph-mon[44005]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:44.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:43 smithi063 ceph-mon[35189]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:45 smithi111 ceph-mon[44005]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:45 smithi063 ceph-mon[35189]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:41:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:46.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:48.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:47 smithi063 ceph-mon[35189]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:48.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:47 smithi111 ceph-mon[44005]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:49.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.494+0000 7f4331600700 1 -- 172.21.15.111:0/101363763 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4334111940 con 0x7f431c0692b0 2024-03-23T11:41:49.480 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 4 on host 'smithi111' 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f431c0692b0 msgr2=0x7f431c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f431c0692b0 0x7f431c06b760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4328009fd0 tx=0x7f4328009e30 comp rx=0 tx=0).stop 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fe0d0 msgr2=0x7f4334110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fe0d0 0x7f4334110060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4320002a50 tx=0x7f43200046b0 comp rx=0 tx=0).stop 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 shutdown_connections 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f431c0692b0 0x7f431c06b760 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43340fcee0 0x7f433410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 --2- 172.21.15.111:0/101363763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43340fe0d0 0x7f4334110060 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 >> 172.21.15.111:0/101363763 conn(0x7f43340f8560 msgr2=0x7f43340f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:41:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 shutdown_connections 2024-03-23T11:41:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:49.497+0000 7f433c000700 1 -- 172.21.15.111:0/101363763 wait complete. 2024-03-23T11:41:50.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:49 smithi063 ceph-mon[35189]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:50.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:49 smithi063 ceph-mon[35189]: from='osd.4 [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:41:50.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:49 smithi063 ceph-mon[35189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:41:50.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:49 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:50.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:49 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:50.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:49 smithi111 ceph-mon[44005]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:50.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:49 smithi111 ceph-mon[44005]: from='osd.4 [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:41:50.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:49 smithi111 ceph-mon[44005]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:41:50.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:49 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:50.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:49 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:50.178 DEBUG:teuthology.orchestra.run.smithi111:osd.4> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.4.service 2024-03-23T11:41:50.183 INFO:tasks.cephadm:Deploying osd.5 on smithi111 with /dev/vg_nvme/lv_2... 2024-03-23T11:41:50.183 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-23T11:41:50.256 INFO:journalctl@ceph.osd.4.smithi111.stdout:-- Logs begin at Sat 2024-03-23 11:22:58 UTC. -- 2024-03-23T11:41:50.648 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:41:51.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:50 smithi063 ceph-mon[35189]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-23T11:41:51.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:50 smithi063 ceph-mon[35189]: osdmap e32: 5 total, 4 up, 5 in 2024-03-23T11:41:51.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:51.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:50 smithi063 ceph-mon[35189]: from='osd.4 [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:41:51.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:50 smithi063 ceph-mon[35189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:41:51.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:51.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:51.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-mon[44005]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-23T11:41:51.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-mon[44005]: osdmap e32: 5 total, 4 up, 5 in 2024-03-23T11:41:51.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:51.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-mon[44005]: from='osd.4 [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:41:51.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-mon[44005]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:41:51.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:51.156 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:51.156 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[58272]: 2024-03-23T11:41:50.887+0000 7f5c86600700 -1 osd.4 0 waiting for initial osdmap 2024-03-23T11:41:51.156 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:41:50 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[58272]: 2024-03-23T11:41:50.897+0000 7f5c82a00700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:41:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:51 smithi063 ceph-mon[35189]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:51 smithi063 ceph-mon[35189]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-23T11:41:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:51 smithi063 ceph-mon[35189]: osdmap e33: 5 total, 4 up, 5 in 2024-03-23T11:41:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:51 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:52.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:51 smithi111 ceph-mon[44005]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:52.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:51 smithi111 ceph-mon[44005]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-23T11:41:52.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:51 smithi111 ceph-mon[44005]: osdmap e33: 5 total, 4 up, 5 in 2024-03-23T11:41:52.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:52.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:51 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:53.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:52 smithi063 ceph-mon[35189]: purged_snaps scrub starts 2024-03-23T11:41:53.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:52 smithi063 ceph-mon[35189]: purged_snaps scrub ok 2024-03-23T11:41:53.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:52 smithi063 ceph-mon[35189]: osd.4 [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] boot 2024-03-23T11:41:53.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:52 smithi063 ceph-mon[35189]: osdmap e34: 5 total, 5 up, 5 in 2024-03-23T11:41:53.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:52 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:53.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:52 smithi111 ceph-mon[44005]: purged_snaps scrub starts 2024-03-23T11:41:53.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:52 smithi111 ceph-mon[44005]: purged_snaps scrub ok 2024-03-23T11:41:53.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:52 smithi111 ceph-mon[44005]: osd.4 [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] boot 2024-03-23T11:41:53.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:52 smithi111 ceph-mon[44005]: osdmap e34: 5 total, 5 up, 5 in 2024-03-23T11:41:53.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:52 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:41:53.689 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-23T11:41:53.689 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-23T11:41:53.689 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-03-23T11:41:53.689 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-03-23T11:41:53.689 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0351724 s, 298 MB/s 2024-03-23T11:41:53.690 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-03-23T11:41:54.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:53 smithi063 ceph-mon[35189]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:54.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:53 smithi063 ceph-mon[35189]: osdmap e35: 5 total, 5 up, 5 in 2024-03-23T11:41:54.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:53 smithi111 ceph-mon[44005]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-23T11:41:54.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:53 smithi111 ceph-mon[44005]: osdmap e35: 5 total, 5 up, 5 in 2024-03-23T11:41:54.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-03-23T11:41:55.217 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:41:56.122 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:55 smithi111 ceph-mon[44005]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:41:56.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:55 smithi063 ceph-mon[35189]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:41:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 -- 172.21.15.111:0/188549110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 msgr2=0x7f46480fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 --2- 172.21.15.111:0/188549110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f46480fd310 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4638004970 tx=0x7f4638009e30 comp rx=0 tx=0).stop 2024-03-23T11:41:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 -- 172.21.15.111:0/188549110 shutdown_connections 2024-03-23T11:41:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 --2- 172.21.15.111:0/188549110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46480fe0d0 0x7f46480fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 --2- 172.21.15.111:0/188549110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f46480fd310 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 -- 172.21.15.111:0/188549110 >> 172.21.15.111:0/188549110 conn(0x7f46480f8560 msgr2=0x7f46480fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:41:57.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 -- 172.21.15.111:0/188549110 shutdown_connections 2024-03-23T11:41:57.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.630+0000 7f464e400700 1 -- 172.21.15.111:0/188549110 wait complete. 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.631+0000 7f464e400700 1 Processor -- start 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.631+0000 7f464e400700 1 -- start start 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.631+0000 7f464e400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f464810fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.631+0000 7f464e400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46480fe0d0 0x7f4648110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.631+0000 7f464e400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4648110660 con 0x7f46480fe0d0 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.631+0000 7f464e400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4648110eb0 con 0x7f46480fcee0 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4647400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f464810fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4647400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f464810fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51500/0 (socket says 172.21.15.111:51500) 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4647400700 1 -- 172.21.15.111:0/404301543 learned_addr learned my addr 172.21.15.111:0/404301543 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:41:57.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4646a00700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46480fe0d0 0x7f4648110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:41:57.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4647400700 1 -- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46480fe0d0 msgr2=0x7f4648110060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:41:57.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4647400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46480fe0d0 0x7f4648110060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:41:57.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4647400700 1 -- 172.21.15.111:0/404301543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4638005040 con 0x7f46480fcee0 2024-03-23T11:41:57.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.632+0000 7f4647400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f464810fb20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4638000c00 tx=0x7f4638009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:41:57.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.633+0000 7f463f400700 1 -- 172.21.15.111:0/404301543 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f463800e040 con 0x7f46480fcee0 2024-03-23T11:41:57.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.633+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4648111130 con 0x7f46480fcee0 2024-03-23T11:41:57.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.633+0000 7f463f400700 1 -- 172.21.15.111:0/404301543 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4638004290 con 0x7f46480fcee0 2024-03-23T11:41:57.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.633+0000 7f463f400700 1 -- 172.21.15.111:0/404301543 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f463801c630 con 0x7f46480fcee0 2024-03-23T11:41:57.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.633+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4648111670 con 0x7f46480fcee0 2024-03-23T11:41:57.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.634+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4648101a60 con 0x7f46480fcee0 2024-03-23T11:41:57.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.635+0000 7f463f400700 1 -- 172.21.15.111:0/404301543 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f463800bc10 con 0x7f46480fcee0 2024-03-23T11:41:57.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.635+0000 7f463f400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f462c0692b0 0x7f462c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:41:57.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.636+0000 7f463f400700 1 -- 172.21.15.111:0/404301543 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f4638087fd0 con 0x7f46480fcee0 2024-03-23T11:41:57.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.636+0000 7f4646a00700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f462c0692b0 0x7f462c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:41:57.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.636+0000 7f4646a00700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f462c0692b0 0x7f462c06b760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4634004f40 tx=0x7f463400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:41:57.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.640+0000 7f463f400700 1 -- 172.21.15.111:0/404301543 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4638058330 con 0x7f46480fcee0 2024-03-23T11:41:57.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:41:57.842+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4648111890 con 0x7f462c0692b0 2024-03-23T11:41:58.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:41:58.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: Detected new or changed devices on smithi111 2024-03-23T11:41:58.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi111 to 8059M 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:41:58.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:58 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi111 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi111 to 8059M 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:41:58.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-23T11:41:58.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-23T11:41:58.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:58 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:59.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:59 smithi063 ceph-mon[35189]: from='client.24189 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:41:59.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:59.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:59.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:41:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:41:59.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:59 smithi111 ceph-mon[44005]: from='client.24189 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:41:59.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:41:59.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:41:59.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:41:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:00.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:00 smithi063 ceph-mon[35189]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:00.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:00.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:00 smithi111 ceph-mon[44005]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:00.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:01.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:01 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/2121794783' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2219a929-a842-4b41-ab46-096be6ef4d13"}]: dispatch 2024-03-23T11:42:01.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:01 smithi063 ceph-mon[35189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2219a929-a842-4b41-ab46-096be6ef4d13"}]: dispatch 2024-03-23T11:42:01.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:01 smithi063 ceph-mon[35189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2219a929-a842-4b41-ab46-096be6ef4d13"}]': finished 2024-03-23T11:42:01.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:01 smithi063 ceph-mon[35189]: osdmap e36: 6 total, 5 up, 6 in 2024-03-23T11:42:01.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:01.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:01 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/2121794783' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2219a929-a842-4b41-ab46-096be6ef4d13"}]: dispatch 2024-03-23T11:42:01.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:01 smithi111 ceph-mon[44005]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2219a929-a842-4b41-ab46-096be6ef4d13"}]: dispatch 2024-03-23T11:42:01.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:01 smithi111 ceph-mon[44005]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2219a929-a842-4b41-ab46-096be6ef4d13"}]': finished 2024-03-23T11:42:01.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:01 smithi111 ceph-mon[44005]: osdmap e36: 6 total, 5 up, 6 in 2024-03-23T11:42:01.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:02.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:02 smithi063 ceph-mon[35189]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:02.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:02 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/3265140942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:42:02.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:02 smithi111 ceph-mon[44005]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:02.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:02 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/3265140942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-23T11:42:04.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:04 smithi063 ceph-mon[35189]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:04.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:04 smithi111 ceph-mon[44005]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:06.515 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:06 smithi111 ceph-mon[44005]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:06.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:06 smithi063 ceph-mon[35189]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:08.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:08 smithi063 ceph-mon[35189]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:08.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:08 smithi111 ceph-mon[44005]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:09.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T11:42:09.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:42:09.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T11:42:09.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:42:10.630 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:10 smithi111 ceph-mon[44005]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:10.631 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:10 smithi111 ceph-mon[44005]: Deploying daemon osd.5 on smithi111 2024-03-23T11:42:10.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:10 smithi063 ceph-mon[35189]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:10.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:10 smithi063 ceph-mon[35189]: Deploying daemon osd.5 on smithi111 2024-03-23T11:42:12.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:12 smithi111 ceph-mon[44005]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:12.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:12 smithi063 ceph-mon[35189]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:14.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:14 smithi111 ceph-mon[44005]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:14 smithi063 ceph-mon[35189]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:15.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:16.478 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:16 smithi111 ceph-mon[44005]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:16.478 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:16.478 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:16.478 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:16 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:42:16.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:16 smithi063 ceph-mon[35189]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:16.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:16.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:16.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:16 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:42:18.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:18 smithi111 ceph-mon[44005]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:18.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:18 smithi063 ceph-mon[35189]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:19.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.730+0000 7f463f400700 1 -- 172.21.15.111:0/404301543 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4648111890 con 0x7f462c0692b0 2024-03-23T11:42:19.717 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 5 on host 'smithi111' 2024-03-23T11:42:19.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f462c0692b0 msgr2=0x7f462c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:19.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f462c0692b0 0x7f462c06b760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4634004f40 tx=0x7f463400b040 comp rx=0 tx=0).stop 2024-03-23T11:42:19.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 msgr2=0x7f464810fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:19.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f464810fb20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4638000c00 tx=0x7f4638009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:19.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 shutdown_connections 2024-03-23T11:42:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f462c0692b0 0x7f462c06b760 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46480fcee0 0x7f464810fb20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 --2- 172.21.15.111:0/404301543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46480fe0d0 0x7f4648110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 >> 172.21.15.111:0/404301543 conn(0x7f46480f8560 msgr2=0x7f46480f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.732+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 shutdown_connections 2024-03-23T11:42:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:19.733+0000 7f464e400700 1 -- 172.21.15.111:0/404301543 wait complete. 2024-03-23T11:42:20.334 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:20 smithi111 ceph-mon[44005]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:20.335 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:20 smithi111 ceph-mon[44005]: from='osd.5 [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:42:20.335 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:20 smithi111 ceph-mon[44005]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:42:20.335 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.335 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.335 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.335 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:20 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.374 DEBUG:teuthology.orchestra.run.smithi111:osd.5> sudo journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.5.service 2024-03-23T11:42:20.378 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-23T11:42:20.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-23T11:42:20.447 INFO:journalctl@ceph.osd.5.smithi111.stdout:-- Logs begin at Sat 2024-03-23 11:22:58 UTC. -- 2024-03-23T11:42:20.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:20 smithi063 ceph-mon[35189]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:20.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:20 smithi063 ceph-mon[35189]: from='osd.5 [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:42:20.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:20 smithi063 ceph-mon[35189]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:42:20.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:20 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:20.820 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:21 smithi063 ceph-mon[35189]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-23T11:42:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:21 smithi063 ceph-mon[35189]: osdmap e37: 6 total, 5 up, 6 in 2024-03-23T11:42:21.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:21 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:21.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:21 smithi063 ceph-mon[35189]: from='osd.5 [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:42:21.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:21 smithi063 ceph-mon[35189]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:42:21.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:21 smithi111 ceph-mon[44005]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-23T11:42:21.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:21 smithi111 ceph-mon[44005]: osdmap e37: 6 total, 5 up, 6 in 2024-03-23T11:42:21.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:21 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:21.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:21 smithi111 ceph-mon[44005]: from='osd.5 [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:42:21.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:21 smithi111 ceph-mon[44005]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:42:21.656 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:42:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[66502]: 2024-03-23T11:42:21.270+0000 7f1c38e00700 -1 osd.5 0 waiting for initial osdmap 2024-03-23T11:42:21.656 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:42:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[66502]: 2024-03-23T11:42:21.280+0000 7f1c32000700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:42:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:22 smithi111 ceph-mon[44005]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:22 smithi111 ceph-mon[44005]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-23T11:42:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:22 smithi111 ceph-mon[44005]: osdmap e38: 6 total, 5 up, 6 in 2024-03-23T11:42:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:22 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:22.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:22 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:22.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:22 smithi063 ceph-mon[35189]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-23T11:42:22.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:22 smithi063 ceph-mon[35189]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-23T11:42:22.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:22 smithi063 ceph-mon[35189]: osdmap e38: 6 total, 5 up, 6 in 2024-03-23T11:42:22.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:22 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:22.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:22 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:22.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.860+0000 7f669f200700 1 -- 172.21.15.63:0/1035023234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fce80 msgr2=0x7f66980fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:22.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.860+0000 7f669f200700 1 --2- 172.21.15.63:0/1035023234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fce80 0x7f66980fd290 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6688004940 tx=0x7f6688009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:22.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.861+0000 7f669f200700 1 -- 172.21.15.63:0/1035023234 shutdown_connections 2024-03-23T11:42:22.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.861+0000 7f669f200700 1 --2- 172.21.15.63:0/1035023234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66980fdff0 0x7f66980fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:22.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.861+0000 7f669f200700 1 --2- 172.21.15.63:0/1035023234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fce80 0x7f66980fd290 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:22.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.861+0000 7f669f200700 1 -- 172.21.15.63:0/1035023234 >> 172.21.15.63:0/1035023234 conn(0x7f66980f8540 msgr2=0x7f66980fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:22.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.861+0000 7f669f200700 1 -- 172.21.15.63:0/1035023234 shutdown_connections 2024-03-23T11:42:22.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.861+0000 7f669f200700 1 -- 172.21.15.63:0/1035023234 wait complete. 2024-03-23T11:42:22.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f669f200700 1 Processor -- start 2024-03-23T11:42:22.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f669f200700 1 -- start start 2024-03-23T11:42:22.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f669f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66980fce80 0x7f669810fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:22.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f669f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fdff0 0x7f6698110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:22.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f669f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6698110610 con 0x7f66980fdff0 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f669f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6698110e60 con 0x7f66980fce80 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f6697400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fdff0 0x7f6698110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f6697400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fdff0 0x7f6698110010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52668/0 (socket says 172.21.15.63:52668) 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f6697400700 1 -- 172.21.15.63:0/1520239761 learned_addr learned my addr 172.21.15.63:0/1520239761 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.862+0000 7f6697e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66980fce80 0x7f669810fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f6697400700 1 -- 172.21.15.63:0/1520239761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66980fce80 msgr2=0x7f669810fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f6697400700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66980fce80 0x7f669810fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f6697400700 1 -- 172.21.15.63:0/1520239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688005040 con 0x7f66980fdff0 2024-03-23T11:42:22.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f6697400700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fdff0 0x7f6698110010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f668400eb70 tx=0x7f668400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:22.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f668fe00700 1 -- 172.21.15.63:0/1520239761 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6684007cc0 con 0x7f66980fdff0 2024-03-23T11:42:22.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f668fe00700 1 -- 172.21.15.63:0/1520239761 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f668400f040 con 0x7f66980fdff0 2024-03-23T11:42:22.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66981110e0 con 0x7f66980fdff0 2024-03-23T11:42:22.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f668fe00700 1 -- 172.21.15.63:0/1520239761 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668400c9f0 con 0x7f66980fdff0 2024-03-23T11:42:22.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.863+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6698111620 con 0x7f66980fdff0 2024-03-23T11:42:22.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.865+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6698101980 con 0x7f66980fdff0 2024-03-23T11:42:22.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.866+0000 7f668fe00700 1 -- 172.21.15.63:0/1520239761 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f668400cb50 con 0x7f66980fdff0 2024-03-23T11:42:22.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.866+0000 7f668fe00700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f667c0692b0 0x7f667c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:22.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.866+0000 7f668fe00700 1 -- 172.21.15.63:0/1520239761 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6684086a00 con 0x7f66980fdff0 2024-03-23T11:42:22.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.867+0000 7f6697e00700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f667c0692b0 0x7f667c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:22.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.867+0000 7f6697e00700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f667c0692b0 0x7f667c06b760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6688009fd0 tx=0x7f6688009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:22.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:22.872+0000 7f668fe00700 1 -- 172.21.15.63:0/1520239761 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6684056a80 con 0x7f66980fdff0 2024-03-23T11:42:23.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.081+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f669805f5f0 con 0x7f66980fdff0 2024-03-23T11:42:23.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.081+0000 7f668fe00700 1 -- 172.21.15.63:0/1520239761 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f6684056610 con 0x7f66980fdff0 2024-03-23T11:42:23.068 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:42:23.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.083+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f667c0692b0 msgr2=0x7f667c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:23.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.083+0000 7f669f200700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f667c0692b0 0x7f667c06b760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6688009fd0 tx=0x7f6688009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:23.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.083+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fdff0 msgr2=0x7f6698110010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:23.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.083+0000 7f669f200700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fdff0 0x7f6698110010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f668400eb70 tx=0x7f668400b040 comp rx=0 tx=0).stop 2024-03-23T11:42:23.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.084+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 shutdown_connections 2024-03-23T11:42:23.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.084+0000 7f669f200700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f667c0692b0 0x7f667c06b760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:23.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.084+0000 7f669f200700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66980fce80 0x7f669810fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:23.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.084+0000 7f669f200700 1 --2- 172.21.15.63:0/1520239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66980fdff0 0x7f6698110010 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:23.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.084+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 >> 172.21.15.63:0/1520239761 conn(0x7f66980f8540 msgr2=0x7f66980f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:23.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.084+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 shutdown_connections 2024-03-23T11:42:23.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:23.084+0000 7f669f200700 1 -- 172.21.15.63:0/1520239761 wait complete. 2024-03-23T11:42:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:23 smithi063 ceph-mon[35189]: purged_snaps scrub starts 2024-03-23T11:42:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:23 smithi063 ceph-mon[35189]: purged_snaps scrub ok 2024-03-23T11:42:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:23 smithi063 ceph-mon[35189]: osd.5 [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] boot 2024-03-23T11:42:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:23 smithi063 ceph-mon[35189]: osdmap e39: 6 total, 6 up, 6 in 2024-03-23T11:42:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:23.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:23 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1520239761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-23T11:42:23.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:23 smithi111 ceph-mon[44005]: purged_snaps scrub starts 2024-03-23T11:42:23.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:23 smithi111 ceph-mon[44005]: purged_snaps scrub ok 2024-03-23T11:42:23.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:23 smithi111 ceph-mon[44005]: osd.5 [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] boot 2024-03-23T11:42:23.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:23 smithi111 ceph-mon[44005]: osdmap e39: 6 total, 6 up, 6 in 2024-03-23T11:42:23.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:42:23.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:23 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1520239761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-23T11:42:23.768 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1711194142,"num_in_osds":6,"osd_in_since":1711194120,"num_remapped_pgs":0} 2024-03-23T11:42:23.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-23T11:42:24.203 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:24.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: osdmap e40: 6 total, 6 up, 6 in 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:42:24.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:24.511 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:24.511 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: osdmap e40: 6 total, 6 up, 6 in 2024-03-23T11:42:24.511 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:24.511 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:24.511 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:42:24.511 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:42:24.512 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:42:24.512 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:24.512 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:42:24.512 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:42:24.512 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:25.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:25 smithi063 ceph-mon[35189]: Detected new or changed devices on smithi111 2024-03-23T11:42:25.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:25 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi111 to 5372M 2024-03-23T11:42:25.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:42:25.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:42:25.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:42:25.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:25.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:25 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi111 2024-03-23T11:42:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:25 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi111 to 5372M 2024-03-23T11:42:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:42:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:42:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:42:25.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:42:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 -- 172.21.15.63:0/2608471205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 msgr2=0x7fd5d41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 --2- 172.21.15.63:0/2608471205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d41031d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8004940 tx=0x7fd5c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 -- 172.21.15.63:0/2608471205 shutdown_connections 2024-03-23T11:42:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 --2- 172.21.15.63:0/2608471205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5d41037a0 0x7fd5d40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 --2- 172.21.15.63:0/2608471205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d41031d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:26.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 -- 172.21.15.63:0/2608471205 >> 172.21.15.63:0/2608471205 conn(0x7fd5d40f89b0 msgr2=0x7fd5d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:26.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 -- 172.21.15.63:0/2608471205 shutdown_connections 2024-03-23T11:42:26.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.239+0000 7fd5dba00700 1 -- 172.21.15.63:0/2608471205 wait complete. 2024-03-23T11:42:26.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.240+0000 7fd5dba00700 1 Processor -- start 2024-03-23T11:42:26.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.240+0000 7fd5dba00700 1 -- start start 2024-03-23T11:42:26.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.240+0000 7fd5dba00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:26.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.240+0000 7fd5dba00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5d41037a0 0x7fd5d4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:26.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.240+0000 7fd5dba00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4070d90 con 0x7fd5d4102e00 2024-03-23T11:42:26.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.240+0000 7fd5dba00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4070ed0 con 0x7fd5d41037a0 2024-03-23T11:42:26.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52698/0 (socket says 172.21.15.63:52698) 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d9000700 1 -- 172.21.15.63:0/3676528647 learned_addr learned my addr 172.21.15.63:0/3676528647 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d3e00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5d41037a0 0x7fd5d4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d9000700 1 -- 172.21.15.63:0/3676528647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5d41037a0 msgr2=0x7fd5d4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d9000700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5d41037a0 0x7fd5d4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d9000700 1 -- 172.21.15.63:0/3676528647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c8005040 con 0x7fd5d4102e00 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d9000700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d40702c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009390 tx=0x7fd5c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d0c00700 1 -- 172.21.15.63:0/3676528647 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c800e040 con 0x7fd5d4102e00 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d0c00700 1 -- 172.21.15.63:0/3676528647 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5c8013420 con 0x7fd5d4102e00 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d4071120 con 0x7fd5d4102e00 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.241+0000 7fd5d0c00700 1 -- 172.21.15.63:0/3676528647 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c801c630 con 0x7fd5d4102e00 2024-03-23T11:42:26.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.242+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d411a170 con 0x7fd5d4102e00 2024-03-23T11:42:26.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.243+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d40fe550 con 0x7fd5d4102e00 2024-03-23T11:42:26.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.243+0000 7fd5d0c00700 1 -- 172.21.15.63:0/3676528647 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd5c80139e0 con 0x7fd5d4102e00 2024-03-23T11:42:26.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.244+0000 7fd5d0c00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd5bc0692b0 0x7fd5bc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:26.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.244+0000 7fd5d0c00700 1 -- 172.21.15.63:0/3676528647 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd5c8087ff0 con 0x7fd5d4102e00 2024-03-23T11:42:26.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.245+0000 7fd5d3e00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd5bc0692b0 0x7fd5bc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:26.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.245+0000 7fd5d3e00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd5bc0692b0 0x7fd5bc06b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009fd0 tx=0x7fd5c0009f60 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:26.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.249+0000 7fd5d0c00700 1 -- 172.21.15.63:0/3676528647 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd5c8054680 con 0x7fd5d4102e00 2024-03-23T11:42:26.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.458+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5d405f5f0 con 0x7fd5d4102e00 2024-03-23T11:42:26.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.459+0000 7fd5d0c00700 1 -- 172.21.15.63:0/3676528647 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11071 (secure 0 0 0) 0x7fd5c8057ca0 con 0x7fd5d4102e00 2024-03-23T11:42:26.444 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:42:26.444 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":40,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","created":"2024-03-23T11:35:55.761926+0000","modified":"2024-03-23T11:42:23.269491+0000","last_up_change":"2024-03-23T11:42:22.261037+0000","last_in_change":"2024-03-23T11:42:00.992881+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:40:53.432498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1e4f29eb-375c-4f37-a8d8-f82b40650ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6803","nonce":1473323264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6805","nonce":1473323264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6809","nonce":1473323264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6807","nonce":1473323264}]},"public_addr":"172.21.15.63:6803/1473323264","cluster_addr":"172.21.15.63:6805/1473323264","heartbeat_back_addr":"172.21.15.63:6809/1473323264","heartbeat_front_addr":"172.21.15.63:6807/1473323264","state":["exists","up"]},{"osd":1,"uuid":"5e3ddf8b-e2c3-4188-b495-ac2743805f33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6811","nonce":823348330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6813","nonce":823348330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6817","nonce":823348330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6815","nonce":823348330}]},"public_addr":"172.21.15.63:6811/823348330","cluster_addr":"172.21.15.63:6813/823348330","heartbeat_back_addr":"172.21.15.63:6817/823348330","heartbeat_front_addr":"172.21.15.63:6815/823348330","state":["exists","up"]},{"osd":2,"uuid":"edc4babe-dff9-462c-9af7-75e30b8095af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6819","nonce":219143073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6821","nonce":219143073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6825","nonce":219143073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6823","nonce":219143073}]},"public_addr":"172.21.15.63:6819/219143073","cluster_addr":"172.21.15.63:6821/219143073","heartbeat_back_addr":"172.21.15.63:6825/219143073","heartbeat_front_addr":"172.21.15.63:6823/219143073","state":["exists","up"]},{"osd":3,"uuid":"7bc94d72-a814-447a-8112-9d50058a2759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6801","nonce":1017350834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6803","nonce":1017350834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6807","nonce":1017350834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6805","nonce":1017350834}]},"public_addr":"172.21.15.111:6801/1017350834","cluster_addr":"172.21.15.111:6803/1017350834","heartbeat_back_addr":"172.21.15.111:6807/1017350834","heartbeat_front_addr":"172.21.15.111:6805/1017350834","state":["exists","up"]},{"osd":4,"uuid":"7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6809","nonce":1242047756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6811","nonce":1242047756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6815","nonce":1242047756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6813","nonce":1242047756}]},"public_addr":"172.21.15.111:6809/1242047756","cluster_addr":"172.21.15.111:6811/1242047756","heartbeat_back_addr":"172.21.15.111:6815/1242047756","heartbeat_front_addr":"172.21.15.111:6813/1242047756","state":["exists","up"]},{"osd":5,"uuid":"2219a929-a842-4b41-ab46-096be6ef4d13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6817","nonce":1318746257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6819","nonce":1318746257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6823","nonce":1318746257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6821","nonce":1318746257}]},"public_addr":"172.21.15.111:6817/1318746257","cluster_addr":"172.21.15.111:6819/1318746257","heartbeat_back_addr":"172.21.15.111:6823/1318746257","heartbeat_front_addr":"172.21.15.111:6821/1318746257","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:39:42.374999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:17.321830+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:50.840928+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:20.211360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:50.314996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:42:20.376924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/3148717714":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6801/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/2981669421":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/1183338694":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/3227457751":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6800/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3491519284":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6801/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/2286910220":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1379382252":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3192464709":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6801/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6800/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6800/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/3356174638":"2024-03-24T11:37:01.693515+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:42:26.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd5bc0692b0 msgr2=0x7fd5bc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd5bc0692b0 0x7fd5bc06b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009fd0 tx=0x7fd5c0009f60 comp rx=0 tx=0).stop 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 msgr2=0x7fd5d40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d40702c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009390 tx=0x7fd5c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 shutdown_connections 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd5bc0692b0 0x7fd5bc06b760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5d4102e00 0x7fd5d40702c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 --2- 172.21.15.63:0/3676528647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5d41037a0 0x7fd5d4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 >> 172.21.15.63:0/3676528647 conn(0x7fd5d40f89b0 msgr2=0x7fd5d40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 shutdown_connections 2024-03-23T11:42:26.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:26.461+0000 7fd5dba00700 1 -- 172.21.15.63:0/3676528647 wait complete. 2024-03-23T11:42:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:26 smithi063 ceph-mon[35189]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:26 smithi111 ceph-mon[44005]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:27.108 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:40:53.432498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-23T11:42:27.108 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-23T11:42:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:27 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3676528647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:42:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:27 smithi063 ceph-mon[35189]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:27.541 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:27.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:27 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3676528647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:42:27.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:27 smithi111 ceph-mon[44005]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 -- 172.21.15.63:0/4020073017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 msgr2=0x7f16b00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 --2- 172.21.15.63:0/4020073017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b00fce10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f169c004940 tx=0x7f169c009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 -- 172.21.15.63:0/4020073017 shutdown_connections 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 --2- 172.21.15.63:0/4020073017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b00fce10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 --2- 172.21.15.63:0/4020073017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b0102e00 0x7f16b01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 -- 172.21.15.63:0/4020073017 >> 172.21.15.63:0/4020073017 conn(0x7f16b00f89b0 msgr2=0x7f16b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 -- 172.21.15.63:0/4020073017 shutdown_connections 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.514+0000 7f16b7800700 1 -- 172.21.15.63:0/4020073017 wait complete. 2024-03-23T11:42:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.515+0000 7f16b7800700 1 Processor -- start 2024-03-23T11:42:29.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.515+0000 7f16b7800700 1 -- start start 2024-03-23T11:42:29.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16b7800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b0102e00 0x7f16b0115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:29.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16b7800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b0116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:29.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16b7800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b0116880 con 0x7f16b01037a0 2024-03-23T11:42:29.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16b7800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b010fc50 con 0x7f16b0102e00 2024-03-23T11:42:29.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16afe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b0116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:29.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16afe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b0116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52720/0 (socket says 172.21.15.63:52720) 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16afe00700 1 -- 172.21.15.63:0/2858784872 learned_addr learned my addr 172.21.15.63:0/2858784872 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16b4e00700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b0102e00 0x7f16b0115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.516+0000 7f16afe00700 1 -- 172.21.15.63:0/2858784872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b0102e00 msgr2=0x7f16b0115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16afe00700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b0102e00 0x7f16b0115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16afe00700 1 -- 172.21.15.63:0/2858784872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c005040 con 0x7f16b01037a0 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16afe00700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b0116230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f169c009390 tx=0x7f169c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16acc00700 1 -- 172.21.15.63:0/2858784872 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169c00e040 con 0x7f16b01037a0 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16acc00700 1 -- 172.21.15.63:0/2858784872 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f169c013420 con 0x7f16b01037a0 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b010fed0 con 0x7f16b01037a0 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16acc00700 1 -- 172.21.15.63:0/2858784872 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169c01c630 con 0x7f16b01037a0 2024-03-23T11:42:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.517+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b0110490 con 0x7f16b01037a0 2024-03-23T11:42:29.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.518+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b00fe550 con 0x7f16b01037a0 2024-03-23T11:42:29.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.519+0000 7f16acc00700 1 -- 172.21.15.63:0/2858784872 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f169c013a00 con 0x7f16b01037a0 2024-03-23T11:42:29.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.520+0000 7f16acc00700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f16980692b0 0x7f169806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:29.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.520+0000 7f16acc00700 1 -- 172.21.15.63:0/2858784872 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f169c087ff0 con 0x7f16b01037a0 2024-03-23T11:42:29.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.520+0000 7f16b4e00700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f16980692b0 0x7f169806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:29.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.521+0000 7f16b4e00700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f16980692b0 0x7f169806b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f16a4004f40 tx=0x7f16a400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:29.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.525+0000 7f16acc00700 1 -- 172.21.15.63:0/2858784872 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f169c0ba190 con 0x7f16b01037a0 2024-03-23T11:42:29.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.732+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f16b005f5f0 con 0x7f16b01037a0 2024-03-23T11:42:29.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.732+0000 7f16acc00700 1 -- 172.21.15.63:0/2858784872 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f169c058050 con 0x7f16b01037a0 2024-03-23T11:42:29.718 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-03-23T11:42:29.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f16980692b0 msgr2=0x7f169806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:29.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f16980692b0 0x7f169806b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f16a4004f40 tx=0x7f16a400b040 comp rx=0 tx=0).stop 2024-03-23T11:42:29.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 msgr2=0x7f16b0116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:29.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b0116230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f169c009390 tx=0x7f169c009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:29.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 shutdown_connections 2024-03-23T11:42:29.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f16980692b0 0x7f169806b760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:29.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b0102e00 0x7f16b0115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:29.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 --2- 172.21.15.63:0/2858784872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16b01037a0 0x7f16b0116230 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:29.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 >> 172.21.15.63:0/2858784872 conn(0x7f16b00f89b0 msgr2=0x7f16b00f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:29.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.734+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 shutdown_connections 2024-03-23T11:42:29.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:29.735+0000 7f16b7800700 1 -- 172.21.15.63:0/2858784872 wait complete. 2024-03-23T11:42:30.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:29 smithi063 ceph-mon[35189]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:30.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:29 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2858784872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T11:42:30.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:29 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:30.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:29 smithi111 ceph-mon[44005]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:30.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:29 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2858784872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T11:42:30.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:29 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:30.493 INFO:tasks.cephadm:Setting up client nodes... 2024-03-23T11:42:30.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-23T11:42:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:32.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:31 smithi063 ceph-mon[35189]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:32.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:31 smithi111 ceph-mon[44005]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:32.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 -- 172.21.15.63:0/2422040343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 msgr2=0x7f08180fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:32.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 --2- 172.21.15.63:0/2422040343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f08180fe4e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0800004940 tx=0x7f0800009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:32.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 -- 172.21.15.63:0/2422040343 shutdown_connections 2024-03-23T11:42:32.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 --2- 172.21.15.63:0/2422040343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f08180fe4e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:32.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 --2- 172.21.15.63:0/2422040343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08180fce70 0x7f08180fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:32.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 -- 172.21.15.63:0/2422040343 >> 172.21.15.63:0/2422040343 conn(0x7f08180f8540 msgr2=0x7f08180fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:32.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 -- 172.21.15.63:0/2422040343 shutdown_connections 2024-03-23T11:42:32.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.722+0000 7f081e200700 1 -- 172.21.15.63:0/2422040343 wait complete. 2024-03-23T11:42:32.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.723+0000 7f081e200700 1 Processor -- start 2024-03-23T11:42:32.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.723+0000 7f081e200700 1 -- start start 2024-03-23T11:42:32.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.723+0000 7f081e200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08180fce70 0x7f081810f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:32.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.723+0000 7f081e200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f081810ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:32.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.723+0000 7f081e200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0818110520 con 0x7f08180fe070 2024-03-23T11:42:32.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.723+0000 7f081e200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0818110690 con 0x7f08180fce70 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f081810ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f081810ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35616/0 (socket says 172.21.15.63:35616) 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080fe00700 1 -- 172.21.15.63:0/4256723730 learned_addr learned my addr 172.21.15.63:0/4256723730 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f0817400700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08180fce70 0x7f081810f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080fe00700 1 -- 172.21.15.63:0/4256723730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08180fce70 msgr2=0x7f081810f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080fe00700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08180fce70 0x7f081810f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080fe00700 1 -- 172.21.15.63:0/4256723730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0800005040 con 0x7f08180fe070 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080fe00700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f081810ff20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f080000b3b0 tx=0x7f0800009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.724+0000 7f080f400700 1 -- 172.21.15.63:0/4256723730 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f080000e040 con 0x7f08180fe070 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.725+0000 7f080f400700 1 -- 172.21.15.63:0/4256723730 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0800004290 con 0x7f08180fe070 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.725+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0818115090 con 0x7f08180fe070 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.725+0000 7f080f400700 1 -- 172.21.15.63:0/4256723730 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f080001c3f0 con 0x7f08180fe070 2024-03-23T11:42:32.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.725+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0818115650 con 0x7f08180fe070 2024-03-23T11:42:32.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.726+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0818101a00 con 0x7f08180fe070 2024-03-23T11:42:32.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.727+0000 7f080f400700 1 -- 172.21.15.63:0/4256723730 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0800013420 con 0x7f08180fe070 2024-03-23T11:42:32.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.728+0000 7f080f400700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f07fc0692b0 0x7f07fc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:32.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.728+0000 7f080f400700 1 -- 172.21.15.63:0/4256723730 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f08000881e0 con 0x7f08180fe070 2024-03-23T11:42:32.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.728+0000 7f0817400700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f07fc0692b0 0x7f07fc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:32.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.728+0000 7f0817400700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f07fc0692b0 0x7f07fc06b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08180fdbe0 tx=0x7f0808007ea0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:32.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:32.732+0000 7f080f400700 1 -- 172.21.15.63:0/4256723730 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f08000582e0 con 0x7f08180fe070 2024-03-23T11:42:33.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.016+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f081804c710 con 0x7f08180fe070 2024-03-23T11:42:33.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.022+0000 7f080f400700 1 -- 172.21.15.63:0/4256723730 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0800057e70 con 0x7f08180fe070 2024-03-23T11:42:33.008 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-03-23T11:42:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQApwP5lX+AJARAAUvAw7ya++uCxWPg5bMtptA== 2024-03-23T11:42:33.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.024+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f07fc0692b0 msgr2=0x7f07fc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:33.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.024+0000 7f081e200700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f07fc0692b0 0x7f07fc06b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08180fdbe0 tx=0x7f0808007ea0 comp rx=0 tx=0).stop 2024-03-23T11:42:33.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.024+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 msgr2=0x7f081810ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:33.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.024+0000 7f081e200700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f081810ff20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f080000b3b0 tx=0x7f0800009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:33.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.025+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 shutdown_connections 2024-03-23T11:42:33.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.025+0000 7f081e200700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f07fc0692b0 0x7f07fc06b760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:33.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.025+0000 7f081e200700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08180fce70 0x7f081810f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:33.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.025+0000 7f081e200700 1 --2- 172.21.15.63:0/4256723730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08180fe070 0x7f081810ff20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:33.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.025+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 >> 172.21.15.63:0/4256723730 conn(0x7f08180f8540 msgr2=0x7f08180f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:33.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.025+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 shutdown_connections 2024-03-23T11:42:33.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:33.025+0000 7f081e200700 1 -- 172.21.15.63:0/4256723730 wait complete. 2024-03-23T11:42:33.856 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:42:33.856 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-23T11:42:33.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-23T11:42:33.954 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-23T11:42:34.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:33 smithi063 ceph-mon[35189]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:34.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:33 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4256723730' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T11:42:34.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:33 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4256723730' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-23T11:42:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:33 smithi111 ceph-mon[44005]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:33 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4256723730' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T11:42:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:33 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4256723730' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-23T11:42:34.383 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 -- 172.21.15.111:0/1827660103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22440708f0 msgr2=0x7f2244070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 --2- 172.21.15.111:0/1827660103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22440708f0 0x7f2244070d60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2230004970 tx=0x7f2230009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 -- 172.21.15.111:0/1827660103 shutdown_connections 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 --2- 172.21.15.111:0/1827660103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22440708f0 0x7f2244070d60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 --2- 172.21.15.111:0/1827660103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2244071ea0 0x7f22440703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 -- 172.21.15.111:0/1827660103 >> 172.21.15.111:0/1827660103 conn(0x7f22440f8540 msgr2=0x7f22440fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 -- 172.21.15.111:0/1827660103 shutdown_connections 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.166+0000 7f2248e00700 1 -- 172.21.15.111:0/1827660103 wait complete. 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.167+0000 7f2248e00700 1 Processor -- start 2024-03-23T11:42:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.167+0000 7f2248e00700 1 -- start start 2024-03-23T11:42:36.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2248e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22440708f0 0x7f224410fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:36.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2248e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2244071ea0 0x7f224410ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:36.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2248e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2244110640 con 0x7f22440708f0 2024-03-23T11:42:36.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2248e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2244114fe0 con 0x7f2244071ea0 2024-03-23T11:42:36.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:35 smithi111 ceph-mon[44005]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2241400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2244071ea0 0x7f224410ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2241e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22440708f0 0x7f224410fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2241400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2244071ea0 0x7f224410ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45976/0 (socket says 172.21.15.111:45976) 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2241400700 1 -- 172.21.15.111:0/1457419998 learned_addr learned my addr 172.21.15.111:0/1457419998 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2241400700 1 -- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22440708f0 msgr2=0x7f224410fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2241400700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22440708f0 0x7f224410fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.168+0000 7f2241400700 1 -- 172.21.15.111:0/1457419998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2230005040 con 0x7f2244071ea0 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.169+0000 7f2241400700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2244071ea0 0x7f224410ffb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2230009380 tx=0x7f2230009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.169+0000 7f223a000700 1 -- 172.21.15.111:0/1457419998 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223000e040 con 0x7f2244071ea0 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.169+0000 7f223a000700 1 -- 172.21.15.111:0/1457419998 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2230004290 con 0x7f2244071ea0 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.169+0000 7f223a000700 1 -- 172.21.15.111:0/1457419998 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223001c3f0 con 0x7f2244071ea0 2024-03-23T11:42:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.169+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22441151e0 con 0x7f2244071ea0 2024-03-23T11:42:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.169+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22441156d0 con 0x7f2244071ea0 2024-03-23T11:42:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.170+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224404c710 con 0x7f2244071ea0 2024-03-23T11:42:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.171+0000 7f223a000700 1 -- 172.21.15.111:0/1457419998 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f223000bc10 con 0x7f2244071ea0 2024-03-23T11:42:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.172+0000 7f223a000700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f22280692b0 0x7f222806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.173+0000 7f223a000700 1 -- 172.21.15.111:0/1457419998 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f22300881e0 con 0x7f2244071ea0 2024-03-23T11:42:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.173+0000 7f2241e00700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f22280692b0 0x7f222806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:36.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.173+0000 7f2241e00700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f22280692b0 0x7f222806b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2244071be0 tx=0x7f22340091a0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.177+0000 7f223a000700 1 -- 172.21.15.111:0/1457419998 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2230058370 con 0x7f2244071ea0 2024-03-23T11:42:36.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:35 smithi063 ceph-mon[35189]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:36.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.444+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f2244107a80 con 0x7f2244071ea0 2024-03-23T11:42:36.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.450+0000 7f223a000700 1 -- 172.21.15.111:0/1457419998 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2230057f00 con 0x7f2244071ea0 2024-03-23T11:42:36.437 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2024-03-23T11:42:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAswP5ldcupGhAAGQiNVEes5EY3SfeDlHv8lw== 2024-03-23T11:42:36.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f22280692b0 msgr2=0x7f222806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f22280692b0 0x7f222806b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2244071be0 tx=0x7f22340091a0 comp rx=0 tx=0).stop 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2244071ea0 msgr2=0x7f224410ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2244071ea0 0x7f224410ffb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2230009380 tx=0x7f2230009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 shutdown_connections 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f22280692b0 0x7f222806b760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22440708f0 0x7f224410fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 --2- 172.21.15.111:0/1457419998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2244071ea0 0x7f224410ffb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 >> 172.21.15.111:0/1457419998 conn(0x7f22440f8540 msgr2=0x7f22440f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 shutdown_connections 2024-03-23T11:42:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-23T11:42:36.453+0000 7f2248e00700 1 -- 172.21.15.111:0/1457419998 wait complete. 2024-03-23T11:42:37.064 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:36 smithi111 ceph-mon[44005]: from='client.? 172.21.15.111:0/1457419998' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T11:42:37.064 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:36 smithi111 ceph-mon[44005]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T11:42:37.064 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:36 smithi111 ceph-mon[44005]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-23T11:42:37.087 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:42:37.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-23T11:42:37.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-23T11:42:37.180 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-23T11:42:37.180 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-23T11:42:37.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-23T11:42:37.249 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:36 smithi063 ceph-mon[35189]: from='client.? 172.21.15.111:0/1457419998' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T11:42:37.249 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:36 smithi063 ceph-mon[35189]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-23T11:42:37.249 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:36 smithi063 ceph-mon[35189]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-23T11:42:37.621 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:38.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:37 smithi063 ceph-mon[35189]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:38.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:37 smithi111 ceph-mon[44005]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:39.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 -- 172.21.15.63:0/3544205666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fde90 msgr2=0x7f73c80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:39.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 --2- 172.21.15.63:0/3544205666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fde90 0x7f73c80fe260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f73bc004940 tx=0x7f73bc009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:39.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 -- 172.21.15.63:0/3544205666 shutdown_connections 2024-03-23T11:42:39.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 --2- 172.21.15.63:0/3544205666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c80fe830 0x7f73c8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:39.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 --2- 172.21.15.63:0/3544205666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fde90 0x7f73c80fe260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:39.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 -- 172.21.15.63:0/3544205666 >> 172.21.15.63:0/3544205666 conn(0x7f73c8072bc0 msgr2=0x7f73c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:39.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 -- 172.21.15.63:0/3544205666 shutdown_connections 2024-03-23T11:42:39.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.602+0000 7f73cfe00700 1 -- 172.21.15.63:0/3544205666 wait complete. 2024-03-23T11:42:39.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.603+0000 7f73cfe00700 1 Processor -- start 2024-03-23T11:42:39.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.603+0000 7f73cfe00700 1 -- start start 2024-03-23T11:42:39.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.603+0000 7f73cfe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c80fde90 0x7f73c8115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:39.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.603+0000 7f73cfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fe830 0x7f73c8116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:39.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.603+0000 7f73cfe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c81168b0 con 0x7f73c80fe830 2024-03-23T11:42:39.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.603+0000 7f73cfe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c810fc80 con 0x7f73c80fde90 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.603+0000 7f73cca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fe830 0x7f73c8116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fe830 0x7f73c8116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35640/0 (socket says 172.21.15.63:35640) 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cca00700 1 -- 172.21.15.63:0/345682673 learned_addr learned my addr 172.21.15.63:0/345682673 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cd400700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c80fde90 0x7f73c8115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cca00700 1 -- 172.21.15.63:0/345682673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c80fde90 msgr2=0x7f73c8115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cca00700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c80fde90 0x7f73c8115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cca00700 1 -- 172.21.15.63:0/345682673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc005040 con 0x7f73c80fe830 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cca00700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fe830 0x7f73c8116260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f73b400c8c0 tx=0x7f73b40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73c5600700 1 -- 172.21.15.63:0/345682673 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73b400a980 con 0x7f73c80fe830 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73c5600700 1 -- 172.21.15.63:0/345682673 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73b400ce60 con 0x7f73c80fe830 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73c5600700 1 -- 172.21.15.63:0/345682673 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73b40113a0 con 0x7f73c80fe830 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.604+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c810ff60 con 0x7f73c80fe830 2024-03-23T11:42:39.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.605+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c81104a0 con 0x7f73c80fe830 2024-03-23T11:42:39.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.606+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c8106a20 con 0x7f73c80fe830 2024-03-23T11:42:39.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.606+0000 7f73c5600700 1 -- 172.21.15.63:0/345682673 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f73b4012400 con 0x7f73c80fe830 2024-03-23T11:42:39.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.607+0000 7f73c5600700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f73b00692b0 0x7f73b006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:39.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.607+0000 7f73c5600700 1 -- 172.21.15.63:0/345682673 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f73b4086ef0 con 0x7f73c80fe830 2024-03-23T11:42:39.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.608+0000 7f73cd400700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f73b00692b0 0x7f73b006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:39.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.608+0000 7f73cd400700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f73b00692b0 0x7f73b006b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00b3b0 tx=0x7f73bc009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:39.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.612+0000 7f73c5600700 1 -- 172.21.15.63:0/345682673 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f73b4056fd0 con 0x7f73c80fe830 2024-03-23T11:42:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.873+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f73c8065e30 con 0x7f73c80fe830 2024-03-23T11:42:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.876+0000 7f73c5600700 1 -- 172.21.15.63:0/345682673 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167169 (secure 0 0 0) 0x7f73b4056b60 con 0x7f73c80fe830 2024-03-23T11:42:39.866 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:42:39.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.879+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f73b00692b0 msgr2=0x7f73b006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:39.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.879+0000 7f73cfe00700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f73b00692b0 0x7f73b006b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00b3b0 tx=0x7f73bc009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:39.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.879+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fe830 msgr2=0x7f73c8116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:39.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.879+0000 7f73cfe00700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fe830 0x7f73c8116260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f73b400c8c0 tx=0x7f73b40044c0 comp rx=0 tx=0).stop 2024-03-23T11:42:39.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.880+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 shutdown_connections 2024-03-23T11:42:39.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.880+0000 7f73cfe00700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f73b00692b0 0x7f73b006b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:39.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.880+0000 7f73cfe00700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c80fde90 0x7f73c8115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:39.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.880+0000 7f73cfe00700 1 --2- 172.21.15.63:0/345682673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73c80fe830 0x7f73c8116260 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:39.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.880+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 >> 172.21.15.63:0/345682673 conn(0x7f73c8072bc0 msgr2=0x7f73c80fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:39.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.880+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 shutdown_connections 2024-03-23T11:42:39.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:39.880+0000 7f73cfe00700 1 -- 172.21.15.63:0/345682673 wait complete. 2024-03-23T11:42:40.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:39 smithi111 ceph-mon[44005]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:40.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:39 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/345682673' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-23T11:42:40.198 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:39 smithi063 ceph-mon[35189]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:40.198 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:39 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/345682673' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-23T11:42:40.618 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":19,"active_gid":14204,"active_name":"smithi063.gehkcr","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2193815059},{"type":"v1","addr":"172.21.15.63:6801","nonce":2193815059}]},"active_addr":"172.21.15.63:6801/2193815059","active_change":"2024-03-23T11:38:14.792439+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14226,"name":"smithi111.nnblsk","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.63:8443/","prometheus":"http://172.21.15.63:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2483586841}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2975088208}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2603926600}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2812522808}]}]} 2024-03-23T11:42:40.623 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-23T11:42:40.623 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-23T11:42:40.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-23T11:42:41.057 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:42.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:41 smithi111 ceph-mon[44005]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:42.241 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:41 smithi063 ceph-mon[35189]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 -- 172.21.15.63:0/3437887635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fde90 msgr2=0x7fa4840fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 --2- 172.21.15.63:0/3437887635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fde90 0x7fa4840fe260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa474004940 tx=0x7fa474009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 -- 172.21.15.63:0/3437887635 shutdown_connections 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 --2- 172.21.15.63:0/3437887635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4840fe830 0x7fa484106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 --2- 172.21.15.63:0/3437887635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fde90 0x7fa4840fe260 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 -- 172.21.15.63:0/3437887635 >> 172.21.15.63:0/3437887635 conn(0x7fa484072bc0 msgr2=0x7fa484072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 -- 172.21.15.63:0/3437887635 shutdown_connections 2024-03-23T11:42:42.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.938+0000 7fa48a400700 1 -- 172.21.15.63:0/3437887635 wait complete. 2024-03-23T11:42:42.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.939+0000 7fa48a400700 1 Processor -- start 2024-03-23T11:42:42.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.939+0000 7fa48a400700 1 -- start start 2024-03-23T11:42:42.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.939+0000 7fa48a400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4840fde90 0x7fa484115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:42.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa48a400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fe830 0x7fa484116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:42.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa48a400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa484116890 con 0x7fa4840fe830 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa48a400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48410fc60 con 0x7fa4840fde90 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa482a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fe830 0x7fa484116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa482a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fe830 0x7fa484116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47868/0 (socket says 172.21.15.63:47868) 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa482a00700 1 -- 172.21.15.63:0/1820378570 learned_addr learned my addr 172.21.15.63:0/1820378570 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa483400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4840fde90 0x7fa484115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa482a00700 1 -- 172.21.15.63:0/1820378570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4840fde90 msgr2=0x7fa484115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa482a00700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4840fde90 0x7fa484115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.940+0000 7fa482a00700 1 -- 172.21.15.63:0/1820378570 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474005040 con 0x7fa4840fe830 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.941+0000 7fa482a00700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fe830 0x7fa484116240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa470002a50 tx=0x7fa4700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.941+0000 7fa47b400700 1 -- 172.21.15.63:0/1820378570 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa470007cc0 con 0x7fa4840fe830 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.941+0000 7fa47b400700 1 -- 172.21.15.63:0/1820378570 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa47000f040 con 0x7fa4840fe830 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.941+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48410ff40 con 0x7fa4840fe830 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.941+0000 7fa47b400700 1 -- 172.21.15.63:0/1820378570 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa47000d9c0 con 0x7fa4840fe830 2024-03-23T11:42:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.941+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa484110480 con 0x7fa4840fe830 2024-03-23T11:42:42.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.942+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa484106a20 con 0x7fa4840fe830 2024-03-23T11:42:42.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.943+0000 7fa47b400700 1 -- 172.21.15.63:0/1820378570 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa47000db20 con 0x7fa4840fe830 2024-03-23T11:42:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.944+0000 7fa47b400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fa4680692a0 0x7fa46806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.944+0000 7fa47b400700 1 -- 172.21.15.63:0/1820378570 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa470086b40 con 0x7fa4840fe830 2024-03-23T11:42:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.944+0000 7fa483400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fa4680692a0 0x7fa46806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.945+0000 7fa483400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fa4680692a0 0x7fa46806b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b7e0 tx=0x7fa474009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:42.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:42.949+0000 7fa47b400700 1 -- 172.21.15.63:0/1820378570 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa470056ba0 con 0x7fa4840fe830 2024-03-23T11:42:43.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.156+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa484065e30 con 0x7fa4840fe830 2024-03-23T11:42:43.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.156+0000 7fa47b400700 1 -- 172.21.15.63:0/1820378570 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11071 (secure 0 0 0) 0x7fa470056730 con 0x7fa4840fe830 2024-03-23T11:42:43.143 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:42:43.143 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":40,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","created":"2024-03-23T11:35:55.761926+0000","modified":"2024-03-23T11:42:23.269491+0000","last_up_change":"2024-03-23T11:42:22.261037+0000","last_in_change":"2024-03-23T11:42:00.992881+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:40:53.432498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1e4f29eb-375c-4f37-a8d8-f82b40650ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6803","nonce":1473323264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6805","nonce":1473323264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6809","nonce":1473323264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6807","nonce":1473323264}]},"public_addr":"172.21.15.63:6803/1473323264","cluster_addr":"172.21.15.63:6805/1473323264","heartbeat_back_addr":"172.21.15.63:6809/1473323264","heartbeat_front_addr":"172.21.15.63:6807/1473323264","state":["exists","up"]},{"osd":1,"uuid":"5e3ddf8b-e2c3-4188-b495-ac2743805f33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6811","nonce":823348330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6813","nonce":823348330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6817","nonce":823348330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6815","nonce":823348330}]},"public_addr":"172.21.15.63:6811/823348330","cluster_addr":"172.21.15.63:6813/823348330","heartbeat_back_addr":"172.21.15.63:6817/823348330","heartbeat_front_addr":"172.21.15.63:6815/823348330","state":["exists","up"]},{"osd":2,"uuid":"edc4babe-dff9-462c-9af7-75e30b8095af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6819","nonce":219143073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6821","nonce":219143073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6825","nonce":219143073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6823","nonce":219143073}]},"public_addr":"172.21.15.63:6819/219143073","cluster_addr":"172.21.15.63:6821/219143073","heartbeat_back_addr":"172.21.15.63:6825/219143073","heartbeat_front_addr":"172.21.15.63:6823/219143073","state":["exists","up"]},{"osd":3,"uuid":"7bc94d72-a814-447a-8112-9d50058a2759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6801","nonce":1017350834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6803","nonce":1017350834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6807","nonce":1017350834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6805","nonce":1017350834}]},"public_addr":"172.21.15.111:6801/1017350834","cluster_addr":"172.21.15.111:6803/1017350834","heartbeat_back_addr":"172.21.15.111:6807/1017350834","heartbeat_front_addr":"172.21.15.111:6805/1017350834","state":["exists","up"]},{"osd":4,"uuid":"7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6809","nonce":1242047756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6811","nonce":1242047756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6815","nonce":1242047756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6813","nonce":1242047756}]},"public_addr":"172.21.15.111:6809/1242047756","cluster_addr":"172.21.15.111:6811/1242047756","heartbeat_back_addr":"172.21.15.111:6815/1242047756","heartbeat_front_addr":"172.21.15.111:6813/1242047756","state":["exists","up"]},{"osd":5,"uuid":"2219a929-a842-4b41-ab46-096be6ef4d13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6817","nonce":1318746257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6819","nonce":1318746257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6823","nonce":1318746257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6821","nonce":1318746257}]},"public_addr":"172.21.15.111:6817/1318746257","cluster_addr":"172.21.15.111:6819/1318746257","heartbeat_back_addr":"172.21.15.111:6823/1318746257","heartbeat_front_addr":"172.21.15.111:6821/1318746257","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:39:42.374999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:17.321830+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:50.840928+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:20.211360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:50.314996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:42:20.376924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/3148717714":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6801/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/2981669421":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/1183338694":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/3227457751":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6800/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3491519284":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6801/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/2286910220":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1379382252":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3192464709":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6801/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6800/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6800/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/3356174638":"2024-03-24T11:37:01.693515+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:42:43.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fa4680692a0 msgr2=0x7fa46806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:43.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fa4680692a0 0x7fa46806b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b7e0 tx=0x7fa474009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:43.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fe830 msgr2=0x7fa484116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:43.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fe830 0x7fa484116240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa470002a50 tx=0x7fa4700046b0 comp rx=0 tx=0).stop 2024-03-23T11:42:43.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 shutdown_connections 2024-03-23T11:42:43.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fa4680692a0 0x7fa46806b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:43.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4840fde90 0x7fa484115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:43.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 --2- 172.21.15.63:0/1820378570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4840fe830 0x7fa484116240 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:43.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 >> 172.21.15.63:0/1820378570 conn(0x7fa484072bc0 msgr2=0x7fa4840fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:43.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 shutdown_connections 2024-03-23T11:42:43.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:43.159+0000 7fa48a400700 1 -- 172.21.15.63:0/1820378570 wait complete. 2024-03-23T11:42:43.831 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-23T11:42:43.831 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-23T11:42:44.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:43 smithi063 ceph-mon[35189]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:44.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:43 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1820378570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:42:44.262 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:44.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:43 smithi111 ceph-mon[44005]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:44.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:43 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1820378570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:42:45.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:45.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:44 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:42:46.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 -- 172.21.15.63:0/1254952377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 msgr2=0x7f79b8109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:46.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 --2- 172.21.15.63:0/1254952377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8109110 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f79a4004970 tx=0x7f79a4009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:46.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 -- 172.21.15.63:0/1254952377 shutdown_connections 2024-03-23T11:42:46.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 --2- 172.21.15.63:0/1254952377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8109110 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:46.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 --2- 172.21.15.63:0/1254952377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79b8070360 0x7f79b8070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:46.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 -- 172.21.15.63:0/1254952377 >> 172.21.15.63:0/1254952377 conn(0x7f79b80f89b0 msgr2=0x7f79b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:46.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 -- 172.21.15.63:0/1254952377 shutdown_connections 2024-03-23T11:42:46.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.118+0000 7f79c0600700 1 -- 172.21.15.63:0/1254952377 wait complete. 2024-03-23T11:42:46.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.119+0000 7f79c0600700 1 Processor -- start 2024-03-23T11:42:46.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.119+0000 7f79c0600700 1 -- start start 2024-03-23T11:42:46.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79c0600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79b8070360 0x7f79b8115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:46.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79c0600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79c0600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b8116930 con 0x7f79b8070c70 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79c0600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b810fc70 con 0x7f79b8070360 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bd200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bd200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47888/0 (socket says 172.21.15.63:47888) 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bd200700 1 -- 172.21.15.63:0/3497984308 learned_addr learned my addr 172.21.15.63:0/3497984308 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bdc00700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79b8070360 0x7f79b8115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bd200700 1 -- 172.21.15.63:0/3497984308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79b8070360 msgr2=0x7f79b8115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bd200700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79b8070360 0x7f79b8115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bd200700 1 -- 172.21.15.63:0/3497984308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a4005040 con 0x7f79b8070c70 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.120+0000 7f79bd200700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8116250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f79a40043d0 tx=0x7f79a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.121+0000 7f79b5600700 1 -- 172.21.15.63:0/3497984308 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79a400e040 con 0x7f79b8070c70 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.121+0000 7f79b5600700 1 -- 172.21.15.63:0/3497984308 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79a4013420 con 0x7f79b8070c70 2024-03-23T11:42:46.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.121+0000 7f79b5600700 1 -- 172.21.15.63:0/3497984308 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79a401c630 con 0x7f79b8070c70 2024-03-23T11:42:46.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.121+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b810fe90 con 0x7f79b8070c70 2024-03-23T11:42:46.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.121+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b81103d0 con 0x7f79b8070c70 2024-03-23T11:42:46.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.122+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8071aa0 con 0x7f79b8070c70 2024-03-23T11:42:46.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.123+0000 7f79b5600700 1 -- 172.21.15.63:0/3497984308 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f79a4013970 con 0x7f79b8070c70 2024-03-23T11:42:46.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.124+0000 7f79b5600700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f79a00692a0 0x7f79a006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:46.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.124+0000 7f79b5600700 1 -- 172.21.15.63:0/3497984308 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f79a4087ff0 con 0x7f79b8070c70 2024-03-23T11:42:46.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.124+0000 7f79bdc00700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f79a00692a0 0x7f79a006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:46.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.125+0000 7f79bdc00700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f79a00692a0 0x7f79a006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f79ac006fd0 tx=0x7f79ac008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:46.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.129+0000 7f79b5600700 1 -- 172.21.15.63:0/3497984308 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f79a40ba190 con 0x7f79b8070c70 2024-03-23T11:42:46.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:45 smithi063 ceph-mon[35189]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:46.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.337+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79b805f5f0 con 0x7f79b8070c70 2024-03-23T11:42:46.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.338+0000 7f79b5600700 1 -- 172.21.15.63:0/3497984308 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11071 (secure 0 0 0) 0x7f79a40580f0 con 0x7f79b8070c70 2024-03-23T11:42:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:42:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":40,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","created":"2024-03-23T11:35:55.761926+0000","modified":"2024-03-23T11:42:23.269491+0000","last_up_change":"2024-03-23T11:42:22.261037+0000","last_in_change":"2024-03-23T11:42:00.992881+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:40:53.432498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1e4f29eb-375c-4f37-a8d8-f82b40650ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6803","nonce":1473323264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6805","nonce":1473323264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6809","nonce":1473323264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6807","nonce":1473323264}]},"public_addr":"172.21.15.63:6803/1473323264","cluster_addr":"172.21.15.63:6805/1473323264","heartbeat_back_addr":"172.21.15.63:6809/1473323264","heartbeat_front_addr":"172.21.15.63:6807/1473323264","state":["exists","up"]},{"osd":1,"uuid":"5e3ddf8b-e2c3-4188-b495-ac2743805f33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6811","nonce":823348330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6813","nonce":823348330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6817","nonce":823348330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6815","nonce":823348330}]},"public_addr":"172.21.15.63:6811/823348330","cluster_addr":"172.21.15.63:6813/823348330","heartbeat_back_addr":"172.21.15.63:6817/823348330","heartbeat_front_addr":"172.21.15.63:6815/823348330","state":["exists","up"]},{"osd":2,"uuid":"edc4babe-dff9-462c-9af7-75e30b8095af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6819","nonce":219143073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6821","nonce":219143073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6825","nonce":219143073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6823","nonce":219143073}]},"public_addr":"172.21.15.63:6819/219143073","cluster_addr":"172.21.15.63:6821/219143073","heartbeat_back_addr":"172.21.15.63:6825/219143073","heartbeat_front_addr":"172.21.15.63:6823/219143073","state":["exists","up"]},{"osd":3,"uuid":"7bc94d72-a814-447a-8112-9d50058a2759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6801","nonce":1017350834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6803","nonce":1017350834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6807","nonce":1017350834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6805","nonce":1017350834}]},"public_addr":"172.21.15.111:6801/1017350834","cluster_addr":"172.21.15.111:6803/1017350834","heartbeat_back_addr":"172.21.15.111:6807/1017350834","heartbeat_front_addr":"172.21.15.111:6805/1017350834","state":["exists","up"]},{"osd":4,"uuid":"7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6809","nonce":1242047756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6811","nonce":1242047756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6815","nonce":1242047756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6813","nonce":1242047756}]},"public_addr":"172.21.15.111:6809/1242047756","cluster_addr":"172.21.15.111:6811/1242047756","heartbeat_back_addr":"172.21.15.111:6815/1242047756","heartbeat_front_addr":"172.21.15.111:6813/1242047756","state":["exists","up"]},{"osd":5,"uuid":"2219a929-a842-4b41-ab46-096be6ef4d13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6817","nonce":1318746257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6819","nonce":1318746257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6823","nonce":1318746257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6821","nonce":1318746257}]},"public_addr":"172.21.15.111:6817/1318746257","cluster_addr":"172.21.15.111:6819/1318746257","heartbeat_back_addr":"172.21.15.111:6823/1318746257","heartbeat_front_addr":"172.21.15.111:6821/1318746257","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:39:42.374999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:17.321830+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:50.840928+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:20.211360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:50.314996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:42:20.376924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/3148717714":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6801/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/2981669421":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/1183338694":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/3227457751":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6800/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3491519284":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6801/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/2286910220":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1379382252":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3192464709":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6801/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6800/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6800/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/3356174638":"2024-03-24T11:37:01.693515+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:42:46.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f79a00692a0 msgr2=0x7f79a006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:46.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f79a00692a0 0x7f79a006b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f79ac006fd0 tx=0x7f79ac008040 comp rx=0 tx=0).stop 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 msgr2=0x7f79b8116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8116250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f79a40043d0 tx=0x7f79a4009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 shutdown_connections 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f79a00692a0 0x7f79a006b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79b8070360 0x7f79b8115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 --2- 172.21.15.63:0/3497984308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b8070c70 0x7f79b8116250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 >> 172.21.15.63:0/3497984308 conn(0x7f79b80f89b0 msgr2=0x7f79b80f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 shutdown_connections 2024-03-23T11:42:46.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:46.340+0000 7f79c0600700 1 -- 172.21.15.63:0/3497984308 wait complete. 2024-03-23T11:42:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:45 smithi111 ceph-mon[44005]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:47.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-23T11:42:47.037 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-23T11:42:47.037 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-23T11:42:47.038 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-23T11:42:47.038 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-23T11:42:47.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-23T11:42:47.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:46 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3497984308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:42:47.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:46 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3497984308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:42:47.627 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:47.670 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:48.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:47 smithi111 ceph-mon[44005]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:48.238 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:47 smithi063 ceph-mon[35189]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:49.264 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:49.269 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:49.269 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:49.270 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:49.998 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:49 smithi063 ceph-mon[35189]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:50.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:49 smithi111 ceph-mon[44005]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.911+0000 7f29b2c00700 1 -- 172.21.15.63:0/1863138516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 msgr2=0x7f29ac0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.911+0000 7f29b2c00700 1 --2- 172.21.15.63:0/1863138516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac0fbd80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2998004940 tx=0x7f2998009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.912+0000 7f29b2c00700 1 -- 172.21.15.63:0/1863138516 shutdown_connections 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.912+0000 7f29b2c00700 1 --2- 172.21.15.63:0/1863138516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac0fbd80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.912+0000 7f29b2c00700 1 --2- 172.21.15.63:0/1863138516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29ac101d70 0x7f29ac102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.912+0000 7f29b2c00700 1 -- 172.21.15.63:0/1863138516 >> 172.21.15.63:0/1863138516 conn(0x7f29ac072bc0 msgr2=0x7f29ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.912+0000 7f29b2c00700 1 -- 172.21.15.63:0/1863138516 shutdown_connections 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.912+0000 7f29b2c00700 1 -- 172.21.15.63:0/1863138516 wait complete. 2024-03-23T11:42:51.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29b2c00700 1 Processor -- start 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29b2c00700 1 -- start start 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29b2c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29ac101d70 0x7f29ac102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29b2c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29b2c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac1036a0 con 0x7f29ac102710 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29b2c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac1037e0 con 0x7f29ac101d70 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29ab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.913+0000 7f29ab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47914/0 (socket says 172.21.15.63:47914) 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29ab400700 1 -- 172.21.15.63:0/2357253200 learned_addr learned my addr 172.21.15.63:0/2357253200 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29abe00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29ac101d70 0x7f29ac102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29ab400700 1 -- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29ac101d70 msgr2=0x7f29ac102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29ab400700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29ac101d70 0x7f29ac102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29ab400700 1 -- 172.21.15.63:0/2357253200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2998005040 con 0x7f29ac102710 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29ab400700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac103110 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2998009390 tx=0x7f2998009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:51.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f299800e040 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2998013420 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac103a30 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.914+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f299801c630 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.915+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac11a1f0 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.916+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f29ac105fe0 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.918+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2998013a00 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29a3e00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f29900692b0 0x7f299006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2998087fd0 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29abe00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f29900692b0 0x7f299006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29a3e00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] conn(0x7f299006ef80 0x7f2990071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 --> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2990071a40 con 0x7f299006ef80 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f299808a530 con 0x7f29ac102710 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29abe00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f29900692b0 0x7f299006b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f299c004f40 tx=0x7f299c00b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:51.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.919+0000 7f29b0c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] conn(0x7f299006ef80 0x7f2990071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.920+0000 7f29b0c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] conn(0x7f299006ef80 0x7f2990071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.921+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== osd.4 v2:172.21.15.111:6808/1242047756 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2990071a40 con 0x7f299006ef80 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.938+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 --> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f29ac0fd4c0 con 0x7f299006ef80 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.939+0000 7f29a3e00700 1 -- 172.21.15.63:0/2357253200 <== osd.4 v2:172.21.15.111:6808/1242047756 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f29ac0fd4c0 con 0x7f299006ef80 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] conn(0x7f299006ef80 msgr2=0x7f2990071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] conn(0x7f299006ef80 0x7f2990071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f29900692b0 msgr2=0x7f299006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f29900692b0 0x7f299006b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f299c004f40 tx=0x7f299c00b040 comp rx=0 tx=0).stop 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 msgr2=0x7f29ac103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac103110 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2998009390 tx=0x7f2998009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 shutdown_connections 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:6808/1242047756,v1:172.21.15.111:6809/1242047756] conn(0x7f299006ef80 0x7f2990071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f29900692b0 0x7f299006b760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29ac101d70 0x7f29ac102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 --2- 172.21.15.63:0/2357253200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29ac102710 0x7f29ac103110 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 >> 172.21.15.63:0/2357253200 conn(0x7f29ac072bc0 msgr2=0x7f29ac065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:51.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 shutdown_connections 2024-03-23T11:42:51.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:50.940+0000 7f29b2c00700 1 -- 172.21.15.63:0/2357253200 wait complete. 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 -- 172.21.15.63:0/2652041490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 msgr2=0x7fde00106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 --2- 172.21.15.63:0/2652041490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00106f00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fddec004940 tx=0x7fddec009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 -- 172.21.15.63:0/2652041490 shutdown_connections 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 --2- 172.21.15.63:0/2652041490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00106f00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 --2- 172.21.15.63:0/2652041490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde000fde90 0x7fde000fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 -- 172.21.15.63:0/2652041490 >> 172.21.15.63:0/2652041490 conn(0x7fde00072bc0 msgr2=0x7fde00072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 -- 172.21.15.63:0/2652041490 shutdown_connections 2024-03-23T11:42:51.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.602+0000 7fde07600700 1 -- 172.21.15.63:0/2652041490 wait complete. 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.603+0000 7fde07600700 1 Processor -- start 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.603+0000 7fde07600700 1 -- start start 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.603+0000 7fde07600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde000fde90 0x7fde00113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.603+0000 7fde07600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.603+0000 7fde07600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde00114670 con 0x7fde000fe830 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.603+0000 7fde07600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0010da60 con 0x7fde000fde90 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44214/0 (socket says 172.21.15.63:44214) 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddffe00700 1 -- 172.21.15.63:0/1081525818 learned_addr learned my addr 172.21.15.63:0/1081525818 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fde04c00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde000fde90 0x7fde00113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddffe00700 1 -- 172.21.15.63:0/1081525818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde000fde90 msgr2=0x7fde00113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddffe00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde000fde90 0x7fde00113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddffe00700 1 -- 172.21.15.63:0/1081525818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec005040 con 0x7fde000fe830 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fde04c00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde000fde90 0x7fde00113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddffe00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00114020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fddec0093b0 tx=0x7fddec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.604+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddec00e040 con 0x7fde000fe830 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.605+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddec004290 con 0x7fde000fe830 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.605+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddec01c3f0 con 0x7fde000fe830 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.605+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0010dce0 con 0x7fde000fe830 2024-03-23T11:42:51.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.605+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0010e1d0 con 0x7fde000fe830 2024-03-23T11:42:51.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.605+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde0010e740 con 0x7fde000fe830 2024-03-23T11:42:51.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.607+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fddec013790 con 0x7fde000fe830 2024-03-23T11:42:51.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.607+0000 7fddfcc00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdde80692b0 0x7fdde806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.608+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fddec088c10 con 0x7fde000fe830 2024-03-23T11:42:51.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.608+0000 7fde04c00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdde80692b0 0x7fdde806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.608+0000 7fddfcc00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] conn(0x7fdde806ef80 0x7fdde8071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:51.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.608+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 --> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdde8071a40 con 0x7fdde806ef80 2024-03-23T11:42:51.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.608+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddec088f50 con 0x7fde000fe830 2024-03-23T11:42:51.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.608+0000 7fde05600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] conn(0x7fdde806ef80 0x7fdde8071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:51.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.608+0000 7fde04c00700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdde80692b0 0x7fdde806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009fd0 tx=0x7fddf400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:51.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.609+0000 7fde05600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] conn(0x7fdde806ef80 0x7fdde8071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:51.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.610+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== osd.1 v2:172.21.15.63:6810/823348330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdde8071a40 con 0x7fdde806ef80 2024-03-23T11:42:51.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.626+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 --> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde0010e900 con 0x7fdde806ef80 2024-03-23T11:42:51.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.626+0000 7fddfcc00700 1 -- 172.21.15.63:0/1081525818 <== osd.1 v2:172.21.15.63:6810/823348330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fde0010e900 con 0x7fdde806ef80 2024-03-23T11:42:51.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.626+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] conn(0x7fdde806ef80 msgr2=0x7fdde8071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.626+0000 7fde07600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] conn(0x7fdde806ef80 0x7fdde8071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.626+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdde80692b0 msgr2=0x7fdde806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.626+0000 7fde07600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdde80692b0 0x7fdde806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009fd0 tx=0x7fddf400b040 comp rx=0 tx=0).stop 2024-03-23T11:42:51.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 msgr2=0x7fde00114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00114020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fddec0093b0 tx=0x7fddec009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 shutdown_connections 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6810/823348330,v1:172.21.15.63:6811/823348330] conn(0x7fdde806ef80 0x7fdde8071390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdde80692b0 0x7fdde806b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde000fde90 0x7fde00113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 --2- 172.21.15.63:0/1081525818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde000fe830 0x7fde00114020 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 >> 172.21.15.63:0/1081525818 conn(0x7fde00072bc0 msgr2=0x7fde00103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 shutdown_connections 2024-03-23T11:42:51.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:51.627+0000 7fde07600700 1 -- 172.21.15.63:0/1081525818 wait complete. 2024-03-23T11:42:52.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:51 smithi063 ceph-mon[35189]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:52.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:51 smithi111 ceph-mon[44005]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:52.437 INFO:teuthology.orchestra.run.smithi063.stdout:146028888078 2024-03-23T11:42:52.437 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-23T11:42:54.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:53 smithi063 ceph-mon[35189]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:54.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:53 smithi111 ceph-mon[44005]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:54.572 INFO:teuthology.orchestra.run.smithi063.stdout:60129542177 2024-03-23T11:42:54.572 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-23T11:42:56.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:55 smithi063 ceph-mon[35189]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:56.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:55 smithi111 ceph-mon[44005]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:56.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 -- 172.21.15.63:0/164870958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c102e00 msgr2=0x7fec4c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:56.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 --2- 172.21.15.63:0/164870958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c102e00 0x7fec4c1031d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fec3c004940 tx=0x7fec3c009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:56.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 -- 172.21.15.63:0/164870958 shutdown_connections 2024-03-23T11:42:56.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 --2- 172.21.15.63:0/164870958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec4c1037a0 0x7fec4c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 --2- 172.21.15.63:0/164870958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c102e00 0x7fec4c1031d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 -- 172.21.15.63:0/164870958 >> 172.21.15.63:0/164870958 conn(0x7fec4c0f89b0 msgr2=0x7fec4c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:56.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 -- 172.21.15.63:0/164870958 shutdown_connections 2024-03-23T11:42:56.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.983+0000 7fec52000700 1 -- 172.21.15.63:0/164870958 wait complete. 2024-03-23T11:42:56.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.984+0000 7fec52000700 1 Processor -- start 2024-03-23T11:42:56.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.984+0000 7fec52000700 1 -- start start 2024-03-23T11:42:56.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.984+0000 7fec52000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec4c102e00 0x7fec4c115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:56.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.984+0000 7fec52000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c1037a0 0x7fec4c116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:56.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.984+0000 7fec52000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c1168e0 con 0x7fec4c1037a0 2024-03-23T11:42:56.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.984+0000 7fec52000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c10fc20 con 0x7fec4c102e00 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c1037a0 0x7fec4c116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c1037a0 0x7fec4c116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44242/0 (socket says 172.21.15.63:44242) 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4aa00700 1 -- 172.21.15.63:0/3669623451 learned_addr learned my addr 172.21.15.63:0/3669623451 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4b400700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec4c102e00 0x7fec4c115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4aa00700 1 -- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec4c102e00 msgr2=0x7fec4c115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4aa00700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec4c102e00 0x7fec4c115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4aa00700 1 -- 172.21.15.63:0/3669623451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c005040 con 0x7fec4c1037a0 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4b400700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec4c102e00 0x7fec4c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:42:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec4aa00700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c1037a0 0x7fec4c116200 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fec3800b790 tx=0x7fec38009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:56.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec3800f6f0 con 0x7fec4c1037a0 2024-03-23T11:42:56.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec3800fd00 con 0x7fec4c1037a0 2024-03-23T11:42:56.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec380103a0 con 0x7fec4c1037a0 2024-03-23T11:42:56.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.985+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4c10fea0 con 0x7fec4c1037a0 2024-03-23T11:42:56.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.986+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c110460 con 0x7fec4c1037a0 2024-03-23T11:42:56.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.987+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec4c1109f0 con 0x7fec4c1037a0 2024-03-23T11:42:56.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.988+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fec3800f850 con 0x7fec4c1037a0 2024-03-23T11:42:56.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec43400700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fec300692a0 0x7fec3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:56.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fec38086cb0 con 0x7fec4c1037a0 2024-03-23T11:42:56.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec4b400700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fec300692a0 0x7fec3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:56.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec43400700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] conn(0x7fec3006ef70 0x7fec30071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:56.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 --> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec30071a30 con 0x7fec3006ef70 2024-03-23T11:42:56.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec3808b220 con 0x7fec4c1037a0 2024-03-23T11:42:56.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec4be00700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] conn(0x7fec3006ef70 0x7fec30071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:56.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.989+0000 7fec4b400700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fec300692a0 0x7fec3006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec3c004560 tx=0x7fec3c009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:56.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.990+0000 7fec4be00700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] conn(0x7fec3006ef70 0x7fec30071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:56.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:56.991+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== osd.5 v2:172.21.15.111:6816/1318746257 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fec30071a30 con 0x7fec3006ef70 2024-03-23T11:42:56.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.007+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 --> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec4c0705f0 con 0x7fec3006ef70 2024-03-23T11:42:56.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec43400700 1 -- 172.21.15.63:0/3669623451 <== osd.5 v2:172.21.15.111:6816/1318746257 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fec4c0705f0 con 0x7fec3006ef70 2024-03-23T11:42:56.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] conn(0x7fec3006ef70 msgr2=0x7fec30071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] conn(0x7fec3006ef70 0x7fec30071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fec300692a0 msgr2=0x7fec3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fec300692a0 0x7fec3006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec3c004560 tx=0x7fec3c009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c1037a0 msgr2=0x7fec4c116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c1037a0 0x7fec4c116200 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fec3800b790 tx=0x7fec38009220 comp rx=0 tx=0).stop 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 shutdown_connections 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:6816/1318746257,v1:172.21.15.111:6817/1318746257] conn(0x7fec3006ef70 0x7fec30071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fec300692a0 0x7fec3006b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec4c102e00 0x7fec4c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 --2- 172.21.15.63:0/3669623451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec4c1037a0 0x7fec4c116200 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.008+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 >> 172.21.15.63:0/3669623451 conn(0x7fec4c0f89b0 msgr2=0x7fec4c071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.009+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 shutdown_connections 2024-03-23T11:42:56.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:57.009+0000 7fec52000700 1 -- 172.21.15.63:0/3669623451 wait complete. 2024-03-23T11:42:58.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:57 smithi063 ceph-mon[35189]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:58.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:57 smithi111 ceph-mon[44005]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:42:58.454 INFO:teuthology.orchestra.run.smithi063.stdout:167503724553 2024-03-23T11:42:58.454 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-23T11:42:58.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 -- 172.21.15.63:0/2378242445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69380708a0 msgr2=0x7f6938070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:58.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 --2- 172.21.15.63:0/2378242445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69380708a0 0x7f6938070d10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6924004970 tx=0x7f6924009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:58.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 -- 172.21.15.63:0/2378242445 shutdown_connections 2024-03-23T11:42:58.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 --2- 172.21.15.63:0/2378242445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69380708a0 0x7f6938070d10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 --2- 172.21.15.63:0/2378242445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6938071f90 0x7f6938070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 -- 172.21.15.63:0/2378242445 >> 172.21.15.63:0/2378242445 conn(0x7f69380f89b0 msgr2=0x7f69380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:58.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 -- 172.21.15.63:0/2378242445 shutdown_connections 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.225+0000 7f693e800700 1 -- 172.21.15.63:0/2378242445 wait complete. 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.226+0000 7f693e800700 1 Processor -- start 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.226+0000 7f693e800700 1 -- start start 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.226+0000 7f693e800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380708a0 0x7f693811a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.226+0000 7f693e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938071f90 0x7f693811a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.226+0000 7f693e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693811ab90 con 0x7f6938071f90 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.226+0000 7f693e800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6938114010 con 0x7f69380708a0 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6936a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938071f90 0x7f693811a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6936a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938071f90 0x7f693811a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44256/0 (socket says 172.21.15.63:44256) 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6936a00700 1 -- 172.21.15.63:0/3132262072 learned_addr learned my addr 172.21.15.63:0/3132262072 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6937400700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380708a0 0x7f693811a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6936a00700 1 -- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380708a0 msgr2=0x7f693811a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6936a00700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380708a0 0x7f693811a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6936a00700 1 -- 172.21.15.63:0/3132262072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924005040 con 0x7f6938071f90 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f6936a00700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938071f90 0x7f693811a540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6924004f50 tx=0x7f6924009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:58.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.227+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f692400e040 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.228+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6924013420 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.228+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6938114290 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.228+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f692401c630 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.228+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69381147d0 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.229+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6938114cf0 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.229+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6924013590 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.230+0000 7f692f400700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f691c0692b0 0x7f691c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.230+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6924088170 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.231+0000 7f6937400700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f691c0692b0 0x7f691c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.231+0000 7f692f400700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] conn(0x7f691c06ef80 0x7f691c071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.231+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 --> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f691c071a40 con 0x7f691c06ef80 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.231+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f692408a6c0 con 0x7f6938071f90 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.231+0000 7f6937e00700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] conn(0x7f691c06ef80 0x7f691c071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:58.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.231+0000 7f6937400700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f691c0692b0 0x7f691c06b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f69280041d0 tx=0x7f69280043c0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:58.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.232+0000 7f6937e00700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] conn(0x7f691c06ef80 0x7f691c071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:58.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.233+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== osd.3 v2:172.21.15.111:6800/1017350834 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f691c071a40 con 0x7f691c06ef80 2024-03-23T11:42:58.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.249+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 --> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6938114eb0 con 0x7f691c06ef80 2024-03-23T11:42:58.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.249+0000 7f692f400700 1 -- 172.21.15.63:0/3132262072 <== osd.3 v2:172.21.15.111:6800/1017350834 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6938114eb0 con 0x7f691c06ef80 2024-03-23T11:42:58.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] conn(0x7f691c06ef80 msgr2=0x7f691c071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:58.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] conn(0x7f691c06ef80 0x7f691c071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f691c0692b0 msgr2=0x7f691c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f691c0692b0 0x7f691c06b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f69280041d0 tx=0x7f69280043c0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938071f90 msgr2=0x7f693811a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938071f90 0x7f693811a540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6924004f50 tx=0x7f6924009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 shutdown_connections 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:6800/1017350834,v1:172.21.15.111:6801/1017350834] conn(0x7f691c06ef80 0x7f691c071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f691c0692b0 0x7f691c06b760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380708a0 0x7f693811a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 --2- 172.21.15.63:0/3132262072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938071f90 0x7f693811a540 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 >> 172.21.15.63:0/3132262072 conn(0x7f69380f89b0 msgr2=0x7f69380f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 shutdown_connections 2024-03-23T11:42:58.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:58.250+0000 7f693e800700 1 -- 172.21.15.63:0/3132262072 wait complete. 2024-03-23T11:42:59.135 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:59.135 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:42:59.547 INFO:teuthology.orchestra.run.smithi063.stdout:111669149718 2024-03-23T11:42:59.547 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-23T11:42:59.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 -- 172.21.15.63:0/1976052377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 msgr2=0x7f84d40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:59.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 --2- 172.21.15.63:0/1976052377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d40fce10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f84c0004940 tx=0x7f84c0009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:59.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 -- 172.21.15.63:0/1976052377 shutdown_connections 2024-03-23T11:42:59.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 --2- 172.21.15.63:0/1976052377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d40fce10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 --2- 172.21.15.63:0/1976052377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d4102e00 0x7f84d41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 -- 172.21.15.63:0/1976052377 >> 172.21.15.63:0/1976052377 conn(0x7f84d40f89b0 msgr2=0x7f84d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:59.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 -- 172.21.15.63:0/1976052377 shutdown_connections 2024-03-23T11:42:59.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.834+0000 7f84dc200700 1 -- 172.21.15.63:0/1976052377 wait complete. 2024-03-23T11:42:59.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.835+0000 7f84dc200700 1 Processor -- start 2024-03-23T11:42:59.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.835+0000 7f84dc200700 1 -- start start 2024-03-23T11:42:59.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.835+0000 7f84dc200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d4102e00 0x7f84d4113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:59.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.835+0000 7f84dc200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:59.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.835+0000 7f84dc200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4114700 con 0x7f84d41037a0 2024-03-23T11:42:59.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.835+0000 7f84dc200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d410da60 con 0x7f84d4102e00 2024-03-23T11:42:59.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.836+0000 7f84d8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:59.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.836+0000 7f84d8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d4114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44272/0 (socket says 172.21.15.63:44272) 2024-03-23T11:42:59.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.836+0000 7f84d8e00700 1 -- 172.21.15.63:0/1043968311 learned_addr learned my addr 172.21.15.63:0/1043968311 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:42:59.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.836+0000 7f84d9800700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d4102e00 0x7f84d4113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:59.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.836+0000 7f84d8e00700 1 -- 172.21.15.63:0/1043968311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d4102e00 msgr2=0x7f84d4113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:59.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.836+0000 7f84d8e00700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d4102e00 0x7f84d4113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.836+0000 7f84d8e00700 1 -- 172.21.15.63:0/1043968311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0005040 con 0x7f84d41037a0 2024-03-23T11:42:59.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.837+0000 7f84d8e00700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d4114020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f84c0008fd0 tx=0x7f84c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:59.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.837+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c000e040 con 0x7f84d41037a0 2024-03-23T11:42:59.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.837+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84c0004290 con 0x7f84d41037a0 2024-03-23T11:42:59.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.837+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c0013660 con 0x7f84d41037a0 2024-03-23T11:42:59.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.837+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d410dce0 con 0x7f84d41037a0 2024-03-23T11:42:59.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.837+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d410e2a0 con 0x7f84d41037a0 2024-03-23T11:42:59.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.838+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f84d410e810 con 0x7f84d41037a0 2024-03-23T11:42:59.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.839+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f84c0022020 con 0x7f84d41037a0 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.840+0000 7f84d1600700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f84bc0692b0 0x7f84bc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.840+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f84c008c190 con 0x7f84d41037a0 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.841+0000 7f84d9800700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f84bc0692b0 0x7f84bc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.841+0000 7f84d1600700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] conn(0x7f84bc06ef80 0x7f84bc071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.841+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 --> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84bc071a40 con 0x7f84bc06ef80 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.841+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84c008c4d0 con 0x7f84d41037a0 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.841+0000 7f84da200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] conn(0x7f84bc06ef80 0x7f84bc071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.841+0000 7f84d9800700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f84bc0692b0 0x7f84bc06b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009bf0 tx=0x7f84c80091a0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.842+0000 7f84da200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] conn(0x7f84bc06ef80 0x7f84bc071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:42:59.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.843+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== osd.0 v2:172.21.15.63:6802/1473323264 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f84bc071a40 con 0x7f84bc06ef80 2024-03-23T11:42:59.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.871+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 --> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f84d40fa310 con 0x7f84bc06ef80 2024-03-23T11:42:59.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.871+0000 7f84d1600700 1 -- 172.21.15.63:0/1043968311 <== osd.0 v2:172.21.15.63:6802/1473323264 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f84d40fa310 con 0x7f84bc06ef80 2024-03-23T11:42:59.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] conn(0x7f84bc06ef80 msgr2=0x7f84bc071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:59.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] conn(0x7f84bc06ef80 0x7f84bc071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f84bc0692b0 msgr2=0x7f84bc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:59.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f84bc0692b0 0x7f84bc06b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009bf0 tx=0x7f84c80091a0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 msgr2=0x7f84d4114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d4114020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f84c0008fd0 tx=0x7f84c0009e30 comp rx=0 tx=0).stop 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 shutdown_connections 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6802/1473323264,v1:172.21.15.63:6803/1473323264] conn(0x7f84bc06ef80 0x7f84bc071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f84bc0692b0 0x7f84bc06b760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d4102e00 0x7f84d4113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 --2- 172.21.15.63:0/1043968311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d41037a0 0x7f84d4114020 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.872+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 >> 172.21.15.63:0/1043968311 conn(0x7f84d40f89b0 msgr2=0x7f84d40f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.873+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 shutdown_connections 2024-03-23T11:42:59.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:42:59.873+0000 7f84dc200700 1 -- 172.21.15.63:0/1043968311 wait complete. 2024-03-23T11:43:00.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:59 smithi063 ceph-mon[35189]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:00.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:42:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:59 smithi111 ceph-mon[44005]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:00.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:42:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 -- 172.21.15.63:0/3003478055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 msgr2=0x7f88fc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 --2- 172.21.15.63:0/3003478055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc1031d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88ec004940 tx=0x7f88ec009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 -- 172.21.15.63:0/3003478055 shutdown_connections 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 --2- 172.21.15.63:0/3003478055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88fc1037a0 0x7f88fc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 --2- 172.21.15.63:0/3003478055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc1031d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 -- 172.21.15.63:0/3003478055 >> 172.21.15.63:0/3003478055 conn(0x7f88fc0f89b0 msgr2=0x7f88fc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 -- 172.21.15.63:0/3003478055 shutdown_connections 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.014+0000 7f8901c00700 1 -- 172.21.15.63:0/3003478055 wait complete. 2024-03-23T11:43:01.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f8901c00700 1 Processor -- start 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f8901c00700 1 -- start start 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f8901c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f8901c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88fc1037a0 0x7f88fc116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f8901c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc116880 con 0x7f88fc102e00 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f8901c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc10fc50 con 0x7f88fc1037a0 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f88faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.015+0000 7f88faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44294/0 (socket says 172.21.15.63:44294) 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88faa00700 1 -- 172.21.15.63:0/2404981758 learned_addr learned my addr 172.21.15.63:0/2404981758 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:01.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88fa000700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88fc1037a0 0x7f88fc116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:01.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88faa00700 1 -- 172.21.15.63:0/2404981758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88fc1037a0 msgr2=0x7f88fc116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:01.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88faa00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88fc1037a0 0x7f88fc116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88faa00700 1 -- 172.21.15.63:0/2404981758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec005040 con 0x7f88fc102e00 2024-03-23T11:43:01.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88fa000700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88fc1037a0 0x7f88fc116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88faa00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc115cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009390 tx=0x7f88ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88ec00e040 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88ec013420 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88ec01c630 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.016+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc10fed0 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.017+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc110440 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.017+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88fc110970 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.018+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f88ec0139e0 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.019+0000 7f88f2a00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f88e00692b0 0x7f88e006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.019+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f88ec088120 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.019+0000 7f88fa000700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f88e00692b0 0x7f88e006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.020+0000 7f88f2a00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] conn(0x7f88e006ef80 0x7f88e0071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.020+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 --> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88e0071a40 con 0x7f88e006ef80 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.020+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec08a670 con 0x7f88fc102e00 2024-03-23T11:43:01.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.020+0000 7f88fb400700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] conn(0x7f88e006ef80 0x7f88e0071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.020+0000 7f88fa000700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f88e00692b0 0x7f88e006b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f88e8007fd0 tx=0x7f88e8009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.020+0000 7f88fb400700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] conn(0x7f88e006ef80 0x7f88e0071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.021+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== osd.2 v2:172.21.15.63:6818/219143073 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f88e0071a40 con 0x7f88e006ef80 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.037+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 --> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f88fc0fe550 con 0x7f88e006ef80 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.037+0000 7f88f2a00700 1 -- 172.21.15.63:0/2404981758 <== osd.2 v2:172.21.15.63:6818/219143073 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f88fc0fe550 con 0x7f88e006ef80 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.038+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] conn(0x7f88e006ef80 msgr2=0x7f88e0071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.038+0000 7f8901c00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] conn(0x7f88e006ef80 0x7f88e0071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.038+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f88e00692b0 msgr2=0x7f88e006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.038+0000 7f8901c00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f88e00692b0 0x7f88e006b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f88e8007fd0 tx=0x7f88e8009040 comp rx=0 tx=0).stop 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.038+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 msgr2=0x7f88fc115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.038+0000 7f8901c00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc115cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009390 tx=0x7f88ec009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 shutdown_connections 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6818/219143073,v1:172.21.15.63:6819/219143073] conn(0x7f88e006ef80 0x7f88e0071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f88e00692b0 0x7f88e006b760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc102e00 0x7f88fc115cf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 --2- 172.21.15.63:0/2404981758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88fc1037a0 0x7f88fc116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 >> 172.21.15.63:0/2404981758 conn(0x7f88fc0f89b0 msgr2=0x7f88fc0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 shutdown_connections 2024-03-23T11:43:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:01.039+0000 7f8901c00700 1 -- 172.21.15.63:0/2404981758 wait complete. 2024-03-23T11:43:01.478 INFO:teuthology.orchestra.run.smithi063.stdout:38654705707 2024-03-23T11:43:01.478 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-23T11:43:02.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:01 smithi063 ceph-mon[35189]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:02.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:01 smithi111 ceph-mon[44005]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:02.808 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:03.639 INFO:teuthology.orchestra.run.smithi063.stdout:81604378651 2024-03-23T11:43:03.639 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-23T11:43:03.884 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:04.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:03 smithi063 ceph-mon[35189]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:04.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:03 smithi111 ceph-mon[44005]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:05.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 -- 172.21.15.63:0/1698641064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 msgr2=0x7fad680fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:05.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 --2- 172.21.15.63:0/1698641064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad680fce10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fad54004940 tx=0x7fad54009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:05.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 -- 172.21.15.63:0/1698641064 shutdown_connections 2024-03-23T11:43:05.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 --2- 172.21.15.63:0/1698641064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad680fce10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:05.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 --2- 172.21.15.63:0/1698641064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102e00 0x7fad681031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:05.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 -- 172.21.15.63:0/1698641064 >> 172.21.15.63:0/1698641064 conn(0x7fad680f89b0 msgr2=0x7fad680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:05.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 -- 172.21.15.63:0/1698641064 shutdown_connections 2024-03-23T11:43:05.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.173+0000 7fad6f200700 1 -- 172.21.15.63:0/1698641064 wait complete. 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.174+0000 7fad6f200700 1 Processor -- start 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.174+0000 7fad6f200700 1 -- start start 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.174+0000 7fad6f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102e00 0x7fad68113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.174+0000 7fad6f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad68113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.174+0000 7fad6f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad68114600 con 0x7fad681037a0 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.174+0000 7fad6f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6810d9f0 con 0x7fad68102e00 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad68113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad68113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40784/0 (socket says 172.21.15.63:40784) 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67400700 1 -- 172.21.15.63:0/4131333612 learned_addr learned my addr 172.21.15.63:0/4131333612 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67400700 1 -- 172.21.15.63:0/4131333612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102e00 msgr2=0x7fad68113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:05.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67e00700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102e00 0x7fad68113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67400700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102e00 0x7fad68113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67400700 1 -- 172.21.15.63:0/4131333612 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad54005040 con 0x7fad681037a0 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad67400700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad68113fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fad54008d50 tx=0x7fad54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad5fe00700 1 -- 172.21.15.63:0/4131333612 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad5400e030 con 0x7fad681037a0 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.175+0000 7fad5fe00700 1 -- 172.21.15.63:0/4131333612 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad54013de0 con 0x7fad681037a0 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.176+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6810dc70 con 0x7fad681037a0 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.176+0000 7fad5fe00700 1 -- 172.21.15.63:0/4131333612 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad5401cd90 con 0x7fad681037a0 2024-03-23T11:43:05.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.176+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6810e230 con 0x7fad681037a0 2024-03-23T11:43:05.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.177+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad680fe550 con 0x7fad681037a0 2024-03-23T11:43:05.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.177+0000 7fad5fe00700 1 -- 172.21.15.63:0/4131333612 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fad54013930 con 0x7fad681037a0 2024-03-23T11:43:05.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.178+0000 7fad5fe00700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fad4c0692b0 0x7fad4c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:05.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.178+0000 7fad5fe00700 1 -- 172.21.15.63:0/4131333612 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fad54088810 con 0x7fad681037a0 2024-03-23T11:43:05.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.179+0000 7fad67e00700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fad4c0692b0 0x7fad4c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:05.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.179+0000 7fad67e00700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fad4c0692b0 0x7fad4c06b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fad58004600 tx=0x7fad5800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:05.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.183+0000 7fad5fe00700 1 -- 172.21.15.63:0/4131333612 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fad540588f0 con 0x7fad681037a0 2024-03-23T11:43:05.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.397+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fad68109dc0 con 0x7fad681037a0 2024-03-23T11:43:05.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.398+0000 7fad5fe00700 1 -- 172.21.15.63:0/4131333612 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fad54058480 con 0x7fad681037a0 2024-03-23T11:43:05.384 INFO:teuthology.orchestra.run.smithi063.stdout:60129542179 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fad4c0692b0 msgr2=0x7fad4c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fad4c0692b0 0x7fad4c06b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fad58004600 tx=0x7fad5800b040 comp rx=0 tx=0).stop 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 msgr2=0x7fad68113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad68113fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fad54008d50 tx=0x7fad54009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 shutdown_connections 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fad4c0692b0 0x7fad4c06b760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102e00 0x7fad68113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 --2- 172.21.15.63:0/4131333612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad681037a0 0x7fad68113fb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 >> 172.21.15.63:0/4131333612 conn(0x7fad680f89b0 msgr2=0x7fad680f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.400+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 shutdown_connections 2024-03-23T11:43:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:05.401+0000 7fad6f200700 1 -- 172.21.15.63:0/4131333612 wait complete. 2024-03-23T11:43:06.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:05 smithi063 ceph-mon[35189]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:06.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:05 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4131333612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-23T11:43:06.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:05 smithi111 ceph-mon[44005]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:06.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:05 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4131333612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-23T11:43:06.651 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 -- 172.21.15.63:0/507578955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 msgr2=0x7f50981031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 --2- 172.21.15.63:0/507578955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f50981031d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f508c004940 tx=0x7f508c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 -- 172.21.15.63:0/507578955 shutdown_connections 2024-03-23T11:43:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 --2- 172.21.15.63:0/507578955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50981037a0 0x7f50980fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 --2- 172.21.15.63:0/507578955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f50981031d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 -- 172.21.15.63:0/507578955 >> 172.21.15.63:0/507578955 conn(0x7f50980f89b0 msgr2=0x7f50980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 -- 172.21.15.63:0/507578955 shutdown_connections 2024-03-23T11:43:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.007+0000 7f509f400700 1 -- 172.21.15.63:0/507578955 wait complete. 2024-03-23T11:43:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.008+0000 7f509f400700 1 Processor -- start 2024-03-23T11:43:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.008+0000 7f509f400700 1 -- start start 2024-03-23T11:43:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.008+0000 7f509f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f5098115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.008+0000 7f509f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50981037a0 0x7f5098116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.008+0000 7f509f400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5098116880 con 0x7f5098102e00 2024-03-23T11:43:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.008+0000 7f509f400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509810fc50 con 0x7f50981037a0 2024-03-23T11:43:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f5098115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f5098115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40810/0 (socket says 172.21.15.63:40810) 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509ca00700 1 -- 172.21.15.63:0/2215656317 learned_addr learned my addr 172.21.15.63:0/2215656317 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f5097e00700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50981037a0 0x7f5098116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509ca00700 1 -- 172.21.15.63:0/2215656317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50981037a0 msgr2=0x7f5098116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509ca00700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50981037a0 0x7f5098116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509ca00700 1 -- 172.21.15.63:0/2215656317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c005040 con 0x7f5098102e00 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f5097e00700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50981037a0 0x7f5098116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:43:06.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509ca00700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f5098115cf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f508c009390 tx=0x7f508c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f5094c00700 1 -- 172.21.15.63:0/2215656317 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f508c00e040 con 0x7f5098102e00 2024-03-23T11:43:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f5094c00700 1 -- 172.21.15.63:0/2215656317 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f508c013420 con 0x7f5098102e00 2024-03-23T11:43:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f5094c00700 1 -- 172.21.15.63:0/2215656317 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f508c01c630 con 0x7f5098102e00 2024-03-23T11:43:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.009+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509810fed0 con 0x7f5098102e00 2024-03-23T11:43:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.010+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5098110490 con 0x7f5098102e00 2024-03-23T11:43:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.011+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50980fe550 con 0x7f5098102e00 2024-03-23T11:43:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.011+0000 7f5094c00700 1 -- 172.21.15.63:0/2215656317 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f508c0139e0 con 0x7f5098102e00 2024-03-23T11:43:06.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.012+0000 7f5094c00700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f50800692b0 0x7f508006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:06.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.012+0000 7f5094c00700 1 -- 172.21.15.63:0/2215656317 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f508c087ff0 con 0x7f5098102e00 2024-03-23T11:43:06.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.013+0000 7f5097e00700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f50800692b0 0x7f508006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:06.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.013+0000 7f5097e00700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f50800692b0 0x7f508006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5084009fd0 tx=0x7f508400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:07.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.017+0000 7f5094c00700 1 -- 172.21.15.63:0/2215656317 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f508c0ba190 con 0x7f5098102e00 2024-03-23T11:43:07.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.223+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f509804bed0 con 0x7f5098102e00 2024-03-23T11:43:07.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.224+0000 7f5094c00700 1 -- 172.21.15.63:0/2215656317 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f508c058180 con 0x7f5098102e00 2024-03-23T11:43:07.210 INFO:teuthology.orchestra.run.smithi063.stdout:146028888081 2024-03-23T11:43:07.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f50800692b0 msgr2=0x7f508006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f50800692b0 0x7f508006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5084009fd0 tx=0x7f508400b040 comp rx=0 tx=0).stop 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 msgr2=0x7f5098115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f5098115cf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f508c009390 tx=0x7f508c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 shutdown_connections 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f50800692b0 0x7f508006b760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5098102e00 0x7f5098115cf0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 --2- 172.21.15.63:0/2215656317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50981037a0 0x7f5098116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 >> 172.21.15.63:0/2215656317 conn(0x7f50980f89b0 msgr2=0x7f50980706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 shutdown_connections 2024-03-23T11:43:07.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:07.226+0000 7f509f400700 1 -- 172.21.15.63:0/2215656317 wait complete. 2024-03-23T11:43:07.293 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542177 got 60129542179 for osd.1 2024-03-23T11:43:07.293 DEBUG:teuthology.parallel:result is None 2024-03-23T11:43:08.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:07 smithi063 ceph-mon[35189]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:08.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:07 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2215656317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-23T11:43:08.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:07 smithi111 ceph-mon[44005]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:08.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:07 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2215656317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-23T11:43:08.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888081 for osd.4 2024-03-23T11:43:08.188 DEBUG:teuthology.parallel:result is None 2024-03-23T11:43:09.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 -- 172.21.15.63:0/2475732009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d8102e00 msgr2=0x7f67d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:09.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 --2- 172.21.15.63:0/2475732009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d8102e00 0x7f67d81031d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f67c8004940 tx=0x7f67c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:09.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 -- 172.21.15.63:0/2475732009 shutdown_connections 2024-03-23T11:43:09.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 --2- 172.21.15.63:0/2475732009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d81037a0 0x7f67d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:09.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 --2- 172.21.15.63:0/2475732009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d8102e00 0x7f67d81031d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:09.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 -- 172.21.15.63:0/2475732009 >> 172.21.15.63:0/2475732009 conn(0x7f67d80f89b0 msgr2=0x7f67d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:09.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 -- 172.21.15.63:0/2475732009 shutdown_connections 2024-03-23T11:43:09.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.585+0000 7f67de400700 1 -- 172.21.15.63:0/2475732009 wait complete. 2024-03-23T11:43:09.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67de400700 1 Processor -- start 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67de400700 1 -- start start 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67de400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d8102e00 0x7f67d8113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67de400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d81037a0 0x7f67d8113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67de400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d81145d0 con 0x7f67d81037a0 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67de400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d810d9c0 con 0x7f67d8102e00 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67d6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d81037a0 0x7f67d8113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67d6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d81037a0 0x7f67d8113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40828/0 (socket says 172.21.15.63:40828) 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.586+0000 7f67d6a00700 1 -- 172.21.15.63:0/3889535796 learned_addr learned my addr 172.21.15.63:0/3889535796 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:09.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67d7400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d8102e00 0x7f67d8113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:09.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67d6a00700 1 -- 172.21.15.63:0/3889535796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d8102e00 msgr2=0x7f67d8113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:09.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67d6a00700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d8102e00 0x7f67d8113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:09.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67d6a00700 1 -- 172.21.15.63:0/3889535796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c8005040 con 0x7f67d81037a0 2024-03-23T11:43:09.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67d7400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d8102e00 0x7f67d8113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:43:09.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67d6a00700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d81037a0 0x7f67d8113f80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f67c400eb40 tx=0x7f67c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67cf400700 1 -- 172.21.15.63:0/3889535796 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67c4007cc0 con 0x7f67d81037a0 2024-03-23T11:43:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67cf400700 1 -- 172.21.15.63:0/3889535796 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67c400f040 con 0x7f67d81037a0 2024-03-23T11:43:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d810dc40 con 0x7f67d81037a0 2024-03-23T11:43:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.587+0000 7f67cf400700 1 -- 172.21.15.63:0/3889535796 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67c4009570 con 0x7f67d81037a0 2024-03-23T11:43:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.588+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d810e200 con 0x7f67d81037a0 2024-03-23T11:43:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.589+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d80fe550 con 0x7f67d81037a0 2024-03-23T11:43:09.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.590+0000 7f67cf400700 1 -- 172.21.15.63:0/3889535796 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f67c40096d0 con 0x7f67d81037a0 2024-03-23T11:43:09.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.590+0000 7f67cf400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f67bc0692b0 0x7f67bc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:09.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.591+0000 7f67cf400700 1 -- 172.21.15.63:0/3889535796 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f67c4086880 con 0x7f67d81037a0 2024-03-23T11:43:09.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.591+0000 7f67d7400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f67bc0692b0 0x7f67bc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:09.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.591+0000 7f67d7400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f67bc0692b0 0x7f67bc06b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67c800b0d0 tx=0x7f67c8009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:09.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.595+0000 7f67cf400700 1 -- 172.21.15.63:0/3889535796 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f67c404e190 con 0x7f67d81037a0 2024-03-23T11:43:09.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.800+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f67d8109dc0 con 0x7f67d81037a0 2024-03-23T11:43:09.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.801+0000 7f67cf400700 1 -- 172.21.15.63:0/3889535796 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f67c4056a30 con 0x7f67d81037a0 2024-03-23T11:43:09.787 INFO:teuthology.orchestra.run.smithi063.stdout:167503724556 2024-03-23T11:43:09.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f67bc0692b0 msgr2=0x7f67bc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:09.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f67bc0692b0 0x7f67bc06b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67c800b0d0 tx=0x7f67c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:09.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d81037a0 msgr2=0x7f67d8113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:09.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d81037a0 0x7f67d8113f80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f67c400eb40 tx=0x7f67c400b040 comp rx=0 tx=0).stop 2024-03-23T11:43:09.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 shutdown_connections 2024-03-23T11:43:09.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f67bc0692b0 0x7f67bc06b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:09.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d8102e00 0x7f67d8113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:09.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 --2- 172.21.15.63:0/3889535796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67d81037a0 0x7f67d8113f80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:09.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 >> 172.21.15.63:0/3889535796 conn(0x7f67d80f89b0 msgr2=0x7f67d80f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:09.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 shutdown_connections 2024-03-23T11:43:09.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:09.803+0000 7f67de400700 1 -- 172.21.15.63:0/3889535796 wait complete. 2024-03-23T11:43:09.850 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:10.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:09 smithi063 ceph-mon[35189]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:10.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:09 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3889535796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-23T11:43:10.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:09 smithi111 ceph-mon[44005]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:10.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:09 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3889535796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-23T11:43:10.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 -- 172.21.15.63:0/1380392103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070360 msgr2=0x7f7c3c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:10.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 --2- 172.21.15.63:0/1380392103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070360 0x7f7c3c070730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7c30004970 tx=0x7f7c30009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:10.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 -- 172.21.15.63:0/1380392103 shutdown_connections 2024-03-23T11:43:10.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 --2- 172.21.15.63:0/1380392103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c070c70 0x7f7c3c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:10.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 --2- 172.21.15.63:0/1380392103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070360 0x7f7c3c070730 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:10.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 -- 172.21.15.63:0/1380392103 >> 172.21.15.63:0/1380392103 conn(0x7f7c3c0f89b0 msgr2=0x7f7c3c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:10.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 -- 172.21.15.63:0/1380392103 shutdown_connections 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.805+0000 7f7c44000700 1 -- 172.21.15.63:0/1380392103 wait complete. 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.806+0000 7f7c44000700 1 Processor -- start 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.806+0000 7f7c44000700 1 -- start start 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c44000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c070360 0x7f7c3c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c44000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070c70 0x7f7c3c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c44000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c116810 con 0x7f7c3c070c70 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c44000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c10fbe0 con 0x7f7c3c070360 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c40c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070c70 0x7f7c3c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c40c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070c70 0x7f7c3c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40856/0 (socket says 172.21.15.63:40856) 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c40c00700 1 -- 172.21.15.63:0/3073341050 learned_addr learned my addr 172.21.15.63:0/3073341050 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c41600700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c070360 0x7f7c3c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c40c00700 1 -- 172.21.15.63:0/3073341050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c070360 msgr2=0x7f7c3c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c40c00700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c070360 0x7f7c3c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c40c00700 1 -- 172.21.15.63:0/3073341050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30005040 con 0x7f7c3c070c70 2024-03-23T11:43:10.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.807+0000 7f7c41600700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c070360 0x7f7c3c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.808+0000 7f7c40c00700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070c70 0x7f7c3c1161c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800c8c0 tx=0x7f7c280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.808+0000 7f7c39600700 1 -- 172.21.15.63:0/3073341050 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2800a980 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.808+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c10fec0 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.808+0000 7f7c39600700 1 -- 172.21.15.63:0/3073341050 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2800ce60 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.808+0000 7f7c39600700 1 -- 172.21.15.63:0/3073341050 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c280113a0 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.808+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c110400 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.809+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c103b90 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.810+0000 7f7c39600700 1 -- 172.21.15.63:0/3073341050 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7c28012400 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.811+0000 7f7c39600700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c240692b0 0x7f7c2406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.811+0000 7f7c39600700 1 -- 172.21.15.63:0/3073341050 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7c28086e30 con 0x7f7c3c070c70 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.811+0000 7f7c41600700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c240692b0 0x7f7c2406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.812+0000 7f7c41600700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c240692b0 0x7f7c2406b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b800 tx=0x7f7c30009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:10.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.819+0000 7f7c39600700 1 -- 172.21.15.63:0/3073341050 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7c28056fd0 con 0x7f7c3c070c70 2024-03-23T11:43:10.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 -- 172.21.15.63:0/2901480846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc0724f0 msgr2=0x7f57cc0728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:10.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 --2- 172.21.15.63:0/2901480846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc0724f0 0x7f57cc0728c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f57c0004940 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:10.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 -- 172.21.15.63:0/2901480846 shutdown_connections 2024-03-23T11:43:10.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 --2- 172.21.15.63:0/2901480846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57cc072e00 0x7f57cc10d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 --2- 172.21.15.63:0/2901480846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc0724f0 0x7f57cc0728c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 -- 172.21.15.63:0/2901480846 >> 172.21.15.63:0/2901480846 conn(0x7f57cc0faba0 msgr2=0x7f57cc0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 -- 172.21.15.63:0/2901480846 shutdown_connections 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.901+0000 7f57d3c00700 1 -- 172.21.15.63:0/2901480846 wait complete. 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.902+0000 7f57d3c00700 1 Processor -- start 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.902+0000 7f57d3c00700 1 -- start start 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d3c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57cc0724f0 0x7f57cc119f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d3c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc072e00 0x7f57cc11a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d3c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc11aae0 con 0x7f57cc072e00 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d3c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc113ef0 con 0x7f57cc0724f0 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57cbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc072e00 0x7f57cc11a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d1200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57cc0724f0 0x7f57cc119f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57cbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc072e00 0x7f57cc11a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40880/0 (socket says 172.21.15.63:40880) 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57cbe00700 1 -- 172.21.15.63:0/371276397 learned_addr learned my addr 172.21.15.63:0/371276397 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:10.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d1200700 1 -- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc072e00 msgr2=0x7f57cc11a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d1200700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc072e00 0x7f57cc11a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.903+0000 7f57d1200700 1 -- 172.21.15.63:0/371276397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0005040 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.904+0000 7f57d1200700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57cc0724f0 0x7f57cc119f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f57c0000c00 tx=0x7f57c0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.904+0000 7f57c8c00700 1 -- 172.21.15.63:0/371276397 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c0021070 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.904+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc114170 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.904+0000 7f57c8c00700 1 -- 172.21.15.63:0/371276397 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57c00093b0 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.904+0000 7f57c8c00700 1 -- 172.21.15.63:0/371276397 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c000e070 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.904+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc114730 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.905+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc073c30 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.907+0000 7f57c8c00700 1 -- 172.21.15.63:0/371276397 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f57c001c490 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.907+0000 7f57c8c00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f57b40692b0 0x7f57b406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.907+0000 7f57c8c00700 1 -- 172.21.15.63:0/371276397 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f57c0088170 con 0x7f57cc0724f0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.908+0000 7f57cbe00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f57b40692b0 0x7f57b406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.908+0000 7f57cbe00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f57b40692b0 0x7f57b406b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f57cc115830 tx=0x7f57b800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:10.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:10.911+0000 7f57c8c00700 1 -- 172.21.15.63:0/371276397 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f57c00547c0 con 0x7f57cc0724f0 2024-03-23T11:43:11.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.034+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7c3c071aa0 con 0x7f7c3c070c70 2024-03-23T11:43:11.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.034+0000 7f7c39600700 1 -- 172.21.15.63:0/3073341050 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7c28056b60 con 0x7f7c3c070c70 2024-03-23T11:43:11.021 INFO:teuthology.orchestra.run.smithi063.stdout:111669149720 2024-03-23T11:43:11.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c240692b0 msgr2=0x7f7c2406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:11.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c240692b0 0x7f7c2406b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b800 tx=0x7f7c30009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:11.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070c70 msgr2=0x7f7c3c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070c70 0x7f7c3c1161c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800c8c0 tx=0x7f7c280044c0 comp rx=0 tx=0).stop 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 shutdown_connections 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c240692b0 0x7f7c2406b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c070360 0x7f7c3c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 --2- 172.21.15.63:0/3073341050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c3c070c70 0x7f7c3c1161c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 >> 172.21.15.63:0/3073341050 conn(0x7f7c3c0f89b0 msgr2=0x7f7c3c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.037+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 shutdown_connections 2024-03-23T11:43:11.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.038+0000 7f7c44000700 1 -- 172.21.15.63:0/3073341050 wait complete. 2024-03-23T11:43:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.126+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f57cc05f5f0 con 0x7f57cc0724f0 2024-03-23T11:43:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.127+0000 7f57c8c00700 1 -- 172.21.15.63:0/371276397 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f57c0057de0 con 0x7f57cc0724f0 2024-03-23T11:43:11.113 INFO:teuthology.orchestra.run.smithi063.stdout:38654705709 2024-03-23T11:43:11.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f57b40692b0 msgr2=0x7f57b406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f57b40692b0 0x7f57b406b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f57cc115830 tx=0x7f57b800b040 comp rx=0 tx=0).stop 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57cc0724f0 msgr2=0x7f57cc119f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57cc0724f0 0x7f57cc119f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f57c0000c00 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 shutdown_connections 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f57b40692b0 0x7f57b406b760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57cc0724f0 0x7f57cc119f50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 --2- 172.21.15.63:0/371276397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57cc072e00 0x7f57cc11a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 >> 172.21.15.63:0/371276397 conn(0x7f57cc0faba0 msgr2=0x7f57cc0faf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 shutdown_connections 2024-03-23T11:43:11.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:11.129+0000 7f57d3c00700 1 -- 172.21.15.63:0/371276397 wait complete. 2024-03-23T11:43:12.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:11 smithi063 ceph-mon[35189]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:12.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:11 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3073341050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-23T11:43:12.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:11 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/371276397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-23T11:43:12.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724556 for osd.5 2024-03-23T11:43:12.080 DEBUG:teuthology.parallel:result is None 2024-03-23T11:43:12.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:11 smithi111 ceph-mon[44005]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:12.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:11 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3073341050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-23T11:43:12.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:11 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/371276397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-23T11:43:12.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149718 got 111669149720 for osd.3 2024-03-23T11:43:12.778 DEBUG:teuthology.parallel:result is None 2024-03-23T11:43:13.933 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705707 got 38654705709 for osd.0 2024-03-23T11:43:13.934 DEBUG:teuthology.parallel:result is None 2024-03-23T11:43:14.285 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:13 smithi063 ceph-mon[35189]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:13 smithi111 ceph-mon[44005]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 -- 172.21.15.63:0/3837608293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fe830 msgr2=0x7f8980106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 --2- 172.21.15.63:0/3837608293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fe830 0x7f8980106f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f896c004940 tx=0x7f896c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 -- 172.21.15.63:0/3837608293 shutdown_connections 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 --2- 172.21.15.63:0/3837608293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fe830 0x7f8980106f00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 --2- 172.21.15.63:0/3837608293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89800fde90 0x7f89800fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 -- 172.21.15.63:0/3837608293 >> 172.21.15.63:0/3837608293 conn(0x7f8980072bc0 msgr2=0x7f8980072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 -- 172.21.15.63:0/3837608293 shutdown_connections 2024-03-23T11:43:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.632+0000 7f8988400700 1 -- 172.21.15.63:0/3837608293 wait complete. 2024-03-23T11:43:14.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.633+0000 7f8988400700 1 Processor -- start 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.633+0000 7f8988400700 1 -- start start 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8988400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fde90 0x7f89800fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8988400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89800fe830 0x7f89800fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8988400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89800fc820 con 0x7f89800fde90 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8988400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89800fc960 con 0x7f89800fe830 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8985a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fde90 0x7f89800fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8985a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fde90 0x7f89800fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57754/0 (socket says 172.21.15.63:57754) 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8985a00700 1 -- 172.21.15.63:0/765613663 learned_addr learned my addr 172.21.15.63:0/765613663 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8985000700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89800fe830 0x7f89800fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8985a00700 1 -- 172.21.15.63:0/765613663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89800fe830 msgr2=0x7f89800fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8985a00700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89800fe830 0x7f89800fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.634+0000 7f8985a00700 1 -- 172.21.15.63:0/765613663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c005040 con 0x7f89800fde90 2024-03-23T11:43:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.635+0000 7f8985a00700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fde90 0x7f89800fbd50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f897400c8c0 tx=0x7f89740044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.635+0000 7f897d600700 1 -- 172.21.15.63:0/765613663 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897400a980 con 0x7f89800fde90 2024-03-23T11:43:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.635+0000 7f897d600700 1 -- 172.21.15.63:0/765613663 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f897400ce60 con 0x7f89800fde90 2024-03-23T11:43:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.635+0000 7f897d600700 1 -- 172.21.15.63:0/765613663 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89740113a0 con 0x7f89800fde90 2024-03-23T11:43:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.635+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8980119de0 con 0x7f89800fde90 2024-03-23T11:43:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.635+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898011a1e0 con 0x7f89800fde90 2024-03-23T11:43:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.636+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8980065e30 con 0x7f89800fde90 2024-03-23T11:43:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.637+0000 7f897d600700 1 -- 172.21.15.63:0/765613663 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8974012400 con 0x7f89800fde90 2024-03-23T11:43:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.638+0000 7f897d600700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f89680692b0 0x7f896806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.638+0000 7f897d600700 1 -- 172.21.15.63:0/765613663 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8974086ea0 con 0x7f89800fde90 2024-03-23T11:43:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.638+0000 7f8985000700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f89680692b0 0x7f896806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:14.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.639+0000 7f8985000700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f89680692b0 0x7f896806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f89800fd690 tx=0x7f896c009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:14.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.643+0000 7f897d600700 1 -- 172.21.15.63:0/765613663 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f89740535b0 con 0x7f89800fde90 2024-03-23T11:43:14.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.848+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f898004bf60 con 0x7f89800fde90 2024-03-23T11:43:14.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.848+0000 7f897d600700 1 -- 172.21.15.63:0/765613663 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8974056bd0 con 0x7f89800fde90 2024-03-23T11:43:14.835 INFO:teuthology.orchestra.run.smithi063.stdout:81604378653 2024-03-23T11:43:14.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f89680692b0 msgr2=0x7f896806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f89680692b0 0x7f896806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f89800fd690 tx=0x7f896c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fde90 msgr2=0x7f89800fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fde90 0x7f89800fbd50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f897400c8c0 tx=0x7f89740044c0 comp rx=0 tx=0).stop 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 shutdown_connections 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f89680692b0 0x7f896806b760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89800fde90 0x7f89800fbd50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 --2- 172.21.15.63:0/765613663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89800fe830 0x7f89800fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 >> 172.21.15.63:0/765613663 conn(0x7f8980072bc0 msgr2=0x7f8980103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.851+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 shutdown_connections 2024-03-23T11:43:14.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:14.852+0000 7f8988400700 1 -- 172.21.15.63:0/765613663 wait complete. 2024-03-23T11:43:15.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:14 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/765613663' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-23T11:43:15.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:15.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:14 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/765613663' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-23T11:43:15.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:15.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378651 got 81604378653 for osd.2 2024-03-23T11:43:15.525 DEBUG:teuthology.parallel:result is None 2024-03-23T11:43:15.525 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-23T11:43:15.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-23T11:43:15.958 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:16.230 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:15 smithi063 ceph-mon[35189]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:16.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:15 smithi111 ceph-mon[44005]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:17.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 -- 172.21.15.63:0/626429388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff794102e00 msgr2=0x7ff7941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:17.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 --2- 172.21.15.63:0/626429388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff794102e00 0x7ff7941031d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff784004940 tx=0x7ff784009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:17.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 -- 172.21.15.63:0/626429388 shutdown_connections 2024-03-23T11:43:17.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 --2- 172.21.15.63:0/626429388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7941037a0 0x7ff7940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:17.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 --2- 172.21.15.63:0/626429388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff794102e00 0x7ff7941031d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:17.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 -- 172.21.15.63:0/626429388 >> 172.21.15.63:0/626429388 conn(0x7ff7940f89b0 msgr2=0x7ff7940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:17.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 -- 172.21.15.63:0/626429388 shutdown_connections 2024-03-23T11:43:17.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.785+0000 7ff79b000700 1 -- 172.21.15.63:0/626429388 wait complete. 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.786+0000 7ff79b000700 1 Processor -- start 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.786+0000 7ff79b000700 1 -- start start 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.786+0000 7ff79b000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794102e00 0x7ff794115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.786+0000 7ff79b000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7941037a0 0x7ff794116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.786+0000 7ff79b000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7941168a0 con 0x7ff7941037a0 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.786+0000 7ff79b000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79410fc70 con 0x7ff794102e00 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7941037a0 0x7ff794116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7941037a0 0x7ff794116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57770/0 (socket says 172.21.15.63:57770) 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793400700 1 -- 172.21.15.63:0/2091011221 learned_addr learned my addr 172.21.15.63:0/2091011221 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:17.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793e00700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794102e00 0x7ff794115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793400700 1 -- 172.21.15.63:0/2091011221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794102e00 msgr2=0x7ff794115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793400700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794102e00 0x7ff794115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793400700 1 -- 172.21.15.63:0/2091011221 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff784005040 con 0x7ff7941037a0 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.787+0000 7ff793400700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7941037a0 0x7ff794116250 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff78000c890 tx=0x7ff7800044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.788+0000 7ff78be00700 1 -- 172.21.15.63:0/2091011221 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78000a700 con 0x7ff7941037a0 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.788+0000 7ff78be00700 1 -- 172.21.15.63:0/2091011221 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff78000ad10 con 0x7ff7941037a0 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.788+0000 7ff78be00700 1 -- 172.21.15.63:0/2091011221 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff780012e70 con 0x7ff7941037a0 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.788+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79410ff50 con 0x7ff7941037a0 2024-03-23T11:43:17.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.788+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7941104c0 con 0x7ff7941037a0 2024-03-23T11:43:17.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.789+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7940fe550 con 0x7ff7941037a0 2024-03-23T11:43:17.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.790+0000 7ff78be00700 1 -- 172.21.15.63:0/2091011221 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff78000a860 con 0x7ff7941037a0 2024-03-23T11:43:17.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.791+0000 7ff78be00700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff7780692b0 0x7ff77806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:17.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.791+0000 7ff78be00700 1 -- 172.21.15.63:0/2091011221 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff780086e90 con 0x7ff7941037a0 2024-03-23T11:43:17.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.791+0000 7ff793e00700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff7780692b0 0x7ff77806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:17.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.792+0000 7ff793e00700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff7780692b0 0x7ff77806b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff784009fd0 tx=0x7ff784009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:17.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.796+0000 7ff78be00700 1 -- 172.21.15.63:0/2091011221 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff780056f70 con 0x7ff7941037a0 2024-03-23T11:43:17.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.997+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff794109e30 con 0x7ff7780692b0 2024-03-23T11:43:17.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:17.998+0000 7ff78be00700 1 -- 172.21.15.63:0/2091011221 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27803 (secure 0 0 0) 0x7ff794109e30 con 0x7ff7780692b0 2024-03-23T11:43:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:43:17.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff7780692b0 msgr2=0x7ff77806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:17.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff7780692b0 0x7ff77806b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff784009fd0 tx=0x7ff784009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:17.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7941037a0 msgr2=0x7ff794116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7941037a0 0x7ff794116250 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff78000c890 tx=0x7ff7800044c0 comp rx=0 tx=0).stop 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 shutdown_connections 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff7780692b0 0x7ff77806b760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794102e00 0x7ff794115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 --2- 172.21.15.63:0/2091011221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7941037a0 0x7ff794116250 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 >> 172.21.15.63:0/2091011221 conn(0x7ff7940f89b0 msgr2=0x7ff7940704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 shutdown_connections 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:18.001+0000 7ff79b000700 1 -- 172.21.15.63:0/2091011221 wait complete. 2024-03-23T11:43:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-03-23T11:43:18.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:17 smithi111 ceph-mon[44005]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:18.300 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:17 smithi063 ceph-mon[35189]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:18.655 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-03-23T11:43:16.850573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002126"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-23T11:42:22.993246+0000","last_change":"2024-03-23T11:41:51.893319+0000","last_active":"2024-03-23T11:42:22.993246+0000","last_peered":"2024-03-23T11:42:22.993246+0000","last_clean":"2024-03-23T11:42:22.993246+0000","last_became_active":"2024-03-23T11:41:51.892934+0000","last_became_peered":"2024-03-23T11:41:51.892934+0000","last_unstale":"2024-03-23T11:42:22.993246+0000","last_undegraded":"2024-03-23T11:42:22.993246+0000","last_fullsized":"2024-03-23T11:42:22.993246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:40:53.482701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:40:53.482701+0000","last_clean_scrub_stamp":"2024-03-23T11:40:53.482701+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:59:30.802891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.68100000000000005}]},{"osd":1,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.63100000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.749},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63900000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58699999999999997,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Mar 23 11:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Mar 23 11:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57199999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.34399999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.56499999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65800000000000003,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.40300000000000002}]},{"osd":3,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542182,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.29699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.36599999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Mar 23 11:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.313},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.36799999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.79100000000000004}]},{"osd":4,"last update":"Sat Mar 23 11:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.68000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:43:18.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-23T11:43:19.088 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:19.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:19 smithi063 ceph-mon[35189]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T11:43:19.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:19 smithi111 ceph-mon[44005]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T11:43:20.242 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:20 smithi063 ceph-mon[35189]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:20.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:20 smithi111 ceph-mon[44005]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:20.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 -- 172.21.15.63:0/2425800007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30104e60 msgr2=0x7fed30105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 --2- 172.21.15.63:0/2425800007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30104e60 0x7fed30105230 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fed20004940 tx=0x7fed20009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 -- 172.21.15.63:0/2425800007 shutdown_connections 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 --2- 172.21.15.63:0/2425800007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed30105800 0x7fed300fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 --2- 172.21.15.63:0/2425800007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30104e60 0x7fed30105230 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 -- 172.21.15.63:0/2425800007 >> 172.21.15.63:0/2425800007 conn(0x7fed300fab10 msgr2=0x7fed300fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 -- 172.21.15.63:0/2425800007 shutdown_connections 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.913+0000 7fed2f200700 1 -- 172.21.15.63:0/2425800007 wait complete. 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.914+0000 7fed2f200700 1 Processor -- start 2024-03-23T11:43:20.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.914+0000 7fed2f200700 1 -- start start 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed30104e60 0x7fed30115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30105800 0x7fed30116070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed301166c0 con 0x7fed30105800 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3010fab0 con 0x7fed30104e60 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30105800 0x7fed30116070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30105800 0x7fed30116070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57792/0 (socket says 172.21.15.63:57792) 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2d400700 1 -- 172.21.15.63:0/3298253631 learned_addr learned my addr 172.21.15.63:0/3298253631 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2de00700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed30104e60 0x7fed30115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2d400700 1 -- 172.21.15.63:0/3298253631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed30104e60 msgr2=0x7fed30115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2d400700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed30104e60 0x7fed30115b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:20.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.915+0000 7fed2d400700 1 -- 172.21.15.63:0/3298253631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20005040 con 0x7fed30105800 2024-03-23T11:43:20.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.916+0000 7fed2d400700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30105800 0x7fed30116070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed1c002a50 tx=0x7fed1c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:20.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.916+0000 7fed26000700 1 -- 172.21.15.63:0/3298253631 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c007cc0 con 0x7fed30105800 2024-03-23T11:43:20.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.916+0000 7fed26000700 1 -- 172.21.15.63:0/3298253631 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed1c00f040 con 0x7fed30105800 2024-03-23T11:43:20.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.916+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3010fd90 con 0x7fed30105800 2024-03-23T11:43:20.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.916+0000 7fed26000700 1 -- 172.21.15.63:0/3298253631 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c00d920 con 0x7fed30105800 2024-03-23T11:43:20.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.916+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed30110350 con 0x7fed30105800 2024-03-23T11:43:20.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.917+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed301005b0 con 0x7fed30105800 2024-03-23T11:43:20.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.918+0000 7fed26000700 1 -- 172.21.15.63:0/3298253631 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fed1c009680 con 0x7fed30105800 2024-03-23T11:43:20.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.919+0000 7fed26000700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fed140692b0 0x7fed1406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:20.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.919+0000 7fed26000700 1 -- 172.21.15.63:0/3298253631 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fed1c086b30 con 0x7fed30105800 2024-03-23T11:43:20.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.919+0000 7fed2de00700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fed140692b0 0x7fed1406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:20.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.919+0000 7fed2de00700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fed140692b0 0x7fed1406b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fed20009fd0 tx=0x7fed20009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:20.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:20.923+0000 7fed26000700 1 -- 172.21.15.63:0/3298253631 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fed1c056b90 con 0x7fed30105800 2024-03-23T11:43:21.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.124+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed3010be20 con 0x7fed140692b0 2024-03-23T11:43:21.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.126+0000 7fed26000700 1 -- 172.21.15.63:0/3298253631 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27803 (secure 0 0 0) 0x7fed3010be20 con 0x7fed140692b0 2024-03-23T11:43:21.113 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:43:21.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fed140692b0 msgr2=0x7fed1406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:21.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fed140692b0 0x7fed1406b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fed20009fd0 tx=0x7fed20009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30105800 msgr2=0x7fed30116070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30105800 0x7fed30116070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed1c002a50 tx=0x7fed1c004600 comp rx=0 tx=0).stop 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 shutdown_connections 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fed140692b0 0x7fed1406b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed30104e60 0x7fed30115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 --2- 172.21.15.63:0/3298253631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed30105800 0x7fed30116070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 >> 172.21.15.63:0/3298253631 conn(0x7fed300fab10 msgr2=0x7fed300fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 shutdown_connections 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:21.128+0000 7fed2f200700 1 -- 172.21.15.63:0/3298253631 wait complete. 2024-03-23T11:43:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-03-23T11:43:21.821 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-03-23T11:43:20.851160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001865"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-23T11:42:22.993246+0000","last_change":"2024-03-23T11:41:51.893319+0000","last_active":"2024-03-23T11:42:22.993246+0000","last_peered":"2024-03-23T11:42:22.993246+0000","last_clean":"2024-03-23T11:42:22.993246+0000","last_became_active":"2024-03-23T11:41:51.892934+0000","last_became_peered":"2024-03-23T11:41:51.892934+0000","last_unstale":"2024-03-23T11:42:22.993246+0000","last_undegraded":"2024-03-23T11:42:22.993246+0000","last_fullsized":"2024-03-23T11:42:22.993246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:40:53.482701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:40:53.482701+0000","last_clean_scrub_stamp":"2024-03-23T11:40:53.482701+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:59:30.802891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":4,"up_from":34,"seq":146028888084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Mar 23 11:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.48299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.40500000000000003}]},{"osd":1,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.443},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.47499999999999998}]},{"osd":2,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.499}]},{"osd":4,"last update":"Sat Mar 23 11:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.67000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378655,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58699999999999997,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46899999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.35999999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.66500000000000004}]},{"osd":4,"last update":"Sat Mar 23 11:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705711,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57199999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.439,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.34399999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.56499999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65800000000000003,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.82999999999999996}]},{"osd":4,"last update":"Sat Mar 23 11:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.76700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542183,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.29699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.36599999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Mar 23 11:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.495},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.51600000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:43:21.823 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-23T11:43:21.823 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-23T11:43:21.823 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-23T11:43:21.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-23T11:43:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:22 smithi063 ceph-mon[35189]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:22 smithi063 ceph-mon[35189]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T11:43:22.255 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:22.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:22 smithi111 ceph-mon[44005]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:22.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:22 smithi111 ceph-mon[44005]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-23T11:43:24.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:24 smithi063 ceph-mon[35189]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:24.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 -- 172.21.15.63:0/3415576953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 msgr2=0x7f5cb0109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:24.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/3415576953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0109110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c98004970 tx=0x7f5c98009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:24.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 -- 172.21.15.63:0/3415576953 shutdown_connections 2024-03-23T11:43:24.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/3415576953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0109110 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:24.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/3415576953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cb0070360 0x7f5cb0070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:24.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 -- 172.21.15.63:0/3415576953 >> 172.21.15.63:0/3415576953 conn(0x7f5cb00f89b0 msgr2=0x7f5cb00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:24.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 -- 172.21.15.63:0/3415576953 shutdown_connections 2024-03-23T11:43:24.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.158+0000 7f5cb6a00700 1 -- 172.21.15.63:0/3415576953 wait complete. 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.159+0000 7f5cb6a00700 1 Processor -- start 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.159+0000 7f5cb6a00700 1 -- start start 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.159+0000 7f5cb6a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cb0070360 0x7f5cb0115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.159+0000 7f5cb6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.159+0000 7f5cb6a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0116930 con 0x7f5cb0070c70 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.159+0000 7f5cb6a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb010fc70 con 0x7f5cb0070360 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5ca7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5ca7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38530/0 (socket says 172.21.15.63:38530) 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5ca7e00700 1 -- 172.21.15.63:0/4217249235 learned_addr learned my addr 172.21.15.63:0/4217249235 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:24.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5cafe00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cb0070360 0x7f5cb0115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5ca7e00700 1 -- 172.21.15.63:0/4217249235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cb0070360 msgr2=0x7f5cb0115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5ca7e00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cb0070360 0x7f5cb0115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5ca7e00700 1 -- 172.21.15.63:0/4217249235 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c98005040 con 0x7f5cb0070c70 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5cafe00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cb0070360 0x7f5cb0115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5ca7e00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0116250 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5c98004f50 tx=0x7f5c98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.160+0000 7f5cacc00700 1 -- 172.21.15.63:0/4217249235 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c9800e040 con 0x7f5cb0070c70 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.161+0000 7f5cacc00700 1 -- 172.21.15.63:0/4217249235 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c98013420 con 0x7f5cb0070c70 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.161+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb010fe90 con 0x7f5cb0070c70 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.161+0000 7f5cacc00700 1 -- 172.21.15.63:0/4217249235 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c9801c610 con 0x7f5cb0070c70 2024-03-23T11:43:24.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.161+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb01103d0 con 0x7f5cb0070c70 2024-03-23T11:43:24.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.162+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb0071aa0 con 0x7f5cb0070c70 2024-03-23T11:43:24.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.163+0000 7f5cacc00700 1 -- 172.21.15.63:0/4217249235 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5c98013590 con 0x7f5cb0070c70 2024-03-23T11:43:24.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.163+0000 7f5cacc00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5c940692b0 0x7f5c9406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:24.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.164+0000 7f5cacc00700 1 -- 172.21.15.63:0/4217249235 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5c98088060 con 0x7f5cb0070c70 2024-03-23T11:43:24.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.164+0000 7f5cafe00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5c940692b0 0x7f5c9406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:24.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.164+0000 7f5cafe00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5c940692b0 0x7f5c9406b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0006fd0 tx=0x7f5ca0008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:24.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.168+0000 7f5cacc00700 1 -- 172.21.15.63:0/4217249235 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5c980580e0 con 0x7f5cb0070c70 2024-03-23T11:43:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.416+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5cb010c0c0 con 0x7f5cb0070c70 2024-03-23T11:43:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.417+0000 7f5cacc00700 1 -- 172.21.15.63:0/4217249235 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5c98057c70 con 0x7f5cb0070c70 2024-03-23T11:43:24.403 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:43:24.403 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T11:43:24.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:24 smithi111 ceph-mon[44005]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:24.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5c940692b0 msgr2=0x7f5c9406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5c940692b0 0x7f5c9406b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0006fd0 tx=0x7f5ca0008040 comp rx=0 tx=0).stop 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 msgr2=0x7f5cb0116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0116250 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5c98004f50 tx=0x7f5c98009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 shutdown_connections 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f5c940692b0 0x7f5c9406b760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cb0070360 0x7f5cb0115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 --2- 172.21.15.63:0/4217249235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cb0070c70 0x7f5cb0116250 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 >> 172.21.15.63:0/4217249235 conn(0x7f5cb00f89b0 msgr2=0x7f5cb00f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 shutdown_connections 2024-03-23T11:43:24.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:24.419+0000 7f5cb6a00700 1 -- 172.21.15.63:0/4217249235 wait complete. 2024-03-23T11:43:25.064 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:25 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4217249235' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-23T11:43:25.064 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:25 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:43:25.088 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-23T11:43:25.088 INFO:tasks.cephadm:Setup complete, yielding 2024-03-23T11:43:25.089 INFO:teuthology.run_tasks:Running task print... 2024-03-23T11:43:25.100 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-23T11:43:25.100 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T11:43:25.154 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:43:25.154 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-23T11:43:25.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:25 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4217249235' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-23T11:43:25.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:25 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:43:25.599 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:26.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:26 smithi063 ceph-mon[35189]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:43:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:26 smithi111 ceph-mon[44005]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:43:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.562+0000 7f6292800700 1 -- 172.21.15.63:0/2181266384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 msgr2=0x7f628c0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.562+0000 7f6292800700 1 --2- 172.21.15.63:0/2181266384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c0fd310 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f627c004940 tx=0x7f627c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.563+0000 7f6292800700 1 -- 172.21.15.63:0/2181266384 shutdown_connections 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.563+0000 7f6292800700 1 --2- 172.21.15.63:0/2181266384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0fe0d0 0x7f628c0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.563+0000 7f6292800700 1 --2- 172.21.15.63:0/2181266384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c0fd310 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.563+0000 7f6292800700 1 -- 172.21.15.63:0/2181266384 >> 172.21.15.63:0/2181266384 conn(0x7f628c0f8560 msgr2=0x7f628c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.563+0000 7f6292800700 1 -- 172.21.15.63:0/2181266384 shutdown_connections 2024-03-23T11:43:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.563+0000 7f6292800700 1 -- 172.21.15.63:0/2181266384 wait complete. 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.563+0000 7f6292800700 1 Processor -- start 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f6292800700 1 -- start start 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f6292800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f6292800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0fe0d0 0x7f628c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f6292800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0708c0 con 0x7f628c0fcee0 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f6292800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c070a30 con 0x7f628c0fe0d0 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f628b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f628b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38550/0 (socket says 172.21.15.63:38550) 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.564+0000 7f628b400700 1 -- 172.21.15.63:0/3454655033 learned_addr learned my addr 172.21.15.63:0/3454655033 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f628aa00700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0fe0d0 0x7f628c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:27.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f628b400700 1 -- 172.21.15.63:0/3454655033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0fe0d0 msgr2=0x7f628c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f628b400700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0fe0d0 0x7f628c070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f628b400700 1 -- 172.21.15.63:0/3454655033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c005040 con 0x7f628c0fcee0 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f628b400700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c071e40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f627c000c00 tx=0x7f627c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f6283400700 1 -- 172.21.15.63:0/3454655033 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f627c00e070 con 0x7f628c0fcee0 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f6283400700 1 -- 172.21.15.63:0/3454655033 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f627c01c400 con 0x7f628c0fcee0 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c070cb0 con 0x7f628c0fcee0 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f6283400700 1 -- 172.21.15.63:0/3454655033 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f627c00b650 con 0x7f628c0fcee0 2024-03-23T11:43:27.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.565+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c115920 con 0x7f628c0fcee0 2024-03-23T11:43:27.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.566+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c05f5f0 con 0x7f628c0fcee0 2024-03-23T11:43:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.567+0000 7f6283400700 1 -- 172.21.15.63:0/3454655033 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f627c01c970 con 0x7f628c0fcee0 2024-03-23T11:43:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.568+0000 7f6283400700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f62700692b0 0x7f627006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.568+0000 7f6283400700 1 -- 172.21.15.63:0/3454655033 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f627c088420 con 0x7f628c0fcee0 2024-03-23T11:43:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.568+0000 7f628aa00700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f62700692b0 0x7f627006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.569+0000 7f628aa00700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f62700692b0 0x7f627006b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6278009fd0 tx=0x7f627800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.573+0000 7f6283400700 1 -- 172.21.15.63:0/3454655033 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f627c054a90 con 0x7f628c0fcee0 2024-03-23T11:43:27.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.775+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f628c04bed0 con 0x7f628c0fcee0 2024-03-23T11:43:27.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.779+0000 7f6283400700 1 -- 172.21.15.63:0/3454655033 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f627c004490 con 0x7f628c0fcee0 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f62700692b0 msgr2=0x7f627006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f62700692b0 0x7f627006b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6278009fd0 tx=0x7f627800b040 comp rx=0 tx=0).stop 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 msgr2=0x7f628c071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c071e40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f627c000c00 tx=0x7f627c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 shutdown_connections 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f62700692b0 0x7f627006b760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c0fcee0 0x7f628c071e40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:27.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 --2- 172.21.15.63:0/3454655033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0fe0d0 0x7f628c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:27.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 >> 172.21.15.63:0/3454655033 conn(0x7f628c0f8560 msgr2=0x7f628c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:27.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 shutdown_connections 2024-03-23T11:43:27.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:27.782+0000 7f6292800700 1 -- 172.21.15.63:0/3454655033 wait complete. 2024-03-23T11:43:28.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:28 smithi063 ceph-mon[35189]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:28.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:28 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3454655033' entity='client.admin' 2024-03-23T11:43:28.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:43:28.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:28.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:43:28.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:28 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:28.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:28 smithi111 ceph-mon[44005]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:28.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:28 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3454655033' entity='client.admin' 2024-03-23T11:43:28.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:43:28.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:28.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:43:28.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:28 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:28.495 INFO:teuthology.run_tasks:Running task print... 2024-03-23T11:43:28.505 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-23T11:43:28.506 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T11:43:28.514 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:43:28.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-23T11:43:28.946 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:30.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:30 smithi063 ceph-mon[35189]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:30.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:30 smithi111 ceph-mon[44005]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 -- 172.21.15.63:0/3483478890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 msgr2=0x7fd70c0fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 --2- 172.21.15.63:0/3483478890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c0fd290 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc004940 tx=0x7fd6fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 -- 172.21.15.63:0/3483478890 shutdown_connections 2024-03-23T11:43:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 --2- 172.21.15.63:0/3483478890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd70c0fdff0 0x7fd70c0fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 --2- 172.21.15.63:0/3483478890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c0fd290 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 -- 172.21.15.63:0/3483478890 >> 172.21.15.63:0/3483478890 conn(0x7fd70c0f8560 msgr2=0x7fd70c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 -- 172.21.15.63:0/3483478890 shutdown_connections 2024-03-23T11:43:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.929+0000 7fd712000700 1 -- 172.21.15.63:0/3483478890 wait complete. 2024-03-23T11:43:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd712000700 1 Processor -- start 2024-03-23T11:43:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd712000700 1 -- start start 2024-03-23T11:43:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd712000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c10fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd712000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd70c0fdff0 0x7fd70c110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd712000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c110690 con 0x7fd70c0fce80 2024-03-23T11:43:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd712000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70c110ee0 con 0x7fd70c0fdff0 2024-03-23T11:43:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd70b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c10fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd70b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c10fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38568/0 (socket says 172.21.15.63:38568) 2024-03-23T11:43:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.930+0000 7fd70b400700 1 -- 172.21.15.63:0/3134407440 learned_addr learned my addr 172.21.15.63:0/3134407440 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd70aa00700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd70c0fdff0 0x7fd70c110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd70b400700 1 -- 172.21.15.63:0/3134407440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd70c0fdff0 msgr2=0x7fd70c110090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd70b400700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd70c0fdff0 0x7fd70c110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd70b400700 1 -- 172.21.15.63:0/3134407440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc005040 con 0x7fd70c0fce80 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd70b400700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c10fb50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00b3b0 tx=0x7fd6fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd703400700 1 -- 172.21.15.63:0/3134407440 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6fc00e040 con 0x7fd70c0fce80 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd703400700 1 -- 172.21.15.63:0/3134407440 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6fc004290 con 0x7fd70c0fce80 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c111160 con 0x7fd70c0fce80 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.931+0000 7fd703400700 1 -- 172.21.15.63:0/3134407440 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6fc01c3f0 con 0x7fd70c0fce80 2024-03-23T11:43:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.932+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70c1116a0 con 0x7fd70c0fce80 2024-03-23T11:43:30.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.932+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70c05f5f0 con 0x7fd70c0fce80 2024-03-23T11:43:30.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.933+0000 7fd703400700 1 -- 172.21.15.63:0/3134407440 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd6fc013420 con 0x7fd70c0fce80 2024-03-23T11:43:30.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.934+0000 7fd703400700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd6f00692b0 0x7fd6f006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:30.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.934+0000 7fd703400700 1 -- 172.21.15.63:0/3134407440 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd6fc0881c0 con 0x7fd70c0fce80 2024-03-23T11:43:30.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.935+0000 7fd70aa00700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd6f00692b0 0x7fd6f006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:30.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.935+0000 7fd70aa00700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd6f00692b0 0x7fd6f006b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8004d60 tx=0x7fd6f800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:30.939+0000 7fd703400700 1 -- 172.21.15.63:0/3134407440 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd6fc0582c0 con 0x7fd70c0fce80 2024-03-23T11:43:31.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.162+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd70c111a40 con 0x7fd6f00692b0 2024-03-23T11:43:31.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.163+0000 7fd703400700 1 -- 172.21.15.63:0/3134407440 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd70c111a40 con 0x7fd6f00692b0 2024-03-23T11:43:31.150 INFO:teuthology.orchestra.run.smithi063.stdout:Backend: cephadm 2024-03-23T11:43:31.150 INFO:teuthology.orchestra.run.smithi063.stdout:Available: Yes 2024-03-23T11:43:31.150 INFO:teuthology.orchestra.run.smithi063.stdout:Paused: No 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd6f00692b0 msgr2=0x7fd6f006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd6f00692b0 0x7fd6f006b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8004d60 tx=0x7fd6f800b040 comp rx=0 tx=0).stop 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 msgr2=0x7fd70c10fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c10fb50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00b3b0 tx=0x7fd6fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 shutdown_connections 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd6f00692b0 0x7fd6f006b760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd70c0fce80 0x7fd70c10fb50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:31.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 --2- 172.21.15.63:0/3134407440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd70c0fdff0 0x7fd70c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:31.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 >> 172.21.15.63:0/3134407440 conn(0x7fd70c0f8560 msgr2=0x7fd70c0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:31.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 shutdown_connections 2024-03-23T11:43:31.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:31.166+0000 7fd712000700 1 -- 172.21.15.63:0/3134407440 wait complete. 2024-03-23T11:43:31.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-23T11:43:32.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:32 smithi063 ceph-mon[35189]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:32.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:32 smithi063 ceph-mon[35189]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:32.323 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:32.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:32 smithi111 ceph-mon[44005]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:32.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:32 smithi111 ceph-mon[44005]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1273370833 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 msgr2=0x7fc7dc0fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1273370833 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc0fe330 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8004940 tx=0x7fc7c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1273370833 shutdown_connections 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1273370833 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc0fe330 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1273370833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc0fcc10 0x7fc7dc0fd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1273370833 >> 172.21.15.63:0/1273370833 conn(0x7fc7dc0f83d0 msgr2=0x7fc7dc0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1273370833 shutdown_connections 2024-03-23T11:43:34.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.249+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1273370833 wait complete. 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.250+0000 7fc7e2c00700 1 Processor -- start 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.250+0000 7fc7e2c00700 1 -- start start 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.250+0000 7fc7e2c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc0fcc10 0x7fc7dc10f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.250+0000 7fc7e2c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc10fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e2c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc110490 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e2c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc110ce0 con 0x7fc7dc0fcc10 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e0e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc10fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e0e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc10fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53072/0 (socket says 172.21.15.63:53072) 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e0e00700 1 -- 172.21.15.63:0/1545873978 learned_addr learned my addr 172.21.15.63:0/1545873978 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:34.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e1800700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc0fcc10 0x7fc7dc10f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e0e00700 1 -- 172.21.15.63:0/1545873978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc0fcc10 msgr2=0x7fc7dc10f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e0e00700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc0fcc10 0x7fc7dc10f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e0e00700 1 -- 172.21.15.63:0/1545873978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8005040 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e1800700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc0fcc10 0x7fc7dc10f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.251+0000 7fc7e0e00700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc10fe90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800b3b0 tx=0x7fc7c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.252+0000 7fc7d9600700 1 -- 172.21.15.63:0/1545873978 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c800e040 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.252+0000 7fc7d9600700 1 -- 172.21.15.63:0/1545873978 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7c800bd20 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.252+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc110f60 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.252+0000 7fc7d9600700 1 -- 172.21.15.63:0/1545873978 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c801c630 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.252+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc1114a0 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.253+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc101850 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.254+0000 7fc7d9600700 1 -- 172.21.15.63:0/1545873978 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc7c8013420 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.255+0000 7fc7d9600700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc7c40692a0 0x7fc7c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:34.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.255+0000 7fc7d9600700 1 -- 172.21.15.63:0/1545873978 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc7c8087c10 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.256+0000 7fc7e1800700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc7c40692a0 0x7fc7c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:34.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.256+0000 7fc7e1800700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc7c40692a0 0x7fc7c406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0007fd0 tx=0x7fc7d0009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:34.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.260+0000 7fc7d9600700 1 -- 172.21.15.63:0/1545873978 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc7c8057c90 con 0x7fc7dc0fdec0 2024-03-23T11:43:34.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:34 smithi063 ceph-mon[35189]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:34.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:34 smithi111 ceph-mon[44005]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:34.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.474+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7dc1056d0 con 0x7fc7c40692a0 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.483+0000 7fc7d9600700 1 -- 172.21.15.63:0/1545873978 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7fc7dc1056d0 con 0x7fc7c40692a0 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (4m) 2m ago 5m 14.1M - 0.25.0 c8568f914cd2 c31e5bfcace0 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (6m) 2m ago 6m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (5m) 74s ago 5m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (4m) 2m ago 5m 74.6M - 9.4.7 954c08fa6188 d05e188def00 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:9283 running (7m) 2m ago 7m 470M - 17.2.7-726-g9c4e0c34 837124da804f 63833d132315 2024-03-23T11:43:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (4m) 74s ago 4m 430M - 17.2.7-726-g9c4e0c34 837124da804f f2f79acf25e6 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (7m) 2m ago 7m 39.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (4m) 74s ago 4m 37.3M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (6m) 2m ago 6m 7738k - 1.5.0 0da6a335fe13 e942e152db56 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (4m) 74s ago 4m 7432k - 1.5.0 0da6a335fe13 68f97d563d92 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (3m) 2m ago 3m 35.5M 4348M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (3m) 2m ago 3m 34.6M 4348M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (2m) 2m ago 2m 10.5M 4348M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (2m) 74s ago 2m 38.2M 5372M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (109s) 74s ago 109s 38.2M 5372M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (79s) 74s ago 79s 10.5M 5372M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:43:34.470 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (4m) 2m ago 5m 32.4M - 2.43.0 a07b618ecd1d 328fbe2c4719 2024-03-23T11:43:34.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc7c40692a0 msgr2=0x7fc7c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc7c40692a0 0x7fc7c406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0007fd0 tx=0x7fc7d0009040 comp rx=0 tx=0).stop 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 msgr2=0x7fc7dc10fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc10fe90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800b3b0 tx=0x7fc7c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 shutdown_connections 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc7c40692a0 0x7fc7c406b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc0fcc10 0x7fc7dc10f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 --2- 172.21.15.63:0/1545873978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7dc0fdec0 0x7fc7dc10fe90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.485+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 >> 172.21.15.63:0/1545873978 conn(0x7fc7dc0f83d0 msgr2=0x7fc7dc0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.486+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 shutdown_connections 2024-03-23T11:43:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:34.486+0000 7fc7e2c00700 1 -- 172.21.15.63:0/1545873978 wait complete. 2024-03-23T11:43:35.214 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-23T11:43:35.648 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:36.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:36 smithi063 ceph-mon[35189]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:36.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:36 smithi063 ceph-mon[35189]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:36.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:36 smithi111 ceph-mon[44005]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:36.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:36 smithi111 ceph-mon[44005]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 -- 172.21.15.63:0/910893243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fcee0 msgr2=0x7f96340fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 --2- 172.21.15.63:0/910893243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fcee0 0x7f96340fd310 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9628004940 tx=0x7f9628009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 -- 172.21.15.63:0/910893243 shutdown_connections 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 --2- 172.21.15.63:0/910893243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96340fe0d0 0x7f96340fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 --2- 172.21.15.63:0/910893243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fcee0 0x7f96340fd310 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 -- 172.21.15.63:0/910893243 >> 172.21.15.63:0/910893243 conn(0x7f96340f8560 msgr2=0x7f96340fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 -- 172.21.15.63:0/910893243 shutdown_connections 2024-03-23T11:43:37.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.552+0000 7f963be00700 1 -- 172.21.15.63:0/910893243 wait complete. 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.553+0000 7f963be00700 1 Processor -- start 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.553+0000 7f963be00700 1 -- start start 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f963be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96340fcee0 0x7f963410fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f963be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fe0d0 0x7f9634110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f963be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634110610 con 0x7f96340fe0d0 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f963be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634110e60 con 0x7f96340fcee0 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9638a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fe0d0 0x7f9634110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9638a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fe0d0 0x7f9634110010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53086/0 (socket says 172.21.15.63:53086) 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9638a00700 1 -- 172.21.15.63:0/2256221816 learned_addr learned my addr 172.21.15.63:0/2256221816 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9639400700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96340fcee0 0x7f963410fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9638a00700 1 -- 172.21.15.63:0/2256221816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96340fcee0 msgr2=0x7f963410fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9638a00700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96340fcee0 0x7f963410fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9638a00700 1 -- 172.21.15.63:0/2256221816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628005040 con 0x7f96340fe0d0 2024-03-23T11:43:37.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9639400700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96340fcee0 0x7f963410fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:43:37.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.554+0000 7f9638a00700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fe0d0 0x7f9634110010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f962000eb40 tx=0x7f962000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:37.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.555+0000 7f9631600700 1 -- 172.21.15.63:0/2256221816 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9620007cc0 con 0x7f96340fe0d0 2024-03-23T11:43:37.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.555+0000 7f9631600700 1 -- 172.21.15.63:0/2256221816 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f962000f040 con 0x7f96340fe0d0 2024-03-23T11:43:37.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.555+0000 7f9631600700 1 -- 172.21.15.63:0/2256221816 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9620009570 con 0x7f96340fe0d0 2024-03-23T11:43:37.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.555+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96341110e0 con 0x7f96340fe0d0 2024-03-23T11:43:37.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.555+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9634111620 con 0x7f96340fe0d0 2024-03-23T11:43:37.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.556+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9634101a60 con 0x7f96340fe0d0 2024-03-23T11:43:37.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.557+0000 7f9631600700 1 -- 172.21.15.63:0/2256221816 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f96200096d0 con 0x7f96340fe0d0 2024-03-23T11:43:37.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.558+0000 7f9631600700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f961c069360 0x7f961c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:37.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.558+0000 7f9631600700 1 -- 172.21.15.63:0/2256221816 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9620086a70 con 0x7f96340fe0d0 2024-03-23T11:43:37.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.558+0000 7f9639400700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f961c069360 0x7f961c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:37.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.559+0000 7f9639400700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f961c069360 0x7f961c06b810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9628013680 tx=0x7f9628009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:37.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.562+0000 7f9631600700 1 -- 172.21.15.63:0/2256221816 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f962004e190 con 0x7f96340fe0d0 2024-03-23T11:43:37.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.777+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9634111890 con 0x7f961c069360 2024-03-23T11:43:37.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.781+0000 7f9631600700 1 -- 172.21.15.63:0/2256221816 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f9634111890 con 0x7f961c069360 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:crash 2/2 2m ago 6m * 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:mgr 2/2 2m ago 6m count:2 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:mon 2/2 2m ago 5m smithi063:172.21.15.63=smithi063;smithi111:172.21.15.111=smithi111;count:2 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:osd 6 2m ago - 2024-03-23T11:43:37.768 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-23T11:43:37.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f961c069360 msgr2=0x7f961c06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:37.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f961c069360 0x7f961c06b810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9628013680 tx=0x7f9628009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fe0d0 msgr2=0x7f9634110010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fe0d0 0x7f9634110010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f962000eb40 tx=0x7f962000b040 comp rx=0 tx=0).stop 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 shutdown_connections 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f961c069360 0x7f961c06b810 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96340fcee0 0x7f963410fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 --2- 172.21.15.63:0/2256221816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96340fe0d0 0x7f9634110010 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 >> 172.21.15.63:0/2256221816 conn(0x7f96340f8560 msgr2=0x7f96340f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 shutdown_connections 2024-03-23T11:43:37.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:37.784+0000 7f963be00700 1 -- 172.21.15.63:0/2256221816 wait complete. 2024-03-23T11:43:38.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:38 smithi063 ceph-mon[35189]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:38.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:38 smithi111 ceph-mon[44005]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:38.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-23T11:43:38.953 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:39.301 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:39 smithi063 ceph-mon[35189]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:39.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:39 smithi111 ceph-mon[44005]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:40.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:40 smithi063 ceph-mon[35189]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:40.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:40 smithi111 ceph-mon[44005]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:40.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 -- 172.21.15.63:0/1592017673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad000709a0 msgr2=0x7fad00070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:40.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 --2- 172.21.15.63:0/1592017673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad000709a0 0x7fad00070e10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7facec004940 tx=0x7facec009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:40.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 -- 172.21.15.63:0/1592017673 shutdown_connections 2024-03-23T11:43:40.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 --2- 172.21.15.63:0/1592017673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad000709a0 0x7fad00070e10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:40.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 --2- 172.21.15.63:0/1592017673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad00071f50 0x7fad00070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:40.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 -- 172.21.15.63:0/1592017673 >> 172.21.15.63:0/1592017673 conn(0x7fad000f8560 msgr2=0x7fad000fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:40.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 -- 172.21.15.63:0/1592017673 shutdown_connections 2024-03-23T11:43:40.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.904+0000 7fad07c00700 1 -- 172.21.15.63:0/1592017673 wait complete. 2024-03-23T11:43:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7fad07c00700 1 Processor -- start 2024-03-23T11:43:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7fad07c00700 1 -- start start 2024-03-23T11:43:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7fad07c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad000709a0 0x7fad0010fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7fad07c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad00071f50 0x7fad00110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7fad07c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00110650 con 0x7fad00071f50 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7fad07c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00110ea0 con 0x7fad000709a0 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7facffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad00071f50 0x7fad00110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7facffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad00071f50 0x7fad00110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53112/0 (socket says 172.21.15.63:53112) 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.905+0000 7facffe00700 1 -- 172.21.15.63:0/207916934 learned_addr learned my addr 172.21.15.63:0/207916934 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7fad05200700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad000709a0 0x7fad0010fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7facffe00700 1 -- 172.21.15.63:0/207916934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad000709a0 msgr2=0x7fad0010fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7facffe00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad000709a0 0x7fad0010fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7facffe00700 1 -- 172.21.15.63:0/207916934 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec005040 con 0x7fad00071f50 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7fad05200700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad000709a0 0x7fad0010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7facffe00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad00071f50 0x7fad00110050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7facec009380 tx=0x7facec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7facfcc00700 1 -- 172.21.15.63:0/207916934 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facec00e040 con 0x7fad00071f50 2024-03-23T11:43:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7facfcc00700 1 -- 172.21.15.63:0/207916934 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facec00bd20 con 0x7fad00071f50 2024-03-23T11:43:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7facfcc00700 1 -- 172.21.15.63:0/207916934 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facec01c630 con 0x7fad00071f50 2024-03-23T11:43:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad00111120 con 0x7fad00071f50 2024-03-23T11:43:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.906+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad00111660 con 0x7fad00071f50 2024-03-23T11:43:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.907+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0005f5f0 con 0x7fad00071f50 2024-03-23T11:43:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.908+0000 7facfcc00700 1 -- 172.21.15.63:0/207916934 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7facec013420 con 0x7fad00071f50 2024-03-23T11:43:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.909+0000 7facfcc00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7face80692a0 0x7face806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.909+0000 7facfcc00700 1 -- 172.21.15.63:0/207916934 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7facec087bc0 con 0x7fad00071f50 2024-03-23T11:43:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.909+0000 7fad05200700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7face80692a0 0x7face806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:40.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.910+0000 7fad05200700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7face80692a0 0x7face806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7facf4009fd0 tx=0x7facf400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:40.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:40.913+0000 7facfcc00700 1 -- 172.21.15.63:0/207916934 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7facec057ca0 con 0x7fad00071f50 2024-03-23T11:43:41.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.126+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fad001119e0 con 0x7face80692a0 2024-03-23T11:43:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.127+0000 7facfcc00700 1 -- 172.21.15.63:0/207916934 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fad001119e0 con 0x7face80692a0 2024-03-23T11:43:41.117 INFO:teuthology.orchestra.run.smithi063.stdout:HOST ADDR LABELS STATUS 2024-03-23T11:43:41.118 INFO:teuthology.orchestra.run.smithi063.stdout:smithi063 172.21.15.63 2024-03-23T11:43:41.118 INFO:teuthology.orchestra.run.smithi063.stdout:smithi111 172.21.15.111 2024-03-23T11:43:41.118 INFO:teuthology.orchestra.run.smithi063.stdout:2 hosts in cluster 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7face80692a0 msgr2=0x7face806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7face80692a0 0x7face806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7facf4009fd0 tx=0x7facf400b040 comp rx=0 tx=0).stop 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad00071f50 msgr2=0x7fad00110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad00071f50 0x7fad00110050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7facec009380 tx=0x7facec009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 shutdown_connections 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7face80692a0 0x7face806b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad000709a0 0x7fad0010fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 --2- 172.21.15.63:0/207916934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad00071f50 0x7fad00110050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 >> 172.21.15.63:0/207916934 conn(0x7fad000f8560 msgr2=0x7fad000f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 shutdown_connections 2024-03-23T11:43:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:41.130+0000 7fad07c00700 1 -- 172.21.15.63:0/207916934 wait complete. 2024-03-23T11:43:41.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-23T11:43:42.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:42 smithi063 ceph-mon[35189]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:42.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:42 smithi063 ceph-mon[35189]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:42.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:42 smithi111 ceph-mon[44005]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:42.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:42 smithi111 ceph-mon[44005]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:44.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 -- 172.21.15.63:0/107967949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 msgr2=0x7fc9f0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:44.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/107967949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f0070460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4004940 tx=0x7fc9e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:44.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 -- 172.21.15.63:0/107967949 shutdown_connections 2024-03-23T11:43:44.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/107967949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00709a0 0x7fc9f0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:44.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/107967949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f0070460 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:44.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 -- 172.21.15.63:0/107967949 >> 172.21.15.63:0/107967949 conn(0x7fc9f00f8560 msgr2=0x7fc9f00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:44.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 -- 172.21.15.63:0/107967949 shutdown_connections 2024-03-23T11:43:44.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.128+0000 7fc9f7e00700 1 -- 172.21.15.63:0/107967949 wait complete. 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.129+0000 7fc9f7e00700 1 Processor -- start 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.129+0000 7fc9f7e00700 1 -- start start 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.129+0000 7fc9f7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00709a0 0x7fc9f0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.129+0000 7fc9f7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f01144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.129+0000 7fc9f7e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f0114aa0 con 0x7fc9f0071f50 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.129+0000 7fc9f7e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f01152f0 con 0x7fc9f00709a0 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.129+0000 7fc9f4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f01144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f01144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59270/0 (socket says 172.21.15.63:59270) 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f4a00700 1 -- 172.21.15.63:0/2866522798 learned_addr learned my addr 172.21.15.63:0/2866522798 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f5400700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00709a0 0x7fc9f0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f4a00700 1 -- 172.21.15.63:0/2866522798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00709a0 msgr2=0x7fc9f0113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:44.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f4a00700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00709a0 0x7fc9f0113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f4a00700 1 -- 172.21.15.63:0/2866522798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e4005040 con 0x7fc9f0071f50 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f4a00700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f01144a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc002a50 tx=0x7fc9dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9ed600700 1 -- 172.21.15.63:0/2866522798 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9dc007cc0 con 0x7fc9f0071f50 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f01155d0 con 0x7fc9f0071f50 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.130+0000 7fc9ed600700 1 -- 172.21.15.63:0/2866522798 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9dc00f040 con 0x7fc9f0071f50 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.131+0000 7fc9ed600700 1 -- 172.21.15.63:0/2866522798 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9dc00d9c0 con 0x7fc9f0071f50 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.131+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f0115b10 con 0x7fc9f0071f50 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.132+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f005f5f0 con 0x7fc9f0071f50 2024-03-23T11:43:44.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.133+0000 7fc9ed600700 1 -- 172.21.15.63:0/2866522798 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc9dc00db20 con 0x7fc9f0071f50 2024-03-23T11:43:44.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.133+0000 7fc9ed600700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc9d80692b0 0x7fc9d806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:44.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.133+0000 7fc9ed600700 1 -- 172.21.15.63:0/2866522798 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc9dc086c10 con 0x7fc9f0071f50 2024-03-23T11:43:44.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.134+0000 7fc9f5400700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc9d80692b0 0x7fc9d806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:44.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.134+0000 7fc9f5400700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc9d80692b0 0x7fc9d806b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400b3b0 tx=0x7fc9e4009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:44.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.138+0000 7fc9ed600700 1 -- 172.21.15.63:0/2866522798 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc9dc056bf0 con 0x7fc9f0071f50 2024-03-23T11:43:44.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:44 smithi063 ceph-mon[35189]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:44.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.350+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9f0115e90 con 0x7fc9d80692b0 2024-03-23T11:43:44.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.351+0000 7fc9ed600700 1 -- 172.21.15.63:0/2866522798 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fc9f0115e90 con 0x7fc9d80692b0 2024-03-23T11:43:44.338 INFO:teuthology.orchestra.run.smithi063.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-23T11:43:44.338 INFO:teuthology.orchestra.run.smithi063.stdout:smithi063 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5344005B400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-23T11:43:44.338 INFO:teuthology.orchestra.run.smithi063.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 372G No 80s ago Has a FileSystem, LVM detected 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.353+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc9d80692b0 msgr2=0x7fc9d806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.353+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc9d80692b0 0x7fc9d806b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400b3b0 tx=0x7fc9e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.353+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 msgr2=0x7fc9f01144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.353+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f01144a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc002a50 tx=0x7fc9dc0046b0 comp rx=0 tx=0).stop 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.354+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 shutdown_connections 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.354+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fc9d80692b0 0x7fc9d806b760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.354+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00709a0 0x7fc9f0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.354+0000 7fc9f7e00700 1 --2- 172.21.15.63:0/2866522798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9f0071f50 0x7fc9f01144a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.354+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 >> 172.21.15.63:0/2866522798 conn(0x7fc9f00f8560 msgr2=0x7fc9f00f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.354+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 shutdown_connections 2024-03-23T11:43:44.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:44.354+0000 7fc9f7e00700 1 -- 172.21.15.63:0/2866522798 wait complete. 2024-03-23T11:43:44.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:44 smithi111 ceph-mon[44005]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:45.063 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T11:43:45.072 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:43:45.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-23T11:43:45.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:45.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:43:45.511 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:46.298 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:46 smithi063 ceph-mon[35189]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:46.298 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:46 smithi063 ceph-mon[35189]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:46 smithi111 ceph-mon[44005]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:46 smithi111 ceph-mon[44005]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 -- 172.21.15.63:0/1942972541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800fff00 msgr2=0x7f1280102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 --2- 172.21.15.63:0/1942972541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800fff00 0x7f1280102320 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f126c004940 tx=0x7f126c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 -- 172.21.15.63:0/1942972541 shutdown_connections 2024-03-23T11:43:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 --2- 172.21.15.63:0/1942972541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800fff00 0x7f1280102320 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 --2- 172.21.15.63:0/1942972541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12800662e0 0x7f12800ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 -- 172.21.15.63:0/1942972541 >> 172.21.15.63:0/1942972541 conn(0x7f12800f7430 msgr2=0x7f12800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 -- 172.21.15.63:0/1942972541 shutdown_connections 2024-03-23T11:43:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.378+0000 7f1284a00700 1 -- 172.21.15.63:0/1942972541 wait complete. 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.379+0000 7f1284a00700 1 Processor -- start 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.379+0000 7f1284a00700 1 -- start start 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.379+0000 7f1284a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800662e0 0x7f128010fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.379+0000 7f1284a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12800fff00 0x7f1280110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.379+0000 7f1284a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280110720 con 0x7f12800662e0 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.379+0000 7f1284a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280110f10 con 0x7f12800fff00 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.379+0000 7f127de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800662e0 0x7f128010fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800662e0 0x7f128010fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59284/0 (socket says 172.21.15.63:59284) 2024-03-23T11:43:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127de00700 1 -- 172.21.15.63:0/601151327 learned_addr learned my addr 172.21.15.63:0/601151327 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127d400700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12800fff00 0x7f1280110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127de00700 1 -- 172.21.15.63:0/601151327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12800fff00 msgr2=0x7f1280110090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127de00700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12800fff00 0x7f1280110090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127de00700 1 -- 172.21.15.63:0/601151327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c005040 con 0x7f12800662e0 2024-03-23T11:43:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127d400700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12800fff00 0x7f1280110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:43:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f127de00700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800662e0 0x7f128010fb50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f127000b790 tx=0x7f1270009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f1276000700 1 -- 172.21.15.63:0/601151327 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f127000f6f0 con 0x7f12800662e0 2024-03-23T11:43:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12801111f0 con 0x7f12800662e0 2024-03-23T11:43:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.380+0000 7f1276000700 1 -- 172.21.15.63:0/601151327 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f127000fd00 con 0x7f12800662e0 2024-03-23T11:43:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.381+0000 7f1276000700 1 -- 172.21.15.63:0/601151327 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12700103a0 con 0x7f12800662e0 2024-03-23T11:43:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.381+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12801117b0 con 0x7f12800662e0 2024-03-23T11:43:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.382+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128005f5f0 con 0x7f12800662e0 2024-03-23T11:43:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.382+0000 7f1276000700 1 -- 172.21.15.63:0/601151327 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f127000f850 con 0x7f12800662e0 2024-03-23T11:43:47.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.383+0000 7f1276000700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f12640692b0 0x7f126406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:47.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.383+0000 7f1276000700 1 -- 172.21.15.63:0/601151327 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1270086d60 con 0x7f12800662e0 2024-03-23T11:43:47.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.384+0000 7f127d400700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f12640692b0 0x7f126406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.384+0000 7f127d400700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f12640692b0 0x7f126406b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f126c004f90 tx=0x7f126c009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.388+0000 7f1276000700 1 -- 172.21.15.63:0/601151327 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f127008b070 con 0x7f12800662e0 2024-03-23T11:43:47.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:47.627+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1280111af0 con 0x7f12640692b0 2024-03-23T11:43:48.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:48 smithi063 ceph-mon[35189]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:48.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:48 smithi111 ceph-mon[44005]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:49.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.235+0000 7f1276000700 1 -- 172.21.15.63:0/601151327 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1280111af0 con 0x7f12640692b0 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.238+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f12640692b0 msgr2=0x7f126406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.238+0000 7f1284a00700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f12640692b0 0x7f126406b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f126c004f90 tx=0x7f126c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800662e0 msgr2=0x7f128010fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800662e0 0x7f128010fb50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f127000b790 tx=0x7f1270009220 comp rx=0 tx=0).stop 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 shutdown_connections 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f12640692b0 0x7f126406b760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12800662e0 0x7f128010fb50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 --2- 172.21.15.63:0/601151327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12800fff00 0x7f1280110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 >> 172.21.15.63:0/601151327 conn(0x7f12800f7430 msgr2=0x7f12800fbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:49.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 shutdown_connections 2024-03-23T11:43:49.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:49.239+0000 7f1284a00700 1 -- 172.21.15.63:0/601151327 wait complete. 2024-03-23T11:43:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:49 smithi111 ceph-mon[44005]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:49 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-23T11:43:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:49 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-23T11:43:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:49 smithi111 ceph-mon[44005]: osdmap e41: 6 total, 6 up, 6 in 2024-03-23T11:43:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:49 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-23T11:43:49.580 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:49 smithi063 ceph-mon[35189]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:43:49.580 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:49 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-23T11:43:49.580 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:49 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-23T11:43:49.580 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:49 smithi063 ceph-mon[35189]: osdmap e41: 6 total, 6 up, 6 in 2024-03-23T11:43:49.580 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:49 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-23T11:43:49.580 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:49 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[35166]: 2024-03-23T11:43:49.208+0000 7f6c57e00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:43:50.038 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-23T11:43:50.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: pgmap v196: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: osdmap e42: 6 total, 6 up, 6 in 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: osdmap e43: 6 total, 6 up, 6 in 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: fsmap cephfs:0 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: Saving service mds.cephfs spec with placement count:4 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:43:50.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:50.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:43:50.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:43:50.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:50.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:50 smithi063 ceph-mon[35189]: Deploying daemon mds.cephfs.smithi063.brtzgy on smithi063 2024-03-23T11:43:50.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: pgmap v196: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:50.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-23T11:43:50.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: osdmap e42: 6 total, 6 up, 6 in 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: osdmap e43: 6 total, 6 up, 6 in 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: fsmap cephfs:0 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: Saving service mds.cephfs spec with placement count:4 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:43:50.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:50.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:50 smithi111 ceph-mon[44005]: Deploying daemon mds.cephfs.smithi063.brtzgy on smithi063 2024-03-23T11:43:51.196 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:51.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:51 smithi063 ceph-mon[35189]: osdmap e44: 6 total, 6 up, 6 in 2024-03-23T11:43:51.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:51 smithi111 ceph-mon[44005]: osdmap e44: 6 total, 6 up, 6 in 2024-03-23T11:43:52.461 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:52 smithi063 ceph-mon[35189]: pgmap v200: 65 pgs: 11 creating+peering, 11 active+clean, 43 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:52.462 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:52 smithi063 ceph-mon[35189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-23T11:43:52.462 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:52 smithi063 ceph-mon[35189]: osdmap e45: 6 total, 6 up, 6 in 2024-03-23T11:43:52.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:52 smithi111 ceph-mon[44005]: pgmap v200: 65 pgs: 11 creating+peering, 11 active+clean, 43 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:52.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:52 smithi111 ceph-mon[44005]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-23T11:43:52.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:52 smithi111 ceph-mon[44005]: osdmap e45: 6 total, 6 up, 6 in 2024-03-23T11:43:54.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 -- 172.21.15.63:0/1284900812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c4100290 msgr2=0x7f06c4100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:54.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 --2- 172.21.15.63:0/1284900812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c4100290 0x7f06c4100700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f06b0004940 tx=0x7f06b0009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:54.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 -- 172.21.15.63:0/1284900812 shutdown_connections 2024-03-23T11:43:54.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 --2- 172.21.15.63:0/1284900812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c4100290 0x7f06c4100700 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:54.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 --2- 172.21.15.63:0/1284900812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06c40fefe0 0x7f06c40ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:54.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 -- 172.21.15.63:0/1284900812 >> 172.21.15.63:0/1284900812 conn(0x7f06c40fa790 msgr2=0x7f06c40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:54.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 -- 172.21.15.63:0/1284900812 shutdown_connections 2024-03-23T11:43:54.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.311+0000 7f06cb800700 1 -- 172.21.15.63:0/1284900812 wait complete. 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.312+0000 7f06cb800700 1 Processor -- start 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.312+0000 7f06cb800700 1 -- start start 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06cb800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c40fefe0 0x7f06c410f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06cb800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06c4100290 0x7f06c410fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06cb800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c41104d0 con 0x7f06c40fefe0 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06cb800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4110640 con 0x7f06c4100290 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c40fefe0 0x7f06c410f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c40fefe0 0x7f06c410f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58112/0 (socket says 172.21.15.63:58112) 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c8e00700 1 -- 172.21.15.63:0/3282877951 learned_addr learned my addr 172.21.15.63:0/3282877951 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:54.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c3e00700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06c4100290 0x7f06c410fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c8e00700 1 -- 172.21.15.63:0/3282877951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06c4100290 msgr2=0x7f06c410fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c8e00700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06c4100290 0x7f06c410fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c8e00700 1 -- 172.21.15.63:0/3282877951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b0005040 con 0x7f06c40fefe0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.313+0000 7f06c8e00700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c40fefe0 0x7f06c410f990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f06b8002a80 tx=0x7f06b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.314+0000 7f06c0c00700 1 -- 172.21.15.63:0/3282877951 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b8007cc0 con 0x7f06c40fefe0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.314+0000 7f06c0c00700 1 -- 172.21.15.63:0/3282877951 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06b800f040 con 0x7f06c40fefe0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.314+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c41150a0 con 0x7f06c40fefe0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.314+0000 7f06c0c00700 1 -- 172.21.15.63:0/3282877951 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b800da60 con 0x7f06c40fefe0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.314+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c4115660 con 0x7f06c40fefe0 2024-03-23T11:43:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.315+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c404c710 con 0x7f06c40fefe0 2024-03-23T11:43:54.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.316+0000 7f06c0c00700 1 -- 172.21.15.63:0/3282877951 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f06b800dbc0 con 0x7f06c40fefe0 2024-03-23T11:43:54.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.317+0000 7f06c0c00700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06ac0692a0 0x7f06ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:54.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.317+0000 7f06c0c00700 1 -- 172.21.15.63:0/3282877951 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f06b8086d40 con 0x7f06c40fefe0 2024-03-23T11:43:54.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.317+0000 7f06c3e00700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06ac0692a0 0x7f06ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:54.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.318+0000 7f06c3e00700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06ac0692a0 0x7f06ac06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f06b0004f90 tx=0x7f06b0009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:54.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.322+0000 7f06c0c00700 1 -- 172.21.15.63:0/3282877951 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f06b8056b00 con 0x7f06c40fefe0 2024-03-23T11:43:54.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.575+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f06c4115b90 con 0x7f06c40fefe0 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.576+0000 7f06c0c00700 1 -- 172.21.15.63:0/3282877951 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7f06b8056690 con 0x7f06c40fefe0 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:e4 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 4 2024-03-23T11:43:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:43:54.230596+0000 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 0 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 1 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:in 0 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462} 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:43:54.563 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 0 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:creating seq 1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:43:54.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06ac0692a0 msgr2=0x7f06ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06ac0692a0 0x7f06ac06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f06b0004f90 tx=0x7f06b0009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c40fefe0 msgr2=0x7f06c410f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c40fefe0 0x7f06c410f990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f06b8002a80 tx=0x7f06b80046b0 comp rx=0 tx=0).stop 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 shutdown_connections 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06ac0692a0 0x7f06ac06b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06c40fefe0 0x7f06c410f990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 --2- 172.21.15.63:0/3282877951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06c4100290 0x7f06c410fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 >> 172.21.15.63:0/3282877951 conn(0x7f06c40fa790 msgr2=0x7f06c40fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 shutdown_connections 2024-03-23T11:43:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:54.578+0000 7f06cb800700 1 -- 172.21.15.63:0/3282877951 wait complete. 2024-03-23T11:43:54.566 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 4 2024-03-23T11:43:54.586 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:54 smithi063 ceph-mon[35189]: pgmap v202: 65 pgs: 11 creating+peering, 39 active+clean, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:54.586 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:54.586 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:54.586 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:54.586 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:43:54.586 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:43:54.586 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:54 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:54 smithi111 ceph-mon[44005]: pgmap v202: 65 pgs: 11 creating+peering, 39 active+clean, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:43:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:54.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:43:54.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:43:54.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:54 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:55.456 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T11:43:55.466 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:43:55.466 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-23T11:43:55.530 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: Deploying daemon mds.cephfs.smithi111.ylnmzy on smithi111 2024-03-23T11:43:55.530 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] up:boot 2024-03-23T11:43:55.530 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: daemon mds.cephfs.smithi063.brtzgy assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: Cluster is now healthy 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: fsmap cephfs:0 1 up:standby 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:creating} 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: daemon mds.cephfs.smithi063.brtzgy is now active in filesystem cephfs as rank 0 2024-03-23T11:43:55.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:55 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3282877951' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:43:55.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: Deploying daemon mds.cephfs.smithi111.ylnmzy on smithi111 2024-03-23T11:43:55.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] up:boot 2024-03-23T11:43:55.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: daemon mds.cephfs.smithi063.brtzgy assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-23T11:43:55.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-23T11:43:55.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-23T11:43:55.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: Cluster is now healthy 2024-03-23T11:43:55.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: fsmap cephfs:0 1 up:standby 2024-03-23T11:43:55.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:43:55.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:creating} 2024-03-23T11:43:55.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: daemon mds.cephfs.smithi063.brtzgy is now active in filesystem cephfs as rank 0 2024-03-23T11:43:55.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:55 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3282877951' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:43:55.903 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:43:56.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:56 smithi111 ceph-mon[44005]: pgmap v203: 65 pgs: 5 creating+peering, 60 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-23T11:43:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:56 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] up:active 2024-03-23T11:43:56.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:56 smithi111 ceph-mon[44005]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 2024-03-23T11:43:56.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:56 smithi063 ceph-mon[35189]: pgmap v203: 65 pgs: 5 creating+peering, 60 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-23T11:43:56.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:56 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] up:active 2024-03-23T11:43:56.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:56 smithi063 ceph-mon[35189]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 2024-03-23T11:43:57.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 -- 172.21.15.63:0/3390063070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fe0d0 msgr2=0x7f7f200fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 --2- 172.21.15.63:0/3390063070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fe0d0 0x7f7f200fe540 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c004940 tx=0x7f7f0c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 -- 172.21.15.63:0/3390063070 shutdown_connections 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 --2- 172.21.15.63:0/3390063070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fe0d0 0x7f7f200fe540 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 --2- 172.21.15.63:0/3390063070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f200fcee0 0x7f7f200fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 -- 172.21.15.63:0/3390063070 >> 172.21.15.63:0/3390063070 conn(0x7f7f200f8560 msgr2=0x7f7f200fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 -- 172.21.15.63:0/3390063070 shutdown_connections 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.893+0000 7f7f27c00700 1 -- 172.21.15.63:0/3390063070 wait complete. 2024-03-23T11:43:57.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.894+0000 7f7f27c00700 1 Processor -- start 2024-03-23T11:43:57.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.894+0000 7f7f27c00700 1 -- start start 2024-03-23T11:43:57.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f27c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fcee0 0x7f7f2010d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:57.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f27c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f200fe0d0 0x7f7f2010deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:57.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f27c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2010e4b0 con 0x7f7f200fcee0 2024-03-23T11:43:57.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f27c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2010ed00 con 0x7f7f200fe0d0 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f25200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fcee0 0x7f7f2010d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f25200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fcee0 0x7f7f2010d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58130/0 (socket says 172.21.15.63:58130) 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f25200700 1 -- 172.21.15.63:0/2532201750 learned_addr learned my addr 172.21.15.63:0/2532201750 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f1fe00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f200fe0d0 0x7f7f2010deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f25200700 1 -- 172.21.15.63:0/2532201750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f200fe0d0 msgr2=0x7f7f2010deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.895+0000 7f7f25200700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f200fe0d0 0x7f7f2010deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.896+0000 7f7f25200700 1 -- 172.21.15.63:0/2532201750 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c005040 con 0x7f7f200fcee0 2024-03-23T11:43:57.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.896+0000 7f7f25200700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fcee0 0x7f7f2010d970 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c8e0 tx=0x7f7f140044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:57.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.896+0000 7f7f1cc00700 1 -- 172.21.15.63:0/2532201750 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f1400a940 con 0x7f7f200fcee0 2024-03-23T11:43:57.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.896+0000 7f7f1cc00700 1 -- 172.21.15.63:0/2532201750 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f1400ce80 con 0x7f7f200fcee0 2024-03-23T11:43:57.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.896+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2010efe0 con 0x7f7f200fcee0 2024-03-23T11:43:57.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.896+0000 7f7f1cc00700 1 -- 172.21.15.63:0/2532201750 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f140113a0 con 0x7f7f200fcee0 2024-03-23T11:43:57.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.896+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2010f520 con 0x7f7f200fcee0 2024-03-23T11:43:57.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.898+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2005f5f0 con 0x7f7f200fcee0 2024-03-23T11:43:57.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.898+0000 7f7f1cc00700 1 -- 172.21.15.63:0/2532201750 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7f14012400 con 0x7f7f200fcee0 2024-03-23T11:43:57.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.899+0000 7f7f1cc00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7f080692b0 0x7f7f0806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:43:57.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.899+0000 7f7f1cc00700 1 -- 172.21.15.63:0/2532201750 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7f14087340 con 0x7f7f200fcee0 2024-03-23T11:43:57.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.900+0000 7f7f1fe00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7f080692b0 0x7f7f0806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:43:57.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.900+0000 7f7f1fe00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7f080692b0 0x7f7f0806b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005010 tx=0x7f7f0c009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:43:57.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:57.905+0000 7f7f1cc00700 1 -- 172.21.15.63:0/2532201750 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7f14057060 con 0x7f7f200fcee0 2024-03-23T11:43:58.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.163+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f7f2010f7d0 con 0x7f7f200fcee0 2024-03-23T11:43:58.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: pgmap v204: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-23T11:43:58.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:58.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:58.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: Deploying daemon mds.cephfs.smithi063.mdbdrz on smithi063 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.111:6824/1161744664,v1:172.21.15.111:6825/1161744664] up:boot 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 1 up:standby 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:43:58.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:57 smithi111 ceph-mon[44005]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 1 up:standby 2024-03-23T11:43:58.211 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: pgmap v204: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-23T11:43:58.211 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:58.211 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:58.211 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:43:58.211 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:43:58.211 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:43:58.211 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:43:58.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: Deploying daemon mds.cephfs.smithi063.mdbdrz on smithi063 2024-03-23T11:43:58.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.111:6824/1161744664,v1:172.21.15.111:6825/1161744664] up:boot 2024-03-23T11:43:58.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 1 up:standby 2024-03-23T11:43:58.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:43:58.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:57 smithi063 ceph-mon[35189]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 1 up:standby 2024-03-23T11:43:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.976+0000 7f7f1cc00700 1 -- 172.21.15.63:0/2532201750 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7f14056bf0 con 0x7f7f200fcee0 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7f080692b0 msgr2=0x7f7f0806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7f080692b0 0x7f7f0806b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005010 tx=0x7f7f0c009e30 comp rx=0 tx=0).stop 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fcee0 msgr2=0x7f7f2010d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fcee0 0x7f7f2010d970 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c8e0 tx=0x7f7f140044c0 comp rx=0 tx=0).stop 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 shutdown_connections 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7f080692b0 0x7f7f0806b760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f200fcee0 0x7f7f2010d970 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 --2- 172.21.15.63:0/2532201750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f200fe0d0 0x7f7f2010deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:43:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 >> 172.21.15.63:0/2532201750 conn(0x7f7f200f8560 msgr2=0x7f7f200f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:43:58.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 shutdown_connections 2024-03-23T11:43:58.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:43:58.979+0000 7f7f27c00700 1 -- 172.21.15.63:0/2532201750 wait complete. 2024-03-23T11:43:59.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:58 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2532201750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-23T11:43:59.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:58 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2532201750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-23T11:44:00.196 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T11:44:00.207 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:44:00.207 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: pgmap v205: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2532201750' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: daemon mds.cephfs.smithi111.ylnmzy assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 1 up:standby 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:creating} 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: daemon mds.cephfs.smithi111.ylnmzy is now active in filesystem cephfs as rank 1 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:00.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:43:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: pgmap v205: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-23T11:44:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-23T11:44:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2532201750' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-23T11:44:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: daemon mds.cephfs.smithi111.ylnmzy assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-23T11:44:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-23T11:44:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-23T11:44:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: fsmap cephfs:1 {0=cephfs.smithi063.brtzgy=up:active} 1 up:standby 2024-03-23T11:44:00.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:creating} 2024-03-23T11:44:00.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: daemon mds.cephfs.smithi111.ylnmzy is now active in filesystem cephfs as rank 1 2024-03-23T11:44:00.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:00.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:43:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:01.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:00 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.111:6824/1161744664,v1:172.21.15.111:6825/1161744664] up:active 2024-03-23T11:44:01.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:00 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2024-03-23T11:44:01.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:01.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:01.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:00 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.111:6824/1161744664,v1:172.21.15.111:6825/1161744664] up:active 2024-03-23T11:44:01.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:00 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2024-03-23T11:44:01.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:01.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:02.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: pgmap v206: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-03-23T11:44:02.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-23T11:44:02.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: Cluster is now healthy 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:boot 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 1 up:standby 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:44:02.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:01 smithi063 ceph-mon[35189]: Deploying daemon mds.cephfs.smithi111.atkibb on smithi111 2024-03-23T11:44:02.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: pgmap v206: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-03-23T11:44:02.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: Cluster is now healthy 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:boot 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 1 up:standby 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:44:02.155 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:01 smithi111 ceph-mon[44005]: Deploying daemon mds.cephfs.smithi111.atkibb on smithi111 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 -- 172.21.15.63:0/4265237357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fcee0 msgr2=0x7f700c0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 --2- 172.21.15.63:0/4265237357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fcee0 0x7f700c0fd310 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc004940 tx=0x7f6ffc009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 -- 172.21.15.63:0/4265237357 shutdown_connections 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 --2- 172.21.15.63:0/4265237357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f700c0fe0d0 0x7f700c0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 --2- 172.21.15.63:0/4265237357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fcee0 0x7f700c0fd310 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 -- 172.21.15.63:0/4265237357 >> 172.21.15.63:0/4265237357 conn(0x7f700c0f8560 msgr2=0x7f700c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 -- 172.21.15.63:0/4265237357 shutdown_connections 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.866+0000 7f7012800700 1 -- 172.21.15.63:0/4265237357 wait complete. 2024-03-23T11:44:02.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.867+0000 7f7012800700 1 Processor -- start 2024-03-23T11:44:02.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.867+0000 7f7012800700 1 -- start start 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.867+0000 7f7012800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f700c0fcee0 0x7f700c10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.867+0000 7f7012800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fe0d0 0x7f700c1100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.867+0000 7f7012800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c110740 con 0x7f700c0fe0d0 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.867+0000 7f7012800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c110f30 con 0x7f700c0fcee0 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fe0d0 0x7f700c1100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fe0d0 0x7f700c1100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34242/0 (socket says 172.21.15.63:34242) 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700aa00700 1 -- 172.21.15.63:0/4207390243 learned_addr learned my addr 172.21.15.63:0/4207390243 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700b400700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f700c0fcee0 0x7f700c10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700aa00700 1 -- 172.21.15.63:0/4207390243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f700c0fcee0 msgr2=0x7f700c10fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700aa00700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f700c0fcee0 0x7f700c10fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700aa00700 1 -- 172.21.15.63:0/4207390243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc005040 con 0x7f700c0fe0d0 2024-03-23T11:44:02.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.868+0000 7f700aa00700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fe0d0 0x7f700c1100b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800b790 tx=0x7f6ff8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.869+0000 7f7003400700 1 -- 172.21.15.63:0/4207390243 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff800f6f0 con 0x7f700c0fe0d0 2024-03-23T11:44:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.869+0000 7f7003400700 1 -- 172.21.15.63:0/4207390243 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ff800fd00 con 0x7f700c0fe0d0 2024-03-23T11:44:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.869+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700c111210 con 0x7f700c0fe0d0 2024-03-23T11:44:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.869+0000 7f7003400700 1 -- 172.21.15.63:0/4207390243 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff80103a0 con 0x7f700c0fe0d0 2024-03-23T11:44:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.869+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700c1117d0 con 0x7f700c0fe0d0 2024-03-23T11:44:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.870+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700c101a60 con 0x7f700c0fe0d0 2024-03-23T11:44:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.871+0000 7f7003400700 1 -- 172.21.15.63:0/4207390243 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6ff800f850 con 0x7f700c0fe0d0 2024-03-23T11:44:02.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.872+0000 7f7003400700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6ff00692b0 0x7f6ff006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:02.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.872+0000 7f7003400700 1 -- 172.21.15.63:0/4207390243 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6ff8086f40 con 0x7f700c0fe0d0 2024-03-23T11:44:02.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.872+0000 7f700b400700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6ff00692b0 0x7f6ff006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:02.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.873+0000 7f700b400700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6ff00692b0 0x7f6ff006b760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00b3b0 tx=0x7f6ffc009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:02.877+0000 7f7003400700 1 -- 172.21.15.63:0/4207390243 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6ff8056ce0 con 0x7f700c0fe0d0 2024-03-23T11:44:03.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.129+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f700c05f5f0 con 0x7f700c0fe0d0 2024-03-23T11:44:03.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.994+0000 7f7003400700 1 -- 172.21.15.63:0/4207390243 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v12) v1 ==== 121+0+0 (secure 0 0 0) 0x7f6ff8056870 con 0x7f700c0fe0d0 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.997+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6ff00692b0 msgr2=0x7f6ff006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.997+0000 7f7012800700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6ff00692b0 0x7f6ff006b760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00b3b0 tx=0x7f6ffc009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.997+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fe0d0 msgr2=0x7f700c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.997+0000 7f7012800700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fe0d0 0x7f700c1100b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800b790 tx=0x7f6ff8009250 comp rx=0 tx=0).stop 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.998+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 shutdown_connections 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.998+0000 7f7012800700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6ff00692b0 0x7f6ff006b760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.998+0000 7f7012800700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f700c0fcee0 0x7f700c10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.998+0000 7f7012800700 1 --2- 172.21.15.63:0/4207390243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f700c0fe0d0 0x7f700c1100b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:03.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.998+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 >> 172.21.15.63:0/4207390243 conn(0x7f700c0f8560 msgr2=0x7f700c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:03.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.998+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 shutdown_connections 2024-03-23T11:44:03.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:03.998+0000 7f7012800700 1 -- 172.21.15.63:0/4207390243 wait complete. 2024-03-23T11:44:04.361 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:03 smithi063 ceph-mon[35189]: pgmap v207: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-23T11:44:04.361 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:03 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4207390243' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T11:44:04.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:03 smithi111 ceph-mon[44005]: pgmap v207: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-23T11:44:04.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:03 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4207390243' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-23T11:44:04.742 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T11:44:04.752 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:44:04.753 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data false' 2024-03-23T11:44:05.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4207390243' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-23T11:44:05.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] up:boot 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2 up:standby 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2 up:standby-replay 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:44:05.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:04 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4207390243' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] up:boot 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2 up:standby 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2 up:standby-replay 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:44:05.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:04 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:06.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: pgmap v208: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-23T11:44:06.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:standby-replay 2024-03-23T11:44:06.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: Dropping low affinity active daemon mds.cephfs.smithi111.ylnmzy in favor of higher affinity standby. 2024-03-23T11:44:06.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: Replacing daemon mds.cephfs.smithi111.ylnmzy as rank 1 with standby daemon mds.cephfs.smithi063.mdbdrz 2024-03-23T11:44:06.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-23T11:44:06.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2 up:standby-replay 2024-03-23T11:44:06.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: osdmap e46: 6 total, 6 up, 6 in 2024-03-23T11:44:06.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:06 smithi063 ceph-mon[35189]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:replay} 1 up:standby-replay 2024-03-23T11:44:06.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: pgmap v208: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-23T11:44:06.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:standby-replay 2024-03-23T11:44:06.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: Dropping low affinity active daemon mds.cephfs.smithi111.ylnmzy in favor of higher affinity standby. 2024-03-23T11:44:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: Replacing daemon mds.cephfs.smithi111.ylnmzy as rank 1 with standby daemon mds.cephfs.smithi063.mdbdrz 2024-03-23T11:44:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-23T11:44:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi111.ylnmzy=up:active} 2 up:standby-replay 2024-03-23T11:44:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: osdmap e46: 6 total, 6 up, 6 in 2024-03-23T11:44:06.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:06 smithi111 ceph-mon[44005]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:replay} 1 up:standby-replay 2024-03-23T11:44:07.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:07 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] up:boot 2024-03-23T11:44:07.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:07 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:resolve 2024-03-23T11:44:07.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:07 smithi063 ceph-mon[35189]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:resolve} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:07.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:07 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:44:07.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:07 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] up:boot 2024-03-23T11:44:07.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:07 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:resolve 2024-03-23T11:44:07.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:07 smithi111 ceph-mon[44005]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:resolve} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:07.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:07 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:44:08.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.021+0000 7ffb56c00700 1 -- 172.21.15.63:0/2087828328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 msgr2=0x7ffb500fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.021+0000 7ffb56c00700 1 --2- 172.21.15.63:0/2087828328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb500fe540 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c004940 tx=0x7ffb3c009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.021+0000 7ffb56c00700 1 -- 172.21.15.63:0/2087828328 shutdown_connections 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.021+0000 7ffb56c00700 1 --2- 172.21.15.63:0/2087828328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb500fe540 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.021+0000 7ffb56c00700 1 --2- 172.21.15.63:0/2087828328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500fcee0 0x7ffb500fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.021+0000 7ffb56c00700 1 -- 172.21.15.63:0/2087828328 >> 172.21.15.63:0/2087828328 conn(0x7ffb500f8560 msgr2=0x7ffb500fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.021+0000 7ffb56c00700 1 -- 172.21.15.63:0/2087828328 shutdown_connections 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.022+0000 7ffb56c00700 1 -- 172.21.15.63:0/2087828328 wait complete. 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.023+0000 7ffb56c00700 1 Processor -- start 2024-03-23T11:44:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.023+0000 7ffb56c00700 1 -- start start 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.023+0000 7ffb56c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500fcee0 0x7ffb5010d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.023+0000 7ffb56c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb5010de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.023+0000 7ffb56c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5010e440 con 0x7ffb500fe0d0 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.023+0000 7ffb56c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5010ec90 con 0x7ffb500fcee0 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb5010de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb5010de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34266/0 (socket says 172.21.15.63:34266) 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4f400700 1 -- 172.21.15.63:0/380362858 learned_addr learned my addr 172.21.15.63:0/380362858 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4fe00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500fcee0 0x7ffb5010d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4f400700 1 -- 172.21.15.63:0/380362858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500fcee0 msgr2=0x7ffb5010d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4f400700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500fcee0 0x7ffb5010d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4f400700 1 -- 172.21.15.63:0/380362858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c005040 con 0x7ffb500fe0d0 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.024+0000 7ffb4f400700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb5010de40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0137b0 tx=0x7ffb3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.025+0000 7ffb47e00700 1 -- 172.21.15.63:0/380362858 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3c00e070 con 0x7ffb500fe0d0 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.025+0000 7ffb47e00700 1 -- 172.21.15.63:0/380362858 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb3c01c400 con 0x7ffb500fe0d0 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.025+0000 7ffb47e00700 1 -- 172.21.15.63:0/380362858 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3c00b650 con 0x7ffb500fe0d0 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.025+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5010ef10 con 0x7ffb500fe0d0 2024-03-23T11:44:08.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.025+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5010f3f0 con 0x7ffb500fe0d0 2024-03-23T11:44:08.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.026+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5005f5f0 con 0x7ffb500fe0d0 2024-03-23T11:44:08.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.027+0000 7ffb47e00700 1 -- 172.21.15.63:0/380362858 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ffb3c01c570 con 0x7ffb500fe0d0 2024-03-23T11:44:08.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.028+0000 7ffb47e00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb340692b0 0x7ffb3406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:08.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.028+0000 7ffb47e00700 1 -- 172.21.15.63:0/380362858 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ffb3c088480 con 0x7ffb500fe0d0 2024-03-23T11:44:08.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.028+0000 7ffb4fe00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb340692b0 0x7ffb3406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:08.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.028+0000 7ffb4fe00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb340692b0 0x7ffb3406b760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009b80 tx=0x7ffb4000b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:08.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.032+0000 7ffb47e00700 1 -- 172.21.15.63:0/380362858 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ffb3c058290 con 0x7ffb500fe0d0 2024-03-23T11:44:08.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:08.287+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7ffb50105870 con 0x7ffb500fe0d0 2024-03-23T11:44:08.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:08 smithi063 ceph-mon[35189]: pgmap v210: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-23T11:44:08.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:08 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:reconnect 2024-03-23T11:44:08.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:08 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] up:active 2024-03-23T11:44:08.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:08 smithi063 ceph-mon[35189]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:reconnect} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:08.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:08 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:08.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:08 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:08.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:08 smithi111 ceph-mon[44005]: pgmap v210: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-23T11:44:08.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:08 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:reconnect 2024-03-23T11:44:08.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:08 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] up:active 2024-03-23T11:44:08.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:08 smithi111 ceph-mon[44005]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:reconnect} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:08.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:08 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:08.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:08 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:09.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.136+0000 7ffb47e00700 1 -- 172.21.15.63:0/380362858 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v19) v1 ==== 133+0+0 (secure 0 0 0) 0x7ffb3c057e20 con 0x7ffb500fe0d0 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.138+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb340692b0 msgr2=0x7ffb3406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.138+0000 7ffb56c00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb340692b0 0x7ffb3406b760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009b80 tx=0x7ffb4000b040 comp rx=0 tx=0).stop 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.138+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 msgr2=0x7ffb5010de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.138+0000 7ffb56c00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb5010de40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0137b0 tx=0x7ffb3c009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.139+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 shutdown_connections 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.139+0000 7ffb56c00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb340692b0 0x7ffb3406b760 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.139+0000 7ffb56c00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500fcee0 0x7ffb5010d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.139+0000 7ffb56c00700 1 --2- 172.21.15.63:0/380362858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb500fe0d0 0x7ffb5010de40 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0137b0 tx=0x7ffb3c009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.139+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 >> 172.21.15.63:0/380362858 conn(0x7ffb500f8560 msgr2=0x7ffb500f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.139+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 shutdown_connections 2024-03-23T11:44:09.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:09.139+0000 7ffb56c00700 1 -- 172.21.15.63:0/380362858 wait complete. 2024-03-23T11:44:09.126 INFO:teuthology.orchestra.run.smithi063.stderr:inline data disabled 2024-03-23T11:44:09.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:09 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:rejoin 2024-03-23T11:44:09.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:09 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] up:standby-replay 2024-03-23T11:44:09.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:09 smithi063 ceph-mon[35189]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:rejoin} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:09.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:09 smithi063 ceph-mon[35189]: daemon mds.cephfs.smithi063.mdbdrz is now active in filesystem cephfs as rank 1 2024-03-23T11:44:09.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:09 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/380362858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-23T11:44:09.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:09 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:rejoin 2024-03-23T11:44:09.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:09 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] up:standby-replay 2024-03-23T11:44:09.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:09 smithi111 ceph-mon[44005]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:rejoin} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:09.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:09 smithi111 ceph-mon[44005]: daemon mds.cephfs.smithi063.mdbdrz is now active in filesystem cephfs as rank 1 2024-03-23T11:44:09.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:09 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/380362858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-23T11:44:09.935 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T11:44:09.946 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:44:09.946 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-23T11:44:10.196 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:10 smithi063 ceph-mon[35189]: pgmap v211: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-23T11:44:10.196 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:10 smithi063 ceph-mon[35189]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-23T11:44:10.197 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:10 smithi063 ceph-mon[35189]: Cluster is now healthy 2024-03-23T11:44:10.197 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:10 smithi063 ceph-mon[35189]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:active 2024-03-23T11:44:10.197 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:10 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/380362858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-23T11:44:10.197 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:10 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:10.197 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:10 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:44:10.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:10 smithi111 ceph-mon[44005]: pgmap v211: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-23T11:44:10.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:10 smithi111 ceph-mon[44005]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-23T11:44:10.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:10 smithi111 ceph-mon[44005]: Cluster is now healthy 2024-03-23T11:44:10.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:10 smithi111 ceph-mon[44005]: mds.? [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] up:active 2024-03-23T11:44:10.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:10 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/380362858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-23T11:44:10.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:10 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 1 up:standby-replay 1 up:standby 2024-03-23T11:44:10.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:10 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:44:10.701 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:12.310 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:12 smithi063 ceph-mon[35189]: pgmap v212: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-23T11:44:12.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:12 smithi111 ceph-mon[44005]: pgmap v212: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 -- 172.21.15.63:0/299834058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 msgr2=0x7f17c4070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 --2- 172.21.15.63:0/299834058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c4070460 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f17b8004940 tx=0x7f17b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 -- 172.21.15.63:0/299834058 shutdown_connections 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 --2- 172.21.15.63:0/299834058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17c40709a0 0x7f17c4070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 --2- 172.21.15.63:0/299834058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c4070460 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 -- 172.21.15.63:0/299834058 >> 172.21.15.63:0/299834058 conn(0x7f17c40f8560 msgr2=0x7f17c40fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 -- 172.21.15.63:0/299834058 shutdown_connections 2024-03-23T11:44:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.186+0000 7f17cbc00700 1 -- 172.21.15.63:0/299834058 wait complete. 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.187+0000 7f17cbc00700 1 Processor -- start 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.187+0000 7f17cbc00700 1 -- start start 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17cbc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17c40709a0 0x7f17c410fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17cbc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c41100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17cbc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c41106b0 con 0x7f17c4071f50 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17cbc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c4110f00 con 0x7f17c40709a0 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17c3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c41100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:13.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17c3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c41100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40034/0 (socket says 172.21.15.63:40034) 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17c3e00700 1 -- 172.21.15.63:0/4085159241 learned_addr learned my addr 172.21.15.63:0/4085159241 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17c9200700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17c40709a0 0x7f17c410fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17c3e00700 1 -- 172.21.15.63:0/4085159241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17c40709a0 msgr2=0x7f17c410fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17c3e00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17c40709a0 0x7f17c410fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.188+0000 7f17c3e00700 1 -- 172.21.15.63:0/4085159241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8005040 con 0x7f17c4071f50 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.189+0000 7f17c3e00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c41100b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f17b0002a50 tx=0x7f17b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.189+0000 7f17c0c00700 1 -- 172.21.15.63:0/4085159241 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b0007cc0 con 0x7f17c4071f50 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.189+0000 7f17c0c00700 1 -- 172.21.15.63:0/4085159241 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17b000f040 con 0x7f17c4071f50 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.189+0000 7f17c0c00700 1 -- 172.21.15.63:0/4085159241 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b000d9c0 con 0x7f17c4071f50 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.189+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c41111e0 con 0x7f17c4071f50 2024-03-23T11:44:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.189+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c41116d0 con 0x7f17c4071f50 2024-03-23T11:44:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.190+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c405f5f0 con 0x7f17c4071f50 2024-03-23T11:44:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.191+0000 7f17c0c00700 1 -- 172.21.15.63:0/4085159241 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f17b000db60 con 0x7f17c4071f50 2024-03-23T11:44:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.192+0000 7f17c0c00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f17ac0692a0 0x7f17ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.192+0000 7f17c0c00700 1 -- 172.21.15.63:0/4085159241 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f17b0086900 con 0x7f17c4071f50 2024-03-23T11:44:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.192+0000 7f17c9200700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f17ac0692a0 0x7f17ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.193+0000 7f17c9200700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f17ac0692a0 0x7f17ac06b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17b800b900 tx=0x7f17b8009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:13.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.197+0000 7f17c0c00700 1 -- 172.21.15.63:0/4085159241 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f17b004d190 con 0x7f17c4071f50 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.473+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f17c404bed0 con 0x7f17c4071f50 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.474+0000 7f17c0c00700 1 -- 172.21.15.63:0/4085159241 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1832 (secure 0 0 0) 0x7f17b0017740 con 0x7f17c4071f50 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stdout:e20 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:44:13.459 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 20 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:44:09.137068+0000 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:44:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:44:13.461 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:44:13.462 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:44:13.462 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:44:13.462 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:44:13.462 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:44:13.462 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:44:13.462 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:44:13.462 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:44:13.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f17ac0692a0 msgr2=0x7f17ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:13.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f17ac0692a0 0x7f17ac06b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17b800b900 tx=0x7f17b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:13.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 msgr2=0x7f17c41100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:13.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c41100b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f17b0002a50 tx=0x7f17b00046b0 comp rx=0 tx=0).stop 2024-03-23T11:44:13.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 shutdown_connections 2024-03-23T11:44:13.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f17ac0692a0 0x7f17ac06b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:13.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17c40709a0 0x7f17c410fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:13.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 --2- 172.21.15.63:0/4085159241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17c4071f50 0x7f17c41100b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:13.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 >> 172.21.15.63:0/4085159241 conn(0x7f17c40f8560 msgr2=0x7f17c40f9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:13.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 shutdown_connections 2024-03-23T11:44:13.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:13.476+0000 7f17cbc00700 1 -- 172.21.15.63:0/4085159241 wait complete. 2024-03-23T11:44:13.464 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 20 2024-03-23T11:44:14.196 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-23T11:44:14.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: pgmap v213: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-23T11:44:14.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:14.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:14.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:44:14.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:44:14.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:14.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4085159241' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:44:14.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:14 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: pgmap v213: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4085159241' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:44:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:14 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:44:15.210 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:16.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:15 smithi063 ceph-mon[35189]: pgmap v214: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-23T11:44:16.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:16.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:16.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:15 smithi111 ceph-mon[44005]: pgmap v214: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-23T11:44:16.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:16.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.603+0000 7ff8d3e00700 1 -- 172.21.15.63:0/2273967243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fdfe0 msgr2=0x7ff8cc0fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.603+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/2273967243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fdfe0 0x7ff8cc0fe450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8004940 tx=0x7ff8b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.603+0000 7ff8d3e00700 1 -- 172.21.15.63:0/2273967243 shutdown_connections 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.603+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/2273967243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fdfe0 0x7ff8cc0fe450 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.603+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/2273967243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc0fce70 0x7ff8cc0fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.603+0000 7ff8d3e00700 1 -- 172.21.15.63:0/2273967243 >> 172.21.15.63:0/2273967243 conn(0x7ff8cc0f8550 msgr2=0x7ff8cc0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.604+0000 7ff8d3e00700 1 -- 172.21.15.63:0/2273967243 shutdown_connections 2024-03-23T11:44:17.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.604+0000 7ff8d3e00700 1 -- 172.21.15.63:0/2273967243 wait complete. 2024-03-23T11:44:17.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.605+0000 7ff8d3e00700 1 Processor -- start 2024-03-23T11:44:17.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.605+0000 7ff8d3e00700 1 -- start start 2024-03-23T11:44:17.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fce70 0x7ff8cc10fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:17.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d3e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc0fdfe0 0x7ff8cc10ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:17.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d3e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc110580 con 0x7ff8cc0fce70 2024-03-23T11:44:17.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d3e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1106f0 con 0x7ff8cc0fdfe0 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fce70 0x7ff8cc10fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fce70 0x7ff8cc10fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40064/0 (socket says 172.21.15.63:40064) 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d1400700 1 -- 172.21.15.63:0/3762204951 learned_addr learned my addr 172.21.15.63:0/3762204951 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d0a00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc0fdfe0 0x7ff8cc10ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d1400700 1 -- 172.21.15.63:0/3762204951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc0fdfe0 msgr2=0x7ff8cc10ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d1400700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc0fdfe0 0x7ff8cc10ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.606+0000 7ff8d1400700 1 -- 172.21.15.63:0/3762204951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b8005040 con 0x7ff8cc0fce70 2024-03-23T11:44:17.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.607+0000 7ff8d1400700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fce70 0x7ff8cc10fa40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0002a50 tx=0x7ff8c00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:17.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.607+0000 7ff8c9600700 1 -- 172.21.15.63:0/3762204951 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c0007cc0 con 0x7ff8cc0fce70 2024-03-23T11:44:17.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.607+0000 7ff8c9600700 1 -- 172.21.15.63:0/3762204951 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8c000f040 con 0x7ff8cc0fce70 2024-03-23T11:44:17.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.607+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc115150 con 0x7ff8cc0fce70 2024-03-23T11:44:17.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.607+0000 7ff8c9600700 1 -- 172.21.15.63:0/3762204951 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c0009540 con 0x7ff8cc0fce70 2024-03-23T11:44:17.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.607+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc115710 con 0x7ff8cc0fce70 2024-03-23T11:44:17.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.608+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc101970 con 0x7ff8cc0fce70 2024-03-23T11:44:17.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.609+0000 7ff8c9600700 1 -- 172.21.15.63:0/3762204951 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff8c00096a0 con 0x7ff8cc0fce70 2024-03-23T11:44:17.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.610+0000 7ff8c9600700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff8b40692a0 0x7ff8b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:17.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.610+0000 7ff8c9600700 1 -- 172.21.15.63:0/3762204951 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff8c0086fc0 con 0x7ff8cc0fce70 2024-03-23T11:44:17.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.610+0000 7ff8d0a00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff8b40692a0 0x7ff8b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:17.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.611+0000 7ff8d0a00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff8b40692a0 0x7ff8b406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800b3b0 tx=0x7ff8b8009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:17.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.615+0000 7ff8c9600700 1 -- 172.21.15.63:0/3762204951 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff8c0056c90 con 0x7ff8cc0fce70 2024-03-23T11:44:17.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:17 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:17.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:17 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:17.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:17 smithi063 ceph-mon[35189]: pgmap v215: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-23T11:44:17.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:17 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:17.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:17 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:17.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:17 smithi111 ceph-mon[44005]: pgmap v215: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-23T11:44:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.872+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8cc04c710 con 0x7ff8cc0fce70 2024-03-23T11:44:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.873+0000 7ff8c9600700 1 -- 172.21.15.63:0/3762204951 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4722 (secure 0 0 0) 0x7ff8c0056820 con 0x7ff8cc0fce70 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.875+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff8b40692a0 msgr2=0x7ff8b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.875+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff8b40692a0 0x7ff8b406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800b3b0 tx=0x7ff8b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.875+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fce70 msgr2=0x7ff8cc10fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.875+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fce70 0x7ff8cc10fa40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0002a50 tx=0x7ff8c00046b0 comp rx=0 tx=0).stop 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.876+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 shutdown_connections 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.876+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff8b40692a0 0x7ff8b406b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.876+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8cc0fce70 0x7ff8cc10fa40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.876+0000 7ff8d3e00700 1 --2- 172.21.15.63:0/3762204951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc0fdfe0 0x7ff8cc10ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.876+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 >> 172.21.15.63:0/3762204951 conn(0x7ff8cc0f8550 msgr2=0x7ff8cc0f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.876+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 shutdown_connections 2024-03-23T11:44:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:17.876+0000 7ff8d3e00700 1 -- 172.21.15.63:0/3762204951 wait complete. 2024-03-23T11:44:17.862 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 20 2024-03-23T11:44:17.877 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:44:18.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:18 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3762204951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T11:44:18.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:18 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3762204951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T11:44:18.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-23T11:44:19.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:19 smithi063 ceph-mon[35189]: pgmap v216: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-23T11:44:19.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:19 smithi111 ceph-mon[44005]: pgmap v216: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-23T11:44:19.857 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:21 smithi063 ceph-mon[35189]: pgmap v217: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-23T11:44:22.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:21 smithi111 ceph-mon[44005]: pgmap v217: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-23T11:44:22.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 -- 172.21.15.63:0/781566262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c071f50 msgr2=0x7f7c9c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:22.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 --2- 172.21.15.63:0/781566262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c071f50 0x7f7c9c070460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c004940 tx=0x7f7c8c009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:22.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 -- 172.21.15.63:0/781566262 shutdown_connections 2024-03-23T11:44:22.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 --2- 172.21.15.63:0/781566262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c9c0709a0 0x7f7c9c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:22.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 --2- 172.21.15.63:0/781566262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c071f50 0x7f7c9c070460 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:22.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 -- 172.21.15.63:0/781566262 >> 172.21.15.63:0/781566262 conn(0x7f7c9c0f8560 msgr2=0x7f7c9c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:22.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 -- 172.21.15.63:0/781566262 shutdown_connections 2024-03-23T11:44:22.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.555+0000 7f7ca2000700 1 -- 172.21.15.63:0/781566262 wait complete. 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.556+0000 7f7ca2000700 1 Processor -- start 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.556+0000 7f7ca2000700 1 -- start start 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.556+0000 7f7ca2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c0709a0 0x7f7c9c10fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.556+0000 7f7ca2000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c9c071f50 0x7f7c9c110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.556+0000 7f7ca2000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c110610 con 0x7f7c9c0709a0 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.556+0000 7f7ca2000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c110e60 con 0x7f7c9c071f50 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c0709a0 0x7f7c9c10fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c0709a0 0x7f7c9c10fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48122/0 (socket says 172.21.15.63:48122) 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9b400700 1 -- 172.21.15.63:0/1511972619 learned_addr learned my addr 172.21.15.63:0/1511972619 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9aa00700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c9c071f50 0x7f7c9c110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:22.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9b400700 1 -- 172.21.15.63:0/1511972619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c9c071f50 msgr2=0x7f7c9c110010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:22.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9b400700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c9c071f50 0x7f7c9c110010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:22.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9b400700 1 -- 172.21.15.63:0/1511972619 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c005040 con 0x7f7c9c0709a0 2024-03-23T11:44:22.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c9b400700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c0709a0 0x7f7c9c10fad0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c000c00 tx=0x7f7c8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:22.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.557+0000 7f7c93400700 1 -- 172.21.15.63:0/1511972619 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c8c00e070 con 0x7f7c9c0709a0 2024-03-23T11:44:22.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.558+0000 7f7c93400700 1 -- 172.21.15.63:0/1511972619 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c8c01c400 con 0x7f7c9c0709a0 2024-03-23T11:44:22.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.558+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c1110e0 con 0x7f7c9c0709a0 2024-03-23T11:44:22.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.558+0000 7f7c93400700 1 -- 172.21.15.63:0/1511972619 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c8c00b650 con 0x7f7c9c0709a0 2024-03-23T11:44:22.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.558+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c1115c0 con 0x7f7c9c0709a0 2024-03-23T11:44:22.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.559+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9c05f5f0 con 0x7f7c9c0709a0 2024-03-23T11:44:22.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.559+0000 7f7c93400700 1 -- 172.21.15.63:0/1511972619 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7c8c01c570 con 0x7f7c9c0709a0 2024-03-23T11:44:22.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.560+0000 7f7c93400700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c800692a0 0x7f7c8006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:22.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.560+0000 7f7c93400700 1 -- 172.21.15.63:0/1511972619 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7c8c088770 con 0x7f7c9c0709a0 2024-03-23T11:44:22.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.561+0000 7f7c9aa00700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c800692a0 0x7f7c8006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:22.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.561+0000 7f7c9aa00700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c800692a0 0x7f7c8006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7c88009b80 tx=0x7f7c8800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:22.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.565+0000 7f7c93400700 1 -- 172.21.15.63:0/1511972619 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7c8c0584c0 con 0x7f7c9c0709a0 2024-03-23T11:44:22.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.854+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f7c9c111990 con 0x7f7c9c0709a0 2024-03-23T11:44:22.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.855+0000 7f7c93400700 1 -- 172.21.15.63:0/1511972619 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v20) v1 ==== 78+0+99 (secure 0 0 0) 0x7f7c8c058050 con 0x7f7c9c0709a0 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.857+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c800692a0 msgr2=0x7f7c8006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.857+0000 7f7ca2000700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c800692a0 0x7f7c8006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7c88009b80 tx=0x7f7c8800b040 comp rx=0 tx=0).stop 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c0709a0 msgr2=0x7f7c9c10fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c0709a0 0x7f7c9c10fad0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c000c00 tx=0x7f7c8c009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 shutdown_connections 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7c800692a0 0x7f7c8006b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c9c0709a0 0x7f7c9c10fad0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 --2- 172.21.15.63:0/1511972619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c9c071f50 0x7f7c9c110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 >> 172.21.15.63:0/1511972619 conn(0x7f7c9c0f8560 msgr2=0x7f7c9c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 shutdown_connections 2024-03-23T11:44:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:22.858+0000 7f7ca2000700 1 -- 172.21.15.63:0/1511972619 wait complete. 2024-03-23T11:44:22.856 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:44:23.587 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:23.587 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:23.587 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:44:23.587 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:44:23.587 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:23.588 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:23 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1511972619' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-23T11:44:23.588 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:23 smithi063 ceph-mon[35189]: pgmap v218: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-23T11:44:23.611 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-23T11:44:23.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:44:23.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:23.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:23.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:44:23.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:44:23.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:44:23.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:23 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1511972619' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-23T11:44:23.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:23 smithi111 ceph-mon[44005]: pgmap v218: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-23T11:44:23.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 -- 172.21.15.63:0/3746961684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fdfc0 msgr2=0x7f7d040fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:23.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 --2- 172.21.15.63:0/3746961684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fdfc0 0x7f7d040fe430 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7cec004940 tx=0x7f7cec009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:23.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 -- 172.21.15.63:0/3746961684 shutdown_connections 2024-03-23T11:44:23.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 --2- 172.21.15.63:0/3746961684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fdfc0 0x7f7d040fe430 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:23.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 --2- 172.21.15.63:0/3746961684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d040fcd10 0x7f7d040fd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:23.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 -- 172.21.15.63:0/3746961684 >> 172.21.15.63:0/3746961684 conn(0x7f7d040f8470 msgr2=0x7f7d040fa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:23.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 -- 172.21.15.63:0/3746961684 shutdown_connections 2024-03-23T11:44:23.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.850+0000 7f7d0a200700 1 -- 172.21.15.63:0/3746961684 wait complete. 2024-03-23T11:44:23.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.851+0000 7f7d0a200700 1 Processor -- start 2024-03-23T11:44:23.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.851+0000 7f7d0a200700 1 -- start start 2024-03-23T11:44:23.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.851+0000 7f7d0a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fcd10 0x7f7d0410d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.851+0000 7f7d0a200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d040fdfc0 0x7f7d0410dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.851+0000 7f7d0a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0410e240 con 0x7f7d040fcd10 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.851+0000 7f7d0a200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0410e3b0 con 0x7f7d040fdfc0 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7d03400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fcd10 0x7f7d0410d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7d03400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fcd10 0x7f7d0410d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48128/0 (socket says 172.21.15.63:48128) 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7d03400700 1 -- 172.21.15.63:0/2008350462 learned_addr learned my addr 172.21.15.63:0/2008350462 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7cfbe00700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d040fdfc0 0x7f7d0410dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7d03400700 1 -- 172.21.15.63:0/2008350462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d040fdfc0 msgr2=0x7f7d0410dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7d03400700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d040fdfc0 0x7f7d0410dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7d03400700 1 -- 172.21.15.63:0/2008350462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec005040 con 0x7f7d040fcd10 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7cfbe00700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d040fdfc0 0x7f7d0410dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.852+0000 7f7d03400700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fcd10 0x7f7d0410d700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4002a50 tx=0x7f7cf40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.853+0000 7f7cfb400700 1 -- 172.21.15.63:0/2008350462 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cf4007cc0 con 0x7f7d040fcd10 2024-03-23T11:44:23.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.853+0000 7f7cfb400700 1 -- 172.21.15.63:0/2008350462 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cf400f040 con 0x7f7d040fcd10 2024-03-23T11:44:23.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.853+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d04112e10 con 0x7f7d040fcd10 2024-03-23T11:44:23.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.853+0000 7f7cfb400700 1 -- 172.21.15.63:0/2008350462 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cf4009540 con 0x7f7d040fcd10 2024-03-23T11:44:23.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.853+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d04113340 con 0x7f7d040fcd10 2024-03-23T11:44:23.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.854+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d04064970 con 0x7f7d040fcd10 2024-03-23T11:44:23.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.855+0000 7f7cfb400700 1 -- 172.21.15.63:0/2008350462 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7cf40096a0 con 0x7f7d040fcd10 2024-03-23T11:44:23.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.855+0000 7f7cfb400700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7ce80692a0 0x7f7ce806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:23.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.856+0000 7f7cfb400700 1 -- 172.21.15.63:0/2008350462 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7cf4086fe0 con 0x7f7d040fcd10 2024-03-23T11:44:23.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.856+0000 7f7cfbe00700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7ce80692a0 0x7f7ce806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:23.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.856+0000 7f7cfbe00700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7ce80692a0 0x7f7ce806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7cec004f90 tx=0x7f7cec009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:23.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:23.860+0000 7f7cfb400700 1 -- 172.21.15.63:0/2008350462 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7cf4056cb0 con 0x7f7d040fcd10 2024-03-23T11:44:24.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.070+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d041135b0 con 0x7f7d040fcd10 2024-03-23T11:44:24.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.071+0000 7f7cfb400700 1 -- 172.21.15.63:0/2008350462 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14388 (secure 0 0 0) 0x7f7cf4056840 con 0x7f7d040fcd10 2024-03-23T11:44:24.056 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:44:24.056 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":46,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","created":"2024-03-23T11:35:55.761926+0000","modified":"2024-03-23T11:44:05.102099+0000","last_up_change":"2024-03-23T11:42:22.261037+0000","last_in_change":"2024-03-23T11:42:00.992881+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:40:53.432498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-23T11:43:48.146323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-23T11:43:48.722717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e4f29eb-375c-4f37-a8d8-f82b40650ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6803","nonce":1473323264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6805","nonce":1473323264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6809","nonce":1473323264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1473323264},{"type":"v1","addr":"172.21.15.63:6807","nonce":1473323264}]},"public_addr":"172.21.15.63:6803/1473323264","cluster_addr":"172.21.15.63:6805/1473323264","heartbeat_back_addr":"172.21.15.63:6809/1473323264","heartbeat_front_addr":"172.21.15.63:6807/1473323264","state":["exists","up"]},{"osd":1,"uuid":"5e3ddf8b-e2c3-4188-b495-ac2743805f33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6811","nonce":823348330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6813","nonce":823348330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6817","nonce":823348330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":823348330},{"type":"v1","addr":"172.21.15.63:6815","nonce":823348330}]},"public_addr":"172.21.15.63:6811/823348330","cluster_addr":"172.21.15.63:6813/823348330","heartbeat_back_addr":"172.21.15.63:6817/823348330","heartbeat_front_addr":"172.21.15.63:6815/823348330","state":["exists","up"]},{"osd":2,"uuid":"edc4babe-dff9-462c-9af7-75e30b8095af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6819","nonce":219143073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6821","nonce":219143073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6825","nonce":219143073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":219143073},{"type":"v1","addr":"172.21.15.63:6823","nonce":219143073}]},"public_addr":"172.21.15.63:6819/219143073","cluster_addr":"172.21.15.63:6821/219143073","heartbeat_back_addr":"172.21.15.63:6825/219143073","heartbeat_front_addr":"172.21.15.63:6823/219143073","state":["exists","up"]},{"osd":3,"uuid":"7bc94d72-a814-447a-8112-9d50058a2759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6801","nonce":1017350834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6803","nonce":1017350834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6807","nonce":1017350834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1017350834},{"type":"v1","addr":"172.21.15.111:6805","nonce":1017350834}]},"public_addr":"172.21.15.111:6801/1017350834","cluster_addr":"172.21.15.111:6803/1017350834","heartbeat_back_addr":"172.21.15.111:6807/1017350834","heartbeat_front_addr":"172.21.15.111:6805/1017350834","state":["exists","up"]},{"osd":4,"uuid":"7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6809","nonce":1242047756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6811","nonce":1242047756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6815","nonce":1242047756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1242047756},{"type":"v1","addr":"172.21.15.111:6813","nonce":1242047756}]},"public_addr":"172.21.15.111:6809/1242047756","cluster_addr":"172.21.15.111:6811/1242047756","heartbeat_back_addr":"172.21.15.111:6815/1242047756","heartbeat_front_addr":"172.21.15.111:6813/1242047756","state":["exists","up"]},{"osd":5,"uuid":"2219a929-a842-4b41-ab46-096be6ef4d13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6817","nonce":1318746257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6819","nonce":1318746257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6823","nonce":1318746257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1318746257},{"type":"v1","addr":"172.21.15.111:6821","nonce":1318746257}]},"public_addr":"172.21.15.111:6817/1318746257","cluster_addr":"172.21.15.111:6819/1318746257","heartbeat_back_addr":"172.21.15.111:6823/1318746257","heartbeat_front_addr":"172.21.15.111:6821/1318746257","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:39:42.374999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:17.321830+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:50.840928+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:20.211360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:50.314996+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:42:20.376924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6825/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:0/3356174638":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6800/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6801/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3192464709":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1379382252":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6801/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/3491519284":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6800/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3227457751":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/1183338694":"2024-03-24T11:38:14.792281+0000","172.21.15.111:6824/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:0/2981669421":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6800/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/2286910220":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6801/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/3148717714":"2024-03-24T11:38:14.792281+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:44:24.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7ce80692a0 msgr2=0x7f7ce806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7ce80692a0 0x7f7ce806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7cec004f90 tx=0x7f7cec009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fcd10 msgr2=0x7f7d0410d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fcd10 0x7f7d0410d700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4002a50 tx=0x7f7cf40046b0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 shutdown_connections 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f7ce80692a0 0x7f7ce806b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d040fcd10 0x7f7d0410d700 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 --2- 172.21.15.63:0/2008350462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d040fdfc0 0x7f7d0410dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 >> 172.21.15.63:0/2008350462 conn(0x7f7d040f8470 msgr2=0x7f7d040f9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 shutdown_connections 2024-03-23T11:44:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.073+0000 7f7d0a200700 1 -- 172.21.15.63:0/2008350462 wait complete. 2024-03-23T11:44:24.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:40:53.432498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-23T11:43:48.146323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-23T11:43:48.722717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T11:44:24.077 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 -- 172.21.15.63:0/12761457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6640065a00 msgr2=0x7f66400ff930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 --2- 172.21.15.63:0/12761457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6640065a00 0x7f66400ff930 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6630004940 tx=0x7f6630009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 -- 172.21.15.63:0/12761457 shutdown_connections 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 --2- 172.21.15.63:0/12761457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66400ffe70 0x7f6640102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 --2- 172.21.15.63:0/12761457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6640065a00 0x7f66400ff930 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 -- 172.21.15.63:0/12761457 >> 172.21.15.63:0/12761457 conn(0x7f66400f7300 msgr2=0x7f66400f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 -- 172.21.15.63:0/12761457 shutdown_connections 2024-03-23T11:44:24.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.289+0000 7f6645a00700 1 -- 172.21.15.63:0/12761457 wait complete. 2024-03-23T11:44:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.290+0000 7f6645a00700 1 Processor -- start 2024-03-23T11:44:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.290+0000 7f6645a00700 1 -- start start 2024-03-23T11:44:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.290+0000 7f6645a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6640065a00 0x7f664010f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.290+0000 7f6645a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66400ffe70 0x7f664010fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.290+0000 7f6645a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640110440 con 0x7f66400ffe70 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.290+0000 7f6645a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66401105b0 con 0x7f6640065a00 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.290+0000 7f663e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66400ffe70 0x7f664010fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66400ffe70 0x7f664010fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48140/0 (socket says 172.21.15.63:48140) 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663e000700 1 -- 172.21.15.63:0/632500789 learned_addr learned my addr 172.21.15.63:0/632500789 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663e000700 1 -- 172.21.15.63:0/632500789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6640065a00 msgr2=0x7f664010f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663ea00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6640065a00 0x7f664010f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663e000700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6640065a00 0x7f664010f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663e000700 1 -- 172.21.15.63:0/632500789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630005040 con 0x7f66400ffe70 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663ea00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6640065a00 0x7f664010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f663e000700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66400ffe70 0x7f664010fe40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f662c00eb40 tx=0x7f662c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f6636a00700 1 -- 172.21.15.63:0/632500789 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f662c007cc0 con 0x7f66400ffe70 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f6636a00700 1 -- 172.21.15.63:0/632500789 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f662c00f040 con 0x7f66400ffe70 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6640114fb0 con 0x7f66400ffe70 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.291+0000 7f6636a00700 1 -- 172.21.15.63:0/632500789 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f662c009570 con 0x7f66400ffe70 2024-03-23T11:44:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.292+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6640115540 con 0x7f66400ffe70 2024-03-23T11:44:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.292+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6640064970 con 0x7f66400ffe70 2024-03-23T11:44:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.293+0000 7f6636a00700 1 -- 172.21.15.63:0/632500789 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f662c0096d0 con 0x7f66400ffe70 2024-03-23T11:44:24.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.294+0000 7f6636a00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f66240692a0 0x7f662406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:24.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.294+0000 7f6636a00700 1 -- 172.21.15.63:0/632500789 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f662c086e50 con 0x7f66400ffe70 2024-03-23T11:44:24.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.294+0000 7f663ea00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f66240692a0 0x7f662406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:24.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.295+0000 7f663ea00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f66240692a0 0x7f662406b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f663000b0d0 tx=0x7f6630009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:24.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.298+0000 7f6636a00700 1 -- 172.21.15.63:0/632500789 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f662c04e190 con 0x7f66400ffe70 2024-03-23T11:44:24.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.504+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6640115790 con 0x7f66400ffe70 2024-03-23T11:44:24.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.504+0000 7f6636a00700 1 -- 172.21.15.63:0/632500789 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f662c056bf0 con 0x7f66400ffe70 2024-03-23T11:44:24.489 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.506+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f66240692a0 msgr2=0x7f662406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.506+0000 7f6645a00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f66240692a0 0x7f662406b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f663000b0d0 tx=0x7f6630009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66400ffe70 msgr2=0x7f664010fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66400ffe70 0x7f664010fe40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f662c00eb40 tx=0x7f662c00b040 comp rx=0 tx=0).stop 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 shutdown_connections 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f66240692a0 0x7f662406b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6640065a00 0x7f664010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 --2- 172.21.15.63:0/632500789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66400ffe70 0x7f664010fe40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 >> 172.21.15.63:0/632500789 conn(0x7f66400f7300 msgr2=0x7f66400fb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 shutdown_connections 2024-03-23T11:44:24.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.507+0000 7f6645a00700 1 -- 172.21.15.63:0/632500789 wait complete. 2024-03-23T11:44:24.510 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-23T11:44:24.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:24 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2008350462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 -- 172.21.15.63:0/1895380944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c0708f0 msgr2=0x7fdd8c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 --2- 172.21.15.63:0/1895380944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c0708f0 0x7fdd8c070d60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdd78004940 tx=0x7fdd78009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 -- 172.21.15.63:0/1895380944 shutdown_connections 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 --2- 172.21.15.63:0/1895380944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c0708f0 0x7fdd8c070d60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 --2- 172.21.15.63:0/1895380944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c071ea0 0x7fdd8c0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 -- 172.21.15.63:0/1895380944 >> 172.21.15.63:0/1895380944 conn(0x7fdd8c0f8440 msgr2=0x7fdd8c0fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 -- 172.21.15.63:0/1895380944 shutdown_connections 2024-03-23T11:44:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 -- 172.21.15.63:0/1895380944 wait complete. 2024-03-23T11:44:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.722+0000 7fdd93a00700 1 Processor -- start 2024-03-23T11:44:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd93a00700 1 -- start start 2024-03-23T11:44:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd93a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c0708f0 0x7fdd8c10f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd93a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c071ea0 0x7fdd8c10fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd93a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c110490 con 0x7fdd8c071ea0 2024-03-23T11:44:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd93a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c110600 con 0x7fdd8c0708f0 2024-03-23T11:44:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd8be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c071ea0 0x7fdd8c10fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd8be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c071ea0 0x7fdd8c10fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48158/0 (socket says 172.21.15.63:48158) 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd8be00700 1 -- 172.21.15.63:0/2877229308 learned_addr learned my addr 172.21.15.63:0/2877229308 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd8be00700 1 -- 172.21.15.63:0/2877229308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c0708f0 msgr2=0x7fdd8c10f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.723+0000 7fdd91000700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c0708f0 0x7fdd8c10f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd8be00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c0708f0 0x7fdd8c10f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd8be00700 1 -- 172.21.15.63:0/2877229308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78005040 con 0x7fdd8c071ea0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd91000700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c0708f0 0x7fdd8c10f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd8be00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c071ea0 0x7fdd8c10fe90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009380 tx=0x7fdd78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd88c00700 1 -- 172.21.15.63:0/2877229308 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd7800e040 con 0x7fdd8c071ea0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd88c00700 1 -- 172.21.15.63:0/2877229308 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd78004290 con 0x7fdd8c071ea0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd88c00700 1 -- 172.21.15.63:0/2877229308 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd7801c3f0 con 0x7fdd8c071ea0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c115000 con 0x7fdd8c071ea0 2024-03-23T11:44:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.724+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c115570 con 0x7fdd8c071ea0 2024-03-23T11:44:24.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.725+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c064970 con 0x7fdd8c071ea0 2024-03-23T11:44:24.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.726+0000 7fdd88c00700 1 -- 172.21.15.63:0/2877229308 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdd78013420 con 0x7fdd8c071ea0 2024-03-23T11:44:24.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.727+0000 7fdd88c00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdd740692a0 0x7fdd7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:24.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.727+0000 7fdd88c00700 1 -- 172.21.15.63:0/2877229308 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdd78088600 con 0x7fdd8c071ea0 2024-03-23T11:44:24.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.727+0000 7fdd91000700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdd740692a0 0x7fdd7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:24.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.728+0000 7fdd91000700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdd740692a0 0x7fdd7406b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009fd0 tx=0x7fdd8000b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:24.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.732+0000 7fdd88c00700 1 -- 172.21.15.63:0/2877229308 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdd78058370 con 0x7fdd8c071ea0 2024-03-23T11:44:24.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:24 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2008350462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T11:44:24.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.941+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fdd8c1157e0 con 0x7fdd8c071ea0 2024-03-23T11:44:24.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.942+0000 7fdd88c00700 1 -- 172.21.15.63:0/2877229308 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fdd8c1157e0 con 0x7fdd8c071ea0 2024-03-23T11:44:24.926 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T11:44:24.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdd740692a0 msgr2=0x7fdd7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdd740692a0 0x7fdd7406b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009fd0 tx=0x7fdd8000b040 comp rx=0 tx=0).stop 2024-03-23T11:44:24.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c071ea0 msgr2=0x7fdd8c10fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c071ea0 0x7fdd8c10fe90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009380 tx=0x7fdd78009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 shutdown_connections 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fdd740692a0 0x7fdd7406b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c0708f0 0x7fdd8c10f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 --2- 172.21.15.63:0/2877229308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd8c071ea0 0x7fdd8c10fe90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 >> 172.21.15.63:0/2877229308 conn(0x7fdd8c0f8440 msgr2=0x7fdd8c0f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 shutdown_connections 2024-03-23T11:44:24.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:24.944+0000 7fdd93a00700 1 -- 172.21.15.63:0/2877229308 wait complete. 2024-03-23T11:44:24.947 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-23T11:44:25.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.157+0000 7fd2e8200700 1 -- 172.21.15.63:0/2312051273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 msgr2=0x7fd2e00fd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.157+0000 7fd2e8200700 1 --2- 172.21.15.63:0/2312051273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e00fd0c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4004940 tx=0x7fd2d4009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:25.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.157+0000 7fd2e8200700 1 -- 172.21.15.63:0/2312051273 shutdown_connections 2024-03-23T11:44:25.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.157+0000 7fd2e8200700 1 --2- 172.21.15.63:0/2312051273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2e00fdf60 0x7fd2e00fe3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.157+0000 7fd2e8200700 1 --2- 172.21.15.63:0/2312051273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e00fd0c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.157+0000 7fd2e8200700 1 -- 172.21.15.63:0/2312051273 >> 172.21.15.63:0/2312051273 conn(0x7fd2e00f8430 msgr2=0x7fd2e00fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:25.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.158+0000 7fd2e8200700 1 -- 172.21.15.63:0/2312051273 shutdown_connections 2024-03-23T11:44:25.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.158+0000 7fd2e8200700 1 -- 172.21.15.63:0/2312051273 wait complete. 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.158+0000 7fd2e8200700 1 Processor -- start 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.158+0000 7fd2e8200700 1 -- start start 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e8200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e010f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e8200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2e00fdf60 0x7fd2e010fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e8200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e0110450 con 0x7fd2e00fccb0 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e8200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e01105c0 con 0x7fd2e00fdf60 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e010f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e010f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48164/0 (socket says 172.21.15.63:48164) 2024-03-23T11:44:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e5800700 1 -- 172.21.15.63:0/3865010504 learned_addr learned my addr 172.21.15.63:0/3865010504 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e4e00700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2e00fdf60 0x7fd2e010fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e5800700 1 -- 172.21.15.63:0/3865010504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2e00fdf60 msgr2=0x7fd2e010fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e5800700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2e00fdf60 0x7fd2e010fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.159+0000 7fd2e5800700 1 -- 172.21.15.63:0/3865010504 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d4005040 con 0x7fd2e00fccb0 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.160+0000 7fd2e5800700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e010f910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400b3b0 tx=0x7fd2d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.160+0000 7fd2dd600700 1 -- 172.21.15.63:0/3865010504 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2d400e040 con 0x7fd2e00fccb0 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.160+0000 7fd2dd600700 1 -- 172.21.15.63:0/3865010504 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2d4004290 con 0x7fd2e00fccb0 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.160+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e0114fc0 con 0x7fd2e00fccb0 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.160+0000 7fd2dd600700 1 -- 172.21.15.63:0/3865010504 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2d401c3f0 con 0x7fd2e00fccb0 2024-03-23T11:44:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.160+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e01154f0 con 0x7fd2e00fccb0 2024-03-23T11:44:25.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.161+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e01018f0 con 0x7fd2e00fccb0 2024-03-23T11:44:25.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.162+0000 7fd2dd600700 1 -- 172.21.15.63:0/3865010504 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd2d4013420 con 0x7fd2e00fccb0 2024-03-23T11:44:25.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.163+0000 7fd2dd600700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd2c80692a0 0x7fd2c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:25.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.163+0000 7fd2dd600700 1 -- 172.21.15.63:0/3865010504 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd2d4088450 con 0x7fd2e00fccb0 2024-03-23T11:44:25.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.163+0000 7fd2e4e00700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd2c80692a0 0x7fd2c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:25.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.164+0000 7fd2e4e00700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd2c80692a0 0x7fd2c806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc007fd0 tx=0x7fd2cc009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:25.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.168+0000 7fd2dd600700 1 -- 172.21.15.63:0/3865010504 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd2d40581a0 con 0x7fd2e00fccb0 2024-03-23T11:44:25.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.373+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fd2e0064970 con 0x7fd2e00fccb0 2024-03-23T11:44:25.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.373+0000 7fd2dd600700 1 -- 172.21.15.63:0/3865010504 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd2d4057d30 con 0x7fd2e00fccb0 2024-03-23T11:44:25.358 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.375+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd2c80692a0 msgr2=0x7fd2c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.375+0000 7fd2e8200700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd2c80692a0 0x7fd2c806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc007fd0 tx=0x7fd2cc009040 comp rx=0 tx=0).stop 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.375+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 msgr2=0x7fd2e010f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.375+0000 7fd2e8200700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e010f910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400b3b0 tx=0x7fd2d4009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.376+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 shutdown_connections 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.376+0000 7fd2e8200700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fd2c80692a0 0x7fd2c806b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.376+0000 7fd2e8200700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2e00fccb0 0x7fd2e010f910 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.376+0000 7fd2e8200700 1 --2- 172.21.15.63:0/3865010504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2e00fdf60 0x7fd2e010fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.376+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 >> 172.21.15.63:0/3865010504 conn(0x7fd2e00f8430 msgr2=0x7fd2e00f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.376+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 shutdown_connections 2024-03-23T11:44:25.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.376+0000 7fd2e8200700 1 -- 172.21.15.63:0/3865010504 wait complete. 2024-03-23T11:44:25.379 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:44:25.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.588+0000 7ff9ec000700 1 -- 172.21.15.63:0/2833017624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fcd10 msgr2=0x7ff9e40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.588+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2833017624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fcd10 0x7ff9e40fd120 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8004940 tx=0x7ff9d8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:25.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.588+0000 7ff9ec000700 1 -- 172.21.15.63:0/2833017624 shutdown_connections 2024-03-23T11:44:25.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.588+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2833017624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e40fdfc0 0x7ff9e40fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.588+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2833017624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fcd10 0x7ff9e40fd120 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.589+0000 7ff9ec000700 1 -- 172.21.15.63:0/2833017624 >> 172.21.15.63:0/2833017624 conn(0x7ff9e40f8470 msgr2=0x7ff9e40fa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:25.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.589+0000 7ff9ec000700 1 -- 172.21.15.63:0/2833017624 shutdown_connections 2024-03-23T11:44:25.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.589+0000 7ff9ec000700 1 -- 172.21.15.63:0/2833017624 wait complete. 2024-03-23T11:44:25.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.589+0000 7ff9ec000700 1 Processor -- start 2024-03-23T11:44:25.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9ec000700 1 -- start start 2024-03-23T11:44:25.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9ec000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e40fcd10 0x7ff9e410f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:25.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9ec000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fdfc0 0x7ff9e410fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:25.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9ec000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e4110480 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9ec000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e41105f0 con 0x7ff9e40fcd10 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fdfc0 0x7ff9e410fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fdfc0 0x7ff9e410fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48194/0 (socket says 172.21.15.63:48194) 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e8c00700 1 -- 172.21.15.63:0/2791538763 learned_addr learned my addr 172.21.15.63:0/2791538763 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e9600700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e40fcd10 0x7ff9e410f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e8c00700 1 -- 172.21.15.63:0/2791538763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e40fcd10 msgr2=0x7ff9e410f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e8c00700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e40fcd10 0x7ff9e410f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e8c00700 1 -- 172.21.15.63:0/2791538763 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8005040 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.590+0000 7ff9e9600700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e40fcd10 0x7ff9e410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.591+0000 7ff9e8c00700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fdfc0 0x7ff9e410fe80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0002a50 tx=0x7ff9d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.591+0000 7ff9e1600700 1 -- 172.21.15.63:0/2791538763 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d0007cc0 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.591+0000 7ff9e1600700 1 -- 172.21.15.63:0/2791538763 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9d000f040 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.591+0000 7ff9e1600700 1 -- 172.21.15.63:0/2791538763 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d0009540 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.591+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e4115050 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.591+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e41155e0 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.592+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e4101950 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.593+0000 7ff9e1600700 1 -- 172.21.15.63:0/2791538763 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff9d00096a0 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.594+0000 7ff9e1600700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff9cc0692a0 0x7ff9cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:25.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.594+0000 7ff9e1600700 1 -- 172.21.15.63:0/2791538763 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff9d0086fe0 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.594+0000 7ff9e9600700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff9cc0692a0 0x7ff9cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:25.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.595+0000 7ff9e9600700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff9cc0692a0 0x7ff9cc06b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8004f90 tx=0x7ff9d8009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:25.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.598+0000 7ff9e1600700 1 -- 172.21.15.63:0/2791538763 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff9d0056cb0 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:25 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/632500789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T11:44:25.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:25 smithi063 ceph-mon[35189]: pgmap v219: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-23T11:44:25.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:25 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2877229308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T11:44:25.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:25 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3865010504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T11:44:25.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:25 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/632500789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T11:44:25.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:25 smithi111 ceph-mon[44005]: pgmap v219: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-23T11:44:25.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:25 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2877229308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T11:44:25.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:25 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3865010504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T11:44:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.853+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9e4064970 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.854+0000 7ff9e1600700 1 -- 172.21.15.63:0/2791538763 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4722 (secure 0 0 0) 0x7ff9e4064970 con 0x7ff9e40fdfc0 2024-03-23T11:44:25.840 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:44:25.840 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:44:09.137068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14462,"mds_1":14478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.63:6827/3230334326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3230334326},{"type":"v1","addr":"172.21.15.63:6827","nonce":3230334326}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14478":{"gid":14478,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.63:6829/2516647170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2516647170},{"type":"v1","addr":"172.21.15.63:6829","nonce":2516647170}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi111.atkibb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6827/3582848147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3582848147},{"type":"v1","addr":"172.21.15.111:6827","nonce":3582848147}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24299":{"gid":24299,"name":"cephfs.smithi111.ylnmzy","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6825/315773837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":315773837},{"type":"v1","addr":"172.21.15.111:6825","nonce":315773837}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-23T11:44:25.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff9cc0692a0 msgr2=0x7ff9cc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff9cc0692a0 0x7ff9cc06b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8004f90 tx=0x7ff9d8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fdfc0 msgr2=0x7ff9e410fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fdfc0 0x7ff9e410fe80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0002a50 tx=0x7ff9d00046b0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 shutdown_connections 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff9cc0692a0 0x7ff9cc06b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e40fcd10 0x7ff9e410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 --2- 172.21.15.63:0/2791538763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e40fdfc0 0x7ff9e410fe80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.856+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 >> 172.21.15.63:0/2791538763 conn(0x7ff9e40f8470 msgr2=0x7ff9e40f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.857+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 shutdown_connections 2024-03-23T11:44:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:25.857+0000 7ff9ec000700 1 -- 172.21.15.63:0/2791538763 wait complete. 2024-03-23T11:44:25.844 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 20 2024-03-23T11:44:25.861 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 20, 'max_mds': 2, 'flags': 50} 2024-03-23T11:44:25.861 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-23T11:44:25.885 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-23T11:44:25.885 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-03-23T11:44:25.885 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-23T11:44:25.885 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T11:44:25.885 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:25.885 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-23T11:44:25.885 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T11:44:25.885 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:25.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:25.885 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list 2024-03-23T11:44:25.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:25.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link delete ceph-brx 2024-03-23T11:44:26.002 INFO:teuthology.orchestra.run.smithi063.stderr:Cannot find device "ceph-brx" 2024-03-23T11:44:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:44:26.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:26.006 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2024-03-23T11:44:26.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:26.059 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2024-03-23T11:44:26.136 INFO:teuthology.orchestra.run.smithi111.stderr:Cannot find device "ceph-brx" 2024-03-23T11:44:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:44:26.140 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-23T11:44:26.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-23T11:44:26.575 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:26 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2791538763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T11:44:26.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:26 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2791538763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T11:44:27.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:27 smithi111 ceph-mon[44005]: pgmap v220: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-23T11:44:27.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:27 smithi063 ceph-mon[35189]: pgmap v220: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-23T11:44:28.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.558+0000 7f06dee00700 1 -- 172.21.15.63:0/4041757611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 msgr2=0x7f06d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:28.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.558+0000 7f06dee00700 1 --2- 172.21.15.63:0/4041757611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d81031d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f06c8004940 tx=0x7f06c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:28.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.558+0000 7f06dee00700 1 -- 172.21.15.63:0/4041757611 shutdown_connections 2024-03-23T11:44:28.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.558+0000 7f06dee00700 1 --2- 172.21.15.63:0/4041757611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d81037a0 0x7f06d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:28.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.558+0000 7f06dee00700 1 --2- 172.21.15.63:0/4041757611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d81031d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:28.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.558+0000 7f06dee00700 1 -- 172.21.15.63:0/4041757611 >> 172.21.15.63:0/4041757611 conn(0x7f06d80f89b0 msgr2=0x7f06d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:28.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.559+0000 7f06dee00700 1 -- 172.21.15.63:0/4041757611 shutdown_connections 2024-03-23T11:44:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.559+0000 7f06dee00700 1 -- 172.21.15.63:0/4041757611 wait complete. 2024-03-23T11:44:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.559+0000 7f06dee00700 1 Processor -- start 2024-03-23T11:44:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06dee00700 1 -- start start 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06dee00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d810af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06dee00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d81037a0 0x7f06d810b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06dee00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d810f0d0 con 0x7f06d8102e00 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06dee00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d810b9c0 con 0x7f06d81037a0 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06d7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d810af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06d7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d810af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48218/0 (socket says 172.21.15.63:48218) 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06d7e00700 1 -- 172.21.15.63:0/1250317010 learned_addr learned my addr 172.21.15.63:0/1250317010 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.560+0000 7f06d7400700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d81037a0 0x7f06d810b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06d7e00700 1 -- 172.21.15.63:0/1250317010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d81037a0 msgr2=0x7f06d810b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06d7e00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d81037a0 0x7f06d810b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06d7e00700 1 -- 172.21.15.63:0/1250317010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c8005040 con 0x7f06d8102e00 2024-03-23T11:44:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06d7400700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d81037a0 0x7f06d810b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06d7e00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d810af40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009360 tx=0x7f06c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06cfe00700 1 -- 172.21.15.63:0/1250317010 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c800e040 con 0x7f06d8102e00 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06cfe00700 1 -- 172.21.15.63:0/1250317010 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06c8013420 con 0x7f06d8102e00 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d810bc40 con 0x7f06d8102e00 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06cfe00700 1 -- 172.21.15.63:0/1250317010 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c801c5f0 con 0x7f06d8102e00 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.561+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d811e660 con 0x7f06d8102e00 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.562+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d80fe550 con 0x7f06d8102e00 2024-03-23T11:44:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.563+0000 7f06cfe00700 1 -- 172.21.15.63:0/1250317010 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f06c8013590 con 0x7f06d8102e00 2024-03-23T11:44:28.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.564+0000 7f06cfe00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06bc0692b0 0x7f06bc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:28.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.565+0000 7f06cfe00700 1 -- 172.21.15.63:0/1250317010 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f06c8088410 con 0x7f06d8102e00 2024-03-23T11:44:28.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.565+0000 7f06d7400700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06bc0692b0 0x7f06bc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.565+0000 7f06d7400700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06bc0692b0 0x7f06bc06b760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009fd0 tx=0x7f06c4009ee0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:28.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.570+0000 7f06cfe00700 1 -- 172.21.15.63:0/1250317010 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f06c80546d0 con 0x7f06d8102e00 2024-03-23T11:44:28.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.822+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f06d810c630 con 0x7f06d8102e00 2024-03-23T11:44:28.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.822+0000 7f06cfe00700 1 -- 172.21.15.63:0/1250317010 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+83 (secure 0 0 0) 0x7f06c8057cf0 con 0x7f06d8102e00 2024-03-23T11:44:28.808 INFO:teuthology.orchestra.run.smithi063.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06bc0692b0 msgr2=0x7f06bc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06bc0692b0 0x7f06bc06b760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009fd0 tx=0x7f06c4009ee0 comp rx=0 tx=0).stop 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 msgr2=0x7f06d810af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d810af40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009360 tx=0x7f06c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 shutdown_connections 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f06bc0692b0 0x7f06bc06b760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06d8102e00 0x7f06d810af40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:28.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 --2- 172.21.15.63:0/1250317010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06d81037a0 0x7f06d810b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:28.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 >> 172.21.15.63:0/1250317010 conn(0x7f06d80f89b0 msgr2=0x7f06d80706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:28.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 shutdown_connections 2024-03-23T11:44:28.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:28.825+0000 7f06dee00700 1 -- 172.21.15.63:0/1250317010 wait complete. 2024-03-23T11:44:29.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:28 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1250317010' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-23T11:44:29.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:28 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1250317010' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi063.front.sepia.ceph.com 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:44:29.585 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:44:29.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:29.585 DEBUG:teuthology.orchestra.run.smithi063:> ip addr 2024-03-23T11:44:29.632 INFO:teuthology.orchestra.run.smithi063.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: inet6 ::1/128 scope host 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:a8:b4:76 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: altname enp7s0f0 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:a8:b4:77 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: altname enp7s0f1 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:b6:f8:d6 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:44:29.633 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:b6:f8:d7 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:29.634 INFO:teuthology.orchestra.run.smithi063.stdout: inet 172.21.15.63/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:44:29.634 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft 41941sec preferred_lft 41941sec 2024-03-23T11:44:29.634 INFO:teuthology.orchestra.run.smithi063.stdout: inet6 fe80::ec4:7aff:feb6:f8d7/64 scope link 2024-03-23T11:44:29.634 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:44:29.634 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T11:44:29.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:29.634 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec bash -c ' 2024-03-23T11:44:29.634 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-03-23T11:44:29.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link add name ceph-brx type bridge 2024-03-23T11:44:29.635 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip addr flush dev ceph-brx 2024-03-23T11:44:29.635 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link set ceph-brx up 2024-03-23T11:44:29.635 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T11:44:29.635 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-03-23T11:44:30.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:30.093 DEBUG:teuthology.orchestra.run.smithi063:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:44:30.161 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:29 smithi063 ceph-mon[35189]: pgmap v221: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:44:30.162 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:29 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:30.167 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-03-23T11:44:30.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:30.171 DEBUG:teuthology.orchestra.run.smithi063:> route 2024-03-23T11:44:30.241 INFO:teuthology.orchestra.run.smithi063.stdout:Kernel IP routing table 2024-03-23T11:44:30.241 INFO:teuthology.orchestra.run.smithi063.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:44:30.242 INFO:teuthology.orchestra.run.smithi063.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:44:30.242 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:44:30.242 INFO:teuthology.orchestra.run.smithi063.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:44:30.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:30.243 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec bash -c ' 2024-03-23T11:44:30.243 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-03-23T11:44:30.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:44:30.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:44:30.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:44:30.243 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-03-23T11:44:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:29 smithi111 ceph-mon[44005]: pgmap v221: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:44:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:29 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:30.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:30.908 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list 2024-03-23T11:44:30.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:30.955 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list-id 2024-03-23T11:44:31.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:31.000 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec bash -c ' 2024-03-23T11:44:31.000 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-03-23T11:44:31.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:44:31.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-23T11:44:31.000 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-03-23T11:44:31.140 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-23T11:44:31.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:31.140 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec bash -c ' 2024-03-23T11:44:31.141 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-03-23T11:44:31.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-23T11:44:31.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T11:44:31.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-23T11:44:31.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-23T11:44:31.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-23T11:44:31.141 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-03-23T11:44:31.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:31.405 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec bash -c ' 2024-03-23T11:44:31.405 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-03-23T11:44:31.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link set brx.0 up 2024-03-23T11:44:31.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T11:44:31.405 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-03-23T11:44:31.575 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:44:31.575 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:44:31.575 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:31.619 INFO:teuthology.orchestra.run.smithi063.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:44:31.619 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:44:31.620 DEBUG:teuthology.orchestra.run.smithi063:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:31.703 DEBUG:teuthology.orchestra.run.smithi063:> sudo modprobe fuse 2024-03-23T11:44:31.812 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:44:31.812 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:44:31.879 INFO:teuthology.orchestra.run.smithi063.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:44:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:44:31.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:31.883 DEBUG:teuthology.orchestra.run.smithi063:> ls /sys/fs/fuse/connections 2024-03-23T11:44:31.929 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:44:31.929 DEBUG:teuthology.orchestra.run.smithi063:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:44:31.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo modprobe fuse 2024-03-23T11:44:32.004 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:44:32.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:44:32.064 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:31 smithi063 ceph-mon[35189]: pgmap v222: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:44:32.072 INFO:teuthology.orchestra.run.smithi063.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:44:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:44:32.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:32.075 DEBUG:teuthology.orchestra.run.smithi063:> ls /sys/fs/fuse/connections 2024-03-23T11:44:32.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi063.stderr:2024-03-23T11:44:32.131+0000 7f6d13b50540 -1 init, newargv = 0x559f20788e80 newargc=15 2024-03-23T11:44:32.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi063.stderr:ceph-fuse[95348]: starting ceph client 2024-03-23T11:44:32.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi063.stderr:ceph-fuse[95348]: starting fuse 2024-03-23T11:44:32.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:31 smithi111 ceph-mon[44005]: pgmap v222: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:44:33.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo modprobe fuse 2024-03-23T11:44:33.300 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:44:33.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:44:33.369 INFO:teuthology.orchestra.run.smithi063.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:44:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:44:33.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:33.372 DEBUG:teuthology.orchestra.run.smithi063:> ls /sys/fs/fuse/connections 2024-03-23T11:44:33.417 INFO:teuthology.orchestra.run.smithi063.stdout:159 2024-03-23T11:44:33.417 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [159] 2024-03-23T11:44:33.417 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:44:33.417 DEBUG:teuthology.orchestra.run.smithi063:> import glob 2024-03-23T11:44:33.417 DEBUG:teuthology.orchestra.run.smithi063:> import re 2024-03-23T11:44:33.417 DEBUG:teuthology.orchestra.run.smithi063:> import os 2024-03-23T11:44:33.417 DEBUG:teuthology.orchestra.run.smithi063:> import subprocess 2024-03-23T11:44:33.417 DEBUG:teuthology.orchestra.run.smithi063:> 2024-03-23T11:44:33.417 DEBUG:teuthology.orchestra.run.smithi063:> def _find_admin_socket(client_name): 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> files = glob.glob(asok_path) 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> # Given a non-glob path, it better be there 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> if "*" not in asok_path: 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> assert(len(files) == 1) 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> return files[0] 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> for f in files: 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> contents = proc_f.read() 2024-03-23T11:44:33.418 DEBUG:teuthology.orchestra.run.smithi063:> if mountpoint in contents: 2024-03-23T11:44:33.419 DEBUG:teuthology.orchestra.run.smithi063:> return f 2024-03-23T11:44:33.419 DEBUG:teuthology.orchestra.run.smithi063:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:44:33.419 DEBUG:teuthology.orchestra.run.smithi063:> 2024-03-23T11:44:33.419 DEBUG:teuthology.orchestra.run.smithi063:> print(_find_admin_socket("client.0")) 2024-03-23T11:44:33.419 DEBUG:teuthology.orchestra.run.smithi063:> ' 2024-03-23T11:44:33.600 INFO:teuthology.orchestra.run.smithi063.stdout:/var/run/ceph/ceph-client.0.95348.asok 2024-03-23T11:44:33.778 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.95348.asok 2024-03-23T11:44:33.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:33.778 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.95348.asok status 2024-03-23T11:44:34.000 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:44:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "metadata": { 2024-03-23T11:44:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_sha1": "9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45", 2024-03-23T11:44:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-23T11:44:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "entity_id": "0", 2024-03-23T11:44:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "hostname": "smithi063", 2024-03-23T11:44:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "pid": "95348", 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "root": "/" 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "dentry_count": 0, 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "dentry_pinned_count": 0, 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 14532, 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "inst": { 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "name": { 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "client", 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 14532 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": { 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:44:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2227074864 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": { 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2227074864 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "inst_str": "client.14532 192.168.0.1:0/2227074864", 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "addr_str": "192.168.0.1:0/2227074864", 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "inode_count": 1, 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_epoch": 20, 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_epoch": 46, 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:44:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklisted": false, 2024-03-23T11:44:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs" 2024-03-23T11:44:34.003 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:44:34.014 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-23T11:44:34.074 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:33 smithi063 ceph-mon[35189]: pgmap v223: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-23T11:44:34.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:33 smithi111 ceph-mon[44005]: pgmap v223: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-23T11:44:34.450 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:36.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:35 smithi063 ceph-mon[35189]: pgmap v224: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-23T11:44:36.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:35 smithi111 ceph-mon[44005]: pgmap v224: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-23T11:44:36.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 -- 172.21.15.63:0/293226653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc0105010 msgr2=0x7fbdc01053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:36.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/293226653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc0105010 0x7fbdc01053e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0004940 tx=0x7fbdb0009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:36.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 -- 172.21.15.63:0/293226653 shutdown_connections 2024-03-23T11:44:36.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/293226653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdc01059b0 0x7fbdc00ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:36.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/293226653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc0105010 0x7fbdc01053e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:36.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 -- 172.21.15.63:0/293226653 >> 172.21.15.63:0/293226653 conn(0x7fbdc00faba0 msgr2=0x7fbdc00fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:36.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 -- 172.21.15.63:0/293226653 shutdown_connections 2024-03-23T11:44:36.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.475+0000 7fbdc6e00700 1 -- 172.21.15.63:0/293226653 wait complete. 2024-03-23T11:44:36.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.476+0000 7fbdc6e00700 1 Processor -- start 2024-03-23T11:44:36.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.476+0000 7fbdc6e00700 1 -- start start 2024-03-23T11:44:36.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdc6e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdc0105010 0x7fbdc0115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:36.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdc6e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc01059b0 0x7fbdc01161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:36.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdc6e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc01168d0 con 0x7fbdc01059b0 2024-03-23T11:44:36.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdc6e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc010fc10 con 0x7fbdc0105010 2024-03-23T11:44:36.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdbf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc01059b0 0x7fbdc01161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:36.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdbf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc01059b0 0x7fbdc01161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60986/0 (socket says 172.21.15.63:60986) 2024-03-23T11:44:36.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdbf400700 1 -- 172.21.15.63:0/3841616941 learned_addr learned my addr 172.21.15.63:0/3841616941 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:36.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.477+0000 7fbdbfe00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdc0105010 0x7fbdc0115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:36.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdbf400700 1 -- 172.21.15.63:0/3841616941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdc0105010 msgr2=0x7fbdc0115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:36.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdbf400700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdc0105010 0x7fbdc0115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:36.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdbf400700 1 -- 172.21.15.63:0/3841616941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0005040 con 0x7fbdc01059b0 2024-03-23T11:44:36.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdbfe00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdc0105010 0x7fbdc0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:44:36.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdbf400700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc01059b0 0x7fbdc01161f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00b790 tx=0x7fbdac009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:36.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdb7e00700 1 -- 172.21.15.63:0/3841616941 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdac00f6f0 con 0x7fbdc01059b0 2024-03-23T11:44:36.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdb7e00700 1 -- 172.21.15.63:0/3841616941 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdac00fd00 con 0x7fbdc01059b0 2024-03-23T11:44:36.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdb7e00700 1 -- 172.21.15.63:0/3841616941 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdac0103a0 con 0x7fbdc01059b0 2024-03-23T11:44:36.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc010fe90 con 0x7fbdc01059b0 2024-03-23T11:44:36.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.478+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc0110450 con 0x7fbdc01059b0 2024-03-23T11:44:36.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.479+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc0100760 con 0x7fbdc01059b0 2024-03-23T11:44:36.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.480+0000 7fbdb7e00700 1 -- 172.21.15.63:0/3841616941 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbdac00f850 con 0x7fbdc01059b0 2024-03-23T11:44:36.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.481+0000 7fbdb7e00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fbda40692b0 0x7fbda406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:36.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.481+0000 7fbdb7e00700 1 -- 172.21.15.63:0/3841616941 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbdac086fe0 con 0x7fbdc01059b0 2024-03-23T11:44:36.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.481+0000 7fbdbfe00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fbda40692b0 0x7fbda406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:36.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.482+0000 7fbdbfe00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fbda40692b0 0x7fbda406b760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0000c00 tx=0x7fbdb0009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:36.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.486+0000 7fbdb7e00700 1 -- 172.21.15.63:0/3841616941 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbdac056d10 con 0x7fbdc01059b0 2024-03-23T11:44:36.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.741+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbdc004bed0 con 0x7fbdc01059b0 2024-03-23T11:44:36.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.741+0000 7fbdb7e00700 1 -- 172.21.15.63:0/3841616941 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+83 (secure 0 0 0) 0x7fbdac0568a0 con 0x7fbdc01059b0 2024-03-23T11:44:36.727 INFO:teuthology.orchestra.run.smithi063.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fbda40692b0 msgr2=0x7fbda406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fbda40692b0 0x7fbda406b760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0000c00 tx=0x7fbdb0009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc01059b0 msgr2=0x7fbdc01161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc01059b0 0x7fbdc01161f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00b790 tx=0x7fbdac009220 comp rx=0 tx=0).stop 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 shutdown_connections 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fbda40692b0 0x7fbda406b760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdc0105010 0x7fbdc0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.743+0000 7fbdc6e00700 1 --2- 172.21.15.63:0/3841616941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdc01059b0 0x7fbdc01161f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.744+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 >> 172.21.15.63:0/3841616941 conn(0x7fbdc00faba0 msgr2=0x7fbdc00fbc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.744+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 shutdown_connections 2024-03-23T11:44:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:36.744+0000 7fbdc6e00700 1 -- 172.21.15.63:0/3841616941 wait complete. 2024-03-23T11:44:37.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:36 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3841616941' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-23T11:44:37.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:36 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3841616941' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-23T11:44:37.458 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:44:37.458 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:44:37.458 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2024-03-23T11:44:37.458 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T11:44:37.458 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:37.459 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:44:37.459 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:44:37.459 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:44:37.459 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T11:44:37.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:37.459 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2024-03-23T11:44:37.510 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2024-03-23T11:44:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:44:37.512 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:37.512 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:44:37.512 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2024-03-23T11:44:37.512 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:44:37.512 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41918sec preferred_lft 41918sec 2024-03-23T11:44:37.512 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2024-03-23T11:44:37.512 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:44:37.513 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T11:44:37.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:37.513 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-23T11:44:37.513 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-23T11:44:37.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2024-03-23T11:44:37.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2024-03-23T11:44:37.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2024-03-23T11:44:37.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T11:44:37.513 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-23T11:44:37.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:37.943 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:44:38.012 INFO:teuthology.orchestra.run.smithi111.stdout:1 2024-03-23T11:44:38.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:38.015 DEBUG:teuthology.orchestra.run.smithi111:> route 2024-03-23T11:44:38.082 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2024-03-23T11:44:38.082 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:44:38.082 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:44:38.082 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:44:38.083 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:44:38.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:38.084 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-23T11:44:38.084 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-23T11:44:38.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:44:38.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:44:38.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:44:38.084 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-23T11:44:38.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:37 smithi111 ceph-mon[44005]: pgmap v225: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-23T11:44:38.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:37 smithi063 ceph-mon[35189]: pgmap v225: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-23T11:44:38.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:38.724 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2024-03-23T11:44:38.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:38.769 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2024-03-23T11:44:38.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:38.855 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-23T11:44:38.855 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-23T11:44:38.855 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T11:44:38.855 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-23T11:44:38.855 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-23T11:44:38.996 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-23T11:44:38.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:38.996 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-23T11:44:38.996 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-23T11:44:38.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-23T11:44:38.997 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T11:44:38.997 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-23T11:44:38.997 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-23T11:44:38.997 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-23T11:44:38.997 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-23T11:44:39.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:39.273 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-23T11:44:39.273 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-23T11:44:39.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2024-03-23T11:44:39.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T11:44:39.273 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-23T11:44:39.440 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-23T11:44:39.440 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:44:39.441 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:39.485 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T11:44:39.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:44:39.485 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:39.568 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2024-03-23T11:44:39.681 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:44:39.681 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:44:39.747 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:44:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:44:39.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:39.750 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2024-03-23T11:44:39.794 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:44:39.795 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-23T11:44:39.839 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2024-03-23T11:44:39.906 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:44:39.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:44:39.971 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:44:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:44:39.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:39.974 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2024-03-23T11:44:40.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi111.stderr:2024-03-23T11:44:40.062+0000 7fa317c40540 -1 init, newargv = 0x562ef3284fa0 newargc=15 2024-03-23T11:44:40.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi111.stderr:ceph-fuse[74224]: starting ceph client 2024-03-23T11:44:40.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi111.stderr:ceph-fuse[74224]: starting fuse 2024-03-23T11:44:40.198 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:39 smithi063 ceph-mon[35189]: pgmap v226: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-23T11:44:40.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:39 smithi111 ceph-mon[44005]: pgmap v226: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-23T11:44:41.124 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2024-03-23T11:44:41.200 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:44:41.200 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:44:41.267 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:44:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:44:41.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:41.270 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2024-03-23T11:44:41.314 INFO:teuthology.orchestra.run.smithi111.stdout:123 2024-03-23T11:44:41.314 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [123] 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> import re 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> import os 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2024-03-23T11:44:41.315 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> return f 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.1")) 2024-03-23T11:44:41.316 DEBUG:teuthology.orchestra.run.smithi111:> ' 2024-03-23T11:44:41.493 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.1.74224.asok 2024-03-23T11:44:41.672 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.74224.asok 2024-03-23T11:44:41.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:41.672 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.74224.asok status 2024-03-23T11:44:41.885 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-23T11:44:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2024-03-23T11:44:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45", 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "1", 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "74224", 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24341, 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2024-03-23T11:44:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 24341 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2514609317 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2514609317 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.24341 192.168.0.1:0/2514609317", 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2514609317", 2024-03-23T11:44:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2024-03-23T11:44:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 20, 2024-03-23T11:44:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 46, 2024-03-23T11:44:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:44:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2024-03-23T11:44:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2024-03-23T11:44:41.888 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-23T11:44:41.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:41.900 DEBUG:teuthology.orchestra.run.smithi063:> stat --file-system '--printf=%T 2024-03-23T11:44:41.900 DEBUG:teuthology.orchestra.run.smithi063:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:41.953 INFO:teuthology.orchestra.run.smithi063.stdout:fuseblk 2024-03-23T11:44:41.953 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:41.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:41.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:42.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:42.031 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2024-03-23T11:44:42.031 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:42.077 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2024-03-23T11:44:42.077 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:42.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:44:42.077 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:42.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:41 smithi063 ceph-mon[35189]: pgmap v227: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-23T11:44:42.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:41 smithi111 ceph-mon[44005]: pgmap v227: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-23T11:44:43.779 INFO:teuthology.run_tasks:Running task print... 2024-03-23T11:44:43.790 INFO:teuthology.task.print:**** done client 2024-03-23T11:44:43.790 INFO:teuthology.run_tasks:Running task parallel... 2024-03-23T11:44:43.895 INFO:teuthology.task.parallel:starting parallel... 2024-03-23T11:44:43.895 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-23T11:44:43.895 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-23T11:44:43.896 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:44:43.896 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-23T11:44:43.896 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-23T11:44:43.897 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-23T11:44:43.900 INFO:tasks.workunit:Pulling workunits from ref b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:44:43.900 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-23T11:44:43.900 INFO:tasks.workunit:timeout=3h 2024-03-23T11:44:43.900 INFO:tasks.workunit:cleanup=True 2024-03-23T11:44:43.900 DEBUG:teuthology.orchestra.run.smithi063:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:43.958 INFO:teuthology.orchestra.run.smithi063.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:43.958 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-23T11:44:43.959 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 9fh/159d Inode: 1 Links: 2 2024-03-23T11:44:43.959 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:44:43.959 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-23T11:44:43.959 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-23T11:44:43.959 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-23 11:43:54.236838901 +0000 2024-03-23T11:44:43.959 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-23 11:44:42.041445780 +0000 2024-03-23T11:44:43.959 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-03-23T11:44:43.959 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-23T11:44:43.959 DEBUG:teuthology.orchestra.run.smithi063:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-23T11:44:44.078 DEBUG:teuthology.orchestra.run.smithi111:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:44.130 INFO:teuthology.orchestra.run.smithi111.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-23 11:44:44.084518136 +0000 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-23 11:44:44.084518136 +0000 2024-03-23T11:44:44.131 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2024-03-23T11:44:44.131 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-23T11:44:44.131 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-23T11:44:44.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:43 smithi111 ceph-mon[44005]: pgmap v228: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-23T11:44:44.211 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:44:44.211 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:44:44.261 INFO:tasks.workunit.client.0.smithi063.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-23T11:44:44.298 INFO:tasks.workunit.client.1.smithi111.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-23T11:44:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:44.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:43 smithi063 ceph-mon[35189]: pgmap v228: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-23T11:44:45.241 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:44 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:45.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:44 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:44:46.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:45 smithi063 ceph-mon[35189]: pgmap v229: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-23T11:44:46.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.406+0000 7ff09c000700 1 -- 172.21.15.63:0/1530311149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 msgr2=0x7ff0940fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:46.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.406+0000 7ff09c000700 1 --2- 172.21.15.63:0/1530311149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff0940fd310 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff088004940 tx=0x7ff088009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:46.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.407+0000 7ff09c000700 1 -- 172.21.15.63:0/1530311149 shutdown_connections 2024-03-23T11:44:46.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.407+0000 7ff09c000700 1 --2- 172.21.15.63:0/1530311149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0940fe0d0 0x7ff0940fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:46.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.407+0000 7ff09c000700 1 --2- 172.21.15.63:0/1530311149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff0940fd310 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:46.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.407+0000 7ff09c000700 1 -- 172.21.15.63:0/1530311149 >> 172.21.15.63:0/1530311149 conn(0x7ff0940f8560 msgr2=0x7ff0940fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:46.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.407+0000 7ff09c000700 1 -- 172.21.15.63:0/1530311149 shutdown_connections 2024-03-23T11:44:46.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.407+0000 7ff09c000700 1 -- 172.21.15.63:0/1530311149 wait complete. 2024-03-23T11:44:46.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.407+0000 7ff09c000700 1 Processor -- start 2024-03-23T11:44:46.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff09c000700 1 -- start start 2024-03-23T11:44:46.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff09c000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff09410fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:46.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff09c000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0940fe0d0 0x7ff094110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:46.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff09c000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff094110660 con 0x7ff0940fcee0 2024-03-23T11:44:46.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff09c000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff094110eb0 con 0x7ff0940fe0d0 2024-03-23T11:44:46.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff099600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff09410fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:46.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff099600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff09410fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48964/0 (socket says 172.21.15.63:48964) 2024-03-23T11:44:46.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.408+0000 7ff099600700 1 -- 172.21.15.63:0/2240236001 learned_addr learned my addr 172.21.15.63:0/2240236001 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:46.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff098c00700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0940fe0d0 0x7ff094110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:46.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff099600700 1 -- 172.21.15.63:0/2240236001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0940fe0d0 msgr2=0x7ff094110060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:46.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff099600700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0940fe0d0 0x7ff094110060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff099600700 1 -- 172.21.15.63:0/2240236001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088005040 con 0x7ff0940fcee0 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff098c00700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0940fe0d0 0x7ff094110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff099600700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff09410fb20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff088009380 tx=0x7ff088009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff091600700 1 -- 172.21.15.63:0/2240236001 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff08800e040 con 0x7ff0940fcee0 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff091600700 1 -- 172.21.15.63:0/2240236001 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff088004290 con 0x7ff0940fcee0 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff091600700 1 -- 172.21.15.63:0/2240236001 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff08801c3f0 con 0x7ff0940fcee0 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff094111130 con 0x7ff0940fcee0 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.409+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff094111670 con 0x7ff0940fcee0 2024-03-23T11:44:46.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.410+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff094101a60 con 0x7ff0940fcee0 2024-03-23T11:44:46.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.411+0000 7ff091600700 1 -- 172.21.15.63:0/2240236001 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff088013420 con 0x7ff0940fcee0 2024-03-23T11:44:46.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.412+0000 7ff091600700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff07c0692b0 0x7ff07c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:46.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.412+0000 7ff091600700 1 -- 172.21.15.63:0/2240236001 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff088088610 con 0x7ff0940fcee0 2024-03-23T11:44:46.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.413+0000 7ff098c00700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff07c0692b0 0x7ff07c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:46.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.413+0000 7ff098c00700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff07c0692b0 0x7ff07c06b760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff080009b80 tx=0x7ff08000b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:46.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.417+0000 7ff091600700 1 -- 172.21.15.63:0/2240236001 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff0880582e0 con 0x7ff0940fcee0 2024-03-23T11:44:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:45 smithi111 ceph-mon[44005]: pgmap v229: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-23T11:44:46.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.621+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7ff09405f5f0 con 0x7ff0940fcee0 2024-03-23T11:44:46.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.621+0000 7ff091600700 1 -- 172.21.15.63:0/2240236001 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7ff088057e70 con 0x7ff0940fcee0 2024-03-23T11:44:46.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff07c0692b0 msgr2=0x7ff07c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:46.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff07c0692b0 0x7ff07c06b760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff080009b80 tx=0x7ff08000b040 comp rx=0 tx=0).stop 2024-03-23T11:44:46.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 msgr2=0x7ff09410fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff09410fb20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff088009380 tx=0x7ff088009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 shutdown_connections 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ff07c0692b0 0x7ff07c06b760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0940fcee0 0x7ff09410fb20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 --2- 172.21.15.63:0/2240236001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0940fe0d0 0x7ff094110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.623+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 >> 172.21.15.63:0/2240236001 conn(0x7ff0940f8560 msgr2=0x7ff0940f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.624+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 shutdown_connections 2024-03-23T11:44:46.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:46.624+0000 7ff09c000700 1 -- 172.21.15.63:0/2240236001 wait complete. 2024-03-23T11:44:47.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-23T11:44:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:48.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:47 smithi111 ceph-mon[44005]: pgmap v230: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:44:48.248 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:47 smithi063 ceph-mon[35189]: pgmap v230: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:44:49.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 -- 172.21.15.63:0/1023900801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fcee0 msgr2=0x7f140c0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:49.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 --2- 172.21.15.63:0/1023900801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fcee0 0x7f140c0fd310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f13fc004940 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:49.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 -- 172.21.15.63:0/1023900801 shutdown_connections 2024-03-23T11:44:49.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 --2- 172.21.15.63:0/1023900801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f140c0fe0d0 0x7f140c0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:49.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 --2- 172.21.15.63:0/1023900801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fcee0 0x7f140c0fd310 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:49.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 -- 172.21.15.63:0/1023900801 >> 172.21.15.63:0/1023900801 conn(0x7f140c0f8560 msgr2=0x7f140c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:49.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 -- 172.21.15.63:0/1023900801 shutdown_connections 2024-03-23T11:44:49.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.702+0000 7f1411400700 1 -- 172.21.15.63:0/1023900801 wait complete. 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.703+0000 7f1411400700 1 Processor -- start 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.703+0000 7f1411400700 1 -- start start 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.704+0000 7f1411400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f140c0fcee0 0x7f140c10fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.704+0000 7f1411400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fe0d0 0x7f140c1100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.704+0000 7f1411400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1106f0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.704+0000 7f1411400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c110f40 con 0x7f140c0fcee0 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.704+0000 7f1409e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fe0d0 0x7f140c1100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.704+0000 7f1409e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fe0d0 0x7f140c1100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48988/0 (socket says 172.21.15.63:48988) 2024-03-23T11:44:49.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.704+0000 7f1409e00700 1 -- 172.21.15.63:0/2365937866 learned_addr learned my addr 172.21.15.63:0/2365937866 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:49.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f140a800700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f140c0fcee0 0x7f140c10fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1409e00700 1 -- 172.21.15.63:0/2365937866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f140c0fcee0 msgr2=0x7f140c10fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1409e00700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f140c0fcee0 0x7f140c10fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1409e00700 1 -- 172.21.15.63:0/2365937866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc005040 con 0x7f140c0fe0d0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1409e00700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fe0d0 0x7f140c1100f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f13f800eb70 tx=0x7f13f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1402a00700 1 -- 172.21.15.63:0/2365937866 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f8007cc0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1402a00700 1 -- 172.21.15.63:0/2365937866 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13f800f040 con 0x7f140c0fe0d0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140c1111c0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1402a00700 1 -- 172.21.15.63:0/2365937866 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f80095f0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.705+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c111700 con 0x7f140c0fe0d0 2024-03-23T11:44:49.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.706+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140c05f5f0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.707+0000 7f1402a00700 1 -- 172.21.15.63:0/2365937866 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f13f8009750 con 0x7f140c0fe0d0 2024-03-23T11:44:49.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.708+0000 7f1402a00700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f13f00692b0 0x7f13f006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:49.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.708+0000 7f1402a00700 1 -- 172.21.15.63:0/2365937866 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f13f8086ef0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.709+0000 7f140a800700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f13f00692b0 0x7f13f006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:49.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.709+0000 7f140a800700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f13f00692b0 0x7f13f006b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00b0d0 tx=0x7f13fc009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:49.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.713+0000 7f1402a00700 1 -- 172.21.15.63:0/2365937866 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f13f8056b40 con 0x7f140c0fe0d0 2024-03-23T11:44:49.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.921+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f140c04bed0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.921+0000 7f1402a00700 1 -- 172.21.15.63:0/2365937866 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f13f80566d0 con 0x7f140c0fe0d0 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f13f00692b0 msgr2=0x7f13f006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f13f00692b0 0x7f13f006b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00b0d0 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fe0d0 msgr2=0x7f140c1100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fe0d0 0x7f140c1100f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f13f800eb70 tx=0x7f13f800b040 comp rx=0 tx=0).stop 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 shutdown_connections 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f13f00692b0 0x7f13f006b760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f140c0fcee0 0x7f140c10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 --2- 172.21.15.63:0/2365937866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f140c0fe0d0 0x7f140c1100f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:49.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.923+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 >> 172.21.15.63:0/2365937866 conn(0x7f140c0f8560 msgr2=0x7f140c0f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:49.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.924+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 shutdown_connections 2024-03-23T11:44:49.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:49.924+0000 7f1411400700 1 -- 172.21.15.63:0/2365937866 wait complete. 2024-03-23T11:44:50.197 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:49 smithi063 ceph-mon[35189]: pgmap v231: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-23T11:44:50.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:49 smithi111 ceph-mon[44005]: pgmap v231: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-23T11:44:50.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-23T11:44:51.061 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:52.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:51 smithi063 ceph-mon[35189]: pgmap v232: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-23T11:44:52.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:51 smithi111 ceph-mon[44005]: pgmap v232: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-23T11:44:53.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 -- 172.21.15.63:0/1067888591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 msgr2=0x7ffb680fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:53.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/1067888591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb680fd310 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ffb58004940 tx=0x7ffb58009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:53.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 -- 172.21.15.63:0/1067888591 shutdown_connections 2024-03-23T11:44:53.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/1067888591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680fe0d0 0x7ffb680fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:53.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/1067888591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb680fd310 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:53.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 -- 172.21.15.63:0/1067888591 >> 172.21.15.63:0/1067888591 conn(0x7ffb680f8560 msgr2=0x7ffb680fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:53.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 -- 172.21.15.63:0/1067888591 shutdown_connections 2024-03-23T11:44:53.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.158+0000 7ffb6dc00700 1 -- 172.21.15.63:0/1067888591 wait complete. 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.159+0000 7ffb6dc00700 1 Processor -- start 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.159+0000 7ffb6dc00700 1 -- start start 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb6dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb6810fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb6dc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680fe0d0 0x7ffb681100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb6dc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681106d0 con 0x7ffb680fcee0 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb6dc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68110f20 con 0x7ffb680fe0d0 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb6810fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:53.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb6810fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56234/0 (socket says 172.21.15.63:56234) 2024-03-23T11:44:53.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66a00700 1 -- 172.21.15.63:0/151412987 learned_addr learned my addr 172.21.15.63:0/151412987 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:53.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66000700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680fe0d0 0x7ffb681100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:53.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66a00700 1 -- 172.21.15.63:0/151412987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680fe0d0 msgr2=0x7ffb681100d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:53.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66a00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680fe0d0 0x7ffb681100d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:53.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66a00700 1 -- 172.21.15.63:0/151412987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58005040 con 0x7ffb680fcee0 2024-03-23T11:44:53.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.160+0000 7ffb66000700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680fe0d0 0x7ffb681100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:44:53.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.161+0000 7ffb66a00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb6810fb90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffb58009380 tx=0x7ffb58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:53.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.161+0000 7ffb5ea00700 1 -- 172.21.15.63:0/151412987 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb5800e040 con 0x7ffb680fcee0 2024-03-23T11:44:53.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.161+0000 7ffb5ea00700 1 -- 172.21.15.63:0/151412987 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb58004290 con 0x7ffb680fcee0 2024-03-23T11:44:53.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.161+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681111a0 con 0x7ffb680fcee0 2024-03-23T11:44:53.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.161+0000 7ffb5ea00700 1 -- 172.21.15.63:0/151412987 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb5801c3f0 con 0x7ffb680fcee0 2024-03-23T11:44:53.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.161+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb681116e0 con 0x7ffb680fcee0 2024-03-23T11:44:53.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.162+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb68101a60 con 0x7ffb680fcee0 2024-03-23T11:44:53.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.163+0000 7ffb5ea00700 1 -- 172.21.15.63:0/151412987 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ffb58013420 con 0x7ffb680fcee0 2024-03-23T11:44:53.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.164+0000 7ffb5ea00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb4c0692b0 0x7ffb4c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:53.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.164+0000 7ffb5ea00700 1 -- 172.21.15.63:0/151412987 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ffb580884d0 con 0x7ffb680fcee0 2024-03-23T11:44:53.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.164+0000 7ffb66000700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb4c0692b0 0x7ffb4c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:53.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.165+0000 7ffb66000700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb4c0692b0 0x7ffb4c06b760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009fd0 tx=0x7ffb5400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:53.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.168+0000 7ffb5ea00700 1 -- 172.21.15.63:0/151412987 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ffb580582e0 con 0x7ffb680fcee0 2024-03-23T11:44:53.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.376+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7ffb681059c0 con 0x7ffb680fcee0 2024-03-23T11:44:53.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.377+0000 7ffb5ea00700 1 -- 172.21.15.63:0/151412987 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7ffb58057e70 con 0x7ffb680fcee0 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb4c0692b0 msgr2=0x7ffb4c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb4c0692b0 0x7ffb4c06b760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009fd0 tx=0x7ffb5400b040 comp rx=0 tx=0).stop 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 msgr2=0x7ffb6810fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb6810fb90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffb58009380 tx=0x7ffb58009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 shutdown_connections 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7ffb4c0692b0 0x7ffb4c06b760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb680fcee0 0x7ffb6810fb90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 --2- 172.21.15.63:0/151412987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680fe0d0 0x7ffb681100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 >> 172.21.15.63:0/151412987 conn(0x7ffb680f8560 msgr2=0x7ffb680f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:53.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 shutdown_connections 2024-03-23T11:44:53.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:53.379+0000 7ffb6dc00700 1 -- 172.21.15.63:0/151412987 wait complete. 2024-03-23T11:44:54.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:53 smithi063 ceph-mon[35189]: pgmap v233: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-23T11:44:54.204 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-23T11:44:54.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:53 smithi111 ceph-mon[44005]: pgmap v233: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-23T11:44:54.643 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:44:56.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:55 smithi063 ceph-mon[35189]: pgmap v234: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-23T11:44:56.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:55 smithi111 ceph-mon[44005]: pgmap v234: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-23T11:44:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 -- 172.21.15.63:0/4080812951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 msgr2=0x7f9d40070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 --2- 172.21.15.63:0/4080812951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d40070e10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c004940 tx=0x7f9d2c009e30 comp rx=0 tx=0).stop 2024-03-23T11:44:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 -- 172.21.15.63:0/4080812951 shutdown_connections 2024-03-23T11:44:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 --2- 172.21.15.63:0/4080812951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d40070e10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 --2- 172.21.15.63:0/4080812951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d40071f50 0x7f9d40070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 -- 172.21.15.63:0/4080812951 >> 172.21.15.63:0/4080812951 conn(0x7f9d400f8560 msgr2=0x7f9d400fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:44:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 -- 172.21.15.63:0/4080812951 shutdown_connections 2024-03-23T11:44:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.741+0000 7f9d48600700 1 -- 172.21.15.63:0/4080812951 wait complete. 2024-03-23T11:44:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.742+0000 7f9d48600700 1 Processor -- start 2024-03-23T11:44:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.742+0000 7f9d48600700 1 -- start start 2024-03-23T11:44:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.742+0000 7f9d48600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d4010faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.742+0000 7f9d48600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d40071f50 0x7f9d4010ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d48600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d401105e0 con 0x7f9d400709a0 2024-03-23T11:44:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d48600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d40110e30 con 0x7f9d40071f50 2024-03-23T11:44:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d4010faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d4010faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56258/0 (socket says 172.21.15.63:56258) 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45c00700 1 -- 172.21.15.63:0/3416121376 learned_addr learned my addr 172.21.15.63:0/3416121376 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45200700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d40071f50 0x7f9d4010ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45c00700 1 -- 172.21.15.63:0/3416121376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d40071f50 msgr2=0x7f9d4010ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45c00700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d40071f50 0x7f9d4010ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45c00700 1 -- 172.21.15.63:0/3416121376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c005040 con 0x7f9d400709a0 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.743+0000 7f9d45200700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d40071f50 0x7f9d4010ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:44:56.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.744+0000 7f9d45c00700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d4010faa0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9d34002a80 tx=0x7f9d340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:56.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.744+0000 7f9d3d600700 1 -- 172.21.15.63:0/3416121376 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d34007cc0 con 0x7f9d400709a0 2024-03-23T11:44:56.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.744+0000 7f9d3d600700 1 -- 172.21.15.63:0/3416121376 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d3400f040 con 0x7f9d400709a0 2024-03-23T11:44:56.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.744+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d40111110 con 0x7f9d400709a0 2024-03-23T11:44:56.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.744+0000 7f9d3d600700 1 -- 172.21.15.63:0/3416121376 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d340095d0 con 0x7f9d400709a0 2024-03-23T11:44:56.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.744+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d40111650 con 0x7f9d400709a0 2024-03-23T11:44:56.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.745+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4005f5f0 con 0x7f9d400709a0 2024-03-23T11:44:56.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.746+0000 7f9d3d600700 1 -- 172.21.15.63:0/3416121376 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9d34009730 con 0x7f9d400709a0 2024-03-23T11:44:56.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.747+0000 7f9d3d600700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d280692a0 0x7f9d2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:44:56.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.747+0000 7f9d3d600700 1 -- 172.21.15.63:0/3416121376 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9d34086b10 con 0x7f9d400709a0 2024-03-23T11:44:56.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.747+0000 7f9d45200700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d280692a0 0x7f9d2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:44:56.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.748+0000 7f9d45200700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d280692a0 0x7f9d2806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c000c00 tx=0x7f9d2c009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:44:56.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.751+0000 7f9d3d600700 1 -- 172.21.15.63:0/3416121376 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9d3404d190 con 0x7f9d400709a0 2024-03-23T11:44:56.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:44:56.984+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d40111a90 con 0x7f9d280692a0 2024-03-23T11:44:58.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:57 smithi111 ceph-mon[44005]: pgmap v235: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-23T11:44:58.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:57 smithi111 ceph-mon[44005]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:44:58.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:57 smithi063 ceph-mon[35189]: pgmap v235: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-23T11:44:58.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:57 smithi063 ceph-mon[35189]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:00.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:59 smithi063 ceph-mon[35189]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-23T11:45:00.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:44:59 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:59 smithi111 ceph-mon[44005]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-23T11:45:00.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:44:59 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:02.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:01 smithi063 ceph-mon[35189]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-23T11:45:02.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:01 smithi111 ceph-mon[44005]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-23T11:45:04.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:03 smithi063 ceph-mon[35189]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:45:04.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:03 smithi111 ceph-mon[44005]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:45:06.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:05 smithi063 ceph-mon[35189]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-23T11:45:06.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:05 smithi111 ceph-mon[44005]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-23T11:45:08.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:07 smithi111 ceph-mon[44005]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:45:08.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:07 smithi063 ceph-mon[35189]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:45:10.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:09 smithi063 ceph-mon[35189]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-23T11:45:10.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:09 smithi111 ceph-mon[44005]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-23T11:45:12.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:12 smithi063 ceph-mon[35189]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:12.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:12 smithi111 ceph-mon[44005]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:14.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:14 smithi063 ceph-mon[35189]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:14.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:14 smithi111 ceph-mon[44005]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:15.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:15.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:16.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:16 smithi063 ceph-mon[35189]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:16.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:16 smithi111 ceph-mon[44005]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:18.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:18 smithi111 ceph-mon[44005]: pgmap v245: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:18.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:18 smithi063 ceph-mon[35189]: pgmap v245: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:20.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:20 smithi063 ceph-mon[35189]: pgmap v246: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:20.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:20 smithi111 ceph-mon[44005]: pgmap v246: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:22.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:22 smithi063 ceph-mon[35189]: pgmap v247: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:22.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:22 smithi111 ceph-mon[44005]: pgmap v247: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:23.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:23 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:45:23.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:23 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:45:24.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:24 smithi063 ceph-mon[35189]: pgmap v248: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:24.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:45:24.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:45:24.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:24 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:24.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:24 smithi111 ceph-mon[44005]: pgmap v248: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:24.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:45:24.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:45:24.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:24 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:25.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.156+0000 7f9d3d600700 1 -- 172.21.15.63:0/3416121376 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f9d40111a90 con 0x7f9d280692a0 2024-03-23T11:45:25.143 INFO:teuthology.orchestra.run.smithi063.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:45:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.159+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d280692a0 msgr2=0x7f9d2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.159+0000 7f9d48600700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d280692a0 0x7f9d2806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c000c00 tx=0x7f9d2c009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.159+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 msgr2=0x7f9d4010faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:25.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.159+0000 7f9d48600700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d4010faa0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9d34002a80 tx=0x7f9d340046b0 comp rx=0 tx=0).stop 2024-03-23T11:45:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.160+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 shutdown_connections 2024-03-23T11:45:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.160+0000 7f9d48600700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d280692a0 0x7f9d2806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.160+0000 7f9d48600700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d400709a0 0x7f9d4010faa0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.160+0000 7f9d48600700 1 --2- 172.21.15.63:0/3416121376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d40071f50 0x7f9d4010ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.160+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 >> 172.21.15.63:0/3416121376 conn(0x7f9d400f8560 msgr2=0x7f9d400f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.160+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 shutdown_connections 2024-03-23T11:45:25.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:25.160+0000 7f9d48600700 1 -- 172.21.15.63:0/3416121376 wait complete. 2024-03-23T11:45:25.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-23T11:45:26.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:26 smithi063 ceph-mon[35189]: pgmap v249: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:26.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:26 smithi063 ceph-mon[35189]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:45:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:45:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:45:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:45:26.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:26 smithi111 ceph-mon[44005]: pgmap v249: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:26 smithi111 ceph-mon[44005]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:45:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:45:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:45:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:45:26.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:26.515 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:45:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:27 smithi063 ceph-mon[35189]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:45:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:27 smithi111 ceph-mon[44005]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:45:28.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:28 smithi111 ceph-mon[44005]: pgmap v250: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:28.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:28 smithi063 ceph-mon[35189]: pgmap v250: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 -- 172.21.15.63:0/647293479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8066cd0 msgr2=0x7f9ea8101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 --2- 172.21.15.63:0/647293479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8066cd0 0x7f9ea8101c00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9e98004940 tx=0x7f9e98009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 -- 172.21.15.63:0/647293479 shutdown_connections 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 --2- 172.21.15.63:0/647293479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ea8102140 0x7f9ea8104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 --2- 172.21.15.63:0/647293479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8066cd0 0x7f9ea8101c00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 -- 172.21.15.63:0/647293479 >> 172.21.15.63:0/647293479 conn(0x7f9ea80f7270 msgr2=0x7f9ea80f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 -- 172.21.15.63:0/647293479 shutdown_connections 2024-03-23T11:45:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.721+0000 7f9eaca00700 1 -- 172.21.15.63:0/647293479 wait complete. 2024-03-23T11:45:29.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.722+0000 7f9eaca00700 1 Processor -- start 2024-03-23T11:45:29.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.722+0000 7f9eaca00700 1 -- start start 2024-03-23T11:45:29.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9eaca00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ea8066cd0 0x7f9ea810f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:29.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9eaca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8102140 0x7f9ea810fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:29.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9eaca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8110400 con 0x7f9ea8102140 2024-03-23T11:45:29.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9eaca00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8110c50 con 0x7f9ea8066cd0 2024-03-23T11:45:29.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9ea6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8102140 0x7f9ea810fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:29.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9ea7400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ea8066cd0 0x7f9ea810f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:29.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9ea6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8102140 0x7f9ea810fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37472/0 (socket says 172.21.15.63:37472) 2024-03-23T11:45:29.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.723+0000 7f9ea6a00700 1 -- 172.21.15.63:0/3050412788 learned_addr learned my addr 172.21.15.63:0/3050412788 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:45:29.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9ea6a00700 1 -- 172.21.15.63:0/3050412788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ea8066cd0 msgr2=0x7f9ea810f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:29.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9ea6a00700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ea8066cd0 0x7f9ea810f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:29.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9ea6a00700 1 -- 172.21.15.63:0/3050412788 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98005040 con 0x7f9ea8102140 2024-03-23T11:45:29.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9ea6a00700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8102140 0x7f9ea810fe00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400eb40 tx=0x7f9e9400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:29.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9e9f400700 1 -- 172.21.15.63:0/3050412788 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e94007cc0 con 0x7f9ea8102140 2024-03-23T11:45:29.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9e9f400700 1 -- 172.21.15.63:0/3050412788 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e9400f040 con 0x7f9ea8102140 2024-03-23T11:45:29.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea8110ed0 con 0x7f9ea8102140 2024-03-23T11:45:29.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9e9f400700 1 -- 172.21.15.63:0/3050412788 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9400c990 con 0x7f9ea8102140 2024-03-23T11:45:29.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.724+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea8111410 con 0x7f9ea8102140 2024-03-23T11:45:29.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.726+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea80faf10 con 0x7f9ea8102140 2024-03-23T11:45:29.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.727+0000 7f9e9f400700 1 -- 172.21.15.63:0/3050412788 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9e9400caf0 con 0x7f9ea8102140 2024-03-23T11:45:29.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.728+0000 7f9e9f400700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e8c0692a0 0x7f9e8c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:29.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.728+0000 7f9e9f400700 1 -- 172.21.15.63:0/3050412788 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9e94086d90 con 0x7f9ea8102140 2024-03-23T11:45:29.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.728+0000 7f9ea7400700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e8c0692a0 0x7f9e8c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:29.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.729+0000 7f9ea7400700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e8c0692a0 0x7f9e8c06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9e980137d0 tx=0x7f9e98009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:29.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.733+0000 7f9e9f400700 1 -- 172.21.15.63:0/3050412788 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9e9404e190 con 0x7f9ea8102140 2024-03-23T11:45:29.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.962+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ea8111790 con 0x7f9e8c0692a0 2024-03-23T11:45:29.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.963+0000 7f9e9f400700 1 -- 172.21.15.63:0/3050412788 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f9ea8111790 con 0x7f9e8c0692a0 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e8c0692a0 msgr2=0x7f9e8c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e8c0692a0 0x7f9e8c06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9e980137d0 tx=0x7f9e98009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8102140 msgr2=0x7f9ea810fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8102140 0x7f9ea810fe00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400eb40 tx=0x7f9e9400b040 comp rx=0 tx=0).stop 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 shutdown_connections 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9e8c0692a0 0x7f9e8c06b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ea8066cd0 0x7f9ea810f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 --2- 172.21.15.63:0/3050412788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9ea8102140 0x7f9ea810fe00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 >> 172.21.15.63:0/3050412788 conn(0x7f9ea80f7270 msgr2=0x7f9ea80f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 shutdown_connections 2024-03-23T11:45:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:29.966+0000 7f9eaca00700 1 -- 172.21.15.63:0/3050412788 wait complete. 2024-03-23T11:45:29.966 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.130+0000 7f4120200700 1 -- 172.21.15.63:0/221924068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 msgr2=0x7f4118102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.130+0000 7f4120200700 1 --2- 172.21.15.63:0/221924068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 0x7f4118102380 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4104004940 tx=0x7f4104009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.130+0000 7f4120200700 1 -- 172.21.15.63:0/221924068 shutdown_connections 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.130+0000 7f4120200700 1 --2- 172.21.15.63:0/221924068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 0x7f4118102380 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.130+0000 7f4120200700 1 --2- 172.21.15.63:0/221924068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 0x7f41180ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.130+0000 7f4120200700 1 -- 172.21.15.63:0/221924068 >> 172.21.15.63:0/221924068 conn(0x7f41180f7430 msgr2=0x7f41180f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.130+0000 7f4120200700 1 -- 172.21.15.63:0/221924068 shutdown_connections 2024-03-23T11:45:30.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.131+0000 7f4120200700 1 -- 172.21.15.63:0/221924068 wait complete. 2024-03-23T11:45:30.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.132+0000 7f4120200700 1 Processor -- start 2024-03-23T11:45:30.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.132+0000 7f4120200700 1 -- start start 2024-03-23T11:45:30.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.132+0000 7f4120200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 0x7f411810fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.132+0000 7f4120200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 0x7f4118110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.132+0000 7f4120200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118110650 con 0x7f41180fffa0 2024-03-23T11:45:30.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.132+0000 7f4120200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118110e50 con 0x7f4118066240 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 0x7f411810fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 0x7f4118110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 0x7f411810fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:40584/0 (socket says 172.21.15.63:40584) 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411d800700 1 -- 172.21.15.63:0/174167961 learned_addr learned my addr 172.21.15.63:0/174167961 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411d800700 1 -- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 msgr2=0x7f4118110050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411d800700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 0x7f4118110050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411d800700 1 -- 172.21.15.63:0/174167961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104005040 con 0x7f4118066240 2024-03-23T11:45:30.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.133+0000 7f411d800700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 0x7f411810fb10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f410c00eb70 tx=0x7f410c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.134+0000 7f4115600700 1 -- 172.21.15.63:0/174167961 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f410c007cc0 con 0x7f4118066240 2024-03-23T11:45:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.134+0000 7f4115600700 1 -- 172.21.15.63:0/174167961 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f410c00f040 con 0x7f4118066240 2024-03-23T11:45:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.134+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41181110d0 con 0x7f4118066240 2024-03-23T11:45:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.134+0000 7f4115600700 1 -- 172.21.15.63:0/174167961 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f410c00c9d0 con 0x7f4118066240 2024-03-23T11:45:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.134+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4118111610 con 0x7f4118066240 2024-03-23T11:45:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.135+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411805f5f0 con 0x7f4118066240 2024-03-23T11:45:30.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.136+0000 7f4115600700 1 -- 172.21.15.63:0/174167961 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f410c00cb30 con 0x7f4118066240 2024-03-23T11:45:30.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.137+0000 7f4115600700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f41000692b0 0x7f410006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.137+0000 7f4115600700 1 -- 172.21.15.63:0/174167961 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f410c086c80 con 0x7f4118066240 2024-03-23T11:45:30.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.137+0000 7f411ce00700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f41000692b0 0x7f410006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.138+0000 7f411ce00700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f41000692b0 0x7f410006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4104009fd0 tx=0x7f4104009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:30.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.142+0000 7f4115600700 1 -- 172.21.15.63:0/174167961 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f410c04e190 con 0x7f4118066240 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: pgmap v251: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: Upgrade: Target is version 18.2.2-820-gb630d6cd (unknown) 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313'] 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: Upgrade: Need to upgrade myself (mgr.smithi063.gehkcr) 2024-03-23T11:45:30.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:30 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:30.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.409+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41181119f0 con 0x7f41000692b0 2024-03-23T11:45:30.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.410+0000 7f4115600700 1 -- 172.21.15.63:0/174167961 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f41181119f0 con 0x7f41000692b0 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f41000692b0 msgr2=0x7f410006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f41000692b0 0x7f410006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4104009fd0 tx=0x7f4104009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 msgr2=0x7f411810fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 0x7f411810fb10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f410c00eb70 tx=0x7f410c00b040 comp rx=0 tx=0).stop 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 shutdown_connections 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f41000692b0 0x7f410006b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4118066240 0x7f411810fb10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 --2- 172.21.15.63:0/174167961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41180fffa0 0x7f4118110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 >> 172.21.15.63:0/174167961 conn(0x7f41180f7430 msgr2=0x7f41180fbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:30.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 shutdown_connections 2024-03-23T11:45:30.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.412+0000 7f4120200700 1 -- 172.21.15.63:0/174167961 wait complete. 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: pgmap v251: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: Upgrade: Target is version 18.2.2-820-gb630d6cd (unknown) 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313'] 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: Upgrade: Need to upgrade myself (mgr.smithi063.gehkcr) 2024-03-23T11:45:30.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:30 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 -- 172.21.15.63:0/3840910122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 msgr2=0x7f6160102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 --2- 172.21.15.63:0/3840910122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f6160102160 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f614c004940 tx=0x7f614c009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 -- 172.21.15.63:0/3840910122 shutdown_connections 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 --2- 172.21.15.63:0/3840910122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f6160102160 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 --2- 172.21.15.63:0/3840910122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6160066cd0 0x7f61600ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 -- 172.21.15.63:0/3840910122 >> 172.21.15.63:0/3840910122 conn(0x7f61600f7270 msgr2=0x7f61600f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 -- 172.21.15.63:0/3840910122 shutdown_connections 2024-03-23T11:45:30.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.556+0000 7f615f200700 1 -- 172.21.15.63:0/3840910122 wait complete. 2024-03-23T11:45:30.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.557+0000 7f615f200700 1 Processor -- start 2024-03-23T11:45:30.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.557+0000 7f615f200700 1 -- start start 2024-03-23T11:45:30.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6160066cd0 0x7f616010f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f616010fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160110420 con 0x7f61600ffd40 2024-03-23T11:45:30.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160110c70 con 0x7f6160066cd0 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f616010fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615de00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6160066cd0 0x7f616010f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f616010fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37490/0 (socket says 172.21.15.63:37490) 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615d400700 1 -- 172.21.15.63:0/2124201533 learned_addr learned my addr 172.21.15.63:0/2124201533 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615d400700 1 -- 172.21.15.63:0/2124201533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6160066cd0 msgr2=0x7f616010f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615d400700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6160066cd0 0x7f616010f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.558+0000 7f615d400700 1 -- 172.21.15.63:0/2124201533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c005040 con 0x7f61600ffd40 2024-03-23T11:45:30.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.559+0000 7f615d400700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f616010fe20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f614c00b0d0 tx=0x7f614c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:30.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.559+0000 7f6156000700 1 -- 172.21.15.63:0/2124201533 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614c00e070 con 0x7f61600ffd40 2024-03-23T11:45:30.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.559+0000 7f6156000700 1 -- 172.21.15.63:0/2124201533 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f614c00b410 con 0x7f61600ffd40 2024-03-23T11:45:30.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.559+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6160110ef0 con 0x7f61600ffd40 2024-03-23T11:45:30.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.559+0000 7f6156000700 1 -- 172.21.15.63:0/2124201533 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614c012600 con 0x7f61600ffd40 2024-03-23T11:45:30.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.559+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6160111430 con 0x7f61600ffd40 2024-03-23T11:45:30.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.560+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61600faf10 con 0x7f61600ffd40 2024-03-23T11:45:30.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.562+0000 7f6156000700 1 -- 172.21.15.63:0/2124201533 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f614c00b9f0 con 0x7f61600ffd40 2024-03-23T11:45:30.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.563+0000 7f6156000700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f61440692a0 0x7f614406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.563+0000 7f6156000700 1 -- 172.21.15.63:0/2124201533 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f614c087e50 con 0x7f61600ffd40 2024-03-23T11:45:30.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.563+0000 7f615de00700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f61440692a0 0x7f614406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.564+0000 7f615de00700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f61440692a0 0x7f614406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6150009b80 tx=0x7f615000b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:30.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.567+0000 7f6156000700 1 -- 172.21.15.63:0/2124201533 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f614c057c50 con 0x7f61600ffd40 2024-03-23T11:45:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.785+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6160111800 con 0x7f61440692a0 2024-03-23T11:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.796+0000 7f6156000700 1 -- 172.21.15.63:0/2124201533 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f6160111800 con 0x7f61440692a0 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (6m) 68s ago 7m 13.4M - 0.25.0 c8568f914cd2 c31e5bfcace0 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (8m) 68s ago 8m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (6m) 74s ago 6m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (6m) 68s ago 7m 73.5M - 9.4.7 954c08fa6188 d05e188def00 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (97s) 68s ago 96s 13.3M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (90s) 68s ago 89s 12.9M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (87s) 74s ago 86s 11.7M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (94s) 74s ago 93s 11.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:9283 running (9m) 68s ago 9m 480M - 17.2.7-726-g9c4e0c34 837124da804f 63833d132315 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (6m) 74s ago 6m 431M - 17.2.7-726-g9c4e0c34 837124da804f f2f79acf25e6 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (9m) 68s ago 9m 44.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (6m) 74s ago 6m 34.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (8m) 68s ago 8m 7537k - 1.5.0 0da6a335fe13 e942e152db56 2024-03-23T11:45:30.782 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (6m) 74s ago 6m 7256k - 1.5.0 0da6a335fe13 68f97d563d92 2024-03-23T11:45:30.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (5m) 68s ago 5m 45.4M 4348M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:45:30.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (5m) 68s ago 5m 45.6M 4348M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:45:30.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (4m) 68s ago 4m 43.4M 4348M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:45:30.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (4m) 74s ago 4m 43.1M 5372M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:45:30.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (3m) 74s ago 3m 42.7M 5372M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:45:30.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (3m) 74s ago 3m 40.8M 5372M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:45:30.783 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (6m) 68s ago 7m 46.3M - 2.43.0 a07b618ecd1d 328fbe2c4719 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f61440692a0 msgr2=0x7f614406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f61440692a0 0x7f614406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6150009b80 tx=0x7f615000b040 comp rx=0 tx=0).stop 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 msgr2=0x7f616010fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f616010fe20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f614c00b0d0 tx=0x7f614c009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 shutdown_connections 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f61440692a0 0x7f614406b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6160066cd0 0x7f616010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 --2- 172.21.15.63:0/2124201533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61600ffd40 0x7f616010fe20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.798+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 >> 172.21.15.63:0/2124201533 conn(0x7f61600f7270 msgr2=0x7f61600fb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.799+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 shutdown_connections 2024-03-23T11:45:30.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.799+0000 7f615f200700 1 -- 172.21.15.63:0/2124201533 wait complete. 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 -- 172.21.15.63:0/2594158012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fbb10 msgr2=0x7f96140fbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 --2- 172.21.15.63:0/2594158012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fbb10 0x7f96140fbf20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9608004940 tx=0x7f9608009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 -- 172.21.15.63:0/2594158012 shutdown_connections 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 --2- 172.21.15.63:0/2594158012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96140fcdc0 0x7f96140fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 --2- 172.21.15.63:0/2594158012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fbb10 0x7f96140fbf20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 -- 172.21.15.63:0/2594158012 >> 172.21.15.63:0/2594158012 conn(0x7f96140f7270 msgr2=0x7f96140f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 -- 172.21.15.63:0/2594158012 shutdown_connections 2024-03-23T11:45:30.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.938+0000 7f961a000700 1 -- 172.21.15.63:0/2594158012 wait complete. 2024-03-23T11:45:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.939+0000 7f961a000700 1 Processor -- start 2024-03-23T11:45:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.939+0000 7f961a000700 1 -- start start 2024-03-23T11:45:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.939+0000 7f961a000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96140fbb10 0x7f961410f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.939+0000 7f961a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fcdc0 0x7f961410fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.939+0000 7f961a000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614110480 con 0x7f96140fcdc0 2024-03-23T11:45:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.939+0000 7f961a000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614110c80 con 0x7f96140fbb10 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9613e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fcdc0 0x7f961410fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9613e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fcdc0 0x7f961410fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37502/0 (socket says 172.21.15.63:37502) 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9613e00700 1 -- 172.21.15.63:0/317842708 learned_addr learned my addr 172.21.15.63:0/317842708 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9618c00700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96140fbb10 0x7f961410f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9613e00700 1 -- 172.21.15.63:0/317842708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96140fbb10 msgr2=0x7f961410f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9613e00700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96140fbb10 0x7f961410f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9613e00700 1 -- 172.21.15.63:0/317842708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9608005040 con 0x7f96140fcdc0 2024-03-23T11:45:30.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.940+0000 7f9613e00700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fcdc0 0x7f961410fe80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f960000eb40 tx=0x7f960000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:30.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.941+0000 7f9610c00700 1 -- 172.21.15.63:0/317842708 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9600007cc0 con 0x7f96140fcdc0 2024-03-23T11:45:30.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.941+0000 7f9610c00700 1 -- 172.21.15.63:0/317842708 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f960000f040 con 0x7f96140fcdc0 2024-03-23T11:45:30.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.941+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614110f00 con 0x7f96140fcdc0 2024-03-23T11:45:30.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.941+0000 7f9610c00700 1 -- 172.21.15.63:0/317842708 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f960000c990 con 0x7f96140fcdc0 2024-03-23T11:45:30.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.941+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9614111440 con 0x7f96140fcdc0 2024-03-23T11:45:30.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.942+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614101750 con 0x7f96140fcdc0 2024-03-23T11:45:30.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.943+0000 7f9610c00700 1 -- 172.21.15.63:0/317842708 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f960000caf0 con 0x7f96140fcdc0 2024-03-23T11:45:30.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.944+0000 7f9610c00700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f95fc0692a0 0x7f95fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:30.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.944+0000 7f9610c00700 1 -- 172.21.15.63:0/317842708 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9600086d90 con 0x7f96140fcdc0 2024-03-23T11:45:30.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.944+0000 7f9618c00700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f95fc0692a0 0x7f95fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:30.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.945+0000 7f9618c00700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f95fc0692a0 0x7f95fc06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f960800b0d0 tx=0x7f9608009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:30.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:30.948+0000 7f9610c00700 1 -- 172.21.15.63:0/317842708 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f960004e190 con 0x7f96140fcdc0 2024-03-23T11:45:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.270+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f96140faf10 con 0x7f96140fcdc0 2024-03-23T11:45:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.271+0000 7f9610c00700 1 -- 172.21.15.63:0/317842708 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f9600056b10 con 0x7f96140fcdc0 2024-03-23T11:45:31.256 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:45:31.256 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:45:31.256 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:45:31.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:45:31.256 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:45:31.256 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 14 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:45:31.257 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f95fc0692a0 msgr2=0x7f95fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f95fc0692a0 0x7f95fc06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f960800b0d0 tx=0x7f9608009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fcdc0 msgr2=0x7f961410fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fcdc0 0x7f961410fe80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f960000eb40 tx=0x7f960000b040 comp rx=0 tx=0).stop 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 shutdown_connections 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f95fc0692a0 0x7f95fc06b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96140fbb10 0x7f961410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 --2- 172.21.15.63:0/317842708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96140fcdc0 0x7f961410fe80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 >> 172.21.15.63:0/317842708 conn(0x7f96140f7270 msgr2=0x7f9614066640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:31.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 shutdown_connections 2024-03-23T11:45:31.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.273+0000 7f961a000700 1 -- 172.21.15.63:0/317842708 wait complete. 2024-03-23T11:45:31.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:31 smithi063 ceph-mon[35189]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 on smithi111 2024-03-23T11:45:31.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:31 smithi063 ceph-mon[35189]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:31.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:31 smithi111 ceph-mon[44005]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 on smithi111 2024-03-23T11:45:31.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:31 smithi111 ceph-mon[44005]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:31.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.421+0000 7f11ff000700 1 -- 172.21.15.63:0/2653487911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f80fffc0 msgr2=0x7f11f81023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:31.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.421+0000 7f11ff000700 1 --2- 172.21.15.63:0/2653487911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f80fffc0 0x7f11f81023a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f11e4004940 tx=0x7f11e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:31.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.421+0000 7f11ff000700 1 -- 172.21.15.63:0/2653487911 shutdown_connections 2024-03-23T11:45:31.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.421+0000 7f11ff000700 1 --2- 172.21.15.63:0/2653487911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f80fffc0 0x7f11f81023a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.421+0000 7f11ff000700 1 --2- 172.21.15.63:0/2653487911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f8066220 0x7f11f80ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.421+0000 7f11ff000700 1 -- 172.21.15.63:0/2653487911 >> 172.21.15.63:0/2653487911 conn(0x7f11f80f7430 msgr2=0x7f11f80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:31.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.422+0000 7f11ff000700 1 -- 172.21.15.63:0/2653487911 shutdown_connections 2024-03-23T11:45:31.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.422+0000 7f11ff000700 1 -- 172.21.15.63:0/2653487911 wait complete. 2024-03-23T11:45:31.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.423+0000 7f11ff000700 1 Processor -- start 2024-03-23T11:45:31.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.423+0000 7f11ff000700 1 -- start start 2024-03-23T11:45:31.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.423+0000 7f11ff000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f8066220 0x7f11f810d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:31.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11ff000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f80fffc0 0x7f11f810de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:31.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11ff000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f810e430 con 0x7f11f8066220 2024-03-23T11:45:31.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11ff000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f810ec80 con 0x7f11f80fffc0 2024-03-23T11:45:31.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11f7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f8066220 0x7f11f810d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:31.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11f7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f8066220 0x7f11f810d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37514/0 (socket says 172.21.15.63:37514) 2024-03-23T11:45:31.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11f7e00700 1 -- 172.21.15.63:0/3575484444 learned_addr learned my addr 172.21.15.63:0/3575484444 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11f7400700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f80fffc0 0x7f11f810de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11f7e00700 1 -- 172.21.15.63:0/3575484444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f80fffc0 msgr2=0x7f11f810de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11f7e00700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f80fffc0 0x7f11f810de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.424+0000 7f11f7e00700 1 -- 172.21.15.63:0/3575484444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e4005040 con 0x7f11f8066220 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.425+0000 7f11f7e00700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f8066220 0x7f11f810d8f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f11e800eb70 tx=0x7f11e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.425+0000 7f11efe00700 1 -- 172.21.15.63:0/3575484444 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e8007cc0 con 0x7f11f8066220 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.425+0000 7f11efe00700 1 -- 172.21.15.63:0/3575484444 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11e800f040 con 0x7f11f8066220 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.425+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f810ef00 con 0x7f11f8066220 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.425+0000 7f11efe00700 1 -- 172.21.15.63:0/3575484444 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e800c9d0 con 0x7f11f8066220 2024-03-23T11:45:31.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.425+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f810f440 con 0x7f11f8066220 2024-03-23T11:45:31.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.427+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f805f5f0 con 0x7f11f8066220 2024-03-23T11:45:31.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.427+0000 7f11efe00700 1 -- 172.21.15.63:0/3575484444 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f11e800cb30 con 0x7f11f8066220 2024-03-23T11:45:31.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.428+0000 7f11efe00700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f11dc0692b0 0x7f11dc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:45:31.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.428+0000 7f11efe00700 1 -- 172.21.15.63:0/3575484444 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f11e8086bf0 con 0x7f11f8066220 2024-03-23T11:45:31.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.428+0000 7f11f7400700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f11dc0692b0 0x7f11dc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:45:31.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.429+0000 7f11f7400700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f11dc0692b0 0x7f11dc06b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f11e40137d0 tx=0x7f11e4009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:45:31.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.433+0000 7f11efe00700 1 -- 172.21.15.63:0/3575484444 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f11e804e190 con 0x7f11f8066220 2024-03-23T11:45:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.679+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11f810f7e0 con 0x7f11dc0692b0 2024-03-23T11:45:31.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.681+0000 7f11efe00700 1 -- 172.21.15.63:0/3575484444 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f11f810f7e0 con 0x7f11dc0692b0 2024-03-23T11:45:31.666 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [], 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "0/2 daemons upgraded", 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 image on host smithi111", 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:45:31.667 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f11dc0692b0 msgr2=0x7f11dc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f11dc0692b0 0x7f11dc06b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f11e40137d0 tx=0x7f11e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f8066220 msgr2=0x7f11f810d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f8066220 0x7f11f810d8f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f11e800eb70 tx=0x7f11e800b040 comp rx=0 tx=0).stop 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 shutdown_connections 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f11dc0692b0 0x7f11dc06b760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f8066220 0x7f11f810d8f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 --2- 172.21.15.63:0/3575484444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f80fffc0 0x7f11f810de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 >> 172.21.15.63:0/3575484444 conn(0x7f11f80f7430 msgr2=0x7f11f80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:45:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 shutdown_connections 2024-03-23T11:45:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:45:31.683+0000 7f11ff000700 1 -- 172.21.15.63:0/3575484444 wait complete. 2024-03-23T11:45:32.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:32 smithi111 ceph-mon[44005]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:32.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:32 smithi111 ceph-mon[44005]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:32.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:32 smithi111 ceph-mon[44005]: pgmap v252: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:32.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:32 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/317842708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:45:32.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:32 smithi063 ceph-mon[35189]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:32.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:32 smithi063 ceph-mon[35189]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:32.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:32 smithi063 ceph-mon[35189]: pgmap v252: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:32.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:32 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/317842708' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:45:33.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:33 smithi111 ceph-mon[44005]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:33.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:33 smithi063 ceph-mon[35189]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:45:34.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:34 smithi111 ceph-mon[44005]: pgmap v253: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:34.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:34 smithi063 ceph-mon[35189]: pgmap v253: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:36.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:36 smithi111 ceph-mon[44005]: pgmap v254: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:36.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:36 smithi063 ceph-mon[35189]: pgmap v254: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:38.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:38 smithi111 ceph-mon[44005]: pgmap v255: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:38.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:38 smithi063 ceph-mon[35189]: pgmap v255: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:40.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:40 smithi111 ceph-mon[44005]: pgmap v256: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:40.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:40 smithi063 ceph-mon[35189]: pgmap v256: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:42.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:42 smithi111 ceph-mon[44005]: pgmap v257: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:42.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:42 smithi063 ceph-mon[35189]: pgmap v257: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:44.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:44 smithi063 ceph-mon[35189]: pgmap v258: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:44.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:44 smithi111 ceph-mon[44005]: pgmap v258: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:45.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:45 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:45.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:45 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:45:46.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:46 smithi063 ceph-mon[35189]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:46.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:46 smithi111 ceph-mon[44005]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:48.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:48 smithi063 ceph-mon[35189]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:48.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:48 smithi111 ceph-mon[44005]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:45:50.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:50 smithi063 ceph-mon[35189]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:50.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:50 smithi111 ceph-mon[44005]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:52.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:52 smithi063 ceph-mon[35189]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:52.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:52 smithi111 ceph-mon[44005]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:54 smithi063 ceph-mon[35189]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:54 smithi111 ceph-mon[44005]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:45:56.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:56 smithi063 ceph-mon[35189]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:56.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:56 smithi111 ceph-mon[44005]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:58.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:45:58 smithi063 ceph-mon[35189]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:45:58.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:45:58 smithi111 ceph-mon[44005]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:46:00.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:00 smithi063 ceph-mon[35189]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:46:00.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:00 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:00.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:00 smithi111 ceph-mon[44005]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:46:00.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:00 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:01.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.852+0000 7f8723200700 1 -- 172.21.15.63:0/3537956492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 msgr2=0x7f87241021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.852+0000 7f8723200700 1 --2- 172.21.15.63:0/3537956492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 0x7f87241021c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8710004940 tx=0x7f8710009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.853+0000 7f8723200700 1 -- 172.21.15.63:0/3537956492 shutdown_connections 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.853+0000 7f8723200700 1 --2- 172.21.15.63:0/3537956492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 0x7f87241021c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.853+0000 7f8723200700 1 --2- 172.21.15.63:0/3537956492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 0x7f87240ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.853+0000 7f8723200700 1 -- 172.21.15.63:0/3537956492 >> 172.21.15.63:0/3537956492 conn(0x7f87240f7270 msgr2=0x7f87240f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.853+0000 7f8723200700 1 -- 172.21.15.63:0/3537956492 shutdown_connections 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.853+0000 7f8723200700 1 -- 172.21.15.63:0/3537956492 wait complete. 2024-03-23T11:46:01.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.854+0000 7f8723200700 1 Processor -- start 2024-03-23T11:46:01.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.854+0000 7f8723200700 1 -- start start 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.855+0000 7f8723200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 0x7f872410d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.855+0000 7f8723200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 0x7f872410dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.855+0000 7f8723200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872410e250 con 0x7f87240ffde0 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.855+0000 7f8723200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872410eaa0 con 0x7f8724066240 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.855+0000 7f8721e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 0x7f872410d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.855+0000 7f8721e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 0x7f872410d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:37288/0 (socket says 172.21.15.63:37288) 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.855+0000 7f8721e00700 1 -- 172.21.15.63:0/1584866462 learned_addr learned my addr 172.21.15.63:0/1584866462 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:01.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.856+0000 7f8721400700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 0x7f872410dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:01.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.856+0000 7f8721e00700 1 -- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 msgr2=0x7f872410dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:01.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.856+0000 7f8721e00700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 0x7f872410dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:01.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.856+0000 7f8721e00700 1 -- 172.21.15.63:0/1584866462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8710005040 con 0x7f8724066240 2024-03-23T11:46:01.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.856+0000 7f8721e00700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 0x7f872410d710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f871400eb70 tx=0x7f871400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:01.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.856+0000 7f871a000700 1 -- 172.21.15.63:0/1584866462 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8714007cc0 con 0x7f8724066240 2024-03-23T11:46:01.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.857+0000 7f871a000700 1 -- 172.21.15.63:0/1584866462 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f871400f040 con 0x7f8724066240 2024-03-23T11:46:01.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.857+0000 7f871a000700 1 -- 172.21.15.63:0/1584866462 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f871400ca60 con 0x7f8724066240 2024-03-23T11:46:01.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.857+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872410ed20 con 0x7f8724066240 2024-03-23T11:46:01.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.857+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872410f260 con 0x7f8724066240 2024-03-23T11:46:01.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.858+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872405f5f0 con 0x7f8724066240 2024-03-23T11:46:01.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.860+0000 7f871a000700 1 -- 172.21.15.63:0/1584866462 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f871400cbc0 con 0x7f8724066240 2024-03-23T11:46:01.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.862+0000 7f871a000700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f87080692b0 0x7f870806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:01.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.862+0000 7f871a000700 1 -- 172.21.15.63:0/1584866462 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8714086ca0 con 0x7f8724066240 2024-03-23T11:46:01.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.862+0000 7f8721400700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f87080692b0 0x7f870806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:01.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.863+0000 7f8721400700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f87080692b0 0x7f870806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f871000b0d0 tx=0x7f8710009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:01.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:01.869+0000 7f871a000700 1 -- 172.21.15.63:0/1584866462 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f871404e190 con 0x7f8724066240 2024-03-23T11:46:02.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.107+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f872410f640 con 0x7f87080692b0 2024-03-23T11:46:02.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.109+0000 7f871a000700 1 -- 172.21.15.63:0/1584866462 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f872410f640 con 0x7f87080692b0 2024-03-23T11:46:02.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f87080692b0 msgr2=0x7f870806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f87080692b0 0x7f870806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f871000b0d0 tx=0x7f8710009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 msgr2=0x7f872410d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 0x7f872410d710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f871400eb70 tx=0x7f871400b040 comp rx=0 tx=0).stop 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 shutdown_connections 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f87080692b0 0x7f870806b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8724066240 0x7f872410d710 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f871400eb70 tx=0x7f871400b040 comp rx=0 tx=0).stop 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 --2- 172.21.15.63:0/1584866462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87240ffde0 0x7f872410dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 >> 172.21.15.63:0/1584866462 conn(0x7f87240f7270 msgr2=0x7f87240f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 shutdown_connections 2024-03-23T11:46:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.112+0000 7f8723200700 1 -- 172.21.15.63:0/1584866462 wait complete. 2024-03-23T11:46:02.111 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:46:02.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 -- 172.21.15.63:0/2939360172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 msgr2=0x7f9d2c0fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 --2- 172.21.15.63:0/2939360172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 0x7f9d2c0fd280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d20004940 tx=0x7f9d20009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:02.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 -- 172.21.15.63:0/2939360172 shutdown_connections 2024-03-23T11:46:02.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 --2- 172.21.15.63:0/2939360172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 0x7f9d2c0fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 --2- 172.21.15.63:0/2939360172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 0x7f9d2c0fd280 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 -- 172.21.15.63:0/2939360172 >> 172.21.15.63:0/2939360172 conn(0x7f9d2c0f8550 msgr2=0x7f9d2c0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:02.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 -- 172.21.15.63:0/2939360172 shutdown_connections 2024-03-23T11:46:02.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.255+0000 7f9d33e00700 1 -- 172.21.15.63:0/2939360172 wait complete. 2024-03-23T11:46:02.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.256+0000 7f9d33e00700 1 Processor -- start 2024-03-23T11:46:02.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.257+0000 7f9d33e00700 1 -- start start 2024-03-23T11:46:02.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.257+0000 7f9d33e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 0x7f9d2c10d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:02.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.257+0000 7f9d33e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 0x7f9d2c10dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:02.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.257+0000 7f9d33e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c10e310 con 0x7f9d2c0fce70 2024-03-23T11:46:02.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.257+0000 7f9d33e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c10e480 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.258+0000 7f9d31400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 0x7f9d2c10d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:02.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.258+0000 7f9d30a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 0x7f9d2c10dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:02.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.258+0000 7f9d30a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 0x7f9d2c10dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:37320/0 (socket says 172.21.15.63:37320) 2024-03-23T11:46:02.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.258+0000 7f9d30a00700 1 -- 172.21.15.63:0/294328172 learned_addr learned my addr 172.21.15.63:0/294328172 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:02.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.258+0000 7f9d30a00700 1 -- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 msgr2=0x7f9d2c10d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.258+0000 7f9d30a00700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 0x7f9d2c10d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.258+0000 7f9d30a00700 1 -- 172.21.15.63:0/294328172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d20005040 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.259+0000 7f9d30a00700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 0x7f9d2c10dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d18002a80 tx=0x7f9d180046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:02.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.259+0000 7f9d29600700 1 -- 172.21.15.63:0/294328172 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d18007cc0 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.259+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2c112ee0 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.259+0000 7f9d29600700 1 -- 172.21.15.63:0/294328172 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d1800f040 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.259+0000 7f9d29600700 1 -- 172.21.15.63:0/294328172 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d1800da60 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.259+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2c1133f0 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.261+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2c04c710 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.263+0000 7f9d29600700 1 -- 172.21.15.63:0/294328172 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9d1800dbc0 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.264+0000 7f9d29600700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d140692a0 0x7f9d1406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:02.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.264+0000 7f9d29600700 1 -- 172.21.15.63:0/294328172 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9d18086ba0 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.265+0000 7f9d31400700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d140692a0 0x7f9d1406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:02.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.265+0000 7f9d31400700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d140692a0 0x7f9d1406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000b690 tx=0x7f9d20009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:02.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.270+0000 7f9d29600700 1 -- 172.21.15.63:0/294328172 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9d180568f0 con 0x7f9d2c0fdfe0 2024-03-23T11:46:02.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.495+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d2c102340 con 0x7f9d140692a0 2024-03-23T11:46:02.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.497+0000 7f9d29600700 1 -- 172.21.15.63:0/294328172 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f9d2c102340 con 0x7f9d140692a0 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d140692a0 msgr2=0x7f9d1406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d140692a0 0x7f9d1406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000b690 tx=0x7f9d20009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 msgr2=0x7f9d2c10dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 0x7f9d2c10dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d18002a80 tx=0x7f9d180046b0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 shutdown_connections 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f9d140692a0 0x7f9d1406b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d2c0fce70 0x7f9d2c10d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.499+0000 7f9d33e00700 1 --2- 172.21.15.63:0/294328172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0fdfe0 0x7f9d2c10dd10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.500+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 >> 172.21.15.63:0/294328172 conn(0x7f9d2c0f8550 msgr2=0x7f9d2c0f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:02.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.500+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 shutdown_connections 2024-03-23T11:46:02.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.500+0000 7f9d33e00700 1 -- 172.21.15.63:0/294328172 wait complete. 2024-03-23T11:46:02.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 -- 172.21.15.63:0/234353953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fe0d0 msgr2=0x7fcf340fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 --2- 172.21.15.63:0/234353953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fe0d0 0x7fcf340fe540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcf200048b0 tx=0x7fcf20009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:02.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 -- 172.21.15.63:0/234353953 shutdown_connections 2024-03-23T11:46:02.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 --2- 172.21.15.63:0/234353953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fe0d0 0x7fcf340fe540 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 --2- 172.21.15.63:0/234353953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fcee0 0x7fcf340fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 -- 172.21.15.63:0/234353953 >> 172.21.15.63:0/234353953 conn(0x7fcf340f8560 msgr2=0x7fcf340fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:02.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 -- 172.21.15.63:0/234353953 shutdown_connections 2024-03-23T11:46:02.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.642+0000 7fcf3a600700 1 -- 172.21.15.63:0/234353953 wait complete. 2024-03-23T11:46:02.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.643+0000 7fcf3a600700 1 Processor -- start 2024-03-23T11:46:02.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.644+0000 7fcf3a600700 1 -- start start 2024-03-23T11:46:02.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.644+0000 7fcf3a600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fcee0 0x7fcf3410d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:02.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.644+0000 7fcf3a600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fe0d0 0x7fcf3410de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:02.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.644+0000 7fcf3a600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410e420 con 0x7fcf340fe0d0 2024-03-23T11:46:02.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.644+0000 7fcf3a600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410ec70 con 0x7fcf340fcee0 2024-03-23T11:46:02.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf33400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fcee0 0x7fcf3410d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:02.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf32a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fe0d0 0x7fcf3410de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:02.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf32a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fe0d0 0x7fcf3410de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55740/0 (socket says 172.21.15.63:55740) 2024-03-23T11:46:02.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf32a00700 1 -- 172.21.15.63:0/1257931852 learned_addr learned my addr 172.21.15.63:0/1257931852 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:02.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf32a00700 1 -- 172.21.15.63:0/1257931852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fcee0 msgr2=0x7fcf3410d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf32a00700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fcee0 0x7fcf3410d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf32a00700 1 -- 172.21.15.63:0/1257931852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf20005040 con 0x7fcf340fe0d0 2024-03-23T11:46:02.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.645+0000 7fcf32a00700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fe0d0 0x7fcf3410de20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcf200137d0 tx=0x7fcf20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:02.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.646+0000 7fcf2b400700 1 -- 172.21.15.63:0/1257931852 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf2000e040 con 0x7fcf340fe0d0 2024-03-23T11:46:02.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.646+0000 7fcf2b400700 1 -- 172.21.15.63:0/1257931852 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf20014c30 con 0x7fcf340fe0d0 2024-03-23T11:46:02.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.646+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3410eef0 con 0x7fcf340fe0d0 2024-03-23T11:46:02.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.646+0000 7fcf2b400700 1 -- 172.21.15.63:0/1257931852 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf20012600 con 0x7fcf340fe0d0 2024-03-23T11:46:02.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.646+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3410f3d0 con 0x7fcf340fe0d0 2024-03-23T11:46:02.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.647+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3405f5f0 con 0x7fcf340fe0d0 2024-03-23T11:46:02.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.649+0000 7fcf2b400700 1 -- 172.21.15.63:0/1257931852 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcf2001c9e0 con 0x7fcf340fe0d0 2024-03-23T11:46:02.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:02 smithi063 ceph-mon[35189]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:02.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.650+0000 7fcf2b400700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fcf180692a0 0x7fcf1806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:02.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.650+0000 7fcf2b400700 1 -- 172.21.15.63:0/1257931852 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcf20089250 con 0x7fcf340fe0d0 2024-03-23T11:46:02.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.651+0000 7fcf33400700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fcf180692a0 0x7fcf1806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:02.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.652+0000 7fcf33400700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fcf180692a0 0x7fcf1806b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009b80 tx=0x7fcf2400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:02.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.658+0000 7fcf2b400700 1 -- 172.21.15.63:0/1257931852 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcf20059120 con 0x7fcf340fe0d0 2024-03-23T11:46:02.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:02 smithi111 ceph-mon[44005]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.876+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf3410f7c0 con 0x7fcf180692a0 2024-03-23T11:46:02.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.886+0000 7fcf2b400700 1 -- 172.21.15.63:0/1257931852 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7fcf3410f7c0 con 0x7fcf180692a0 2024-03-23T11:46:02.872 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:46:02.872 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (6m) 100s ago 8m 13.4M - 0.25.0 c8568f914cd2 c31e5bfcace0 2024-03-23T11:46:02.872 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (8m) 100s ago 8m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:46:02.872 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (7m) 106s ago 7m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:46:02.872 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (6m) 100s ago 8m 73.5M - 9.4.7 954c08fa6188 d05e188def00 2024-03-23T11:46:02.872 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (2m) 100s ago 2m 13.3M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (2m) 100s ago 2m 12.9M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (119s) 106s ago 118s 11.7M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (2m) 106s ago 2m 11.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:9283 running (9m) 100s ago 9m 480M - 17.2.7-726-g9c4e0c34 837124da804f 63833d132315 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (7m) 106s ago 7m 431M - 17.2.7-726-g9c4e0c34 837124da804f f2f79acf25e6 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (10m) 100s ago 10m 44.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (7m) 106s ago 7m 34.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:46:02.873 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (8m) 100s ago 8m 7537k - 1.5.0 0da6a335fe13 e942e152db56 2024-03-23T11:46:02.874 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 106s ago 7m 7256k - 1.5.0 0da6a335fe13 68f97d563d92 2024-03-23T11:46:02.874 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (6m) 100s ago 6m 45.4M 4348M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:46:02.874 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (5m) 100s ago 5m 45.6M 4348M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:46:02.874 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (5m) 100s ago 5m 43.4M 4348M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:46:02.874 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (4m) 106s ago 4m 43.1M 5372M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:46:02.874 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (4m) 106s ago 4m 42.7M 5372M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:46:02.875 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (3m) 106s ago 3m 40.8M 5372M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:46:02.875 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (6m) 100s ago 7m 46.3M - 2.43.0 a07b618ecd1d 328fbe2c4719 2024-03-23T11:46:02.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fcf180692a0 msgr2=0x7fcf1806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fcf180692a0 0x7fcf1806b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009b80 tx=0x7fcf2400b040 comp rx=0 tx=0).stop 2024-03-23T11:46:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fe0d0 msgr2=0x7fcf3410de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fe0d0 0x7fcf3410de20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcf200137d0 tx=0x7fcf20009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 shutdown_connections 2024-03-23T11:46:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7fcf180692a0 0x7fcf1806b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf340fcee0 0x7fcf3410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 --2- 172.21.15.63:0/1257931852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fe0d0 0x7fcf3410de20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:02.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 >> 172.21.15.63:0/1257931852 conn(0x7fcf340f8560 msgr2=0x7fcf340f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:02.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 shutdown_connections 2024-03-23T11:46:02.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:02.890+0000 7fcf3a600700 1 -- 172.21.15.63:0/1257931852 wait complete. 2024-03-23T11:46:03.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.031+0000 7f3f69200700 1 -- 172.21.15.63:0/3315290088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64070890 msgr2=0x7f3f64070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.031+0000 7f3f69200700 1 --2- 172.21.15.63:0/3315290088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64070890 0x7f3f64070d00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3f50004940 tx=0x7f3f50009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.032+0000 7f3f69200700 1 -- 172.21.15.63:0/3315290088 shutdown_connections 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.032+0000 7f3f69200700 1 --2- 172.21.15.63:0/3315290088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64070890 0x7f3f64070d00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.032+0000 7f3f69200700 1 --2- 172.21.15.63:0/3315290088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64071e40 0x7f3f64070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.032+0000 7f3f69200700 1 -- 172.21.15.63:0/3315290088 >> 172.21.15.63:0/3315290088 conn(0x7f3f640f83a0 msgr2=0x7f3f640fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.032+0000 7f3f69200700 1 -- 172.21.15.63:0/3315290088 shutdown_connections 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.032+0000 7f3f69200700 1 -- 172.21.15.63:0/3315290088 wait complete. 2024-03-23T11:46:03.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.033+0000 7f3f69200700 1 Processor -- start 2024-03-23T11:46:03.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.033+0000 7f3f69200700 1 -- start start 2024-03-23T11:46:03.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f69200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64070890 0x7f3f6410f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:03.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f69200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64071e40 0x7f3f6410fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:03.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f69200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641104e0 con 0x7f3f64071e40 2024-03-23T11:46:03.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f69200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64110d30 con 0x7f3f64070890 2024-03-23T11:46:03.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f63400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64070890 0x7f3f6410f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:03.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f63400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64070890 0x7f3f6410f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:37354/0 (socket says 172.21.15.63:37354) 2024-03-23T11:46:03.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f63400700 1 -- 172.21.15.63:0/3877079171 learned_addr learned my addr 172.21.15.63:0/3877079171 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:03.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.034+0000 7f3f62a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64071e40 0x7f3f6410fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:03.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.035+0000 7f3f63400700 1 -- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64071e40 msgr2=0x7f3f6410fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.035+0000 7f3f63400700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64071e40 0x7f3f6410fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.035+0000 7f3f63400700 1 -- 172.21.15.63:0/3877079171 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50005040 con 0x7f3f64070890 2024-03-23T11:46:03.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.035+0000 7f3f63400700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64070890 0x7f3f6410f9a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400eb40 tx=0x7f3f5400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:03.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.036+0000 7f3f5b400700 1 -- 172.21.15.63:0/3877079171 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f54007cc0 con 0x7f3f64070890 2024-03-23T11:46:03.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.036+0000 7f3f5b400700 1 -- 172.21.15.63:0/3877079171 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f5400f040 con 0x7f3f64070890 2024-03-23T11:46:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.036+0000 7f3f5b400700 1 -- 172.21.15.63:0/3877079171 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f5400c9f0 con 0x7f3f64070890 2024-03-23T11:46:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.036+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f64110fb0 con 0x7f3f64070890 2024-03-23T11:46:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.036+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f641114f0 con 0x7f3f64070890 2024-03-23T11:46:03.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.038+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6405f5f0 con 0x7f3f64070890 2024-03-23T11:46:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.039+0000 7f3f5b400700 1 -- 172.21.15.63:0/3877079171 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3f5400cb50 con 0x7f3f64070890 2024-03-23T11:46:03.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.041+0000 7f3f5b400700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f480692b0 0x7f3f4806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:03.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.041+0000 7f3f5b400700 1 -- 172.21.15.63:0/3877079171 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3f54086d00 con 0x7f3f64070890 2024-03-23T11:46:03.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.041+0000 7f3f62a00700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f480692b0 0x7f3f4806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:03.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.042+0000 7f3f62a00700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f480692b0 0x7f3f4806b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3f5000b0d0 tx=0x7f3f50009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:03.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.050+0000 7f3f5b400700 1 -- 172.21.15.63:0/3877079171 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3f540569d0 con 0x7f3f64070890 2024-03-23T11:46:03.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.396+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f641118c0 con 0x7f3f64070890 2024-03-23T11:46:03.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.397+0000 7f3f5b400700 1 -- 172.21.15.63:0/3877079171 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f3f54056560 con 0x7f3f64070890 2024-03-23T11:46:03.381 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:46:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:46:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:46:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 14 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:46:03.382 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:46:03.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f480692b0 msgr2=0x7f3f4806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f480692b0 0x7f3f4806b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3f5000b0d0 tx=0x7f3f50009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:03.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64070890 msgr2=0x7f3f6410f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64070890 0x7f3f6410f9a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400eb40 tx=0x7f3f5400b040 comp rx=0 tx=0).stop 2024-03-23T11:46:03.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 shutdown_connections 2024-03-23T11:46:03.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f3f480692b0 0x7f3f4806b760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f64070890 0x7f3f6410f9a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 --2- 172.21.15.63:0/3877079171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f64071e40 0x7f3f6410fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.400+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 >> 172.21.15.63:0/3877079171 conn(0x7f3f640f83a0 msgr2=0x7f3f640f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:03.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.401+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 shutdown_connections 2024-03-23T11:46:03.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.401+0000 7f3f69200700 1 -- 172.21.15.63:0/3877079171 wait complete. 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 -- 172.21.15.63:0/3030719267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fcc10 msgr2=0x7f6d540fd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 --2- 172.21.15.63:0/3030719267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fcc10 0x7f6d540fd020 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6d44004940 tx=0x7f6d44009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 -- 172.21.15.63:0/3030719267 shutdown_connections 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 --2- 172.21.15.63:0/3030719267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d540fdec0 0x7f6d540fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 --2- 172.21.15.63:0/3030719267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fcc10 0x7f6d540fd020 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 -- 172.21.15.63:0/3030719267 >> 172.21.15.63:0/3030719267 conn(0x7f6d540f83d0 msgr2=0x7f6d540fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 -- 172.21.15.63:0/3030719267 shutdown_connections 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.607+0000 7f6d53e00700 1 -- 172.21.15.63:0/3030719267 wait complete. 2024-03-23T11:46:03.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.608+0000 7f6d53e00700 1 Processor -- start 2024-03-23T11:46:03.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d53e00700 1 -- start start 2024-03-23T11:46:03.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d53e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d540fcc10 0x7f6d54071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:03.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d53e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fdec0 0x7f6d54070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:03.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d53e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540708c0 con 0x7f6d540fdec0 2024-03-23T11:46:03.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d53e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54070a30 con 0x7f6d540fcc10 2024-03-23T11:46:03.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d52000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fdec0 0x7f6d54070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:03.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d52000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fdec0 0x7f6d54070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55780/0 (socket says 172.21.15.63:55780) 2024-03-23T11:46:03.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.609+0000 7f6d52000700 1 -- 172.21.15.63:0/2532715071 learned_addr learned my addr 172.21.15.63:0/2532715071 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:03.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.610+0000 7f6d52a00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d540fcc10 0x7f6d54071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:03.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.610+0000 7f6d52000700 1 -- 172.21.15.63:0/2532715071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d540fcc10 msgr2=0x7f6d54071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.610+0000 7f6d52000700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d540fcc10 0x7f6d54071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.610+0000 7f6d52000700 1 -- 172.21.15.63:0/2532715071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44005040 con 0x7f6d540fdec0 2024-03-23T11:46:03.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.610+0000 7f6d52000700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fdec0 0x7f6d54070310 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000eb40 tx=0x7f6d4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:03.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.611+0000 7f6d4aa00700 1 -- 172.21.15.63:0/2532715071 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d40007cc0 con 0x7f6d540fdec0 2024-03-23T11:46:03.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.611+0000 7f6d4aa00700 1 -- 172.21.15.63:0/2532715071 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d4000f040 con 0x7f6d540fdec0 2024-03-23T11:46:03.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.611+0000 7f6d4aa00700 1 -- 172.21.15.63:0/2532715071 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d4000c990 con 0x7f6d540fdec0 2024-03-23T11:46:03.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.611+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d54070cb0 con 0x7f6d540fdec0 2024-03-23T11:46:03.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.611+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d54115730 con 0x7f6d540fdec0 2024-03-23T11:46:03.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.612+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5405f5f0 con 0x7f6d540fdec0 2024-03-23T11:46:03.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.613+0000 7f6d4aa00700 1 -- 172.21.15.63:0/2532715071 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6d4000cc00 con 0x7f6d540fdec0 2024-03-23T11:46:03.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.614+0000 7f6d4aa00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6d380692b0 0x7f6d3806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:03.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.614+0000 7f6d4aa00700 1 -- 172.21.15.63:0/2532715071 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6d40086df0 con 0x7f6d540fdec0 2024-03-23T11:46:03.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.615+0000 7f6d52a00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6d380692b0 0x7f6d3806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:03.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.615+0000 7f6d52a00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6d380692b0 0x7f6d3806b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6d440137d0 tx=0x7f6d44009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:03.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.621+0000 7f6d4aa00700 1 -- 172.21.15.63:0/2532715071 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6d4004e190 con 0x7f6d540fdec0 2024-03-23T11:46:03.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:03 smithi063 ceph-mon[35189]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:03.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:03 smithi111 ceph-mon[44005]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:03.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.849+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 --> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d54102100 con 0x7f6d380692b0 2024-03-23T11:46:03.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.850+0000 7f6d4aa00700 1 -- 172.21.15.63:0/2532715071 <== mgr.14204 v2:172.21.15.63:6800/2193815059 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f6d54102100 con 0x7f6d380692b0 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [], 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "0/2 daemons upgraded", 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 image on host smithi111", 2024-03-23T11:46:03.835 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:46:03.836 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.852+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6d380692b0 msgr2=0x7f6d3806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.852+0000 7f6d53e00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6d380692b0 0x7f6d3806b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6d440137d0 tx=0x7f6d44009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.852+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fdec0 msgr2=0x7f6d54070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.852+0000 7f6d53e00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fdec0 0x7f6d54070310 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000eb40 tx=0x7f6d4000b040 comp rx=0 tx=0).stop 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.853+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 shutdown_connections 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.853+0000 7f6d53e00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:6800/2193815059,v1:172.21.15.63:6801/2193815059] conn(0x7f6d380692b0 0x7f6d3806b760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.853+0000 7f6d53e00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d540fcc10 0x7f6d54071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.853+0000 7f6d53e00700 1 --2- 172.21.15.63:0/2532715071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d540fdec0 0x7f6d54070310 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:03.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.853+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 >> 172.21.15.63:0/2532715071 conn(0x7f6d540f83d0 msgr2=0x7f6d540f9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:03.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.853+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 shutdown_connections 2024-03-23T11:46:03.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:03.853+0000 7f6d53e00700 1 -- 172.21.15.63:0/2532715071 wait complete. 2024-03-23T11:46:04.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:04 smithi063 ceph-mon[35189]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:04.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:04 smithi063 ceph-mon[35189]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:04.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:04 smithi063 ceph-mon[35189]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:46:04.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:04 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3877079171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:46:04.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:04 smithi111 ceph-mon[44005]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:04.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:04 smithi111 ceph-mon[44005]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:04.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:04 smithi111 ceph-mon[44005]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:46:04.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:04 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3877079171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:46:05.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:05 smithi063 ceph-mon[35189]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:05.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:05 smithi111 ceph-mon[44005]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:06.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:06 smithi063 ceph-mon[35189]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:06.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:06 smithi111 ceph-mon[44005]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:08.269 INFO:tasks.workunit.client.0.smithi063.stderr:Updating files: 61% (7526/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-23T11:46:08.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:08 smithi063 ceph-mon[35189]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:08.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:08 smithi111 ceph-mon[44005]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:09.127 INFO:tasks.workunit.client.0.smithi063.stderr:Note: switching to 'b630d6cd702af63dfefcf906294a092ddce1c609'. 2024-03-23T11:46:09.127 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:state without impacting any branches by switching back to a branch. 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: git switch -c 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:Or undo this operation with: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: git switch - 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-03-23T11:46:09.128 INFO:tasks.workunit.client.0.smithi063.stderr:HEAD is now at b630d6cd702 DNM: new commit, just to trigger builds 2024-03-23T11:46:09.143 DEBUG:teuthology.orchestra.run.smithi063:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-23T11:46:09.200 INFO:tasks.workunit.client.0.smithi063.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-23T11:46:09.205 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-23T11:46:09.205 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-23T11:46:09.844 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-23T11:46:09.931 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-23T11:46:09.996 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-23T11:46:10.000 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-23T11:46:10.000 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-23T11:46:10.063 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-23T11:46:10.070 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T11:46:10.070 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-23T11:46:10.115 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-23T11:46:10.116 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-23T11:46:10.116 DEBUG:teuthology.orchestra.run.smithi063:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b630d6cd702af63dfefcf906294a092ddce1c609 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-23T11:46:10.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:09 smithi063 ceph-mon[35189]: Upgrade: Updating mgr.smithi111.nnblsk 2024-03-23T11:46:10.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:10.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:46:10.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:46:10.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:09 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:46:10.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:09 smithi063 ceph-mon[35189]: Deploying daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:46:10.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:09 smithi063 ceph-mon[35189]: pgmap v271: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:46:10.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:09 smithi111 ceph-mon[44005]: Upgrade: Updating mgr.smithi111.nnblsk 2024-03-23T11:46:10.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:10.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:46:10.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:46:10.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:09 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:46:10.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:09 smithi111 ceph-mon[44005]: Deploying daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:46:10.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:09 smithi111 ceph-mon[44005]: pgmap v271: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:46:10.199 INFO:tasks.workunit.client.0.smithi063.stderr:+ mkdir -p fsstress 2024-03-23T11:46:10.204 INFO:tasks.workunit.client.0.smithi063.stderr:+ pushd fsstress 2024-03-23T11:46:10.206 INFO:tasks.workunit.client.0.smithi063.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-23T11:46:10.206 INFO:tasks.workunit.client.0.smithi063.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-23T11:46:10.974 INFO:tasks.workunit.client.0.smithi063.stderr:+ tar xzf ltp-full.tgz 2024-03-23T11:46:12.110 INFO:tasks.workunit.client.1.smithi111.stderr:Updating files: 61% (7565/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 96% (11800/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-23T11:46:12.177 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:11 smithi111 ceph-mon[44005]: pgmap v272: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:12.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:11 smithi063 ceph-mon[35189]: pgmap v272: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr:Note: switching to 'b630d6cd702af63dfefcf906294a092ddce1c609'. 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr: 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr:state without impacting any branches by switching back to a branch. 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr: 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-23T11:46:12.989 INFO:tasks.workunit.client.1.smithi111.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr: 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr: git switch -c 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr: 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr:Or undo this operation with: 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr: 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr: git switch - 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr: 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr: 2024-03-23T11:46:12.990 INFO:tasks.workunit.client.1.smithi111.stderr:HEAD is now at b630d6cd702 DNM: new commit, just to trigger builds 2024-03-23T11:46:13.006 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-23T11:46:13.067 INFO:tasks.workunit.client.1.smithi111.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-23T11:46:13.073 INFO:tasks.workunit.client.1.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-23T11:46:13.073 INFO:tasks.workunit.client.1.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-23T11:46:14.102 INFO:tasks.workunit.client.1.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-23T11:46:14.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:13 smithi111 ceph-mon[44005]: pgmap v273: 65 pgs: 65 active+clean; 7.5 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 601 KiB/s wr, 17 op/s 2024-03-23T11:46:14.244 INFO:tasks.workunit.client.1.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-23T11:46:14.363 INFO:tasks.workunit.client.1.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-23T11:46:14.368 INFO:tasks.workunit.client.1.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-23T11:46:14.368 INFO:tasks.workunit.client.1.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-23T11:46:14.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:13 smithi063 ceph-mon[35189]: pgmap v273: 65 pgs: 65 active+clean; 7.5 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 601 KiB/s wr, 17 op/s 2024-03-23T11:46:14.436 INFO:tasks.workunit.client.1.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-23T11:46:14.445 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-23T11:46:14.445 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-23T11:46:14.495 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-23T11:46:14.496 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-23T11:46:14.496 DEBUG:teuthology.orchestra.run.smithi111:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b630d6cd702af63dfefcf906294a092ddce1c609 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-23T11:46:14.572 INFO:tasks.workunit.client.1.smithi111.stderr:+ mkdir -p fsstress 2024-03-23T11:46:14.578 INFO:tasks.workunit.client.1.smithi111.stderr:+ pushd fsstress 2024-03-23T11:46:14.580 INFO:tasks.workunit.client.1.smithi111.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-23T11:46:14.580 INFO:tasks.workunit.client.1.smithi111.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-23T11:46:15.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:15.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:15.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:15.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:15 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:15.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:15.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:15.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:15.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:15 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:15.698 INFO:tasks.workunit.client.1.smithi111.stderr:+ tar xzf ltp-full.tgz 2024-03-23T11:46:16.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:16 smithi063 ceph-mon[35189]: pgmap v274: 65 pgs: 65 active+clean; 12 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-03-23T11:46:16.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:16 smithi111 ceph-mon[44005]: pgmap v274: 65 pgs: 65 active+clean; 12 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-03-23T11:46:18.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:18 smithi063 ceph-mon[35189]: pgmap v275: 65 pgs: 65 active+clean; 13 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-23T11:46:18.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:18 smithi111 ceph-mon[44005]: pgmap v275: 65 pgs: 65 active+clean; 13 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-03-23T11:46:19.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:19 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:19.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:19 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:19.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:19 smithi111 ceph-mon[44005]: pgmap v276: 65 pgs: 65 active+clean; 23 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 215 KiB/s rd, 1.9 MiB/s wr, 128 op/s 2024-03-23T11:46:19.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:19 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:19.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:19 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:19.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:19 smithi063 ceph-mon[35189]: pgmap v276: 65 pgs: 65 active+clean; 23 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 215 KiB/s rd, 1.9 MiB/s wr, 128 op/s 2024-03-23T11:46:21.070 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:21 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:21.070 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:21 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:21.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:21 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:21.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:21 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:22.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:22 smithi063 ceph-mon[35189]: pgmap v277: 65 pgs: 65 active+clean; 34 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 467 KiB/s rd, 2.9 MiB/s wr, 181 op/s 2024-03-23T11:46:22.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:22 smithi111 ceph-mon[44005]: pgmap v277: 65 pgs: 65 active+clean; 34 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 467 KiB/s rd, 2.9 MiB/s wr, 181 op/s 2024-03-23T11:46:24.340 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:24 smithi111 ceph-mon[44005]: pgmap v278: 65 pgs: 65 active+clean; 38 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 597 KiB/s rd, 3.3 MiB/s wr, 239 op/s 2024-03-23T11:46:24.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:24 smithi063 ceph-mon[35189]: pgmap v278: 65 pgs: 65 active+clean; 38 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 597 KiB/s rd, 3.3 MiB/s wr, 239 op/s 2024-03-23T11:46:25.275 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:25 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:46:25.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:25 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:46:26.364 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: pgmap v279: 65 pgs: 65 active+clean; 42 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 595 KiB/s rd, 3.1 MiB/s wr, 282 op/s 2024-03-23T11:46:26.364 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk restarted 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:46:26.365 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:26 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:26.371 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: pgmap v279: 65 pgs: 65 active+clean; 42 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 595 KiB/s rd, 3.1 MiB/s wr, 282 op/s 2024-03-23T11:46:26.371 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: Standby manager daemon smithi111.nnblsk restarted 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:46:26.372 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:26 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: mgrmap e20: smithi063.gehkcr(active, since 8m), standbys: smithi111.nnblsk 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr fail", "who": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: osdmap e47: 6 total, 6 up, 6 in 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "mgr fail", "who": "smithi063.gehkcr"}]': finished 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: mgrmap e21: smithi111.nnblsk(active, starting, since 0.0104416s) 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:46:27.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: Manager daemon smithi111.nnblsk is now available 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:27.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:27.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:27.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:46:27.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:46:27.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/trash_purge_schedule"}]: dispatch 2024-03-23T11:46:27.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/trash_purge_schedule"}]: dispatch 2024-03-23T11:46:27.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:27.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:27 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: mgrmap e20: smithi063.gehkcr(active, since 8m), standbys: smithi111.nnblsk 2024-03-23T11:46:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr fail", "who": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:46:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: osdmap e47: 6 total, 6 up, 6 in 2024-03-23T11:46:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.14204 172.21.15.63:0/1574091977' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "mgr fail", "who": "smithi063.gehkcr"}]': finished 2024-03-23T11:46:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: mgrmap e21: smithi111.nnblsk(active, starting, since 0.0104416s) 2024-03-23T11:46:27.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:46:27.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: Manager daemon smithi111.nnblsk is now available 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/trash_purge_schedule"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.nnblsk/trash_purge_schedule"}]: dispatch 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:27.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:27 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: Deploying cephadm binary to smithi111 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: Deploying cephadm binary to smithi063 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: mgrmap e22: smithi111.nnblsk(active, since 1.0136s) 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: pgmap v3: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: [23/Mar/2024:11:46:27] ENGINE Bus STARTING 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: [23/Mar/2024:11:46:27] ENGINE Serving on https://172.21.15.111:7150 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: [23/Mar/2024:11:46:27] ENGINE Serving on http://172.21.15.111:8765 2024-03-23T11:46:28.113 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:28 smithi111 ceph-mon[44005]: [23/Mar/2024:11:46:27] ENGINE Bus STARTED 2024-03-23T11:46:28.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: Deploying cephadm binary to smithi111 2024-03-23T11:46:28.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: Deploying cephadm binary to smithi063 2024-03-23T11:46:28.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: mgrmap e22: smithi111.nnblsk(active, since 1.0136s) 2024-03-23T11:46:28.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: pgmap v3: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:46:28.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: [23/Mar/2024:11:46:27] ENGINE Bus STARTING 2024-03-23T11:46:28.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: [23/Mar/2024:11:46:27] ENGINE Serving on https://172.21.15.111:7150 2024-03-23T11:46:28.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: [23/Mar/2024:11:46:27] ENGINE Serving on http://172.21.15.111:8765 2024-03-23T11:46:28.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:28 smithi063 ceph-mon[35189]: [23/Mar/2024:11:46:27] ENGINE Bus STARTED 2024-03-23T11:46:29.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:29 smithi063 ceph-mon[35189]: mgrmap e23: smithi111.nnblsk(active, since 2s) 2024-03-23T11:46:29.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:29 smithi063 ceph-mon[35189]: pgmap v4: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:46:29.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:29 smithi111 ceph-mon[44005]: mgrmap e23: smithi111.nnblsk(active, since 2s) 2024-03-23T11:46:29.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:29 smithi111 ceph-mon[44005]: pgmap v4: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:46:31.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:31 smithi063 ceph-mon[35189]: pgmap v5: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:46:31.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:31 smithi063 ceph-mon[35189]: mgrmap e24: smithi111.nnblsk(active, since 4s) 2024-03-23T11:46:31.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:31 smithi111 ceph-mon[44005]: pgmap v5: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:46:31.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:31 smithi111 ceph-mon[44005]: mgrmap e24: smithi111.nnblsk(active, since 4s) 2024-03-23T11:46:33.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:33 smithi063 ceph-mon[35189]: pgmap v6: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 304 op/s 2024-03-23T11:46:33.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:33 smithi111 ceph-mon[44005]: pgmap v6: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 304 op/s 2024-03-23T11:46:34.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 -- 172.21.15.63:0/3012606872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3080071e40 msgr2=0x7f3080070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:34.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 --2- 172.21.15.63:0/3012606872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3080071e40 0x7f3080070350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3070004940 tx=0x7f3070009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:34.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 -- 172.21.15.63:0/3012606872 shutdown_connections 2024-03-23T11:46:34.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 --2- 172.21.15.63:0/3012606872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 0x7f3080070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 --2- 172.21.15.63:0/3012606872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3080071e40 0x7f3080070350 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 -- 172.21.15.63:0/3012606872 >> 172.21.15.63:0/3012606872 conn(0x7f30800f8580 msgr2=0x7f30800fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:34.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 -- 172.21.15.63:0/3012606872 shutdown_connections 2024-03-23T11:46:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.099+0000 7f3086000700 1 -- 172.21.15.63:0/3012606872 wait complete. 2024-03-23T11:46:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.101+0000 7f3086000700 1 Processor -- start 2024-03-23T11:46:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.102+0000 7f3086000700 1 -- start start 2024-03-23T11:46:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.102+0000 7f3086000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 0x7f30801141d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.102+0000 7f3086000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3080114710 0x7f3080115510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.102+0000 7f3086000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080114bd0 con 0x7f3080114710 2024-03-23T11:46:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.102+0000 7f3086000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080114d10 con 0x7f3080070890 2024-03-23T11:46:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.103+0000 7f307f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 0x7f30801141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.103+0000 7f307f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 0x7f30801141d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33572/0 (socket says 172.21.15.63:33572) 2024-03-23T11:46:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.103+0000 7f307f400700 1 -- 172.21.15.63:0/3745032994 learned_addr learned my addr 172.21.15.63:0/3745032994 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.103+0000 7f307f400700 1 -- 172.21.15.63:0/3745032994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3080114710 msgr2=0x7f3080115510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:46:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.103+0000 7f307f400700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3080114710 0x7f3080115510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.103+0000 7f307f400700 1 -- 172.21.15.63:0/3745032994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30780048f0 con 0x7f3080070890 2024-03-23T11:46:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.103+0000 7f307f400700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 0x7f30801141d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3070013870 tx=0x7f3070009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.104+0000 7f3077400700 1 -- 172.21.15.63:0/3745032994 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f307000e070 con 0x7f3080070890 2024-03-23T11:46:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.104+0000 7f3077400700 1 -- 172.21.15.63:0/3745032994 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3070013dc0 con 0x7f3080070890 2024-03-23T11:46:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.104+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3070005040 con 0x7f3080070890 2024-03-23T11:46:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.104+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3080115f10 con 0x7f3080070890 2024-03-23T11:46:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.105+0000 7f3077400700 1 -- 172.21.15.63:0/3745032994 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f307001ce60 con 0x7f3080070890 2024-03-23T11:46:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.106+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3080103a50 con 0x7f3080070890 2024-03-23T11:46:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.106+0000 7f3077400700 1 -- 172.21.15.63:0/3745032994 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 45523+0+0 (secure 0 0 0) 0x7f3070025400 con 0x7f3080070890 2024-03-23T11:46:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.107+0000 7f3077400700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f3064038470 0x7f306403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.107+0000 7f3077400700 1 -- 172.21.15.63:0/3745032994 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f307004dff0 con 0x7f3080070890 2024-03-23T11:46:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.107+0000 7f307ea00700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f3064038470 0x7f306403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.108+0000 7f307ea00700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f3064038470 0x7f306403a920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3078007b80 tx=0x7f3078009ca0 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:34.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.115+0000 7f3077400700 1 -- 172.21.15.63:0/3745032994 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3070011730 con 0x7f3080070890 2024-03-23T11:46:34.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.496+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3080116150 con 0x7f3064038470 2024-03-23T11:46:34.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.498+0000 7f3077400700 1 -- 172.21.15.63:0/3745032994 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f3080116150 con 0x7f3064038470 2024-03-23T11:46:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.500+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f3064038470 msgr2=0x7f306403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f3064038470 0x7f306403a920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3078007b80 tx=0x7f3078009ca0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 msgr2=0x7f30801141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 0x7f30801141d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3070013870 tx=0x7f3070009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:34.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 shutdown_connections 2024-03-23T11:46:34.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f3064038470 0x7f306403a920 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3080070890 0x7f30801141d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 --2- 172.21.15.63:0/3745032994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3080114710 0x7f3080115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.501+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 >> 172.21.15.63:0/3745032994 conn(0x7f30800f8580 msgr2=0x7f3080100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:34.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.502+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 shutdown_connections 2024-03-23T11:46:34.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.502+0000 7f3086000700 1 -- 172.21.15.63:0/3745032994 wait complete. 2024-03-23T11:46:34.508 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:46:34.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.707+0000 7fb861200700 1 -- 172.21.15.63:0/3030219378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 msgr2=0x7fb85c0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:34.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.707+0000 7fb861200700 1 --2- 172.21.15.63:0/3030219378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 0x7fb85c0fc0e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb848004940 tx=0x7fb848009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:34.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.708+0000 7fb861200700 1 -- 172.21.15.63:0/3030219378 shutdown_connections 2024-03-23T11:46:34.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.708+0000 7fb861200700 1 --2- 172.21.15.63:0/3030219378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb85c0fcf80 0x7fb85c0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.708+0000 7fb861200700 1 --2- 172.21.15.63:0/3030219378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 0x7fb85c0fc0e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.708+0000 7fb861200700 1 -- 172.21.15.63:0/3030219378 >> 172.21.15.63:0/3030219378 conn(0x7fb85c0f7430 msgr2=0x7fb85c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:34.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.709+0000 7fb861200700 1 -- 172.21.15.63:0/3030219378 shutdown_connections 2024-03-23T11:46:34.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.709+0000 7fb861200700 1 -- 172.21.15.63:0/3030219378 wait complete. 2024-03-23T11:46:34.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.710+0000 7fb861200700 1 Processor -- start 2024-03-23T11:46:34.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.711+0000 7fb861200700 1 -- start start 2024-03-23T11:46:34.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.711+0000 7fb861200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 0x7fb85c113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:34.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.711+0000 7fb861200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb85c0fcf80 0x7fb85c114510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:34.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.711+0000 7fb861200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c114ba0 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.711+0000 7fb861200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c115390 con 0x7fb85c0fcf80 2024-03-23T11:46:34.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.712+0000 7fb859400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb85c0fcf80 0x7fb85c114510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:34.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.712+0000 7fb859400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb85c0fcf80 0x7fb85c114510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33596/0 (socket says 172.21.15.63:33596) 2024-03-23T11:46:34.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.712+0000 7fb859400700 1 -- 172.21.15.63:0/4100495765 learned_addr learned my addr 172.21.15.63:0/4100495765 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:34.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.713+0000 7fb859e00700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 0x7fb85c113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:34.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.714+0000 7fb859e00700 1 -- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb85c0fcf80 msgr2=0x7fb85c114510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:34.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.714+0000 7fb859e00700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb85c0fcf80 0x7fb85c114510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:34.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.714+0000 7fb859e00700 1 -- 172.21.15.63:0/4100495765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848005040 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.716+0000 7fb859e00700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 0x7fb85c113fd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb848004890 tx=0x7fb848009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:34.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.716+0000 7fb852000700 1 -- 172.21.15.63:0/4100495765 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb84800e030 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.717+0000 7fb852000700 1 -- 172.21.15.63:0/4100495765 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb848013420 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.717+0000 7fb852000700 1 -- 172.21.15.63:0/4100495765 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb84801c640 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.717+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c115610 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.717+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c115bd0 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.719+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c05f5f0 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.720+0000 7fb852000700 1 -- 172.21.15.63:0/4100495765 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 24) v1 ==== 45523+0+0 (secure 0 0 0) 0x7fb848013970 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.721+0000 7fb852000700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb8400384b0 0x7fb84003a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:34.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.721+0000 7fb852000700 1 -- 172.21.15.63:0/4100495765 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fb84804d6c0 con 0x7fb85c0fbcd0 2024-03-23T11:46:34.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.725+0000 7fb859400700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb8400384b0 0x7fb84003a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:34.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.726+0000 7fb859400700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb8400384b0 0x7fb84003a960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009aa0 tx=0x7fb84c008040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:34.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:34.738+0000 7fb852000700 1 -- 172.21.15.63:0/4100495765 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb848011550 con 0x7fb85c0fbcd0 2024-03-23T11:46:35.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.141+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb85c115fc0 con 0x7fb8400384b0 2024-03-23T11:46:35.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.143+0000 7fb852000700 1 -- 172.21.15.63:0/4100495765 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fb85c115fc0 con 0x7fb8400384b0 2024-03-23T11:46:35.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.146+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb8400384b0 msgr2=0x7fb84003a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.146+0000 7fb861200700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb8400384b0 0x7fb84003a960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009aa0 tx=0x7fb84c008040 comp rx=0 tx=0).stop 2024-03-23T11:46:35.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 msgr2=0x7fb85c113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 0x7fb85c113fd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb848004890 tx=0x7fb848009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 shutdown_connections 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb85c0fbcd0 0x7fb85c113fd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb8400384b0 0x7fb84003a960 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 --2- 172.21.15.63:0/4100495765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb85c0fcf80 0x7fb85c114510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 >> 172.21.15.63:0/4100495765 conn(0x7fb85c0f7430 msgr2=0x7fb85c0f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 shutdown_connections 2024-03-23T11:46:35.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.147+0000 7fb861200700 1 -- 172.21.15.63:0/4100495765 wait complete. 2024-03-23T11:46:35.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.365+0000 7f9e61200700 1 -- 172.21.15.63:0/3135669702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c066280 msgr2=0x7f9e5c0ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.365+0000 7f9e61200700 1 --2- 172.21.15.63:0/3135669702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c066280 0x7f9e5c0ffa20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009ce0 tx=0x7f9e54004d10 comp rx=0 tx=0).stop 2024-03-23T11:46:35.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.365+0000 7f9e61200700 1 -- 172.21.15.63:0/3135669702 shutdown_connections 2024-03-23T11:46:35.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.366+0000 7f9e61200700 1 --2- 172.21.15.63:0/3135669702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c0fff60 0x7f9e5c102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.366+0000 7f9e61200700 1 --2- 172.21.15.63:0/3135669702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c066280 0x7f9e5c0ffa20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.366+0000 7f9e61200700 1 -- 172.21.15.63:0/3135669702 >> 172.21.15.63:0/3135669702 conn(0x7f9e5c0f7430 msgr2=0x7f9e5c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:35.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.366+0000 7f9e61200700 1 -- 172.21.15.63:0/3135669702 shutdown_connections 2024-03-23T11:46:35.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.366+0000 7f9e61200700 1 -- 172.21.15.63:0/3135669702 wait complete. 2024-03-23T11:46:35.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.367+0000 7f9e61200700 1 Processor -- start 2024-03-23T11:46:35.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e61200700 1 -- start start 2024-03-23T11:46:35.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e61200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c066280 0x7f9e5c10b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:35.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e61200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c0fff60 0x7f9e5c10bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:35.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e61200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c10c3e0 con 0x7f9e5c066280 2024-03-23T11:46:35.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e61200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c10cc30 con 0x7f9e5c0fff60 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e59e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c066280 0x7f9e5c10b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e59e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c066280 0x7f9e5c10b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47312/0 (socket says 172.21.15.63:47312) 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e59e00700 1 -- 172.21.15.63:0/2598665366 learned_addr learned my addr 172.21.15.63:0/2598665366 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.368+0000 7f9e59400700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c0fff60 0x7f9e5c10bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.369+0000 7f9e59e00700 1 -- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c0fff60 msgr2=0x7f9e5c10bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.369+0000 7f9e59e00700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c0fff60 0x7f9e5c10bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.369+0000 7f9e59e00700 1 -- 172.21.15.63:0/2598665366 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54009940 con 0x7f9e5c066280 2024-03-23T11:46:35.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.369+0000 7f9e59e00700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c066280 0x7f9e5c10b8a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009490 tx=0x7f9e54004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:35.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.370+0000 7f9e52000700 1 -- 172.21.15.63:0/2598665366 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e5400e070 con 0x7f9e5c066280 2024-03-23T11:46:35.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.370+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c10ce30 con 0x7f9e5c066280 2024-03-23T11:46:35.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.370+0000 7f9e52000700 1 -- 172.21.15.63:0/2598665366 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e54007d30 con 0x7f9e5c066280 2024-03-23T11:46:35.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.370+0000 7f9e52000700 1 -- 172.21.15.63:0/2598665366 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e54021680 con 0x7f9e5c066280 2024-03-23T11:46:35.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.370+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c10d3f0 con 0x7f9e5c066280 2024-03-23T11:46:35.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.371+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c05f5f0 con 0x7f9e5c066280 2024-03-23T11:46:35.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.372+0000 7f9e52000700 1 -- 172.21.15.63:0/2598665366 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 24) v1 ==== 45523+0+0 (secure 0 0 0) 0x7f9e54016070 con 0x7f9e5c066280 2024-03-23T11:46:35.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.373+0000 7f9e52000700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f9e40038460 0x7f9e4003a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:35.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.373+0000 7f9e52000700 1 -- 172.21.15.63:0/2598665366 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f9e5404cee0 con 0x7f9e5c066280 2024-03-23T11:46:35.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.375+0000 7f9e59400700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f9e40038460 0x7f9e4003a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:35.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.375+0000 7f9e59400700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f9e40038460 0x7f9e4003a910 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9e480041d0 tx=0x7f9e48016040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:35.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.383+0000 7f9e52000700 1 -- 172.21.15.63:0/2598665366 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f9e54004e30 con 0x7f9e5c066280 2024-03-23T11:46:35.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:35 smithi063 ceph-mon[35189]: pgmap v7: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 916 KiB/s rd, 3.0 MiB/s wr, 269 op/s 2024-03-23T11:46:35.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:35 smithi063 ceph-mon[35189]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:35.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:35 smithi111 ceph-mon[44005]: pgmap v7: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 916 KiB/s rd, 3.0 MiB/s wr, 269 op/s 2024-03-23T11:46:35.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:35 smithi111 ceph-mon[44005]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:35.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.730+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e5c10d810 con 0x7f9e40038460 2024-03-23T11:46:35.731 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:46:35.731 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (7m) 2m ago 8m 13.4M - 0.25.0 c8568f914cd2 c31e5bfcace0 2024-03-23T11:46:35.732 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (9m) 2m ago 9m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:46:35.732 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (8m) 17s ago 8m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:46:35.732 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (7m) 2m ago 8m 73.5M - 9.4.7 954c08fa6188 d05e188def00 2024-03-23T11:46:35.732 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (2m) 2m ago 2m 13.3M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:46:35.732 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (2m) 2m ago 2m 12.9M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:46:35.732 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (2m) 17s ago 2m 25.8M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:46:35.733 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (2m) 17s ago 2m 16.8M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:46:35.733 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:9283 running (10m) 2m ago 10m 480M - 17.2.7-726-g9c4e0c34 837124da804f 63833d132315 2024-03-23T11:46:35.733 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (22s) 17s ago 7m 57.5M - 18.2.2-820-gb630d6cd 881b71eac2fe 278e3d19ec5d 2024-03-23T11:46:35.733 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (10m) 2m ago 10m 44.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:46:35.733 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (7m) 17s ago 7m 35.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:46:35.733 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (9m) 2m ago 9m 7537k - 1.5.0 0da6a335fe13 e942e152db56 2024-03-23T11:46:35.734 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 17s ago 7m 7860k - 1.5.0 0da6a335fe13 68f97d563d92 2024-03-23T11:46:35.734 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (6m) 2m ago 6m 45.4M 4348M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:46:35.734 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (6m) 2m ago 6m 45.6M 4348M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:46:35.734 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (5m) 2m ago 5m 43.4M 4348M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:46:35.734 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (5m) 17s ago 5m 52.3M 5372M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:46:35.734 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (4m) 17s ago 4m 53.2M 5372M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:46:35.735 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (4m) 17s ago 4m 49.8M 5372M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:46:35.735 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (7m) 2m ago 8m 46.3M - 2.43.0 a07b618ecd1d 328fbe2c4719 2024-03-23T11:46:35.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.745+0000 7f9e52000700 1 -- 172.21.15.63:0/2598665366 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f9e5c10d810 con 0x7f9e40038460 2024-03-23T11:46:35.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f9e40038460 msgr2=0x7f9e4003a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f9e40038460 0x7f9e4003a910 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9e480041d0 tx=0x7f9e48016040 comp rx=0 tx=0).stop 2024-03-23T11:46:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c066280 msgr2=0x7f9e5c10b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c066280 0x7f9e5c10b8a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009490 tx=0x7f9e54004d10 comp rx=0 tx=0).stop 2024-03-23T11:46:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 shutdown_connections 2024-03-23T11:46:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e5c066280 0x7f9e5c10b8a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f9e40038460 0x7f9e4003a910 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 --2- 172.21.15.63:0/2598665366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c0fff60 0x7f9e5c10bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 >> 172.21.15.63:0/2598665366 conn(0x7f9e5c0f7430 msgr2=0x7f9e5c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:35.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 shutdown_connections 2024-03-23T11:46:35.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.750+0000 7f9e61200700 1 -- 172.21.15.63:0/2598665366 wait complete. 2024-03-23T11:46:35.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.969+0000 7fde6b200700 1 -- 172.21.15.63:0/2107338996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 msgr2=0x7fde6406ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.969+0000 7fde6b200700 1 --2- 172.21.15.63:0/2107338996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 0x7fde6406ee10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde4c004940 tx=0x7fde4c009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:35.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.969+0000 7fde6b200700 1 -- 172.21.15.63:0/2107338996 shutdown_connections 2024-03-23T11:46:35.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.969+0000 7fde6b200700 1 --2- 172.21.15.63:0/2107338996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 0x7fde6406ee10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.969+0000 7fde6b200700 1 --2- 172.21.15.63:0/2107338996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 0x7fde6406e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.969+0000 7fde6b200700 1 -- 172.21.15.63:0/2107338996 >> 172.21.15.63:0/2107338996 conn(0x7fde64069de0 msgr2=0x7fde6406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:35.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.969+0000 7fde6b200700 1 -- 172.21.15.63:0/2107338996 shutdown_connections 2024-03-23T11:46:35.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.970+0000 7fde6b200700 1 -- 172.21.15.63:0/2107338996 wait complete. 2024-03-23T11:46:35.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.970+0000 7fde6b200700 1 Processor -- start 2024-03-23T11:46:35.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.971+0000 7fde6b200700 1 -- start start 2024-03-23T11:46:35.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.971+0000 7fde6b200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 0x7fde64115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:35.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.972+0000 7fde6b200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 0x7fde641164f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:35.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.972+0000 7fde6b200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde64116af0 con 0x7fde6406f9d0 2024-03-23T11:46:35.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.972+0000 7fde6b200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde64116c60 con 0x7fde6406e9a0 2024-03-23T11:46:35.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.972+0000 7fde5be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 0x7fde641164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:35.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.972+0000 7fde5be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 0x7fde641164f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47326/0 (socket says 172.21.15.63:47326) 2024-03-23T11:46:35.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.972+0000 7fde5be00700 1 -- 172.21.15.63:0/3201220844 learned_addr learned my addr 172.21.15.63:0/3201220844 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:35.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.973+0000 7fde63e00700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 0x7fde64115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:35.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.973+0000 7fde5be00700 1 -- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 msgr2=0x7fde64115fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:35.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.973+0000 7fde5be00700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 0x7fde64115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:35.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.973+0000 7fde5be00700 1 -- 172.21.15.63:0/3201220844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c005040 con 0x7fde6406f9d0 2024-03-23T11:46:35.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.973+0000 7fde5be00700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 0x7fde641164f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00b3b0 tx=0x7fde4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:35.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.974+0000 7fde60c00700 1 -- 172.21.15.63:0/3201220844 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde4c00e040 con 0x7fde6406f9d0 2024-03-23T11:46:35.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.974+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde64104fc0 con 0x7fde6406f9d0 2024-03-23T11:46:35.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.974+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde64105580 con 0x7fde6406f9d0 2024-03-23T11:46:35.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.974+0000 7fde60c00700 1 -- 172.21.15.63:0/3201220844 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde4c004290 con 0x7fde6406f9d0 2024-03-23T11:46:35.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.974+0000 7fde60c00700 1 -- 172.21.15.63:0/3201220844 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde4c01c3f0 con 0x7fde6406f9d0 2024-03-23T11:46:35.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.976+0000 7fde60c00700 1 -- 172.21.15.63:0/3201220844 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 24) v1 ==== 45523+0+0 (secure 0 0 0) 0x7fde4c01c660 con 0x7fde6406f9d0 2024-03-23T11:46:35.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.977+0000 7fde60c00700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fde48038460 0x7fde4803a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:35.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.977+0000 7fde60c00700 1 -- 172.21.15.63:0/3201220844 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fde4c04cfc0 con 0x7fde6406f9d0 2024-03-23T11:46:35.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.977+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde400052e0 con 0x7fde6406f9d0 2024-03-23T11:46:35.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.978+0000 7fde63e00700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fde48038460 0x7fde4803a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:35.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.980+0000 7fde63e00700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fde48038460 0x7fde4803a910 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde540099f0 tx=0x7fde5400b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:35.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:35.984+0000 7fde60c00700 1 -- 172.21.15.63:0/3201220844 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fde4c025400 con 0x7fde6406f9d0 2024-03-23T11:46:36.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:36 smithi111 ceph-mon[44005]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:36.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:36 smithi111 ceph-mon[44005]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:36.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.425+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fde40005860 con 0x7fde6406f9d0 2024-03-23T11:46:36.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.426+0000 7fde60c00700 1 -- 172.21.15.63:0/3201220844 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7fde4c011b60 con 0x7fde6406f9d0 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:46:36.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:46:36.416 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fde48038460 msgr2=0x7fde4803a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fde48038460 0x7fde4803a910 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde540099f0 tx=0x7fde5400b040 comp rx=0 tx=0).stop 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 msgr2=0x7fde641164f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 0x7fde641164f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00b3b0 tx=0x7fde4c009e30 comp rx=0 tx=0).stop 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 shutdown_connections 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fde48038460 0x7fde4803a910 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6406e9a0 0x7fde64115fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:36.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 --2- 172.21.15.63:0/3201220844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde6406f9d0 0x7fde641164f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:36.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.430+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 >> 172.21.15.63:0/3201220844 conn(0x7fde64069de0 msgr2=0x7fde6406b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:36.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.431+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 shutdown_connections 2024-03-23T11:46:36.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.431+0000 7fde6b200700 1 -- 172.21.15.63:0/3201220844 wait complete. 2024-03-23T11:46:36.595 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:36 smithi063 ceph-mon[35189]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:36.595 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:36 smithi063 ceph-mon[35189]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:36.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 -- 172.21.15.63:0/866499174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 msgr2=0x7fa9f4070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:36.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 --2- 172.21.15.63:0/866499174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f4070d00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009ce0 tx=0x7fa9dc004d10 comp rx=0 tx=0).stop 2024-03-23T11:46:36.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 -- 172.21.15.63:0/866499174 shutdown_connections 2024-03-23T11:46:36.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 --2- 172.21.15.63:0/866499174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f4070d00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:36.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 --2- 172.21.15.63:0/866499174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9f4071e40 0x7fa9f4070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:36.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 -- 172.21.15.63:0/866499174 >> 172.21.15.63:0/866499174 conn(0x7fa9f40f8580 msgr2=0x7fa9f40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:36.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 -- 172.21.15.63:0/866499174 shutdown_connections 2024-03-23T11:46:36.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.631+0000 7fa9fb200700 1 -- 172.21.15.63:0/866499174 wait complete. 2024-03-23T11:46:36.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.632+0000 7fa9fb200700 1 Processor -- start 2024-03-23T11:46:36.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.633+0000 7fa9fb200700 1 -- start start 2024-03-23T11:46:36.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.634+0000 7fa9fb200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f410fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:36.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.634+0000 7fa9fb200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9f4071e40 0x7fa9f4110330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:36.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.634+0000 7fa9fb200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f4111050 con 0x7fa9f4070890 2024-03-23T11:46:36.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.634+0000 7fa9fb200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f41111c0 con 0x7fa9f4071e40 2024-03-23T11:46:36.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.635+0000 7fa9f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f410fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:36.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.635+0000 7fa9f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f410fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47336/0 (socket says 172.21.15.63:47336) 2024-03-23T11:46:36.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.635+0000 7fa9f3e00700 1 -- 172.21.15.63:0/87594819 learned_addr learned my addr 172.21.15.63:0/87594819 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:46:36.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.635+0000 7fa9ebe00700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9f4071e40 0x7fa9f4110330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.636+0000 7fa9f3e00700 1 -- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9f4071e40 msgr2=0x7fa9f4110330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.636+0000 7fa9f3e00700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9f4071e40 0x7fa9f4110330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.636+0000 7fa9f3e00700 1 -- 172.21.15.63:0/87594819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc009940 con 0x7fa9f4070890 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.636+0000 7fa9f3e00700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f410fdf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400b790 tx=0x7fa9e4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.637+0000 7fa9f0c00700 1 -- 172.21.15.63:0/87594819 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9e400f6f0 con 0x7fa9f4070890 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.637+0000 7fa9f0c00700 1 -- 172.21.15.63:0/87594819 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9e400fd00 con 0x7fa9f4070890 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.637+0000 7fa9f0c00700 1 -- 172.21.15.63:0/87594819 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9e4011e90 con 0x7fa9f4070890 2024-03-23T11:46:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.637+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f4111440 con 0x7fa9f4070890 2024-03-23T11:46:36.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.637+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f41119b0 con 0x7fa9f4070890 2024-03-23T11:46:36.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.639+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f405f5f0 con 0x7fa9f4070890 2024-03-23T11:46:36.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.640+0000 7fa9f0c00700 1 -- 172.21.15.63:0/87594819 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 24) v1 ==== 45523+0+0 (secure 0 0 0) 0x7fa9e400f850 con 0x7fa9f4070890 2024-03-23T11:46:36.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.641+0000 7fa9f0c00700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa9d8038470 0x7fa9d803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:46:36.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.641+0000 7fa9f0c00700 1 -- 172.21.15.63:0/87594819 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fa9e404c4f0 con 0x7fa9f4070890 2024-03-23T11:46:36.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.644+0000 7fa9ebe00700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa9d8038470 0x7fa9d803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:46:36.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.646+0000 7fa9ebe00700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa9d8038470 0x7fa9d803a920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc015040 tx=0x7fa9dc004d10 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:46:36.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:36.659+0000 7fa9f0c00700 1 -- 172.21.15.63:0/87594819 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa9e4011690 con 0x7fa9f4070890 2024-03-23T11:46:37.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.051+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9f4111d00 con 0x7fa9d8038470 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [], 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "1/2 daemons upgraded", 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "", 2024-03-23T11:46:37.038 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:46:37.039 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:46:37.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.053+0000 7fa9f0c00700 1 -- 172.21.15.63:0/87594819 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fa9f4111d00 con 0x7fa9d8038470 2024-03-23T11:46:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.055+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa9d8038470 msgr2=0x7fa9d803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.055+0000 7fa9fb200700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa9d8038470 0x7fa9d803a920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc015040 tx=0x7fa9dc004d10 comp rx=0 tx=0).stop 2024-03-23T11:46:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.055+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 msgr2=0x7fa9f410fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:46:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.055+0000 7fa9fb200700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f410fdf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400b790 tx=0x7fa9e4009220 comp rx=0 tx=0).stop 2024-03-23T11:46:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.057+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 shutdown_connections 2024-03-23T11:46:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.057+0000 7fa9fb200700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9f4070890 0x7fa9f410fdf0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.057+0000 7fa9fb200700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa9d8038470 0x7fa9d803a920 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.057+0000 7fa9fb200700 1 --2- 172.21.15.63:0/87594819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9f4071e40 0x7fa9f4110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:46:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.057+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 >> 172.21.15.63:0/87594819 conn(0x7fa9f40f8580 msgr2=0x7fa9f40fa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:46:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.058+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 shutdown_connections 2024-03-23T11:46:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:46:37.058+0000 7fa9fb200700 1 -- 172.21.15.63:0/87594819 wait complete. 2024-03-23T11:46:37.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:37 smithi111 ceph-mon[44005]: pgmap v8: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 712 KiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-03-23T11:46:37.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:37 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3201220844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:46:37.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:37 smithi063 ceph-mon[35189]: pgmap v8: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 712 KiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-03-23T11:46:37.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:37 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3201220844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:46:37.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:37 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:37.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:37 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:37.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:37 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:37.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:37 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:38.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:38 smithi111 ceph-mon[44005]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:38.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:38 smithi063 ceph-mon[35189]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: pgmap v9: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.3 MiB/s wr, 323 op/s 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi063 to 1618M 2024-03-23T11:46:39.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:39.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: Standby manager daemon smithi063.gehkcr started 2024-03-23T11:46:39.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/crt"}]: dispatch 2024-03-23T11:46:39.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:46:39.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/key"}]: dispatch 2024-03-23T11:46:39.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:39 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:46:39.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: pgmap v9: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.3 MiB/s wr, 323 op/s 2024-03-23T11:46:39.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi063 to 1618M 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: Standby manager daemon smithi063.gehkcr started 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/crt"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:46:39.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/key"}]: dispatch 2024-03-23T11:46:39.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:39 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/3340924304' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:46:40.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:40 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:46:40.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:40 smithi063 ceph-mon[35189]: mgrmap e25: smithi111.nnblsk(active, since 13s), standbys: smithi063.gehkcr 2024-03-23T11:46:40.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:40 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:40.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:40 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:40.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:40 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:46:40.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:40 smithi111 ceph-mon[44005]: mgrmap e25: smithi111.nnblsk(active, since 13s), standbys: smithi063.gehkcr 2024-03-23T11:46:40.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:40 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:40.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:40 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:42.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:41 smithi063 ceph-mon[35189]: pgmap v10: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 295 op/s 2024-03-23T11:46:42.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:41 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:42.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:41 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:42.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:41 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:42.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:41 smithi111 ceph-mon[44005]: pgmap v10: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 295 op/s 2024-03-23T11:46:42.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:41 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:42.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:41 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:42.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:41 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:44.120 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:43 smithi111 ceph-mon[44005]: pgmap v11: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 430 op/s 2024-03-23T11:46:44.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:43 smithi063 ceph-mon[35189]: pgmap v11: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 430 op/s 2024-03-23T11:46:46.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:45 smithi063 ceph-mon[35189]: pgmap v12: 65 pgs: 65 active+clean; 104 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 340 op/s 2024-03-23T11:46:46.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:45 smithi111 ceph-mon[44005]: pgmap v12: 65 pgs: 65 active+clean; 104 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 340 op/s 2024-03-23T11:46:48.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:47 smithi063 ceph-mon[35189]: pgmap v13: 65 pgs: 65 active+clean; 104 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 310 op/s 2024-03-23T11:46:48.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:47 smithi111 ceph-mon[44005]: pgmap v13: 65 pgs: 65 active+clean; 104 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 310 op/s 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: Adjusting osd_memory_target on smithi111 to 2642M 2024-03-23T11:46:49.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:49.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:46:49.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:46:49.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:46:49.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:48 smithi063 ceph-mon[35189]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:46:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: Adjusting osd_memory_target on smithi111 to 2642M 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:46:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:48 smithi111 ceph-mon[44005]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:46:50.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:49 smithi063 ceph-mon[35189]: pgmap v14: 65 pgs: 65 active+clean; 117 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 474 op/s 2024-03-23T11:46:50.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:49 smithi063 ceph-mon[35189]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:46:50.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:49 smithi063 ceph-mon[35189]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:46:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:49 smithi111 ceph-mon[44005]: pgmap v14: 65 pgs: 65 active+clean; 117 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 474 op/s 2024-03-23T11:46:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:49 smithi111 ceph-mon[44005]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:46:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:49 smithi111 ceph-mon[44005]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:46:51.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:50 smithi063 ceph-mon[35189]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:46:51.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:50 smithi063 ceph-mon[35189]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:46:51.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:50 smithi063 ceph-mon[35189]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:46:51.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:50 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:51.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:50 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:51.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:50 smithi111 ceph-mon[44005]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:46:51.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:50 smithi111 ceph-mon[44005]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:46:51.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:50 smithi111 ceph-mon[44005]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:46:51.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:50 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:51.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:50 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: pgmap v15: 65 pgs: 65 active+clean; 117 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 335 op/s 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:51 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: pgmap v15: 65 pgs: 65 active+clean; 117 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 335 op/s 2024-03-23T11:46:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:46:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:52.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:51 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:46:53.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:52 smithi063 ceph-mon[35189]: Reconfiguring node-exporter.smithi063 (dependencies changed)... 2024-03-23T11:46:53.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:52 smithi063 ceph-mon[35189]: Reconfiguring daemon node-exporter.smithi063 on smithi063 2024-03-23T11:46:53.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:52 smithi111 ceph-mon[44005]: Reconfiguring node-exporter.smithi063 (dependencies changed)... 2024-03-23T11:46:53.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:52 smithi111 ceph-mon[44005]: Reconfiguring daemon node-exporter.smithi063 on smithi063 2024-03-23T11:46:54.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:53 smithi063 ceph-mon[35189]: pgmap v16: 65 pgs: 65 active+clean; 126 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 445 op/s 2024-03-23T11:46:54.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:53 smithi111 ceph-mon[44005]: pgmap v16: 65 pgs: 65 active+clean; 126 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 445 op/s 2024-03-23T11:46:55.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:54 smithi063 ceph-mon[35189]: pgmap v17: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 338 op/s 2024-03-23T11:46:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:54 smithi111 ceph-mon[44005]: pgmap v17: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 338 op/s 2024-03-23T11:46:57.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:57 smithi063 ceph-mon[35189]: pgmap v18: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 302 op/s 2024-03-23T11:46:57.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:57 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:57.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:57 smithi111 ceph-mon[44005]: pgmap v18: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 302 op/s 2024-03-23T11:46:57.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:57 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:46:59.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:46:59 smithi063 ceph-mon[35189]: pgmap v19: 65 pgs: 65 active+clean; 148 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-03-23T11:46:59.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:46:59 smithi111 ceph-mon[44005]: pgmap v19: 65 pgs: 65 active+clean; 148 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-03-23T11:47:01.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:01 smithi063 ceph-mon[35189]: pgmap v20: 65 pgs: 65 active+clean; 148 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-03-23T11:47:01.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:01 smithi111 ceph-mon[44005]: pgmap v20: 65 pgs: 65 active+clean; 148 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 286 op/s 2024-03-23T11:47:03.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:03 smithi063 ceph-mon[35189]: pgmap v21: 65 pgs: 65 active+clean; 161 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 413 op/s 2024-03-23T11:47:03.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:03 smithi111 ceph-mon[44005]: pgmap v21: 65 pgs: 65 active+clean; 161 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 413 op/s 2024-03-23T11:47:05.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:05 smithi063 ceph-mon[35189]: pgmap v22: 65 pgs: 65 active+clean; 164 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 318 op/s 2024-03-23T11:47:05.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:05 smithi111 ceph-mon[44005]: pgmap v22: 65 pgs: 65 active+clean; 164 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 318 op/s 2024-03-23T11:47:07.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.281+0000 7f508a000700 1 -- 172.21.15.63:0/2702244877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 msgr2=0x7f50840fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:07.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.281+0000 7f508a000700 1 --2- 172.21.15.63:0/2702244877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f50840fd3f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5070004940 tx=0x7f5070009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:07.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.281+0000 7f508a000700 1 -- 172.21.15.63:0/2702244877 shutdown_connections 2024-03-23T11:47:07.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.281+0000 7f508a000700 1 --2- 172.21.15.63:0/2702244877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f50840fd3f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.281+0000 7f508a000700 1 --2- 172.21.15.63:0/2702244877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50840fbcd0 0x7f50840fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.281+0000 7f508a000700 1 -- 172.21.15.63:0/2702244877 >> 172.21.15.63:0/2702244877 conn(0x7f50840f7430 msgr2=0x7f50840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.282+0000 7f508a000700 1 -- 172.21.15.63:0/2702244877 shutdown_connections 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.282+0000 7f508a000700 1 -- 172.21.15.63:0/2702244877 wait complete. 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.283+0000 7f508a000700 1 Processor -- start 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.283+0000 7f508a000700 1 -- start start 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.284+0000 7f508a000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50840fbcd0 0x7f5084065720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.284+0000 7f508a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f5084065c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.284+0000 7f508a000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508406ff60 con 0x7f50840fcf80 2024-03-23T11:47:07.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.284+0000 7f508a000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50840661a0 con 0x7f50840fbcd0 2024-03-23T11:47:07.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5082a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f5084065c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:07.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5083400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50840fbcd0 0x7f5084065720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:07.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5082a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f5084065c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43980/0 (socket says 172.21.15.63:43980) 2024-03-23T11:47:07.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5082a00700 1 -- 172.21.15.63:0/1291460382 learned_addr learned my addr 172.21.15.63:0/1291460382 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:07.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5083400700 1 -- 172.21.15.63:0/1291460382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 msgr2=0x7f5084065c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:07.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5083400700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f5084065c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5083400700 1 -- 172.21.15.63:0/1291460382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5070005040 con 0x7f50840fbcd0 2024-03-23T11:47:07.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.285+0000 7f5082a00700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f5084065c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:47:07.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.286+0000 7f5083400700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50840fbcd0 0x7f5084065720 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5074009fd0 tx=0x7f507400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:07.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.286+0000 7f507b400700 1 -- 172.21.15.63:0/1291460382 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5074007cc0 con 0x7f50840fbcd0 2024-03-23T11:47:07.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.286+0000 7f507b400700 1 -- 172.21.15.63:0/1291460382 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f507400f040 con 0x7f50840fbcd0 2024-03-23T11:47:07.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.286+0000 7f507b400700 1 -- 172.21.15.63:0/1291460382 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f507400ca60 con 0x7f50840fbcd0 2024-03-23T11:47:07.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.286+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5084066420 con 0x7f50840fbcd0 2024-03-23T11:47:07.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.286+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508406e580 con 0x7f50840fbcd0 2024-03-23T11:47:07.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.288+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508405f5f0 con 0x7f50840fbcd0 2024-03-23T11:47:07.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.288+0000 7f507b400700 1 -- 172.21.15.63:0/1291460382 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f507400cbc0 con 0x7f50840fbcd0 2024-03-23T11:47:07.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.290+0000 7f507b400700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f506806ac90 0x7f506806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:07.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.290+0000 7f507b400700 1 -- 172.21.15.63:0/1291460382 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f5074089080 con 0x7f50840fbcd0 2024-03-23T11:47:07.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.290+0000 7f5082a00700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f506806ac90 0x7f506806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:07.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.291+0000 7f5082a00700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f506806ac90 0x7f506806d140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5070013600 tx=0x7f5070009e30 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:07.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.295+0000 7f507b400700 1 -- 172.21.15.63:0/1291460382 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f507404f560 con 0x7f50840fbcd0 2024-03-23T11:47:07.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.639+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50841012e0 con 0x7f506806ac90 2024-03-23T11:47:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.641+0000 7f507b400700 1 -- 172.21.15.63:0/1291460382 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f50841012e0 con 0x7f506806ac90 2024-03-23T11:47:07.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f506806ac90 msgr2=0x7f506806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:07.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f506806ac90 0x7f506806d140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5070013600 tx=0x7f5070009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50840fbcd0 msgr2=0x7f5084065720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50840fbcd0 0x7f5084065720 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5074009fd0 tx=0x7f507400b040 comp rx=0 tx=0).stop 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 shutdown_connections 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f506806ac90 0x7f506806d140 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50840fbcd0 0x7f5084065720 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 --2- 172.21.15.63:0/1291460382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50840fcf80 0x7f5084065c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.644+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 >> 172.21.15.63:0/1291460382 conn(0x7f50840f7430 msgr2=0x7f50840f8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.645+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 shutdown_connections 2024-03-23T11:47:07.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.645+0000 7f508a000700 1 -- 172.21.15.63:0/1291460382 wait complete. 2024-03-23T11:47:07.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:07 smithi063 ceph-mon[35189]: pgmap v23: 65 pgs: 65 active+clean; 164 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 290 op/s 2024-03-23T11:47:07.649 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:47:07.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:07 smithi111 ceph-mon[44005]: pgmap v23: 65 pgs: 65 active+clean; 164 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 290 op/s 2024-03-23T11:47:07.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.840+0000 7f19d4a00700 1 -- 172.21.15.63:0/4132647771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 msgr2=0x7f19d00fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:07.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.840+0000 7f19d4a00700 1 --2- 172.21.15.63:0/4132647771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d00fe540 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f19bc004940 tx=0x7f19bc009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:07.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.840+0000 7f19d4a00700 1 -- 172.21.15.63:0/4132647771 shutdown_connections 2024-03-23T11:47:07.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.840+0000 7f19d4a00700 1 --2- 172.21.15.63:0/4132647771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d00fe540 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.840+0000 7f19d4a00700 1 --2- 172.21.15.63:0/4132647771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d00fcee0 0x7f19d00fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.840+0000 7f19d4a00700 1 -- 172.21.15.63:0/4132647771 >> 172.21.15.63:0/4132647771 conn(0x7f19d00f8560 msgr2=0x7f19d00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:07.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.842+0000 7f19d4a00700 1 -- 172.21.15.63:0/4132647771 shutdown_connections 2024-03-23T11:47:07.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.843+0000 7f19d4a00700 1 -- 172.21.15.63:0/4132647771 wait complete. 2024-03-23T11:47:07.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.844+0000 7f19d4a00700 1 Processor -- start 2024-03-23T11:47:07.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.844+0000 7f19d4a00700 1 -- start start 2024-03-23T11:47:07.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.844+0000 7f19d4a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d00fcee0 0x7f19d0106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:07.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.844+0000 7f19d4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d0107510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:07.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.844+0000 7f19d4a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0107b60 con 0x7f19d00fe0d0 2024-03-23T11:47:07.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.844+0000 7f19d4a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d01083b0 con 0x7f19d00fcee0 2024-03-23T11:47:07.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.845+0000 7f19cd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d0107510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:07.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.845+0000 7f19cd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d0107510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44002/0 (socket says 172.21.15.63:44002) 2024-03-23T11:47:07.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.845+0000 7f19cd400700 1 -- 172.21.15.63:0/1031218045 learned_addr learned my addr 172.21.15.63:0/1031218045 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:07.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.845+0000 7f19cde00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d00fcee0 0x7f19d0106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.846+0000 7f19cd400700 1 -- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d00fcee0 msgr2=0x7f19d0106fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.846+0000 7f19cd400700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d00fcee0 0x7f19d0106fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.846+0000 7f19cd400700 1 -- 172.21.15.63:0/1031218045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc005040 con 0x7f19d00fe0d0 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.846+0000 7f19cde00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d00fcee0 0x7f19d0106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.846+0000 7f19cd400700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d0107510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f19bc013870 tx=0x7f19bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.847+0000 7f19c6000700 1 -- 172.21.15.63:0/1031218045 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19bc021070 con 0x7f19d00fe0d0 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.847+0000 7f19c6000700 1 -- 172.21.15.63:0/1031218045 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19bc013dc0 con 0x7f19d00fe0d0 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.847+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d01085b0 con 0x7f19d00fe0d0 2024-03-23T11:47:07.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.847+0000 7f19c6000700 1 -- 172.21.15.63:0/1031218045 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19bc00e070 con 0x7f19d00fe0d0 2024-03-23T11:47:07.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.847+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d0108ae0 con 0x7f19d00fe0d0 2024-03-23T11:47:07.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.849+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d005f5f0 con 0x7f19d00fe0d0 2024-03-23T11:47:07.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.850+0000 7f19c6000700 1 -- 172.21.15.63:0/1031218045 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f19bc01c600 con 0x7f19d00fe0d0 2024-03-23T11:47:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.851+0000 7f19c6000700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f19b406ad50 0x7f19b406d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.851+0000 7f19c6000700 1 -- 172.21.15.63:0/1031218045 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f19bc08a730 con 0x7f19d00fe0d0 2024-03-23T11:47:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.851+0000 7f19cde00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f19b406ad50 0x7f19b406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.852+0000 7f19cde00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f19b406ad50 0x7f19b406d200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f19d0106d60 tx=0x7f19c000b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:07.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:07.862+0000 7f19c6000700 1 -- 172.21.15.63:0/1031218045 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f19bc054dc0 con 0x7f19d00fe0d0 2024-03-23T11:47:08.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.194+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19d0108ec0 con 0x7f19b406ad50 2024-03-23T11:47:08.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.195+0000 7f19c6000700 1 -- 172.21.15.63:0/1031218045 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f19d0108ec0 con 0x7f19b406ad50 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.198+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f19b406ad50 msgr2=0x7f19b406d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.198+0000 7f19d4a00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f19b406ad50 0x7f19b406d200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f19d0106d60 tx=0x7f19c000b040 comp rx=0 tx=0).stop 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.198+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 msgr2=0x7f19d0107510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.198+0000 7f19d4a00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d0107510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f19bc013870 tx=0x7f19bc009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.199+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 shutdown_connections 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.199+0000 7f19d4a00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f19b406ad50 0x7f19b406d200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.199+0000 7f19d4a00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d00fcee0 0x7f19d0106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.199+0000 7f19d4a00700 1 --2- 172.21.15.63:0/1031218045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19d00fe0d0 0x7f19d0107510 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.199+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 >> 172.21.15.63:0/1031218045 conn(0x7f19d00f8560 msgr2=0x7f19d00f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:08.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.199+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 shutdown_connections 2024-03-23T11:47:08.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.199+0000 7f19d4a00700 1 -- 172.21.15.63:0/1031218045 wait complete. 2024-03-23T11:47:08.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 -- 172.21.15.63:0/2748394793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 msgr2=0x7fa1a8006b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:08.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/2748394793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 0x7fa1a8006b60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0004940 tx=0x7fa1a0009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:08.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 -- 172.21.15.63:0/2748394793 shutdown_connections 2024-03-23T11:47:08.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/2748394793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 0x7fa1a8007510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/2748394793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 0x7fa1a8006b60 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 -- 172.21.15.63:0/2748394793 >> 172.21.15.63:0/2748394793 conn(0x7fa1a8090360 msgr2=0x7fa1a8092770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:08.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 -- 172.21.15.63:0/2748394793 shutdown_connections 2024-03-23T11:47:08.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.394+0000 7fa1b4a00700 1 -- 172.21.15.63:0/2748394793 wait complete. 2024-03-23T11:47:08.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.395+0000 7fa1b4a00700 1 Processor -- start 2024-03-23T11:47:08.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1b4a00700 1 -- start start 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1b4a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 0x7fa1a80abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1b4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 0x7fa1a80ac140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1b4a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80ac740 con 0x7fa1a8008650 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1b4a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80ac8b0 con 0x7fa1a80070a0 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1ae800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 0x7fa1a80abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1ae800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 0x7fa1a80abc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:36552/0 (socket says 172.21.15.63:36552) 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.396+0000 7fa1ae800700 1 -- 172.21.15.63:0/1706561016 learned_addr learned my addr 172.21.15.63:0/1706561016 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:08.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.397+0000 7fa1ae800700 1 -- 172.21.15.63:0/1706561016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 msgr2=0x7fa1a80ac140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:47:08.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.398+0000 7fa1ade00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 0x7fa1a80ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:08.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.398+0000 7fa1ae800700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 0x7fa1a80ac140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.398+0000 7fa1ae800700 1 -- 172.21.15.63:0/1706561016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0005040 con 0x7fa1a80070a0 2024-03-23T11:47:08.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.398+0000 7fa1ade00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 0x7fa1a80ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:47:08.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.399+0000 7fa1ae800700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 0x7fa1a80abc00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0000c00 tx=0x7fa1a0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:08.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.399+0000 7fa1a6a00700 1 -- 172.21.15.63:0/1706561016 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1a000e070 con 0x7fa1a80070a0 2024-03-23T11:47:08.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.399+0000 7fa1a6a00700 1 -- 172.21.15.63:0/1706561016 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1a0013d00 con 0x7fa1a80070a0 2024-03-23T11:47:08.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.399+0000 7fa1a6a00700 1 -- 172.21.15.63:0/1706561016 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1a001cdf0 con 0x7fa1a80070a0 2024-03-23T11:47:08.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.399+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a80b12b0 con 0x7fa1a80070a0 2024-03-23T11:47:08.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.399+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a80b1810 con 0x7fa1a80070a0 2024-03-23T11:47:08.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.401+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a8004470 con 0x7fa1a80070a0 2024-03-23T11:47:08.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.403+0000 7fa1a6a00700 1 -- 172.21.15.63:0/1706561016 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa1a001c600 con 0x7fa1a80070a0 2024-03-23T11:47:08.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.405+0000 7fa1a6a00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa19406ad60 0x7fa19406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:08.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.405+0000 7fa1a6a00700 1 -- 172.21.15.63:0/1706561016 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fa1a008ac00 con 0x7fa1a80070a0 2024-03-23T11:47:08.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.406+0000 7fa1ade00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa19406ad60 0x7fa19406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:08.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.407+0000 7fa1ade00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa19406ad60 0x7fa19406d210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa19c004600 tx=0x7fa19c00b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:08.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.410+0000 7fa1a6a00700 1 -- 172.21.15.63:0/1706561016 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa1a0058d20 con 0x7fa1a80070a0 2024-03-23T11:47:08.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:08 smithi063 ceph-mon[35189]: from='client.24441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:08.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:08 smithi111 ceph-mon[44005]: from='client.24441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (7m) 32s ago 9m 13.5M - 0.25.0 c8568f914cd2 c31e5bfcace0 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (9m) 32s ago 9m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (8m) 29s ago 8m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (7m) 32s ago 9m 74.1M - 9.4.7 954c08fa6188 d05e188def00 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (3m) 32s ago 3m 70.5M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (3m) 32s ago 3m 42.0M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (3m) 29s ago 3m 41.1M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (3m) 29s ago 3m 31.6M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:47:08.847 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:9283 running (11m) 32s ago 11m 156M - 17.2.7-726-g9c4e0c34 837124da804f 63833d132315 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (55s) 29s ago 8m 478M - 18.2.2-820-gb630d6cd 881b71eac2fe 278e3d19ec5d 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (11m) 32s ago 11m 48.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (8m) 29s ago 8m 37.0M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 starting - - - - 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (8m) 29s ago 8m 7889k - 1.5.0 0da6a335fe13 68f97d563d92 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (7m) 32s ago 7m 73.2M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (6m) 32s ago 6m 78.9M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (6m) 32s ago 6m 67.7M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (5m) 29s ago 5m 78.8M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (5m) 29s ago 5m 74.8M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (4m) 29s ago 4m 71.7M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (7m) 32s ago 9m 47.5M - 2.43.0 a07b618ecd1d 328fbe2c4719 2024-03-23T11:47:08.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.778+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1a80976d0 con 0x7fa19406ad60 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.789+0000 7fa1a6a00700 1 -- 172.21.15.63:0/1706561016 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fa1a80976d0 con 0x7fa19406ad60 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa19406ad60 msgr2=0x7fa19406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa19406ad60 0x7fa19406d210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa19c004600 tx=0x7fa19c00b040 comp rx=0 tx=0).stop 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 msgr2=0x7fa1a80abc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 0x7fa1a80abc00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0000c00 tx=0x7fa1a0009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 shutdown_connections 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa19406ad60 0x7fa19406d210 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a80070a0 0x7fa1a80abc00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 --2- 172.21.15.63:0/1706561016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1a8008650 0x7fa1a80ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 >> 172.21.15.63:0/1706561016 conn(0x7fa1a8090360 msgr2=0x7fa1a8091dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.792+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 shutdown_connections 2024-03-23T11:47:08.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.793+0000 7fa1b4a00700 1 -- 172.21.15.63:0/1706561016 wait complete. 2024-03-23T11:47:08.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.985+0000 7fee22200700 1 -- 172.21.15.63:0/3907928256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c102300 msgr2=0x7fee1c1046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:08.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.985+0000 7fee22200700 1 --2- 172.21.15.63:0/3907928256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c102300 0x7fee1c1046e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fee04004940 tx=0x7fee04009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:08.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.986+0000 7fee22200700 1 -- 172.21.15.63:0/3907928256 shutdown_connections 2024-03-23T11:47:08.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.986+0000 7fee22200700 1 --2- 172.21.15.63:0/3907928256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c102300 0x7fee1c1046e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.986+0000 7fee22200700 1 --2- 172.21.15.63:0/3907928256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c0662e0 0x7fee1c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.986+0000 7fee22200700 1 -- 172.21.15.63:0/3907928256 >> 172.21.15.63:0/3907928256 conn(0x7fee1c0f7430 msgr2=0x7fee1c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:08.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.987+0000 7fee22200700 1 -- 172.21.15.63:0/3907928256 shutdown_connections 2024-03-23T11:47:08.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.987+0000 7fee22200700 1 -- 172.21.15.63:0/3907928256 wait complete. 2024-03-23T11:47:08.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.988+0000 7fee22200700 1 Processor -- start 2024-03-23T11:47:08.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.989+0000 7fee22200700 1 -- start start 2024-03-23T11:47:08.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.991+0000 7fee22200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c0662e0 0x7fee1c06f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:08.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.991+0000 7fee22200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c06de70 0x7fee1c06e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:08.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.991+0000 7fee22200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c06fee0 con 0x7fee1c0662e0 2024-03-23T11:47:08.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.991+0000 7fee22200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c070050 con 0x7fee1c06de70 2024-03-23T11:47:08.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.992+0000 7fee12a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c06de70 0x7fee1c06e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:08.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.992+0000 7fee12a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c06de70 0x7fee1c06e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:36564/0 (socket says 172.21.15.63:36564) 2024-03-23T11:47:08.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.992+0000 7fee12a00700 1 -- 172.21.15.63:0/1981462612 learned_addr learned my addr 172.21.15.63:0/1981462612 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:08.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.993+0000 7fee12a00700 1 -- 172.21.15.63:0/1981462612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c0662e0 msgr2=0x7fee1c06f950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:47:08.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.993+0000 7fee1b400700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c0662e0 0x7fee1c06f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:08.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.993+0000 7fee12a00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c0662e0 0x7fee1c06f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.993+0000 7fee12a00700 1 -- 172.21.15.63:0/1981462612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04005040 con 0x7fee1c06de70 2024-03-23T11:47:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.993+0000 7fee1b400700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c0662e0 0x7fee1c06f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:47:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.995+0000 7fee12a00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c06de70 0x7fee1c06e2e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee04004890 tx=0x7fee04009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.995+0000 7fee13e00700 1 -- 172.21.15.63:0/1981462612 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0400e030 con 0x7fee1c06de70 2024-03-23T11:47:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.996+0000 7fee13e00700 1 -- 172.21.15.63:0/1981462612 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee04013420 con 0x7fee1c06de70 2024-03-23T11:47:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.996+0000 7fee13e00700 1 -- 172.21.15.63:0/1981462612 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0401c640 con 0x7fee1c06de70 2024-03-23T11:47:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.996+0000 7fee22200700 1 -- 172.21.15.63:0/1981462612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c06e910 con 0x7fee1c06de70 2024-03-23T11:47:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.996+0000 7fee22200700 1 -- 172.21.15.63:0/1981462612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c1091c0 con 0x7fee1c06de70 2024-03-23T11:47:08.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:08.998+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf40052b0 con 0x7fee1c06de70 2024-03-23T11:47:08.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.000+0000 7fee13e00700 1 -- 172.21.15.63:0/1981462612 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fee04013900 con 0x7fee1c06de70 2024-03-23T11:47:08.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.003+0000 7fee13e00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fee0006ae70 0x7fee0006d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:08.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.003+0000 7fee13e00700 1 -- 172.21.15.63:0/1981462612 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fee0408ac00 con 0x7fee1c06de70 2024-03-23T11:47:08.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.003+0000 7fee1b400700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fee0006ae70 0x7fee0006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:08.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.005+0000 7fee1b400700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fee0006ae70 0x7fee0006d320 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0041d0 tx=0x7fee0c009040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:08.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.007+0000 7fee13e00700 1 -- 172.21.15.63:0/1981462612 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fee04055290 con 0x7fee1c06de70 2024-03-23T11:47:09.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.487+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fedf4005830 con 0x7fee1c06de70 2024-03-23T11:47:09.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.487+0000 7fee13e00700 1 -- 172.21.15.63:0/1981462612 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+848 (secure 0 0 0) 0x7fee040588b0 con 0x7fee1c06de70 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:47:09.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:09.474 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:47:09.474 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 13, 2024-03-23T11:47:09.474 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:47:09.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:47:09.474 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:47:09.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.490+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fee0006ae70 msgr2=0x7fee0006d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:09.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.490+0000 7fee10c00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fee0006ae70 0x7fee0006d320 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0041d0 tx=0x7fee0c009040 comp rx=0 tx=0).stop 2024-03-23T11:47:09.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.490+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c06de70 msgr2=0x7fee1c06e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:09.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.490+0000 7fee10c00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c06de70 0x7fee1c06e2e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee04004890 tx=0x7fee04009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:09.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.491+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 shutdown_connections 2024-03-23T11:47:09.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.491+0000 7fee10c00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee1c0662e0 0x7fee1c06f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:09.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.491+0000 7fee10c00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fee0006ae70 0x7fee0006d320 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0041d0 tx=0x7fee0c009040 comp rx=0 tx=0).stop 2024-03-23T11:47:09.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.491+0000 7fee10c00700 1 --2- 172.21.15.63:0/1981462612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee1c06de70 0x7fee1c06e2e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:09.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.491+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 >> 172.21.15.63:0/1981462612 conn(0x7fee1c0f7430 msgr2=0x7fee1c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:09.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.491+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 shutdown_connections 2024-03-23T11:47:09.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.491+0000 7fee10c00700 1 -- 172.21.15.63:0/1981462612 wait complete. 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.679+0000 7fb76d800700 1 -- 172.21.15.63:0/2697334084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 msgr2=0x7fb768070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.679+0000 7fb76d800700 1 --2- 172.21.15.63:0/2697334084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 0x7fb768070460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb758004940 tx=0x7fb758009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.680+0000 7fb76d800700 1 -- 172.21.15.63:0/2697334084 shutdown_connections 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.680+0000 7fb76d800700 1 --2- 172.21.15.63:0/2697334084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 0x7fb768070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.680+0000 7fb76d800700 1 --2- 172.21.15.63:0/2697334084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 0x7fb768070460 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.680+0000 7fb76d800700 1 -- 172.21.15.63:0/2697334084 >> 172.21.15.63:0/2697334084 conn(0x7fb7680f8560 msgr2=0x7fb7680fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.680+0000 7fb76d800700 1 -- 172.21.15.63:0/2697334084 shutdown_connections 2024-03-23T11:47:09.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.680+0000 7fb76d800700 1 -- 172.21.15.63:0/2697334084 wait complete. 2024-03-23T11:47:09.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.681+0000 7fb76d800700 1 Processor -- start 2024-03-23T11:47:09.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.682+0000 7fb76d800700 1 -- start start 2024-03-23T11:47:09.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.682+0000 7fb76d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 0x7fb768113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:09.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.682+0000 7fb76d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 0x7fb768114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:09.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.682+0000 7fb76d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb768114a10 con 0x7fb7680709a0 2024-03-23T11:47:09.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.682+0000 7fb76d800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb768115260 con 0x7fb768071f50 2024-03-23T11:47:09.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 0x7fb768113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:09.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 0x7fb768113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44066/0 (socket says 172.21.15.63:44066) 2024-03-23T11:47:09.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766a00700 1 -- 172.21.15.63:0/2416794972 learned_addr learned my addr 172.21.15.63:0/2416794972 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:09.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766000700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 0x7fb768114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:09.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766000700 1 -- 172.21.15.63:0/2416794972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 msgr2=0x7fb768113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:09.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766000700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 0x7fb768113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:09.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766000700 1 -- 172.21.15.63:0/2416794972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758005040 con 0x7fb768071f50 2024-03-23T11:47:09.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.683+0000 7fb766a00700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 0x7fb768113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:47:09.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.684+0000 7fb766000700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 0x7fb768114410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb754009fd0 tx=0x7fb75400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:09.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.684+0000 7fb75ea00700 1 -- 172.21.15.63:0/2416794972 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb754004e90 con 0x7fb768071f50 2024-03-23T11:47:09.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.684+0000 7fb75ea00700 1 -- 172.21.15.63:0/2416794972 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb75400f040 con 0x7fb768071f50 2024-03-23T11:47:09.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.684+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7681154e0 con 0x7fb768071f50 2024-03-23T11:47:09.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.684+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb768115a20 con 0x7fb768071f50 2024-03-23T11:47:09.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.685+0000 7fb75ea00700 1 -- 172.21.15.63:0/2416794972 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb75400c9b0 con 0x7fb768071f50 2024-03-23T11:47:09.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.686+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76805f5f0 con 0x7fb768071f50 2024-03-23T11:47:09.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.687+0000 7fb75ea00700 1 -- 172.21.15.63:0/2416794972 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb754007cc0 con 0x7fb768071f50 2024-03-23T11:47:09.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.688+0000 7fb75ea00700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb74c06ac90 0x7fb74c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:09.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.688+0000 7fb75ea00700 1 -- 172.21.15.63:0/2416794972 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fb754088fd0 con 0x7fb768071f50 2024-03-23T11:47:09.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.688+0000 7fb766a00700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb74c06ac90 0x7fb74c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:09.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.689+0000 7fb766a00700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb74c06ac90 0x7fb74c06d140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb758013680 tx=0x7fb758009e30 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:09.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:09.694+0000 7fb75ea00700 1 -- 172.21.15.63:0/2416794972 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb75404f560 con 0x7fb768071f50 2024-03-23T11:47:09.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: pgmap v24: 65 pgs: 65 active+clean; 178 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 417 op/s 2024-03-23T11:47:09.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:09.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:09.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:09.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: Reconfiguring alertmanager.smithi063 (dependencies changed)... 2024-03-23T11:47:09.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: Reconfiguring daemon alertmanager.smithi063 on smithi063 2024-03-23T11:47:09.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:09.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:09 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1981462612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:47:09.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: pgmap v24: 65 pgs: 65 active+clean; 178 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 417 op/s 2024-03-23T11:47:09.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:09.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:09.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:09.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: Reconfiguring alertmanager.smithi063 (dependencies changed)... 2024-03-23T11:47:09.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: Reconfiguring daemon alertmanager.smithi063 on smithi063 2024-03-23T11:47:09.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:09.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:09 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1981462612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:47:10.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.051+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb768115e10 con 0x7fb74c06ac90 2024-03-23T11:47:10.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.053+0000 7fb75ea00700 1 -- 172.21.15.63:0/2416794972 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fb768115e10 con 0x7fb74c06ac90 2024-03-23T11:47:10.038 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [], 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "1/2 daemons upgraded", 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "", 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:47:10.039 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:47:10.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb74c06ac90 msgr2=0x7fb74c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:10.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb74c06ac90 0x7fb74c06d140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb758013680 tx=0x7fb758009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:10.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 msgr2=0x7fb768114410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:10.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 0x7fb768114410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb754009fd0 tx=0x7fb75400b040 comp rx=0 tx=0).stop 2024-03-23T11:47:10.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 shutdown_connections 2024-03-23T11:47:10.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7680709a0 0x7fb768113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:10.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fb74c06ac90 0x7fb74c06d140 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:10.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 --2- 172.21.15.63:0/2416794972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb768071f50 0x7fb768114410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:10.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 >> 172.21.15.63:0/2416794972 conn(0x7fb7680f8560 msgr2=0x7fb7680f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:10.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 shutdown_connections 2024-03-23T11:47:10.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:10.057+0000 7fb76d800700 1 -- 172.21.15.63:0/2416794972 wait complete. 2024-03-23T11:47:11.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:11 smithi063 ceph-mon[35189]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:11.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:11 smithi063 ceph-mon[35189]: pgmap v25: 65 pgs: 65 active+clean; 178 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 269 op/s 2024-03-23T11:47:11.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:11 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:11.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:11 smithi111 ceph-mon[44005]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:11.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:11 smithi111 ceph-mon[44005]: pgmap v25: 65 pgs: 65 active+clean; 178 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 269 op/s 2024-03-23T11:47:11.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:11 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:13.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:13 smithi063 ceph-mon[35189]: pgmap v26: 65 pgs: 65 active+clean; 197 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 393 op/s 2024-03-23T11:47:13.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:13 smithi111 ceph-mon[44005]: pgmap v26: 65 pgs: 65 active+clean; 197 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 393 op/s 2024-03-23T11:47:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:15 smithi063 ceph-mon[35189]: pgmap v27: 65 pgs: 65 active+clean; 198 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 277 op/s 2024-03-23T11:47:15.850 INFO:tasks.workunit.client.0.smithi063.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-23T11:47:15.876 INFO:tasks.workunit.client.0.smithi063.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-23T11:47:15.877 INFO:tasks.workunit.client.0.smithi063.stderr:+ make 2024-03-23T11:47:15.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:15 smithi111 ceph-mon[44005]: pgmap v27: 65 pgs: 65 active+clean; 198 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 277 op/s 2024-03-23T11:47:16.126 INFO:tasks.workunit.client.0.smithi063.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-23T11:47:17.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:17 smithi111 ceph-mon[44005]: pgmap v28: 65 pgs: 65 active+clean; 198 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 263 op/s 2024-03-23T11:47:17.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:17 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:47:17.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:17 smithi063 ceph-mon[35189]: pgmap v28: 65 pgs: 65 active+clean; 198 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 263 op/s 2024-03-23T11:47:17.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:17 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:47:18.598 INFO:tasks.workunit.client.1.smithi111.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-23T11:47:18.618 INFO:tasks.workunit.client.1.smithi111.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-23T11:47:18.618 INFO:tasks.workunit.client.1.smithi111.stderr:+ make 2024-03-23T11:47:18.837 INFO:tasks.workunit.client.1.smithi111.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-23T11:47:18.998 INFO:tasks.workunit.client.0.smithi063.stderr:++ readlink -f fsstress 2024-03-23T11:47:19.002 INFO:tasks.workunit.client.0.smithi063.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-23T11:47:19.002 INFO:tasks.workunit.client.0.smithi063.stderr:+ popd 2024-03-23T11:47:19.004 INFO:tasks.workunit.client.0.smithi063.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-23T11:47:19.004 INFO:tasks.workunit.client.0.smithi063.stderr:+ popd 2024-03-23T11:47:19.005 INFO:tasks.workunit.client.0.smithi063.stdout:~/cephtest/mnt.0/client.0/tmp 2024-03-23T11:47:19.006 INFO:tasks.workunit.client.0.smithi063.stderr:++ mktemp -d -p . 2024-03-23T11:47:19.010 INFO:tasks.workunit.client.0.smithi063.stderr:+ T=./tmp.3LUi8G1z5V 2024-03-23T11:47:19.010 INFO:tasks.workunit.client.0.smithi063.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.3LUi8G1z5V -l 1 -n 1000 -p 10 -v 2024-03-23T11:47:19.015 INFO:tasks.workunit.client.0.smithi063.stdout:seed = 1711193531 2024-03-23T11:47:19.021 INFO:tasks.workunit.client.0.smithi063.stdout:0/0: fdatasync - no filename 2024-03-23T11:47:19.022 INFO:tasks.workunit.client.0.smithi063.stdout:0/1: mknod c0 0 2024-03-23T11:47:19.023 INFO:tasks.workunit.client.0.smithi063.stdout:0/2: write - no filename 2024-03-23T11:47:19.023 INFO:tasks.workunit.client.0.smithi063.stdout:1/0: unlink - no file 2024-03-23T11:47:19.023 INFO:tasks.workunit.client.0.smithi063.stdout:1/1: dwrite - no filename 2024-03-23T11:47:19.024 INFO:tasks.workunit.client.0.smithi063.stdout:2/0: truncate - no filename 2024-03-23T11:47:19.024 INFO:tasks.workunit.client.0.smithi063.stdout:2/1: stat - no entries 2024-03-23T11:47:19.024 INFO:tasks.workunit.client.0.smithi063.stdout:2/2: fsync - no filename 2024-03-23T11:47:19.024 INFO:tasks.workunit.client.0.smithi063.stdout:2/3: dread - no filename 2024-03-23T11:47:19.024 INFO:tasks.workunit.client.0.smithi063.stdout:2/4: write - no filename 2024-03-23T11:47:19.024 INFO:tasks.workunit.client.0.smithi063.stdout:2/5: dwrite - no filename 2024-03-23T11:47:19.025 INFO:tasks.workunit.client.0.smithi063.stdout:0/3: unlink c0 0 2024-03-23T11:47:19.026 INFO:tasks.workunit.client.0.smithi063.stdout:1/2: mknod c0 0 2024-03-23T11:47:19.028 INFO:tasks.workunit.client.0.smithi063.stdout:1/3: symlink l1 0 2024-03-23T11:47:19.028 INFO:tasks.workunit.client.0.smithi063.stdout:3/0: chown . 1943141146 1 2024-03-23T11:47:19.028 INFO:tasks.workunit.client.0.smithi063.stdout:3/1: dwrite - no filename 2024-03-23T11:47:19.028 INFO:tasks.workunit.client.0.smithi063.stdout:3/2: chown . 54572 1 2024-03-23T11:47:19.029 INFO:tasks.workunit.client.0.smithi063.stdout:2/6: creat f0 x:0 0 0 2024-03-23T11:47:19.031 INFO:tasks.workunit.client.0.smithi063.stdout:1/4: creat f2 x:0 0 0 2024-03-23T11:47:19.033 INFO:tasks.workunit.client.0.smithi063.stdout:3/3: creat f0 x:0 0 0 2024-03-23T11:47:19.033 INFO:tasks.workunit.client.0.smithi063.stdout:2/7: truncate f0 651099 0 2024-03-23T11:47:19.034 INFO:tasks.workunit.client.0.smithi063.stdout:3/4: chown f0 102140296 1 2024-03-23T11:47:19.035 INFO:tasks.workunit.client.0.smithi063.stdout:1/5: mkdir d3 0 2024-03-23T11:47:19.036 INFO:tasks.workunit.client.0.smithi063.stdout:5/0: fdatasync - no filename 2024-03-23T11:47:19.039 INFO:tasks.workunit.client.0.smithi063.stdout:1/6: write f2 [378791,123420] 0 2024-03-23T11:47:19.040 INFO:tasks.workunit.client.0.smithi063.stdout:3/5: rename f0 to f1 0 2024-03-23T11:47:19.040 INFO:tasks.workunit.client.0.smithi063.stdout:3/6: rmdir - no directory 2024-03-23T11:47:19.040 INFO:tasks.workunit.client.0.smithi063.stdout:2/8: rename f0 to f1 0 2024-03-23T11:47:19.044 INFO:tasks.workunit.client.0.smithi063.stdout:4/0: symlink l0 0 2024-03-23T11:47:19.044 INFO:tasks.workunit.client.0.smithi063.stdout:6/0: chown . 120417 1 2024-03-23T11:47:19.044 INFO:tasks.workunit.client.0.smithi063.stdout:6/1: write - no filename 2024-03-23T11:47:19.044 INFO:tasks.workunit.client.0.smithi063.stdout:6/2: truncate - no filename 2024-03-23T11:47:19.044 INFO:tasks.workunit.client.0.smithi063.stdout:2/9: read f1 [527934,121472] 0 2024-03-23T11:47:19.047 INFO:tasks.workunit.client.0.smithi063.stdout:3/7: creat f2 x:0 0 0 2024-03-23T11:47:19.048 INFO:tasks.workunit.client.0.smithi063.stdout:3/8: chown f1 1149 1 2024-03-23T11:47:19.049 INFO:tasks.workunit.client.0.smithi063.stdout:0/4: getdents . 0 2024-03-23T11:47:19.049 INFO:tasks.workunit.client.0.smithi063.stdout:7/0: write - no filename 2024-03-23T11:47:19.050 INFO:tasks.workunit.client.0.smithi063.stdout:3/9: truncate f2 947320 0 2024-03-23T11:47:19.052 INFO:tasks.workunit.client.0.smithi063.stdout:4/1: symlink l1 0 2024-03-23T11:47:19.052 INFO:tasks.workunit.client.0.smithi063.stdout:4/2: truncate - no filename 2024-03-23T11:47:19.052 INFO:tasks.workunit.client.0.smithi063.stdout:4/3: write - no filename 2024-03-23T11:47:19.055 INFO:tasks.workunit.client.0.smithi063.stdout:1/7: creat d3/f4 x:0 0 0 2024-03-23T11:47:19.060 INFO:tasks.workunit.client.0.smithi063.stdout:0/5: creat f1 x:0 0 0 2024-03-23T11:47:19.061 INFO:tasks.workunit.client.0.smithi063.stdout:6/3: creat f0 x:0 0 0 2024-03-23T11:47:19.063 INFO:tasks.workunit.client.0.smithi063.stdout:4/4: creat f2 x:0 0 0 2024-03-23T11:47:19.063 INFO:tasks.workunit.client.0.smithi063.stdout:1/8: mknod d3/c5 0 2024-03-23T11:47:19.065 INFO:tasks.workunit.client.0.smithi063.stdout:6/4: dread - f0 zero size 2024-03-23T11:47:19.066 INFO:tasks.workunit.client.0.smithi063.stdout:7/1: symlink l0 0 2024-03-23T11:47:19.067 INFO:tasks.workunit.client.0.smithi063.stdout:4/5: dread - f2 zero size 2024-03-23T11:47:19.070 INFO:tasks.workunit.client.0.smithi063.stdout:0/6: mkdir d2 0 2024-03-23T11:47:19.074 INFO:tasks.workunit.client.0.smithi063.stdout:8/0: creat f0 x:0 0 0 2024-03-23T11:47:19.076 INFO:tasks.workunit.client.0.smithi063.stdout:7/2: creat f1 x:0 0 0 2024-03-23T11:47:19.082 INFO:tasks.workunit.client.0.smithi063.stdout:8/1: rename f0 to f1 0 2024-03-23T11:47:19.085 INFO:tasks.workunit.client.0.smithi063.stdout:7/3: creat f2 x:0 0 0 2024-03-23T11:47:19.087 INFO:tasks.workunit.client.0.smithi063.stdout:4/6: rename l0 to l3 0 2024-03-23T11:47:19.088 INFO:tasks.workunit.client.0.smithi063.stdout:8/2: creat f2 x:0 0 0 2024-03-23T11:47:19.101 INFO:tasks.workunit.client.0.smithi063.stdout:7/4: write f2 [298742,4607] 0 2024-03-23T11:47:19.105 INFO:tasks.workunit.client.0.smithi063.stdout:8/3: truncate f1 1035657 0 2024-03-23T11:47:19.107 INFO:tasks.workunit.client.0.smithi063.stdout:4/7: truncate f2 87479 0 2024-03-23T11:47:19.113 INFO:tasks.workunit.client.0.smithi063.stdout:2/10: dwrite f1 [0,4194304] 0 2024-03-23T11:47:19.122 INFO:tasks.workunit.client.0.smithi063.stdout:7/5: chown l0 28993 1 2024-03-23T11:47:19.135 INFO:tasks.workunit.client.0.smithi063.stdout:4/8: symlink l4 0 2024-03-23T11:47:19.140 INFO:tasks.workunit.client.0.smithi063.stdout:2/11: mkdir d2 0 2024-03-23T11:47:19.140 INFO:tasks.workunit.client.0.smithi063.stdout:8/4: write f2 [453437,34160] 0 2024-03-23T11:47:19.140 INFO:tasks.workunit.client.0.smithi063.stdout:3/10: dwrite f1 [0,4194304] 0 2024-03-23T11:47:19.142 INFO:tasks.workunit.client.0.smithi063.stdout:2/12: chown d2 232 1 2024-03-23T11:47:19.145 INFO:tasks.workunit.client.0.smithi063.stdout:7/6: symlink l3 0 2024-03-23T11:47:19.147 INFO:tasks.workunit.client.0.smithi063.stdout:4/9: mkdir d5 0 2024-03-23T11:47:19.151 INFO:tasks.workunit.client.0.smithi063.stdout:1/9: dwrite d3/f4 [0,4194304] 0 2024-03-23T11:47:19.154 INFO:tasks.workunit.client.0.smithi063.stdout:2/13: chown d2 1454110795 1 2024-03-23T11:47:19.154 INFO:tasks.workunit.client.0.smithi063.stdout:6/5: dwrite f0 [0,4194304] 0 2024-03-23T11:47:19.155 INFO:tasks.workunit.client.0.smithi063.stdout:0/7: dwrite f1 [0,4194304] 0 2024-03-23T11:47:19.156 INFO:tasks.workunit.client.0.smithi063.stdout:8/5: write f2 [309595,13941] 0 2024-03-23T11:47:19.159 INFO:tasks.workunit.client.0.smithi063.stdout:1/10: truncate f2 572073 0 2024-03-23T11:47:19.160 INFO:tasks.workunit.client.0.smithi063.stdout:4/10: stat l3 0 2024-03-23T11:47:19.162 INFO:tasks.workunit.client.0.smithi063.stdout:7/7: truncate f1 543782 0 2024-03-23T11:47:19.166 INFO:tasks.workunit.client.0.smithi063.stdout:8/6: read f1 [762811,108431] 0 2024-03-23T11:47:19.169 INFO:tasks.workunit.client.0.smithi063.stdout:7/8: read f1 [210503,126970] 0 2024-03-23T11:47:19.169 INFO:tasks.workunit.client.0.smithi063.stdout:8/7: fdatasync f2 0 2024-03-23T11:47:19.171 INFO:tasks.workunit.client.0.smithi063.stdout:1/11: mknod d3/c6 0 2024-03-23T11:47:19.178 INFO:tasks.workunit.client.0.smithi063.stdout:3/11: link f2 f3 0 2024-03-23T11:47:19.187 INFO:tasks.workunit.client.0.smithi063.stdout:2/14: symlink d2/l3 0 2024-03-23T11:47:19.187 INFO:tasks.workunit.client.0.smithi063.stdout:1/12: dread f2 [0,4194304] 0 2024-03-23T11:47:19.188 INFO:tasks.workunit.client.0.smithi063.stdout:8/8: symlink l3 0 2024-03-23T11:47:19.189 INFO:tasks.workunit.client.0.smithi063.stdout:3/12: unlink f1 0 2024-03-23T11:47:19.194 INFO:tasks.workunit.client.0.smithi063.stdout:2/15: creat d2/f4 x:0 0 0 2024-03-23T11:47:19.194 INFO:tasks.workunit.client.0.smithi063.stdout:8/9: rename l3 to l4 0 2024-03-23T11:47:19.196 INFO:tasks.workunit.client.0.smithi063.stdout:6/6: dread f0 [0,4194304] 0 2024-03-23T11:47:19.197 INFO:tasks.workunit.client.0.smithi063.stdout:3/13: creat f4 x:0 0 0 2024-03-23T11:47:19.198 INFO:tasks.workunit.client.0.smithi063.stdout:4/11: link l3 d5/l6 0 2024-03-23T11:47:19.199 INFO:tasks.workunit.client.0.smithi063.stdout:2/16: truncate f1 4767355 0 2024-03-23T11:47:19.201 INFO:tasks.workunit.client.0.smithi063.stdout:2/17: dread - d2/f4 zero size 2024-03-23T11:47:19.201 INFO:tasks.workunit.client.0.smithi063.stdout:4/12: write f2 [1062802,52531] 0 2024-03-23T11:47:19.208 INFO:tasks.workunit.client.0.smithi063.stdout:4/13: unlink l4 0 2024-03-23T11:47:19.208 INFO:tasks.workunit.client.0.smithi063.stdout:6/7: write f0 [5044068,128812] 0 2024-03-23T11:47:19.211 INFO:tasks.workunit.client.0.smithi063.stdout:3/14: fsync f2 0 2024-03-23T11:47:19.215 INFO:tasks.workunit.client.0.smithi063.stdout:7/9: dwrite f2 [0,4194304] 0 2024-03-23T11:47:19.218 INFO:tasks.workunit.client.0.smithi063.stdout:6/8: mkdir d1 0 2024-03-23T11:47:19.231 INFO:tasks.workunit.client.0.smithi063.stdout:4/14: symlink d5/l7 0 2024-03-23T11:47:19.234 INFO:tasks.workunit.client.0.smithi063.stdout:6/9: unlink f0 0 2024-03-23T11:47:19.234 INFO:tasks.workunit.client.0.smithi063.stdout:6/10: write - no filename 2024-03-23T11:47:19.235 INFO:tasks.workunit.client.0.smithi063.stdout:6/11: dread - no filename 2024-03-23T11:47:19.235 INFO:tasks.workunit.client.0.smithi063.stdout:6/12: write - no filename 2024-03-23T11:47:19.235 INFO:tasks.workunit.client.0.smithi063.stdout:6/13: link - no file 2024-03-23T11:47:19.235 INFO:tasks.workunit.client.0.smithi063.stdout:6/14: dwrite - no filename 2024-03-23T11:47:19.235 INFO:tasks.workunit.client.0.smithi063.stdout:6/15: fdatasync - no filename 2024-03-23T11:47:19.235 INFO:tasks.workunit.client.0.smithi063.stdout:3/15: dread f2 [0,4194304] 0 2024-03-23T11:47:19.235 INFO:tasks.workunit.client.0.smithi063.stdout:7/10: write f1 [576548,96388] 0 2024-03-23T11:47:19.238 INFO:tasks.workunit.client.0.smithi063.stdout:4/15: truncate f2 1960721 0 2024-03-23T11:47:19.242 INFO:tasks.workunit.client.0.smithi063.stdout:1/13: dwrite f2 [0,4194304] 0 2024-03-23T11:47:19.244 INFO:tasks.workunit.client.0.smithi063.stdout:6/16: symlink d1/l2 0 2024-03-23T11:47:19.244 INFO:tasks.workunit.client.0.smithi063.stdout:6/17: write - no filename 2024-03-23T11:47:19.245 INFO:tasks.workunit.client.0.smithi063.stdout:1/14: readlink l1 0 2024-03-23T11:47:19.246 INFO:tasks.workunit.client.0.smithi063.stdout:8/10: dwrite f2 [0,4194304] 0 2024-03-23T11:47:19.247 INFO:tasks.workunit.client.0.smithi063.stdout:4/16: chown d5/l7 270363665 1 2024-03-23T11:47:19.249 INFO:tasks.workunit.client.0.smithi063.stdout:4/17: chown l1 97780375 1 2024-03-23T11:47:19.250 INFO:tasks.workunit.client.0.smithi063.stdout:4/18: stat f2 0 2024-03-23T11:47:19.251 INFO:tasks.workunit.client.0.smithi063.stdout:6/18: creat d1/f3 x:0 0 0 2024-03-23T11:47:19.251 INFO:tasks.workunit.client.0.smithi063.stdout:3/16: link f2 f5 0 2024-03-23T11:47:19.251 INFO:tasks.workunit.client.0.smithi063.stdout:7/11: getdents . 0 2024-03-23T11:47:19.256 INFO:tasks.workunit.client.0.smithi063.stdout:4/19: write f2 [2846146,7729] 0 2024-03-23T11:47:19.284 INFO:tasks.workunit.client.0.smithi063.stdout:7/12: creat f4 x:0 0 0 2024-03-23T11:47:19.291 INFO:tasks.workunit.client.0.smithi063.stdout:7/13: truncate f4 706007 0 2024-03-23T11:47:19.292 INFO:tasks.workunit.client.0.smithi063.stdout:8/11: dread f2 [0,4194304] 0 2024-03-23T11:47:19.295 INFO:tasks.workunit.client.0.smithi063.stdout:7/14: chown l3 1974354 1 2024-03-23T11:47:19.301 INFO:tasks.workunit.client.0.smithi063.stdout:7/15: readlink l0 0 2024-03-23T11:47:19.304 INFO:tasks.workunit.client.0.smithi063.stdout:4/20: dwrite f2 [0,4194304] 0 2024-03-23T11:47:19.305 INFO:tasks.workunit.client.0.smithi063.stdout:7/16: fsync f2 0 2024-03-23T11:47:19.306 INFO:tasks.workunit.client.0.smithi063.stdout:1/15: dwrite d3/f4 [0,4194304] 0 2024-03-23T11:47:19.310 INFO:tasks.workunit.client.0.smithi063.stdout:6/19: dwrite d1/f3 [0,4194304] 0 2024-03-23T11:47:19.315 INFO:tasks.workunit.client.0.smithi063.stdout:7/17: write f1 [1081583,85287] 0 2024-03-23T11:47:19.338 INFO:tasks.workunit.client.0.smithi063.stdout:7/18: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:19.341 INFO:tasks.workunit.client.0.smithi063.stdout:4/21: fdatasync f2 0 2024-03-23T11:47:19.341 INFO:tasks.workunit.client.0.smithi063.stdout:7/19: write f1 [1610900,4781] 0 2024-03-23T11:47:19.341 INFO:tasks.workunit.client.0.smithi063.stdout:7/20: rmdir - no directory 2024-03-23T11:47:19.386 INFO:tasks.workunit.client.0.smithi063.stdout:3/17: fdatasync f2 0 2024-03-23T11:47:19.387 INFO:tasks.workunit.client.0.smithi063.stdout:3/18: chown f4 2 1 2024-03-23T11:47:19.389 INFO:tasks.workunit.client.0.smithi063.stdout:1/16: creat d3/f7 x:0 0 0 2024-03-23T11:47:19.389 INFO:tasks.workunit.client.0.smithi063.stdout:6/20: creat d1/f4 x:0 0 0 2024-03-23T11:47:19.392 INFO:tasks.workunit.client.0.smithi063.stdout:6/21: write d1/f3 [1646149,61264] 0 2024-03-23T11:47:19.394 INFO:tasks.workunit.client.0.smithi063.stdout:4/22: readlink d5/l6 0 2024-03-23T11:47:19.397 INFO:tasks.workunit.client.0.smithi063.stdout:7/21: mkdir d5 0 2024-03-23T11:47:19.400 INFO:tasks.workunit.client.0.smithi063.stdout:3/19: rename f4 to f6 0 2024-03-23T11:47:19.401 INFO:tasks.workunit.client.0.smithi063.stdout:1/17: mkdir d3/d8 0 2024-03-23T11:47:19.403 INFO:tasks.workunit.client.0.smithi063.stdout:1/18: read d3/f4 [3401232,20887] 0 2024-03-23T11:47:19.406 INFO:tasks.workunit.client.0.smithi063.stdout:8/12: dwrite f2 [0,4194304] 0 2024-03-23T11:47:19.408 INFO:tasks.workunit.client.0.smithi063.stdout:4/23: rename l1 to d5/l8 0 2024-03-23T11:47:19.410 INFO:tasks.workunit.client.0.smithi063.stdout:3/20: write f6 [375762,36325] 0 2024-03-23T11:47:19.413 INFO:tasks.workunit.client.0.smithi063.stdout:8/13: creat f5 x:0 0 0 2024-03-23T11:47:19.414 INFO:tasks.workunit.client.0.smithi063.stdout:7/22: mkdir d5/d6 0 2024-03-23T11:47:19.415 INFO:tasks.workunit.client.0.smithi063.stdout:7/23: write f1 [1708527,38395] 0 2024-03-23T11:47:19.416 INFO:tasks.workunit.client.0.smithi063.stdout:4/24: mkdir d5/d9 0 2024-03-23T11:47:19.419 INFO:tasks.workunit.client.0.smithi063.stdout:3/21: creat f7 x:0 0 0 2024-03-23T11:47:19.422 INFO:tasks.workunit.client.0.smithi063.stdout:8/14: stat l4 0 2024-03-23T11:47:19.426 INFO:tasks.workunit.client.0.smithi063.stdout:4/25: mkdir d5/da 0 2024-03-23T11:47:19.428 INFO:tasks.workunit.client.0.smithi063.stdout:8/15: dread f1 [0,4194304] 0 2024-03-23T11:47:19.430 INFO:tasks.workunit.client.0.smithi063.stdout:4/26: write f2 [2588387,75994] 0 2024-03-23T11:47:19.430 INFO:tasks.workunit.client.0.smithi063.stdout:1/19: rmdir d3/d8 0 2024-03-23T11:47:19.432 INFO:tasks.workunit.client.0.smithi063.stdout:7/24: mkdir d5/d6/d7 0 2024-03-23T11:47:19.433 INFO:tasks.workunit.client.0.smithi063.stdout:8/16: mkdir d6 0 2024-03-23T11:47:19.436 INFO:tasks.workunit.client.0.smithi063.stdout:4/27: write f2 [198418,17373] 0 2024-03-23T11:47:19.437 INFO:tasks.workunit.client.0.smithi063.stdout:7/25: write f2 [4216386,108913] 0 2024-03-23T11:47:19.437 INFO:tasks.workunit.client.0.smithi063.stdout:1/20: creat d3/f9 x:0 0 0 2024-03-23T11:47:19.441 INFO:tasks.workunit.client.0.smithi063.stdout:4/28: chown d5/l6 17170 1 2024-03-23T11:47:19.442 INFO:tasks.workunit.client.0.smithi063.stdout:7/26: write f1 [630706,78174] 0 2024-03-23T11:47:19.445 INFO:tasks.workunit.client.0.smithi063.stdout:8/17: getdents d6 0 2024-03-23T11:47:19.446 INFO:tasks.workunit.client.0.smithi063.stdout:4/29: write f2 [4265726,97419] 0 2024-03-23T11:47:19.450 INFO:tasks.workunit.client.0.smithi063.stdout:8/18: fdatasync f1 0 2024-03-23T11:47:19.453 INFO:tasks.workunit.client.0.smithi063.stdout:3/22: dwrite f6 [0,4194304] 0 2024-03-23T11:47:19.457 INFO:tasks.workunit.client.0.smithi063.stdout:7/27: rmdir d5/d6/d7 0 2024-03-23T11:47:19.457 INFO:tasks.workunit.client.0.smithi063.stdout:8/19: rename f2 to d6/f7 0 2024-03-23T11:47:19.459 INFO:tasks.workunit.client.0.smithi063.stdout:7/28: write f1 [2216299,42173] 0 2024-03-23T11:47:19.460 INFO:tasks.workunit.client.0.smithi063.stdout:8/20: chown f5 1 1 2024-03-23T11:47:19.464 INFO:tasks.workunit.client.0.smithi063.stdout:8/21: write f5 [428358,11725] 0 2024-03-23T11:47:19.468 INFO:tasks.workunit.client.1.smithi111.stderr:++ readlink -f fsstress 2024-03-23T11:47:19.468 INFO:tasks.workunit.client.0.smithi063.stdout:8/22: truncate f5 644797 0 2024-03-23T11:47:19.468 INFO:tasks.workunit.client.0.smithi063.stdout:7/29: fsync f1 0 2024-03-23T11:47:19.472 INFO:tasks.workunit.client.1.smithi111.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-23T11:47:19.472 INFO:tasks.workunit.client.1.smithi111.stderr:+ popd 2024-03-23T11:47:19.474 INFO:tasks.workunit.client.1.smithi111.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-23T11:47:19.474 INFO:tasks.workunit.client.1.smithi111.stderr:+ popd 2024-03-23T11:47:19.475 INFO:tasks.workunit.client.1.smithi111.stdout:~/cephtest/mnt.1/client.1/tmp 2024-03-23T11:47:19.476 INFO:tasks.workunit.client.1.smithi111.stderr:++ mktemp -d -p . 2024-03-23T11:47:19.477 INFO:tasks.workunit.client.0.smithi063.stdout:4/30: dwrite f2 [0,4194304] 0 2024-03-23T11:47:19.477 INFO:tasks.workunit.client.0.smithi063.stdout:7/30: rename f2 to d5/f8 0 2024-03-23T11:47:19.477 INFO:tasks.workunit.client.0.smithi063.stdout:4/31: chown d5/d9 29 1 2024-03-23T11:47:19.477 INFO:tasks.workunit.client.0.smithi063.stdout:4/32: stat d5/d9 0 2024-03-23T11:47:19.483 INFO:tasks.workunit.client.1.smithi111.stderr:+ T=./tmp.v3ZauUBSr9 2024-03-23T11:47:19.483 INFO:tasks.workunit.client.1.smithi111.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.v3ZauUBSr9 -l 1 -n 1000 -p 10 -v 2024-03-23T11:47:19.484 INFO:tasks.workunit.client.0.smithi063.stdout:4/33: write f2 [3438938,7653] 0 2024-03-23T11:47:19.487 INFO:tasks.workunit.client.0.smithi063.stdout:4/34: write f2 [2844582,18220] 0 2024-03-23T11:47:19.493 INFO:tasks.workunit.client.1.smithi111.stdout:seed = 1710819303 2024-03-23T11:47:19.496 INFO:tasks.workunit.client.0.smithi063.stdout:4/35: rmdir d5/d9 0 2024-03-23T11:47:19.501 INFO:tasks.workunit.client.1.smithi111.stdout:0/0: link - no file 2024-03-23T11:47:19.501 INFO:tasks.workunit.client.1.smithi111.stdout:0/1: fsync - no filename 2024-03-23T11:47:19.502 INFO:tasks.workunit.client.1.smithi111.stdout:0/2: fsync - no filename 2024-03-23T11:47:19.502 INFO:tasks.workunit.client.0.smithi063.stdout:8/23: dwrite f1 [0,4194304] 0 2024-03-23T11:47:19.505 INFO:tasks.workunit.client.1.smithi111.stdout:0/3: mkdir d0 0 2024-03-23T11:47:19.506 INFO:tasks.workunit.client.0.smithi063.stdout:4/36: creat d5/da/fb x:0 0 0 2024-03-23T11:47:19.507 INFO:tasks.workunit.client.1.smithi111.stdout:0/4: rename d0 to d0/d1 22 2024-03-23T11:47:19.507 INFO:tasks.workunit.client.1.smithi111.stdout:0/5: dwrite - no filename 2024-03-23T11:47:19.507 INFO:tasks.workunit.client.1.smithi111.stdout:0/6: fdatasync - no filename 2024-03-23T11:47:19.508 INFO:tasks.workunit.client.1.smithi111.stdout:3/0: fsync - no filename 2024-03-23T11:47:19.508 INFO:tasks.workunit.client.1.smithi111.stdout:3/1: rename - no filename 2024-03-23T11:47:19.508 INFO:tasks.workunit.client.1.smithi111.stdout:3/2: write - no filename 2024-03-23T11:47:19.508 INFO:tasks.workunit.client.1.smithi111.stdout:3/3: write - no filename 2024-03-23T11:47:19.509 INFO:tasks.workunit.client.1.smithi111.stdout:3/4: readlink - no filename 2024-03-23T11:47:19.509 INFO:tasks.workunit.client.1.smithi111.stdout:3/5: dwrite - no filename 2024-03-23T11:47:19.509 INFO:tasks.workunit.client.1.smithi111.stdout:3/6: dread - no filename 2024-03-23T11:47:19.509 INFO:tasks.workunit.client.1.smithi111.stdout:3/7: dwrite - no filename 2024-03-23T11:47:19.509 INFO:tasks.workunit.client.1.smithi111.stdout:3/8: unlink - no file 2024-03-23T11:47:19.509 INFO:tasks.workunit.client.1.smithi111.stdout:1/0: getdents . 0 2024-03-23T11:47:19.510 INFO:tasks.workunit.client.1.smithi111.stdout:1/1: truncate - no filename 2024-03-23T11:47:19.512 INFO:tasks.workunit.client.1.smithi111.stdout:4/0: dread - no filename 2024-03-23T11:47:19.513 INFO:tasks.workunit.client.1.smithi111.stdout:3/9: mkdir d0 0 2024-03-23T11:47:19.513 INFO:tasks.workunit.client.1.smithi111.stdout:4/1: chown . 1 1 2024-03-23T11:47:19.515 INFO:tasks.workunit.client.1.smithi111.stdout:1/2: mkdir d0 0 2024-03-23T11:47:19.516 INFO:tasks.workunit.client.1.smithi111.stdout:0/7: rmdir d0 0 2024-03-23T11:47:19.517 INFO:tasks.workunit.client.0.smithi063.stdout:4/37: creat d5/fc x:0 0 0 2024-03-23T11:47:19.518 INFO:tasks.workunit.client.1.smithi111.stdout:5/0: truncate - no filename 2024-03-23T11:47:19.519 INFO:tasks.workunit.client.1.smithi111.stdout:4/2: mkdir d0 0 2024-03-23T11:47:19.520 INFO:tasks.workunit.client.1.smithi111.stdout:1/3: creat d0/f1 x:0 0 0 2024-03-23T11:47:19.522 INFO:tasks.workunit.client.1.smithi111.stdout:6/0: rename - no filename 2024-03-23T11:47:19.523 INFO:tasks.workunit.client.1.smithi111.stdout:1/4: mknod d0/c2 0 2024-03-23T11:47:19.524 INFO:tasks.workunit.client.1.smithi111.stdout:5/1: symlink l0 0 2024-03-23T11:47:19.524 INFO:tasks.workunit.client.1.smithi111.stdout:5/2: dwrite - no filename 2024-03-23T11:47:19.524 INFO:tasks.workunit.client.1.smithi111.stdout:5/3: write - no filename 2024-03-23T11:47:19.525 INFO:tasks.workunit.client.0.smithi063.stdout:4/38: creat d5/da/fd x:0 0 0 2024-03-23T11:47:19.525 INFO:tasks.workunit.client.1.smithi111.stdout:4/3: creat d0/f1 x:0 0 0 2024-03-23T11:47:19.527 INFO:tasks.workunit.client.1.smithi111.stdout:2/0: rename - no filename 2024-03-23T11:47:19.527 INFO:tasks.workunit.client.1.smithi111.stdout:2/1: fsync - no filename 2024-03-23T11:47:19.527 INFO:tasks.workunit.client.1.smithi111.stdout:2/2: unlink - no file 2024-03-23T11:47:19.527 INFO:tasks.workunit.client.1.smithi111.stdout:2/3: rmdir - no directory 2024-03-23T11:47:19.530 INFO:tasks.workunit.client.1.smithi111.stdout:1/5: symlink d0/l3 0 2024-03-23T11:47:19.530 INFO:tasks.workunit.client.1.smithi111.stdout:5/4: mkdir d1 0 2024-03-23T11:47:19.531 INFO:tasks.workunit.client.1.smithi111.stdout:5/5: chown d1 1393960140 1 2024-03-23T11:47:19.532 INFO:tasks.workunit.client.1.smithi111.stdout:1/6: write d0/f1 [417875,60692] 0 2024-03-23T11:47:19.532 INFO:tasks.workunit.client.1.smithi111.stdout:4/4: mknod d0/c2 0 2024-03-23T11:47:19.532 INFO:tasks.workunit.client.1.smithi111.stdout:4/5: readlink - no filename 2024-03-23T11:47:19.536 INFO:tasks.workunit.client.1.smithi111.stdout:7/0: readlink - no filename 2024-03-23T11:47:19.540 INFO:tasks.workunit.client.0.smithi063.stdout:6/22: rmdir d1 39 2024-03-23T11:47:19.540 INFO:tasks.workunit.client.0.smithi063.stdout:8/24: dwrite f1 [0,4194304] 0 2024-03-23T11:47:19.541 INFO:tasks.workunit.client.1.smithi111.stdout:6/1: symlink l0 0 2024-03-23T11:47:19.541 INFO:tasks.workunit.client.1.smithi111.stdout:6/2: dwrite - no filename 2024-03-23T11:47:19.542 INFO:tasks.workunit.client.1.smithi111.stdout:6/3: chown l0 5535 1 2024-03-23T11:47:19.542 INFO:tasks.workunit.client.0.smithi063.stdout:8/25: stat f1 0 2024-03-23T11:47:19.544 INFO:tasks.workunit.client.0.smithi063.stdout:3/23: truncate f2 1603388 0 2024-03-23T11:47:19.545 INFO:tasks.workunit.client.1.smithi111.stdout:2/4: creat f0 x:0 0 0 2024-03-23T11:47:19.546 INFO:tasks.workunit.client.1.smithi111.stdout:2/5: dread - f0 zero size 2024-03-23T11:47:19.551 INFO:tasks.workunit.client.0.smithi063.stdout:8/26: chown d6/f7 9220442 1 2024-03-23T11:47:19.552 INFO:tasks.workunit.client.1.smithi111.stdout:6/4: rename l0 to l1 0 2024-03-23T11:47:19.552 INFO:tasks.workunit.client.1.smithi111.stdout:6/5: dread - no filename 2024-03-23T11:47:19.555 INFO:tasks.workunit.client.1.smithi111.stdout:9/0: truncate - no filename 2024-03-23T11:47:19.555 INFO:tasks.workunit.client.1.smithi111.stdout:9/1: chown . 12550322 1 2024-03-23T11:47:19.555 INFO:tasks.workunit.client.1.smithi111.stdout:9/2: dread - no filename 2024-03-23T11:47:19.555 INFO:tasks.workunit.client.1.smithi111.stdout:9/3: unlink - no file 2024-03-23T11:47:19.556 INFO:tasks.workunit.client.1.smithi111.stdout:9/4: unlink - no file 2024-03-23T11:47:19.556 INFO:tasks.workunit.client.1.smithi111.stdout:9/5: rename - no filename 2024-03-23T11:47:19.556 INFO:tasks.workunit.client.1.smithi111.stdout:9/6: rename - no filename 2024-03-23T11:47:19.556 INFO:tasks.workunit.client.0.smithi063.stdout:6/23: getdents d1 0 2024-03-23T11:47:19.558 INFO:tasks.workunit.client.1.smithi111.stdout:2/6: mknod c1 0 2024-03-23T11:47:19.560 INFO:tasks.workunit.client.1.smithi111.stdout:7/1: getdents . 0 2024-03-23T11:47:19.560 INFO:tasks.workunit.client.1.smithi111.stdout:7/2: readlink - no filename 2024-03-23T11:47:19.562 INFO:tasks.workunit.client.0.smithi063.stdout:8/27: mknod d6/c8 0 2024-03-23T11:47:19.562 INFO:tasks.workunit.client.0.smithi063.stdout:4/39: dwrite d5/da/fd [0,4194304] 0 2024-03-23T11:47:19.563 INFO:tasks.workunit.client.0.smithi063.stdout:4/40: read f2 [1399934,53458] 0 2024-03-23T11:47:19.564 INFO:tasks.workunit.client.1.smithi111.stdout:5/6: link l0 d1/l2 0 2024-03-23T11:47:19.564 INFO:tasks.workunit.client.1.smithi111.stdout:5/7: write - no filename 2024-03-23T11:47:19.564 INFO:tasks.workunit.client.1.smithi111.stdout:5/8: dwrite - no filename 2024-03-23T11:47:19.566 INFO:tasks.workunit.client.1.smithi111.stdout:4/6: link d0/c2 d0/c3 0 2024-03-23T11:47:19.570 INFO:tasks.workunit.client.1.smithi111.stdout:6/6: mkdir d2 0 2024-03-23T11:47:19.570 INFO:tasks.workunit.client.1.smithi111.stdout:6/7: dwrite - no filename 2024-03-23T11:47:19.570 INFO:tasks.workunit.client.1.smithi111.stdout:6/8: write - no filename 2024-03-23T11:47:19.572 INFO:tasks.workunit.client.1.smithi111.stdout:8/0: mknod c0 0 2024-03-23T11:47:19.576 INFO:tasks.workunit.client.1.smithi111.stdout:2/7: dwrite f0 [0,4194304] 0 2024-03-23T11:47:19.576 INFO:tasks.workunit.client.1.smithi111.stdout:4/7: creat d0/f4 x:0 0 0 2024-03-23T11:47:19.577 INFO:tasks.workunit.client.0.smithi063.stdout:8/28: dread f5 [0,4194304] 0 2024-03-23T11:47:19.578 INFO:tasks.workunit.client.0.smithi063.stdout:4/41: symlink d5/le 0 2024-03-23T11:47:19.578 INFO:tasks.workunit.client.1.smithi111.stdout:9/7: mkdir d0 0 2024-03-23T11:47:19.579 INFO:tasks.workunit.client.1.smithi111.stdout:4/8: dread - d0/f1 zero size 2024-03-23T11:47:19.582 INFO:tasks.workunit.client.0.smithi063.stdout:7/31: truncate d5/f8 75122 0 2024-03-23T11:47:19.587 INFO:tasks.workunit.client.1.smithi111.stdout:2/8: dread f0 [0,4194304] 0 2024-03-23T11:47:19.588 INFO:tasks.workunit.client.0.smithi063.stdout:7/32: mkdir d5/d9 0 2024-03-23T11:47:19.590 INFO:tasks.workunit.client.0.smithi063.stdout:7/33: fsync f4 0 2024-03-23T11:47:19.591 INFO:tasks.workunit.client.1.smithi111.stdout:7/3: creat f0 x:0 0 0 2024-03-23T11:47:19.593 INFO:tasks.workunit.client.0.smithi063.stdout:6/24: dwrite d1/f3 [0,4194304] 0 2024-03-23T11:47:19.593 INFO:tasks.workunit.client.1.smithi111.stdout:5/9: mkdir d1/d3 0 2024-03-23T11:47:19.593 INFO:tasks.workunit.client.1.smithi111.stdout:5/10: write - no filename 2024-03-23T11:47:19.594 INFO:tasks.workunit.client.0.smithi063.stdout:3/24: dwrite f7 [0,4194304] 0 2024-03-23T11:47:19.594 INFO:tasks.workunit.client.1.smithi111.stdout:8/1: creat f1 x:0 0 0 2024-03-23T11:47:19.594 INFO:tasks.workunit.client.1.smithi111.stdout:5/11: chown d1 512 1 2024-03-23T11:47:19.594 INFO:tasks.workunit.client.1.smithi111.stdout:5/12: write - no filename 2024-03-23T11:47:19.595 INFO:tasks.workunit.client.1.smithi111.stdout:7/4: write f0 [791234,91901] 0 2024-03-23T11:47:19.595 INFO:tasks.workunit.client.1.smithi111.stdout:8/2: dread - f1 zero size 2024-03-23T11:47:19.600 INFO:tasks.workunit.client.1.smithi111.stdout:8/3: chown f1 5710700 1 2024-03-23T11:47:19.601 INFO:tasks.workunit.client.1.smithi111.stdout:7/5: write f0 [1539621,35352] 0 2024-03-23T11:47:19.601 INFO:tasks.workunit.client.1.smithi111.stdout:4/9: dwrite d0/f4 [0,4194304] 0 2024-03-23T11:47:19.611 INFO:tasks.workunit.client.1.smithi111.stdout:9/8: mkdir d0/d1 0 2024-03-23T11:47:19.612 INFO:tasks.workunit.client.1.smithi111.stdout:9/9: fsync - no filename 2024-03-23T11:47:19.612 INFO:tasks.workunit.client.1.smithi111.stdout:9/10: dread - no filename 2024-03-23T11:47:19.612 INFO:tasks.workunit.client.1.smithi111.stdout:9/11: fdatasync - no filename 2024-03-23T11:47:19.613 INFO:tasks.workunit.client.1.smithi111.stdout:4/10: mknod d0/c5 0 2024-03-23T11:47:19.616 INFO:tasks.workunit.client.0.smithi063.stdout:8/29: dread f1 [0,4194304] 0 2024-03-23T11:47:19.618 INFO:tasks.workunit.client.0.smithi063.stdout:4/42: dwrite d5/da/fb [0,4194304] 0 2024-03-23T11:47:19.618 INFO:tasks.workunit.client.1.smithi111.stdout:5/13: unlink d1/l2 0 2024-03-23T11:47:19.620 INFO:tasks.workunit.client.1.smithi111.stdout:9/12: mknod d0/c2 0 2024-03-23T11:47:19.620 INFO:tasks.workunit.client.1.smithi111.stdout:9/13: dwrite - no filename 2024-03-23T11:47:19.620 INFO:tasks.workunit.client.1.smithi111.stdout:9/14: dread - no filename 2024-03-23T11:47:19.620 INFO:tasks.workunit.client.1.smithi111.stdout:9/15: write - no filename 2024-03-23T11:47:19.620 INFO:tasks.workunit.client.1.smithi111.stdout:9/16: read - no filename 2024-03-23T11:47:19.621 INFO:tasks.workunit.client.0.smithi063.stdout:3/25: write f2 [1069452,35790] 0 2024-03-23T11:47:19.624 INFO:tasks.workunit.client.1.smithi111.stdout:4/11: mknod d0/c6 0 2024-03-23T11:47:19.625 INFO:tasks.workunit.client.0.smithi063.stdout:3/26: write f3 [821360,26469] 0 2024-03-23T11:47:19.625 INFO:tasks.workunit.client.0.smithi063.stdout:3/27: rmdir - no directory 2024-03-23T11:47:19.632 INFO:tasks.workunit.client.0.smithi063.stdout:8/30: rename d6/f7 to d6/f9 0 2024-03-23T11:47:19.633 INFO:tasks.workunit.client.0.smithi063.stdout:4/43: read f2 [700770,13205] 0 2024-03-23T11:47:19.633 INFO:tasks.workunit.client.1.smithi111.stdout:8/4: dwrite f1 [0,4194304] 0 2024-03-23T11:47:19.634 INFO:tasks.workunit.client.1.smithi111.stdout:5/14: creat d1/d3/f4 x:0 0 0 2024-03-23T11:47:19.634 INFO:tasks.workunit.client.1.smithi111.stdout:7/6: dwrite f0 [0,4194304] 0 2024-03-23T11:47:19.637 INFO:tasks.workunit.client.1.smithi111.stdout:4/12: mkdir d0/d7 0 2024-03-23T11:47:19.637 INFO:tasks.workunit.client.0.smithi063.stdout:8/31: rmdir d6 39 2024-03-23T11:47:19.639 INFO:tasks.workunit.client.1.smithi111.stdout:4/13: chown d0/f4 229976 1 2024-03-23T11:47:19.640 INFO:tasks.workunit.client.1.smithi111.stdout:4/14: chown d0/c6 432864474 1 2024-03-23T11:47:19.645 INFO:tasks.workunit.client.0.smithi063.stdout:6/25: dwrite d1/f3 [0,4194304] 0 2024-03-23T11:47:19.647 INFO:tasks.workunit.client.0.smithi063.stdout:6/26: read - d1/f4 zero size 2024-03-23T11:47:19.648 INFO:tasks.workunit.client.1.smithi111.stdout:5/15: mknod d1/c5 0 2024-03-23T11:47:19.648 INFO:tasks.workunit.client.1.smithi111.stdout:7/7: creat f1 x:0 0 0 2024-03-23T11:47:19.654 INFO:tasks.workunit.client.0.smithi063.stdout:7/34: truncate d5/f8 336519 0 2024-03-23T11:47:19.654 INFO:tasks.workunit.client.0.smithi063.stdout:6/27: dread - d1/f4 zero size 2024-03-23T11:47:19.654 INFO:tasks.workunit.client.0.smithi063.stdout:8/32: mknod d6/ca 0 2024-03-23T11:47:19.656 INFO:tasks.workunit.client.1.smithi111.stdout:4/15: mknod d0/c8 0 2024-03-23T11:47:19.656 INFO:tasks.workunit.client.1.smithi111.stdout:5/16: mkdir d1/d3/d6 0 2024-03-23T11:47:19.656 INFO:tasks.workunit.client.0.smithi063.stdout:8/33: chown d6/ca 0 1 2024-03-23T11:47:19.657 INFO:tasks.workunit.client.0.smithi063.stdout:6/28: creat d1/f5 x:0 0 0 2024-03-23T11:47:19.658 INFO:tasks.workunit.client.0.smithi063.stdout:4/44: dread d5/da/fd [0,4194304] 0 2024-03-23T11:47:19.659 INFO:tasks.workunit.client.0.smithi063.stdout:6/29: write d1/f3 [3957900,108931] 0 2024-03-23T11:47:19.660 INFO:tasks.workunit.client.0.smithi063.stdout:7/35: mkdir d5/d9/da 0 2024-03-23T11:47:19.661 INFO:tasks.workunit.client.1.smithi111.stdout:9/17: link d0/c2 d0/c3 0 2024-03-23T11:47:19.661 INFO:tasks.workunit.client.1.smithi111.stdout:9/18: dwrite - no filename 2024-03-23T11:47:19.665 INFO:tasks.workunit.client.0.smithi063.stdout:4/45: truncate d5/fc 852283 0 2024-03-23T11:47:19.666 INFO:tasks.workunit.client.0.smithi063.stdout:6/30: chown d1/f4 613840 1 2024-03-23T11:47:19.667 INFO:tasks.workunit.client.0.smithi063.stdout:3/28: truncate f2 432250 0 2024-03-23T11:47:19.672 INFO:tasks.workunit.client.0.smithi063.stdout:7/36: mknod d5/d9/da/cb 0 2024-03-23T11:47:19.674 INFO:tasks.workunit.client.0.smithi063.stdout:3/29: creat f8 x:0 0 0 2024-03-23T11:47:19.674 INFO:tasks.workunit.client.0.smithi063.stdout:3/30: rmdir - no directory 2024-03-23T11:47:19.675 INFO:tasks.workunit.client.1.smithi111.stdout:9/19: mkdir d0/d4 0 2024-03-23T11:47:19.675 INFO:tasks.workunit.client.1.smithi111.stdout:9/20: write - no filename 2024-03-23T11:47:19.677 INFO:tasks.workunit.client.0.smithi063.stdout:7/37: write f1 [1245115,28783] 0 2024-03-23T11:47:19.679 INFO:tasks.workunit.client.0.smithi063.stdout:8/34: dread d6/f9 [0,4194304] 0 2024-03-23T11:47:19.685 INFO:tasks.workunit.client.0.smithi063.stdout:3/31: truncate f3 863266 0 2024-03-23T11:47:19.686 INFO:tasks.workunit.client.0.smithi063.stdout:4/46: write f2 [11517,47487] 0 2024-03-23T11:47:19.689 INFO:tasks.workunit.client.0.smithi063.stdout:4/47: chown d5/da 1366 1 2024-03-23T11:47:19.691 INFO:tasks.workunit.client.0.smithi063.stdout:6/31: dread d1/f3 [0,4194304] 0 2024-03-23T11:47:19.692 INFO:tasks.workunit.client.1.smithi111.stdout:0/8: getdents . 0 2024-03-23T11:47:19.693 INFO:tasks.workunit.client.0.smithi063.stdout:6/32: dread - d1/f4 zero size 2024-03-23T11:47:19.694 INFO:tasks.workunit.client.1.smithi111.stdout:4/16: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:19.694 INFO:tasks.workunit.client.1.smithi111.stdout:0/9: chown . 6 1 2024-03-23T11:47:19.694 INFO:tasks.workunit.client.1.smithi111.stdout:0/10: rename - no filename 2024-03-23T11:47:19.699 INFO:tasks.workunit.client.1.smithi111.stdout:9/21: creat d0/d1/f5 x:0 0 0 2024-03-23T11:47:19.699 INFO:tasks.workunit.client.1.smithi111.stdout:5/17: dwrite d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:19.703 INFO:tasks.workunit.client.1.smithi111.stdout:5/18: write d1/d3/f4 [341794,95883] 0 2024-03-23T11:47:19.704 INFO:tasks.workunit.client.0.smithi063.stdout:7/38: truncate f4 113077 0 2024-03-23T11:47:19.705 INFO:tasks.workunit.client.0.smithi063.stdout:8/35: dread d6/f9 [0,4194304] 0 2024-03-23T11:47:19.708 INFO:tasks.workunit.client.1.smithi111.stdout:0/11: symlink l2 0 2024-03-23T11:47:19.708 INFO:tasks.workunit.client.1.smithi111.stdout:0/12: stat l2 0 2024-03-23T11:47:19.721 INFO:tasks.workunit.client.1.smithi111.stdout:0/13: mknod c3 0 2024-03-23T11:47:19.721 INFO:tasks.workunit.client.1.smithi111.stdout:0/14: rmdir - no directory 2024-03-23T11:47:19.723 INFO:tasks.workunit.client.1.smithi111.stdout:0/15: readlink l2 0 2024-03-23T11:47:19.723 INFO:tasks.workunit.client.1.smithi111.stdout:0/16: fsync - no filename 2024-03-23T11:47:19.723 INFO:tasks.workunit.client.1.smithi111.stdout:0/17: truncate - no filename 2024-03-23T11:47:19.723 INFO:tasks.workunit.client.1.smithi111.stdout:0/18: write - no filename 2024-03-23T11:47:19.723 INFO:tasks.workunit.client.0.smithi063.stdout:4/48: dwrite d5/fc [0,4194304] 0 2024-03-23T11:47:19.724 INFO:tasks.workunit.client.0.smithi063.stdout:8/36: write d6/f9 [3816256,123616] 0 2024-03-23T11:47:19.727 INFO:tasks.workunit.client.1.smithi111.stdout:5/19: mkdir d1/d3/d6/d7 0 2024-03-23T11:47:19.728 INFO:tasks.workunit.client.0.smithi063.stdout:3/32: link f3 f9 0 2024-03-23T11:47:19.731 INFO:tasks.workunit.client.0.smithi063.stdout:4/49: write d5/da/fb [1403114,100955] 0 2024-03-23T11:47:19.731 INFO:tasks.workunit.client.0.smithi063.stdout:3/33: read f7 [2268652,26737] 0 2024-03-23T11:47:19.736 INFO:tasks.workunit.client.0.smithi063.stdout:6/33: getdents d1 0 2024-03-23T11:47:19.739 INFO:tasks.workunit.client.1.smithi111.stdout:9/22: dwrite d0/d1/f5 [0,4194304] 0 2024-03-23T11:47:19.739 INFO:tasks.workunit.client.0.smithi063.stdout:7/39: fsync d5/f8 0 2024-03-23T11:47:19.741 INFO:tasks.workunit.client.1.smithi111.stdout:0/19: creat f4 x:0 0 0 2024-03-23T11:47:19.743 INFO:tasks.workunit.client.1.smithi111.stdout:2/9: write f0 [4352363,115272] 0 2024-03-23T11:47:19.745 INFO:tasks.workunit.client.1.smithi111.stdout:9/23: read d0/d1/f5 [329580,41520] 0 2024-03-23T11:47:19.745 INFO:tasks.workunit.client.1.smithi111.stdout:2/10: read f0 [197196,47052] 0 2024-03-23T11:47:19.746 INFO:tasks.workunit.client.0.smithi063.stdout:7/40: truncate f1 2834887 0 2024-03-23T11:47:19.746 INFO:tasks.workunit.client.1.smithi111.stdout:8/5: getdents . 0 2024-03-23T11:47:19.749 INFO:tasks.workunit.client.1.smithi111.stdout:6/9: link l1 d2/l3 0 2024-03-23T11:47:19.754 INFO:tasks.workunit.client.1.smithi111.stdout:8/6: write f1 [367733,24022] 0 2024-03-23T11:47:19.756 INFO:tasks.workunit.client.0.smithi063.stdout:8/37: dread f1 [0,4194304] 0 2024-03-23T11:47:19.756 INFO:tasks.workunit.client.1.smithi111.stdout:9/24: mknod d0/d1/c6 0 2024-03-23T11:47:19.757 INFO:tasks.workunit.client.0.smithi063.stdout:7/41: rename l0 to d5/lc 0 2024-03-23T11:47:19.758 INFO:tasks.workunit.client.1.smithi111.stdout:9/25: chown d0 6434 1 2024-03-23T11:47:19.761 INFO:tasks.workunit.client.0.smithi063.stdout:6/34: link d1/l2 d1/l6 0 2024-03-23T11:47:19.763 INFO:tasks.workunit.client.0.smithi063.stdout:8/38: symlink d6/lb 0 2024-03-23T11:47:19.764 INFO:tasks.workunit.client.1.smithi111.stdout:6/10: symlink d2/l4 0 2024-03-23T11:47:19.764 INFO:tasks.workunit.client.0.smithi063.stdout:4/50: truncate d5/fc 3561767 0 2024-03-23T11:47:19.768 INFO:tasks.workunit.client.0.smithi063.stdout:6/35: creat d1/f7 x:0 0 0 2024-03-23T11:47:19.768 INFO:tasks.workunit.client.1.smithi111.stdout:4/17: getdents d0 0 2024-03-23T11:47:19.771 INFO:tasks.workunit.client.1.smithi111.stdout:5/20: dwrite d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:19.771 INFO:tasks.workunit.client.1.smithi111.stdout:0/20: fsync f4 0 2024-03-23T11:47:19.773 INFO:tasks.workunit.client.0.smithi063.stdout:6/36: truncate d1/f5 204097 0 2024-03-23T11:47:19.773 INFO:tasks.workunit.client.1.smithi111.stdout:4/18: write d0/f4 [103908,66475] 0 2024-03-23T11:47:19.775 INFO:tasks.workunit.client.1.smithi111.stdout:6/11: creat d2/f5 x:0 0 0 2024-03-23T11:47:19.775 INFO:tasks.workunit.client.1.smithi111.stdout:2/11: dread f0 [0,4194304] 0 2024-03-23T11:47:19.777 INFO:tasks.workunit.client.1.smithi111.stdout:5/21: write d1/d3/f4 [957961,123281] 0 2024-03-23T11:47:19.777 INFO:tasks.workunit.client.1.smithi111.stdout:0/21: mkdir d5 0 2024-03-23T11:47:19.779 INFO:tasks.workunit.client.0.smithi063.stdout:3/34: dwrite f6 [4194304,4194304] 0 2024-03-23T11:47:19.779 INFO:tasks.workunit.client.0.smithi063.stdout:8/39: link d6/c8 d6/cc 0 2024-03-23T11:47:19.780 INFO:tasks.workunit.client.1.smithi111.stdout:2/12: chown c1 1571 1 2024-03-23T11:47:19.781 INFO:tasks.workunit.client.0.smithi063.stdout:6/37: creat d1/f8 x:0 0 0 2024-03-23T11:47:19.781 INFO:tasks.workunit.client.1.smithi111.stdout:0/22: fsync f4 0 2024-03-23T11:47:19.781 INFO:tasks.workunit.client.1.smithi111.stdout:6/12: mknod d2/c6 0 2024-03-23T11:47:19.781 INFO:tasks.workunit.client.0.smithi063.stdout:4/51: write d5/da/fd [1741219,770] 0 2024-03-23T11:47:19.785 INFO:tasks.workunit.client.1.smithi111.stdout:9/26: rename d0/c2 to d0/d1/c7 0 2024-03-23T11:47:19.785 INFO:tasks.workunit.client.1.smithi111.stdout:6/13: creat d2/f7 x:0 0 0 2024-03-23T11:47:19.786 INFO:tasks.workunit.client.0.smithi063.stdout:3/35: creat fa x:0 0 0 2024-03-23T11:47:19.786 INFO:tasks.workunit.client.1.smithi111.stdout:2/13: write f0 [3889672,23543] 0 2024-03-23T11:47:19.786 INFO:tasks.workunit.client.1.smithi111.stdout:5/22: creat d1/d3/d6/f8 x:0 0 0 2024-03-23T11:47:19.787 INFO:tasks.workunit.client.1.smithi111.stdout:6/14: dread - d2/f5 zero size 2024-03-23T11:47:19.789 INFO:tasks.workunit.client.1.smithi111.stdout:5/23: write d1/d3/d6/f8 [578408,2894] 0 2024-03-23T11:47:19.789 INFO:tasks.workunit.client.1.smithi111.stdout:2/14: dread f0 [4194304,4194304] 0 2024-03-23T11:47:19.790 INFO:tasks.workunit.client.1.smithi111.stdout:0/23: symlink d5/l6 0 2024-03-23T11:47:19.790 INFO:tasks.workunit.client.0.smithi063.stdout:6/38: readlink d1/l2 0 2024-03-23T11:47:19.790 INFO:tasks.workunit.client.1.smithi111.stdout:9/27: symlink d0/d1/l8 0 2024-03-23T11:47:19.792 INFO:tasks.workunit.client.0.smithi063.stdout:8/40: write f1 [4041954,13033] 0 2024-03-23T11:47:19.792 INFO:tasks.workunit.client.1.smithi111.stdout:6/15: readlink l1 0 2024-03-23T11:47:19.793 INFO:tasks.workunit.client.0.smithi063.stdout:6/39: stat d1/f8 0 2024-03-23T11:47:19.793 INFO:tasks.workunit.client.1.smithi111.stdout:0/24: mknod d5/c7 0 2024-03-23T11:47:19.794 INFO:tasks.workunit.client.1.smithi111.stdout:6/16: write d2/f5 [544083,128367] 0 2024-03-23T11:47:19.796 INFO:tasks.workunit.client.1.smithi111.stdout:0/25: write f4 [841794,49295] 0 2024-03-23T11:47:19.797 INFO:tasks.workunit.client.1.smithi111.stdout:0/26: readlink l2 0 2024-03-23T11:47:19.798 INFO:tasks.workunit.client.1.smithi111.stdout:9/28: creat d0/d4/f9 x:0 0 0 2024-03-23T11:47:19.799 INFO:tasks.workunit.client.0.smithi063.stdout:7/42: link d5/lc d5/ld 0 2024-03-23T11:47:19.800 INFO:tasks.workunit.client.1.smithi111.stdout:5/24: creat d1/d3/d6/d7/f9 x:0 0 0 2024-03-23T11:47:19.800 INFO:tasks.workunit.client.1.smithi111.stdout:6/17: creat d2/f8 x:0 0 0 2024-03-23T11:47:19.800 INFO:tasks.workunit.client.1.smithi111.stdout:0/27: write f4 [1563601,103792] 0 2024-03-23T11:47:19.803 INFO:tasks.workunit.client.0.smithi063.stdout:4/52: dread d5/da/fb [0,4194304] 0 2024-03-23T11:47:19.804 INFO:tasks.workunit.client.0.smithi063.stdout:7/43: read d5/f8 [244375,9587] 0 2024-03-23T11:47:19.805 INFO:tasks.workunit.client.0.smithi063.stdout:4/53: fsync d5/da/fd 0 2024-03-23T11:47:19.805 INFO:tasks.workunit.client.1.smithi111.stdout:2/15: dread f0 [0,4194304] 0 2024-03-23T11:47:19.807 INFO:tasks.workunit.client.0.smithi063.stdout:3/36: truncate f9 805724 0 2024-03-23T11:47:19.808 INFO:tasks.workunit.client.1.smithi111.stdout:5/25: readlink l0 0 2024-03-23T11:47:19.810 INFO:tasks.workunit.client.1.smithi111.stdout:0/28: creat d5/f8 x:0 0 0 2024-03-23T11:47:19.811 INFO:tasks.workunit.client.0.smithi063.stdout:8/41: mkdir d6/dd 0 2024-03-23T11:47:19.813 INFO:tasks.workunit.client.0.smithi063.stdout:3/37: symlink lb 0 2024-03-23T11:47:19.815 INFO:tasks.workunit.client.1.smithi111.stdout:6/18: rename d2/l3 to d2/l9 0 2024-03-23T11:47:19.816 INFO:tasks.workunit.client.1.smithi111.stdout:5/26: creat d1/d3/d6/fa x:0 0 0 2024-03-23T11:47:19.818 INFO:tasks.workunit.client.1.smithi111.stdout:6/19: dread - d2/f7 zero size 2024-03-23T11:47:19.820 INFO:tasks.workunit.client.1.smithi111.stdout:6/20: dread - d2/f7 zero size 2024-03-23T11:47:19.821 INFO:tasks.workunit.client.0.smithi063.stdout:8/42: mkdir d6/de 0 2024-03-23T11:47:19.823 INFO:tasks.workunit.client.1.smithi111.stdout:6/21: creat d2/fa x:0 0 0 2024-03-23T11:47:19.824 INFO:tasks.workunit.client.1.smithi111.stdout:5/27: write d1/d3/d6/d7/f9 [753692,2181] 0 2024-03-23T11:47:19.826 INFO:tasks.workunit.client.1.smithi111.stdout:6/22: chown d2/f8 29 1 2024-03-23T11:47:19.826 INFO:tasks.workunit.client.0.smithi063.stdout:3/38: link f8 fc 0 2024-03-23T11:47:19.828 INFO:tasks.workunit.client.1.smithi111.stdout:9/29: dwrite d0/d1/f5 [0,4194304] 0 2024-03-23T11:47:19.828 INFO:tasks.workunit.client.1.smithi111.stdout:6/23: write d2/fa [120623,63673] 0 2024-03-23T11:47:19.830 INFO:tasks.workunit.client.1.smithi111.stdout:5/28: symlink d1/d3/d6/d7/lb 0 2024-03-23T11:47:19.830 INFO:tasks.workunit.client.1.smithi111.stdout:6/24: write d2/f7 [271202,81597] 0 2024-03-23T11:47:19.832 INFO:tasks.workunit.client.1.smithi111.stdout:5/29: chown d1/d3 21804088 1 2024-03-23T11:47:19.833 INFO:tasks.workunit.client.1.smithi111.stdout:0/29: rmdir d5 39 2024-03-23T11:47:19.834 INFO:tasks.workunit.client.1.smithi111.stdout:0/30: chown l2 329 1 2024-03-23T11:47:19.838 INFO:tasks.workunit.client.1.smithi111.stdout:9/30: rename d0/d1/c6 to d0/ca 0 2024-03-23T11:47:19.841 INFO:tasks.workunit.client.1.smithi111.stdout:2/16: truncate f0 921156 0 2024-03-23T11:47:19.842 INFO:tasks.workunit.client.0.smithi063.stdout:6/40: dwrite d1/f4 [0,4194304] 0 2024-03-23T11:47:19.845 INFO:tasks.workunit.client.0.smithi063.stdout:3/39: unlink fa 0 2024-03-23T11:47:19.847 INFO:tasks.workunit.client.1.smithi111.stdout:6/25: dread d2/f7 [0,4194304] 0 2024-03-23T11:47:19.848 INFO:tasks.workunit.client.1.smithi111.stdout:2/17: mkdir d2 0 2024-03-23T11:47:19.850 INFO:tasks.workunit.client.1.smithi111.stdout:5/30: dwrite d1/d3/d6/fa [0,4194304] 0 2024-03-23T11:47:19.850 INFO:tasks.workunit.client.0.smithi063.stdout:4/54: dread d5/da/fd [0,4194304] 0 2024-03-23T11:47:19.851 INFO:tasks.workunit.client.0.smithi063.stdout:6/41: symlink d1/l9 0 2024-03-23T11:47:19.851 INFO:tasks.workunit.client.1.smithi111.stdout:0/31: creat d5/f9 x:0 0 0 2024-03-23T11:47:19.852 INFO:tasks.workunit.client.0.smithi063.stdout:3/40: truncate f8 681706 0 2024-03-23T11:47:19.854 INFO:tasks.workunit.client.1.smithi111.stdout:9/31: unlink d0/c3 0 2024-03-23T11:47:19.857 INFO:tasks.workunit.client.1.smithi111.stdout:5/31: creat d1/d3/fc x:0 0 0 2024-03-23T11:47:19.858 INFO:tasks.workunit.client.1.smithi111.stdout:2/18: getdents d2 0 2024-03-23T11:47:19.860 INFO:tasks.workunit.client.1.smithi111.stdout:9/32: mknod d0/d4/cb 0 2024-03-23T11:47:19.864 INFO:tasks.workunit.client.1.smithi111.stdout:5/32: symlink d1/d3/d6/d7/ld 0 2024-03-23T11:47:19.866 INFO:tasks.workunit.client.0.smithi063.stdout:8/43: dwrite f1 [0,4194304] 0 2024-03-23T11:47:19.870 INFO:tasks.workunit.client.0.smithi063.stdout:7/44: dwrite d5/f8 [0,4194304] 0 2024-03-23T11:47:19.873 INFO:tasks.workunit.client.1.smithi111.stdout:9/33: chown d0/ca 1590 1 2024-03-23T11:47:19.874 INFO:tasks.workunit.client.1.smithi111.stdout:2/19: read f0 [397996,46125] 0 2024-03-23T11:47:19.877 INFO:tasks.workunit.client.1.smithi111.stdout:6/26: dwrite d2/f7 [0,4194304] 0 2024-03-23T11:47:19.883 INFO:tasks.workunit.client.0.smithi063.stdout:3/41: truncate f7 414684 0 2024-03-23T11:47:19.885 INFO:tasks.workunit.client.0.smithi063.stdout:3/42: readlink lb 0 2024-03-23T11:47:19.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:19 smithi063 ceph-mon[35189]: pgmap v29: 65 pgs: 65 active+clean; 220 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 397 op/s 2024-03-23T11:47:19.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:19 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:19.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:19 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:19.886 INFO:tasks.workunit.client.1.smithi111.stdout:9/34: mknod d0/d4/cc 0 2024-03-23T11:47:19.887 INFO:tasks.workunit.client.0.smithi063.stdout:4/55: dwrite f2 [0,4194304] 0 2024-03-23T11:47:19.892 INFO:tasks.workunit.client.0.smithi063.stdout:4/56: fsync d5/da/fd 0 2024-03-23T11:47:19.896 INFO:tasks.workunit.client.0.smithi063.stdout:4/57: chown d5/le 102425 1 2024-03-23T11:47:19.898 INFO:tasks.workunit.client.0.smithi063.stdout:8/44: dread d6/f9 [0,4194304] 0 2024-03-23T11:47:19.899 INFO:tasks.workunit.client.1.smithi111.stdout:0/32: rmdir d5 39 2024-03-23T11:47:19.900 INFO:tasks.workunit.client.1.smithi111.stdout:6/27: rename d2/f7 to d2/fb 0 2024-03-23T11:47:19.901 INFO:tasks.workunit.client.0.smithi063.stdout:7/45: getdents d5/d9/da 0 2024-03-23T11:47:19.901 INFO:tasks.workunit.client.1.smithi111.stdout:2/20: mknod d2/c3 0 2024-03-23T11:47:19.901 INFO:tasks.workunit.client.0.smithi063.stdout:3/43: dread f3 [0,4194304] 0 2024-03-23T11:47:19.903 INFO:tasks.workunit.client.0.smithi063.stdout:8/45: stat d6 0 2024-03-23T11:47:19.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:19 smithi111 ceph-mon[44005]: pgmap v29: 65 pgs: 65 active+clean; 220 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 397 op/s 2024-03-23T11:47:19.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:19 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:19.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:19 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:19.905 INFO:tasks.workunit.client.1.smithi111.stdout:5/33: dwrite d1/d3/d6/f8 [0,4194304] 0 2024-03-23T11:47:19.906 INFO:tasks.workunit.client.0.smithi063.stdout:7/46: readlink d5/ld 0 2024-03-23T11:47:19.907 INFO:tasks.workunit.client.0.smithi063.stdout:7/47: stat d5/f8 0 2024-03-23T11:47:19.910 INFO:tasks.workunit.client.0.smithi063.stdout:3/44: rename f2 to fd 0 2024-03-23T11:47:19.911 INFO:tasks.workunit.client.0.smithi063.stdout:8/46: read f5 [584369,127105] 0 2024-03-23T11:47:19.915 INFO:tasks.workunit.client.0.smithi063.stdout:3/45: symlink le 0 2024-03-23T11:47:19.915 INFO:tasks.workunit.client.0.smithi063.stdout:8/47: read d6/f9 [3465761,7834] 0 2024-03-23T11:47:19.916 INFO:tasks.workunit.client.0.smithi063.stdout:3/46: readlink le 0 2024-03-23T11:47:19.922 INFO:tasks.workunit.client.1.smithi111.stdout:0/33: dread f4 [0,4194304] 0 2024-03-23T11:47:19.923 INFO:tasks.workunit.client.0.smithi063.stdout:4/58: dread d5/da/fd [0,4194304] 0 2024-03-23T11:47:19.924 INFO:tasks.workunit.client.1.smithi111.stdout:0/34: readlink l2 0 2024-03-23T11:47:19.924 INFO:tasks.workunit.client.0.smithi063.stdout:8/48: mkdir d6/dd/df 0 2024-03-23T11:47:19.926 INFO:tasks.workunit.client.1.smithi111.stdout:9/35: dwrite d0/d4/f9 [0,4194304] 0 2024-03-23T11:47:19.929 INFO:tasks.workunit.client.0.smithi063.stdout:7/48: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:19.930 INFO:tasks.workunit.client.0.smithi063.stdout:3/47: chown f9 57440 1 2024-03-23T11:47:19.932 INFO:tasks.workunit.client.0.smithi063.stdout:4/59: write d5/fc [2984999,80744] 0 2024-03-23T11:47:19.934 INFO:tasks.workunit.client.0.smithi063.stdout:8/49: getdents d6/dd/df 0 2024-03-23T11:47:19.938 INFO:tasks.workunit.client.1.smithi111.stdout:9/36: dread d0/d1/f5 [0,4194304] 0 2024-03-23T11:47:19.939 INFO:tasks.workunit.client.0.smithi063.stdout:7/49: creat d5/d6/fe x:0 0 0 2024-03-23T11:47:19.948 INFO:tasks.workunit.client.0.smithi063.stdout:3/48: rename f7 to ff 0 2024-03-23T11:47:19.957 INFO:tasks.workunit.client.0.smithi063.stdout:4/60: dwrite d5/da/fd [0,4194304] 0 2024-03-23T11:47:19.957 INFO:tasks.workunit.client.0.smithi063.stdout:3/49: mknod c10 0 2024-03-23T11:47:19.965 INFO:tasks.workunit.client.0.smithi063.stdout:3/50: fsync f8 0 2024-03-23T11:47:19.968 INFO:tasks.workunit.client.0.smithi063.stdout:3/51: fsync f8 0 2024-03-23T11:47:19.970 INFO:tasks.workunit.client.0.smithi063.stdout:8/50: dwrite f5 [0,4194304] 0 2024-03-23T11:47:19.980 INFO:tasks.workunit.client.0.smithi063.stdout:3/52: creat f11 x:0 0 0 2024-03-23T11:47:19.987 INFO:tasks.workunit.client.0.smithi063.stdout:7/50: dwrite d5/f8 [0,4194304] 0 2024-03-23T11:47:19.989 INFO:tasks.workunit.client.0.smithi063.stdout:8/51: creat d6/de/f10 x:0 0 0 2024-03-23T11:47:19.990 INFO:tasks.workunit.client.0.smithi063.stdout:4/61: dwrite d5/fc [0,4194304] 0 2024-03-23T11:47:19.991 INFO:tasks.workunit.client.0.smithi063.stdout:3/53: creat f12 x:0 0 0 2024-03-23T11:47:19.999 INFO:tasks.workunit.client.0.smithi063.stdout:3/54: dread f9 [0,4194304] 0 2024-03-23T11:47:20.000 INFO:tasks.workunit.client.0.smithi063.stdout:3/55: readlink le 0 2024-03-23T11:47:20.008 INFO:tasks.workunit.client.0.smithi063.stdout:8/52: creat d6/dd/df/f11 x:0 0 0 2024-03-23T11:47:20.010 INFO:tasks.workunit.client.0.smithi063.stdout:8/53: chown d6/de/f10 112016 1 2024-03-23T11:47:20.012 INFO:tasks.workunit.client.0.smithi063.stdout:8/54: stat d6/lb 0 2024-03-23T11:47:20.013 INFO:tasks.workunit.client.0.smithi063.stdout:8/55: dread - d6/de/f10 zero size 2024-03-23T11:47:20.016 INFO:tasks.workunit.client.0.smithi063.stdout:8/56: fdatasync d6/dd/df/f11 0 2024-03-23T11:47:20.017 INFO:tasks.workunit.client.0.smithi063.stdout:4/62: truncate d5/da/fd 1915576 0 2024-03-23T11:47:20.018 INFO:tasks.workunit.client.0.smithi063.stdout:3/56: write f3 [1434218,62631] 0 2024-03-23T11:47:20.020 INFO:tasks.workunit.client.0.smithi063.stdout:4/63: chown d5/l7 82040 1 2024-03-23T11:47:20.021 INFO:tasks.workunit.client.0.smithi063.stdout:3/57: fsync f12 0 2024-03-23T11:47:20.026 INFO:tasks.workunit.client.0.smithi063.stdout:3/58: truncate f12 884423 0 2024-03-23T11:47:20.029 INFO:tasks.workunit.client.0.smithi063.stdout:4/64: rename d5/da/fb to d5/ff 0 2024-03-23T11:47:20.029 INFO:tasks.workunit.client.0.smithi063.stdout:7/51: dwrite f4 [0,4194304] 0 2024-03-23T11:47:20.032 INFO:tasks.workunit.client.0.smithi063.stdout:8/57: getdents d6/dd/df 0 2024-03-23T11:47:20.039 INFO:tasks.workunit.client.0.smithi063.stdout:3/59: dread f3 [0,4194304] 0 2024-03-23T11:47:20.039 INFO:tasks.workunit.client.0.smithi063.stdout:7/52: mknod d5/d9/da/cf 0 2024-03-23T11:47:20.045 INFO:tasks.workunit.client.0.smithi063.stdout:7/53: rename d5/ld to d5/l10 0 2024-03-23T11:47:20.045 INFO:tasks.workunit.client.0.smithi063.stdout:3/60: dread f5 [0,4194304] 0 2024-03-23T11:47:20.047 INFO:tasks.workunit.client.0.smithi063.stdout:3/61: readlink lb 0 2024-03-23T11:47:20.047 INFO:tasks.workunit.client.0.smithi063.stdout:7/54: fsync d5/d6/fe 0 2024-03-23T11:47:20.049 INFO:tasks.workunit.client.0.smithi063.stdout:8/58: dread d6/f9 [0,4194304] 0 2024-03-23T11:47:20.057 INFO:tasks.workunit.client.0.smithi063.stdout:4/65: write d5/da/fd [277005,90077] 0 2024-03-23T11:47:20.062 INFO:tasks.workunit.client.1.smithi111.stdout:2/21: creat d2/f4 x:0 0 0 2024-03-23T11:47:20.062 INFO:tasks.workunit.client.1.smithi111.stdout:6/28: mknod d2/cc 0 2024-03-23T11:47:20.062 INFO:tasks.workunit.client.1.smithi111.stdout:5/34: mknod d1/d3/d6/d7/ce 0 2024-03-23T11:47:20.063 INFO:tasks.workunit.client.0.smithi063.stdout:7/55: fsync f4 0 2024-03-23T11:47:20.063 INFO:tasks.workunit.client.0.smithi063.stdout:8/59: rename d6/c8 to d6/dd/c12 0 2024-03-23T11:47:20.064 INFO:tasks.workunit.client.1.smithi111.stdout:5/35: write d1/d3/f4 [1629518,67808] 0 2024-03-23T11:47:20.064 INFO:tasks.workunit.client.1.smithi111.stdout:0/35: dread - d5/f9 zero size 2024-03-23T11:47:20.066 INFO:tasks.workunit.client.1.smithi111.stdout:2/22: symlink d2/l5 0 2024-03-23T11:47:20.067 INFO:tasks.workunit.client.1.smithi111.stdout:2/23: stat c1 0 2024-03-23T11:47:20.069 INFO:tasks.workunit.client.0.smithi063.stdout:8/60: creat d6/de/f13 x:0 0 0 2024-03-23T11:47:20.072 INFO:tasks.workunit.client.0.smithi063.stdout:4/66: write d5/ff [183129,36632] 0 2024-03-23T11:47:20.074 INFO:tasks.workunit.client.1.smithi111.stdout:9/37: link d0/d4/f9 d0/fd 0 2024-03-23T11:47:20.077 INFO:tasks.workunit.client.1.smithi111.stdout:0/36: mknod d5/ca 0 2024-03-23T11:47:20.078 INFO:tasks.workunit.client.1.smithi111.stdout:5/36: creat d1/d3/d6/ff x:0 0 0 2024-03-23T11:47:20.079 INFO:tasks.workunit.client.0.smithi063.stdout:4/67: chown d5/ff 4 1 2024-03-23T11:47:20.080 INFO:tasks.workunit.client.1.smithi111.stdout:6/29: link d2/f8 d2/fd 0 2024-03-23T11:47:20.080 INFO:tasks.workunit.client.1.smithi111.stdout:5/37: chown d1/d3/d6/d7/ce 800629271 1 2024-03-23T11:47:20.080 INFO:tasks.workunit.client.0.smithi063.stdout:3/62: dwrite ff [0,4194304] 0 2024-03-23T11:47:20.081 INFO:tasks.workunit.client.1.smithi111.stdout:2/24: creat d2/f6 x:0 0 0 2024-03-23T11:47:20.082 INFO:tasks.workunit.client.1.smithi111.stdout:5/38: write d1/d3/d6/fa [780410,98792] 0 2024-03-23T11:47:20.083 INFO:tasks.workunit.client.0.smithi063.stdout:3/63: write f6 [1853565,114549] 0 2024-03-23T11:47:20.084 INFO:tasks.workunit.client.1.smithi111.stdout:0/37: fsync f4 0 2024-03-23T11:47:20.086 INFO:tasks.workunit.client.1.smithi111.stdout:0/38: stat d5/f8 0 2024-03-23T11:47:20.087 INFO:tasks.workunit.client.1.smithi111.stdout:0/39: write d5/f9 [421153,80464] 0 2024-03-23T11:47:20.088 INFO:tasks.workunit.client.1.smithi111.stdout:0/40: chown c3 505 1 2024-03-23T11:47:20.089 INFO:tasks.workunit.client.1.smithi111.stdout:0/41: stat c3 0 2024-03-23T11:47:20.089 INFO:tasks.workunit.client.0.smithi063.stdout:4/68: truncate d5/da/fd 2675257 0 2024-03-23T11:47:20.090 INFO:tasks.workunit.client.1.smithi111.stdout:0/42: write d5/f8 [163016,13184] 0 2024-03-23T11:47:20.090 INFO:tasks.workunit.client.1.smithi111.stdout:2/25: mkdir d2/d7 0 2024-03-23T11:47:20.091 INFO:tasks.workunit.client.1.smithi111.stdout:9/38: link d0/d4/f9 d0/d4/fe 0 2024-03-23T11:47:20.092 INFO:tasks.workunit.client.1.smithi111.stdout:5/39: creat d1/d3/d6/d7/f10 x:0 0 0 2024-03-23T11:47:20.092 INFO:tasks.workunit.client.1.smithi111.stdout:0/43: readlink d5/l6 0 2024-03-23T11:47:20.092 INFO:tasks.workunit.client.0.smithi063.stdout:3/64: mkdir d13 0 2024-03-23T11:47:20.093 INFO:tasks.workunit.client.0.smithi063.stdout:4/69: fsync d5/da/fd 0 2024-03-23T11:47:20.095 INFO:tasks.workunit.client.0.smithi063.stdout:4/70: readlink l3 0 2024-03-23T11:47:20.095 INFO:tasks.workunit.client.1.smithi111.stdout:5/40: dread - d1/d3/fc zero size 2024-03-23T11:47:20.096 INFO:tasks.workunit.client.1.smithi111.stdout:6/30: link d2/c6 d2/ce 0 2024-03-23T11:47:20.097 INFO:tasks.workunit.client.1.smithi111.stdout:2/26: mknod d2/d7/c8 0 2024-03-23T11:47:20.099 INFO:tasks.workunit.client.0.smithi063.stdout:7/56: dwrite f4 [0,4194304] 0 2024-03-23T11:47:20.100 INFO:tasks.workunit.client.0.smithi063.stdout:3/65: mknod d13/c14 0 2024-03-23T11:47:20.103 INFO:tasks.workunit.client.0.smithi063.stdout:7/57: fdatasync d5/d6/fe 0 2024-03-23T11:47:20.105 INFO:tasks.workunit.client.0.smithi063.stdout:8/61: dwrite d6/dd/df/f11 [0,4194304] 0 2024-03-23T11:47:20.106 INFO:tasks.workunit.client.0.smithi063.stdout:8/62: read - d6/de/f13 zero size 2024-03-23T11:47:20.108 INFO:tasks.workunit.client.1.smithi111.stdout:6/31: creat d2/ff x:0 0 0 2024-03-23T11:47:20.109 INFO:tasks.workunit.client.1.smithi111.stdout:5/41: mknod d1/d3/d6/c11 0 2024-03-23T11:47:20.110 INFO:tasks.workunit.client.0.smithi063.stdout:7/58: symlink d5/d9/da/l11 0 2024-03-23T11:47:20.110 INFO:tasks.workunit.client.0.smithi063.stdout:3/66: write f3 [2231815,117432] 0 2024-03-23T11:47:20.111 INFO:tasks.workunit.client.0.smithi063.stdout:8/63: creat d6/dd/df/f14 x:0 0 0 2024-03-23T11:47:20.112 INFO:tasks.workunit.client.1.smithi111.stdout:2/27: fsync f0 0 2024-03-23T11:47:20.116 INFO:tasks.workunit.client.0.smithi063.stdout:8/64: mkdir d6/de/d15 0 2024-03-23T11:47:20.119 INFO:tasks.workunit.client.0.smithi063.stdout:8/65: creat d6/f16 x:0 0 0 2024-03-23T11:47:20.123 INFO:tasks.workunit.client.0.smithi063.stdout:8/66: write d6/de/f13 [508448,37563] 0 2024-03-23T11:47:20.127 INFO:tasks.workunit.client.1.smithi111.stdout:6/32: dwrite d2/ff [0,4194304] 0 2024-03-23T11:47:20.128 INFO:tasks.workunit.client.1.smithi111.stdout:6/33: chown d2/l4 136027 1 2024-03-23T11:47:20.128 INFO:tasks.workunit.client.0.smithi063.stdout:8/67: creat d6/de/d15/f17 x:0 0 0 2024-03-23T11:47:20.134 INFO:tasks.workunit.client.0.smithi063.stdout:3/67: dwrite ff [0,4194304] 0 2024-03-23T11:47:20.135 INFO:tasks.workunit.client.0.smithi063.stdout:8/68: creat d6/dd/df/f18 x:0 0 0 2024-03-23T11:47:20.136 INFO:tasks.workunit.client.0.smithi063.stdout:3/68: chown ff 3 1 2024-03-23T11:47:20.138 INFO:tasks.workunit.client.0.smithi063.stdout:3/69: write f6 [948029,128001] 0 2024-03-23T11:47:20.141 INFO:tasks.workunit.client.0.smithi063.stdout:8/69: creat d6/dd/df/f19 x:0 0 0 2024-03-23T11:47:20.145 INFO:tasks.workunit.client.0.smithi063.stdout:8/70: creat d6/de/f1a x:0 0 0 2024-03-23T11:47:20.147 INFO:tasks.workunit.client.0.smithi063.stdout:8/71: truncate d6/f16 374052 0 2024-03-23T11:47:20.148 INFO:tasks.workunit.client.0.smithi063.stdout:8/72: chown f1 696 1 2024-03-23T11:47:20.151 INFO:tasks.workunit.client.0.smithi063.stdout:8/73: mknod d6/de/c1b 0 2024-03-23T11:47:20.153 INFO:tasks.workunit.client.0.smithi063.stdout:3/70: dread f6 [4194304,4194304] 0 2024-03-23T11:47:20.154 INFO:tasks.workunit.client.0.smithi063.stdout:8/74: truncate d6/de/f1a 44036 0 2024-03-23T11:47:20.156 INFO:tasks.workunit.client.0.smithi063.stdout:3/71: mkdir d13/d15 0 2024-03-23T11:47:20.157 INFO:tasks.workunit.client.0.smithi063.stdout:8/75: mkdir d6/d1c 0 2024-03-23T11:47:20.161 INFO:tasks.workunit.client.0.smithi063.stdout:8/76: unlink l4 0 2024-03-23T11:47:20.161 INFO:tasks.workunit.client.0.smithi063.stdout:3/72: link lb d13/l16 0 2024-03-23T11:47:20.163 INFO:tasks.workunit.client.0.smithi063.stdout:3/73: chown d13/c14 53025 1 2024-03-23T11:47:20.164 INFO:tasks.workunit.client.0.smithi063.stdout:8/77: creat d6/d1c/f1d x:0 0 0 2024-03-23T11:47:20.165 INFO:tasks.workunit.client.0.smithi063.stdout:3/74: mkdir d13/d15/d17 0 2024-03-23T11:47:20.430 INFO:tasks.workunit.client.1.smithi111.stdout:0/44: truncate f4 562941 0 2024-03-23T11:47:20.433 INFO:tasks.workunit.client.1.smithi111.stdout:9/39: getdents d0/d4 0 2024-03-23T11:47:20.434 INFO:tasks.workunit.client.1.smithi111.stdout:5/42: rmdir d1/d3 39 2024-03-23T11:47:20.434 INFO:tasks.workunit.client.1.smithi111.stdout:6/34: getdents d2 0 2024-03-23T11:47:20.434 INFO:tasks.workunit.client.1.smithi111.stdout:0/45: mkdir d5/db 0 2024-03-23T11:47:20.438 INFO:tasks.workunit.client.1.smithi111.stdout:6/35: truncate d2/fa 296393 0 2024-03-23T11:47:20.438 INFO:tasks.workunit.client.1.smithi111.stdout:9/40: creat d0/d4/ff x:0 0 0 2024-03-23T11:47:20.438 INFO:tasks.workunit.client.0.smithi063.stdout:7/59: truncate f1 672592 0 2024-03-23T11:47:20.440 INFO:tasks.workunit.client.1.smithi111.stdout:5/43: symlink d1/d3/d6/l12 0 2024-03-23T11:47:20.441 INFO:tasks.workunit.client.1.smithi111.stdout:5/44: readlink l0 0 2024-03-23T11:47:20.442 INFO:tasks.workunit.client.1.smithi111.stdout:6/36: stat d2/l9 0 2024-03-23T11:47:20.442 INFO:tasks.workunit.client.0.smithi063.stdout:7/60: creat d5/f12 x:0 0 0 2024-03-23T11:47:20.443 INFO:tasks.workunit.client.1.smithi111.stdout:9/41: mkdir d0/d4/d10 0 2024-03-23T11:47:20.444 INFO:tasks.workunit.client.0.smithi063.stdout:7/61: dread - d5/d6/fe zero size 2024-03-23T11:47:20.444 INFO:tasks.workunit.client.1.smithi111.stdout:5/45: rmdir d1/d3 39 2024-03-23T11:47:20.446 INFO:tasks.workunit.client.0.smithi063.stdout:3/75: getdents d13/d15 0 2024-03-23T11:47:20.446 INFO:tasks.workunit.client.1.smithi111.stdout:6/37: write d2/ff [4216623,113798] 0 2024-03-23T11:47:20.447 INFO:tasks.workunit.client.1.smithi111.stdout:9/42: fdatasync d0/d4/ff 0 2024-03-23T11:47:20.448 INFO:tasks.workunit.client.0.smithi063.stdout:7/62: mknod d5/c13 0 2024-03-23T11:47:20.448 INFO:tasks.workunit.client.0.smithi063.stdout:3/76: write f8 [1662096,1534] 0 2024-03-23T11:47:20.449 INFO:tasks.workunit.client.0.smithi063.stdout:8/78: truncate f5 3669452 0 2024-03-23T11:47:20.449 INFO:tasks.workunit.client.1.smithi111.stdout:6/38: read - d2/fd zero size 2024-03-23T11:47:20.449 INFO:tasks.workunit.client.0.smithi063.stdout:7/63: stat d5/d9 0 2024-03-23T11:47:20.451 INFO:tasks.workunit.client.1.smithi111.stdout:5/46: dread - d1/d3/d6/d7/f10 zero size 2024-03-23T11:47:20.453 INFO:tasks.workunit.client.1.smithi111.stdout:5/47: fdatasync d1/d3/d6/f8 0 2024-03-23T11:47:20.455 INFO:tasks.workunit.client.0.smithi063.stdout:8/79: creat d6/d1c/f1e x:0 0 0 2024-03-23T11:47:20.456 INFO:tasks.workunit.client.1.smithi111.stdout:5/48: rmdir d1/d3 39 2024-03-23T11:47:20.456 INFO:tasks.workunit.client.0.smithi063.stdout:8/80: chown d6/de/c1b 439771757 1 2024-03-23T11:47:20.461 INFO:tasks.workunit.client.0.smithi063.stdout:3/77: fsync fc 0 2024-03-23T11:47:20.461 INFO:tasks.workunit.client.0.smithi063.stdout:8/81: dread d6/f16 [0,4194304] 0 2024-03-23T11:47:20.462 INFO:tasks.workunit.client.1.smithi111.stdout:5/49: creat d1/d3/d6/d7/f13 x:0 0 0 2024-03-23T11:47:20.462 INFO:tasks.workunit.client.0.smithi063.stdout:3/78: chown f6 0 1 2024-03-23T11:47:20.464 INFO:tasks.workunit.client.0.smithi063.stdout:3/79: fdatasync f11 0 2024-03-23T11:47:20.465 INFO:tasks.workunit.client.0.smithi063.stdout:8/82: rmdir d6/dd/df 39 2024-03-23T11:47:20.470 INFO:tasks.workunit.client.1.smithi111.stdout:0/46: truncate f4 50222 0 2024-03-23T11:47:20.470 INFO:tasks.workunit.client.1.smithi111.stdout:9/43: dread d0/d4/f9 [0,4194304] 0 2024-03-23T11:47:20.470 INFO:tasks.workunit.client.0.smithi063.stdout:3/80: write f9 [1249234,86330] 0 2024-03-23T11:47:20.471 INFO:tasks.workunit.client.0.smithi063.stdout:7/64: dwrite f4 [0,4194304] 0 2024-03-23T11:47:20.473 INFO:tasks.workunit.client.0.smithi063.stdout:8/83: truncate d6/dd/df/f18 717169 0 2024-03-23T11:47:20.474 INFO:tasks.workunit.client.0.smithi063.stdout:7/65: symlink d5/l14 0 2024-03-23T11:47:20.475 INFO:tasks.workunit.client.1.smithi111.stdout:0/47: chown f4 170 1 2024-03-23T11:47:20.477 INFO:tasks.workunit.client.0.smithi063.stdout:3/81: dread f12 [0,4194304] 0 2024-03-23T11:47:20.478 INFO:tasks.workunit.client.0.smithi063.stdout:8/84: mkdir d6/d1f 0 2024-03-23T11:47:20.478 INFO:tasks.workunit.client.0.smithi063.stdout:7/66: chown d5/d9/da/cb 241 1 2024-03-23T11:47:20.479 INFO:tasks.workunit.client.0.smithi063.stdout:3/82: fsync fc 0 2024-03-23T11:47:20.479 INFO:tasks.workunit.client.1.smithi111.stdout:5/50: getdents d1/d3/d6/d7 0 2024-03-23T11:47:20.481 INFO:tasks.workunit.client.0.smithi063.stdout:7/67: write f4 [4381654,4874] 0 2024-03-23T11:47:20.482 INFO:tasks.workunit.client.1.smithi111.stdout:6/39: dwrite d2/fa [0,4194304] 0 2024-03-23T11:47:20.483 INFO:tasks.workunit.client.1.smithi111.stdout:6/40: chown d2/fb 186657 1 2024-03-23T11:47:20.485 INFO:tasks.workunit.client.0.smithi063.stdout:8/85: write d6/f16 [358141,127798] 0 2024-03-23T11:47:20.486 INFO:tasks.workunit.client.1.smithi111.stdout:6/41: readlink d2/l4 0 2024-03-23T11:47:20.486 INFO:tasks.workunit.client.1.smithi111.stdout:5/51: write d1/d3/d6/d7/f10 [850405,59625] 0 2024-03-23T11:47:20.488 INFO:tasks.workunit.client.1.smithi111.stdout:6/42: chown d2/cc 2621 1 2024-03-23T11:47:20.488 INFO:tasks.workunit.client.1.smithi111.stdout:5/52: chown d1/d3 58963 1 2024-03-23T11:47:20.494 INFO:tasks.workunit.client.0.smithi063.stdout:8/86: read d6/de/f13 [450572,105041] 0 2024-03-23T11:47:20.494 INFO:tasks.workunit.client.0.smithi063.stdout:3/83: dread f5 [0,4194304] 0 2024-03-23T11:47:20.495 INFO:tasks.workunit.client.1.smithi111.stdout:6/43: read d2/f5 [150659,75820] 0 2024-03-23T11:47:20.498 INFO:tasks.workunit.client.1.smithi111.stdout:6/44: rmdir d2 39 2024-03-23T11:47:20.503 INFO:tasks.workunit.client.1.smithi111.stdout:9/44: dread d0/d1/f5 [0,4194304] 0 2024-03-23T11:47:20.504 INFO:tasks.workunit.client.1.smithi111.stdout:0/48: dwrite d5/f9 [0,4194304] 0 2024-03-23T11:47:20.507 INFO:tasks.workunit.client.0.smithi063.stdout:8/87: rename d6/lb to d6/d1c/l20 0 2024-03-23T11:47:20.511 INFO:tasks.workunit.client.1.smithi111.stdout:6/45: symlink d2/l10 0 2024-03-23T11:47:20.512 INFO:tasks.workunit.client.1.smithi111.stdout:5/53: dwrite d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:20.515 INFO:tasks.workunit.client.1.smithi111.stdout:6/46: creat d2/f11 x:0 0 0 2024-03-23T11:47:20.520 INFO:tasks.workunit.client.0.smithi063.stdout:7/68: dwrite f4 [0,4194304] 0 2024-03-23T11:47:20.521 INFO:tasks.workunit.client.1.smithi111.stdout:5/54: rename d1/d3/d6/d7/lb to d1/l14 0 2024-03-23T11:47:20.525 INFO:tasks.workunit.client.0.smithi063.stdout:3/84: dwrite f5 [0,4194304] 0 2024-03-23T11:47:20.527 INFO:tasks.workunit.client.1.smithi111.stdout:5/55: write d1/d3/d6/f8 [5034305,6150] 0 2024-03-23T11:47:20.542 INFO:tasks.workunit.client.1.smithi111.stdout:6/47: dwrite d2/f11 [0,4194304] 0 2024-03-23T11:47:20.543 INFO:tasks.workunit.client.1.smithi111.stdout:6/48: chown d2 1722892 1 2024-03-23T11:47:20.544 INFO:tasks.workunit.client.1.smithi111.stdout:6/49: fdatasync d2/ff 0 2024-03-23T11:47:20.550 INFO:tasks.workunit.client.1.smithi111.stdout:5/56: dread d1/d3/d6/d7/f10 [0,4194304] 0 2024-03-23T11:47:20.566 INFO:tasks.workunit.client.1.smithi111.stdout:6/50: dwrite d2/fa [4194304,4194304] 0 2024-03-23T11:47:20.621 INFO:tasks.workunit.client.0.smithi063.stdout:8/88: mknod d6/d1f/c21 0 2024-03-23T11:47:20.622 INFO:tasks.workunit.client.1.smithi111.stdout:0/49: dread f4 [0,4194304] 0 2024-03-23T11:47:20.625 INFO:tasks.workunit.client.0.smithi063.stdout:3/85: symlink d13/d15/d17/l18 0 2024-03-23T11:47:20.625 INFO:tasks.workunit.client.1.smithi111.stdout:0/50: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:20.626 INFO:tasks.workunit.client.0.smithi063.stdout:8/89: symlink d6/d1c/l22 0 2024-03-23T11:47:20.626 INFO:tasks.workunit.client.1.smithi111.stdout:0/51: fsync d5/f9 0 2024-03-23T11:47:20.628 INFO:tasks.workunit.client.0.smithi063.stdout:8/90: chown d6/d1c/f1d 57 1 2024-03-23T11:47:20.629 INFO:tasks.workunit.client.1.smithi111.stdout:9/45: write d0/d1/f5 [1024052,12462] 0 2024-03-23T11:47:20.630 INFO:tasks.workunit.client.1.smithi111.stdout:5/57: rmdir d1/d3 39 2024-03-23T11:47:20.631 INFO:tasks.workunit.client.0.smithi063.stdout:8/91: write d6/f16 [556554,34079] 0 2024-03-23T11:47:20.631 INFO:tasks.workunit.client.1.smithi111.stdout:9/46: chown d0/fd 0 1 2024-03-23T11:47:20.631 INFO:tasks.workunit.client.1.smithi111.stdout:6/51: symlink d2/l12 0 2024-03-23T11:47:20.633 INFO:tasks.workunit.client.1.smithi111.stdout:6/52: chown d2/f11 57532722 1 2024-03-23T11:47:20.637 INFO:tasks.workunit.client.1.smithi111.stdout:0/52: creat d5/fc x:0 0 0 2024-03-23T11:47:20.638 INFO:tasks.workunit.client.0.smithi063.stdout:8/92: rename d6/d1c/l22 to d6/dd/l23 0 2024-03-23T11:47:20.641 INFO:tasks.workunit.client.0.smithi063.stdout:8/93: write d6/f9 [2960247,118258] 0 2024-03-23T11:47:20.642 INFO:tasks.workunit.client.1.smithi111.stdout:6/53: symlink d2/l13 0 2024-03-23T11:47:20.643 INFO:tasks.workunit.client.1.smithi111.stdout:0/53: read d5/f8 [107009,101306] 0 2024-03-23T11:47:20.644 INFO:tasks.workunit.client.1.smithi111.stdout:9/47: link d0/d1/f5 d0/d1/f11 0 2024-03-23T11:47:20.645 INFO:tasks.workunit.client.1.smithi111.stdout:6/54: mknod d2/c14 0 2024-03-23T11:47:20.646 INFO:tasks.workunit.client.0.smithi063.stdout:8/94: link d6/f16 d6/de/d15/f24 0 2024-03-23T11:47:20.646 INFO:tasks.workunit.client.1.smithi111.stdout:0/54: truncate d5/fc 52076 0 2024-03-23T11:47:20.648 INFO:tasks.workunit.client.0.smithi063.stdout:8/95: creat d6/d1f/f25 x:0 0 0 2024-03-23T11:47:20.649 INFO:tasks.workunit.client.1.smithi111.stdout:9/48: write d0/d4/fe [2322838,34521] 0 2024-03-23T11:47:20.652 INFO:tasks.workunit.client.1.smithi111.stdout:6/55: rename d2/c14 to d2/c15 0 2024-03-23T11:47:20.657 INFO:tasks.workunit.client.1.smithi111.stdout:6/56: rename d2/f5 to d2/f16 0 2024-03-23T11:47:20.661 INFO:tasks.workunit.client.1.smithi111.stdout:6/57: fsync d2/ff 0 2024-03-23T11:47:20.661 INFO:tasks.workunit.client.1.smithi111.stdout:5/58: dwrite d1/d3/d6/d7/f13 [0,4194304] 0 2024-03-23T11:47:20.667 INFO:tasks.workunit.client.1.smithi111.stdout:5/59: write d1/d3/fc [629460,37169] 0 2024-03-23T11:47:20.668 INFO:tasks.workunit.client.0.smithi063.stdout:8/96: dwrite d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:20.674 INFO:tasks.workunit.client.1.smithi111.stdout:9/49: dread d0/d4/f9 [0,4194304] 0 2024-03-23T11:47:20.674 INFO:tasks.workunit.client.0.smithi063.stdout:8/97: stat d6/d1c/f1d 0 2024-03-23T11:47:20.675 INFO:tasks.workunit.client.1.smithi111.stdout:0/55: dwrite f4 [0,4194304] 0 2024-03-23T11:47:20.678 INFO:tasks.workunit.client.1.smithi111.stdout:6/58: fsync d2/f16 0 2024-03-23T11:47:20.680 INFO:tasks.workunit.client.1.smithi111.stdout:9/50: mknod d0/d4/c12 0 2024-03-23T11:47:20.683 INFO:tasks.workunit.client.1.smithi111.stdout:6/59: mkdir d2/d17 0 2024-03-23T11:47:20.684 INFO:tasks.workunit.client.1.smithi111.stdout:5/60: link d1/c5 d1/c15 0 2024-03-23T11:47:20.685 INFO:tasks.workunit.client.1.smithi111.stdout:6/60: truncate d2/f11 4737641 0 2024-03-23T11:47:20.686 INFO:tasks.workunit.client.1.smithi111.stdout:9/51: creat d0/f13 x:0 0 0 2024-03-23T11:47:20.690 INFO:tasks.workunit.client.1.smithi111.stdout:5/61: creat d1/d3/d6/d7/f16 x:0 0 0 2024-03-23T11:47:20.691 INFO:tasks.workunit.client.1.smithi111.stdout:6/61: creat d2/f18 x:0 0 0 2024-03-23T11:47:20.692 INFO:tasks.workunit.client.1.smithi111.stdout:6/62: fdatasync d2/f16 0 2024-03-23T11:47:20.693 INFO:tasks.workunit.client.1.smithi111.stdout:9/52: unlink d0/d4/ff 0 2024-03-23T11:47:20.696 INFO:tasks.workunit.client.1.smithi111.stdout:9/53: creat d0/d4/f14 x:0 0 0 2024-03-23T11:47:20.697 INFO:tasks.workunit.client.1.smithi111.stdout:5/62: creat d1/f17 x:0 0 0 2024-03-23T11:47:20.698 INFO:tasks.workunit.client.1.smithi111.stdout:6/63: symlink d2/d17/l19 0 2024-03-23T11:47:20.698 INFO:tasks.workunit.client.1.smithi111.stdout:6/64: dread - d2/f18 zero size 2024-03-23T11:47:20.700 INFO:tasks.workunit.client.1.smithi111.stdout:5/63: creat d1/d3/f18 x:0 0 0 2024-03-23T11:47:20.702 INFO:tasks.workunit.client.1.smithi111.stdout:6/65: write d2/fb [4229436,107322] 0 2024-03-23T11:47:20.702 INFO:tasks.workunit.client.1.smithi111.stdout:5/64: stat d1/d3/d6/f8 0 2024-03-23T11:47:20.703 INFO:tasks.workunit.client.1.smithi111.stdout:6/66: dread - d2/fd zero size 2024-03-23T11:47:20.703 INFO:tasks.workunit.client.1.smithi111.stdout:6/67: stat d2/f18 0 2024-03-23T11:47:20.707 INFO:tasks.workunit.client.1.smithi111.stdout:5/65: creat d1/d3/d6/f19 x:0 0 0 2024-03-23T11:47:20.708 INFO:tasks.workunit.client.1.smithi111.stdout:9/54: write d0/d1/f11 [3352799,20443] 0 2024-03-23T11:47:20.711 INFO:tasks.workunit.client.1.smithi111.stdout:5/66: symlink d1/d3/d6/l1a 0 2024-03-23T11:47:20.713 INFO:tasks.workunit.client.1.smithi111.stdout:9/55: creat d0/d4/f15 x:0 0 0 2024-03-23T11:47:20.715 INFO:tasks.workunit.client.0.smithi063.stdout:7/69: rmdir d5 39 2024-03-23T11:47:20.718 INFO:tasks.workunit.client.0.smithi063.stdout:7/70: fsync d5/f8 0 2024-03-23T11:47:20.720 INFO:tasks.workunit.client.0.smithi063.stdout:7/71: mknod d5/d9/c15 0 2024-03-23T11:47:20.721 INFO:tasks.workunit.client.1.smithi111.stdout:6/68: dwrite d2/f11 [0,4194304] 0 2024-03-23T11:47:20.722 INFO:tasks.workunit.client.0.smithi063.stdout:3/86: truncate f9 221615 0 2024-03-23T11:47:20.723 INFO:tasks.workunit.client.0.smithi063.stdout:7/72: rmdir d5/d9/da 39 2024-03-23T11:47:20.724 INFO:tasks.workunit.client.1.smithi111.stdout:6/69: unlink d2/l12 0 2024-03-23T11:47:20.724 INFO:tasks.workunit.client.0.smithi063.stdout:7/73: fdatasync d5/d6/fe 0 2024-03-23T11:47:20.728 INFO:tasks.workunit.client.0.smithi063.stdout:8/98: rename d6/de to d6/d1f/d26 0 2024-03-23T11:47:20.731 INFO:tasks.workunit.client.1.smithi111.stdout:6/70: getdents d2/d17 0 2024-03-23T11:47:20.731 INFO:tasks.workunit.client.0.smithi063.stdout:8/99: write d6/d1c/f1d [5171470,28388] 0 2024-03-23T11:47:20.734 INFO:tasks.workunit.client.1.smithi111.stdout:5/67: dwrite d1/d3/fc [0,4194304] 0 2024-03-23T11:47:20.736 INFO:tasks.workunit.client.0.smithi063.stdout:8/100: truncate d6/d1f/d26/f1a 290430 0 2024-03-23T11:47:20.736 INFO:tasks.workunit.client.0.smithi063.stdout:3/87: dread f3 [0,4194304] 0 2024-03-23T11:47:20.736 INFO:tasks.workunit.client.1.smithi111.stdout:6/71: mkdir d2/d1a 0 2024-03-23T11:47:20.736 INFO:tasks.workunit.client.1.smithi111.stdout:6/72: stat d2/fa 0 2024-03-23T11:47:20.737 INFO:tasks.workunit.client.0.smithi063.stdout:3/88: chown fd 91 1 2024-03-23T11:47:20.739 INFO:tasks.workunit.client.1.smithi111.stdout:5/68: symlink d1/d3/d6/d7/l1b 0 2024-03-23T11:47:20.742 INFO:tasks.workunit.client.1.smithi111.stdout:5/69: write d1/d3/d6/ff [274471,89190] 0 2024-03-23T11:47:20.746 INFO:tasks.workunit.client.0.smithi063.stdout:3/89: write f3 [1169225,45098] 0 2024-03-23T11:47:20.747 INFO:tasks.workunit.client.0.smithi063.stdout:3/90: rename d13/d15 to d13/d15/d17/d19 22 2024-03-23T11:47:20.750 INFO:tasks.workunit.client.0.smithi063.stdout:3/91: fdatasync f12 0 2024-03-23T11:47:20.753 INFO:tasks.workunit.client.0.smithi063.stdout:3/92: mknod d13/d15/c1a 0 2024-03-23T11:47:20.753 INFO:tasks.workunit.client.0.smithi063.stdout:8/101: dwrite d6/d1c/f1e [0,4194304] 0 2024-03-23T11:47:20.755 INFO:tasks.workunit.client.0.smithi063.stdout:3/93: read fc [530405,76144] 0 2024-03-23T11:47:20.756 INFO:tasks.workunit.client.0.smithi063.stdout:8/102: read d6/f9 [166249,130682] 0 2024-03-23T11:47:20.759 INFO:tasks.workunit.client.0.smithi063.stdout:3/94: creat d13/d15/f1b x:0 0 0 2024-03-23T11:47:20.760 INFO:tasks.workunit.client.1.smithi111.stdout:6/73: dwrite d2/fb [4194304,4194304] 0 2024-03-23T11:47:20.760 INFO:tasks.workunit.client.0.smithi063.stdout:8/103: write d6/d1f/d26/f1a [934198,85152] 0 2024-03-23T11:47:20.764 INFO:tasks.workunit.client.0.smithi063.stdout:8/104: mkdir d6/d1c/d27 0 2024-03-23T11:47:20.765 INFO:tasks.workunit.client.1.smithi111.stdout:6/74: write d2/f18 [233334,40090] 0 2024-03-23T11:47:20.766 INFO:tasks.workunit.client.1.smithi111.stdout:5/70: dwrite d1/d3/d6/fa [0,4194304] 0 2024-03-23T11:47:20.771 INFO:tasks.workunit.client.1.smithi111.stdout:6/75: mknod d2/c1b 0 2024-03-23T11:47:20.772 INFO:tasks.workunit.client.0.smithi063.stdout:3/95: dread f5 [0,4194304] 0 2024-03-23T11:47:20.772 INFO:tasks.workunit.client.1.smithi111.stdout:5/71: symlink d1/d3/d6/l1c 0 2024-03-23T11:47:20.774 INFO:tasks.workunit.client.1.smithi111.stdout:5/72: dread - d1/d3/f18 zero size 2024-03-23T11:47:20.775 INFO:tasks.workunit.client.0.smithi063.stdout:8/105: link d6/d1f/d26/d15/f17 d6/d1f/d26/f28 0 2024-03-23T11:47:20.782 INFO:tasks.workunit.client.1.smithi111.stdout:5/73: rename d1/d3/d6/d7/ld to d1/d3/d6/l1d 0 2024-03-23T11:47:20.784 INFO:tasks.workunit.client.1.smithi111.stdout:6/76: dread d2/fb [0,4194304] 0 2024-03-23T11:47:20.784 INFO:tasks.workunit.client.0.smithi063.stdout:8/106: symlink d6/d1f/l29 0 2024-03-23T11:47:20.789 INFO:tasks.workunit.client.1.smithi111.stdout:5/74: mkdir d1/d1e 0 2024-03-23T11:47:20.791 INFO:tasks.workunit.client.0.smithi063.stdout:8/107: mknod d6/dd/c2a 0 2024-03-23T11:47:20.791 INFO:tasks.workunit.client.1.smithi111.stdout:5/75: write d1/d3/d6/d7/f9 [1649557,4983] 0 2024-03-23T11:47:20.795 INFO:tasks.workunit.client.0.smithi063.stdout:8/108: chown d6/dd/df/f19 128467 1 2024-03-23T11:47:20.797 INFO:tasks.workunit.client.0.smithi063.stdout:8/109: mknod d6/d1c/c2b 0 2024-03-23T11:47:20.798 INFO:tasks.workunit.client.0.smithi063.stdout:3/96: dwrite ff [0,4194304] 0 2024-03-23T11:47:20.799 INFO:tasks.workunit.client.0.smithi063.stdout:8/110: dread - d6/d1f/f25 zero size 2024-03-23T11:47:20.800 INFO:tasks.workunit.client.0.smithi063.stdout:3/97: write fc [1816618,101836] 0 2024-03-23T11:47:20.801 INFO:tasks.workunit.client.0.smithi063.stdout:8/111: chown d6/d1f/d26/c1b 366246 1 2024-03-23T11:47:20.802 INFO:tasks.workunit.client.1.smithi111.stdout:6/77: fsync d2/fb 0 2024-03-23T11:47:20.805 INFO:tasks.workunit.client.1.smithi111.stdout:6/78: rmdir d2 39 2024-03-23T11:47:20.806 INFO:tasks.workunit.client.1.smithi111.stdout:5/76: dwrite d1/d3/fc [0,4194304] 0 2024-03-23T11:47:20.806 INFO:tasks.workunit.client.0.smithi063.stdout:8/112: creat d6/dd/df/f2c x:0 0 0 2024-03-23T11:47:20.808 INFO:tasks.workunit.client.0.smithi063.stdout:8/113: chown d6/dd/c12 9 1 2024-03-23T11:47:20.810 INFO:tasks.workunit.client.1.smithi111.stdout:5/77: rmdir d1/d3/d6/d7 39 2024-03-23T11:47:20.810 INFO:tasks.workunit.client.0.smithi063.stdout:8/114: chown d6/dd/c12 39 1 2024-03-23T11:47:20.811 INFO:tasks.workunit.client.1.smithi111.stdout:6/79: dread - d2/f8 zero size 2024-03-23T11:47:20.812 INFO:tasks.workunit.client.1.smithi111.stdout:5/78: fsync d1/d3/d6/f8 0 2024-03-23T11:47:20.812 INFO:tasks.workunit.client.0.smithi063.stdout:3/98: getdents d13/d15 0 2024-03-23T11:47:20.814 INFO:tasks.workunit.client.0.smithi063.stdout:3/99: write f8 [1669468,17040] 0 2024-03-23T11:47:20.814 INFO:tasks.workunit.client.1.smithi111.stdout:5/79: fdatasync d1/d3/d6/ff 0 2024-03-23T11:47:20.815 INFO:tasks.workunit.client.1.smithi111.stdout:5/80: truncate d1/f17 252819 0 2024-03-23T11:47:20.820 INFO:tasks.workunit.client.0.smithi063.stdout:8/115: readlink d6/d1c/l20 0 2024-03-23T11:47:20.820 INFO:tasks.workunit.client.0.smithi063.stdout:3/100: creat d13/d15/f1c x:0 0 0 2024-03-23T11:47:20.821 INFO:tasks.workunit.client.0.smithi063.stdout:8/116: creat d6/d1c/f2d x:0 0 0 2024-03-23T11:47:20.823 INFO:tasks.workunit.client.0.smithi063.stdout:3/101: write d13/d15/f1c [693792,82776] 0 2024-03-23T11:47:20.826 INFO:tasks.workunit.client.0.smithi063.stdout:8/117: dread - d6/d1f/d26/f10 zero size 2024-03-23T11:47:20.827 INFO:tasks.workunit.client.0.smithi063.stdout:3/102: write fd [2133120,9160] 0 2024-03-23T11:47:20.828 INFO:tasks.workunit.client.1.smithi111.stdout:5/81: dread d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:20.831 INFO:tasks.workunit.client.0.smithi063.stdout:3/103: symlink d13/d15/d17/l1d 0 2024-03-23T11:47:20.835 INFO:tasks.workunit.client.0.smithi063.stdout:3/104: symlink d13/d15/l1e 0 2024-03-23T11:47:20.842 INFO:tasks.workunit.client.0.smithi063.stdout:8/118: truncate d6/dd/df/f11 2782460 0 2024-03-23T11:47:20.843 INFO:tasks.workunit.client.1.smithi111.stdout:6/80: mkdir d2/d1a/d1c 0 2024-03-23T11:47:20.846 INFO:tasks.workunit.client.1.smithi111.stdout:5/82: link d1/d3/fc d1/d3/d6/d7/f1f 0 2024-03-23T11:47:20.847 INFO:tasks.workunit.client.0.smithi063.stdout:8/119: truncate d6/dd/df/f2c 744043 0 2024-03-23T11:47:20.849 INFO:tasks.workunit.client.0.smithi063.stdout:8/120: symlink d6/l2e 0 2024-03-23T11:47:20.853 INFO:tasks.workunit.client.0.smithi063.stdout:3/105: dwrite ff [0,4194304] 0 2024-03-23T11:47:20.857 INFO:tasks.workunit.client.0.smithi063.stdout:3/106: unlink f11 0 2024-03-23T11:47:20.863 INFO:tasks.workunit.client.1.smithi111.stdout:6/81: dwrite d2/f16 [0,4194304] 0 2024-03-23T11:47:20.867 INFO:tasks.workunit.client.1.smithi111.stdout:6/82: write d2/fb [7295576,38262] 0 2024-03-23T11:47:20.870 INFO:tasks.workunit.client.1.smithi111.stdout:6/83: chown d2/fa 0 1 2024-03-23T11:47:20.872 INFO:tasks.workunit.client.0.smithi063.stdout:3/107: dread f5 [0,4194304] 0 2024-03-23T11:47:20.874 INFO:tasks.workunit.client.0.smithi063.stdout:3/108: fsync fd 0 2024-03-23T11:47:20.878 INFO:tasks.workunit.client.0.smithi063.stdout:3/109: readlink d13/d15/d17/l1d 0 2024-03-23T11:47:20.878 INFO:tasks.workunit.client.1.smithi111.stdout:6/84: creat d2/d1a/f1d x:0 0 0 2024-03-23T11:47:20.878 INFO:tasks.workunit.client.1.smithi111.stdout:5/83: dwrite d1/d3/f4 [4194304,4194304] 0 2024-03-23T11:47:20.879 INFO:tasks.workunit.client.1.smithi111.stdout:5/84: chown d1/c5 838 1 2024-03-23T11:47:20.880 INFO:tasks.workunit.client.0.smithi063.stdout:3/110: dread f5 [0,4194304] 0 2024-03-23T11:47:20.881 INFO:tasks.workunit.client.0.smithi063.stdout:3/111: fsync f8 0 2024-03-23T11:47:20.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:20 smithi063 ceph-mon[35189]: Reconfiguring grafana.smithi063 (dependencies changed)... 2024-03-23T11:47:20.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:20 smithi063 ceph-mon[35189]: Reconfiguring daemon grafana.smithi063 on smithi063 2024-03-23T11:47:20.895 INFO:tasks.workunit.client.0.smithi063.stdout:8/121: truncate d6/dd/df/f11 631744 0 2024-03-23T11:47:20.898 INFO:tasks.workunit.client.0.smithi063.stdout:8/122: symlink d6/dd/l2f 0 2024-03-23T11:47:20.900 INFO:tasks.workunit.client.0.smithi063.stdout:3/112: dwrite d13/d15/f1c [0,4194304] 0 2024-03-23T11:47:20.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:20 smithi111 ceph-mon[44005]: Reconfiguring grafana.smithi063 (dependencies changed)... 2024-03-23T11:47:20.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:20 smithi111 ceph-mon[44005]: Reconfiguring daemon grafana.smithi063 on smithi063 2024-03-23T11:47:20.911 INFO:tasks.workunit.client.0.smithi063.stdout:3/113: truncate d13/d15/f1c 3324271 0 2024-03-23T11:47:20.928 INFO:tasks.workunit.client.0.smithi063.stdout:8/123: dwrite d6/d1f/d26/d15/f17 [0,4194304] 0 2024-03-23T11:47:20.932 INFO:tasks.workunit.client.0.smithi063.stdout:3/114: dwrite d13/d15/f1c [0,4194304] 0 2024-03-23T11:47:20.935 INFO:tasks.workunit.client.0.smithi063.stdout:3/115: rmdir d13/d15/d17 39 2024-03-23T11:47:20.938 INFO:tasks.workunit.client.0.smithi063.stdout:3/116: write fc [833125,123992] 0 2024-03-23T11:47:20.941 INFO:tasks.workunit.client.0.smithi063.stdout:3/117: truncate f3 2864193 0 2024-03-23T11:47:20.945 INFO:tasks.workunit.client.0.smithi063.stdout:8/124: dread d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:20.955 INFO:tasks.workunit.client.0.smithi063.stdout:8/125: write d6/d1f/d26/d15/f24 [1065993,104212] 0 2024-03-23T11:47:20.960 INFO:tasks.workunit.client.0.smithi063.stdout:8/126: mkdir d6/d30 0 2024-03-23T11:47:20.965 INFO:tasks.workunit.client.0.smithi063.stdout:8/127: creat d6/d1c/d27/f31 x:0 0 0 2024-03-23T11:47:20.965 INFO:tasks.workunit.client.0.smithi063.stdout:3/118: dwrite fc [0,4194304] 0 2024-03-23T11:47:20.967 INFO:tasks.workunit.client.0.smithi063.stdout:3/119: chown f12 811 1 2024-03-23T11:47:20.978 INFO:tasks.workunit.client.0.smithi063.stdout:8/128: dread d6/d1c/f1e [0,4194304] 0 2024-03-23T11:47:20.979 INFO:tasks.workunit.client.0.smithi063.stdout:8/129: stat d6/dd 0 2024-03-23T11:47:20.995 INFO:tasks.workunit.client.0.smithi063.stdout:3/120: dwrite f3 [0,4194304] 0 2024-03-23T11:47:21.001 INFO:tasks.workunit.client.0.smithi063.stdout:8/130: dwrite d6/dd/df/f19 [0,4194304] 0 2024-03-23T11:47:21.004 INFO:tasks.workunit.client.0.smithi063.stdout:3/121: truncate fd 1433423 0 2024-03-23T11:47:21.008 INFO:tasks.workunit.client.0.smithi063.stdout:3/122: truncate fc 4633340 0 2024-03-23T11:47:21.028 INFO:tasks.workunit.client.0.smithi063.stdout:3/123: dread ff [0,4194304] 0 2024-03-23T11:47:21.028 INFO:tasks.workunit.client.0.smithi063.stdout:8/131: dwrite d6/d1c/f2d [0,4194304] 0 2024-03-23T11:47:21.030 INFO:tasks.workunit.client.0.smithi063.stdout:3/124: write f8 [4624038,51629] 0 2024-03-23T11:47:21.032 INFO:tasks.workunit.client.0.smithi063.stdout:8/132: mkdir d6/d1c/d32 0 2024-03-23T11:47:21.033 INFO:tasks.workunit.client.0.smithi063.stdout:3/125: chown ff 424832 1 2024-03-23T11:47:21.038 INFO:tasks.workunit.client.0.smithi063.stdout:3/126: dread f12 [0,4194304] 0 2024-03-23T11:47:21.040 INFO:tasks.workunit.client.0.smithi063.stdout:3/127: rmdir d13 39 2024-03-23T11:47:21.041 INFO:tasks.workunit.client.0.smithi063.stdout:3/128: stat f6 0 2024-03-23T11:47:21.045 INFO:tasks.workunit.client.0.smithi063.stdout:3/129: creat d13/d15/f1f x:0 0 0 2024-03-23T11:47:21.047 INFO:tasks.workunit.client.0.smithi063.stdout:3/130: read f12 [454012,9582] 0 2024-03-23T11:47:21.051 INFO:tasks.workunit.client.0.smithi063.stdout:3/131: rename d13/c14 to d13/d15/c20 0 2024-03-23T11:47:21.052 INFO:tasks.workunit.client.0.smithi063.stdout:3/132: chown f6 27848412 1 2024-03-23T11:47:21.053 INFO:tasks.workunit.client.0.smithi063.stdout:8/133: fsync d6/d1c/f2d 0 2024-03-23T11:47:21.055 INFO:tasks.workunit.client.0.smithi063.stdout:3/133: symlink d13/l21 0 2024-03-23T11:47:21.056 INFO:tasks.workunit.client.0.smithi063.stdout:8/134: mknod d6/d1f/c33 0 2024-03-23T11:47:21.059 INFO:tasks.workunit.client.0.smithi063.stdout:8/135: rmdir d6/d1c 39 2024-03-23T11:47:21.059 INFO:tasks.workunit.client.0.smithi063.stdout:8/136: chown d6/d1f 2 1 2024-03-23T11:47:21.059 INFO:tasks.workunit.client.0.smithi063.stdout:3/134: dread f3 [0,4194304] 0 2024-03-23T11:47:21.063 INFO:tasks.workunit.client.0.smithi063.stdout:8/137: creat d6/d1c/d27/f34 x:0 0 0 2024-03-23T11:47:21.063 INFO:tasks.workunit.client.0.smithi063.stdout:8/138: readlink d6/d1c/l20 0 2024-03-23T11:47:21.065 INFO:tasks.workunit.client.0.smithi063.stdout:8/139: creat d6/d1f/d26/f35 x:0 0 0 2024-03-23T11:47:21.069 INFO:tasks.workunit.client.0.smithi063.stdout:8/140: rename d6/dd/l23 to d6/d1c/d32/l36 0 2024-03-23T11:47:21.071 INFO:tasks.workunit.client.0.smithi063.stdout:8/141: write d6/d1f/d26/f35 [1000486,79193] 0 2024-03-23T11:47:21.072 INFO:tasks.workunit.client.0.smithi063.stdout:8/142: chown d6/f16 2923 1 2024-03-23T11:47:21.084 INFO:tasks.workunit.client.0.smithi063.stdout:8/143: dwrite d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:21.086 INFO:tasks.workunit.client.0.smithi063.stdout:8/144: read d6/d1f/d26/d15/f17 [2904879,66125] 0 2024-03-23T11:47:21.087 INFO:tasks.workunit.client.0.smithi063.stdout:8/145: stat d6/d1f/f25 0 2024-03-23T11:47:21.099 INFO:tasks.workunit.client.0.smithi063.stdout:8/146: dread f5 [0,4194304] 0 2024-03-23T11:47:21.099 INFO:tasks.workunit.client.0.smithi063.stdout:8/147: read f5 [877304,44935] 0 2024-03-23T11:47:21.101 INFO:tasks.workunit.client.0.smithi063.stdout:8/148: write d6/d1f/d26/f35 [1838613,89730] 0 2024-03-23T11:47:21.104 INFO:tasks.workunit.client.0.smithi063.stdout:8/149: mkdir d6/d1c/d32/d37 0 2024-03-23T11:47:21.115 INFO:tasks.workunit.client.0.smithi063.stdout:8/150: dwrite d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:21.133 INFO:tasks.workunit.client.0.smithi063.stdout:8/151: dwrite d6/f16 [0,4194304] 0 2024-03-23T11:47:21.140 INFO:tasks.workunit.client.0.smithi063.stdout:8/152: link d6/dd/df/f11 d6/d30/f38 0 2024-03-23T11:47:21.143 INFO:tasks.workunit.client.0.smithi063.stdout:8/153: getdents d6/d1c/d32/d37 0 2024-03-23T11:47:21.145 INFO:tasks.workunit.client.0.smithi063.stdout:8/154: symlink d6/d1c/d32/l39 0 2024-03-23T11:47:21.164 INFO:tasks.workunit.client.0.smithi063.stdout:8/155: dwrite d6/d30/f38 [0,4194304] 0 2024-03-23T11:47:21.165 INFO:tasks.workunit.client.0.smithi063.stdout:8/156: chown d6/d1c 0 1 2024-03-23T11:47:21.165 INFO:tasks.workunit.client.0.smithi063.stdout:8/157: readlink d6/l2e 0 2024-03-23T11:47:21.170 INFO:tasks.workunit.client.0.smithi063.stdout:8/158: getdents d6/d1c 0 2024-03-23T11:47:21.172 INFO:tasks.workunit.client.0.smithi063.stdout:8/159: symlink d6/d1c/d32/l3a 0 2024-03-23T11:47:21.182 INFO:tasks.workunit.client.0.smithi063.stdout:8/160: dread d6/d1f/d26/d15/f24 [0,4194304] 0 2024-03-23T11:47:21.183 INFO:tasks.workunit.client.0.smithi063.stdout:8/161: dread - d6/dd/df/f14 zero size 2024-03-23T11:47:21.185 INFO:tasks.workunit.client.0.smithi063.stdout:8/162: rmdir d6/dd 39 2024-03-23T11:47:21.186 INFO:tasks.workunit.client.0.smithi063.stdout:8/163: read - d6/d1f/f25 zero size 2024-03-23T11:47:21.189 INFO:tasks.workunit.client.0.smithi063.stdout:8/164: rename d6/d30/f38 to d6/d1f/d26/f3b 0 2024-03-23T11:47:21.195 INFO:tasks.workunit.client.0.smithi063.stdout:8/165: dread d6/dd/df/f18 [0,4194304] 0 2024-03-23T11:47:21.197 INFO:tasks.workunit.client.0.smithi063.stdout:8/166: chown d6/d1f/d26/d15/f24 61 1 2024-03-23T11:47:21.198 INFO:tasks.workunit.client.0.smithi063.stdout:8/167: stat d6/cc 0 2024-03-23T11:47:21.199 INFO:tasks.workunit.client.0.smithi063.stdout:8/168: dread - d6/dd/df/f14 zero size 2024-03-23T11:47:21.199 INFO:tasks.workunit.client.0.smithi063.stdout:8/169: stat d6/d1f/l29 0 2024-03-23T11:47:21.202 INFO:tasks.workunit.client.0.smithi063.stdout:8/170: rename d6/d1f/d26/f1a to d6/d1c/f3c 0 2024-03-23T11:47:21.205 INFO:tasks.workunit.client.0.smithi063.stdout:8/171: write d6/d1f/d26/f13 [944784,48059] 0 2024-03-23T11:47:21.210 INFO:tasks.workunit.client.0.smithi063.stdout:8/172: write f5 [4512816,10865] 0 2024-03-23T11:47:21.217 INFO:tasks.workunit.client.0.smithi063.stdout:8/173: truncate d6/dd/df/f11 3466940 0 2024-03-23T11:47:21.220 INFO:tasks.workunit.client.0.smithi063.stdout:8/174: getdents d6/d1f/d26/d15 0 2024-03-23T11:47:21.236 INFO:tasks.workunit.client.0.smithi063.stdout:8/175: dwrite d6/dd/df/f2c [0,4194304] 0 2024-03-23T11:47:21.237 INFO:tasks.workunit.client.0.smithi063.stdout:8/176: stat d6 0 2024-03-23T11:47:21.240 INFO:tasks.workunit.client.0.smithi063.stdout:8/177: symlink d6/d1f/l3d 0 2024-03-23T11:47:21.243 INFO:tasks.workunit.client.0.smithi063.stdout:8/178: mkdir d6/d1f/d3e 0 2024-03-23T11:47:21.245 INFO:tasks.workunit.client.0.smithi063.stdout:8/179: mkdir d6/d1c/d27/d3f 0 2024-03-23T11:47:21.246 INFO:tasks.workunit.client.0.smithi063.stdout:8/180: chown d6/d1f 16 1 2024-03-23T11:47:21.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:21 smithi063 ceph-mon[35189]: pgmap v30: 65 pgs: 65 active+clean; 220 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-23T11:47:21.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:21 smithi111 ceph-mon[44005]: pgmap v30: 65 pgs: 65 active+clean; 220 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/1: sync 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/2: rename - no filename 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/3: dwrite - no filename 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/4: write - no filename 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/5: truncate - no filename 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/6: stat - no entries 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/7: dwrite - no filename 2024-03-23T11:47:22.770 INFO:tasks.workunit.client.0.smithi063.stdout:5/8: fdatasync - no filename 2024-03-23T11:47:22.773 INFO:tasks.workunit.client.0.smithi063.stdout:5/9: mknod c0 0 2024-03-23T11:47:22.773 INFO:tasks.workunit.client.0.smithi063.stdout:5/10: write - no filename 2024-03-23T11:47:22.773 INFO:tasks.workunit.client.0.smithi063.stdout:5/11: read - no filename 2024-03-23T11:47:22.775 INFO:tasks.workunit.client.0.smithi063.stdout:5/12: creat f1 x:0 0 0 2024-03-23T11:47:22.776 INFO:tasks.workunit.client.0.smithi063.stdout:5/13: mkdir d2 0 2024-03-23T11:47:22.777 INFO:tasks.workunit.client.0.smithi063.stdout:5/14: dread - f1 zero size 2024-03-23T11:47:22.780 INFO:tasks.workunit.client.0.smithi063.stdout:5/15: creat d2/f3 x:0 0 0 2024-03-23T11:47:22.782 INFO:tasks.workunit.client.0.smithi063.stdout:5/16: unlink d2/f3 0 2024-03-23T11:47:22.783 INFO:tasks.workunit.client.0.smithi063.stdout:5/17: write f1 [740031,122845] 0 2024-03-23T11:47:22.785 INFO:tasks.workunit.client.0.smithi063.stdout:5/18: write f1 [792047,129986] 0 2024-03-23T11:47:22.788 INFO:tasks.workunit.client.0.smithi063.stdout:5/19: creat d2/f4 x:0 0 0 2024-03-23T11:47:22.790 INFO:tasks.workunit.client.0.smithi063.stdout:5/20: creat d2/f5 x:0 0 0 2024-03-23T11:47:22.792 INFO:tasks.workunit.client.0.smithi063.stdout:5/21: creat d2/f6 x:0 0 0 2024-03-23T11:47:22.793 INFO:tasks.workunit.client.0.smithi063.stdout:5/22: chown f1 5 1 2024-03-23T11:47:22.795 INFO:tasks.workunit.client.0.smithi063.stdout:5/23: mkdir d2/d7 0 2024-03-23T11:47:22.795 INFO:tasks.workunit.client.0.smithi063.stdout:5/24: dread - d2/f4 zero size 2024-03-23T11:47:22.796 INFO:tasks.workunit.client.0.smithi063.stdout:5/25: truncate f1 1421160 0 2024-03-23T11:47:22.798 INFO:tasks.workunit.client.0.smithi063.stdout:5/26: write d2/f4 [828248,14285] 0 2024-03-23T11:47:22.800 INFO:tasks.workunit.client.0.smithi063.stdout:5/27: mkdir d2/d7/d8 0 2024-03-23T11:47:22.808 INFO:tasks.workunit.client.0.smithi063.stdout:5/28: dread f1 [0,4194304] 0 2024-03-23T11:47:22.810 INFO:tasks.workunit.client.0.smithi063.stdout:5/29: write d2/f5 [686897,122462] 0 2024-03-23T11:47:22.813 INFO:tasks.workunit.client.0.smithi063.stdout:5/30: mknod d2/d7/c9 0 2024-03-23T11:47:22.813 INFO:tasks.workunit.client.0.smithi063.stdout:5/31: chown d2/d7 423 1 2024-03-23T11:47:22.814 INFO:tasks.workunit.client.0.smithi063.stdout:5/32: rename d2 to d2/d7/da 22 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:1/21: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:0/8: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:2/18: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:4/71: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:9/0: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:9/1: dread - no filename 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:3/135: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:7/74: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:8/181: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:6/42: sync 2024-03-23T11:47:22.829 INFO:tasks.workunit.client.0.smithi063.stdout:0/9: stat f1 0 2024-03-23T11:47:22.833 INFO:tasks.workunit.client.0.smithi063.stdout:7/75: chown d5/l14 4 1 2024-03-23T11:47:22.837 INFO:tasks.workunit.client.0.smithi063.stdout:6/43: write d1/f7 [15024,17108] 0 2024-03-23T11:47:22.838 INFO:tasks.workunit.client.0.smithi063.stdout:4/72: rmdir d5 39 2024-03-23T11:47:22.841 INFO:tasks.workunit.client.0.smithi063.stdout:3/136: chown d13/d15/c20 103 1 2024-03-23T11:47:22.843 INFO:tasks.workunit.client.0.smithi063.stdout:0/10: mknod d2/c3 0 2024-03-23T11:47:22.843 INFO:tasks.workunit.client.0.smithi063.stdout:7/76: chown d5/d9/c15 29435674 1 2024-03-23T11:47:22.844 INFO:tasks.workunit.client.0.smithi063.stdout:4/73: fdatasync d5/fc 0 2024-03-23T11:47:22.847 INFO:tasks.workunit.client.0.smithi063.stdout:0/11: write f1 [4769315,142] 0 2024-03-23T11:47:22.848 INFO:tasks.workunit.client.0.smithi063.stdout:6/44: fdatasync d1/f3 0 2024-03-23T11:47:22.850 INFO:tasks.workunit.client.0.smithi063.stdout:4/74: mkdir d5/d10 0 2024-03-23T11:47:22.852 INFO:tasks.workunit.client.0.smithi063.stdout:8/182: symlink d6/d1f/d3e/l40 0 2024-03-23T11:47:22.853 INFO:tasks.workunit.client.0.smithi063.stdout:3/137: dread f3 [0,4194304] 0 2024-03-23T11:47:22.856 INFO:tasks.workunit.client.0.smithi063.stdout:0/12: mkdir d2/d4 0 2024-03-23T11:47:22.859 INFO:tasks.workunit.client.0.smithi063.stdout:8/183: chown d6/d1f/d26/d15/f24 1386671 1 2024-03-23T11:47:22.862 INFO:tasks.workunit.client.0.smithi063.stdout:8/184: stat d6/d1c/d27/f31 0 2024-03-23T11:47:22.863 INFO:tasks.workunit.client.0.smithi063.stdout:1/22: dread f2 [0,4194304] 0 2024-03-23T11:47:22.863 INFO:tasks.workunit.client.0.smithi063.stdout:6/45: rename d1/l6 to d1/la 0 2024-03-23T11:47:22.866 INFO:tasks.workunit.client.0.smithi063.stdout:2/19: dwrite d2/f4 [0,4194304] 0 2024-03-23T11:47:22.866 INFO:tasks.workunit.client.0.smithi063.stdout:3/138: mkdir d13/d22 0 2024-03-23T11:47:22.870 INFO:tasks.workunit.client.0.smithi063.stdout:6/46: fdatasync d1/f8 0 2024-03-23T11:47:22.872 INFO:tasks.workunit.client.0.smithi063.stdout:1/23: fdatasync d3/f4 0 2024-03-23T11:47:22.872 INFO:tasks.workunit.client.0.smithi063.stdout:7/77: getdents d5/d6 0 2024-03-23T11:47:22.875 INFO:tasks.workunit.client.0.smithi063.stdout:1/24: fsync d3/f9 0 2024-03-23T11:47:22.876 INFO:tasks.workunit.client.0.smithi063.stdout:8/185: symlink d6/d1f/d26/d15/l41 0 2024-03-23T11:47:22.876 INFO:tasks.workunit.client.0.smithi063.stdout:4/75: link d5/le d5/da/l11 0 2024-03-23T11:47:22.878 INFO:tasks.workunit.client.0.smithi063.stdout:4/76: fsync d5/ff 0 2024-03-23T11:47:22.880 INFO:tasks.workunit.client.0.smithi063.stdout:4/77: write f2 [1529220,11086] 0 2024-03-23T11:47:22.882 INFO:tasks.workunit.client.0.smithi063.stdout:7/78: mkdir d5/d16 0 2024-03-23T11:47:22.884 INFO:tasks.workunit.client.0.smithi063.stdout:4/78: write d5/ff [3173654,130807] 0 2024-03-23T11:47:22.886 INFO:tasks.workunit.client.0.smithi063.stdout:2/20: dread d2/f4 [0,4194304] 0 2024-03-23T11:47:22.886 INFO:tasks.workunit.client.0.smithi063.stdout:3/139: dread f6 [4194304,4194304] 0 2024-03-23T11:47:22.886 INFO:tasks.workunit.client.0.smithi063.stdout:7/79: chown d5/c13 6235 1 2024-03-23T11:47:22.888 INFO:tasks.workunit.client.0.smithi063.stdout:4/79: rename d5/d10 to d5/d10/d12 22 2024-03-23T11:47:22.889 INFO:tasks.workunit.client.0.smithi063.stdout:3/140: dread - d13/d15/f1f zero size 2024-03-23T11:47:22.889 INFO:tasks.workunit.client.0.smithi063.stdout:7/80: chown l3 957 1 2024-03-23T11:47:22.890 INFO:tasks.workunit.client.0.smithi063.stdout:4/80: fsync d5/ff 0 2024-03-23T11:47:22.892 INFO:tasks.workunit.client.0.smithi063.stdout:2/21: unlink d2/l3 0 2024-03-23T11:47:22.897 INFO:tasks.workunit.client.0.smithi063.stdout:3/141: write f9 [1780110,66079] 0 2024-03-23T11:47:22.901 INFO:tasks.workunit.client.0.smithi063.stdout:7/81: creat d5/d9/da/f17 x:0 0 0 2024-03-23T11:47:22.902 INFO:tasks.workunit.client.0.smithi063.stdout:2/22: write d2/f4 [685087,127599] 0 2024-03-23T11:47:22.903 INFO:tasks.workunit.client.0.smithi063.stdout:3/142: readlink d13/d15/d17/l18 0 2024-03-23T11:47:22.905 INFO:tasks.workunit.client.0.smithi063.stdout:3/143: read - d13/d15/f1b zero size 2024-03-23T11:47:22.905 INFO:tasks.workunit.client.0.smithi063.stdout:4/81: dread d5/da/fd [0,4194304] 0 2024-03-23T11:47:22.905 INFO:tasks.workunit.client.0.smithi063.stdout:2/23: creat d2/f5 x:0 0 0 2024-03-23T11:47:22.908 INFO:tasks.workunit.client.0.smithi063.stdout:4/82: read d5/ff [80788,58593] 0 2024-03-23T11:47:22.911 INFO:tasks.workunit.client.0.smithi063.stdout:2/24: symlink d2/l6 0 2024-03-23T11:47:22.914 INFO:tasks.workunit.client.0.smithi063.stdout:2/25: chown d2/f5 32 1 2024-03-23T11:47:22.917 INFO:tasks.workunit.client.0.smithi063.stdout:2/26: chown d2/f5 61073257 1 2024-03-23T11:47:22.920 INFO:tasks.workunit.client.0.smithi063.stdout:2/27: mknod d2/c7 0 2024-03-23T11:47:22.924 INFO:tasks.workunit.client.0.smithi063.stdout:7/82: dwrite d5/d9/da/f17 [0,4194304] 0 2024-03-23T11:47:22.937 INFO:tasks.workunit.client.0.smithi063.stdout:7/83: symlink d5/l18 0 2024-03-23T11:47:22.939 INFO:tasks.workunit.client.0.smithi063.stdout:5/33: fsync d2/f6 0 2024-03-23T11:47:22.941 INFO:tasks.workunit.client.0.smithi063.stdout:7/84: mkdir d5/d6/d19 0 2024-03-23T11:47:22.942 INFO:tasks.workunit.client.0.smithi063.stdout:5/34: write d2/f4 [1537761,39739] 0 2024-03-23T11:47:22.949 INFO:tasks.workunit.client.0.smithi063.stdout:7/85: mkdir d5/d6/d19/d1a 0 2024-03-23T11:47:22.951 INFO:tasks.workunit.client.0.smithi063.stdout:8/186: getdents d6/d1f/d3e 0 2024-03-23T11:47:22.954 INFO:tasks.workunit.client.0.smithi063.stdout:7/86: unlink d5/l18 0 2024-03-23T11:47:22.960 INFO:tasks.workunit.client.0.smithi063.stdout:8/187: symlink d6/l42 0 2024-03-23T11:47:22.961 INFO:tasks.workunit.client.0.smithi063.stdout:6/47: rename d1/la to d1/lb 0 2024-03-23T11:47:22.962 INFO:tasks.workunit.client.0.smithi063.stdout:1/25: write d3/f4 [703087,23591] 0 2024-03-23T11:47:22.963 INFO:tasks.workunit.client.0.smithi063.stdout:7/87: symlink d5/d9/da/l1b 0 2024-03-23T11:47:22.965 INFO:tasks.workunit.client.0.smithi063.stdout:0/13: truncate f1 448964 0 2024-03-23T11:47:22.965 INFO:tasks.workunit.client.0.smithi063.stdout:1/26: stat d3/c5 0 2024-03-23T11:47:22.966 INFO:tasks.workunit.client.0.smithi063.stdout:6/48: mkdir d1/dc 0 2024-03-23T11:47:22.967 INFO:tasks.workunit.client.0.smithi063.stdout:8/188: stat d6/d1c/d27 0 2024-03-23T11:47:22.972 INFO:tasks.workunit.client.0.smithi063.stdout:5/35: dwrite d2/f5 [0,4194304] 0 2024-03-23T11:47:22.972 INFO:tasks.workunit.client.0.smithi063.stdout:6/49: read d1/f3 [3236183,15581] 0 2024-03-23T11:47:22.972 INFO:tasks.workunit.client.0.smithi063.stdout:0/14: creat d2/f5 x:0 0 0 2024-03-23T11:47:22.975 INFO:tasks.workunit.client.0.smithi063.stdout:3/144: truncate f9 1347787 0 2024-03-23T11:47:22.976 INFO:tasks.workunit.client.0.smithi063.stdout:0/15: truncate d2/f5 750924 0 2024-03-23T11:47:22.978 INFO:tasks.workunit.client.0.smithi063.stdout:8/189: fsync d6/d1c/f1e 0 2024-03-23T11:47:22.978 INFO:tasks.workunit.client.0.smithi063.stdout:0/16: stat d2/d4 0 2024-03-23T11:47:22.978 INFO:tasks.workunit.client.0.smithi063.stdout:2/28: rmdir d2 39 2024-03-23T11:47:22.984 INFO:tasks.workunit.client.0.smithi063.stdout:6/50: rename d1/f4 to d1/dc/fd 0 2024-03-23T11:47:22.991 INFO:tasks.workunit.client.0.smithi063.stdout:7/88: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:23.001 INFO:tasks.workunit.client.0.smithi063.stdout:2/29: write d2/f5 [403188,59358] 0 2024-03-23T11:47:23.003 INFO:tasks.workunit.client.0.smithi063.stdout:1/27: getdents d3 0 2024-03-23T11:47:23.008 INFO:tasks.workunit.client.0.smithi063.stdout:3/145: symlink d13/d22/l23 0 2024-03-23T11:47:23.013 INFO:tasks.workunit.client.0.smithi063.stdout:0/17: creat d2/d4/f6 x:0 0 0 2024-03-23T11:47:23.013 INFO:tasks.workunit.client.0.smithi063.stdout:1/28: mknod d3/ca 0 2024-03-23T11:47:23.016 INFO:tasks.workunit.client.0.smithi063.stdout:8/190: dwrite f5 [4194304,4194304] 0 2024-03-23T11:47:23.021 INFO:tasks.workunit.client.0.smithi063.stdout:1/29: mknod d3/cb 0 2024-03-23T11:47:23.024 INFO:tasks.workunit.client.0.smithi063.stdout:7/89: rmdir d5/d6/d19/d1a 0 2024-03-23T11:47:23.029 INFO:tasks.workunit.client.0.smithi063.stdout:3/146: link d13/d15/f1c d13/f24 0 2024-03-23T11:47:23.032 INFO:tasks.workunit.client.0.smithi063.stdout:0/18: rename d2/f5 to d2/f7 0 2024-03-23T11:47:23.035 INFO:tasks.workunit.client.0.smithi063.stdout:5/36: dwrite f1 [0,4194304] 0 2024-03-23T11:47:23.041 INFO:tasks.workunit.client.0.smithi063.stdout:0/19: unlink d2/c3 0 2024-03-23T11:47:23.046 INFO:tasks.workunit.client.0.smithi063.stdout:2/30: dwrite f1 [0,4194304] 0 2024-03-23T11:47:23.049 INFO:tasks.workunit.client.0.smithi063.stdout:8/191: getdents d6/d30 0 2024-03-23T11:47:23.050 INFO:tasks.workunit.client.0.smithi063.stdout:5/37: dread - d2/f6 zero size 2024-03-23T11:47:23.058 INFO:tasks.workunit.client.0.smithi063.stdout:0/20: rmdir d2 39 2024-03-23T11:47:23.067 INFO:tasks.workunit.client.0.smithi063.stdout:5/38: mkdir d2/d7/db 0 2024-03-23T11:47:23.070 INFO:tasks.workunit.client.0.smithi063.stdout:8/192: getdents d6/d1c/d32/d37 0 2024-03-23T11:47:23.071 INFO:tasks.workunit.client.0.smithi063.stdout:0/21: rename d2 to d2/d4/d8 22 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:4/19: sync 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:6/85: sync 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:2/28: sync 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:9/56: sync 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:8/7: sync 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:7/8: sync 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:5/85: sync 2024-03-23T11:47:23.076 INFO:tasks.workunit.client.1.smithi111.stdout:3/10: sync 2024-03-23T11:47:23.077 INFO:tasks.workunit.client.1.smithi111.stdout:0/56: sync 2024-03-23T11:47:23.077 INFO:tasks.workunit.client.1.smithi111.stdout:1/7: sync 2024-03-23T11:47:23.077 INFO:tasks.workunit.client.1.smithi111.stdout:7/9: stat f1 0 2024-03-23T11:47:23.077 INFO:tasks.workunit.client.1.smithi111.stdout:7/10: rmdir - no directory 2024-03-23T11:47:23.078 INFO:tasks.workunit.client.1.smithi111.stdout:6/86: readlink d2/l9 0 2024-03-23T11:47:23.080 INFO:tasks.workunit.client.1.smithi111.stdout:1/8: rmdir d0 39 2024-03-23T11:47:23.083 INFO:tasks.workunit.client.0.smithi063.stdout:7/90: dwrite d5/f12 [0,4194304] 0 2024-03-23T11:47:23.084 INFO:tasks.workunit.client.0.smithi063.stdout:1/30: dwrite d3/f9 [0,4194304] 0 2024-03-23T11:47:23.085 INFO:tasks.workunit.client.0.smithi063.stdout:8/193: chown d6/d1f/c21 761 1 2024-03-23T11:47:23.086 INFO:tasks.workunit.client.0.smithi063.stdout:9/2: sync 2024-03-23T11:47:23.086 INFO:tasks.workunit.client.0.smithi063.stdout:9/3: rmdir - no directory 2024-03-23T11:47:23.087 INFO:tasks.workunit.client.1.smithi111.stdout:3/11: symlink d0/l1 0 2024-03-23T11:47:23.088 INFO:tasks.workunit.client.1.smithi111.stdout:3/12: write - no filename 2024-03-23T11:47:23.088 INFO:tasks.workunit.client.1.smithi111.stdout:3/13: truncate - no filename 2024-03-23T11:47:23.089 INFO:tasks.workunit.client.1.smithi111.stdout:4/20: mkdir d0/d7/d9 0 2024-03-23T11:47:23.091 INFO:tasks.workunit.client.1.smithi111.stdout:7/11: link f1 f2 0 2024-03-23T11:47:23.092 INFO:tasks.workunit.client.1.smithi111.stdout:8/8: dread f1 [0,4194304] 0 2024-03-23T11:47:23.093 INFO:tasks.workunit.client.1.smithi111.stdout:6/87: symlink d2/d1a/d1c/l1e 0 2024-03-23T11:47:23.094 INFO:tasks.workunit.client.1.smithi111.stdout:7/12: truncate f1 183159 0 2024-03-23T11:47:23.094 INFO:tasks.workunit.client.0.smithi063.stdout:5/39: rmdir d2/d7/db 0 2024-03-23T11:47:23.094 INFO:tasks.workunit.client.1.smithi111.stdout:9/57: mknod d0/d1/c16 0 2024-03-23T11:47:23.096 INFO:tasks.workunit.client.0.smithi063.stdout:3/147: dwrite f12 [0,4194304] 0 2024-03-23T11:47:23.097 INFO:tasks.workunit.client.1.smithi111.stdout:5/86: dread d1/d3/d6/d7/f1f [0,4194304] 0 2024-03-23T11:47:23.099 INFO:tasks.workunit.client.1.smithi111.stdout:7/13: dread f1 [0,4194304] 0 2024-03-23T11:47:23.100 INFO:tasks.workunit.client.1.smithi111.stdout:7/14: read f1 [32298,39438] 0 2024-03-23T11:47:23.103 INFO:tasks.workunit.client.0.smithi063.stdout:2/31: dread f1 [0,4194304] 0 2024-03-23T11:47:23.103 INFO:tasks.workunit.client.1.smithi111.stdout:3/14: symlink d0/l2 0 2024-03-23T11:47:23.103 INFO:tasks.workunit.client.1.smithi111.stdout:1/9: creat d0/f4 x:0 0 0 2024-03-23T11:47:23.104 INFO:tasks.workunit.client.1.smithi111.stdout:1/10: chown d0/l3 5262024 1 2024-03-23T11:47:23.105 INFO:tasks.workunit.client.1.smithi111.stdout:4/21: dread d0/f4 [0,4194304] 0 2024-03-23T11:47:23.105 INFO:tasks.workunit.client.0.smithi063.stdout:1/31: truncate d3/f7 864226 0 2024-03-23T11:47:23.106 INFO:tasks.workunit.client.1.smithi111.stdout:1/11: truncate d0/f4 761996 0 2024-03-23T11:47:23.107 INFO:tasks.workunit.client.0.smithi063.stdout:5/40: rmdir d2 39 2024-03-23T11:47:23.109 INFO:tasks.workunit.client.1.smithi111.stdout:8/9: creat f2 x:0 0 0 2024-03-23T11:47:23.110 INFO:tasks.workunit.client.1.smithi111.stdout:8/10: chown c0 1735353 1 2024-03-23T11:47:23.111 INFO:tasks.workunit.client.1.smithi111.stdout:6/88: unlink d2/fa 0 2024-03-23T11:47:23.114 INFO:tasks.workunit.client.0.smithi063.stdout:9/4: creat f0 x:0 0 0 2024-03-23T11:47:23.115 INFO:tasks.workunit.client.1.smithi111.stdout:9/58: write d0/d4/fe [686994,92797] 0 2024-03-23T11:47:23.116 INFO:tasks.workunit.client.1.smithi111.stdout:9/59: readlink d0/d1/l8 0 2024-03-23T11:47:23.121 INFO:tasks.workunit.client.0.smithi063.stdout:8/194: write d6/d1f/d26/f3b [1398769,26833] 0 2024-03-23T11:47:23.123 INFO:tasks.workunit.client.0.smithi063.stdout:1/32: write f2 [1304436,21818] 0 2024-03-23T11:47:23.125 INFO:tasks.workunit.client.1.smithi111.stdout:3/15: creat d0/f3 x:0 0 0 2024-03-23T11:47:23.127 INFO:tasks.workunit.client.1.smithi111.stdout:4/22: rmdir d0 39 2024-03-23T11:47:23.129 INFO:tasks.workunit.client.1.smithi111.stdout:3/16: truncate d0/f3 768862 0 2024-03-23T11:47:23.129 INFO:tasks.workunit.client.0.smithi063.stdout:0/22: dwrite d2/d4/f6 [0,4194304] 0 2024-03-23T11:47:23.130 INFO:tasks.workunit.client.1.smithi111.stdout:1/12: mkdir d0/d5 0 2024-03-23T11:47:23.133 INFO:tasks.workunit.client.0.smithi063.stdout:8/195: stat d6/d1f/d26/c1b 0 2024-03-23T11:47:23.135 INFO:tasks.workunit.client.1.smithi111.stdout:3/17: write d0/f3 [1085375,66856] 0 2024-03-23T11:47:23.135 INFO:tasks.workunit.client.1.smithi111.stdout:8/11: unlink f2 0 2024-03-23T11:47:23.136 INFO:tasks.workunit.client.0.smithi063.stdout:9/5: mknod c1 0 2024-03-23T11:47:23.138 INFO:tasks.workunit.client.1.smithi111.stdout:1/13: read d0/f1 [147072,82153] 0 2024-03-23T11:47:23.138 INFO:tasks.workunit.client.1.smithi111.stdout:6/89: dwrite d2/fb [0,4194304] 0 2024-03-23T11:47:23.138 INFO:tasks.workunit.client.0.smithi063.stdout:3/148: creat d13/d15/f25 x:0 0 0 2024-03-23T11:47:23.139 INFO:tasks.workunit.client.0.smithi063.stdout:9/6: read - f0 zero size 2024-03-23T11:47:23.142 INFO:tasks.workunit.client.0.smithi063.stdout:7/91: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:23.142 INFO:tasks.workunit.client.0.smithi063.stdout:1/33: mknod d3/cc 0 2024-03-23T11:47:23.145 INFO:tasks.workunit.client.1.smithi111.stdout:2/29: link d2/l5 d2/l9 0 2024-03-23T11:47:23.146 INFO:tasks.workunit.client.1.smithi111.stdout:0/57: getdents d5 0 2024-03-23T11:47:23.148 INFO:tasks.workunit.client.0.smithi063.stdout:0/23: creat d2/d4/f9 x:0 0 0 2024-03-23T11:47:23.149 INFO:tasks.workunit.client.1.smithi111.stdout:7/15: dwrite f1 [0,4194304] 0 2024-03-23T11:47:23.151 INFO:tasks.workunit.client.1.smithi111.stdout:2/30: write d2/f6 [892915,87943] 0 2024-03-23T11:47:23.151 INFO:tasks.workunit.client.0.smithi063.stdout:1/34: write d3/f7 [898941,76383] 0 2024-03-23T11:47:23.151 INFO:tasks.workunit.client.0.smithi063.stdout:9/7: chown f0 760 1 2024-03-23T11:47:23.151 INFO:tasks.workunit.client.0.smithi063.stdout:9/8: rmdir - no directory 2024-03-23T11:47:23.152 INFO:tasks.workunit.client.0.smithi063.stdout:3/149: write d13/d15/f1f [1039856,111301] 0 2024-03-23T11:47:23.152 INFO:tasks.workunit.client.1.smithi111.stdout:0/58: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:23.154 INFO:tasks.workunit.client.1.smithi111.stdout:3/18: creat d0/f4 x:0 0 0 2024-03-23T11:47:23.154 INFO:tasks.workunit.client.0.smithi063.stdout:8/196: creat d6/f43 x:0 0 0 2024-03-23T11:47:23.158 INFO:tasks.workunit.client.0.smithi063.stdout:7/92: chown d5/d6/fe 890905444 1 2024-03-23T11:47:23.159 INFO:tasks.workunit.client.1.smithi111.stdout:3/19: write d0/f4 [351088,29313] 0 2024-03-23T11:47:23.164 INFO:tasks.workunit.client.0.smithi063.stdout:2/32: dwrite d2/f5 [0,4194304] 0 2024-03-23T11:47:23.164 INFO:tasks.workunit.client.0.smithi063.stdout:5/41: dread f1 [0,4194304] 0 2024-03-23T11:47:23.165 INFO:tasks.workunit.client.1.smithi111.stdout:1/14: dwrite d0/f4 [0,4194304] 0 2024-03-23T11:47:23.165 INFO:tasks.workunit.client.0.smithi063.stdout:1/35: creat d3/fd x:0 0 0 2024-03-23T11:47:23.166 INFO:tasks.workunit.client.0.smithi063.stdout:9/9: write f0 [802125,90165] 0 2024-03-23T11:47:23.166 INFO:tasks.workunit.client.0.smithi063.stdout:9/10: rmdir - no directory 2024-03-23T11:47:23.167 INFO:tasks.workunit.client.1.smithi111.stdout:6/90: unlink d2/f11 0 2024-03-23T11:47:23.168 INFO:tasks.workunit.client.1.smithi111.stdout:9/60: creat d0/d4/d10/f17 x:0 0 0 2024-03-23T11:47:23.169 INFO:tasks.workunit.client.1.smithi111.stdout:6/91: read - d2/fd zero size 2024-03-23T11:47:23.169 INFO:tasks.workunit.client.1.smithi111.stdout:9/61: fdatasync d0/d1/f11 0 2024-03-23T11:47:23.170 INFO:tasks.workunit.client.1.smithi111.stdout:6/92: readlink l1 0 2024-03-23T11:47:23.171 INFO:tasks.workunit.client.0.smithi063.stdout:8/197: unlink d6/f9 0 2024-03-23T11:47:23.173 INFO:tasks.workunit.client.1.smithi111.stdout:6/93: write d2/d1a/f1d [595387,108450] 0 2024-03-23T11:47:23.177 INFO:tasks.workunit.client.1.smithi111.stdout:7/16: symlink l3 0 2024-03-23T11:47:23.181 INFO:tasks.workunit.client.1.smithi111.stdout:0/59: rename l2 to d5/db/ld 0 2024-03-23T11:47:23.184 INFO:tasks.workunit.client.1.smithi111.stdout:0/60: chown d5/f9 3373664 1 2024-03-23T11:47:23.185 INFO:tasks.workunit.client.1.smithi111.stdout:0/61: chown d5/f9 0 1 2024-03-23T11:47:23.185 INFO:tasks.workunit.client.1.smithi111.stdout:1/15: mkdir d0/d6 0 2024-03-23T11:47:23.191 INFO:tasks.workunit.client.1.smithi111.stdout:4/23: dwrite d0/f4 [0,4194304] 0 2024-03-23T11:47:23.192 INFO:tasks.workunit.client.0.smithi063.stdout:0/24: dwrite d2/d4/f6 [4194304,4194304] 0 2024-03-23T11:47:23.193 INFO:tasks.workunit.client.1.smithi111.stdout:4/24: write d0/f4 [1782055,9597] 0 2024-03-23T11:47:23.193 INFO:tasks.workunit.client.0.smithi063.stdout:8/198: mkdir d6/d1c/d32/d44 0 2024-03-23T11:47:23.194 INFO:tasks.workunit.client.0.smithi063.stdout:3/150: dread f12 [0,4194304] 0 2024-03-23T11:47:23.194 INFO:tasks.workunit.client.1.smithi111.stdout:7/17: dread f0 [0,4194304] 0 2024-03-23T11:47:23.195 INFO:tasks.workunit.client.1.smithi111.stdout:9/62: creat d0/d1/f18 x:0 0 0 2024-03-23T11:47:23.196 INFO:tasks.workunit.client.0.smithi063.stdout:9/11: link f0 f2 0 2024-03-23T11:47:23.196 INFO:tasks.workunit.client.0.smithi063.stdout:0/25: dread - d2/d4/f9 zero size 2024-03-23T11:47:23.199 INFO:tasks.workunit.client.0.smithi063.stdout:5/42: link d2/f5 d2/fc 0 2024-03-23T11:47:23.199 INFO:tasks.workunit.client.1.smithi111.stdout:6/94: chown d2/c15 11655734 1 2024-03-23T11:47:23.206 INFO:tasks.workunit.client.0.smithi063.stdout:9/12: rename c1 to c3 0 2024-03-23T11:47:23.206 INFO:tasks.workunit.client.0.smithi063.stdout:9/13: readlink - no filename 2024-03-23T11:47:23.209 INFO:tasks.workunit.client.0.smithi063.stdout:7/93: getdents d5/d6 0 2024-03-23T11:47:23.212 INFO:tasks.workunit.client.0.smithi063.stdout:2/33: getdents d2 0 2024-03-23T11:47:23.215 INFO:tasks.workunit.client.0.smithi063.stdout:8/199: dread d6/d1c/f2d [0,4194304] 0 2024-03-23T11:47:23.218 INFO:tasks.workunit.client.1.smithi111.stdout:0/62: mkdir d5/db/de 0 2024-03-23T11:47:23.218 INFO:tasks.workunit.client.0.smithi063.stdout:5/43: creat d2/d7/fd x:0 0 0 2024-03-23T11:47:23.219 INFO:tasks.workunit.client.0.smithi063.stdout:3/151: truncate f3 1327878 0 2024-03-23T11:47:23.220 INFO:tasks.workunit.client.0.smithi063.stdout:0/26: dread d2/d4/f6 [4194304,4194304] 0 2024-03-23T11:47:23.220 INFO:tasks.workunit.client.1.smithi111.stdout:7/18: symlink l4 0 2024-03-23T11:47:23.221 INFO:tasks.workunit.client.0.smithi063.stdout:5/44: chown d2/fc 811860 1 2024-03-23T11:47:23.221 INFO:tasks.workunit.client.1.smithi111.stdout:9/63: mkdir d0/d4/d10/d19 0 2024-03-23T11:47:23.222 INFO:tasks.workunit.client.0.smithi063.stdout:2/34: symlink d2/l8 0 2024-03-23T11:47:23.222 INFO:tasks.workunit.client.0.smithi063.stdout:3/152: dread - d13/d15/f25 zero size 2024-03-23T11:47:23.224 INFO:tasks.workunit.client.0.smithi063.stdout:9/14: fsync f0 0 2024-03-23T11:47:23.224 INFO:tasks.workunit.client.1.smithi111.stdout:6/95: mkdir d2/d1f 0 2024-03-23T11:47:23.224 INFO:tasks.workunit.client.0.smithi063.stdout:5/45: write d2/f4 [1335241,8460] 0 2024-03-23T11:47:23.225 INFO:tasks.workunit.client.0.smithi063.stdout:5/46: readlink - no filename 2024-03-23T11:47:23.225 INFO:tasks.workunit.client.0.smithi063.stdout:5/47: readlink - no filename 2024-03-23T11:47:23.227 INFO:tasks.workunit.client.0.smithi063.stdout:8/200: creat d6/d1f/f45 x:0 0 0 2024-03-23T11:47:23.228 INFO:tasks.workunit.client.0.smithi063.stdout:0/27: mknod d2/ca 0 2024-03-23T11:47:23.229 INFO:tasks.workunit.client.0.smithi063.stdout:0/28: stat d2/f7 0 2024-03-23T11:47:23.229 INFO:tasks.workunit.client.1.smithi111.stdout:1/16: symlink d0/d5/l7 0 2024-03-23T11:47:23.230 INFO:tasks.workunit.client.0.smithi063.stdout:2/35: symlink d2/l9 0 2024-03-23T11:47:23.235 INFO:tasks.workunit.client.1.smithi111.stdout:0/63: dwrite f4 [0,4194304] 0 2024-03-23T11:47:23.235 INFO:tasks.workunit.client.1.smithi111.stdout:6/96: creat d2/d17/f20 x:0 0 0 2024-03-23T11:47:23.237 INFO:tasks.workunit.client.0.smithi063.stdout:9/15: fsync f0 0 2024-03-23T11:47:23.238 INFO:tasks.workunit.client.1.smithi111.stdout:6/97: write d2/f16 [658808,13797] 0 2024-03-23T11:47:23.238 INFO:tasks.workunit.client.1.smithi111.stdout:1/17: mknod d0/c8 0 2024-03-23T11:47:23.240 INFO:tasks.workunit.client.0.smithi063.stdout:8/201: truncate d6/d1f/d26/f28 1522967 0 2024-03-23T11:47:23.241 INFO:tasks.workunit.client.0.smithi063.stdout:7/94: truncate d5/f12 2808524 0 2024-03-23T11:47:23.242 INFO:tasks.workunit.client.0.smithi063.stdout:8/202: fdatasync f1 0 2024-03-23T11:47:23.243 INFO:tasks.workunit.client.0.smithi063.stdout:2/36: creat d2/fa x:0 0 0 2024-03-23T11:47:23.244 INFO:tasks.workunit.client.1.smithi111.stdout:0/64: rename d5/db/de to d5/df 0 2024-03-23T11:47:23.244 INFO:tasks.workunit.client.0.smithi063.stdout:2/37: fdatasync d2/f4 0 2024-03-23T11:47:23.245 INFO:tasks.workunit.client.1.smithi111.stdout:7/19: dread f0 [0,4194304] 0 2024-03-23T11:47:23.247 INFO:tasks.workunit.client.1.smithi111.stdout:7/20: write f2 [943635,6618] 0 2024-03-23T11:47:23.247 INFO:tasks.workunit.client.0.smithi063.stdout:0/29: rename d2/ca to d2/d4/cb 0 2024-03-23T11:47:23.247 INFO:tasks.workunit.client.1.smithi111.stdout:6/98: mknod d2/c21 0 2024-03-23T11:47:23.248 INFO:tasks.workunit.client.1.smithi111.stdout:6/99: dread - d2/f8 zero size 2024-03-23T11:47:23.251 INFO:tasks.workunit.client.0.smithi063.stdout:5/48: mkdir d2/d7/d8/de 0 2024-03-23T11:47:23.254 INFO:tasks.workunit.client.1.smithi111.stdout:7/21: symlink l5 0 2024-03-23T11:47:23.257 INFO:tasks.workunit.client.0.smithi063.stdout:2/38: symlink d2/lb 0 2024-03-23T11:47:23.259 INFO:tasks.workunit.client.0.smithi063.stdout:2/39: dread - d2/fa zero size 2024-03-23T11:47:23.259 INFO:tasks.workunit.client.0.smithi063.stdout:0/30: creat d2/fc x:0 0 0 2024-03-23T11:47:23.261 INFO:tasks.workunit.client.0.smithi063.stdout:2/40: chown d2/c7 598 1 2024-03-23T11:47:23.261 INFO:tasks.workunit.client.0.smithi063.stdout:3/153: link d13/d15/c20 d13/d22/c26 0 2024-03-23T11:47:23.271 INFO:tasks.workunit.client.1.smithi111.stdout:1/18: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:23.272 INFO:tasks.workunit.client.0.smithi063.stdout:8/203: dread d6/dd/df/f19 [0,4194304] 0 2024-03-23T11:47:23.274 INFO:tasks.workunit.client.1.smithi111.stdout:6/100: dwrite d2/ff [0,4194304] 0 2024-03-23T11:47:23.274 INFO:tasks.workunit.client.0.smithi063.stdout:2/41: mknod d2/cc 0 2024-03-23T11:47:23.275 INFO:tasks.workunit.client.1.smithi111.stdout:7/22: dread f2 [0,4194304] 0 2024-03-23T11:47:23.275 INFO:tasks.workunit.client.1.smithi111.stdout:1/19: write d0/f1 [2832863,40806] 0 2024-03-23T11:47:23.279 INFO:tasks.workunit.client.0.smithi063.stdout:9/16: dwrite f0 [0,4194304] 0 2024-03-23T11:47:23.279 INFO:tasks.workunit.client.1.smithi111.stdout:6/101: creat d2/d1a/f22 x:0 0 0 2024-03-23T11:47:23.279 INFO:tasks.workunit.client.1.smithi111.stdout:1/20: write d0/f1 [130596,107313] 0 2024-03-23T11:47:23.283 INFO:tasks.workunit.client.0.smithi063.stdout:8/204: symlink d6/dd/df/l46 0 2024-03-23T11:47:23.286 INFO:tasks.workunit.client.0.smithi063.stdout:2/42: mknod d2/cd 0 2024-03-23T11:47:23.291 INFO:tasks.workunit.client.0.smithi063.stdout:3/154: symlink d13/d15/l27 0 2024-03-23T11:47:23.291 INFO:tasks.workunit.client.1.smithi111.stdout:6/102: rename d2/c6 to d2/d17/c23 0 2024-03-23T11:47:23.294 INFO:tasks.workunit.client.1.smithi111.stdout:6/103: truncate d2/f18 802522 0 2024-03-23T11:47:23.295 INFO:tasks.workunit.client.0.smithi063.stdout:2/43: mkdir d2/de 0 2024-03-23T11:47:23.296 INFO:tasks.workunit.client.0.smithi063.stdout:5/49: getdents d2 0 2024-03-23T11:47:23.296 INFO:tasks.workunit.client.0.smithi063.stdout:2/44: stat d2/f4 0 2024-03-23T11:47:23.296 INFO:tasks.workunit.client.0.smithi063.stdout:7/95: write f1 [1662201,122203] 0 2024-03-23T11:47:23.297 INFO:tasks.workunit.client.0.smithi063.stdout:5/50: stat d2/d7 0 2024-03-23T11:47:23.298 INFO:tasks.workunit.client.1.smithi111.stdout:6/104: creat d2/d1a/d1c/f24 x:0 0 0 2024-03-23T11:47:23.300 INFO:tasks.workunit.client.0.smithi063.stdout:7/96: fsync d5/d6/fe 0 2024-03-23T11:47:23.301 INFO:tasks.workunit.client.1.smithi111.stdout:6/105: symlink d2/d1a/d1c/l25 0 2024-03-23T11:47:23.303 INFO:tasks.workunit.client.0.smithi063.stdout:2/45: truncate d2/fa 999447 0 2024-03-23T11:47:23.305 INFO:tasks.workunit.client.1.smithi111.stdout:3/20: sync 2024-03-23T11:47:23.305 INFO:tasks.workunit.client.1.smithi111.stdout:6/106: creat d2/d1f/f26 x:0 0 0 2024-03-23T11:47:23.307 INFO:tasks.workunit.client.1.smithi111.stdout:3/21: write d0/f3 [853684,86872] 0 2024-03-23T11:47:23.308 INFO:tasks.workunit.client.1.smithi111.stdout:6/107: write d2/d1a/d1c/f24 [140130,76487] 0 2024-03-23T11:47:23.312 INFO:tasks.workunit.client.1.smithi111.stdout:6/108: creat d2/f27 x:0 0 0 2024-03-23T11:47:23.313 INFO:tasks.workunit.client.0.smithi063.stdout:2/46: rename d2/f5 to d2/de/ff 0 2024-03-23T11:47:23.314 INFO:tasks.workunit.client.0.smithi063.stdout:2/47: fsync d2/fa 0 2024-03-23T11:47:23.315 INFO:tasks.workunit.client.0.smithi063.stdout:5/51: mknod d2/d7/d8/de/cf 0 2024-03-23T11:47:23.318 INFO:tasks.workunit.client.0.smithi063.stdout:9/17: truncate f0 3085691 0 2024-03-23T11:47:23.318 INFO:tasks.workunit.client.0.smithi063.stdout:8/205: getdents d6/dd 0 2024-03-23T11:47:23.320 INFO:tasks.workunit.client.0.smithi063.stdout:7/97: read d5/f12 [887612,85796] 0 2024-03-23T11:47:23.322 INFO:tasks.workunit.client.1.smithi111.stdout:6/109: dread d2/f16 [0,4194304] 0 2024-03-23T11:47:23.322 INFO:tasks.workunit.client.0.smithi063.stdout:2/48: symlink d2/de/l10 0 2024-03-23T11:47:23.325 INFO:tasks.workunit.client.1.smithi111.stdout:6/110: creat d2/f28 x:0 0 0 2024-03-23T11:47:23.326 INFO:tasks.workunit.client.1.smithi111.stdout:6/111: readlink d2/l13 0 2024-03-23T11:47:23.326 INFO:tasks.workunit.client.1.smithi111.stdout:6/112: stat d2/f27 0 2024-03-23T11:47:23.327 INFO:tasks.workunit.client.0.smithi063.stdout:5/52: creat d2/d7/f10 x:0 0 0 2024-03-23T11:47:23.328 INFO:tasks.workunit.client.0.smithi063.stdout:2/49: unlink d2/cc 0 2024-03-23T11:47:23.328 INFO:tasks.workunit.client.1.smithi111.stdout:6/113: dread - d2/d17/f20 zero size 2024-03-23T11:47:23.329 INFO:tasks.workunit.client.1.smithi111.stdout:6/114: truncate d2/f27 429124 0 2024-03-23T11:47:23.334 INFO:tasks.workunit.client.0.smithi063.stdout:5/53: chown d2/d7/c9 5047 1 2024-03-23T11:47:23.335 INFO:tasks.workunit.client.0.smithi063.stdout:2/50: chown d2/fa 0 1 2024-03-23T11:47:23.341 INFO:tasks.workunit.client.0.smithi063.stdout:8/206: dread d6/d1c/f3c [0,4194304] 0 2024-03-23T11:47:23.344 INFO:tasks.workunit.client.0.smithi063.stdout:3/155: dwrite f5 [0,4194304] 0 2024-03-23T11:47:23.347 INFO:tasks.workunit.client.0.smithi063.stdout:5/54: getdents d2 0 2024-03-23T11:47:23.348 INFO:tasks.workunit.client.0.smithi063.stdout:2/51: rename d2/l6 to d2/de/l11 0 2024-03-23T11:47:23.355 INFO:tasks.workunit.client.0.smithi063.stdout:8/207: unlink d6/l2e 0 2024-03-23T11:47:23.360 INFO:tasks.workunit.client.0.smithi063.stdout:9/18: dwrite f0 [0,4194304] 0 2024-03-23T11:47:23.360 INFO:tasks.workunit.client.0.smithi063.stdout:2/52: mknod d2/c12 0 2024-03-23T11:47:23.362 INFO:tasks.workunit.client.0.smithi063.stdout:8/208: chown d6/dd/c12 7643 1 2024-03-23T11:47:23.363 INFO:tasks.workunit.client.0.smithi063.stdout:7/98: dwrite d5/d6/fe [0,4194304] 0 2024-03-23T11:47:23.364 INFO:tasks.workunit.client.0.smithi063.stdout:2/53: readlink d2/de/l10 0 2024-03-23T11:47:23.369 INFO:tasks.workunit.client.0.smithi063.stdout:9/19: mkdir d4 0 2024-03-23T11:47:23.372 INFO:tasks.workunit.client.0.smithi063.stdout:7/99: mknod d5/d6/c1c 0 2024-03-23T11:47:23.374 INFO:tasks.workunit.client.0.smithi063.stdout:8/209: creat d6/dd/f47 x:0 0 0 2024-03-23T11:47:23.382 INFO:tasks.workunit.client.0.smithi063.stdout:3/156: dread f8 [0,4194304] 0 2024-03-23T11:47:23.382 INFO:tasks.workunit.client.0.smithi063.stdout:8/210: dread - d6/f43 zero size 2024-03-23T11:47:23.387 INFO:tasks.workunit.client.1.smithi111.stdout:4/25: sync 2024-03-23T11:47:23.387 INFO:tasks.workunit.client.0.smithi063.stdout:5/55: dwrite d2/f5 [4194304,4194304] 0 2024-03-23T11:47:23.388 INFO:tasks.workunit.client.0.smithi063.stdout:3/157: dread - d13/d15/f25 zero size 2024-03-23T11:47:23.391 INFO:tasks.workunit.client.0.smithi063.stdout:5/56: fdatasync d2/f6 0 2024-03-23T11:47:23.391 INFO:tasks.workunit.client.1.smithi111.stdout:4/26: mkdir d0/d7/da 0 2024-03-23T11:47:23.394 INFO:tasks.workunit.client.0.smithi063.stdout:3/158: mknod d13/c28 0 2024-03-23T11:47:23.398 INFO:tasks.workunit.client.1.smithi111.stdout:4/27: rmdir d0 39 2024-03-23T11:47:23.399 INFO:tasks.workunit.client.0.smithi063.stdout:3/159: stat f6 0 2024-03-23T11:47:23.404 INFO:tasks.workunit.client.0.smithi063.stdout:7/100: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:23.408 INFO:tasks.workunit.client.0.smithi063.stdout:2/54: dread d2/de/ff [0,4194304] 0 2024-03-23T11:47:23.408 INFO:tasks.workunit.client.0.smithi063.stdout:8/211: chown d6/dd/df/f14 143619 1 2024-03-23T11:47:23.408 INFO:tasks.workunit.client.0.smithi063.stdout:5/57: getdents d2/d7/d8 0 2024-03-23T11:47:23.415 INFO:tasks.workunit.client.1.smithi111.stdout:4/28: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:23.416 INFO:tasks.workunit.client.0.smithi063.stdout:5/58: dread - d2/d7/fd zero size 2024-03-23T11:47:23.417 INFO:tasks.workunit.client.0.smithi063.stdout:3/160: getdents d13/d22 0 2024-03-23T11:47:23.418 INFO:tasks.workunit.client.0.smithi063.stdout:9/20: dwrite f2 [0,4194304] 0 2024-03-23T11:47:23.420 INFO:tasks.workunit.client.1.smithi111.stdout:4/29: creat d0/d7/da/fb x:0 0 0 2024-03-23T11:47:23.421 INFO:tasks.workunit.client.0.smithi063.stdout:5/59: write d2/f6 [729453,91500] 0 2024-03-23T11:47:23.421 INFO:tasks.workunit.client.1.smithi111.stdout:4/30: chown d0/f1 940 1 2024-03-23T11:47:23.423 INFO:tasks.workunit.client.1.smithi111.stdout:4/31: write d0/f4 [2193746,120130] 0 2024-03-23T11:47:23.431 INFO:tasks.workunit.client.1.smithi111.stdout:4/32: mkdir d0/dc 0 2024-03-23T11:47:23.432 INFO:tasks.workunit.client.1.smithi111.stdout:3/22: sync 2024-03-23T11:47:23.436 INFO:tasks.workunit.client.1.smithi111.stdout:3/23: symlink d0/l5 0 2024-03-23T11:47:23.438 INFO:tasks.workunit.client.1.smithi111.stdout:4/33: mknod d0/dc/cd 0 2024-03-23T11:47:23.440 INFO:tasks.workunit.client.1.smithi111.stdout:3/24: rename d0/l1 to d0/l6 0 2024-03-23T11:47:23.444 INFO:tasks.workunit.client.0.smithi063.stdout:3/161: dread d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:23.444 INFO:tasks.workunit.client.0.smithi063.stdout:2/55: dread f1 [0,4194304] 0 2024-03-23T11:47:23.445 INFO:tasks.workunit.client.1.smithi111.stdout:3/25: mkdir d0/d7 0 2024-03-23T11:47:23.446 INFO:tasks.workunit.client.1.smithi111.stdout:3/26: rename d0 to d0/d8 22 2024-03-23T11:47:23.448 INFO:tasks.workunit.client.0.smithi063.stdout:7/101: dwrite d5/f8 [0,4194304] 0 2024-03-23T11:47:23.451 INFO:tasks.workunit.client.1.smithi111.stdout:3/27: dread d0/f3 [0,4194304] 0 2024-03-23T11:47:23.453 INFO:tasks.workunit.client.0.smithi063.stdout:8/212: dwrite d6/d1c/f2d [0,4194304] 0 2024-03-23T11:47:23.454 INFO:tasks.workunit.client.0.smithi063.stdout:3/162: mknod d13/c29 0 2024-03-23T11:47:23.455 INFO:tasks.workunit.client.0.smithi063.stdout:2/56: write d2/f4 [2171644,44348] 0 2024-03-23T11:47:23.458 INFO:tasks.workunit.client.0.smithi063.stdout:9/21: fdatasync f2 0 2024-03-23T11:47:23.459 INFO:tasks.workunit.client.1.smithi111.stdout:3/28: rename d0/f4 to d0/f9 0 2024-03-23T11:47:23.463 INFO:tasks.workunit.client.1.smithi111.stdout:3/29: dread d0/f3 [0,4194304] 0 2024-03-23T11:47:23.464 INFO:tasks.workunit.client.1.smithi111.stdout:5/87: write d1/d3/fc [1953301,24804] 0 2024-03-23T11:47:23.464 INFO:tasks.workunit.client.0.smithi063.stdout:5/60: dwrite d2/d7/fd [0,4194304] 0 2024-03-23T11:47:23.465 INFO:tasks.workunit.client.0.smithi063.stdout:9/22: write f2 [2388376,76244] 0 2024-03-23T11:47:23.465 INFO:tasks.workunit.client.0.smithi063.stdout:2/57: write d2/f4 [3658052,85589] 0 2024-03-23T11:47:23.469 INFO:tasks.workunit.client.1.smithi111.stdout:3/30: dread d0/f3 [0,4194304] 0 2024-03-23T11:47:23.471 INFO:tasks.workunit.client.0.smithi063.stdout:8/213: truncate d6/dd/df/f18 1728856 0 2024-03-23T11:47:23.474 INFO:tasks.workunit.client.0.smithi063.stdout:3/163: dread d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:23.475 INFO:tasks.workunit.client.0.smithi063.stdout:2/58: symlink d2/de/l13 0 2024-03-23T11:47:23.475 INFO:tasks.workunit.client.1.smithi111.stdout:2/31: getdents d2 0 2024-03-23T11:47:23.476 INFO:tasks.workunit.client.1.smithi111.stdout:3/31: mknod d0/d7/ca 0 2024-03-23T11:47:23.477 INFO:tasks.workunit.client.0.smithi063.stdout:8/214: stat d6/d1c/f2d 0 2024-03-23T11:47:23.479 INFO:tasks.workunit.client.1.smithi111.stdout:2/32: mkdir d2/d7/da 0 2024-03-23T11:47:23.480 INFO:tasks.workunit.client.0.smithi063.stdout:5/61: symlink d2/d7/d8/l11 0 2024-03-23T11:47:23.481 INFO:tasks.workunit.client.0.smithi063.stdout:8/215: mknod d6/d1f/d26/c48 0 2024-03-23T11:47:23.482 INFO:tasks.workunit.client.0.smithi063.stdout:5/62: fdatasync d2/f6 0 2024-03-23T11:47:23.482 INFO:tasks.workunit.client.1.smithi111.stdout:2/33: truncate d2/f6 1943681 0 2024-03-23T11:47:23.484 INFO:tasks.workunit.client.0.smithi063.stdout:3/164: truncate d13/d15/f1b 148593 0 2024-03-23T11:47:23.487 INFO:tasks.workunit.client.0.smithi063.stdout:5/63: creat d2/d7/d8/f12 x:0 0 0 2024-03-23T11:47:23.488 INFO:tasks.workunit.client.0.smithi063.stdout:2/59: link d2/cd d2/c14 0 2024-03-23T11:47:23.489 INFO:tasks.workunit.client.0.smithi063.stdout:5/64: truncate d2/f4 2469094 0 2024-03-23T11:47:23.491 INFO:tasks.workunit.client.0.smithi063.stdout:3/165: truncate d13/d15/f1c 22356 0 2024-03-23T11:47:23.491 INFO:tasks.workunit.client.0.smithi063.stdout:8/216: link d6/d1c/f3c d6/d1f/d26/f49 0 2024-03-23T11:47:23.492 INFO:tasks.workunit.client.1.smithi111.stdout:2/34: rename d2/l5 to d2/d7/da/lb 0 2024-03-23T11:47:23.494 INFO:tasks.workunit.client.1.smithi111.stdout:2/35: creat d2/fc x:0 0 0 2024-03-23T11:47:23.494 INFO:tasks.workunit.client.0.smithi063.stdout:5/65: link f1 d2/d7/d8/f13 0 2024-03-23T11:47:23.496 INFO:tasks.workunit.client.1.smithi111.stdout:2/36: creat d2/d7/fd x:0 0 0 2024-03-23T11:47:23.497 INFO:tasks.workunit.client.1.smithi111.stdout:8/12: truncate f1 1691174 0 2024-03-23T11:47:23.497 INFO:tasks.workunit.client.0.smithi063.stdout:3/166: creat d13/d15/f2a x:0 0 0 2024-03-23T11:47:23.501 INFO:tasks.workunit.client.0.smithi063.stdout:3/167: link le d13/d15/d17/l2b 0 2024-03-23T11:47:23.502 INFO:tasks.workunit.client.0.smithi063.stdout:8/217: dread d6/d1f/d26/d15/f17 [0,4194304] 0 2024-03-23T11:47:23.503 INFO:tasks.workunit.client.0.smithi063.stdout:3/168: chown d13/l21 2 1 2024-03-23T11:47:23.504 INFO:tasks.workunit.client.0.smithi063.stdout:3/169: fdatasync fd 0 2024-03-23T11:47:23.505 INFO:tasks.workunit.client.0.smithi063.stdout:3/170: chown f9 9833312 1 2024-03-23T11:47:23.506 INFO:tasks.workunit.client.0.smithi063.stdout:3/171: stat d13/d15/l1e 0 2024-03-23T11:47:23.506 INFO:tasks.workunit.client.0.smithi063.stdout:8/218: creat d6/d1c/d27/f4a x:0 0 0 2024-03-23T11:47:23.507 INFO:tasks.workunit.client.1.smithi111.stdout:9/64: truncate d0/fd 2412286 0 2024-03-23T11:47:23.509 INFO:tasks.workunit.client.1.smithi111.stdout:9/65: truncate d0/d4/f14 156516 0 2024-03-23T11:47:23.511 INFO:tasks.workunit.client.0.smithi063.stdout:8/219: creat d6/dd/df/f4b x:0 0 0 2024-03-23T11:47:23.512 INFO:tasks.workunit.client.0.smithi063.stdout:8/220: dread - d6/dd/df/f14 zero size 2024-03-23T11:47:23.513 INFO:tasks.workunit.client.1.smithi111.stdout:0/65: rmdir d5/df 0 2024-03-23T11:47:23.514 INFO:tasks.workunit.client.1.smithi111.stdout:7/23: getdents . 0 2024-03-23T11:47:23.515 INFO:tasks.workunit.client.1.smithi111.stdout:7/24: read f0 [11970,32963] 0 2024-03-23T11:47:23.516 INFO:tasks.workunit.client.1.smithi111.stdout:0/66: mknod d5/c10 0 2024-03-23T11:47:23.517 INFO:tasks.workunit.client.1.smithi111.stdout:0/67: chown f4 6206 1 2024-03-23T11:47:23.518 INFO:tasks.workunit.client.1.smithi111.stdout:0/68: chown d5/f8 240338 1 2024-03-23T11:47:23.521 INFO:tasks.workunit.client.1.smithi111.stdout:1/21: truncate d0/f1 315525 0 2024-03-23T11:47:23.524 INFO:tasks.workunit.client.1.smithi111.stdout:1/22: symlink d0/l9 0 2024-03-23T11:47:23.530 INFO:tasks.workunit.client.1.smithi111.stdout:4/34: truncate d0/f1 1572299 0 2024-03-23T11:47:23.531 INFO:tasks.workunit.client.0.smithi063.stdout:3/172: dwrite d13/d15/f2a [0,4194304] 0 2024-03-23T11:47:23.536 INFO:tasks.workunit.client.1.smithi111.stdout:2/37: stat d2/d7/da/lb 0 2024-03-23T11:47:23.539 INFO:tasks.workunit.client.1.smithi111.stdout:7/25: dwrite f1 [0,4194304] 0 2024-03-23T11:47:23.539 INFO:tasks.workunit.client.0.smithi063.stdout:3/173: dread d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:23.540 INFO:tasks.workunit.client.1.smithi111.stdout:7/26: stat l3 0 2024-03-23T11:47:23.540 INFO:tasks.workunit.client.1.smithi111.stdout:0/69: dread f4 [0,4194304] 0 2024-03-23T11:47:23.542 INFO:tasks.workunit.client.1.smithi111.stdout:8/13: truncate f1 1924045 0 2024-03-23T11:47:23.543 INFO:tasks.workunit.client.0.smithi063.stdout:8/221: dwrite d6/dd/df/f18 [0,4194304] 0 2024-03-23T11:47:23.545 INFO:tasks.workunit.client.0.smithi063.stdout:8/222: chown d6/d1c/d32/d37 452422 1 2024-03-23T11:47:23.546 INFO:tasks.workunit.client.1.smithi111.stdout:6/115: dwrite d2/fd [0,4194304] 0 2024-03-23T11:47:23.546 INFO:tasks.workunit.client.1.smithi111.stdout:4/35: rename d0/d7/da/fb to d0/dc/fe 0 2024-03-23T11:47:23.548 INFO:tasks.workunit.client.1.smithi111.stdout:3/32: sync 2024-03-23T11:47:23.548 INFO:tasks.workunit.client.1.smithi111.stdout:0/70: truncate d5/fc 153789 0 2024-03-23T11:47:23.550 INFO:tasks.workunit.client.0.smithi063.stdout:3/174: dread d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:23.550 INFO:tasks.workunit.client.1.smithi111.stdout:6/116: chown d2/fd 4726891 1 2024-03-23T11:47:23.550 INFO:tasks.workunit.client.1.smithi111.stdout:1/23: dread d0/f4 [0,4194304] 0 2024-03-23T11:47:23.550 INFO:tasks.workunit.client.1.smithi111.stdout:7/27: mknod c6 0 2024-03-23T11:47:23.552 INFO:tasks.workunit.client.0.smithi063.stdout:8/223: rename d6/dd/df/f19 to d6/d1c/d32/d44/f4c 0 2024-03-23T11:47:23.553 INFO:tasks.workunit.client.1.smithi111.stdout:5/88: dwrite d1/d3/d6/d7/f10 [0,4194304] 0 2024-03-23T11:47:23.553 INFO:tasks.workunit.client.1.smithi111.stdout:8/14: creat f3 x:0 0 0 2024-03-23T11:47:23.553 INFO:tasks.workunit.client.1.smithi111.stdout:3/33: chown d0/d7/ca 14053729 1 2024-03-23T11:47:23.555 INFO:tasks.workunit.client.0.smithi063.stdout:8/224: fsync d6/d1c/d27/f34 0 2024-03-23T11:47:23.560 INFO:tasks.workunit.client.0.smithi063.stdout:8/225: mknod d6/d1c/d32/d44/c4d 0 2024-03-23T11:47:23.560 INFO:tasks.workunit.client.1.smithi111.stdout:0/71: symlink d5/db/l11 0 2024-03-23T11:47:23.565 INFO:tasks.workunit.client.1.smithi111.stdout:6/117: creat d2/f29 x:0 0 0 2024-03-23T11:47:23.567 INFO:tasks.workunit.client.0.smithi063.stdout:8/226: creat d6/d1c/f4e x:0 0 0 2024-03-23T11:47:23.567 INFO:tasks.workunit.client.1.smithi111.stdout:7/28: rename f0 to f7 0 2024-03-23T11:47:23.568 INFO:tasks.workunit.client.1.smithi111.stdout:3/34: rmdir d0/d7 39 2024-03-23T11:47:23.568 INFO:tasks.workunit.client.1.smithi111.stdout:2/38: dwrite d2/f4 [0,4194304] 0 2024-03-23T11:47:23.568 INFO:tasks.workunit.client.1.smithi111.stdout:0/72: fsync f4 0 2024-03-23T11:47:23.568 INFO:tasks.workunit.client.1.smithi111.stdout:7/29: stat f1 0 2024-03-23T11:47:23.569 INFO:tasks.workunit.client.1.smithi111.stdout:6/118: write d2/f28 [524710,86305] 0 2024-03-23T11:47:23.571 INFO:tasks.workunit.client.0.smithi063.stdout:8/227: rename f1 to d6/d1c/f4f 0 2024-03-23T11:47:23.572 INFO:tasks.workunit.client.1.smithi111.stdout:4/36: creat d0/d7/d9/ff x:0 0 0 2024-03-23T11:47:23.572 INFO:tasks.workunit.client.1.smithi111.stdout:4/37: readlink - no filename 2024-03-23T11:47:23.573 INFO:tasks.workunit.client.0.smithi063.stdout:8/228: write d6/d1c/f4e [493964,101468] 0 2024-03-23T11:47:23.576 INFO:tasks.workunit.client.1.smithi111.stdout:5/89: mkdir d1/d20 0 2024-03-23T11:47:23.577 INFO:tasks.workunit.client.1.smithi111.stdout:0/73: read d5/f8 [95084,87582] 0 2024-03-23T11:47:23.577 INFO:tasks.workunit.client.1.smithi111.stdout:6/119: dread - d2/d17/f20 zero size 2024-03-23T11:47:23.578 INFO:tasks.workunit.client.0.smithi063.stdout:3/175: write d13/d15/f1c [790554,74875] 0 2024-03-23T11:47:23.578 INFO:tasks.workunit.client.1.smithi111.stdout:7/30: rename l4 to l8 0 2024-03-23T11:47:23.581 INFO:tasks.workunit.client.0.smithi063.stdout:8/229: dread d6/d1c/f3c [0,4194304] 0 2024-03-23T11:47:23.583 INFO:tasks.workunit.client.1.smithi111.stdout:4/38: mknod d0/d7/c10 0 2024-03-23T11:47:23.586 INFO:tasks.workunit.client.1.smithi111.stdout:3/35: symlink d0/d7/lb 0 2024-03-23T11:47:23.586 INFO:tasks.workunit.client.1.smithi111.stdout:4/39: read - d0/dc/fe zero size 2024-03-23T11:47:23.591 INFO:tasks.workunit.client.1.smithi111.stdout:8/15: dread f1 [0,4194304] 0 2024-03-23T11:47:23.592 INFO:tasks.workunit.client.0.smithi063.stdout:8/230: unlink d6/d1f/d26/d15/l41 0 2024-03-23T11:47:23.592 INFO:tasks.workunit.client.1.smithi111.stdout:0/74: mknod d5/c12 0 2024-03-23T11:47:23.602 INFO:tasks.workunit.client.0.smithi063.stdout:3/176: dwrite f6 [4194304,4194304] 0 2024-03-23T11:47:23.609 INFO:tasks.workunit.client.0.smithi063.stdout:3/177: getdents d13/d15 0 2024-03-23T11:47:23.612 INFO:tasks.workunit.client.0.smithi063.stdout:3/178: chown d13/d15/d17 167 1 2024-03-23T11:47:23.622 INFO:tasks.workunit.client.0.smithi063.stdout:3/179: creat d13/d22/f2c x:0 0 0 2024-03-23T11:47:23.623 INFO:tasks.workunit.client.0.smithi063.stdout:8/231: write d6/d1f/d26/d15/f17 [359169,6109] 0 2024-03-23T11:47:23.626 INFO:tasks.workunit.client.0.smithi063.stdout:3/180: write d13/d15/f2a [668387,27097] 0 2024-03-23T11:47:23.626 INFO:tasks.workunit.client.0.smithi063.stdout:8/232: dread - d6/dd/f47 zero size 2024-03-23T11:47:23.633 INFO:tasks.workunit.client.0.smithi063.stdout:8/233: creat d6/dd/df/f50 x:0 0 0 2024-03-23T11:47:23.636 INFO:tasks.workunit.client.1.smithi111.stdout:9/66: dwrite d0/d4/f9 [0,4194304] 0 2024-03-23T11:47:23.636 INFO:tasks.workunit.client.1.smithi111.stdout:3/36: chown d0/d7/ca 9209 1 2024-03-23T11:47:23.637 INFO:tasks.workunit.client.0.smithi063.stdout:8/234: creat d6/d1c/d32/d44/f51 x:0 0 0 2024-03-23T11:47:23.638 INFO:tasks.workunit.client.1.smithi111.stdout:4/40: symlink d0/l11 0 2024-03-23T11:47:23.641 INFO:tasks.workunit.client.1.smithi111.stdout:8/16: symlink l4 0 2024-03-23T11:47:23.642 INFO:tasks.workunit.client.0.smithi063.stdout:8/235: getdents d6/d1f/d3e 0 2024-03-23T11:47:23.646 INFO:tasks.workunit.client.1.smithi111.stdout:0/75: creat d5/f13 x:0 0 0 2024-03-23T11:47:23.651 INFO:tasks.workunit.client.1.smithi111.stdout:4/41: symlink d0/d7/d9/l12 0 2024-03-23T11:47:23.651 INFO:tasks.workunit.client.1.smithi111.stdout:3/37: write d0/f9 [675328,87372] 0 2024-03-23T11:47:23.651 INFO:tasks.workunit.client.1.smithi111.stdout:8/17: write f3 [691064,14142] 0 2024-03-23T11:47:23.651 INFO:tasks.workunit.client.1.smithi111.stdout:7/31: chown l8 46838 1 2024-03-23T11:47:23.652 INFO:tasks.workunit.client.1.smithi111.stdout:9/67: rmdir d0/d4/d10/d19 0 2024-03-23T11:47:23.655 INFO:tasks.workunit.client.1.smithi111.stdout:0/76: mkdir d5/db/d14 0 2024-03-23T11:47:23.656 INFO:tasks.workunit.client.0.smithi063.stdout:3/181: dwrite ff [0,4194304] 0 2024-03-23T11:47:23.659 INFO:tasks.workunit.client.1.smithi111.stdout:2/39: dwrite d2/fc [0,4194304] 0 2024-03-23T11:47:23.661 INFO:tasks.workunit.client.1.smithi111.stdout:8/18: write f1 [2704893,81690] 0 2024-03-23T11:47:23.663 INFO:tasks.workunit.client.1.smithi111.stdout:6/120: dwrite d2/f8 [0,4194304] 0 2024-03-23T11:47:23.663 INFO:tasks.workunit.client.1.smithi111.stdout:0/77: write d5/f9 [4754017,22830] 0 2024-03-23T11:47:23.664 INFO:tasks.workunit.client.0.smithi063.stdout:8/236: dread d6/d1f/d26/d15/f24 [0,4194304] 0 2024-03-23T11:47:23.666 INFO:tasks.workunit.client.1.smithi111.stdout:3/38: mknod d0/cc 0 2024-03-23T11:47:23.668 INFO:tasks.workunit.client.0.smithi063.stdout:8/237: chown d6/dd/df/f2c 1143 1 2024-03-23T11:47:23.669 INFO:tasks.workunit.client.1.smithi111.stdout:1/24: sync 2024-03-23T11:47:23.669 INFO:tasks.workunit.client.1.smithi111.stdout:2/40: dread f0 [0,4194304] 0 2024-03-23T11:47:23.672 INFO:tasks.workunit.client.1.smithi111.stdout:4/42: creat d0/d7/da/f13 x:0 0 0 2024-03-23T11:47:23.672 INFO:tasks.workunit.client.1.smithi111.stdout:3/39: write d0/f9 [931118,3819] 0 2024-03-23T11:47:23.673 INFO:tasks.workunit.client.0.smithi063.stdout:3/182: getdents d13/d15 0 2024-03-23T11:47:23.673 INFO:tasks.workunit.client.0.smithi063.stdout:8/238: creat d6/d1c/f52 x:0 0 0 2024-03-23T11:47:23.674 INFO:tasks.workunit.client.1.smithi111.stdout:2/41: chown d2/d7/c8 1 1 2024-03-23T11:47:23.674 INFO:tasks.workunit.client.1.smithi111.stdout:7/32: fsync f7 0 2024-03-23T11:47:23.675 INFO:tasks.workunit.client.0.smithi063.stdout:8/239: dread - d6/d1f/f45 zero size 2024-03-23T11:47:23.678 INFO:tasks.workunit.client.1.smithi111.stdout:2/42: readlink d2/d7/da/lb 0 2024-03-23T11:47:23.678 INFO:tasks.workunit.client.0.smithi063.stdout:3/183: rename d13/d15/d17 to d13/d22/d2d 0 2024-03-23T11:47:23.679 INFO:tasks.workunit.client.1.smithi111.stdout:0/78: stat d5/db/ld 0 2024-03-23T11:47:23.680 INFO:tasks.workunit.client.1.smithi111.stdout:2/43: fdatasync d2/f6 0 2024-03-23T11:47:23.680 INFO:tasks.workunit.client.0.smithi063.stdout:8/240: symlink d6/dd/l53 0 2024-03-23T11:47:23.681 INFO:tasks.workunit.client.1.smithi111.stdout:6/121: mkdir d2/d2a 0 2024-03-23T11:47:23.682 INFO:tasks.workunit.client.0.smithi063.stdout:3/184: symlink d13/l2e 0 2024-03-23T11:47:23.686 INFO:tasks.workunit.client.1.smithi111.stdout:0/79: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:23.686 INFO:tasks.workunit.client.1.smithi111.stdout:6/122: fdatasync d2/f28 0 2024-03-23T11:47:23.688 INFO:tasks.workunit.client.1.smithi111.stdout:9/68: dwrite d0/fd [0,4194304] 0 2024-03-23T11:47:23.692 INFO:tasks.workunit.client.1.smithi111.stdout:2/44: mkdir d2/de 0 2024-03-23T11:47:23.692 INFO:tasks.workunit.client.1.smithi111.stdout:1/25: mknod d0/d6/ca 0 2024-03-23T11:47:23.695 INFO:tasks.workunit.client.1.smithi111.stdout:3/40: stat d0/l6 0 2024-03-23T11:47:23.699 INFO:tasks.workunit.client.1.smithi111.stdout:9/69: mknod d0/d1/c1a 0 2024-03-23T11:47:23.701 INFO:tasks.workunit.client.1.smithi111.stdout:1/26: fsync d0/f4 0 2024-03-23T11:47:23.702 INFO:tasks.workunit.client.1.smithi111.stdout:0/80: write d5/f9 [1064007,72670] 0 2024-03-23T11:47:23.703 INFO:tasks.workunit.client.1.smithi111.stdout:3/41: readlink d0/l2 0 2024-03-23T11:47:23.704 INFO:tasks.workunit.client.1.smithi111.stdout:6/123: dread d2/f27 [0,4194304] 0 2024-03-23T11:47:23.704 INFO:tasks.workunit.client.1.smithi111.stdout:2/45: creat d2/d7/da/ff x:0 0 0 2024-03-23T11:47:23.705 INFO:tasks.workunit.client.1.smithi111.stdout:8/19: dwrite f3 [0,4194304] 0 2024-03-23T11:47:23.706 INFO:tasks.workunit.client.0.smithi063.stdout:8/241: dwrite d6/d1c/d27/f31 [0,4194304] 0 2024-03-23T11:47:23.707 INFO:tasks.workunit.client.1.smithi111.stdout:7/33: dread f1 [0,4194304] 0 2024-03-23T11:47:23.710 INFO:tasks.workunit.client.1.smithi111.stdout:9/70: dread - d0/d4/d10/f17 zero size 2024-03-23T11:47:23.710 INFO:tasks.workunit.client.1.smithi111.stdout:8/20: mkdir d5 0 2024-03-23T11:47:23.711 INFO:tasks.workunit.client.1.smithi111.stdout:6/124: mknod d2/d1f/c2b 0 2024-03-23T11:47:23.712 INFO:tasks.workunit.client.1.smithi111.stdout:4/43: dwrite d0/f4 [4194304,4194304] 0 2024-03-23T11:47:23.713 INFO:tasks.workunit.client.1.smithi111.stdout:2/46: dread - d2/d7/da/ff zero size 2024-03-23T11:47:23.715 INFO:tasks.workunit.client.1.smithi111.stdout:1/27: write d0/f1 [450610,58103] 0 2024-03-23T11:47:23.716 INFO:tasks.workunit.client.1.smithi111.stdout:0/81: creat d5/db/f15 x:0 0 0 2024-03-23T11:47:23.717 INFO:tasks.workunit.client.1.smithi111.stdout:6/125: mkdir d2/d1a/d1c/d2c 0 2024-03-23T11:47:23.718 INFO:tasks.workunit.client.0.smithi063.stdout:3/185: truncate f8 4319247 0 2024-03-23T11:47:23.718 INFO:tasks.workunit.client.0.smithi063.stdout:8/242: write d6/d1f/d26/d15/f24 [3416674,130755] 0 2024-03-23T11:47:23.719 INFO:tasks.workunit.client.1.smithi111.stdout:7/34: write f2 [1692314,45678] 0 2024-03-23T11:47:23.719 INFO:tasks.workunit.client.1.smithi111.stdout:2/47: dread - d2/d7/da/ff zero size 2024-03-23T11:47:23.719 INFO:tasks.workunit.client.1.smithi111.stdout:9/71: creat d0/d4/f1b x:0 0 0 2024-03-23T11:47:23.719 INFO:tasks.workunit.client.1.smithi111.stdout:1/28: chown d0/f1 11 1 2024-03-23T11:47:23.720 INFO:tasks.workunit.client.1.smithi111.stdout:0/82: chown d5/c12 370413399 1 2024-03-23T11:47:23.720 INFO:tasks.workunit.client.1.smithi111.stdout:4/44: creat d0/dc/f14 x:0 0 0 2024-03-23T11:47:23.721 INFO:tasks.workunit.client.1.smithi111.stdout:1/29: readlink d0/l9 0 2024-03-23T11:47:23.722 INFO:tasks.workunit.client.0.smithi063.stdout:3/186: write d13/d15/f1c [1615784,12787] 0 2024-03-23T11:47:23.722 INFO:tasks.workunit.client.1.smithi111.stdout:3/42: dread d0/f3 [0,4194304] 0 2024-03-23T11:47:23.722 INFO:tasks.workunit.client.1.smithi111.stdout:9/72: dread - d0/d1/f18 zero size 2024-03-23T11:47:23.723 INFO:tasks.workunit.client.1.smithi111.stdout:5/90: sync 2024-03-23T11:47:23.723 INFO:tasks.workunit.client.1.smithi111.stdout:8/21: link f1 d5/f6 0 2024-03-23T11:47:23.724 INFO:tasks.workunit.client.0.smithi063.stdout:3/187: chown d13/d15/f2a 201980024 1 2024-03-23T11:47:23.727 INFO:tasks.workunit.client.1.smithi111.stdout:7/35: write f7 [1025685,49998] 0 2024-03-23T11:47:23.727 INFO:tasks.workunit.client.1.smithi111.stdout:0/83: creat d5/db/f16 x:0 0 0 2024-03-23T11:47:23.728 INFO:tasks.workunit.client.1.smithi111.stdout:1/30: mknod d0/d6/cb 0 2024-03-23T11:47:23.729 INFO:tasks.workunit.client.0.smithi063.stdout:8/243: mkdir d6/d1f/d54 0 2024-03-23T11:47:23.729 INFO:tasks.workunit.client.1.smithi111.stdout:6/126: write d2/f27 [101993,7764] 0 2024-03-23T11:47:23.729 INFO:tasks.workunit.client.1.smithi111.stdout:4/45: rmdir d0/d7/d9 39 2024-03-23T11:47:23.730 INFO:tasks.workunit.client.0.smithi063.stdout:8/244: chown d6/d1c/d32/d44 1 1 2024-03-23T11:47:23.731 INFO:tasks.workunit.client.1.smithi111.stdout:3/43: rmdir d0/d7 39 2024-03-23T11:47:23.734 INFO:tasks.workunit.client.1.smithi111.stdout:1/31: truncate d0/f1 889619 0 2024-03-23T11:47:23.734 INFO:tasks.workunit.client.1.smithi111.stdout:8/22: fsync f3 0 2024-03-23T11:47:23.734 INFO:tasks.workunit.client.0.smithi063.stdout:8/245: write d6/d1c/d27/f31 [2251019,27515] 0 2024-03-23T11:47:23.734 INFO:tasks.workunit.client.0.smithi063.stdout:3/188: creat d13/d22/f2f x:0 0 0 2024-03-23T11:47:23.734 INFO:tasks.workunit.client.1.smithi111.stdout:5/91: dread d1/d3/d6/ff [0,4194304] 0 2024-03-23T11:47:23.735 INFO:tasks.workunit.client.0.smithi063.stdout:8/246: stat d6/d1f/d26/c48 0 2024-03-23T11:47:23.736 INFO:tasks.workunit.client.1.smithi111.stdout:7/36: creat f9 x:0 0 0 2024-03-23T11:47:23.739 INFO:tasks.workunit.client.0.smithi063.stdout:3/189: rmdir d13/d22/d2d 39 2024-03-23T11:47:23.741 INFO:tasks.workunit.client.1.smithi111.stdout:1/32: creat d0/d6/fc x:0 0 0 2024-03-23T11:47:23.742 INFO:tasks.workunit.client.0.smithi063.stdout:8/247: readlink d6/dd/df/l46 0 2024-03-23T11:47:23.744 INFO:tasks.workunit.client.0.smithi063.stdout:3/190: creat d13/f30 x:0 0 0 2024-03-23T11:47:23.745 INFO:tasks.workunit.client.1.smithi111.stdout:1/33: fsync d0/f1 0 2024-03-23T11:47:23.745 INFO:tasks.workunit.client.1.smithi111.stdout:2/48: read d2/f6 [508236,111906] 0 2024-03-23T11:47:23.750 INFO:tasks.workunit.client.1.smithi111.stdout:8/23: write d5/f6 [3608003,20570] 0 2024-03-23T11:47:23.752 INFO:tasks.workunit.client.1.smithi111.stdout:2/49: read - d2/d7/da/ff zero size 2024-03-23T11:47:23.753 INFO:tasks.workunit.client.1.smithi111.stdout:4/46: creat d0/d7/d9/f15 x:0 0 0 2024-03-23T11:47:23.759 INFO:tasks.workunit.client.1.smithi111.stdout:9/73: dwrite d0/d4/f1b [0,4194304] 0 2024-03-23T11:47:23.760 INFO:tasks.workunit.client.1.smithi111.stdout:1/34: mkdir d0/d5/dd 0 2024-03-23T11:47:23.760 INFO:tasks.workunit.client.0.smithi063.stdout:3/191: rename d13/f24 to d13/d22/d2d/f31 0 2024-03-23T11:47:23.761 INFO:tasks.workunit.client.1.smithi111.stdout:6/127: rmdir d2/d1f 39 2024-03-23T11:47:23.764 INFO:tasks.workunit.client.1.smithi111.stdout:5/92: dwrite d1/d3/d6/d7/f9 [0,4194304] 0 2024-03-23T11:47:23.765 INFO:tasks.workunit.client.1.smithi111.stdout:6/128: read d2/f8 [3386637,767] 0 2024-03-23T11:47:23.766 INFO:tasks.workunit.client.0.smithi063.stdout:8/248: dwrite d6/d1c/f52 [0,4194304] 0 2024-03-23T11:47:23.766 INFO:tasks.workunit.client.0.smithi063.stdout:3/192: rename c10 to d13/c32 0 2024-03-23T11:47:23.766 INFO:tasks.workunit.client.1.smithi111.stdout:5/93: write d1/d3/d6/ff [532788,16668] 0 2024-03-23T11:47:23.767 INFO:tasks.workunit.client.1.smithi111.stdout:3/44: creat d0/fd x:0 0 0 2024-03-23T11:47:23.768 INFO:tasks.workunit.client.1.smithi111.stdout:6/129: write d2/f18 [1313810,93945] 0 2024-03-23T11:47:23.770 INFO:tasks.workunit.client.1.smithi111.stdout:2/50: symlink d2/d7/da/l10 0 2024-03-23T11:47:23.774 INFO:tasks.workunit.client.1.smithi111.stdout:0/84: getdents d5 0 2024-03-23T11:47:23.774 INFO:tasks.workunit.client.1.smithi111.stdout:2/51: write d2/d7/fd [717733,52723] 0 2024-03-23T11:47:23.775 INFO:tasks.workunit.client.0.smithi063.stdout:8/249: dread d6/d1c/d27/f31 [0,4194304] 0 2024-03-23T11:47:23.776 INFO:tasks.workunit.client.1.smithi111.stdout:1/35: creat d0/fe x:0 0 0 2024-03-23T11:47:23.781 INFO:tasks.workunit.client.1.smithi111.stdout:8/24: creat d5/f7 x:0 0 0 2024-03-23T11:47:23.790 INFO:tasks.workunit.client.1.smithi111.stdout:6/130: creat d2/d1a/f2d x:0 0 0 2024-03-23T11:47:23.790 INFO:tasks.workunit.client.1.smithi111.stdout:0/85: creat d5/f17 x:0 0 0 2024-03-23T11:47:23.793 INFO:tasks.workunit.client.1.smithi111.stdout:1/36: rename d0/c2 to d0/d5/cf 0 2024-03-23T11:47:23.793 INFO:tasks.workunit.client.1.smithi111.stdout:3/45: dwrite d0/fd [0,4194304] 0 2024-03-23T11:47:23.794 INFO:tasks.workunit.client.1.smithi111.stdout:7/37: getdents . 0 2024-03-23T11:47:23.795 INFO:tasks.workunit.client.0.smithi063.stdout:3/193: dwrite f5 [0,4194304] 0 2024-03-23T11:47:23.797 INFO:tasks.workunit.client.1.smithi111.stdout:8/25: mknod d5/c8 0 2024-03-23T11:47:23.798 INFO:tasks.workunit.client.0.smithi063.stdout:4/83: sync 2024-03-23T11:47:23.800 INFO:tasks.workunit.client.1.smithi111.stdout:8/26: write d5/f6 [3186886,104124] 0 2024-03-23T11:47:23.800 INFO:tasks.workunit.client.1.smithi111.stdout:5/94: link d1/d3/d6/f8 d1/d1e/f21 0 2024-03-23T11:47:23.806 INFO:tasks.workunit.client.1.smithi111.stdout:7/38: symlink la 0 2024-03-23T11:47:23.808 INFO:tasks.workunit.client.1.smithi111.stdout:4/47: getdents d0/d7 0 2024-03-23T11:47:23.809 INFO:tasks.workunit.client.1.smithi111.stdout:0/86: creat d5/db/d14/f18 x:0 0 0 2024-03-23T11:47:23.809 INFO:tasks.workunit.client.1.smithi111.stdout:3/46: stat d0/d7/lb 0 2024-03-23T11:47:23.810 INFO:tasks.workunit.client.1.smithi111.stdout:6/131: creat d2/d1f/f2e x:0 0 0 2024-03-23T11:47:23.810 INFO:tasks.workunit.client.1.smithi111.stdout:1/37: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:23.811 INFO:tasks.workunit.client.1.smithi111.stdout:4/48: stat d0/d7/d9 0 2024-03-23T11:47:23.811 INFO:tasks.workunit.client.1.smithi111.stdout:6/132: chown d2/d17 2 1 2024-03-23T11:47:23.812 INFO:tasks.workunit.client.0.smithi063.stdout:3/194: creat d13/d15/f33 x:0 0 0 2024-03-23T11:47:23.812 INFO:tasks.workunit.client.1.smithi111.stdout:8/27: symlink d5/l9 0 2024-03-23T11:47:23.814 INFO:tasks.workunit.client.0.smithi063.stdout:4/84: dread d5/ff [0,4194304] 0 2024-03-23T11:47:23.814 INFO:tasks.workunit.client.1.smithi111.stdout:4/49: read - d0/d7/da/f13 zero size 2024-03-23T11:47:23.815 INFO:tasks.workunit.client.0.smithi063.stdout:8/250: getdents d6 0 2024-03-23T11:47:23.815 INFO:tasks.workunit.client.1.smithi111.stdout:5/95: rename d1/d3/d6/fa to d1/d1e/f22 0 2024-03-23T11:47:23.815 INFO:tasks.workunit.client.1.smithi111.stdout:7/39: creat fb x:0 0 0 2024-03-23T11:47:23.816 INFO:tasks.workunit.client.1.smithi111.stdout:8/28: read f1 [1840270,46632] 0 2024-03-23T11:47:23.817 INFO:tasks.workunit.client.1.smithi111.stdout:4/50: stat d0/f1 0 2024-03-23T11:47:23.817 INFO:tasks.workunit.client.1.smithi111.stdout:2/52: getdents d2 0 2024-03-23T11:47:23.818 INFO:tasks.workunit.client.1.smithi111.stdout:0/87: mkdir d5/db/d14/d19 0 2024-03-23T11:47:23.819 INFO:tasks.workunit.client.1.smithi111.stdout:7/40: truncate f9 322462 0 2024-03-23T11:47:23.820 INFO:tasks.workunit.client.1.smithi111.stdout:8/29: chown c0 15513591 1 2024-03-23T11:47:23.821 INFO:tasks.workunit.client.0.smithi063.stdout:8/251: creat d6/d1f/f55 x:0 0 0 2024-03-23T11:47:23.823 INFO:tasks.workunit.client.1.smithi111.stdout:6/133: write d2/f8 [1903851,26317] 0 2024-03-23T11:47:23.824 INFO:tasks.workunit.client.1.smithi111.stdout:5/96: mkdir d1/d3/d23 0 2024-03-23T11:47:23.825 INFO:tasks.workunit.client.0.smithi063.stdout:8/252: write d6/d1f/d26/f13 [165574,82368] 0 2024-03-23T11:47:23.825 INFO:tasks.workunit.client.1.smithi111.stdout:2/53: mkdir d2/d11 0 2024-03-23T11:47:23.827 INFO:tasks.workunit.client.1.smithi111.stdout:0/88: creat d5/db/d14/f1a x:0 0 0 2024-03-23T11:47:23.829 INFO:tasks.workunit.client.0.smithi063.stdout:4/85: truncate d5/da/fd 1862410 0 2024-03-23T11:47:23.829 INFO:tasks.workunit.client.1.smithi111.stdout:1/38: creat d0/d5/dd/f10 x:0 0 0 2024-03-23T11:47:23.829 INFO:tasks.workunit.client.1.smithi111.stdout:8/30: creat d5/fa x:0 0 0 2024-03-23T11:47:23.830 INFO:tasks.workunit.client.1.smithi111.stdout:0/89: write f4 [3123071,55287] 0 2024-03-23T11:47:23.832 INFO:tasks.workunit.client.1.smithi111.stdout:5/97: write d1/d3/d6/f8 [5651817,1257] 0 2024-03-23T11:47:23.832 INFO:tasks.workunit.client.1.smithi111.stdout:2/54: mkdir d2/de/d12 0 2024-03-23T11:47:23.833 INFO:tasks.workunit.client.1.smithi111.stdout:8/31: mknod d5/cb 0 2024-03-23T11:47:23.835 INFO:tasks.workunit.client.1.smithi111.stdout:1/39: write d0/d6/fc [661264,71786] 0 2024-03-23T11:47:23.835 INFO:tasks.workunit.client.0.smithi063.stdout:8/253: dread d6/d1f/d26/f28 [0,4194304] 0 2024-03-23T11:47:23.841 INFO:tasks.workunit.client.1.smithi111.stdout:5/98: creat d1/d3/d6/d7/f24 x:0 0 0 2024-03-23T11:47:23.846 INFO:tasks.workunit.client.1.smithi111.stdout:2/55: creat d2/d11/f13 x:0 0 0 2024-03-23T11:47:23.846 INFO:tasks.workunit.client.1.smithi111.stdout:1/40: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:23.848 INFO:tasks.workunit.client.0.smithi063.stdout:8/254: dread d6/d1c/f1e [0,4194304] 0 2024-03-23T11:47:23.850 INFO:tasks.workunit.client.1.smithi111.stdout:0/90: symlink d5/db/d14/d19/l1b 0 2024-03-23T11:47:23.852 INFO:tasks.workunit.client.1.smithi111.stdout:2/56: dread - d2/d11/f13 zero size 2024-03-23T11:47:23.852 INFO:tasks.workunit.client.1.smithi111.stdout:7/41: dwrite f9 [0,4194304] 0 2024-03-23T11:47:23.852 INFO:tasks.workunit.client.1.smithi111.stdout:8/32: link d5/cb d5/cc 0 2024-03-23T11:47:23.852 INFO:tasks.workunit.client.1.smithi111.stdout:9/74: truncate d0/d4/f9 1940808 0 2024-03-23T11:47:23.854 INFO:tasks.workunit.client.0.smithi063.stdout:8/255: rename d6/d1f/f25 to d6/dd/f56 0 2024-03-23T11:47:23.855 INFO:tasks.workunit.client.1.smithi111.stdout:7/42: chown fb 6028420 1 2024-03-23T11:47:23.856 INFO:tasks.workunit.client.0.smithi063.stdout:8/256: dread - d6/d1f/f45 zero size 2024-03-23T11:47:23.857 INFO:tasks.workunit.client.1.smithi111.stdout:1/41: creat d0/d6/f11 x:0 0 0 2024-03-23T11:47:23.859 INFO:tasks.workunit.client.1.smithi111.stdout:6/134: dwrite d2/ff [0,4194304] 0 2024-03-23T11:47:23.859 INFO:tasks.workunit.client.0.smithi063.stdout:8/257: creat d6/d1c/d32/f57 x:0 0 0 2024-03-23T11:47:23.860 INFO:tasks.workunit.client.1.smithi111.stdout:5/99: truncate d1/d3/d6/d7/f24 50043 0 2024-03-23T11:47:23.860 INFO:tasks.workunit.client.0.smithi063.stdout:8/258: chown f5 59 1 2024-03-23T11:47:23.871 INFO:tasks.workunit.client.1.smithi111.stdout:6/135: write d2/d17/f20 [138496,77658] 0 2024-03-23T11:47:23.871 INFO:tasks.workunit.client.1.smithi111.stdout:5/100: fdatasync d1/d3/d6/f8 0 2024-03-23T11:47:23.875 INFO:tasks.workunit.client.1.smithi111.stdout:0/91: getdents d5/db/d14/d19 0 2024-03-23T11:47:23.875 INFO:tasks.workunit.client.1.smithi111.stdout:9/75: getdents d0/d4 0 2024-03-23T11:47:23.875 INFO:tasks.workunit.client.1.smithi111.stdout:6/136: dread - d2/f29 zero size 2024-03-23T11:47:23.877 INFO:tasks.workunit.client.1.smithi111.stdout:5/101: mkdir d1/d20/d25 0 2024-03-23T11:47:23.879 INFO:tasks.workunit.client.1.smithi111.stdout:9/76: chown d0/d4 5604 1 2024-03-23T11:47:23.880 INFO:tasks.workunit.client.1.smithi111.stdout:1/42: dread d0/d6/fc [0,4194304] 0 2024-03-23T11:47:23.881 INFO:tasks.workunit.client.0.smithi063.stdout:8/259: fsync d6/d1c/d32/f57 0 2024-03-23T11:47:23.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:23 smithi063 ceph-mon[35189]: pgmap v31: 65 pgs: 65 active+clean; 436 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.4 MiB/s rd, 33 MiB/s wr, 350 op/s 2024-03-23T11:47:23.885 INFO:tasks.workunit.client.1.smithi111.stdout:0/92: creat d5/db/d14/d19/f1c x:0 0 0 2024-03-23T11:47:23.886 INFO:tasks.workunit.client.1.smithi111.stdout:6/137: chown d2/fd 9296423 1 2024-03-23T11:47:23.886 INFO:tasks.workunit.client.0.smithi063.stdout:8/260: link d6/d1f/c33 d6/c58 0 2024-03-23T11:47:23.887 INFO:tasks.workunit.client.1.smithi111.stdout:7/43: truncate f9 3387351 0 2024-03-23T11:47:23.888 INFO:tasks.workunit.client.1.smithi111.stdout:1/43: creat d0/f12 x:0 0 0 2024-03-23T11:47:23.888 INFO:tasks.workunit.client.1.smithi111.stdout:9/77: chown d0/d4/d10/f17 42724947 1 2024-03-23T11:47:23.890 INFO:tasks.workunit.client.1.smithi111.stdout:5/102: write d1/d3/d6/d7/f13 [3553156,121737] 0 2024-03-23T11:47:23.891 INFO:tasks.workunit.client.1.smithi111.stdout:8/33: dread f1 [0,4194304] 0 2024-03-23T11:47:23.891 INFO:tasks.workunit.client.1.smithi111.stdout:7/44: chown l5 11122 1 2024-03-23T11:47:23.894 INFO:tasks.workunit.client.1.smithi111.stdout:2/57: dwrite d2/d7/fd [0,4194304] 0 2024-03-23T11:47:23.897 INFO:tasks.workunit.client.1.smithi111.stdout:5/103: chown d1/d1e/f22 639638691 1 2024-03-23T11:47:23.897 INFO:tasks.workunit.client.1.smithi111.stdout:0/93: write d5/f17 [175530,129581] 0 2024-03-23T11:47:23.898 INFO:tasks.workunit.client.1.smithi111.stdout:6/138: rename d2/f29 to d2/d2a/f2f 0 2024-03-23T11:47:23.899 INFO:tasks.workunit.client.0.smithi063.stdout:8/261: fsync d6/d1c/f4f 0 2024-03-23T11:47:23.900 INFO:tasks.workunit.client.1.smithi111.stdout:1/44: fsync d0/d5/dd/f10 0 2024-03-23T11:47:23.900 INFO:tasks.workunit.client.1.smithi111.stdout:2/58: chown d2/d7 935126 1 2024-03-23T11:47:23.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:23 smithi111 ceph-mon[44005]: pgmap v31: 65 pgs: 65 active+clean; 436 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.4 MiB/s rd, 33 MiB/s wr, 350 op/s 2024-03-23T11:47:23.904 INFO:tasks.workunit.client.0.smithi063.stdout:8/262: truncate d6/d1c/f1e 1068448 0 2024-03-23T11:47:23.905 INFO:tasks.workunit.client.1.smithi111.stdout:5/104: write d1/d3/f18 [208664,88346] 0 2024-03-23T11:47:23.907 INFO:tasks.workunit.client.1.smithi111.stdout:1/45: write d0/f4 [4719918,105550] 0 2024-03-23T11:47:23.907 INFO:tasks.workunit.client.1.smithi111.stdout:9/78: read d0/d4/f9 [1763937,99217] 0 2024-03-23T11:47:23.908 INFO:tasks.workunit.client.1.smithi111.stdout:0/94: truncate d5/db/d14/f18 251814 0 2024-03-23T11:47:23.910 INFO:tasks.workunit.client.1.smithi111.stdout:6/139: rename d2/c1b to d2/d1a/c30 0 2024-03-23T11:47:23.911 INFO:tasks.workunit.client.1.smithi111.stdout:2/59: chown d2/d7/da/l10 342484 1 2024-03-23T11:47:23.912 INFO:tasks.workunit.client.1.smithi111.stdout:9/79: fdatasync d0/d4/f1b 0 2024-03-23T11:47:23.913 INFO:tasks.workunit.client.1.smithi111.stdout:2/60: stat d2/d7 0 2024-03-23T11:47:23.913 INFO:tasks.workunit.client.1.smithi111.stdout:6/140: rmdir d2/d1a/d1c 39 2024-03-23T11:47:23.914 INFO:tasks.workunit.client.1.smithi111.stdout:1/46: chown d0/fe 48975371 1 2024-03-23T11:47:23.916 INFO:tasks.workunit.client.1.smithi111.stdout:5/105: creat d1/d20/d25/f26 x:0 0 0 2024-03-23T11:47:23.917 INFO:tasks.workunit.client.1.smithi111.stdout:9/80: mkdir d0/d4/d1c 0 2024-03-23T11:47:23.921 INFO:tasks.workunit.client.1.smithi111.stdout:1/47: creat d0/f13 x:0 0 0 2024-03-23T11:47:23.923 INFO:tasks.workunit.client.1.smithi111.stdout:8/34: dread f3 [0,4194304] 0 2024-03-23T11:47:23.923 INFO:tasks.workunit.client.1.smithi111.stdout:7/45: dread f7 [0,4194304] 0 2024-03-23T11:47:23.924 INFO:tasks.workunit.client.1.smithi111.stdout:9/81: mknod d0/d4/d1c/c1d 0 2024-03-23T11:47:23.925 INFO:tasks.workunit.client.1.smithi111.stdout:6/141: link d2/d2a/f2f d2/f31 0 2024-03-23T11:47:23.926 INFO:tasks.workunit.client.0.smithi063.stdout:8/263: dwrite d6/d1f/d26/f49 [0,4194304] 0 2024-03-23T11:47:23.927 INFO:tasks.workunit.client.1.smithi111.stdout:8/35: write d5/fa [395365,103161] 0 2024-03-23T11:47:23.928 INFO:tasks.workunit.client.1.smithi111.stdout:7/46: mknod cc 0 2024-03-23T11:47:23.928 INFO:tasks.workunit.client.1.smithi111.stdout:7/47: rmdir - no directory 2024-03-23T11:47:23.931 INFO:tasks.workunit.client.0.smithi063.stdout:8/264: mkdir d6/dd/d59 0 2024-03-23T11:47:23.937 INFO:tasks.workunit.client.0.smithi063.stdout:8/265: rename d6/d1c/l20 to d6/d1c/d27/l5a 0 2024-03-23T11:47:23.937 INFO:tasks.workunit.client.0.smithi063.stdout:8/266: readlink d6/d1c/d32/l3a 0 2024-03-23T11:47:23.945 INFO:tasks.workunit.client.1.smithi111.stdout:5/106: dread d1/d3/d6/d7/f13 [0,4194304] 0 2024-03-23T11:47:23.956 INFO:tasks.workunit.client.1.smithi111.stdout:7/48: dwrite fb [0,4194304] 0 2024-03-23T11:47:23.963 INFO:tasks.workunit.client.1.smithi111.stdout:7/49: dread fb [0,4194304] 0 2024-03-23T11:47:23.999 INFO:tasks.workunit.client.1.smithi111.stdout:3/47: sync 2024-03-23T11:47:23.999 INFO:tasks.workunit.client.1.smithi111.stdout:3/48: read d0/f3 [454716,52457] 0 2024-03-23T11:47:24.001 INFO:tasks.workunit.client.1.smithi111.stdout:3/49: read d0/f9 [771799,25718] 0 2024-03-23T11:47:24.002 INFO:tasks.workunit.client.1.smithi111.stdout:3/50: stat d0 0 2024-03-23T11:47:24.007 INFO:tasks.workunit.client.0.smithi063.stdout:6/51: sync 2024-03-23T11:47:24.007 INFO:tasks.workunit.client.0.smithi063.stdout:1/36: sync 2024-03-23T11:47:24.009 INFO:tasks.workunit.client.0.smithi063.stdout:6/52: truncate d1/f5 996522 0 2024-03-23T11:47:24.009 INFO:tasks.workunit.client.0.smithi063.stdout:1/37: write d3/f7 [830009,119751] 0 2024-03-23T11:47:24.010 INFO:tasks.workunit.client.0.smithi063.stdout:6/53: chown d1/f8 1429 1 2024-03-23T11:47:24.010 INFO:tasks.workunit.client.0.smithi063.stdout:1/38: chown d3/cb 1 1 2024-03-23T11:47:24.011 INFO:tasks.workunit.client.0.smithi063.stdout:1/39: fdatasync d3/f4 0 2024-03-23T11:47:24.013 INFO:tasks.workunit.client.0.smithi063.stdout:6/54: mknod d1/ce 0 2024-03-23T11:47:24.014 INFO:tasks.workunit.client.0.smithi063.stdout:1/40: mknod d3/ce 0 2024-03-23T11:47:24.015 INFO:tasks.workunit.client.0.smithi063.stdout:1/41: write f2 [54721,34623] 0 2024-03-23T11:47:24.016 INFO:tasks.workunit.client.0.smithi063.stdout:1/42: stat l1 0 2024-03-23T11:47:24.017 INFO:tasks.workunit.client.0.smithi063.stdout:1/43: write f2 [498480,25993] 0 2024-03-23T11:47:24.018 INFO:tasks.workunit.client.0.smithi063.stdout:1/44: chown d3/ca 521995583 1 2024-03-23T11:47:24.019 INFO:tasks.workunit.client.0.smithi063.stdout:1/45: dread - d3/fd zero size 2024-03-23T11:47:24.021 INFO:tasks.workunit.client.0.smithi063.stdout:1/46: mkdir d3/df 0 2024-03-23T11:47:24.030 INFO:tasks.workunit.client.0.smithi063.stdout:6/55: dwrite d1/f3 [0,4194304] 0 2024-03-23T11:47:24.031 INFO:tasks.workunit.client.0.smithi063.stdout:1/47: truncate d3/f9 2085348 0 2024-03-23T11:47:24.036 INFO:tasks.workunit.client.0.smithi063.stdout:1/48: fsync d3/f4 0 2024-03-23T11:47:24.037 INFO:tasks.workunit.client.0.smithi063.stdout:6/56: getdents d1 0 2024-03-23T11:47:24.040 INFO:tasks.workunit.client.0.smithi063.stdout:6/57: symlink d1/lf 0 2024-03-23T11:47:24.044 INFO:tasks.workunit.client.0.smithi063.stdout:6/58: rename d1/lf to d1/l10 0 2024-03-23T11:47:24.055 INFO:tasks.workunit.client.0.smithi063.stdout:1/49: dwrite d3/fd [0,4194304] 0 2024-03-23T11:47:24.058 INFO:tasks.workunit.client.0.smithi063.stdout:1/50: creat d3/f10 x:0 0 0 2024-03-23T11:47:24.060 INFO:tasks.workunit.client.0.smithi063.stdout:6/59: dread d1/f3 [0,4194304] 0 2024-03-23T11:47:24.063 INFO:tasks.workunit.client.0.smithi063.stdout:1/51: rename d3/fd to d3/f11 0 2024-03-23T11:47:24.068 INFO:tasks.workunit.client.0.smithi063.stdout:1/52: creat d3/df/f12 x:0 0 0 2024-03-23T11:47:24.069 INFO:tasks.workunit.client.0.smithi063.stdout:6/60: getdents d1/dc 0 2024-03-23T11:47:24.071 INFO:tasks.workunit.client.0.smithi063.stdout:1/53: symlink d3/df/l13 0 2024-03-23T11:47:24.074 INFO:tasks.workunit.client.0.smithi063.stdout:1/54: fdatasync d3/f11 0 2024-03-23T11:47:24.076 INFO:tasks.workunit.client.0.smithi063.stdout:6/61: getdents d1 0 2024-03-23T11:47:24.078 INFO:tasks.workunit.client.0.smithi063.stdout:6/62: read d1/f3 [4069762,83148] 0 2024-03-23T11:47:24.079 INFO:tasks.workunit.client.0.smithi063.stdout:1/55: getdents d3 0 2024-03-23T11:47:24.079 INFO:tasks.workunit.client.0.smithi063.stdout:6/63: readlink d1/l9 0 2024-03-23T11:47:24.081 INFO:tasks.workunit.client.0.smithi063.stdout:1/56: write f2 [1610485,16751] 0 2024-03-23T11:47:24.082 INFO:tasks.workunit.client.0.smithi063.stdout:6/64: truncate d1/f8 728719 0 2024-03-23T11:47:24.083 INFO:tasks.workunit.client.0.smithi063.stdout:1/57: write d3/f11 [2546607,128924] 0 2024-03-23T11:47:24.084 INFO:tasks.workunit.client.0.smithi063.stdout:6/65: mknod d1/dc/c11 0 2024-03-23T11:47:24.086 INFO:tasks.workunit.client.0.smithi063.stdout:6/66: fdatasync d1/dc/fd 0 2024-03-23T11:47:24.088 INFO:tasks.workunit.client.0.smithi063.stdout:1/58: getdents d3 0 2024-03-23T11:47:24.088 INFO:tasks.workunit.client.0.smithi063.stdout:1/59: stat d3/ce 0 2024-03-23T11:47:24.089 INFO:tasks.workunit.client.0.smithi063.stdout:6/67: creat d1/f12 x:0 0 0 2024-03-23T11:47:24.090 INFO:tasks.workunit.client.0.smithi063.stdout:1/60: symlink d3/df/l14 0 2024-03-23T11:47:24.093 INFO:tasks.workunit.client.0.smithi063.stdout:1/61: mknod d3/df/c15 0 2024-03-23T11:47:24.100 INFO:tasks.workunit.client.0.smithi063.stdout:6/68: truncate d1/f3 562866 0 2024-03-23T11:47:24.108 INFO:tasks.workunit.client.0.smithi063.stdout:1/62: dwrite d3/f9 [0,4194304] 0 2024-03-23T11:47:24.120 INFO:tasks.workunit.client.0.smithi063.stdout:6/69: dwrite d1/f5 [0,4194304] 0 2024-03-23T11:47:24.124 INFO:tasks.workunit.client.1.smithi111.stdout:4/51: sync 2024-03-23T11:47:24.124 INFO:tasks.workunit.client.0.smithi063.stdout:6/70: write d1/f8 [869245,13356] 0 2024-03-23T11:47:24.129 INFO:tasks.workunit.client.1.smithi111.stdout:4/52: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:24.133 INFO:tasks.workunit.client.0.smithi063.stdout:1/63: dwrite f2 [0,4194304] 0 2024-03-23T11:47:24.151 INFO:tasks.workunit.client.0.smithi063.stdout:6/71: dwrite d1/f12 [0,4194304] 0 2024-03-23T11:47:24.155 INFO:tasks.workunit.client.0.smithi063.stdout:6/72: mkdir d1/d13 0 2024-03-23T11:47:24.158 INFO:tasks.workunit.client.1.smithi111.stdout:0/95: sync 2024-03-23T11:47:24.158 INFO:tasks.workunit.client.1.smithi111.stdout:2/61: sync 2024-03-23T11:47:24.160 INFO:tasks.workunit.client.1.smithi111.stdout:0/96: write d5/db/d14/f18 [504938,99868] 0 2024-03-23T11:47:24.161 INFO:tasks.workunit.client.0.smithi063.stdout:1/64: dwrite d3/f4 [0,4194304] 0 2024-03-23T11:47:24.162 INFO:tasks.workunit.client.0.smithi063.stdout:6/73: readlink d1/l2 0 2024-03-23T11:47:24.162 INFO:tasks.workunit.client.1.smithi111.stdout:0/97: read - d5/db/d14/d19/f1c zero size 2024-03-23T11:47:24.167 INFO:tasks.workunit.client.0.smithi063.stdout:1/65: creat d3/f16 x:0 0 0 2024-03-23T11:47:24.168 INFO:tasks.workunit.client.1.smithi111.stdout:9/82: rename d0/d1/c7 to d0/d1/c1e 0 2024-03-23T11:47:24.170 INFO:tasks.workunit.client.1.smithi111.stdout:8/36: unlink d5/f7 0 2024-03-23T11:47:24.170 INFO:tasks.workunit.client.0.smithi063.stdout:1/66: mknod d3/c17 0 2024-03-23T11:47:24.171 INFO:tasks.workunit.client.1.smithi111.stdout:5/107: symlink d1/d3/d6/d7/l27 0 2024-03-23T11:47:24.173 INFO:tasks.workunit.client.0.smithi063.stdout:1/67: chown d3/df/c15 545548 1 2024-03-23T11:47:24.175 INFO:tasks.workunit.client.1.smithi111.stdout:3/51: truncate d0/f3 1057119 0 2024-03-23T11:47:24.177 INFO:tasks.workunit.client.1.smithi111.stdout:2/62: creat d2/f14 x:0 0 0 2024-03-23T11:47:24.181 INFO:tasks.workunit.client.1.smithi111.stdout:9/83: symlink d0/d4/d10/l1f 0 2024-03-23T11:47:24.182 INFO:tasks.workunit.client.0.smithi063.stdout:1/68: getdents d3/df 0 2024-03-23T11:47:24.188 INFO:tasks.workunit.client.1.smithi111.stdout:8/37: dwrite d5/fa [0,4194304] 0 2024-03-23T11:47:24.188 INFO:tasks.workunit.client.1.smithi111.stdout:4/53: link d0/f4 d0/d7/f16 0 2024-03-23T11:47:24.189 INFO:tasks.workunit.client.0.smithi063.stdout:1/69: creat d3/df/f18 x:0 0 0 2024-03-23T11:47:24.189 INFO:tasks.workunit.client.0.smithi063.stdout:6/74: dwrite d1/f7 [0,4194304] 0 2024-03-23T11:47:24.191 INFO:tasks.workunit.client.1.smithi111.stdout:3/52: unlink d0/d7/lb 0 2024-03-23T11:47:24.193 INFO:tasks.workunit.client.1.smithi111.stdout:2/63: mknod d2/c15 0 2024-03-23T11:47:24.194 INFO:tasks.workunit.client.1.smithi111.stdout:2/64: readlink d2/d7/da/l10 0 2024-03-23T11:47:24.194 INFO:tasks.workunit.client.0.smithi063.stdout:1/70: creat d3/df/f19 x:0 0 0 2024-03-23T11:47:24.204 INFO:tasks.workunit.client.1.smithi111.stdout:8/38: dread f3 [0,4194304] 0 2024-03-23T11:47:24.204 INFO:tasks.workunit.client.1.smithi111.stdout:7/50: getdents . 0 2024-03-23T11:47:24.205 INFO:tasks.workunit.client.1.smithi111.stdout:2/65: creat d2/de/f16 x:0 0 0 2024-03-23T11:47:24.207 INFO:tasks.workunit.client.1.smithi111.stdout:2/66: write d2/f6 [1974729,56633] 0 2024-03-23T11:47:24.212 INFO:tasks.workunit.client.1.smithi111.stdout:8/39: mknod d5/cd 0 2024-03-23T11:47:24.212 INFO:tasks.workunit.client.0.smithi063.stdout:6/75: dwrite d1/f8 [0,4194304] 0 2024-03-23T11:47:24.213 INFO:tasks.workunit.client.1.smithi111.stdout:2/67: mkdir d2/d11/d17 0 2024-03-23T11:47:24.213 INFO:tasks.workunit.client.0.smithi063.stdout:1/71: dread d3/f11 [0,4194304] 0 2024-03-23T11:47:24.214 INFO:tasks.workunit.client.1.smithi111.stdout:8/40: fsync f1 0 2024-03-23T11:47:24.215 INFO:tasks.workunit.client.0.smithi063.stdout:6/76: write d1/f5 [617216,115233] 0 2024-03-23T11:47:24.218 INFO:tasks.workunit.client.0.smithi063.stdout:1/72: unlink d3/f16 0 2024-03-23T11:47:24.219 INFO:tasks.workunit.client.1.smithi111.stdout:2/68: truncate f0 449732 0 2024-03-23T11:47:24.222 INFO:tasks.workunit.client.0.smithi063.stdout:6/77: symlink d1/d13/l14 0 2024-03-23T11:47:24.223 INFO:tasks.workunit.client.1.smithi111.stdout:2/69: fdatasync d2/d7/fd 0 2024-03-23T11:47:24.223 INFO:tasks.workunit.client.0.smithi063.stdout:6/78: fsync d1/f12 0 2024-03-23T11:47:24.227 INFO:tasks.workunit.client.0.smithi063.stdout:6/79: mkdir d1/d15 0 2024-03-23T11:47:24.228 INFO:tasks.workunit.client.0.smithi063.stdout:6/80: stat d1/d13 0 2024-03-23T11:47:24.230 INFO:tasks.workunit.client.1.smithi111.stdout:4/54: dwrite d0/f4 [0,4194304] 0 2024-03-23T11:47:24.231 INFO:tasks.workunit.client.1.smithi111.stdout:2/70: creat d2/de/f18 x:0 0 0 2024-03-23T11:47:24.231 INFO:tasks.workunit.client.0.smithi063.stdout:6/81: write d1/f5 [293772,97342] 0 2024-03-23T11:47:24.232 INFO:tasks.workunit.client.1.smithi111.stdout:1/48: sync 2024-03-23T11:47:24.234 INFO:tasks.workunit.client.0.smithi063.stdout:1/73: dread f2 [0,4194304] 0 2024-03-23T11:47:24.235 INFO:tasks.workunit.client.1.smithi111.stdout:7/51: dwrite f9 [0,4194304] 0 2024-03-23T11:47:24.236 INFO:tasks.workunit.client.1.smithi111.stdout:1/49: write d0/f4 [1118116,125308] 0 2024-03-23T11:47:24.237 INFO:tasks.workunit.client.1.smithi111.stdout:2/71: mkdir d2/d7/da/d19 0 2024-03-23T11:47:24.237 INFO:tasks.workunit.client.0.smithi063.stdout:0/31: sync 2024-03-23T11:47:24.237 INFO:tasks.workunit.client.1.smithi111.stdout:7/52: chown l3 23 1 2024-03-23T11:47:24.238 INFO:tasks.workunit.client.1.smithi111.stdout:1/50: write d0/fe [157273,37448] 0 2024-03-23T11:47:24.239 INFO:tasks.workunit.client.1.smithi111.stdout:4/55: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:24.240 INFO:tasks.workunit.client.1.smithi111.stdout:7/53: write f1 [156834,106309] 0 2024-03-23T11:47:24.246 INFO:tasks.workunit.client.0.smithi063.stdout:1/74: getdents d3 0 2024-03-23T11:47:24.249 INFO:tasks.workunit.client.0.smithi063.stdout:0/32: mkdir d2/d4/dd 0 2024-03-23T11:47:24.250 INFO:tasks.workunit.client.1.smithi111.stdout:1/51: mknod d0/d5/c14 0 2024-03-23T11:47:24.251 INFO:tasks.workunit.client.1.smithi111.stdout:1/52: fdatasync d0/f13 0 2024-03-23T11:47:24.251 INFO:tasks.workunit.client.1.smithi111.stdout:7/54: dread f7 [0,4194304] 0 2024-03-23T11:47:24.252 INFO:tasks.workunit.client.0.smithi063.stdout:1/75: unlink d3/c6 0 2024-03-23T11:47:24.255 INFO:tasks.workunit.client.0.smithi063.stdout:1/76: stat c0 0 2024-03-23T11:47:24.256 INFO:tasks.workunit.client.0.smithi063.stdout:6/82: dread d1/f5 [0,4194304] 0 2024-03-23T11:47:24.256 INFO:tasks.workunit.client.0.smithi063.stdout:0/33: creat d2/d4/fe x:0 0 0 2024-03-23T11:47:24.258 INFO:tasks.workunit.client.1.smithi111.stdout:1/53: symlink d0/l15 0 2024-03-23T11:47:24.262 INFO:tasks.workunit.client.1.smithi111.stdout:1/54: creat d0/d5/f16 x:0 0 0 2024-03-23T11:47:24.267 INFO:tasks.workunit.client.1.smithi111.stdout:1/55: write d0/d6/fc [193521,76558] 0 2024-03-23T11:47:24.269 INFO:tasks.workunit.client.0.smithi063.stdout:1/77: dread d3/f4 [0,4194304] 0 2024-03-23T11:47:24.272 INFO:tasks.workunit.client.0.smithi063.stdout:1/78: chown d3/df/f12 1840857706 1 2024-03-23T11:47:24.273 INFO:tasks.workunit.client.1.smithi111.stdout:7/55: dwrite fb [0,4194304] 0 2024-03-23T11:47:24.273 INFO:tasks.workunit.client.1.smithi111.stdout:1/56: link d0/l9 d0/d5/dd/l17 0 2024-03-23T11:47:24.274 INFO:tasks.workunit.client.1.smithi111.stdout:7/56: readlink l3 0 2024-03-23T11:47:24.274 INFO:tasks.workunit.client.0.smithi063.stdout:1/79: read - d3/df/f19 zero size 2024-03-23T11:47:24.277 INFO:tasks.workunit.client.1.smithi111.stdout:7/57: creat fd x:0 0 0 2024-03-23T11:47:24.281 INFO:tasks.workunit.client.0.smithi063.stdout:0/34: dwrite d2/f7 [0,4194304] 0 2024-03-23T11:47:24.281 INFO:tasks.workunit.client.0.smithi063.stdout:1/80: rename f2 to d3/df/f1a 0 2024-03-23T11:47:24.285 INFO:tasks.workunit.client.0.smithi063.stdout:1/81: write d3/df/f19 [1001140,52441] 0 2024-03-23T11:47:24.286 INFO:tasks.workunit.client.0.smithi063.stdout:0/35: creat d2/d4/dd/ff x:0 0 0 2024-03-23T11:47:24.287 INFO:tasks.workunit.client.0.smithi063.stdout:0/36: dread - d2/d4/f9 zero size 2024-03-23T11:47:24.291 INFO:tasks.workunit.client.0.smithi063.stdout:0/37: mknod d2/d4/dd/c10 0 2024-03-23T11:47:24.292 INFO:tasks.workunit.client.0.smithi063.stdout:1/82: link d3/cb d3/c1b 0 2024-03-23T11:47:24.295 INFO:tasks.workunit.client.0.smithi063.stdout:0/38: write d2/d4/f6 [7204210,105960] 0 2024-03-23T11:47:24.295 INFO:tasks.workunit.client.0.smithi063.stdout:1/83: read d3/df/f19 [421806,24362] 0 2024-03-23T11:47:24.299 INFO:tasks.workunit.client.1.smithi111.stdout:1/57: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:24.299 INFO:tasks.workunit.client.0.smithi063.stdout:1/84: fdatasync d3/df/f1a 0 2024-03-23T11:47:24.305 INFO:tasks.workunit.client.1.smithi111.stdout:7/58: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:24.306 INFO:tasks.workunit.client.1.smithi111.stdout:7/59: chown cc 492201690 1 2024-03-23T11:47:24.308 INFO:tasks.workunit.client.1.smithi111.stdout:1/58: chown d0/d5/dd/l17 2422 1 2024-03-23T11:47:24.314 INFO:tasks.workunit.client.1.smithi111.stdout:6/142: truncate d2/f16 798293 0 2024-03-23T11:47:24.318 INFO:tasks.workunit.client.1.smithi111.stdout:7/60: dread f9 [0,4194304] 0 2024-03-23T11:47:24.319 INFO:tasks.workunit.client.1.smithi111.stdout:4/56: fdatasync d0/d7/d9/ff 0 2024-03-23T11:47:24.321 INFO:tasks.workunit.client.1.smithi111.stdout:5/108: truncate d1/d3/d6/d7/f9 2138432 0 2024-03-23T11:47:24.324 INFO:tasks.workunit.client.1.smithi111.stdout:4/57: chown d0/dc/cd 2288 1 2024-03-23T11:47:24.327 INFO:tasks.workunit.client.1.smithi111.stdout:0/98: dwrite d5/f8 [0,4194304] 0 2024-03-23T11:47:24.327 INFO:tasks.workunit.client.1.smithi111.stdout:3/53: truncate d0/fd 4039758 0 2024-03-23T11:47:24.330 INFO:tasks.workunit.client.1.smithi111.stdout:2/72: getdents d2/d11 0 2024-03-23T11:47:24.330 INFO:tasks.workunit.client.1.smithi111.stdout:8/41: truncate d5/fa 76515 0 2024-03-23T11:47:24.333 INFO:tasks.workunit.client.0.smithi063.stdout:0/39: dwrite d2/d4/f6 [4194304,4194304] 0 2024-03-23T11:47:24.335 INFO:tasks.workunit.client.1.smithi111.stdout:8/42: creat d5/fe x:0 0 0 2024-03-23T11:47:24.335 INFO:tasks.workunit.client.1.smithi111.stdout:4/58: write d0/d7/da/f13 [666761,23378] 0 2024-03-23T11:47:24.337 INFO:tasks.workunit.client.1.smithi111.stdout:0/99: symlink d5/db/d14/l1d 0 2024-03-23T11:47:24.338 INFO:tasks.workunit.client.1.smithi111.stdout:9/84: dwrite d0/fd [0,4194304] 0 2024-03-23T11:47:24.338 INFO:tasks.workunit.client.1.smithi111.stdout:0/100: dread - d5/f13 zero size 2024-03-23T11:47:24.339 INFO:tasks.workunit.client.0.smithi063.stdout:1/85: dwrite d3/df/f12 [0,4194304] 0 2024-03-23T11:47:24.340 INFO:tasks.workunit.client.1.smithi111.stdout:9/85: fsync d0/d1/f18 0 2024-03-23T11:47:24.341 INFO:tasks.workunit.client.1.smithi111.stdout:6/143: rename d2/ce to d2/d1a/c32 0 2024-03-23T11:47:24.342 INFO:tasks.workunit.client.1.smithi111.stdout:2/73: symlink d2/de/l1a 0 2024-03-23T11:47:24.346 INFO:tasks.workunit.client.1.smithi111.stdout:5/109: truncate d1/d3/d6/f8 4504022 0 2024-03-23T11:47:24.346 INFO:tasks.workunit.client.1.smithi111.stdout:2/74: dread - d2/d11/f13 zero size 2024-03-23T11:47:24.347 INFO:tasks.workunit.client.0.smithi063.stdout:0/40: unlink d2/d4/fe 0 2024-03-23T11:47:24.354 INFO:tasks.workunit.client.0.smithi063.stdout:1/86: getdents d3/df 0 2024-03-23T11:47:24.355 INFO:tasks.workunit.client.0.smithi063.stdout:0/41: creat d2/d4/dd/f11 x:0 0 0 2024-03-23T11:47:24.358 INFO:tasks.workunit.client.1.smithi111.stdout:4/59: mkdir d0/d17 0 2024-03-23T11:47:24.359 INFO:tasks.workunit.client.1.smithi111.stdout:9/86: creat d0/d1/f20 x:0 0 0 2024-03-23T11:47:24.360 INFO:tasks.workunit.client.1.smithi111.stdout:6/144: fsync d2/f31 0 2024-03-23T11:47:24.362 INFO:tasks.workunit.client.0.smithi063.stdout:1/87: mkdir d3/d1c 0 2024-03-23T11:47:24.363 INFO:tasks.workunit.client.1.smithi111.stdout:2/75: creat d2/d11/f1b x:0 0 0 2024-03-23T11:47:24.363 INFO:tasks.workunit.client.1.smithi111.stdout:4/60: truncate d0/dc/f14 106408 0 2024-03-23T11:47:24.365 INFO:tasks.workunit.client.1.smithi111.stdout:0/101: dwrite d5/f8 [0,4194304] 0 2024-03-23T11:47:24.366 INFO:tasks.workunit.client.1.smithi111.stdout:4/61: dread - d0/d7/d9/ff zero size 2024-03-23T11:47:24.370 INFO:tasks.workunit.client.1.smithi111.stdout:2/76: mknod d2/c1c 0 2024-03-23T11:47:24.371 INFO:tasks.workunit.client.1.smithi111.stdout:2/77: write d2/f6 [1097257,68810] 0 2024-03-23T11:47:24.374 INFO:tasks.workunit.client.1.smithi111.stdout:9/87: unlink d0/d1/c1e 0 2024-03-23T11:47:24.380 INFO:tasks.workunit.client.0.smithi063.stdout:1/88: dread d3/f9 [0,4194304] 0 2024-03-23T11:47:24.382 INFO:tasks.workunit.client.1.smithi111.stdout:4/62: creat d0/dc/f18 x:0 0 0 2024-03-23T11:47:24.387 INFO:tasks.workunit.client.1.smithi111.stdout:2/78: creat d2/d7/da/f1d x:0 0 0 2024-03-23T11:47:24.387 INFO:tasks.workunit.client.0.smithi063.stdout:0/42: dwrite d2/d4/dd/ff [0,4194304] 0 2024-03-23T11:47:24.389 INFO:tasks.workunit.client.1.smithi111.stdout:2/79: rename d2/de/d12 to d2/de/d12/d1e 22 2024-03-23T11:47:24.390 INFO:tasks.workunit.client.0.smithi063.stdout:0/43: mkdir d2/d12 0 2024-03-23T11:47:24.391 INFO:tasks.workunit.client.1.smithi111.stdout:9/88: creat d0/f21 x:0 0 0 2024-03-23T11:47:24.393 INFO:tasks.workunit.client.1.smithi111.stdout:9/89: chown d0/d4/f9 3998 1 2024-03-23T11:47:24.394 INFO:tasks.workunit.client.1.smithi111.stdout:9/90: readlink d0/d1/l8 0 2024-03-23T11:47:24.395 INFO:tasks.workunit.client.0.smithi063.stdout:0/44: getdents d2/d12 0 2024-03-23T11:47:24.397 INFO:tasks.workunit.client.1.smithi111.stdout:0/102: link d5/c7 d5/db/d14/d19/c1e 0 2024-03-23T11:47:24.398 INFO:tasks.workunit.client.1.smithi111.stdout:9/91: write d0/d4/d10/f17 [819522,35793] 0 2024-03-23T11:47:24.398 INFO:tasks.workunit.client.1.smithi111.stdout:4/63: dwrite d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:24.400 INFO:tasks.workunit.client.1.smithi111.stdout:0/103: chown d5/db/d14/f18 1545 1 2024-03-23T11:47:24.401 INFO:tasks.workunit.client.1.smithi111.stdout:2/80: mkdir d2/d11/d17/d1f 0 2024-03-23T11:47:24.402 INFO:tasks.workunit.client.0.smithi063.stdout:0/45: mkdir d2/d4/d13 0 2024-03-23T11:47:24.403 INFO:tasks.workunit.client.1.smithi111.stdout:2/81: readlink d2/d7/da/l10 0 2024-03-23T11:47:24.404 INFO:tasks.workunit.client.1.smithi111.stdout:4/64: chown d0/c3 121298 1 2024-03-23T11:47:24.405 INFO:tasks.workunit.client.1.smithi111.stdout:6/145: truncate d2/fb 241448 0 2024-03-23T11:47:24.405 INFO:tasks.workunit.client.1.smithi111.stdout:4/65: read d0/f4 [3275412,49677] 0 2024-03-23T11:47:24.406 INFO:tasks.workunit.client.1.smithi111.stdout:4/66: chown d0/c5 126049 1 2024-03-23T11:47:24.407 INFO:tasks.workunit.client.1.smithi111.stdout:4/67: chown d0/d7/d9 0 1 2024-03-23T11:47:24.408 INFO:tasks.workunit.client.0.smithi063.stdout:0/46: mkdir d2/d14 0 2024-03-23T11:47:24.408 INFO:tasks.workunit.client.1.smithi111.stdout:9/92: mknod d0/d4/d10/c22 0 2024-03-23T11:47:24.411 INFO:tasks.workunit.client.0.smithi063.stdout:1/89: dwrite d3/f7 [0,4194304] 0 2024-03-23T11:47:24.412 INFO:tasks.workunit.client.0.smithi063.stdout:0/47: truncate d2/fc 483344 0 2024-03-23T11:47:24.421 INFO:tasks.workunit.client.1.smithi111.stdout:2/82: creat d2/d11/d17/d1f/f20 x:0 0 0 2024-03-23T11:47:24.426 INFO:tasks.workunit.client.1.smithi111.stdout:2/83: creat d2/d11/d17/f21 x:0 0 0 2024-03-23T11:47:24.427 INFO:tasks.workunit.client.1.smithi111.stdout:0/104: getdents d5/db 0 2024-03-23T11:47:24.429 INFO:tasks.workunit.client.1.smithi111.stdout:9/93: getdents d0/d4 0 2024-03-23T11:47:24.429 INFO:tasks.workunit.client.1.smithi111.stdout:0/105: dread - d5/db/d14/f1a zero size 2024-03-23T11:47:24.430 INFO:tasks.workunit.client.1.smithi111.stdout:2/84: mknod d2/de/d12/c22 0 2024-03-23T11:47:24.430 INFO:tasks.workunit.client.1.smithi111.stdout:9/94: write d0/d4/f1b [4586236,27796] 0 2024-03-23T11:47:24.431 INFO:tasks.workunit.client.1.smithi111.stdout:2/85: write d2/fc [3669968,119529] 0 2024-03-23T11:47:24.432 INFO:tasks.workunit.client.0.smithi063.stdout:1/90: dwrite d3/df/f1a [4194304,4194304] 0 2024-03-23T11:47:24.434 INFO:tasks.workunit.client.0.smithi063.stdout:1/91: fdatasync d3/f11 0 2024-03-23T11:47:24.435 INFO:tasks.workunit.client.1.smithi111.stdout:2/86: unlink f0 0 2024-03-23T11:47:24.437 INFO:tasks.workunit.client.0.smithi063.stdout:1/92: stat d3/c1b 0 2024-03-23T11:47:24.440 INFO:tasks.workunit.client.1.smithi111.stdout:9/95: link d0/d4/cb d0/d4/d1c/c23 0 2024-03-23T11:47:24.443 INFO:tasks.workunit.client.1.smithi111.stdout:9/96: unlink d0/d4/d10/c22 0 2024-03-23T11:47:24.444 INFO:tasks.workunit.client.1.smithi111.stdout:1/59: sync 2024-03-23T11:47:24.446 INFO:tasks.workunit.client.1.smithi111.stdout:1/60: mknod d0/c18 0 2024-03-23T11:47:24.447 INFO:tasks.workunit.client.1.smithi111.stdout:9/97: mknod d0/d4/c24 0 2024-03-23T11:47:24.447 INFO:tasks.workunit.client.1.smithi111.stdout:1/61: stat d0/f4 0 2024-03-23T11:47:24.447 INFO:tasks.workunit.client.1.smithi111.stdout:9/98: fdatasync d0/d4/f14 0 2024-03-23T11:47:24.449 INFO:tasks.workunit.client.1.smithi111.stdout:9/99: write d0/d4/d10/f17 [1726368,121479] 0 2024-03-23T11:47:24.450 INFO:tasks.workunit.client.0.smithi063.stdout:1/93: dread d3/df/f19 [0,4194304] 0 2024-03-23T11:47:24.451 INFO:tasks.workunit.client.0.smithi063.stdout:7/102: sync 2024-03-23T11:47:24.451 INFO:tasks.workunit.client.1.smithi111.stdout:9/100: symlink d0/d4/l25 0 2024-03-23T11:47:24.452 INFO:tasks.workunit.client.0.smithi063.stdout:1/94: chown d3/df/f19 9020592 1 2024-03-23T11:47:24.455 INFO:tasks.workunit.client.0.smithi063.stdout:1/95: rmdir d3 39 2024-03-23T11:47:24.456 INFO:tasks.workunit.client.1.smithi111.stdout:9/101: rename d0/d4/f9 to d0/d4/d1c/f26 0 2024-03-23T11:47:24.459 INFO:tasks.workunit.client.0.smithi063.stdout:1/96: chown d3/d1c 718 1 2024-03-23T11:47:24.461 INFO:tasks.workunit.client.1.smithi111.stdout:9/102: symlink d0/l27 0 2024-03-23T11:47:24.461 INFO:tasks.workunit.client.0.smithi063.stdout:0/48: write f1 [197051,5071] 0 2024-03-23T11:47:24.462 INFO:tasks.workunit.client.0.smithi063.stdout:1/97: symlink d3/d1c/l1d 0 2024-03-23T11:47:24.463 INFO:tasks.workunit.client.1.smithi111.stdout:9/103: dread - d0/d1/f18 zero size 2024-03-23T11:47:24.465 INFO:tasks.workunit.client.1.smithi111.stdout:1/62: dwrite d0/d5/f16 [0,4194304] 0 2024-03-23T11:47:24.465 INFO:tasks.workunit.client.1.smithi111.stdout:2/87: fsync d2/d11/d17/f21 0 2024-03-23T11:47:24.466 INFO:tasks.workunit.client.0.smithi063.stdout:1/98: creat d3/d1c/f1e x:0 0 0 2024-03-23T11:47:24.466 INFO:tasks.workunit.client.1.smithi111.stdout:1/63: stat d0/d6/f11 0 2024-03-23T11:47:24.466 INFO:tasks.workunit.client.1.smithi111.stdout:2/88: chown d2/d11/d17/f21 81 1 2024-03-23T11:47:24.467 INFO:tasks.workunit.client.0.smithi063.stdout:0/49: getdents d2/d4/d13 0 2024-03-23T11:47:24.467 INFO:tasks.workunit.client.1.smithi111.stdout:9/104: mkdir d0/d1/d28 0 2024-03-23T11:47:24.468 INFO:tasks.workunit.client.0.smithi063.stdout:7/103: dread f4 [0,4194304] 0 2024-03-23T11:47:24.468 INFO:tasks.workunit.client.1.smithi111.stdout:2/89: write d2/d7/fd [2381201,99898] 0 2024-03-23T11:47:24.469 INFO:tasks.workunit.client.0.smithi063.stdout:7/104: truncate f1 1848136 0 2024-03-23T11:47:24.469 INFO:tasks.workunit.client.0.smithi063.stdout:1/99: mknod d3/d1c/c1f 0 2024-03-23T11:47:24.469 INFO:tasks.workunit.client.0.smithi063.stdout:0/50: write d2/f7 [2546113,113590] 0 2024-03-23T11:47:24.470 INFO:tasks.workunit.client.1.smithi111.stdout:9/105: truncate d0/d1/f11 5156536 0 2024-03-23T11:47:24.471 INFO:tasks.workunit.client.1.smithi111.stdout:2/90: readlink d2/d7/da/l10 0 2024-03-23T11:47:24.472 INFO:tasks.workunit.client.0.smithi063.stdout:1/100: write d3/d1c/f1e [359060,70575] 0 2024-03-23T11:47:24.472 INFO:tasks.workunit.client.0.smithi063.stdout:9/23: sync 2024-03-23T11:47:24.472 INFO:tasks.workunit.client.0.smithi063.stdout:3/195: sync 2024-03-23T11:47:24.473 INFO:tasks.workunit.client.1.smithi111.stdout:1/64: getdents d0 0 2024-03-23T11:47:24.473 INFO:tasks.workunit.client.1.smithi111.stdout:9/106: read d0/d4/fe [1493323,102708] 0 2024-03-23T11:47:24.473 INFO:tasks.workunit.client.0.smithi063.stdout:5/66: sync 2024-03-23T11:47:24.474 INFO:tasks.workunit.client.0.smithi063.stdout:2/60: sync 2024-03-23T11:47:24.474 INFO:tasks.workunit.client.0.smithi063.stdout:8/267: sync 2024-03-23T11:47:24.474 INFO:tasks.workunit.client.0.smithi063.stdout:4/86: sync 2024-03-23T11:47:24.474 INFO:tasks.workunit.client.0.smithi063.stdout:6/83: sync 2024-03-23T11:47:24.474 INFO:tasks.workunit.client.0.smithi063.stdout:7/105: symlink d5/d6/l1d 0 2024-03-23T11:47:24.474 INFO:tasks.workunit.client.0.smithi063.stdout:9/24: chown d4 1343 1 2024-03-23T11:47:24.477 INFO:tasks.workunit.client.1.smithi111.stdout:1/65: write d0/d5/dd/f10 [870910,13976] 0 2024-03-23T11:47:24.478 INFO:tasks.workunit.client.0.smithi063.stdout:0/51: rename d2/d4/dd/f11 to d2/d14/f15 0 2024-03-23T11:47:24.478 INFO:tasks.workunit.client.1.smithi111.stdout:2/91: rename d2/d7/da/l10 to d2/d7/da/l23 0 2024-03-23T11:47:24.479 INFO:tasks.workunit.client.1.smithi111.stdout:9/107: mkdir d0/d4/d10/d29 0 2024-03-23T11:47:24.482 INFO:tasks.workunit.client.1.smithi111.stdout:2/92: write d2/f14 [899488,41209] 0 2024-03-23T11:47:24.484 INFO:tasks.workunit.client.1.smithi111.stdout:1/66: creat d0/d6/f19 x:0 0 0 2024-03-23T11:47:24.486 INFO:tasks.workunit.client.1.smithi111.stdout:2/93: dread - d2/d11/f1b zero size 2024-03-23T11:47:24.488 INFO:tasks.workunit.client.1.smithi111.stdout:1/67: creat d0/f1a x:0 0 0 2024-03-23T11:47:24.490 INFO:tasks.workunit.client.0.smithi063.stdout:7/106: chown d5/l14 470078047 1 2024-03-23T11:47:24.490 INFO:tasks.workunit.client.1.smithi111.stdout:6/146: dwrite d2/fb [0,4194304] 0 2024-03-23T11:47:24.490 INFO:tasks.workunit.client.0.smithi063.stdout:2/61: chown d2/lb 219348883 1 2024-03-23T11:47:24.493 INFO:tasks.workunit.client.1.smithi111.stdout:6/147: chown d2/cc 468264354 1 2024-03-23T11:47:24.493 INFO:tasks.workunit.client.0.smithi063.stdout:6/84: rename d1/f3 to d1/d13/f16 0 2024-03-23T11:47:24.494 INFO:tasks.workunit.client.1.smithi111.stdout:6/148: dread - d2/d1f/f2e zero size 2024-03-23T11:47:24.495 INFO:tasks.workunit.client.1.smithi111.stdout:1/68: symlink d0/d5/l1b 0 2024-03-23T11:47:24.497 INFO:tasks.workunit.client.1.smithi111.stdout:9/108: dread d0/fd [0,4194304] 0 2024-03-23T11:47:24.498 INFO:tasks.workunit.client.0.smithi063.stdout:5/67: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:24.498 INFO:tasks.workunit.client.0.smithi063.stdout:0/52: unlink d2/f7 0 2024-03-23T11:47:24.499 INFO:tasks.workunit.client.1.smithi111.stdout:2/94: rename d2/d7/da/f1d to d2/d11/f24 0 2024-03-23T11:47:24.500 INFO:tasks.workunit.client.1.smithi111.stdout:9/109: write d0/d4/f15 [309887,58196] 0 2024-03-23T11:47:24.500 INFO:tasks.workunit.client.1.smithi111.stdout:2/95: readlink d2/l9 0 2024-03-23T11:47:24.503 INFO:tasks.workunit.client.1.smithi111.stdout:1/69: symlink d0/d5/l1c 0 2024-03-23T11:47:24.504 INFO:tasks.workunit.client.0.smithi063.stdout:0/53: dread - d2/d4/f9 zero size 2024-03-23T11:47:24.504 INFO:tasks.workunit.client.0.smithi063.stdout:7/107: mknod d5/d16/c1e 0 2024-03-23T11:47:24.505 INFO:tasks.workunit.client.1.smithi111.stdout:1/70: chown d0/d6/f19 3369735 1 2024-03-23T11:47:24.505 INFO:tasks.workunit.client.1.smithi111.stdout:6/149: creat d2/d17/f33 x:0 0 0 2024-03-23T11:47:24.505 INFO:tasks.workunit.client.0.smithi063.stdout:2/62: chown d2/de/l13 5456675 1 2024-03-23T11:47:24.506 INFO:tasks.workunit.client.1.smithi111.stdout:9/110: symlink d0/d4/l2a 0 2024-03-23T11:47:24.506 INFO:tasks.workunit.client.1.smithi111.stdout:6/150: chown d2/fb 233723 1 2024-03-23T11:47:24.507 INFO:tasks.workunit.client.1.smithi111.stdout:2/96: mknod d2/d11/d17/c25 0 2024-03-23T11:47:24.507 INFO:tasks.workunit.client.0.smithi063.stdout:5/68: fsync d2/fc 0 2024-03-23T11:47:24.509 INFO:tasks.workunit.client.1.smithi111.stdout:6/151: write d2/d1a/f2d [558823,100058] 0 2024-03-23T11:47:24.511 INFO:tasks.workunit.client.1.smithi111.stdout:2/97: symlink d2/de/d12/l26 0 2024-03-23T11:47:24.513 INFO:tasks.workunit.client.1.smithi111.stdout:2/98: truncate d2/d11/f13 647019 0 2024-03-23T11:47:24.515 INFO:tasks.workunit.client.0.smithi063.stdout:7/108: write d5/d9/da/f17 [1321423,126189] 0 2024-03-23T11:47:24.517 INFO:tasks.workunit.client.1.smithi111.stdout:2/99: mkdir d2/de/d27 0 2024-03-23T11:47:24.518 INFO:tasks.workunit.client.1.smithi111.stdout:1/71: rmdir d0 39 2024-03-23T11:47:24.518 INFO:tasks.workunit.client.0.smithi063.stdout:6/85: mknod d1/d15/c17 0 2024-03-23T11:47:24.524 INFO:tasks.workunit.client.1.smithi111.stdout:6/152: getdents d2/d1f 0 2024-03-23T11:47:24.525 INFO:tasks.workunit.client.1.smithi111.stdout:6/153: fdatasync d2/fb 0 2024-03-23T11:47:24.527 INFO:tasks.workunit.client.0.smithi063.stdout:2/63: creat d2/de/f15 x:0 0 0 2024-03-23T11:47:24.527 INFO:tasks.workunit.client.1.smithi111.stdout:2/100: write d2/d11/f24 [310093,68636] 0 2024-03-23T11:47:24.529 INFO:tasks.workunit.client.1.smithi111.stdout:9/111: dwrite d0/d4/d1c/f26 [0,4194304] 0 2024-03-23T11:47:24.529 INFO:tasks.workunit.client.0.smithi063.stdout:5/69: creat d2/d7/f14 x:0 0 0 2024-03-23T11:47:24.531 INFO:tasks.workunit.client.1.smithi111.stdout:9/112: dread - d0/d1/f18 zero size 2024-03-23T11:47:24.532 INFO:tasks.workunit.client.1.smithi111.stdout:6/154: symlink d2/d1a/d1c/l34 0 2024-03-23T11:47:24.534 INFO:tasks.workunit.client.0.smithi063.stdout:8/268: link d6/dd/l2f d6/d1c/d32/d37/l5b 0 2024-03-23T11:47:24.534 INFO:tasks.workunit.client.1.smithi111.stdout:2/101: creat d2/de/d27/f28 x:0 0 0 2024-03-23T11:47:24.534 INFO:tasks.workunit.client.1.smithi111.stdout:6/155: dread - d2/d1a/f22 zero size 2024-03-23T11:47:24.538 INFO:tasks.workunit.client.0.smithi063.stdout:9/25: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:24.543 INFO:tasks.workunit.client.1.smithi111.stdout:1/72: dwrite d0/d6/fc [0,4194304] 0 2024-03-23T11:47:24.544 INFO:tasks.workunit.client.1.smithi111.stdout:2/102: creat d2/de/d12/f29 x:0 0 0 2024-03-23T11:47:24.544 INFO:tasks.workunit.client.0.smithi063.stdout:3/196: dread f6 [0,4194304] 0 2024-03-23T11:47:24.544 INFO:tasks.workunit.client.1.smithi111.stdout:1/73: readlink d0/d5/l1b 0 2024-03-23T11:47:24.545 INFO:tasks.workunit.client.0.smithi063.stdout:5/70: mknod d2/d7/c15 0 2024-03-23T11:47:24.547 INFO:tasks.workunit.client.0.smithi063.stdout:1/101: dwrite d3/f7 [4194304,4194304] 0 2024-03-23T11:47:24.547 INFO:tasks.workunit.client.1.smithi111.stdout:6/156: mkdir d2/d35 0 2024-03-23T11:47:24.547 INFO:tasks.workunit.client.1.smithi111.stdout:6/157: stat d2/d17/f20 0 2024-03-23T11:47:24.548 INFO:tasks.workunit.client.1.smithi111.stdout:9/113: getdents d0/d4/d10 0 2024-03-23T11:47:24.548 INFO:tasks.workunit.client.1.smithi111.stdout:1/74: write d0/d5/f16 [3177710,80377] 0 2024-03-23T11:47:24.548 INFO:tasks.workunit.client.1.smithi111.stdout:2/103: write d2/d11/f13 [167381,50906] 0 2024-03-23T11:47:24.549 INFO:tasks.workunit.client.1.smithi111.stdout:7/61: sync 2024-03-23T11:47:24.550 INFO:tasks.workunit.client.0.smithi063.stdout:2/64: rename d2/de/l13 to d2/l16 0 2024-03-23T11:47:24.551 INFO:tasks.workunit.client.1.smithi111.stdout:7/62: read f2 [5669884,10990] 0 2024-03-23T11:47:24.552 INFO:tasks.workunit.client.1.smithi111.stdout:9/114: mknod d0/c2b 0 2024-03-23T11:47:24.554 INFO:tasks.workunit.client.1.smithi111.stdout:2/104: symlink d2/d7/da/l2a 0 2024-03-23T11:47:24.555 INFO:tasks.workunit.client.1.smithi111.stdout:6/158: mknod d2/c36 0 2024-03-23T11:47:24.558 INFO:tasks.workunit.client.0.smithi063.stdout:4/87: dwrite d5/da/fd [0,4194304] 0 2024-03-23T11:47:24.558 INFO:tasks.workunit.client.1.smithi111.stdout:9/115: getdents d0/d1/d28 0 2024-03-23T11:47:24.561 INFO:tasks.workunit.client.1.smithi111.stdout:2/105: symlink d2/d7/l2b 0 2024-03-23T11:47:24.562 INFO:tasks.workunit.client.0.smithi063.stdout:6/86: link d1/dc/c11 d1/d15/c18 0 2024-03-23T11:47:24.563 INFO:tasks.workunit.client.0.smithi063.stdout:3/197: symlink d13/d15/l34 0 2024-03-23T11:47:24.564 INFO:tasks.workunit.client.1.smithi111.stdout:2/106: chown d2/de/d12/f29 322598430 1 2024-03-23T11:47:24.565 INFO:tasks.workunit.client.0.smithi063.stdout:8/269: mkdir d6/d1f/d54/d5c 0 2024-03-23T11:47:24.566 INFO:tasks.workunit.client.1.smithi111.stdout:7/63: getdents . 0 2024-03-23T11:47:24.567 INFO:tasks.workunit.client.1.smithi111.stdout:6/159: dread d2/d1a/d1c/f24 [0,4194304] 0 2024-03-23T11:47:24.567 INFO:tasks.workunit.client.0.smithi063.stdout:9/26: creat d4/f5 x:0 0 0 2024-03-23T11:47:24.568 INFO:tasks.workunit.client.1.smithi111.stdout:9/116: unlink d0/fd 0 2024-03-23T11:47:24.568 INFO:tasks.workunit.client.1.smithi111.stdout:2/107: dread - d2/d11/d17/f21 zero size 2024-03-23T11:47:24.569 INFO:tasks.workunit.client.1.smithi111.stdout:7/64: truncate f7 4592220 0 2024-03-23T11:47:24.575 INFO:tasks.workunit.client.0.smithi063.stdout:2/65: read d2/f4 [719447,124856] 0 2024-03-23T11:47:24.577 INFO:tasks.workunit.client.0.smithi063.stdout:7/109: dwrite f4 [0,4194304] 0 2024-03-23T11:47:24.577 INFO:tasks.workunit.client.0.smithi063.stdout:1/102: symlink d3/df/l20 0 2024-03-23T11:47:24.577 INFO:tasks.workunit.client.0.smithi063.stdout:5/71: symlink d2/d7/d8/de/l16 0 2024-03-23T11:47:24.579 INFO:tasks.workunit.client.0.smithi063.stdout:2/66: readlink d2/de/l10 0 2024-03-23T11:47:24.584 INFO:tasks.workunit.client.1.smithi111.stdout:9/117: dread d0/d4/f1b [0,4194304] 0 2024-03-23T11:47:24.584 INFO:tasks.workunit.client.0.smithi063.stdout:0/54: dread d2/d4/dd/ff [0,4194304] 0 2024-03-23T11:47:24.584 INFO:tasks.workunit.client.1.smithi111.stdout:6/160: dread d2/f8 [0,4194304] 0 2024-03-23T11:47:24.585 INFO:tasks.workunit.client.0.smithi063.stdout:6/87: chown d1/d13/f16 96510 1 2024-03-23T11:47:24.586 INFO:tasks.workunit.client.0.smithi063.stdout:3/198: symlink d13/d22/d2d/l35 0 2024-03-23T11:47:24.587 INFO:tasks.workunit.client.1.smithi111.stdout:7/65: creat fe x:0 0 0 2024-03-23T11:47:24.589 INFO:tasks.workunit.client.1.smithi111.stdout:2/108: dwrite d2/f14 [0,4194304] 0 2024-03-23T11:47:24.591 INFO:tasks.workunit.client.0.smithi063.stdout:3/199: readlink d13/d15/l1e 0 2024-03-23T11:47:24.595 INFO:tasks.workunit.client.0.smithi063.stdout:8/270: mkdir d6/d1f/d26/d5d 0 2024-03-23T11:47:24.599 INFO:tasks.workunit.client.1.smithi111.stdout:6/161: mknod d2/d1a/d1c/d2c/c37 0 2024-03-23T11:47:24.600 INFO:tasks.workunit.client.0.smithi063.stdout:5/72: mkdir d2/d17 0 2024-03-23T11:47:24.603 INFO:tasks.workunit.client.1.smithi111.stdout:2/109: creat d2/de/d12/f2c x:0 0 0 2024-03-23T11:47:24.604 INFO:tasks.workunit.client.1.smithi111.stdout:6/162: symlink d2/d17/l38 0 2024-03-23T11:47:24.605 INFO:tasks.workunit.client.1.smithi111.stdout:6/163: chown d2/f31 10358247 1 2024-03-23T11:47:24.606 INFO:tasks.workunit.client.1.smithi111.stdout:2/110: truncate d2/d11/f13 1609333 0 2024-03-23T11:47:24.609 INFO:tasks.workunit.client.0.smithi063.stdout:7/110: mkdir d5/d6/d19/d1f 0 2024-03-23T11:47:24.610 INFO:tasks.workunit.client.1.smithi111.stdout:9/118: dread d0/d4/f15 [0,4194304] 0 2024-03-23T11:47:24.616 INFO:tasks.workunit.client.0.smithi063.stdout:1/103: rename d3/c17 to d3/df/c21 0 2024-03-23T11:47:24.616 INFO:tasks.workunit.client.1.smithi111.stdout:7/66: dwrite f9 [0,4194304] 0 2024-03-23T11:47:24.619 INFO:tasks.workunit.client.1.smithi111.stdout:2/111: dread d2/fc [0,4194304] 0 2024-03-23T11:47:24.620 INFO:tasks.workunit.client.1.smithi111.stdout:6/164: truncate d2/f28 598698 0 2024-03-23T11:47:24.621 INFO:tasks.workunit.client.1.smithi111.stdout:2/112: chown d2/c15 209422 1 2024-03-23T11:47:24.625 INFO:tasks.workunit.client.1.smithi111.stdout:7/67: fsync f1 0 2024-03-23T11:47:24.625 INFO:tasks.workunit.client.0.smithi063.stdout:5/73: symlink d2/d7/l18 0 2024-03-23T11:47:24.626 INFO:tasks.workunit.client.0.smithi063.stdout:2/67: link d2/f4 d2/f17 0 2024-03-23T11:47:24.631 INFO:tasks.workunit.client.1.smithi111.stdout:6/165: rename d2/d1a/c30 to d2/d2a/c39 0 2024-03-23T11:47:24.632 INFO:tasks.workunit.client.0.smithi063.stdout:4/88: dwrite d5/ff [0,4194304] 0 2024-03-23T11:47:24.633 INFO:tasks.workunit.client.0.smithi063.stdout:1/104: chown d3/df/l20 20732126 1 2024-03-23T11:47:24.634 INFO:tasks.workunit.client.0.smithi063.stdout:5/74: fsync d2/d7/fd 0 2024-03-23T11:47:24.637 INFO:tasks.workunit.client.0.smithi063.stdout:0/55: getdents d2/d4/d13 0 2024-03-23T11:47:24.643 INFO:tasks.workunit.client.0.smithi063.stdout:5/75: stat d2/d7/f14 0 2024-03-23T11:47:24.643 INFO:tasks.workunit.client.0.smithi063.stdout:6/88: dread d1/f7 [0,4194304] 0 2024-03-23T11:47:24.643 INFO:tasks.workunit.client.1.smithi111.stdout:2/113: dwrite d2/d11/d17/d1f/f20 [0,4194304] 0 2024-03-23T11:47:24.645 INFO:tasks.workunit.client.1.smithi111.stdout:8/43: sync 2024-03-23T11:47:24.645 INFO:tasks.workunit.client.1.smithi111.stdout:3/54: sync 2024-03-23T11:47:24.645 INFO:tasks.workunit.client.1.smithi111.stdout:2/114: chown d2 148795691 1 2024-03-23T11:47:24.645 INFO:tasks.workunit.client.0.smithi063.stdout:4/89: creat d5/f13 x:0 0 0 2024-03-23T11:47:24.648 INFO:tasks.workunit.client.1.smithi111.stdout:8/44: mknod d5/cf 0 2024-03-23T11:47:24.649 INFO:tasks.workunit.client.0.smithi063.stdout:6/89: readlink d1/lb 0 2024-03-23T11:47:24.649 INFO:tasks.workunit.client.0.smithi063.stdout:1/105: mknod d3/d1c/c22 0 2024-03-23T11:47:24.650 INFO:tasks.workunit.client.0.smithi063.stdout:2/68: write d2/f4 [2639177,83511] 0 2024-03-23T11:47:24.651 INFO:tasks.workunit.client.1.smithi111.stdout:8/45: mknod d5/c10 0 2024-03-23T11:47:24.656 INFO:tasks.workunit.client.1.smithi111.stdout:6/166: write d2/f31 [640967,51571] 0 2024-03-23T11:47:24.656 INFO:tasks.workunit.client.0.smithi063.stdout:7/111: getdents d5/d6/d19 0 2024-03-23T11:47:24.656 INFO:tasks.workunit.client.1.smithi111.stdout:8/46: creat d5/f11 x:0 0 0 2024-03-23T11:47:24.660 INFO:tasks.workunit.client.1.smithi111.stdout:3/55: dread d0/f9 [0,4194304] 0 2024-03-23T11:47:24.660 INFO:tasks.workunit.client.0.smithi063.stdout:8/271: dwrite d6/dd/df/f14 [0,4194304] 0 2024-03-23T11:47:24.661 INFO:tasks.workunit.client.0.smithi063.stdout:0/56: symlink d2/d14/l16 0 2024-03-23T11:47:24.666 INFO:tasks.workunit.client.1.smithi111.stdout:6/167: chown d2/d1a/d1c/l34 592890816 1 2024-03-23T11:47:24.669 INFO:tasks.workunit.client.0.smithi063.stdout:8/272: fsync d6/d1c/d27/f34 0 2024-03-23T11:47:24.670 INFO:tasks.workunit.client.0.smithi063.stdout:2/69: fsync d2/f17 0 2024-03-23T11:47:24.671 INFO:tasks.workunit.client.0.smithi063.stdout:1/106: write d3/f9 [625238,17444] 0 2024-03-23T11:47:24.673 INFO:tasks.workunit.client.0.smithi063.stdout:3/200: dwrite d13/d15/f1b [0,4194304] 0 2024-03-23T11:47:24.677 INFO:tasks.workunit.client.0.smithi063.stdout:5/76: getdents d2/d17 0 2024-03-23T11:47:24.679 INFO:tasks.workunit.client.1.smithi111.stdout:3/56: truncate d0/fd 318442 0 2024-03-23T11:47:24.681 INFO:tasks.workunit.client.1.smithi111.stdout:7/68: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:24.686 INFO:tasks.workunit.client.0.smithi063.stdout:3/201: write f12 [1438547,32905] 0 2024-03-23T11:47:24.686 INFO:tasks.workunit.client.1.smithi111.stdout:9/119: dwrite d0/d4/fe [0,4194304] 0 2024-03-23T11:47:24.686 INFO:tasks.workunit.client.1.smithi111.stdout:5/110: sync 2024-03-23T11:47:24.686 INFO:tasks.workunit.client.1.smithi111.stdout:2/115: dwrite d2/de/d12/f29 [0,4194304] 0 2024-03-23T11:47:24.691 INFO:tasks.workunit.client.0.smithi063.stdout:7/112: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:24.694 INFO:tasks.workunit.client.0.smithi063.stdout:3/202: write d13/d22/f2c [734190,41461] 0 2024-03-23T11:47:24.697 INFO:tasks.workunit.client.0.smithi063.stdout:1/107: dread d3/f9 [0,4194304] 0 2024-03-23T11:47:24.698 INFO:tasks.workunit.client.1.smithi111.stdout:7/69: symlink lf 0 2024-03-23T11:47:24.699 INFO:tasks.workunit.client.0.smithi063.stdout:6/90: dwrite d1/dc/fd [0,4194304] 0 2024-03-23T11:47:24.701 INFO:tasks.workunit.client.1.smithi111.stdout:9/120: truncate d0/f21 279553 0 2024-03-23T11:47:24.701 INFO:tasks.workunit.client.1.smithi111.stdout:8/47: dread d5/f6 [0,4194304] 0 2024-03-23T11:47:24.701 INFO:tasks.workunit.client.1.smithi111.stdout:7/70: chown cc 1245 1 2024-03-23T11:47:24.703 INFO:tasks.workunit.client.0.smithi063.stdout:3/203: truncate d13/d15/f1f 1367424 0 2024-03-23T11:47:24.705 INFO:tasks.workunit.client.0.smithi063.stdout:1/108: chown d3/f7 201017974 1 2024-03-23T11:47:24.705 INFO:tasks.workunit.client.0.smithi063.stdout:3/204: stat d13/d15/f2a 0 2024-03-23T11:47:24.708 INFO:tasks.workunit.client.1.smithi111.stdout:5/111: creat d1/d3/d23/f28 x:0 0 0 2024-03-23T11:47:24.708 INFO:tasks.workunit.client.0.smithi063.stdout:1/109: stat d3/d1c/c1f 0 2024-03-23T11:47:24.708 INFO:tasks.workunit.client.1.smithi111.stdout:8/48: dread - d5/fe zero size 2024-03-23T11:47:24.710 INFO:tasks.workunit.client.1.smithi111.stdout:2/116: rmdir d2/de/d12 39 2024-03-23T11:47:24.711 INFO:tasks.workunit.client.0.smithi063.stdout:4/90: truncate d5/da/fd 1846727 0 2024-03-23T11:47:24.711 INFO:tasks.workunit.client.0.smithi063.stdout:2/70: dwrite f1 [0,4194304] 0 2024-03-23T11:47:24.712 INFO:tasks.workunit.client.0.smithi063.stdout:0/57: truncate d2/d4/f6 2622171 0 2024-03-23T11:47:24.713 INFO:tasks.workunit.client.0.smithi063.stdout:1/110: write d3/df/f1a [3926734,67726] 0 2024-03-23T11:47:24.715 INFO:tasks.workunit.client.1.smithi111.stdout:5/112: dread - d1/d20/d25/f26 zero size 2024-03-23T11:47:24.717 INFO:tasks.workunit.client.1.smithi111.stdout:9/121: creat d0/d4/d10/f2c x:0 0 0 2024-03-23T11:47:24.717 INFO:tasks.workunit.client.0.smithi063.stdout:2/71: truncate d2/f4 5139212 0 2024-03-23T11:47:24.719 INFO:tasks.workunit.client.1.smithi111.stdout:8/49: mknod d5/c12 0 2024-03-23T11:47:24.720 INFO:tasks.workunit.client.0.smithi063.stdout:2/72: chown d2/f17 1822 1 2024-03-23T11:47:24.722 INFO:tasks.workunit.client.0.smithi063.stdout:4/91: unlink d5/f13 0 2024-03-23T11:47:24.724 INFO:tasks.workunit.client.1.smithi111.stdout:2/117: rmdir d2/de 39 2024-03-23T11:47:24.724 INFO:tasks.workunit.client.0.smithi063.stdout:3/205: fsync f8 0 2024-03-23T11:47:24.725 INFO:tasks.workunit.client.1.smithi111.stdout:6/168: dwrite d2/d1a/f1d [0,4194304] 0 2024-03-23T11:47:24.725 INFO:tasks.workunit.client.0.smithi063.stdout:0/58: read f1 [332433,118591] 0 2024-03-23T11:47:24.727 INFO:tasks.workunit.client.0.smithi063.stdout:0/59: readlink d2/d14/l16 0 2024-03-23T11:47:24.730 INFO:tasks.workunit.client.0.smithi063.stdout:7/113: dwrite d5/d9/da/f17 [0,4194304] 0 2024-03-23T11:47:24.730 INFO:tasks.workunit.client.0.smithi063.stdout:6/91: dread d1/f8 [0,4194304] 0 2024-03-23T11:47:24.731 INFO:tasks.workunit.client.0.smithi063.stdout:1/111: symlink d3/d1c/l23 0 2024-03-23T11:47:24.733 INFO:tasks.workunit.client.1.smithi111.stdout:9/122: mkdir d0/d4/d1c/d2d 0 2024-03-23T11:47:24.733 INFO:tasks.workunit.client.0.smithi063.stdout:2/73: rename d2/f4 to d2/f18 0 2024-03-23T11:47:24.735 INFO:tasks.workunit.client.1.smithi111.stdout:2/118: write d2/d11/d17/d1f/f20 [2015847,34840] 0 2024-03-23T11:47:24.735 INFO:tasks.workunit.client.0.smithi063.stdout:4/92: rmdir d5/da 39 2024-03-23T11:47:24.737 INFO:tasks.workunit.client.0.smithi063.stdout:3/206: mknod d13/c36 0 2024-03-23T11:47:24.743 INFO:tasks.workunit.client.0.smithi063.stdout:6/92: mknod d1/d13/c19 0 2024-03-23T11:47:24.744 INFO:tasks.workunit.client.1.smithi111.stdout:5/113: rename d1/d1e to d1/d3/d6/d29 0 2024-03-23T11:47:24.744 INFO:tasks.workunit.client.1.smithi111.stdout:9/123: dread - d0/d4/d10/f2c zero size 2024-03-23T11:47:24.744 INFO:tasks.workunit.client.0.smithi063.stdout:0/60: read f1 [120463,90727] 0 2024-03-23T11:47:24.745 INFO:tasks.workunit.client.1.smithi111.stdout:7/71: dwrite fd [0,4194304] 0 2024-03-23T11:47:24.745 INFO:tasks.workunit.client.0.smithi063.stdout:3/207: mknod d13/d15/c37 0 2024-03-23T11:47:24.748 INFO:tasks.workunit.client.0.smithi063.stdout:3/208: chown f12 91 1 2024-03-23T11:47:24.749 INFO:tasks.workunit.client.1.smithi111.stdout:2/119: unlink d2/d11/f24 0 2024-03-23T11:47:24.749 INFO:tasks.workunit.client.0.smithi063.stdout:7/114: mkdir d5/d9/d20 0 2024-03-23T11:47:24.752 INFO:tasks.workunit.client.1.smithi111.stdout:9/124: read d0/d4/f14 [29857,69936] 0 2024-03-23T11:47:24.753 INFO:tasks.workunit.client.0.smithi063.stdout:6/93: rename d1/l2 to d1/l1a 0 2024-03-23T11:47:24.754 INFO:tasks.workunit.client.1.smithi111.stdout:3/57: write d0/f3 [886094,12914] 0 2024-03-23T11:47:24.755 INFO:tasks.workunit.client.0.smithi063.stdout:4/93: mknod d5/da/c14 0 2024-03-23T11:47:24.755 INFO:tasks.workunit.client.0.smithi063.stdout:0/61: rmdir d2/d14 39 2024-03-23T11:47:24.755 INFO:tasks.workunit.client.1.smithi111.stdout:9/125: stat d0/d4/f15 0 2024-03-23T11:47:24.756 INFO:tasks.workunit.client.1.smithi111.stdout:6/169: rename d2/l13 to d2/d1a/d1c/l3a 0 2024-03-23T11:47:24.756 INFO:tasks.workunit.client.0.smithi063.stdout:4/94: read d5/ff [2207058,114093] 0 2024-03-23T11:47:24.757 INFO:tasks.workunit.client.1.smithi111.stdout:4/68: sync 2024-03-23T11:47:24.757 INFO:tasks.workunit.client.1.smithi111.stdout:1/75: sync 2024-03-23T11:47:24.758 INFO:tasks.workunit.client.1.smithi111.stdout:0/106: sync 2024-03-23T11:47:24.758 INFO:tasks.workunit.client.1.smithi111.stdout:8/50: dwrite f1 [4194304,4194304] 0 2024-03-23T11:47:24.758 INFO:tasks.workunit.client.0.smithi063.stdout:2/74: fdatasync f1 0 2024-03-23T11:47:24.760 INFO:tasks.workunit.client.1.smithi111.stdout:3/58: chown d0/f9 431846 1 2024-03-23T11:47:24.761 INFO:tasks.workunit.client.1.smithi111.stdout:7/72: creat f10 x:0 0 0 2024-03-23T11:47:24.761 INFO:tasks.workunit.client.1.smithi111.stdout:9/126: read d0/d4/fe [1136648,35490] 0 2024-03-23T11:47:24.766 INFO:tasks.workunit.client.1.smithi111.stdout:1/76: creat d0/f1d x:0 0 0 2024-03-23T11:47:24.767 INFO:tasks.workunit.client.1.smithi111.stdout:7/73: creat f11 x:0 0 0 2024-03-23T11:47:24.767 INFO:tasks.workunit.client.0.smithi063.stdout:1/112: dwrite d3/f9 [0,4194304] 0 2024-03-23T11:47:24.770 INFO:tasks.workunit.client.1.smithi111.stdout:7/74: write fb [1702076,96842] 0 2024-03-23T11:47:24.771 INFO:tasks.workunit.client.0.smithi063.stdout:2/75: truncate d2/de/f15 561280 0 2024-03-23T11:47:24.774 INFO:tasks.workunit.client.1.smithi111.stdout:4/69: rename d0/c6 to d0/d7/d9/c19 0 2024-03-23T11:47:24.774 INFO:tasks.workunit.client.0.smithi063.stdout:2/76: truncate d2/f18 5864420 0 2024-03-23T11:47:24.776 INFO:tasks.workunit.client.1.smithi111.stdout:4/70: truncate d0/f1 1762912 0 2024-03-23T11:47:24.781 INFO:tasks.workunit.client.1.smithi111.stdout:0/107: rename d5/db/d14 to d5/db/d1f 0 2024-03-23T11:47:24.783 INFO:tasks.workunit.client.1.smithi111.stdout:0/108: stat d5/ca 0 2024-03-23T11:47:24.784 INFO:tasks.workunit.client.0.smithi063.stdout:4/95: dwrite f2 [0,4194304] 0 2024-03-23T11:47:24.784 INFO:tasks.workunit.client.0.smithi063.stdout:0/62: fsync d2/d4/f6 0 2024-03-23T11:47:24.784 INFO:tasks.workunit.client.0.smithi063.stdout:6/94: dread d1/dc/fd [0,4194304] 0 2024-03-23T11:47:24.786 INFO:tasks.workunit.client.1.smithi111.stdout:1/77: mkdir d0/d5/dd/d1e 0 2024-03-23T11:47:24.788 INFO:tasks.workunit.client.1.smithi111.stdout:1/78: stat d0/d6/f19 0 2024-03-23T11:47:24.789 INFO:tasks.workunit.client.1.smithi111.stdout:8/51: truncate f3 2564277 0 2024-03-23T11:47:24.791 INFO:tasks.workunit.client.1.smithi111.stdout:4/71: symlink d0/l1a 0 2024-03-23T11:47:24.792 INFO:tasks.workunit.client.1.smithi111.stdout:6/170: creat d2/f3b x:0 0 0 2024-03-23T11:47:24.793 INFO:tasks.workunit.client.0.smithi063.stdout:1/113: mknod d3/c24 0 2024-03-23T11:47:24.794 INFO:tasks.workunit.client.1.smithi111.stdout:0/109: unlink d5/f9 0 2024-03-23T11:47:24.796 INFO:tasks.workunit.client.1.smithi111.stdout:5/114: getdents d1/d3/d6/d29 0 2024-03-23T11:47:24.796 INFO:tasks.workunit.client.1.smithi111.stdout:7/75: dwrite fb [4194304,4194304] 0 2024-03-23T11:47:24.796 INFO:tasks.workunit.client.0.smithi063.stdout:2/77: creat d2/f19 x:0 0 0 2024-03-23T11:47:24.797 INFO:tasks.workunit.client.0.smithi063.stdout:1/114: write d3/f11 [3400,56831] 0 2024-03-23T11:47:24.809 INFO:tasks.workunit.client.0.smithi063.stdout:0/63: dread d2/d4/dd/ff [0,4194304] 0 2024-03-23T11:47:24.810 INFO:tasks.workunit.client.0.smithi063.stdout:3/209: dwrite f8 [0,4194304] 0 2024-03-23T11:47:24.814 INFO:tasks.workunit.client.1.smithi111.stdout:5/115: unlink d1/d3/f18 0 2024-03-23T11:47:24.816 INFO:tasks.workunit.client.1.smithi111.stdout:1/79: dread d0/d5/f16 [0,4194304] 0 2024-03-23T11:47:24.818 INFO:tasks.workunit.client.0.smithi063.stdout:7/115: rename d5/lc to d5/d9/l21 0 2024-03-23T11:47:24.818 INFO:tasks.workunit.client.0.smithi063.stdout:2/78: symlink d2/de/l1a 0 2024-03-23T11:47:24.820 INFO:tasks.workunit.client.0.smithi063.stdout:1/115: rename d3/ca to d3/c25 0 2024-03-23T11:47:24.825 INFO:tasks.workunit.client.1.smithi111.stdout:4/72: creat d0/d7/f1b x:0 0 0 2024-03-23T11:47:24.825 INFO:tasks.workunit.client.0.smithi063.stdout:2/79: creat d2/de/f1b x:0 0 0 2024-03-23T11:47:24.827 INFO:tasks.workunit.client.0.smithi063.stdout:2/80: stat d2 0 2024-03-23T11:47:24.827 INFO:tasks.workunit.client.1.smithi111.stdout:2/120: truncate d2/f4 3276399 0 2024-03-23T11:47:24.828 INFO:tasks.workunit.client.0.smithi063.stdout:1/116: creat d3/df/f26 x:0 0 0 2024-03-23T11:47:24.828 INFO:tasks.workunit.client.1.smithi111.stdout:6/171: dwrite d2/f27 [0,4194304] 0 2024-03-23T11:47:24.832 INFO:tasks.workunit.client.0.smithi063.stdout:0/64: rmdir d2/d4/d13 0 2024-03-23T11:47:24.832 INFO:tasks.workunit.client.0.smithi063.stdout:3/210: link d13/d15/f1c d13/d22/d2d/f38 0 2024-03-23T11:47:24.833 INFO:tasks.workunit.client.0.smithi063.stdout:2/81: mkdir d2/d1c 0 2024-03-23T11:47:24.834 INFO:tasks.workunit.client.0.smithi063.stdout:7/116: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:24.837 INFO:tasks.workunit.client.1.smithi111.stdout:9/127: truncate d0/d4/fe 2525348 0 2024-03-23T11:47:24.837 INFO:tasks.workunit.client.0.smithi063.stdout:2/82: dread - d2/de/f1b zero size 2024-03-23T11:47:24.837 INFO:tasks.workunit.client.0.smithi063.stdout:7/117: readlink d5/d9/da/l11 0 2024-03-23T11:47:24.838 INFO:tasks.workunit.client.1.smithi111.stdout:1/80: symlink d0/d5/dd/d1e/l1f 0 2024-03-23T11:47:24.838 INFO:tasks.workunit.client.0.smithi063.stdout:1/117: creat d3/d1c/f27 x:0 0 0 2024-03-23T11:47:24.839 INFO:tasks.workunit.client.1.smithi111.stdout:6/172: chown d2/d1a/d1c/l34 110 1 2024-03-23T11:47:24.840 INFO:tasks.workunit.client.1.smithi111.stdout:7/76: dwrite f7 [0,4194304] 0 2024-03-23T11:47:24.842 INFO:tasks.workunit.client.0.smithi063.stdout:3/211: mknod d13/c39 0 2024-03-23T11:47:24.842 INFO:tasks.workunit.client.0.smithi063.stdout:6/95: dwrite d1/f7 [0,4194304] 0 2024-03-23T11:47:24.843 INFO:tasks.workunit.client.1.smithi111.stdout:0/110: dwrite d5/f13 [0,4194304] 0 2024-03-23T11:47:24.843 INFO:tasks.workunit.client.1.smithi111.stdout:1/81: chown d0/f1a 428 1 2024-03-23T11:47:24.843 INFO:tasks.workunit.client.1.smithi111.stdout:7/77: chown c6 0 1 2024-03-23T11:47:24.847 INFO:tasks.workunit.client.1.smithi111.stdout:5/116: truncate d1/d3/d6/d7/f9 567821 0 2024-03-23T11:47:24.847 INFO:tasks.workunit.client.0.smithi063.stdout:2/83: symlink d2/de/l1d 0 2024-03-23T11:47:24.847 INFO:tasks.workunit.client.0.smithi063.stdout:7/118: mkdir d5/d22 0 2024-03-23T11:47:24.850 INFO:tasks.workunit.client.0.smithi063.stdout:6/96: dread d1/d13/f16 [0,4194304] 0 2024-03-23T11:47:24.850 INFO:tasks.workunit.client.1.smithi111.stdout:6/173: symlink d2/d1f/l3c 0 2024-03-23T11:47:24.850 INFO:tasks.workunit.client.1.smithi111.stdout:7/78: chown f9 55309900 1 2024-03-23T11:47:24.851 INFO:tasks.workunit.client.0.smithi063.stdout:1/118: symlink d3/l28 0 2024-03-23T11:47:24.852 INFO:tasks.workunit.client.0.smithi063.stdout:2/84: creat d2/f1e x:0 0 0 2024-03-23T11:47:24.853 INFO:tasks.workunit.client.1.smithi111.stdout:1/82: write d0/d6/f11 [397398,51577] 0 2024-03-23T11:47:24.855 INFO:tasks.workunit.client.0.smithi063.stdout:6/97: creat d1/dc/f1b x:0 0 0 2024-03-23T11:47:24.856 INFO:tasks.workunit.client.1.smithi111.stdout:5/117: creat d1/d20/f2a x:0 0 0 2024-03-23T11:47:24.856 INFO:tasks.workunit.client.1.smithi111.stdout:9/128: creat d0/d4/d1c/d2d/f2e x:0 0 0 2024-03-23T11:47:24.857 INFO:tasks.workunit.client.1.smithi111.stdout:7/79: write fb [7844285,114586] 0 2024-03-23T11:47:24.858 INFO:tasks.workunit.client.0.smithi063.stdout:7/119: mknod d5/d6/c23 0 2024-03-23T11:47:24.860 INFO:tasks.workunit.client.1.smithi111.stdout:9/129: chown d0/c2b 1712780448 1 2024-03-23T11:47:24.861 INFO:tasks.workunit.client.1.smithi111.stdout:6/174: stat d2/d2a/c39 0 2024-03-23T11:47:24.863 INFO:tasks.workunit.client.1.smithi111.stdout:1/83: creat d0/f20 x:0 0 0 2024-03-23T11:47:24.865 INFO:tasks.workunit.client.1.smithi111.stdout:7/80: creat f12 x:0 0 0 2024-03-23T11:47:24.866 INFO:tasks.workunit.client.1.smithi111.stdout:9/130: symlink d0/d4/d1c/l2f 0 2024-03-23T11:47:24.866 INFO:tasks.workunit.client.1.smithi111.stdout:4/73: write d0/f4 [4774308,128249] 0 2024-03-23T11:47:24.867 INFO:tasks.workunit.client.0.smithi063.stdout:1/119: chown d3/df/c21 910 1 2024-03-23T11:47:24.870 INFO:tasks.workunit.client.1.smithi111.stdout:7/81: chown f2 1849 1 2024-03-23T11:47:24.871 INFO:tasks.workunit.client.1.smithi111.stdout:5/118: creat d1/d3/d6/f2b x:0 0 0 2024-03-23T11:47:24.871 INFO:tasks.workunit.client.1.smithi111.stdout:1/84: creat d0/d5/dd/d1e/f21 x:0 0 0 2024-03-23T11:47:24.872 INFO:tasks.workunit.client.0.smithi063.stdout:6/98: stat d1/l1a 0 2024-03-23T11:47:24.877 INFO:tasks.workunit.client.1.smithi111.stdout:8/52: truncate f3 1227735 0 2024-03-23T11:47:24.877 INFO:tasks.workunit.client.1.smithi111.stdout:7/82: write f2 [7668423,109046] 0 2024-03-23T11:47:24.878 INFO:tasks.workunit.client.1.smithi111.stdout:1/85: stat d0/d5/l1b 0 2024-03-23T11:47:24.883 INFO:tasks.workunit.client.1.smithi111.stdout:4/74: rename d0/f4 to d0/dc/f1c 0 2024-03-23T11:47:24.883 INFO:tasks.workunit.client.1.smithi111.stdout:9/131: truncate d0/d4/f14 850960 0 2024-03-23T11:47:24.884 INFO:tasks.workunit.client.1.smithi111.stdout:0/111: dwrite d5/f13 [4194304,4194304] 0 2024-03-23T11:47:24.885 INFO:tasks.workunit.client.1.smithi111.stdout:5/119: mkdir d1/d3/d23/d2c 0 2024-03-23T11:47:24.885 INFO:tasks.workunit.client.1.smithi111.stdout:1/86: dread - d0/d6/f19 zero size 2024-03-23T11:47:24.886 INFO:tasks.workunit.client.0.smithi063.stdout:2/85: creat d2/d1c/f1f x:0 0 0 2024-03-23T11:47:24.886 INFO:tasks.workunit.client.1.smithi111.stdout:8/53: rmdir d5 39 2024-03-23T11:47:24.888 INFO:tasks.workunit.client.1.smithi111.stdout:3/59: write d0/fd [1198694,112300] 0 2024-03-23T11:47:24.888 INFO:tasks.workunit.client.0.smithi063.stdout:7/120: creat d5/d22/f24 x:0 0 0 2024-03-23T11:47:24.892 INFO:tasks.workunit.client.1.smithi111.stdout:0/112: dread - d5/db/f15 zero size 2024-03-23T11:47:24.892 INFO:tasks.workunit.client.0.smithi063.stdout:0/65: dwrite d2/d4/f6 [0,4194304] 0 2024-03-23T11:47:24.895 INFO:tasks.workunit.client.1.smithi111.stdout:1/87: read - d0/f1d zero size 2024-03-23T11:47:24.898 INFO:tasks.workunit.client.1.smithi111.stdout:5/120: creat d1/d3/d23/d2c/f2d x:0 0 0 2024-03-23T11:47:24.899 INFO:tasks.workunit.client.1.smithi111.stdout:3/60: chown d0/cc 565534 1 2024-03-23T11:47:24.900 INFO:tasks.workunit.client.0.smithi063.stdout:3/212: truncate d13/d22/f2c 517667 0 2024-03-23T11:47:24.900 INFO:tasks.workunit.client.1.smithi111.stdout:0/113: stat d5/db/d1f/d19/f1c 0 2024-03-23T11:47:24.901 INFO:tasks.workunit.client.1.smithi111.stdout:6/175: dwrite d2/d2a/f2f [0,4194304] 0 2024-03-23T11:47:24.902 INFO:tasks.workunit.client.0.smithi063.stdout:1/120: dwrite d3/f10 [0,4194304] 0 2024-03-23T11:47:24.903 INFO:tasks.workunit.client.1.smithi111.stdout:6/176: stat d2/ff 0 2024-03-23T11:47:24.904 INFO:tasks.workunit.client.0.smithi063.stdout:7/121: write d5/d9/da/f17 [3452190,91367] 0 2024-03-23T11:47:24.904 INFO:tasks.workunit.client.1.smithi111.stdout:9/132: mknod d0/d4/d1c/d2d/c30 0 2024-03-23T11:47:24.909 INFO:tasks.workunit.client.1.smithi111.stdout:1/88: rmdir d0 39 2024-03-23T11:47:24.910 INFO:tasks.workunit.client.0.smithi063.stdout:6/99: dwrite d1/f12 [0,4194304] 0 2024-03-23T11:47:24.912 INFO:tasks.workunit.client.0.smithi063.stdout:6/100: readlink d1/l9 0 2024-03-23T11:47:24.913 INFO:tasks.workunit.client.1.smithi111.stdout:0/114: truncate d5/db/f15 301747 0 2024-03-23T11:47:24.914 INFO:tasks.workunit.client.1.smithi111.stdout:6/177: rmdir d2/d1a 39 2024-03-23T11:47:24.914 INFO:tasks.workunit.client.0.smithi063.stdout:6/101: write d1/f12 [1360868,112786] 0 2024-03-23T11:47:24.915 INFO:tasks.workunit.client.0.smithi063.stdout:4/96: dread d5/da/fd [0,4194304] 0 2024-03-23T11:47:24.915 INFO:tasks.workunit.client.0.smithi063.stdout:7/122: read d5/d9/da/f17 [1502911,53801] 0 2024-03-23T11:47:24.917 INFO:tasks.workunit.client.1.smithi111.stdout:9/133: mkdir d0/d31 0 2024-03-23T11:47:24.919 INFO:tasks.workunit.client.0.smithi063.stdout:0/66: rename d2/d14/l16 to d2/d4/l17 0 2024-03-23T11:47:24.921 INFO:tasks.workunit.client.1.smithi111.stdout:3/61: symlink d0/le 0 2024-03-23T11:47:24.922 INFO:tasks.workunit.client.0.smithi063.stdout:6/102: creat d1/d13/f1c x:0 0 0 2024-03-23T11:47:24.927 INFO:tasks.workunit.client.0.smithi063.stdout:7/123: creat d5/d6/f25 x:0 0 0 2024-03-23T11:47:24.927 INFO:tasks.workunit.client.0.smithi063.stdout:6/103: dread - d1/d13/f1c zero size 2024-03-23T11:47:24.927 INFO:tasks.workunit.client.0.smithi063.stdout:0/67: write d2/d4/f9 [77403,39694] 0 2024-03-23T11:47:24.928 INFO:tasks.workunit.client.1.smithi111.stdout:1/89: readlink d0/l3 0 2024-03-23T11:47:24.929 INFO:tasks.workunit.client.1.smithi111.stdout:6/178: mknod d2/d35/c3d 0 2024-03-23T11:47:24.932 INFO:tasks.workunit.client.1.smithi111.stdout:7/83: dwrite f11 [0,4194304] 0 2024-03-23T11:47:24.932 INFO:tasks.workunit.client.1.smithi111.stdout:6/179: stat d2 0 2024-03-23T11:47:24.933 INFO:tasks.workunit.client.0.smithi063.stdout:1/121: rename d3/df/c21 to d3/d1c/c29 0 2024-03-23T11:47:24.933 INFO:tasks.workunit.client.0.smithi063.stdout:0/68: chown d2/d4/dd/c10 148490516 1 2024-03-23T11:47:24.933 INFO:tasks.workunit.client.0.smithi063.stdout:3/213: dwrite d13/d15/f1b [4194304,4194304] 0 2024-03-23T11:47:24.936 INFO:tasks.workunit.client.1.smithi111.stdout:3/62: truncate d0/f9 285468 0 2024-03-23T11:47:24.936 INFO:tasks.workunit.client.0.smithi063.stdout:3/214: fsync ff 0 2024-03-23T11:47:24.937 INFO:tasks.workunit.client.1.smithi111.stdout:6/180: read d2/ff [327006,38181] 0 2024-03-23T11:47:24.937 INFO:tasks.workunit.client.0.smithi063.stdout:7/124: mkdir d5/d26 0 2024-03-23T11:47:24.939 INFO:tasks.workunit.client.1.smithi111.stdout:4/75: dwrite d0/dc/fe [0,4194304] 0 2024-03-23T11:47:24.939 INFO:tasks.workunit.client.1.smithi111.stdout:8/54: dwrite d5/f11 [0,4194304] 0 2024-03-23T11:47:24.941 INFO:tasks.workunit.client.1.smithi111.stdout:6/181: chown d2/d17/f20 93 1 2024-03-23T11:47:24.941 INFO:tasks.workunit.client.1.smithi111.stdout:4/76: readlink d0/l11 0 2024-03-23T11:47:24.946 INFO:tasks.workunit.client.0.smithi063.stdout:1/122: creat d3/f2a x:0 0 0 2024-03-23T11:47:24.949 INFO:tasks.workunit.client.0.smithi063.stdout:6/104: dread d1/f12 [0,4194304] 0 2024-03-23T11:47:24.949 INFO:tasks.workunit.client.0.smithi063.stdout:6/105: chown d1 5404 1 2024-03-23T11:47:24.950 INFO:tasks.workunit.client.1.smithi111.stdout:7/84: creat f13 x:0 0 0 2024-03-23T11:47:24.950 INFO:tasks.workunit.client.1.smithi111.stdout:9/134: truncate d0/d4/d1c/f26 2114943 0 2024-03-23T11:47:24.951 INFO:tasks.workunit.client.1.smithi111.stdout:8/55: mknod d5/c13 0 2024-03-23T11:47:24.952 INFO:tasks.workunit.client.1.smithi111.stdout:4/77: unlink d0/dc/cd 0 2024-03-23T11:47:24.952 INFO:tasks.workunit.client.1.smithi111.stdout:3/63: creat d0/d7/ff x:0 0 0 2024-03-23T11:47:24.955 INFO:tasks.workunit.client.1.smithi111.stdout:7/85: unlink f13 0 2024-03-23T11:47:24.956 INFO:tasks.workunit.client.0.smithi063.stdout:3/215: creat d13/d22/d2d/f3a x:0 0 0 2024-03-23T11:47:24.956 INFO:tasks.workunit.client.1.smithi111.stdout:8/56: rename d5 to d5/d14 22 2024-03-23T11:47:24.961 INFO:tasks.workunit.client.0.smithi063.stdout:7/125: creat d5/d6/f27 x:0 0 0 2024-03-23T11:47:24.962 INFO:tasks.workunit.client.1.smithi111.stdout:4/78: rename d0/d7 to d0/d7/da/d1d 22 2024-03-23T11:47:24.964 INFO:tasks.workunit.client.1.smithi111.stdout:4/79: chown d0/dc/f14 60311 1 2024-03-23T11:47:24.964 INFO:tasks.workunit.client.0.smithi063.stdout:7/126: write d5/d6/f25 [223256,102836] 0 2024-03-23T11:47:24.966 INFO:tasks.workunit.client.1.smithi111.stdout:4/80: chown d0/d17 904 1 2024-03-23T11:47:24.966 INFO:tasks.workunit.client.1.smithi111.stdout:3/64: creat d0/d7/f10 x:0 0 0 2024-03-23T11:47:24.967 INFO:tasks.workunit.client.1.smithi111.stdout:7/86: mkdir d14 0 2024-03-23T11:47:24.968 INFO:tasks.workunit.client.0.smithi063.stdout:1/123: symlink d3/df/l2b 0 2024-03-23T11:47:24.969 INFO:tasks.workunit.client.0.smithi063.stdout:0/69: read d2/d4/f9 [77760,126134] 0 2024-03-23T11:47:24.970 INFO:tasks.workunit.client.1.smithi111.stdout:7/87: fsync f12 0 2024-03-23T11:47:24.970 INFO:tasks.workunit.client.1.smithi111.stdout:3/65: fdatasync d0/f3 0 2024-03-23T11:47:24.973 INFO:tasks.workunit.client.1.smithi111.stdout:6/182: link d2/d17/l38 d2/d2a/l3e 0 2024-03-23T11:47:24.974 INFO:tasks.workunit.client.0.smithi063.stdout:0/70: chown d2/d4/f9 229873545 1 2024-03-23T11:47:24.974 INFO:tasks.workunit.client.0.smithi063.stdout:6/106: creat d1/dc/f1d x:0 0 0 2024-03-23T11:47:24.976 INFO:tasks.workunit.client.1.smithi111.stdout:9/135: dwrite d0/f21 [0,4194304] 0 2024-03-23T11:47:24.977 INFO:tasks.workunit.client.1.smithi111.stdout:9/136: fdatasync d0/f13 0 2024-03-23T11:47:24.978 INFO:tasks.workunit.client.1.smithi111.stdout:4/81: mknod d0/d7/c1e 0 2024-03-23T11:47:24.980 INFO:tasks.workunit.client.0.smithi063.stdout:3/216: dread f5 [0,4194304] 0 2024-03-23T11:47:24.981 INFO:tasks.workunit.client.0.smithi063.stdout:6/107: creat d1/d13/f1e x:0 0 0 2024-03-23T11:47:24.985 INFO:tasks.workunit.client.0.smithi063.stdout:7/127: creat d5/d9/d20/f28 x:0 0 0 2024-03-23T11:47:24.985 INFO:tasks.workunit.client.0.smithi063.stdout:4/97: dwrite d5/ff [4194304,4194304] 0 2024-03-23T11:47:24.986 INFO:tasks.workunit.client.0.smithi063.stdout:3/217: read - d13/d22/f2f zero size 2024-03-23T11:47:24.989 INFO:tasks.workunit.client.0.smithi063.stdout:3/218: dread - d13/d15/f33 zero size 2024-03-23T11:47:24.989 INFO:tasks.workunit.client.1.smithi111.stdout:9/137: symlink d0/d1/d28/l32 0 2024-03-23T11:47:24.990 INFO:tasks.workunit.client.1.smithi111.stdout:3/66: link d0/l6 d0/l11 0 2024-03-23T11:47:24.991 INFO:tasks.workunit.client.1.smithi111.stdout:9/138: read - d0/d4/d10/f2c zero size 2024-03-23T11:47:24.992 INFO:tasks.workunit.client.0.smithi063.stdout:0/71: fsync d2/d4/f9 0 2024-03-23T11:47:24.992 INFO:tasks.workunit.client.0.smithi063.stdout:6/108: creat d1/dc/f1f x:0 0 0 2024-03-23T11:47:24.992 INFO:tasks.workunit.client.1.smithi111.stdout:5/121: write d1/d3/d6/d7/f9 [1352212,42773] 0 2024-03-23T11:47:24.993 INFO:tasks.workunit.client.1.smithi111.stdout:3/67: chown d0/d7/f10 5 1 2024-03-23T11:47:24.996 INFO:tasks.workunit.client.0.smithi063.stdout:4/98: mknod d5/d10/c15 0 2024-03-23T11:47:24.997 INFO:tasks.workunit.client.1.smithi111.stdout:6/183: dread d2/d1a/f1d [0,4194304] 0 2024-03-23T11:47:24.998 INFO:tasks.workunit.client.1.smithi111.stdout:9/139: write d0/d1/f5 [2826070,108276] 0 2024-03-23T11:47:24.998 INFO:tasks.workunit.client.1.smithi111.stdout:3/68: chown d0/d7/f10 0 1 2024-03-23T11:47:24.998 INFO:tasks.workunit.client.0.smithi063.stdout:7/128: link d5/d9/da/cb d5/d6/d19/c29 0 2024-03-23T11:47:24.999 INFO:tasks.workunit.client.0.smithi063.stdout:4/99: write f2 [5360278,29172] 0 2024-03-23T11:47:24.999 INFO:tasks.workunit.client.0.smithi063.stdout:3/219: creat d13/d22/f3b x:0 0 0 2024-03-23T11:47:25.001 INFO:tasks.workunit.client.1.smithi111.stdout:5/122: mknod d1/d3/d23/d2c/c2e 0 2024-03-23T11:47:25.004 INFO:tasks.workunit.client.0.smithi063.stdout:0/72: mknod d2/d4/dd/c18 0 2024-03-23T11:47:25.005 INFO:tasks.workunit.client.0.smithi063.stdout:6/109: dread d1/d13/f16 [0,4194304] 0 2024-03-23T11:47:25.006 INFO:tasks.workunit.client.1.smithi111.stdout:2/121: sync 2024-03-23T11:47:25.007 INFO:tasks.workunit.client.0.smithi063.stdout:1/124: dwrite d3/df/f19 [0,4194304] 0 2024-03-23T11:47:25.010 INFO:tasks.workunit.client.1.smithi111.stdout:3/69: creat d0/d7/f12 x:0 0 0 2024-03-23T11:47:25.011 INFO:tasks.workunit.client.0.smithi063.stdout:3/220: chown d13/d22/c26 88772 1 2024-03-23T11:47:25.012 INFO:tasks.workunit.client.0.smithi063.stdout:6/110: fsync d1/d13/f1c 0 2024-03-23T11:47:25.013 INFO:tasks.workunit.client.0.smithi063.stdout:7/129: symlink d5/d9/da/l2a 0 2024-03-23T11:47:25.014 INFO:tasks.workunit.client.1.smithi111.stdout:3/70: dread - d0/d7/ff zero size 2024-03-23T11:47:25.015 INFO:tasks.workunit.client.0.smithi063.stdout:0/73: dread d2/fc [0,4194304] 0 2024-03-23T11:47:25.015 INFO:tasks.workunit.client.0.smithi063.stdout:7/130: readlink d5/d6/l1d 0 2024-03-23T11:47:25.017 INFO:tasks.workunit.client.0.smithi063.stdout:7/131: stat d5/d6/f25 0 2024-03-23T11:47:25.021 INFO:tasks.workunit.client.1.smithi111.stdout:2/122: creat d2/d7/da/f2d x:0 0 0 2024-03-23T11:47:25.022 INFO:tasks.workunit.client.0.smithi063.stdout:3/221: creat d13/d15/f3c x:0 0 0 2024-03-23T11:47:25.024 INFO:tasks.workunit.client.0.smithi063.stdout:1/125: dread d3/f10 [0,4194304] 0 2024-03-23T11:47:25.024 INFO:tasks.workunit.client.0.smithi063.stdout:0/74: creat d2/d14/f19 x:0 0 0 2024-03-23T11:47:25.026 INFO:tasks.workunit.client.1.smithi111.stdout:8/57: write f3 [878913,85867] 0 2024-03-23T11:47:25.030 INFO:tasks.workunit.client.1.smithi111.stdout:4/82: truncate d0/d7/f16 374272 0 2024-03-23T11:47:25.031 INFO:tasks.workunit.client.1.smithi111.stdout:3/71: stat d0/l6 0 2024-03-23T11:47:25.036 INFO:tasks.workunit.client.1.smithi111.stdout:6/184: write d2/f28 [636969,103379] 0 2024-03-23T11:47:25.036 INFO:tasks.workunit.client.0.smithi063.stdout:7/132: rename d5/d6/l1d to d5/d26/l2b 0 2024-03-23T11:47:25.036 INFO:tasks.workunit.client.1.smithi111.stdout:3/72: rename d0 to d0/d13 22 2024-03-23T11:47:25.037 INFO:tasks.workunit.client.1.smithi111.stdout:8/58: write d5/f6 [3675485,65994] 0 2024-03-23T11:47:25.037 INFO:tasks.workunit.client.1.smithi111.stdout:5/123: dwrite d1/f17 [0,4194304] 0 2024-03-23T11:47:25.038 INFO:tasks.workunit.client.0.smithi063.stdout:6/111: dwrite d1/dc/f1d [0,4194304] 0 2024-03-23T11:47:25.039 INFO:tasks.workunit.client.1.smithi111.stdout:2/123: chown d2/de/l1a 7 1 2024-03-23T11:47:25.040 INFO:tasks.workunit.client.1.smithi111.stdout:5/124: chown d1/d3 17 1 2024-03-23T11:47:25.041 INFO:tasks.workunit.client.0.smithi063.stdout:7/133: write d5/d22/f24 [80630,28721] 0 2024-03-23T11:47:25.041 INFO:tasks.workunit.client.1.smithi111.stdout:9/140: dwrite d0/d4/f15 [0,4194304] 0 2024-03-23T11:47:25.044 INFO:tasks.workunit.client.1.smithi111.stdout:2/124: write d2/d11/d17/d1f/f20 [1421419,17274] 0 2024-03-23T11:47:25.045 INFO:tasks.workunit.client.1.smithi111.stdout:3/73: chown d0/d7/f10 158359 1 2024-03-23T11:47:25.049 INFO:tasks.workunit.client.1.smithi111.stdout:5/125: symlink d1/d3/d6/d7/l2f 0 2024-03-23T11:47:25.049 INFO:tasks.workunit.client.0.smithi063.stdout:0/75: mknod d2/c1a 0 2024-03-23T11:47:25.053 INFO:tasks.workunit.client.0.smithi063.stdout:6/112: mknod d1/d15/c20 0 2024-03-23T11:47:25.055 INFO:tasks.workunit.client.0.smithi063.stdout:7/134: readlink d5/l10 0 2024-03-23T11:47:25.055 INFO:tasks.workunit.client.0.smithi063.stdout:3/222: creat d13/d22/f3d x:0 0 0 2024-03-23T11:47:25.057 INFO:tasks.workunit.client.1.smithi111.stdout:2/125: mkdir d2/de/d27/d2e 0 2024-03-23T11:47:25.061 INFO:tasks.workunit.client.1.smithi111.stdout:8/59: link l4 d5/l15 0 2024-03-23T11:47:25.062 INFO:tasks.workunit.client.1.smithi111.stdout:9/141: link d0/d1/l8 d0/d4/d1c/l33 0 2024-03-23T11:47:25.064 INFO:tasks.workunit.client.0.smithi063.stdout:0/76: mknod d2/d14/c1b 0 2024-03-23T11:47:25.067 INFO:tasks.workunit.client.1.smithi111.stdout:4/83: dread d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:25.067 INFO:tasks.workunit.client.0.smithi063.stdout:1/126: getdents d3/df 0 2024-03-23T11:47:25.067 INFO:tasks.workunit.client.0.smithi063.stdout:6/113: mkdir d1/dc/d21 0 2024-03-23T11:47:25.071 INFO:tasks.workunit.client.1.smithi111.stdout:8/60: mknod d5/c16 0 2024-03-23T11:47:25.072 INFO:tasks.workunit.client.0.smithi063.stdout:4/100: dwrite d5/da/fd [0,4194304] 0 2024-03-23T11:47:25.073 INFO:tasks.workunit.client.0.smithi063.stdout:1/127: rename d3/c24 to d3/df/c2c 0 2024-03-23T11:47:25.074 INFO:tasks.workunit.client.1.smithi111.stdout:9/142: write d0/d1/f18 [7592,58959] 0 2024-03-23T11:47:25.075 INFO:tasks.workunit.client.1.smithi111.stdout:6/185: dwrite d2/d1a/f2d [0,4194304] 0 2024-03-23T11:47:25.077 INFO:tasks.workunit.client.0.smithi063.stdout:0/77: write d2/d4/dd/ff [3541303,121958] 0 2024-03-23T11:47:25.080 INFO:tasks.workunit.client.1.smithi111.stdout:4/84: mkdir d0/dc/d1f 0 2024-03-23T11:47:25.082 INFO:tasks.workunit.client.1.smithi111.stdout:9/143: chown d0/d4/d1c 447 1 2024-03-23T11:47:25.083 INFO:tasks.workunit.client.1.smithi111.stdout:2/126: chown d2/de/d12/l26 17157533 1 2024-03-23T11:47:25.083 INFO:tasks.workunit.client.1.smithi111.stdout:8/61: readlink d5/l15 0 2024-03-23T11:47:25.086 INFO:tasks.workunit.client.1.smithi111.stdout:6/186: chown d2/d2a/c39 148 1 2024-03-23T11:47:25.086 INFO:tasks.workunit.client.0.smithi063.stdout:1/128: rename d3/ce to d3/c2d 0 2024-03-23T11:47:25.091 INFO:tasks.workunit.client.1.smithi111.stdout:9/144: readlink d0/d1/l8 0 2024-03-23T11:47:25.092 INFO:tasks.workunit.client.0.smithi063.stdout:1/129: chown d3/d1c/c1f 254957 1 2024-03-23T11:47:25.093 INFO:tasks.workunit.client.0.smithi063.stdout:3/223: dwrite d13/d15/f2a [0,4194304] 0 2024-03-23T11:47:25.095 INFO:tasks.workunit.client.1.smithi111.stdout:5/126: dwrite d1/d3/fc [4194304,4194304] 0 2024-03-23T11:47:25.097 INFO:tasks.workunit.client.1.smithi111.stdout:3/74: dwrite d0/d7/f12 [0,4194304] 0 2024-03-23T11:47:25.098 INFO:tasks.workunit.client.0.smithi063.stdout:7/135: dwrite d5/d6/f25 [0,4194304] 0 2024-03-23T11:47:25.098 INFO:tasks.workunit.client.0.smithi063.stdout:0/78: rename d2/fc to d2/d14/f1c 0 2024-03-23T11:47:25.099 INFO:tasks.workunit.client.1.smithi111.stdout:3/75: chown d0/l2 125597096 1 2024-03-23T11:47:25.099 INFO:tasks.workunit.client.0.smithi063.stdout:6/114: dread d1/f5 [0,4194304] 0 2024-03-23T11:47:25.100 INFO:tasks.workunit.client.1.smithi111.stdout:8/62: creat d5/f17 x:0 0 0 2024-03-23T11:47:25.102 INFO:tasks.workunit.client.1.smithi111.stdout:8/63: write d5/fe [359613,51601] 0 2024-03-23T11:47:25.105 INFO:tasks.workunit.client.0.smithi063.stdout:4/101: dread d5/ff [4194304,4194304] 0 2024-03-23T11:47:25.105 INFO:tasks.workunit.client.0.smithi063.stdout:7/136: write d5/d9/d20/f28 [44643,115128] 0 2024-03-23T11:47:25.108 INFO:tasks.workunit.client.0.smithi063.stdout:3/224: rmdir d13 39 2024-03-23T11:47:25.109 INFO:tasks.workunit.client.1.smithi111.stdout:6/187: creat d2/d35/f3f x:0 0 0 2024-03-23T11:47:25.113 INFO:tasks.workunit.client.0.smithi063.stdout:6/115: write d1/f5 [1275849,51490] 0 2024-03-23T11:47:25.114 INFO:tasks.workunit.client.1.smithi111.stdout:5/127: dread d1/d3/d6/d7/f1f [0,4194304] 0 2024-03-23T11:47:25.115 INFO:tasks.workunit.client.1.smithi111.stdout:5/128: stat d1/d3/d23 0 2024-03-23T11:47:25.118 INFO:tasks.workunit.client.1.smithi111.stdout:9/145: mkdir d0/d1/d28/d34 0 2024-03-23T11:47:25.119 INFO:tasks.workunit.client.0.smithi063.stdout:4/102: mkdir d5/d10/d16 0 2024-03-23T11:47:25.122 INFO:tasks.workunit.client.0.smithi063.stdout:1/130: dwrite d3/d1c/f27 [0,4194304] 0 2024-03-23T11:47:25.123 INFO:tasks.workunit.client.1.smithi111.stdout:8/64: dread d5/fe [0,4194304] 0 2024-03-23T11:47:25.124 INFO:tasks.workunit.client.0.smithi063.stdout:9/27: sync 2024-03-23T11:47:25.125 INFO:tasks.workunit.client.1.smithi111.stdout:8/65: chown d5/l9 1116405 1 2024-03-23T11:47:25.127 INFO:tasks.workunit.client.0.smithi063.stdout:6/116: mknod d1/d13/c22 0 2024-03-23T11:47:25.128 INFO:tasks.workunit.client.1.smithi111.stdout:3/76: fdatasync d0/d7/f12 0 2024-03-23T11:47:25.131 INFO:tasks.workunit.client.1.smithi111.stdout:3/77: write d0/d7/f10 [473940,80493] 0 2024-03-23T11:47:25.131 INFO:tasks.workunit.client.1.smithi111.stdout:6/188: creat d2/d35/f40 x:0 0 0 2024-03-23T11:47:25.136 INFO:tasks.workunit.client.1.smithi111.stdout:2/127: link d2/de/d27/f28 d2/de/d27/f2f 0 2024-03-23T11:47:25.136 INFO:tasks.workunit.client.1.smithi111.stdout:3/78: dread d0/f3 [0,4194304] 0 2024-03-23T11:47:25.138 INFO:tasks.workunit.client.1.smithi111.stdout:0/115: sync 2024-03-23T11:47:25.139 INFO:tasks.workunit.client.1.smithi111.stdout:3/79: write d0/d7/f12 [446318,123541] 0 2024-03-23T11:47:25.142 INFO:tasks.workunit.client.0.smithi063.stdout:9/28: dread f0 [4194304,4194304] 0 2024-03-23T11:47:25.142 INFO:tasks.workunit.client.1.smithi111.stdout:8/66: dwrite d5/f6 [4194304,4194304] 0 2024-03-23T11:47:25.147 INFO:tasks.workunit.client.0.smithi063.stdout:6/117: symlink d1/d13/l23 0 2024-03-23T11:47:25.148 INFO:tasks.workunit.client.0.smithi063.stdout:4/103: mknod d5/c17 0 2024-03-23T11:47:25.148 INFO:tasks.workunit.client.0.smithi063.stdout:4/104: stat l3 0 2024-03-23T11:47:25.149 INFO:tasks.workunit.client.1.smithi111.stdout:2/128: mkdir d2/d11/d17/d1f/d30 0 2024-03-23T11:47:25.152 INFO:tasks.workunit.client.1.smithi111.stdout:5/129: mkdir d1/d3/d6/d30 0 2024-03-23T11:47:25.153 INFO:tasks.workunit.client.1.smithi111.stdout:3/80: mknod d0/d7/c14 0 2024-03-23T11:47:25.156 INFO:tasks.workunit.client.1.smithi111.stdout:2/129: write d2/de/f16 [432982,43173] 0 2024-03-23T11:47:25.156 INFO:tasks.workunit.client.1.smithi111.stdout:8/67: unlink d5/c13 0 2024-03-23T11:47:25.157 INFO:tasks.workunit.client.0.smithi063.stdout:6/118: rmdir d1/d15 39 2024-03-23T11:47:25.157 INFO:tasks.workunit.client.0.smithi063.stdout:1/131: link d3/df/l14 d3/l2e 0 2024-03-23T11:47:25.159 INFO:tasks.workunit.client.1.smithi111.stdout:5/130: write d1/d3/d6/ff [298631,123166] 0 2024-03-23T11:47:25.159 INFO:tasks.workunit.client.1.smithi111.stdout:4/85: rename d0/d7/d9/c19 to d0/d7/c20 0 2024-03-23T11:47:25.160 INFO:tasks.workunit.client.0.smithi063.stdout:6/119: write d1/f5 [1486977,45492] 0 2024-03-23T11:47:25.163 INFO:tasks.workunit.client.1.smithi111.stdout:6/189: creat d2/f41 x:0 0 0 2024-03-23T11:47:25.164 INFO:tasks.workunit.client.1.smithi111.stdout:4/86: dread - d0/dc/f18 zero size 2024-03-23T11:47:25.166 INFO:tasks.workunit.client.1.smithi111.stdout:0/116: dwrite d5/f13 [0,4194304] 0 2024-03-23T11:47:25.168 INFO:tasks.workunit.client.1.smithi111.stdout:2/130: symlink d2/de/d27/l31 0 2024-03-23T11:47:25.177 INFO:tasks.workunit.client.1.smithi111.stdout:2/131: mkdir d2/d32 0 2024-03-23T11:47:25.178 INFO:tasks.workunit.client.1.smithi111.stdout:5/131: rename d1/d3/d6/d7/f9 to d1/d3/d6/f31 0 2024-03-23T11:47:25.181 INFO:tasks.workunit.client.1.smithi111.stdout:8/68: dwrite f3 [0,4194304] 0 2024-03-23T11:47:25.185 INFO:tasks.workunit.client.1.smithi111.stdout:8/69: chown d5/fa 24770 1 2024-03-23T11:47:25.192 INFO:tasks.workunit.client.1.smithi111.stdout:0/117: truncate f4 2230875 0 2024-03-23T11:47:25.195 INFO:tasks.workunit.client.1.smithi111.stdout:4/87: dwrite d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:25.197 INFO:tasks.workunit.client.1.smithi111.stdout:0/118: write d5/db/d1f/f18 [1402809,71887] 0 2024-03-23T11:47:25.197 INFO:tasks.workunit.client.1.smithi111.stdout:2/132: dread d2/f4 [0,4194304] 0 2024-03-23T11:47:25.200 INFO:tasks.workunit.client.1.smithi111.stdout:5/132: dread d1/d3/d6/d7/f13 [0,4194304] 0 2024-03-23T11:47:25.205 INFO:tasks.workunit.client.1.smithi111.stdout:2/133: write d2/d11/d17/f21 [926100,14706] 0 2024-03-23T11:47:25.206 INFO:tasks.workunit.client.1.smithi111.stdout:0/119: mknod d5/db/d1f/d19/c20 0 2024-03-23T11:47:25.210 INFO:tasks.workunit.client.1.smithi111.stdout:0/120: mknod d5/db/c21 0 2024-03-23T11:47:25.211 INFO:tasks.workunit.client.1.smithi111.stdout:8/70: dwrite d5/f11 [0,4194304] 0 2024-03-23T11:47:25.212 INFO:tasks.workunit.client.1.smithi111.stdout:0/121: chown d5/f13 4899335 1 2024-03-23T11:47:25.213 INFO:tasks.workunit.client.0.smithi063.stdout:0/79: write d2/d14/f1c [309561,11657] 0 2024-03-23T11:47:25.215 INFO:tasks.workunit.client.1.smithi111.stdout:5/133: truncate d1/d3/f4 5504668 0 2024-03-23T11:47:25.220 INFO:tasks.workunit.client.1.smithi111.stdout:8/71: dread d5/fe [0,4194304] 0 2024-03-23T11:47:25.220 INFO:tasks.workunit.client.1.smithi111.stdout:4/88: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:25.224 INFO:tasks.workunit.client.1.smithi111.stdout:5/134: rename d1/c5 to d1/d20/c32 0 2024-03-23T11:47:25.224 INFO:tasks.workunit.client.0.smithi063.stdout:0/80: mknod d2/d14/c1d 0 2024-03-23T11:47:25.227 INFO:tasks.workunit.client.0.smithi063.stdout:9/29: dwrite f2 [0,4194304] 0 2024-03-23T11:47:25.229 INFO:tasks.workunit.client.0.smithi063.stdout:1/132: dread d3/f4 [0,4194304] 0 2024-03-23T11:47:25.230 INFO:tasks.workunit.client.1.smithi111.stdout:0/122: rmdir d5 39 2024-03-23T11:47:25.233 INFO:tasks.workunit.client.1.smithi111.stdout:8/72: rename d5/cc to d5/c18 0 2024-03-23T11:47:25.234 INFO:tasks.workunit.client.1.smithi111.stdout:4/89: truncate d0/dc/f1c 420398 0 2024-03-23T11:47:25.234 INFO:tasks.workunit.client.1.smithi111.stdout:8/73: fsync f3 0 2024-03-23T11:47:25.237 INFO:tasks.workunit.client.0.smithi063.stdout:1/133: chown d3/df/l13 11548506 1 2024-03-23T11:47:25.237 INFO:tasks.workunit.client.0.smithi063.stdout:9/30: write f0 [6407293,105211] 0 2024-03-23T11:47:25.239 INFO:tasks.workunit.client.0.smithi063.stdout:0/81: mknod d2/c1e 0 2024-03-23T11:47:25.239 INFO:tasks.workunit.client.0.smithi063.stdout:6/120: dwrite d1/dc/f1f [0,4194304] 0 2024-03-23T11:47:25.245 INFO:tasks.workunit.client.0.smithi063.stdout:1/134: mkdir d3/df/d2f 0 2024-03-23T11:47:25.247 INFO:tasks.workunit.client.0.smithi063.stdout:6/121: write d1/f12 [3078327,48850] 0 2024-03-23T11:47:25.248 INFO:tasks.workunit.client.1.smithi111.stdout:5/135: dwrite d1/d3/d6/f31 [0,4194304] 0 2024-03-23T11:47:25.251 INFO:tasks.workunit.client.0.smithi063.stdout:9/31: creat d4/f6 x:0 0 0 2024-03-23T11:47:25.251 INFO:tasks.workunit.client.0.smithi063.stdout:9/32: readlink - no filename 2024-03-23T11:47:25.252 INFO:tasks.workunit.client.0.smithi063.stdout:0/82: readlink d2/d4/l17 0 2024-03-23T11:47:25.252 INFO:tasks.workunit.client.1.smithi111.stdout:5/136: fsync d1/d3/d23/d2c/f2d 0 2024-03-23T11:47:25.254 INFO:tasks.workunit.client.1.smithi111.stdout:5/137: dread - d1/d3/d6/f19 zero size 2024-03-23T11:47:25.255 INFO:tasks.workunit.client.0.smithi063.stdout:9/33: mknod d4/c7 0 2024-03-23T11:47:25.256 INFO:tasks.workunit.client.0.smithi063.stdout:6/122: write d1/f5 [198332,24752] 0 2024-03-23T11:47:25.257 INFO:tasks.workunit.client.0.smithi063.stdout:9/34: stat d4/c7 0 2024-03-23T11:47:25.257 INFO:tasks.workunit.client.1.smithi111.stdout:4/90: mknod d0/dc/d1f/c21 0 2024-03-23T11:47:25.258 INFO:tasks.workunit.client.0.smithi063.stdout:1/135: creat d3/df/f30 x:0 0 0 2024-03-23T11:47:25.259 INFO:tasks.workunit.client.1.smithi111.stdout:4/91: rename d0 to d0/d7/da/d22 22 2024-03-23T11:47:25.259 INFO:tasks.workunit.client.0.smithi063.stdout:0/83: mkdir d2/d4/dd/d1f 0 2024-03-23T11:47:25.259 INFO:tasks.workunit.client.1.smithi111.stdout:8/74: link d5/f6 d5/f19 0 2024-03-23T11:47:25.260 INFO:tasks.workunit.client.1.smithi111.stdout:0/123: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:25.261 INFO:tasks.workunit.client.0.smithi063.stdout:0/84: fdatasync d2/d14/f19 0 2024-03-23T11:47:25.262 INFO:tasks.workunit.client.0.smithi063.stdout:4/105: dread d5/fc [0,4194304] 0 2024-03-23T11:47:25.263 INFO:tasks.workunit.client.1.smithi111.stdout:0/124: write d5/db/d1f/d19/f1c [323549,92040] 0 2024-03-23T11:47:25.263 INFO:tasks.workunit.client.0.smithi063.stdout:9/35: chown c3 65427 1 2024-03-23T11:47:25.264 INFO:tasks.workunit.client.0.smithi063.stdout:0/85: stat d2/d4/dd/d1f 0 2024-03-23T11:47:25.265 INFO:tasks.workunit.client.0.smithi063.stdout:1/136: mkdir d3/df/d31 0 2024-03-23T11:47:25.266 INFO:tasks.workunit.client.1.smithi111.stdout:8/75: creat d5/f1a x:0 0 0 2024-03-23T11:47:25.266 INFO:tasks.workunit.client.1.smithi111.stdout:4/92: read d0/d7/f16 [333602,52375] 0 2024-03-23T11:47:25.268 INFO:tasks.workunit.client.1.smithi111.stdout:4/93: dread - d0/dc/f18 zero size 2024-03-23T11:47:25.268 INFO:tasks.workunit.client.0.smithi063.stdout:9/36: write d4/f5 [611337,64316] 0 2024-03-23T11:47:25.270 INFO:tasks.workunit.client.1.smithi111.stdout:4/94: truncate d0/d7/d9/f15 1038890 0 2024-03-23T11:47:25.271 INFO:tasks.workunit.client.1.smithi111.stdout:0/125: creat d5/db/d1f/f22 x:0 0 0 2024-03-23T11:47:25.272 INFO:tasks.workunit.client.0.smithi063.stdout:9/37: symlink d4/l8 0 2024-03-23T11:47:25.273 INFO:tasks.workunit.client.1.smithi111.stdout:8/76: mkdir d5/d1b 0 2024-03-23T11:47:25.276 INFO:tasks.workunit.client.1.smithi111.stdout:8/77: fdatasync d5/f19 0 2024-03-23T11:47:25.277 INFO:tasks.workunit.client.1.smithi111.stdout:8/78: readlink d5/l9 0 2024-03-23T11:47:25.278 INFO:tasks.workunit.client.1.smithi111.stdout:8/79: write d5/f17 [53808,104067] 0 2024-03-23T11:47:25.280 INFO:tasks.workunit.client.1.smithi111.stdout:8/80: read f3 [2285588,112591] 0 2024-03-23T11:47:25.286 INFO:tasks.workunit.client.1.smithi111.stdout:8/81: mknod d5/d1b/c1c 0 2024-03-23T11:47:25.289 INFO:tasks.workunit.client.0.smithi063.stdout:4/106: dwrite d5/ff [4194304,4194304] 0 2024-03-23T11:47:25.289 INFO:tasks.workunit.client.0.smithi063.stdout:0/86: dwrite d2/d4/f9 [0,4194304] 0 2024-03-23T11:47:25.290 INFO:tasks.workunit.client.1.smithi111.stdout:5/138: truncate d1/d3/fc 5799170 0 2024-03-23T11:47:25.290 INFO:tasks.workunit.client.1.smithi111.stdout:4/95: dwrite d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:25.292 INFO:tasks.workunit.client.1.smithi111.stdout:8/82: mknod d5/d1b/c1d 0 2024-03-23T11:47:25.294 INFO:tasks.workunit.client.1.smithi111.stdout:5/139: dread - d1/d3/d6/f19 zero size 2024-03-23T11:47:25.294 INFO:tasks.workunit.client.1.smithi111.stdout:4/96: chown d0/c3 5484705 1 2024-03-23T11:47:25.294 INFO:tasks.workunit.client.0.smithi063.stdout:9/38: dread d4/f5 [0,4194304] 0 2024-03-23T11:47:25.296 INFO:tasks.workunit.client.0.smithi063.stdout:4/107: readlink d5/le 0 2024-03-23T11:47:25.296 INFO:tasks.workunit.client.0.smithi063.stdout:1/137: dwrite d3/df/f1a [4194304,4194304] 0 2024-03-23T11:47:25.296 INFO:tasks.workunit.client.1.smithi111.stdout:1/90: sync 2024-03-23T11:47:25.296 INFO:tasks.workunit.client.1.smithi111.stdout:7/88: sync 2024-03-23T11:47:25.298 INFO:tasks.workunit.client.0.smithi063.stdout:0/87: creat d2/d4/dd/d1f/f20 x:0 0 0 2024-03-23T11:47:25.298 INFO:tasks.workunit.client.0.smithi063.stdout:9/39: fdatasync d4/f6 0 2024-03-23T11:47:25.300 INFO:tasks.workunit.client.1.smithi111.stdout:0/126: dwrite d5/db/d1f/f1a [0,4194304] 0 2024-03-23T11:47:25.304 INFO:tasks.workunit.client.0.smithi063.stdout:1/138: creat d3/df/f32 x:0 0 0 2024-03-23T11:47:25.305 INFO:tasks.workunit.client.1.smithi111.stdout:5/140: symlink d1/d3/d23/d2c/l33 0 2024-03-23T11:47:25.306 INFO:tasks.workunit.client.0.smithi063.stdout:4/108: mknod d5/da/c18 0 2024-03-23T11:47:25.307 INFO:tasks.workunit.client.0.smithi063.stdout:9/40: creat d4/f9 x:0 0 0 2024-03-23T11:47:25.310 INFO:tasks.workunit.client.1.smithi111.stdout:1/91: link d0/d6/f11 d0/d6/f22 0 2024-03-23T11:47:25.310 INFO:tasks.workunit.client.0.smithi063.stdout:0/88: creat d2/d12/f21 x:0 0 0 2024-03-23T11:47:25.310 INFO:tasks.workunit.client.0.smithi063.stdout:1/139: mknod d3/c33 0 2024-03-23T11:47:25.312 INFO:tasks.workunit.client.0.smithi063.stdout:4/109: creat d5/da/f19 x:0 0 0 2024-03-23T11:47:25.312 INFO:tasks.workunit.client.0.smithi063.stdout:0/89: dread - d2/d14/f19 zero size 2024-03-23T11:47:25.313 INFO:tasks.workunit.client.1.smithi111.stdout:1/92: write d0/f20 [212523,100080] 0 2024-03-23T11:47:25.314 INFO:tasks.workunit.client.1.smithi111.stdout:5/141: rename d1/d3/d23/f28 to d1/d3/d23/d2c/f34 0 2024-03-23T11:47:25.314 INFO:tasks.workunit.client.0.smithi063.stdout:9/41: creat d4/fa x:0 0 0 2024-03-23T11:47:25.314 INFO:tasks.workunit.client.1.smithi111.stdout:4/97: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:25.315 INFO:tasks.workunit.client.0.smithi063.stdout:4/110: write f2 [6139461,82219] 0 2024-03-23T11:47:25.315 INFO:tasks.workunit.client.0.smithi063.stdout:0/90: chown f1 2084 1 2024-03-23T11:47:25.315 INFO:tasks.workunit.client.1.smithi111.stdout:1/93: fsync d0/f1 0 2024-03-23T11:47:25.318 INFO:tasks.workunit.client.0.smithi063.stdout:9/42: mknod d4/cb 0 2024-03-23T11:47:25.320 INFO:tasks.workunit.client.0.smithi063.stdout:0/91: write f1 [475521,79400] 0 2024-03-23T11:47:25.323 INFO:tasks.workunit.client.1.smithi111.stdout:0/127: dwrite d5/db/f15 [0,4194304] 0 2024-03-23T11:47:25.326 INFO:tasks.workunit.client.0.smithi063.stdout:0/92: write d2/d4/dd/ff [3348715,71921] 0 2024-03-23T11:47:25.331 INFO:tasks.workunit.client.0.smithi063.stdout:0/93: symlink d2/d12/l22 0 2024-03-23T11:47:25.333 INFO:tasks.workunit.client.0.smithi063.stdout:1/140: dread d3/d1c/f27 [0,4194304] 0 2024-03-23T11:47:25.333 INFO:tasks.workunit.client.0.smithi063.stdout:0/94: chown d2/d4 3627 1 2024-03-23T11:47:25.334 INFO:tasks.workunit.client.1.smithi111.stdout:0/128: rename d5/f17 to d5/db/d1f/d19/f23 0 2024-03-23T11:47:25.335 INFO:tasks.workunit.client.1.smithi111.stdout:4/98: dread d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:25.337 INFO:tasks.workunit.client.0.smithi063.stdout:1/141: chown d3/c2d 2 1 2024-03-23T11:47:25.337 INFO:tasks.workunit.client.0.smithi063.stdout:0/95: truncate d2/d12/f21 601855 0 2024-03-23T11:47:25.339 INFO:tasks.workunit.client.1.smithi111.stdout:5/142: dwrite d1/d3/d6/f31 [0,4194304] 0 2024-03-23T11:47:25.340 INFO:tasks.workunit.client.0.smithi063.stdout:9/43: dwrite f0 [0,4194304] 0 2024-03-23T11:47:25.340 INFO:tasks.workunit.client.0.smithi063.stdout:1/142: read d3/f11 [3841976,77455] 0 2024-03-23T11:47:25.341 INFO:tasks.workunit.client.1.smithi111.stdout:4/99: symlink d0/dc/l23 0 2024-03-23T11:47:25.342 INFO:tasks.workunit.client.0.smithi063.stdout:9/44: fdatasync d4/f6 0 2024-03-23T11:47:25.342 INFO:tasks.workunit.client.1.smithi111.stdout:1/94: dwrite d0/d5/dd/d1e/f21 [0,4194304] 0 2024-03-23T11:47:25.343 INFO:tasks.workunit.client.0.smithi063.stdout:0/96: creat d2/d14/f23 x:0 0 0 2024-03-23T11:47:25.343 INFO:tasks.workunit.client.0.smithi063.stdout:1/143: chown d3/df/f12 398936676 1 2024-03-23T11:47:25.345 INFO:tasks.workunit.client.0.smithi063.stdout:9/45: write f2 [1221520,56651] 0 2024-03-23T11:47:25.345 INFO:tasks.workunit.client.0.smithi063.stdout:1/144: stat d3/d1c/l1d 0 2024-03-23T11:47:25.346 INFO:tasks.workunit.client.0.smithi063.stdout:0/97: read - d2/d14/f19 zero size 2024-03-23T11:47:25.349 INFO:tasks.workunit.client.1.smithi111.stdout:8/83: getdents d5/d1b 0 2024-03-23T11:47:25.349 INFO:tasks.workunit.client.0.smithi063.stdout:9/46: creat d4/fc x:0 0 0 2024-03-23T11:47:25.351 INFO:tasks.workunit.client.1.smithi111.stdout:0/129: link d5/db/l11 d5/db/d1f/d19/l24 0 2024-03-23T11:47:25.356 INFO:tasks.workunit.client.0.smithi063.stdout:9/47: creat d4/fd x:0 0 0 2024-03-23T11:47:25.356 INFO:tasks.workunit.client.0.smithi063.stdout:0/98: rmdir d2/d4 39 2024-03-23T11:47:25.364 INFO:tasks.workunit.client.1.smithi111.stdout:4/100: link d0/d7/d9/l12 d0/l24 0 2024-03-23T11:47:25.364 INFO:tasks.workunit.client.1.smithi111.stdout:8/84: rename c0 to d5/d1b/c1e 0 2024-03-23T11:47:25.364 INFO:tasks.workunit.client.0.smithi063.stdout:0/99: dread - d2/d14/f15 zero size 2024-03-23T11:47:25.364 INFO:tasks.workunit.client.0.smithi063.stdout:9/48: unlink c3 0 2024-03-23T11:47:25.367 INFO:tasks.workunit.client.1.smithi111.stdout:7/89: truncate f9 990125 0 2024-03-23T11:47:25.368 INFO:tasks.workunit.client.1.smithi111.stdout:8/85: chown d5/cd 190 1 2024-03-23T11:47:25.369 INFO:tasks.workunit.client.0.smithi063.stdout:9/49: write d4/f9 [508021,75484] 0 2024-03-23T11:47:25.370 INFO:tasks.workunit.client.0.smithi063.stdout:1/145: dwrite d3/df/f32 [0,4194304] 0 2024-03-23T11:47:25.371 INFO:tasks.workunit.client.0.smithi063.stdout:4/111: truncate f2 4541903 0 2024-03-23T11:47:25.372 INFO:tasks.workunit.client.0.smithi063.stdout:0/100: chown d2/d4/dd/c10 4383 1 2024-03-23T11:47:25.373 INFO:tasks.workunit.client.1.smithi111.stdout:5/143: dwrite d1/d3/d6/f31 [0,4194304] 0 2024-03-23T11:47:25.374 INFO:tasks.workunit.client.1.smithi111.stdout:7/90: unlink l8 0 2024-03-23T11:47:25.376 INFO:tasks.workunit.client.0.smithi063.stdout:4/112: stat d5/da/c18 0 2024-03-23T11:47:25.376 INFO:tasks.workunit.client.1.smithi111.stdout:8/86: write d5/f6 [7781931,30887] 0 2024-03-23T11:47:25.377 INFO:tasks.workunit.client.0.smithi063.stdout:0/101: dread d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:25.378 INFO:tasks.workunit.client.0.smithi063.stdout:9/50: creat d4/fe x:0 0 0 2024-03-23T11:47:25.381 INFO:tasks.workunit.client.0.smithi063.stdout:0/102: dread - d2/d4/dd/d1f/f20 zero size 2024-03-23T11:47:25.383 INFO:tasks.workunit.client.1.smithi111.stdout:4/101: rename d0/dc/f18 to d0/d7/f25 0 2024-03-23T11:47:25.384 INFO:tasks.workunit.client.1.smithi111.stdout:1/95: dwrite d0/f4 [0,4194304] 0 2024-03-23T11:47:25.384 INFO:tasks.workunit.client.1.smithi111.stdout:5/144: fsync d1/d3/d23/d2c/f34 0 2024-03-23T11:47:25.386 INFO:tasks.workunit.client.0.smithi063.stdout:4/113: rename d5/le to d5/da/l1a 0 2024-03-23T11:47:25.387 INFO:tasks.workunit.client.1.smithi111.stdout:0/130: link d5/db/d1f/d19/c1e d5/db/d1f/d19/c25 0 2024-03-23T11:47:25.389 INFO:tasks.workunit.client.0.smithi063.stdout:4/114: chown d5/da 55 1 2024-03-23T11:47:25.389 INFO:tasks.workunit.client.1.smithi111.stdout:8/87: creat d5/f1f x:0 0 0 2024-03-23T11:47:25.390 INFO:tasks.workunit.client.1.smithi111.stdout:0/131: fdatasync d5/fc 0 2024-03-23T11:47:25.391 INFO:tasks.workunit.client.1.smithi111.stdout:1/96: chown d0/fe 27 1 2024-03-23T11:47:25.391 INFO:tasks.workunit.client.1.smithi111.stdout:8/88: read d5/f17 [153555,45174] 0 2024-03-23T11:47:25.393 INFO:tasks.workunit.client.1.smithi111.stdout:4/102: write d0/d7/f1b [3752414,51198] 0 2024-03-23T11:47:25.394 INFO:tasks.workunit.client.1.smithi111.stdout:5/145: fdatasync d1/d3/d23/d2c/f2d 0 2024-03-23T11:47:25.395 INFO:tasks.workunit.client.1.smithi111.stdout:1/97: write d0/f1d [55759,99932] 0 2024-03-23T11:47:25.396 INFO:tasks.workunit.client.0.smithi063.stdout:4/115: creat d5/f1b x:0 0 0 2024-03-23T11:47:25.396 INFO:tasks.workunit.client.0.smithi063.stdout:1/146: dwrite d3/f10 [0,4194304] 0 2024-03-23T11:47:25.397 INFO:tasks.workunit.client.1.smithi111.stdout:8/89: mknod d5/c20 0 2024-03-23T11:47:25.401 INFO:tasks.workunit.client.1.smithi111.stdout:1/98: chown d0/d5/dd/d1e/l1f 79385 1 2024-03-23T11:47:25.402 INFO:tasks.workunit.client.0.smithi063.stdout:0/103: dread f1 [0,4194304] 0 2024-03-23T11:47:25.403 INFO:tasks.workunit.client.0.smithi063.stdout:4/116: creat d5/f1c x:0 0 0 2024-03-23T11:47:25.404 INFO:tasks.workunit.client.1.smithi111.stdout:1/99: write d0/f1a [163368,10125] 0 2024-03-23T11:47:25.408 INFO:tasks.workunit.client.1.smithi111.stdout:1/100: stat d0/d5/dd/d1e/l1f 0 2024-03-23T11:47:25.410 INFO:tasks.workunit.client.0.smithi063.stdout:0/104: rename d2/d4/l17 to d2/d4/dd/l24 0 2024-03-23T11:47:25.410 INFO:tasks.workunit.client.0.smithi063.stdout:9/51: rmdir d4 39 2024-03-23T11:47:25.410 INFO:tasks.workunit.client.0.smithi063.stdout:4/117: fsync d5/ff 0 2024-03-23T11:47:25.412 INFO:tasks.workunit.client.1.smithi111.stdout:7/91: dread f1 [4194304,4194304] 0 2024-03-23T11:47:25.413 INFO:tasks.workunit.client.1.smithi111.stdout:4/103: getdents d0/d7/d9 0 2024-03-23T11:47:25.413 INFO:tasks.workunit.client.1.smithi111.stdout:9/146: sync 2024-03-23T11:47:25.418 INFO:tasks.workunit.client.0.smithi063.stdout:0/105: dread d2/d12/f21 [0,4194304] 0 2024-03-23T11:47:25.418 INFO:tasks.workunit.client.0.smithi063.stdout:4/118: creat d5/d10/d16/f1d x:0 0 0 2024-03-23T11:47:25.419 INFO:tasks.workunit.client.1.smithi111.stdout:7/92: chown l5 3351824 1 2024-03-23T11:47:25.419 INFO:tasks.workunit.client.0.smithi063.stdout:4/119: chown d5/da/fd 727944672 1 2024-03-23T11:47:25.420 INFO:tasks.workunit.client.1.smithi111.stdout:1/101: write d0/d5/f16 [1286018,104226] 0 2024-03-23T11:47:25.420 INFO:tasks.workunit.client.0.smithi063.stdout:4/120: chown d5/l7 80025 1 2024-03-23T11:47:25.421 INFO:tasks.workunit.client.0.smithi063.stdout:9/52: rename d4/f6 to d4/ff 0 2024-03-23T11:47:25.423 INFO:tasks.workunit.client.0.smithi063.stdout:9/53: read - d4/ff zero size 2024-03-23T11:47:25.423 INFO:tasks.workunit.client.1.smithi111.stdout:9/147: mkdir d0/d31/d35 0 2024-03-23T11:47:25.424 INFO:tasks.workunit.client.0.smithi063.stdout:0/106: mkdir d2/d4/dd/d25 0 2024-03-23T11:47:25.425 INFO:tasks.workunit.client.0.smithi063.stdout:4/121: write d5/da/f19 [534058,83978] 0 2024-03-23T11:47:25.425 INFO:tasks.workunit.client.1.smithi111.stdout:7/93: chown f10 207 1 2024-03-23T11:47:25.430 INFO:tasks.workunit.client.1.smithi111.stdout:1/102: unlink d0/f12 0 2024-03-23T11:47:25.435 INFO:tasks.workunit.client.1.smithi111.stdout:7/94: write fb [3864372,4234] 0 2024-03-23T11:47:25.436 INFO:tasks.workunit.client.1.smithi111.stdout:8/90: dwrite d5/f19 [0,4194304] 0 2024-03-23T11:47:25.437 INFO:tasks.workunit.client.1.smithi111.stdout:1/103: write d0/f20 [1034917,121006] 0 2024-03-23T11:47:25.437 INFO:tasks.workunit.client.1.smithi111.stdout:4/104: dread d0/d7/da/f13 [0,4194304] 0 2024-03-23T11:47:25.438 INFO:tasks.workunit.client.1.smithi111.stdout:0/132: dwrite d5/db/d1f/f22 [0,4194304] 0 2024-03-23T11:47:25.439 INFO:tasks.workunit.client.1.smithi111.stdout:4/105: readlink d0/l1a 0 2024-03-23T11:47:25.439 INFO:tasks.workunit.client.1.smithi111.stdout:7/95: dread - fe zero size 2024-03-23T11:47:25.443 INFO:tasks.workunit.client.1.smithi111.stdout:8/91: truncate d5/f1f 636656 0 2024-03-23T11:47:25.443 INFO:tasks.workunit.client.1.smithi111.stdout:1/104: creat d0/d5/f23 x:0 0 0 2024-03-23T11:47:25.445 INFO:tasks.workunit.client.0.smithi063.stdout:9/54: dwrite d4/fd [0,4194304] 0 2024-03-23T11:47:25.445 INFO:tasks.workunit.client.1.smithi111.stdout:0/133: creat d5/db/f26 x:0 0 0 2024-03-23T11:47:25.448 INFO:tasks.workunit.client.1.smithi111.stdout:8/92: symlink d5/l21 0 2024-03-23T11:47:25.450 INFO:tasks.workunit.client.1.smithi111.stdout:1/105: creat d0/d6/f24 x:0 0 0 2024-03-23T11:47:25.451 INFO:tasks.workunit.client.0.smithi063.stdout:9/55: unlink d4/c7 0 2024-03-23T11:47:25.452 INFO:tasks.workunit.client.0.smithi063.stdout:4/122: dwrite d5/fc [4194304,4194304] 0 2024-03-23T11:47:25.452 INFO:tasks.workunit.client.0.smithi063.stdout:9/56: stat d4/l8 0 2024-03-23T11:47:25.454 INFO:tasks.workunit.client.0.smithi063.stdout:4/123: truncate d5/f1c 656846 0 2024-03-23T11:47:25.456 INFO:tasks.workunit.client.1.smithi111.stdout:9/148: dread d0/d4/fe [0,4194304] 0 2024-03-23T11:47:25.459 INFO:tasks.workunit.client.0.smithi063.stdout:9/57: mknod d4/c10 0 2024-03-23T11:47:25.460 INFO:tasks.workunit.client.1.smithi111.stdout:8/93: truncate f3 3997279 0 2024-03-23T11:47:25.463 INFO:tasks.workunit.client.1.smithi111.stdout:8/94: chown d5/f6 59391270 1 2024-03-23T11:47:25.465 INFO:tasks.workunit.client.0.smithi063.stdout:9/58: mkdir d4/d11 0 2024-03-23T11:47:25.465 INFO:tasks.workunit.client.0.smithi063.stdout:9/59: fdatasync f2 0 2024-03-23T11:47:25.467 INFO:tasks.workunit.client.0.smithi063.stdout:9/60: stat d4/fa 0 2024-03-23T11:47:25.468 INFO:tasks.workunit.client.1.smithi111.stdout:9/149: rename d0/d1 to d0/d4/d1c/d36 0 2024-03-23T11:47:25.470 INFO:tasks.workunit.client.0.smithi063.stdout:9/61: creat d4/d11/f12 x:0 0 0 2024-03-23T11:47:25.473 INFO:tasks.workunit.client.1.smithi111.stdout:5/146: dwrite d1/d3/d6/d29/f21 [4194304,4194304] 0 2024-03-23T11:47:25.473 INFO:tasks.workunit.client.0.smithi063.stdout:9/62: symlink d4/d11/l13 0 2024-03-23T11:47:25.474 INFO:tasks.workunit.client.0.smithi063.stdout:9/63: readlink d4/l8 0 2024-03-23T11:47:25.474 INFO:tasks.workunit.client.0.smithi063.stdout:4/124: dwrite d5/ff [0,4194304] 0 2024-03-23T11:47:25.479 INFO:tasks.workunit.client.1.smithi111.stdout:4/106: dread d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:25.483 INFO:tasks.workunit.client.1.smithi111.stdout:7/96: truncate f1 8385141 0 2024-03-23T11:47:25.484 INFO:tasks.workunit.client.1.smithi111.stdout:8/95: dread d5/f17 [0,4194304] 0 2024-03-23T11:47:25.487 INFO:tasks.workunit.client.1.smithi111.stdout:7/97: unlink cc 0 2024-03-23T11:47:25.487 INFO:tasks.workunit.client.1.smithi111.stdout:3/81: sync 2024-03-23T11:47:25.490 INFO:tasks.workunit.client.0.smithi063.stdout:4/125: dwrite d5/f1c [0,4194304] 0 2024-03-23T11:47:25.490 INFO:tasks.workunit.client.1.smithi111.stdout:5/147: creat d1/f35 x:0 0 0 2024-03-23T11:47:25.491 INFO:tasks.workunit.client.1.smithi111.stdout:0/134: dwrite d5/db/d1f/f22 [0,4194304] 0 2024-03-23T11:47:25.495 INFO:tasks.workunit.client.1.smithi111.stdout:3/82: stat d0/d7/c14 0 2024-03-23T11:47:25.496 INFO:tasks.workunit.client.1.smithi111.stdout:7/98: truncate f10 876333 0 2024-03-23T11:47:25.499 INFO:tasks.workunit.client.1.smithi111.stdout:8/96: truncate d5/f17 687990 0 2024-03-23T11:47:25.501 INFO:tasks.workunit.client.1.smithi111.stdout:1/106: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:25.504 INFO:tasks.workunit.client.0.smithi063.stdout:4/126: dread f2 [4194304,4194304] 0 2024-03-23T11:47:25.508 INFO:tasks.workunit.client.1.smithi111.stdout:0/135: creat d5/db/f27 x:0 0 0 2024-03-23T11:47:25.508 INFO:tasks.workunit.client.1.smithi111.stdout:5/148: read d1/d3/d6/f31 [111617,86890] 0 2024-03-23T11:47:25.509 INFO:tasks.workunit.client.1.smithi111.stdout:9/150: dread d0/d4/d1c/f26 [0,4194304] 0 2024-03-23T11:47:25.516 INFO:tasks.workunit.client.1.smithi111.stdout:3/83: dread d0/f3 [0,4194304] 0 2024-03-23T11:47:25.517 INFO:tasks.workunit.client.1.smithi111.stdout:3/84: dread - d0/d7/ff zero size 2024-03-23T11:47:25.517 INFO:tasks.workunit.client.1.smithi111.stdout:7/99: link fb d14/f15 0 2024-03-23T11:47:25.521 INFO:tasks.workunit.client.1.smithi111.stdout:4/107: getdents d0/d7/da 0 2024-03-23T11:47:25.523 INFO:tasks.workunit.client.1.smithi111.stdout:0/136: creat d5/db/d1f/f28 x:0 0 0 2024-03-23T11:47:25.524 INFO:tasks.workunit.client.1.smithi111.stdout:5/149: chown d1/d3/d6/d7/f1f 1399 1 2024-03-23T11:47:25.525 INFO:tasks.workunit.client.1.smithi111.stdout:9/151: chown d0/d4/d1c/c23 799 1 2024-03-23T11:47:25.528 INFO:tasks.workunit.client.0.smithi063.stdout:4/127: dwrite d5/da/fd [4194304,4194304] 0 2024-03-23T11:47:25.529 INFO:tasks.workunit.client.1.smithi111.stdout:7/100: mknod d14/c16 0 2024-03-23T11:47:25.533 INFO:tasks.workunit.client.1.smithi111.stdout:0/137: chown d5/db/d1f/d19/l24 1827950 1 2024-03-23T11:47:25.533 INFO:tasks.workunit.client.1.smithi111.stdout:5/150: rmdir d1/d3/d23 39 2024-03-23T11:47:25.536 INFO:tasks.workunit.client.0.smithi063.stdout:9/64: rmdir d4 39 2024-03-23T11:47:25.537 INFO:tasks.workunit.client.0.smithi063.stdout:9/65: truncate d4/d11/f12 212516 0 2024-03-23T11:47:25.542 INFO:tasks.workunit.client.0.smithi063.stdout:9/66: rename f0 to d4/f14 0 2024-03-23T11:47:25.544 INFO:tasks.workunit.client.1.smithi111.stdout:5/151: creat d1/d3/d6/d7/f36 x:0 0 0 2024-03-23T11:47:25.544 INFO:tasks.workunit.client.0.smithi063.stdout:9/67: dread - d4/fa zero size 2024-03-23T11:47:25.544 INFO:tasks.workunit.client.1.smithi111.stdout:4/108: link d0/c5 d0/dc/d1f/c26 0 2024-03-23T11:47:25.545 INFO:tasks.workunit.client.1.smithi111.stdout:1/107: write d0/d6/f11 [567593,130568] 0 2024-03-23T11:47:25.546 INFO:tasks.workunit.client.1.smithi111.stdout:9/152: mknod d0/d31/d35/c37 0 2024-03-23T11:47:25.548 INFO:tasks.workunit.client.1.smithi111.stdout:6/190: sync 2024-03-23T11:47:25.548 INFO:tasks.workunit.client.1.smithi111.stdout:2/134: sync 2024-03-23T11:47:25.549 INFO:tasks.workunit.client.1.smithi111.stdout:5/152: chown d1/d3/d23 33382140 1 2024-03-23T11:47:25.553 INFO:tasks.workunit.client.0.smithi063.stdout:4/128: dwrite d5/f1b [0,4194304] 0 2024-03-23T11:47:25.554 INFO:tasks.workunit.client.1.smithi111.stdout:5/153: stat d1/f35 0 2024-03-23T11:47:25.555 INFO:tasks.workunit.client.1.smithi111.stdout:2/135: chown d2/de/d12/c22 1265393365 1 2024-03-23T11:47:25.557 INFO:tasks.workunit.client.1.smithi111.stdout:0/138: dwrite d5/db/f26 [0,4194304] 0 2024-03-23T11:47:25.558 INFO:tasks.workunit.client.0.smithi063.stdout:4/129: mkdir d5/d1e 0 2024-03-23T11:47:25.559 INFO:tasks.workunit.client.1.smithi111.stdout:7/101: getdents d14 0 2024-03-23T11:47:25.560 INFO:tasks.workunit.client.1.smithi111.stdout:6/191: dread d2/f28 [0,4194304] 0 2024-03-23T11:47:25.560 INFO:tasks.workunit.client.0.smithi063.stdout:9/68: truncate f2 5993435 0 2024-03-23T11:47:25.560 INFO:tasks.workunit.client.0.smithi063.stdout:4/130: write d5/da/f19 [70287,56835] 0 2024-03-23T11:47:25.562 INFO:tasks.workunit.client.0.smithi063.stdout:9/69: dread - d4/ff zero size 2024-03-23T11:47:25.563 INFO:tasks.workunit.client.1.smithi111.stdout:9/153: write d0/d4/fe [2254720,74567] 0 2024-03-23T11:47:25.564 INFO:tasks.workunit.client.1.smithi111.stdout:5/154: chown d1/d3/d23/d2c/f34 56 1 2024-03-23T11:47:25.564 INFO:tasks.workunit.client.0.smithi063.stdout:9/70: truncate d4/fc 255800 0 2024-03-23T11:47:25.564 INFO:tasks.workunit.client.0.smithi063.stdout:4/131: mkdir d5/d10/d1f 0 2024-03-23T11:47:25.565 INFO:tasks.workunit.client.1.smithi111.stdout:2/136: write d2/d11/f13 [877886,78192] 0 2024-03-23T11:47:25.567 INFO:tasks.workunit.client.0.smithi063.stdout:9/71: write d4/fa [350978,12257] 0 2024-03-23T11:47:25.567 INFO:tasks.workunit.client.0.smithi063.stdout:4/132: mkdir d5/da/d20 0 2024-03-23T11:47:25.569 INFO:tasks.workunit.client.0.smithi063.stdout:4/133: chown d5/f1c 628 1 2024-03-23T11:47:25.571 INFO:tasks.workunit.client.1.smithi111.stdout:9/154: stat d0/d4/d1c/d36/d28/l32 0 2024-03-23T11:47:25.573 INFO:tasks.workunit.client.0.smithi063.stdout:9/72: link d4/cb d4/c15 0 2024-03-23T11:47:25.575 INFO:tasks.workunit.client.0.smithi063.stdout:4/134: creat d5/d10/d1f/f21 x:0 0 0 2024-03-23T11:47:25.577 INFO:tasks.workunit.client.1.smithi111.stdout:7/102: chown f1 85 1 2024-03-23T11:47:25.577 INFO:tasks.workunit.client.0.smithi063.stdout:9/73: creat d4/f16 x:0 0 0 2024-03-23T11:47:25.578 INFO:tasks.workunit.client.1.smithi111.stdout:7/103: readlink l5 0 2024-03-23T11:47:25.579 INFO:tasks.workunit.client.1.smithi111.stdout:5/155: creat d1/d20/f37 x:0 0 0 2024-03-23T11:47:25.581 INFO:tasks.workunit.client.0.smithi063.stdout:4/135: rename d5/l8 to d5/da/l22 0 2024-03-23T11:47:25.581 INFO:tasks.workunit.client.1.smithi111.stdout:1/108: dwrite d0/f20 [0,4194304] 0 2024-03-23T11:47:25.582 INFO:tasks.workunit.client.1.smithi111.stdout:9/155: fdatasync d0/d4/f1b 0 2024-03-23T11:47:25.583 INFO:tasks.workunit.client.0.smithi063.stdout:9/74: creat d4/d11/f17 x:0 0 0 2024-03-23T11:47:25.583 INFO:tasks.workunit.client.1.smithi111.stdout:3/85: dwrite d0/f9 [0,4194304] 0 2024-03-23T11:47:25.583 INFO:tasks.workunit.client.1.smithi111.stdout:7/104: symlink d14/l17 0 2024-03-23T11:47:25.584 INFO:tasks.workunit.client.0.smithi063.stdout:9/75: readlink d4/d11/l13 0 2024-03-23T11:47:25.586 INFO:tasks.workunit.client.1.smithi111.stdout:5/156: readlink d1/d3/d6/d7/l2f 0 2024-03-23T11:47:25.588 INFO:tasks.workunit.client.1.smithi111.stdout:2/137: creat d2/d7/da/d19/f33 x:0 0 0 2024-03-23T11:47:25.588 INFO:tasks.workunit.client.1.smithi111.stdout:4/109: dwrite d0/d7/d9/f15 [0,4194304] 0 2024-03-23T11:47:25.588 INFO:tasks.workunit.client.0.smithi063.stdout:4/136: creat d5/f23 x:0 0 0 2024-03-23T11:47:25.589 INFO:tasks.workunit.client.0.smithi063.stdout:9/76: symlink d4/d11/l18 0 2024-03-23T11:47:25.590 INFO:tasks.workunit.client.1.smithi111.stdout:6/192: link d2/d1a/f2d d2/d1a/d1c/f42 0 2024-03-23T11:47:25.590 INFO:tasks.workunit.client.1.smithi111.stdout:4/110: stat d0/f1 0 2024-03-23T11:47:25.591 INFO:tasks.workunit.client.0.smithi063.stdout:4/137: fsync d5/da/f19 0 2024-03-23T11:47:25.591 INFO:tasks.workunit.client.1.smithi111.stdout:5/157: dread - d1/d20/f37 zero size 2024-03-23T11:47:25.592 INFO:tasks.workunit.client.1.smithi111.stdout:3/86: rmdir d0 39 2024-03-23T11:47:25.593 INFO:tasks.workunit.client.0.smithi063.stdout:9/77: mkdir d4/d11/d19 0 2024-03-23T11:47:25.595 INFO:tasks.workunit.client.1.smithi111.stdout:1/109: creat d0/d5/dd/f25 x:0 0 0 2024-03-23T11:47:25.596 INFO:tasks.workunit.client.1.smithi111.stdout:9/156: symlink d0/d31/l38 0 2024-03-23T11:47:25.599 INFO:tasks.workunit.client.1.smithi111.stdout:2/138: creat d2/d7/da/d19/f34 x:0 0 0 2024-03-23T11:47:25.602 INFO:tasks.workunit.client.1.smithi111.stdout:5/158: mknod d1/d3/d23/c38 0 2024-03-23T11:47:25.605 INFO:tasks.workunit.client.1.smithi111.stdout:1/110: stat d0/d5/dd/l17 0 2024-03-23T11:47:25.606 INFO:tasks.workunit.client.1.smithi111.stdout:9/157: stat d0/d4/d1c/d36/l8 0 2024-03-23T11:47:25.606 INFO:tasks.workunit.client.1.smithi111.stdout:2/139: read d2/d11/d17/f21 [838172,36778] 0 2024-03-23T11:47:25.609 INFO:tasks.workunit.client.1.smithi111.stdout:3/87: mknod d0/c15 0 2024-03-23T11:47:25.611 INFO:tasks.workunit.client.1.smithi111.stdout:9/158: chown d0/d4/cc 226792837 1 2024-03-23T11:47:25.613 INFO:tasks.workunit.client.0.smithi063.stdout:4/138: dwrite d5/f1c [0,4194304] 0 2024-03-23T11:47:25.614 INFO:tasks.workunit.client.1.smithi111.stdout:6/193: link d2/d35/c3d d2/d1a/d1c/c43 0 2024-03-23T11:47:25.614 INFO:tasks.workunit.client.0.smithi063.stdout:4/139: fsync d5/da/fd 0 2024-03-23T11:47:25.615 INFO:tasks.workunit.client.1.smithi111.stdout:5/159: mkdir d1/d3/d23/d39 0 2024-03-23T11:47:25.617 INFO:tasks.workunit.client.1.smithi111.stdout:4/111: link d0/c5 d0/d7/da/c27 0 2024-03-23T11:47:25.620 INFO:tasks.workunit.client.1.smithi111.stdout:7/105: dwrite f12 [0,4194304] 0 2024-03-23T11:47:25.621 INFO:tasks.workunit.client.1.smithi111.stdout:6/194: creat d2/d35/f44 x:0 0 0 2024-03-23T11:47:25.624 INFO:tasks.workunit.client.1.smithi111.stdout:4/112: stat d0/d7/f16 0 2024-03-23T11:47:25.630 INFO:tasks.workunit.client.1.smithi111.stdout:4/113: write d0/d7/f1b [4767804,31494] 0 2024-03-23T11:47:25.630 INFO:tasks.workunit.client.1.smithi111.stdout:5/160: mkdir d1/d3/d3a 0 2024-03-23T11:47:25.635 INFO:tasks.workunit.client.0.smithi063.stdout:4/140: dwrite d5/f23 [0,4194304] 0 2024-03-23T11:47:25.636 INFO:tasks.workunit.client.1.smithi111.stdout:1/111: dwrite d0/d6/f24 [0,4194304] 0 2024-03-23T11:47:25.638 INFO:tasks.workunit.client.1.smithi111.stdout:5/161: dread - d1/d20/f37 zero size 2024-03-23T11:47:25.638 INFO:tasks.workunit.client.0.smithi063.stdout:4/141: chown d5/da/c18 327364 1 2024-03-23T11:47:25.639 INFO:tasks.workunit.client.1.smithi111.stdout:2/140: link d2/d11/f13 d2/d11/d17/d1f/d30/f35 0 2024-03-23T11:47:25.641 INFO:tasks.workunit.client.1.smithi111.stdout:6/195: unlink d2/d35/c3d 0 2024-03-23T11:47:25.642 INFO:tasks.workunit.client.1.smithi111.stdout:9/159: creat d0/d4/d1c/d36/d28/d34/f39 x:0 0 0 2024-03-23T11:47:25.643 INFO:tasks.workunit.client.1.smithi111.stdout:5/162: dread - d1/d20/f37 zero size 2024-03-23T11:47:25.644 INFO:tasks.workunit.client.1.smithi111.stdout:1/112: mknod d0/c26 0 2024-03-23T11:47:25.647 INFO:tasks.workunit.client.1.smithi111.stdout:6/196: dread - d2/d17/f33 zero size 2024-03-23T11:47:25.648 INFO:tasks.workunit.client.1.smithi111.stdout:4/114: write d0/d7/da/f13 [1316515,11161] 0 2024-03-23T11:47:25.648 INFO:tasks.workunit.client.1.smithi111.stdout:5/163: chown d1/d3/d6/d7 451 1 2024-03-23T11:47:25.650 INFO:tasks.workunit.client.1.smithi111.stdout:7/106: write d14/f15 [4767574,129137] 0 2024-03-23T11:47:25.651 INFO:tasks.workunit.client.0.smithi063.stdout:4/142: rename d5/d10 to d5/da/d20/d24 0 2024-03-23T11:47:25.652 INFO:tasks.workunit.client.0.smithi063.stdout:9/78: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:25.652 INFO:tasks.workunit.client.1.smithi111.stdout:3/88: dwrite d0/f3 [0,4194304] 0 2024-03-23T11:47:25.654 INFO:tasks.workunit.client.1.smithi111.stdout:8/97: sync 2024-03-23T11:47:25.654 INFO:tasks.workunit.client.0.smithi063.stdout:4/143: write d5/da/f19 [974047,83384] 0 2024-03-23T11:47:25.655 INFO:tasks.workunit.client.1.smithi111.stdout:2/141: link c1 d2/d32/c36 0 2024-03-23T11:47:25.656 INFO:tasks.workunit.client.1.smithi111.stdout:6/197: write d2/d1f/f2e [1028445,73874] 0 2024-03-23T11:47:25.657 INFO:tasks.workunit.client.1.smithi111.stdout:9/160: rmdir d0/d4/d1c/d36/d28 39 2024-03-23T11:47:25.659 INFO:tasks.workunit.client.1.smithi111.stdout:6/198: chown d2/d1f 248 1 2024-03-23T11:47:25.661 INFO:tasks.workunit.client.1.smithi111.stdout:1/113: dread d0/fe [0,4194304] 0 2024-03-23T11:47:25.661 INFO:tasks.workunit.client.1.smithi111.stdout:5/164: truncate d1/d3/d23/d2c/f2d 101685 0 2024-03-23T11:47:25.662 INFO:tasks.workunit.client.1.smithi111.stdout:5/165: chown d1/d20 37613478 1 2024-03-23T11:47:25.664 INFO:tasks.workunit.client.1.smithi111.stdout:2/142: creat d2/d11/f37 x:0 0 0 2024-03-23T11:47:25.667 INFO:tasks.workunit.client.1.smithi111.stdout:7/107: write f1 [1714851,59272] 0 2024-03-23T11:47:25.668 INFO:tasks.workunit.client.1.smithi111.stdout:6/199: mkdir d2/d1f/d45 0 2024-03-23T11:47:25.670 INFO:tasks.workunit.client.1.smithi111.stdout:7/108: dread - fe zero size 2024-03-23T11:47:25.673 INFO:tasks.workunit.client.0.smithi063.stdout:4/144: dread d5/f1b [0,4194304] 0 2024-03-23T11:47:25.673 INFO:tasks.workunit.client.1.smithi111.stdout:9/161: mknod d0/d4/d1c/c3a 0 2024-03-23T11:47:25.675 INFO:tasks.workunit.client.1.smithi111.stdout:4/115: link d0/d7/f25 d0/d17/f28 0 2024-03-23T11:47:25.675 INFO:tasks.workunit.client.1.smithi111.stdout:1/114: rename d0/d6/f11 to d0/d6/f27 0 2024-03-23T11:47:25.676 INFO:tasks.workunit.client.1.smithi111.stdout:8/98: link d5/fa d5/f22 0 2024-03-23T11:47:25.677 INFO:tasks.workunit.client.1.smithi111.stdout:8/99: stat d5/d1b 0 2024-03-23T11:47:25.677 INFO:tasks.workunit.client.1.smithi111.stdout:3/89: creat d0/f16 x:0 0 0 2024-03-23T11:47:25.680 INFO:tasks.workunit.client.1.smithi111.stdout:4/116: write d0/d7/f1b [2523482,111138] 0 2024-03-23T11:47:25.681 INFO:tasks.workunit.client.0.smithi063.stdout:4/145: truncate d5/da/fd 7591179 0 2024-03-23T11:47:25.686 INFO:tasks.workunit.client.1.smithi111.stdout:9/162: mkdir d0/d4/d1c/d2d/d3b 0 2024-03-23T11:47:25.687 INFO:tasks.workunit.client.1.smithi111.stdout:5/166: creat d1/d3/d6/f3b x:0 0 0 2024-03-23T11:47:25.692 INFO:tasks.workunit.client.0.smithi063.stdout:4/146: truncate d5/ff 6579884 0 2024-03-23T11:47:25.692 INFO:tasks.workunit.client.1.smithi111.stdout:4/117: truncate d0/dc/f14 4770258 0 2024-03-23T11:47:25.693 INFO:tasks.workunit.client.0.smithi063.stdout:4/147: dread f2 [4194304,4194304] 0 2024-03-23T11:47:25.696 INFO:tasks.workunit.client.0.smithi063.stdout:4/148: mknod d5/da/d20/d24/c25 0 2024-03-23T11:47:25.697 INFO:tasks.workunit.client.0.smithi063.stdout:8/273: sync 2024-03-23T11:47:25.697 INFO:tasks.workunit.client.0.smithi063.stdout:5/77: sync 2024-03-23T11:47:25.700 INFO:tasks.workunit.client.0.smithi063.stdout:8/274: readlink d6/l42 0 2024-03-23T11:47:25.700 INFO:tasks.workunit.client.1.smithi111.stdout:5/167: unlink d1/d3/d6/d7/f13 0 2024-03-23T11:47:25.701 INFO:tasks.workunit.client.1.smithi111.stdout:7/109: dwrite f10 [0,4194304] 0 2024-03-23T11:47:25.704 INFO:tasks.workunit.client.0.smithi063.stdout:8/275: creat d6/dd/df/f5e x:0 0 0 2024-03-23T11:47:25.705 INFO:tasks.workunit.client.1.smithi111.stdout:2/143: dwrite d2/f6 [0,4194304] 0 2024-03-23T11:47:25.707 INFO:tasks.workunit.client.1.smithi111.stdout:6/200: rename d2/d1a/d1c/d2c/c37 to d2/d17/c46 0 2024-03-23T11:47:25.708 INFO:tasks.workunit.client.1.smithi111.stdout:9/163: truncate d0/d4/d1c/d2d/f2e 336398 0 2024-03-23T11:47:25.712 INFO:tasks.workunit.client.1.smithi111.stdout:5/168: symlink d1/d3/d3a/l3c 0 2024-03-23T11:47:25.716 INFO:tasks.workunit.client.1.smithi111.stdout:8/100: dread d5/f19 [4194304,4194304] 0 2024-03-23T11:47:25.717 INFO:tasks.workunit.client.1.smithi111.stdout:6/201: read d2/d1a/f2d [2937429,82811] 0 2024-03-23T11:47:25.720 INFO:tasks.workunit.client.1.smithi111.stdout:1/115: dwrite d0/d5/dd/f10 [0,4194304] 0 2024-03-23T11:47:25.724 INFO:tasks.workunit.client.1.smithi111.stdout:3/90: dwrite d0/d7/f12 [0,4194304] 0 2024-03-23T11:47:25.728 INFO:tasks.workunit.client.0.smithi063.stdout:8/276: dwrite d6/d1f/f55 [0,4194304] 0 2024-03-23T11:47:25.729 INFO:tasks.workunit.client.0.smithi063.stdout:4/149: dwrite d5/ff [4194304,4194304] 0 2024-03-23T11:47:25.729 INFO:tasks.workunit.client.1.smithi111.stdout:4/118: truncate d0/f1 3044758 0 2024-03-23T11:47:25.729 INFO:tasks.workunit.client.1.smithi111.stdout:2/144: truncate d2/d7/da/f2d 600413 0 2024-03-23T11:47:25.729 INFO:tasks.workunit.client.1.smithi111.stdout:5/169: stat d1/d3/d6/d29/f22 0 2024-03-23T11:47:25.731 INFO:tasks.workunit.client.0.smithi063.stdout:8/277: fdatasync d6/d1c/f4e 0 2024-03-23T11:47:25.732 INFO:tasks.workunit.client.1.smithi111.stdout:1/116: stat d0/d5/dd/d1e 0 2024-03-23T11:47:25.735 INFO:tasks.workunit.client.0.smithi063.stdout:4/150: chown d5/da/d20/d24/c25 77116802 1 2024-03-23T11:47:25.735 INFO:tasks.workunit.client.0.smithi063.stdout:8/278: creat d6/d1f/d26/f5f x:0 0 0 2024-03-23T11:47:25.739 INFO:tasks.workunit.client.1.smithi111.stdout:2/145: dread - d2/de/d27/f2f zero size 2024-03-23T11:47:25.739 INFO:tasks.workunit.client.1.smithi111.stdout:8/101: dread d5/f22 [0,4194304] 0 2024-03-23T11:47:25.745 INFO:tasks.workunit.client.1.smithi111.stdout:5/170: write d1/d3/d6/f8 [9428136,41256] 0 2024-03-23T11:47:25.747 INFO:tasks.workunit.client.1.smithi111.stdout:3/91: link d0/cc d0/c17 0 2024-03-23T11:47:25.748 INFO:tasks.workunit.client.1.smithi111.stdout:4/119: write d0/dc/f14 [3596377,36501] 0 2024-03-23T11:47:25.754 INFO:tasks.workunit.client.1.smithi111.stdout:2/146: dread - d2/de/d12/f2c zero size 2024-03-23T11:47:25.756 INFO:tasks.workunit.client.1.smithi111.stdout:3/92: stat d0/d7/f10 0 2024-03-23T11:47:25.758 INFO:tasks.workunit.client.1.smithi111.stdout:5/171: chown d1/d20/f37 1921503 1 2024-03-23T11:47:25.758 INFO:tasks.workunit.client.1.smithi111.stdout:2/147: read - d2/d11/f1b zero size 2024-03-23T11:47:25.759 INFO:tasks.workunit.client.1.smithi111.stdout:4/120: write d0/d7/d9/ff [274498,27158] 0 2024-03-23T11:47:25.760 INFO:tasks.workunit.client.0.smithi063.stdout:4/151: dwrite f2 [0,4194304] 0 2024-03-23T11:47:25.760 INFO:tasks.workunit.client.0.smithi063.stdout:8/279: dread d6/d1f/d26/f35 [0,4194304] 0 2024-03-23T11:47:25.760 INFO:tasks.workunit.client.1.smithi111.stdout:9/164: dread d0/d4/f15 [0,4194304] 0 2024-03-23T11:47:25.763 INFO:tasks.workunit.client.1.smithi111.stdout:6/202: dwrite d2/d35/f3f [0,4194304] 0 2024-03-23T11:47:25.766 INFO:tasks.workunit.client.1.smithi111.stdout:6/203: readlink d2/l9 0 2024-03-23T11:47:25.766 INFO:tasks.workunit.client.1.smithi111.stdout:3/93: creat d0/d7/f18 x:0 0 0 2024-03-23T11:47:25.767 INFO:tasks.workunit.client.1.smithi111.stdout:2/148: truncate d2/f14 4198158 0 2024-03-23T11:47:25.768 INFO:tasks.workunit.client.0.smithi063.stdout:4/152: rename d5/f23 to d5/da/d20/d24/d1f/f26 0 2024-03-23T11:47:25.769 INFO:tasks.workunit.client.1.smithi111.stdout:6/204: read - d2/d35/f44 zero size 2024-03-23T11:47:25.769 INFO:tasks.workunit.client.1.smithi111.stdout:3/94: readlink d0/l2 0 2024-03-23T11:47:25.770 INFO:tasks.workunit.client.0.smithi063.stdout:8/280: rename d6/dd/df/f5e to d6/dd/df/f60 0 2024-03-23T11:47:25.771 INFO:tasks.workunit.client.0.smithi063.stdout:8/281: chown d6/d1c/d32 63 1 2024-03-23T11:47:25.772 INFO:tasks.workunit.client.0.smithi063.stdout:8/282: dread - d6/f43 zero size 2024-03-23T11:47:25.774 INFO:tasks.workunit.client.1.smithi111.stdout:9/165: write d0/d4/d1c/d36/f20 [594588,75680] 0 2024-03-23T11:47:25.775 INFO:tasks.workunit.client.0.smithi063.stdout:8/283: chown d6/dd/c2a 153902 1 2024-03-23T11:47:25.775 INFO:tasks.workunit.client.1.smithi111.stdout:2/149: chown d2/de/d27/f2f 64846528 1 2024-03-23T11:47:25.777 INFO:tasks.workunit.client.0.smithi063.stdout:8/284: creat d6/dd/df/f61 x:0 0 0 2024-03-23T11:47:25.779 INFO:tasks.workunit.client.1.smithi111.stdout:5/172: write d1/d3/d23/d2c/f34 [188946,16748] 0 2024-03-23T11:47:25.780 INFO:tasks.workunit.client.1.smithi111.stdout:9/166: dread - d0/d4/d10/f2c zero size 2024-03-23T11:47:25.780 INFO:tasks.workunit.client.0.smithi063.stdout:8/285: symlink d6/d1f/d54/d5c/l62 0 2024-03-23T11:47:25.782 INFO:tasks.workunit.client.1.smithi111.stdout:2/150: creat d2/d7/f38 x:0 0 0 2024-03-23T11:47:25.784 INFO:tasks.workunit.client.0.smithi063.stdout:8/286: unlink d6/d1f/d26/f3b 0 2024-03-23T11:47:25.785 INFO:tasks.workunit.client.1.smithi111.stdout:1/117: dwrite d0/d5/dd/d1e/f21 [0,4194304] 0 2024-03-23T11:47:25.786 INFO:tasks.workunit.client.1.smithi111.stdout:0/139: sync 2024-03-23T11:47:25.787 INFO:tasks.workunit.client.0.smithi063.stdout:8/287: mkdir d6/d1f/d54/d63 0 2024-03-23T11:47:25.790 INFO:tasks.workunit.client.0.smithi063.stdout:8/288: fsync d6/d1f/d26/f28 0 2024-03-23T11:47:25.791 INFO:tasks.workunit.client.1.smithi111.stdout:5/173: creat d1/d3/d3a/f3d x:0 0 0 2024-03-23T11:47:25.792 INFO:tasks.workunit.client.1.smithi111.stdout:0/140: rename d5/db/d1f/d19/c20 to d5/db/c29 0 2024-03-23T11:47:25.793 INFO:tasks.workunit.client.0.smithi063.stdout:8/289: chown d6/d1c/d32/d44/c4d 0 1 2024-03-23T11:47:25.793 INFO:tasks.workunit.client.0.smithi063.stdout:3/225: sync 2024-03-23T11:47:25.793 INFO:tasks.workunit.client.0.smithi063.stdout:2/86: sync 2024-03-23T11:47:25.793 INFO:tasks.workunit.client.0.smithi063.stdout:7/137: sync 2024-03-23T11:47:25.793 INFO:tasks.workunit.client.1.smithi111.stdout:1/118: chown d0/d6/f24 3649557 1 2024-03-23T11:47:25.795 INFO:tasks.workunit.client.1.smithi111.stdout:8/102: truncate d5/fa 626223 0 2024-03-23T11:47:25.797 INFO:tasks.workunit.client.0.smithi063.stdout:7/138: write d5/d9/d20/f28 [173551,99217] 0 2024-03-23T11:47:25.798 INFO:tasks.workunit.client.0.smithi063.stdout:3/226: write d13/d22/f3d [429472,11797] 0 2024-03-23T11:47:25.800 INFO:tasks.workunit.client.0.smithi063.stdout:2/87: rmdir d2/de 39 2024-03-23T11:47:25.801 INFO:tasks.workunit.client.1.smithi111.stdout:4/121: dwrite d0/dc/fe [0,4194304] 0 2024-03-23T11:47:25.801 INFO:tasks.workunit.client.1.smithi111.stdout:4/122: chown d0/c8 9389 1 2024-03-23T11:47:25.803 INFO:tasks.workunit.client.0.smithi063.stdout:4/153: truncate f2 2081130 0 2024-03-23T11:47:25.803 INFO:tasks.workunit.client.0.smithi063.stdout:2/88: write d2/f19 [818616,118368] 0 2024-03-23T11:47:25.805 INFO:tasks.workunit.client.1.smithi111.stdout:9/167: rename d0/d4/d1c/l33 to d0/d4/d1c/d2d/d3b/l3c 0 2024-03-23T11:47:25.806 INFO:tasks.workunit.client.0.smithi063.stdout:4/154: stat d5/da/d20 0 2024-03-23T11:47:25.807 INFO:tasks.workunit.client.1.smithi111.stdout:9/168: stat d0/d4/f1b 0 2024-03-23T11:47:25.808 INFO:tasks.workunit.client.1.smithi111.stdout:9/169: chown d0/d4/f1b 1417 1 2024-03-23T11:47:25.808 INFO:tasks.workunit.client.0.smithi063.stdout:3/227: mkdir d13/d22/d2d/d3e 0 2024-03-23T11:47:25.810 INFO:tasks.workunit.client.1.smithi111.stdout:2/151: creat d2/de/d27/d2e/f39 x:0 0 0 2024-03-23T11:47:25.811 INFO:tasks.workunit.client.1.smithi111.stdout:2/152: chown d2/de/d27/f2f 1803808 1 2024-03-23T11:47:25.812 INFO:tasks.workunit.client.1.smithi111.stdout:3/95: truncate d0/fd 1239443 0 2024-03-23T11:47:25.813 INFO:tasks.workunit.client.1.smithi111.stdout:3/96: dread - d0/d7/ff zero size 2024-03-23T11:47:25.813 INFO:tasks.workunit.client.1.smithi111.stdout:2/153: fdatasync d2/d7/da/d19/f34 0 2024-03-23T11:47:25.813 INFO:tasks.workunit.client.0.smithi063.stdout:2/89: chown d2/de/f1b 468937 1 2024-03-23T11:47:25.814 INFO:tasks.workunit.client.1.smithi111.stdout:0/141: creat d5/db/d1f/d19/f2a x:0 0 0 2024-03-23T11:47:25.815 INFO:tasks.workunit.client.0.smithi063.stdout:7/139: rename d5/d9/da/l2a to d5/d6/d19/d1f/l2c 0 2024-03-23T11:47:25.818 INFO:tasks.workunit.client.0.smithi063.stdout:2/90: dread - d2/de/f1b zero size 2024-03-23T11:47:25.819 INFO:tasks.workunit.client.0.smithi063.stdout:8/290: dwrite d6/dd/df/f61 [0,4194304] 0 2024-03-23T11:47:25.823 INFO:tasks.workunit.client.0.smithi063.stdout:4/155: fdatasync d5/da/fd 0 2024-03-23T11:47:25.823 INFO:tasks.workunit.client.0.smithi063.stdout:2/91: stat d2/lb 0 2024-03-23T11:47:25.826 INFO:tasks.workunit.client.1.smithi111.stdout:8/103: mknod d5/d1b/c23 0 2024-03-23T11:47:25.827 INFO:tasks.workunit.client.0.smithi063.stdout:7/140: mknod d5/d6/d19/d1f/c2d 0 2024-03-23T11:47:25.828 INFO:tasks.workunit.client.0.smithi063.stdout:8/291: chown d6/c58 57957468 1 2024-03-23T11:47:25.828 INFO:tasks.workunit.client.1.smithi111.stdout:8/104: chown d5/d1b/c1d 25184167 1 2024-03-23T11:47:25.828 INFO:tasks.workunit.client.0.smithi063.stdout:2/92: read - d2/de/f1b zero size 2024-03-23T11:47:25.831 INFO:tasks.workunit.client.0.smithi063.stdout:7/141: write f1 [1855659,3899] 0 2024-03-23T11:47:25.834 INFO:tasks.workunit.client.0.smithi063.stdout:2/93: chown d2/c12 5703 1 2024-03-23T11:47:25.834 INFO:tasks.workunit.client.0.smithi063.stdout:3/228: dwrite ff [0,4194304] 0 2024-03-23T11:47:25.840 INFO:tasks.workunit.client.1.smithi111.stdout:0/142: dwrite d5/f13 [4194304,4194304] 0 2024-03-23T11:47:25.840 INFO:tasks.workunit.client.1.smithi111.stdout:6/205: link d2/d1a/d1c/l25 d2/d2a/l47 0 2024-03-23T11:47:25.841 INFO:tasks.workunit.client.0.smithi063.stdout:2/94: creat d2/d1c/f20 x:0 0 0 2024-03-23T11:47:25.844 INFO:tasks.workunit.client.1.smithi111.stdout:0/143: truncate d5/db/f27 329107 0 2024-03-23T11:47:25.850 INFO:tasks.workunit.client.1.smithi111.stdout:5/174: dread d1/d3/d6/f8 [0,4194304] 0 2024-03-23T11:47:25.854 INFO:tasks.workunit.client.1.smithi111.stdout:4/123: chown d0/c5 47840937 1 2024-03-23T11:47:25.856 INFO:tasks.workunit.client.1.smithi111.stdout:4/124: fdatasync d0/d7/f1b 0 2024-03-23T11:47:25.857 INFO:tasks.workunit.client.1.smithi111.stdout:9/170: truncate d0/d4/f1b 424772 0 2024-03-23T11:47:25.857 INFO:tasks.workunit.client.1.smithi111.stdout:7/110: sync 2024-03-23T11:47:25.860 INFO:tasks.workunit.client.1.smithi111.stdout:9/171: chown d0/d31/l38 30373681 1 2024-03-23T11:47:25.867 INFO:tasks.workunit.client.1.smithi111.stdout:3/97: mkdir d0/d7/d19 0 2024-03-23T11:47:25.867 INFO:tasks.workunit.client.0.smithi063.stdout:3/229: dwrite d13/d22/f3b [0,4194304] 0 2024-03-23T11:47:25.871 INFO:tasks.workunit.client.0.smithi063.stdout:3/230: stat d13/c28 0 2024-03-23T11:47:25.873 INFO:tasks.workunit.client.0.smithi063.stdout:7/142: truncate f1 1611821 0 2024-03-23T11:47:25.875 INFO:tasks.workunit.client.1.smithi111.stdout:2/154: dwrite d2/fc [4194304,4194304] 0 2024-03-23T11:47:25.877 INFO:tasks.workunit.client.1.smithi111.stdout:6/206: creat d2/d1a/d1c/d2c/f48 x:0 0 0 2024-03-23T11:47:25.879 INFO:tasks.workunit.client.0.smithi063.stdout:3/231: mkdir d13/d22/d2d/d3f 0 2024-03-23T11:47:25.881 INFO:tasks.workunit.client.1.smithi111.stdout:0/144: symlink d5/db/l2b 0 2024-03-23T11:47:25.882 INFO:tasks.workunit.client.0.smithi063.stdout:3/232: fsync d13/d15/f3c 0 2024-03-23T11:47:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:25 smithi063 ceph-mon[35189]: pgmap v32: 65 pgs: 65 active+clean; 436 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 1.9 MiB/s rd, 31 MiB/s wr, 226 op/s 2024-03-23T11:47:25.888 INFO:tasks.workunit.client.1.smithi111.stdout:4/125: truncate d0/dc/f1c 984904 0 2024-03-23T11:47:25.890 INFO:tasks.workunit.client.1.smithi111.stdout:7/111: symlink d14/l18 0 2024-03-23T11:47:25.890 INFO:tasks.workunit.client.0.smithi063.stdout:3/233: symlink d13/d22/l40 0 2024-03-23T11:47:25.892 INFO:tasks.workunit.client.0.smithi063.stdout:8/292: dread d6/d1c/f52 [0,4194304] 0 2024-03-23T11:47:25.894 INFO:tasks.workunit.client.0.smithi063.stdout:8/293: fdatasync d6/d1f/d26/f49 0 2024-03-23T11:47:25.894 INFO:tasks.workunit.client.0.smithi063.stdout:3/234: write d13/f30 [704684,46199] 0 2024-03-23T11:47:25.896 INFO:tasks.workunit.client.0.smithi063.stdout:8/294: fdatasync d6/d1f/d26/f5f 0 2024-03-23T11:47:25.897 INFO:tasks.workunit.client.0.smithi063.stdout:3/235: write d13/f30 [644259,75481] 0 2024-03-23T11:47:25.898 INFO:tasks.workunit.client.0.smithi063.stdout:8/295: readlink d6/d1c/d32/l3a 0 2024-03-23T11:47:25.899 INFO:tasks.workunit.client.0.smithi063.stdout:7/143: dread f4 [0,4194304] 0 2024-03-23T11:47:25.901 INFO:tasks.workunit.client.1.smithi111.stdout:2/155: dwrite d2/de/d12/f2c [0,4194304] 0 2024-03-23T11:47:25.903 INFO:tasks.workunit.client.1.smithi111.stdout:6/207: dwrite d2/d1f/f2e [0,4194304] 0 2024-03-23T11:47:25.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:25 smithi111 ceph-mon[44005]: pgmap v32: 65 pgs: 65 active+clean; 436 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 1.9 MiB/s rd, 31 MiB/s wr, 226 op/s 2024-03-23T11:47:25.907 INFO:tasks.workunit.client.1.smithi111.stdout:9/172: chown d0/d4/f1b 4817708 1 2024-03-23T11:47:25.908 INFO:tasks.workunit.client.0.smithi063.stdout:8/296: link d6/d1f/d26/c48 d6/d1f/d26/d5d/c64 0 2024-03-23T11:47:25.909 INFO:tasks.workunit.client.1.smithi111.stdout:7/112: creat d14/f19 x:0 0 0 2024-03-23T11:47:25.915 INFO:tasks.workunit.client.0.smithi063.stdout:8/297: write d6/dd/df/f2c [4425642,126330] 0 2024-03-23T11:47:25.915 INFO:tasks.workunit.client.0.smithi063.stdout:3/236: dread f9 [0,4194304] 0 2024-03-23T11:47:25.917 INFO:tasks.workunit.client.0.smithi063.stdout:2/95: dread d2/de/ff [0,4194304] 0 2024-03-23T11:47:25.918 INFO:tasks.workunit.client.0.smithi063.stdout:2/96: stat d2/f17 0 2024-03-23T11:47:25.922 INFO:tasks.workunit.client.1.smithi111.stdout:5/175: rmdir d1/d3/d23/d39 0 2024-03-23T11:47:25.924 INFO:tasks.workunit.client.0.smithi063.stdout:3/237: fsync fd 0 2024-03-23T11:47:25.925 INFO:tasks.workunit.client.0.smithi063.stdout:7/144: write d5/f12 [454293,770] 0 2024-03-23T11:47:25.929 INFO:tasks.workunit.client.0.smithi063.stdout:7/145: write d5/d6/fe [1644375,63633] 0 2024-03-23T11:47:25.930 INFO:tasks.workunit.client.1.smithi111.stdout:8/105: dread d5/fa [0,4194304] 0 2024-03-23T11:47:25.934 INFO:tasks.workunit.client.0.smithi063.stdout:3/238: symlink d13/d22/l41 0 2024-03-23T11:47:25.934 INFO:tasks.workunit.client.0.smithi063.stdout:7/146: fsync d5/d6/f25 0 2024-03-23T11:47:25.937 INFO:tasks.workunit.client.0.smithi063.stdout:2/97: rename d2/fa to d2/f21 0 2024-03-23T11:47:25.941 INFO:tasks.workunit.client.0.smithi063.stdout:8/298: dwrite d6/d1c/f4e [0,4194304] 0 2024-03-23T11:47:25.942 INFO:tasks.workunit.client.1.smithi111.stdout:1/119: truncate d0/d5/dd/d1e/f21 206491 0 2024-03-23T11:47:25.943 INFO:tasks.workunit.client.1.smithi111.stdout:1/120: fsync d0/d6/fc 0 2024-03-23T11:47:25.945 INFO:tasks.workunit.client.0.smithi063.stdout:8/299: fdatasync d6/dd/df/f4b 0 2024-03-23T11:47:25.946 INFO:tasks.workunit.client.1.smithi111.stdout:6/208: mkdir d2/d1f/d45/d49 0 2024-03-23T11:47:25.948 INFO:tasks.workunit.client.1.smithi111.stdout:9/173: mknod d0/d4/d10/d29/c3d 0 2024-03-23T11:47:25.950 INFO:tasks.workunit.client.1.smithi111.stdout:8/106: fdatasync d5/f6 0 2024-03-23T11:47:25.951 INFO:tasks.workunit.client.0.smithi063.stdout:8/300: creat d6/d1f/d26/d15/f65 x:0 0 0 2024-03-23T11:47:25.952 INFO:tasks.workunit.client.1.smithi111.stdout:9/174: stat d0/d4/fe 0 2024-03-23T11:47:25.954 INFO:tasks.workunit.client.0.smithi063.stdout:7/147: creat d5/d9/f2e x:0 0 0 2024-03-23T11:47:25.955 INFO:tasks.workunit.client.1.smithi111.stdout:5/176: dwrite d1/d3/d6/d29/f21 [8388608,4194304] 0 2024-03-23T11:47:25.956 INFO:tasks.workunit.client.1.smithi111.stdout:8/107: symlink d5/l24 0 2024-03-23T11:47:25.957 INFO:tasks.workunit.client.0.smithi063.stdout:7/148: stat f4 0 2024-03-23T11:47:25.957 INFO:tasks.workunit.client.0.smithi063.stdout:8/301: fdatasync d6/d1c/f4f 0 2024-03-23T11:47:25.958 INFO:tasks.workunit.client.1.smithi111.stdout:9/175: mknod d0/d4/c3e 0 2024-03-23T11:47:25.961 INFO:tasks.workunit.client.1.smithi111.stdout:2/156: rmdir d2 39 2024-03-23T11:47:25.962 INFO:tasks.workunit.client.1.smithi111.stdout:7/113: rmdir d14 39 2024-03-23T11:47:25.964 INFO:tasks.workunit.client.0.smithi063.stdout:3/239: dwrite f8 [4194304,4194304] 0 2024-03-23T11:47:25.968 INFO:tasks.workunit.client.1.smithi111.stdout:1/121: dwrite d0/d5/dd/f10 [4194304,4194304] 0 2024-03-23T11:47:25.970 INFO:tasks.workunit.client.1.smithi111.stdout:2/157: mknod d2/d11/d17/d1f/c3a 0 2024-03-23T11:47:25.971 INFO:tasks.workunit.client.1.smithi111.stdout:1/122: truncate d0/f1a 1107794 0 2024-03-23T11:47:25.972 INFO:tasks.workunit.client.0.smithi063.stdout:2/98: dwrite d2/d1c/f1f [0,4194304] 0 2024-03-23T11:47:25.975 INFO:tasks.workunit.client.1.smithi111.stdout:0/145: truncate d5/db/f27 77000 0 2024-03-23T11:47:25.978 INFO:tasks.workunit.client.1.smithi111.stdout:2/158: creat d2/d7/da/f3b x:0 0 0 2024-03-23T11:47:25.980 INFO:tasks.workunit.client.0.smithi063.stdout:2/99: truncate d2/f19 1080983 0 2024-03-23T11:47:25.981 INFO:tasks.workunit.client.0.smithi063.stdout:7/149: dread d5/d9/d20/f28 [0,4194304] 0 2024-03-23T11:47:25.983 INFO:tasks.workunit.client.1.smithi111.stdout:9/176: dwrite d0/d4/d10/f2c [0,4194304] 0 2024-03-23T11:47:25.984 INFO:tasks.workunit.client.0.smithi063.stdout:3/240: rmdir d13/d22/d2d 39 2024-03-23T11:47:25.984 INFO:tasks.workunit.client.0.smithi063.stdout:8/302: rename d6/d1c/d32/d44 to d6/d1f/d54/d5c/d66 0 2024-03-23T11:47:25.986 INFO:tasks.workunit.client.1.smithi111.stdout:9/177: write d0/f21 [1270063,13126] 0 2024-03-23T11:47:25.986 INFO:tasks.workunit.client.0.smithi063.stdout:8/303: read d6/dd/df/f61 [550253,33309] 0 2024-03-23T11:47:25.986 INFO:tasks.workunit.client.0.smithi063.stdout:3/241: write ff [1636808,76129] 0 2024-03-23T11:47:25.988 INFO:tasks.workunit.client.1.smithi111.stdout:8/108: rmdir d5 39 2024-03-23T11:47:25.988 INFO:tasks.workunit.client.1.smithi111.stdout:0/146: creat d5/f2c x:0 0 0 2024-03-23T11:47:25.989 INFO:tasks.workunit.client.0.smithi063.stdout:2/100: write d2/f18 [4293975,10273] 0 2024-03-23T11:47:25.991 INFO:tasks.workunit.client.1.smithi111.stdout:9/178: truncate d0/f13 828305 0 2024-03-23T11:47:25.993 INFO:tasks.workunit.client.1.smithi111.stdout:2/159: rmdir d2/d11/d17 39 2024-03-23T11:47:25.994 INFO:tasks.workunit.client.1.smithi111.stdout:2/160: stat d2/d11/f1b 0 2024-03-23T11:47:25.998 INFO:tasks.workunit.client.1.smithi111.stdout:2/161: write d2/d7/da/f3b [472888,41146] 0 2024-03-23T11:47:25.999 INFO:tasks.workunit.client.0.smithi063.stdout:8/304: fdatasync d6/d1f/f55 0 2024-03-23T11:47:25.999 INFO:tasks.workunit.client.0.smithi063.stdout:7/150: chown d5/d6/d19/c29 37704 1 2024-03-23T11:47:26.001 INFO:tasks.workunit.client.1.smithi111.stdout:6/209: dwrite d2/d1a/d1c/f42 [0,4194304] 0 2024-03-23T11:47:26.001 INFO:tasks.workunit.client.1.smithi111.stdout:2/162: dread - d2/de/f18 zero size 2024-03-23T11:47:26.002 INFO:tasks.workunit.client.1.smithi111.stdout:7/114: getdents d14 0 2024-03-23T11:47:26.002 INFO:tasks.workunit.client.0.smithi063.stdout:7/151: dread - d5/d6/f27 zero size 2024-03-23T11:47:26.002 INFO:tasks.workunit.client.0.smithi063.stdout:2/101: mkdir d2/de/d22 0 2024-03-23T11:47:26.004 INFO:tasks.workunit.client.0.smithi063.stdout:2/102: chown d2/de/ff 10065 1 2024-03-23T11:47:26.005 INFO:tasks.workunit.client.1.smithi111.stdout:0/147: creat d5/f2d x:0 0 0 2024-03-23T11:47:26.006 INFO:tasks.workunit.client.1.smithi111.stdout:1/123: getdents d0/d5/dd/d1e 0 2024-03-23T11:47:26.006 INFO:tasks.workunit.client.1.smithi111.stdout:9/179: readlink d0/d4/d1c/d36/d28/l32 0 2024-03-23T11:47:26.007 INFO:tasks.workunit.client.1.smithi111.stdout:8/109: mknod d5/d1b/c25 0 2024-03-23T11:47:26.008 INFO:tasks.workunit.client.1.smithi111.stdout:7/115: fsync d14/f19 0 2024-03-23T11:47:26.009 INFO:tasks.workunit.client.1.smithi111.stdout:9/180: fdatasync d0/d4/fe 0 2024-03-23T11:47:26.009 INFO:tasks.workunit.client.0.smithi063.stdout:7/152: unlink f4 0 2024-03-23T11:47:26.010 INFO:tasks.workunit.client.0.smithi063.stdout:6/123: sync 2024-03-23T11:47:26.011 INFO:tasks.workunit.client.1.smithi111.stdout:2/163: write d2/d7/da/d19/f34 [788308,83337] 0 2024-03-23T11:47:26.011 INFO:tasks.workunit.client.1.smithi111.stdout:9/181: chown d0/d4/cc 0 1 2024-03-23T11:47:26.013 INFO:tasks.workunit.client.0.smithi063.stdout:7/153: dread - d5/d6/f27 zero size 2024-03-23T11:47:26.014 INFO:tasks.workunit.client.0.smithi063.stdout:3/242: dwrite d13/d22/f3d [0,4194304] 0 2024-03-23T11:47:26.015 INFO:tasks.workunit.client.0.smithi063.stdout:2/103: mkdir d2/d23 0 2024-03-23T11:47:26.017 INFO:tasks.workunit.client.1.smithi111.stdout:1/124: mknod d0/c28 0 2024-03-23T11:47:26.017 INFO:tasks.workunit.client.1.smithi111.stdout:6/210: creat d2/d17/f4a x:0 0 0 2024-03-23T11:47:26.017 INFO:tasks.workunit.client.1.smithi111.stdout:8/110: unlink d5/d1b/c1d 0 2024-03-23T11:47:26.019 INFO:tasks.workunit.client.1.smithi111.stdout:6/211: chown d2/fd 28057 1 2024-03-23T11:47:26.020 INFO:tasks.workunit.client.1.smithi111.stdout:1/125: write d0/d6/f24 [4398641,37352] 0 2024-03-23T11:47:26.021 INFO:tasks.workunit.client.1.smithi111.stdout:0/148: mknod d5/db/d1f/d19/c2e 0 2024-03-23T11:47:26.023 INFO:tasks.workunit.client.1.smithi111.stdout:1/126: chown d0/d6/f24 1 1 2024-03-23T11:47:26.025 INFO:tasks.workunit.client.1.smithi111.stdout:1/127: read - d0/d5/dd/f25 zero size 2024-03-23T11:47:26.026 INFO:tasks.workunit.client.0.smithi063.stdout:3/243: creat d13/d22/d2d/d3f/f42 x:0 0 0 2024-03-23T11:47:26.027 INFO:tasks.workunit.client.1.smithi111.stdout:2/164: unlink d2/de/d12/c22 0 2024-03-23T11:47:26.029 INFO:tasks.workunit.client.1.smithi111.stdout:9/182: dwrite d0/f13 [0,4194304] 0 2024-03-23T11:47:26.030 INFO:tasks.workunit.client.0.smithi063.stdout:2/104: symlink d2/d23/l24 0 2024-03-23T11:47:26.031 INFO:tasks.workunit.client.1.smithi111.stdout:2/165: write d2/d7/da/d19/f34 [628048,25454] 0 2024-03-23T11:47:26.032 INFO:tasks.workunit.client.1.smithi111.stdout:9/183: stat d0/ca 0 2024-03-23T11:47:26.037 INFO:tasks.workunit.client.1.smithi111.stdout:0/149: symlink d5/db/d1f/l2f 0 2024-03-23T11:47:26.038 INFO:tasks.workunit.client.0.smithi063.stdout:6/124: dwrite d1/f5 [0,4194304] 0 2024-03-23T11:47:26.038 INFO:tasks.workunit.client.1.smithi111.stdout:6/212: dwrite d2/d1f/f2e [0,4194304] 0 2024-03-23T11:47:26.041 INFO:tasks.workunit.client.0.smithi063.stdout:6/125: write d1/f7 [1991227,48885] 0 2024-03-23T11:47:26.041 INFO:tasks.workunit.client.0.smithi063.stdout:3/244: dread fc [4194304,4194304] 0 2024-03-23T11:47:26.042 INFO:tasks.workunit.client.1.smithi111.stdout:1/128: symlink d0/d5/l29 0 2024-03-23T11:47:26.046 INFO:tasks.workunit.client.0.smithi063.stdout:6/126: creat d1/dc/f24 x:0 0 0 2024-03-23T11:47:26.048 INFO:tasks.workunit.client.0.smithi063.stdout:2/105: dread f1 [0,4194304] 0 2024-03-23T11:47:26.053 INFO:tasks.workunit.client.0.smithi063.stdout:2/106: unlink d2/f19 0 2024-03-23T11:47:26.055 INFO:tasks.workunit.client.1.smithi111.stdout:9/184: dwrite d0/d4/d10/f2c [0,4194304] 0 2024-03-23T11:47:26.057 INFO:tasks.workunit.client.0.smithi063.stdout:3/245: getdents d13/d22/d2d/d3f 0 2024-03-23T11:47:26.058 INFO:tasks.workunit.client.1.smithi111.stdout:0/150: link d5/f2d d5/f30 0 2024-03-23T11:47:26.060 INFO:tasks.workunit.client.1.smithi111.stdout:6/213: symlink d2/l4b 0 2024-03-23T11:47:26.060 INFO:tasks.workunit.client.1.smithi111.stdout:3/98: sync 2024-03-23T11:47:26.060 INFO:tasks.workunit.client.1.smithi111.stdout:4/126: sync 2024-03-23T11:47:26.063 INFO:tasks.workunit.client.0.smithi063.stdout:3/246: readlink d13/d22/d2d/l2b 0 2024-03-23T11:47:26.064 INFO:tasks.workunit.client.1.smithi111.stdout:6/214: write d2/d35/f44 [596084,57859] 0 2024-03-23T11:47:26.071 INFO:tasks.workunit.client.1.smithi111.stdout:1/129: dwrite d0/d5/f23 [0,4194304] 0 2024-03-23T11:47:26.071 INFO:tasks.workunit.client.1.smithi111.stdout:7/116: truncate fb 6458721 0 2024-03-23T11:47:26.081 INFO:tasks.workunit.client.0.smithi063.stdout:3/247: dwrite f5 [4194304,4194304] 0 2024-03-23T11:47:26.084 INFO:tasks.workunit.client.1.smithi111.stdout:1/130: symlink d0/l2a 0 2024-03-23T11:47:26.084 INFO:tasks.workunit.client.1.smithi111.stdout:8/111: truncate d5/f11 3414777 0 2024-03-23T11:47:26.086 INFO:tasks.workunit.client.0.smithi063.stdout:3/248: dread - d13/d22/d2d/d3f/f42 zero size 2024-03-23T11:47:26.086 INFO:tasks.workunit.client.1.smithi111.stdout:7/117: write fd [1073496,83562] 0 2024-03-23T11:47:26.088 INFO:tasks.workunit.client.0.smithi063.stdout:2/107: rmdir d2 39 2024-03-23T11:47:26.090 INFO:tasks.workunit.client.0.smithi063.stdout:3/249: creat d13/d15/f43 x:0 0 0 2024-03-23T11:47:26.091 INFO:tasks.workunit.client.1.smithi111.stdout:6/215: creat d2/d1a/d1c/d2c/f4c x:0 0 0 2024-03-23T11:47:26.096 INFO:tasks.workunit.client.0.smithi063.stdout:3/250: creat d13/d22/d2d/f44 x:0 0 0 2024-03-23T11:47:26.098 INFO:tasks.workunit.client.1.smithi111.stdout:6/216: fdatasync d2/d35/f3f 0 2024-03-23T11:47:26.101 INFO:tasks.workunit.client.0.smithi063.stdout:3/251: creat d13/d22/d2d/f45 x:0 0 0 2024-03-23T11:47:26.103 INFO:tasks.workunit.client.1.smithi111.stdout:8/112: truncate f1 4046104 0 2024-03-23T11:47:26.104 INFO:tasks.workunit.client.1.smithi111.stdout:0/151: dwrite d5/db/f16 [0,4194304] 0 2024-03-23T11:47:26.105 INFO:tasks.workunit.client.1.smithi111.stdout:4/127: dwrite d0/dc/fe [0,4194304] 0 2024-03-23T11:47:26.108 INFO:tasks.workunit.client.1.smithi111.stdout:3/99: dread d0/d7/f12 [0,4194304] 0 2024-03-23T11:47:26.111 INFO:tasks.workunit.client.0.smithi063.stdout:2/108: dwrite d2/d1c/f1f [0,4194304] 0 2024-03-23T11:47:26.114 INFO:tasks.workunit.client.0.smithi063.stdout:2/109: write d2/de/f15 [489804,25983] 0 2024-03-23T11:47:26.118 INFO:tasks.workunit.client.0.smithi063.stdout:3/252: dread fd [4194304,4194304] 0 2024-03-23T11:47:26.120 INFO:tasks.workunit.client.1.smithi111.stdout:4/128: write d0/d7/f1b [2075720,130019] 0 2024-03-23T11:47:26.120 INFO:tasks.workunit.client.1.smithi111.stdout:3/100: write d0/d7/f18 [91656,86718] 0 2024-03-23T11:47:26.122 INFO:tasks.workunit.client.1.smithi111.stdout:1/131: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:26.125 INFO:tasks.workunit.client.0.smithi063.stdout:2/110: getdents d2/de 0 2024-03-23T11:47:26.129 INFO:tasks.workunit.client.1.smithi111.stdout:7/118: truncate f11 712641 0 2024-03-23T11:47:26.130 INFO:tasks.workunit.client.1.smithi111.stdout:9/185: dwrite d0/d4/f1b [0,4194304] 0 2024-03-23T11:47:26.133 INFO:tasks.workunit.client.1.smithi111.stdout:1/132: mkdir d0/d2b 0 2024-03-23T11:47:26.134 INFO:tasks.workunit.client.1.smithi111.stdout:6/217: dwrite d2/f8 [0,4194304] 0 2024-03-23T11:47:26.135 INFO:tasks.workunit.client.0.smithi063.stdout:2/111: creat d2/f25 x:0 0 0 2024-03-23T11:47:26.136 INFO:tasks.workunit.client.1.smithi111.stdout:4/129: mkdir d0/d17/d29 0 2024-03-23T11:47:26.137 INFO:tasks.workunit.client.1.smithi111.stdout:1/133: symlink d0/l2c 0 2024-03-23T11:47:26.139 INFO:tasks.workunit.client.1.smithi111.stdout:9/186: creat d0/d4/d1c/d36/d28/f3f x:0 0 0 2024-03-23T11:47:26.142 INFO:tasks.workunit.client.0.smithi063.stdout:2/112: symlink d2/de/l26 0 2024-03-23T11:47:26.143 INFO:tasks.workunit.client.0.smithi063.stdout:3/253: dwrite d13/d15/f3c [0,4194304] 0 2024-03-23T11:47:26.144 INFO:tasks.workunit.client.1.smithi111.stdout:9/187: fsync d0/d4/d1c/d36/d28/d34/f39 0 2024-03-23T11:47:26.144 INFO:tasks.workunit.client.1.smithi111.stdout:6/218: creat d2/d17/f4d x:0 0 0 2024-03-23T11:47:26.145 INFO:tasks.workunit.client.1.smithi111.stdout:5/177: sync 2024-03-23T11:47:26.146 INFO:tasks.workunit.client.1.smithi111.stdout:3/101: dread d0/f9 [0,4194304] 0 2024-03-23T11:47:26.146 INFO:tasks.workunit.client.1.smithi111.stdout:0/152: write d5/f30 [965452,33033] 0 2024-03-23T11:47:26.147 INFO:tasks.workunit.client.1.smithi111.stdout:8/113: write f1 [4796151,112475] 0 2024-03-23T11:47:26.148 INFO:tasks.workunit.client.1.smithi111.stdout:6/219: fsync d2/d1a/d1c/d2c/f4c 0 2024-03-23T11:47:26.150 INFO:tasks.workunit.client.0.smithi063.stdout:3/254: symlink d13/d22/d2d/l46 0 2024-03-23T11:47:26.151 INFO:tasks.workunit.client.1.smithi111.stdout:5/178: write d1/d3/d6/d7/f24 [414902,104782] 0 2024-03-23T11:47:26.152 INFO:tasks.workunit.client.1.smithi111.stdout:1/134: symlink d0/d2b/l2d 0 2024-03-23T11:47:26.152 INFO:tasks.workunit.client.1.smithi111.stdout:6/220: write d2/d35/f44 [952888,18996] 0 2024-03-23T11:47:26.160 INFO:tasks.workunit.client.1.smithi111.stdout:0/153: rename d5/db/d1f/d19/c2e to d5/db/d1f/c31 0 2024-03-23T11:47:26.166 INFO:tasks.workunit.client.1.smithi111.stdout:2/166: sync 2024-03-23T11:47:26.166 INFO:tasks.workunit.client.1.smithi111.stdout:3/102: mkdir d0/d1a 0 2024-03-23T11:47:26.167 INFO:tasks.workunit.client.0.smithi063.stdout:3/255: rename d13/d22/d2d/d3f/f42 to d13/f47 0 2024-03-23T11:47:26.167 INFO:tasks.workunit.client.0.smithi063.stdout:2/113: dread d2/de/ff [0,4194304] 0 2024-03-23T11:47:26.168 INFO:tasks.workunit.client.1.smithi111.stdout:6/221: rmdir d2/d35 39 2024-03-23T11:47:26.168 INFO:tasks.workunit.client.1.smithi111.stdout:3/103: stat d0/d7/c14 0 2024-03-23T11:47:26.169 INFO:tasks.workunit.client.1.smithi111.stdout:2/167: write d2/de/d12/f29 [90499,86053] 0 2024-03-23T11:47:26.169 INFO:tasks.workunit.client.1.smithi111.stdout:5/179: creat d1/d3/d6/d7/f3e x:0 0 0 2024-03-23T11:47:26.169 INFO:tasks.workunit.client.1.smithi111.stdout:3/104: fsync d0/f16 0 2024-03-23T11:47:26.169 INFO:tasks.workunit.client.0.smithi063.stdout:2/114: chown d2/d23/l24 117 1 2024-03-23T11:47:26.172 INFO:tasks.workunit.client.1.smithi111.stdout:1/135: creat d0/d2b/f2e x:0 0 0 2024-03-23T11:47:26.177 INFO:tasks.workunit.client.1.smithi111.stdout:9/188: getdents d0/d4/d1c/d2d 0 2024-03-23T11:47:26.178 INFO:tasks.workunit.client.1.smithi111.stdout:8/114: getdents d5/d1b 0 2024-03-23T11:47:26.178 INFO:tasks.workunit.client.1.smithi111.stdout:2/168: symlink d2/de/d27/d2e/l3c 0 2024-03-23T11:47:26.179 INFO:tasks.workunit.client.1.smithi111.stdout:9/189: read d0/f13 [3967568,9381] 0 2024-03-23T11:47:26.181 INFO:tasks.workunit.client.1.smithi111.stdout:1/136: creat d0/d5/f2f x:0 0 0 2024-03-23T11:47:26.183 INFO:tasks.workunit.client.0.smithi063.stdout:2/115: dread d2/f17 [4194304,4194304] 0 2024-03-23T11:47:26.184 INFO:tasks.workunit.client.1.smithi111.stdout:8/115: fsync d5/f22 0 2024-03-23T11:47:26.185 INFO:tasks.workunit.client.1.smithi111.stdout:9/190: mknod d0/d31/c40 0 2024-03-23T11:47:26.186 INFO:tasks.workunit.client.1.smithi111.stdout:2/169: mkdir d2/d3d 0 2024-03-23T11:47:26.187 INFO:tasks.workunit.client.0.smithi063.stdout:2/116: mkdir d2/d27 0 2024-03-23T11:47:26.187 INFO:tasks.workunit.client.1.smithi111.stdout:1/137: symlink d0/l30 0 2024-03-23T11:47:26.188 INFO:tasks.workunit.client.1.smithi111.stdout:0/154: dwrite d5/f30 [0,4194304] 0 2024-03-23T11:47:26.189 INFO:tasks.workunit.client.0.smithi063.stdout:2/117: fsync d2/f18 0 2024-03-23T11:47:26.191 INFO:tasks.workunit.client.0.smithi063.stdout:3/256: dwrite d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:26.192 INFO:tasks.workunit.client.0.smithi063.stdout:2/118: chown d2/de/l1a 8 1 2024-03-23T11:47:26.193 INFO:tasks.workunit.client.1.smithi111.stdout:9/191: mkdir d0/d31/d41 0 2024-03-23T11:47:26.194 INFO:tasks.workunit.client.1.smithi111.stdout:2/170: mkdir d2/d11/d17/d3e 0 2024-03-23T11:47:26.196 INFO:tasks.workunit.client.0.smithi063.stdout:3/257: mkdir d13/d15/d48 0 2024-03-23T11:47:26.196 INFO:tasks.workunit.client.1.smithi111.stdout:6/222: dwrite d2/d1a/f22 [0,4194304] 0 2024-03-23T11:47:26.198 INFO:tasks.workunit.client.1.smithi111.stdout:8/116: truncate d5/fe 147467 0 2024-03-23T11:47:26.199 INFO:tasks.workunit.client.0.smithi063.stdout:0/107: sync 2024-03-23T11:47:26.199 INFO:tasks.workunit.client.0.smithi063.stdout:1/147: sync 2024-03-23T11:47:26.200 INFO:tasks.workunit.client.1.smithi111.stdout:0/155: mkdir d5/db/d1f/d32 0 2024-03-23T11:47:26.200 INFO:tasks.workunit.client.0.smithi063.stdout:0/108: chown d2/d4 525688369 1 2024-03-23T11:47:26.202 INFO:tasks.workunit.client.0.smithi063.stdout:3/258: mknod d13/d22/c49 0 2024-03-23T11:47:26.202 INFO:tasks.workunit.client.1.smithi111.stdout:3/105: dwrite d0/d7/f12 [0,4194304] 0 2024-03-23T11:47:26.202 INFO:tasks.workunit.client.1.smithi111.stdout:0/156: chown d5/f2c 237083 1 2024-03-23T11:47:26.204 INFO:tasks.workunit.client.1.smithi111.stdout:6/223: dread d2/f28 [0,4194304] 0 2024-03-23T11:47:26.204 INFO:tasks.workunit.client.0.smithi063.stdout:0/109: rmdir d2/d4/dd 39 2024-03-23T11:47:26.207 INFO:tasks.workunit.client.1.smithi111.stdout:2/171: mkdir d2/de/d27/d2e/d3f 0 2024-03-23T11:47:26.208 INFO:tasks.workunit.client.1.smithi111.stdout:0/157: fsync d5/fc 0 2024-03-23T11:47:26.209 INFO:tasks.workunit.client.1.smithi111.stdout:0/158: chown d5/db/l11 2511344 1 2024-03-23T11:47:26.210 INFO:tasks.workunit.client.0.smithi063.stdout:2/119: dread f1 [0,4194304] 0 2024-03-23T11:47:26.211 INFO:tasks.workunit.client.1.smithi111.stdout:6/224: creat d2/d1f/d45/f4e x:0 0 0 2024-03-23T11:47:26.213 INFO:tasks.workunit.client.1.smithi111.stdout:0/159: symlink d5/db/l33 0 2024-03-23T11:47:26.215 INFO:tasks.workunit.client.1.smithi111.stdout:1/138: dread d0/d5/dd/f10 [0,4194304] 0 2024-03-23T11:47:26.217 INFO:tasks.workunit.client.1.smithi111.stdout:4/130: truncate d0/d7/d9/f15 1091554 0 2024-03-23T11:47:26.221 INFO:tasks.workunit.client.1.smithi111.stdout:1/139: write d0/d5/f16 [4164657,93137] 0 2024-03-23T11:47:26.230 INFO:tasks.workunit.client.1.smithi111.stdout:6/225: symlink d2/d35/l4f 0 2024-03-23T11:47:26.231 INFO:tasks.workunit.client.1.smithi111.stdout:8/117: dwrite d5/f1f [0,4194304] 0 2024-03-23T11:47:26.234 INFO:tasks.workunit.client.0.smithi063.stdout:3/259: getdents d13/d22 0 2024-03-23T11:47:26.237 INFO:tasks.workunit.client.0.smithi063.stdout:2/120: dwrite d2/f18 [4194304,4194304] 0 2024-03-23T11:47:26.239 INFO:tasks.workunit.client.1.smithi111.stdout:3/106: dread d0/fd [0,4194304] 0 2024-03-23T11:47:26.242 INFO:tasks.workunit.client.1.smithi111.stdout:8/118: chown d5/l15 0 1 2024-03-23T11:47:26.244 INFO:tasks.workunit.client.1.smithi111.stdout:8/119: readlink l4 0 2024-03-23T11:47:26.247 INFO:tasks.workunit.client.0.smithi063.stdout:0/110: dwrite d2/d4/dd/d1f/f20 [0,4194304] 0 2024-03-23T11:47:26.247 INFO:tasks.workunit.client.0.smithi063.stdout:2/121: link d2/f18 d2/d23/f28 0 2024-03-23T11:47:26.249 INFO:tasks.workunit.client.0.smithi063.stdout:0/111: mknod d2/d4/c26 0 2024-03-23T11:47:26.250 INFO:tasks.workunit.client.1.smithi111.stdout:8/120: fdatasync d5/fa 0 2024-03-23T11:47:26.251 INFO:tasks.workunit.client.0.smithi063.stdout:2/122: write d2/d23/f28 [6732571,39382] 0 2024-03-23T11:47:26.253 INFO:tasks.workunit.client.0.smithi063.stdout:2/123: write d2/f18 [8845061,1348] 0 2024-03-23T11:47:26.254 INFO:tasks.workunit.client.1.smithi111.stdout:1/140: dwrite d0/d6/f24 [0,4194304] 0 2024-03-23T11:47:26.255 INFO:tasks.workunit.client.0.smithi063.stdout:0/112: creat d2/f27 x:0 0 0 2024-03-23T11:47:26.259 INFO:tasks.workunit.client.1.smithi111.stdout:8/121: mkdir d5/d26 0 2024-03-23T11:47:26.259 INFO:tasks.workunit.client.0.smithi063.stdout:2/124: creat d2/f29 x:0 0 0 2024-03-23T11:47:26.265 INFO:tasks.workunit.client.0.smithi063.stdout:2/125: mknod d2/de/d22/c2a 0 2024-03-23T11:47:26.267 INFO:tasks.workunit.client.1.smithi111.stdout:3/107: dwrite d0/f9 [4194304,4194304] 0 2024-03-23T11:47:26.269 INFO:tasks.workunit.client.0.smithi063.stdout:2/126: write d2/f25 [920119,83104] 0 2024-03-23T11:47:26.269 INFO:tasks.workunit.client.1.smithi111.stdout:0/160: truncate d5/db/d1f/f1a 1147857 0 2024-03-23T11:47:26.269 INFO:tasks.workunit.client.1.smithi111.stdout:4/131: write d0/f1 [2972879,113567] 0 2024-03-23T11:47:26.273 INFO:tasks.workunit.client.0.smithi063.stdout:2/127: stat d2/de/l1a 0 2024-03-23T11:47:26.274 INFO:tasks.workunit.client.1.smithi111.stdout:7/119: dread fb [0,4194304] 0 2024-03-23T11:47:26.277 INFO:tasks.workunit.client.1.smithi111.stdout:1/141: getdents d0/d5/dd 0 2024-03-23T11:47:26.278 INFO:tasks.workunit.client.1.smithi111.stdout:0/161: mknod d5/db/d1f/c34 0 2024-03-23T11:47:26.281 INFO:tasks.workunit.client.1.smithi111.stdout:5/180: dread d1/d3/d6/d29/f22 [0,4194304] 0 2024-03-23T11:47:26.282 INFO:tasks.workunit.client.0.smithi063.stdout:0/113: dwrite d2/f27 [0,4194304] 0 2024-03-23T11:47:26.282 INFO:tasks.workunit.client.1.smithi111.stdout:3/108: chown d0/cc 21152 1 2024-03-23T11:47:26.287 INFO:tasks.workunit.client.1.smithi111.stdout:1/142: dread d0/f1d [0,4194304] 0 2024-03-23T11:47:26.287 INFO:tasks.workunit.client.1.smithi111.stdout:7/120: creat d14/f1a x:0 0 0 2024-03-23T11:47:26.289 INFO:tasks.workunit.client.1.smithi111.stdout:0/162: write d5/f8 [262140,120194] 0 2024-03-23T11:47:26.290 INFO:tasks.workunit.client.1.smithi111.stdout:7/121: dread - d14/f1a zero size 2024-03-23T11:47:26.299 INFO:tasks.workunit.client.1.smithi111.stdout:8/122: dread d5/f11 [0,4194304] 0 2024-03-23T11:47:26.301 INFO:tasks.workunit.client.0.smithi063.stdout:2/128: dwrite d2/f1e [0,4194304] 0 2024-03-23T11:47:26.302 INFO:tasks.workunit.client.1.smithi111.stdout:5/181: getdents d1/d3/d6 0 2024-03-23T11:47:26.303 INFO:tasks.workunit.client.1.smithi111.stdout:8/123: chown d5/d1b 22198812 1 2024-03-23T11:47:26.303 INFO:tasks.workunit.client.1.smithi111.stdout:1/143: creat d0/d5/dd/d1e/f31 x:0 0 0 2024-03-23T11:47:26.304 INFO:tasks.workunit.client.1.smithi111.stdout:4/132: dread d0/dc/fe [0,4194304] 0 2024-03-23T11:47:26.304 INFO:tasks.workunit.client.0.smithi063.stdout:2/129: fsync d2/de/f15 0 2024-03-23T11:47:26.305 INFO:tasks.workunit.client.1.smithi111.stdout:5/182: write d1/d3/d6/ff [389965,27772] 0 2024-03-23T11:47:26.308 INFO:tasks.workunit.client.1.smithi111.stdout:8/124: mkdir d5/d27 0 2024-03-23T11:47:26.310 INFO:tasks.workunit.client.1.smithi111.stdout:3/109: truncate d0/f3 185507 0 2024-03-23T11:47:26.312 INFO:tasks.workunit.client.1.smithi111.stdout:1/144: symlink d0/l32 0 2024-03-23T11:47:26.313 INFO:tasks.workunit.client.1.smithi111.stdout:0/163: dread d5/f30 [0,4194304] 0 2024-03-23T11:47:26.314 INFO:tasks.workunit.client.0.smithi063.stdout:2/130: creat d2/de/f2b x:0 0 0 2024-03-23T11:47:26.316 INFO:tasks.workunit.client.1.smithi111.stdout:0/164: chown d5/ca 31432 1 2024-03-23T11:47:26.317 INFO:tasks.workunit.client.0.smithi063.stdout:2/131: creat d2/de/f2c x:0 0 0 2024-03-23T11:47:26.318 INFO:tasks.workunit.client.1.smithi111.stdout:4/133: mkdir d0/d17/d29/d2a 0 2024-03-23T11:47:26.318 INFO:tasks.workunit.client.1.smithi111.stdout:8/125: mknod d5/d27/c28 0 2024-03-23T11:47:26.319 INFO:tasks.workunit.client.1.smithi111.stdout:1/145: symlink d0/d2b/l33 0 2024-03-23T11:47:26.320 INFO:tasks.workunit.client.1.smithi111.stdout:3/110: dread d0/fd [0,4194304] 0 2024-03-23T11:47:26.320 INFO:tasks.workunit.client.1.smithi111.stdout:0/165: mknod d5/c35 0 2024-03-23T11:47:26.321 INFO:tasks.workunit.client.0.smithi063.stdout:2/132: truncate d2/de/f1b 905126 0 2024-03-23T11:47:26.325 INFO:tasks.workunit.client.1.smithi111.stdout:1/146: creat d0/d6/f34 x:0 0 0 2024-03-23T11:47:26.327 INFO:tasks.workunit.client.1.smithi111.stdout:3/111: mknod d0/c1b 0 2024-03-23T11:47:26.333 INFO:tasks.workunit.client.1.smithi111.stdout:1/147: creat d0/d5/f35 x:0 0 0 2024-03-23T11:47:26.333 INFO:tasks.workunit.client.1.smithi111.stdout:9/192: sync 2024-03-23T11:47:26.333 INFO:tasks.workunit.client.1.smithi111.stdout:2/172: sync 2024-03-23T11:47:26.333 INFO:tasks.workunit.client.1.smithi111.stdout:6/226: sync 2024-03-23T11:47:26.334 INFO:tasks.workunit.client.1.smithi111.stdout:4/134: dread d0/d7/f16 [0,4194304] 0 2024-03-23T11:47:26.334 INFO:tasks.workunit.client.0.smithi063.stdout:2/133: truncate d2/f25 210964 0 2024-03-23T11:47:26.335 INFO:tasks.workunit.client.1.smithi111.stdout:0/166: truncate d5/db/d1f/d19/f2a 577961 0 2024-03-23T11:47:26.335 INFO:tasks.workunit.client.1.smithi111.stdout:7/122: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:26.337 INFO:tasks.workunit.client.0.smithi063.stdout:2/134: write d2/de/f2c [373401,87912] 0 2024-03-23T11:47:26.337 INFO:tasks.workunit.client.1.smithi111.stdout:0/167: chown d5/db/d1f 2882 1 2024-03-23T11:47:26.338 INFO:tasks.workunit.client.0.smithi063.stdout:2/135: chown d2/f17 716599 1 2024-03-23T11:47:26.339 INFO:tasks.workunit.client.1.smithi111.stdout:5/183: dwrite d1/d3/d6/d7/f10 [0,4194304] 0 2024-03-23T11:47:26.340 INFO:tasks.workunit.client.0.smithi063.stdout:3/260: dread f6 [4194304,4194304] 0 2024-03-23T11:47:26.341 INFO:tasks.workunit.client.1.smithi111.stdout:8/126: write d5/f17 [1380114,4340] 0 2024-03-23T11:47:26.341 INFO:tasks.workunit.client.1.smithi111.stdout:7/123: unlink f2 0 2024-03-23T11:47:26.341 INFO:tasks.workunit.client.1.smithi111.stdout:6/227: mknod d2/d17/c50 0 2024-03-23T11:47:26.342 INFO:tasks.workunit.client.1.smithi111.stdout:3/112: creat d0/d1a/f1c x:0 0 0 2024-03-23T11:47:26.343 INFO:tasks.workunit.client.1.smithi111.stdout:5/184: dread - d1/d3/d3a/f3d zero size 2024-03-23T11:47:26.344 INFO:tasks.workunit.client.1.smithi111.stdout:6/228: stat d2/c21 0 2024-03-23T11:47:26.344 INFO:tasks.workunit.client.1.smithi111.stdout:8/127: readlink d5/l9 0 2024-03-23T11:47:26.345 INFO:tasks.workunit.client.1.smithi111.stdout:4/135: symlink d0/d7/da/l2b 0 2024-03-23T11:47:26.345 INFO:tasks.workunit.client.1.smithi111.stdout:9/193: mkdir d0/d4/d1c/d2d/d3b/d42 0 2024-03-23T11:47:26.346 INFO:tasks.workunit.client.1.smithi111.stdout:2/173: truncate d2/d11/d17/f21 814258 0 2024-03-23T11:47:26.349 INFO:tasks.workunit.client.1.smithi111.stdout:7/124: mkdir d14/d1b 0 2024-03-23T11:47:26.349 INFO:tasks.workunit.client.1.smithi111.stdout:3/113: write d0/f9 [6908505,127992] 0 2024-03-23T11:47:26.351 INFO:tasks.workunit.client.1.smithi111.stdout:2/174: fdatasync d2/d11/f1b 0 2024-03-23T11:47:26.357 INFO:tasks.workunit.client.1.smithi111.stdout:5/185: creat d1/d3/d3a/f3f x:0 0 0 2024-03-23T11:47:26.360 INFO:tasks.workunit.client.0.smithi063.stdout:2/136: dwrite d2/de/f1b [0,4194304] 0 2024-03-23T11:47:26.360 INFO:tasks.workunit.client.1.smithi111.stdout:3/114: mkdir d0/d1a/d1d 0 2024-03-23T11:47:26.361 INFO:tasks.workunit.client.1.smithi111.stdout:9/194: dread d0/d4/d1c/d2d/f2e [0,4194304] 0 2024-03-23T11:47:26.364 INFO:tasks.workunit.client.0.smithi063.stdout:2/137: write f1 [5302174,101765] 0 2024-03-23T11:47:26.365 INFO:tasks.workunit.client.1.smithi111.stdout:9/195: write d0/d4/f1b [328376,83778] 0 2024-03-23T11:47:26.365 INFO:tasks.workunit.client.1.smithi111.stdout:1/148: dwrite d0/d2b/f2e [0,4194304] 0 2024-03-23T11:47:26.366 INFO:tasks.workunit.client.0.smithi063.stdout:3/261: dwrite d13/d15/f1b [4194304,4194304] 0 2024-03-23T11:47:26.366 INFO:tasks.workunit.client.0.smithi063.stdout:2/138: fdatasync d2/de/f2b 0 2024-03-23T11:47:26.367 INFO:tasks.workunit.client.1.smithi111.stdout:1/149: write d0/d6/fc [4354929,116256] 0 2024-03-23T11:47:26.369 INFO:tasks.workunit.client.1.smithi111.stdout:6/229: rename d2/d2a to d2/d1a/d1c/d2c/d51 0 2024-03-23T11:47:26.371 INFO:tasks.workunit.client.1.smithi111.stdout:5/186: creat d1/d3/d6/d29/f40 x:0 0 0 2024-03-23T11:47:26.372 INFO:tasks.workunit.client.1.smithi111.stdout:1/150: write d0/d5/dd/f25 [6451,42280] 0 2024-03-23T11:47:26.373 INFO:tasks.workunit.client.0.smithi063.stdout:2/139: write d2/de/f1b [2046447,115704] 0 2024-03-23T11:47:26.376 INFO:tasks.workunit.client.1.smithi111.stdout:1/151: write d0/d5/dd/f25 [129469,10483] 0 2024-03-23T11:47:26.377 INFO:tasks.workunit.client.1.smithi111.stdout:4/136: link d0/d7/da/f13 d0/d17/f2c 0 2024-03-23T11:47:26.379 INFO:tasks.workunit.client.0.smithi063.stdout:2/140: mknod d2/d1c/c2d 0 2024-03-23T11:47:26.380 INFO:tasks.workunit.client.1.smithi111.stdout:2/175: creat d2/d11/d17/d3e/f40 x:0 0 0 2024-03-23T11:47:26.383 INFO:tasks.workunit.client.1.smithi111.stdout:2/176: chown d2/d7/da/lb 5290502 1 2024-03-23T11:47:26.383 INFO:tasks.workunit.client.1.smithi111.stdout:5/187: mkdir d1/d20/d41 0 2024-03-23T11:47:26.384 INFO:tasks.workunit.client.0.smithi063.stdout:2/141: mkdir d2/d23/d2e 0 2024-03-23T11:47:26.387 INFO:tasks.workunit.client.1.smithi111.stdout:5/188: fdatasync d1/d3/d3a/f3d 0 2024-03-23T11:47:26.389 INFO:tasks.workunit.client.1.smithi111.stdout:9/196: dwrite d0/d4/d1c/d36/f11 [0,4194304] 0 2024-03-23T11:47:26.390 INFO:tasks.workunit.client.1.smithi111.stdout:6/230: dwrite d2/d1f/f2e [0,4194304] 0 2024-03-23T11:47:26.391 INFO:tasks.workunit.client.0.smithi063.stdout:3/262: dread d13/d22/d2d/f31 [0,4194304] 0 2024-03-23T11:47:26.392 INFO:tasks.workunit.client.1.smithi111.stdout:8/128: write d5/fe [972241,8791] 0 2024-03-23T11:47:26.393 INFO:tasks.workunit.client.1.smithi111.stdout:7/125: getdents d14 0 2024-03-23T11:47:26.394 INFO:tasks.workunit.client.1.smithi111.stdout:6/231: truncate d2/d17/f33 328189 0 2024-03-23T11:47:26.402 INFO:tasks.workunit.client.1.smithi111.stdout:4/137: mknod d0/d7/c2d 0 2024-03-23T11:47:26.406 INFO:tasks.workunit.client.1.smithi111.stdout:5/189: dwrite d1/d20/f2a [0,4194304] 0 2024-03-23T11:47:26.408 INFO:tasks.workunit.client.1.smithi111.stdout:9/197: creat d0/d4/f43 x:0 0 0 2024-03-23T11:47:26.411 INFO:tasks.workunit.client.1.smithi111.stdout:2/177: creat d2/d11/d17/d1f/d30/f41 x:0 0 0 2024-03-23T11:47:26.411 INFO:tasks.workunit.client.1.smithi111.stdout:7/126: creat d14/f1c x:0 0 0 2024-03-23T11:47:26.412 INFO:tasks.workunit.client.1.smithi111.stdout:5/190: dread - d1/d3/d6/f2b zero size 2024-03-23T11:47:26.413 INFO:tasks.workunit.client.0.smithi063.stdout:3/263: dread d13/d15/f3c [0,4194304] 0 2024-03-23T11:47:26.413 INFO:tasks.workunit.client.1.smithi111.stdout:4/138: dread d0/d17/f2c [0,4194304] 0 2024-03-23T11:47:26.417 INFO:tasks.workunit.client.1.smithi111.stdout:5/191: write d1/d3/d3a/f3d [830791,31082] 0 2024-03-23T11:47:26.418 INFO:tasks.workunit.client.0.smithi063.stdout:3/264: unlink d13/c36 0 2024-03-23T11:47:26.418 INFO:tasks.workunit.client.1.smithi111.stdout:9/198: readlink d0/d4/d1c/d2d/d3b/l3c 0 2024-03-23T11:47:26.418 INFO:tasks.workunit.client.1.smithi111.stdout:1/152: getdents d0/d5 0 2024-03-23T11:47:26.420 INFO:tasks.workunit.client.1.smithi111.stdout:4/139: unlink d0/d7/f25 0 2024-03-23T11:47:26.425 INFO:tasks.workunit.client.0.smithi063.stdout:3/265: mknod d13/d22/d2d/c4a 0 2024-03-23T11:47:26.427 INFO:tasks.workunit.client.1.smithi111.stdout:9/199: fsync d0/d4/d1c/d36/d28/d34/f39 0 2024-03-23T11:47:26.428 INFO:tasks.workunit.client.0.smithi063.stdout:3/266: symlink d13/d22/l4b 0 2024-03-23T11:47:26.429 INFO:tasks.workunit.client.1.smithi111.stdout:9/200: read d0/d4/fe [504532,36598] 0 2024-03-23T11:47:26.432 INFO:tasks.workunit.client.1.smithi111.stdout:6/232: dread d2/f16 [0,4194304] 0 2024-03-23T11:47:26.432 INFO:tasks.workunit.client.1.smithi111.stdout:1/153: truncate d0/d5/dd/d1e/f21 856611 0 2024-03-23T11:47:26.432 INFO:tasks.workunit.client.1.smithi111.stdout:4/140: symlink d0/d17/d29/d2a/l2e 0 2024-03-23T11:47:26.435 INFO:tasks.workunit.client.1.smithi111.stdout:4/141: write d0/d7/f1b [2180252,6658] 0 2024-03-23T11:47:26.436 INFO:tasks.workunit.client.1.smithi111.stdout:6/233: rmdir d2 39 2024-03-23T11:47:26.443 INFO:tasks.workunit.client.1.smithi111.stdout:1/154: link d0/d6/f22 d0/d5/dd/d1e/f36 0 2024-03-23T11:47:26.445 INFO:tasks.workunit.client.0.smithi063.stdout:3/267: dwrite f5 [0,4194304] 0 2024-03-23T11:47:26.445 INFO:tasks.workunit.client.1.smithi111.stdout:6/234: creat d2/d1a/d1c/d2c/f52 x:0 0 0 2024-03-23T11:47:26.447 INFO:tasks.workunit.client.1.smithi111.stdout:4/142: truncate d0/d7/d9/f15 39431 0 2024-03-23T11:47:26.447 INFO:tasks.workunit.client.1.smithi111.stdout:9/201: getdents d0/d4/d1c/d36/d28/d34 0 2024-03-23T11:47:26.449 INFO:tasks.workunit.client.1.smithi111.stdout:2/178: write d2/de/d27/f28 [318857,55587] 0 2024-03-23T11:47:26.450 INFO:tasks.workunit.client.1.smithi111.stdout:6/235: write d2/d1a/f1d [290475,111699] 0 2024-03-23T11:47:26.451 INFO:tasks.workunit.client.1.smithi111.stdout:9/202: readlink d0/d31/l38 0 2024-03-23T11:47:26.455 INFO:tasks.workunit.client.1.smithi111.stdout:5/192: truncate d1/d3/d6/d29/f22 944147 0 2024-03-23T11:47:26.456 INFO:tasks.workunit.client.0.smithi063.stdout:3/268: link d13/f30 d13/d22/d2d/d3e/f4c 0 2024-03-23T11:47:26.459 INFO:tasks.workunit.client.1.smithi111.stdout:4/143: fdatasync d0/dc/f1c 0 2024-03-23T11:47:26.459 INFO:tasks.workunit.client.1.smithi111.stdout:2/179: unlink d2/de/f18 0 2024-03-23T11:47:26.461 INFO:tasks.workunit.client.1.smithi111.stdout:9/203: dread - d0/d4/f43 zero size 2024-03-23T11:47:26.465 INFO:tasks.workunit.client.1.smithi111.stdout:5/193: symlink d1/l42 0 2024-03-23T11:47:26.465 INFO:tasks.workunit.client.0.smithi063.stdout:3/269: link d13/d22/f2c d13/f4d 0 2024-03-23T11:47:26.466 INFO:tasks.workunit.client.0.smithi063.stdout:3/270: chown d13/d22 0 1 2024-03-23T11:47:26.466 INFO:tasks.workunit.client.1.smithi111.stdout:9/204: chown d0/d4/d10/d29 430463 1 2024-03-23T11:47:26.470 INFO:tasks.workunit.client.0.smithi063.stdout:3/271: mknod d13/d22/c4e 0 2024-03-23T11:47:26.470 INFO:tasks.workunit.client.1.smithi111.stdout:6/236: mknod d2/d1a/d1c/c53 0 2024-03-23T11:47:26.471 INFO:tasks.workunit.client.1.smithi111.stdout:5/194: mkdir d1/d3/d23/d2c/d43 0 2024-03-23T11:47:26.472 INFO:tasks.workunit.client.1.smithi111.stdout:2/180: symlink d2/de/d27/d2e/l42 0 2024-03-23T11:47:26.473 INFO:tasks.workunit.client.1.smithi111.stdout:1/155: dwrite d0/fe [0,4194304] 0 2024-03-23T11:47:26.473 INFO:tasks.workunit.client.0.smithi063.stdout:3/272: creat d13/d22/d2d/f4f x:0 0 0 2024-03-23T11:47:26.475 INFO:tasks.workunit.client.1.smithi111.stdout:5/195: stat d1/d3/d23/d2c/d43 0 2024-03-23T11:47:26.475 INFO:tasks.workunit.client.1.smithi111.stdout:6/237: chown d2/d1a/d1c/d2c 22205235 1 2024-03-23T11:47:26.476 INFO:tasks.workunit.client.1.smithi111.stdout:7/127: dread fd [0,4194304] 0 2024-03-23T11:47:26.477 INFO:tasks.workunit.client.1.smithi111.stdout:9/205: truncate d0/d4/d1c/d36/d28/f3f 10906 0 2024-03-23T11:47:26.478 INFO:tasks.workunit.client.0.smithi063.stdout:3/273: rename fd to d13/f50 0 2024-03-23T11:47:26.479 INFO:tasks.workunit.client.1.smithi111.stdout:6/238: creat d2/d1f/f54 x:0 0 0 2024-03-23T11:47:26.480 INFO:tasks.workunit.client.1.smithi111.stdout:5/196: symlink d1/d20/l44 0 2024-03-23T11:47:26.482 INFO:tasks.workunit.client.1.smithi111.stdout:5/197: fsync d1/d3/d3a/f3f 0 2024-03-23T11:47:26.482 INFO:tasks.workunit.client.1.smithi111.stdout:7/128: rename la to d14/d1b/l1d 0 2024-03-23T11:47:26.482 INFO:tasks.workunit.client.1.smithi111.stdout:4/144: dwrite d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:26.485 INFO:tasks.workunit.client.1.smithi111.stdout:4/145: chown d0/d7 0 1 2024-03-23T11:47:26.487 INFO:tasks.workunit.client.1.smithi111.stdout:5/198: symlink d1/d20/l45 0 2024-03-23T11:47:26.488 INFO:tasks.workunit.client.1.smithi111.stdout:0/168: sync 2024-03-23T11:47:26.489 INFO:tasks.workunit.client.0.smithi063.stdout:3/274: dread d13/d22/d2d/d3e/f4c [0,4194304] 0 2024-03-23T11:47:26.490 INFO:tasks.workunit.client.1.smithi111.stdout:6/239: rename d2/d1a/d1c/f42 to d2/d17/f55 0 2024-03-23T11:47:26.492 INFO:tasks.workunit.client.1.smithi111.stdout:9/206: mkdir d0/d31/d41/d44 0 2024-03-23T11:47:26.493 INFO:tasks.workunit.client.1.smithi111.stdout:1/156: write d0/d6/f27 [1525284,33556] 0 2024-03-23T11:47:26.495 INFO:tasks.workunit.client.0.smithi063.stdout:3/275: symlink d13/d22/l51 0 2024-03-23T11:47:26.497 INFO:tasks.workunit.client.1.smithi111.stdout:5/199: symlink d1/d3/d3a/l46 0 2024-03-23T11:47:26.499 INFO:tasks.workunit.client.0.smithi063.stdout:3/276: symlink d13/d22/d2d/d3e/l52 0 2024-03-23T11:47:26.499 INFO:tasks.workunit.client.1.smithi111.stdout:6/240: rename d2/f18 to d2/d1f/d45/f56 0 2024-03-23T11:47:26.500 INFO:tasks.workunit.client.1.smithi111.stdout:5/200: chown d1/d3 34217292 1 2024-03-23T11:47:26.501 INFO:tasks.workunit.client.0.smithi063.stdout:3/277: write d13/d15/f33 [529017,38938] 0 2024-03-23T11:47:26.503 INFO:tasks.workunit.client.1.smithi111.stdout:1/157: mkdir d0/d2b/d37 0 2024-03-23T11:47:26.504 INFO:tasks.workunit.client.0.smithi063.stdout:3/278: creat d13/d22/d2d/f53 x:0 0 0 2024-03-23T11:47:26.505 INFO:tasks.workunit.client.1.smithi111.stdout:5/201: mknod d1/d3/d6/d30/c47 0 2024-03-23T11:47:26.506 INFO:tasks.workunit.client.1.smithi111.stdout:4/146: mknod d0/d17/c2f 0 2024-03-23T11:47:26.508 INFO:tasks.workunit.client.1.smithi111.stdout:6/241: creat d2/f57 x:0 0 0 2024-03-23T11:47:26.509 INFO:tasks.workunit.client.1.smithi111.stdout:5/202: truncate d1/f35 517437 0 2024-03-23T11:47:26.509 INFO:tasks.workunit.client.1.smithi111.stdout:1/158: symlink d0/d2b/d37/l38 0 2024-03-23T11:47:26.512 INFO:tasks.workunit.client.1.smithi111.stdout:0/169: dread d5/db/d1f/f18 [0,4194304] 0 2024-03-23T11:47:26.515 INFO:tasks.workunit.client.1.smithi111.stdout:0/170: readlink d5/db/l33 0 2024-03-23T11:47:26.518 INFO:tasks.workunit.client.1.smithi111.stdout:5/203: creat d1/d3/d6/d7/f48 x:0 0 0 2024-03-23T11:47:26.518 INFO:tasks.workunit.client.1.smithi111.stdout:0/171: readlink d5/db/d1f/d19/l1b 0 2024-03-23T11:47:26.522 INFO:tasks.workunit.client.1.smithi111.stdout:5/204: write d1/f17 [1302865,104282] 0 2024-03-23T11:47:26.526 INFO:tasks.workunit.client.1.smithi111.stdout:0/172: mknod d5/db/d1f/d19/c36 0 2024-03-23T11:47:26.527 INFO:tasks.workunit.client.1.smithi111.stdout:9/207: dwrite d0/d4/d1c/f26 [0,4194304] 0 2024-03-23T11:47:26.538 INFO:tasks.workunit.client.1.smithi111.stdout:5/205: write d1/d20/d25/f26 [327330,25388] 0 2024-03-23T11:47:26.542 INFO:tasks.workunit.client.1.smithi111.stdout:6/242: dwrite d2/f57 [0,4194304] 0 2024-03-23T11:47:26.543 INFO:tasks.workunit.client.1.smithi111.stdout:9/208: creat d0/d4/d1c/f45 x:0 0 0 2024-03-23T11:47:26.543 INFO:tasks.workunit.client.1.smithi111.stdout:5/206: stat d1/d20/d41 0 2024-03-23T11:47:26.545 INFO:tasks.workunit.client.1.smithi111.stdout:0/173: getdents d5/db/d1f/d32 0 2024-03-23T11:47:26.545 INFO:tasks.workunit.client.1.smithi111.stdout:4/147: dwrite d0/dc/f1c [0,4194304] 0 2024-03-23T11:47:26.549 INFO:tasks.workunit.client.1.smithi111.stdout:7/129: dread f7 [0,4194304] 0 2024-03-23T11:47:26.551 INFO:tasks.workunit.client.1.smithi111.stdout:6/243: chown d2/d1a/f1d 453126 1 2024-03-23T11:47:26.554 INFO:tasks.workunit.client.1.smithi111.stdout:5/207: rename d1/f35 to d1/f49 0 2024-03-23T11:47:26.555 INFO:tasks.workunit.client.1.smithi111.stdout:9/209: creat d0/d4/d1c/f46 x:0 0 0 2024-03-23T11:47:26.559 INFO:tasks.workunit.client.1.smithi111.stdout:6/244: chown d2/d1a/d1c/c53 982945 1 2024-03-23T11:47:26.560 INFO:tasks.workunit.client.1.smithi111.stdout:9/210: chown d0/d31/d35/c37 24657 1 2024-03-23T11:47:26.562 INFO:tasks.workunit.client.1.smithi111.stdout:1/159: dread d0/d6/fc [0,4194304] 0 2024-03-23T11:47:26.562 INFO:tasks.workunit.client.1.smithi111.stdout:5/208: mknod d1/d3/d23/d2c/d43/c4a 0 2024-03-23T11:47:26.572 INFO:tasks.workunit.client.1.smithi111.stdout:9/211: creat d0/d4/f47 x:0 0 0 2024-03-23T11:47:26.574 INFO:tasks.workunit.client.1.smithi111.stdout:6/245: getdents d2/d1a/d1c 0 2024-03-23T11:47:26.580 INFO:tasks.workunit.client.1.smithi111.stdout:9/212: chown d0/d4/d1c/d36/f11 31 1 2024-03-23T11:47:26.582 INFO:tasks.workunit.client.1.smithi111.stdout:6/246: creat d2/d1a/d1c/f58 x:0 0 0 2024-03-23T11:47:26.582 INFO:tasks.workunit.client.0.smithi063.stdout:9/79: sync 2024-03-23T11:47:26.583 INFO:tasks.workunit.client.1.smithi111.stdout:0/174: dwrite d5/f2d [4194304,4194304] 0 2024-03-23T11:47:26.585 INFO:tasks.workunit.client.0.smithi063.stdout:9/80: mknod d4/d11/d19/c1a 0 2024-03-23T11:47:26.586 INFO:tasks.workunit.client.1.smithi111.stdout:9/213: readlink d0/d4/l2a 0 2024-03-23T11:47:26.586 INFO:tasks.workunit.client.1.smithi111.stdout:4/148: dwrite d0/dc/f1c [0,4194304] 0 2024-03-23T11:47:26.587 INFO:tasks.workunit.client.0.smithi063.stdout:9/81: creat d4/d11/d19/f1b x:0 0 0 2024-03-23T11:47:26.588 INFO:tasks.workunit.client.1.smithi111.stdout:7/130: dwrite d14/f19 [0,4194304] 0 2024-03-23T11:47:26.593 INFO:tasks.workunit.client.1.smithi111.stdout:4/149: chown d0/l11 53949 1 2024-03-23T11:47:26.594 INFO:tasks.workunit.client.1.smithi111.stdout:9/214: chown d0/d4/d1c/f45 35 1 2024-03-23T11:47:26.596 INFO:tasks.workunit.client.1.smithi111.stdout:6/247: write d2/f28 [129591,101078] 0 2024-03-23T11:47:26.599 INFO:tasks.workunit.client.1.smithi111.stdout:4/150: chown d0/dc/d1f 6894 1 2024-03-23T11:47:26.604 INFO:tasks.workunit.client.1.smithi111.stdout:5/209: dwrite d1/d3/d6/f3b [0,4194304] 0 2024-03-23T11:47:26.605 INFO:tasks.workunit.client.1.smithi111.stdout:9/215: chown d0/d4/f14 15477 1 2024-03-23T11:47:26.608 INFO:tasks.workunit.client.0.smithi063.stdout:9/82: dwrite d4/fe [0,4194304] 0 2024-03-23T11:47:26.609 INFO:tasks.workunit.client.1.smithi111.stdout:1/160: dwrite d0/d5/dd/f25 [0,4194304] 0 2024-03-23T11:47:26.611 INFO:tasks.workunit.client.1.smithi111.stdout:9/216: write d0/d4/f1b [2828202,123302] 0 2024-03-23T11:47:26.611 INFO:tasks.workunit.client.0.smithi063.stdout:9/83: write d4/fa [1379116,92171] 0 2024-03-23T11:47:26.614 INFO:tasks.workunit.client.1.smithi111.stdout:3/115: sync 2024-03-23T11:47:26.615 INFO:tasks.workunit.client.0.smithi063.stdout:9/84: write d4/ff [685336,13410] 0 2024-03-23T11:47:26.616 INFO:tasks.workunit.client.1.smithi111.stdout:1/161: dread - d0/d6/f34 zero size 2024-03-23T11:47:26.625 INFO:tasks.workunit.client.1.smithi111.stdout:1/162: rmdir d0/d2b/d37 39 2024-03-23T11:47:26.632 INFO:tasks.workunit.client.1.smithi111.stdout:2/181: sync 2024-03-23T11:47:26.632 INFO:tasks.workunit.client.1.smithi111.stdout:8/129: sync 2024-03-23T11:47:26.633 INFO:tasks.workunit.client.1.smithi111.stdout:3/116: truncate d0/d1a/f1c 319178 0 2024-03-23T11:47:26.636 INFO:tasks.workunit.client.1.smithi111.stdout:7/131: dwrite d14/f15 [0,4194304] 0 2024-03-23T11:47:26.637 INFO:tasks.workunit.client.0.smithi063.stdout:9/85: dwrite d4/f16 [0,4194304] 0 2024-03-23T11:47:26.638 INFO:tasks.workunit.client.1.smithi111.stdout:0/175: dwrite d5/f8 [0,4194304] 0 2024-03-23T11:47:26.640 INFO:tasks.workunit.client.0.smithi063.stdout:9/86: mknod d4/c1c 0 2024-03-23T11:47:26.642 INFO:tasks.workunit.client.0.smithi063.stdout:9/87: write d4/fa [839026,15845] 0 2024-03-23T11:47:26.654 INFO:tasks.workunit.client.1.smithi111.stdout:8/130: rename d5/cb to d5/d1b/c29 0 2024-03-23T11:47:26.655 INFO:tasks.workunit.client.1.smithi111.stdout:5/210: dread d1/d3/d6/d29/f21 [0,4194304] 0 2024-03-23T11:47:26.661 INFO:tasks.workunit.client.1.smithi111.stdout:6/248: dwrite d2/d1a/f22 [0,4194304] 0 2024-03-23T11:47:26.663 INFO:tasks.workunit.client.1.smithi111.stdout:8/131: readlink d5/l24 0 2024-03-23T11:47:26.668 INFO:tasks.workunit.client.0.smithi063.stdout:9/88: rmdir d4 39 2024-03-23T11:47:26.668 INFO:tasks.workunit.client.1.smithi111.stdout:9/217: dread d0/f13 [0,4194304] 0 2024-03-23T11:47:26.670 INFO:tasks.workunit.client.1.smithi111.stdout:1/163: getdents d0/d6 0 2024-03-23T11:47:26.672 INFO:tasks.workunit.client.1.smithi111.stdout:0/176: rename d5/f2c to d5/db/d1f/f37 0 2024-03-23T11:47:26.672 INFO:tasks.workunit.client.0.smithi063.stdout:9/89: rmdir d4 39 2024-03-23T11:47:26.673 INFO:tasks.workunit.client.1.smithi111.stdout:6/249: unlink d2/f8 0 2024-03-23T11:47:26.675 INFO:tasks.workunit.client.1.smithi111.stdout:7/132: write f9 [492901,46847] 0 2024-03-23T11:47:26.677 INFO:tasks.workunit.client.1.smithi111.stdout:5/211: creat d1/d3/d6/d29/f4b x:0 0 0 2024-03-23T11:47:26.678 INFO:tasks.workunit.client.0.smithi063.stdout:9/90: fsync d4/fa 0 2024-03-23T11:47:26.678 INFO:tasks.workunit.client.1.smithi111.stdout:9/218: creat d0/f48 x:0 0 0 2024-03-23T11:47:26.679 INFO:tasks.workunit.client.1.smithi111.stdout:4/151: dwrite d0/d7/da/f13 [0,4194304] 0 2024-03-23T11:47:26.679 INFO:tasks.workunit.client.1.smithi111.stdout:8/132: mknod d5/c2a 0 2024-03-23T11:47:26.680 INFO:tasks.workunit.client.1.smithi111.stdout:3/117: write d0/f3 [1136355,43149] 0 2024-03-23T11:47:26.681 INFO:tasks.workunit.client.0.smithi063.stdout:9/91: symlink d4/d11/d19/l1d 0 2024-03-23T11:47:26.683 INFO:tasks.workunit.client.1.smithi111.stdout:1/164: rmdir d0/d5/dd 39 2024-03-23T11:47:26.683 INFO:tasks.workunit.client.1.smithi111.stdout:9/219: readlink d0/d4/l2a 0 2024-03-23T11:47:26.684 INFO:tasks.workunit.client.0.smithi063.stdout:9/92: link d4/fe d4/f1e 0 2024-03-23T11:47:26.685 INFO:tasks.workunit.client.1.smithi111.stdout:6/250: write d2/d1a/f1d [700478,19450] 0 2024-03-23T11:47:26.687 INFO:tasks.workunit.client.0.smithi063.stdout:9/93: unlink d4/f9 0 2024-03-23T11:47:26.688 INFO:tasks.workunit.client.1.smithi111.stdout:3/118: unlink d0/l5 0 2024-03-23T11:47:26.689 INFO:tasks.workunit.client.1.smithi111.stdout:0/177: creat d5/db/f38 x:0 0 0 2024-03-23T11:47:26.689 INFO:tasks.workunit.client.1.smithi111.stdout:7/133: mkdir d14/d1b/d1e 0 2024-03-23T11:47:26.692 INFO:tasks.workunit.client.1.smithi111.stdout:3/119: stat d0/d1a 0 2024-03-23T11:47:26.693 INFO:tasks.workunit.client.1.smithi111.stdout:1/165: truncate d0/d5/f2f 983606 0 2024-03-23T11:47:26.694 INFO:tasks.workunit.client.1.smithi111.stdout:5/212: write d1/d3/d23/d2c/f2d [758369,87470] 0 2024-03-23T11:47:26.694 INFO:tasks.workunit.client.1.smithi111.stdout:7/134: readlink l3 0 2024-03-23T11:47:26.697 INFO:tasks.workunit.client.1.smithi111.stdout:6/251: write d2/d17/f4d [534041,100994] 0 2024-03-23T11:47:26.697 INFO:tasks.workunit.client.1.smithi111.stdout:2/182: dwrite d2/d11/d17/d1f/f20 [0,4194304] 0 2024-03-23T11:47:26.697 INFO:tasks.workunit.client.1.smithi111.stdout:4/152: rename d0/d7/d9 to d0/dc/d30 0 2024-03-23T11:47:26.702 INFO:tasks.workunit.client.1.smithi111.stdout:5/213: unlink d1/d3/d6/l1d 0 2024-03-23T11:47:26.705 INFO:tasks.workunit.client.1.smithi111.stdout:4/153: mknod d0/dc/d30/c31 0 2024-03-23T11:47:26.707 INFO:tasks.workunit.client.1.smithi111.stdout:9/220: link d0/d4/d1c/d36/l8 d0/d31/d41/l49 0 2024-03-23T11:47:26.708 INFO:tasks.workunit.client.1.smithi111.stdout:5/214: mknod d1/d20/c4c 0 2024-03-23T11:47:26.708 INFO:tasks.workunit.client.0.smithi063.stdout:9/94: dwrite d4/f1e [4194304,4194304] 0 2024-03-23T11:47:26.709 INFO:tasks.workunit.client.1.smithi111.stdout:1/166: getdents d0/d5/dd 0 2024-03-23T11:47:26.711 INFO:tasks.workunit.client.0.smithi063.stdout:9/95: mknod d4/c1f 0 2024-03-23T11:47:26.713 INFO:tasks.workunit.client.1.smithi111.stdout:1/167: fdatasync d0/d6/f22 0 2024-03-23T11:47:26.713 INFO:tasks.workunit.client.1.smithi111.stdout:9/221: dread - d0/d4/f43 zero size 2024-03-23T11:47:26.716 INFO:tasks.workunit.client.1.smithi111.stdout:9/222: chown d0/l27 350345981 1 2024-03-23T11:47:26.719 INFO:tasks.workunit.client.1.smithi111.stdout:4/154: unlink d0/d17/c2f 0 2024-03-23T11:47:26.721 INFO:tasks.workunit.client.1.smithi111.stdout:3/120: dwrite d0/d1a/f1c [0,4194304] 0 2024-03-23T11:47:26.722 INFO:tasks.workunit.client.1.smithi111.stdout:2/183: link d2/d11/d17/f21 d2/de/d12/f43 0 2024-03-23T11:47:26.727 INFO:tasks.workunit.client.1.smithi111.stdout:4/155: write d0/dc/d30/ff [1067711,26495] 0 2024-03-23T11:47:26.727 INFO:tasks.workunit.client.1.smithi111.stdout:7/135: dwrite fd [0,4194304] 0 2024-03-23T11:47:26.728 INFO:tasks.workunit.client.1.smithi111.stdout:9/223: rename d0/d31/d35/c37 to d0/d31/c4a 0 2024-03-23T11:47:26.730 INFO:tasks.workunit.client.1.smithi111.stdout:3/121: creat d0/d7/f1e x:0 0 0 2024-03-23T11:47:26.731 INFO:tasks.workunit.client.1.smithi111.stdout:3/122: chown d0/f9 1 1 2024-03-23T11:47:26.732 INFO:tasks.workunit.client.0.smithi063.stdout:9/96: dwrite d4/fa [0,4194304] 0 2024-03-23T11:47:26.732 INFO:tasks.workunit.client.1.smithi111.stdout:3/123: chown d0/d1a 10456 1 2024-03-23T11:47:26.735 INFO:tasks.workunit.client.1.smithi111.stdout:3/124: write d0/d7/f12 [1250587,126593] 0 2024-03-23T11:47:26.736 INFO:tasks.workunit.client.0.smithi063.stdout:9/97: creat d4/f20 x:0 0 0 2024-03-23T11:47:26.738 INFO:tasks.workunit.client.1.smithi111.stdout:2/184: rename d2/d7/da/f2d to d2/de/d27/d2e/d3f/f44 0 2024-03-23T11:47:26.738 INFO:tasks.workunit.client.0.smithi063.stdout:9/98: fdatasync d4/fd 0 2024-03-23T11:47:26.741 INFO:tasks.workunit.client.0.smithi063.stdout:9/99: write d4/fe [1774085,37346] 0 2024-03-23T11:47:26.747 INFO:tasks.workunit.client.0.smithi063.stdout:9/100: link d4/f5 d4/f21 0 2024-03-23T11:47:26.747 INFO:tasks.workunit.client.1.smithi111.stdout:7/136: dwrite d14/f15 [0,4194304] 0 2024-03-23T11:47:26.749 INFO:tasks.workunit.client.0.smithi063.stdout:9/101: write d4/f16 [5178352,30182] 0 2024-03-23T11:47:26.751 INFO:tasks.workunit.client.0.smithi063.stdout:9/102: readlink d4/d11/d19/l1d 0 2024-03-23T11:47:26.754 INFO:tasks.workunit.client.0.smithi063.stdout:9/103: chown d4/l8 162 1 2024-03-23T11:47:26.757 INFO:tasks.workunit.client.0.smithi063.stdout:9/104: unlink d4/fd 0 2024-03-23T11:47:26.764 INFO:tasks.workunit.client.0.smithi063.stdout:9/105: creat d4/d11/d19/f22 x:0 0 0 2024-03-23T11:47:26.765 INFO:tasks.workunit.client.0.smithi063.stdout:9/106: readlink d4/l8 0 2024-03-23T11:47:26.765 INFO:tasks.workunit.client.1.smithi111.stdout:2/185: symlink d2/d3d/l45 0 2024-03-23T11:47:26.767 INFO:tasks.workunit.client.0.smithi063.stdout:5/78: sync 2024-03-23T11:47:26.767 INFO:tasks.workunit.client.0.smithi063.stdout:4/156: sync 2024-03-23T11:47:26.768 INFO:tasks.workunit.client.1.smithi111.stdout:5/215: fsync d1/d3/d6/d7/f3e 0 2024-03-23T11:47:26.769 INFO:tasks.workunit.client.0.smithi063.stdout:5/79: fsync d2/d7/f14 0 2024-03-23T11:47:26.774 INFO:tasks.workunit.client.1.smithi111.stdout:3/125: creat d0/f1f x:0 0 0 2024-03-23T11:47:26.776 INFO:tasks.workunit.client.1.smithi111.stdout:2/186: rmdir d2/d11 39 2024-03-23T11:47:26.777 INFO:tasks.workunit.client.0.smithi063.stdout:4/157: symlink d5/d1e/l27 0 2024-03-23T11:47:26.778 INFO:tasks.workunit.client.1.smithi111.stdout:1/168: truncate d0/d5/dd/f25 1267725 0 2024-03-23T11:47:26.781 INFO:tasks.workunit.client.0.smithi063.stdout:9/107: truncate d4/fa 403259 0 2024-03-23T11:47:26.783 INFO:tasks.workunit.client.0.smithi063.stdout:4/158: rename d5/c17 to d5/da/d20/d24/c28 0 2024-03-23T11:47:26.786 INFO:tasks.workunit.client.1.smithi111.stdout:5/216: creat d1/d3/d6/d29/f4d x:0 0 0 2024-03-23T11:47:26.787 INFO:tasks.workunit.client.0.smithi063.stdout:8/305: sync 2024-03-23T11:47:26.787 INFO:tasks.workunit.client.1.smithi111.stdout:4/156: symlink d0/d17/d29/d2a/l32 0 2024-03-23T11:47:26.787 INFO:tasks.workunit.client.0.smithi063.stdout:6/127: sync 2024-03-23T11:47:26.787 INFO:tasks.workunit.client.0.smithi063.stdout:2/142: sync 2024-03-23T11:47:26.788 INFO:tasks.workunit.client.1.smithi111.stdout:4/157: read d0/dc/fe [1368382,27772] 0 2024-03-23T11:47:26.790 INFO:tasks.workunit.client.0.smithi063.stdout:7/154: sync 2024-03-23T11:47:26.790 INFO:tasks.workunit.client.0.smithi063.stdout:0/114: sync 2024-03-23T11:47:26.790 INFO:tasks.workunit.client.0.smithi063.stdout:1/148: sync 2024-03-23T11:47:26.790 INFO:tasks.workunit.client.0.smithi063.stdout:3/279: sync 2024-03-23T11:47:26.790 INFO:tasks.workunit.client.0.smithi063.stdout:2/143: dread - d2/de/f2b zero size 2024-03-23T11:47:26.792 INFO:tasks.workunit.client.1.smithi111.stdout:7/137: write f11 [559354,33076] 0 2024-03-23T11:47:26.794 INFO:tasks.workunit.client.1.smithi111.stdout:5/217: mknod d1/d20/c4e 0 2024-03-23T11:47:26.796 INFO:tasks.workunit.client.1.smithi111.stdout:4/158: rmdir d0/d7 39 2024-03-23T11:47:26.796 INFO:tasks.workunit.client.1.smithi111.stdout:4/159: stat d0/c2 0 2024-03-23T11:47:26.798 INFO:tasks.workunit.client.0.smithi063.stdout:1/149: symlink d3/l34 0 2024-03-23T11:47:26.798 INFO:tasks.workunit.client.0.smithi063.stdout:3/280: readlink d13/d22/d2d/l46 0 2024-03-23T11:47:26.801 INFO:tasks.workunit.client.0.smithi063.stdout:6/128: symlink d1/d13/l25 0 2024-03-23T11:47:26.802 INFO:tasks.workunit.client.0.smithi063.stdout:0/115: mkdir d2/d4/dd/d28 0 2024-03-23T11:47:26.804 INFO:tasks.workunit.client.1.smithi111.stdout:1/169: rmdir d0 39 2024-03-23T11:47:26.807 INFO:tasks.workunit.client.1.smithi111.stdout:3/126: truncate d0/d7/f12 1319986 0 2024-03-23T11:47:26.807 INFO:tasks.workunit.client.0.smithi063.stdout:8/306: mknod d6/d1f/d54/d5c/c67 0 2024-03-23T11:47:26.811 INFO:tasks.workunit.client.1.smithi111.stdout:1/170: fsync d0/d5/dd/d1e/f36 0 2024-03-23T11:47:26.813 INFO:tasks.workunit.client.1.smithi111.stdout:5/218: dwrite d1/d3/d23/d2c/f34 [0,4194304] 0 2024-03-23T11:47:26.815 INFO:tasks.workunit.client.0.smithi063.stdout:3/281: mknod d13/d22/d2d/d3e/c54 0 2024-03-23T11:47:26.817 INFO:tasks.workunit.client.1.smithi111.stdout:7/138: truncate f12 1624439 0 2024-03-23T11:47:26.820 INFO:tasks.workunit.client.0.smithi063.stdout:9/108: write d4/fa [456110,126184] 0 2024-03-23T11:47:26.821 INFO:tasks.workunit.client.1.smithi111.stdout:1/171: symlink d0/d5/dd/l39 0 2024-03-23T11:47:26.822 INFO:tasks.workunit.client.0.smithi063.stdout:7/155: dwrite d5/d6/f27 [0,4194304] 0 2024-03-23T11:47:26.823 INFO:tasks.workunit.client.0.smithi063.stdout:5/80: dread d2/d7/fd [0,4194304] 0 2024-03-23T11:47:26.825 INFO:tasks.workunit.client.0.smithi063.stdout:4/159: dwrite d5/da/d20/d24/d1f/f21 [0,4194304] 0 2024-03-23T11:47:26.831 INFO:tasks.workunit.client.0.smithi063.stdout:8/307: symlink d6/d1c/d32/l68 0 2024-03-23T11:47:26.832 INFO:tasks.workunit.client.1.smithi111.stdout:4/160: link d0/dc/d30/l12 d0/dc/l33 0 2024-03-23T11:47:26.832 INFO:tasks.workunit.client.0.smithi063.stdout:9/109: write d4/f14 [2736026,95959] 0 2024-03-23T11:47:26.832 INFO:tasks.workunit.client.0.smithi063.stdout:6/129: creat d1/d15/f26 x:0 0 0 2024-03-23T11:47:26.833 INFO:tasks.workunit.client.1.smithi111.stdout:5/219: rename d1/d3/d6/d7/ce to d1/d3/d6/d29/c4f 0 2024-03-23T11:47:26.835 INFO:tasks.workunit.client.1.smithi111.stdout:3/127: write d0/d7/f12 [1880212,125696] 0 2024-03-23T11:47:26.836 INFO:tasks.workunit.client.0.smithi063.stdout:1/150: dwrite d3/f7 [4194304,4194304] 0 2024-03-23T11:47:26.838 INFO:tasks.workunit.client.0.smithi063.stdout:3/282: creat d13/f55 x:0 0 0 2024-03-23T11:47:26.841 INFO:tasks.workunit.client.1.smithi111.stdout:7/139: rename l3 to d14/d1b/d1e/l1f 0 2024-03-23T11:47:26.841 INFO:tasks.workunit.client.1.smithi111.stdout:7/140: fdatasync f11 0 2024-03-23T11:47:26.842 INFO:tasks.workunit.client.1.smithi111.stdout:7/141: fdatasync f9 0 2024-03-23T11:47:26.843 INFO:tasks.workunit.client.0.smithi063.stdout:2/144: dwrite d2/de/f2b [0,4194304] 0 2024-03-23T11:47:26.844 INFO:tasks.workunit.client.0.smithi063.stdout:8/308: stat d6/d1c/d27/d3f 0 2024-03-23T11:47:26.845 INFO:tasks.workunit.client.0.smithi063.stdout:0/116: symlink d2/l29 0 2024-03-23T11:47:26.846 INFO:tasks.workunit.client.0.smithi063.stdout:1/151: readlink d3/df/l20 0 2024-03-23T11:47:26.848 INFO:tasks.workunit.client.1.smithi111.stdout:5/220: rename d1/d20/f2a to d1/d20/f50 0 2024-03-23T11:47:26.849 INFO:tasks.workunit.client.0.smithi063.stdout:4/160: mkdir d5/da/d20/d24/d29 0 2024-03-23T11:47:26.849 INFO:tasks.workunit.client.1.smithi111.stdout:3/128: unlink d0/f3 0 2024-03-23T11:47:26.850 INFO:tasks.workunit.client.0.smithi063.stdout:8/309: write d6/d1f/f55 [4861353,67439] 0 2024-03-23T11:47:26.851 INFO:tasks.workunit.client.0.smithi063.stdout:0/117: chown d2/d12 1 1 2024-03-23T11:47:26.851 INFO:tasks.workunit.client.0.smithi063.stdout:3/283: stat d13/d22/d2d/f45 0 2024-03-23T11:47:26.852 INFO:tasks.workunit.client.1.smithi111.stdout:5/221: write d1/d3/d3a/f3d [1574088,101953] 0 2024-03-23T11:47:26.854 INFO:tasks.workunit.client.0.smithi063.stdout:2/145: chown d2/f1e 28275 1 2024-03-23T11:47:26.856 INFO:tasks.workunit.client.0.smithi063.stdout:5/81: creat d2/d17/f19 x:0 0 0 2024-03-23T11:47:26.856 INFO:tasks.workunit.client.1.smithi111.stdout:3/129: dread d0/fd [0,4194304] 0 2024-03-23T11:47:26.857 INFO:tasks.workunit.client.1.smithi111.stdout:1/172: link d0/l2a d0/d5/l3a 0 2024-03-23T11:47:26.858 INFO:tasks.workunit.client.1.smithi111.stdout:3/130: write d0/d7/ff [748966,107613] 0 2024-03-23T11:47:26.859 INFO:tasks.workunit.client.0.smithi063.stdout:6/130: creat d1/d13/f27 x:0 0 0 2024-03-23T11:47:26.863 INFO:tasks.workunit.client.0.smithi063.stdout:1/152: mkdir d3/d35 0 2024-03-23T11:47:26.866 INFO:tasks.workunit.client.1.smithi111.stdout:5/222: creat d1/d3/d23/d2c/f51 x:0 0 0 2024-03-23T11:47:26.868 INFO:tasks.workunit.client.0.smithi063.stdout:0/118: readlink d2/d4/dd/l24 0 2024-03-23T11:47:26.869 INFO:tasks.workunit.client.1.smithi111.stdout:5/223: dread - d1/d3/d6/d29/f4d zero size 2024-03-23T11:47:26.870 INFO:tasks.workunit.client.1.smithi111.stdout:1/173: mkdir d0/d6/d3b 0 2024-03-23T11:47:26.870 INFO:tasks.workunit.client.0.smithi063.stdout:6/131: dread - d1/dc/f1b zero size 2024-03-23T11:47:26.873 INFO:tasks.workunit.client.0.smithi063.stdout:5/82: creat d2/d7/f1a x:0 0 0 2024-03-23T11:47:26.874 INFO:tasks.workunit.client.1.smithi111.stdout:4/161: link d0/d7/da/c27 d0/d7/da/c34 0 2024-03-23T11:47:26.875 INFO:tasks.workunit.client.1.smithi111.stdout:3/131: dwrite d0/f16 [0,4194304] 0 2024-03-23T11:47:26.875 INFO:tasks.workunit.client.1.smithi111.stdout:4/162: chown d0/d7/f16 44 1 2024-03-23T11:47:26.876 INFO:tasks.workunit.client.0.smithi063.stdout:9/110: dread d4/f1e [4194304,4194304] 0 2024-03-23T11:47:26.877 INFO:tasks.workunit.client.0.smithi063.stdout:1/153: creat d3/df/f36 x:0 0 0 2024-03-23T11:47:26.877 INFO:tasks.workunit.client.1.smithi111.stdout:4/163: write d0/d7/f16 [1000336,37114] 0 2024-03-23T11:47:26.881 INFO:tasks.workunit.client.1.smithi111.stdout:1/174: creat d0/d6/f3c x:0 0 0 2024-03-23T11:47:26.881 INFO:tasks.workunit.client.0.smithi063.stdout:0/119: symlink d2/d12/l2a 0 2024-03-23T11:47:26.882 INFO:tasks.workunit.client.0.smithi063.stdout:6/132: chown d1/d13/f1c 23 1 2024-03-23T11:47:26.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:26 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:26.883 INFO:tasks.workunit.client.0.smithi063.stdout:9/111: read - d4/f20 zero size 2024-03-23T11:47:26.884 INFO:tasks.workunit.client.0.smithi063.stdout:4/161: dread d5/ff [4194304,4194304] 0 2024-03-23T11:47:26.885 INFO:tasks.workunit.client.1.smithi111.stdout:1/175: truncate d0/d5/dd/d1e/f36 1961426 0 2024-03-23T11:47:26.885 INFO:tasks.workunit.client.1.smithi111.stdout:1/176: chown d0/f1d 3429201 1 2024-03-23T11:47:26.886 INFO:tasks.workunit.client.1.smithi111.stdout:5/224: creat d1/d3/d23/d2c/f52 x:0 0 0 2024-03-23T11:47:26.887 INFO:tasks.workunit.client.1.smithi111.stdout:4/164: creat d0/dc/f35 x:0 0 0 2024-03-23T11:47:26.887 INFO:tasks.workunit.client.1.smithi111.stdout:1/177: dread - d0/d5/f35 zero size 2024-03-23T11:47:26.889 INFO:tasks.workunit.client.1.smithi111.stdout:3/132: dread d0/d7/f10 [0,4194304] 0 2024-03-23T11:47:26.890 INFO:tasks.workunit.client.1.smithi111.stdout:5/225: unlink d1/d3/d6/d7/f48 0 2024-03-23T11:47:26.891 INFO:tasks.workunit.client.1.smithi111.stdout:5/226: chown d1/d3/d6/d7/f16 133811916 1 2024-03-23T11:47:26.892 INFO:tasks.workunit.client.0.smithi063.stdout:8/310: mknod d6/d1c/d32/d37/c69 0 2024-03-23T11:47:26.892 INFO:tasks.workunit.client.0.smithi063.stdout:5/83: mknod d2/d7/d8/de/c1b 0 2024-03-23T11:47:26.892 INFO:tasks.workunit.client.1.smithi111.stdout:3/133: fsync d0/d7/f10 0 2024-03-23T11:47:26.894 INFO:tasks.workunit.client.1.smithi111.stdout:4/165: rename d0/d7/c1e to d0/c36 0 2024-03-23T11:47:26.896 INFO:tasks.workunit.client.0.smithi063.stdout:8/311: readlink d6/d1c/d32/l3a 0 2024-03-23T11:47:26.897 INFO:tasks.workunit.client.1.smithi111.stdout:3/134: mknod d0/d7/c20 0 2024-03-23T11:47:26.897 INFO:tasks.workunit.client.0.smithi063.stdout:7/156: dread d5/d6/fe [0,4194304] 0 2024-03-23T11:47:26.898 INFO:tasks.workunit.client.1.smithi111.stdout:5/227: unlink d1/d3/d6/f8 0 2024-03-23T11:47:26.899 INFO:tasks.workunit.client.0.smithi063.stdout:9/112: creat d4/d11/d19/f23 x:0 0 0 2024-03-23T11:47:26.899 INFO:tasks.workunit.client.0.smithi063.stdout:2/146: rename d2/de/f2c to d2/f2f 0 2024-03-23T11:47:26.901 INFO:tasks.workunit.client.0.smithi063.stdout:5/84: write d2/d7/f10 [17398,40119] 0 2024-03-23T11:47:26.902 INFO:tasks.workunit.client.1.smithi111.stdout:4/166: mknod d0/d7/da/c37 0 2024-03-23T11:47:26.902 INFO:tasks.workunit.client.1.smithi111.stdout:1/178: dwrite d0/d5/f16 [0,4194304] 0 2024-03-23T11:47:26.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:26 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:26.905 INFO:tasks.workunit.client.0.smithi063.stdout:1/154: getdents d3/d35 0 2024-03-23T11:47:26.906 INFO:tasks.workunit.client.0.smithi063.stdout:8/312: write d6/d1c/f4e [4974719,115616] 0 2024-03-23T11:47:26.906 INFO:tasks.workunit.client.0.smithi063.stdout:9/113: fsync d4/fa 0 2024-03-23T11:47:26.907 INFO:tasks.workunit.client.0.smithi063.stdout:7/157: unlink d5/d6/c23 0 2024-03-23T11:47:26.909 INFO:tasks.workunit.client.1.smithi111.stdout:3/135: unlink d0/d7/ff 0 2024-03-23T11:47:26.911 INFO:tasks.workunit.client.1.smithi111.stdout:3/136: dread - d0/f1f zero size 2024-03-23T11:47:26.911 INFO:tasks.workunit.client.1.smithi111.stdout:5/228: write d1/d3/d6/ff [1346182,129464] 0 2024-03-23T11:47:26.913 INFO:tasks.workunit.client.1.smithi111.stdout:1/179: creat d0/f3d x:0 0 0 2024-03-23T11:47:26.921 INFO:tasks.workunit.client.1.smithi111.stdout:3/137: symlink d0/d7/l21 0 2024-03-23T11:47:26.922 INFO:tasks.workunit.client.0.smithi063.stdout:7/158: mkdir d5/d16/d2f 0 2024-03-23T11:47:26.923 INFO:tasks.workunit.client.0.smithi063.stdout:3/284: dwrite d13/f30 [0,4194304] 0 2024-03-23T11:47:26.926 INFO:tasks.workunit.client.0.smithi063.stdout:8/313: unlink d6/d1f/d54/d5c/l62 0 2024-03-23T11:47:26.928 INFO:tasks.workunit.client.0.smithi063.stdout:6/133: getdents d1/d15 0 2024-03-23T11:47:26.930 INFO:tasks.workunit.client.1.smithi111.stdout:3/138: symlink d0/d1a/l22 0 2024-03-23T11:47:26.931 INFO:tasks.workunit.client.0.smithi063.stdout:1/155: dread d3/d1c/f1e [0,4194304] 0 2024-03-23T11:47:26.933 INFO:tasks.workunit.client.0.smithi063.stdout:0/120: dwrite d2/d14/f23 [0,4194304] 0 2024-03-23T11:47:26.936 INFO:tasks.workunit.client.1.smithi111.stdout:1/180: rename d0/d6/f24 to d0/d2b/f3e 0 2024-03-23T11:47:26.937 INFO:tasks.workunit.client.1.smithi111.stdout:8/133: sync 2024-03-23T11:47:26.937 INFO:tasks.workunit.client.1.smithi111.stdout:6/252: sync 2024-03-23T11:47:26.937 INFO:tasks.workunit.client.1.smithi111.stdout:9/224: sync 2024-03-23T11:47:26.937 INFO:tasks.workunit.client.1.smithi111.stdout:0/178: sync 2024-03-23T11:47:26.938 INFO:tasks.workunit.client.1.smithi111.stdout:0/179: stat d5/db/f26 0 2024-03-23T11:47:26.939 INFO:tasks.workunit.client.0.smithi063.stdout:8/314: chown d6/d1f/d26/f49 10562 1 2024-03-23T11:47:26.939 INFO:tasks.workunit.client.0.smithi063.stdout:3/285: creat d13/f56 x:0 0 0 2024-03-23T11:47:26.939 INFO:tasks.workunit.client.1.smithi111.stdout:9/225: chown d0/d4/d1c/d2d/f2e 246 1 2024-03-23T11:47:26.940 INFO:tasks.workunit.client.0.smithi063.stdout:4/162: dwrite d5/da/fd [0,4194304] 0 2024-03-23T11:47:26.943 INFO:tasks.workunit.client.0.smithi063.stdout:2/147: dwrite d2/d1c/f20 [0,4194304] 0 2024-03-23T11:47:26.946 INFO:tasks.workunit.client.0.smithi063.stdout:7/159: rename d5/d6/c1c to d5/d22/c30 0 2024-03-23T11:47:26.947 INFO:tasks.workunit.client.0.smithi063.stdout:8/315: stat d6/d1c/d27/f34 0 2024-03-23T11:47:26.948 INFO:tasks.workunit.client.0.smithi063.stdout:8/316: stat d6/dd/f56 0 2024-03-23T11:47:26.948 INFO:tasks.workunit.client.0.smithi063.stdout:9/114: dread d4/f14 [4194304,4194304] 0 2024-03-23T11:47:26.955 INFO:tasks.workunit.client.0.smithi063.stdout:8/317: chown d6/d1f/d26/d5d 853149979 1 2024-03-23T11:47:26.956 INFO:tasks.workunit.client.1.smithi111.stdout:0/180: dwrite d5/db/d1f/f28 [0,4194304] 0 2024-03-23T11:47:26.957 INFO:tasks.workunit.client.0.smithi063.stdout:3/286: symlink d13/d22/d2d/l57 0 2024-03-23T11:47:26.962 INFO:tasks.workunit.client.0.smithi063.stdout:5/85: getdents d2/d7 0 2024-03-23T11:47:26.964 INFO:tasks.workunit.client.0.smithi063.stdout:7/160: creat d5/d9/d20/f31 x:0 0 0 2024-03-23T11:47:26.966 INFO:tasks.workunit.client.1.smithi111.stdout:0/181: mkdir d5/db/d1f/d39 0 2024-03-23T11:47:26.967 INFO:tasks.workunit.client.0.smithi063.stdout:0/121: symlink d2/d4/dd/d28/l2b 0 2024-03-23T11:47:26.968 INFO:tasks.workunit.client.0.smithi063.stdout:9/115: fsync d4/f14 0 2024-03-23T11:47:26.968 INFO:tasks.workunit.client.1.smithi111.stdout:4/167: dwrite d0/dc/d30/f15 [0,4194304] 0 2024-03-23T11:47:26.969 INFO:tasks.workunit.client.0.smithi063.stdout:2/148: write d2/f21 [809464,71044] 0 2024-03-23T11:47:26.969 INFO:tasks.workunit.client.0.smithi063.stdout:6/134: dwrite d1/d15/f26 [0,4194304] 0 2024-03-23T11:47:26.972 INFO:tasks.workunit.client.1.smithi111.stdout:1/181: truncate d0/d5/dd/f25 2052851 0 2024-03-23T11:47:26.975 INFO:tasks.workunit.client.1.smithi111.stdout:6/253: creat d2/d1f/f59 x:0 0 0 2024-03-23T11:47:26.976 INFO:tasks.workunit.client.0.smithi063.stdout:6/135: readlink d1/l10 0 2024-03-23T11:47:26.981 INFO:tasks.workunit.client.0.smithi063.stdout:9/116: chown d4/d11/d19/f23 6132 1 2024-03-23T11:47:26.983 INFO:tasks.workunit.client.0.smithi063.stdout:8/318: chown d6/d1c/d27/l5a 452267917 1 2024-03-23T11:47:26.985 INFO:tasks.workunit.client.1.smithi111.stdout:4/168: creat d0/d17/d29/d2a/f38 x:0 0 0 2024-03-23T11:47:26.986 INFO:tasks.workunit.client.1.smithi111.stdout:1/182: mknod d0/d6/c3f 0 2024-03-23T11:47:26.986 INFO:tasks.workunit.client.0.smithi063.stdout:4/163: dwrite d5/da/d20/d24/d16/f1d [0,4194304] 0 2024-03-23T11:47:26.987 INFO:tasks.workunit.client.0.smithi063.stdout:0/122: mknod d2/d4/dd/d1f/c2c 0 2024-03-23T11:47:26.988 INFO:tasks.workunit.client.1.smithi111.stdout:1/183: write d0/f4 [597029,66414] 0 2024-03-23T11:47:26.989 INFO:tasks.workunit.client.1.smithi111.stdout:0/182: dwrite d5/db/f38 [0,4194304] 0 2024-03-23T11:47:26.989 INFO:tasks.workunit.client.0.smithi063.stdout:8/319: fdatasync d6/d1f/f55 0 2024-03-23T11:47:26.990 INFO:tasks.workunit.client.1.smithi111.stdout:1/184: chown d0/l2c 3 1 2024-03-23T11:47:26.991 INFO:tasks.workunit.client.1.smithi111.stdout:0/183: chown d5/db/d1f/f22 16939988 1 2024-03-23T11:47:26.992 INFO:tasks.workunit.client.0.smithi063.stdout:3/287: write fc [5958596,51131] 0 2024-03-23T11:47:26.993 INFO:tasks.workunit.client.0.smithi063.stdout:9/117: creat d4/d11/f24 x:0 0 0 2024-03-23T11:47:27.004 INFO:tasks.workunit.client.1.smithi111.stdout:9/226: getdents d0 0 2024-03-23T11:47:27.005 INFO:tasks.workunit.client.1.smithi111.stdout:8/134: write d5/fa [651016,117318] 0 2024-03-23T11:47:27.006 INFO:tasks.workunit.client.0.smithi063.stdout:8/320: fdatasync d6/dd/df/f4b 0 2024-03-23T11:47:27.006 INFO:tasks.workunit.client.1.smithi111.stdout:8/135: chown d5/d27 2784 1 2024-03-23T11:47:27.007 INFO:tasks.workunit.client.0.smithi063.stdout:4/164: chown d5/da/d20/d24/d29 12303 1 2024-03-23T11:47:27.008 INFO:tasks.workunit.client.1.smithi111.stdout:4/169: creat d0/d17/d29/f39 x:0 0 0 2024-03-23T11:47:27.008 INFO:tasks.workunit.client.1.smithi111.stdout:8/136: write d5/f1f [4368037,107077] 0 2024-03-23T11:47:27.009 INFO:tasks.workunit.client.0.smithi063.stdout:0/123: creat d2/d4/dd/d1f/f2d x:0 0 0 2024-03-23T11:47:27.010 INFO:tasks.workunit.client.1.smithi111.stdout:4/170: fdatasync d0/f1 0 2024-03-23T11:47:27.019 INFO:tasks.workunit.client.0.smithi063.stdout:8/321: write d6/d1c/f4f [1205394,120762] 0 2024-03-23T11:47:27.019 INFO:tasks.workunit.client.0.smithi063.stdout:7/161: dwrite d5/d9/f2e [0,4194304] 0 2024-03-23T11:47:27.020 INFO:tasks.workunit.client.0.smithi063.stdout:0/124: mknod d2/d14/c2e 0 2024-03-23T11:47:27.023 INFO:tasks.workunit.client.1.smithi111.stdout:9/227: dwrite d0/d4/d1c/d36/d28/d34/f39 [0,4194304] 0 2024-03-23T11:47:27.030 INFO:tasks.workunit.client.1.smithi111.stdout:4/171: dwrite d0/d17/d29/d2a/f38 [0,4194304] 0 2024-03-23T11:47:27.030 INFO:tasks.workunit.client.0.smithi063.stdout:2/149: truncate d2/f21 428218 0 2024-03-23T11:47:27.031 INFO:tasks.workunit.client.0.smithi063.stdout:9/118: dread d4/f21 [0,4194304] 0 2024-03-23T11:47:27.034 INFO:tasks.workunit.client.1.smithi111.stdout:2/187: sync 2024-03-23T11:47:27.034 INFO:tasks.workunit.client.0.smithi063.stdout:0/125: creat d2/d4/dd/d1f/f2f x:0 0 0 2024-03-23T11:47:27.034 INFO:tasks.workunit.client.0.smithi063.stdout:8/322: chown d6/d1f/d3e/l40 56 1 2024-03-23T11:47:27.035 INFO:tasks.workunit.client.1.smithi111.stdout:1/185: creat d0/d5/dd/d1e/f40 x:0 0 0 2024-03-23T11:47:27.035 INFO:tasks.workunit.client.1.smithi111.stdout:0/184: symlink d5/db/l3a 0 2024-03-23T11:47:27.040 INFO:tasks.workunit.client.0.smithi063.stdout:6/136: dwrite d1/d15/f26 [0,4194304] 0 2024-03-23T11:47:27.042 INFO:tasks.workunit.client.1.smithi111.stdout:0/185: write d5/db/f15 [1917003,73564] 0 2024-03-23T11:47:27.044 INFO:tasks.workunit.client.1.smithi111.stdout:6/254: chown d2/d1a/d1c/d2c/d51 419189 1 2024-03-23T11:47:27.044 INFO:tasks.workunit.client.1.smithi111.stdout:5/229: link d1/d3/d6/d29/f21 d1/d3/d6/d7/f53 0 2024-03-23T11:47:27.044 INFO:tasks.workunit.client.0.smithi063.stdout:7/162: write d5/d9/da/f17 [3476259,108681] 0 2024-03-23T11:47:27.047 INFO:tasks.workunit.client.1.smithi111.stdout:5/230: fdatasync d1/d3/d6/d7/f36 0 2024-03-23T11:47:27.050 INFO:tasks.workunit.client.0.smithi063.stdout:9/119: write d4/f1e [896932,84292] 0 2024-03-23T11:47:27.050 INFO:tasks.workunit.client.0.smithi063.stdout:7/163: chown d5/d26 770 1 2024-03-23T11:47:27.052 INFO:tasks.workunit.client.0.smithi063.stdout:6/137: truncate d1/d13/f1c 752118 0 2024-03-23T11:47:27.052 INFO:tasks.workunit.client.0.smithi063.stdout:9/120: fsync d4/fa 0 2024-03-23T11:47:27.052 INFO:tasks.workunit.client.0.smithi063.stdout:4/165: dread d5/f1c [0,4194304] 0 2024-03-23T11:47:27.053 INFO:tasks.workunit.client.1.smithi111.stdout:4/172: dread d0/d7/f16 [0,4194304] 0 2024-03-23T11:47:27.054 INFO:tasks.workunit.client.1.smithi111.stdout:1/186: rename d0/d6/d3b to d0/d5/dd/d1e/d41 0 2024-03-23T11:47:27.056 INFO:tasks.workunit.client.0.smithi063.stdout:0/126: unlink d2/f27 0 2024-03-23T11:47:27.058 INFO:tasks.workunit.client.0.smithi063.stdout:6/138: fdatasync d1/dc/f1d 0 2024-03-23T11:47:27.058 INFO:tasks.workunit.client.0.smithi063.stdout:2/150: dread d2/f25 [0,4194304] 0 2024-03-23T11:47:27.059 INFO:tasks.workunit.client.1.smithi111.stdout:1/187: dread - d0/d5/f35 zero size 2024-03-23T11:47:27.060 INFO:tasks.workunit.client.0.smithi063.stdout:8/323: creat d6/d1f/d54/f6a x:0 0 0 2024-03-23T11:47:27.060 INFO:tasks.workunit.client.0.smithi063.stdout:7/164: rmdir d5/d22 39 2024-03-23T11:47:27.063 INFO:tasks.workunit.client.1.smithi111.stdout:1/188: write d0/d6/f22 [1464542,113012] 0 2024-03-23T11:47:27.065 INFO:tasks.workunit.client.1.smithi111.stdout:6/255: dwrite d2/d1f/f26 [0,4194304] 0 2024-03-23T11:47:27.066 INFO:tasks.workunit.client.1.smithi111.stdout:3/139: sync 2024-03-23T11:47:27.066 INFO:tasks.workunit.client.1.smithi111.stdout:7/142: sync 2024-03-23T11:47:27.067 INFO:tasks.workunit.client.0.smithi063.stdout:3/288: dwrite f5 [0,4194304] 0 2024-03-23T11:47:27.067 INFO:tasks.workunit.client.0.smithi063.stdout:0/127: rmdir d2/d4/dd/d28 39 2024-03-23T11:47:27.067 INFO:tasks.workunit.client.0.smithi063.stdout:6/139: chown d1/lb 80 1 2024-03-23T11:47:27.068 INFO:tasks.workunit.client.1.smithi111.stdout:2/188: symlink d2/d11/d17/l46 0 2024-03-23T11:47:27.068 INFO:tasks.workunit.client.1.smithi111.stdout:7/143: write fd [1872960,60303] 0 2024-03-23T11:47:27.069 INFO:tasks.workunit.client.0.smithi063.stdout:9/121: link d4/ff d4/d11/f25 0 2024-03-23T11:47:27.069 INFO:tasks.workunit.client.1.smithi111.stdout:7/144: chown c6 1 1 2024-03-23T11:47:27.070 INFO:tasks.workunit.client.0.smithi063.stdout:6/140: stat d1/dc/f1b 0 2024-03-23T11:47:27.071 INFO:tasks.workunit.client.0.smithi063.stdout:2/151: write d2/de/f2b [241615,103781] 0 2024-03-23T11:47:27.072 INFO:tasks.workunit.client.1.smithi111.stdout:2/189: write d2/d7/da/ff [821638,125501] 0 2024-03-23T11:47:27.074 INFO:tasks.workunit.client.0.smithi063.stdout:8/324: read d6/dd/df/f2c [3342388,80131] 0 2024-03-23T11:47:27.079 INFO:tasks.workunit.client.0.smithi063.stdout:8/325: dread - d6/d1c/d27/f4a zero size 2024-03-23T11:47:27.079 INFO:tasks.workunit.client.1.smithi111.stdout:9/228: link d0/d4/c12 d0/d4/d1c/d36/d28/c4b 0 2024-03-23T11:47:27.080 INFO:tasks.workunit.client.1.smithi111.stdout:0/186: creat d5/db/d1f/d32/f3b x:0 0 0 2024-03-23T11:47:27.081 INFO:tasks.workunit.client.1.smithi111.stdout:9/229: chown d0/d4/c3e 719600352 1 2024-03-23T11:47:27.084 INFO:tasks.workunit.client.0.smithi063.stdout:9/122: mknod d4/d11/c26 0 2024-03-23T11:47:27.084 INFO:tasks.workunit.client.0.smithi063.stdout:0/128: write d2/d4/dd/d1f/f20 [97854,60782] 0 2024-03-23T11:47:27.084 INFO:tasks.workunit.client.0.smithi063.stdout:2/152: creat d2/d1c/f30 x:0 0 0 2024-03-23T11:47:27.089 INFO:tasks.workunit.client.0.smithi063.stdout:4/166: dread d5/f1b [0,4194304] 0 2024-03-23T11:47:27.090 INFO:tasks.workunit.client.1.smithi111.stdout:8/137: getdents d5/d1b 0 2024-03-23T11:47:27.090 INFO:tasks.workunit.client.1.smithi111.stdout:2/190: unlink d2/de/d27/d2e/f39 0 2024-03-23T11:47:27.092 INFO:tasks.workunit.client.0.smithi063.stdout:3/289: unlink d13/d22/l23 0 2024-03-23T11:47:27.094 INFO:tasks.workunit.client.0.smithi063.stdout:9/123: rmdir d4/d11/d19 39 2024-03-23T11:47:27.098 INFO:tasks.workunit.client.1.smithi111.stdout:9/230: dwrite d0/f48 [0,4194304] 0 2024-03-23T11:47:27.099 INFO:tasks.workunit.client.0.smithi063.stdout:9/124: write d4/fc [785070,119251] 0 2024-03-23T11:47:27.102 INFO:tasks.workunit.client.0.smithi063.stdout:2/153: symlink d2/l31 0 2024-03-23T11:47:27.105 INFO:tasks.workunit.client.0.smithi063.stdout:4/167: creat d5/da/f2a x:0 0 0 2024-03-23T11:47:27.106 INFO:tasks.workunit.client.0.smithi063.stdout:4/168: fdatasync d5/fc 0 2024-03-23T11:47:27.108 INFO:tasks.workunit.client.1.smithi111.stdout:4/173: truncate d0/dc/fe 30078 0 2024-03-23T11:47:27.109 INFO:tasks.workunit.client.0.smithi063.stdout:8/326: truncate d6/d1c/f3c 1212520 0 2024-03-23T11:47:27.111 INFO:tasks.workunit.client.0.smithi063.stdout:3/290: dread d13/f4d [0,4194304] 0 2024-03-23T11:47:27.113 INFO:tasks.workunit.client.0.smithi063.stdout:8/327: write d6/f43 [872501,40665] 0 2024-03-23T11:47:27.114 INFO:tasks.workunit.client.0.smithi063.stdout:0/129: link d2/d4/dd/ff d2/d4/f30 0 2024-03-23T11:47:27.118 INFO:tasks.workunit.client.1.smithi111.stdout:5/231: creat d1/d3/d6/d29/f54 x:0 0 0 2024-03-23T11:47:27.119 INFO:tasks.workunit.client.1.smithi111.stdout:3/140: dwrite d0/fd [0,4194304] 0 2024-03-23T11:47:27.119 INFO:tasks.workunit.client.0.smithi063.stdout:9/125: creat d4/d11/d19/f27 x:0 0 0 2024-03-23T11:47:27.120 INFO:tasks.workunit.client.0.smithi063.stdout:2/154: symlink d2/l32 0 2024-03-23T11:47:27.121 INFO:tasks.workunit.client.0.smithi063.stdout:3/291: mkdir d13/d22/d2d/d3f/d58 0 2024-03-23T11:47:27.124 INFO:tasks.workunit.client.0.smithi063.stdout:8/328: fdatasync d6/d1f/d54/d5c/d66/f4c 0 2024-03-23T11:47:27.124 INFO:tasks.workunit.client.1.smithi111.stdout:6/256: link d2/d17/f33 d2/d17/f5a 0 2024-03-23T11:47:27.125 INFO:tasks.workunit.client.1.smithi111.stdout:2/191: mknod d2/de/c47 0 2024-03-23T11:47:27.126 INFO:tasks.workunit.client.0.smithi063.stdout:9/126: symlink d4/l28 0 2024-03-23T11:47:27.126 INFO:tasks.workunit.client.0.smithi063.stdout:2/155: chown d2/d1c 0 1 2024-03-23T11:47:27.127 INFO:tasks.workunit.client.0.smithi063.stdout:3/292: mknod d13/d15/c59 0 2024-03-23T11:47:27.129 INFO:tasks.workunit.client.1.smithi111.stdout:8/138: link d5/fa d5/d1b/f2b 0 2024-03-23T11:47:27.129 INFO:tasks.workunit.client.1.smithi111.stdout:1/189: rmdir d0/d5/dd/d1e/d41 0 2024-03-23T11:47:27.129 INFO:tasks.workunit.client.1.smithi111.stdout:0/187: link d5/db/d1f/c34 d5/db/c3c 0 2024-03-23T11:47:27.130 INFO:tasks.workunit.client.1.smithi111.stdout:9/231: dwrite d0/d4/d1c/d36/d28/f3f [0,4194304] 0 2024-03-23T11:47:27.131 INFO:tasks.workunit.client.0.smithi063.stdout:9/127: unlink d4/l28 0 2024-03-23T11:47:27.134 INFO:tasks.workunit.client.0.smithi063.stdout:8/329: mknod d6/d1c/c6b 0 2024-03-23T11:47:27.135 INFO:tasks.workunit.client.0.smithi063.stdout:9/128: fsync d4/f1e 0 2024-03-23T11:47:27.137 INFO:tasks.workunit.client.1.smithi111.stdout:5/232: chown d1/d3/d6/d30 4 1 2024-03-23T11:47:27.137 INFO:tasks.workunit.client.1.smithi111.stdout:3/141: creat d0/d7/f23 x:0 0 0 2024-03-23T11:47:27.139 INFO:tasks.workunit.client.0.smithi063.stdout:9/129: fdatasync d4/d11/d19/f27 0 2024-03-23T11:47:27.142 INFO:tasks.workunit.client.1.smithi111.stdout:3/142: readlink d0/d1a/l22 0 2024-03-23T11:47:27.142 INFO:tasks.workunit.client.1.smithi111.stdout:0/188: write d5/db/f16 [1420230,76343] 0 2024-03-23T11:47:27.142 INFO:tasks.workunit.client.1.smithi111.stdout:7/145: dwrite f12 [0,4194304] 0 2024-03-23T11:47:27.146 INFO:tasks.workunit.client.0.smithi063.stdout:7/165: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:27.146 INFO:tasks.workunit.client.1.smithi111.stdout:4/174: dread d0/dc/f1c [0,4194304] 0 2024-03-23T11:47:27.146 INFO:tasks.workunit.client.1.smithi111.stdout:7/146: fdatasync fb 0 2024-03-23T11:47:27.148 INFO:tasks.workunit.client.0.smithi063.stdout:4/169: dwrite d5/da/d20/d24/d16/f1d [0,4194304] 0 2024-03-23T11:47:27.149 INFO:tasks.workunit.client.0.smithi063.stdout:0/130: dread f1 [0,4194304] 0 2024-03-23T11:47:27.153 INFO:tasks.workunit.client.1.smithi111.stdout:4/175: write d0/f1 [88586,15299] 0 2024-03-23T11:47:27.153 INFO:tasks.workunit.client.0.smithi063.stdout:7/166: fdatasync d5/d6/fe 0 2024-03-23T11:47:27.159 INFO:tasks.workunit.client.1.smithi111.stdout:2/192: rename d2/de/c47 to d2/d11/d17/d3e/c48 0 2024-03-23T11:47:27.162 INFO:tasks.workunit.client.1.smithi111.stdout:8/139: symlink d5/l2c 0 2024-03-23T11:47:27.163 INFO:tasks.workunit.client.1.smithi111.stdout:9/232: rmdir d0/d4/d1c/d36/d28/d34 39 2024-03-23T11:47:27.167 INFO:tasks.workunit.client.1.smithi111.stdout:2/193: chown d2/c1c 41504 1 2024-03-23T11:47:27.168 INFO:tasks.workunit.client.1.smithi111.stdout:9/233: fdatasync d0/d4/d10/f2c 0 2024-03-23T11:47:27.171 INFO:tasks.workunit.client.1.smithi111.stdout:5/233: dread d1/d3/d23/d2c/f2d [0,4194304] 0 2024-03-23T11:47:27.173 INFO:tasks.workunit.client.1.smithi111.stdout:0/189: creat d5/db/d1f/d39/f3d x:0 0 0 2024-03-23T11:47:27.173 INFO:tasks.workunit.client.1.smithi111.stdout:8/140: creat d5/f2d x:0 0 0 2024-03-23T11:47:27.173 INFO:tasks.workunit.client.1.smithi111.stdout:1/190: dread d0/d6/f22 [0,4194304] 0 2024-03-23T11:47:27.175 INFO:tasks.workunit.client.1.smithi111.stdout:9/234: rename d0 to d0/d31/d41/d44/d4c 22 2024-03-23T11:47:27.175 INFO:tasks.workunit.client.0.smithi063.stdout:8/330: truncate d6/d1f/d26/f49 336508 0 2024-03-23T11:47:27.175 INFO:tasks.workunit.client.0.smithi063.stdout:2/156: dwrite d2/de/ff [0,4194304] 0 2024-03-23T11:47:27.177 INFO:tasks.workunit.client.0.smithi063.stdout:4/170: dread d5/fc [4194304,4194304] 0 2024-03-23T11:47:27.177 INFO:tasks.workunit.client.1.smithi111.stdout:8/141: stat d5/l2c 0 2024-03-23T11:47:27.177 INFO:tasks.workunit.client.0.smithi063.stdout:0/131: dwrite d2/d4/dd/d1f/f20 [0,4194304] 0 2024-03-23T11:47:27.178 INFO:tasks.workunit.client.1.smithi111.stdout:5/234: write d1/d3/d6/ff [850557,111935] 0 2024-03-23T11:47:27.180 INFO:tasks.workunit.client.1.smithi111.stdout:3/143: dread d0/d1a/f1c [0,4194304] 0 2024-03-23T11:47:27.181 INFO:tasks.workunit.client.0.smithi063.stdout:7/167: mkdir d5/d26/d32 0 2024-03-23T11:47:27.181 INFO:tasks.workunit.client.1.smithi111.stdout:8/142: read d5/d1b/f2b [665341,33244] 0 2024-03-23T11:47:27.182 INFO:tasks.workunit.client.0.smithi063.stdout:0/132: chown d2/d4/dd/ff 8160320 1 2024-03-23T11:47:27.182 INFO:tasks.workunit.client.0.smithi063.stdout:9/130: write d4/ff [1115132,48928] 0 2024-03-23T11:47:27.182 INFO:tasks.workunit.client.1.smithi111.stdout:6/257: dwrite d2/f57 [4194304,4194304] 0 2024-03-23T11:47:27.182 INFO:tasks.workunit.client.0.smithi063.stdout:2/157: truncate d2/de/f15 999546 0 2024-03-23T11:47:27.183 INFO:tasks.workunit.client.1.smithi111.stdout:2/194: link d2/f14 d2/d7/da/d19/f49 0 2024-03-23T11:47:27.185 INFO:tasks.workunit.client.0.smithi063.stdout:4/171: mkdir d5/d1e/d2b 0 2024-03-23T11:47:27.186 INFO:tasks.workunit.client.1.smithi111.stdout:1/191: creat d0/d6/f42 x:0 0 0 2024-03-23T11:47:27.186 INFO:tasks.workunit.client.0.smithi063.stdout:8/331: creat d6/d1c/f6c x:0 0 0 2024-03-23T11:47:27.186 INFO:tasks.workunit.client.1.smithi111.stdout:5/235: truncate d1/d3/d23/d2c/f2d 1701625 0 2024-03-23T11:47:27.187 INFO:tasks.workunit.client.1.smithi111.stdout:0/190: creat d5/db/d1f/d32/f3e x:0 0 0 2024-03-23T11:47:27.188 INFO:tasks.workunit.client.0.smithi063.stdout:8/332: fsync f5 0 2024-03-23T11:47:27.189 INFO:tasks.workunit.client.0.smithi063.stdout:0/133: readlink d2/d4/dd/d28/l2b 0 2024-03-23T11:47:27.190 INFO:tasks.workunit.client.0.smithi063.stdout:9/131: symlink d4/d11/d19/l29 0 2024-03-23T11:47:27.192 INFO:tasks.workunit.client.0.smithi063.stdout:0/134: stat d2/d14/f15 0 2024-03-23T11:47:27.193 INFO:tasks.workunit.client.1.smithi111.stdout:9/235: creat d0/d31/d35/f4d x:0 0 0 2024-03-23T11:47:27.194 INFO:tasks.workunit.client.0.smithi063.stdout:7/168: mknod d5/c33 0 2024-03-23T11:47:27.195 INFO:tasks.workunit.client.1.smithi111.stdout:3/144: rename d0/d7/f18 to d0/d1a/d1d/f24 0 2024-03-23T11:47:27.197 INFO:tasks.workunit.client.1.smithi111.stdout:1/192: symlink d0/d6/l43 0 2024-03-23T11:47:27.200 INFO:tasks.workunit.client.0.smithi063.stdout:2/158: creat d2/d27/f33 x:0 0 0 2024-03-23T11:47:27.200 INFO:tasks.workunit.client.1.smithi111.stdout:5/236: fdatasync d1/d3/d6/f19 0 2024-03-23T11:47:27.202 INFO:tasks.workunit.client.0.smithi063.stdout:4/172: rename d5/f1b to d5/da/d20/d24/d16/f2c 0 2024-03-23T11:47:27.206 INFO:tasks.workunit.client.1.smithi111.stdout:6/258: mknod d2/d1a/c5b 0 2024-03-23T11:47:27.207 INFO:tasks.workunit.client.1.smithi111.stdout:9/236: mknod d0/d31/d41/c4e 0 2024-03-23T11:47:27.207 INFO:tasks.workunit.client.0.smithi063.stdout:4/173: fsync d5/da/d20/d24/d1f/f21 0 2024-03-23T11:47:27.208 INFO:tasks.workunit.client.1.smithi111.stdout:8/143: dwrite d5/f17 [0,4194304] 0 2024-03-23T11:47:27.208 INFO:tasks.workunit.client.0.smithi063.stdout:0/135: rename d2/d4/dd/ff to d2/d14/f31 0 2024-03-23T11:47:27.209 INFO:tasks.workunit.client.1.smithi111.stdout:5/237: fdatasync d1/d3/d6/d7/f1f 0 2024-03-23T11:47:27.210 INFO:tasks.workunit.client.1.smithi111.stdout:1/193: write d0/d2b/f3e [3234140,40686] 0 2024-03-23T11:47:27.211 INFO:tasks.workunit.client.1.smithi111.stdout:3/145: dread d0/d7/f12 [0,4194304] 0 2024-03-23T11:47:27.214 INFO:tasks.workunit.client.0.smithi063.stdout:2/159: mknod d2/de/d22/c34 0 2024-03-23T11:47:27.216 INFO:tasks.workunit.client.1.smithi111.stdout:5/238: write d1/d3/d23/d2c/f52 [15800,42292] 0 2024-03-23T11:47:27.224 INFO:tasks.workunit.client.0.smithi063.stdout:4/174: creat d5/da/d20/d24/f2d x:0 0 0 2024-03-23T11:47:27.226 INFO:tasks.workunit.client.0.smithi063.stdout:9/132: write d4/f5 [1416584,35677] 0 2024-03-23T11:47:27.228 INFO:tasks.workunit.client.1.smithi111.stdout:2/195: dread d2/d7/da/d19/f49 [0,4194304] 0 2024-03-23T11:47:27.229 INFO:tasks.workunit.client.0.smithi063.stdout:4/175: fdatasync d5/da/fd 0 2024-03-23T11:47:27.231 INFO:tasks.workunit.client.0.smithi063.stdout:9/133: read - d4/f20 zero size 2024-03-23T11:47:27.232 INFO:tasks.workunit.client.0.smithi063.stdout:2/160: creat d2/d23/d2e/f35 x:0 0 0 2024-03-23T11:47:27.232 INFO:tasks.workunit.client.1.smithi111.stdout:6/259: dwrite d2/d35/f40 [0,4194304] 0 2024-03-23T11:47:27.234 INFO:tasks.workunit.client.1.smithi111.stdout:5/239: mkdir d1/d55 0 2024-03-23T11:47:27.238 INFO:tasks.workunit.client.1.smithi111.stdout:9/237: mkdir d0/d4/d1c/d2d/d3b/d42/d4f 0 2024-03-23T11:47:27.239 INFO:tasks.workunit.client.0.smithi063.stdout:9/134: creat d4/f2a x:0 0 0 2024-03-23T11:47:27.244 INFO:tasks.workunit.client.0.smithi063.stdout:8/333: dwrite d6/d1f/d26/f35 [0,4194304] 0 2024-03-23T11:47:27.245 INFO:tasks.workunit.client.0.smithi063.stdout:4/176: symlink d5/da/l2e 0 2024-03-23T11:47:27.245 INFO:tasks.workunit.client.0.smithi063.stdout:9/135: fsync d4/d11/f25 0 2024-03-23T11:47:27.247 INFO:tasks.workunit.client.1.smithi111.stdout:2/196: symlink d2/d11/d17/d1f/l4a 0 2024-03-23T11:47:27.250 INFO:tasks.workunit.client.1.smithi111.stdout:5/240: dwrite d1/d3/d6/f19 [0,4194304] 0 2024-03-23T11:47:27.250 INFO:tasks.workunit.client.1.smithi111.stdout:3/146: link d0/l2 d0/d7/l25 0 2024-03-23T11:47:27.254 INFO:tasks.workunit.client.0.smithi063.stdout:4/177: fdatasync d5/da/d20/d24/d16/f1d 0 2024-03-23T11:47:27.254 INFO:tasks.workunit.client.0.smithi063.stdout:9/136: symlink d4/d11/d19/l2b 0 2024-03-23T11:47:27.255 INFO:tasks.workunit.client.0.smithi063.stdout:2/161: link d2/d27/f33 d2/d23/d2e/f36 0 2024-03-23T11:47:27.257 INFO:tasks.workunit.client.0.smithi063.stdout:0/136: dwrite d2/d14/f19 [0,4194304] 0 2024-03-23T11:47:27.258 INFO:tasks.workunit.client.1.smithi111.stdout:6/260: symlink d2/d17/l5c 0 2024-03-23T11:47:27.259 INFO:tasks.workunit.client.1.smithi111.stdout:0/191: dread f4 [0,4194304] 0 2024-03-23T11:47:27.260 INFO:tasks.workunit.client.0.smithi063.stdout:7/169: dread d5/d9/da/f17 [0,4194304] 0 2024-03-23T11:47:27.265 INFO:tasks.workunit.client.0.smithi063.stdout:4/178: mknod d5/da/c2f 0 2024-03-23T11:47:27.265 INFO:tasks.workunit.client.0.smithi063.stdout:9/137: mkdir d4/d2c 0 2024-03-23T11:47:27.267 INFO:tasks.workunit.client.0.smithi063.stdout:9/138: chown f2 420469193 1 2024-03-23T11:47:27.268 INFO:tasks.workunit.client.1.smithi111.stdout:9/238: mkdir d0/d31/d41/d44/d50 0 2024-03-23T11:47:27.270 INFO:tasks.workunit.client.0.smithi063.stdout:2/162: fdatasync d2/f21 0 2024-03-23T11:47:27.272 INFO:tasks.workunit.client.0.smithi063.stdout:8/334: rename d6/d1c/f1e to d6/d1f/d54/d5c/f6d 0 2024-03-23T11:47:27.273 INFO:tasks.workunit.client.1.smithi111.stdout:9/239: dread - d0/d4/f47 zero size 2024-03-23T11:47:27.273 INFO:tasks.workunit.client.1.smithi111.stdout:5/241: mknod d1/d3/d6/c56 0 2024-03-23T11:47:27.277 INFO:tasks.workunit.client.1.smithi111.stdout:2/197: dwrite d2/d11/d17/d1f/f20 [0,4194304] 0 2024-03-23T11:47:27.278 INFO:tasks.workunit.client.0.smithi063.stdout:0/137: dread d2/d14/f19 [0,4194304] 0 2024-03-23T11:47:27.278 INFO:tasks.workunit.client.1.smithi111.stdout:8/144: dwrite f3 [4194304,4194304] 0 2024-03-23T11:47:27.279 INFO:tasks.workunit.client.1.smithi111.stdout:3/147: dwrite d0/d7/f1e [0,4194304] 0 2024-03-23T11:47:27.287 INFO:tasks.workunit.client.0.smithi063.stdout:8/335: mknod d6/d1c/c6e 0 2024-03-23T11:47:27.289 INFO:tasks.workunit.client.0.smithi063.stdout:9/139: unlink d4/c15 0 2024-03-23T11:47:27.290 INFO:tasks.workunit.client.0.smithi063.stdout:8/336: write d6/d1c/d27/f34 [520487,68090] 0 2024-03-23T11:47:27.290 INFO:tasks.workunit.client.0.smithi063.stdout:4/179: rmdir d5/da/d20/d24 39 2024-03-23T11:47:27.291 INFO:tasks.workunit.client.1.smithi111.stdout:2/198: fdatasync d2/f14 0 2024-03-23T11:47:27.291 INFO:tasks.workunit.client.1.smithi111.stdout:5/242: rmdir d1/d3/d3a 39 2024-03-23T11:47:27.292 INFO:tasks.workunit.client.0.smithi063.stdout:9/140: stat d4/d2c 0 2024-03-23T11:47:27.292 INFO:tasks.workunit.client.1.smithi111.stdout:8/145: symlink d5/l2e 0 2024-03-23T11:47:27.292 INFO:tasks.workunit.client.1.smithi111.stdout:9/240: rmdir d0/d4/d1c/d2d/d3b/d42 39 2024-03-23T11:47:27.293 INFO:tasks.workunit.client.1.smithi111.stdout:3/148: mkdir d0/d1a/d1d/d26 0 2024-03-23T11:47:27.293 INFO:tasks.workunit.client.0.smithi063.stdout:9/141: stat d4/f1e 0 2024-03-23T11:47:27.300 INFO:tasks.workunit.client.1.smithi111.stdout:3/149: write d0/d7/f23 [464376,117743] 0 2024-03-23T11:47:27.300 INFO:tasks.workunit.client.1.smithi111.stdout:5/243: fdatasync d1/d3/d23/d2c/f52 0 2024-03-23T11:47:27.300 INFO:tasks.workunit.client.0.smithi063.stdout:2/163: creat d2/f37 x:0 0 0 2024-03-23T11:47:27.303 INFO:tasks.workunit.client.0.smithi063.stdout:9/142: unlink d4/fa 0 2024-03-23T11:47:27.304 INFO:tasks.workunit.client.1.smithi111.stdout:2/199: mkdir d2/d32/d4b 0 2024-03-23T11:47:27.304 INFO:tasks.workunit.client.1.smithi111.stdout:3/150: write d0/d7/f23 [751446,104691] 0 2024-03-23T11:47:27.306 INFO:tasks.workunit.client.1.smithi111.stdout:9/241: readlink d0/d4/d1c/d36/l8 0 2024-03-23T11:47:27.308 INFO:tasks.workunit.client.0.smithi063.stdout:4/180: symlink d5/da/d20/d24/d29/l30 0 2024-03-23T11:47:27.309 INFO:tasks.workunit.client.1.smithi111.stdout:0/192: write d5/db/d1f/f1a [1936615,108673] 0 2024-03-23T11:47:27.312 INFO:tasks.workunit.client.1.smithi111.stdout:2/200: symlink d2/de/d27/l4c 0 2024-03-23T11:47:27.312 INFO:tasks.workunit.client.0.smithi063.stdout:9/143: rename d4/d11/f12 to d4/d11/d19/f2d 0 2024-03-23T11:47:27.316 INFO:tasks.workunit.client.0.smithi063.stdout:2/164: getdents d2/d27 0 2024-03-23T11:47:27.317 INFO:tasks.workunit.client.1.smithi111.stdout:9/242: truncate d0/d4/d1c/d2d/f2e 396012 0 2024-03-23T11:47:27.317 INFO:tasks.workunit.client.1.smithi111.stdout:3/151: read d0/d1a/d1d/f24 [16277,102634] 0 2024-03-23T11:47:27.318 INFO:tasks.workunit.client.1.smithi111.stdout:8/146: dwrite f1 [0,4194304] 0 2024-03-23T11:47:27.319 INFO:tasks.workunit.client.0.smithi063.stdout:0/138: write d2/d12/f21 [1390254,19599] 0 2024-03-23T11:47:27.320 INFO:tasks.workunit.client.0.smithi063.stdout:7/170: truncate d5/f8 1940483 0 2024-03-23T11:47:27.321 INFO:tasks.workunit.client.0.smithi063.stdout:2/165: rename d2/l9 to d2/d27/l38 0 2024-03-23T11:47:27.322 INFO:tasks.workunit.client.0.smithi063.stdout:0/139: chown d2/d14/f1c 96143481 1 2024-03-23T11:47:27.323 INFO:tasks.workunit.client.0.smithi063.stdout:7/171: mknod d5/d16/c34 0 2024-03-23T11:47:27.323 INFO:tasks.workunit.client.0.smithi063.stdout:2/166: fsync d2/de/f1b 0 2024-03-23T11:47:27.324 INFO:tasks.workunit.client.1.smithi111.stdout:2/201: link d2/f4 d2/de/d27/d2e/d3f/f4d 0 2024-03-23T11:47:27.325 INFO:tasks.workunit.client.1.smithi111.stdout:8/147: truncate d5/f22 968616 0 2024-03-23T11:47:27.326 INFO:tasks.workunit.client.1.smithi111.stdout:9/243: chown d0/d4/d1c/d2d/f2e 1 1 2024-03-23T11:47:27.327 INFO:tasks.workunit.client.1.smithi111.stdout:3/152: rename d0/cc to d0/d7/d19/c27 0 2024-03-23T11:47:27.328 INFO:tasks.workunit.client.0.smithi063.stdout:0/140: mkdir d2/d12/d32 0 2024-03-23T11:47:27.329 INFO:tasks.workunit.client.1.smithi111.stdout:6/261: dread d2/ff [0,4194304] 0 2024-03-23T11:47:27.331 INFO:tasks.workunit.client.1.smithi111.stdout:0/193: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:27.331 INFO:tasks.workunit.client.1.smithi111.stdout:5/244: dread d1/d3/d6/ff [0,4194304] 0 2024-03-23T11:47:27.332 INFO:tasks.workunit.client.1.smithi111.stdout:2/202: creat d2/d11/d17/d3e/f4e x:0 0 0 2024-03-23T11:47:27.332 INFO:tasks.workunit.client.0.smithi063.stdout:7/172: creat d5/d9/d20/f35 x:0 0 0 2024-03-23T11:47:27.333 INFO:tasks.workunit.client.1.smithi111.stdout:3/153: unlink d0/d7/c14 0 2024-03-23T11:47:27.333 INFO:tasks.workunit.client.1.smithi111.stdout:0/194: truncate d5/f2d 8577132 0 2024-03-23T11:47:27.334 INFO:tasks.workunit.client.1.smithi111.stdout:6/262: dread - d2/f3b zero size 2024-03-23T11:47:27.334 INFO:tasks.workunit.client.1.smithi111.stdout:3/154: chown d0/c1b 6822210 1 2024-03-23T11:47:27.336 INFO:tasks.workunit.client.0.smithi063.stdout:9/144: dwrite d4/f14 [0,4194304] 0 2024-03-23T11:47:27.336 INFO:tasks.workunit.client.1.smithi111.stdout:8/148: fsync d5/f11 0 2024-03-23T11:47:27.337 INFO:tasks.workunit.client.0.smithi063.stdout:7/173: write d5/d9/d20/f31 [429419,21520] 0 2024-03-23T11:47:27.338 INFO:tasks.workunit.client.1.smithi111.stdout:8/149: stat d5/fe 0 2024-03-23T11:47:27.339 INFO:tasks.workunit.client.1.smithi111.stdout:0/195: creat d5/db/d1f/d39/f3f x:0 0 0 2024-03-23T11:47:27.339 INFO:tasks.workunit.client.0.smithi063.stdout:2/167: unlink d2/c14 0 2024-03-23T11:47:27.340 INFO:tasks.workunit.client.1.smithi111.stdout:6/263: dread d2/d17/f5a [0,4194304] 0 2024-03-23T11:47:27.340 INFO:tasks.workunit.client.1.smithi111.stdout:3/155: creat d0/d1a/f28 x:0 0 0 2024-03-23T11:47:27.341 INFO:tasks.workunit.client.1.smithi111.stdout:0/196: chown f4 50775 1 2024-03-23T11:47:27.344 INFO:tasks.workunit.client.0.smithi063.stdout:8/337: dread d6/d1f/d26/f49 [0,4194304] 0 2024-03-23T11:47:27.345 INFO:tasks.workunit.client.1.smithi111.stdout:9/244: link d0/d4/d1c/d36/f18 d0/d4/d10/f51 0 2024-03-23T11:47:27.351 INFO:tasks.workunit.client.1.smithi111.stdout:6/264: truncate d2/d17/f5a 1072520 0 2024-03-23T11:47:27.351 INFO:tasks.workunit.client.0.smithi063.stdout:7/174: mknod d5/d26/d32/c36 0 2024-03-23T11:47:27.353 INFO:tasks.workunit.client.0.smithi063.stdout:8/338: rmdir d6/dd 39 2024-03-23T11:47:27.354 INFO:tasks.workunit.client.1.smithi111.stdout:2/203: rename d2/de/f16 to d2/f4f 0 2024-03-23T11:47:27.355 INFO:tasks.workunit.client.0.smithi063.stdout:0/141: rmdir d2 39 2024-03-23T11:47:27.355 INFO:tasks.workunit.client.1.smithi111.stdout:8/150: getdents d5/d26 0 2024-03-23T11:47:27.357 INFO:tasks.workunit.client.1.smithi111.stdout:3/156: dwrite d0/d7/f23 [0,4194304] 0 2024-03-23T11:47:27.359 INFO:tasks.workunit.client.1.smithi111.stdout:3/157: fsync d0/d7/f10 0 2024-03-23T11:47:27.359 INFO:tasks.workunit.client.1.smithi111.stdout:8/151: read d5/f1f [3107555,118985] 0 2024-03-23T11:47:27.360 INFO:tasks.workunit.client.1.smithi111.stdout:8/152: stat d5/d1b/c23 0 2024-03-23T11:47:27.361 INFO:tasks.workunit.client.0.smithi063.stdout:2/168: dwrite f1 [0,4194304] 0 2024-03-23T11:47:27.362 INFO:tasks.workunit.client.1.smithi111.stdout:8/153: fsync d5/fe 0 2024-03-23T11:47:27.362 INFO:tasks.workunit.client.1.smithi111.stdout:8/154: readlink d5/l24 0 2024-03-23T11:47:27.363 INFO:tasks.workunit.client.0.smithi063.stdout:9/145: dwrite d4/f2a [0,4194304] 0 2024-03-23T11:47:27.367 INFO:tasks.workunit.client.0.smithi063.stdout:9/146: read d4/f16 [1516613,46444] 0 2024-03-23T11:47:27.372 INFO:tasks.workunit.client.0.smithi063.stdout:8/339: creat d6/d1f/d26/d15/f6f x:0 0 0 2024-03-23T11:47:27.377 INFO:tasks.workunit.client.0.smithi063.stdout:0/142: write d2/d14/f31 [2218995,60219] 0 2024-03-23T11:47:27.379 INFO:tasks.workunit.client.1.smithi111.stdout:9/245: symlink d0/d4/d1c/d2d/d3b/d42/d4f/l52 0 2024-03-23T11:47:27.380 INFO:tasks.workunit.client.0.smithi063.stdout:9/147: mknod d4/d2c/c2e 0 2024-03-23T11:47:27.380 INFO:tasks.workunit.client.1.smithi111.stdout:3/158: rmdir d0 39 2024-03-23T11:47:27.381 INFO:tasks.workunit.client.1.smithi111.stdout:9/246: fdatasync d0/d4/f47 0 2024-03-23T11:47:27.384 INFO:tasks.workunit.client.0.smithi063.stdout:7/175: dwrite d5/d9/d20/f31 [0,4194304] 0 2024-03-23T11:47:27.385 INFO:tasks.workunit.client.1.smithi111.stdout:2/204: truncate d2/f14 1622028 0 2024-03-23T11:47:27.387 INFO:tasks.workunit.client.1.smithi111.stdout:2/205: write d2/d7/f38 [886330,59337] 0 2024-03-23T11:47:27.389 INFO:tasks.workunit.client.1.smithi111.stdout:2/206: chown d2/d32/d4b 79255799 1 2024-03-23T11:47:27.389 INFO:tasks.workunit.client.0.smithi063.stdout:9/148: symlink d4/d2c/l2f 0 2024-03-23T11:47:27.390 INFO:tasks.workunit.client.0.smithi063.stdout:8/340: mknod d6/dd/c70 0 2024-03-23T11:47:27.391 INFO:tasks.workunit.client.1.smithi111.stdout:8/155: mkdir d5/d26/d2f 0 2024-03-23T11:47:27.391 INFO:tasks.workunit.client.1.smithi111.stdout:2/207: readlink d2/de/d27/d2e/l42 0 2024-03-23T11:47:27.394 INFO:tasks.workunit.client.0.smithi063.stdout:0/143: creat d2/d4/dd/f33 x:0 0 0 2024-03-23T11:47:27.397 INFO:tasks.workunit.client.0.smithi063.stdout:8/341: creat d6/f71 x:0 0 0 2024-03-23T11:47:27.397 INFO:tasks.workunit.client.1.smithi111.stdout:9/247: mknod d0/d4/d1c/d36/d28/d34/c53 0 2024-03-23T11:47:27.399 INFO:tasks.workunit.client.1.smithi111.stdout:9/248: readlink d0/d4/d10/l1f 0 2024-03-23T11:47:27.399 INFO:tasks.workunit.client.0.smithi063.stdout:0/144: creat d2/d14/f34 x:0 0 0 2024-03-23T11:47:27.399 INFO:tasks.workunit.client.0.smithi063.stdout:8/342: chown d6/d1f/d26/f49 15 1 2024-03-23T11:47:27.402 INFO:tasks.workunit.client.0.smithi063.stdout:9/149: rename d4/d11/f24 to d4/d11/f30 0 2024-03-23T11:47:27.403 INFO:tasks.workunit.client.0.smithi063.stdout:8/343: truncate d6/d1f/f55 5916534 0 2024-03-23T11:47:27.405 INFO:tasks.workunit.client.1.smithi111.stdout:2/208: rename d2/f14 to d2/de/d27/f50 0 2024-03-23T11:47:27.406 INFO:tasks.workunit.client.1.smithi111.stdout:6/265: write d2/d17/f33 [852797,15051] 0 2024-03-23T11:47:27.407 INFO:tasks.workunit.client.0.smithi063.stdout:0/145: unlink d2/d4/dd/l24 0 2024-03-23T11:47:27.407 INFO:tasks.workunit.client.0.smithi063.stdout:8/344: write d6/d1f/d54/f6a [377444,41448] 0 2024-03-23T11:47:27.407 INFO:tasks.workunit.client.1.smithi111.stdout:8/156: dread d5/f6 [0,4194304] 0 2024-03-23T11:47:27.408 INFO:tasks.workunit.client.0.smithi063.stdout:8/345: chown d6/d1f/d54/d63 285936177 1 2024-03-23T11:47:27.409 INFO:tasks.workunit.client.0.smithi063.stdout:9/150: mkdir d4/d31 0 2024-03-23T11:47:27.411 INFO:tasks.workunit.client.0.smithi063.stdout:9/151: readlink d4/d11/d19/l1d 0 2024-03-23T11:47:27.411 INFO:tasks.workunit.client.0.smithi063.stdout:0/146: mkdir d2/d12/d35 0 2024-03-23T11:47:27.411 INFO:tasks.workunit.client.1.smithi111.stdout:2/209: unlink d2/d11/d17/d1f/c3a 0 2024-03-23T11:47:27.412 INFO:tasks.workunit.client.1.smithi111.stdout:7/147: sync 2024-03-23T11:47:27.413 INFO:tasks.workunit.client.0.smithi063.stdout:8/346: chown d6/d1f/d26/f10 192030900 1 2024-03-23T11:47:27.413 INFO:tasks.workunit.client.1.smithi111.stdout:4/176: sync 2024-03-23T11:47:27.417 INFO:tasks.workunit.client.0.smithi063.stdout:0/147: mkdir d2/d14/d36 0 2024-03-23T11:47:27.419 INFO:tasks.workunit.client.0.smithi063.stdout:7/176: dread f1 [0,4194304] 0 2024-03-23T11:47:27.420 INFO:tasks.workunit.client.0.smithi063.stdout:8/347: rename d6/d1f/d26/d15/f6f to d6/d1f/d54/d5c/d66/f72 0 2024-03-23T11:47:27.420 INFO:tasks.workunit.client.1.smithi111.stdout:3/159: truncate d0/d7/f12 1545006 0 2024-03-23T11:47:27.423 INFO:tasks.workunit.client.0.smithi063.stdout:7/177: stat d5/d9/da/cf 0 2024-03-23T11:47:27.423 INFO:tasks.workunit.client.0.smithi063.stdout:0/148: mknod d2/d14/d36/c37 0 2024-03-23T11:47:27.424 INFO:tasks.workunit.client.0.smithi063.stdout:8/348: chown d6/d1f/c33 827519 1 2024-03-23T11:47:27.424 INFO:tasks.workunit.client.1.smithi111.stdout:6/266: creat d2/d1f/d45/d49/f5d x:0 0 0 2024-03-23T11:47:27.426 INFO:tasks.workunit.client.1.smithi111.stdout:3/160: truncate d0/d1a/d1d/f24 525071 0 2024-03-23T11:47:27.426 INFO:tasks.workunit.client.1.smithi111.stdout:7/148: write f1 [4418871,66547] 0 2024-03-23T11:47:27.429 INFO:tasks.workunit.client.1.smithi111.stdout:4/177: creat d0/f3a x:0 0 0 2024-03-23T11:47:27.431 INFO:tasks.workunit.client.1.smithi111.stdout:4/178: readlink d0/l1a 0 2024-03-23T11:47:27.432 INFO:tasks.workunit.client.0.smithi063.stdout:0/149: symlink d2/d4/l38 0 2024-03-23T11:47:27.433 INFO:tasks.workunit.client.1.smithi111.stdout:8/157: dwrite d5/f6 [0,4194304] 0 2024-03-23T11:47:27.433 INFO:tasks.workunit.client.0.smithi063.stdout:9/152: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:27.433 INFO:tasks.workunit.client.1.smithi111.stdout:7/149: truncate f11 1000125 0 2024-03-23T11:47:27.435 INFO:tasks.workunit.client.1.smithi111.stdout:2/210: dwrite d2/d7/fd [0,4194304] 0 2024-03-23T11:47:27.435 INFO:tasks.workunit.client.1.smithi111.stdout:7/150: read f12 [3762469,117806] 0 2024-03-23T11:47:27.436 INFO:tasks.workunit.client.0.smithi063.stdout:8/349: dread d6/d1c/d27/f34 [0,4194304] 0 2024-03-23T11:47:27.438 INFO:tasks.workunit.client.0.smithi063.stdout:9/153: mknod d4/d11/c32 0 2024-03-23T11:47:27.438 INFO:tasks.workunit.client.0.smithi063.stdout:8/350: stat d6/d1f/d54/d5c/d66/f51 0 2024-03-23T11:47:27.438 INFO:tasks.workunit.client.0.smithi063.stdout:0/150: getdents d2/d4/dd/d25 0 2024-03-23T11:47:27.438 INFO:tasks.workunit.client.1.smithi111.stdout:4/179: rmdir d0/d7/da 39 2024-03-23T11:47:27.440 INFO:tasks.workunit.client.1.smithi111.stdout:2/211: fdatasync d2/de/d12/f2c 0 2024-03-23T11:47:27.442 INFO:tasks.workunit.client.1.smithi111.stdout:3/161: link d0/d7/f1e d0/d7/d19/f29 0 2024-03-23T11:47:27.442 INFO:tasks.workunit.client.0.smithi063.stdout:8/351: chown d6/d1c/f3c 12 1 2024-03-23T11:47:27.443 INFO:tasks.workunit.client.1.smithi111.stdout:2/212: fdatasync d2/d11/d17/d3e/f40 0 2024-03-23T11:47:27.444 INFO:tasks.workunit.client.1.smithi111.stdout:4/180: read d0/d17/d29/d2a/f38 [438001,99187] 0 2024-03-23T11:47:27.445 INFO:tasks.workunit.client.0.smithi063.stdout:8/352: chown d6/d1f/c21 654951868 1 2024-03-23T11:47:27.446 INFO:tasks.workunit.client.0.smithi063.stdout:9/154: unlink d4/f20 0 2024-03-23T11:47:27.446 INFO:tasks.workunit.client.0.smithi063.stdout:8/353: chown d6/d1c/f4f 5717 1 2024-03-23T11:47:27.447 INFO:tasks.workunit.client.1.smithi111.stdout:6/267: rename d2/d35/f3f to d2/d1a/f5e 0 2024-03-23T11:47:27.448 INFO:tasks.workunit.client.1.smithi111.stdout:9/249: dread d0/d4/d10/f2c [0,4194304] 0 2024-03-23T11:47:27.449 INFO:tasks.workunit.client.1.smithi111.stdout:3/162: truncate d0/d1a/f1c 5126417 0 2024-03-23T11:47:27.450 INFO:tasks.workunit.client.0.smithi063.stdout:9/155: write d4/d11/d19/f23 [271365,107428] 0 2024-03-23T11:47:27.451 INFO:tasks.workunit.client.1.smithi111.stdout:7/151: getdents d14/d1b/d1e 0 2024-03-23T11:47:27.456 INFO:tasks.workunit.client.0.smithi063.stdout:8/354: chown d6/d1f/d26/d5d/c64 4 1 2024-03-23T11:47:27.457 INFO:tasks.workunit.client.1.smithi111.stdout:2/213: creat d2/d11/f51 x:0 0 0 2024-03-23T11:47:27.459 INFO:tasks.workunit.client.0.smithi063.stdout:0/151: dread d2/d4/dd/d1f/f20 [0,4194304] 0 2024-03-23T11:47:27.459 INFO:tasks.workunit.client.1.smithi111.stdout:4/181: write d0/d7/da/f13 [3985386,52660] 0 2024-03-23T11:47:27.459 INFO:tasks.workunit.client.1.smithi111.stdout:6/268: rename d2/d1f/f54 to d2/d1a/d1c/d2c/d51/f5f 0 2024-03-23T11:47:27.460 INFO:tasks.workunit.client.1.smithi111.stdout:3/163: creat d0/d7/f2a x:0 0 0 2024-03-23T11:47:27.461 INFO:tasks.workunit.client.0.smithi063.stdout:9/156: link d4/fe d4/d11/d19/f33 0 2024-03-23T11:47:27.462 INFO:tasks.workunit.client.0.smithi063.stdout:0/152: mknod d2/d12/d32/c39 0 2024-03-23T11:47:27.464 INFO:tasks.workunit.client.0.smithi063.stdout:9/157: mknod d4/d11/d19/c34 0 2024-03-23T11:47:27.465 INFO:tasks.workunit.client.0.smithi063.stdout:8/355: read d6/d1f/d26/f28 [152197,83912] 0 2024-03-23T11:47:27.465 INFO:tasks.workunit.client.0.smithi063.stdout:0/153: symlink d2/d14/d36/l3a 0 2024-03-23T11:47:27.466 INFO:tasks.workunit.client.1.smithi111.stdout:3/164: mkdir d0/d1a/d2b 0 2024-03-23T11:47:27.466 INFO:tasks.workunit.client.1.smithi111.stdout:2/214: read d2/de/d27/f2f [59579,55109] 0 2024-03-23T11:47:27.467 INFO:tasks.workunit.client.1.smithi111.stdout:6/269: creat d2/d17/f60 x:0 0 0 2024-03-23T11:47:27.468 INFO:tasks.workunit.client.0.smithi063.stdout:0/154: chown d2/d12/d32/c39 5270626 1 2024-03-23T11:47:27.471 INFO:tasks.workunit.client.0.smithi063.stdout:9/158: creat d4/f35 x:0 0 0 2024-03-23T11:47:27.472 INFO:tasks.workunit.client.1.smithi111.stdout:8/158: dwrite d5/f19 [0,4194304] 0 2024-03-23T11:47:27.472 INFO:tasks.workunit.client.0.smithi063.stdout:9/159: write d4/f35 [415362,76771] 0 2024-03-23T11:47:27.475 INFO:tasks.workunit.client.1.smithi111.stdout:1/194: sync 2024-03-23T11:47:27.476 INFO:tasks.workunit.client.1.smithi111.stdout:7/152: dread f7 [0,4194304] 0 2024-03-23T11:47:27.476 INFO:tasks.workunit.client.0.smithi063.stdout:9/160: fdatasync d4/d11/f30 0 2024-03-23T11:47:27.477 INFO:tasks.workunit.client.1.smithi111.stdout:9/250: dread d0/d4/fe [0,4194304] 0 2024-03-23T11:47:27.478 INFO:tasks.workunit.client.0.smithi063.stdout:0/155: link d2/d4/dd/d28/l2b d2/l3b 0 2024-03-23T11:47:27.478 INFO:tasks.workunit.client.1.smithi111.stdout:8/159: rename d5/c16 to d5/d27/c30 0 2024-03-23T11:47:27.478 INFO:tasks.workunit.client.1.smithi111.stdout:6/270: truncate d2/d1f/d45/d49/f5d 540765 0 2024-03-23T11:47:27.480 INFO:tasks.workunit.client.0.smithi063.stdout:7/178: dread d5/d6/f25 [0,4194304] 0 2024-03-23T11:47:27.480 INFO:tasks.workunit.client.1.smithi111.stdout:7/153: mkdir d14/d1b/d20 0 2024-03-23T11:47:27.481 INFO:tasks.workunit.client.1.smithi111.stdout:4/182: dread d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:27.482 INFO:tasks.workunit.client.0.smithi063.stdout:9/161: mknod d4/d31/c36 0 2024-03-23T11:47:27.482 INFO:tasks.workunit.client.1.smithi111.stdout:6/271: readlink d2/l4 0 2024-03-23T11:47:27.483 INFO:tasks.workunit.client.1.smithi111.stdout:1/195: symlink d0/d2b/d37/l44 0 2024-03-23T11:47:27.484 INFO:tasks.workunit.client.1.smithi111.stdout:9/251: mknod d0/d4/d1c/d2d/c54 0 2024-03-23T11:47:27.485 INFO:tasks.workunit.client.0.smithi063.stdout:8/356: dwrite d6/d1f/f55 [0,4194304] 0 2024-03-23T11:47:27.487 INFO:tasks.workunit.client.0.smithi063.stdout:8/357: readlink d6/d1c/d32/d37/l5b 0 2024-03-23T11:47:27.489 INFO:tasks.workunit.client.0.smithi063.stdout:9/162: write d4/d11/d19/f1b [792014,122669] 0 2024-03-23T11:47:27.490 INFO:tasks.workunit.client.0.smithi063.stdout:0/156: rename d2/d4/dd/d28/l2b to d2/d12/d32/l3c 0 2024-03-23T11:47:27.492 INFO:tasks.workunit.client.0.smithi063.stdout:8/358: mknod d6/c73 0 2024-03-23T11:47:27.493 INFO:tasks.workunit.client.0.smithi063.stdout:0/157: write d2/d4/dd/f33 [545355,127217] 0 2024-03-23T11:47:27.493 INFO:tasks.workunit.client.1.smithi111.stdout:4/183: mknod d0/d7/da/c3b 0 2024-03-23T11:47:27.495 INFO:tasks.workunit.client.0.smithi063.stdout:9/163: symlink d4/d11/d19/l37 0 2024-03-23T11:47:27.496 INFO:tasks.workunit.client.0.smithi063.stdout:8/359: fsync d6/d1f/d26/f13 0 2024-03-23T11:47:27.498 INFO:tasks.workunit.client.1.smithi111.stdout:6/272: creat d2/d1a/d1c/d2c/d51/f61 x:0 0 0 2024-03-23T11:47:27.499 INFO:tasks.workunit.client.0.smithi063.stdout:8/360: creat d6/d1f/f74 x:0 0 0 2024-03-23T11:47:27.501 INFO:tasks.workunit.client.1.smithi111.stdout:7/154: link d14/c16 d14/c21 0 2024-03-23T11:47:27.503 INFO:tasks.workunit.client.1.smithi111.stdout:3/165: dwrite d0/d7/f2a [0,4194304] 0 2024-03-23T11:47:27.507 INFO:tasks.workunit.client.1.smithi111.stdout:4/184: readlink d0/l24 0 2024-03-23T11:47:27.507 INFO:tasks.workunit.client.1.smithi111.stdout:6/273: stat d2/d1f/f2e 0 2024-03-23T11:47:27.507 INFO:tasks.workunit.client.0.smithi063.stdout:1/156: sync 2024-03-23T11:47:27.508 INFO:tasks.workunit.client.1.smithi111.stdout:3/166: chown d0/f16 6608 1 2024-03-23T11:47:27.509 INFO:tasks.workunit.client.1.smithi111.stdout:2/215: dwrite d2/d7/da/ff [0,4194304] 0 2024-03-23T11:47:27.510 INFO:tasks.workunit.client.1.smithi111.stdout:9/252: symlink d0/d4/d1c/d36/d28/l55 0 2024-03-23T11:47:27.512 INFO:tasks.workunit.client.0.smithi063.stdout:7/179: dwrite d5/d22/f24 [0,4194304] 0 2024-03-23T11:47:27.513 INFO:tasks.workunit.client.0.smithi063.stdout:1/157: chown d3/d1c/l23 2 1 2024-03-23T11:47:27.519 INFO:tasks.workunit.client.0.smithi063.stdout:9/164: dwrite d4/d11/f30 [0,4194304] 0 2024-03-23T11:47:27.520 INFO:tasks.workunit.client.1.smithi111.stdout:1/196: rename d0/f4 to d0/d2b/f45 0 2024-03-23T11:47:27.521 INFO:tasks.workunit.client.1.smithi111.stdout:9/253: symlink d0/d4/l56 0 2024-03-23T11:47:27.526 INFO:tasks.workunit.client.1.smithi111.stdout:0/197: sync 2024-03-23T11:47:27.526 INFO:tasks.workunit.client.1.smithi111.stdout:5/245: sync 2024-03-23T11:47:27.527 INFO:tasks.workunit.client.1.smithi111.stdout:2/216: mkdir d2/d32/d4b/d52 0 2024-03-23T11:47:27.530 INFO:tasks.workunit.client.0.smithi063.stdout:0/158: dwrite d2/d4/dd/d1f/f20 [0,4194304] 0 2024-03-23T11:47:27.531 INFO:tasks.workunit.client.1.smithi111.stdout:1/197: truncate d0/d5/f2f 1157412 0 2024-03-23T11:47:27.531 INFO:tasks.workunit.client.0.smithi063.stdout:9/165: truncate d4/f21 2387842 0 2024-03-23T11:47:27.532 INFO:tasks.workunit.client.0.smithi063.stdout:7/180: rename d5/d9 to d5/d6/d19/d1f/d37 0 2024-03-23T11:47:27.535 INFO:tasks.workunit.client.1.smithi111.stdout:9/254: read - d0/d4/d1c/f45 zero size 2024-03-23T11:47:27.535 INFO:tasks.workunit.client.1.smithi111.stdout:5/246: chown d1/d3/d6/d7/f16 3106562 1 2024-03-23T11:47:27.535 INFO:tasks.workunit.client.1.smithi111.stdout:1/198: dread - d0/f13 zero size 2024-03-23T11:47:27.535 INFO:tasks.workunit.client.0.smithi063.stdout:0/159: mknod d2/d12/d35/c3d 0 2024-03-23T11:47:27.535 INFO:tasks.workunit.client.0.smithi063.stdout:9/166: stat d4/d11/d19/f2d 0 2024-03-23T11:47:27.536 INFO:tasks.workunit.client.1.smithi111.stdout:4/185: link d0/c8 d0/dc/d30/c3c 0 2024-03-23T11:47:27.536 INFO:tasks.workunit.client.1.smithi111.stdout:7/155: dwrite fd [0,4194304] 0 2024-03-23T11:47:27.544 INFO:tasks.workunit.client.1.smithi111.stdout:9/255: stat d0/d31/c4a 0 2024-03-23T11:47:27.545 INFO:tasks.workunit.client.0.smithi063.stdout:1/158: dwrite d3/df/f12 [0,4194304] 0 2024-03-23T11:47:27.545 INFO:tasks.workunit.client.1.smithi111.stdout:9/256: stat d0/d4/d1c/d2d/d3b/d42/d4f 0 2024-03-23T11:47:27.554 INFO:tasks.workunit.client.1.smithi111.stdout:1/199: fdatasync d0/d5/dd/f10 0 2024-03-23T11:47:27.555 INFO:tasks.workunit.client.1.smithi111.stdout:4/186: fsync d0/dc/f1c 0 2024-03-23T11:47:27.556 INFO:tasks.workunit.client.0.smithi063.stdout:7/181: mknod d5/d6/d19/d1f/d37/d20/c38 0 2024-03-23T11:47:27.557 INFO:tasks.workunit.client.1.smithi111.stdout:9/257: unlink d0/d4/d1c/d36/d28/f3f 0 2024-03-23T11:47:27.557 INFO:tasks.workunit.client.1.smithi111.stdout:2/217: write d2/de/d27/d2e/d3f/f44 [1396798,110285] 0 2024-03-23T11:47:27.559 INFO:tasks.workunit.client.0.smithi063.stdout:7/182: readlink d5/l10 0 2024-03-23T11:47:27.561 INFO:tasks.workunit.client.1.smithi111.stdout:0/198: dread d5/db/d1f/d19/f1c [0,4194304] 0 2024-03-23T11:47:27.566 INFO:tasks.workunit.client.1.smithi111.stdout:9/258: chown d0/d31/d41/d44 5001527 1 2024-03-23T11:47:27.566 INFO:tasks.workunit.client.1.smithi111.stdout:1/200: write d0/d2b/f3e [519996,108192] 0 2024-03-23T11:47:27.569 INFO:tasks.workunit.client.1.smithi111.stdout:5/247: rename d1/d3/d6/d7/f16 to d1/d3/d6/f57 0 2024-03-23T11:47:27.574 INFO:tasks.workunit.client.1.smithi111.stdout:2/218: creat d2/d11/d17/d1f/d30/f53 x:0 0 0 2024-03-23T11:47:27.575 INFO:tasks.workunit.client.1.smithi111.stdout:4/187: read d0/d17/f2c [2489938,75469] 0 2024-03-23T11:47:27.575 INFO:tasks.workunit.client.0.smithi063.stdout:0/160: dwrite f1 [0,4194304] 0 2024-03-23T11:47:27.576 INFO:tasks.workunit.client.0.smithi063.stdout:9/167: dwrite d4/f2a [0,4194304] 0 2024-03-23T11:47:27.576 INFO:tasks.workunit.client.1.smithi111.stdout:5/248: chown d1/d3/fc 502 1 2024-03-23T11:47:27.577 INFO:tasks.workunit.client.1.smithi111.stdout:8/160: dwrite d5/f22 [0,4194304] 0 2024-03-23T11:47:27.579 INFO:tasks.workunit.client.1.smithi111.stdout:6/274: dwrite d2/ff [0,4194304] 0 2024-03-23T11:47:27.583 INFO:tasks.workunit.client.0.smithi063.stdout:1/159: dwrite d3/df/f30 [0,4194304] 0 2024-03-23T11:47:27.599 INFO:tasks.workunit.client.1.smithi111.stdout:9/259: rename d0/d4/l2a to d0/d31/d35/l57 0 2024-03-23T11:47:27.600 INFO:tasks.workunit.client.0.smithi063.stdout:7/183: dread d5/d6/d19/d1f/d37/f2e [0,4194304] 0 2024-03-23T11:47:27.600 INFO:tasks.workunit.client.0.smithi063.stdout:9/168: mknod d4/d2c/c38 0 2024-03-23T11:47:27.604 INFO:tasks.workunit.client.1.smithi111.stdout:3/167: chown d0/d7/f12 1 1 2024-03-23T11:47:27.605 INFO:tasks.workunit.client.1.smithi111.stdout:4/188: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:27.605 INFO:tasks.workunit.client.1.smithi111.stdout:3/168: rename d0 to d0/d1a/d1d/d2c 22 2024-03-23T11:47:27.607 INFO:tasks.workunit.client.1.smithi111.stdout:3/169: dread - d0/f1f zero size 2024-03-23T11:47:27.609 INFO:tasks.workunit.client.0.smithi063.stdout:1/160: stat d3/d1c/c29 0 2024-03-23T11:47:27.609 INFO:tasks.workunit.client.0.smithi063.stdout:0/161: rmdir d2/d4/dd 39 2024-03-23T11:47:27.616 INFO:tasks.workunit.client.0.smithi063.stdout:0/162: rmdir d2/d12 39 2024-03-23T11:47:27.617 INFO:tasks.workunit.client.0.smithi063.stdout:8/361: write d6/d1f/d26/f49 [1093692,80265] 0 2024-03-23T11:47:27.619 INFO:tasks.workunit.client.0.smithi063.stdout:1/161: symlink d3/d35/l37 0 2024-03-23T11:47:27.622 INFO:tasks.workunit.client.0.smithi063.stdout:8/362: fdatasync d6/dd/df/f18 0 2024-03-23T11:47:27.626 INFO:tasks.workunit.client.0.smithi063.stdout:0/163: write d2/d4/dd/f33 [750787,44133] 0 2024-03-23T11:47:27.626 INFO:tasks.workunit.client.1.smithi111.stdout:8/161: unlink d5/c12 0 2024-03-23T11:47:27.628 INFO:tasks.workunit.client.1.smithi111.stdout:7/156: dread f11 [0,4194304] 0 2024-03-23T11:47:27.629 INFO:tasks.workunit.client.1.smithi111.stdout:8/162: truncate d5/f6 5526425 0 2024-03-23T11:47:27.629 INFO:tasks.workunit.client.0.smithi063.stdout:0/164: fsync d2/d4/dd/d1f/f2d 0 2024-03-23T11:47:27.630 INFO:tasks.workunit.client.0.smithi063.stdout:1/162: creat d3/f38 x:0 0 0 2024-03-23T11:47:27.630 INFO:tasks.workunit.client.0.smithi063.stdout:9/169: dwrite f2 [4194304,4194304] 0 2024-03-23T11:47:27.631 INFO:tasks.workunit.client.0.smithi063.stdout:7/184: dread d5/d6/d19/d1f/d37/da/f17 [0,4194304] 0 2024-03-23T11:47:27.633 INFO:tasks.workunit.client.0.smithi063.stdout:1/163: readlink d3/df/l13 0 2024-03-23T11:47:27.638 INFO:tasks.workunit.client.0.smithi063.stdout:0/165: read d2/d14/f31 [2668680,62020] 0 2024-03-23T11:47:27.639 INFO:tasks.workunit.client.1.smithi111.stdout:1/201: symlink d0/d2b/l46 0 2024-03-23T11:47:27.640 INFO:tasks.workunit.client.1.smithi111.stdout:1/202: read d0/d6/f22 [410820,2035] 0 2024-03-23T11:47:27.641 INFO:tasks.workunit.client.0.smithi063.stdout:8/363: symlink d6/l75 0 2024-03-23T11:47:27.642 INFO:tasks.workunit.client.1.smithi111.stdout:0/199: truncate d5/f2d 457877 0 2024-03-23T11:47:27.645 INFO:tasks.workunit.client.0.smithi063.stdout:9/170: mknod d4/d11/c39 0 2024-03-23T11:47:27.645 INFO:tasks.workunit.client.0.smithi063.stdout:1/164: dread d3/d1c/f1e [0,4194304] 0 2024-03-23T11:47:27.645 INFO:tasks.workunit.client.1.smithi111.stdout:3/170: mkdir d0/d1a/d2d 0 2024-03-23T11:47:27.646 INFO:tasks.workunit.client.1.smithi111.stdout:5/249: rmdir d1/d3/d6/d29 39 2024-03-23T11:47:27.646 INFO:tasks.workunit.client.1.smithi111.stdout:4/189: truncate d0/d17/f28 46469 0 2024-03-23T11:47:27.647 INFO:tasks.workunit.client.0.smithi063.stdout:7/185: mkdir d5/d6/d39 0 2024-03-23T11:47:27.650 INFO:tasks.workunit.client.0.smithi063.stdout:0/166: creat d2/d14/f3e x:0 0 0 2024-03-23T11:47:27.650 INFO:tasks.workunit.client.1.smithi111.stdout:8/163: unlink d5/l2c 0 2024-03-23T11:47:27.651 INFO:tasks.workunit.client.0.smithi063.stdout:8/364: read d6/f16 [4184003,101483] 0 2024-03-23T11:47:27.652 INFO:tasks.workunit.client.0.smithi063.stdout:9/171: creat d4/d11/d19/f3a x:0 0 0 2024-03-23T11:47:27.653 INFO:tasks.workunit.client.1.smithi111.stdout:1/203: rename d0/c8 to d0/d6/c47 0 2024-03-23T11:47:27.653 INFO:tasks.workunit.client.1.smithi111.stdout:2/219: creat d2/f54 x:0 0 0 2024-03-23T11:47:27.654 INFO:tasks.workunit.client.0.smithi063.stdout:7/186: mkdir d5/d26/d32/d3a 0 2024-03-23T11:47:27.655 INFO:tasks.workunit.client.1.smithi111.stdout:0/200: write d5/db/d1f/d19/f1c [430547,72135] 0 2024-03-23T11:47:27.655 INFO:tasks.workunit.client.1.smithi111.stdout:2/220: chown d2/de 752037 1 2024-03-23T11:47:27.655 INFO:tasks.workunit.client.0.smithi063.stdout:9/172: write d4/d11/f25 [2016226,28415] 0 2024-03-23T11:47:27.657 INFO:tasks.workunit.client.1.smithi111.stdout:9/260: rmdir d0/d31/d41/d44/d50 0 2024-03-23T11:47:27.658 INFO:tasks.workunit.client.0.smithi063.stdout:9/173: write d4/ff [213432,49046] 0 2024-03-23T11:47:27.659 INFO:tasks.workunit.client.1.smithi111.stdout:8/164: mkdir d5/d26/d31 0 2024-03-23T11:47:27.665 INFO:tasks.workunit.client.0.smithi063.stdout:0/167: mknod d2/c3f 0 2024-03-23T11:47:27.666 INFO:tasks.workunit.client.1.smithi111.stdout:0/201: write d5/db/d1f/f18 [1104923,36258] 0 2024-03-23T11:47:27.669 INFO:tasks.workunit.client.1.smithi111.stdout:4/190: creat d0/d17/f3d x:0 0 0 2024-03-23T11:47:27.672 INFO:tasks.workunit.client.1.smithi111.stdout:8/165: mkdir d5/d32 0 2024-03-23T11:47:27.675 INFO:tasks.workunit.client.1.smithi111.stdout:8/166: fsync d5/f17 0 2024-03-23T11:47:27.675 INFO:tasks.workunit.client.1.smithi111.stdout:3/171: link d0/d7/ca d0/d1a/d1d/c2e 0 2024-03-23T11:47:27.675 INFO:tasks.workunit.client.1.smithi111.stdout:1/204: dwrite d0/f3d [0,4194304] 0 2024-03-23T11:47:27.677 INFO:tasks.workunit.client.1.smithi111.stdout:7/157: getdents d14 0 2024-03-23T11:47:27.686 INFO:tasks.workunit.client.1.smithi111.stdout:9/261: symlink d0/d4/d1c/d36/d28/l58 0 2024-03-23T11:47:27.691 INFO:tasks.workunit.client.0.smithi063.stdout:9/174: dwrite d4/ff [0,4194304] 0 2024-03-23T11:47:27.692 INFO:tasks.workunit.client.0.smithi063.stdout:7/187: write d5/f8 [101504,108697] 0 2024-03-23T11:47:27.699 INFO:tasks.workunit.client.1.smithi111.stdout:3/172: chown d0/d1a/d1d/c2e 14 1 2024-03-23T11:47:27.702 INFO:tasks.workunit.client.1.smithi111.stdout:1/205: mkdir d0/d6/d48 0 2024-03-23T11:47:27.703 INFO:tasks.workunit.client.0.smithi063.stdout:1/165: dread d3/df/f19 [0,4194304] 0 2024-03-23T11:47:27.705 INFO:tasks.workunit.client.0.smithi063.stdout:0/168: dwrite d2/d4/f30 [0,4194304] 0 2024-03-23T11:47:27.706 INFO:tasks.workunit.client.0.smithi063.stdout:8/365: dread d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:27.707 INFO:tasks.workunit.client.1.smithi111.stdout:7/158: fsync f11 0 2024-03-23T11:47:27.708 INFO:tasks.workunit.client.0.smithi063.stdout:0/169: chown d2/d14/d36 1 1 2024-03-23T11:47:27.709 INFO:tasks.workunit.client.0.smithi063.stdout:0/170: fdatasync d2/d4/f9 0 2024-03-23T11:47:27.710 INFO:tasks.workunit.client.1.smithi111.stdout:2/221: rename d2/c15 to d2/c55 0 2024-03-23T11:47:27.710 INFO:tasks.workunit.client.1.smithi111.stdout:1/206: dread d0/d2b/f45 [4194304,4194304] 0 2024-03-23T11:47:27.712 INFO:tasks.workunit.client.0.smithi063.stdout:1/166: creat d3/f39 x:0 0 0 2024-03-23T11:47:27.714 INFO:tasks.workunit.client.1.smithi111.stdout:0/202: link d5/db/d1f/f28 d5/f40 0 2024-03-23T11:47:27.715 INFO:tasks.workunit.client.1.smithi111.stdout:4/191: mknod d0/d17/c3e 0 2024-03-23T11:47:27.715 INFO:tasks.workunit.client.1.smithi111.stdout:5/250: dread d1/d3/d6/d29/f21 [8388608,4194304] 0 2024-03-23T11:47:27.715 INFO:tasks.workunit.client.0.smithi063.stdout:0/171: mknod d2/d4/dd/d1f/c40 0 2024-03-23T11:47:27.715 INFO:tasks.workunit.client.0.smithi063.stdout:9/175: link d4/d11/d19/l2b d4/l3b 0 2024-03-23T11:47:27.715 INFO:tasks.workunit.client.1.smithi111.stdout:8/167: mknod d5/d32/c33 0 2024-03-23T11:47:27.716 INFO:tasks.workunit.client.0.smithi063.stdout:5/86: sync 2024-03-23T11:47:27.716 INFO:tasks.workunit.client.1.smithi111.stdout:3/173: symlink d0/d1a/l2f 0 2024-03-23T11:47:27.717 INFO:tasks.workunit.client.0.smithi063.stdout:6/141: sync 2024-03-23T11:47:27.717 INFO:tasks.workunit.client.1.smithi111.stdout:4/192: chown d0/d17 8 1 2024-03-23T11:47:27.719 INFO:tasks.workunit.client.1.smithi111.stdout:7/159: unlink d14/f1a 0 2024-03-23T11:47:27.719 INFO:tasks.workunit.client.1.smithi111.stdout:4/193: write d0/f1 [1486647,104093] 0 2024-03-23T11:47:27.721 INFO:tasks.workunit.client.1.smithi111.stdout:4/194: readlink d0/l11 0 2024-03-23T11:47:27.721 INFO:tasks.workunit.client.0.smithi063.stdout:7/188: creat d5/d6/d19/d1f/d37/f3b x:0 0 0 2024-03-23T11:47:27.721 INFO:tasks.workunit.client.0.smithi063.stdout:5/87: write d2/d7/f1a [1008382,17672] 0 2024-03-23T11:47:27.722 INFO:tasks.workunit.client.1.smithi111.stdout:4/195: chown d0/d17/f3d 122252994 1 2024-03-23T11:47:27.723 INFO:tasks.workunit.client.0.smithi063.stdout:1/167: mknod d3/d1c/c3a 0 2024-03-23T11:47:27.723 INFO:tasks.workunit.client.1.smithi111.stdout:4/196: stat d0/dc/d1f 0 2024-03-23T11:47:27.724 INFO:tasks.workunit.client.0.smithi063.stdout:5/88: readlink d2/d7/l18 0 2024-03-23T11:47:27.725 INFO:tasks.workunit.client.0.smithi063.stdout:6/142: fsync d1/d13/f16 0 2024-03-23T11:47:27.727 INFO:tasks.workunit.client.1.smithi111.stdout:6/275: sync 2024-03-23T11:47:27.727 INFO:tasks.workunit.client.1.smithi111.stdout:9/262: rename d0/d4/f1b to d0/d4/d1c/d36/f59 0 2024-03-23T11:47:27.728 INFO:tasks.workunit.client.1.smithi111.stdout:7/160: mkdir d14/d1b/d22 0 2024-03-23T11:47:27.729 INFO:tasks.workunit.client.0.smithi063.stdout:7/189: chown d5/d6/d19/d1f/d37/da 24105867 1 2024-03-23T11:47:27.729 INFO:tasks.workunit.client.0.smithi063.stdout:6/143: chown d1/d13/f1c 425096 1 2024-03-23T11:47:27.730 INFO:tasks.workunit.client.0.smithi063.stdout:8/366: dwrite d6/d1c/d32/f57 [0,4194304] 0 2024-03-23T11:47:27.731 INFO:tasks.workunit.client.1.smithi111.stdout:6/276: write d2/d1f/f26 [3238175,43766] 0 2024-03-23T11:47:27.733 INFO:tasks.workunit.client.1.smithi111.stdout:7/161: write f1 [2720009,98692] 0 2024-03-23T11:47:27.735 INFO:tasks.workunit.client.1.smithi111.stdout:8/168: rename d5/d1b/c1e to d5/d26/d31/c34 0 2024-03-23T11:47:27.736 INFO:tasks.workunit.client.1.smithi111.stdout:3/174: dwrite d0/f9 [4194304,4194304] 0 2024-03-23T11:47:27.740 INFO:tasks.workunit.client.1.smithi111.stdout:1/207: creat d0/d2b/f49 x:0 0 0 2024-03-23T11:47:27.740 INFO:tasks.workunit.client.1.smithi111.stdout:9/263: write d0/d4/d1c/f26 [1624031,63771] 0 2024-03-23T11:47:27.741 INFO:tasks.workunit.client.1.smithi111.stdout:2/222: getdents d2/d7/da/d19 0 2024-03-23T11:47:27.741 INFO:tasks.workunit.client.0.smithi063.stdout:9/176: link f2 d4/d11/f3c 0 2024-03-23T11:47:27.741 INFO:tasks.workunit.client.0.smithi063.stdout:1/168: getdents d3/df/d2f 0 2024-03-23T11:47:27.747 INFO:tasks.workunit.client.0.smithi063.stdout:6/144: mkdir d1/d15/d28 0 2024-03-23T11:47:27.747 INFO:tasks.workunit.client.1.smithi111.stdout:0/203: getdents d5/db/d1f 0 2024-03-23T11:47:27.747 INFO:tasks.workunit.client.1.smithi111.stdout:5/251: dread d1/d3/d6/d7/f1f [4194304,4194304] 0 2024-03-23T11:47:27.748 INFO:tasks.workunit.client.1.smithi111.stdout:7/162: symlink d14/d1b/l23 0 2024-03-23T11:47:27.750 INFO:tasks.workunit.client.1.smithi111.stdout:6/277: write d2/d1a/f2d [3102814,75162] 0 2024-03-23T11:47:27.750 INFO:tasks.workunit.client.1.smithi111.stdout:8/169: mkdir d5/d32/d35 0 2024-03-23T11:47:27.751 INFO:tasks.workunit.client.1.smithi111.stdout:7/163: write f1 [536582,122763] 0 2024-03-23T11:47:27.752 INFO:tasks.workunit.client.0.smithi063.stdout:1/169: symlink d3/d35/l3b 0 2024-03-23T11:47:27.754 INFO:tasks.workunit.client.0.smithi063.stdout:1/170: readlink d3/d35/l37 0 2024-03-23T11:47:27.756 INFO:tasks.workunit.client.1.smithi111.stdout:1/208: write d0/d2b/f45 [5477456,126078] 0 2024-03-23T11:47:27.757 INFO:tasks.workunit.client.1.smithi111.stdout:2/223: creat d2/d7/f56 x:0 0 0 2024-03-23T11:47:27.758 INFO:tasks.workunit.client.0.smithi063.stdout:8/367: truncate d6/d1f/d54/d5c/d66/f4c 1190578 0 2024-03-23T11:47:27.759 INFO:tasks.workunit.client.1.smithi111.stdout:6/278: fdatasync d2/d1a/f22 0 2024-03-23T11:47:27.761 INFO:tasks.workunit.client.0.smithi063.stdout:0/172: write d2/d14/f1c [1409179,53195] 0 2024-03-23T11:47:27.761 INFO:tasks.workunit.client.1.smithi111.stdout:0/204: rmdir d5 39 2024-03-23T11:47:27.762 INFO:tasks.workunit.client.0.smithi063.stdout:1/171: creat d3/d1c/f3c x:0 0 0 2024-03-23T11:47:27.763 INFO:tasks.workunit.client.1.smithi111.stdout:2/224: write d2/d11/f37 [352444,66099] 0 2024-03-23T11:47:27.763 INFO:tasks.workunit.client.0.smithi063.stdout:6/145: chown d1/d15/c18 2 1 2024-03-23T11:47:27.763 INFO:tasks.workunit.client.0.smithi063.stdout:7/190: rename d5/d6/d19/d1f/d37/f2e to d5/d6/d19/d1f/d37/f3c 0 2024-03-23T11:47:27.765 INFO:tasks.workunit.client.1.smithi111.stdout:2/225: readlink d2/de/d27/l31 0 2024-03-23T11:47:27.765 INFO:tasks.workunit.client.1.smithi111.stdout:3/175: dwrite d0/d7/f2a [0,4194304] 0 2024-03-23T11:47:27.767 INFO:tasks.workunit.client.0.smithi063.stdout:8/368: mkdir d6/d1f/d54/d5c/d76 0 2024-03-23T11:47:27.767 INFO:tasks.workunit.client.0.smithi063.stdout:1/172: readlink d3/d35/l37 0 2024-03-23T11:47:27.768 INFO:tasks.workunit.client.0.smithi063.stdout:6/146: chown d1/d13/c22 374 1 2024-03-23T11:47:27.772 INFO:tasks.workunit.client.0.smithi063.stdout:0/173: unlink d2/d14/f3e 0 2024-03-23T11:47:27.774 INFO:tasks.workunit.client.0.smithi063.stdout:0/174: fdatasync d2/d14/f23 0 2024-03-23T11:47:27.778 INFO:tasks.workunit.client.1.smithi111.stdout:5/252: dwrite d1/d3/d6/d7/f10 [0,4194304] 0 2024-03-23T11:47:27.778 INFO:tasks.workunit.client.1.smithi111.stdout:6/279: creat d2/d1f/d45/f62 x:0 0 0 2024-03-23T11:47:27.782 INFO:tasks.workunit.client.1.smithi111.stdout:7/164: truncate f9 351273 0 2024-03-23T11:47:27.783 INFO:tasks.workunit.client.1.smithi111.stdout:1/209: mkdir d0/d6/d48/d4a 0 2024-03-23T11:47:27.784 INFO:tasks.workunit.client.0.smithi063.stdout:0/175: mknod d2/d12/d32/c41 0 2024-03-23T11:47:27.784 INFO:tasks.workunit.client.1.smithi111.stdout:7/165: stat l5 0 2024-03-23T11:47:27.788 INFO:tasks.workunit.client.1.smithi111.stdout:0/205: dread d5/db/d1f/d19/f1c [0,4194304] 0 2024-03-23T11:47:27.789 INFO:tasks.workunit.client.0.smithi063.stdout:1/173: dwrite d3/f10 [0,4194304] 0 2024-03-23T11:47:27.790 INFO:tasks.workunit.client.1.smithi111.stdout:4/197: dwrite d0/d17/f28 [0,4194304] 0 2024-03-23T11:47:27.793 INFO:tasks.workunit.client.1.smithi111.stdout:9/264: getdents d0/d31/d35 0 2024-03-23T11:47:27.793 INFO:tasks.workunit.client.1.smithi111.stdout:7/166: write d14/f1c [648129,67979] 0 2024-03-23T11:47:27.795 INFO:tasks.workunit.client.1.smithi111.stdout:8/170: dread d5/f11 [0,4194304] 0 2024-03-23T11:47:27.799 INFO:tasks.workunit.client.0.smithi063.stdout:0/176: creat d2/d12/f42 x:0 0 0 2024-03-23T11:47:27.799 INFO:tasks.workunit.client.0.smithi063.stdout:6/147: write d1/d13/f16 [250434,33225] 0 2024-03-23T11:47:27.799 INFO:tasks.workunit.client.0.smithi063.stdout:7/191: dwrite d5/f12 [0,4194304] 0 2024-03-23T11:47:27.804 INFO:tasks.workunit.client.0.smithi063.stdout:1/174: symlink d3/df/d31/l3d 0 2024-03-23T11:47:27.804 INFO:tasks.workunit.client.1.smithi111.stdout:0/206: mknod d5/c41 0 2024-03-23T11:47:27.805 INFO:tasks.workunit.client.0.smithi063.stdout:8/369: truncate d6/d1c/f52 2869526 0 2024-03-23T11:47:27.807 INFO:tasks.workunit.client.0.smithi063.stdout:8/370: chown d6/d1c/d32/d37 57 1 2024-03-23T11:47:27.809 INFO:tasks.workunit.client.0.smithi063.stdout:0/177: rename d2/d14/f19 to d2/d12/d32/f43 0 2024-03-23T11:47:27.810 INFO:tasks.workunit.client.1.smithi111.stdout:4/198: mknod d0/dc/d1f/c3f 0 2024-03-23T11:47:27.811 INFO:tasks.workunit.client.1.smithi111.stdout:2/226: dwrite d2/d7/da/d19/f34 [0,4194304] 0 2024-03-23T11:47:27.812 INFO:tasks.workunit.client.0.smithi063.stdout:0/178: write d2/d4/f9 [3537645,34194] 0 2024-03-23T11:47:27.814 INFO:tasks.workunit.client.1.smithi111.stdout:9/265: symlink d0/d4/d10/d29/l5a 0 2024-03-23T11:47:27.814 INFO:tasks.workunit.client.0.smithi063.stdout:1/175: mknod d3/df/d2f/c3e 0 2024-03-23T11:47:27.814 INFO:tasks.workunit.client.0.smithi063.stdout:7/192: mknod d5/c3d 0 2024-03-23T11:47:27.815 INFO:tasks.workunit.client.0.smithi063.stdout:0/179: write d2/d4/f30 [369643,73707] 0 2024-03-23T11:47:27.816 INFO:tasks.workunit.client.0.smithi063.stdout:8/371: mkdir d6/d1f/d54/d5c/d77 0 2024-03-23T11:47:27.816 INFO:tasks.workunit.client.1.smithi111.stdout:5/253: write d1/d3/d6/d29/f22 [593682,113229] 0 2024-03-23T11:47:27.817 INFO:tasks.workunit.client.1.smithi111.stdout:3/176: link d0/c17 d0/d1a/c30 0 2024-03-23T11:47:27.819 INFO:tasks.workunit.client.0.smithi063.stdout:8/372: fdatasync d6/dd/df/f18 0 2024-03-23T11:47:27.823 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:27 smithi111 ceph-mon[44005]: pgmap v33: 65 pgs: 65 active+clean; 436 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 1.9 MiB/s rd, 31 MiB/s wr, 214 op/s 2024-03-23T11:47:27.827 INFO:tasks.workunit.client.1.smithi111.stdout:5/254: chown d1/f49 1708083 1 2024-03-23T11:47:27.828 INFO:tasks.workunit.client.0.smithi063.stdout:1/176: mkdir d3/d3f 0 2024-03-23T11:47:27.831 INFO:tasks.workunit.client.0.smithi063.stdout:1/177: readlink d3/df/d31/l3d 0 2024-03-23T11:47:27.832 INFO:tasks.workunit.client.1.smithi111.stdout:9/266: mkdir d0/d4/d10/d29/d5b 0 2024-03-23T11:47:27.834 INFO:tasks.workunit.client.0.smithi063.stdout:1/178: stat d3/df/f18 0 2024-03-23T11:47:27.835 INFO:tasks.workunit.client.0.smithi063.stdout:8/373: mknod d6/d1c/d32/d37/c78 0 2024-03-23T11:47:27.836 INFO:tasks.workunit.client.1.smithi111.stdout:8/171: dwrite d5/f1a [0,4194304] 0 2024-03-23T11:47:27.836 INFO:tasks.workunit.client.1.smithi111.stdout:7/167: dread d14/f15 [4194304,4194304] 0 2024-03-23T11:47:27.837 INFO:tasks.workunit.client.1.smithi111.stdout:2/227: unlink d2/de/d27/f50 0 2024-03-23T11:47:27.837 INFO:tasks.workunit.client.0.smithi063.stdout:0/180: link d2/d4/dd/d1f/f2f d2/d12/f44 0 2024-03-23T11:47:27.838 INFO:tasks.workunit.client.0.smithi063.stdout:8/374: readlink d6/l75 0 2024-03-23T11:47:27.839 INFO:tasks.workunit.client.1.smithi111.stdout:6/280: dwrite d2/fb [0,4194304] 0 2024-03-23T11:47:27.842 INFO:tasks.workunit.client.0.smithi063.stdout:1/179: mkdir d3/d35/d40 0 2024-03-23T11:47:27.843 INFO:tasks.workunit.client.0.smithi063.stdout:0/181: stat d2/d14/f23 0 2024-03-23T11:47:27.843 INFO:tasks.workunit.client.1.smithi111.stdout:4/199: rmdir d0 39 2024-03-23T11:47:27.844 INFO:tasks.workunit.client.0.smithi063.stdout:8/375: stat d6/d1c/f4f 0 2024-03-23T11:47:27.847 INFO:tasks.workunit.client.1.smithi111.stdout:5/255: write d1/d3/d6/d29/f4d [298243,1490] 0 2024-03-23T11:47:27.847 INFO:tasks.workunit.client.0.smithi063.stdout:7/193: dwrite d5/d6/d19/d1f/d37/da/f17 [0,4194304] 0 2024-03-23T11:47:27.851 INFO:tasks.workunit.client.0.smithi063.stdout:7/194: chown d5/d16/c34 404003470 1 2024-03-23T11:47:27.855 INFO:tasks.workunit.client.1.smithi111.stdout:2/228: read d2/d11/f37 [234248,55295] 0 2024-03-23T11:47:27.856 INFO:tasks.workunit.client.1.smithi111.stdout:7/168: write f7 [5197642,109662] 0 2024-03-23T11:47:27.857 INFO:tasks.workunit.client.1.smithi111.stdout:9/267: creat d0/d4/d1c/d2d/d3b/d42/d4f/f5c x:0 0 0 2024-03-23T11:47:27.858 INFO:tasks.workunit.client.0.smithi063.stdout:8/376: mknod d6/d1f/d54/d5c/d77/c79 0 2024-03-23T11:47:27.868 INFO:tasks.workunit.client.1.smithi111.stdout:0/207: dwrite d5/db/d1f/d39/f3d [0,4194304] 0 2024-03-23T11:47:27.869 INFO:tasks.workunit.client.1.smithi111.stdout:3/177: dwrite d0/fd [0,4194304] 0 2024-03-23T11:47:27.869 INFO:tasks.workunit.client.1.smithi111.stdout:1/210: sync 2024-03-23T11:47:27.870 INFO:tasks.workunit.client.0.smithi063.stdout:6/148: dread d1/dc/f1f [0,4194304] 0 2024-03-23T11:47:27.872 INFO:tasks.workunit.client.1.smithi111.stdout:6/281: dread d2/f16 [0,4194304] 0 2024-03-23T11:47:27.872 INFO:tasks.workunit.client.0.smithi063.stdout:1/180: dwrite d3/d1c/f1e [0,4194304] 0 2024-03-23T11:47:27.874 INFO:tasks.workunit.client.0.smithi063.stdout:8/377: mknod d6/dd/c7a 0 2024-03-23T11:47:27.876 INFO:tasks.workunit.client.1.smithi111.stdout:1/211: readlink d0/d2b/l46 0 2024-03-23T11:47:27.876 INFO:tasks.workunit.client.1.smithi111.stdout:3/178: write d0/f16 [3711265,76411] 0 2024-03-23T11:47:27.876 INFO:tasks.workunit.client.1.smithi111.stdout:9/268: dread - d0/d31/d35/f4d zero size 2024-03-23T11:47:27.876 INFO:tasks.workunit.client.1.smithi111.stdout:5/256: rename d1/d20/c4e to d1/d3/d23/c58 0 2024-03-23T11:47:27.878 INFO:tasks.workunit.client.1.smithi111.stdout:4/200: write d0/d7/f1b [4893681,97976] 0 2024-03-23T11:47:27.881 INFO:tasks.workunit.client.0.smithi063.stdout:8/378: creat d6/d1c/f7b x:0 0 0 2024-03-23T11:47:27.882 INFO:tasks.workunit.client.0.smithi063.stdout:7/195: dread d5/d6/d19/d1f/d37/d20/f28 [0,4194304] 0 2024-03-23T11:47:27.882 INFO:tasks.workunit.client.0.smithi063.stdout:0/182: dwrite d2/d12/d32/f43 [4194304,4194304] 0 2024-03-23T11:47:27.883 INFO:tasks.workunit.client.1.smithi111.stdout:0/208: symlink d5/l42 0 2024-03-23T11:47:27.883 INFO:tasks.workunit.client.1.smithi111.stdout:8/172: write d5/f1f [4773835,9435] 0 2024-03-23T11:47:27.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:27 smithi063 ceph-mon[35189]: pgmap v33: 65 pgs: 65 active+clean; 436 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 1.9 MiB/s rd, 31 MiB/s wr, 214 op/s 2024-03-23T11:47:27.887 INFO:tasks.workunit.client.0.smithi063.stdout:7/196: chown d5/d6/d19/d1f/d37/f3c 40 1 2024-03-23T11:47:27.887 INFO:tasks.workunit.client.0.smithi063.stdout:0/183: truncate d2/d14/f1c 1734061 0 2024-03-23T11:47:27.888 INFO:tasks.workunit.client.1.smithi111.stdout:8/173: chown d5/d27/c28 80 1 2024-03-23T11:47:27.890 INFO:tasks.workunit.client.0.smithi063.stdout:0/184: chown d2/d14/f1c 477242 1 2024-03-23T11:47:27.890 INFO:tasks.workunit.client.1.smithi111.stdout:8/174: read f3 [4963236,19665] 0 2024-03-23T11:47:27.894 INFO:tasks.workunit.client.0.smithi063.stdout:6/149: getdents d1 0 2024-03-23T11:47:27.898 INFO:tasks.workunit.client.1.smithi111.stdout:2/229: getdents d2/d11 0 2024-03-23T11:47:27.900 INFO:tasks.workunit.client.1.smithi111.stdout:7/169: dwrite d14/f15 [0,4194304] 0 2024-03-23T11:47:27.901 INFO:tasks.workunit.client.0.smithi063.stdout:8/379: rename d6/dd/c70 to d6/d1f/d3e/c7c 0 2024-03-23T11:47:27.904 INFO:tasks.workunit.client.0.smithi063.stdout:7/197: symlink d5/d26/l3e 0 2024-03-23T11:47:27.905 INFO:tasks.workunit.client.1.smithi111.stdout:0/209: creat d5/db/d1f/f43 x:0 0 0 2024-03-23T11:47:27.905 INFO:tasks.workunit.client.1.smithi111.stdout:7/170: fdatasync fd 0 2024-03-23T11:47:27.905 INFO:tasks.workunit.client.1.smithi111.stdout:2/230: readlink d2/de/d27/l4c 0 2024-03-23T11:47:27.907 INFO:tasks.workunit.client.1.smithi111.stdout:4/201: creat d0/d17/d29/d2a/f40 x:0 0 0 2024-03-23T11:47:27.908 INFO:tasks.workunit.client.0.smithi063.stdout:1/181: link d3/df/f26 d3/df/d31/f41 0 2024-03-23T11:47:27.908 INFO:tasks.workunit.client.0.smithi063.stdout:6/150: rmdir d1/dc 39 2024-03-23T11:47:27.908 INFO:tasks.workunit.client.1.smithi111.stdout:0/210: stat d5/db/d1f/f37 0 2024-03-23T11:47:27.909 INFO:tasks.workunit.client.0.smithi063.stdout:0/185: mknod d2/d4/dd/d25/c45 0 2024-03-23T11:47:27.910 INFO:tasks.workunit.client.1.smithi111.stdout:3/179: write d0/d7/f12 [1010626,110618] 0 2024-03-23T11:47:27.912 INFO:tasks.workunit.client.1.smithi111.stdout:2/231: fsync d2/d11/d17/d1f/f20 0 2024-03-23T11:47:27.913 INFO:tasks.workunit.client.1.smithi111.stdout:1/212: dread d0/d5/dd/d1e/f36 [0,4194304] 0 2024-03-23T11:47:27.915 INFO:tasks.workunit.client.1.smithi111.stdout:9/269: creat d0/d4/d1c/d36/d28/f5d x:0 0 0 2024-03-23T11:47:27.917 INFO:tasks.workunit.client.1.smithi111.stdout:4/202: unlink d0/d17/f28 0 2024-03-23T11:47:27.917 INFO:tasks.workunit.client.0.smithi063.stdout:7/198: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:27.923 INFO:tasks.workunit.client.0.smithi063.stdout:8/380: rename d6/d1f/d54/d5c/d77 to d6/d1f/d26/d5d/d7d 0 2024-03-23T11:47:27.926 INFO:tasks.workunit.client.0.smithi063.stdout:0/186: mkdir d2/d4/dd/d28/d46 0 2024-03-23T11:47:27.926 INFO:tasks.workunit.client.1.smithi111.stdout:8/175: dwrite f3 [4194304,4194304] 0 2024-03-23T11:47:27.927 INFO:tasks.workunit.client.1.smithi111.stdout:7/171: link d14/l17 d14/d1b/l24 0 2024-03-23T11:47:27.930 INFO:tasks.workunit.client.0.smithi063.stdout:1/182: creat d3/d35/d40/f42 x:0 0 0 2024-03-23T11:47:27.935 INFO:tasks.workunit.client.0.smithi063.stdout:8/381: mknod d6/d1f/d26/c7e 0 2024-03-23T11:47:27.935 INFO:tasks.workunit.client.1.smithi111.stdout:2/232: symlink d2/d11/d17/d1f/l57 0 2024-03-23T11:47:27.936 INFO:tasks.workunit.client.1.smithi111.stdout:0/211: link d5/db/d1f/f28 d5/db/d1f/d32/f44 0 2024-03-23T11:47:27.937 INFO:tasks.workunit.client.0.smithi063.stdout:8/382: write d6/d1c/f2d [325000,99955] 0 2024-03-23T11:47:27.938 INFO:tasks.workunit.client.1.smithi111.stdout:4/203: rename d0/dc/fe to d0/dc/f41 0 2024-03-23T11:47:27.939 INFO:tasks.workunit.client.0.smithi063.stdout:7/199: mknod d5/c3f 0 2024-03-23T11:47:27.941 INFO:tasks.workunit.client.0.smithi063.stdout:1/183: rename d3/l28 to d3/d35/l43 0 2024-03-23T11:47:27.943 INFO:tasks.workunit.client.1.smithi111.stdout:9/270: dread d0/f48 [0,4194304] 0 2024-03-23T11:47:27.943 INFO:tasks.workunit.client.1.smithi111.stdout:8/176: creat d5/d27/f36 x:0 0 0 2024-03-23T11:47:27.944 INFO:tasks.workunit.client.0.smithi063.stdout:6/151: creat d1/dc/d21/f29 x:0 0 0 2024-03-23T11:47:27.945 INFO:tasks.workunit.client.0.smithi063.stdout:1/184: write d3/f39 [328480,53275] 0 2024-03-23T11:47:27.946 INFO:tasks.workunit.client.1.smithi111.stdout:8/177: readlink d5/l2e 0 2024-03-23T11:47:27.947 INFO:tasks.workunit.client.0.smithi063.stdout:8/383: mkdir d6/dd/d59/d7f 0 2024-03-23T11:47:27.949 INFO:tasks.workunit.client.1.smithi111.stdout:2/233: truncate d2/d11/f37 816880 0 2024-03-23T11:47:27.950 INFO:tasks.workunit.client.0.smithi063.stdout:0/187: dwrite f1 [0,4194304] 0 2024-03-23T11:47:27.951 INFO:tasks.workunit.client.0.smithi063.stdout:7/200: mkdir d5/d6/d19/d1f/d37/d40 0 2024-03-23T11:47:27.952 INFO:tasks.workunit.client.1.smithi111.stdout:4/204: mknod d0/dc/c42 0 2024-03-23T11:47:27.952 INFO:tasks.workunit.client.1.smithi111.stdout:6/282: dread d2/d17/f55 [0,4194304] 0 2024-03-23T11:47:27.953 INFO:tasks.workunit.client.0.smithi063.stdout:1/185: mknod d3/c44 0 2024-03-23T11:47:27.954 INFO:tasks.workunit.client.0.smithi063.stdout:0/188: stat d2/d14/d36/l3a 0 2024-03-23T11:47:27.954 INFO:tasks.workunit.client.0.smithi063.stdout:8/384: rmdir d6/d1f/d54/d5c/d66 39 2024-03-23T11:47:27.954 INFO:tasks.workunit.client.0.smithi063.stdout:7/201: write d5/d22/f24 [3468225,24797] 0 2024-03-23T11:47:27.957 INFO:tasks.workunit.client.1.smithi111.stdout:0/212: mkdir d5/db/d1f/d19/d45 0 2024-03-23T11:47:27.959 INFO:tasks.workunit.client.0.smithi063.stdout:0/189: write d2/d4/f30 [395003,104088] 0 2024-03-23T11:47:27.961 INFO:tasks.workunit.client.1.smithi111.stdout:1/213: dwrite d0/d2b/f49 [0,4194304] 0 2024-03-23T11:47:27.961 INFO:tasks.workunit.client.1.smithi111.stdout:3/180: dread d0/d7/d19/f29 [0,4194304] 0 2024-03-23T11:47:27.962 INFO:tasks.workunit.client.0.smithi063.stdout:4/181: sync 2024-03-23T11:47:27.962 INFO:tasks.workunit.client.0.smithi063.stdout:3/293: sync 2024-03-23T11:47:27.962 INFO:tasks.workunit.client.0.smithi063.stdout:2/169: sync 2024-03-23T11:47:27.962 INFO:tasks.workunit.client.1.smithi111.stdout:5/257: dread d1/d3/d6/f3b [0,4194304] 0 2024-03-23T11:47:27.963 INFO:tasks.workunit.client.1.smithi111.stdout:3/181: read d0/d7/f23 [1047336,110154] 0 2024-03-23T11:47:27.964 INFO:tasks.workunit.client.1.smithi111.stdout:2/234: write d2/d11/d17/d1f/d30/f41 [999289,34369] 0 2024-03-23T11:47:27.969 INFO:tasks.workunit.client.1.smithi111.stdout:0/213: symlink d5/db/d1f/d19/l46 0 2024-03-23T11:47:27.969 INFO:tasks.workunit.client.0.smithi063.stdout:0/190: unlink d2/d4/dd/c18 0 2024-03-23T11:47:27.971 INFO:tasks.workunit.client.0.smithi063.stdout:4/182: chown d5/da/f2a 412387635 1 2024-03-23T11:47:27.971 INFO:tasks.workunit.client.1.smithi111.stdout:4/205: mknod d0/c43 0 2024-03-23T11:47:27.971 INFO:tasks.workunit.client.1.smithi111.stdout:7/172: dwrite d14/f19 [0,4194304] 0 2024-03-23T11:47:27.972 INFO:tasks.workunit.client.1.smithi111.stdout:6/283: rename d2/d1a/d1c/d2c/f52 to d2/d1f/d45/f63 0 2024-03-23T11:47:27.974 INFO:tasks.workunit.client.1.smithi111.stdout:9/271: creat d0/d4/d1c/d36/d28/f5e x:0 0 0 2024-03-23T11:47:27.975 INFO:tasks.workunit.client.1.smithi111.stdout:4/206: readlink d0/d7/da/l2b 0 2024-03-23T11:47:27.977 INFO:tasks.workunit.client.1.smithi111.stdout:6/284: stat d2/d17/f5a 0 2024-03-23T11:47:27.980 INFO:tasks.workunit.client.1.smithi111.stdout:6/285: dread - d2/d1f/d45/f63 zero size 2024-03-23T11:47:27.982 INFO:tasks.workunit.client.1.smithi111.stdout:5/258: mkdir d1/d20/d25/d59 0 2024-03-23T11:47:27.982 INFO:tasks.workunit.client.1.smithi111.stdout:2/235: rmdir d2/de/d27/d2e/d3f 39 2024-03-23T11:47:27.983 INFO:tasks.workunit.client.1.smithi111.stdout:5/259: readlink l0 0 2024-03-23T11:47:27.984 INFO:tasks.workunit.client.0.smithi063.stdout:8/385: mknod d6/d1c/d32/d37/c80 0 2024-03-23T11:47:27.985 INFO:tasks.workunit.client.1.smithi111.stdout:8/178: dwrite d5/f11 [0,4194304] 0 2024-03-23T11:47:27.987 INFO:tasks.workunit.client.0.smithi063.stdout:2/170: symlink d2/d1c/l39 0 2024-03-23T11:47:27.989 INFO:tasks.workunit.client.0.smithi063.stdout:3/294: dwrite d13/d15/f2a [0,4194304] 0 2024-03-23T11:47:27.990 INFO:tasks.workunit.client.0.smithi063.stdout:4/183: write d5/ff [4378377,28936] 0 2024-03-23T11:47:27.992 INFO:tasks.workunit.client.1.smithi111.stdout:4/207: mknod d0/d17/d29/c44 0 2024-03-23T11:47:27.996 INFO:tasks.workunit.client.1.smithi111.stdout:1/214: link d0/d6/l43 d0/d5/dd/l4b 0 2024-03-23T11:47:27.996 INFO:tasks.workunit.client.1.smithi111.stdout:5/260: fdatasync d1/d3/d6/d29/f21 0 2024-03-23T11:47:27.998 INFO:tasks.workunit.client.0.smithi063.stdout:0/191: mkdir d2/d47 0 2024-03-23T11:47:27.998 INFO:tasks.workunit.client.1.smithi111.stdout:5/261: fdatasync d1/f17 0 2024-03-23T11:47:27.999 INFO:tasks.workunit.client.0.smithi063.stdout:8/386: symlink d6/d1f/l81 0 2024-03-23T11:47:28.001 INFO:tasks.workunit.client.1.smithi111.stdout:5/262: chown d1/d3/d6/d7/f24 4537752 1 2024-03-23T11:47:28.005 INFO:tasks.workunit.client.0.smithi063.stdout:1/186: truncate d3/df/f30 2881690 0 2024-03-23T11:47:28.006 INFO:tasks.workunit.client.1.smithi111.stdout:8/179: symlink d5/d32/l37 0 2024-03-23T11:47:28.006 INFO:tasks.workunit.client.0.smithi063.stdout:0/192: dread - d2/d14/f15 zero size 2024-03-23T11:47:28.008 INFO:tasks.workunit.client.0.smithi063.stdout:6/152: dread d1/f7 [0,4194304] 0 2024-03-23T11:47:28.008 INFO:tasks.workunit.client.1.smithi111.stdout:3/182: creat d0/f31 x:0 0 0 2024-03-23T11:47:28.009 INFO:tasks.workunit.client.0.smithi063.stdout:4/184: unlink d5/da/fd 0 2024-03-23T11:47:28.011 INFO:tasks.workunit.client.0.smithi063.stdout:3/295: mknod d13/d15/d48/c5a 0 2024-03-23T11:47:28.013 INFO:tasks.workunit.client.1.smithi111.stdout:2/236: mkdir d2/d32/d4b/d52/d58 0 2024-03-23T11:47:28.014 INFO:tasks.workunit.client.0.smithi063.stdout:1/187: unlink d3/c33 0 2024-03-23T11:47:28.014 INFO:tasks.workunit.client.0.smithi063.stdout:8/387: chown d6/d1c/f52 101938971 1 2024-03-23T11:47:28.019 INFO:tasks.workunit.client.1.smithi111.stdout:4/208: unlink d0/d17/d29/d2a/l2e 0 2024-03-23T11:47:28.020 INFO:tasks.workunit.client.0.smithi063.stdout:2/171: creat d2/f3a x:0 0 0 2024-03-23T11:47:28.021 INFO:tasks.workunit.client.0.smithi063.stdout:6/153: mkdir d1/d2a 0 2024-03-23T11:47:28.021 INFO:tasks.workunit.client.1.smithi111.stdout:4/209: stat d0/dc/d30/ff 0 2024-03-23T11:47:28.022 INFO:tasks.workunit.client.0.smithi063.stdout:3/296: unlink d13/d22/d2d/d3e/f4c 0 2024-03-23T11:47:28.023 INFO:tasks.workunit.client.1.smithi111.stdout:8/180: dwrite d5/f2d [0,4194304] 0 2024-03-23T11:47:28.024 INFO:tasks.workunit.client.1.smithi111.stdout:4/210: write d0/dc/f14 [3732785,53311] 0 2024-03-23T11:47:28.024 INFO:tasks.workunit.client.0.smithi063.stdout:8/388: dread - d6/d1f/d26/d15/f65 zero size 2024-03-23T11:47:28.024 INFO:tasks.workunit.client.0.smithi063.stdout:7/202: dwrite d5/d6/d19/d1f/d37/d20/f28 [0,4194304] 0 2024-03-23T11:47:28.025 INFO:tasks.workunit.client.0.smithi063.stdout:0/193: truncate d2/d4/dd/d1f/f2d 789659 0 2024-03-23T11:47:28.026 INFO:tasks.workunit.client.1.smithi111.stdout:4/211: chown d0/d7/da/c3b 2332 1 2024-03-23T11:47:28.029 INFO:tasks.workunit.client.1.smithi111.stdout:1/215: rmdir d0/d2b/d37 39 2024-03-23T11:47:28.034 INFO:tasks.workunit.client.1.smithi111.stdout:5/263: chown d1/d3/d3a/f3f 7559 1 2024-03-23T11:47:28.034 INFO:tasks.workunit.client.1.smithi111.stdout:0/214: write d5/db/d1f/d32/f44 [2482507,69366] 0 2024-03-23T11:47:28.037 INFO:tasks.workunit.client.1.smithi111.stdout:8/181: dread d5/d1b/f2b [0,4194304] 0 2024-03-23T11:47:28.037 INFO:tasks.workunit.client.0.smithi063.stdout:6/154: chown d1/d13/c19 205062834 1 2024-03-23T11:47:28.038 INFO:tasks.workunit.client.1.smithi111.stdout:8/182: fsync d5/f1a 0 2024-03-23T11:47:28.039 INFO:tasks.workunit.client.0.smithi063.stdout:8/389: fsync d6/dd/f56 0 2024-03-23T11:47:28.040 INFO:tasks.workunit.client.0.smithi063.stdout:3/297: mkdir d13/d22/d2d/d3f/d5b 0 2024-03-23T11:47:28.041 INFO:tasks.workunit.client.1.smithi111.stdout:8/183: chown d5/l2e 167338208 1 2024-03-23T11:47:28.043 INFO:tasks.workunit.client.0.smithi063.stdout:3/298: chown d13/d22/d2d/c4a 0 1 2024-03-23T11:47:28.043 INFO:tasks.workunit.client.1.smithi111.stdout:5/264: creat d1/d3/d6/d7/f5a x:0 0 0 2024-03-23T11:47:28.044 INFO:tasks.workunit.client.1.smithi111.stdout:7/173: getdents d14/d1b 0 2024-03-23T11:47:28.044 INFO:tasks.workunit.client.1.smithi111.stdout:0/215: rmdir d5/db 39 2024-03-23T11:47:28.045 INFO:tasks.workunit.client.0.smithi063.stdout:7/203: mknod d5/d6/d19/d1f/c41 0 2024-03-23T11:47:28.046 INFO:tasks.workunit.client.0.smithi063.stdout:1/188: rename d3/l34 to d3/df/d2f/l45 0 2024-03-23T11:47:28.050 INFO:tasks.workunit.client.1.smithi111.stdout:1/216: creat d0/d2b/d37/f4c x:0 0 0 2024-03-23T11:47:28.050 INFO:tasks.workunit.client.0.smithi063.stdout:2/172: mkdir d2/d3b 0 2024-03-23T11:47:28.051 INFO:tasks.workunit.client.1.smithi111.stdout:9/272: dread d0/f21 [0,4194304] 0 2024-03-23T11:47:28.052 INFO:tasks.workunit.client.1.smithi111.stdout:7/174: rename fd to d14/f25 0 2024-03-23T11:47:28.053 INFO:tasks.workunit.client.0.smithi063.stdout:0/194: mkdir d2/d48 0 2024-03-23T11:47:28.054 INFO:tasks.workunit.client.1.smithi111.stdout:7/175: truncate d14/f1c 1202639 0 2024-03-23T11:47:28.054 INFO:tasks.workunit.client.0.smithi063.stdout:7/204: chown d5/d6/d19/d1f/d37/d20/f28 7011 1 2024-03-23T11:47:28.055 INFO:tasks.workunit.client.1.smithi111.stdout:3/183: link d0/d7/l21 d0/d1a/d1d/d26/l32 0 2024-03-23T11:47:28.055 INFO:tasks.workunit.client.0.smithi063.stdout:0/195: chown d2/d14/f23 3034 1 2024-03-23T11:47:28.056 INFO:tasks.workunit.client.0.smithi063.stdout:6/155: write d1/dc/fd [3757783,37012] 0 2024-03-23T11:47:28.057 INFO:tasks.workunit.client.1.smithi111.stdout:5/265: chown d1/d3/d6/c11 232791460 1 2024-03-23T11:47:28.057 INFO:tasks.workunit.client.1.smithi111.stdout:9/273: truncate d0/d4/d1c/d36/f18 954973 0 2024-03-23T11:47:28.057 INFO:tasks.workunit.client.1.smithi111.stdout:8/184: dread f1 [0,4194304] 0 2024-03-23T11:47:28.058 INFO:tasks.workunit.client.0.smithi063.stdout:3/299: symlink d13/d22/d2d/l5c 0 2024-03-23T11:47:28.059 INFO:tasks.workunit.client.1.smithi111.stdout:4/212: creat d0/f45 x:0 0 0 2024-03-23T11:47:28.061 INFO:tasks.workunit.client.1.smithi111.stdout:0/216: write d5/db/f38 [3776432,126096] 0 2024-03-23T11:47:28.062 INFO:tasks.workunit.client.1.smithi111.stdout:7/176: mknod d14/c26 0 2024-03-23T11:47:28.063 INFO:tasks.workunit.client.0.smithi063.stdout:7/205: creat d5/d6/d19/d1f/f42 x:0 0 0 2024-03-23T11:47:28.065 INFO:tasks.workunit.client.1.smithi111.stdout:5/266: write d1/d20/d25/f26 [878667,15683] 0 2024-03-23T11:47:28.066 INFO:tasks.workunit.client.0.smithi063.stdout:6/156: truncate d1/dc/d21/f29 746861 0 2024-03-23T11:47:28.068 INFO:tasks.workunit.client.1.smithi111.stdout:4/213: write d0/dc/d30/f15 [4998802,35433] 0 2024-03-23T11:47:28.069 INFO:tasks.workunit.client.1.smithi111.stdout:7/177: truncate d14/f1c 1510640 0 2024-03-23T11:47:28.069 INFO:tasks.workunit.client.0.smithi063.stdout:7/206: symlink d5/d6/d19/l43 0 2024-03-23T11:47:28.069 INFO:tasks.workunit.client.0.smithi063.stdout:7/207: readlink d5/d26/l3e 0 2024-03-23T11:47:28.070 INFO:tasks.workunit.client.1.smithi111.stdout:7/178: readlink l5 0 2024-03-23T11:47:28.070 INFO:tasks.workunit.client.1.smithi111.stdout:5/267: dread - d1/d3/d6/d29/f54 zero size 2024-03-23T11:47:28.073 INFO:tasks.workunit.client.1.smithi111.stdout:5/268: fsync d1/d3/d6/f19 0 2024-03-23T11:47:28.073 INFO:tasks.workunit.client.1.smithi111.stdout:1/217: dwrite d0/d6/f34 [0,4194304] 0 2024-03-23T11:47:28.073 INFO:tasks.workunit.client.0.smithi063.stdout:0/196: mknod d2/d4/dd/c49 0 2024-03-23T11:47:28.074 INFO:tasks.workunit.client.0.smithi063.stdout:6/157: readlink d1/l10 0 2024-03-23T11:47:28.076 INFO:tasks.workunit.client.1.smithi111.stdout:0/217: mkdir d5/db/d1f/d47 0 2024-03-23T11:47:28.079 INFO:tasks.workunit.client.0.smithi063.stdout:4/185: truncate d5/da/d20/d24/d1f/f26 431717 0 2024-03-23T11:47:28.080 INFO:tasks.workunit.client.0.smithi063.stdout:0/197: truncate d2/d12/f44 128953 0 2024-03-23T11:47:28.081 INFO:tasks.workunit.client.0.smithi063.stdout:1/189: write d3/df/f26 [665691,6259] 0 2024-03-23T11:47:28.082 INFO:tasks.workunit.client.1.smithi111.stdout:3/184: creat d0/d1a/d2b/f33 x:0 0 0 2024-03-23T11:47:28.084 INFO:tasks.workunit.client.1.smithi111.stdout:2/237: dwrite d2/d11/f13 [0,4194304] 0 2024-03-23T11:47:28.087 INFO:tasks.workunit.client.1.smithi111.stdout:7/179: symlink d14/l27 0 2024-03-23T11:47:28.089 INFO:tasks.workunit.client.0.smithi063.stdout:3/300: dwrite d13/d22/d2d/f45 [0,4194304] 0 2024-03-23T11:47:28.092 INFO:tasks.workunit.client.0.smithi063.stdout:0/198: mknod d2/d4/dd/d25/c4a 0 2024-03-23T11:47:28.093 INFO:tasks.workunit.client.0.smithi063.stdout:1/190: mknod d3/df/d2f/c46 0 2024-03-23T11:47:28.095 INFO:tasks.workunit.client.1.smithi111.stdout:4/214: creat d0/d7/da/f46 x:0 0 0 2024-03-23T11:47:28.095 INFO:tasks.workunit.client.0.smithi063.stdout:8/390: dread d6/d1c/f4f [0,4194304] 0 2024-03-23T11:47:28.099 INFO:tasks.workunit.client.0.smithi063.stdout:7/208: dread d5/d22/f24 [0,4194304] 0 2024-03-23T11:47:28.108 INFO:tasks.workunit.client.0.smithi063.stdout:7/209: write d5/d6/f27 [4260858,110051] 0 2024-03-23T11:47:28.110 INFO:tasks.workunit.client.0.smithi063.stdout:0/199: symlink d2/d4/dd/d28/l4b 0 2024-03-23T11:47:28.111 INFO:tasks.workunit.client.1.smithi111.stdout:3/185: symlink d0/d1a/d1d/l34 0 2024-03-23T11:47:28.112 INFO:tasks.workunit.client.1.smithi111.stdout:0/218: dread d5/db/d1f/d19/f23 [0,4194304] 0 2024-03-23T11:47:28.114 INFO:tasks.workunit.client.1.smithi111.stdout:0/219: dread - d5/db/d1f/f37 zero size 2024-03-23T11:47:28.115 INFO:tasks.workunit.client.0.smithi063.stdout:8/391: creat d6/dd/d59/f82 x:0 0 0 2024-03-23T11:47:28.115 INFO:tasks.workunit.client.1.smithi111.stdout:0/220: fdatasync d5/db/f16 0 2024-03-23T11:47:28.116 INFO:tasks.workunit.client.0.smithi063.stdout:0/200: stat d2/d14/d36/l3a 0 2024-03-23T11:47:28.116 INFO:tasks.workunit.client.0.smithi063.stdout:1/191: dread d3/d1c/f1e [0,4194304] 0 2024-03-23T11:47:28.117 INFO:tasks.workunit.client.1.smithi111.stdout:0/221: write d5/f40 [1348739,20118] 0 2024-03-23T11:47:28.118 INFO:tasks.workunit.client.0.smithi063.stdout:3/301: creat d13/d22/d2d/d3f/d5b/f5d x:0 0 0 2024-03-23T11:47:28.118 INFO:tasks.workunit.client.0.smithi063.stdout:0/201: fdatasync f1 0 2024-03-23T11:47:28.120 INFO:tasks.workunit.client.0.smithi063.stdout:2/173: dread d2/d1c/f1f [0,4194304] 0 2024-03-23T11:47:28.120 INFO:tasks.workunit.client.0.smithi063.stdout:4/186: getdents d5/da/d20/d24/d16 0 2024-03-23T11:47:28.121 INFO:tasks.workunit.client.0.smithi063.stdout:8/392: stat d6/d1f/d26/f28 0 2024-03-23T11:47:28.123 INFO:tasks.workunit.client.0.smithi063.stdout:7/210: link d5/f12 d5/d22/f44 0 2024-03-23T11:47:28.126 INFO:tasks.workunit.client.1.smithi111.stdout:8/185: link d5/l2e d5/d26/d2f/l38 0 2024-03-23T11:47:28.126 INFO:tasks.workunit.client.1.smithi111.stdout:4/215: rmdir d0 39 2024-03-23T11:47:28.127 INFO:tasks.workunit.client.1.smithi111.stdout:6/286: sync 2024-03-23T11:47:28.130 INFO:tasks.workunit.client.1.smithi111.stdout:9/274: truncate d0/d4/d1c/f26 1369063 0 2024-03-23T11:47:28.133 INFO:tasks.workunit.client.0.smithi063.stdout:7/211: mkdir d5/d26/d45 0 2024-03-23T11:47:28.133 INFO:tasks.workunit.client.1.smithi111.stdout:3/186: symlink d0/d1a/l35 0 2024-03-23T11:47:28.135 INFO:tasks.workunit.client.1.smithi111.stdout:3/187: fsync d0/d1a/d2b/f33 0 2024-03-23T11:47:28.135 INFO:tasks.workunit.client.1.smithi111.stdout:2/238: mknod d2/de/c59 0 2024-03-23T11:47:28.137 INFO:tasks.workunit.client.1.smithi111.stdout:2/239: write d2/de/d12/f2c [628159,76210] 0 2024-03-23T11:47:28.140 INFO:tasks.workunit.client.1.smithi111.stdout:1/218: truncate d0/f3d 377910 0 2024-03-23T11:47:28.140 INFO:tasks.workunit.client.0.smithi063.stdout:4/187: truncate d5/da/d20/d24/f2d 485608 0 2024-03-23T11:47:28.142 INFO:tasks.workunit.client.0.smithi063.stdout:2/174: link d2/f17 d2/d1c/f3c 0 2024-03-23T11:47:28.143 INFO:tasks.workunit.client.1.smithi111.stdout:0/222: mkdir d5/db/d1f/d48 0 2024-03-23T11:47:28.143 INFO:tasks.workunit.client.0.smithi063.stdout:7/212: mknod d5/d16/c46 0 2024-03-23T11:47:28.143 INFO:tasks.workunit.client.1.smithi111.stdout:0/223: chown d5/db/d1f 0 1 2024-03-23T11:47:28.144 INFO:tasks.workunit.client.1.smithi111.stdout:8/186: creat d5/d26/d31/f39 x:0 0 0 2024-03-23T11:47:28.147 INFO:tasks.workunit.client.0.smithi063.stdout:6/158: dread d1/f5 [0,4194304] 0 2024-03-23T11:47:28.150 INFO:tasks.workunit.client.0.smithi063.stdout:2/175: readlink d2/de/l26 0 2024-03-23T11:47:28.150 INFO:tasks.workunit.client.0.smithi063.stdout:4/188: creat d5/da/d20/d24/f31 x:0 0 0 2024-03-23T11:47:28.151 INFO:tasks.workunit.client.1.smithi111.stdout:2/240: unlink d2/d7/da/l2a 0 2024-03-23T11:47:28.152 INFO:tasks.workunit.client.0.smithi063.stdout:1/192: dread d3/df/f30 [0,4194304] 0 2024-03-23T11:47:28.153 INFO:tasks.workunit.client.1.smithi111.stdout:1/219: mkdir d0/d6/d48/d4d 0 2024-03-23T11:47:28.154 INFO:tasks.workunit.client.1.smithi111.stdout:4/216: stat d0/d7/c10 0 2024-03-23T11:47:28.156 INFO:tasks.workunit.client.0.smithi063.stdout:1/193: chown d3/d3f 778 1 2024-03-23T11:47:28.156 INFO:tasks.workunit.client.0.smithi063.stdout:2/176: readlink d2/l31 0 2024-03-23T11:47:28.157 INFO:tasks.workunit.client.1.smithi111.stdout:6/287: unlink d2/l9 0 2024-03-23T11:47:28.157 INFO:tasks.workunit.client.0.smithi063.stdout:3/302: dwrite f12 [0,4194304] 0 2024-03-23T11:47:28.158 INFO:tasks.workunit.client.1.smithi111.stdout:0/224: dwrite d5/db/f26 [4194304,4194304] 0 2024-03-23T11:47:28.160 INFO:tasks.workunit.client.1.smithi111.stdout:6/288: write d2/d1a/d1c/d2c/d51/f61 [429639,34294] 0 2024-03-23T11:47:28.161 INFO:tasks.workunit.client.0.smithi063.stdout:4/189: mkdir d5/da/d20/d24/d16/d32 0 2024-03-23T11:47:28.161 INFO:tasks.workunit.client.0.smithi063.stdout:0/202: dwrite d2/d4/f9 [0,4194304] 0 2024-03-23T11:47:28.162 INFO:tasks.workunit.client.1.smithi111.stdout:8/187: unlink d5/cd 0 2024-03-23T11:47:28.162 INFO:tasks.workunit.client.0.smithi063.stdout:6/159: write d1/d13/f1e [841907,27886] 0 2024-03-23T11:47:28.166 INFO:tasks.workunit.client.0.smithi063.stdout:3/303: readlink d13/d22/d2d/l35 0 2024-03-23T11:47:28.166 INFO:tasks.workunit.client.0.smithi063.stdout:1/194: unlink d3/df/f32 0 2024-03-23T11:47:28.173 INFO:tasks.workunit.client.0.smithi063.stdout:3/304: write f5 [5704653,16663] 0 2024-03-23T11:47:28.176 INFO:tasks.workunit.client.0.smithi063.stdout:8/393: dread d6/f16 [0,4194304] 0 2024-03-23T11:47:28.177 INFO:tasks.workunit.client.0.smithi063.stdout:3/305: dread - d13/d22/d2d/d3f/d5b/f5d zero size 2024-03-23T11:47:28.179 INFO:tasks.workunit.client.1.smithi111.stdout:7/180: getdents d14 0 2024-03-23T11:47:28.183 INFO:tasks.workunit.client.0.smithi063.stdout:3/306: write f12 [4972695,95976] 0 2024-03-23T11:47:28.185 INFO:tasks.workunit.client.0.smithi063.stdout:7/213: dread d5/d6/d19/d1f/d37/d20/f28 [0,4194304] 0 2024-03-23T11:47:28.186 INFO:tasks.workunit.client.1.smithi111.stdout:4/217: symlink d0/dc/l47 0 2024-03-23T11:47:28.187 INFO:tasks.workunit.client.0.smithi063.stdout:7/214: chown d5 49047337 1 2024-03-23T11:47:28.188 INFO:tasks.workunit.client.0.smithi063.stdout:4/190: stat d5/da/l11 0 2024-03-23T11:47:28.188 INFO:tasks.workunit.client.0.smithi063.stdout:1/195: fdatasync d3/df/f19 0 2024-03-23T11:47:28.189 INFO:tasks.workunit.client.1.smithi111.stdout:3/188: truncate d0/d1a/f1c 3515667 0 2024-03-23T11:47:28.189 INFO:tasks.workunit.client.1.smithi111.stdout:8/188: creat d5/d1b/f3a x:0 0 0 2024-03-23T11:47:28.193 INFO:tasks.workunit.client.0.smithi063.stdout:6/160: symlink d1/l2b 0 2024-03-23T11:47:28.193 INFO:tasks.workunit.client.0.smithi063.stdout:8/394: write d6/d1c/d27/f31 [705226,21944] 0 2024-03-23T11:47:28.193 INFO:tasks.workunit.client.0.smithi063.stdout:0/203: dread d2/d14/f31 [0,4194304] 0 2024-03-23T11:47:28.194 INFO:tasks.workunit.client.0.smithi063.stdout:3/307: mkdir d13/d22/d2d/d5e 0 2024-03-23T11:47:28.195 INFO:tasks.workunit.client.1.smithi111.stdout:2/241: dread d2/de/d12/f29 [0,4194304] 0 2024-03-23T11:47:28.195 INFO:tasks.workunit.client.0.smithi063.stdout:0/204: readlink d2/d4/l38 0 2024-03-23T11:47:28.196 INFO:tasks.workunit.client.1.smithi111.stdout:8/189: write d5/f1a [4323243,34251] 0 2024-03-23T11:47:28.197 INFO:tasks.workunit.client.0.smithi063.stdout:6/161: write d1/dc/f1b [68858,34903] 0 2024-03-23T11:47:28.200 INFO:tasks.workunit.client.0.smithi063.stdout:0/205: stat d2/d4/l38 0 2024-03-23T11:47:28.200 INFO:tasks.workunit.client.0.smithi063.stdout:1/196: rename c0 to d3/df/c47 0 2024-03-23T11:47:28.202 INFO:tasks.workunit.client.0.smithi063.stdout:7/215: truncate d5/f12 3983313 0 2024-03-23T11:47:28.203 INFO:tasks.workunit.client.0.smithi063.stdout:6/162: truncate d1/dc/f24 540696 0 2024-03-23T11:47:28.206 INFO:tasks.workunit.client.0.smithi063.stdout:9/177: sync 2024-03-23T11:47:28.206 INFO:tasks.workunit.client.0.smithi063.stdout:5/89: sync 2024-03-23T11:47:28.207 INFO:tasks.workunit.client.0.smithi063.stdout:2/177: truncate d2/f1e 179250 0 2024-03-23T11:47:28.208 INFO:tasks.workunit.client.0.smithi063.stdout:1/197: chown d3/d35/d40/f42 8413 1 2024-03-23T11:47:28.208 INFO:tasks.workunit.client.1.smithi111.stdout:6/289: getdents d2 0 2024-03-23T11:47:28.208 INFO:tasks.workunit.client.1.smithi111.stdout:4/218: dwrite d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:28.210 INFO:tasks.workunit.client.0.smithi063.stdout:3/308: rename d13/d15/c59 to d13/d15/d48/c5f 0 2024-03-23T11:47:28.210 INFO:tasks.workunit.client.0.smithi063.stdout:6/163: rmdir d1 39 2024-03-23T11:47:28.211 INFO:tasks.workunit.client.0.smithi063.stdout:2/178: write d2/de/f2b [125626,17542] 0 2024-03-23T11:47:28.211 INFO:tasks.workunit.client.0.smithi063.stdout:1/198: fdatasync d3/df/f1a 0 2024-03-23T11:47:28.212 INFO:tasks.workunit.client.1.smithi111.stdout:1/220: dread d0/d5/dd/f25 [0,4194304] 0 2024-03-23T11:47:28.213 INFO:tasks.workunit.client.0.smithi063.stdout:7/216: write d5/d22/f24 [4129151,127866] 0 2024-03-23T11:47:28.213 INFO:tasks.workunit.client.1.smithi111.stdout:7/181: dread f12 [0,4194304] 0 2024-03-23T11:47:28.214 INFO:tasks.workunit.client.0.smithi063.stdout:9/178: creat d4/d31/f3d x:0 0 0 2024-03-23T11:47:28.217 INFO:tasks.workunit.client.1.smithi111.stdout:6/290: unlink l1 0 2024-03-23T11:47:28.219 INFO:tasks.workunit.client.1.smithi111.stdout:7/182: symlink d14/d1b/d1e/l28 0 2024-03-23T11:47:28.220 INFO:tasks.workunit.client.1.smithi111.stdout:3/189: dwrite d0/d7/f12 [0,4194304] 0 2024-03-23T11:47:28.223 INFO:tasks.workunit.client.1.smithi111.stdout:3/190: fdatasync d0/fd 0 2024-03-23T11:47:28.225 INFO:tasks.workunit.client.1.smithi111.stdout:1/221: truncate d0/d5/dd/f10 6096975 0 2024-03-23T11:47:28.226 INFO:tasks.workunit.client.1.smithi111.stdout:9/275: link d0/d4/fe d0/f5f 0 2024-03-23T11:47:28.226 INFO:tasks.workunit.client.0.smithi063.stdout:3/309: chown lb 2088019226 1 2024-03-23T11:47:28.226 INFO:tasks.workunit.client.0.smithi063.stdout:4/191: dread d5/da/d20/d24/d16/f2c [0,4194304] 0 2024-03-23T11:47:28.227 INFO:tasks.workunit.client.0.smithi063.stdout:7/217: dread d5/d6/f27 [4194304,4194304] 0 2024-03-23T11:47:28.228 INFO:tasks.workunit.client.0.smithi063.stdout:5/90: creat d2/d7/f1c x:0 0 0 2024-03-23T11:47:28.230 INFO:tasks.workunit.client.0.smithi063.stdout:3/310: write d13/d15/f1f [2992825,63419] 0 2024-03-23T11:47:28.233 INFO:tasks.workunit.client.1.smithi111.stdout:3/191: creat d0/d1a/d1d/f36 x:0 0 0 2024-03-23T11:47:28.235 INFO:tasks.workunit.client.1.smithi111.stdout:1/222: mknod d0/d6/c4e 0 2024-03-23T11:47:28.236 INFO:tasks.workunit.client.0.smithi063.stdout:8/395: dwrite d6/dd/f47 [0,4194304] 0 2024-03-23T11:47:28.237 INFO:tasks.workunit.client.1.smithi111.stdout:2/242: write d2/f4 [2867831,110763] 0 2024-03-23T11:47:28.237 INFO:tasks.workunit.client.1.smithi111.stdout:4/219: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:28.237 INFO:tasks.workunit.client.0.smithi063.stdout:6/164: mknod d1/dc/d21/c2c 0 2024-03-23T11:47:28.241 INFO:tasks.workunit.client.1.smithi111.stdout:9/276: rename d0/d4/d1c/c3a to d0/d31/d41/c60 0 2024-03-23T11:47:28.242 INFO:tasks.workunit.client.1.smithi111.stdout:7/183: creat d14/d1b/d22/f29 x:0 0 0 2024-03-23T11:47:28.242 INFO:tasks.workunit.client.1.smithi111.stdout:6/291: link d2/d1f/d45/f62 d2/d1a/f64 0 2024-03-23T11:47:28.245 INFO:tasks.workunit.client.1.smithi111.stdout:1/223: write d0/f1 [3101263,38413] 0 2024-03-23T11:47:28.246 INFO:tasks.workunit.client.1.smithi111.stdout:4/220: chown d0/d7/da/c27 3 1 2024-03-23T11:47:28.246 INFO:tasks.workunit.client.1.smithi111.stdout:6/292: chown d2/l4 138254782 1 2024-03-23T11:47:28.247 INFO:tasks.workunit.client.1.smithi111.stdout:7/184: write fe [1023922,64521] 0 2024-03-23T11:47:28.250 INFO:tasks.workunit.client.0.smithi063.stdout:1/199: dwrite d3/d1c/f3c [0,4194304] 0 2024-03-23T11:47:28.251 INFO:tasks.workunit.client.1.smithi111.stdout:2/243: dread - d2/d7/da/d19/f33 zero size 2024-03-23T11:47:28.251 INFO:tasks.workunit.client.1.smithi111.stdout:4/221: chown d0/d7/da/c37 630 1 2024-03-23T11:47:28.255 INFO:tasks.workunit.client.1.smithi111.stdout:2/244: fdatasync d2/d7/fd 0 2024-03-23T11:47:28.256 INFO:tasks.workunit.client.0.smithi063.stdout:3/311: chown d13/c32 281 1 2024-03-23T11:47:28.259 INFO:tasks.workunit.client.1.smithi111.stdout:9/277: getdents d0/d4/d10 0 2024-03-23T11:47:28.260 INFO:tasks.workunit.client.1.smithi111.stdout:7/185: getdents d14/d1b/d20 0 2024-03-23T11:47:28.262 INFO:tasks.workunit.client.0.smithi063.stdout:7/218: rename d5/d26/d45 to d5/d6/d19/d1f/d47 0 2024-03-23T11:47:28.262 INFO:tasks.workunit.client.1.smithi111.stdout:3/192: dwrite d0/d1a/d1d/f36 [0,4194304] 0 2024-03-23T11:47:28.266 INFO:tasks.workunit.client.0.smithi063.stdout:8/396: write d6/d1f/d54/d5c/d66/f51 [65371,32757] 0 2024-03-23T11:47:28.267 INFO:tasks.workunit.client.0.smithi063.stdout:5/91: dwrite d2/d7/f14 [0,4194304] 0 2024-03-23T11:47:28.268 INFO:tasks.workunit.client.0.smithi063.stdout:6/165: rename d1/dc/f1b to d1/dc/f2d 0 2024-03-23T11:47:28.269 INFO:tasks.workunit.client.1.smithi111.stdout:9/278: creat d0/d4/d1c/d2d/d3b/d42/d4f/f61 x:0 0 0 2024-03-23T11:47:28.270 INFO:tasks.workunit.client.1.smithi111.stdout:4/222: getdents d0/d17 0 2024-03-23T11:47:28.271 INFO:tasks.workunit.client.1.smithi111.stdout:2/245: rename d2/d7/da/l23 to d2/d32/l5a 0 2024-03-23T11:47:28.271 INFO:tasks.workunit.client.1.smithi111.stdout:7/186: mknod d14/d1b/d20/c2a 0 2024-03-23T11:47:28.273 INFO:tasks.workunit.client.1.smithi111.stdout:4/223: dread - d0/d17/d29/d2a/f40 zero size 2024-03-23T11:47:28.273 INFO:tasks.workunit.client.1.smithi111.stdout:7/187: stat f1 0 2024-03-23T11:47:28.273 INFO:tasks.workunit.client.0.smithi063.stdout:5/92: truncate d2/d7/f1c 166513 0 2024-03-23T11:47:28.273 INFO:tasks.workunit.client.0.smithi063.stdout:2/179: link d2/de/l10 d2/d1c/l3d 0 2024-03-23T11:47:28.275 INFO:tasks.workunit.client.1.smithi111.stdout:2/246: write d2/d11/d17/d1f/f20 [4241500,49844] 0 2024-03-23T11:47:28.275 INFO:tasks.workunit.client.1.smithi111.stdout:9/279: symlink d0/d31/d35/l62 0 2024-03-23T11:47:28.276 INFO:tasks.workunit.client.0.smithi063.stdout:0/206: truncate d2/d14/f31 3544362 0 2024-03-23T11:47:28.278 INFO:tasks.workunit.client.0.smithi063.stdout:9/179: write d4/d11/f3c [7035000,36903] 0 2024-03-23T11:47:28.278 INFO:tasks.workunit.client.1.smithi111.stdout:3/193: rename d0/d7/c20 to d0/d1a/d1d/c37 0 2024-03-23T11:47:28.282 INFO:tasks.workunit.client.0.smithi063.stdout:0/207: write d2/d12/d32/f43 [6518407,114390] 0 2024-03-23T11:47:28.282 INFO:tasks.workunit.client.1.smithi111.stdout:4/224: mknod d0/d17/d29/d2a/c48 0 2024-03-23T11:47:28.287 INFO:tasks.workunit.client.1.smithi111.stdout:9/280: creat d0/d4/d1c/d2d/f63 x:0 0 0 2024-03-23T11:47:28.289 INFO:tasks.workunit.client.1.smithi111.stdout:9/281: dread - d0/d4/d1c/d2d/d3b/d42/d4f/f5c zero size 2024-03-23T11:47:28.290 INFO:tasks.workunit.client.0.smithi063.stdout:8/397: creat d6/d1f/d54/d5c/d66/f83 x:0 0 0 2024-03-23T11:47:28.290 INFO:tasks.workunit.client.1.smithi111.stdout:7/188: rename d14/f19 to d14/d1b/d20/f2b 0 2024-03-23T11:47:28.293 INFO:tasks.workunit.client.1.smithi111.stdout:9/282: write d0/d4/d1c/d2d/d3b/d42/d4f/f5c [385987,112840] 0 2024-03-23T11:47:28.294 INFO:tasks.workunit.client.1.smithi111.stdout:3/194: mkdir d0/d1a/d2b/d38 0 2024-03-23T11:47:28.295 INFO:tasks.workunit.client.1.smithi111.stdout:9/283: write d0/d4/d1c/d36/f11 [136849,66911] 0 2024-03-23T11:47:28.309 INFO:tasks.workunit.client.0.smithi063.stdout:5/93: mkdir d2/d7/d8/d1d 0 2024-03-23T11:47:28.312 INFO:tasks.workunit.client.0.smithi063.stdout:7/219: mknod d5/c48 0 2024-03-23T11:47:28.315 INFO:tasks.workunit.client.0.smithi063.stdout:9/180: rmdir d4/d2c 39 2024-03-23T11:47:28.315 INFO:tasks.workunit.client.0.smithi063.stdout:1/200: dread d3/df/d31/f41 [0,4194304] 0 2024-03-23T11:47:28.315 INFO:tasks.workunit.client.0.smithi063.stdout:8/398: dread d6/d1f/d26/f35 [0,4194304] 0 2024-03-23T11:47:28.319 INFO:tasks.workunit.client.0.smithi063.stdout:9/181: chown d4/d11/c39 2201060 1 2024-03-23T11:47:28.320 INFO:tasks.workunit.client.1.smithi111.stdout:7/189: dread d14/f1c [0,4194304] 0 2024-03-23T11:47:28.322 INFO:tasks.workunit.client.1.smithi111.stdout:4/225: unlink d0/dc/f41 0 2024-03-23T11:47:28.323 INFO:tasks.workunit.client.0.smithi063.stdout:3/312: rename d13/d22/l4b to d13/d22/l60 0 2024-03-23T11:47:28.323 INFO:tasks.workunit.client.1.smithi111.stdout:9/284: dread d0/d4/fe [0,4194304] 0 2024-03-23T11:47:28.323 INFO:tasks.workunit.client.0.smithi063.stdout:5/94: read d2/d7/f10 [17085,96773] 0 2024-03-23T11:47:28.334 INFO:tasks.workunit.client.0.smithi063.stdout:2/180: dread d2/d1c/f3c [4194304,4194304] 0 2024-03-23T11:47:28.341 INFO:tasks.workunit.client.0.smithi063.stdout:1/201: dread d3/f10 [0,4194304] 0 2024-03-23T11:47:28.342 INFO:tasks.workunit.client.0.smithi063.stdout:6/166: dwrite d1/f8 [0,4194304] 0 2024-03-23T11:47:28.342 INFO:tasks.workunit.client.0.smithi063.stdout:5/95: mknod d2/d7/c1e 0 2024-03-23T11:47:28.347 INFO:tasks.workunit.client.0.smithi063.stdout:0/208: rename d2/d4/dd/d25/c4a to d2/c4c 0 2024-03-23T11:47:28.349 INFO:tasks.workunit.client.0.smithi063.stdout:2/181: creat d2/d27/f3e x:0 0 0 2024-03-23T11:47:28.356 INFO:tasks.workunit.client.0.smithi063.stdout:0/209: chown d2/d4/f30 7333849 1 2024-03-23T11:47:28.356 INFO:tasks.workunit.client.0.smithi063.stdout:7/220: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:28.357 INFO:tasks.workunit.client.0.smithi063.stdout:8/399: dwrite d6/dd/df/f2c [0,4194304] 0 2024-03-23T11:47:28.358 INFO:tasks.workunit.client.0.smithi063.stdout:9/182: link d4/c1f d4/c3e 0 2024-03-23T11:47:28.360 INFO:tasks.workunit.client.0.smithi063.stdout:2/182: rmdir d2/d27 39 2024-03-23T11:47:28.361 INFO:tasks.workunit.client.0.smithi063.stdout:0/210: write f1 [509406,101719] 0 2024-03-23T11:47:28.363 INFO:tasks.workunit.client.0.smithi063.stdout:3/313: write d13/f30 [2829040,50568] 0 2024-03-23T11:47:28.366 INFO:tasks.workunit.client.0.smithi063.stdout:2/183: dread - d2/d1c/f30 zero size 2024-03-23T11:47:28.366 INFO:tasks.workunit.client.0.smithi063.stdout:7/221: creat d5/d26/d32/f49 x:0 0 0 2024-03-23T11:47:28.367 INFO:tasks.workunit.client.0.smithi063.stdout:0/211: fdatasync d2/d12/f21 0 2024-03-23T11:47:28.367 INFO:tasks.workunit.client.1.smithi111.stdout:5/269: sync 2024-03-23T11:47:28.367 INFO:tasks.workunit.client.1.smithi111.stdout:4/226: mkdir d0/dc/d49 0 2024-03-23T11:47:28.368 INFO:tasks.workunit.client.0.smithi063.stdout:8/400: creat d6/d1f/d3e/f84 x:0 0 0 2024-03-23T11:47:28.368 INFO:tasks.workunit.client.1.smithi111.stdout:6/293: dwrite d2/f16 [0,4194304] 0 2024-03-23T11:47:28.370 INFO:tasks.workunit.client.0.smithi063.stdout:6/167: link d1/d13/l23 d1/d15/d28/l2e 0 2024-03-23T11:47:28.372 INFO:tasks.workunit.client.1.smithi111.stdout:6/294: dread - d2/f3b zero size 2024-03-23T11:47:28.373 INFO:tasks.workunit.client.0.smithi063.stdout:1/202: dwrite d3/df/f12 [0,4194304] 0 2024-03-23T11:47:28.373 INFO:tasks.workunit.client.1.smithi111.stdout:5/270: rename d1/d3 to d1/d3/d23/d5b 22 2024-03-23T11:47:28.374 INFO:tasks.workunit.client.0.smithi063.stdout:2/184: write d2/d23/d2e/f35 [438270,55745] 0 2024-03-23T11:47:28.374 INFO:tasks.workunit.client.0.smithi063.stdout:3/314: rmdir d13/d15 39 2024-03-23T11:47:28.375 INFO:tasks.workunit.client.1.smithi111.stdout:4/227: write d0/dc/f1c [4326050,11802] 0 2024-03-23T11:47:28.376 INFO:tasks.workunit.client.0.smithi063.stdout:6/168: creat d1/dc/d21/f2f x:0 0 0 2024-03-23T11:47:28.377 INFO:tasks.workunit.client.0.smithi063.stdout:9/183: dread d4/fc [0,4194304] 0 2024-03-23T11:47:28.378 INFO:tasks.workunit.client.0.smithi063.stdout:8/401: fsync d6/dd/df/f60 0 2024-03-23T11:47:28.379 INFO:tasks.workunit.client.0.smithi063.stdout:1/203: write d3/f7 [4470606,12669] 0 2024-03-23T11:47:28.383 INFO:tasks.workunit.client.0.smithi063.stdout:8/402: fdatasync d6/d1f/d26/f49 0 2024-03-23T11:47:28.383 INFO:tasks.workunit.client.1.smithi111.stdout:7/190: dread fb [0,4194304] 0 2024-03-23T11:47:28.384 INFO:tasks.workunit.client.0.smithi063.stdout:3/315: mknod d13/d22/d2d/c61 0 2024-03-23T11:47:28.389 INFO:tasks.workunit.client.1.smithi111.stdout:5/271: rmdir d1/d20/d25 39 2024-03-23T11:47:28.389 INFO:tasks.workunit.client.1.smithi111.stdout:4/228: creat d0/dc/d1f/f4a x:0 0 0 2024-03-23T11:47:28.390 INFO:tasks.workunit.client.1.smithi111.stdout:9/285: dwrite d0/d4/d1c/d36/f5 [4194304,4194304] 0 2024-03-23T11:47:28.392 INFO:tasks.workunit.client.0.smithi063.stdout:0/212: symlink d2/d4/dd/l4d 0 2024-03-23T11:47:28.393 INFO:tasks.workunit.client.0.smithi063.stdout:9/184: creat d4/d2c/f3f x:0 0 0 2024-03-23T11:47:28.394 INFO:tasks.workunit.client.0.smithi063.stdout:1/204: rename d3/d1c/l1d to d3/df/l48 0 2024-03-23T11:47:28.396 INFO:tasks.workunit.client.0.smithi063.stdout:9/185: readlink d4/d11/d19/l1d 0 2024-03-23T11:47:28.398 INFO:tasks.workunit.client.0.smithi063.stdout:7/222: link d5/c33 d5/c4a 0 2024-03-23T11:47:28.399 INFO:tasks.workunit.client.0.smithi063.stdout:0/213: fsync d2/d14/f31 0 2024-03-23T11:47:28.401 INFO:tasks.workunit.client.1.smithi111.stdout:2/247: dwrite d2/d7/da/d19/f49 [0,4194304] 0 2024-03-23T11:47:28.401 INFO:tasks.workunit.client.1.smithi111.stdout:5/272: stat d1/d3/d6/d7/f36 0 2024-03-23T11:47:28.401 INFO:tasks.workunit.client.0.smithi063.stdout:5/96: dread d2/fc [0,4194304] 0 2024-03-23T11:47:28.403 INFO:tasks.workunit.client.0.smithi063.stdout:9/186: write d4/f14 [4149578,84841] 0 2024-03-23T11:47:28.403 INFO:tasks.workunit.client.0.smithi063.stdout:7/223: symlink d5/d22/l4b 0 2024-03-23T11:47:28.404 INFO:tasks.workunit.client.0.smithi063.stdout:5/97: stat d2 0 2024-03-23T11:47:28.406 INFO:tasks.workunit.client.1.smithi111.stdout:2/248: readlink d2/d7/da/lb 0 2024-03-23T11:47:28.409 INFO:tasks.workunit.client.0.smithi063.stdout:7/224: write d5/d6/d19/d1f/d37/f3b [921228,130159] 0 2024-03-23T11:47:28.411 INFO:tasks.workunit.client.1.smithi111.stdout:4/229: getdents d0/dc/d30 0 2024-03-23T11:47:28.411 INFO:tasks.workunit.client.1.smithi111.stdout:9/286: truncate d0/d4/d1c/f45 155937 0 2024-03-23T11:47:28.412 INFO:tasks.workunit.client.1.smithi111.stdout:2/249: write d2/fc [8078016,42525] 0 2024-03-23T11:47:28.414 INFO:tasks.workunit.client.1.smithi111.stdout:5/273: mkdir d1/d20/d5c 0 2024-03-23T11:47:28.414 INFO:tasks.workunit.client.0.smithi063.stdout:8/403: getdents d6/dd/df 0 2024-03-23T11:47:28.416 INFO:tasks.workunit.client.1.smithi111.stdout:4/230: read - d0/dc/f35 zero size 2024-03-23T11:47:28.416 INFO:tasks.workunit.client.0.smithi063.stdout:8/404: read - d6/d1c/d27/f4a zero size 2024-03-23T11:47:28.418 INFO:tasks.workunit.client.1.smithi111.stdout:6/295: dwrite d2/ff [4194304,4194304] 0 2024-03-23T11:47:28.418 INFO:tasks.workunit.client.1.smithi111.stdout:7/191: dwrite f11 [0,4194304] 0 2024-03-23T11:47:28.418 INFO:tasks.workunit.client.0.smithi063.stdout:7/225: write d5/d6/d19/d1f/d37/d20/f35 [769878,41389] 0 2024-03-23T11:47:28.425 INFO:tasks.workunit.client.0.smithi063.stdout:3/316: getdents d13/d22/d2d/d3f 0 2024-03-23T11:47:28.427 INFO:tasks.workunit.client.0.smithi063.stdout:5/98: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:28.427 INFO:tasks.workunit.client.0.smithi063.stdout:0/214: mknod d2/d4/dd/c4e 0 2024-03-23T11:47:28.431 INFO:tasks.workunit.client.0.smithi063.stdout:1/205: dwrite d3/df/f26 [0,4194304] 0 2024-03-23T11:47:28.431 INFO:tasks.workunit.client.0.smithi063.stdout:3/317: chown d13/d22/d2d/f3a 2675 1 2024-03-23T11:47:28.438 INFO:tasks.workunit.client.1.smithi111.stdout:9/287: dread d0/d4/d10/f17 [0,4194304] 0 2024-03-23T11:47:28.440 INFO:tasks.workunit.client.1.smithi111.stdout:7/192: mkdir d14/d1b/d1e/d2c 0 2024-03-23T11:47:28.440 INFO:tasks.workunit.client.0.smithi063.stdout:9/187: creat d4/d11/f40 x:0 0 0 2024-03-23T11:47:28.442 INFO:tasks.workunit.client.0.smithi063.stdout:8/405: mkdir d6/d1f/d54/d5c/d76/d85 0 2024-03-23T11:47:28.445 INFO:tasks.workunit.client.1.smithi111.stdout:6/296: mkdir d2/d1f/d45/d65 0 2024-03-23T11:47:28.447 INFO:tasks.workunit.client.1.smithi111.stdout:5/274: truncate d1/d3/d6/f57 775698 0 2024-03-23T11:47:28.448 INFO:tasks.workunit.client.0.smithi063.stdout:9/188: chown d4/d11/d19/c1a 12303 1 2024-03-23T11:47:28.450 INFO:tasks.workunit.client.0.smithi063.stdout:8/406: write d6/d1c/f2d [177561,88147] 0 2024-03-23T11:47:28.451 INFO:tasks.workunit.client.0.smithi063.stdout:2/185: dwrite d2/d23/d2e/f36 [0,4194304] 0 2024-03-23T11:47:28.451 INFO:tasks.workunit.client.1.smithi111.stdout:9/288: mknod d0/d4/d10/d29/c64 0 2024-03-23T11:47:28.452 INFO:tasks.workunit.client.0.smithi063.stdout:5/99: dread d2/d7/f14 [0,4194304] 0 2024-03-23T11:47:28.454 INFO:tasks.workunit.client.0.smithi063.stdout:9/189: write d4/d11/f17 [820445,37582] 0 2024-03-23T11:47:28.455 INFO:tasks.workunit.client.1.smithi111.stdout:9/289: chown d0/d4/d1c/d36/f5 216614697 1 2024-03-23T11:47:28.456 INFO:tasks.workunit.client.0.smithi063.stdout:2/186: chown d2/f18 4629 1 2024-03-23T11:47:28.458 INFO:tasks.workunit.client.1.smithi111.stdout:2/250: creat d2/f5b x:0 0 0 2024-03-23T11:47:28.460 INFO:tasks.workunit.client.0.smithi063.stdout:0/215: mkdir d2/d4/dd/d1f/d4f 0 2024-03-23T11:47:28.460 INFO:tasks.workunit.client.0.smithi063.stdout:5/100: stat d2/d7/c9 0 2024-03-23T11:47:28.460 INFO:tasks.workunit.client.0.smithi063.stdout:1/206: write d3/f4 [60155,62918] 0 2024-03-23T11:47:28.460 INFO:tasks.workunit.client.1.smithi111.stdout:9/290: mkdir d0/d31/d41/d65 0 2024-03-23T11:47:28.462 INFO:tasks.workunit.client.1.smithi111.stdout:0/225: sync 2024-03-23T11:47:28.466 INFO:tasks.workunit.client.1.smithi111.stdout:6/297: getdents d2/d1a/d1c 0 2024-03-23T11:47:28.467 INFO:tasks.workunit.client.1.smithi111.stdout:9/291: creat d0/d4/d1c/f66 x:0 0 0 2024-03-23T11:47:28.472 INFO:tasks.workunit.client.1.smithi111.stdout:9/292: chown d0/d4/d1c/d2d/c30 797429221 1 2024-03-23T11:47:28.475 INFO:tasks.workunit.client.1.smithi111.stdout:2/251: dread d2/d11/d17/d1f/f20 [0,4194304] 0 2024-03-23T11:47:28.478 INFO:tasks.workunit.client.1.smithi111.stdout:2/252: dread - d2/d11/d17/d1f/d30/f53 zero size 2024-03-23T11:47:28.478 INFO:tasks.workunit.client.1.smithi111.stdout:4/231: truncate d0/f1 2011626 0 2024-03-23T11:47:28.480 INFO:tasks.workunit.client.1.smithi111.stdout:5/275: rename d1/d20/d25 to d1/d5d 0 2024-03-23T11:47:28.480 INFO:tasks.workunit.client.1.smithi111.stdout:0/226: dwrite d5/db/f26 [4194304,4194304] 0 2024-03-23T11:47:28.481 INFO:tasks.workunit.client.0.smithi063.stdout:9/190: write d4/d11/d19/f27 [779664,28728] 0 2024-03-23T11:47:28.482 INFO:tasks.workunit.client.0.smithi063.stdout:7/226: link d5/d26/d32/c36 d5/d6/d19/d1f/d37/d20/c4c 0 2024-03-23T11:47:28.482 INFO:tasks.workunit.client.0.smithi063.stdout:2/187: write d2/de/ff [3258250,63879] 0 2024-03-23T11:47:28.482 INFO:tasks.workunit.client.0.smithi063.stdout:3/318: creat d13/d15/d48/f62 x:0 0 0 2024-03-23T11:47:28.483 INFO:tasks.workunit.client.1.smithi111.stdout:6/298: symlink d2/l66 0 2024-03-23T11:47:28.488 INFO:tasks.workunit.client.1.smithi111.stdout:9/293: creat d0/d4/d10/d29/f67 x:0 0 0 2024-03-23T11:47:28.497 INFO:tasks.workunit.client.0.smithi063.stdout:7/227: truncate d5/d6/f27 2603358 0 2024-03-23T11:47:28.498 INFO:tasks.workunit.client.0.smithi063.stdout:0/216: dread d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:28.499 INFO:tasks.workunit.client.1.smithi111.stdout:2/253: mkdir d2/d5c 0 2024-03-23T11:47:28.499 INFO:tasks.workunit.client.0.smithi063.stdout:8/407: getdents d6/d1c/d32 0 2024-03-23T11:47:28.501 INFO:tasks.workunit.client.1.smithi111.stdout:2/254: chown d2/d11/d17/d1f/d30/f41 3065676 1 2024-03-23T11:47:28.502 INFO:tasks.workunit.client.0.smithi063.stdout:7/228: rename d5/d6 to d5/d6/d19/d4d 22 2024-03-23T11:47:28.504 INFO:tasks.workunit.client.0.smithi063.stdout:5/101: dwrite d2/d7/d8/f12 [0,4194304] 0 2024-03-23T11:47:28.505 INFO:tasks.workunit.client.0.smithi063.stdout:8/408: fsync d6/d1f/d26/d15/f65 0 2024-03-23T11:47:28.506 INFO:tasks.workunit.client.0.smithi063.stdout:3/319: unlink d13/d22/c26 0 2024-03-23T11:47:28.507 INFO:tasks.workunit.client.1.smithi111.stdout:0/227: creat d5/db/d1f/d19/d45/f49 x:0 0 0 2024-03-23T11:47:28.508 INFO:tasks.workunit.client.0.smithi063.stdout:7/229: chown d5/d6/d19 3208 1 2024-03-23T11:47:28.509 INFO:tasks.workunit.client.0.smithi063.stdout:9/191: creat d4/d11/f41 x:0 0 0 2024-03-23T11:47:28.510 INFO:tasks.workunit.client.1.smithi111.stdout:0/228: write d5/db/d1f/d19/f2a [1309663,24290] 0 2024-03-23T11:47:28.512 INFO:tasks.workunit.client.1.smithi111.stdout:2/255: unlink d2/d32/c36 0 2024-03-23T11:47:28.512 INFO:tasks.workunit.client.1.smithi111.stdout:9/294: dwrite d0/f48 [0,4194304] 0 2024-03-23T11:47:28.513 INFO:tasks.workunit.client.0.smithi063.stdout:2/188: rmdir d2/d3b 0 2024-03-23T11:47:28.514 INFO:tasks.workunit.client.1.smithi111.stdout:8/190: sync 2024-03-23T11:47:28.514 INFO:tasks.workunit.client.1.smithi111.stdout:3/195: sync 2024-03-23T11:47:28.514 INFO:tasks.workunit.client.1.smithi111.stdout:1/224: sync 2024-03-23T11:47:28.514 INFO:tasks.workunit.client.1.smithi111.stdout:6/299: mknod d2/d1f/d45/d49/c67 0 2024-03-23T11:47:28.515 INFO:tasks.workunit.client.0.smithi063.stdout:1/207: dwrite d3/f10 [0,4194304] 0 2024-03-23T11:47:28.517 INFO:tasks.workunit.client.0.smithi063.stdout:8/409: fsync d6/d1f/d54/d5c/d66/f51 0 2024-03-23T11:47:28.517 INFO:tasks.workunit.client.1.smithi111.stdout:0/229: dread - d5/db/d1f/f43 zero size 2024-03-23T11:47:28.520 INFO:tasks.workunit.client.1.smithi111.stdout:7/193: dread f1 [0,4194304] 0 2024-03-23T11:47:28.520 INFO:tasks.workunit.client.0.smithi063.stdout:2/189: chown d2/f18 18993 1 2024-03-23T11:47:28.522 INFO:tasks.workunit.client.0.smithi063.stdout:9/192: rmdir d4/d2c 39 2024-03-23T11:47:28.524 INFO:tasks.workunit.client.1.smithi111.stdout:1/225: creat d0/d2b/d37/f4f x:0 0 0 2024-03-23T11:47:28.525 INFO:tasks.workunit.client.0.smithi063.stdout:3/320: dread d13/f4d [0,4194304] 0 2024-03-23T11:47:28.525 INFO:tasks.workunit.client.1.smithi111.stdout:5/276: dwrite d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:28.530 INFO:tasks.workunit.client.0.smithi063.stdout:3/321: stat d13/d22/d2d/f45 0 2024-03-23T11:47:28.536 INFO:tasks.workunit.client.0.smithi063.stdout:0/217: dread f1 [0,4194304] 0 2024-03-23T11:47:28.538 INFO:tasks.workunit.client.0.smithi063.stdout:1/208: mkdir d3/d49 0 2024-03-23T11:47:28.542 INFO:tasks.workunit.client.1.smithi111.stdout:7/194: rename f7 to d14/d1b/f2d 0 2024-03-23T11:47:28.543 INFO:tasks.workunit.client.0.smithi063.stdout:8/410: rename d6/d1f/d26 to d6/d1c/d32/d37/d86 0 2024-03-23T11:47:28.545 INFO:tasks.workunit.client.1.smithi111.stdout:8/191: dread d5/fa [0,4194304] 0 2024-03-23T11:47:28.545 INFO:tasks.workunit.client.1.smithi111.stdout:2/256: dwrite d2/d7/da/d19/f34 [0,4194304] 0 2024-03-23T11:47:28.545 INFO:tasks.workunit.client.0.smithi063.stdout:8/411: chown d6/d1f/l81 2285 1 2024-03-23T11:47:28.548 INFO:tasks.workunit.client.0.smithi063.stdout:5/102: truncate d2/f5 6555455 0 2024-03-23T11:47:28.548 INFO:tasks.workunit.client.1.smithi111.stdout:0/230: link d5/db/d1f/f43 d5/db/f4a 0 2024-03-23T11:47:28.549 INFO:tasks.workunit.client.1.smithi111.stdout:3/196: write d0/d1a/f1c [1755574,23053] 0 2024-03-23T11:47:28.550 INFO:tasks.workunit.client.0.smithi063.stdout:8/412: chown d6/d1c/d32/d37/d86/d15/f24 4273 1 2024-03-23T11:47:28.550 INFO:tasks.workunit.client.1.smithi111.stdout:9/295: write d0/d4/d1c/f26 [1674479,21944] 0 2024-03-23T11:47:28.551 INFO:tasks.workunit.client.0.smithi063.stdout:9/193: creat d4/d11/d19/f42 x:0 0 0 2024-03-23T11:47:28.553 INFO:tasks.workunit.client.0.smithi063.stdout:3/322: fdatasync d13/d22/d2d/f38 0 2024-03-23T11:47:28.554 INFO:tasks.workunit.client.0.smithi063.stdout:3/323: readlink d13/l21 0 2024-03-23T11:47:28.560 INFO:tasks.workunit.client.1.smithi111.stdout:4/232: dread d0/dc/f1c [0,4194304] 0 2024-03-23T11:47:28.560 INFO:tasks.workunit.client.1.smithi111.stdout:2/257: getdents d2/d5c 0 2024-03-23T11:47:28.560 INFO:tasks.workunit.client.1.smithi111.stdout:8/192: mkdir d5/d26/d31/d3b 0 2024-03-23T11:47:28.561 INFO:tasks.workunit.client.1.smithi111.stdout:5/277: rename d1/d3/d3a/f3f to d1/d3/d6/d29/f5e 0 2024-03-23T11:47:28.561 INFO:tasks.workunit.client.0.smithi063.stdout:9/194: mkdir d4/d31/d43 0 2024-03-23T11:47:28.562 INFO:tasks.workunit.client.1.smithi111.stdout:0/231: write d5/db/d1f/d19/f1c [635385,71493] 0 2024-03-23T11:47:28.564 INFO:tasks.workunit.client.0.smithi063.stdout:7/230: write d5/d22/f44 [1283348,65917] 0 2024-03-23T11:47:28.565 INFO:tasks.workunit.client.1.smithi111.stdout:1/226: getdents d0/d5/dd/d1e 0 2024-03-23T11:47:28.565 INFO:tasks.workunit.client.1.smithi111.stdout:6/300: dwrite d2/d17/f33 [0,4194304] 0 2024-03-23T11:47:28.565 INFO:tasks.workunit.client.1.smithi111.stdout:8/193: mknod d5/d26/c3c 0 2024-03-23T11:47:28.567 INFO:tasks.workunit.client.0.smithi063.stdout:1/209: symlink d3/d49/l4a 0 2024-03-23T11:47:28.570 INFO:tasks.workunit.client.1.smithi111.stdout:0/232: chown d5/db/d1f/f18 251721 1 2024-03-23T11:47:28.571 INFO:tasks.workunit.client.1.smithi111.stdout:2/258: mkdir d2/d11/d17/d3e/d5d 0 2024-03-23T11:47:28.576 INFO:tasks.workunit.client.0.smithi063.stdout:9/195: mkdir d4/d11/d44 0 2024-03-23T11:47:28.576 INFO:tasks.workunit.client.0.smithi063.stdout:8/413: dwrite d6/d1f/d54/d5c/d66/f51 [0,4194304] 0 2024-03-23T11:47:28.578 INFO:tasks.workunit.client.0.smithi063.stdout:9/196: stat d4/d11/f17 0 2024-03-23T11:47:28.578 INFO:tasks.workunit.client.1.smithi111.stdout:8/194: write d5/d1b/f3a [938764,10989] 0 2024-03-23T11:47:28.580 INFO:tasks.workunit.client.1.smithi111.stdout:9/296: rename d0/d4/d1c/d36/f20 to d0/d4/d1c/d36/d28/f68 0 2024-03-23T11:47:28.580 INFO:tasks.workunit.client.0.smithi063.stdout:3/324: creat d13/d22/f63 x:0 0 0 2024-03-23T11:47:28.582 INFO:tasks.workunit.client.0.smithi063.stdout:0/218: getdents d2/d12/d35 0 2024-03-23T11:47:28.584 INFO:tasks.workunit.client.1.smithi111.stdout:0/233: read d5/db/d1f/d39/f3d [3781242,27793] 0 2024-03-23T11:47:28.586 INFO:tasks.workunit.client.1.smithi111.stdout:1/227: truncate d0/d6/f19 922090 0 2024-03-23T11:47:28.588 INFO:tasks.workunit.client.0.smithi063.stdout:3/325: dread d13/d22/f2c [0,4194304] 0 2024-03-23T11:47:28.590 INFO:tasks.workunit.client.0.smithi063.stdout:9/197: creat d4/d2c/f45 x:0 0 0 2024-03-23T11:47:28.591 INFO:tasks.workunit.client.1.smithi111.stdout:4/233: rename d0/dc/d49 to d0/d17/d29/d4b 0 2024-03-23T11:47:28.591 INFO:tasks.workunit.client.0.smithi063.stdout:0/219: truncate d2/d12/f42 74997 0 2024-03-23T11:47:28.593 INFO:tasks.workunit.client.0.smithi063.stdout:3/326: read - d13/d22/d2d/f4f zero size 2024-03-23T11:47:28.594 INFO:tasks.workunit.client.0.smithi063.stdout:3/327: fsync f3 0 2024-03-23T11:47:28.594 INFO:tasks.workunit.client.0.smithi063.stdout:8/414: rename f5 to d6/d1c/d32/d37/d86/d5d/f87 0 2024-03-23T11:47:28.597 INFO:tasks.workunit.client.1.smithi111.stdout:9/297: mkdir d0/d4/d1c/d36/d28/d69 0 2024-03-23T11:47:28.597 INFO:tasks.workunit.client.0.smithi063.stdout:0/220: read d2/d4/f30 [2184111,104940] 0 2024-03-23T11:47:28.598 INFO:tasks.workunit.client.1.smithi111.stdout:7/195: dwrite fe [0,4194304] 0 2024-03-23T11:47:28.599 INFO:tasks.workunit.client.1.smithi111.stdout:8/195: creat d5/d26/d31/d3b/f3d x:0 0 0 2024-03-23T11:47:28.600 INFO:tasks.workunit.client.0.smithi063.stdout:0/221: fsync d2/d12/f21 0 2024-03-23T11:47:28.601 INFO:tasks.workunit.client.1.smithi111.stdout:5/278: rename d1/d3/d6/ff to d1/d5d/f5f 0 2024-03-23T11:47:28.601 INFO:tasks.workunit.client.0.smithi063.stdout:7/231: dwrite d5/d6/d19/d1f/d37/f3c [0,4194304] 0 2024-03-23T11:47:28.603 INFO:tasks.workunit.client.1.smithi111.stdout:8/196: chown l4 5432 1 2024-03-23T11:47:28.606 INFO:tasks.workunit.client.1.smithi111.stdout:3/197: dwrite d0/d1a/f28 [0,4194304] 0 2024-03-23T11:47:28.606 INFO:tasks.workunit.client.1.smithi111.stdout:4/234: truncate d0/d17/f3d 878050 0 2024-03-23T11:47:28.606 INFO:tasks.workunit.client.0.smithi063.stdout:8/415: mknod d6/d1f/d54/d63/c88 0 2024-03-23T11:47:28.607 INFO:tasks.workunit.client.0.smithi063.stdout:7/232: write d5/d6/d19/d1f/d37/f3b [1076177,6681] 0 2024-03-23T11:47:28.612 INFO:tasks.workunit.client.0.smithi063.stdout:3/328: rename le to d13/d22/l64 0 2024-03-23T11:47:28.614 INFO:tasks.workunit.client.1.smithi111.stdout:4/235: readlink d0/l1a 0 2024-03-23T11:47:28.617 INFO:tasks.workunit.client.0.smithi063.stdout:3/329: truncate d13/d22/d2d/f53 610673 0 2024-03-23T11:47:28.618 INFO:tasks.workunit.client.0.smithi063.stdout:0/222: getdents d2/d14/d36 0 2024-03-23T11:47:28.622 INFO:tasks.workunit.client.0.smithi063.stdout:8/416: rename d6/d1c/d32/d37/d86/f10 to d6/f89 0 2024-03-23T11:47:28.623 INFO:tasks.workunit.client.0.smithi063.stdout:3/330: fdatasync d13/d22/f3b 0 2024-03-23T11:47:28.623 INFO:tasks.workunit.client.1.smithi111.stdout:6/301: rename d2/d1a/d1c/c53 to d2/d1a/d1c/d2c/d51/c68 0 2024-03-23T11:47:28.629 INFO:tasks.workunit.client.0.smithi063.stdout:7/233: dread d5/d6/d19/d1f/d37/f3c [0,4194304] 0 2024-03-23T11:47:28.630 INFO:tasks.workunit.client.1.smithi111.stdout:2/259: dread d2/d7/da/ff [0,4194304] 0 2024-03-23T11:47:28.634 INFO:tasks.workunit.client.1.smithi111.stdout:0/234: dwrite d5/f40 [0,4194304] 0 2024-03-23T11:47:28.637 INFO:tasks.workunit.client.1.smithi111.stdout:3/198: chown d0/d1a/d1d/d26/l32 2118 1 2024-03-23T11:47:28.640 INFO:tasks.workunit.client.1.smithi111.stdout:5/279: truncate d1/d3/d6/d7/f53 5288214 0 2024-03-23T11:47:28.641 INFO:tasks.workunit.client.1.smithi111.stdout:7/196: mkdir d14/d1b/d1e/d2c/d2e 0 2024-03-23T11:47:28.642 INFO:tasks.workunit.client.1.smithi111.stdout:9/298: rename d0/d4/d1c/d36/d28/d34 to d0/d4/d6a 0 2024-03-23T11:47:28.642 INFO:tasks.workunit.client.0.smithi063.stdout:7/234: creat d5/d6/d19/d1f/d37/da/f4e x:0 0 0 2024-03-23T11:47:28.644 INFO:tasks.workunit.client.1.smithi111.stdout:8/197: dread d5/f22 [0,4194304] 0 2024-03-23T11:47:28.644 INFO:tasks.workunit.client.1.smithi111.stdout:2/260: symlink d2/d11/d17/l5e 0 2024-03-23T11:47:28.647 INFO:tasks.workunit.client.0.smithi063.stdout:1/210: dread d3/f9 [0,4194304] 0 2024-03-23T11:47:28.649 INFO:tasks.workunit.client.0.smithi063.stdout:7/235: mkdir d5/d6/d19/d1f/d47/d4f 0 2024-03-23T11:47:28.649 INFO:tasks.workunit.client.0.smithi063.stdout:3/331: dwrite f8 [0,4194304] 0 2024-03-23T11:47:28.650 INFO:tasks.workunit.client.1.smithi111.stdout:7/197: symlink d14/d1b/d1e/l2f 0 2024-03-23T11:47:28.652 INFO:tasks.workunit.client.1.smithi111.stdout:4/236: getdents d0/d17/d29 0 2024-03-23T11:47:28.653 INFO:tasks.workunit.client.1.smithi111.stdout:8/198: rmdir d5/d26/d2f 39 2024-03-23T11:47:28.654 INFO:tasks.workunit.client.0.smithi063.stdout:9/198: write d4/f16 [2363570,99522] 0 2024-03-23T11:47:28.657 INFO:tasks.workunit.client.1.smithi111.stdout:6/302: rename d2/d1a/c32 to d2/d1f/d45/c69 0 2024-03-23T11:47:28.658 INFO:tasks.workunit.client.0.smithi063.stdout:9/199: dread - d4/d11/f41 zero size 2024-03-23T11:47:28.662 INFO:tasks.workunit.client.1.smithi111.stdout:7/198: mknod d14/d1b/c30 0 2024-03-23T11:47:28.663 INFO:tasks.workunit.client.1.smithi111.stdout:5/280: creat d1/d20/f60 x:0 0 0 2024-03-23T11:47:28.664 INFO:tasks.workunit.client.1.smithi111.stdout:3/199: dwrite d0/f31 [0,4194304] 0 2024-03-23T11:47:28.664 INFO:tasks.workunit.client.1.smithi111.stdout:9/299: creat d0/d4/d1c/d36/f6b x:0 0 0 2024-03-23T11:47:28.665 INFO:tasks.workunit.client.0.smithi063.stdout:9/200: symlink d4/d31/l46 0 2024-03-23T11:47:28.667 INFO:tasks.workunit.client.0.smithi063.stdout:4/192: sync 2024-03-23T11:47:28.668 INFO:tasks.workunit.client.1.smithi111.stdout:9/300: chown d0/d4/d1c/d2d/d3b 47266402 1 2024-03-23T11:47:28.670 INFO:tasks.workunit.client.1.smithi111.stdout:4/237: rename d0/d17/d29 to d0/dc/d4c 0 2024-03-23T11:47:28.670 INFO:tasks.workunit.client.0.smithi063.stdout:1/211: dread d3/d1c/f1e [0,4194304] 0 2024-03-23T11:47:28.671 INFO:tasks.workunit.client.1.smithi111.stdout:6/303: mknod d2/d1f/c6a 0 2024-03-23T11:47:28.671 INFO:tasks.workunit.client.0.smithi063.stdout:8/417: write d6/d1f/d54/d5c/f6d [1017159,119712] 0 2024-03-23T11:47:28.672 INFO:tasks.workunit.client.0.smithi063.stdout:9/201: read d4/d11/d19/f27 [720795,95757] 0 2024-03-23T11:47:28.672 INFO:tasks.workunit.client.0.smithi063.stdout:0/223: truncate d2/d12/d32/f43 3620557 0 2024-03-23T11:47:28.672 INFO:tasks.workunit.client.1.smithi111.stdout:7/199: creat d14/d1b/d1e/d2c/f31 x:0 0 0 2024-03-23T11:47:28.672 INFO:tasks.workunit.client.0.smithi063.stdout:4/193: write d5/da/f2a [336430,35140] 0 2024-03-23T11:47:28.674 INFO:tasks.workunit.client.1.smithi111.stdout:2/261: getdents d2/de/d27 0 2024-03-23T11:47:28.676 INFO:tasks.workunit.client.1.smithi111.stdout:9/301: write d0/d4/f15 [859296,43893] 0 2024-03-23T11:47:28.676 INFO:tasks.workunit.client.1.smithi111.stdout:7/200: chown f12 614 1 2024-03-23T11:47:28.676 INFO:tasks.workunit.client.0.smithi063.stdout:1/212: fdatasync d3/f39 0 2024-03-23T11:47:28.678 INFO:tasks.workunit.client.0.smithi063.stdout:9/202: chown d4/d11/d19/l29 3 1 2024-03-23T11:47:28.679 INFO:tasks.workunit.client.1.smithi111.stdout:2/262: readlink d2/d3d/l45 0 2024-03-23T11:47:28.681 INFO:tasks.workunit.client.0.smithi063.stdout:7/236: dread d5/d6/d19/d1f/d37/da/f17 [0,4194304] 0 2024-03-23T11:47:28.685 INFO:tasks.workunit.client.1.smithi111.stdout:1/228: dread d0/d5/f23 [0,4194304] 0 2024-03-23T11:47:28.687 INFO:tasks.workunit.client.1.smithi111.stdout:7/201: creat d14/d1b/f32 x:0 0 0 2024-03-23T11:47:28.687 INFO:tasks.workunit.client.0.smithi063.stdout:8/418: truncate d6/d1c/d32/d37/d86/f5f 1047423 0 2024-03-23T11:47:28.687 INFO:tasks.workunit.client.1.smithi111.stdout:2/263: symlink d2/d32/l5f 0 2024-03-23T11:47:28.690 INFO:tasks.workunit.client.1.smithi111.stdout:6/304: unlink d2/d1a/d1c/l25 0 2024-03-23T11:47:28.692 INFO:tasks.workunit.client.0.smithi063.stdout:8/419: read - d6/d1f/d54/d5c/d66/f83 zero size 2024-03-23T11:47:28.696 INFO:tasks.workunit.client.0.smithi063.stdout:1/213: mkdir d3/d1c/d4b 0 2024-03-23T11:47:28.697 INFO:tasks.workunit.client.1.smithi111.stdout:0/235: dread d5/f8 [0,4194304] 0 2024-03-23T11:47:28.700 INFO:tasks.workunit.client.1.smithi111.stdout:2/264: fsync d2/de/d12/f29 0 2024-03-23T11:47:28.700 INFO:tasks.workunit.client.1.smithi111.stdout:4/238: dwrite d0/d7/f1b [4194304,4194304] 0 2024-03-23T11:47:28.701 INFO:tasks.workunit.client.1.smithi111.stdout:1/229: mknod d0/d6/d48/d4a/c50 0 2024-03-23T11:47:28.702 INFO:tasks.workunit.client.1.smithi111.stdout:5/281: dwrite d1/d5d/f5f [0,4194304] 0 2024-03-23T11:47:28.702 INFO:tasks.workunit.client.0.smithi063.stdout:0/224: link d2/d14/f34 d2/d12/f50 0 2024-03-23T11:47:28.703 INFO:tasks.workunit.client.1.smithi111.stdout:1/230: fdatasync d0/d6/f42 0 2024-03-23T11:47:28.707 INFO:tasks.workunit.client.1.smithi111.stdout:7/202: mkdir d14/d1b/d1e/d2c/d2e/d33 0 2024-03-23T11:47:28.709 INFO:tasks.workunit.client.0.smithi063.stdout:4/194: dwrite d5/ff [4194304,4194304] 0 2024-03-23T11:47:28.711 INFO:tasks.workunit.client.0.smithi063.stdout:0/225: write d2/d4/dd/d1f/f2d [751425,92480] 0 2024-03-23T11:47:28.712 INFO:tasks.workunit.client.1.smithi111.stdout:8/199: truncate d5/d1b/f3a 307038 0 2024-03-23T11:47:28.715 INFO:tasks.workunit.client.1.smithi111.stdout:0/236: mkdir d5/db/d1f/d19/d45/d4b 0 2024-03-23T11:47:28.715 INFO:tasks.workunit.client.0.smithi063.stdout:3/332: dread f6 [0,4194304] 0 2024-03-23T11:47:28.716 INFO:tasks.workunit.client.1.smithi111.stdout:4/239: mkdir d0/dc/d4c/d4b/d4d 0 2024-03-23T11:47:28.718 INFO:tasks.workunit.client.1.smithi111.stdout:2/265: dread d2/d7/da/f3b [0,4194304] 0 2024-03-23T11:47:28.718 INFO:tasks.workunit.client.0.smithi063.stdout:9/203: truncate d4/d11/d19/f27 419002 0 2024-03-23T11:47:28.719 INFO:tasks.workunit.client.1.smithi111.stdout:7/203: write d14/d1b/d22/f29 [623221,98933] 0 2024-03-23T11:47:28.720 INFO:tasks.workunit.client.0.smithi063.stdout:8/420: symlink d6/d30/l8a 0 2024-03-23T11:47:28.722 INFO:tasks.workunit.client.0.smithi063.stdout:9/204: stat d4/d11/f30 0 2024-03-23T11:47:28.724 INFO:tasks.workunit.client.0.smithi063.stdout:3/333: write d13/d22/d2d/f53 [532305,94211] 0 2024-03-23T11:47:28.726 INFO:tasks.workunit.client.0.smithi063.stdout:9/205: write d4/d11/f41 [134798,81049] 0 2024-03-23T11:47:28.728 INFO:tasks.workunit.client.0.smithi063.stdout:7/237: rmdir d5/d6/d19/d1f/d37/d40 0 2024-03-23T11:47:28.728 INFO:tasks.workunit.client.0.smithi063.stdout:1/214: dwrite d3/df/f19 [0,4194304] 0 2024-03-23T11:47:28.733 INFO:tasks.workunit.client.1.smithi111.stdout:1/231: dwrite d0/d5/dd/d1e/f31 [0,4194304] 0 2024-03-23T11:47:28.734 INFO:tasks.workunit.client.0.smithi063.stdout:7/238: truncate d5/f8 2294646 0 2024-03-23T11:47:28.738 INFO:tasks.workunit.client.0.smithi063.stdout:7/239: chown d5/d6/d19/d1f/d37/d20/f28 18430 1 2024-03-23T11:47:28.741 INFO:tasks.workunit.client.1.smithi111.stdout:9/302: dread d0/f13 [0,4194304] 0 2024-03-23T11:47:28.742 INFO:tasks.workunit.client.0.smithi063.stdout:4/195: mknod d5/da/d20/d24/d16/d32/c33 0 2024-03-23T11:47:28.743 INFO:tasks.workunit.client.1.smithi111.stdout:6/305: link d2/d17/f4d d2/d1a/d1c/f6b 0 2024-03-23T11:47:28.745 INFO:tasks.workunit.client.0.smithi063.stdout:9/206: rmdir d4/d31 39 2024-03-23T11:47:28.745 INFO:tasks.workunit.client.0.smithi063.stdout:0/226: mkdir d2/d4/dd/d28/d46/d51 0 2024-03-23T11:47:28.746 INFO:tasks.workunit.client.1.smithi111.stdout:9/303: fsync d0/d4/d1c/f26 0 2024-03-23T11:47:28.748 INFO:tasks.workunit.client.0.smithi063.stdout:1/215: creat d3/d35/d40/f4c x:0 0 0 2024-03-23T11:47:28.749 INFO:tasks.workunit.client.0.smithi063.stdout:4/196: rename d5/da/d20 to d5/da/d20/d24/d16/d32/d34 22 2024-03-23T11:47:28.750 INFO:tasks.workunit.client.1.smithi111.stdout:5/282: rename d1/d3/d6/f31 to d1/d20/f61 0 2024-03-23T11:47:28.750 INFO:tasks.workunit.client.1.smithi111.stdout:2/266: symlink d2/d11/d17/d3e/d5d/l60 0 2024-03-23T11:47:28.750 INFO:tasks.workunit.client.1.smithi111.stdout:4/240: link d0/dc/d4c/d2a/l32 d0/dc/d1f/l4e 0 2024-03-23T11:47:28.752 INFO:tasks.workunit.client.1.smithi111.stdout:1/232: mknod d0/d6/d48/d4d/c51 0 2024-03-23T11:47:28.756 INFO:tasks.workunit.client.0.smithi063.stdout:9/207: dread d4/d11/f17 [0,4194304] 0 2024-03-23T11:47:28.756 INFO:tasks.workunit.client.1.smithi111.stdout:9/304: dread - d0/d4/f43 zero size 2024-03-23T11:47:28.756 INFO:tasks.workunit.client.1.smithi111.stdout:4/241: mkdir d0/dc/d4c/d4b/d4f 0 2024-03-23T11:47:28.757 INFO:tasks.workunit.client.1.smithi111.stdout:2/267: chown d2/d7/da/f3b 635426415 1 2024-03-23T11:47:28.759 INFO:tasks.workunit.client.1.smithi111.stdout:8/200: dread d5/f19 [0,4194304] 0 2024-03-23T11:47:28.761 INFO:tasks.workunit.client.1.smithi111.stdout:1/233: creat d0/d6/d48/f52 x:0 0 0 2024-03-23T11:47:28.762 INFO:tasks.workunit.client.0.smithi063.stdout:3/334: getdents d13 0 2024-03-23T11:47:28.763 INFO:tasks.workunit.client.1.smithi111.stdout:9/305: read d0/f5f [53058,35225] 0 2024-03-23T11:47:28.763 INFO:tasks.workunit.client.1.smithi111.stdout:2/268: readlink d2/d11/d17/l5e 0 2024-03-23T11:47:28.764 INFO:tasks.workunit.client.0.smithi063.stdout:0/227: dread d2/d14/f31 [0,4194304] 0 2024-03-23T11:47:28.766 INFO:tasks.workunit.client.0.smithi063.stdout:4/197: creat d5/d1e/d2b/f35 x:0 0 0 2024-03-23T11:47:28.767 INFO:tasks.workunit.client.0.smithi063.stdout:3/335: readlink d13/d22/d2d/d3e/l52 0 2024-03-23T11:47:28.769 INFO:tasks.workunit.client.1.smithi111.stdout:1/234: fsync d0/d5/f23 0 2024-03-23T11:47:28.771 INFO:tasks.workunit.client.1.smithi111.stdout:0/237: dwrite d5/db/d1f/f37 [0,4194304] 0 2024-03-23T11:47:28.772 INFO:tasks.workunit.client.0.smithi063.stdout:4/198: chown d5 0 1 2024-03-23T11:47:28.775 INFO:tasks.workunit.client.0.smithi063.stdout:0/228: chown d2/d14/f31 0 1 2024-03-23T11:47:28.775 INFO:tasks.workunit.client.1.smithi111.stdout:6/306: dwrite d2/d17/f60 [0,4194304] 0 2024-03-23T11:47:28.777 INFO:tasks.workunit.client.1.smithi111.stdout:0/238: creat d5/db/d1f/d19/f4c x:0 0 0 2024-03-23T11:47:28.779 INFO:tasks.workunit.client.1.smithi111.stdout:1/235: mknod d0/d2b/d37/c53 0 2024-03-23T11:47:28.780 INFO:tasks.workunit.client.1.smithi111.stdout:5/283: rename d1/d3/d6/d7/f36 to d1/d3/d6/d29/f62 0 2024-03-23T11:47:28.781 INFO:tasks.workunit.client.0.smithi063.stdout:9/208: dread d4/f21 [0,4194304] 0 2024-03-23T11:47:28.783 INFO:tasks.workunit.client.0.smithi063.stdout:9/209: stat d4/f2a 0 2024-03-23T11:47:28.787 INFO:tasks.workunit.client.1.smithi111.stdout:0/239: mkdir d5/db/d4d 0 2024-03-23T11:47:28.789 INFO:tasks.workunit.client.0.smithi063.stdout:3/336: creat d13/d22/d2d/d3f/d5b/f65 x:0 0 0 2024-03-23T11:47:28.789 INFO:tasks.workunit.client.0.smithi063.stdout:7/240: dwrite d5/d6/d19/d1f/d37/da/f17 [4194304,4194304] 0 2024-03-23T11:47:28.789 INFO:tasks.workunit.client.1.smithi111.stdout:5/284: chown d1/d20/c4c 393007864 1 2024-03-23T11:47:28.790 INFO:tasks.workunit.client.1.smithi111.stdout:4/242: rename d0/dc/d1f/c26 to d0/dc/d4c/d4b/d4f/c50 0 2024-03-23T11:47:28.794 INFO:tasks.workunit.client.1.smithi111.stdout:1/236: mkdir d0/d5/dd/d1e/d54 0 2024-03-23T11:47:28.805 INFO:tasks.workunit.client.1.smithi111.stdout:0/240: fdatasync d5/db/f4a 0 2024-03-23T11:47:28.805 INFO:tasks.workunit.client.1.smithi111.stdout:3/200: sync 2024-03-23T11:47:28.808 INFO:tasks.workunit.client.1.smithi111.stdout:2/269: dwrite d2/d7/f38 [0,4194304] 0 2024-03-23T11:47:28.808 INFO:tasks.workunit.client.1.smithi111.stdout:5/285: rename d1/d3/d23 to d1/d3/d6/d7/d63 0 2024-03-23T11:47:28.812 INFO:tasks.workunit.client.1.smithi111.stdout:8/201: dwrite d5/d1b/f2b [0,4194304] 0 2024-03-23T11:47:28.812 INFO:tasks.workunit.client.0.smithi063.stdout:7/241: dwrite d5/d6/d19/d1f/d37/d20/f31 [0,4194304] 0 2024-03-23T11:47:28.813 INFO:tasks.workunit.client.1.smithi111.stdout:1/237: creat d0/d6/d48/f55 x:0 0 0 2024-03-23T11:47:28.816 INFO:tasks.workunit.client.0.smithi063.stdout:7/242: write d5/d6/d19/d1f/d37/da/f4e [229642,106104] 0 2024-03-23T11:47:28.819 INFO:tasks.workunit.client.1.smithi111.stdout:9/306: dwrite d0/d4/d1c/f46 [0,4194304] 0 2024-03-23T11:47:28.819 INFO:tasks.workunit.client.1.smithi111.stdout:2/270: fsync d2/f5b 0 2024-03-23T11:47:28.821 INFO:tasks.workunit.client.1.smithi111.stdout:4/243: creat d0/dc/d4c/d4b/d4d/f51 x:0 0 0 2024-03-23T11:47:28.825 INFO:tasks.workunit.client.1.smithi111.stdout:6/307: dwrite d2/d35/f44 [0,4194304] 0 2024-03-23T11:47:28.826 INFO:tasks.workunit.client.1.smithi111.stdout:8/202: rmdir d5/d1b 39 2024-03-23T11:47:28.835 INFO:tasks.workunit.client.0.smithi063.stdout:4/199: mknod d5/da/d20/d24/d16/d32/c36 0 2024-03-23T11:47:28.842 INFO:tasks.workunit.client.1.smithi111.stdout:2/271: symlink d2/d5c/l61 0 2024-03-23T11:47:28.843 INFO:tasks.workunit.client.1.smithi111.stdout:4/244: dread d0/d17/f3d [0,4194304] 0 2024-03-23T11:47:28.846 INFO:tasks.workunit.client.0.smithi063.stdout:9/210: creat d4/f47 x:0 0 0 2024-03-23T11:47:28.851 INFO:tasks.workunit.client.0.smithi063.stdout:0/229: getdents d2/d4/dd/d25 0 2024-03-23T11:47:28.852 INFO:tasks.workunit.client.1.smithi111.stdout:9/307: symlink d0/d4/d10/d29/d5b/l6c 0 2024-03-23T11:47:28.852 INFO:tasks.workunit.client.1.smithi111.stdout:8/203: creat d5/d32/d35/f3e x:0 0 0 2024-03-23T11:47:28.854 INFO:tasks.workunit.client.0.smithi063.stdout:7/243: getdents d5/d6/d19/d1f 0 2024-03-23T11:47:28.857 INFO:tasks.workunit.client.1.smithi111.stdout:3/201: dread d0/d7/f23 [0,4194304] 0 2024-03-23T11:47:28.861 INFO:tasks.workunit.client.1.smithi111.stdout:4/245: mkdir d0/dc/d1f/d52 0 2024-03-23T11:47:28.864 INFO:tasks.workunit.client.0.smithi063.stdout:4/200: dwrite d5/da/d20/d24/d16/f1d [0,4194304] 0 2024-03-23T11:47:28.864 INFO:tasks.workunit.client.0.smithi063.stdout:4/201: stat d5/l6 0 2024-03-23T11:47:28.869 INFO:tasks.workunit.client.1.smithi111.stdout:5/286: dwrite d1/d3/d6/d29/f4d [0,4194304] 0 2024-03-23T11:47:28.869 INFO:tasks.workunit.client.1.smithi111.stdout:3/202: chown d0/d1a/d1d/d26/l32 1278 1 2024-03-23T11:47:28.870 INFO:tasks.workunit.client.1.smithi111.stdout:6/308: dread d2/d1f/d45/f56 [0,4194304] 0 2024-03-23T11:47:28.871 INFO:tasks.workunit.client.0.smithi063.stdout:4/202: dread d5/da/d20/d24/f2d [0,4194304] 0 2024-03-23T11:47:28.872 INFO:tasks.workunit.client.1.smithi111.stdout:9/308: symlink d0/d31/d35/l6d 0 2024-03-23T11:47:28.876 INFO:tasks.workunit.client.1.smithi111.stdout:1/238: dwrite d0/d5/dd/d1e/f31 [0,4194304] 0 2024-03-23T11:47:28.876 INFO:tasks.workunit.client.1.smithi111.stdout:9/309: readlink d0/d4/l56 0 2024-03-23T11:47:28.876 INFO:tasks.workunit.client.0.smithi063.stdout:6/169: sync 2024-03-23T11:47:28.876 INFO:tasks.workunit.client.1.smithi111.stdout:5/287: fsync d1/d5d/f26 0 2024-03-23T11:47:28.878 INFO:tasks.workunit.client.0.smithi063.stdout:4/203: mkdir d5/da/d20/d24/d16/d37 0 2024-03-23T11:47:28.881 INFO:tasks.workunit.client.1.smithi111.stdout:4/246: symlink d0/dc/d4c/d4b/d4f/l53 0 2024-03-23T11:47:28.884 INFO:tasks.workunit.client.1.smithi111.stdout:6/309: chown d2/d1f/d45/d49 1827226 1 2024-03-23T11:47:28.885 INFO:tasks.workunit.client.0.smithi063.stdout:6/170: write d1/f12 [2585292,44858] 0 2024-03-23T11:47:28.892 INFO:tasks.workunit.client.1.smithi111.stdout:9/310: mkdir d0/d4/d1c/d2d/d6e 0 2024-03-23T11:47:28.893 INFO:tasks.workunit.client.1.smithi111.stdout:2/272: dwrite d2/f4 [0,4194304] 0 2024-03-23T11:47:28.896 INFO:tasks.workunit.client.0.smithi063.stdout:4/204: mkdir d5/da/d20/d24/d29/d38 0 2024-03-23T11:47:28.904 INFO:tasks.workunit.client.0.smithi063.stdout:6/171: dwrite d1/dc/d21/f29 [0,4194304] 0 2024-03-23T11:47:28.905 INFO:tasks.workunit.client.1.smithi111.stdout:8/204: dwrite d5/f1a [0,4194304] 0 2024-03-23T11:47:28.906 INFO:tasks.workunit.client.1.smithi111.stdout:7/204: sync 2024-03-23T11:47:28.909 INFO:tasks.workunit.client.0.smithi063.stdout:4/205: truncate d5/fc 6029310 0 2024-03-23T11:47:28.913 INFO:tasks.workunit.client.0.smithi063.stdout:6/172: unlink d1/d13/f1e 0 2024-03-23T11:47:28.916 INFO:tasks.workunit.client.1.smithi111.stdout:6/310: fsync d2/d1a/d1c/d2c/d51/f61 0 2024-03-23T11:47:28.917 INFO:tasks.workunit.client.1.smithi111.stdout:5/288: link d1/d5d/f26 d1/d3/d6/d7/f64 0 2024-03-23T11:47:28.917 INFO:tasks.workunit.client.0.smithi063.stdout:4/206: creat d5/da/d20/d24/d1f/f39 x:0 0 0 2024-03-23T11:47:28.921 INFO:tasks.workunit.client.0.smithi063.stdout:4/207: readlink d5/da/d20/d24/d29/l30 0 2024-03-23T11:47:28.926 INFO:tasks.workunit.client.0.smithi063.stdout:6/173: creat d1/d15/d28/f30 x:0 0 0 2024-03-23T11:47:28.926 INFO:tasks.workunit.client.1.smithi111.stdout:3/203: dwrite d0/f9 [0,4194304] 0 2024-03-23T11:47:28.928 INFO:tasks.workunit.client.1.smithi111.stdout:5/289: dread - d1/d3/d6/d7/f3e zero size 2024-03-23T11:47:28.928 INFO:tasks.workunit.client.0.smithi063.stdout:4/208: rename d5/d1e/l27 to d5/d1e/d2b/l3a 0 2024-03-23T11:47:28.929 INFO:tasks.workunit.client.1.smithi111.stdout:7/205: link f1 d14/d1b/d1e/d2c/d2e/f34 0 2024-03-23T11:47:28.929 INFO:tasks.workunit.client.0.smithi063.stdout:6/174: rename d1/d13 to d1/d13/d31 22 2024-03-23T11:47:28.929 INFO:tasks.workunit.client.0.smithi063.stdout:4/209: stat f2 0 2024-03-23T11:47:28.930 INFO:tasks.workunit.client.0.smithi063.stdout:3/337: mkdir d13/d66 0 2024-03-23T11:47:28.933 INFO:tasks.workunit.client.1.smithi111.stdout:3/204: stat d0/d7/f23 0 2024-03-23T11:47:28.933 INFO:tasks.workunit.client.0.smithi063.stdout:6/175: write d1/dc/f1d [4130755,51304] 0 2024-03-23T11:47:28.938 INFO:tasks.workunit.client.0.smithi063.stdout:4/210: mknod d5/d1e/d2b/c3b 0 2024-03-23T11:47:28.939 INFO:tasks.workunit.client.1.smithi111.stdout:1/239: dwrite d0/d6/f34 [0,4194304] 0 2024-03-23T11:47:28.939 INFO:tasks.workunit.client.1.smithi111.stdout:7/206: rmdir d14/d1b/d1e 39 2024-03-23T11:47:28.940 INFO:tasks.workunit.client.0.smithi063.stdout:4/211: chown d5/da/d20 5780 1 2024-03-23T11:47:28.943 INFO:tasks.workunit.client.0.smithi063.stdout:6/176: rename d1/d13/f27 to d1/dc/f32 0 2024-03-23T11:47:28.944 INFO:tasks.workunit.client.0.smithi063.stdout:0/230: dwrite d2/d12/f44 [0,4194304] 0 2024-03-23T11:47:28.945 INFO:tasks.workunit.client.1.smithi111.stdout:2/273: dwrite d2/f5b [0,4194304] 0 2024-03-23T11:47:28.945 INFO:tasks.workunit.client.1.smithi111.stdout:9/311: dwrite d0/d4/d1c/d2d/d3b/d42/d4f/f5c [0,4194304] 0 2024-03-23T11:47:28.948 INFO:tasks.workunit.client.0.smithi063.stdout:6/177: write d1/d15/f26 [549689,57216] 0 2024-03-23T11:47:28.950 INFO:tasks.workunit.client.1.smithi111.stdout:6/311: creat d2/d1f/d45/d49/f6c x:0 0 0 2024-03-23T11:47:28.950 INFO:tasks.workunit.client.0.smithi063.stdout:6/178: chown d1/dc/f32 7 1 2024-03-23T11:47:28.950 INFO:tasks.workunit.client.0.smithi063.stdout:7/244: symlink d5/d6/d19/d1f/d37/l50 0 2024-03-23T11:47:28.951 INFO:tasks.workunit.client.1.smithi111.stdout:4/247: write d0/dc/d4c/d2a/f38 [4124545,112482] 0 2024-03-23T11:47:28.955 INFO:tasks.workunit.client.1.smithi111.stdout:9/312: write d0/d4/d1c/d36/f6b [860350,7938] 0 2024-03-23T11:47:28.956 INFO:tasks.workunit.client.1.smithi111.stdout:8/205: dwrite d5/f11 [0,4194304] 0 2024-03-23T11:47:28.964 INFO:tasks.workunit.client.1.smithi111.stdout:2/274: symlink d2/d32/d4b/d52/l62 0 2024-03-23T11:47:28.966 INFO:tasks.workunit.client.1.smithi111.stdout:2/275: chown d2/d32/d4b 363 1 2024-03-23T11:47:28.972 INFO:tasks.workunit.client.1.smithi111.stdout:0/241: sync 2024-03-23T11:47:28.972 INFO:tasks.workunit.client.0.smithi063.stdout:4/212: rename d5/da/d20/d24/d16/f2c to d5/da/d20/f3c 0 2024-03-23T11:47:28.979 INFO:tasks.workunit.client.1.smithi111.stdout:6/312: write d2/f41 [981098,87279] 0 2024-03-23T11:47:28.983 INFO:tasks.workunit.client.1.smithi111.stdout:5/290: dread d1/d5d/f26 [0,4194304] 0 2024-03-23T11:47:28.983 INFO:tasks.workunit.client.0.smithi063.stdout:0/231: dwrite d2/d4/dd/d1f/f2d [0,4194304] 0 2024-03-23T11:47:28.985 INFO:tasks.workunit.client.0.smithi063.stdout:6/179: dwrite d1/dc/f1d [4194304,4194304] 0 2024-03-23T11:47:28.987 INFO:tasks.workunit.client.0.smithi063.stdout:3/338: readlink d13/l2e 0 2024-03-23T11:47:28.989 INFO:tasks.workunit.client.0.smithi063.stdout:7/245: dwrite d5/d6/d19/d1f/f42 [0,4194304] 0 2024-03-23T11:47:28.994 INFO:tasks.workunit.client.0.smithi063.stdout:4/213: unlink d5/da/d20/d24/d16/f1d 0 2024-03-23T11:47:28.998 INFO:tasks.workunit.client.0.smithi063.stdout:4/214: readlink d5/l7 0 2024-03-23T11:47:28.998 INFO:tasks.workunit.client.0.smithi063.stdout:3/339: dread - d13/f56 zero size 2024-03-23T11:47:29.001 INFO:tasks.workunit.client.1.smithi111.stdout:4/248: write d0/d7/f16 [2143682,122156] 0 2024-03-23T11:47:29.004 INFO:tasks.workunit.client.1.smithi111.stdout:9/313: write d0/d4/d1c/d36/d28/f68 [9996,64910] 0 2024-03-23T11:47:29.005 INFO:tasks.workunit.client.0.smithi063.stdout:6/180: rename d1/dc/d21/c2c to d1/d13/c33 0 2024-03-23T11:47:29.007 INFO:tasks.workunit.client.1.smithi111.stdout:9/314: fsync d0/d4/d1c/d2d/d3b/d42/d4f/f61 0 2024-03-23T11:47:29.009 INFO:tasks.workunit.client.0.smithi063.stdout:3/340: mkdir d13/d22/d2d/d3f/d67 0 2024-03-23T11:47:29.009 INFO:tasks.workunit.client.0.smithi063.stdout:6/181: chown d1/dc/c11 9 1 2024-03-23T11:47:29.012 INFO:tasks.workunit.client.1.smithi111.stdout:1/240: dread d0/d6/f22 [0,4194304] 0 2024-03-23T11:47:29.014 INFO:tasks.workunit.client.1.smithi111.stdout:2/276: creat d2/d5c/f63 x:0 0 0 2024-03-23T11:47:29.014 INFO:tasks.workunit.client.1.smithi111.stdout:1/241: chown d0/d2b/d37/l44 506663952 1 2024-03-23T11:47:29.016 INFO:tasks.workunit.client.0.smithi063.stdout:7/246: rename d5/d6/d19/d1f/d37/d20/c4c to d5/d6/d39/c51 0 2024-03-23T11:47:29.017 INFO:tasks.workunit.client.0.smithi063.stdout:6/182: chown d1/dc/d21/f29 245029 1 2024-03-23T11:47:29.018 INFO:tasks.workunit.client.0.smithi063.stdout:0/232: dwrite d2/d4/f6 [0,4194304] 0 2024-03-23T11:47:29.018 INFO:tasks.workunit.client.1.smithi111.stdout:0/242: mknod d5/db/d1f/d19/c4e 0 2024-03-23T11:47:29.018 INFO:tasks.workunit.client.1.smithi111.stdout:6/313: creat d2/d1f/d45/d49/f6d x:0 0 0 2024-03-23T11:47:29.019 INFO:tasks.workunit.client.0.smithi063.stdout:4/215: dread d5/da/d20/f3c [0,4194304] 0 2024-03-23T11:47:29.021 INFO:tasks.workunit.client.1.smithi111.stdout:0/243: write d5/db/d1f/d32/f44 [1123329,9176] 0 2024-03-23T11:47:29.022 INFO:tasks.workunit.client.0.smithi063.stdout:6/183: rmdir d1/d15 39 2024-03-23T11:47:29.022 INFO:tasks.workunit.client.0.smithi063.stdout:4/216: write d5/ff [8560554,105972] 0 2024-03-23T11:47:29.024 INFO:tasks.workunit.client.1.smithi111.stdout:3/205: creat d0/f39 x:0 0 0 2024-03-23T11:47:29.027 INFO:tasks.workunit.client.1.smithi111.stdout:3/206: read d0/d7/d19/f29 [1687908,70361] 0 2024-03-23T11:47:29.028 INFO:tasks.workunit.client.1.smithi111.stdout:4/249: creat d0/dc/d4c/d4b/d4d/f54 x:0 0 0 2024-03-23T11:47:29.030 INFO:tasks.workunit.client.1.smithi111.stdout:9/315: rename d0/d4/d10/f17 to d0/d31/d41/d65/f6f 0 2024-03-23T11:47:29.031 INFO:tasks.workunit.client.0.smithi063.stdout:7/247: unlink d5/d6/d19/d1f/d37/da/cb 0 2024-03-23T11:47:29.034 INFO:tasks.workunit.client.1.smithi111.stdout:7/207: dwrite f9 [0,4194304] 0 2024-03-23T11:47:29.034 INFO:tasks.workunit.client.1.smithi111.stdout:2/277: mknod d2/d5c/c64 0 2024-03-23T11:47:29.036 INFO:tasks.workunit.client.0.smithi063.stdout:0/233: symlink d2/d4/dd/d1f/d4f/l52 0 2024-03-23T11:47:29.037 INFO:tasks.workunit.client.0.smithi063.stdout:7/248: mknod d5/d6/d19/d1f/d37/da/c52 0 2024-03-23T11:47:29.039 INFO:tasks.workunit.client.1.smithi111.stdout:5/291: getdents d1/d5d/d59 0 2024-03-23T11:47:29.040 INFO:tasks.workunit.client.0.smithi063.stdout:6/184: creat d1/d15/d28/f34 x:0 0 0 2024-03-23T11:47:29.040 INFO:tasks.workunit.client.0.smithi063.stdout:0/234: write d2/d4/dd/d1f/f2f [3887325,31005] 0 2024-03-23T11:47:29.041 INFO:tasks.workunit.client.1.smithi111.stdout:0/244: symlink d5/db/d1f/d39/l4f 0 2024-03-23T11:47:29.041 INFO:tasks.workunit.client.0.smithi063.stdout:4/217: symlink d5/da/d20/d24/d29/d38/l3d 0 2024-03-23T11:47:29.043 INFO:tasks.workunit.client.0.smithi063.stdout:4/218: fsync d5/da/f2a 0 2024-03-23T11:47:29.045 INFO:tasks.workunit.client.1.smithi111.stdout:0/245: write d5/db/d1f/d39/f3f [209951,128647] 0 2024-03-23T11:47:29.047 INFO:tasks.workunit.client.0.smithi063.stdout:0/235: symlink d2/d14/d36/l53 0 2024-03-23T11:47:29.048 INFO:tasks.workunit.client.1.smithi111.stdout:3/207: dwrite d0/d7/f10 [0,4194304] 0 2024-03-23T11:47:29.053 INFO:tasks.workunit.client.0.smithi063.stdout:7/249: dread d5/d6/d19/d1f/d37/f3c [0,4194304] 0 2024-03-23T11:47:29.056 INFO:tasks.workunit.client.0.smithi063.stdout:6/185: getdents d1/d2a 0 2024-03-23T11:47:29.058 INFO:tasks.workunit.client.1.smithi111.stdout:6/314: mknod d2/d1f/d45/d65/c6e 0 2024-03-23T11:47:29.058 INFO:tasks.workunit.client.0.smithi063.stdout:6/186: chown d1/d13/c22 1156 1 2024-03-23T11:47:29.060 INFO:tasks.workunit.client.0.smithi063.stdout:3/341: truncate d13/f30 3288577 0 2024-03-23T11:47:29.062 INFO:tasks.workunit.client.1.smithi111.stdout:0/246: dwrite d5/db/d1f/f37 [0,4194304] 0 2024-03-23T11:47:29.064 INFO:tasks.workunit.client.1.smithi111.stdout:0/247: chown d5/f40 8911 1 2024-03-23T11:47:29.066 INFO:tasks.workunit.client.1.smithi111.stdout:3/208: symlink d0/d7/d19/l3a 0 2024-03-23T11:47:29.067 INFO:tasks.workunit.client.0.smithi063.stdout:7/250: getdents d5/d6/d19/d1f/d47 0 2024-03-23T11:47:29.068 INFO:tasks.workunit.client.0.smithi063.stdout:3/342: truncate d13/f47 984803 0 2024-03-23T11:47:29.070 INFO:tasks.workunit.client.1.smithi111.stdout:1/242: dread d0/fe [0,4194304] 0 2024-03-23T11:47:29.071 INFO:tasks.workunit.client.1.smithi111.stdout:0/248: write d5/fc [750472,17080] 0 2024-03-23T11:47:29.072 INFO:tasks.workunit.client.0.smithi063.stdout:6/187: creat d1/f35 x:0 0 0 2024-03-23T11:47:29.074 INFO:tasks.workunit.client.1.smithi111.stdout:8/206: dwrite d5/d1b/f3a [0,4194304] 0 2024-03-23T11:47:29.076 INFO:tasks.workunit.client.1.smithi111.stdout:6/315: rename d2/d1f/f2e to d2/d35/f6f 0 2024-03-23T11:47:29.078 INFO:tasks.workunit.client.1.smithi111.stdout:4/250: write d0/d17/f3d [113511,9001] 0 2024-03-23T11:47:29.080 INFO:tasks.workunit.client.0.smithi063.stdout:7/251: unlink d5/d6/fe 0 2024-03-23T11:47:29.084 INFO:tasks.workunit.client.1.smithi111.stdout:5/292: creat d1/d3/f65 x:0 0 0 2024-03-23T11:47:29.084 INFO:tasks.workunit.client.0.smithi063.stdout:0/236: dwrite d2/d14/f15 [0,4194304] 0 2024-03-23T11:47:29.085 INFO:tasks.workunit.client.0.smithi063.stdout:6/188: unlink d1/d13/l25 0 2024-03-23T11:47:29.089 INFO:tasks.workunit.client.0.smithi063.stdout:0/237: fsync d2/d12/f21 0 2024-03-23T11:47:29.089 INFO:tasks.workunit.client.1.smithi111.stdout:3/209: rmdir d0/d7/d19 39 2024-03-23T11:47:29.090 INFO:tasks.workunit.client.1.smithi111.stdout:0/249: read d5/fc [437624,89029] 0 2024-03-23T11:47:29.092 INFO:tasks.workunit.client.1.smithi111.stdout:1/243: symlink d0/d6/l56 0 2024-03-23T11:47:29.093 INFO:tasks.workunit.client.1.smithi111.stdout:3/210: truncate d0/d1a/d1d/f24 1467331 0 2024-03-23T11:47:29.094 INFO:tasks.workunit.client.0.smithi063.stdout:2/190: sync 2024-03-23T11:47:29.095 INFO:tasks.workunit.client.0.smithi063.stdout:7/252: dread d5/d6/d19/d1f/d37/d20/f35 [0,4194304] 0 2024-03-23T11:47:29.098 INFO:tasks.workunit.client.0.smithi063.stdout:4/219: truncate d5/da/f2a 66446 0 2024-03-23T11:47:29.099 INFO:tasks.workunit.client.1.smithi111.stdout:9/316: getdents d0/d4/d1c/d2d/d3b/d42/d4f 0 2024-03-23T11:47:29.100 INFO:tasks.workunit.client.0.smithi063.stdout:6/189: symlink d1/l36 0 2024-03-23T11:47:29.101 INFO:tasks.workunit.client.1.smithi111.stdout:4/251: creat d0/dc/f55 x:0 0 0 2024-03-23T11:47:29.102 INFO:tasks.workunit.client.1.smithi111.stdout:2/278: getdents d2/d7/da 0 2024-03-23T11:47:29.105 INFO:tasks.workunit.client.0.smithi063.stdout:2/191: mknod d2/d27/c3f 0 2024-03-23T11:47:29.106 INFO:tasks.workunit.client.1.smithi111.stdout:5/293: fsync d1/d3/d6/f2b 0 2024-03-23T11:47:29.107 INFO:tasks.workunit.client.0.smithi063.stdout:4/220: rename d5/da/l2e to d5/da/d20/d24/d16/d37/l3e 0 2024-03-23T11:47:29.109 INFO:tasks.workunit.client.0.smithi063.stdout:0/238: dread d2/d14/f31 [0,4194304] 0 2024-03-23T11:47:29.109 INFO:tasks.workunit.client.1.smithi111.stdout:0/250: mknod d5/db/d1f/d32/c50 0 2024-03-23T11:47:29.109 INFO:tasks.workunit.client.1.smithi111.stdout:5/294: write d1/d3/d6/d29/f22 [715340,120954] 0 2024-03-23T11:47:29.111 INFO:tasks.workunit.client.0.smithi063.stdout:0/239: chown d2/d14/f23 513821979 1 2024-03-23T11:47:29.114 INFO:tasks.workunit.client.1.smithi111.stdout:9/317: rename d0/d4/cb to d0/d4/d10/c70 0 2024-03-23T11:47:29.116 INFO:tasks.workunit.client.0.smithi063.stdout:3/343: unlink d13/f30 0 2024-03-23T11:47:29.117 INFO:tasks.workunit.client.1.smithi111.stdout:9/318: readlink d0/d31/d35/l6d 0 2024-03-23T11:47:29.118 INFO:tasks.workunit.client.0.smithi063.stdout:4/221: creat d5/da/d20/d24/d1f/f3f x:0 0 0 2024-03-23T11:47:29.118 INFO:tasks.workunit.client.0.smithi063.stdout:2/192: link d2/c7 d2/d1c/c40 0 2024-03-23T11:47:29.120 INFO:tasks.workunit.client.1.smithi111.stdout:0/251: creat d5/db/f51 x:0 0 0 2024-03-23T11:47:29.120 INFO:tasks.workunit.client.0.smithi063.stdout:6/190: dread d1/dc/d21/f29 [0,4194304] 0 2024-03-23T11:47:29.122 INFO:tasks.workunit.client.1.smithi111.stdout:9/319: chown d0/d4/d1c/d36/f59 2818505 1 2024-03-23T11:47:29.122 INFO:tasks.workunit.client.0.smithi063.stdout:6/191: chown d1/d13/c22 0 1 2024-03-23T11:47:29.123 INFO:tasks.workunit.client.1.smithi111.stdout:4/252: symlink d0/d7/l56 0 2024-03-23T11:47:29.123 INFO:tasks.workunit.client.1.smithi111.stdout:6/316: dwrite d2/d1a/f64 [0,4194304] 0 2024-03-23T11:47:29.123 INFO:tasks.workunit.client.1.smithi111.stdout:5/295: mkdir d1/d55/d66 0 2024-03-23T11:47:29.125 INFO:tasks.workunit.client.0.smithi063.stdout:6/192: dread - d1/f35 zero size 2024-03-23T11:47:29.125 INFO:tasks.workunit.client.1.smithi111.stdout:5/296: chown d1/d3/d6/d29/f54 93 1 2024-03-23T11:47:29.127 INFO:tasks.workunit.client.1.smithi111.stdout:0/252: unlink d5/l42 0 2024-03-23T11:47:29.130 INFO:tasks.workunit.client.0.smithi063.stdout:2/193: write d2/d23/f28 [2573113,84040] 0 2024-03-23T11:47:29.131 INFO:tasks.workunit.client.1.smithi111.stdout:2/279: getdents d2/d11/d17 0 2024-03-23T11:47:29.132 INFO:tasks.workunit.client.1.smithi111.stdout:1/244: dwrite d0/d6/f22 [0,4194304] 0 2024-03-23T11:47:29.133 INFO:tasks.workunit.client.0.smithi063.stdout:6/193: creat d1/d15/d28/f37 x:0 0 0 2024-03-23T11:47:29.138 INFO:tasks.workunit.client.1.smithi111.stdout:9/320: mkdir d0/d31/d41/d71 0 2024-03-23T11:47:29.138 INFO:tasks.workunit.client.1.smithi111.stdout:3/211: dwrite d0/d7/f23 [0,4194304] 0 2024-03-23T11:47:29.140 INFO:tasks.workunit.client.1.smithi111.stdout:6/317: stat d2/fb 0 2024-03-23T11:47:29.141 INFO:tasks.workunit.client.1.smithi111.stdout:2/280: chown c1 7 1 2024-03-23T11:47:29.143 INFO:tasks.workunit.client.1.smithi111.stdout:6/318: write d2/d1f/f59 [177678,89024] 0 2024-03-23T11:47:29.146 INFO:tasks.workunit.client.1.smithi111.stdout:1/245: creat d0/d5/dd/d1e/d54/f57 x:0 0 0 2024-03-23T11:47:29.146 INFO:tasks.workunit.client.1.smithi111.stdout:5/297: creat d1/d5d/d59/f67 x:0 0 0 2024-03-23T11:47:29.146 INFO:tasks.workunit.client.0.smithi063.stdout:0/240: dwrite d2/d4/dd/d1f/f2f [0,4194304] 0 2024-03-23T11:47:29.147 INFO:tasks.workunit.client.1.smithi111.stdout:7/208: sync 2024-03-23T11:47:29.149 INFO:tasks.workunit.client.1.smithi111.stdout:9/321: creat d0/d4/d6a/f72 x:0 0 0 2024-03-23T11:47:29.151 INFO:tasks.workunit.client.1.smithi111.stdout:2/281: creat d2/d3d/f65 x:0 0 0 2024-03-23T11:47:29.153 INFO:tasks.workunit.client.1.smithi111.stdout:1/246: write d0/d5/f23 [1121214,110791] 0 2024-03-23T11:47:29.153 INFO:tasks.workunit.client.1.smithi111.stdout:6/319: mkdir d2/d35/d70 0 2024-03-23T11:47:29.158 INFO:tasks.workunit.client.1.smithi111.stdout:4/253: getdents d0 0 2024-03-23T11:47:29.159 INFO:tasks.workunit.client.1.smithi111.stdout:8/207: dread d5/f2d [0,4194304] 0 2024-03-23T11:47:29.159 INFO:tasks.workunit.client.1.smithi111.stdout:9/322: stat d0/d4/d1c/d2d/f63 0 2024-03-23T11:47:29.160 INFO:tasks.workunit.client.1.smithi111.stdout:3/212: symlink d0/l3b 0 2024-03-23T11:47:29.161 INFO:tasks.workunit.client.0.smithi063.stdout:3/344: dwrite d13/d22/d2d/f44 [0,4194304] 0 2024-03-23T11:47:29.162 INFO:tasks.workunit.client.0.smithi063.stdout:7/253: creat d5/d6/d19/d1f/d37/d20/f53 x:0 0 0 2024-03-23T11:47:29.162 INFO:tasks.workunit.client.1.smithi111.stdout:1/247: write d0/d2b/d37/f4f [152625,83339] 0 2024-03-23T11:47:29.164 INFO:tasks.workunit.client.0.smithi063.stdout:0/241: mknod d2/d14/c54 0 2024-03-23T11:47:29.167 INFO:tasks.workunit.client.1.smithi111.stdout:0/253: dread d5/db/f16 [0,4194304] 0 2024-03-23T11:47:29.171 INFO:tasks.workunit.client.1.smithi111.stdout:5/298: dread d1/d3/d6/d29/f4d [0,4194304] 0 2024-03-23T11:47:29.175 INFO:tasks.workunit.client.1.smithi111.stdout:6/320: fsync d2/d17/f55 0 2024-03-23T11:47:29.182 INFO:tasks.workunit.client.0.smithi063.stdout:3/345: creat d13/d15/d48/f68 x:0 0 0 2024-03-23T11:47:29.182 INFO:tasks.workunit.client.0.smithi063.stdout:6/194: dwrite d1/f7 [0,4194304] 0 2024-03-23T11:47:29.183 INFO:tasks.workunit.client.1.smithi111.stdout:1/248: dwrite d0/d5/f23 [0,4194304] 0 2024-03-23T11:47:29.183 INFO:tasks.workunit.client.1.smithi111.stdout:4/254: creat d0/dc/d1f/f57 x:0 0 0 2024-03-23T11:47:29.186 INFO:tasks.workunit.client.1.smithi111.stdout:1/249: readlink d0/d5/dd/d1e/l1f 0 2024-03-23T11:47:29.187 INFO:tasks.workunit.client.1.smithi111.stdout:2/282: dread d2/de/d27/d2e/d3f/f44 [0,4194304] 0 2024-03-23T11:47:29.188 INFO:tasks.workunit.client.0.smithi063.stdout:2/194: write d2/f1e [591871,89004] 0 2024-03-23T11:47:29.192 INFO:tasks.workunit.client.0.smithi063.stdout:3/346: mknod d13/d22/d2d/c69 0 2024-03-23T11:47:29.193 INFO:tasks.workunit.client.0.smithi063.stdout:7/254: link d5/d16/c46 d5/d6/d19/c54 0 2024-03-23T11:47:29.195 INFO:tasks.workunit.client.0.smithi063.stdout:6/195: chown d1/l1a 2 1 2024-03-23T11:47:29.199 INFO:tasks.workunit.client.1.smithi111.stdout:9/323: mknod d0/d31/d41/d71/c73 0 2024-03-23T11:47:29.199 INFO:tasks.workunit.client.0.smithi063.stdout:2/195: creat d2/d23/f41 x:0 0 0 2024-03-23T11:47:29.201 INFO:tasks.workunit.client.1.smithi111.stdout:4/255: mknod d0/dc/d1f/c58 0 2024-03-23T11:47:29.201 INFO:tasks.workunit.client.0.smithi063.stdout:3/347: dread - d13/d15/d48/f68 zero size 2024-03-23T11:47:29.202 INFO:tasks.workunit.client.0.smithi063.stdout:6/196: chown d1/dc/f2d 53399586 1 2024-03-23T11:47:29.204 INFO:tasks.workunit.client.0.smithi063.stdout:6/197: rename d1/d2a to d1/d2a/d38 22 2024-03-23T11:47:29.204 INFO:tasks.workunit.client.0.smithi063.stdout:3/348: fdatasync d13/d22/d2d/f4f 0 2024-03-23T11:47:29.204 INFO:tasks.workunit.client.1.smithi111.stdout:8/208: link d5/f1a d5/d32/d35/f3f 0 2024-03-23T11:47:29.205 INFO:tasks.workunit.client.0.smithi063.stdout:0/242: dwrite f1 [0,4194304] 0 2024-03-23T11:47:29.208 INFO:tasks.workunit.client.0.smithi063.stdout:0/243: chown d2/d12/l22 2 1 2024-03-23T11:47:29.208 INFO:tasks.workunit.client.0.smithi063.stdout:6/198: readlink d1/d13/l14 0 2024-03-23T11:47:29.210 INFO:tasks.workunit.client.0.smithi063.stdout:2/196: symlink d2/d23/d2e/l42 0 2024-03-23T11:47:29.211 INFO:tasks.workunit.client.0.smithi063.stdout:3/349: mknod d13/d15/c6a 0 2024-03-23T11:47:29.212 INFO:tasks.workunit.client.1.smithi111.stdout:2/283: creat d2/d32/d4b/f66 x:0 0 0 2024-03-23T11:47:29.214 INFO:tasks.workunit.client.1.smithi111.stdout:1/250: rename d0/d6/c4e to d0/d6/d48/d4d/c58 0 2024-03-23T11:47:29.214 INFO:tasks.workunit.client.0.smithi063.stdout:2/197: chown d2/d1c/f20 47893 1 2024-03-23T11:47:29.215 INFO:tasks.workunit.client.1.smithi111.stdout:7/209: symlink d14/d1b/d1e/d2c/d2e/d33/l35 0 2024-03-23T11:47:29.215 INFO:tasks.workunit.client.1.smithi111.stdout:0/254: dwrite d5/fc [0,4194304] 0 2024-03-23T11:47:29.215 INFO:tasks.workunit.client.0.smithi063.stdout:2/198: readlink d2/l31 0 2024-03-23T11:47:29.217 INFO:tasks.workunit.client.1.smithi111.stdout:1/251: fsync d0/d6/f42 0 2024-03-23T11:47:29.217 INFO:tasks.workunit.client.1.smithi111.stdout:4/256: dwrite d0/dc/f35 [0,4194304] 0 2024-03-23T11:47:29.218 INFO:tasks.workunit.client.0.smithi063.stdout:0/244: truncate d2/d4/f30 1015817 0 2024-03-23T11:47:29.218 INFO:tasks.workunit.client.1.smithi111.stdout:2/284: write d2/de/d27/d2e/d3f/f4d [754064,106539] 0 2024-03-23T11:47:29.219 INFO:tasks.workunit.client.1.smithi111.stdout:5/299: unlink d1/d3/d6/d29/f40 0 2024-03-23T11:47:29.219 INFO:tasks.workunit.client.1.smithi111.stdout:0/255: fdatasync d5/db/f38 0 2024-03-23T11:47:29.221 INFO:tasks.workunit.client.0.smithi063.stdout:2/199: chown d2/de/f2b 52695 1 2024-03-23T11:47:29.221 INFO:tasks.workunit.client.0.smithi063.stdout:6/199: mknod d1/d2a/c39 0 2024-03-23T11:47:29.221 INFO:tasks.workunit.client.1.smithi111.stdout:1/252: chown d0/d5/f23 1907074342 1 2024-03-23T11:47:29.222 INFO:tasks.workunit.client.1.smithi111.stdout:6/321: symlink d2/d35/d70/l71 0 2024-03-23T11:47:29.224 INFO:tasks.workunit.client.1.smithi111.stdout:0/256: readlink d5/db/d1f/l2f 0 2024-03-23T11:47:29.225 INFO:tasks.workunit.client.1.smithi111.stdout:9/324: rmdir d0/d31/d41 39 2024-03-23T11:47:29.225 INFO:tasks.workunit.client.1.smithi111.stdout:1/253: readlink d0/d6/l56 0 2024-03-23T11:47:29.226 INFO:tasks.workunit.client.0.smithi063.stdout:3/350: mknod d13/d22/d2d/d3f/c6b 0 2024-03-23T11:47:29.228 INFO:tasks.workunit.client.1.smithi111.stdout:7/210: stat d14/c21 0 2024-03-23T11:47:29.231 INFO:tasks.workunit.client.1.smithi111.stdout:6/322: fsync d2/f57 0 2024-03-23T11:47:29.231 INFO:tasks.workunit.client.0.smithi063.stdout:3/351: truncate d13/d22/d2d/d3f/d5b/f65 678333 0 2024-03-23T11:47:29.232 INFO:tasks.workunit.client.1.smithi111.stdout:5/300: symlink d1/d3/d6/d30/l68 0 2024-03-23T11:47:29.232 INFO:tasks.workunit.client.0.smithi063.stdout:6/200: rename d1/d15/d28 to d1/dc/d3a 0 2024-03-23T11:47:29.233 INFO:tasks.workunit.client.1.smithi111.stdout:3/213: dread d0/d7/d19/f29 [0,4194304] 0 2024-03-23T11:47:29.233 INFO:tasks.workunit.client.1.smithi111.stdout:9/325: chown d0/d4/d10/d29/l5a 457893464 1 2024-03-23T11:47:29.236 INFO:tasks.workunit.client.0.smithi063.stdout:0/245: symlink d2/d4/dd/d28/d46/l55 0 2024-03-23T11:47:29.236 INFO:tasks.workunit.client.1.smithi111.stdout:8/209: mkdir d5/d40 0 2024-03-23T11:47:29.237 INFO:tasks.workunit.client.0.smithi063.stdout:0/246: readlink d2/d4/dd/d28/d46/l55 0 2024-03-23T11:47:29.238 INFO:tasks.workunit.client.1.smithi111.stdout:9/326: chown d0/d4/c24 2 1 2024-03-23T11:47:29.238 INFO:tasks.workunit.client.1.smithi111.stdout:3/214: dread - d0/f1f zero size 2024-03-23T11:47:29.242 INFO:tasks.workunit.client.1.smithi111.stdout:3/215: stat d0/d1a/d2b/d38 0 2024-03-23T11:47:29.244 INFO:tasks.workunit.client.1.smithi111.stdout:6/323: chown d2/f27 2837 1 2024-03-23T11:47:29.244 INFO:tasks.workunit.client.1.smithi111.stdout:4/257: dread d0/d7/f1b [4194304,4194304] 0 2024-03-23T11:47:29.245 INFO:tasks.workunit.client.1.smithi111.stdout:0/257: unlink d5/c7 0 2024-03-23T11:47:29.246 INFO:tasks.workunit.client.1.smithi111.stdout:6/324: readlink d2/d35/l4f 0 2024-03-23T11:47:29.247 INFO:tasks.workunit.client.1.smithi111.stdout:3/216: read d0/f16 [2808818,82038] 0 2024-03-23T11:47:29.248 INFO:tasks.workunit.client.1.smithi111.stdout:4/258: fsync d0/dc/d4c/d2a/f38 0 2024-03-23T11:47:29.250 INFO:tasks.workunit.client.1.smithi111.stdout:0/258: read - d5/db/f51 zero size 2024-03-23T11:47:29.252 INFO:tasks.workunit.client.0.smithi063.stdout:2/200: dwrite d2/de/ff [0,4194304] 0 2024-03-23T11:47:29.254 INFO:tasks.workunit.client.1.smithi111.stdout:5/301: creat d1/d3/d6/d7/f69 x:0 0 0 2024-03-23T11:47:29.254 INFO:tasks.workunit.client.1.smithi111.stdout:7/211: dread fe [0,4194304] 0 2024-03-23T11:47:29.254 INFO:tasks.workunit.client.0.smithi063.stdout:0/247: link f1 d2/f56 0 2024-03-23T11:47:29.256 INFO:tasks.workunit.client.1.smithi111.stdout:8/210: mkdir d5/d26/d31/d3b/d41 0 2024-03-23T11:47:29.256 INFO:tasks.workunit.client.0.smithi063.stdout:7/255: truncate d5/d6/d19/d1f/d37/d20/f31 1920955 0 2024-03-23T11:47:29.258 INFO:tasks.workunit.client.1.smithi111.stdout:1/254: dwrite d0/d2b/f49 [0,4194304] 0 2024-03-23T11:47:29.259 INFO:tasks.workunit.client.0.smithi063.stdout:6/201: rename d1/dc/f2d to d1/f3b 0 2024-03-23T11:47:29.260 INFO:tasks.workunit.client.0.smithi063.stdout:2/201: rmdir d2/d27 39 2024-03-23T11:47:29.264 INFO:tasks.workunit.client.0.smithi063.stdout:0/248: creat d2/d12/f57 x:0 0 0 2024-03-23T11:47:29.268 INFO:tasks.workunit.client.0.smithi063.stdout:7/256: mknod d5/d6/d19/d1f/d37/d20/c55 0 2024-03-23T11:47:29.269 INFO:tasks.workunit.client.0.smithi063.stdout:6/202: readlink d1/dc/d3a/l2e 0 2024-03-23T11:47:29.270 INFO:tasks.workunit.client.0.smithi063.stdout:3/352: dread d13/d15/f1b [4194304,4194304] 0 2024-03-23T11:47:29.271 INFO:tasks.workunit.client.0.smithi063.stdout:2/202: symlink d2/d23/l43 0 2024-03-23T11:47:29.273 INFO:tasks.workunit.client.0.smithi063.stdout:2/203: chown d2/f25 0 1 2024-03-23T11:47:29.273 INFO:tasks.workunit.client.1.smithi111.stdout:0/259: dwrite d5/db/f38 [0,4194304] 0 2024-03-23T11:47:29.274 INFO:tasks.workunit.client.0.smithi063.stdout:0/249: mknod d2/d47/c58 0 2024-03-23T11:47:29.274 INFO:tasks.workunit.client.0.smithi063.stdout:2/204: dread - d2/f37 zero size 2024-03-23T11:47:29.275 INFO:tasks.workunit.client.1.smithi111.stdout:8/211: dwrite d5/fe [0,4194304] 0 2024-03-23T11:47:29.277 INFO:tasks.workunit.client.0.smithi063.stdout:0/250: readlink d2/d14/d36/l53 0 2024-03-23T11:47:29.277 INFO:tasks.workunit.client.1.smithi111.stdout:6/325: fdatasync d2/d1a/d1c/f6b 0 2024-03-23T11:47:29.278 INFO:tasks.workunit.client.0.smithi063.stdout:3/353: symlink d13/d22/d2d/d3f/d58/l6c 0 2024-03-23T11:47:29.278 INFO:tasks.workunit.client.1.smithi111.stdout:2/285: getdents d2/de/d27/d2e/d3f 0 2024-03-23T11:47:29.280 INFO:tasks.workunit.client.1.smithi111.stdout:5/302: creat d1/d5d/d59/f6a x:0 0 0 2024-03-23T11:47:29.280 INFO:tasks.workunit.client.0.smithi063.stdout:0/251: getdents d2/d12/d35 0 2024-03-23T11:47:29.291 INFO:tasks.workunit.client.1.smithi111.stdout:1/255: creat d0/f59 x:0 0 0 2024-03-23T11:47:29.294 INFO:tasks.workunit.client.1.smithi111.stdout:8/212: mkdir d5/d26/d2f/d42 0 2024-03-23T11:47:29.298 INFO:tasks.workunit.client.1.smithi111.stdout:9/327: dwrite d0/d4/d10/f51 [0,4194304] 0 2024-03-23T11:47:29.299 INFO:tasks.workunit.client.0.smithi063.stdout:2/205: dwrite d2/f18 [4194304,4194304] 0 2024-03-23T11:47:29.305 INFO:tasks.workunit.client.1.smithi111.stdout:3/217: link d0/d7/f10 d0/d1a/d2b/d38/f3c 0 2024-03-23T11:47:29.307 INFO:tasks.workunit.client.1.smithi111.stdout:7/212: write f1 [3731644,104073] 0 2024-03-23T11:47:29.308 INFO:tasks.workunit.client.0.smithi063.stdout:0/252: truncate d2/d12/f42 64963 0 2024-03-23T11:47:29.309 INFO:tasks.workunit.client.1.smithi111.stdout:3/218: write d0/d1a/f28 [2115472,120764] 0 2024-03-23T11:47:29.311 INFO:tasks.workunit.client.0.smithi063.stdout:3/354: dwrite d13/f47 [0,4194304] 0 2024-03-23T11:47:29.314 INFO:tasks.workunit.client.1.smithi111.stdout:2/286: rename d2/de/d27/d2e/l3c to d2/d11/d17/d3e/d5d/l67 0 2024-03-23T11:47:29.318 INFO:tasks.workunit.client.1.smithi111.stdout:8/213: truncate d5/f6 5937413 0 2024-03-23T11:47:29.318 INFO:tasks.workunit.client.0.smithi063.stdout:3/355: rmdir d13 39 2024-03-23T11:47:29.318 INFO:tasks.workunit.client.0.smithi063.stdout:0/253: symlink d2/d4/l59 0 2024-03-23T11:47:29.322 INFO:tasks.workunit.client.1.smithi111.stdout:9/328: dwrite d0/d4/d1c/f66 [0,4194304] 0 2024-03-23T11:47:29.322 INFO:tasks.workunit.client.0.smithi063.stdout:3/356: chown d13/d22/d2d/f4f 19285690 1 2024-03-23T11:47:29.324 INFO:tasks.workunit.client.0.smithi063.stdout:0/254: mkdir d2/d48/d5a 0 2024-03-23T11:47:29.329 INFO:tasks.workunit.client.0.smithi063.stdout:3/357: link d13/d22/f3b d13/d22/d2d/d3f/d67/f6d 0 2024-03-23T11:47:29.331 INFO:tasks.workunit.client.0.smithi063.stdout:8/421: sync 2024-03-23T11:47:29.332 INFO:tasks.workunit.client.0.smithi063.stdout:5/103: sync 2024-03-23T11:47:29.336 INFO:tasks.workunit.client.1.smithi111.stdout:7/213: rename d14/d1b/f32 to d14/f36 0 2024-03-23T11:47:29.336 INFO:tasks.workunit.client.0.smithi063.stdout:3/358: fsync d13/d22/d2d/f38 0 2024-03-23T11:47:29.337 INFO:tasks.workunit.client.0.smithi063.stdout:0/255: getdents d2/d4/dd/d1f/d4f 0 2024-03-23T11:47:29.339 INFO:tasks.workunit.client.0.smithi063.stdout:3/359: stat d13/d22/d2d/f45 0 2024-03-23T11:47:29.340 INFO:tasks.workunit.client.0.smithi063.stdout:5/104: chown d2/d7/f10 147 1 2024-03-23T11:47:29.344 INFO:tasks.workunit.client.0.smithi063.stdout:3/360: mkdir d13/d22/d2d/d3f/d58/d6e 0 2024-03-23T11:47:29.346 INFO:tasks.workunit.client.0.smithi063.stdout:0/256: dread d2/d4/f30 [0,4194304] 0 2024-03-23T11:47:29.347 INFO:tasks.workunit.client.1.smithi111.stdout:7/214: dread f9 [0,4194304] 0 2024-03-23T11:47:29.347 INFO:tasks.workunit.client.1.smithi111.stdout:6/326: creat d2/d1a/f72 x:0 0 0 2024-03-23T11:47:29.348 INFO:tasks.workunit.client.0.smithi063.stdout:5/105: creat d2/d7/d8/d1d/f1f x:0 0 0 2024-03-23T11:47:29.349 INFO:tasks.workunit.client.1.smithi111.stdout:0/260: getdents d5/db/d1f/d19/d45 0 2024-03-23T11:47:29.349 INFO:tasks.workunit.client.0.smithi063.stdout:0/257: read d2/d4/dd/d1f/f2f [2377866,31537] 0 2024-03-23T11:47:29.351 INFO:tasks.workunit.client.1.smithi111.stdout:1/256: link d0/d2b/f45 d0/d2b/f5a 0 2024-03-23T11:47:29.354 INFO:tasks.workunit.client.1.smithi111.stdout:5/303: getdents d1 0 2024-03-23T11:47:29.355 INFO:tasks.workunit.client.1.smithi111.stdout:1/257: dread - d0/d5/dd/d1e/d54/f57 zero size 2024-03-23T11:47:29.357 INFO:tasks.workunit.client.0.smithi063.stdout:3/361: creat d13/d22/f6f x:0 0 0 2024-03-23T11:47:29.357 INFO:tasks.workunit.client.1.smithi111.stdout:9/329: write d0/f13 [3569155,9646] 0 2024-03-23T11:47:29.358 INFO:tasks.workunit.client.0.smithi063.stdout:5/106: truncate d2/fc 5386852 0 2024-03-23T11:47:29.361 INFO:tasks.workunit.client.1.smithi111.stdout:9/330: chown d0/d4/d1c/d36/d28/l58 7 1 2024-03-23T11:47:29.363 INFO:tasks.workunit.client.0.smithi063.stdout:5/107: truncate f1 1192063 0 2024-03-23T11:47:29.363 INFO:tasks.workunit.client.1.smithi111.stdout:2/287: link d2/d11/d17/d1f/l57 d2/d32/d4b/d52/l68 0 2024-03-23T11:47:29.363 INFO:tasks.workunit.client.0.smithi063.stdout:3/362: dread d13/d22/f2c [0,4194304] 0 2024-03-23T11:47:29.365 INFO:tasks.workunit.client.1.smithi111.stdout:1/258: symlink d0/d6/l5b 0 2024-03-23T11:47:29.365 INFO:tasks.workunit.client.1.smithi111.stdout:6/327: dwrite d2/d1f/d45/f4e [0,4194304] 0 2024-03-23T11:47:29.366 INFO:tasks.workunit.client.1.smithi111.stdout:5/304: chown d1/d3/d6/f57 92 1 2024-03-23T11:47:29.366 INFO:tasks.workunit.client.1.smithi111.stdout:2/288: dread - d2/d11/d17/d3e/f40 zero size 2024-03-23T11:47:29.366 INFO:tasks.workunit.client.1.smithi111.stdout:9/331: write d0/d4/d1c/d2d/d3b/d42/d4f/f5c [2356121,121878] 0 2024-03-23T11:47:29.367 INFO:tasks.workunit.client.0.smithi063.stdout:3/363: stat d13/d15/d48/c5f 0 2024-03-23T11:47:29.371 INFO:tasks.workunit.client.1.smithi111.stdout:6/328: fdatasync d2/d35/f40 0 2024-03-23T11:47:29.371 INFO:tasks.workunit.client.0.smithi063.stdout:8/422: dread d6/dd/df/f11 [0,4194304] 0 2024-03-23T11:47:29.373 INFO:tasks.workunit.client.0.smithi063.stdout:3/364: unlink d13/d22/d2d/f31 0 2024-03-23T11:47:29.374 INFO:tasks.workunit.client.1.smithi111.stdout:3/219: dread d0/fd [0,4194304] 0 2024-03-23T11:47:29.374 INFO:tasks.workunit.client.1.smithi111.stdout:6/329: write d2/d17/f55 [4896531,42581] 0 2024-03-23T11:47:29.375 INFO:tasks.workunit.client.1.smithi111.stdout:5/305: rmdir d1/d3/d6/d7/d63 39 2024-03-23T11:47:29.378 INFO:tasks.workunit.client.0.smithi063.stdout:8/423: creat d6/dd/d59/d7f/f8b x:0 0 0 2024-03-23T11:47:29.378 INFO:tasks.workunit.client.1.smithi111.stdout:5/306: stat d1/d20/d5c 0 2024-03-23T11:47:29.380 INFO:tasks.workunit.client.0.smithi063.stdout:8/424: stat d6/dd/d59/d7f 0 2024-03-23T11:47:29.380 INFO:tasks.workunit.client.1.smithi111.stdout:0/261: write d5/f30 [1185240,77207] 0 2024-03-23T11:47:29.380 INFO:tasks.workunit.client.1.smithi111.stdout:6/330: dread - d2/d1a/d1c/d2c/f4c zero size 2024-03-23T11:47:29.384 INFO:tasks.workunit.client.1.smithi111.stdout:9/332: getdents d0/d4/d1c/d36 0 2024-03-23T11:47:29.384 INFO:tasks.workunit.client.0.smithi063.stdout:8/425: creat d6/d1f/d3e/f8c x:0 0 0 2024-03-23T11:47:29.387 INFO:tasks.workunit.client.0.smithi063.stdout:8/426: fdatasync d6/d1c/d27/f4a 0 2024-03-23T11:47:29.388 INFO:tasks.workunit.client.0.smithi063.stdout:5/108: dread d2/fc [4194304,4194304] 0 2024-03-23T11:47:29.389 INFO:tasks.workunit.client.1.smithi111.stdout:1/259: dread d0/f3d [0,4194304] 0 2024-03-23T11:47:29.390 INFO:tasks.workunit.client.1.smithi111.stdout:2/289: write d2/de/d12/f43 [1371552,118237] 0 2024-03-23T11:47:29.391 INFO:tasks.workunit.client.1.smithi111.stdout:7/215: dwrite d14/f1c [0,4194304] 0 2024-03-23T11:47:29.393 INFO:tasks.workunit.client.0.smithi063.stdout:5/109: creat d2/d7/d8/f20 x:0 0 0 2024-03-23T11:47:29.393 INFO:tasks.workunit.client.1.smithi111.stdout:9/333: creat d0/d4/d10/f74 x:0 0 0 2024-03-23T11:47:29.396 INFO:tasks.workunit.client.1.smithi111.stdout:3/220: dread d0/f31 [0,4194304] 0 2024-03-23T11:47:29.397 INFO:tasks.workunit.client.1.smithi111.stdout:9/334: write d0/d4/d1c/d36/d28/f5e [339758,40463] 0 2024-03-23T11:47:29.398 INFO:tasks.workunit.client.1.smithi111.stdout:7/216: write d14/f36 [603272,81079] 0 2024-03-23T11:47:29.399 INFO:tasks.workunit.client.0.smithi063.stdout:8/427: link d6/d1c/f6c d6/d1c/d27/f8d 0 2024-03-23T11:47:29.401 INFO:tasks.workunit.client.0.smithi063.stdout:8/428: chown d6/cc 631833 1 2024-03-23T11:47:29.402 INFO:tasks.workunit.client.1.smithi111.stdout:0/262: getdents d5/db/d1f/d19 0 2024-03-23T11:47:29.403 INFO:tasks.workunit.client.1.smithi111.stdout:4/259: sync 2024-03-23T11:47:29.404 INFO:tasks.workunit.client.1.smithi111.stdout:3/221: fsync d0/fd 0 2024-03-23T11:47:29.407 INFO:tasks.workunit.client.0.smithi063.stdout:8/429: rmdir d6/d1c 39 2024-03-23T11:47:29.407 INFO:tasks.workunit.client.1.smithi111.stdout:7/217: unlink d14/c21 0 2024-03-23T11:47:29.409 INFO:tasks.workunit.client.0.smithi063.stdout:8/430: dread - d6/d1f/f45 zero size 2024-03-23T11:47:29.411 INFO:tasks.workunit.client.1.smithi111.stdout:6/331: truncate d2/d35/f6f 570608 0 2024-03-23T11:47:29.413 INFO:tasks.workunit.client.0.smithi063.stdout:8/431: symlink d6/d1f/d3e/l8e 0 2024-03-23T11:47:29.414 INFO:tasks.workunit.client.1.smithi111.stdout:3/222: fdatasync d0/f16 0 2024-03-23T11:47:29.414 INFO:tasks.workunit.client.0.smithi063.stdout:5/110: chown f1 481834 1 2024-03-23T11:47:29.416 INFO:tasks.workunit.client.1.smithi111.stdout:5/307: dwrite d1/d3/d6/d7/f64 [0,4194304] 0 2024-03-23T11:47:29.418 INFO:tasks.workunit.client.1.smithi111.stdout:7/218: mkdir d14/d37 0 2024-03-23T11:47:29.419 INFO:tasks.workunit.client.0.smithi063.stdout:5/111: write d2/d7/f14 [3266394,91619] 0 2024-03-23T11:47:29.420 INFO:tasks.workunit.client.1.smithi111.stdout:3/223: dread - d0/f1f zero size 2024-03-23T11:47:29.422 INFO:tasks.workunit.client.1.smithi111.stdout:4/260: truncate d0/dc/d4c/d4b/d4d/f51 235382 0 2024-03-23T11:47:29.422 INFO:tasks.workunit.client.1.smithi111.stdout:1/260: dwrite d0/d6/d48/f52 [0,4194304] 0 2024-03-23T11:47:29.424 INFO:tasks.workunit.client.0.smithi063.stdout:5/112: rmdir d2/d7/d8/d1d 39 2024-03-23T11:47:29.428 INFO:tasks.workunit.client.1.smithi111.stdout:6/332: write d2/ff [4117888,24046] 0 2024-03-23T11:47:29.429 INFO:tasks.workunit.client.1.smithi111.stdout:1/261: dread - d0/f59 zero size 2024-03-23T11:47:29.432 INFO:tasks.workunit.client.1.smithi111.stdout:7/219: creat d14/d1b/d1e/d2c/f38 x:0 0 0 2024-03-23T11:47:29.434 INFO:tasks.workunit.client.0.smithi063.stdout:5/113: symlink d2/d7/d8/d1d/l21 0 2024-03-23T11:47:29.438 INFO:tasks.workunit.client.0.smithi063.stdout:8/432: dwrite d6/d1f/d3e/f8c [0,4194304] 0 2024-03-23T11:47:29.439 INFO:tasks.workunit.client.1.smithi111.stdout:9/335: dwrite d0/d4/f47 [0,4194304] 0 2024-03-23T11:47:29.441 INFO:tasks.workunit.client.1.smithi111.stdout:4/261: write d0/d7/da/f46 [211627,43250] 0 2024-03-23T11:47:29.443 INFO:tasks.workunit.client.0.smithi063.stdout:8/433: fdatasync d6/d1c/f2d 0 2024-03-23T11:47:29.444 INFO:tasks.workunit.client.1.smithi111.stdout:6/333: rmdir d2/d1a/d1c/d2c 39 2024-03-23T11:47:29.445 INFO:tasks.workunit.client.1.smithi111.stdout:0/263: dread d5/db/f38 [0,4194304] 0 2024-03-23T11:47:29.445 INFO:tasks.workunit.client.1.smithi111.stdout:3/224: symlink d0/d1a/d2b/d38/l3d 0 2024-03-23T11:47:29.445 INFO:tasks.workunit.client.1.smithi111.stdout:2/290: dwrite d2/d11/f37 [0,4194304] 0 2024-03-23T11:47:29.449 INFO:tasks.workunit.client.0.smithi063.stdout:8/434: symlink d6/d1f/d54/d5c/l8f 0 2024-03-23T11:47:29.452 INFO:tasks.workunit.client.1.smithi111.stdout:7/220: write d14/d1b/f2d [1580430,97953] 0 2024-03-23T11:47:29.453 INFO:tasks.workunit.client.1.smithi111.stdout:9/336: read - d0/d4/d10/d29/f67 zero size 2024-03-23T11:47:29.454 INFO:tasks.workunit.client.0.smithi063.stdout:8/435: unlink d6/d1f/f45 0 2024-03-23T11:47:29.455 INFO:tasks.workunit.client.1.smithi111.stdout:0/264: chown d5/db/d1f/d32/c50 969757425 1 2024-03-23T11:47:29.456 INFO:tasks.workunit.client.0.smithi063.stdout:8/436: chown d6/d1c/d32/d37/l5b 77537 1 2024-03-23T11:47:29.456 INFO:tasks.workunit.client.1.smithi111.stdout:5/308: write d1/d3/d6/d7/f1f [4955426,110803] 0 2024-03-23T11:47:29.457 INFO:tasks.workunit.client.0.smithi063.stdout:5/114: write d2/f4 [2203806,87173] 0 2024-03-23T11:47:29.458 INFO:tasks.workunit.client.1.smithi111.stdout:0/265: fsync d5/f13 0 2024-03-23T11:47:29.458 INFO:tasks.workunit.client.1.smithi111.stdout:2/291: write d2/d11/f51 [454829,62916] 0 2024-03-23T11:47:29.458 INFO:tasks.workunit.client.1.smithi111.stdout:6/334: chown d2/d1a/d1c/d2c/d51 0 1 2024-03-23T11:47:29.459 INFO:tasks.workunit.client.0.smithi063.stdout:8/437: chown d6/dd 2893102 1 2024-03-23T11:47:29.460 INFO:tasks.workunit.client.0.smithi063.stdout:5/115: chown d2/d7/d8/de 0 1 2024-03-23T11:47:29.460 INFO:tasks.workunit.client.1.smithi111.stdout:9/337: write d0/d4/d10/f74 [42758,7761] 0 2024-03-23T11:47:29.461 INFO:tasks.workunit.client.1.smithi111.stdout:1/262: link d0/d6/fc d0/d2b/f5c 0 2024-03-23T11:47:29.462 INFO:tasks.workunit.client.1.smithi111.stdout:7/221: creat d14/d1b/d1e/f39 x:0 0 0 2024-03-23T11:47:29.464 INFO:tasks.workunit.client.0.smithi063.stdout:5/116: rmdir d2/d7/d8/d1d 39 2024-03-23T11:47:29.465 INFO:tasks.workunit.client.1.smithi111.stdout:5/309: write d1/d3/fc [798775,23981] 0 2024-03-23T11:47:29.465 INFO:tasks.workunit.client.1.smithi111.stdout:3/225: rmdir d0 39 2024-03-23T11:47:29.470 INFO:tasks.workunit.client.0.smithi063.stdout:5/117: symlink d2/d7/l22 0 2024-03-23T11:47:29.470 INFO:tasks.workunit.client.1.smithi111.stdout:4/262: getdents d0/d7/da 0 2024-03-23T11:47:29.470 INFO:tasks.workunit.client.1.smithi111.stdout:2/292: mknod d2/de/c69 0 2024-03-23T11:47:29.472 INFO:tasks.workunit.client.1.smithi111.stdout:6/335: write d2/d1f/d45/f63 [585597,67940] 0 2024-03-23T11:47:29.472 INFO:tasks.workunit.client.1.smithi111.stdout:1/263: chown d0/d2b/d37/l38 2280 1 2024-03-23T11:47:29.473 INFO:tasks.workunit.client.1.smithi111.stdout:9/338: write d0/d4/d1c/d36/f11 [1722470,3833] 0 2024-03-23T11:47:29.473 INFO:tasks.workunit.client.0.smithi063.stdout:5/118: write d2/d7/d8/f12 [3044520,92674] 0 2024-03-23T11:47:29.474 INFO:tasks.workunit.client.1.smithi111.stdout:8/214: sync 2024-03-23T11:47:29.474 INFO:tasks.workunit.client.0.smithi063.stdout:8/438: dread d6/d1c/f3c [0,4194304] 0 2024-03-23T11:47:29.475 INFO:tasks.workunit.client.1.smithi111.stdout:3/226: chown d0/d1a/d1d/f36 1982102904 1 2024-03-23T11:47:29.476 INFO:tasks.workunit.client.1.smithi111.stdout:2/293: write d2/d32/d4b/f66 [185253,31855] 0 2024-03-23T11:47:29.476 INFO:tasks.workunit.client.1.smithi111.stdout:9/339: fsync d0/f5f 0 2024-03-23T11:47:29.477 INFO:tasks.workunit.client.1.smithi111.stdout:7/222: write d14/d1b/d20/f2b [1784170,67580] 0 2024-03-23T11:47:29.477 INFO:tasks.workunit.client.0.smithi063.stdout:5/119: creat d2/d7/f23 x:0 0 0 2024-03-23T11:47:29.477 INFO:tasks.workunit.client.1.smithi111.stdout:1/264: dread - d0/f13 zero size 2024-03-23T11:47:29.477 INFO:tasks.workunit.client.1.smithi111.stdout:0/266: rename d5/db/d1f/d32/f44 to d5/db/d4d/f52 0 2024-03-23T11:47:29.480 INFO:tasks.workunit.client.1.smithi111.stdout:5/310: write d1/d3/d6/d7/d63/d2c/f52 [767763,18790] 0 2024-03-23T11:47:29.483 INFO:tasks.workunit.client.0.smithi063.stdout:5/120: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:29.483 INFO:tasks.workunit.client.1.smithi111.stdout:3/227: mknod d0/c3e 0 2024-03-23T11:47:29.484 INFO:tasks.workunit.client.1.smithi111.stdout:0/267: write d5/db/f26 [308897,46034] 0 2024-03-23T11:47:29.485 INFO:tasks.workunit.client.1.smithi111.stdout:7/223: write d14/d1b/f2d [663938,90401] 0 2024-03-23T11:47:29.485 INFO:tasks.workunit.client.0.smithi063.stdout:8/439: rename d6/d1c/d32/d37/d86 to d6/d1f/d54/d63/d90 0 2024-03-23T11:47:29.486 INFO:tasks.workunit.client.1.smithi111.stdout:5/311: chown d1/d20/d41 24 1 2024-03-23T11:47:29.487 INFO:tasks.workunit.client.0.smithi063.stdout:5/121: mknod d2/d7/d8/de/c24 0 2024-03-23T11:47:29.487 INFO:tasks.workunit.client.0.smithi063.stdout:8/440: fdatasync d6/dd/df/f4b 0 2024-03-23T11:47:29.489 INFO:tasks.workunit.client.1.smithi111.stdout:9/340: mkdir d0/d4/d1c/d2d/d75 0 2024-03-23T11:47:29.492 INFO:tasks.workunit.client.1.smithi111.stdout:8/215: symlink d5/d26/d2f/d42/l43 0 2024-03-23T11:47:29.493 INFO:tasks.workunit.client.0.smithi063.stdout:5/122: dread d2/d7/f1c [0,4194304] 0 2024-03-23T11:47:29.493 INFO:tasks.workunit.client.0.smithi063.stdout:8/441: symlink d6/d1f/d54/l91 0 2024-03-23T11:47:29.493 INFO:tasks.workunit.client.1.smithi111.stdout:0/268: dread d5/f30 [0,4194304] 0 2024-03-23T11:47:29.494 INFO:tasks.workunit.client.1.smithi111.stdout:4/263: getdents d0/d17 0 2024-03-23T11:47:29.495 INFO:tasks.workunit.client.1.smithi111.stdout:5/312: write d1/d3/d6/d29/f4b [691727,16754] 0 2024-03-23T11:47:29.495 INFO:tasks.workunit.client.0.smithi063.stdout:5/123: creat d2/f25 x:0 0 0 2024-03-23T11:47:29.496 INFO:tasks.workunit.client.1.smithi111.stdout:1/265: link d0/d5/f16 d0/d6/d48/f5d 0 2024-03-23T11:47:29.497 INFO:tasks.workunit.client.0.smithi063.stdout:8/442: creat d6/d1f/d54/d5c/d76/f92 x:0 0 0 2024-03-23T11:47:29.497 INFO:tasks.workunit.client.1.smithi111.stdout:2/294: creat d2/d7/da/d19/f6a x:0 0 0 2024-03-23T11:47:29.498 INFO:tasks.workunit.client.1.smithi111.stdout:3/228: truncate d0/d7/f10 1461328 0 2024-03-23T11:47:29.500 INFO:tasks.workunit.client.0.smithi063.stdout:5/124: rename d2/d7/l18 to d2/d17/l26 0 2024-03-23T11:47:29.501 INFO:tasks.workunit.client.0.smithi063.stdout:8/443: creat d6/d1c/d32/f93 x:0 0 0 2024-03-23T11:47:29.502 INFO:tasks.workunit.client.1.smithi111.stdout:9/341: rename d0/d4/d1c/d36/c1a to d0/d4/d1c/d2d/c76 0 2024-03-23T11:47:29.503 INFO:tasks.workunit.client.0.smithi063.stdout:5/125: mkdir d2/d7/d27 0 2024-03-23T11:47:29.504 INFO:tasks.workunit.client.1.smithi111.stdout:0/269: creat d5/db/d1f/d19/f53 x:0 0 0 2024-03-23T11:47:29.505 INFO:tasks.workunit.client.1.smithi111.stdout:5/313: fsync d1/d20/f61 0 2024-03-23T11:47:29.506 INFO:tasks.workunit.client.0.smithi063.stdout:5/126: fsync d2/d7/d8/d1d/f1f 0 2024-03-23T11:47:29.508 INFO:tasks.workunit.client.0.smithi063.stdout:5/127: readlink d2/d7/d8/de/l16 0 2024-03-23T11:47:29.508 INFO:tasks.workunit.client.1.smithi111.stdout:3/229: write d0/d7/d19/f29 [938995,78693] 0 2024-03-23T11:47:29.509 INFO:tasks.workunit.client.0.smithi063.stdout:8/444: truncate d6/d1c/f2d 3864961 0 2024-03-23T11:47:29.509 INFO:tasks.workunit.client.0.smithi063.stdout:5/128: fdatasync d2/f4 0 2024-03-23T11:47:29.510 INFO:tasks.workunit.client.1.smithi111.stdout:5/314: chown d1/d5d/f26 1 1 2024-03-23T11:47:29.511 INFO:tasks.workunit.client.0.smithi063.stdout:5/129: mkdir d2/d28 0 2024-03-23T11:47:29.512 INFO:tasks.workunit.client.1.smithi111.stdout:2/295: rename d2/de/d27 to d2/d11/d17/d3e/d6b 0 2024-03-23T11:47:29.513 INFO:tasks.workunit.client.0.smithi063.stdout:5/130: chown d2/f4 5 1 2024-03-23T11:47:29.515 INFO:tasks.workunit.client.1.smithi111.stdout:7/224: dread f12 [0,4194304] 0 2024-03-23T11:47:29.516 INFO:tasks.workunit.client.1.smithi111.stdout:9/342: symlink d0/d31/d41/d65/l77 0 2024-03-23T11:47:29.517 INFO:tasks.workunit.client.1.smithi111.stdout:3/230: mknod d0/d1a/c3f 0 2024-03-23T11:47:29.518 INFO:tasks.workunit.client.1.smithi111.stdout:8/216: dwrite d5/fe [0,4194304] 0 2024-03-23T11:47:29.519 INFO:tasks.workunit.client.1.smithi111.stdout:7/225: chown d14/d1b/d1e/d2c/d2e/f34 82421861 1 2024-03-23T11:47:29.519 INFO:tasks.workunit.client.1.smithi111.stdout:3/231: chown d0/f16 1 1 2024-03-23T11:47:29.523 INFO:tasks.workunit.client.1.smithi111.stdout:2/296: rmdir d2/d11/d17 39 2024-03-23T11:47:29.523 INFO:tasks.workunit.client.1.smithi111.stdout:8/217: write d5/d27/f36 [385833,26818] 0 2024-03-23T11:47:29.524 INFO:tasks.workunit.client.0.smithi063.stdout:8/445: dread d6/d1f/d54/d5c/f6d [0,4194304] 0 2024-03-23T11:47:29.525 INFO:tasks.workunit.client.1.smithi111.stdout:7/226: read d14/f36 [393146,68393] 0 2024-03-23T11:47:29.525 INFO:tasks.workunit.client.1.smithi111.stdout:9/343: mknod d0/d4/d10/c78 0 2024-03-23T11:47:29.527 INFO:tasks.workunit.client.0.smithi063.stdout:5/131: rename d2/d7/l22 to d2/d7/d8/l29 0 2024-03-23T11:47:29.529 INFO:tasks.workunit.client.1.smithi111.stdout:8/218: write d5/f2d [159137,8048] 0 2024-03-23T11:47:29.530 INFO:tasks.workunit.client.1.smithi111.stdout:4/264: dread d0/dc/f1c [0,4194304] 0 2024-03-23T11:47:29.532 INFO:tasks.workunit.client.0.smithi063.stdout:5/132: truncate f1 854556 0 2024-03-23T11:47:29.532 INFO:tasks.workunit.client.1.smithi111.stdout:5/315: creat d1/d3/d6/d29/f6b x:0 0 0 2024-03-23T11:47:29.534 INFO:tasks.workunit.client.1.smithi111.stdout:4/265: read - d0/dc/d1f/f4a zero size 2024-03-23T11:47:29.534 INFO:tasks.workunit.client.1.smithi111.stdout:1/266: write d0/d5/dd/f10 [187639,17474] 0 2024-03-23T11:47:29.535 INFO:tasks.workunit.client.0.smithi063.stdout:5/133: write d2/d7/d8/f20 [763539,95384] 0 2024-03-23T11:47:29.536 INFO:tasks.workunit.client.1.smithi111.stdout:4/266: dread - d0/dc/f55 zero size 2024-03-23T11:47:29.536 INFO:tasks.workunit.client.1.smithi111.stdout:9/344: symlink d0/d4/d10/d29/l79 0 2024-03-23T11:47:29.538 INFO:tasks.workunit.client.0.smithi063.stdout:8/446: dread d6/d1f/d54/d63/d90/f13 [0,4194304] 0 2024-03-23T11:47:29.539 INFO:tasks.workunit.client.1.smithi111.stdout:3/232: dwrite d0/f9 [0,4194304] 0 2024-03-23T11:47:29.540 INFO:tasks.workunit.client.0.smithi063.stdout:8/447: write d6/f71 [841819,21968] 0 2024-03-23T11:47:29.543 INFO:tasks.workunit.client.1.smithi111.stdout:8/219: symlink d5/d1b/l44 0 2024-03-23T11:47:29.543 INFO:tasks.workunit.client.1.smithi111.stdout:7/227: creat d14/d1b/d20/f3a x:0 0 0 2024-03-23T11:47:29.545 INFO:tasks.workunit.client.1.smithi111.stdout:9/345: unlink d0/d31/d41/l49 0 2024-03-23T11:47:29.548 INFO:tasks.workunit.client.1.smithi111.stdout:2/297: rename d2/d3d to d2/de/d6c 0 2024-03-23T11:47:29.548 INFO:tasks.workunit.client.1.smithi111.stdout:3/233: write d0/d7/f12 [3766926,20659] 0 2024-03-23T11:47:29.551 INFO:tasks.workunit.client.0.smithi063.stdout:8/448: symlink d6/d1c/d27/d3f/l94 0 2024-03-23T11:47:29.552 INFO:tasks.workunit.client.1.smithi111.stdout:8/220: rmdir d5/d26 39 2024-03-23T11:47:29.553 INFO:tasks.workunit.client.1.smithi111.stdout:5/316: fsync d1/d3/d6/d7/f53 0 2024-03-23T11:47:29.557 INFO:tasks.workunit.client.0.smithi063.stdout:8/449: unlink d6/d1c/f4f 0 2024-03-23T11:47:29.557 INFO:tasks.workunit.client.1.smithi111.stdout:3/234: chown d0/d1a/l22 42 1 2024-03-23T11:47:29.559 INFO:tasks.workunit.client.1.smithi111.stdout:9/346: read d0/d31/d41/d65/f6f [1837789,66607] 0 2024-03-23T11:47:29.559 INFO:tasks.workunit.client.0.smithi063.stdout:8/450: read - d6/d1f/d3e/f84 zero size 2024-03-23T11:47:29.560 INFO:tasks.workunit.client.1.smithi111.stdout:7/228: mknod d14/d1b/d1e/c3b 0 2024-03-23T11:47:29.561 INFO:tasks.workunit.client.1.smithi111.stdout:2/298: creat d2/d32/d4b/d52/d58/f6d x:0 0 0 2024-03-23T11:47:29.562 INFO:tasks.workunit.client.0.smithi063.stdout:5/134: dwrite d2/d7/f10 [0,4194304] 0 2024-03-23T11:47:29.564 INFO:tasks.workunit.client.1.smithi111.stdout:7/229: chown d14/d1b/f2d 88439 1 2024-03-23T11:47:29.565 INFO:tasks.workunit.client.0.smithi063.stdout:8/451: creat d6/d1f/d54/d63/d90/d15/f95 x:0 0 0 2024-03-23T11:47:29.565 INFO:tasks.workunit.client.0.smithi063.stdout:1/216: sync 2024-03-23T11:47:29.565 INFO:tasks.workunit.client.0.smithi063.stdout:9/211: sync 2024-03-23T11:47:29.565 INFO:tasks.workunit.client.1.smithi111.stdout:4/267: dwrite d0/dc/f35 [0,4194304] 0 2024-03-23T11:47:29.566 INFO:tasks.workunit.client.1.smithi111.stdout:9/347: chown d0/d4/d10/f51 0 1 2024-03-23T11:47:29.569 INFO:tasks.workunit.client.0.smithi063.stdout:9/212: dread d4/d11/f17 [0,4194304] 0 2024-03-23T11:47:29.571 INFO:tasks.workunit.client.1.smithi111.stdout:2/299: stat d2/d11/d17/d3e/d6b/d2e/d3f/f44 0 2024-03-23T11:47:29.571 INFO:tasks.workunit.client.1.smithi111.stdout:8/221: truncate d5/f22 408387 0 2024-03-23T11:47:29.576 INFO:tasks.workunit.client.1.smithi111.stdout:8/222: truncate f1 3769382 0 2024-03-23T11:47:29.576 INFO:tasks.workunit.client.0.smithi063.stdout:5/135: mkdir d2/d7/d8/de/d2a 0 2024-03-23T11:47:29.578 INFO:tasks.workunit.client.0.smithi063.stdout:8/452: symlink d6/d1c/d27/d3f/l96 0 2024-03-23T11:47:29.578 INFO:tasks.workunit.client.0.smithi063.stdout:1/217: getdents d3/d1c/d4b 0 2024-03-23T11:47:29.580 INFO:tasks.workunit.client.1.smithi111.stdout:3/235: dread d0/f31 [0,4194304] 0 2024-03-23T11:47:29.580 INFO:tasks.workunit.client.0.smithi063.stdout:5/136: mkdir d2/d7/d2b 0 2024-03-23T11:47:29.581 INFO:tasks.workunit.client.0.smithi063.stdout:8/453: readlink d6/d1c/d32/l39 0 2024-03-23T11:47:29.583 INFO:tasks.workunit.client.0.smithi063.stdout:1/218: mkdir d3/d3f/d4d 0 2024-03-23T11:47:29.586 INFO:tasks.workunit.client.1.smithi111.stdout:7/230: dwrite d14/d1b/d1e/d2c/f38 [0,4194304] 0 2024-03-23T11:47:29.587 INFO:tasks.workunit.client.0.smithi063.stdout:5/137: creat d2/f2c x:0 0 0 2024-03-23T11:47:29.591 INFO:tasks.workunit.client.0.smithi063.stdout:9/213: dwrite d4/ff [0,4194304] 0 2024-03-23T11:47:29.591 INFO:tasks.workunit.client.1.smithi111.stdout:7/231: readlink d14/d1b/d1e/l28 0 2024-03-23T11:47:29.595 INFO:tasks.workunit.client.1.smithi111.stdout:7/232: fsync d14/f25 0 2024-03-23T11:47:29.598 INFO:tasks.workunit.client.0.smithi063.stdout:9/214: mknod d4/c48 0 2024-03-23T11:47:29.598 INFO:tasks.workunit.client.0.smithi063.stdout:8/454: creat d6/dd/d59/d7f/f97 x:0 0 0 2024-03-23T11:47:29.599 INFO:tasks.workunit.client.0.smithi063.stdout:1/219: link d3/df/l20 d3/d1c/l4e 0 2024-03-23T11:47:29.599 INFO:tasks.workunit.client.1.smithi111.stdout:7/233: dread - d14/d1b/d1e/d2c/f31 zero size 2024-03-23T11:47:29.602 INFO:tasks.workunit.client.0.smithi063.stdout:8/455: read d6/d1f/d54/d63/d90/f13 [450313,43057] 0 2024-03-23T11:47:29.604 INFO:tasks.workunit.client.1.smithi111.stdout:3/236: dwrite d0/d7/d19/f29 [0,4194304] 0 2024-03-23T11:47:29.604 INFO:tasks.workunit.client.0.smithi063.stdout:9/215: mkdir d4/d31/d43/d49 0 2024-03-23T11:47:29.609 INFO:tasks.workunit.client.1.smithi111.stdout:3/237: rmdir d0/d1a/d1d 39 2024-03-23T11:47:29.609 INFO:tasks.workunit.client.1.smithi111.stdout:2/300: dread d2/d11/f13 [0,4194304] 0 2024-03-23T11:47:29.613 INFO:tasks.workunit.client.1.smithi111.stdout:2/301: fdatasync d2/d7/da/d19/f33 0 2024-03-23T11:47:29.614 INFO:tasks.workunit.client.1.smithi111.stdout:7/234: dread fe [0,4194304] 0 2024-03-23T11:47:29.618 INFO:tasks.workunit.client.1.smithi111.stdout:7/235: creat d14/d1b/d20/f3c x:0 0 0 2024-03-23T11:47:29.619 INFO:tasks.workunit.client.0.smithi063.stdout:9/216: rename d4/d11/f17 to d4/d2c/f4a 0 2024-03-23T11:47:29.619 INFO:tasks.workunit.client.1.smithi111.stdout:2/302: symlink d2/d32/d4b/d52/l6e 0 2024-03-23T11:47:29.619 INFO:tasks.workunit.client.1.smithi111.stdout:6/336: sync 2024-03-23T11:47:29.620 INFO:tasks.workunit.client.0.smithi063.stdout:1/220: link d3/df/l14 d3/d35/d40/l4f 0 2024-03-23T11:47:29.620 INFO:tasks.workunit.client.0.smithi063.stdout:5/138: dwrite d2/d17/f19 [0,4194304] 0 2024-03-23T11:47:29.621 INFO:tasks.workunit.client.1.smithi111.stdout:2/303: chown d2/d7/l2b 951292562 1 2024-03-23T11:47:29.622 INFO:tasks.workunit.client.1.smithi111.stdout:2/304: readlink d2/d32/d4b/d52/l62 0 2024-03-23T11:47:29.623 INFO:tasks.workunit.client.1.smithi111.stdout:7/236: creat d14/d1b/d1e/d2c/d2e/d33/f3d x:0 0 0 2024-03-23T11:47:29.623 INFO:tasks.workunit.client.1.smithi111.stdout:6/337: mkdir d2/d73 0 2024-03-23T11:47:29.623 INFO:tasks.workunit.client.1.smithi111.stdout:7/237: stat d14/d1b 0 2024-03-23T11:47:29.631 INFO:tasks.workunit.client.1.smithi111.stdout:2/305: creat d2/d11/d17/d3e/d6b/d2e/f6f x:0 0 0 2024-03-23T11:47:29.633 INFO:tasks.workunit.client.1.smithi111.stdout:6/338: mknod d2/d1a/d1c/d2c/d51/c74 0 2024-03-23T11:47:29.638 INFO:tasks.workunit.client.0.smithi063.stdout:8/456: getdents d6/d1f/d54/d63/d90/d5d 0 2024-03-23T11:47:29.638 INFO:tasks.workunit.client.0.smithi063.stdout:9/217: dread d4/d11/f25 [0,4194304] 0 2024-03-23T11:47:29.639 INFO:tasks.workunit.client.1.smithi111.stdout:6/339: mknod d2/d35/d70/c75 0 2024-03-23T11:47:29.643 INFO:tasks.workunit.client.1.smithi111.stdout:7/238: dwrite d14/d1b/d1e/d2c/f38 [0,4194304] 0 2024-03-23T11:47:29.644 INFO:tasks.workunit.client.1.smithi111.stdout:2/306: getdents d2/d32/d4b/d52 0 2024-03-23T11:47:29.645 INFO:tasks.workunit.client.0.smithi063.stdout:9/218: mknod d4/d11/d44/c4b 0 2024-03-23T11:47:29.648 INFO:tasks.workunit.client.0.smithi063.stdout:9/219: write d4/d11/d19/f23 [754484,51332] 0 2024-03-23T11:47:29.649 INFO:tasks.workunit.client.1.smithi111.stdout:2/307: mkdir d2/de/d12/d70 0 2024-03-23T11:47:29.651 INFO:tasks.workunit.client.1.smithi111.stdout:7/239: creat d14/d1b/d20/f3e x:0 0 0 2024-03-23T11:47:29.651 INFO:tasks.workunit.client.0.smithi063.stdout:9/220: write d4/d11/d19/f22 [798165,15452] 0 2024-03-23T11:47:29.655 INFO:tasks.workunit.client.1.smithi111.stdout:3/238: dread d0/d7/f12 [0,4194304] 0 2024-03-23T11:47:29.659 INFO:tasks.workunit.client.1.smithi111.stdout:3/239: mknod d0/d1a/d2d/c40 0 2024-03-23T11:47:29.663 INFO:tasks.workunit.client.0.smithi063.stdout:8/457: truncate d6/f71 243402 0 2024-03-23T11:47:29.663 INFO:tasks.workunit.client.1.smithi111.stdout:6/340: dwrite d2/d17/f5a [0,4194304] 0 2024-03-23T11:47:29.668 INFO:tasks.workunit.client.0.smithi063.stdout:8/458: creat d6/f98 x:0 0 0 2024-03-23T11:47:29.669 INFO:tasks.workunit.client.1.smithi111.stdout:6/341: fsync d2/d35/f44 0 2024-03-23T11:47:29.677 INFO:tasks.workunit.client.1.smithi111.stdout:6/342: dread d2/d17/f20 [0,4194304] 0 2024-03-23T11:47:29.678 INFO:tasks.workunit.client.0.smithi063.stdout:8/459: creat d6/d1c/d27/f99 x:0 0 0 2024-03-23T11:47:29.679 INFO:tasks.workunit.client.0.smithi063.stdout:5/139: dwrite d2/d7/d8/f13 [0,4194304] 0 2024-03-23T11:47:29.680 INFO:tasks.workunit.client.0.smithi063.stdout:9/221: dwrite d4/f21 [0,4194304] 0 2024-03-23T11:47:29.682 INFO:tasks.workunit.client.0.smithi063.stdout:8/460: truncate d6/dd/df/f4b 492858 0 2024-03-23T11:47:29.684 INFO:tasks.workunit.client.0.smithi063.stdout:9/222: truncate d4/f16 5297031 0 2024-03-23T11:47:29.686 INFO:tasks.workunit.client.0.smithi063.stdout:9/223: write d4/f35 [1163469,14360] 0 2024-03-23T11:47:29.686 INFO:tasks.workunit.client.1.smithi111.stdout:2/308: dread d2/de/d12/f2c [0,4194304] 0 2024-03-23T11:47:29.687 INFO:tasks.workunit.client.0.smithi063.stdout:9/224: dread - d4/f47 zero size 2024-03-23T11:47:29.690 INFO:tasks.workunit.client.0.smithi063.stdout:9/225: readlink d4/d11/d19/l29 0 2024-03-23T11:47:29.691 INFO:tasks.workunit.client.1.smithi111.stdout:2/309: mknod d2/d11/d17/c71 0 2024-03-23T11:47:29.693 INFO:tasks.workunit.client.1.smithi111.stdout:6/343: dread d2/d1a/d1c/f24 [0,4194304] 0 2024-03-23T11:47:29.693 INFO:tasks.workunit.client.1.smithi111.stdout:2/310: chown d2/d5c/f63 90105751 1 2024-03-23T11:47:29.695 INFO:tasks.workunit.client.0.smithi063.stdout:5/140: creat d2/d28/f2d x:0 0 0 2024-03-23T11:47:29.696 INFO:tasks.workunit.client.1.smithi111.stdout:7/240: dread f1 [0,4194304] 0 2024-03-23T11:47:29.697 INFO:tasks.workunit.client.1.smithi111.stdout:6/344: creat d2/d1f/d45/f76 x:0 0 0 2024-03-23T11:47:29.699 INFO:tasks.workunit.client.0.smithi063.stdout:5/141: write d2/d28/f2d [400762,117915] 0 2024-03-23T11:47:29.700 INFO:tasks.workunit.client.1.smithi111.stdout:6/345: creat d2/d35/f77 x:0 0 0 2024-03-23T11:47:29.700 INFO:tasks.workunit.client.1.smithi111.stdout:7/241: truncate d14/d1b/d1e/d2c/f31 1040587 0 2024-03-23T11:47:29.706 INFO:tasks.workunit.client.0.smithi063.stdout:9/226: rename d4/d11 to d4/d4c 0 2024-03-23T11:47:29.708 INFO:tasks.workunit.client.1.smithi111.stdout:6/346: link d2/d17/l19 d2/d73/l78 0 2024-03-23T11:47:29.709 INFO:tasks.workunit.client.0.smithi063.stdout:5/142: link d2/d7/d8/de/c24 d2/d7/d8/d1d/c2e 0 2024-03-23T11:47:29.712 INFO:tasks.workunit.client.0.smithi063.stdout:8/461: dread d6/d1f/d54/d63/d90/f35 [0,4194304] 0 2024-03-23T11:47:29.712 INFO:tasks.workunit.client.0.smithi063.stdout:5/143: mkdir d2/d2f 0 2024-03-23T11:47:29.713 INFO:tasks.workunit.client.1.smithi111.stdout:2/311: truncate d2/d7/da/d19/f34 2120937 0 2024-03-23T11:47:29.713 INFO:tasks.workunit.client.0.smithi063.stdout:9/227: read d4/fe [1361141,23427] 0 2024-03-23T11:47:29.715 INFO:tasks.workunit.client.1.smithi111.stdout:7/242: dwrite d14/d1b/d20/f2b [0,4194304] 0 2024-03-23T11:47:29.716 INFO:tasks.workunit.client.0.smithi063.stdout:5/144: readlink d2/d17/l26 0 2024-03-23T11:47:29.717 INFO:tasks.workunit.client.0.smithi063.stdout:8/462: mkdir d6/d1f/d54/d63/d90/d5d/d7d/d9a 0 2024-03-23T11:47:29.718 INFO:tasks.workunit.client.1.smithi111.stdout:7/243: creat d14/d37/f3f x:0 0 0 2024-03-23T11:47:29.719 INFO:tasks.workunit.client.0.smithi063.stdout:9/228: mknod d4/d4c/c4d 0 2024-03-23T11:47:29.724 INFO:tasks.workunit.client.0.smithi063.stdout:8/463: symlink d6/d1c/l9b 0 2024-03-23T11:47:29.724 INFO:tasks.workunit.client.0.smithi063.stdout:9/229: symlink d4/d31/l4e 0 2024-03-23T11:47:29.725 INFO:tasks.workunit.client.0.smithi063.stdout:5/145: symlink d2/d2f/l30 0 2024-03-23T11:47:29.728 INFO:tasks.workunit.client.0.smithi063.stdout:9/230: symlink d4/d31/l4f 0 2024-03-23T11:47:29.728 INFO:tasks.workunit.client.0.smithi063.stdout:5/146: write d2/d7/d8/f20 [724674,32380] 0 2024-03-23T11:47:29.730 INFO:tasks.workunit.client.0.smithi063.stdout:8/464: write d6/d1f/d54/d63/d90/d15/f17 [403727,49335] 0 2024-03-23T11:47:29.732 INFO:tasks.workunit.client.0.smithi063.stdout:5/147: mknod d2/c31 0 2024-03-23T11:47:29.733 INFO:tasks.workunit.client.1.smithi111.stdout:7/244: dwrite d14/d1b/d1e/d2c/f38 [0,4194304] 0 2024-03-23T11:47:29.734 INFO:tasks.workunit.client.0.smithi063.stdout:9/231: rename d4/d4c/d19/f3a to d4/d31/d43/f50 0 2024-03-23T11:47:29.736 INFO:tasks.workunit.client.0.smithi063.stdout:8/465: getdents d6/d30 0 2024-03-23T11:47:29.736 INFO:tasks.workunit.client.0.smithi063.stdout:9/232: truncate d4/d4c/f30 4683312 0 2024-03-23T11:47:29.737 INFO:tasks.workunit.client.1.smithi111.stdout:2/312: dread d2/d7/da/ff [0,4194304] 0 2024-03-23T11:47:29.740 INFO:tasks.workunit.client.1.smithi111.stdout:2/313: rmdir d2/d11/d17 39 2024-03-23T11:47:29.740 INFO:tasks.workunit.client.0.smithi063.stdout:8/466: mknod d6/d30/c9c 0 2024-03-23T11:47:29.742 INFO:tasks.workunit.client.1.smithi111.stdout:2/314: write d2/d11/f1b [897728,121243] 0 2024-03-23T11:47:29.744 INFO:tasks.workunit.client.0.smithi063.stdout:9/233: dread d4/f16 [4194304,4194304] 0 2024-03-23T11:47:29.745 INFO:tasks.workunit.client.1.smithi111.stdout:2/315: write d2/d32/d4b/f66 [914102,73060] 0 2024-03-23T11:47:29.745 INFO:tasks.workunit.client.0.smithi063.stdout:8/467: truncate d6/d1f/d54/d63/d90/f28 2284524 0 2024-03-23T11:47:29.747 INFO:tasks.workunit.client.1.smithi111.stdout:2/316: write d2/de/d12/f43 [1425762,130301] 0 2024-03-23T11:47:29.750 INFO:tasks.workunit.client.0.smithi063.stdout:9/234: creat d4/d4c/d44/f51 x:0 0 0 2024-03-23T11:47:29.752 INFO:tasks.workunit.client.1.smithi111.stdout:2/317: getdents d2/de/d12/d70 0 2024-03-23T11:47:29.752 INFO:tasks.workunit.client.1.smithi111.stdout:7/245: dwrite d14/d1b/d1e/d2c/d2e/d33/f3d [0,4194304] 0 2024-03-23T11:47:29.754 INFO:tasks.workunit.client.0.smithi063.stdout:9/235: symlink d4/d31/d43/l52 0 2024-03-23T11:47:29.755 INFO:tasks.workunit.client.0.smithi063.stdout:9/236: dread - d4/f47 zero size 2024-03-23T11:47:29.755 INFO:tasks.workunit.client.1.smithi111.stdout:7/246: symlink d14/d37/l40 0 2024-03-23T11:47:29.759 INFO:tasks.workunit.client.0.smithi063.stdout:9/237: chown d4/cb 23553664 1 2024-03-23T11:47:29.763 INFO:tasks.workunit.client.0.smithi063.stdout:9/238: chown d4/d4c/d19/f1b 137082 1 2024-03-23T11:47:29.765 INFO:tasks.workunit.client.1.smithi111.stdout:2/318: dwrite d2/f6 [0,4194304] 0 2024-03-23T11:47:29.767 INFO:tasks.workunit.client.0.smithi063.stdout:9/239: symlink d4/d4c/d44/l53 0 2024-03-23T11:47:29.767 INFO:tasks.workunit.client.1.smithi111.stdout:1/267: sync 2024-03-23T11:47:29.767 INFO:tasks.workunit.client.1.smithi111.stdout:0/270: sync 2024-03-23T11:47:29.767 INFO:tasks.workunit.client.1.smithi111.stdout:5/317: sync 2024-03-23T11:47:29.768 INFO:tasks.workunit.client.1.smithi111.stdout:4/268: sync 2024-03-23T11:47:29.768 INFO:tasks.workunit.client.0.smithi063.stdout:8/468: dwrite d6/dd/df/f60 [0,4194304] 0 2024-03-23T11:47:29.775 INFO:tasks.workunit.client.0.smithi063.stdout:8/469: write d6/d1f/d54/d63/d90/f28 [822195,120151] 0 2024-03-23T11:47:29.775 INFO:tasks.workunit.client.1.smithi111.stdout:4/269: rmdir d0/d7/da 39 2024-03-23T11:47:29.778 INFO:tasks.workunit.client.1.smithi111.stdout:7/247: write d14/f36 [1240090,55858] 0 2024-03-23T11:47:29.778 INFO:tasks.workunit.client.1.smithi111.stdout:4/270: chown d0/d7/c10 12293406 1 2024-03-23T11:47:29.779 INFO:tasks.workunit.client.1.smithi111.stdout:5/318: mkdir d1/d5d/d59/d6c 0 2024-03-23T11:47:29.780 INFO:tasks.workunit.client.1.smithi111.stdout:4/271: chown d0/d7/f1b 152094 1 2024-03-23T11:47:29.782 INFO:tasks.workunit.client.1.smithi111.stdout:2/319: creat d2/d11/d17/d3e/f72 x:0 0 0 2024-03-23T11:47:29.784 INFO:tasks.workunit.client.1.smithi111.stdout:0/271: readlink d5/db/d1f/d19/l1b 0 2024-03-23T11:47:29.785 INFO:tasks.workunit.client.0.smithi063.stdout:9/240: dread f2 [4194304,4194304] 0 2024-03-23T11:47:29.785 INFO:tasks.workunit.client.1.smithi111.stdout:4/272: write d0/dc/f55 [347537,91406] 0 2024-03-23T11:47:29.787 INFO:tasks.workunit.client.1.smithi111.stdout:5/319: dread d1/d3/d6/f57 [0,4194304] 0 2024-03-23T11:47:29.787 INFO:tasks.workunit.client.0.smithi063.stdout:8/470: truncate d6/d1f/d54/d63/d90/f13 52098 0 2024-03-23T11:47:29.788 INFO:tasks.workunit.client.0.smithi063.stdout:9/241: symlink d4/d4c/d44/l54 0 2024-03-23T11:47:29.788 INFO:tasks.workunit.client.1.smithi111.stdout:0/272: mknod d5/db/d4d/c54 0 2024-03-23T11:47:29.789 INFO:tasks.workunit.client.1.smithi111.stdout:5/320: symlink d1/d3/d3a/l6d 0 2024-03-23T11:47:29.791 INFO:tasks.workunit.client.0.smithi063.stdout:9/242: symlink d4/d4c/d19/l55 0 2024-03-23T11:47:29.791 INFO:tasks.workunit.client.1.smithi111.stdout:2/320: link d2/d7/da/d19/f6a d2/d32/d4b/d52/f73 0 2024-03-23T11:47:29.792 INFO:tasks.workunit.client.0.smithi063.stdout:8/471: mkdir d6/d1f/d54/d63/d90/d15/d9d 0 2024-03-23T11:47:29.792 INFO:tasks.workunit.client.0.smithi063.stdout:9/243: chown d4/d31/l4e 1209791 1 2024-03-23T11:47:29.793 INFO:tasks.workunit.client.1.smithi111.stdout:5/321: symlink d1/d3/d6/d7/l6e 0 2024-03-23T11:47:29.794 INFO:tasks.workunit.client.1.smithi111.stdout:2/321: rmdir d2/d11/d17 39 2024-03-23T11:47:29.795 INFO:tasks.workunit.client.0.smithi063.stdout:4/222: sync 2024-03-23T11:47:29.795 INFO:tasks.workunit.client.0.smithi063.stdout:9/244: chown d4/f21 174 1 2024-03-23T11:47:29.796 INFO:tasks.workunit.client.0.smithi063.stdout:8/472: unlink d6/d1c/d32/f93 0 2024-03-23T11:47:29.796 INFO:tasks.workunit.client.1.smithi111.stdout:2/322: chown d2/d32/d4b/d52/l62 2888 1 2024-03-23T11:47:29.800 INFO:tasks.workunit.client.1.smithi111.stdout:2/323: creat d2/f74 x:0 0 0 2024-03-23T11:47:29.803 INFO:tasks.workunit.client.1.smithi111.stdout:2/324: mknod d2/d32/d4b/d52/c75 0 2024-03-23T11:47:29.803 INFO:tasks.workunit.client.0.smithi063.stdout:8/473: mkdir d6/d1f/d54/d63/d90/d9e 0 2024-03-23T11:47:29.804 INFO:tasks.workunit.client.1.smithi111.stdout:1/268: dread d0/d5/dd/f10 [4194304,4194304] 0 2024-03-23T11:47:29.804 INFO:tasks.workunit.client.0.smithi063.stdout:8/474: dread - d6/f98 zero size 2024-03-23T11:47:29.806 INFO:tasks.workunit.client.0.smithi063.stdout:9/245: rename d4/d4c/c32 to d4/c56 0 2024-03-23T11:47:29.809 INFO:tasks.workunit.client.1.smithi111.stdout:5/322: dwrite d1/d3/d6/d29/f4b [0,4194304] 0 2024-03-23T11:47:29.810 INFO:tasks.workunit.client.1.smithi111.stdout:2/325: mkdir d2/d11/d17/d3e/d6b/d2e/d76 0 2024-03-23T11:47:29.811 INFO:tasks.workunit.client.1.smithi111.stdout:5/323: fdatasync d1/d3/d6/d29/f22 0 2024-03-23T11:47:29.812 INFO:tasks.workunit.client.1.smithi111.stdout:5/324: chown d1 43463689 1 2024-03-23T11:47:29.816 INFO:tasks.workunit.client.1.smithi111.stdout:5/325: mkdir d1/d3/d6/d30/d6f 0 2024-03-23T11:47:29.817 INFO:tasks.workunit.client.1.smithi111.stdout:5/326: write d1/d20/f37 [479118,106664] 0 2024-03-23T11:47:29.817 INFO:tasks.workunit.client.1.smithi111.stdout:2/326: creat d2/d7/da/d19/f77 x:0 0 0 2024-03-23T11:47:29.823 INFO:tasks.workunit.client.1.smithi111.stdout:5/327: rename d1/d3/d6/d29/f62 to d1/d3/d6/d30/f70 0 2024-03-23T11:47:29.825 INFO:tasks.workunit.client.0.smithi063.stdout:4/223: dwrite d5/da/d20/d24/f2d [0,4194304] 0 2024-03-23T11:47:29.825 INFO:tasks.workunit.client.1.smithi111.stdout:5/328: dread - d1/d3/d6/d30/f70 zero size 2024-03-23T11:47:29.826 INFO:tasks.workunit.client.1.smithi111.stdout:2/327: link d2/d11/d17/d3e/f72 d2/d11/f78 0 2024-03-23T11:47:29.829 INFO:tasks.workunit.client.0.smithi063.stdout:4/224: chown d5/da/d20/d24/d16/d32/c36 74 1 2024-03-23T11:47:29.831 INFO:tasks.workunit.client.0.smithi063.stdout:8/475: dread d6/dd/df/f61 [0,4194304] 0 2024-03-23T11:47:29.836 INFO:tasks.workunit.client.0.smithi063.stdout:4/225: getdents d5/da/d20/d24/d16 0 2024-03-23T11:47:29.836 INFO:tasks.workunit.client.1.smithi111.stdout:2/328: getdents d2/d11/d17/d1f 0 2024-03-23T11:47:29.838 INFO:tasks.workunit.client.1.smithi111.stdout:2/329: truncate d2/d7/f56 570647 0 2024-03-23T11:47:29.839 INFO:tasks.workunit.client.0.smithi063.stdout:4/226: truncate d5/da/d20/d24/d1f/f39 45327 0 2024-03-23T11:47:29.840 INFO:tasks.workunit.client.0.smithi063.stdout:4/227: dread - d5/da/d20/d24/d1f/f3f zero size 2024-03-23T11:47:29.842 INFO:tasks.workunit.client.0.smithi063.stdout:4/228: write d5/da/f19 [817263,49181] 0 2024-03-23T11:47:29.845 INFO:tasks.workunit.client.1.smithi111.stdout:2/330: getdents d2/d7/da 0 2024-03-23T11:47:29.848 INFO:tasks.workunit.client.1.smithi111.stdout:5/329: dwrite d1/d3/d6/d7/f53 [0,4194304] 0 2024-03-23T11:47:29.848 INFO:tasks.workunit.client.0.smithi063.stdout:4/229: link d5/da/d20/d24/d16/d32/c33 d5/da/d20/d24/d29/c40 0 2024-03-23T11:47:29.849 INFO:tasks.workunit.client.1.smithi111.stdout:1/269: dread d0/d6/fc [0,4194304] 0 2024-03-23T11:47:29.850 INFO:tasks.workunit.client.1.smithi111.stdout:5/330: stat d1/d3/d6/d7/f1f 0 2024-03-23T11:47:29.859 INFO:tasks.workunit.client.1.smithi111.stdout:9/348: sync 2024-03-23T11:47:29.860 INFO:tasks.workunit.client.1.smithi111.stdout:9/349: write d0/d4/d6a/f72 [236885,14751] 0 2024-03-23T11:47:29.863 INFO:tasks.workunit.client.1.smithi111.stdout:1/270: dread d0/d6/fc [0,4194304] 0 2024-03-23T11:47:29.865 INFO:tasks.workunit.client.1.smithi111.stdout:9/350: mkdir d0/d4/d1c/d2d/d75/d7a 0 2024-03-23T11:47:29.866 INFO:tasks.workunit.client.1.smithi111.stdout:1/271: write d0/d6/d48/f52 [2217812,21350] 0 2024-03-23T11:47:29.866 INFO:tasks.workunit.client.0.smithi063.stdout:4/230: fdatasync d5/da/d20/d24/f2d 0 2024-03-23T11:47:29.867 INFO:tasks.workunit.client.1.smithi111.stdout:5/331: dwrite d1/d5d/d59/f67 [0,4194304] 0 2024-03-23T11:47:29.869 INFO:tasks.workunit.client.1.smithi111.stdout:5/332: fsync d1/d3/fc 0 2024-03-23T11:47:29.872 INFO:tasks.workunit.client.1.smithi111.stdout:9/351: getdents d0/d4 0 2024-03-23T11:47:29.873 INFO:tasks.workunit.client.0.smithi063.stdout:8/476: dread d6/dd/f47 [0,4194304] 0 2024-03-23T11:47:29.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:29 smithi063 ceph-mon[35189]: pgmap v34: 65 pgs: 65 active+clean; 1.0 GiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 9.2 MiB/s rd, 125 MiB/s wr, 352 op/s 2024-03-23T11:47:29.891 INFO:tasks.workunit.client.0.smithi063.stdout:4/231: dwrite d5/da/f2a [0,4194304] 0 2024-03-23T11:47:29.891 INFO:tasks.workunit.client.0.smithi063.stdout:8/477: dwrite d6/d1c/d27/f34 [0,4194304] 0 2024-03-23T11:47:29.893 INFO:tasks.workunit.client.1.smithi111.stdout:1/272: creat d0/d5/f5e x:0 0 0 2024-03-23T11:47:29.894 INFO:tasks.workunit.client.1.smithi111.stdout:9/352: write d0/d4/f14 [313398,7038] 0 2024-03-23T11:47:29.894 INFO:tasks.workunit.client.0.smithi063.stdout:4/232: creat d5/da/d20/d24/d1f/f41 x:0 0 0 2024-03-23T11:47:29.895 INFO:tasks.workunit.client.1.smithi111.stdout:1/273: readlink d0/d5/dd/d1e/l1f 0 2024-03-23T11:47:29.899 INFO:tasks.workunit.client.0.smithi063.stdout:4/233: creat d5/da/d20/d24/f42 x:0 0 0 2024-03-23T11:47:29.900 INFO:tasks.workunit.client.1.smithi111.stdout:9/353: fdatasync d0/d31/d41/d65/f6f 0 2024-03-23T11:47:29.900 INFO:tasks.workunit.client.1.smithi111.stdout:1/274: symlink d0/d5/dd/d1e/d54/l5f 0 2024-03-23T11:47:29.901 INFO:tasks.workunit.client.1.smithi111.stdout:1/275: chown d0/d2b 636 1 2024-03-23T11:47:29.902 INFO:tasks.workunit.client.0.smithi063.stdout:4/234: creat d5/da/d20/d24/d1f/f43 x:0 0 0 2024-03-23T11:47:29.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:29 smithi111 ceph-mon[44005]: pgmap v34: 65 pgs: 65 active+clean; 1.0 GiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 9.2 MiB/s rd, 125 MiB/s wr, 352 op/s 2024-03-23T11:47:29.904 INFO:tasks.workunit.client.0.smithi063.stdout:4/235: fdatasync d5/d1e/d2b/f35 0 2024-03-23T11:47:29.907 INFO:tasks.workunit.client.0.smithi063.stdout:4/236: symlink d5/l44 0 2024-03-23T11:47:29.908 INFO:tasks.workunit.client.0.smithi063.stdout:8/478: write d6/d1f/d54/d63/d90/f35 [1209526,110311] 0 2024-03-23T11:47:29.910 INFO:tasks.workunit.client.0.smithi063.stdout:4/237: write d5/da/d20/d24/d1f/f3f [264658,55323] 0 2024-03-23T11:47:29.913 INFO:tasks.workunit.client.0.smithi063.stdout:4/238: creat d5/d1e/d2b/f45 x:0 0 0 2024-03-23T11:47:29.914 INFO:tasks.workunit.client.0.smithi063.stdout:8/479: rename d6/d1c/d27/d3f/l96 to d6/d1f/d54/d5c/d76/d85/l9f 0 2024-03-23T11:47:29.916 INFO:tasks.workunit.client.0.smithi063.stdout:4/239: write d5/f1c [4690813,39132] 0 2024-03-23T11:47:29.917 INFO:tasks.workunit.client.0.smithi063.stdout:8/480: creat d6/dd/d59/fa0 x:0 0 0 2024-03-23T11:47:29.918 INFO:tasks.workunit.client.0.smithi063.stdout:4/240: write d5/d1e/d2b/f45 [281877,55375] 0 2024-03-23T11:47:29.919 INFO:tasks.workunit.client.0.smithi063.stdout:8/481: rmdir d6/d30 39 2024-03-23T11:47:29.921 INFO:tasks.workunit.client.0.smithi063.stdout:8/482: read d6/d1c/f1d [4851618,14756] 0 2024-03-23T11:47:29.924 INFO:tasks.workunit.client.0.smithi063.stdout:8/483: symlink d6/d1f/d3e/la1 0 2024-03-23T11:47:29.927 INFO:tasks.workunit.client.0.smithi063.stdout:8/484: mknod d6/d1f/d54/d63/d90/d5d/d7d/ca2 0 2024-03-23T11:47:29.927 INFO:tasks.workunit.client.1.smithi111.stdout:6/347: sync 2024-03-23T11:47:29.927 INFO:tasks.workunit.client.1.smithi111.stdout:8/223: sync 2024-03-23T11:47:29.927 INFO:tasks.workunit.client.1.smithi111.stdout:3/240: sync 2024-03-23T11:47:29.930 INFO:tasks.workunit.client.0.smithi063.stdout:8/485: creat d6/d1f/d54/d63/d90/d5d/fa3 x:0 0 0 2024-03-23T11:47:29.932 INFO:tasks.workunit.client.1.smithi111.stdout:1/276: dread d0/d2b/f5c [0,4194304] 0 2024-03-23T11:47:29.933 INFO:tasks.workunit.client.1.smithi111.stdout:8/224: write f3 [3911452,87527] 0 2024-03-23T11:47:29.935 INFO:tasks.workunit.client.1.smithi111.stdout:3/241: fsync d0/d7/f12 0 2024-03-23T11:47:29.935 INFO:tasks.workunit.client.0.smithi063.stdout:8/486: unlink d6/dd/d59/d7f/f8b 0 2024-03-23T11:47:29.935 INFO:tasks.workunit.client.1.smithi111.stdout:8/225: chown d5/d1b/c25 1 1 2024-03-23T11:47:29.936 INFO:tasks.workunit.client.1.smithi111.stdout:6/348: mknod d2/d1a/c79 0 2024-03-23T11:47:29.937 INFO:tasks.workunit.client.0.smithi063.stdout:4/241: dwrite d5/da/d20/d24/f31 [0,4194304] 0 2024-03-23T11:47:29.938 INFO:tasks.workunit.client.1.smithi111.stdout:6/349: fsync d2/d1f/d45/f76 0 2024-03-23T11:47:29.939 INFO:tasks.workunit.client.0.smithi063.stdout:4/242: fdatasync d5/da/f2a 0 2024-03-23T11:47:29.940 INFO:tasks.workunit.client.1.smithi111.stdout:9/354: dwrite d0/d4/d1c/d36/d28/f5e [0,4194304] 0 2024-03-23T11:47:29.940 INFO:tasks.workunit.client.0.smithi063.stdout:8/487: mkdir d6/d1f/d54/d5c/d66/da4 0 2024-03-23T11:47:29.946 INFO:tasks.workunit.client.1.smithi111.stdout:3/242: mkdir d0/d1a/d2d/d41 0 2024-03-23T11:47:29.948 INFO:tasks.workunit.client.1.smithi111.stdout:9/355: creat d0/d4/d10/d29/d5b/f7b x:0 0 0 2024-03-23T11:47:29.956 INFO:tasks.workunit.client.1.smithi111.stdout:3/243: symlink d0/d7/d19/l42 0 2024-03-23T11:47:29.957 INFO:tasks.workunit.client.1.smithi111.stdout:9/356: rmdir d0/d4/d1c/d2d 39 2024-03-23T11:47:29.962 INFO:tasks.workunit.client.1.smithi111.stdout:1/277: dwrite d0/d2b/d37/f4c [0,4194304] 0 2024-03-23T11:47:29.966 INFO:tasks.workunit.client.1.smithi111.stdout:9/357: mknod d0/d31/d41/d44/c7c 0 2024-03-23T11:47:29.968 INFO:tasks.workunit.client.1.smithi111.stdout:8/226: dwrite d5/d26/d31/f39 [0,4194304] 0 2024-03-23T11:47:29.969 INFO:tasks.workunit.client.1.smithi111.stdout:3/244: creat d0/d1a/d1d/f43 x:0 0 0 2024-03-23T11:47:29.970 INFO:tasks.workunit.client.0.smithi063.stdout:4/243: write d5/da/d20/d24/d1f/f26 [1324441,64643] 0 2024-03-23T11:47:29.970 INFO:tasks.workunit.client.1.smithi111.stdout:6/350: dwrite d2/d1a/f2d [4194304,4194304] 0 2024-03-23T11:47:29.974 INFO:tasks.workunit.client.0.smithi063.stdout:4/244: write d5/da/f2a [4739790,25385] 0 2024-03-23T11:47:29.976 INFO:tasks.workunit.client.1.smithi111.stdout:3/245: fdatasync d0/f16 0 2024-03-23T11:47:29.977 INFO:tasks.workunit.client.1.smithi111.stdout:1/278: creat d0/d5/dd/d1e/d54/f60 x:0 0 0 2024-03-23T11:47:29.979 INFO:tasks.workunit.client.0.smithi063.stdout:4/245: fsync d5/da/f19 0 2024-03-23T11:47:29.979 INFO:tasks.workunit.client.1.smithi111.stdout:3/246: truncate d0/f1f 734689 0 2024-03-23T11:47:29.981 INFO:tasks.workunit.client.1.smithi111.stdout:6/351: dread - d2/f3b zero size 2024-03-23T11:47:29.981 INFO:tasks.workunit.client.1.smithi111.stdout:4/273: sync 2024-03-23T11:47:29.982 INFO:tasks.workunit.client.1.smithi111.stdout:7/248: sync 2024-03-23T11:47:29.985 INFO:tasks.workunit.client.1.smithi111.stdout:6/352: dread - d2/d1a/d1c/d2c/f48 zero size 2024-03-23T11:47:29.985 INFO:tasks.workunit.client.0.smithi063.stdout:8/488: dwrite d6/d1c/f2d [4194304,4194304] 0 2024-03-23T11:47:29.987 INFO:tasks.workunit.client.1.smithi111.stdout:8/227: rename d5/cf to d5/c45 0 2024-03-23T11:47:29.988 INFO:tasks.workunit.client.1.smithi111.stdout:1/279: truncate d0/d5/dd/d1e/f31 4701667 0 2024-03-23T11:47:29.988 INFO:tasks.workunit.client.1.smithi111.stdout:7/249: truncate f9 2873030 0 2024-03-23T11:47:29.993 INFO:tasks.workunit.client.1.smithi111.stdout:8/228: mkdir d5/d32/d35/d46 0 2024-03-23T11:47:29.994 INFO:tasks.workunit.client.1.smithi111.stdout:7/250: creat d14/d1b/d20/f41 x:0 0 0 2024-03-23T11:47:29.995 INFO:tasks.workunit.client.0.smithi063.stdout:8/489: rename d6/dd/f47 to d6/d1f/d54/d63/d90/d15/d9d/fa5 0 2024-03-23T11:47:29.996 INFO:tasks.workunit.client.1.smithi111.stdout:6/353: creat d2/d1f/f7a x:0 0 0 2024-03-23T11:47:30.000 INFO:tasks.workunit.client.1.smithi111.stdout:6/354: chown d2/d1f/d45/f62 414934353 1 2024-03-23T11:47:30.000 INFO:tasks.workunit.client.1.smithi111.stdout:1/280: symlink d0/d6/d48/d4d/l61 0 2024-03-23T11:47:30.001 INFO:tasks.workunit.client.1.smithi111.stdout:0/273: sync 2024-03-23T11:47:30.001 INFO:tasks.workunit.client.1.smithi111.stdout:5/333: sync 2024-03-23T11:47:30.001 INFO:tasks.workunit.client.1.smithi111.stdout:2/331: sync 2024-03-23T11:47:30.002 INFO:tasks.workunit.client.0.smithi063.stdout:8/490: chown d6/d1f/d54/d63/d90/f13 83 1 2024-03-23T11:47:30.004 INFO:tasks.workunit.client.0.smithi063.stdout:4/246: dwrite d5/da/d20/d24/f42 [0,4194304] 0 2024-03-23T11:47:30.004 INFO:tasks.workunit.client.1.smithi111.stdout:4/274: getdents d0/dc/d4c/d4b/d4d 0 2024-03-23T11:47:30.004 INFO:tasks.workunit.client.1.smithi111.stdout:3/247: dwrite d0/d1a/f1c [0,4194304] 0 2024-03-23T11:47:30.006 INFO:tasks.workunit.client.1.smithi111.stdout:1/281: read d0/d6/f22 [2014699,72190] 0 2024-03-23T11:47:30.006 INFO:tasks.workunit.client.0.smithi063.stdout:8/491: dread - d6/d1f/d54/d5c/d66/f72 zero size 2024-03-23T11:47:30.008 INFO:tasks.workunit.client.1.smithi111.stdout:5/334: chown d1/d3/d6/d7/d63/d2c/d43 1440405 1 2024-03-23T11:47:30.009 INFO:tasks.workunit.client.0.smithi063.stdout:4/247: write d5/da/d20/d24/d1f/f41 [630264,17680] 0 2024-03-23T11:47:30.010 INFO:tasks.workunit.client.1.smithi111.stdout:4/275: chown d0/dc/l33 2608 1 2024-03-23T11:47:30.010 INFO:tasks.workunit.client.0.smithi063.stdout:4/248: chown d5 8280404 1 2024-03-23T11:47:30.012 INFO:tasks.workunit.client.1.smithi111.stdout:1/282: chown d0/d5/f5e 3445 1 2024-03-23T11:47:30.013 INFO:tasks.workunit.client.0.smithi063.stdout:8/492: truncate d6/dd/df/f61 3099923 0 2024-03-23T11:47:30.014 INFO:tasks.workunit.client.1.smithi111.stdout:0/274: write d5/f40 [4038510,104422] 0 2024-03-23T11:47:30.015 INFO:tasks.workunit.client.0.smithi063.stdout:4/249: mknod d5/da/c46 0 2024-03-23T11:47:30.020 INFO:tasks.workunit.client.1.smithi111.stdout:3/248: mknod d0/d1a/c44 0 2024-03-23T11:47:30.020 INFO:tasks.workunit.client.0.smithi063.stdout:4/250: chown d5/da/d20/d24/d29/l30 12293 1 2024-03-23T11:47:30.022 INFO:tasks.workunit.client.1.smithi111.stdout:8/229: dread d5/d26/d31/f39 [0,4194304] 0 2024-03-23T11:47:30.023 INFO:tasks.workunit.client.1.smithi111.stdout:7/251: getdents d14 0 2024-03-23T11:47:30.026 INFO:tasks.workunit.client.1.smithi111.stdout:3/249: readlink d0/l6 0 2024-03-23T11:47:30.026 INFO:tasks.workunit.client.1.smithi111.stdout:2/332: link d2/d7/da/ff d2/d11/d17/d3e/d6b/f79 0 2024-03-23T11:47:30.027 INFO:tasks.workunit.client.0.smithi063.stdout:4/251: rename d5/da/d20/d24/d29/d38/l3d to d5/da/d20/d24/d1f/l47 0 2024-03-23T11:47:30.031 INFO:tasks.workunit.client.1.smithi111.stdout:9/358: dwrite d0/d4/d1c/d2d/f2e [0,4194304] 0 2024-03-23T11:47:30.034 INFO:tasks.workunit.client.0.smithi063.stdout:4/252: truncate d5/da/d20/d24/d1f/f21 3882610 0 2024-03-23T11:47:30.034 INFO:tasks.workunit.client.1.smithi111.stdout:8/230: chown d5/d26/d31/d3b/f3d 29972582 1 2024-03-23T11:47:30.036 INFO:tasks.workunit.client.1.smithi111.stdout:0/275: symlink d5/db/d1f/d47/l55 0 2024-03-23T11:47:30.037 INFO:tasks.workunit.client.1.smithi111.stdout:5/335: rename d1/d3/d6/d7/d63/d2c/f2d to d1/d55/d66/f71 0 2024-03-23T11:47:30.040 INFO:tasks.workunit.client.0.smithi063.stdout:4/253: creat d5/da/d20/f48 x:0 0 0 2024-03-23T11:47:30.041 INFO:tasks.workunit.client.1.smithi111.stdout:2/333: symlink d2/d5c/l7a 0 2024-03-23T11:47:30.044 INFO:tasks.workunit.client.0.smithi063.stdout:4/254: creat d5/d1e/d2b/f49 x:0 0 0 2024-03-23T11:47:30.046 INFO:tasks.workunit.client.0.smithi063.stdout:4/255: fsync d5/d1e/d2b/f45 0 2024-03-23T11:47:30.047 INFO:tasks.workunit.client.0.smithi063.stdout:8/493: truncate d6/d1f/d3e/f8c 1554159 0 2024-03-23T11:47:30.048 INFO:tasks.workunit.client.1.smithi111.stdout:8/231: rmdir d5/d26/d31/d3b 39 2024-03-23T11:47:30.049 INFO:tasks.workunit.client.0.smithi063.stdout:8/494: dread - d6/d1c/f7b zero size 2024-03-23T11:47:30.051 INFO:tasks.workunit.client.0.smithi063.stdout:8/495: chown d6/d1f/d54/d5c/d66/da4 467 1 2024-03-23T11:47:30.052 INFO:tasks.workunit.client.1.smithi111.stdout:4/276: write d0/d17/f2c [3348289,61270] 0 2024-03-23T11:47:30.052 INFO:tasks.workunit.client.0.smithi063.stdout:4/256: symlink d5/da/d20/d24/d16/d32/l4a 0 2024-03-23T11:47:30.054 INFO:tasks.workunit.client.0.smithi063.stdout:8/496: creat d6/d1f/d54/fa6 x:0 0 0 2024-03-23T11:47:30.056 INFO:tasks.workunit.client.0.smithi063.stdout:8/497: chown d6/dd/l2f 7628 1 2024-03-23T11:47:30.057 INFO:tasks.workunit.client.1.smithi111.stdout:4/277: dread - d0/dc/d1f/f4a zero size 2024-03-23T11:47:30.057 INFO:tasks.workunit.client.1.smithi111.stdout:7/252: dread d14/f1c [0,4194304] 0 2024-03-23T11:47:30.059 INFO:tasks.workunit.client.1.smithi111.stdout:5/336: creat d1/d20/f72 x:0 0 0 2024-03-23T11:47:30.059 INFO:tasks.workunit.client.1.smithi111.stdout:9/359: link d0/d4/c12 d0/d4/d1c/d2d/d3b/d42/d4f/c7d 0 2024-03-23T11:47:30.060 INFO:tasks.workunit.client.0.smithi063.stdout:8/498: fdatasync d6/d1f/d54/d63/d90/d5d/f87 0 2024-03-23T11:47:30.061 INFO:tasks.workunit.client.1.smithi111.stdout:8/232: mknod d5/d32/d35/d46/c47 0 2024-03-23T11:47:30.062 INFO:tasks.workunit.client.1.smithi111.stdout:7/253: chown d14/d37/l40 9222 1 2024-03-23T11:47:30.062 INFO:tasks.workunit.client.1.smithi111.stdout:6/355: dread d2/f16 [0,4194304] 0 2024-03-23T11:47:30.063 INFO:tasks.workunit.client.1.smithi111.stdout:2/334: link d2/d7/da/d19/f77 d2/d7/f7b 0 2024-03-23T11:47:30.063 INFO:tasks.workunit.client.1.smithi111.stdout:0/276: getdents d5/db/d1f/d19/d45 0 2024-03-23T11:47:30.063 INFO:tasks.workunit.client.1.smithi111.stdout:3/250: dwrite d0/d7/f23 [0,4194304] 0 2024-03-23T11:47:30.064 INFO:tasks.workunit.client.0.smithi063.stdout:8/499: rmdir d6/d1c/d27/d3f 39 2024-03-23T11:47:30.067 INFO:tasks.workunit.client.0.smithi063.stdout:8/500: symlink d6/d1f/d54/la7 0 2024-03-23T11:47:30.071 INFO:tasks.workunit.client.0.smithi063.stdout:4/257: dwrite d5/da/f2a [0,4194304] 0 2024-03-23T11:47:30.071 INFO:tasks.workunit.client.1.smithi111.stdout:1/283: truncate d0/d5/dd/d1e/f31 2513285 0 2024-03-23T11:47:30.071 INFO:tasks.workunit.client.1.smithi111.stdout:4/278: dread d0/d7/da/f46 [0,4194304] 0 2024-03-23T11:47:30.074 INFO:tasks.workunit.client.0.smithi063.stdout:8/501: truncate d6/d1c/d27/f8d 821747 0 2024-03-23T11:47:30.075 INFO:tasks.workunit.client.1.smithi111.stdout:0/277: unlink d5/f8 0 2024-03-23T11:47:30.076 INFO:tasks.workunit.client.1.smithi111.stdout:7/254: mkdir d14/d1b/d1e/d2c/d42 0 2024-03-23T11:47:30.076 INFO:tasks.workunit.client.1.smithi111.stdout:3/251: creat d0/d1a/d2d/f45 x:0 0 0 2024-03-23T11:47:30.079 INFO:tasks.workunit.client.1.smithi111.stdout:6/356: mknod d2/d1a/d1c/d2c/c7b 0 2024-03-23T11:47:30.083 INFO:tasks.workunit.client.1.smithi111.stdout:1/284: symlink d0/d5/dd/d1e/d54/l62 0 2024-03-23T11:47:30.085 INFO:tasks.workunit.client.1.smithi111.stdout:0/278: chown d5/db/d1f/d32/f3e 37697451 1 2024-03-23T11:47:30.087 INFO:tasks.workunit.client.1.smithi111.stdout:2/335: dread d2/d11/d17/f21 [0,4194304] 0 2024-03-23T11:47:30.087 INFO:tasks.workunit.client.1.smithi111.stdout:5/337: dwrite d1/d20/f37 [0,4194304] 0 2024-03-23T11:47:30.088 INFO:tasks.workunit.client.1.smithi111.stdout:1/285: fsync d0/d2b/d37/f4c 0 2024-03-23T11:47:30.090 INFO:tasks.workunit.client.1.smithi111.stdout:7/255: mkdir d14/d37/d43 0 2024-03-23T11:47:30.090 INFO:tasks.workunit.client.1.smithi111.stdout:8/233: rmdir d5 39 2024-03-23T11:47:30.090 INFO:tasks.workunit.client.1.smithi111.stdout:0/279: chown d5/db/c21 112 1 2024-03-23T11:47:30.094 INFO:tasks.workunit.client.1.smithi111.stdout:6/357: creat d2/d1a/d1c/d2c/f7c x:0 0 0 2024-03-23T11:47:30.095 INFO:tasks.workunit.client.0.smithi063.stdout:8/502: write d6/dd/df/f11 [4068344,55351] 0 2024-03-23T11:47:30.096 INFO:tasks.workunit.client.1.smithi111.stdout:2/336: symlink d2/de/d12/l7c 0 2024-03-23T11:47:30.098 INFO:tasks.workunit.client.1.smithi111.stdout:1/286: chown d0/d5/dd/d1e/f36 7 1 2024-03-23T11:47:30.099 INFO:tasks.workunit.client.1.smithi111.stdout:7/256: fdatasync f12 0 2024-03-23T11:47:30.101 INFO:tasks.workunit.client.1.smithi111.stdout:0/280: rmdir d5/db/d1f 39 2024-03-23T11:47:30.102 INFO:tasks.workunit.client.1.smithi111.stdout:4/279: dwrite d0/dc/d1f/f4a [0,4194304] 0 2024-03-23T11:47:30.104 INFO:tasks.workunit.client.1.smithi111.stdout:6/358: truncate d2/f41 1957625 0 2024-03-23T11:47:30.105 INFO:tasks.workunit.client.0.smithi063.stdout:8/503: creat d6/d1f/d54/d63/d90/d9e/fa8 x:0 0 0 2024-03-23T11:47:30.106 INFO:tasks.workunit.client.1.smithi111.stdout:2/337: chown d2/d7/da/f3b 473285495 1 2024-03-23T11:47:30.106 INFO:tasks.workunit.client.0.smithi063.stdout:4/258: dwrite d5/da/d20/d24/d1f/f21 [0,4194304] 0 2024-03-23T11:47:30.109 INFO:tasks.workunit.client.1.smithi111.stdout:6/359: chown d2/d1a/f5e 1 1 2024-03-23T11:47:30.110 INFO:tasks.workunit.client.0.smithi063.stdout:8/504: write d6/dd/df/f60 [4255422,73598] 0 2024-03-23T11:47:30.111 INFO:tasks.workunit.client.1.smithi111.stdout:2/338: write d2/d7/fd [2897419,441] 0 2024-03-23T11:47:30.113 INFO:tasks.workunit.client.1.smithi111.stdout:7/257: mkdir d14/d44 0 2024-03-23T11:47:30.114 INFO:tasks.workunit.client.0.smithi063.stdout:8/505: write d6/d1f/d54/fa6 [538045,65698] 0 2024-03-23T11:47:30.115 INFO:tasks.workunit.client.1.smithi111.stdout:8/234: read d5/fa [11800,17545] 0 2024-03-23T11:47:30.117 INFO:tasks.workunit.client.0.smithi063.stdout:8/506: fsync d6/d1c/f3c 0 2024-03-23T11:47:30.118 INFO:tasks.workunit.client.1.smithi111.stdout:5/338: creat d1/d3/d6/d29/f73 x:0 0 0 2024-03-23T11:47:30.118 INFO:tasks.workunit.client.0.smithi063.stdout:4/259: getdents d5/da/d20/d24 0 2024-03-23T11:47:30.119 INFO:tasks.workunit.client.0.smithi063.stdout:8/507: fsync d6/dd/df/f18 0 2024-03-23T11:47:30.125 INFO:tasks.workunit.client.1.smithi111.stdout:9/360: dread d0/d4/d1c/f46 [0,4194304] 0 2024-03-23T11:47:30.126 INFO:tasks.workunit.client.0.smithi063.stdout:8/508: symlink d6/d1f/d54/d63/d90/d5d/d7d/la9 0 2024-03-23T11:47:30.128 INFO:tasks.workunit.client.1.smithi111.stdout:0/281: mknod d5/db/d1f/d48/c56 0 2024-03-23T11:47:30.130 INFO:tasks.workunit.client.0.smithi063.stdout:4/260: rename d5/da/d20/d24 to d5/da/d20/d4b 0 2024-03-23T11:47:30.131 INFO:tasks.workunit.client.0.smithi063.stdout:8/509: mkdir d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa 0 2024-03-23T11:47:30.132 INFO:tasks.workunit.client.1.smithi111.stdout:7/258: creat d14/d1b/d1e/d2c/d42/f45 x:0 0 0 2024-03-23T11:47:30.133 INFO:tasks.workunit.client.0.smithi063.stdout:4/261: chown d5/fc 15387 1 2024-03-23T11:47:30.133 INFO:tasks.workunit.client.1.smithi111.stdout:1/287: getdents d0 0 2024-03-23T11:47:30.134 INFO:tasks.workunit.client.0.smithi063.stdout:4/262: chown d5/da/d20/d4b/d1f 33 1 2024-03-23T11:47:30.135 INFO:tasks.workunit.client.0.smithi063.stdout:8/510: fdatasync d6/d1f/d54/d63/d90/d15/d9d/fa5 0 2024-03-23T11:47:30.136 INFO:tasks.workunit.client.0.smithi063.stdout:4/263: chown d5 3 1 2024-03-23T11:47:30.136 INFO:tasks.workunit.client.0.smithi063.stdout:8/511: chown d6/d1f/d3e/l8e 513 1 2024-03-23T11:47:30.138 INFO:tasks.workunit.client.1.smithi111.stdout:4/280: dread d0/d7/f16 [0,4194304] 0 2024-03-23T11:47:30.138 INFO:tasks.workunit.client.1.smithi111.stdout:2/339: truncate d2/d11/d17/d3e/d6b/f79 53874 0 2024-03-23T11:47:30.138 INFO:tasks.workunit.client.1.smithi111.stdout:0/282: write d5/db/d1f/d32/f3e [1033181,25000] 0 2024-03-23T11:47:30.139 INFO:tasks.workunit.client.1.smithi111.stdout:1/288: mkdir d0/d5/dd/d1e/d63 0 2024-03-23T11:47:30.141 INFO:tasks.workunit.client.1.smithi111.stdout:7/259: mknod d14/d37/c46 0 2024-03-23T11:47:30.144 INFO:tasks.workunit.client.1.smithi111.stdout:4/281: fdatasync d0/f45 0 2024-03-23T11:47:30.145 INFO:tasks.workunit.client.1.smithi111.stdout:3/252: dread d0/d1a/f28 [0,4194304] 0 2024-03-23T11:47:30.146 INFO:tasks.workunit.client.1.smithi111.stdout:2/340: unlink d2/d7/c8 0 2024-03-23T11:47:30.147 INFO:tasks.workunit.client.1.smithi111.stdout:9/361: truncate d0/d4/d1c/f26 2421412 0 2024-03-23T11:47:30.149 INFO:tasks.workunit.client.0.smithi063.stdout:8/512: dread d6/f43 [0,4194304] 0 2024-03-23T11:47:30.151 INFO:tasks.workunit.client.1.smithi111.stdout:3/253: chown d0/d1a/c44 1110 1 2024-03-23T11:47:30.151 INFO:tasks.workunit.client.0.smithi063.stdout:8/513: unlink d6/cc 0 2024-03-23T11:47:30.153 INFO:tasks.workunit.client.1.smithi111.stdout:8/235: dwrite d5/f17 [0,4194304] 0 2024-03-23T11:47:30.153 INFO:tasks.workunit.client.1.smithi111.stdout:2/341: chown d2/de/d12/f29 1 1 2024-03-23T11:47:30.155 INFO:tasks.workunit.client.0.smithi063.stdout:8/514: creat d6/d1f/d54/d5c/d66/da4/fab x:0 0 0 2024-03-23T11:47:30.157 INFO:tasks.workunit.client.0.smithi063.stdout:8/515: chown d6/dd/df/l46 304334 1 2024-03-23T11:47:30.158 INFO:tasks.workunit.client.1.smithi111.stdout:5/339: dwrite d1/d3/d6/d29/f54 [0,4194304] 0 2024-03-23T11:47:30.158 INFO:tasks.workunit.client.1.smithi111.stdout:7/260: creat d14/d1b/d1e/f47 x:0 0 0 2024-03-23T11:47:30.161 INFO:tasks.workunit.client.0.smithi063.stdout:2/206: sync 2024-03-23T11:47:30.161 INFO:tasks.workunit.client.0.smithi063.stdout:6/203: sync 2024-03-23T11:47:30.161 INFO:tasks.workunit.client.0.smithi063.stdout:7/257: sync 2024-03-23T11:47:30.162 INFO:tasks.workunit.client.0.smithi063.stdout:8/516: mknod d6/d1c/d27/cac 0 2024-03-23T11:47:30.163 INFO:tasks.workunit.client.1.smithi111.stdout:6/360: dread d2/d35/f40 [0,4194304] 0 2024-03-23T11:47:30.163 INFO:tasks.workunit.client.1.smithi111.stdout:0/283: dread d5/db/d1f/d39/f3f [0,4194304] 0 2024-03-23T11:47:30.165 INFO:tasks.workunit.client.1.smithi111.stdout:1/289: creat d0/d5/dd/d1e/d63/f64 x:0 0 0 2024-03-23T11:47:30.167 INFO:tasks.workunit.client.0.smithi063.stdout:6/204: creat d1/dc/d21/f3c x:0 0 0 2024-03-23T11:47:30.168 INFO:tasks.workunit.client.0.smithi063.stdout:7/258: rmdir d5/d6/d19/d1f/d37/d20 39 2024-03-23T11:47:30.173 INFO:tasks.workunit.client.0.smithi063.stdout:2/207: mkdir d2/d27/d44 0 2024-03-23T11:47:30.173 INFO:tasks.workunit.client.1.smithi111.stdout:5/340: rename d1/d3/d6/d29 to d1/d5d/d59/d74 0 2024-03-23T11:47:30.174 INFO:tasks.workunit.client.0.smithi063.stdout:2/208: stat d2/l8 0 2024-03-23T11:47:30.177 INFO:tasks.workunit.client.0.smithi063.stdout:8/517: getdents d6/d1f/d54/d63 0 2024-03-23T11:47:30.177 INFO:tasks.workunit.client.1.smithi111.stdout:5/341: chown d1/d3/d6/f57 454 1 2024-03-23T11:47:30.180 INFO:tasks.workunit.client.1.smithi111.stdout:7/261: write f12 [5096162,23537] 0 2024-03-23T11:47:30.180 INFO:tasks.workunit.client.1.smithi111.stdout:6/361: mknod d2/d1a/d1c/c7d 0 2024-03-23T11:47:30.181 INFO:tasks.workunit.client.0.smithi063.stdout:8/518: write d6/d1f/d54/d5c/d66/f83 [221212,115978] 0 2024-03-23T11:47:30.183 INFO:tasks.workunit.client.1.smithi111.stdout:4/282: dwrite d0/f45 [0,4194304] 0 2024-03-23T11:47:30.187 INFO:tasks.workunit.client.1.smithi111.stdout:4/283: fdatasync d0/dc/f14 0 2024-03-23T11:47:30.188 INFO:tasks.workunit.client.1.smithi111.stdout:5/342: truncate d1/d3/d3a/f3d 2376838 0 2024-03-23T11:47:30.188 INFO:tasks.workunit.client.1.smithi111.stdout:7/262: read - d14/d1b/d20/f3e zero size 2024-03-23T11:47:30.190 INFO:tasks.workunit.client.1.smithi111.stdout:1/290: getdents d0/d6/d48/d4d 0 2024-03-23T11:47:30.194 INFO:tasks.workunit.client.1.smithi111.stdout:5/343: readlink d1/d3/d3a/l6d 0 2024-03-23T11:47:30.199 INFO:tasks.workunit.client.0.smithi063.stdout:6/205: dwrite d1/f7 [0,4194304] 0 2024-03-23T11:47:30.199 INFO:tasks.workunit.client.1.smithi111.stdout:2/342: dwrite d2/d7/f56 [0,4194304] 0 2024-03-23T11:47:30.200 INFO:tasks.workunit.client.1.smithi111.stdout:4/284: write d0/dc/f14 [1675161,97137] 0 2024-03-23T11:47:30.200 INFO:tasks.workunit.client.1.smithi111.stdout:8/236: dwrite d5/d26/d31/f39 [0,4194304] 0 2024-03-23T11:47:30.201 INFO:tasks.workunit.client.0.smithi063.stdout:2/209: dwrite d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:30.206 INFO:tasks.workunit.client.0.smithi063.stdout:7/259: dwrite d5/d6/d19/d1f/d37/d20/f28 [0,4194304] 0 2024-03-23T11:47:30.213 INFO:tasks.workunit.client.1.smithi111.stdout:1/291: symlink d0/d2b/l65 0 2024-03-23T11:47:30.213 INFO:tasks.workunit.client.1.smithi111.stdout:5/344: creat d1/d5d/f75 x:0 0 0 2024-03-23T11:47:30.214 INFO:tasks.workunit.client.1.smithi111.stdout:4/285: chown d0/dc/d30/l12 929775 1 2024-03-23T11:47:30.214 INFO:tasks.workunit.client.1.smithi111.stdout:0/284: dwrite d5/db/d1f/f1a [0,4194304] 0 2024-03-23T11:47:30.214 INFO:tasks.workunit.client.1.smithi111.stdout:1/292: chown d0/c28 1130165881 1 2024-03-23T11:47:30.214 INFO:tasks.workunit.client.0.smithi063.stdout:2/210: creat d2/f45 x:0 0 0 2024-03-23T11:47:30.215 INFO:tasks.workunit.client.1.smithi111.stdout:2/343: creat d2/d32/f7d x:0 0 0 2024-03-23T11:47:30.219 INFO:tasks.workunit.client.1.smithi111.stdout:8/237: creat d5/d27/f48 x:0 0 0 2024-03-23T11:47:30.219 INFO:tasks.workunit.client.0.smithi063.stdout:7/260: unlink d5/d26/d32/f49 0 2024-03-23T11:47:30.219 INFO:tasks.workunit.client.1.smithi111.stdout:5/345: chown d1/d3/d3a/f3d 41471 1 2024-03-23T11:47:30.221 INFO:tasks.workunit.client.0.smithi063.stdout:8/519: write d6/d1f/d54/d63/d90/d15/f24 [1082016,47181] 0 2024-03-23T11:47:30.223 INFO:tasks.workunit.client.0.smithi063.stdout:2/211: fsync d2/f2f 0 2024-03-23T11:47:30.225 INFO:tasks.workunit.client.0.smithi063.stdout:8/520: creat d6/dd/d59/fad x:0 0 0 2024-03-23T11:47:30.226 INFO:tasks.workunit.client.0.smithi063.stdout:2/212: write d2/de/ff [3554731,19612] 0 2024-03-23T11:47:30.229 INFO:tasks.workunit.client.0.smithi063.stdout:2/213: dread - d2/f45 zero size 2024-03-23T11:47:30.232 INFO:tasks.workunit.client.1.smithi111.stdout:0/285: unlink f4 0 2024-03-23T11:47:30.232 INFO:tasks.workunit.client.1.smithi111.stdout:7/263: link lf d14/d1b/d1e/d2c/d2e/d33/l48 0 2024-03-23T11:47:30.235 INFO:tasks.workunit.client.1.smithi111.stdout:6/362: dwrite d2/ff [0,4194304] 0 2024-03-23T11:47:30.235 INFO:tasks.workunit.client.1.smithi111.stdout:2/344: write d2/de/d12/f43 [1238295,2627] 0 2024-03-23T11:47:30.237 INFO:tasks.workunit.client.0.smithi063.stdout:2/214: unlink f1 0 2024-03-23T11:47:30.237 INFO:tasks.workunit.client.1.smithi111.stdout:7/264: truncate fb 7157888 0 2024-03-23T11:47:30.239 INFO:tasks.workunit.client.1.smithi111.stdout:5/346: creat d1/d5d/d59/d6c/f76 x:0 0 0 2024-03-23T11:47:30.243 INFO:tasks.workunit.client.1.smithi111.stdout:9/362: dread d0/d4/f15 [0,4194304] 0 2024-03-23T11:47:30.247 INFO:tasks.workunit.client.1.smithi111.stdout:0/286: mknod d5/db/d1f/d48/c57 0 2024-03-23T11:47:30.247 INFO:tasks.workunit.client.1.smithi111.stdout:8/238: link d5/d32/d35/f3f d5/d32/d35/f49 0 2024-03-23T11:47:30.255 INFO:tasks.workunit.client.1.smithi111.stdout:4/286: dread d0/d17/f3d [0,4194304] 0 2024-03-23T11:47:30.255 INFO:tasks.workunit.client.0.smithi063.stdout:2/215: dwrite d2/de/f1b [0,4194304] 0 2024-03-23T11:47:30.255 INFO:tasks.workunit.client.1.smithi111.stdout:2/345: creat d2/de/d12/d70/f7e x:0 0 0 2024-03-23T11:47:30.258 INFO:tasks.workunit.client.0.smithi063.stdout:2/216: mkdir d2/d46 0 2024-03-23T11:47:30.259 INFO:tasks.workunit.client.0.smithi063.stdout:2/217: chown d2/d27 48 1 2024-03-23T11:47:30.259 INFO:tasks.workunit.client.0.smithi063.stdout:2/218: chown d2/de 1031 1 2024-03-23T11:47:30.260 INFO:tasks.workunit.client.1.smithi111.stdout:8/239: mkdir d5/d32/d35/d4a 0 2024-03-23T11:47:30.260 INFO:tasks.workunit.client.1.smithi111.stdout:0/287: write d5/db/d1f/d39/f3f [245145,100300] 0 2024-03-23T11:47:30.264 INFO:tasks.workunit.client.1.smithi111.stdout:2/346: write d2/d11/f51 [666275,9804] 0 2024-03-23T11:47:30.266 INFO:tasks.workunit.client.0.smithi063.stdout:2/219: creat d2/f47 x:0 0 0 2024-03-23T11:47:30.270 INFO:tasks.workunit.client.0.smithi063.stdout:2/220: symlink d2/de/d22/l48 0 2024-03-23T11:47:30.274 INFO:tasks.workunit.client.1.smithi111.stdout:8/240: mkdir d5/d32/d35/d46/d4b 0 2024-03-23T11:47:30.278 INFO:tasks.workunit.client.0.smithi063.stdout:6/206: dread d1/dc/fd [0,4194304] 0 2024-03-23T11:47:30.278 INFO:tasks.workunit.client.1.smithi111.stdout:2/347: chown d2/f5b 28 1 2024-03-23T11:47:30.283 INFO:tasks.workunit.client.1.smithi111.stdout:1/293: dwrite d0/d5/dd/d1e/d54/f60 [0,4194304] 0 2024-03-23T11:47:30.283 INFO:tasks.workunit.client.0.smithi063.stdout:7/261: dread d5/d6/f25 [0,4194304] 0 2024-03-23T11:47:30.286 INFO:tasks.workunit.client.0.smithi063.stdout:6/207: symlink d1/d2a/l3d 0 2024-03-23T11:47:30.289 INFO:tasks.workunit.client.1.smithi111.stdout:0/288: rename d5/db/f27 to d5/db/d1f/d48/f58 0 2024-03-23T11:47:30.289 INFO:tasks.workunit.client.1.smithi111.stdout:7/265: dwrite d14/f25 [0,4194304] 0 2024-03-23T11:47:30.292 INFO:tasks.workunit.client.1.smithi111.stdout:6/363: write d2/d17/f4d [1377708,8916] 0 2024-03-23T11:47:30.292 INFO:tasks.workunit.client.0.smithi063.stdout:7/262: read d5/d6/d19/d1f/d37/f3b [794287,14467] 0 2024-03-23T11:47:30.292 INFO:tasks.workunit.client.1.smithi111.stdout:0/289: fdatasync d5/db/f26 0 2024-03-23T11:47:30.293 INFO:tasks.workunit.client.1.smithi111.stdout:5/347: dwrite d1/d3/d6/d7/f3e [0,4194304] 0 2024-03-23T11:47:30.295 INFO:tasks.workunit.client.1.smithi111.stdout:8/241: symlink d5/d26/d31/d3b/l4c 0 2024-03-23T11:47:30.295 INFO:tasks.workunit.client.0.smithi063.stdout:8/521: dread d6/d1c/f4e [0,4194304] 0 2024-03-23T11:47:30.298 INFO:tasks.workunit.client.0.smithi063.stdout:6/208: rename d1/dc/f24 to d1/dc/d3a/f3e 0 2024-03-23T11:47:30.299 INFO:tasks.workunit.client.1.smithi111.stdout:5/348: write d1/d3/f4 [4368441,28790] 0 2024-03-23T11:47:30.301 INFO:tasks.workunit.client.1.smithi111.stdout:1/294: mknod d0/d6/d48/d4a/c66 0 2024-03-23T11:47:30.303 INFO:tasks.workunit.client.1.smithi111.stdout:9/363: dwrite d0/d4/d10/f51 [0,4194304] 0 2024-03-23T11:47:30.304 INFO:tasks.workunit.client.1.smithi111.stdout:7/266: creat d14/d37/f49 x:0 0 0 2024-03-23T11:47:30.309 INFO:tasks.workunit.client.0.smithi063.stdout:9/246: sync 2024-03-23T11:47:30.309 INFO:tasks.workunit.client.0.smithi063.stdout:1/221: sync 2024-03-23T11:47:30.309 INFO:tasks.workunit.client.0.smithi063.stdout:3/365: sync 2024-03-23T11:47:30.309 INFO:tasks.workunit.client.0.smithi063.stdout:5/148: sync 2024-03-23T11:47:30.309 INFO:tasks.workunit.client.0.smithi063.stdout:0/258: sync 2024-03-23T11:47:30.309 INFO:tasks.workunit.client.1.smithi111.stdout:0/290: mkdir d5/db/d1f/d47/d59 0 2024-03-23T11:47:30.312 INFO:tasks.workunit.client.0.smithi063.stdout:6/209: creat d1/dc/d3a/f3f x:0 0 0 2024-03-23T11:47:30.315 INFO:tasks.workunit.client.1.smithi111.stdout:2/348: write d2/d11/f78 [395682,88435] 0 2024-03-23T11:47:30.315 INFO:tasks.workunit.client.1.smithi111.stdout:1/295: symlink d0/d6/l67 0 2024-03-23T11:47:30.315 INFO:tasks.workunit.client.0.smithi063.stdout:0/259: fsync d2/d4/dd/d1f/f2d 0 2024-03-23T11:47:30.315 INFO:tasks.workunit.client.0.smithi063.stdout:8/522: mkdir d6/dd/d59/d7f/dae 0 2024-03-23T11:47:30.316 INFO:tasks.workunit.client.0.smithi063.stdout:5/149: truncate d2/f4 3285984 0 2024-03-23T11:47:30.317 INFO:tasks.workunit.client.0.smithi063.stdout:3/366: chown d13/d22/c4e 0 1 2024-03-23T11:47:30.321 INFO:tasks.workunit.client.0.smithi063.stdout:6/210: chown d1/d13/c19 6748592 1 2024-03-23T11:47:30.322 INFO:tasks.workunit.client.1.smithi111.stdout:3/254: sync 2024-03-23T11:47:30.323 INFO:tasks.workunit.client.0.smithi063.stdout:1/222: rename d3/d35/l3b to d3/d3f/d4d/l50 0 2024-03-23T11:47:30.324 INFO:tasks.workunit.client.0.smithi063.stdout:5/150: stat d2/d7/d8/de/cf 0 2024-03-23T11:47:30.324 INFO:tasks.workunit.client.1.smithi111.stdout:0/291: chown d5/db/d1f/d19/f1c 58676 1 2024-03-23T11:47:30.325 INFO:tasks.workunit.client.0.smithi063.stdout:9/247: symlink d4/d31/d43/d49/l57 0 2024-03-23T11:47:30.325 INFO:tasks.workunit.client.1.smithi111.stdout:9/364: rename d0/d4/f14 to d0/d4/d1c/d36/f7e 0 2024-03-23T11:47:30.328 INFO:tasks.workunit.client.1.smithi111.stdout:6/364: dread d2/d1a/d1c/d2c/d51/f61 [0,4194304] 0 2024-03-23T11:47:30.329 INFO:tasks.workunit.client.0.smithi063.stdout:2/221: dread d2/d23/d2e/f36 [0,4194304] 0 2024-03-23T11:47:30.330 INFO:tasks.workunit.client.0.smithi063.stdout:5/151: symlink d2/d2f/l32 0 2024-03-23T11:47:30.332 INFO:tasks.workunit.client.1.smithi111.stdout:1/296: dread d0/f3d [0,4194304] 0 2024-03-23T11:47:30.333 INFO:tasks.workunit.client.0.smithi063.stdout:5/152: chown d2/d7/d8/d1d/f1f 483735 1 2024-03-23T11:47:30.335 INFO:tasks.workunit.client.0.smithi063.stdout:5/153: write d2/f25 [91396,62026] 0 2024-03-23T11:47:30.336 INFO:tasks.workunit.client.1.smithi111.stdout:8/242: dwrite f3 [0,4194304] 0 2024-03-23T11:47:30.336 INFO:tasks.workunit.client.0.smithi063.stdout:1/223: mkdir d3/d51 0 2024-03-23T11:47:30.340 INFO:tasks.workunit.client.1.smithi111.stdout:5/349: dwrite d1/f49 [0,4194304] 0 2024-03-23T11:47:30.340 INFO:tasks.workunit.client.1.smithi111.stdout:3/255: creat d0/d1a/d1d/d26/f46 x:0 0 0 2024-03-23T11:47:30.340 INFO:tasks.workunit.client.0.smithi063.stdout:6/211: creat d1/dc/d3a/f40 x:0 0 0 2024-03-23T11:47:30.341 INFO:tasks.workunit.client.0.smithi063.stdout:3/367: truncate d13/d22/d2d/f38 1810955 0 2024-03-23T11:47:30.344 INFO:tasks.workunit.client.0.smithi063.stdout:8/523: dread d6/d1c/f4e [0,4194304] 0 2024-03-23T11:47:30.346 INFO:tasks.workunit.client.0.smithi063.stdout:6/212: write d1/d13/f16 [428881,103739] 0 2024-03-23T11:47:30.346 INFO:tasks.workunit.client.1.smithi111.stdout:1/297: write d0/f20 [3828434,129205] 0 2024-03-23T11:47:30.347 INFO:tasks.workunit.client.1.smithi111.stdout:8/243: write d5/f1f [650155,19886] 0 2024-03-23T11:47:30.354 INFO:tasks.workunit.client.1.smithi111.stdout:3/256: fsync d0/d1a/f28 0 2024-03-23T11:47:30.354 INFO:tasks.workunit.client.1.smithi111.stdout:7/267: dwrite d14/d1b/d1e/d2c/d42/f45 [0,4194304] 0 2024-03-23T11:47:30.354 INFO:tasks.workunit.client.1.smithi111.stdout:1/298: readlink d0/d5/dd/d1e/d54/l5f 0 2024-03-23T11:47:30.354 INFO:tasks.workunit.client.0.smithi063.stdout:0/260: write d2/d12/f42 [1048314,93637] 0 2024-03-23T11:47:30.357 INFO:tasks.workunit.client.0.smithi063.stdout:2/222: mknod d2/d27/d44/c49 0 2024-03-23T11:47:30.358 INFO:tasks.workunit.client.1.smithi111.stdout:8/244: rmdir d5/d32/d35 39 2024-03-23T11:47:30.358 INFO:tasks.workunit.client.0.smithi063.stdout:0/261: read - d2/d12/f57 zero size 2024-03-23T11:47:30.359 INFO:tasks.workunit.client.0.smithi063.stdout:9/248: dwrite d4/f47 [0,4194304] 0 2024-03-23T11:47:30.361 INFO:tasks.workunit.client.0.smithi063.stdout:7/263: dwrite d5/d6/d19/d1f/d37/d20/f31 [0,4194304] 0 2024-03-23T11:47:30.361 INFO:tasks.workunit.client.0.smithi063.stdout:3/368: creat d13/d22/d2d/d3f/d58/f70 x:0 0 0 2024-03-23T11:47:30.368 INFO:tasks.workunit.client.1.smithi111.stdout:1/299: symlink d0/d5/dd/d1e/d54/l68 0 2024-03-23T11:47:30.369 INFO:tasks.workunit.client.1.smithi111.stdout:6/365: dwrite d2/d35/f77 [0,4194304] 0 2024-03-23T11:47:30.373 INFO:tasks.workunit.client.0.smithi063.stdout:0/262: dread d2/d14/f31 [0,4194304] 0 2024-03-23T11:47:30.373 INFO:tasks.workunit.client.0.smithi063.stdout:8/524: mkdir d6/dd/d59/daf 0 2024-03-23T11:47:30.373 INFO:tasks.workunit.client.0.smithi063.stdout:8/525: chown d6/d1f/d54/d5c/d76 2582097 1 2024-03-23T11:47:30.373 INFO:tasks.workunit.client.0.smithi063.stdout:9/249: creat d4/d2c/f58 x:0 0 0 2024-03-23T11:47:30.374 INFO:tasks.workunit.client.0.smithi063.stdout:6/213: write d1/dc/d3a/f3e [875732,52032] 0 2024-03-23T11:47:30.376 INFO:tasks.workunit.client.0.smithi063.stdout:3/369: readlink d13/l2e 0 2024-03-23T11:47:30.376 INFO:tasks.workunit.client.0.smithi063.stdout:7/264: symlink d5/d16/l56 0 2024-03-23T11:47:30.376 INFO:tasks.workunit.client.0.smithi063.stdout:4/264: sync 2024-03-23T11:47:30.380 INFO:tasks.workunit.client.0.smithi063.stdout:1/224: truncate d3/f2a 158129 0 2024-03-23T11:47:30.384 INFO:tasks.workunit.client.0.smithi063.stdout:9/250: creat d4/d31/f59 x:0 0 0 2024-03-23T11:47:30.385 INFO:tasks.workunit.client.1.smithi111.stdout:7/268: rename d14/d1b/d20/f3e to d14/d1b/d1e/d2c/d42/f4a 0 2024-03-23T11:47:30.387 INFO:tasks.workunit.client.0.smithi063.stdout:9/251: chown d4/f1e 1 1 2024-03-23T11:47:30.390 INFO:tasks.workunit.client.0.smithi063.stdout:4/265: symlink d5/da/l4c 0 2024-03-23T11:47:30.391 INFO:tasks.workunit.client.0.smithi063.stdout:8/526: rmdir d6/d1c/d27 39 2024-03-23T11:47:30.393 INFO:tasks.workunit.client.1.smithi111.stdout:4/287: sync 2024-03-23T11:47:30.393 INFO:tasks.workunit.client.1.smithi111.stdout:6/366: symlink d2/d1f/d45/d49/l7e 0 2024-03-23T11:47:30.394 INFO:tasks.workunit.client.1.smithi111.stdout:8/245: mkdir d5/d26/d31/d4d 0 2024-03-23T11:47:30.396 INFO:tasks.workunit.client.0.smithi063.stdout:4/266: dread - d5/d1e/d2b/f49 zero size 2024-03-23T11:47:30.396 INFO:tasks.workunit.client.0.smithi063.stdout:5/154: getdents d2/d7/d8/de 0 2024-03-23T11:47:30.401 INFO:tasks.workunit.client.1.smithi111.stdout:2/349: truncate d2/d11/d17/d1f/d30/f35 2279260 0 2024-03-23T11:47:30.401 INFO:tasks.workunit.client.1.smithi111.stdout:3/257: creat d0/d7/f47 x:0 0 0 2024-03-23T11:47:30.402 INFO:tasks.workunit.client.0.smithi063.stdout:5/155: write d2/d7/d8/f13 [423016,33792] 0 2024-03-23T11:47:30.403 INFO:tasks.workunit.client.1.smithi111.stdout:3/258: fsync d0/d1a/f28 0 2024-03-23T11:47:30.404 INFO:tasks.workunit.client.0.smithi063.stdout:2/223: getdents d2/d23 0 2024-03-23T11:47:30.405 INFO:tasks.workunit.client.0.smithi063.stdout:5/156: creat d2/f33 x:0 0 0 2024-03-23T11:47:30.405 INFO:tasks.workunit.client.1.smithi111.stdout:0/292: truncate d5/f30 1033027 0 2024-03-23T11:47:30.406 INFO:tasks.workunit.client.1.smithi111.stdout:1/300: mkdir d0/d2b/d69 0 2024-03-23T11:47:30.407 INFO:tasks.workunit.client.0.smithi063.stdout:4/267: rename d5/da/d20/d4b/d1f/l47 to d5/da/l4d 0 2024-03-23T11:47:30.408 INFO:tasks.workunit.client.1.smithi111.stdout:1/301: readlink d0/d5/dd/l39 0 2024-03-23T11:47:30.409 INFO:tasks.workunit.client.0.smithi063.stdout:2/224: fdatasync d2/f18 0 2024-03-23T11:47:30.409 INFO:tasks.workunit.client.1.smithi111.stdout:7/269: mkdir d14/d44/d4b 0 2024-03-23T11:47:30.409 INFO:tasks.workunit.client.1.smithi111.stdout:1/302: chown d0/d6/f19 220926 1 2024-03-23T11:47:30.410 INFO:tasks.workunit.client.1.smithi111.stdout:7/270: readlink d14/l18 0 2024-03-23T11:47:30.410 INFO:tasks.workunit.client.1.smithi111.stdout:5/350: getdents d1 0 2024-03-23T11:47:30.412 INFO:tasks.workunit.client.0.smithi063.stdout:8/527: symlink d6/d1f/d54/d5c/lb0 0 2024-03-23T11:47:30.413 INFO:tasks.workunit.client.1.smithi111.stdout:8/246: creat d5/d26/d31/d3b/d41/f4e x:0 0 0 2024-03-23T11:47:30.414 INFO:tasks.workunit.client.0.smithi063.stdout:5/157: creat d2/d17/f34 x:0 0 0 2024-03-23T11:47:30.414 INFO:tasks.workunit.client.1.smithi111.stdout:9/365: dread d0/d4/d1c/d36/f59 [0,4194304] 0 2024-03-23T11:47:30.415 INFO:tasks.workunit.client.0.smithi063.stdout:4/268: mknod d5/da/d20/d4b/d16/c4e 0 2024-03-23T11:47:30.417 INFO:tasks.workunit.client.1.smithi111.stdout:2/350: rename d2/d11/d17/d1f/l57 to d2/de/d6c/l7f 0 2024-03-23T11:47:30.419 INFO:tasks.workunit.client.1.smithi111.stdout:1/303: mkdir d0/d2b/d37/d6a 0 2024-03-23T11:47:30.421 INFO:tasks.workunit.client.1.smithi111.stdout:5/351: mkdir d1/d55/d66/d77 0 2024-03-23T11:47:30.422 INFO:tasks.workunit.client.1.smithi111.stdout:0/293: fdatasync d5/db/d1f/d19/f23 0 2024-03-23T11:47:30.423 INFO:tasks.workunit.client.1.smithi111.stdout:4/288: dwrite d0/dc/d4c/d2a/f40 [0,4194304] 0 2024-03-23T11:47:30.424 INFO:tasks.workunit.client.1.smithi111.stdout:7/271: readlink d14/d1b/d1e/d2c/d2e/d33/l48 0 2024-03-23T11:47:30.425 INFO:tasks.workunit.client.1.smithi111.stdout:4/289: dread - d0/dc/d4c/f39 zero size 2024-03-23T11:47:30.426 INFO:tasks.workunit.client.0.smithi063.stdout:5/158: mkdir d2/d7/d8/de/d35 0 2024-03-23T11:47:30.427 INFO:tasks.workunit.client.0.smithi063.stdout:4/269: write d5/da/d20/f48 [465016,43168] 0 2024-03-23T11:47:30.427 INFO:tasks.workunit.client.1.smithi111.stdout:9/366: mkdir d0/d4/d10/d29/d7f 0 2024-03-23T11:47:30.428 INFO:tasks.workunit.client.0.smithi063.stdout:1/225: write d3/df/f30 [624415,37562] 0 2024-03-23T11:47:30.428 INFO:tasks.workunit.client.0.smithi063.stdout:9/252: dwrite d4/d2c/f3f [0,4194304] 0 2024-03-23T11:47:30.428 INFO:tasks.workunit.client.0.smithi063.stdout:6/214: truncate d1/dc/d21/f29 3908366 0 2024-03-23T11:47:30.430 INFO:tasks.workunit.client.1.smithi111.stdout:9/367: chown d0/d4/d1c/f45 238145447 1 2024-03-23T11:47:30.438 INFO:tasks.workunit.client.0.smithi063.stdout:7/265: dwrite d5/d6/d19/d1f/d37/f3c [0,4194304] 0 2024-03-23T11:47:30.442 INFO:tasks.workunit.client.0.smithi063.stdout:5/159: write d2/f25 [1017774,111121] 0 2024-03-23T11:47:30.443 INFO:tasks.workunit.client.1.smithi111.stdout:1/304: mknod d0/d6/d48/d4a/c6b 0 2024-03-23T11:47:30.445 INFO:tasks.workunit.client.0.smithi063.stdout:9/253: write d4/d4c/d19/f22 [1052400,118584] 0 2024-03-23T11:47:30.448 INFO:tasks.workunit.client.1.smithi111.stdout:8/247: mknod d5/c4f 0 2024-03-23T11:47:30.448 INFO:tasks.workunit.client.0.smithi063.stdout:4/270: fsync d5/da/d20/d4b/f42 0 2024-03-23T11:47:30.449 INFO:tasks.workunit.client.0.smithi063.stdout:8/528: creat d6/dd/d59/d7f/dae/fb1 x:0 0 0 2024-03-23T11:47:30.451 INFO:tasks.workunit.client.1.smithi111.stdout:8/248: write d5/d27/f36 [1074339,90528] 0 2024-03-23T11:47:30.451 INFO:tasks.workunit.client.0.smithi063.stdout:6/215: mkdir d1/d2a/d41 0 2024-03-23T11:47:30.453 INFO:tasks.workunit.client.1.smithi111.stdout:8/249: write d5/d27/f36 [857196,56475] 0 2024-03-23T11:47:30.455 INFO:tasks.workunit.client.0.smithi063.stdout:7/266: rmdir d5/d6/d19 39 2024-03-23T11:47:30.457 INFO:tasks.workunit.client.0.smithi063.stdout:5/160: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:30.460 INFO:tasks.workunit.client.1.smithi111.stdout:3/259: link d0/d1a/c3f d0/d1a/c48 0 2024-03-23T11:47:30.460 INFO:tasks.workunit.client.1.smithi111.stdout:6/367: truncate d2/d1f/d45/f62 542245 0 2024-03-23T11:47:30.460 INFO:tasks.workunit.client.1.smithi111.stdout:4/290: dread d0/d7/da/f46 [0,4194304] 0 2024-03-23T11:47:30.461 INFO:tasks.workunit.client.0.smithi063.stdout:0/263: dwrite d2/d4/f30 [0,4194304] 0 2024-03-23T11:47:30.463 INFO:tasks.workunit.client.1.smithi111.stdout:4/291: read d0/d17/f3d [719691,92163] 0 2024-03-23T11:47:30.465 INFO:tasks.workunit.client.1.smithi111.stdout:2/351: truncate d2/d7/da/ff 799950 0 2024-03-23T11:47:30.468 INFO:tasks.workunit.client.1.smithi111.stdout:5/352: mknod d1/d20/c78 0 2024-03-23T11:47:30.468 INFO:tasks.workunit.client.0.smithi063.stdout:6/216: write d1/d13/f1c [835962,86023] 0 2024-03-23T11:47:30.468 INFO:tasks.workunit.client.1.smithi111.stdout:6/368: dread d2/d1a/d1c/d2c/d51/f61 [0,4194304] 0 2024-03-23T11:47:30.469 INFO:tasks.workunit.client.0.smithi063.stdout:3/370: dwrite d13/f4d [0,4194304] 0 2024-03-23T11:47:30.469 INFO:tasks.workunit.client.0.smithi063.stdout:7/267: chown d5/d22/f24 32264 1 2024-03-23T11:47:30.469 INFO:tasks.workunit.client.1.smithi111.stdout:0/294: dread d5/fc [0,4194304] 0 2024-03-23T11:47:30.470 INFO:tasks.workunit.client.0.smithi063.stdout:8/529: write d6/d1f/d54/f6a [546952,44959] 0 2024-03-23T11:47:30.473 INFO:tasks.workunit.client.0.smithi063.stdout:5/161: symlink d2/l36 0 2024-03-23T11:47:30.473 INFO:tasks.workunit.client.1.smithi111.stdout:1/305: mkdir d0/d6/d48/d4a/d6c 0 2024-03-23T11:47:30.474 INFO:tasks.workunit.client.1.smithi111.stdout:0/295: write d5/db/d1f/d32/f3e [1997000,72797] 0 2024-03-23T11:47:30.475 INFO:tasks.workunit.client.1.smithi111.stdout:5/353: read d1/d3/fc [5756804,53441] 0 2024-03-23T11:47:30.479 INFO:tasks.workunit.client.1.smithi111.stdout:7/272: symlink d14/d37/d43/l4c 0 2024-03-23T11:47:30.481 INFO:tasks.workunit.client.1.smithi111.stdout:7/273: write d14/d1b/d1e/d2c/f38 [3554635,40455] 0 2024-03-23T11:47:30.481 INFO:tasks.workunit.client.0.smithi063.stdout:2/225: dwrite d2/f25 [0,4194304] 0 2024-03-23T11:47:30.482 INFO:tasks.workunit.client.1.smithi111.stdout:8/250: symlink d5/d27/l50 0 2024-03-23T11:47:30.482 INFO:tasks.workunit.client.0.smithi063.stdout:9/254: creat d4/f5a x:0 0 0 2024-03-23T11:47:30.483 INFO:tasks.workunit.client.1.smithi111.stdout:8/251: truncate d5/d27/f48 475415 0 2024-03-23T11:47:30.486 INFO:tasks.workunit.client.1.smithi111.stdout:3/260: creat d0/d1a/d1d/f49 x:0 0 0 2024-03-23T11:47:30.487 INFO:tasks.workunit.client.0.smithi063.stdout:5/162: fsync d2/d28/f2d 0 2024-03-23T11:47:30.487 INFO:tasks.workunit.client.1.smithi111.stdout:3/261: chown d0/l6 45 1 2024-03-23T11:47:30.489 INFO:tasks.workunit.client.0.smithi063.stdout:8/530: read d6/d1f/d54/d63/d90/f13 [2515,118833] 0 2024-03-23T11:47:30.490 INFO:tasks.workunit.client.1.smithi111.stdout:3/262: chown d0/d7/f1e 31 1 2024-03-23T11:47:30.491 INFO:tasks.workunit.client.0.smithi063.stdout:0/264: read d2/d12/f44 [3936127,82638] 0 2024-03-23T11:47:30.492 INFO:tasks.workunit.client.1.smithi111.stdout:4/292: mkdir d0/dc/d4c/d4b/d4f/d59 0 2024-03-23T11:47:30.492 INFO:tasks.workunit.client.0.smithi063.stdout:1/226: dwrite d3/d35/d40/f4c [0,4194304] 0 2024-03-23T11:47:30.493 INFO:tasks.workunit.client.0.smithi063.stdout:4/271: rename d5/da/d20/d4b/c28 to d5/da/d20/d4b/d29/d38/c4f 0 2024-03-23T11:47:30.498 INFO:tasks.workunit.client.1.smithi111.stdout:2/352: fdatasync d2/d11/d17/d3e/d6b/f2f 0 2024-03-23T11:47:30.498 INFO:tasks.workunit.client.1.smithi111.stdout:1/306: symlink d0/d5/dd/d1e/d63/l6d 0 2024-03-23T11:47:30.501 INFO:tasks.workunit.client.1.smithi111.stdout:0/296: chown d5/db/d1f/d19/l24 1 1 2024-03-23T11:47:30.503 INFO:tasks.workunit.client.0.smithi063.stdout:9/255: mknod d4/d31/d43/c5b 0 2024-03-23T11:47:30.504 INFO:tasks.workunit.client.0.smithi063.stdout:2/226: creat d2/de/d22/f4a x:0 0 0 2024-03-23T11:47:30.510 INFO:tasks.workunit.client.1.smithi111.stdout:4/293: creat d0/dc/d4c/d4b/d4d/f5a x:0 0 0 2024-03-23T11:47:30.511 INFO:tasks.workunit.client.0.smithi063.stdout:9/256: write d4/d4c/f30 [3238753,101440] 0 2024-03-23T11:47:30.512 INFO:tasks.workunit.client.0.smithi063.stdout:6/217: mkdir d1/d2a/d41/d42 0 2024-03-23T11:47:30.513 INFO:tasks.workunit.client.1.smithi111.stdout:1/307: dread d0/f1a [0,4194304] 0 2024-03-23T11:47:30.513 INFO:tasks.workunit.client.1.smithi111.stdout:3/263: dwrite d0/d1a/d1d/d26/f46 [0,4194304] 0 2024-03-23T11:47:30.515 INFO:tasks.workunit.client.1.smithi111.stdout:5/354: symlink d1/d3/d6/d30/d6f/l79 0 2024-03-23T11:47:30.515 INFO:tasks.workunit.client.1.smithi111.stdout:2/353: unlink d2/d11/d17/d3e/d5d/l60 0 2024-03-23T11:47:30.516 INFO:tasks.workunit.client.0.smithi063.stdout:5/163: creat d2/d2f/f37 x:0 0 0 2024-03-23T11:47:30.520 INFO:tasks.workunit.client.1.smithi111.stdout:5/355: write d1/d5d/d59/d6c/f76 [4056,79252] 0 2024-03-23T11:47:30.525 INFO:tasks.workunit.client.1.smithi111.stdout:5/356: truncate d1/d5d/d59/d74/f22 1776107 0 2024-03-23T11:47:30.526 INFO:tasks.workunit.client.0.smithi063.stdout:4/272: mkdir d5/da/d20/d4b/d16/d50 0 2024-03-23T11:47:30.527 INFO:tasks.workunit.client.0.smithi063.stdout:0/265: dread d2/d14/f15 [0,4194304] 0 2024-03-23T11:47:30.527 INFO:tasks.workunit.client.0.smithi063.stdout:5/164: chown d2/d7/d8/de/l16 0 1 2024-03-23T11:47:30.528 INFO:tasks.workunit.client.1.smithi111.stdout:5/357: write d1/d3/d6/d7/f1f [4486878,87283] 0 2024-03-23T11:47:30.528 INFO:tasks.workunit.client.1.smithi111.stdout:1/308: creat d0/d5/dd/f6e x:0 0 0 2024-03-23T11:47:30.530 INFO:tasks.workunit.client.0.smithi063.stdout:8/531: dread d6/dd/df/f11 [0,4194304] 0 2024-03-23T11:47:30.530 INFO:tasks.workunit.client.1.smithi111.stdout:1/309: readlink d0/d5/dd/d1e/d54/l62 0 2024-03-23T11:47:30.531 INFO:tasks.workunit.client.1.smithi111.stdout:3/264: symlink d0/d1a/d2b/d38/l4a 0 2024-03-23T11:47:30.533 INFO:tasks.workunit.client.0.smithi063.stdout:0/266: readlink d2/l29 0 2024-03-23T11:47:30.533 INFO:tasks.workunit.client.0.smithi063.stdout:3/371: dwrite d13/d15/d48/f62 [0,4194304] 0 2024-03-23T11:47:30.537 INFO:tasks.workunit.client.1.smithi111.stdout:8/252: mkdir d5/d32/d35/d46/d4b/d51 0 2024-03-23T11:47:30.539 INFO:tasks.workunit.client.1.smithi111.stdout:8/253: dread - d5/d32/d35/f3e zero size 2024-03-23T11:47:30.540 INFO:tasks.workunit.client.1.smithi111.stdout:8/254: read f3 [3356304,8627] 0 2024-03-23T11:47:30.542 INFO:tasks.workunit.client.1.smithi111.stdout:8/255: chown d5/d1b/c25 0 1 2024-03-23T11:47:30.543 INFO:tasks.workunit.client.1.smithi111.stdout:9/368: truncate d0/d4/d1c/d36/f6b 261360 0 2024-03-23T11:47:30.543 INFO:tasks.workunit.client.0.smithi063.stdout:6/218: symlink d1/d2a/l43 0 2024-03-23T11:47:30.544 INFO:tasks.workunit.client.0.smithi063.stdout:7/268: write d5/d6/d19/d1f/d37/f3b [1821951,26946] 0 2024-03-23T11:47:30.544 INFO:tasks.workunit.client.0.smithi063.stdout:5/165: rmdir d2/d2f 39 2024-03-23T11:47:30.545 INFO:tasks.workunit.client.1.smithi111.stdout:0/297: dread d5/db/f26 [4194304,4194304] 0 2024-03-23T11:47:30.545 INFO:tasks.workunit.client.0.smithi063.stdout:8/532: mkdir d6/dd/d59/db2 0 2024-03-23T11:47:30.546 INFO:tasks.workunit.client.1.smithi111.stdout:0/298: read d5/fc [2385553,97284] 0 2024-03-23T11:47:30.546 INFO:tasks.workunit.client.0.smithi063.stdout:1/227: dread d3/df/f30 [0,4194304] 0 2024-03-23T11:47:30.549 INFO:tasks.workunit.client.0.smithi063.stdout:2/227: dread d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:30.553 INFO:tasks.workunit.client.0.smithi063.stdout:7/269: mkdir d5/d22/d57 0 2024-03-23T11:47:30.553 INFO:tasks.workunit.client.1.smithi111.stdout:3/265: mknod d0/d1a/d2d/c4b 0 2024-03-23T11:47:30.553 INFO:tasks.workunit.client.1.smithi111.stdout:1/310: mknod d0/d6/d48/c6f 0 2024-03-23T11:47:30.554 INFO:tasks.workunit.client.0.smithi063.stdout:2/228: chown d2/d46 12938 1 2024-03-23T11:47:30.556 INFO:tasks.workunit.client.1.smithi111.stdout:1/311: write d0/d6/d48/f52 [4334291,98267] 0 2024-03-23T11:47:30.556 INFO:tasks.workunit.client.1.smithi111.stdout:2/354: link d2/d11/d17/d1f/f20 d2/d11/d17/d1f/d30/f80 0 2024-03-23T11:47:30.560 INFO:tasks.workunit.client.0.smithi063.stdout:8/533: creat d6/d1f/d54/d63/d90/fb3 x:0 0 0 2024-03-23T11:47:30.562 INFO:tasks.workunit.client.1.smithi111.stdout:8/256: chown d5/f6 7205 1 2024-03-23T11:47:30.565 INFO:tasks.workunit.client.1.smithi111.stdout:9/369: write d0/d4/f15 [4792407,52597] 0 2024-03-23T11:47:30.568 INFO:tasks.workunit.client.1.smithi111.stdout:0/299: creat d5/db/d1f/d47/f5a x:0 0 0 2024-03-23T11:47:30.569 INFO:tasks.workunit.client.1.smithi111.stdout:0/300: dread - d5/db/f51 zero size 2024-03-23T11:47:30.569 INFO:tasks.workunit.client.0.smithi063.stdout:9/257: getdents d4 0 2024-03-23T11:47:30.572 INFO:tasks.workunit.client.1.smithi111.stdout:5/358: link d1/d3/d6/d7/f1f d1/d55/d66/d77/f7a 0 2024-03-23T11:47:30.573 INFO:tasks.workunit.client.0.smithi063.stdout:0/267: getdents d2/d4/dd/d1f 0 2024-03-23T11:47:30.574 INFO:tasks.workunit.client.0.smithi063.stdout:8/534: fdatasync d6/dd/df/f60 0 2024-03-23T11:47:30.575 INFO:tasks.workunit.client.0.smithi063.stdout:2/229: creat d2/d23/d2e/f4b x:0 0 0 2024-03-23T11:47:30.576 INFO:tasks.workunit.client.1.smithi111.stdout:4/294: dread d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:30.577 INFO:tasks.workunit.client.1.smithi111.stdout:2/355: rmdir d2/d32/d4b/d52/d58 39 2024-03-23T11:47:30.582 INFO:tasks.workunit.client.0.smithi063.stdout:4/273: dwrite d5/da/f2a [0,4194304] 0 2024-03-23T11:47:30.586 INFO:tasks.workunit.client.0.smithi063.stdout:8/535: chown d6/dd/d59/fad 195479 1 2024-03-23T11:47:30.587 INFO:tasks.workunit.client.1.smithi111.stdout:9/370: mkdir d0/d4/d10/d80 0 2024-03-23T11:47:30.588 INFO:tasks.workunit.client.1.smithi111.stdout:9/371: chown d0/d4/d1c 12669611 1 2024-03-23T11:47:30.593 INFO:tasks.workunit.client.0.smithi063.stdout:3/372: dwrite f12 [4194304,4194304] 0 2024-03-23T11:47:30.593 INFO:tasks.workunit.client.1.smithi111.stdout:6/369: sync 2024-03-23T11:47:30.594 INFO:tasks.workunit.client.0.smithi063.stdout:0/268: creat d2/d4/dd/d28/d46/f5b x:0 0 0 2024-03-23T11:47:30.595 INFO:tasks.workunit.client.0.smithi063.stdout:4/274: chown d5/da/d20/d4b/d1f/f26 91 1 2024-03-23T11:47:30.595 INFO:tasks.workunit.client.0.smithi063.stdout:5/166: dwrite d2/f33 [0,4194304] 0 2024-03-23T11:47:30.598 INFO:tasks.workunit.client.1.smithi111.stdout:3/266: symlink d0/d7/l4c 0 2024-03-23T11:47:30.598 INFO:tasks.workunit.client.0.smithi063.stdout:2/230: rename d2/d1c/f1f to d2/d27/f4c 0 2024-03-23T11:47:30.600 INFO:tasks.workunit.client.1.smithi111.stdout:5/359: write d1/d5d/d59/d74/f4d [1179391,20872] 0 2024-03-23T11:47:30.601 INFO:tasks.workunit.client.1.smithi111.stdout:4/295: creat d0/d7/da/f5b x:0 0 0 2024-03-23T11:47:30.602 INFO:tasks.workunit.client.1.smithi111.stdout:5/360: dread - d1/d5d/d59/d74/f6b zero size 2024-03-23T11:47:30.608 INFO:tasks.workunit.client.1.smithi111.stdout:2/356: creat d2/de/d6c/f81 x:0 0 0 2024-03-23T11:47:30.608 INFO:tasks.workunit.client.0.smithi063.stdout:2/231: chown d2/d23/d2e/f35 1047996 1 2024-03-23T11:47:30.608 INFO:tasks.workunit.client.0.smithi063.stdout:7/270: dwrite d5/d6/d19/d1f/f42 [0,4194304] 0 2024-03-23T11:47:30.609 INFO:tasks.workunit.client.0.smithi063.stdout:6/219: dread d1/f5 [0,4194304] 0 2024-03-23T11:47:30.610 INFO:tasks.workunit.client.0.smithi063.stdout:3/373: unlink d13/d22/d2d/d3f/c6b 0 2024-03-23T11:47:30.613 INFO:tasks.workunit.client.1.smithi111.stdout:8/257: creat d5/d32/d35/d4a/f52 x:0 0 0 2024-03-23T11:47:30.614 INFO:tasks.workunit.client.0.smithi063.stdout:4/275: chown d5/da/d20/d4b/d1f/f26 0 1 2024-03-23T11:47:30.615 INFO:tasks.workunit.client.0.smithi063.stdout:2/232: chown d2/f47 4406486 1 2024-03-23T11:47:30.616 INFO:tasks.workunit.client.0.smithi063.stdout:1/228: dread d3/d35/d40/f4c [0,4194304] 0 2024-03-23T11:47:30.616 INFO:tasks.workunit.client.0.smithi063.stdout:3/374: dread - d13/d22/d2d/f4f zero size 2024-03-23T11:47:30.617 INFO:tasks.workunit.client.1.smithi111.stdout:3/267: dwrite d0/fd [4194304,4194304] 0 2024-03-23T11:47:30.618 INFO:tasks.workunit.client.0.smithi063.stdout:6/220: readlink d1/l9 0 2024-03-23T11:47:30.619 INFO:tasks.workunit.client.1.smithi111.stdout:0/301: mkdir d5/db/d1f/d47/d59/d5b 0 2024-03-23T11:47:30.619 INFO:tasks.workunit.client.0.smithi063.stdout:9/258: dread d4/ff [0,4194304] 0 2024-03-23T11:47:30.620 INFO:tasks.workunit.client.0.smithi063.stdout:3/375: fdatasync d13/d22/d2d/f53 0 2024-03-23T11:47:30.622 INFO:tasks.workunit.client.0.smithi063.stdout:2/233: stat d2/d23/d2e/l42 0 2024-03-23T11:47:30.623 INFO:tasks.workunit.client.1.smithi111.stdout:6/370: readlink d2/d1a/d1c/l3a 0 2024-03-23T11:47:30.624 INFO:tasks.workunit.client.0.smithi063.stdout:2/234: stat d2/de/l1a 0 2024-03-23T11:47:30.627 INFO:tasks.workunit.client.0.smithi063.stdout:5/167: creat d2/d7/d2b/f38 x:0 0 0 2024-03-23T11:47:30.627 INFO:tasks.workunit.client.1.smithi111.stdout:8/258: readlink d5/d26/d2f/l38 0 2024-03-23T11:47:30.628 INFO:tasks.workunit.client.1.smithi111.stdout:9/372: dread d0/d4/d1c/f66 [0,4194304] 0 2024-03-23T11:47:30.628 INFO:tasks.workunit.client.0.smithi063.stdout:8/536: getdents d6/d1c/d32/d37 0 2024-03-23T11:47:30.629 INFO:tasks.workunit.client.0.smithi063.stdout:3/376: mkdir d13/d15/d48/d71 0 2024-03-23T11:47:30.633 INFO:tasks.workunit.client.0.smithi063.stdout:1/229: rename d3/d51 to d3/d3f/d4d/d52 0 2024-03-23T11:47:30.642 INFO:tasks.workunit.client.1.smithi111.stdout:1/312: getdents d0/d5/dd/d1e 0 2024-03-23T11:47:30.644 INFO:tasks.workunit.client.0.smithi063.stdout:7/271: symlink d5/d26/d32/d3a/l58 0 2024-03-23T11:47:30.645 INFO:tasks.workunit.client.0.smithi063.stdout:0/269: link d2/d12/f50 d2/d12/d32/f5c 0 2024-03-23T11:47:30.649 INFO:tasks.workunit.client.0.smithi063.stdout:7/272: chown d5/d16 60784 1 2024-03-23T11:47:30.650 INFO:tasks.workunit.client.0.smithi063.stdout:0/270: chown f1 284 1 2024-03-23T11:47:30.652 INFO:tasks.workunit.client.1.smithi111.stdout:5/361: mkdir d1/d3/d6/d7/d7b 0 2024-03-23T11:47:30.653 INFO:tasks.workunit.client.0.smithi063.stdout:9/259: creat d4/d4c/f5c x:0 0 0 2024-03-23T11:47:30.656 INFO:tasks.workunit.client.0.smithi063.stdout:1/230: fdatasync d3/f2a 0 2024-03-23T11:47:30.656 INFO:tasks.workunit.client.0.smithi063.stdout:7/273: symlink d5/d16/l59 0 2024-03-23T11:47:30.658 INFO:tasks.workunit.client.1.smithi111.stdout:8/259: mknod d5/d26/d2f/d42/c53 0 2024-03-23T11:47:30.659 INFO:tasks.workunit.client.1.smithi111.stdout:3/268: truncate d0/d7/f12 4042253 0 2024-03-23T11:47:30.661 INFO:tasks.workunit.client.1.smithi111.stdout:1/313: mkdir d0/d5/dd/d1e/d70 0 2024-03-23T11:47:30.662 INFO:tasks.workunit.client.1.smithi111.stdout:1/314: truncate d0/d5/f23 4844236 0 2024-03-23T11:47:30.663 INFO:tasks.workunit.client.1.smithi111.stdout:4/296: rmdir d0/dc/d4c/d4b/d4f/d59 0 2024-03-23T11:47:30.664 INFO:tasks.workunit.client.1.smithi111.stdout:6/371: fsync d2/f27 0 2024-03-23T11:47:30.665 INFO:tasks.workunit.client.1.smithi111.stdout:4/297: dread - d0/dc/d4c/d4b/d4d/f5a zero size 2024-03-23T11:47:30.668 INFO:tasks.workunit.client.0.smithi063.stdout:6/221: dwrite d1/dc/d3a/f3e [0,4194304] 0 2024-03-23T11:47:30.670 INFO:tasks.workunit.client.1.smithi111.stdout:3/269: creat d0/d1a/d2b/f4d x:0 0 0 2024-03-23T11:47:30.675 INFO:tasks.workunit.client.0.smithi063.stdout:8/537: dwrite d6/d1f/d54/d63/d90/d9e/fa8 [0,4194304] 0 2024-03-23T11:47:30.677 INFO:tasks.workunit.client.0.smithi063.stdout:9/260: creat d4/d31/f5d x:0 0 0 2024-03-23T11:47:30.677 INFO:tasks.workunit.client.0.smithi063.stdout:2/235: dwrite d2/d27/f4c [0,4194304] 0 2024-03-23T11:47:30.678 INFO:tasks.workunit.client.1.smithi111.stdout:7/274: sync 2024-03-23T11:47:30.678 INFO:tasks.workunit.client.0.smithi063.stdout:3/377: dwrite d13/d22/d2d/d3f/d5b/f65 [0,4194304] 0 2024-03-23T11:47:30.680 INFO:tasks.workunit.client.1.smithi111.stdout:7/275: dread - d14/d1b/d1e/f47 zero size 2024-03-23T11:47:30.691 INFO:tasks.workunit.client.0.smithi063.stdout:4/276: truncate d5/da/d20/f3c 3342576 0 2024-03-23T11:47:30.692 INFO:tasks.workunit.client.1.smithi111.stdout:4/298: rmdir d0/dc/d4c/d4b 39 2024-03-23T11:47:30.693 INFO:tasks.workunit.client.1.smithi111.stdout:9/373: dwrite d0/d4/d1c/d36/d28/f5e [4194304,4194304] 0 2024-03-23T11:47:30.699 INFO:tasks.workunit.client.1.smithi111.stdout:0/302: truncate d5/db/d4d/f52 772171 0 2024-03-23T11:47:30.700 INFO:tasks.workunit.client.1.smithi111.stdout:5/362: creat d1/d3/d6/d7/f7c x:0 0 0 2024-03-23T11:47:30.700 INFO:tasks.workunit.client.0.smithi063.stdout:4/277: read d5/da/d20/f48 [342772,113664] 0 2024-03-23T11:47:30.701 INFO:tasks.workunit.client.1.smithi111.stdout:3/270: creat d0/d1a/d2d/d41/f4e x:0 0 0 2024-03-23T11:47:30.702 INFO:tasks.workunit.client.0.smithi063.stdout:0/271: symlink d2/d4/dd/d25/l5d 0 2024-03-23T11:47:30.703 INFO:tasks.workunit.client.1.smithi111.stdout:0/303: dread - d5/db/f4a zero size 2024-03-23T11:47:30.703 INFO:tasks.workunit.client.1.smithi111.stdout:2/357: sync 2024-03-23T11:47:30.710 INFO:tasks.workunit.client.0.smithi063.stdout:2/236: mkdir d2/d27/d44/d4d 0 2024-03-23T11:47:30.711 INFO:tasks.workunit.client.1.smithi111.stdout:5/363: rmdir d1/d5d/d59 39 2024-03-23T11:47:30.713 INFO:tasks.workunit.client.1.smithi111.stdout:0/304: symlink d5/db/d4d/l5c 0 2024-03-23T11:47:30.714 INFO:tasks.workunit.client.0.smithi063.stdout:0/272: chown d2/d12/f44 1088083755 1 2024-03-23T11:47:30.716 INFO:tasks.workunit.client.1.smithi111.stdout:3/271: dread d0/f1f [0,4194304] 0 2024-03-23T11:47:30.716 INFO:tasks.workunit.client.0.smithi063.stdout:3/378: creat d13/d22/d2d/d3f/f72 x:0 0 0 2024-03-23T11:47:30.718 INFO:tasks.workunit.client.1.smithi111.stdout:7/276: dwrite d14/d1b/d1e/d2c/f31 [0,4194304] 0 2024-03-23T11:47:30.723 INFO:tasks.workunit.client.1.smithi111.stdout:3/272: write d0/d1a/d1d/f43 [61251,64743] 0 2024-03-23T11:47:30.723 INFO:tasks.workunit.client.1.smithi111.stdout:4/299: dwrite d0/dc/d30/ff [0,4194304] 0 2024-03-23T11:47:30.724 INFO:tasks.workunit.client.0.smithi063.stdout:9/261: dwrite d4/d4c/f30 [0,4194304] 0 2024-03-23T11:47:30.732 INFO:tasks.workunit.client.0.smithi063.stdout:5/168: dread d2/d7/d8/f12 [0,4194304] 0 2024-03-23T11:47:30.735 INFO:tasks.workunit.client.0.smithi063.stdout:2/237: unlink d2/de/l11 0 2024-03-23T11:47:30.736 INFO:tasks.workunit.client.1.smithi111.stdout:7/277: chown f9 360850068 1 2024-03-23T11:47:30.737 INFO:tasks.workunit.client.1.smithi111.stdout:7/278: readlink d14/d1b/l23 0 2024-03-23T11:47:30.737 INFO:tasks.workunit.client.0.smithi063.stdout:2/238: write d2/f18 [4778756,64740] 0 2024-03-23T11:47:30.740 INFO:tasks.workunit.client.0.smithi063.stdout:2/239: chown d2/lb 856 1 2024-03-23T11:47:30.745 INFO:tasks.workunit.client.1.smithi111.stdout:1/315: dwrite d0/d2b/f45 [4194304,4194304] 0 2024-03-23T11:47:30.746 INFO:tasks.workunit.client.0.smithi063.stdout:3/379: mknod d13/d15/c73 0 2024-03-23T11:47:30.746 INFO:tasks.workunit.client.0.smithi063.stdout:1/231: write d3/d1c/f27 [4916032,40755] 0 2024-03-23T11:47:30.750 INFO:tasks.workunit.client.1.smithi111.stdout:0/305: chown d5/db/d1f/d19/f4c 1602 1 2024-03-23T11:47:30.750 INFO:tasks.workunit.client.1.smithi111.stdout:8/260: dwrite d5/f1a [0,4194304] 0 2024-03-23T11:47:30.752 INFO:tasks.workunit.client.0.smithi063.stdout:1/232: read d3/d1c/f27 [2145679,105222] 0 2024-03-23T11:47:30.754 INFO:tasks.workunit.client.0.smithi063.stdout:7/274: getdents d5/d26 0 2024-03-23T11:47:30.755 INFO:tasks.workunit.client.1.smithi111.stdout:3/273: write d0/f31 [4704372,71464] 0 2024-03-23T11:47:30.756 INFO:tasks.workunit.client.1.smithi111.stdout:1/316: read - d0/d5/f5e zero size 2024-03-23T11:47:30.763 INFO:tasks.workunit.client.0.smithi063.stdout:5/169: rename d2/d7/d2b/f38 to d2/d7/d2b/f39 0 2024-03-23T11:47:30.764 INFO:tasks.workunit.client.1.smithi111.stdout:3/274: dread - d0/d7/f47 zero size 2024-03-23T11:47:30.764 INFO:tasks.workunit.client.1.smithi111.stdout:2/358: dwrite d2/de/d12/f2c [0,4194304] 0 2024-03-23T11:47:30.768 INFO:tasks.workunit.client.1.smithi111.stdout:9/374: truncate d0/d4/d1c/d36/d28/f5e 6075508 0 2024-03-23T11:47:30.770 INFO:tasks.workunit.client.1.smithi111.stdout:8/261: truncate d5/f6 4273760 0 2024-03-23T11:47:30.776 INFO:tasks.workunit.client.0.smithi063.stdout:0/273: creat d2/d4/f5e x:0 0 0 2024-03-23T11:47:30.777 INFO:tasks.workunit.client.1.smithi111.stdout:0/306: write d5/fc [2379640,46392] 0 2024-03-23T11:47:30.778 INFO:tasks.workunit.client.0.smithi063.stdout:0/274: chown d2/d14/d36/l53 7368 1 2024-03-23T11:47:30.779 INFO:tasks.workunit.client.1.smithi111.stdout:1/317: fsync d0/d5/dd/d1e/d63/f64 0 2024-03-23T11:47:30.780 INFO:tasks.workunit.client.1.smithi111.stdout:6/372: dwrite d2/d1f/d45/f62 [0,4194304] 0 2024-03-23T11:47:30.782 INFO:tasks.workunit.client.0.smithi063.stdout:5/170: mkdir d2/d28/d3a 0 2024-03-23T11:47:30.783 INFO:tasks.workunit.client.1.smithi111.stdout:3/275: write d0/d1a/d1d/f43 [408642,73400] 0 2024-03-23T11:47:30.784 INFO:tasks.workunit.client.0.smithi063.stdout:3/380: mknod d13/d22/d2d/d3f/d67/c74 0 2024-03-23T11:47:30.786 INFO:tasks.workunit.client.1.smithi111.stdout:5/364: dwrite d1/d3/d6/d30/f70 [0,4194304] 0 2024-03-23T11:47:30.787 INFO:tasks.workunit.client.0.smithi063.stdout:8/538: truncate d6/d1c/d27/f31 3439636 0 2024-03-23T11:47:30.787 INFO:tasks.workunit.client.1.smithi111.stdout:4/300: dwrite d0/dc/d1f/f4a [0,4194304] 0 2024-03-23T11:47:30.787 INFO:tasks.workunit.client.1.smithi111.stdout:8/262: write d5/d32/d35/f49 [4136945,126659] 0 2024-03-23T11:47:30.790 INFO:tasks.workunit.client.0.smithi063.stdout:5/171: write d2/d7/d8/d1d/f1f [17863,89731] 0 2024-03-23T11:47:30.790 INFO:tasks.workunit.client.0.smithi063.stdout:2/240: link d2/d1c/l39 d2/d46/l4e 0 2024-03-23T11:47:30.791 INFO:tasks.workunit.client.0.smithi063.stdout:3/381: write d13/d22/d2d/f45 [3869947,32818] 0 2024-03-23T11:47:30.792 INFO:tasks.workunit.client.1.smithi111.stdout:7/279: dwrite d14/f1c [0,4194304] 0 2024-03-23T11:47:30.794 INFO:tasks.workunit.client.0.smithi063.stdout:0/275: write d2/d4/f9 [1042591,129245] 0 2024-03-23T11:47:30.795 INFO:tasks.workunit.client.1.smithi111.stdout:0/307: write d5/db/d1f/f22 [3704379,110800] 0 2024-03-23T11:47:30.798 INFO:tasks.workunit.client.1.smithi111.stdout:9/375: mknod d0/d4/d1c/d2d/d6e/c81 0 2024-03-23T11:47:30.798 INFO:tasks.workunit.client.1.smithi111.stdout:1/318: symlink d0/d6/l71 0 2024-03-23T11:47:30.799 INFO:tasks.workunit.client.0.smithi063.stdout:3/382: write d13/d22/d2d/d3f/d5b/f5d [634688,52287] 0 2024-03-23T11:47:30.800 INFO:tasks.workunit.client.1.smithi111.stdout:7/280: chown d14/l27 14785210 1 2024-03-23T11:47:30.805 INFO:tasks.workunit.client.1.smithi111.stdout:2/359: creat d2/d11/d17/d3e/d6b/d2e/f82 x:0 0 0 2024-03-23T11:47:30.809 INFO:tasks.workunit.client.0.smithi063.stdout:7/275: creat d5/d6/f5a x:0 0 0 2024-03-23T11:47:30.809 INFO:tasks.workunit.client.1.smithi111.stdout:2/360: chown d2/d11/d17/d1f/d30/f53 40493587 1 2024-03-23T11:47:30.811 INFO:tasks.workunit.client.1.smithi111.stdout:6/373: dread d2/d1a/f64 [0,4194304] 0 2024-03-23T11:47:30.813 INFO:tasks.workunit.client.0.smithi063.stdout:7/276: chown d5/d6/d19/d1f/d47/d4f 36492476 1 2024-03-23T11:47:30.816 INFO:tasks.workunit.client.0.smithi063.stdout:6/222: dread d1/f12 [0,4194304] 0 2024-03-23T11:47:30.818 INFO:tasks.workunit.client.0.smithi063.stdout:7/277: write d5/d6/d19/d1f/d37/d20/f31 [1178898,85304] 0 2024-03-23T11:47:30.819 INFO:tasks.workunit.client.0.smithi063.stdout:8/539: mkdir d6/d1f/d54/d63/d90/d5d/d7d/db4 0 2024-03-23T11:47:30.820 INFO:tasks.workunit.client.0.smithi063.stdout:9/262: write d4/f1e [5158933,34009] 0 2024-03-23T11:47:30.824 INFO:tasks.workunit.client.0.smithi063.stdout:0/276: unlink d2/d14/c1b 0 2024-03-23T11:47:30.834 INFO:tasks.workunit.client.0.smithi063.stdout:2/241: mknod d2/d46/c4f 0 2024-03-23T11:47:30.839 INFO:tasks.workunit.client.0.smithi063.stdout:2/242: chown d2/d23/l24 20955 1 2024-03-23T11:47:30.842 INFO:tasks.workunit.client.1.smithi111.stdout:1/319: read d0/d6/f34 [3611754,43820] 0 2024-03-23T11:47:30.845 INFO:tasks.workunit.client.1.smithi111.stdout:7/281: dread f12 [0,4194304] 0 2024-03-23T11:47:30.847 INFO:tasks.workunit.client.0.smithi063.stdout:8/540: mkdir d6/dd/d59/d7f/db5 0 2024-03-23T11:47:30.847 INFO:tasks.workunit.client.0.smithi063.stdout:6/223: dwrite d1/dc/d3a/f3f [0,4194304] 0 2024-03-23T11:47:30.853 INFO:tasks.workunit.client.0.smithi063.stdout:3/383: dwrite d13/d22/f3d [0,4194304] 0 2024-03-23T11:47:30.857 INFO:tasks.workunit.client.0.smithi063.stdout:3/384: write fc [9332087,2350] 0 2024-03-23T11:47:30.858 INFO:tasks.workunit.client.0.smithi063.stdout:3/385: fsync f3 0 2024-03-23T11:47:30.866 INFO:tasks.workunit.client.0.smithi063.stdout:3/386: write d13/d22/f63 [884876,31865] 0 2024-03-23T11:47:30.867 INFO:tasks.workunit.client.0.smithi063.stdout:5/172: mkdir d2/d7/d8/de/d2a/d3b 0 2024-03-23T11:47:30.873 INFO:tasks.workunit.client.0.smithi063.stdout:6/224: dread d1/dc/fd [0,4194304] 0 2024-03-23T11:47:30.874 INFO:tasks.workunit.client.0.smithi063.stdout:3/387: dread - d13/d22/d2d/d3f/f72 zero size 2024-03-23T11:47:30.877 INFO:tasks.workunit.client.0.smithi063.stdout:1/233: dwrite d3/f2a [0,4194304] 0 2024-03-23T11:47:30.877 INFO:tasks.workunit.client.0.smithi063.stdout:5/173: read d2/d7/d8/f20 [214363,93124] 0 2024-03-23T11:47:30.880 INFO:tasks.workunit.client.0.smithi063.stdout:2/243: dwrite d2/f29 [0,4194304] 0 2024-03-23T11:47:30.881 INFO:tasks.workunit.client.0.smithi063.stdout:6/225: chown d1/d2a/l43 7193 1 2024-03-23T11:47:30.885 INFO:tasks.workunit.client.1.smithi111.stdout:0/308: stat d5/db/c3c 0 2024-03-23T11:47:30.889 INFO:tasks.workunit.client.0.smithi063.stdout:7/278: unlink d5/d6/d19/c29 0 2024-03-23T11:47:30.889 INFO:tasks.workunit.client.0.smithi063.stdout:2/244: write d2/d23/d2e/f4b [396488,10832] 0 2024-03-23T11:47:30.899 INFO:tasks.workunit.client.0.smithi063.stdout:0/277: mknod d2/d4/c5f 0 2024-03-23T11:47:30.901 INFO:tasks.workunit.client.1.smithi111.stdout:9/376: write d0/d4/d1c/f66 [3894155,53080] 0 2024-03-23T11:47:30.904 INFO:tasks.workunit.client.0.smithi063.stdout:1/234: creat d3/d35/d40/f53 x:0 0 0 2024-03-23T11:47:30.906 INFO:tasks.workunit.client.1.smithi111.stdout:0/309: dwrite d5/db/f15 [0,4194304] 0 2024-03-23T11:47:30.908 INFO:tasks.workunit.client.1.smithi111.stdout:0/310: dread - d5/db/d1f/f43 zero size 2024-03-23T11:47:30.909 INFO:tasks.workunit.client.1.smithi111.stdout:9/377: truncate d0/d4/d1c/d2d/d3b/d42/d4f/f61 1028864 0 2024-03-23T11:47:30.910 INFO:tasks.workunit.client.0.smithi063.stdout:6/226: dwrite d1/f35 [0,4194304] 0 2024-03-23T11:47:30.912 INFO:tasks.workunit.client.1.smithi111.stdout:9/378: truncate d0/f48 4797584 0 2024-03-23T11:47:30.915 INFO:tasks.workunit.client.0.smithi063.stdout:8/541: truncate d6/dd/df/f60 1683072 0 2024-03-23T11:47:30.923 INFO:tasks.workunit.client.0.smithi063.stdout:8/542: read d6/d1c/f6c [644323,32069] 0 2024-03-23T11:47:30.926 INFO:tasks.workunit.client.0.smithi063.stdout:2/245: dread d2/de/f15 [0,4194304] 0 2024-03-23T11:47:30.928 INFO:tasks.workunit.client.1.smithi111.stdout:0/311: dwrite d5/db/f26 [0,4194304] 0 2024-03-23T11:47:30.930 INFO:tasks.workunit.client.0.smithi063.stdout:0/278: creat d2/d4/dd/d28/f60 x:0 0 0 2024-03-23T11:47:30.930 INFO:tasks.workunit.client.0.smithi063.stdout:1/235: mkdir d3/d35/d54 0 2024-03-23T11:47:30.931 INFO:tasks.workunit.client.0.smithi063.stdout:6/227: mknod d1/dc/d21/c44 0 2024-03-23T11:47:30.933 INFO:tasks.workunit.client.0.smithi063.stdout:0/279: write d2/d14/f31 [4955062,36773] 0 2024-03-23T11:47:30.934 INFO:tasks.workunit.client.1.smithi111.stdout:7/282: dread f11 [0,4194304] 0 2024-03-23T11:47:30.934 INFO:tasks.workunit.client.0.smithi063.stdout:9/263: getdents d4/d31/d43/d49 0 2024-03-23T11:47:30.942 INFO:tasks.workunit.client.1.smithi111.stdout:5/365: creat d1/d3/f7d x:0 0 0 2024-03-23T11:47:30.947 INFO:tasks.workunit.client.0.smithi063.stdout:5/174: rename d2/d7/d8/de/l16 to d2/l3c 0 2024-03-23T11:47:30.948 INFO:tasks.workunit.client.0.smithi063.stdout:2/246: unlink d2/f17 0 2024-03-23T11:47:30.949 INFO:tasks.workunit.client.0.smithi063.stdout:8/543: rename d6/d1f/d54 to d6/d1f/d54/d63/d90/d15/db6 22 2024-03-23T11:47:30.950 INFO:tasks.workunit.client.1.smithi111.stdout:3/276: rename d0/d1a/d2d/d41 to d0/d1a/d4f 0 2024-03-23T11:47:30.950 INFO:tasks.workunit.client.0.smithi063.stdout:5/175: read - d2/d7/d2b/f39 zero size 2024-03-23T11:47:30.952 INFO:tasks.workunit.client.0.smithi063.stdout:1/236: unlink d3/d1c/f3c 0 2024-03-23T11:47:30.953 INFO:tasks.workunit.client.0.smithi063.stdout:9/264: symlink d4/d31/l5e 0 2024-03-23T11:47:30.957 INFO:tasks.workunit.client.0.smithi063.stdout:0/280: creat d2/d14/d36/f61 x:0 0 0 2024-03-23T11:47:30.958 INFO:tasks.workunit.client.0.smithi063.stdout:3/388: dread f6 [0,4194304] 0 2024-03-23T11:47:30.960 INFO:tasks.workunit.client.1.smithi111.stdout:9/379: write d0/d4/d1c/d36/f11 [8462541,95290] 0 2024-03-23T11:47:30.962 INFO:tasks.workunit.client.0.smithi063.stdout:5/176: creat d2/d28/f3d x:0 0 0 2024-03-23T11:47:30.964 INFO:tasks.workunit.client.0.smithi063.stdout:0/281: write d2/d4/dd/d28/f60 [249724,35395] 0 2024-03-23T11:47:30.971 INFO:tasks.workunit.client.0.smithi063.stdout:1/237: dread d3/df/f30 [0,4194304] 0 2024-03-23T11:47:30.971 INFO:tasks.workunit.client.0.smithi063.stdout:7/279: dread d5/d6/d19/d1f/d37/da/f17 [4194304,4194304] 0 2024-03-23T11:47:30.972 INFO:tasks.workunit.client.0.smithi063.stdout:1/238: chown d3/df 1867624 1 2024-03-23T11:47:30.977 INFO:tasks.workunit.client.0.smithi063.stdout:7/280: truncate d5/d6/d19/d1f/d37/f3c 4814966 0 2024-03-23T11:47:30.979 INFO:tasks.workunit.client.1.smithi111.stdout:3/277: mknod d0/d1a/d2b/c50 0 2024-03-23T11:47:30.982 INFO:tasks.workunit.client.0.smithi063.stdout:7/281: write d5/d6/d19/d1f/f42 [3592052,69910] 0 2024-03-23T11:47:30.985 INFO:tasks.workunit.client.1.smithi111.stdout:5/366: dread d1/d55/d66/d77/f7a [4194304,4194304] 0 2024-03-23T11:47:30.985 INFO:tasks.workunit.client.0.smithi063.stdout:0/282: creat d2/d12/d35/f62 x:0 0 0 2024-03-23T11:47:30.989 INFO:tasks.workunit.client.1.smithi111.stdout:5/367: fsync d1/d3/d6/f2b 0 2024-03-23T11:47:30.990 INFO:tasks.workunit.client.1.smithi111.stdout:4/301: getdents d0/d7 0 2024-03-23T11:47:30.991 INFO:tasks.workunit.client.0.smithi063.stdout:9/265: link d4/f5 d4/d31/f5f 0 2024-03-23T11:47:30.991 INFO:tasks.workunit.client.1.smithi111.stdout:5/368: stat d1/d20/f60 0 2024-03-23T11:47:30.992 INFO:tasks.workunit.client.1.smithi111.stdout:7/283: dwrite f1 [4194304,4194304] 0 2024-03-23T11:47:30.993 INFO:tasks.workunit.client.0.smithi063.stdout:3/389: creat d13/d66/f75 x:0 0 0 2024-03-23T11:47:30.994 INFO:tasks.workunit.client.1.smithi111.stdout:0/312: symlink d5/db/d1f/d19/l5d 0 2024-03-23T11:47:30.995 INFO:tasks.workunit.client.0.smithi063.stdout:9/266: chown d4/d31/f59 27626886 1 2024-03-23T11:47:30.997 INFO:tasks.workunit.client.0.smithi063.stdout:9/267: chown d4/f1e 4 1 2024-03-23T11:47:30.998 INFO:tasks.workunit.client.1.smithi111.stdout:2/361: getdents d2/d11/d17/d3e/d5d 0 2024-03-23T11:47:30.999 INFO:tasks.workunit.client.1.smithi111.stdout:8/263: symlink d5/l54 0 2024-03-23T11:47:31.000 INFO:tasks.workunit.client.0.smithi063.stdout:2/247: getdents d2 0 2024-03-23T11:47:31.001 INFO:tasks.workunit.client.1.smithi111.stdout:3/278: dwrite d0/d7/f47 [0,4194304] 0 2024-03-23T11:47:31.001 INFO:tasks.workunit.client.0.smithi063.stdout:6/228: dwrite d1/dc/d21/f29 [0,4194304] 0 2024-03-23T11:47:31.003 INFO:tasks.workunit.client.1.smithi111.stdout:2/362: truncate d2/d7/da/d19/f6a 1010064 0 2024-03-23T11:47:31.005 INFO:tasks.workunit.client.0.smithi063.stdout:2/248: chown d2/d23/d2e/l42 22 1 2024-03-23T11:47:31.010 INFO:tasks.workunit.client.0.smithi063.stdout:3/390: creat d13/d22/d2d/d3f/d58/f76 x:0 0 0 2024-03-23T11:47:31.010 INFO:tasks.workunit.client.0.smithi063.stdout:8/544: link d6/d1f/d54/d63/d90/c7e d6/d1f/d54/d5c/cb7 0 2024-03-23T11:47:31.011 INFO:tasks.workunit.client.1.smithi111.stdout:7/284: creat d14/d1b/d22/f4d x:0 0 0 2024-03-23T11:47:31.011 INFO:tasks.workunit.client.0.smithi063.stdout:0/283: mkdir d2/d4/d63 0 2024-03-23T11:47:31.011 INFO:tasks.workunit.client.0.smithi063.stdout:5/177: creat d2/f3e x:0 0 0 2024-03-23T11:47:31.011 INFO:tasks.workunit.client.0.smithi063.stdout:6/229: creat d1/dc/d3a/f45 x:0 0 0 2024-03-23T11:47:31.013 INFO:tasks.workunit.client.1.smithi111.stdout:7/285: write d14/d1b/d20/f2b [3202046,18535] 0 2024-03-23T11:47:31.015 INFO:tasks.workunit.client.0.smithi063.stdout:2/249: creat d2/d27/d44/f50 x:0 0 0 2024-03-23T11:47:31.015 INFO:tasks.workunit.client.1.smithi111.stdout:5/369: dwrite d1/d20/f61 [4194304,4194304] 0 2024-03-23T11:47:31.015 INFO:tasks.workunit.client.0.smithi063.stdout:7/282: mkdir d5/d6/d19/d1f/d47/d4f/d5b 0 2024-03-23T11:47:31.020 INFO:tasks.workunit.client.1.smithi111.stdout:8/264: mkdir d5/d26/d31/d3b/d55 0 2024-03-23T11:47:31.020 INFO:tasks.workunit.client.1.smithi111.stdout:5/370: fsync d1/f49 0 2024-03-23T11:47:31.027 INFO:tasks.workunit.client.1.smithi111.stdout:7/286: read d14/d1b/d1e/d2c/d2e/d33/f3d [2537006,70952] 0 2024-03-23T11:47:31.027 INFO:tasks.workunit.client.1.smithi111.stdout:9/380: unlink d0/d4/d1c/d2d/c30 0 2024-03-23T11:47:31.028 INFO:tasks.workunit.client.0.smithi063.stdout:0/284: rename f1 to d2/d14/f64 0 2024-03-23T11:47:31.028 INFO:tasks.workunit.client.0.smithi063.stdout:6/230: creat d1/dc/f46 x:0 0 0 2024-03-23T11:47:31.029 INFO:tasks.workunit.client.0.smithi063.stdout:3/391: write d13/d22/d2d/d3f/f72 [387822,96553] 0 2024-03-23T11:47:31.031 INFO:tasks.workunit.client.0.smithi063.stdout:5/178: unlink d2/d2f/l30 0 2024-03-23T11:47:31.033 INFO:tasks.workunit.client.1.smithi111.stdout:2/363: mknod d2/d11/d17/d3e/c83 0 2024-03-23T11:47:31.034 INFO:tasks.workunit.client.0.smithi063.stdout:5/179: chown d2 0 1 2024-03-23T11:47:31.035 INFO:tasks.workunit.client.0.smithi063.stdout:7/283: rmdir d5 39 2024-03-23T11:47:31.035 INFO:tasks.workunit.client.0.smithi063.stdout:1/239: dread d3/df/f1a [0,4194304] 0 2024-03-23T11:47:31.036 INFO:tasks.workunit.client.0.smithi063.stdout:2/250: mkdir d2/d1c/d51 0 2024-03-23T11:47:31.037 INFO:tasks.workunit.client.1.smithi111.stdout:3/279: dread d0/f16 [0,4194304] 0 2024-03-23T11:47:31.040 INFO:tasks.workunit.client.0.smithi063.stdout:9/268: dwrite d4/f35 [0,4194304] 0 2024-03-23T11:47:31.040 INFO:tasks.workunit.client.1.smithi111.stdout:8/265: dwrite d5/d26/d31/d3b/f3d [0,4194304] 0 2024-03-23T11:47:31.042 INFO:tasks.workunit.client.0.smithi063.stdout:3/392: fsync d13/d15/d48/f68 0 2024-03-23T11:47:31.050 INFO:tasks.workunit.client.0.smithi063.stdout:8/545: rename d6/l75 to d6/d1f/d54/d63/d90/d5d/d7d/d9a/lb8 0 2024-03-23T11:47:31.052 INFO:tasks.workunit.client.0.smithi063.stdout:3/393: readlink d13/d15/l1e 0 2024-03-23T11:47:31.053 INFO:tasks.workunit.client.1.smithi111.stdout:1/320: dwrite d0/d5/dd/d1e/f21 [0,4194304] 0 2024-03-23T11:47:31.053 INFO:tasks.workunit.client.1.smithi111.stdout:4/302: rename d0/l24 to d0/dc/d4c/l5c 0 2024-03-23T11:47:31.053 INFO:tasks.workunit.client.0.smithi063.stdout:6/231: readlink d1/dc/d3a/l2e 0 2024-03-23T11:47:31.055 INFO:tasks.workunit.client.1.smithi111.stdout:4/303: write d0/dc/f55 [1430235,3847] 0 2024-03-23T11:47:31.056 INFO:tasks.workunit.client.1.smithi111.stdout:9/381: symlink d0/d4/d10/d29/d7f/l82 0 2024-03-23T11:47:31.057 INFO:tasks.workunit.client.1.smithi111.stdout:9/382: dread - d0/d4/d10/d29/f67 zero size 2024-03-23T11:47:31.058 INFO:tasks.workunit.client.0.smithi063.stdout:4/278: sync 2024-03-23T11:47:31.058 INFO:tasks.workunit.client.0.smithi063.stdout:3/394: chown d13/d22/f63 286736 1 2024-03-23T11:47:31.058 INFO:tasks.workunit.client.0.smithi063.stdout:5/180: rename d2/f25 to d2/d2f/f3f 0 2024-03-23T11:47:31.059 INFO:tasks.workunit.client.1.smithi111.stdout:6/374: truncate d2/ff 4512586 0 2024-03-23T11:47:31.059 INFO:tasks.workunit.client.1.smithi111.stdout:8/266: symlink d5/d40/l56 0 2024-03-23T11:47:31.059 INFO:tasks.workunit.client.0.smithi063.stdout:4/279: stat d5/l44 0 2024-03-23T11:47:31.062 INFO:tasks.workunit.client.1.smithi111.stdout:9/383: write d0/d4/d1c/d2d/d3b/d42/d4f/f61 [352331,66171] 0 2024-03-23T11:47:31.063 INFO:tasks.workunit.client.1.smithi111.stdout:3/280: unlink d0/d7/f10 0 2024-03-23T11:47:31.065 INFO:tasks.workunit.client.1.smithi111.stdout:9/384: fsync d0/d4/d1c/d36/f11 0 2024-03-23T11:47:31.066 INFO:tasks.workunit.client.0.smithi063.stdout:8/546: symlink d6/d1f/d54/lb9 0 2024-03-23T11:47:31.067 INFO:tasks.workunit.client.0.smithi063.stdout:4/280: write d5/d1e/d2b/f49 [682893,86499] 0 2024-03-23T11:47:31.070 INFO:tasks.workunit.client.1.smithi111.stdout:5/371: rename d1/d3/d6/d7/f3e to d1/d20/d5c/f7e 0 2024-03-23T11:47:31.070 INFO:tasks.workunit.client.1.smithi111.stdout:4/304: mknod d0/c5d 0 2024-03-23T11:47:31.071 INFO:tasks.workunit.client.0.smithi063.stdout:3/395: creat d13/d22/d2d/d3f/d5b/f77 x:0 0 0 2024-03-23T11:47:31.072 INFO:tasks.workunit.client.0.smithi063.stdout:8/547: stat d6/d1f/d54/d5c/l8f 0 2024-03-23T11:47:31.072 INFO:tasks.workunit.client.1.smithi111.stdout:2/364: creat d2/de/f84 x:0 0 0 2024-03-23T11:47:31.074 INFO:tasks.workunit.client.0.smithi063.stdout:7/284: rename d5/d16 to d5/d5c 0 2024-03-23T11:47:31.075 INFO:tasks.workunit.client.1.smithi111.stdout:8/267: creat d5/d26/d2f/d42/f57 x:0 0 0 2024-03-23T11:47:31.075 INFO:tasks.workunit.client.1.smithi111.stdout:2/365: fsync d2/d11/d17/d3e/d6b/d2e/f82 0 2024-03-23T11:47:31.076 INFO:tasks.workunit.client.1.smithi111.stdout:5/372: read d1/d5d/f5f [391664,98036] 0 2024-03-23T11:47:31.077 INFO:tasks.workunit.client.0.smithi063.stdout:0/285: truncate d2/d4/dd/d1f/f2d 616682 0 2024-03-23T11:47:31.081 INFO:tasks.workunit.client.0.smithi063.stdout:3/396: readlink d13/d22/d2d/l18 0 2024-03-23T11:47:31.081 INFO:tasks.workunit.client.1.smithi111.stdout:3/281: creat d0/d1a/d2b/d38/f51 x:0 0 0 2024-03-23T11:47:31.083 INFO:tasks.workunit.client.1.smithi111.stdout:3/282: write d0/d1a/d4f/f4e [508801,99046] 0 2024-03-23T11:47:31.084 INFO:tasks.workunit.client.1.smithi111.stdout:9/385: write d0/d4/d1c/d36/f7e [463292,82188] 0 2024-03-23T11:47:31.085 INFO:tasks.workunit.client.0.smithi063.stdout:5/181: mknod d2/d7/d8/de/d35/c40 0 2024-03-23T11:47:31.090 INFO:tasks.workunit.client.1.smithi111.stdout:6/375: symlink d2/d73/l7f 0 2024-03-23T11:47:31.091 INFO:tasks.workunit.client.1.smithi111.stdout:5/373: mknod d1/d3/d3a/c7f 0 2024-03-23T11:47:31.093 INFO:tasks.workunit.client.1.smithi111.stdout:5/374: read d1/d20/f37 [3423331,103290] 0 2024-03-23T11:47:31.094 INFO:tasks.workunit.client.1.smithi111.stdout:0/313: symlink d5/db/d1f/l5e 0 2024-03-23T11:47:31.094 INFO:tasks.workunit.client.0.smithi063.stdout:1/240: getdents d3/df 0 2024-03-23T11:47:31.095 INFO:tasks.workunit.client.0.smithi063.stdout:5/182: readlink d2/d17/l26 0 2024-03-23T11:47:31.096 INFO:tasks.workunit.client.1.smithi111.stdout:8/268: truncate d5/d1b/f2b 436326 0 2024-03-23T11:47:31.097 INFO:tasks.workunit.client.0.smithi063.stdout:9/269: dwrite d4/d4c/f41 [0,4194304] 0 2024-03-23T11:47:31.099 INFO:tasks.workunit.client.0.smithi063.stdout:6/232: dwrite d1/d13/f16 [0,4194304] 0 2024-03-23T11:47:31.100 INFO:tasks.workunit.client.1.smithi111.stdout:4/305: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:31.100 INFO:tasks.workunit.client.1.smithi111.stdout:3/283: creat d0/d7/d19/f52 x:0 0 0 2024-03-23T11:47:31.102 INFO:tasks.workunit.client.1.smithi111.stdout:9/386: creat d0/d4/f83 x:0 0 0 2024-03-23T11:47:31.102 INFO:tasks.workunit.client.1.smithi111.stdout:3/284: truncate d0/f39 481166 0 2024-03-23T11:47:31.104 INFO:tasks.workunit.client.0.smithi063.stdout:5/183: stat d2/d7/d8/de/d2a 0 2024-03-23T11:47:31.107 INFO:tasks.workunit.client.0.smithi063.stdout:0/286: unlink d2/d14/d36/l3a 0 2024-03-23T11:47:31.107 INFO:tasks.workunit.client.0.smithi063.stdout:6/233: write d1/dc/d3a/f3f [4916034,28430] 0 2024-03-23T11:47:31.110 INFO:tasks.workunit.client.1.smithi111.stdout:0/314: fdatasync d5/db/d1f/f43 0 2024-03-23T11:47:31.110 INFO:tasks.workunit.client.0.smithi063.stdout:9/270: chown d4/d4c/d19/f2d 90495 1 2024-03-23T11:47:31.111 INFO:tasks.workunit.client.0.smithi063.stdout:2/251: rename d2/d27 to d2/d23/d2e/d52 0 2024-03-23T11:47:31.112 INFO:tasks.workunit.client.1.smithi111.stdout:2/366: unlink d2/d32/d4b/d52/l68 0 2024-03-23T11:47:31.112 INFO:tasks.workunit.client.0.smithi063.stdout:8/548: mkdir d6/dd/d59/daf/dba 0 2024-03-23T11:47:31.113 INFO:tasks.workunit.client.1.smithi111.stdout:0/315: dread d5/db/d1f/d39/f3f [0,4194304] 0 2024-03-23T11:47:31.113 INFO:tasks.workunit.client.0.smithi063.stdout:6/234: chown d1/dc/f1f 33691 1 2024-03-23T11:47:31.114 INFO:tasks.workunit.client.1.smithi111.stdout:2/367: chown d2/d7/f56 0 1 2024-03-23T11:47:31.116 INFO:tasks.workunit.client.0.smithi063.stdout:8/549: chown d6/d1c/d32/d37 208 1 2024-03-23T11:47:31.116 INFO:tasks.workunit.client.0.smithi063.stdout:0/287: chown d2/d4/dd/c4e 211655694 1 2024-03-23T11:47:31.121 INFO:tasks.workunit.client.0.smithi063.stdout:6/235: readlink d1/l10 0 2024-03-23T11:47:31.123 INFO:tasks.workunit.client.1.smithi111.stdout:9/387: chown d0/d4/d1c/d36/f5 7532 1 2024-03-23T11:47:31.124 INFO:tasks.workunit.client.0.smithi063.stdout:5/184: creat d2/d7/d8/de/f41 x:0 0 0 2024-03-23T11:47:31.126 INFO:tasks.workunit.client.1.smithi111.stdout:9/388: chown d0/d4/d1c/d36/d28/f68 13 1 2024-03-23T11:47:31.126 INFO:tasks.workunit.client.0.smithi063.stdout:7/285: symlink d5/d6/d19/d1f/d47/d4f/d5b/l5d 0 2024-03-23T11:47:31.127 INFO:tasks.workunit.client.0.smithi063.stdout:9/271: unlink d4/d4c/d44/f51 0 2024-03-23T11:47:31.127 INFO:tasks.workunit.client.0.smithi063.stdout:5/185: chown d2/d17/f19 151342629 1 2024-03-23T11:47:31.128 INFO:tasks.workunit.client.0.smithi063.stdout:3/397: dwrite ff [4194304,4194304] 0 2024-03-23T11:47:31.128 INFO:tasks.workunit.client.1.smithi111.stdout:5/375: mknod d1/d20/c80 0 2024-03-23T11:47:31.134 INFO:tasks.workunit.client.1.smithi111.stdout:0/316: dwrite d5/db/f16 [0,4194304] 0 2024-03-23T11:47:31.134 INFO:tasks.workunit.client.1.smithi111.stdout:3/285: mkdir d0/d53 0 2024-03-23T11:47:31.135 INFO:tasks.workunit.client.1.smithi111.stdout:4/306: dwrite d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:31.135 INFO:tasks.workunit.client.1.smithi111.stdout:1/321: fdatasync d0/d5/dd/d1e/f21 0 2024-03-23T11:47:31.144 INFO:tasks.workunit.client.0.smithi063.stdout:4/281: write d5/fc [125966,104164] 0 2024-03-23T11:47:31.149 INFO:tasks.workunit.client.1.smithi111.stdout:2/368: creat d2/d32/d4b/d52/d58/f85 x:0 0 0 2024-03-23T11:47:31.150 INFO:tasks.workunit.client.0.smithi063.stdout:5/186: unlink d2/d2f/l32 0 2024-03-23T11:47:31.152 INFO:tasks.workunit.client.1.smithi111.stdout:9/389: dwrite d0/d4/d1c/d36/d28/f68 [0,4194304] 0 2024-03-23T11:47:31.156 INFO:tasks.workunit.client.1.smithi111.stdout:0/317: creat d5/db/d4d/f5f x:0 0 0 2024-03-23T11:47:31.158 INFO:tasks.workunit.client.0.smithi063.stdout:5/187: write d2/f6 [959958,33408] 0 2024-03-23T11:47:31.159 INFO:tasks.workunit.client.1.smithi111.stdout:9/390: fsync d0/d4/d10/d29/d5b/f7b 0 2024-03-23T11:47:31.161 INFO:tasks.workunit.client.1.smithi111.stdout:5/376: creat d1/d5d/d59/d6c/f81 x:0 0 0 2024-03-23T11:47:31.163 INFO:tasks.workunit.client.0.smithi063.stdout:3/398: mkdir d13/d22/d2d/d78 0 2024-03-23T11:47:31.165 INFO:tasks.workunit.client.1.smithi111.stdout:4/307: dwrite d0/dc/d4c/d2a/f38 [0,4194304] 0 2024-03-23T11:47:31.166 INFO:tasks.workunit.client.0.smithi063.stdout:8/550: dwrite d6/d1c/f2d [4194304,4194304] 0 2024-03-23T11:47:31.166 INFO:tasks.workunit.client.0.smithi063.stdout:0/288: creat d2/d4/dd/d28/d46/d51/f65 x:0 0 0 2024-03-23T11:47:31.167 INFO:tasks.workunit.client.1.smithi111.stdout:0/318: mknod d5/db/d1f/d39/c60 0 2024-03-23T11:47:31.169 INFO:tasks.workunit.client.1.smithi111.stdout:5/377: dread - d1/d3/d6/d7/d63/d2c/f51 zero size 2024-03-23T11:47:31.173 INFO:tasks.workunit.client.0.smithi063.stdout:4/282: creat d5/d1e/f51 x:0 0 0 2024-03-23T11:47:31.175 INFO:tasks.workunit.client.0.smithi063.stdout:5/188: dread d2/d28/f2d [0,4194304] 0 2024-03-23T11:47:31.178 INFO:tasks.workunit.client.1.smithi111.stdout:5/378: write d1/d5d/d59/d6c/f81 [1006099,8362] 0 2024-03-23T11:47:31.179 INFO:tasks.workunit.client.0.smithi063.stdout:2/252: truncate d2/f37 924730 0 2024-03-23T11:47:31.179 INFO:tasks.workunit.client.1.smithi111.stdout:1/322: dread d0/d6/d48/f5d [0,4194304] 0 2024-03-23T11:47:31.180 INFO:tasks.workunit.client.1.smithi111.stdout:2/369: dwrite d2/d7/f56 [0,4194304] 0 2024-03-23T11:47:31.181 INFO:tasks.workunit.client.0.smithi063.stdout:0/289: creat d2/d14/f66 x:0 0 0 2024-03-23T11:47:31.181 INFO:tasks.workunit.client.0.smithi063.stdout:4/283: chown d5/da/d20/d4b/d16/d32/l4a 11 1 2024-03-23T11:47:31.181 INFO:tasks.workunit.client.1.smithi111.stdout:1/323: chown d0/d6/f42 1021202 1 2024-03-23T11:47:31.183 INFO:tasks.workunit.client.1.smithi111.stdout:9/391: getdents d0/d31/d41 0 2024-03-23T11:47:31.184 INFO:tasks.workunit.client.1.smithi111.stdout:1/324: readlink d0/d5/dd/d1e/d54/l5f 0 2024-03-23T11:47:31.186 INFO:tasks.workunit.client.1.smithi111.stdout:9/392: fdatasync d0/d4/d1c/d2d/d3b/d42/d4f/f5c 0 2024-03-23T11:47:31.187 INFO:tasks.workunit.client.0.smithi063.stdout:9/272: dwrite d4/d4c/d19/f2d [0,4194304] 0 2024-03-23T11:47:31.189 INFO:tasks.workunit.client.1.smithi111.stdout:0/319: dread d5/fc [0,4194304] 0 2024-03-23T11:47:31.191 INFO:tasks.workunit.client.1.smithi111.stdout:5/379: fdatasync d1/d3/d6/d7/f1f 0 2024-03-23T11:47:31.191 INFO:tasks.workunit.client.1.smithi111.stdout:2/370: chown d2/d11/f13 23270 1 2024-03-23T11:47:31.192 INFO:tasks.workunit.client.1.smithi111.stdout:1/325: symlink d0/d5/dd/d1e/d54/l72 0 2024-03-23T11:47:31.192 INFO:tasks.workunit.client.1.smithi111.stdout:0/320: read d5/db/f26 [6788013,72907] 0 2024-03-23T11:47:31.196 INFO:tasks.workunit.client.1.smithi111.stdout:0/321: chown d5/db/d4d/l5c 317854 1 2024-03-23T11:47:31.196 INFO:tasks.workunit.client.1.smithi111.stdout:2/371: mknod d2/d32/c86 0 2024-03-23T11:47:31.198 INFO:tasks.workunit.client.0.smithi063.stdout:5/189: creat d2/d17/f42 x:0 0 0 2024-03-23T11:47:31.198 INFO:tasks.workunit.client.0.smithi063.stdout:6/236: getdents d1/d2a 0 2024-03-23T11:47:31.199 INFO:tasks.workunit.client.1.smithi111.stdout:2/372: stat d2/d7/l2b 0 2024-03-23T11:47:31.199 INFO:tasks.workunit.client.0.smithi063.stdout:3/399: dwrite f9 [8388608,4194304] 0 2024-03-23T11:47:31.202 INFO:tasks.workunit.client.0.smithi063.stdout:7/286: link d5/d6/d19/d1f/c2d d5/d6/d19/d1f/d47/d4f/d5b/c5e 0 2024-03-23T11:47:31.202 INFO:tasks.workunit.client.0.smithi063.stdout:5/190: write d2/d17/f19 [2555884,3052] 0 2024-03-23T11:47:31.202 INFO:tasks.workunit.client.0.smithi063.stdout:6/237: fdatasync d1/f35 0 2024-03-23T11:47:31.203 INFO:tasks.workunit.client.1.smithi111.stdout:5/380: rename d1/d55 to d1/d3/d6/d7/d63/d2c/d82 0 2024-03-23T11:47:31.207 INFO:tasks.workunit.client.0.smithi063.stdout:4/284: readlink d5/d1e/d2b/l3a 0 2024-03-23T11:47:31.208 INFO:tasks.workunit.client.0.smithi063.stdout:0/290: mkdir d2/d12/d35/d67 0 2024-03-23T11:47:31.212 INFO:tasks.workunit.client.0.smithi063.stdout:7/287: dread - d5/d6/f5a zero size 2024-03-23T11:47:31.213 INFO:tasks.workunit.client.1.smithi111.stdout:3/286: truncate d0/d7/f1e 2066297 0 2024-03-23T11:47:31.215 INFO:tasks.workunit.client.1.smithi111.stdout:1/326: dread d0/d2b/f45 [4194304,4194304] 0 2024-03-23T11:47:31.215 INFO:tasks.workunit.client.1.smithi111.stdout:2/373: getdents d2/d11/d17/d3e/d6b/d2e 0 2024-03-23T11:47:31.221 INFO:tasks.workunit.client.1.smithi111.stdout:4/308: dread d0/d7/da/f13 [0,4194304] 0 2024-03-23T11:47:31.221 INFO:tasks.workunit.client.0.smithi063.stdout:4/285: write d5/da/d20/d4b/d1f/f39 [503679,37620] 0 2024-03-23T11:47:31.221 INFO:tasks.workunit.client.0.smithi063.stdout:0/291: dread - d2/d4/dd/d28/d46/f5b zero size 2024-03-23T11:47:31.222 INFO:tasks.workunit.client.1.smithi111.stdout:4/309: chown d0/f3a 24389 1 2024-03-23T11:47:31.223 INFO:tasks.workunit.client.0.smithi063.stdout:6/238: write d1/dc/f32 [488481,20126] 0 2024-03-23T11:47:31.223 INFO:tasks.workunit.client.1.smithi111.stdout:1/327: rename d0/l30 to d0/d6/d48/l73 0 2024-03-23T11:47:31.223 INFO:tasks.workunit.client.0.smithi063.stdout:3/400: mknod d13/d22/d2d/d3f/c79 0 2024-03-23T11:47:31.226 INFO:tasks.workunit.client.0.smithi063.stdout:9/273: dwrite d4/d4c/d19/f42 [0,4194304] 0 2024-03-23T11:47:31.227 INFO:tasks.workunit.client.1.smithi111.stdout:4/310: write d0/d17/f2c [257160,92656] 0 2024-03-23T11:47:31.227 INFO:tasks.workunit.client.1.smithi111.stdout:1/328: creat d0/d2b/d37/f74 x:0 0 0 2024-03-23T11:47:31.231 INFO:tasks.workunit.client.1.smithi111.stdout:1/329: write d0/fe [1055611,129427] 0 2024-03-23T11:47:31.232 INFO:tasks.workunit.client.0.smithi063.stdout:3/401: dread - d13/d22/d2d/d3f/d58/f76 zero size 2024-03-23T11:47:31.234 INFO:tasks.workunit.client.0.smithi063.stdout:7/288: symlink d5/d6/d19/d1f/d47/d4f/l5f 0 2024-03-23T11:47:31.236 INFO:tasks.workunit.client.1.smithi111.stdout:2/374: dwrite d2/de/f84 [0,4194304] 0 2024-03-23T11:47:31.237 INFO:tasks.workunit.client.0.smithi063.stdout:4/286: read d5/da/d20/d4b/d1f/f3f [233556,52824] 0 2024-03-23T11:47:31.238 INFO:tasks.workunit.client.1.smithi111.stdout:7/287: sync 2024-03-23T11:47:31.238 INFO:tasks.workunit.client.1.smithi111.stdout:2/375: fdatasync d2/de/d12/f43 0 2024-03-23T11:47:31.239 INFO:tasks.workunit.client.1.smithi111.stdout:1/330: creat d0/d2b/f75 x:0 0 0 2024-03-23T11:47:31.240 INFO:tasks.workunit.client.1.smithi111.stdout:7/288: dread - d14/d1b/d1e/f39 zero size 2024-03-23T11:47:31.241 INFO:tasks.workunit.client.1.smithi111.stdout:7/289: stat d14/f36 0 2024-03-23T11:47:31.241 INFO:tasks.workunit.client.1.smithi111.stdout:1/331: chown d0/d5/dd/f6e 1558 1 2024-03-23T11:47:31.244 INFO:tasks.workunit.client.1.smithi111.stdout:2/376: truncate d2/d32/d4b/d52/d58/f85 87496 0 2024-03-23T11:47:31.245 INFO:tasks.workunit.client.1.smithi111.stdout:7/290: symlink d14/d1b/d22/l4e 0 2024-03-23T11:47:31.249 INFO:tasks.workunit.client.0.smithi063.stdout:0/292: creat d2/d12/f68 x:0 0 0 2024-03-23T11:47:31.249 INFO:tasks.workunit.client.1.smithi111.stdout:9/393: dread d0/d4/d1c/d2d/d3b/d42/d4f/f61 [0,4194304] 0 2024-03-23T11:47:31.250 INFO:tasks.workunit.client.1.smithi111.stdout:7/291: symlink d14/d1b/d1e/d2c/d2e/l4f 0 2024-03-23T11:47:31.252 INFO:tasks.workunit.client.0.smithi063.stdout:3/402: truncate d13/d22/d2d/f53 1601300 0 2024-03-23T11:47:31.253 INFO:tasks.workunit.client.1.smithi111.stdout:0/322: truncate d5/db/f26 7669992 0 2024-03-23T11:47:31.254 INFO:tasks.workunit.client.0.smithi063.stdout:2/253: dread d2/d23/d2e/d52/f33 [0,4194304] 0 2024-03-23T11:47:31.255 INFO:tasks.workunit.client.0.smithi063.stdout:6/239: unlink d1/dc/d3a/f37 0 2024-03-23T11:47:31.256 INFO:tasks.workunit.client.0.smithi063.stdout:5/191: mkdir d2/d7/d8/de/d2a/d3b/d43 0 2024-03-23T11:47:31.257 INFO:tasks.workunit.client.1.smithi111.stdout:1/332: getdents d0/d6 0 2024-03-23T11:47:31.259 INFO:tasks.workunit.client.0.smithi063.stdout:8/551: dwrite d6/d1c/d27/f8d [0,4194304] 0 2024-03-23T11:47:31.261 INFO:tasks.workunit.client.1.smithi111.stdout:0/323: rmdir d5/db/d1f 39 2024-03-23T11:47:31.262 INFO:tasks.workunit.client.0.smithi063.stdout:0/293: write d2/d4/dd/d1f/f20 [3485675,36350] 0 2024-03-23T11:47:31.263 INFO:tasks.workunit.client.1.smithi111.stdout:2/377: dwrite d2/d7/da/d19/f49 [0,4194304] 0 2024-03-23T11:47:31.265 INFO:tasks.workunit.client.1.smithi111.stdout:1/333: read d0/fe [1231873,125705] 0 2024-03-23T11:47:31.268 INFO:tasks.workunit.client.0.smithi063.stdout:7/289: rename d5/d26/d32/d3a to d5/d6/d19/d1f/d37/da/d60 0 2024-03-23T11:47:31.270 INFO:tasks.workunit.client.0.smithi063.stdout:6/240: unlink d1/f7 0 2024-03-23T11:47:31.272 INFO:tasks.workunit.client.1.smithi111.stdout:5/381: dwrite d1/d20/f37 [0,4194304] 0 2024-03-23T11:47:31.273 INFO:tasks.workunit.client.0.smithi063.stdout:4/287: mknod d5/c52 0 2024-03-23T11:47:31.275 INFO:tasks.workunit.client.1.smithi111.stdout:0/324: truncate d5/db/d1f/d47/f5a 957579 0 2024-03-23T11:47:31.276 INFO:tasks.workunit.client.1.smithi111.stdout:1/334: write d0/d5/dd/d1e/d54/f60 [443472,67531] 0 2024-03-23T11:47:31.278 INFO:tasks.workunit.client.0.smithi063.stdout:9/274: dwrite d4/f14 [8388608,4194304] 0 2024-03-23T11:47:31.279 INFO:tasks.workunit.client.0.smithi063.stdout:7/290: write d5/d6/d19/d1f/d37/d20/f53 [97447,121401] 0 2024-03-23T11:47:31.280 INFO:tasks.workunit.client.0.smithi063.stdout:4/288: dread - d5/da/d20/d4b/d1f/f43 zero size 2024-03-23T11:47:31.284 INFO:tasks.workunit.client.1.smithi111.stdout:0/325: fsync d5/db/d1f/d32/f3b 0 2024-03-23T11:47:31.285 INFO:tasks.workunit.client.0.smithi063.stdout:8/552: mkdir d6/d1f/dbb 0 2024-03-23T11:47:31.291 INFO:tasks.workunit.client.1.smithi111.stdout:4/311: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:31.293 INFO:tasks.workunit.client.0.smithi063.stdout:2/254: mknod d2/de/c53 0 2024-03-23T11:47:31.293 INFO:tasks.workunit.client.0.smithi063.stdout:6/241: symlink d1/d2a/l47 0 2024-03-23T11:47:31.296 INFO:tasks.workunit.client.1.smithi111.stdout:5/382: creat d1/d20/f83 x:0 0 0 2024-03-23T11:47:31.296 INFO:tasks.workunit.client.1.smithi111.stdout:7/292: dwrite d14/d1b/d1e/d2c/d2e/d33/f3d [0,4194304] 0 2024-03-23T11:47:31.296 INFO:tasks.workunit.client.1.smithi111.stdout:2/378: write d2/d7/da/d19/f6a [2056342,15849] 0 2024-03-23T11:47:31.298 INFO:tasks.workunit.client.1.smithi111.stdout:1/335: symlink d0/d2b/d69/l76 0 2024-03-23T11:47:31.298 INFO:tasks.workunit.client.1.smithi111.stdout:7/293: fdatasync f1 0 2024-03-23T11:47:31.302 INFO:tasks.workunit.client.0.smithi063.stdout:7/291: symlink d5/d6/d19/d1f/d47/d4f/d5b/l61 0 2024-03-23T11:47:31.303 INFO:tasks.workunit.client.1.smithi111.stdout:1/336: fdatasync d0/d2b/d37/f4f 0 2024-03-23T11:47:31.303 INFO:tasks.workunit.client.1.smithi111.stdout:0/326: truncate d5/f2d 1042178 0 2024-03-23T11:47:31.304 INFO:tasks.workunit.client.1.smithi111.stdout:4/312: write d0/d17/f3d [295223,28614] 0 2024-03-23T11:47:31.305 INFO:tasks.workunit.client.0.smithi063.stdout:9/275: dwrite d4/d4c/d19/f1b [0,4194304] 0 2024-03-23T11:47:31.305 INFO:tasks.workunit.client.1.smithi111.stdout:5/383: read d1/d3/d6/d7/f53 [2027239,112994] 0 2024-03-23T11:47:31.305 INFO:tasks.workunit.client.1.smithi111.stdout:1/337: chown d0/f1 2766 1 2024-03-23T11:47:31.308 INFO:tasks.workunit.client.1.smithi111.stdout:2/379: creat d2/d11/d17/d3e/d6b/f87 x:0 0 0 2024-03-23T11:47:31.309 INFO:tasks.workunit.client.0.smithi063.stdout:0/294: getdents d2/d48/d5a 0 2024-03-23T11:47:31.310 INFO:tasks.workunit.client.0.smithi063.stdout:8/553: creat d6/dd/d59/d7f/dae/fbc x:0 0 0 2024-03-23T11:47:31.315 INFO:tasks.workunit.client.1.smithi111.stdout:2/380: dread d2/d7/da/f3b [0,4194304] 0 2024-03-23T11:47:31.319 INFO:tasks.workunit.client.1.smithi111.stdout:7/294: dread d14/d1b/d1e/d2c/d2e/d33/f3d [0,4194304] 0 2024-03-23T11:47:31.320 INFO:tasks.workunit.client.0.smithi063.stdout:5/192: link d2/d28/f2d d2/f44 0 2024-03-23T11:47:31.321 INFO:tasks.workunit.client.0.smithi063.stdout:7/292: unlink d5/d6/d19/d1f/c41 0 2024-03-23T11:47:31.322 INFO:tasks.workunit.client.0.smithi063.stdout:2/255: mkdir d2/d46/d54 0 2024-03-23T11:47:31.325 INFO:tasks.workunit.client.0.smithi063.stdout:5/193: write d2/d7/f10 [1736593,13631] 0 2024-03-23T11:47:31.325 INFO:tasks.workunit.client.1.smithi111.stdout:2/381: mknod d2/d32/d4b/c88 0 2024-03-23T11:47:31.325 INFO:tasks.workunit.client.1.smithi111.stdout:7/295: chown f12 17946764 1 2024-03-23T11:47:31.325 INFO:tasks.workunit.client.1.smithi111.stdout:0/327: creat d5/db/d1f/d19/d45/d4b/f61 x:0 0 0 2024-03-23T11:47:31.326 INFO:tasks.workunit.client.0.smithi063.stdout:9/276: creat d4/f60 x:0 0 0 2024-03-23T11:47:31.326 INFO:tasks.workunit.client.1.smithi111.stdout:2/382: truncate d2/d7/f7b 116833 0 2024-03-23T11:47:31.330 INFO:tasks.workunit.client.0.smithi063.stdout:0/295: rename d2/d4/dd/l4d to d2/d14/d36/l69 0 2024-03-23T11:47:31.332 INFO:tasks.workunit.client.1.smithi111.stdout:1/338: write d0/d5/dd/d1e/f36 [68668,86558] 0 2024-03-23T11:47:31.332 INFO:tasks.workunit.client.1.smithi111.stdout:0/328: creat d5/db/d1f/d39/f62 x:0 0 0 2024-03-23T11:47:31.333 INFO:tasks.workunit.client.0.smithi063.stdout:8/554: mknod d6/d1f/d54/d63/d90/d15/d9d/cbd 0 2024-03-23T11:47:31.334 INFO:tasks.workunit.client.1.smithi111.stdout:0/329: chown d5/c10 60858242 1 2024-03-23T11:47:31.334 INFO:tasks.workunit.client.0.smithi063.stdout:6/242: dwrite d1/f35 [0,4194304] 0 2024-03-23T11:47:31.343 INFO:tasks.workunit.client.1.smithi111.stdout:4/313: truncate d0/d7/f1b 7973129 0 2024-03-23T11:47:31.343 INFO:tasks.workunit.client.1.smithi111.stdout:5/384: write d1/d3/d6/d7/f1f [6832728,108971] 0 2024-03-23T11:47:31.343 INFO:tasks.workunit.client.1.smithi111.stdout:0/330: link d5/c10 d5/db/d4d/c63 0 2024-03-23T11:47:31.346 INFO:tasks.workunit.client.0.smithi063.stdout:2/256: creat d2/d23/d2e/f55 x:0 0 0 2024-03-23T11:47:31.346 INFO:tasks.workunit.client.0.smithi063.stdout:7/293: rename d5/d6/d19/d1f/f42 to d5/d6/d39/f62 0 2024-03-23T11:47:31.347 INFO:tasks.workunit.client.1.smithi111.stdout:8/269: sync 2024-03-23T11:47:31.349 INFO:tasks.workunit.client.0.smithi063.stdout:6/243: unlink d1/dc/d3a/l2e 0 2024-03-23T11:47:31.349 INFO:tasks.workunit.client.1.smithi111.stdout:6/376: sync 2024-03-23T11:47:31.350 INFO:tasks.workunit.client.1.smithi111.stdout:7/296: dwrite d14/d1b/d1e/d2c/d42/f4a [0,4194304] 0 2024-03-23T11:47:31.350 INFO:tasks.workunit.client.0.smithi063.stdout:6/244: readlink d1/l10 0 2024-03-23T11:47:31.351 INFO:tasks.workunit.client.1.smithi111.stdout:5/385: dread - d1/d5d/d59/d74/f6b zero size 2024-03-23T11:47:31.353 INFO:tasks.workunit.client.1.smithi111.stdout:2/383: write d2/d11/d17/d3e/d6b/d2e/d3f/f44 [2024313,76498] 0 2024-03-23T11:47:31.353 INFO:tasks.workunit.client.1.smithi111.stdout:0/331: mkdir d5/db/d1f/d39/d64 0 2024-03-23T11:47:31.354 INFO:tasks.workunit.client.0.smithi063.stdout:3/403: truncate d13/d15/f3c 1977782 0 2024-03-23T11:47:31.356 INFO:tasks.workunit.client.1.smithi111.stdout:2/384: write d2/d11/f51 [586984,54978] 0 2024-03-23T11:47:31.358 INFO:tasks.workunit.client.0.smithi063.stdout:0/296: dwrite d2/d12/f21 [0,4194304] 0 2024-03-23T11:47:31.358 INFO:tasks.workunit.client.0.smithi063.stdout:2/257: fdatasync d2/d1c/f20 0 2024-03-23T11:47:31.359 INFO:tasks.workunit.client.1.smithi111.stdout:7/297: mkdir d14/d37/d50 0 2024-03-23T11:47:31.361 INFO:tasks.workunit.client.0.smithi063.stdout:3/404: chown d13/d22/d2d/d3f/c79 2001290 1 2024-03-23T11:47:31.362 INFO:tasks.workunit.client.1.smithi111.stdout:8/270: mkdir d5/d26/d2f/d42/d58 0 2024-03-23T11:47:31.362 INFO:tasks.workunit.client.1.smithi111.stdout:1/339: dwrite d0/d6/f34 [0,4194304] 0 2024-03-23T11:47:31.366 INFO:tasks.workunit.client.1.smithi111.stdout:5/386: mknod d1/d3/d3a/c84 0 2024-03-23T11:47:31.367 INFO:tasks.workunit.client.0.smithi063.stdout:8/555: mkdir d6/dd/d59/db2/dbe 0 2024-03-23T11:47:31.367 INFO:tasks.workunit.client.1.smithi111.stdout:1/340: write d0/d5/f35 [330751,127165] 0 2024-03-23T11:47:31.369 INFO:tasks.workunit.client.1.smithi111.stdout:1/341: stat d0/d5/f5e 0 2024-03-23T11:47:31.371 INFO:tasks.workunit.client.1.smithi111.stdout:4/314: write d0/d7/da/f46 [257845,74413] 0 2024-03-23T11:47:31.371 INFO:tasks.workunit.client.1.smithi111.stdout:0/332: readlink d5/db/d1f/d19/l46 0 2024-03-23T11:47:31.373 INFO:tasks.workunit.client.0.smithi063.stdout:5/194: link d2/d7/f23 d2/d7/d8/de/d2a/d3b/d43/f45 0 2024-03-23T11:47:31.373 INFO:tasks.workunit.client.1.smithi111.stdout:7/298: mkdir d14/d51 0 2024-03-23T11:47:31.376 INFO:tasks.workunit.client.0.smithi063.stdout:9/277: chown d4/f60 1 1 2024-03-23T11:47:31.377 INFO:tasks.workunit.client.1.smithi111.stdout:5/387: readlink d1/d3/d6/l12 0 2024-03-23T11:47:31.379 INFO:tasks.workunit.client.1.smithi111.stdout:2/385: dread d2/de/d12/f2c [0,4194304] 0 2024-03-23T11:47:31.379 INFO:tasks.workunit.client.1.smithi111.stdout:5/388: truncate d1/d20/f72 176185 0 2024-03-23T11:47:31.380 INFO:tasks.workunit.client.1.smithi111.stdout:1/342: dread d0/f1a [0,4194304] 0 2024-03-23T11:47:31.382 INFO:tasks.workunit.client.1.smithi111.stdout:2/386: fdatasync d2/de/f84 0 2024-03-23T11:47:31.384 INFO:tasks.workunit.client.1.smithi111.stdout:1/343: dread - d0/d6/d48/f55 zero size 2024-03-23T11:47:31.386 INFO:tasks.workunit.client.0.smithi063.stdout:4/289: dwrite d5/da/d20/f3c [0,4194304] 0 2024-03-23T11:47:31.389 INFO:tasks.workunit.client.0.smithi063.stdout:7/294: dwrite d5/d6/d19/d1f/d37/f3c [0,4194304] 0 2024-03-23T11:47:31.389 INFO:tasks.workunit.client.0.smithi063.stdout:6/245: creat d1/d2a/d41/d42/f48 x:0 0 0 2024-03-23T11:47:31.392 INFO:tasks.workunit.client.0.smithi063.stdout:4/290: readlink d5/da/d20/d4b/d29/l30 0 2024-03-23T11:47:31.396 INFO:tasks.workunit.client.0.smithi063.stdout:4/291: fdatasync d5/da/d20/d4b/d1f/f26 0 2024-03-23T11:47:31.396 INFO:tasks.workunit.client.0.smithi063.stdout:7/295: chown d5/d6/d19/d1f/d37/d20/f31 5086 1 2024-03-23T11:47:31.399 INFO:tasks.workunit.client.1.smithi111.stdout:0/333: rename d5/l6 to d5/l65 0 2024-03-23T11:47:31.400 INFO:tasks.workunit.client.1.smithi111.stdout:6/377: dread d2/d17/f33 [0,4194304] 0 2024-03-23T11:47:31.401 INFO:tasks.workunit.client.0.smithi063.stdout:4/292: write d5/da/f2a [3169487,27179] 0 2024-03-23T11:47:31.405 INFO:tasks.workunit.client.0.smithi063.stdout:9/278: symlink d4/d31/d43/d49/l61 0 2024-03-23T11:47:31.406 INFO:tasks.workunit.client.1.smithi111.stdout:6/378: fdatasync d2/d1f/d45/f63 0 2024-03-23T11:47:31.408 INFO:tasks.workunit.client.0.smithi063.stdout:4/293: read d5/da/d20/d4b/f42 [4038221,66213] 0 2024-03-23T11:47:31.410 INFO:tasks.workunit.client.1.smithi111.stdout:0/334: creat d5/db/d1f/d48/f66 x:0 0 0 2024-03-23T11:47:31.412 INFO:tasks.workunit.client.1.smithi111.stdout:8/271: truncate d5/f11 369080 0 2024-03-23T11:47:31.416 INFO:tasks.workunit.client.1.smithi111.stdout:7/299: rename f12 to d14/d1b/f52 0 2024-03-23T11:47:31.416 INFO:tasks.workunit.client.0.smithi063.stdout:5/195: rmdir d2/d7/d8 39 2024-03-23T11:47:31.417 INFO:tasks.workunit.client.0.smithi063.stdout:3/405: dread d13/d15/f1c [0,4194304] 0 2024-03-23T11:47:31.422 INFO:tasks.workunit.client.1.smithi111.stdout:1/344: creat d0/d6/d48/d4a/d6c/f77 x:0 0 0 2024-03-23T11:47:31.424 INFO:tasks.workunit.client.1.smithi111.stdout:4/315: getdents d0/dc/d4c 0 2024-03-23T11:47:31.426 INFO:tasks.workunit.client.0.smithi063.stdout:2/258: getdents d2/de/d22 0 2024-03-23T11:47:31.428 INFO:tasks.workunit.client.1.smithi111.stdout:2/387: rename d2/d11/d17/l5e to d2/d11/d17/d3e/d6b/d2e/d76/l89 0 2024-03-23T11:47:31.430 INFO:tasks.workunit.client.0.smithi063.stdout:5/196: mkdir d2/d28/d3a/d46 0 2024-03-23T11:47:31.430 INFO:tasks.workunit.client.1.smithi111.stdout:2/388: write d2/de/d6c/f65 [318234,73111] 0 2024-03-23T11:47:31.431 INFO:tasks.workunit.client.0.smithi063.stdout:9/279: rename d4/d4c/l18 to d4/d4c/d19/l62 0 2024-03-23T11:47:31.432 INFO:tasks.workunit.client.1.smithi111.stdout:6/379: mkdir d2/d80 0 2024-03-23T11:47:31.433 INFO:tasks.workunit.client.1.smithi111.stdout:6/380: truncate d2/d1a/f72 940564 0 2024-03-23T11:47:31.434 INFO:tasks.workunit.client.0.smithi063.stdout:2/259: write d2/de/f2b [1312925,20571] 0 2024-03-23T11:47:31.434 INFO:tasks.workunit.client.0.smithi063.stdout:0/297: dwrite d2/d12/f44 [0,4194304] 0 2024-03-23T11:47:31.436 INFO:tasks.workunit.client.1.smithi111.stdout:4/316: mkdir d0/dc/d1f/d5e 0 2024-03-23T11:47:31.438 INFO:tasks.workunit.client.1.smithi111.stdout:8/272: creat d5/d26/d31/d4d/f59 x:0 0 0 2024-03-23T11:47:31.440 INFO:tasks.workunit.client.1.smithi111.stdout:0/335: rename d5/db/d4d/c63 to d5/db/d4d/c67 0 2024-03-23T11:47:31.441 INFO:tasks.workunit.client.1.smithi111.stdout:8/273: chown d5/d26/d31/d3b/d41/f4e 70668 1 2024-03-23T11:47:31.441 INFO:tasks.workunit.client.0.smithi063.stdout:3/406: creat d13/d22/d2d/d3f/d58/d6e/f7a x:0 0 0 2024-03-23T11:47:31.442 INFO:tasks.workunit.client.1.smithi111.stdout:1/345: symlink d0/d5/dd/d1e/d70/l78 0 2024-03-23T11:47:31.443 INFO:tasks.workunit.client.1.smithi111.stdout:0/336: chown d5/db/d1f/d19/f53 7349631 1 2024-03-23T11:47:31.444 INFO:tasks.workunit.client.1.smithi111.stdout:9/394: sync 2024-03-23T11:47:31.444 INFO:tasks.workunit.client.1.smithi111.stdout:3/287: sync 2024-03-23T11:47:31.444 INFO:tasks.workunit.client.1.smithi111.stdout:8/274: dread - d5/d26/d2f/d42/f57 zero size 2024-03-23T11:47:31.445 INFO:tasks.workunit.client.0.smithi063.stdout:0/298: rmdir d2/d47 39 2024-03-23T11:47:31.447 INFO:tasks.workunit.client.0.smithi063.stdout:2/260: mkdir d2/d23/d56 0 2024-03-23T11:47:31.449 INFO:tasks.workunit.client.1.smithi111.stdout:7/300: creat d14/d1b/f53 x:0 0 0 2024-03-23T11:47:31.451 INFO:tasks.workunit.client.1.smithi111.stdout:6/381: symlink d2/d1a/l81 0 2024-03-23T11:47:31.451 INFO:tasks.workunit.client.1.smithi111.stdout:4/317: dread d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:31.452 INFO:tasks.workunit.client.0.smithi063.stdout:0/299: fdatasync d2/d12/f21 0 2024-03-23T11:47:31.453 INFO:tasks.workunit.client.1.smithi111.stdout:2/389: creat d2/d11/f8a x:0 0 0 2024-03-23T11:47:31.453 INFO:tasks.workunit.client.1.smithi111.stdout:8/275: write d5/d26/d31/d3b/d41/f4e [166354,1185] 0 2024-03-23T11:47:31.454 INFO:tasks.workunit.client.0.smithi063.stdout:8/556: dread d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:31.456 INFO:tasks.workunit.client.1.smithi111.stdout:7/301: truncate d14/d1b/d20/f3a 212053 0 2024-03-23T11:47:31.457 INFO:tasks.workunit.client.0.smithi063.stdout:2/261: write d2/d23/d2e/f4b [53152,66057] 0 2024-03-23T11:47:31.461 INFO:tasks.workunit.client.1.smithi111.stdout:4/318: write d0/d17/f2c [2643360,72618] 0 2024-03-23T11:47:31.462 INFO:tasks.workunit.client.1.smithi111.stdout:6/382: symlink d2/d35/d70/l82 0 2024-03-23T11:47:31.462 INFO:tasks.workunit.client.1.smithi111.stdout:8/276: chown d5/l15 697056078 1 2024-03-23T11:47:31.463 INFO:tasks.workunit.client.0.smithi063.stdout:3/407: mkdir d13/d22/d2d/d3f/d67/d7b 0 2024-03-23T11:47:31.463 INFO:tasks.workunit.client.1.smithi111.stdout:9/395: mknod d0/d4/d1c/d2d/d75/d7a/c84 0 2024-03-23T11:47:31.464 INFO:tasks.workunit.client.0.smithi063.stdout:6/246: dread d1/dc/f1f [0,4194304] 0 2024-03-23T11:47:31.464 INFO:tasks.workunit.client.1.smithi111.stdout:2/390: chown d2/d32/d4b/d52/f73 60882368 1 2024-03-23T11:47:31.464 INFO:tasks.workunit.client.0.smithi063.stdout:4/294: dwrite d5/da/d20/f48 [0,4194304] 0 2024-03-23T11:47:31.465 INFO:tasks.workunit.client.0.smithi063.stdout:0/300: creat d2/d12/d35/f6a x:0 0 0 2024-03-23T11:47:31.466 INFO:tasks.workunit.client.1.smithi111.stdout:3/288: mknod d0/d1a/d2b/d38/c54 0 2024-03-23T11:47:31.471 INFO:tasks.workunit.client.1.smithi111.stdout:2/391: fsync d2/d32/d4b/d52/f73 0 2024-03-23T11:47:31.471 INFO:tasks.workunit.client.1.smithi111.stdout:9/396: truncate d0/d4/f43 366148 0 2024-03-23T11:47:31.472 INFO:tasks.workunit.client.1.smithi111.stdout:4/319: unlink d0/d17/f3d 0 2024-03-23T11:47:31.474 INFO:tasks.workunit.client.0.smithi063.stdout:5/197: dwrite d2/d28/f3d [0,4194304] 0 2024-03-23T11:47:31.475 INFO:tasks.workunit.client.1.smithi111.stdout:9/397: chown d0/d4/fe 128564 1 2024-03-23T11:47:31.478 INFO:tasks.workunit.client.0.smithi063.stdout:7/296: dread d5/d22/f44 [0,4194304] 0 2024-03-23T11:47:31.478 INFO:tasks.workunit.client.0.smithi063.stdout:6/247: readlink d1/d13/l23 0 2024-03-23T11:47:31.478 INFO:tasks.workunit.client.1.smithi111.stdout:0/337: getdents d5/db/d1f 0 2024-03-23T11:47:31.481 INFO:tasks.workunit.client.1.smithi111.stdout:6/383: unlink d2/d1a/d1c/f58 0 2024-03-23T11:47:31.483 INFO:tasks.workunit.client.1.smithi111.stdout:1/346: dwrite d0/d6/d48/f55 [0,4194304] 0 2024-03-23T11:47:31.483 INFO:tasks.workunit.client.0.smithi063.stdout:2/262: dread d2/f1e [0,4194304] 0 2024-03-23T11:47:31.483 INFO:tasks.workunit.client.0.smithi063.stdout:3/408: rename d13/l2e to d13/d22/d2d/d3f/d58/l7c 0 2024-03-23T11:47:31.484 INFO:tasks.workunit.client.0.smithi063.stdout:3/409: chown d13/d15/f1f 445 1 2024-03-23T11:47:31.485 INFO:tasks.workunit.client.1.smithi111.stdout:1/347: readlink d0/d5/l7 0 2024-03-23T11:47:31.486 INFO:tasks.workunit.client.0.smithi063.stdout:1/241: sync 2024-03-23T11:47:31.491 INFO:tasks.workunit.client.1.smithi111.stdout:8/277: rename d5/d27 to d5/d32/d35/d46/d4b/d51/d5a 0 2024-03-23T11:47:31.492 INFO:tasks.workunit.client.1.smithi111.stdout:1/348: write d0/f20 [1844917,57523] 0 2024-03-23T11:47:31.493 INFO:tasks.workunit.client.0.smithi063.stdout:8/557: fdatasync d6/dd/df/f60 0 2024-03-23T11:47:31.493 INFO:tasks.workunit.client.0.smithi063.stdout:0/301: creat d2/d4/d63/f6b x:0 0 0 2024-03-23T11:47:31.493 INFO:tasks.workunit.client.0.smithi063.stdout:3/410: write d13/d22/f3d [1170399,49632] 0 2024-03-23T11:47:31.493 INFO:tasks.workunit.client.0.smithi063.stdout:6/248: mkdir d1/dc/d21/d49 0 2024-03-23T11:47:31.495 INFO:tasks.workunit.client.1.smithi111.stdout:3/289: dwrite d0/d1a/d2d/f45 [0,4194304] 0 2024-03-23T11:47:31.500 INFO:tasks.workunit.client.0.smithi063.stdout:6/249: truncate d1/dc/d3a/f30 399137 0 2024-03-23T11:47:31.501 INFO:tasks.workunit.client.1.smithi111.stdout:6/384: stat d2/d1a/d1c/d2c/d51/f2f 0 2024-03-23T11:47:31.506 INFO:tasks.workunit.client.0.smithi063.stdout:5/198: mkdir d2/d7/d8/de/d47 0 2024-03-23T11:47:31.509 INFO:tasks.workunit.client.1.smithi111.stdout:2/392: mknod d2/d11/c8b 0 2024-03-23T11:47:31.514 INFO:tasks.workunit.client.1.smithi111.stdout:3/290: read d0/d1a/d1d/d26/f46 [2296519,76486] 0 2024-03-23T11:47:31.516 INFO:tasks.workunit.client.0.smithi063.stdout:3/411: mkdir d13/d22/d2d/d7d 0 2024-03-23T11:47:31.516 INFO:tasks.workunit.client.0.smithi063.stdout:0/302: dread - d2/d12/d32/f5c zero size 2024-03-23T11:47:31.516 INFO:tasks.workunit.client.1.smithi111.stdout:4/320: dwrite d0/d17/f2c [0,4194304] 0 2024-03-23T11:47:31.521 INFO:tasks.workunit.client.1.smithi111.stdout:1/349: mkdir d0/d5/dd/d1e/d63/d79 0 2024-03-23T11:47:31.522 INFO:tasks.workunit.client.1.smithi111.stdout:7/302: dread d14/d1b/f2d [0,4194304] 0 2024-03-23T11:47:31.522 INFO:tasks.workunit.client.1.smithi111.stdout:4/321: write d0/dc/f55 [1566458,4227] 0 2024-03-23T11:47:31.522 INFO:tasks.workunit.client.0.smithi063.stdout:9/280: dread d4/f47 [0,4194304] 0 2024-03-23T11:47:31.523 INFO:tasks.workunit.client.0.smithi063.stdout:0/303: fsync d2/d4/f6 0 2024-03-23T11:47:31.523 INFO:tasks.workunit.client.0.smithi063.stdout:3/412: stat d13/d22/d2d/d3e/l52 0 2024-03-23T11:47:31.523 INFO:tasks.workunit.client.0.smithi063.stdout:5/199: symlink d2/d28/d3a/l48 0 2024-03-23T11:47:31.523 INFO:tasks.workunit.client.1.smithi111.stdout:1/350: chown d0/f3d 12135 1 2024-03-23T11:47:31.524 INFO:tasks.workunit.client.0.smithi063.stdout:6/250: dread d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:31.525 INFO:tasks.workunit.client.0.smithi063.stdout:8/558: creat d6/fbf x:0 0 0 2024-03-23T11:47:31.525 INFO:tasks.workunit.client.1.smithi111.stdout:1/351: fdatasync d0/d5/f2f 0 2024-03-23T11:47:31.526 INFO:tasks.workunit.client.1.smithi111.stdout:0/338: getdents d5/db/d1f/d32 0 2024-03-23T11:47:31.530 INFO:tasks.workunit.client.0.smithi063.stdout:6/251: truncate d1/dc/fd 4905297 0 2024-03-23T11:47:31.531 INFO:tasks.workunit.client.0.smithi063.stdout:7/297: truncate d5/d22/f44 286866 0 2024-03-23T11:47:31.532 INFO:tasks.workunit.client.0.smithi063.stdout:3/413: mkdir d13/d15/d7e 0 2024-03-23T11:47:31.533 INFO:tasks.workunit.client.1.smithi111.stdout:0/339: read d5/db/f38 [2056048,34662] 0 2024-03-23T11:47:31.537 INFO:tasks.workunit.client.0.smithi063.stdout:0/304: creat d2/d12/d32/f6c x:0 0 0 2024-03-23T11:47:31.537 INFO:tasks.workunit.client.1.smithi111.stdout:2/393: dwrite d2/d11/d17/d3e/f4e [0,4194304] 0 2024-03-23T11:47:31.538 INFO:tasks.workunit.client.0.smithi063.stdout:5/200: dread d2/d7/d8/d1d/f1f [0,4194304] 0 2024-03-23T11:47:31.539 INFO:tasks.workunit.client.1.smithi111.stdout:7/303: mkdir d14/d37/d54 0 2024-03-23T11:47:31.544 INFO:tasks.workunit.client.1.smithi111.stdout:2/394: dread d2/d7/da/f3b [0,4194304] 0 2024-03-23T11:47:31.546 INFO:tasks.workunit.client.1.smithi111.stdout:6/385: dwrite d2/d1a/f5e [0,4194304] 0 2024-03-23T11:47:31.546 INFO:tasks.workunit.client.1.smithi111.stdout:4/322: readlink d0/dc/d1f/l4e 0 2024-03-23T11:47:31.548 INFO:tasks.workunit.client.0.smithi063.stdout:4/295: dread d5/da/d20/d4b/f31 [0,4194304] 0 2024-03-23T11:47:31.549 INFO:tasks.workunit.client.1.smithi111.stdout:1/352: dwrite d0/d2b/f49 [4194304,4194304] 0 2024-03-23T11:47:31.550 INFO:tasks.workunit.client.0.smithi063.stdout:6/252: write d1/dc/d3a/f30 [1325893,9352] 0 2024-03-23T11:47:31.552 INFO:tasks.workunit.client.1.smithi111.stdout:0/340: rename d5/db/l11 to d5/db/d1f/l68 0 2024-03-23T11:47:31.553 INFO:tasks.workunit.client.0.smithi063.stdout:0/305: mkdir d2/d12/d32/d6d 0 2024-03-23T11:47:31.554 INFO:tasks.workunit.client.0.smithi063.stdout:4/296: readlink d5/d1e/d2b/l3a 0 2024-03-23T11:47:31.554 INFO:tasks.workunit.client.1.smithi111.stdout:2/395: mkdir d2/d11/d17/d3e/d5d/d8c 0 2024-03-23T11:47:31.555 INFO:tasks.workunit.client.1.smithi111.stdout:4/323: truncate d0/d7/da/f5b 206875 0 2024-03-23T11:47:31.556 INFO:tasks.workunit.client.0.smithi063.stdout:6/253: creat d1/d2a/f4a x:0 0 0 2024-03-23T11:47:31.557 INFO:tasks.workunit.client.1.smithi111.stdout:8/278: write d5/d1b/f2b [1415906,99714] 0 2024-03-23T11:47:31.558 INFO:tasks.workunit.client.1.smithi111.stdout:6/386: creat d2/d1a/d1c/d2c/f83 x:0 0 0 2024-03-23T11:47:31.558 INFO:tasks.workunit.client.1.smithi111.stdout:0/341: creat d5/db/d1f/d19/d45/d4b/f69 x:0 0 0 2024-03-23T11:47:31.559 INFO:tasks.workunit.client.0.smithi063.stdout:9/281: dwrite d4/d2c/f3f [4194304,4194304] 0 2024-03-23T11:47:31.559 INFO:tasks.workunit.client.1.smithi111.stdout:1/353: mknod d0/d5/c7a 0 2024-03-23T11:47:31.560 INFO:tasks.workunit.client.1.smithi111.stdout:0/342: dread - d5/db/d1f/f43 zero size 2024-03-23T11:47:31.563 INFO:tasks.workunit.client.1.smithi111.stdout:1/354: write d0/d6/f34 [2290809,127162] 0 2024-03-23T11:47:31.565 INFO:tasks.workunit.client.1.smithi111.stdout:1/355: fsync d0/d6/d48/f55 0 2024-03-23T11:47:31.565 INFO:tasks.workunit.client.0.smithi063.stdout:3/414: rename d13/l21 to d13/d22/l7f 0 2024-03-23T11:47:31.566 INFO:tasks.workunit.client.1.smithi111.stdout:4/324: symlink d0/dc/d4c/d4b/d4f/l5f 0 2024-03-23T11:47:31.566 INFO:tasks.workunit.client.0.smithi063.stdout:1/242: dread d3/f7 [4194304,4194304] 0 2024-03-23T11:47:31.567 INFO:tasks.workunit.client.1.smithi111.stdout:7/304: rmdir d14/d37 39 2024-03-23T11:47:31.568 INFO:tasks.workunit.client.0.smithi063.stdout:6/254: readlink d1/lb 0 2024-03-23T11:47:31.569 INFO:tasks.workunit.client.1.smithi111.stdout:5/389: sync 2024-03-23T11:47:31.571 INFO:tasks.workunit.client.1.smithi111.stdout:8/279: rename d5/d1b/l44 to d5/d40/l5b 0 2024-03-23T11:47:31.578 INFO:tasks.workunit.client.1.smithi111.stdout:6/387: symlink d2/d35/d70/l84 0 2024-03-23T11:47:31.582 INFO:tasks.workunit.client.1.smithi111.stdout:4/325: rmdir d0/dc/d4c/d4b 39 2024-03-23T11:47:31.583 INFO:tasks.workunit.client.1.smithi111.stdout:6/388: read d2/d1f/d45/f62 [862914,56717] 0 2024-03-23T11:47:31.583 INFO:tasks.workunit.client.0.smithi063.stdout:9/282: rename d4/d2c/f58 to d4/d31/f63 0 2024-03-23T11:47:31.584 INFO:tasks.workunit.client.0.smithi063.stdout:5/201: dwrite d2/d7/f1a [0,4194304] 0 2024-03-23T11:47:31.585 INFO:tasks.workunit.client.0.smithi063.stdout:7/298: dwrite d5/d6/d19/d1f/d37/da/f4e [0,4194304] 0 2024-03-23T11:47:31.587 INFO:tasks.workunit.client.1.smithi111.stdout:6/389: write d2/d1f/d45/f63 [1094204,98655] 0 2024-03-23T11:47:31.590 INFO:tasks.workunit.client.1.smithi111.stdout:3/291: dwrite d0/d7/d19/f29 [0,4194304] 0 2024-03-23T11:47:31.591 INFO:tasks.workunit.client.1.smithi111.stdout:5/390: symlink d1/d3/d6/d30/l85 0 2024-03-23T11:47:31.592 INFO:tasks.workunit.client.0.smithi063.stdout:0/306: dwrite d2/d4/f5e [0,4194304] 0 2024-03-23T11:47:31.592 INFO:tasks.workunit.client.0.smithi063.stdout:3/415: mknod d13/d22/c80 0 2024-03-23T11:47:31.597 INFO:tasks.workunit.client.0.smithi063.stdout:4/297: rename d5/da/d20/d4b/f31 to d5/da/d20/d4b/d16/d37/f53 0 2024-03-23T11:47:31.598 INFO:tasks.workunit.client.1.smithi111.stdout:2/396: write d2/d11/d17/d1f/f20 [1989768,125969] 0 2024-03-23T11:47:31.598 INFO:tasks.workunit.client.0.smithi063.stdout:6/255: write d1/f12 [1325812,85855] 0 2024-03-23T11:47:31.599 INFO:tasks.workunit.client.0.smithi063.stdout:5/202: mkdir d2/d7/d8/de/d49 0 2024-03-23T11:47:31.599 INFO:tasks.workunit.client.1.smithi111.stdout:4/326: rmdir d0/dc/d30 39 2024-03-23T11:47:31.600 INFO:tasks.workunit.client.0.smithi063.stdout:0/307: dread - d2/d4/d63/f6b zero size 2024-03-23T11:47:31.601 INFO:tasks.workunit.client.0.smithi063.stdout:8/559: write d6/d1c/d27/f31 [126212,79692] 0 2024-03-23T11:47:31.605 INFO:tasks.workunit.client.1.smithi111.stdout:8/280: dwrite d5/fa [0,4194304] 0 2024-03-23T11:47:31.606 INFO:tasks.workunit.client.0.smithi063.stdout:6/256: dread - d1/dc/d21/f2f zero size 2024-03-23T11:47:31.606 INFO:tasks.workunit.client.0.smithi063.stdout:9/283: mkdir d4/d4c/d44/d64 0 2024-03-23T11:47:31.609 INFO:tasks.workunit.client.1.smithi111.stdout:2/397: write d2/d11/d17/d1f/f20 [622706,36228] 0 2024-03-23T11:47:31.610 INFO:tasks.workunit.client.1.smithi111.stdout:7/305: dwrite d14/d1b/d1e/d2c/d42/f45 [0,4194304] 0 2024-03-23T11:47:31.611 INFO:tasks.workunit.client.0.smithi063.stdout:9/284: write d4/f60 [1008152,130515] 0 2024-03-23T11:47:31.612 INFO:tasks.workunit.client.1.smithi111.stdout:5/391: write d1/d3/d6/f3b [391530,130689] 0 2024-03-23T11:47:31.616 INFO:tasks.workunit.client.1.smithi111.stdout:6/390: chown d2/d73/l78 2261358 1 2024-03-23T11:47:31.618 INFO:tasks.workunit.client.1.smithi111.stdout:3/292: mkdir d0/d55 0 2024-03-23T11:47:31.622 INFO:tasks.workunit.client.1.smithi111.stdout:6/391: truncate d2/d1f/d45/f76 324355 0 2024-03-23T11:47:31.623 INFO:tasks.workunit.client.0.smithi063.stdout:4/298: creat d5/d1e/d2b/f54 x:0 0 0 2024-03-23T11:47:31.624 INFO:tasks.workunit.client.0.smithi063.stdout:7/299: dread d5/d6/d19/d1f/d37/da/f17 [4194304,4194304] 0 2024-03-23T11:47:31.625 INFO:tasks.workunit.client.1.smithi111.stdout:3/293: chown d0/d1a/d2b/c50 15057 1 2024-03-23T11:47:31.625 INFO:tasks.workunit.client.0.smithi063.stdout:9/285: dread d4/d4c/f41 [0,4194304] 0 2024-03-23T11:47:31.625 INFO:tasks.workunit.client.0.smithi063.stdout:4/299: dread - d5/d1e/d2b/f35 zero size 2024-03-23T11:47:31.627 INFO:tasks.workunit.client.0.smithi063.stdout:5/203: mknod d2/d28/d3a/c4a 0 2024-03-23T11:47:31.628 INFO:tasks.workunit.client.1.smithi111.stdout:4/327: fdatasync d0/dc/d4c/f39 0 2024-03-23T11:47:31.628 INFO:tasks.workunit.client.1.smithi111.stdout:1/356: truncate d0/d5/f23 3096565 0 2024-03-23T11:47:31.628 INFO:tasks.workunit.client.0.smithi063.stdout:5/204: fdatasync d2/d17/f42 0 2024-03-23T11:47:31.630 INFO:tasks.workunit.client.1.smithi111.stdout:6/392: read d2/d35/f77 [2485199,22145] 0 2024-03-23T11:47:31.631 INFO:tasks.workunit.client.0.smithi063.stdout:4/300: write d5/da/d20/f48 [3965132,38030] 0 2024-03-23T11:47:31.633 INFO:tasks.workunit.client.0.smithi063.stdout:6/257: mkdir d1/d15/d4b 0 2024-03-23T11:47:31.638 INFO:tasks.workunit.client.0.smithi063.stdout:1/243: link d3/l2e d3/d3f/d4d/l55 0 2024-03-23T11:47:31.641 INFO:tasks.workunit.client.1.smithi111.stdout:2/398: rename d2/d11/d17/d3e/d6b/l4c to d2/d11/d17/d3e/d6b/d2e/l8d 0 2024-03-23T11:47:31.641 INFO:tasks.workunit.client.1.smithi111.stdout:7/306: dwrite d14/d1b/d20/f3a [0,4194304] 0 2024-03-23T11:47:31.642 INFO:tasks.workunit.client.1.smithi111.stdout:5/392: mkdir d1/d3/d6/d7/d86 0 2024-03-23T11:47:31.648 INFO:tasks.workunit.client.0.smithi063.stdout:8/560: dwrite d6/f43 [0,4194304] 0 2024-03-23T11:47:31.651 INFO:tasks.workunit.client.0.smithi063.stdout:0/308: mkdir d2/d6e 0 2024-03-23T11:47:31.658 INFO:tasks.workunit.client.1.smithi111.stdout:3/294: creat d0/d55/f56 x:0 0 0 2024-03-23T11:47:31.659 INFO:tasks.workunit.client.0.smithi063.stdout:7/300: rmdir d5/d5c 39 2024-03-23T11:47:31.659 INFO:tasks.workunit.client.0.smithi063.stdout:5/205: unlink d2/d7/c1e 0 2024-03-23T11:47:31.661 INFO:tasks.workunit.client.1.smithi111.stdout:1/357: unlink d0/d6/l43 0 2024-03-23T11:47:31.663 INFO:tasks.workunit.client.0.smithi063.stdout:5/206: chown d2/f44 1442 1 2024-03-23T11:47:31.663 INFO:tasks.workunit.client.0.smithi063.stdout:4/301: dwrite d5/da/d20/d4b/d1f/f41 [0,4194304] 0 2024-03-23T11:47:31.664 INFO:tasks.workunit.client.1.smithi111.stdout:4/328: rmdir d0/dc/d30 39 2024-03-23T11:47:31.664 INFO:tasks.workunit.client.0.smithi063.stdout:6/258: rmdir d1/dc/d3a 39 2024-03-23T11:47:31.668 INFO:tasks.workunit.client.1.smithi111.stdout:5/393: creat d1/d3/d6/d7/d63/d2c/d82/d66/f87 x:0 0 0 2024-03-23T11:47:31.672 INFO:tasks.workunit.client.1.smithi111.stdout:7/307: creat d14/d37/d43/f55 x:0 0 0 2024-03-23T11:47:31.673 INFO:tasks.workunit.client.0.smithi063.stdout:1/244: mkdir d3/d49/d56 0 2024-03-23T11:47:31.674 INFO:tasks.workunit.client.1.smithi111.stdout:7/308: write d14/f36 [967436,36245] 0 2024-03-23T11:47:31.676 INFO:tasks.workunit.client.1.smithi111.stdout:9/398: sync 2024-03-23T11:47:31.676 INFO:tasks.workunit.client.0.smithi063.stdout:8/561: dwrite d6/d1f/d54/fa6 [0,4194304] 0 2024-03-23T11:47:31.677 INFO:tasks.workunit.client.1.smithi111.stdout:9/399: fdatasync d0/d4/f43 0 2024-03-23T11:47:31.677 INFO:tasks.workunit.client.0.smithi063.stdout:6/259: write d1/dc/d21/f29 [3970377,88428] 0 2024-03-23T11:47:31.678 INFO:tasks.workunit.client.1.smithi111.stdout:3/295: symlink d0/d7/d19/l57 0 2024-03-23T11:47:31.679 INFO:tasks.workunit.client.1.smithi111.stdout:9/400: chown d0/d4/c3e 507728 1 2024-03-23T11:47:31.679 INFO:tasks.workunit.client.0.smithi063.stdout:6/260: readlink d1/d13/l23 0 2024-03-23T11:47:31.680 INFO:tasks.workunit.client.0.smithi063.stdout:9/286: mknod d4/c65 0 2024-03-23T11:47:31.681 INFO:tasks.workunit.client.0.smithi063.stdout:8/562: dread - d6/d1f/d54/d63/d90/d15/f95 zero size 2024-03-23T11:47:31.682 INFO:tasks.workunit.client.0.smithi063.stdout:7/301: dread f1 [0,4194304] 0 2024-03-23T11:47:31.683 INFO:tasks.workunit.client.0.smithi063.stdout:6/261: write d1/dc/f46 [956905,94797] 0 2024-03-23T11:47:31.686 INFO:tasks.workunit.client.1.smithi111.stdout:5/394: dwrite d1/d3/fc [4194304,4194304] 0 2024-03-23T11:47:31.687 INFO:tasks.workunit.client.0.smithi063.stdout:6/262: dread - d1/dc/d21/f3c zero size 2024-03-23T11:47:31.688 INFO:tasks.workunit.client.0.smithi063.stdout:0/309: rename d2/d48/d5a to d2/d4/dd/d25/d6f 0 2024-03-23T11:47:31.689 INFO:tasks.workunit.client.0.smithi063.stdout:8/563: write d6/d1f/d54/d63/d90/f49 [1057649,85266] 0 2024-03-23T11:47:31.689 INFO:tasks.workunit.client.1.smithi111.stdout:5/395: chown d1/d3/d6/d7/d63/d2c/f34 373033 1 2024-03-23T11:47:31.690 INFO:tasks.workunit.client.1.smithi111.stdout:2/399: truncate d2/d32/d4b/d52/f73 916606 0 2024-03-23T11:47:31.695 INFO:tasks.workunit.client.0.smithi063.stdout:4/302: rmdir d5 39 2024-03-23T11:47:31.696 INFO:tasks.workunit.client.1.smithi111.stdout:8/281: truncate d5/d26/d31/f39 2225829 0 2024-03-23T11:47:31.699 INFO:tasks.workunit.client.1.smithi111.stdout:6/393: truncate d2/f57 2212044 0 2024-03-23T11:47:31.702 INFO:tasks.workunit.client.1.smithi111.stdout:7/309: mknod d14/d1b/d1e/d2c/c56 0 2024-03-23T11:47:31.705 INFO:tasks.workunit.client.1.smithi111.stdout:7/310: truncate d14/d1b/d1e/f47 134357 0 2024-03-23T11:47:31.707 INFO:tasks.workunit.client.1.smithi111.stdout:7/311: stat d14/d37/d43 0 2024-03-23T11:47:31.711 INFO:tasks.workunit.client.0.smithi063.stdout:3/416: truncate f9 7531341 0 2024-03-23T11:47:31.712 INFO:tasks.workunit.client.1.smithi111.stdout:0/343: sync 2024-03-23T11:47:31.720 INFO:tasks.workunit.client.0.smithi063.stdout:7/302: symlink d5/d6/d19/d1f/d37/da/l63 0 2024-03-23T11:47:31.722 INFO:tasks.workunit.client.1.smithi111.stdout:4/329: rmdir d0/dc/d4c/d4b/d4f 39 2024-03-23T11:47:31.725 INFO:tasks.workunit.client.1.smithi111.stdout:8/282: mknod d5/d32/d35/c5c 0 2024-03-23T11:47:31.725 INFO:tasks.workunit.client.1.smithi111.stdout:1/358: rmdir d0/d5/dd/d1e/d63/d79 0 2024-03-23T11:47:31.727 INFO:tasks.workunit.client.0.smithi063.stdout:8/564: creat d6/d1c/d32/d37/fc0 x:0 0 0 2024-03-23T11:47:31.731 INFO:tasks.workunit.client.0.smithi063.stdout:8/565: fsync d6/dd/f56 0 2024-03-23T11:47:31.733 INFO:tasks.workunit.client.1.smithi111.stdout:7/312: dwrite d14/d1b/d22/f4d [0,4194304] 0 2024-03-23T11:47:31.734 INFO:tasks.workunit.client.1.smithi111.stdout:7/313: fdatasync d14/f1c 0 2024-03-23T11:47:31.734 INFO:tasks.workunit.client.0.smithi063.stdout:8/566: fdatasync d6/d1f/d54/d5c/d66/f83 0 2024-03-23T11:47:31.735 INFO:tasks.workunit.client.1.smithi111.stdout:0/344: dwrite d5/db/d1f/f37 [0,4194304] 0 2024-03-23T11:47:31.735 INFO:tasks.workunit.client.0.smithi063.stdout:3/417: unlink d13/d15/f43 0 2024-03-23T11:47:31.739 INFO:tasks.workunit.client.0.smithi063.stdout:3/418: fdatasync d13/f4d 0 2024-03-23T11:47:31.741 INFO:tasks.workunit.client.1.smithi111.stdout:8/283: dread d5/d32/d35/d46/d4b/d51/d5a/f36 [0,4194304] 0 2024-03-23T11:47:31.742 INFO:tasks.workunit.client.1.smithi111.stdout:1/359: dread d0/d6/d48/f55 [0,4194304] 0 2024-03-23T11:47:31.743 INFO:tasks.workunit.client.1.smithi111.stdout:1/360: fsync d0/f13 0 2024-03-23T11:47:31.743 INFO:tasks.workunit.client.1.smithi111.stdout:8/284: chown d5/l2e 1587802 1 2024-03-23T11:47:31.744 INFO:tasks.workunit.client.0.smithi063.stdout:0/310: dread d2/d4/dd/d1f/f2d [0,4194304] 0 2024-03-23T11:47:31.744 INFO:tasks.workunit.client.0.smithi063.stdout:5/207: dread d2/d7/f14 [0,4194304] 0 2024-03-23T11:47:31.745 INFO:tasks.workunit.client.1.smithi111.stdout:3/296: write d0/d1a/d1d/d26/f46 [4046955,99456] 0 2024-03-23T11:47:31.745 INFO:tasks.workunit.client.0.smithi063.stdout:6/263: creat d1/d15/d4b/f4c x:0 0 0 2024-03-23T11:47:31.748 INFO:tasks.workunit.client.1.smithi111.stdout:9/401: symlink d0/d4/d10/d80/l85 0 2024-03-23T11:47:31.748 INFO:tasks.workunit.client.1.smithi111.stdout:4/330: mknod d0/d17/c60 0 2024-03-23T11:47:31.750 INFO:tasks.workunit.client.0.smithi063.stdout:4/303: creat d5/da/d20/d4b/d29/d38/f55 x:0 0 0 2024-03-23T11:47:31.752 INFO:tasks.workunit.client.1.smithi111.stdout:5/396: write d1/d5d/f5f [1565043,66128] 0 2024-03-23T11:47:31.753 INFO:tasks.workunit.client.1.smithi111.stdout:4/331: truncate d0/d7/da/f46 529909 0 2024-03-23T11:47:31.753 INFO:tasks.workunit.client.0.smithi063.stdout:4/304: read d5/d1e/d2b/f45 [311493,12411] 0 2024-03-23T11:47:31.755 INFO:tasks.workunit.client.0.smithi063.stdout:7/303: mkdir d5/d22/d57/d64 0 2024-03-23T11:47:31.756 INFO:tasks.workunit.client.1.smithi111.stdout:7/314: creat d14/d1b/d20/f57 x:0 0 0 2024-03-23T11:47:31.756 INFO:tasks.workunit.client.1.smithi111.stdout:5/397: write d1/d3/d6/d7/d63/d2c/f51 [679228,8850] 0 2024-03-23T11:47:31.760 INFO:tasks.workunit.client.1.smithi111.stdout:0/345: rmdir d5/db/d1f/d47 39 2024-03-23T11:47:31.761 INFO:tasks.workunit.client.0.smithi063.stdout:8/567: unlink d6/d1f/d54/d63/d90/d5d/fa3 0 2024-03-23T11:47:31.765 INFO:tasks.workunit.client.0.smithi063.stdout:3/419: symlink d13/d22/d2d/d3f/d67/l81 0 2024-03-23T11:47:31.765 INFO:tasks.workunit.client.1.smithi111.stdout:9/402: dread d0/d4/d1c/d36/d28/f68 [0,4194304] 0 2024-03-23T11:47:31.767 INFO:tasks.workunit.client.0.smithi063.stdout:5/208: creat d2/d28/f4b x:0 0 0 2024-03-23T11:47:31.769 INFO:tasks.workunit.client.1.smithi111.stdout:8/285: read d5/f6 [2167874,1464] 0 2024-03-23T11:47:31.770 INFO:tasks.workunit.client.0.smithi063.stdout:1/245: write d3/df/f18 [642431,29601] 0 2024-03-23T11:47:31.770 INFO:tasks.workunit.client.1.smithi111.stdout:2/400: rename d2/d11/d17/d1f/d30/f53 to d2/d11/d17/d3e/d6b/d2e/d3f/f8e 0 2024-03-23T11:47:31.771 INFO:tasks.workunit.client.0.smithi063.stdout:6/264: symlink d1/d15/d4b/l4d 0 2024-03-23T11:47:31.775 INFO:tasks.workunit.client.0.smithi063.stdout:6/265: write d1/f35 [1654487,39977] 0 2024-03-23T11:47:31.777 INFO:tasks.workunit.client.0.smithi063.stdout:4/305: mkdir d5/da/d20/d4b/d16/d32/d56 0 2024-03-23T11:47:31.779 INFO:tasks.workunit.client.1.smithi111.stdout:1/361: write d0/d5/f16 [949734,129218] 0 2024-03-23T11:47:31.781 INFO:tasks.workunit.client.1.smithi111.stdout:5/398: creat d1/d3/d6/d7/d63/d2c/d82/f88 x:0 0 0 2024-03-23T11:47:31.781 INFO:tasks.workunit.client.0.smithi063.stdout:8/568: chown d6/dd/c12 11628083 1 2024-03-23T11:47:31.784 INFO:tasks.workunit.client.1.smithi111.stdout:8/286: rmdir d5/d32/d35/d46/d4b/d51 39 2024-03-23T11:47:31.785 INFO:tasks.workunit.client.0.smithi063.stdout:5/209: mknod d2/d28/c4c 0 2024-03-23T11:47:31.786 INFO:tasks.workunit.client.1.smithi111.stdout:2/401: write d2/f4f [1474522,29861] 0 2024-03-23T11:47:31.786 INFO:tasks.workunit.client.1.smithi111.stdout:7/315: mkdir d14/d37/d50/d58 0 2024-03-23T11:47:31.787 INFO:tasks.workunit.client.0.smithi063.stdout:5/210: readlink d2/l36 0 2024-03-23T11:47:31.788 INFO:tasks.workunit.client.0.smithi063.stdout:8/569: truncate d6/d1f/d54/d63/d90/f5f 1485710 0 2024-03-23T11:47:31.789 INFO:tasks.workunit.client.1.smithi111.stdout:4/332: dwrite d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:31.790 INFO:tasks.workunit.client.0.smithi063.stdout:7/304: mkdir d5/d6/d19/d65 0 2024-03-23T11:47:31.790 INFO:tasks.workunit.client.0.smithi063.stdout:5/211: dread - d2/f3e zero size 2024-03-23T11:47:31.790 INFO:tasks.workunit.client.1.smithi111.stdout:2/402: write d2/d7/da/d19/f49 [3231663,93061] 0 2024-03-23T11:47:31.791 INFO:tasks.workunit.client.1.smithi111.stdout:3/297: truncate d0/d1a/f1c 2656505 0 2024-03-23T11:47:31.791 INFO:tasks.workunit.client.1.smithi111.stdout:5/399: rmdir d1 39 2024-03-23T11:47:31.795 INFO:tasks.workunit.client.0.smithi063.stdout:8/570: stat d6/d1f/d54/d63/d90/d15/f17 0 2024-03-23T11:47:31.795 INFO:tasks.workunit.client.0.smithi063.stdout:5/212: read d2/d28/f3d [920445,129539] 0 2024-03-23T11:47:31.797 INFO:tasks.workunit.client.1.smithi111.stdout:6/394: sync 2024-03-23T11:47:31.798 INFO:tasks.workunit.client.0.smithi063.stdout:9/287: dwrite d4/d31/f5f [0,4194304] 0 2024-03-23T11:47:31.800 INFO:tasks.workunit.client.1.smithi111.stdout:8/287: rmdir d5/d32/d35/d46/d4b/d51/d5a 39 2024-03-23T11:47:31.800 INFO:tasks.workunit.client.0.smithi063.stdout:0/311: rmdir d2/d12/d35/d67 0 2024-03-23T11:47:31.802 INFO:tasks.workunit.client.1.smithi111.stdout:6/395: read d2/d1f/d45/f56 [912117,92064] 0 2024-03-23T11:47:31.805 INFO:tasks.workunit.client.1.smithi111.stdout:1/362: rename d0/d5/dd/d1e/d63/f64 to d0/d6/d48/d4a/d6c/f7b 0 2024-03-23T11:47:31.808 INFO:tasks.workunit.client.1.smithi111.stdout:6/396: chown d2/d35/d70/l84 21 1 2024-03-23T11:47:31.810 INFO:tasks.workunit.client.1.smithi111.stdout:9/403: getdents d0/d31/d41/d44 0 2024-03-23T11:47:31.811 INFO:tasks.workunit.client.1.smithi111.stdout:5/400: rmdir d1/d3/d6/d7/d63/d2c/d82/d66/d77 39 2024-03-23T11:47:31.813 INFO:tasks.workunit.client.0.smithi063.stdout:0/312: readlink d2/d14/d36/l69 0 2024-03-23T11:47:31.814 INFO:tasks.workunit.client.1.smithi111.stdout:0/346: dwrite d5/db/d1f/f28 [0,4194304] 0 2024-03-23T11:47:31.814 INFO:tasks.workunit.client.0.smithi063.stdout:1/246: link d3/df/l48 d3/l57 0 2024-03-23T11:47:31.814 INFO:tasks.workunit.client.0.smithi063.stdout:5/213: write d2/d2f/f3f [418159,128549] 0 2024-03-23T11:47:31.814 INFO:tasks.workunit.client.1.smithi111.stdout:3/298: write d0/f1f [615510,93698] 0 2024-03-23T11:47:31.818 INFO:tasks.workunit.client.0.smithi063.stdout:6/266: creat d1/dc/f4e x:0 0 0 2024-03-23T11:47:31.819 INFO:tasks.workunit.client.0.smithi063.stdout:8/571: truncate d6/f89 738428 0 2024-03-23T11:47:31.820 INFO:tasks.workunit.client.1.smithi111.stdout:2/403: dwrite d2/d11/d17/f21 [0,4194304] 0 2024-03-23T11:47:31.820 INFO:tasks.workunit.client.0.smithi063.stdout:0/313: dread - d2/d12/d35/f62 zero size 2024-03-23T11:47:31.824 INFO:tasks.workunit.client.1.smithi111.stdout:2/404: read - d2/d5c/f63 zero size 2024-03-23T11:47:31.826 INFO:tasks.workunit.client.1.smithi111.stdout:9/404: write d0/d4/d10/d29/d5b/f7b [565570,125721] 0 2024-03-23T11:47:31.827 INFO:tasks.workunit.client.1.smithi111.stdout:1/363: mkdir d0/d2b/d7c 0 2024-03-23T11:47:31.829 INFO:tasks.workunit.client.1.smithi111.stdout:3/299: write d0/d1a/d1d/f24 [1356277,65086] 0 2024-03-23T11:47:31.831 INFO:tasks.workunit.client.0.smithi063.stdout:9/288: dwrite d4/f60 [0,4194304] 0 2024-03-23T11:47:31.833 INFO:tasks.workunit.client.1.smithi111.stdout:8/288: rename d5/l24 to d5/d32/l5d 0 2024-03-23T11:47:31.833 INFO:tasks.workunit.client.1.smithi111.stdout:4/333: dread d0/f45 [0,4194304] 0 2024-03-23T11:47:31.834 INFO:tasks.workunit.client.1.smithi111.stdout:5/401: mknod d1/d20/c89 0 2024-03-23T11:47:31.834 INFO:tasks.workunit.client.0.smithi063.stdout:1/247: mkdir d3/d3f/d4d/d52/d58 0 2024-03-23T11:47:31.835 INFO:tasks.workunit.client.1.smithi111.stdout:2/405: mknod d2/de/d12/d70/c8f 0 2024-03-23T11:47:31.837 INFO:tasks.workunit.client.1.smithi111.stdout:2/406: chown d2/f5b 0 1 2024-03-23T11:47:31.840 INFO:tasks.workunit.client.1.smithi111.stdout:2/407: write d2/d7/da/d19/f33 [259651,68355] 0 2024-03-23T11:47:31.843 INFO:tasks.workunit.client.0.smithi063.stdout:0/314: mknod d2/d14/d36/c70 0 2024-03-23T11:47:31.846 INFO:tasks.workunit.client.0.smithi063.stdout:0/315: chown d2/d14/d36 1179395 1 2024-03-23T11:47:31.846 INFO:tasks.workunit.client.1.smithi111.stdout:2/408: write d2/d11/f37 [2378372,127642] 0 2024-03-23T11:47:31.847 INFO:tasks.workunit.client.1.smithi111.stdout:3/300: symlink d0/l58 0 2024-03-23T11:47:31.848 INFO:tasks.workunit.client.1.smithi111.stdout:5/402: symlink d1/d3/d6/d7/d63/d2c/d82/l8a 0 2024-03-23T11:47:31.849 INFO:tasks.workunit.client.1.smithi111.stdout:1/364: dwrite d0/f59 [0,4194304] 0 2024-03-23T11:47:31.850 INFO:tasks.workunit.client.1.smithi111.stdout:6/397: write d2/ff [659534,50997] 0 2024-03-23T11:47:31.850 INFO:tasks.workunit.client.0.smithi063.stdout:5/214: dwrite d2/f33 [0,4194304] 0 2024-03-23T11:47:31.853 INFO:tasks.workunit.client.0.smithi063.stdout:0/316: chown d2/d4/f30 77 1 2024-03-23T11:47:31.853 INFO:tasks.workunit.client.1.smithi111.stdout:8/289: dwrite d5/f17 [0,4194304] 0 2024-03-23T11:47:31.854 INFO:tasks.workunit.client.1.smithi111.stdout:4/334: symlink d0/dc/l61 0 2024-03-23T11:47:31.855 INFO:tasks.workunit.client.1.smithi111.stdout:2/409: creat d2/d5c/f90 x:0 0 0 2024-03-23T11:47:31.856 INFO:tasks.workunit.client.1.smithi111.stdout:8/290: write d5/d1b/f2b [3887770,116385] 0 2024-03-23T11:47:31.856 INFO:tasks.workunit.client.1.smithi111.stdout:3/301: symlink d0/d1a/d2d/l59 0 2024-03-23T11:47:31.858 INFO:tasks.workunit.client.0.smithi063.stdout:3/420: truncate d13/d22/d2d/f45 2568036 0 2024-03-23T11:47:31.858 INFO:tasks.workunit.client.1.smithi111.stdout:3/302: fsync d0/d1a/d2b/f4d 0 2024-03-23T11:47:31.858 INFO:tasks.workunit.client.1.smithi111.stdout:8/291: write d5/d1b/f3a [3234595,85012] 0 2024-03-23T11:47:31.861 INFO:tasks.workunit.client.0.smithi063.stdout:1/248: mknod d3/d1c/c59 0 2024-03-23T11:47:31.865 INFO:tasks.workunit.client.0.smithi063.stdout:6/267: creat d1/dc/d3a/f4f x:0 0 0 2024-03-23T11:47:31.866 INFO:tasks.workunit.client.0.smithi063.stdout:1/249: dread - d3/df/f36 zero size 2024-03-23T11:47:31.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:31 smithi063 ceph-mon[35189]: pgmap v35: 65 pgs: 65 active+clean; 1.0 GiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 8.7 MiB/s rd, 123 MiB/s wr, 218 op/s 2024-03-23T11:47:31.885 INFO:tasks.workunit.client.0.smithi063.stdout:5/215: dwrite d2/f3e [0,4194304] 0 2024-03-23T11:47:31.888 INFO:tasks.workunit.client.0.smithi063.stdout:9/289: dread d4/d4c/d19/f2d [0,4194304] 0 2024-03-23T11:47:31.889 INFO:tasks.workunit.client.0.smithi063.stdout:5/216: stat d2/d28/f4b 0 2024-03-23T11:47:31.889 INFO:tasks.workunit.client.0.smithi063.stdout:0/317: rename d2/d4/dd/c10 to d2/d14/c71 0 2024-03-23T11:47:31.894 INFO:tasks.workunit.client.0.smithi063.stdout:6/268: dread d1/f35 [0,4194304] 0 2024-03-23T11:47:31.896 INFO:tasks.workunit.client.0.smithi063.stdout:8/572: link d6/d1f/d54/l91 d6/d1f/d54/d63/d90/d15/d9d/lc1 0 2024-03-23T11:47:31.897 INFO:tasks.workunit.client.0.smithi063.stdout:1/250: write d3/d35/d40/f4c [370095,81841] 0 2024-03-23T11:47:31.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:31 smithi111 ceph-mon[44005]: pgmap v35: 65 pgs: 65 active+clean; 1.0 GiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 8.7 MiB/s rd, 123 MiB/s wr, 218 op/s 2024-03-23T11:47:31.904 INFO:tasks.workunit.client.0.smithi063.stdout:9/290: write d4/f14 [5183610,121236] 0 2024-03-23T11:47:31.908 INFO:tasks.workunit.client.0.smithi063.stdout:0/318: mknod d2/d4/dd/d1f/c72 0 2024-03-23T11:47:31.911 INFO:tasks.workunit.client.1.smithi111.stdout:9/405: link d0/d31/d35/l57 d0/d4/d10/d80/l86 0 2024-03-23T11:47:31.911 INFO:tasks.workunit.client.0.smithi063.stdout:3/421: dread f8 [4194304,4194304] 0 2024-03-23T11:47:31.911 INFO:tasks.workunit.client.0.smithi063.stdout:8/573: creat d6/d1f/d54/d63/fc2 x:0 0 0 2024-03-23T11:47:31.912 INFO:tasks.workunit.client.0.smithi063.stdout:0/319: chown d2/d4/dd/d1f/d4f/l52 1 1 2024-03-23T11:47:31.917 INFO:tasks.workunit.client.1.smithi111.stdout:7/316: dread f10 [0,4194304] 0 2024-03-23T11:47:31.917 INFO:tasks.workunit.client.1.smithi111.stdout:3/303: mkdir d0/d1a/d1d/d26/d5a 0 2024-03-23T11:47:31.919 INFO:tasks.workunit.client.0.smithi063.stdout:8/574: truncate d6/d1f/d3e/f84 792911 0 2024-03-23T11:47:31.919 INFO:tasks.workunit.client.0.smithi063.stdout:5/217: truncate d2/d7/fd 2981049 0 2024-03-23T11:47:31.923 INFO:tasks.workunit.client.1.smithi111.stdout:5/403: rename d1/d5d to d1/d3/d6/d8b 0 2024-03-23T11:47:31.926 INFO:tasks.workunit.client.1.smithi111.stdout:5/404: dread d1/d20/f72 [0,4194304] 0 2024-03-23T11:47:31.926 INFO:tasks.workunit.client.0.smithi063.stdout:0/320: creat d2/d4/dd/d28/f73 x:0 0 0 2024-03-23T11:47:31.928 INFO:tasks.workunit.client.1.smithi111.stdout:5/405: write d1/d3/f4 [2383074,95046] 0 2024-03-23T11:47:31.929 INFO:tasks.workunit.client.1.smithi111.stdout:4/335: creat d0/dc/d4c/d4b/d4f/f62 x:0 0 0 2024-03-23T11:47:31.930 INFO:tasks.workunit.client.1.smithi111.stdout:7/317: creat d14/d1b/d1e/f59 x:0 0 0 2024-03-23T11:47:31.930 INFO:tasks.workunit.client.1.smithi111.stdout:9/406: chown d0/d4/d1c/c23 13468 1 2024-03-23T11:47:31.932 INFO:tasks.workunit.client.0.smithi063.stdout:5/218: unlink d2/d7/f1c 0 2024-03-23T11:47:31.933 INFO:tasks.workunit.client.1.smithi111.stdout:2/410: dread d2/d11/d17/d1f/d30/f41 [0,4194304] 0 2024-03-23T11:47:31.934 INFO:tasks.workunit.client.0.smithi063.stdout:5/219: chown f1 37630934 1 2024-03-23T11:47:31.934 INFO:tasks.workunit.client.1.smithi111.stdout:2/411: chown d2/d11/d17/d1f 1922708 1 2024-03-23T11:47:31.936 INFO:tasks.workunit.client.1.smithi111.stdout:0/347: sync 2024-03-23T11:47:31.937 INFO:tasks.workunit.client.0.smithi063.stdout:5/220: truncate d2/d17/f34 577224 0 2024-03-23T11:47:31.937 INFO:tasks.workunit.client.1.smithi111.stdout:2/412: write d2/d7/da/d19/f49 [287429,90461] 0 2024-03-23T11:47:31.941 INFO:tasks.workunit.client.1.smithi111.stdout:5/406: mkdir d1/d3/d6/d7/d8c 0 2024-03-23T11:47:31.942 INFO:tasks.workunit.client.1.smithi111.stdout:4/336: fdatasync d0/d7/f1b 0 2024-03-23T11:47:31.942 INFO:tasks.workunit.client.1.smithi111.stdout:7/318: dread d14/d1b/d1e/d2c/d42/f45 [0,4194304] 0 2024-03-23T11:47:31.942 INFO:tasks.workunit.client.0.smithi063.stdout:8/575: creat d6/d1f/dbb/fc3 x:0 0 0 2024-03-23T11:47:31.943 INFO:tasks.workunit.client.1.smithi111.stdout:4/337: chown d0/dc/d4c/d4b 6979 1 2024-03-23T11:47:31.945 INFO:tasks.workunit.client.0.smithi063.stdout:8/576: fsync d6/d1f/d54/d63/d90/d15/f24 0 2024-03-23T11:47:31.946 INFO:tasks.workunit.client.1.smithi111.stdout:8/292: rename d5/d32/d35 to d5/d26/d31/d5e 0 2024-03-23T11:47:31.951 INFO:tasks.workunit.client.1.smithi111.stdout:4/338: rmdir d0 39 2024-03-23T11:47:31.953 INFO:tasks.workunit.client.0.smithi063.stdout:9/291: getdents d4/d4c 0 2024-03-23T11:47:31.954 INFO:tasks.workunit.client.1.smithi111.stdout:0/348: unlink d5/db/d1f/d19/l24 0 2024-03-23T11:47:31.955 INFO:tasks.workunit.client.1.smithi111.stdout:8/293: rmdir d5/d26/d31/d5e/d46 39 2024-03-23T11:47:31.956 INFO:tasks.workunit.client.0.smithi063.stdout:3/422: getdents d13 0 2024-03-23T11:47:31.957 INFO:tasks.workunit.client.1.smithi111.stdout:8/294: write d5/f17 [2174338,112487] 0 2024-03-23T11:47:31.958 INFO:tasks.workunit.client.0.smithi063.stdout:1/251: truncate d3/d1c/f1e 759213 0 2024-03-23T11:47:31.962 INFO:tasks.workunit.client.0.smithi063.stdout:9/292: creat d4/d31/d43/d49/f66 x:0 0 0 2024-03-23T11:47:31.962 INFO:tasks.workunit.client.1.smithi111.stdout:7/319: dwrite d14/d1b/d1e/d2c/d42/f45 [0,4194304] 0 2024-03-23T11:47:31.965 INFO:tasks.workunit.client.0.smithi063.stdout:8/577: getdents d6/dd/d59/db2/dbe 0 2024-03-23T11:47:31.965 INFO:tasks.workunit.client.0.smithi063.stdout:5/221: dwrite d2/d7/d8/de/f41 [0,4194304] 0 2024-03-23T11:47:31.965 INFO:tasks.workunit.client.1.smithi111.stdout:0/349: creat d5/db/d1f/d48/f6a x:0 0 0 2024-03-23T11:47:31.969 INFO:tasks.workunit.client.1.smithi111.stdout:3/304: rename d0/d7/d19/l57 to d0/d1a/d2b/d38/l5b 0 2024-03-23T11:47:31.969 INFO:tasks.workunit.client.0.smithi063.stdout:8/578: write d6/d1f/d54/d63/fc2 [324047,17192] 0 2024-03-23T11:47:31.972 INFO:tasks.workunit.client.1.smithi111.stdout:8/295: symlink d5/d26/d31/d5e/l5f 0 2024-03-23T11:47:31.975 INFO:tasks.workunit.client.1.smithi111.stdout:0/350: chown d5/db/d1f/d47/f5a 1421690 1 2024-03-23T11:47:31.977 INFO:tasks.workunit.client.1.smithi111.stdout:0/351: dread - d5/db/d1f/f43 zero size 2024-03-23T11:47:31.978 INFO:tasks.workunit.client.0.smithi063.stdout:5/222: mkdir d2/d17/d4d 0 2024-03-23T11:47:31.979 INFO:tasks.workunit.client.1.smithi111.stdout:9/407: rename d0/d4/d1c/d36/f5 to d0/d4/d1c/d36/d28/d69/f87 0 2024-03-23T11:47:31.980 INFO:tasks.workunit.client.1.smithi111.stdout:3/305: creat d0/d1a/d2b/f5c x:0 0 0 2024-03-23T11:47:31.981 INFO:tasks.workunit.client.1.smithi111.stdout:4/339: dwrite d0/dc/d1f/f4a [4194304,4194304] 0 2024-03-23T11:47:31.981 INFO:tasks.workunit.client.0.smithi063.stdout:9/293: rename d4/f60 to d4/d31/d43/d49/f67 0 2024-03-23T11:47:31.982 INFO:tasks.workunit.client.1.smithi111.stdout:8/296: creat d5/d26/d31/d3b/f60 x:0 0 0 2024-03-23T11:47:31.986 INFO:tasks.workunit.client.1.smithi111.stdout:8/297: fsync d5/d26/d31/d3b/d41/f4e 0 2024-03-23T11:47:31.986 INFO:tasks.workunit.client.1.smithi111.stdout:5/407: rename d1/d3/d6/c11 to d1/d20/d5c/c8d 0 2024-03-23T11:47:31.989 INFO:tasks.workunit.client.0.smithi063.stdout:8/579: mkdir d6/dd/d59/daf/dc4 0 2024-03-23T11:47:31.989 INFO:tasks.workunit.client.0.smithi063.stdout:0/321: dwrite d2/d12/d32/f43 [0,4194304] 0 2024-03-23T11:47:31.991 INFO:tasks.workunit.client.1.smithi111.stdout:3/306: rmdir d0/d1a/d4f 39 2024-03-23T11:47:31.991 INFO:tasks.workunit.client.1.smithi111.stdout:8/298: stat d5/d26/d31/d3b/d55 0 2024-03-23T11:47:31.991 INFO:tasks.workunit.client.1.smithi111.stdout:3/307: chown d0/le 6871647 1 2024-03-23T11:47:31.991 INFO:tasks.workunit.client.1.smithi111.stdout:9/408: creat d0/d31/d41/d44/f88 x:0 0 0 2024-03-23T11:47:31.994 INFO:tasks.workunit.client.0.smithi063.stdout:5/223: creat d2/d28/d3a/f4e x:0 0 0 2024-03-23T11:47:31.996 INFO:tasks.workunit.client.1.smithi111.stdout:5/408: creat d1/d3/d6/d7/d8c/f8e x:0 0 0 2024-03-23T11:47:31.998 INFO:tasks.workunit.client.0.smithi063.stdout:1/252: getdents d3/d35 0 2024-03-23T11:47:31.998 INFO:tasks.workunit.client.1.smithi111.stdout:8/299: truncate d5/d26/d31/d4d/f59 729413 0 2024-03-23T11:47:32.000 INFO:tasks.workunit.client.1.smithi111.stdout:5/409: dread - d1/d3/d6/d8b/d59/f6a zero size 2024-03-23T11:47:32.000 INFO:tasks.workunit.client.1.smithi111.stdout:8/300: chown d5/c10 469952486 1 2024-03-23T11:47:32.001 INFO:tasks.workunit.client.1.smithi111.stdout:7/320: rename fb to d14/d1b/f5a 0 2024-03-23T11:47:32.001 INFO:tasks.workunit.client.1.smithi111.stdout:1/365: sync 2024-03-23T11:47:32.001 INFO:tasks.workunit.client.0.smithi063.stdout:5/224: creat d2/d28/f4f x:0 0 0 2024-03-23T11:47:32.002 INFO:tasks.workunit.client.1.smithi111.stdout:9/409: mkdir d0/d4/d6a/d89 0 2024-03-23T11:47:32.002 INFO:tasks.workunit.client.0.smithi063.stdout:8/580: dread d6/f71 [0,4194304] 0 2024-03-23T11:47:32.002 INFO:tasks.workunit.client.1.smithi111.stdout:4/340: dread d0/d7/da/f13 [0,4194304] 0 2024-03-23T11:47:32.005 INFO:tasks.workunit.client.1.smithi111.stdout:0/352: dwrite d5/db/d1f/d32/f3b [0,4194304] 0 2024-03-23T11:47:32.005 INFO:tasks.workunit.client.0.smithi063.stdout:5/225: write d2/d17/f34 [65878,7056] 0 2024-03-23T11:47:32.008 INFO:tasks.workunit.client.1.smithi111.stdout:7/321: write d14/d1b/d20/f57 [63777,52812] 0 2024-03-23T11:47:32.008 INFO:tasks.workunit.client.0.smithi063.stdout:0/322: unlink d2/d14/c71 0 2024-03-23T11:47:32.010 INFO:tasks.workunit.client.1.smithi111.stdout:0/353: chown d5/f13 4 1 2024-03-23T11:47:32.012 INFO:tasks.workunit.client.0.smithi063.stdout:0/323: dread - d2/d12/d32/f5c zero size 2024-03-23T11:47:32.013 INFO:tasks.workunit.client.1.smithi111.stdout:1/366: creat d0/d2b/d37/f7d x:0 0 0 2024-03-23T11:47:32.014 INFO:tasks.workunit.client.0.smithi063.stdout:8/581: rmdir d6/dd/d59 39 2024-03-23T11:47:32.016 INFO:tasks.workunit.client.0.smithi063.stdout:3/423: dread ff [0,4194304] 0 2024-03-23T11:47:32.018 INFO:tasks.workunit.client.0.smithi063.stdout:9/294: dwrite d4/f1e [0,4194304] 0 2024-03-23T11:47:32.019 INFO:tasks.workunit.client.1.smithi111.stdout:4/341: symlink d0/d7/da/l63 0 2024-03-23T11:47:32.020 INFO:tasks.workunit.client.1.smithi111.stdout:0/354: write d5/db/d1f/d19/d45/d4b/f69 [814417,52808] 0 2024-03-23T11:47:32.020 INFO:tasks.workunit.client.1.smithi111.stdout:3/308: dwrite d0/d7/d19/f52 [0,4194304] 0 2024-03-23T11:47:32.021 INFO:tasks.workunit.client.0.smithi063.stdout:3/424: fsync d13/d22/d2d/d3f/d58/f76 0 2024-03-23T11:47:32.022 INFO:tasks.workunit.client.0.smithi063.stdout:9/295: readlink d4/d31/d43/l52 0 2024-03-23T11:47:32.023 INFO:tasks.workunit.client.1.smithi111.stdout:2/413: sync 2024-03-23T11:47:32.023 INFO:tasks.workunit.client.1.smithi111.stdout:6/398: sync 2024-03-23T11:47:32.024 INFO:tasks.workunit.client.0.smithi063.stdout:9/296: dread - d4/d31/f59 zero size 2024-03-23T11:47:32.031 INFO:tasks.workunit.client.0.smithi063.stdout:3/425: write f8 [1967569,13979] 0 2024-03-23T11:47:32.036 INFO:tasks.workunit.client.1.smithi111.stdout:5/410: rmdir d1/d3/d6/d7/d63/d2c/d82/d66 39 2024-03-23T11:47:32.037 INFO:tasks.workunit.client.0.smithi063.stdout:8/582: truncate d6/d1f/d54/d5c/d66/f4c 1261646 0 2024-03-23T11:47:32.039 INFO:tasks.workunit.client.0.smithi063.stdout:3/426: unlink d13/d15/c20 0 2024-03-23T11:47:32.041 INFO:tasks.workunit.client.0.smithi063.stdout:5/226: write d2/f44 [1005332,122813] 0 2024-03-23T11:47:32.041 INFO:tasks.workunit.client.1.smithi111.stdout:1/367: symlink d0/d5/dd/d1e/d54/l7e 0 2024-03-23T11:47:32.045 INFO:tasks.workunit.client.0.smithi063.stdout:3/427: fsync d13/d22/f2c 0 2024-03-23T11:47:32.046 INFO:tasks.workunit.client.0.smithi063.stdout:8/583: mkdir d6/d1f/d54/d5c/dc5 0 2024-03-23T11:47:32.047 INFO:tasks.workunit.client.1.smithi111.stdout:1/368: dread d0/d6/f19 [0,4194304] 0 2024-03-23T11:47:32.049 INFO:tasks.workunit.client.1.smithi111.stdout:1/369: truncate d0/d2b/f75 903100 0 2024-03-23T11:47:32.049 INFO:tasks.workunit.client.1.smithi111.stdout:4/342: rmdir d0/dc/d4c/d4b 39 2024-03-23T11:47:32.050 INFO:tasks.workunit.client.0.smithi063.stdout:5/227: dread - d2/d7/d2b/f39 zero size 2024-03-23T11:47:32.051 INFO:tasks.workunit.client.0.smithi063.stdout:9/297: dread d4/d31/d43/d49/f67 [0,4194304] 0 2024-03-23T11:47:32.052 INFO:tasks.workunit.client.0.smithi063.stdout:1/253: truncate d3/d1c/f1e 1644250 0 2024-03-23T11:47:32.054 INFO:tasks.workunit.client.1.smithi111.stdout:0/355: rename d5/ca to d5/db/d1f/d47/c6b 0 2024-03-23T11:47:32.055 INFO:tasks.workunit.client.0.smithi063.stdout:3/428: mknod d13/d22/d2d/d3f/d67/c82 0 2024-03-23T11:47:32.060 INFO:tasks.workunit.client.0.smithi063.stdout:8/584: mknod d6/d1f/d54/d63/d90/d9e/cc6 0 2024-03-23T11:47:32.062 INFO:tasks.workunit.client.0.smithi063.stdout:0/324: dwrite d2/f56 [4194304,4194304] 0 2024-03-23T11:47:32.063 INFO:tasks.workunit.client.0.smithi063.stdout:3/429: readlink d13/d22/d2d/l57 0 2024-03-23T11:47:32.063 INFO:tasks.workunit.client.0.smithi063.stdout:1/254: truncate d3/d35/d40/f42 976175 0 2024-03-23T11:47:32.063 INFO:tasks.workunit.client.0.smithi063.stdout:5/228: creat d2/f50 x:0 0 0 2024-03-23T11:47:32.066 INFO:tasks.workunit.client.1.smithi111.stdout:6/399: dwrite d2/d17/f33 [0,4194304] 0 2024-03-23T11:47:32.069 INFO:tasks.workunit.client.0.smithi063.stdout:8/585: write d6/d1c/d27/f34 [3844302,27699] 0 2024-03-23T11:47:32.073 INFO:tasks.workunit.client.0.smithi063.stdout:1/255: fsync d3/f7 0 2024-03-23T11:47:32.075 INFO:tasks.workunit.client.0.smithi063.stdout:1/256: chown d3/d49/d56 62634 1 2024-03-23T11:47:32.078 INFO:tasks.workunit.client.1.smithi111.stdout:2/414: rename d2/d32/d4b/d52/d58 to d2/d32/d91 0 2024-03-23T11:47:32.079 INFO:tasks.workunit.client.1.smithi111.stdout:0/356: chown d5/db/d1f/d19/c36 179 1 2024-03-23T11:47:32.081 INFO:tasks.workunit.client.0.smithi063.stdout:5/229: rename d2/d7/d8/f12 to d2/d17/f51 0 2024-03-23T11:47:32.081 INFO:tasks.workunit.client.1.smithi111.stdout:3/309: creat d0/f5d x:0 0 0 2024-03-23T11:47:32.082 INFO:tasks.workunit.client.0.smithi063.stdout:3/430: symlink d13/d22/d2d/d3f/d67/d7b/l83 0 2024-03-23T11:47:32.083 INFO:tasks.workunit.client.0.smithi063.stdout:5/230: chown d2/d7/d8/de/c1b 1 1 2024-03-23T11:47:32.084 INFO:tasks.workunit.client.1.smithi111.stdout:6/400: creat d2/d73/f85 x:0 0 0 2024-03-23T11:47:32.085 INFO:tasks.workunit.client.0.smithi063.stdout:8/586: mkdir d6/d1f/d54/d63/dc7 0 2024-03-23T11:47:32.087 INFO:tasks.workunit.client.0.smithi063.stdout:1/257: creat d3/f5a x:0 0 0 2024-03-23T11:47:32.087 INFO:tasks.workunit.client.1.smithi111.stdout:9/410: truncate d0/f5f 1399924 0 2024-03-23T11:47:32.089 INFO:tasks.workunit.client.0.smithi063.stdout:1/258: write d3/f10 [3564933,109491] 0 2024-03-23T11:47:32.089 INFO:tasks.workunit.client.1.smithi111.stdout:8/301: dwrite d5/f6 [0,4194304] 0 2024-03-23T11:47:32.091 INFO:tasks.workunit.client.1.smithi111.stdout:7/322: truncate d14/d1b/d1e/d2c/f38 3992935 0 2024-03-23T11:47:32.091 INFO:tasks.workunit.client.0.smithi063.stdout:1/259: stat d3/d35/d54 0 2024-03-23T11:47:32.092 INFO:tasks.workunit.client.0.smithi063.stdout:9/298: link d4/d31/d43/d49/l61 d4/d2c/l68 0 2024-03-23T11:47:32.092 INFO:tasks.workunit.client.1.smithi111.stdout:2/415: dread d2/d7/da/d19/f49 [0,4194304] 0 2024-03-23T11:47:32.097 INFO:tasks.workunit.client.0.smithi063.stdout:1/260: write d3/d35/d40/f53 [177364,118470] 0 2024-03-23T11:47:32.097 INFO:tasks.workunit.client.0.smithi063.stdout:9/299: write d4/f1e [1062328,98659] 0 2024-03-23T11:47:32.098 INFO:tasks.workunit.client.1.smithi111.stdout:3/310: readlink d0/d1a/d2b/d38/l3d 0 2024-03-23T11:47:32.101 INFO:tasks.workunit.client.1.smithi111.stdout:9/411: rmdir d0/d31/d41/d65 39 2024-03-23T11:47:32.103 INFO:tasks.workunit.client.1.smithi111.stdout:1/370: truncate d0/d2b/f5c 2405123 0 2024-03-23T11:47:32.104 INFO:tasks.workunit.client.1.smithi111.stdout:4/343: dwrite d0/d7/f1b [4194304,4194304] 0 2024-03-23T11:47:32.107 INFO:tasks.workunit.client.1.smithi111.stdout:8/302: symlink d5/l61 0 2024-03-23T11:47:32.108 INFO:tasks.workunit.client.0.smithi063.stdout:1/261: write d3/df/f12 [624089,72967] 0 2024-03-23T11:47:32.108 INFO:tasks.workunit.client.1.smithi111.stdout:4/344: fsync d0/dc/d1f/f4a 0 2024-03-23T11:47:32.109 INFO:tasks.workunit.client.1.smithi111.stdout:3/311: dread - d0/d1a/d2b/d38/f51 zero size 2024-03-23T11:47:32.112 INFO:tasks.workunit.client.0.smithi063.stdout:8/587: dwrite d6/d1f/d3e/f84 [0,4194304] 0 2024-03-23T11:47:32.112 INFO:tasks.workunit.client.1.smithi111.stdout:2/416: symlink d2/d11/d17/d3e/d6b/d2e/l92 0 2024-03-23T11:47:32.113 INFO:tasks.workunit.client.0.smithi063.stdout:9/300: creat d4/d4c/f69 x:0 0 0 2024-03-23T11:47:32.114 INFO:tasks.workunit.client.1.smithi111.stdout:8/303: dread - d5/d26/d31/d3b/f60 zero size 2024-03-23T11:47:32.114 INFO:tasks.workunit.client.1.smithi111.stdout:1/371: getdents d0/d2b/d37/d6a 0 2024-03-23T11:47:32.115 INFO:tasks.workunit.client.1.smithi111.stdout:7/323: getdents d14/d1b/d1e/d2c/d2e 0 2024-03-23T11:47:32.117 INFO:tasks.workunit.client.0.smithi063.stdout:5/231: creat d2/f52 x:0 0 0 2024-03-23T11:47:32.118 INFO:tasks.workunit.client.1.smithi111.stdout:3/312: getdents d0/d1a/d1d/d26/d5a 0 2024-03-23T11:47:32.120 INFO:tasks.workunit.client.1.smithi111.stdout:3/313: readlink d0/l11 0 2024-03-23T11:47:32.121 INFO:tasks.workunit.client.1.smithi111.stdout:1/372: unlink d0/f1d 0 2024-03-23T11:47:32.122 INFO:tasks.workunit.client.0.smithi063.stdout:8/588: write d6/d1f/dbb/fc3 [823020,111987] 0 2024-03-23T11:47:32.123 INFO:tasks.workunit.client.1.smithi111.stdout:4/345: dread d0/dc/d4c/d4b/d4d/f51 [0,4194304] 0 2024-03-23T11:47:32.123 INFO:tasks.workunit.client.0.smithi063.stdout:9/301: chown d4/d2c/c38 6 1 2024-03-23T11:47:32.124 INFO:tasks.workunit.client.1.smithi111.stdout:8/304: mkdir d5/d26/d2f/d62 0 2024-03-23T11:47:32.124 INFO:tasks.workunit.client.1.smithi111.stdout:3/314: chown d0/f5d 620889414 1 2024-03-23T11:47:32.127 INFO:tasks.workunit.client.0.smithi063.stdout:1/262: creat d3/d3f/d4d/f5b x:0 0 0 2024-03-23T11:47:32.127 INFO:tasks.workunit.client.1.smithi111.stdout:7/324: unlink d14/d1b/d20/f3c 0 2024-03-23T11:47:32.128 INFO:tasks.workunit.client.0.smithi063.stdout:3/431: write d13/d22/d2d/f3a [1035174,127436] 0 2024-03-23T11:47:32.128 INFO:tasks.workunit.client.1.smithi111.stdout:3/315: write d0/fd [1917720,10566] 0 2024-03-23T11:47:32.129 INFO:tasks.workunit.client.0.smithi063.stdout:5/232: mknod d2/d28/c53 0 2024-03-23T11:47:32.130 INFO:tasks.workunit.client.0.smithi063.stdout:0/325: dwrite d2/d12/f50 [0,4194304] 0 2024-03-23T11:47:32.133 INFO:tasks.workunit.client.0.smithi063.stdout:0/326: stat d2/d12/d32/f5c 0 2024-03-23T11:47:32.134 INFO:tasks.workunit.client.1.smithi111.stdout:3/316: dread d0/f39 [0,4194304] 0 2024-03-23T11:47:32.135 INFO:tasks.workunit.client.0.smithi063.stdout:9/302: chown d4/f35 4248911 1 2024-03-23T11:47:32.136 INFO:tasks.workunit.client.1.smithi111.stdout:9/412: dwrite d0/d4/d1c/d2d/f2e [4194304,4194304] 0 2024-03-23T11:47:32.137 INFO:tasks.workunit.client.0.smithi063.stdout:8/589: mkdir d6/dd/d59/d7f/dc8 0 2024-03-23T11:47:32.140 INFO:tasks.workunit.client.1.smithi111.stdout:9/413: readlink d0/d4/d10/d29/d7f/l82 0 2024-03-23T11:47:32.140 INFO:tasks.workunit.client.0.smithi063.stdout:5/233: fdatasync d2/d7/d8/f20 0 2024-03-23T11:47:32.142 INFO:tasks.workunit.client.1.smithi111.stdout:2/417: rename d2/de/c69 to d2/d32/c93 0 2024-03-23T11:47:32.144 INFO:tasks.workunit.client.0.smithi063.stdout:5/234: write d2/f44 [1304862,74508] 0 2024-03-23T11:47:32.146 INFO:tasks.workunit.client.0.smithi063.stdout:8/590: fdatasync d6/f89 0 2024-03-23T11:47:32.154 INFO:tasks.workunit.client.0.smithi063.stdout:0/327: creat d2/d12/d32/d6d/f74 x:0 0 0 2024-03-23T11:47:32.155 INFO:tasks.workunit.client.1.smithi111.stdout:6/401: write d2/f28 [379816,156] 0 2024-03-23T11:47:32.157 INFO:tasks.workunit.client.1.smithi111.stdout:3/317: dwrite d0/d55/f56 [0,4194304] 0 2024-03-23T11:47:32.158 INFO:tasks.workunit.client.0.smithi063.stdout:3/432: dwrite d13/d15/f2a [0,4194304] 0 2024-03-23T11:47:32.163 INFO:tasks.workunit.client.1.smithi111.stdout:5/411: sync 2024-03-23T11:47:32.165 INFO:tasks.workunit.client.0.smithi063.stdout:9/303: write d4/d4c/d19/f42 [4979463,51387] 0 2024-03-23T11:47:32.165 INFO:tasks.workunit.client.0.smithi063.stdout:5/235: mknod d2/d7/d8/c54 0 2024-03-23T11:47:32.167 INFO:tasks.workunit.client.1.smithi111.stdout:7/325: mknod d14/d1b/d1e/d2c/c5b 0 2024-03-23T11:47:32.169 INFO:tasks.workunit.client.1.smithi111.stdout:7/326: chown d14/d1b/d20/f41 1859663 1 2024-03-23T11:47:32.169 INFO:tasks.workunit.client.0.smithi063.stdout:5/236: dread - d2/d2f/f37 zero size 2024-03-23T11:47:32.169 INFO:tasks.workunit.client.0.smithi063.stdout:9/304: write d4/f1e [4298612,116184] 0 2024-03-23T11:47:32.170 INFO:tasks.workunit.client.0.smithi063.stdout:3/433: creat d13/d22/d2d/d3f/d67/f84 x:0 0 0 2024-03-23T11:47:32.171 INFO:tasks.workunit.client.1.smithi111.stdout:8/305: dread d5/f1f [0,4194304] 0 2024-03-23T11:47:32.173 INFO:tasks.workunit.client.1.smithi111.stdout:2/418: rmdir d2/d7 39 2024-03-23T11:47:32.174 INFO:tasks.workunit.client.0.smithi063.stdout:5/237: truncate d2/fc 1913453 0 2024-03-23T11:47:32.175 INFO:tasks.workunit.client.1.smithi111.stdout:5/412: dread d1/d3/fc [4194304,4194304] 0 2024-03-23T11:47:32.180 INFO:tasks.workunit.client.0.smithi063.stdout:3/434: mknod d13/d22/d2d/d3e/c85 0 2024-03-23T11:47:32.181 INFO:tasks.workunit.client.1.smithi111.stdout:9/414: creat d0/d4/d10/d29/d7f/f8a x:0 0 0 2024-03-23T11:47:32.183 INFO:tasks.workunit.client.1.smithi111.stdout:9/415: chown d0/d4/d1c/d36/d28/f5d 220058517 1 2024-03-23T11:47:32.184 INFO:tasks.workunit.client.1.smithi111.stdout:1/373: mknod d0/d2b/c7f 0 2024-03-23T11:47:32.184 INFO:tasks.workunit.client.0.smithi063.stdout:8/591: write d6/dd/df/f60 [2554124,121700] 0 2024-03-23T11:47:32.188 INFO:tasks.workunit.client.0.smithi063.stdout:1/263: dwrite d3/f11 [0,4194304] 0 2024-03-23T11:47:32.189 INFO:tasks.workunit.client.1.smithi111.stdout:9/416: dread d0/d4/d1c/d36/f7e [0,4194304] 0 2024-03-23T11:47:32.189 INFO:tasks.workunit.client.0.smithi063.stdout:3/435: getdents d13/d66 0 2024-03-23T11:47:32.192 INFO:tasks.workunit.client.1.smithi111.stdout:6/402: mkdir d2/d1f/d45/d86 0 2024-03-23T11:47:32.194 INFO:tasks.workunit.client.0.smithi063.stdout:8/592: symlink d6/d1f/d54/d63/d90/d5d/d7d/db4/lc9 0 2024-03-23T11:47:32.195 INFO:tasks.workunit.client.1.smithi111.stdout:3/318: mkdir d0/d7/d19/d5e 0 2024-03-23T11:47:32.199 INFO:tasks.workunit.client.0.smithi063.stdout:5/238: dwrite f1 [0,4194304] 0 2024-03-23T11:47:32.201 INFO:tasks.workunit.client.1.smithi111.stdout:8/306: mkdir d5/d26/d2f/d42/d63 0 2024-03-23T11:47:32.203 INFO:tasks.workunit.client.1.smithi111.stdout:2/419: rename d2/d11/d17/d1f/l4a to d2/d32/d91/l94 0 2024-03-23T11:47:32.210 INFO:tasks.workunit.client.0.smithi063.stdout:5/239: symlink d2/d28/d3a/l55 0 2024-03-23T11:47:32.216 INFO:tasks.workunit.client.1.smithi111.stdout:9/417: write d0/d4/d1c/d36/d28/f68 [3371731,36680] 0 2024-03-23T11:47:32.216 INFO:tasks.workunit.client.0.smithi063.stdout:5/240: symlink d2/d28/d3a/l56 0 2024-03-23T11:47:32.219 INFO:tasks.workunit.client.0.smithi063.stdout:5/241: chown d2/d17 72340 1 2024-03-23T11:47:32.219 INFO:tasks.workunit.client.0.smithi063.stdout:3/436: write d13/d15/f1c [2323023,114139] 0 2024-03-23T11:47:32.228 INFO:tasks.workunit.client.0.smithi063.stdout:8/593: dwrite d6/d1c/d32/f57 [0,4194304] 0 2024-03-23T11:47:32.233 INFO:tasks.workunit.client.1.smithi111.stdout:4/346: link d0/c8 d0/c64 0 2024-03-23T11:47:32.234 INFO:tasks.workunit.client.1.smithi111.stdout:9/418: creat d0/d4/d10/f8b x:0 0 0 2024-03-23T11:47:32.235 INFO:tasks.workunit.client.0.smithi063.stdout:8/594: mknod d6/d1c/cca 0 2024-03-23T11:47:32.236 INFO:tasks.workunit.client.1.smithi111.stdout:6/403: truncate d2/f57 1164511 0 2024-03-23T11:47:32.237 INFO:tasks.workunit.client.1.smithi111.stdout:9/419: readlink d0/d4/d1c/l2f 0 2024-03-23T11:47:32.238 INFO:tasks.workunit.client.0.smithi063.stdout:9/305: dread d4/d4c/f25 [0,4194304] 0 2024-03-23T11:47:32.239 INFO:tasks.workunit.client.1.smithi111.stdout:7/327: rmdir d14/d37/d50/d58 0 2024-03-23T11:47:32.239 INFO:tasks.workunit.client.1.smithi111.stdout:3/319: creat d0/d1a/d1d/d26/d5a/f5f x:0 0 0 2024-03-23T11:47:32.240 INFO:tasks.workunit.client.0.smithi063.stdout:8/595: creat d6/d1f/d3e/fcb x:0 0 0 2024-03-23T11:47:32.242 INFO:tasks.workunit.client.1.smithi111.stdout:2/420: symlink d2/l95 0 2024-03-23T11:47:32.242 INFO:tasks.workunit.client.1.smithi111.stdout:8/307: creat d5/d26/d31/f64 x:0 0 0 2024-03-23T11:47:32.243 INFO:tasks.workunit.client.0.smithi063.stdout:3/437: dwrite d13/d22/f2c [4194304,4194304] 0 2024-03-23T11:47:32.246 INFO:tasks.workunit.client.1.smithi111.stdout:3/320: chown d0/d1a/c30 13 1 2024-03-23T11:47:32.247 INFO:tasks.workunit.client.1.smithi111.stdout:2/421: mkdir d2/de/d12/d70/d96 0 2024-03-23T11:47:32.249 INFO:tasks.workunit.client.0.smithi063.stdout:5/242: dwrite d2/f33 [0,4194304] 0 2024-03-23T11:47:32.254 INFO:tasks.workunit.client.0.smithi063.stdout:8/596: creat d6/d1f/d54/fcc x:0 0 0 2024-03-23T11:47:32.255 INFO:tasks.workunit.client.1.smithi111.stdout:4/347: dread d0/d7/da/f13 [0,4194304] 0 2024-03-23T11:47:32.255 INFO:tasks.workunit.client.1.smithi111.stdout:2/422: write d2/d11/d17/d3e/d6b/f87 [1016874,47673] 0 2024-03-23T11:47:32.258 INFO:tasks.workunit.client.1.smithi111.stdout:3/321: mkdir d0/d1a/d1d/d26/d5a/d60 0 2024-03-23T11:47:32.263 INFO:tasks.workunit.client.0.smithi063.stdout:5/243: dread d2/d7/d8/d1d/f1f [0,4194304] 0 2024-03-23T11:47:32.264 INFO:tasks.workunit.client.1.smithi111.stdout:9/420: dwrite d0/d4/d1c/f45 [0,4194304] 0 2024-03-23T11:47:32.265 INFO:tasks.workunit.client.0.smithi063.stdout:8/597: symlink d6/d1c/d27/d3f/lcd 0 2024-03-23T11:47:32.265 INFO:tasks.workunit.client.1.smithi111.stdout:6/404: dwrite d2/d1f/d45/d49/f6c [0,4194304] 0 2024-03-23T11:47:32.267 INFO:tasks.workunit.client.0.smithi063.stdout:5/244: dread - d2/d28/d3a/f4e zero size 2024-03-23T11:47:32.267 INFO:tasks.workunit.client.1.smithi111.stdout:7/328: dwrite d14/d1b/d1e/d2c/d42/f45 [0,4194304] 0 2024-03-23T11:47:32.268 INFO:tasks.workunit.client.1.smithi111.stdout:6/405: chown d2/d1f/d45/f4e 0 1 2024-03-23T11:47:32.269 INFO:tasks.workunit.client.1.smithi111.stdout:7/329: fsync d14/d1b/d20/f57 0 2024-03-23T11:47:32.272 INFO:tasks.workunit.client.1.smithi111.stdout:9/421: mknod d0/d4/d10/d29/d5b/c8c 0 2024-03-23T11:47:32.275 INFO:tasks.workunit.client.0.smithi063.stdout:5/245: creat d2/d7/d8/d1d/f57 x:0 0 0 2024-03-23T11:47:32.276 INFO:tasks.workunit.client.1.smithi111.stdout:4/348: symlink d0/dc/d30/l65 0 2024-03-23T11:47:32.277 INFO:tasks.workunit.client.0.smithi063.stdout:9/306: dwrite d4/d4c/d19/f2d [0,4194304] 0 2024-03-23T11:47:32.277 INFO:tasks.workunit.client.1.smithi111.stdout:7/330: read d14/d1b/f5a [806971,126614] 0 2024-03-23T11:47:32.278 INFO:tasks.workunit.client.1.smithi111.stdout:0/357: sync 2024-03-23T11:47:32.278 INFO:tasks.workunit.client.1.smithi111.stdout:3/322: symlink d0/d7/l61 0 2024-03-23T11:47:32.279 INFO:tasks.workunit.client.1.smithi111.stdout:7/331: write d14/d37/f3f [1021110,74093] 0 2024-03-23T11:47:32.282 INFO:tasks.workunit.client.0.smithi063.stdout:8/598: fsync d6/d1f/d54/d5c/d66/f4c 0 2024-03-23T11:47:32.283 INFO:tasks.workunit.client.1.smithi111.stdout:7/332: fdatasync d14/d1b/d20/f3a 0 2024-03-23T11:47:32.284 INFO:tasks.workunit.client.0.smithi063.stdout:3/438: dwrite d13/d22/d2d/d3f/d5b/f5d [0,4194304] 0 2024-03-23T11:47:32.285 INFO:tasks.workunit.client.1.smithi111.stdout:8/308: write f3 [7319705,67358] 0 2024-03-23T11:47:32.289 INFO:tasks.workunit.client.1.smithi111.stdout:4/349: rmdir d0/dc/d4c/d4b 39 2024-03-23T11:47:32.294 INFO:tasks.workunit.client.0.smithi063.stdout:3/439: mkdir d13/d22/d2d/d3f/d58/d86 0 2024-03-23T11:47:32.295 INFO:tasks.workunit.client.1.smithi111.stdout:0/358: rename d5/db/d1f/l1d to d5/db/d1f/d39/l6c 0 2024-03-23T11:47:32.295 INFO:tasks.workunit.client.1.smithi111.stdout:4/350: stat d0/dc/d30/ff 0 2024-03-23T11:47:32.298 INFO:tasks.workunit.client.1.smithi111.stdout:0/359: dread - d5/db/d1f/d48/f66 zero size 2024-03-23T11:47:32.298 INFO:tasks.workunit.client.1.smithi111.stdout:6/406: mknod d2/d1a/d1c/c87 0 2024-03-23T11:47:32.299 INFO:tasks.workunit.client.0.smithi063.stdout:8/599: rename d6/d1f/d54/d63/d90/d15/f95 to d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/fce 0 2024-03-23T11:47:32.302 INFO:tasks.workunit.client.1.smithi111.stdout:7/333: write f11 [1815067,60634] 0 2024-03-23T11:47:32.305 INFO:tasks.workunit.client.1.smithi111.stdout:3/323: mknod d0/d53/c62 0 2024-03-23T11:47:32.306 INFO:tasks.workunit.client.1.smithi111.stdout:4/351: mknod d0/c66 0 2024-03-23T11:47:32.308 INFO:tasks.workunit.client.0.smithi063.stdout:3/440: creat d13/d15/f87 x:0 0 0 2024-03-23T11:47:32.309 INFO:tasks.workunit.client.1.smithi111.stdout:7/334: fsync d14/d1b/d1e/f59 0 2024-03-23T11:47:32.309 INFO:tasks.workunit.client.1.smithi111.stdout:0/360: symlink d5/db/d4d/l6d 0 2024-03-23T11:47:32.311 INFO:tasks.workunit.client.1.smithi111.stdout:1/374: sync 2024-03-23T11:47:32.311 INFO:tasks.workunit.client.1.smithi111.stdout:5/413: sync 2024-03-23T11:47:32.313 INFO:tasks.workunit.client.0.smithi063.stdout:8/600: creat d6/d1c/d32/d37/fcf x:0 0 0 2024-03-23T11:47:32.313 INFO:tasks.workunit.client.0.smithi063.stdout:3/441: symlink d13/d15/l88 0 2024-03-23T11:47:32.317 INFO:tasks.workunit.client.1.smithi111.stdout:9/422: dwrite d0/d4/d1c/d2d/d3b/d42/d4f/f61 [0,4194304] 0 2024-03-23T11:47:32.317 INFO:tasks.workunit.client.1.smithi111.stdout:8/309: creat d5/d26/d31/d5e/f65 x:0 0 0 2024-03-23T11:47:32.319 INFO:tasks.workunit.client.1.smithi111.stdout:1/375: read d0/d6/d48/f5d [2675834,29823] 0 2024-03-23T11:47:32.319 INFO:tasks.workunit.client.0.smithi063.stdout:5/246: dwrite d2/d7/d2b/f39 [0,4194304] 0 2024-03-23T11:47:32.320 INFO:tasks.workunit.client.0.smithi063.stdout:9/307: dread d4/f47 [0,4194304] 0 2024-03-23T11:47:32.320 INFO:tasks.workunit.client.0.smithi063.stdout:3/442: dread - d13/d22/d2d/d3f/d58/f70 zero size 2024-03-23T11:47:32.321 INFO:tasks.workunit.client.1.smithi111.stdout:9/423: chown d0/d4/d6a/d89 25314 1 2024-03-23T11:47:32.322 INFO:tasks.workunit.client.0.smithi063.stdout:3/443: stat d13/f4d 0 2024-03-23T11:47:32.324 INFO:tasks.workunit.client.0.smithi063.stdout:8/601: unlink d6/d1f/d54/d63/d90/d15/d9d/cbd 0 2024-03-23T11:47:32.325 INFO:tasks.workunit.client.1.smithi111.stdout:4/352: unlink d0/dc/d30/l65 0 2024-03-23T11:47:32.329 INFO:tasks.workunit.client.1.smithi111.stdout:2/423: dwrite d2/d7/da/d19/f77 [0,4194304] 0 2024-03-23T11:47:32.330 INFO:tasks.workunit.client.1.smithi111.stdout:5/414: read d1/d20/f72 [91169,100907] 0 2024-03-23T11:47:32.331 INFO:tasks.workunit.client.0.smithi063.stdout:5/247: creat d2/d7/d8/d1d/f58 x:0 0 0 2024-03-23T11:47:32.332 INFO:tasks.workunit.client.0.smithi063.stdout:5/248: fsync d2/f52 0 2024-03-23T11:47:32.332 INFO:tasks.workunit.client.1.smithi111.stdout:7/335: write d14/d1b/d1e/d2c/d2e/d33/f3d [2182794,36173] 0 2024-03-23T11:47:32.334 INFO:tasks.workunit.client.1.smithi111.stdout:5/415: fsync d1/d20/f37 0 2024-03-23T11:47:32.334 INFO:tasks.workunit.client.0.smithi063.stdout:9/308: creat d4/d4c/d44/f6a x:0 0 0 2024-03-23T11:47:32.337 INFO:tasks.workunit.client.0.smithi063.stdout:8/602: creat d6/d1f/d54/d63/d90/d5d/d7d/fd0 x:0 0 0 2024-03-23T11:47:32.338 INFO:tasks.workunit.client.1.smithi111.stdout:7/336: chown d14/d37/f49 1070444741 1 2024-03-23T11:47:32.338 INFO:tasks.workunit.client.1.smithi111.stdout:2/424: dread - d2/de/d12/d70/f7e zero size 2024-03-23T11:47:32.340 INFO:tasks.workunit.client.0.smithi063.stdout:3/444: getdents d13/d15/d7e 0 2024-03-23T11:47:32.342 INFO:tasks.workunit.client.1.smithi111.stdout:0/361: symlink d5/db/d1f/d47/d59/l6e 0 2024-03-23T11:47:32.343 INFO:tasks.workunit.client.0.smithi063.stdout:5/249: mknod d2/d28/d3a/d46/c59 0 2024-03-23T11:47:32.345 INFO:tasks.workunit.client.1.smithi111.stdout:0/362: chown d5/db/d4d/l6d 0 1 2024-03-23T11:47:32.345 INFO:tasks.workunit.client.0.smithi063.stdout:9/309: write d4/f5a [853231,42768] 0 2024-03-23T11:47:32.346 INFO:tasks.workunit.client.1.smithi111.stdout:8/310: dread d5/f22 [0,4194304] 0 2024-03-23T11:47:32.347 INFO:tasks.workunit.client.1.smithi111.stdout:1/376: mknod d0/d6/d48/d4d/c80 0 2024-03-23T11:47:32.349 INFO:tasks.workunit.client.1.smithi111.stdout:8/311: fdatasync d5/d26/d31/d5e/f49 0 2024-03-23T11:47:32.351 INFO:tasks.workunit.client.0.smithi063.stdout:8/603: rmdir d6/dd/d59/d7f/dae 39 2024-03-23T11:47:32.351 INFO:tasks.workunit.client.1.smithi111.stdout:8/312: chown d5/c20 1917122 1 2024-03-23T11:47:32.352 INFO:tasks.workunit.client.0.smithi063.stdout:5/250: write d2/f44 [890316,17498] 0 2024-03-23T11:47:32.353 INFO:tasks.workunit.client.1.smithi111.stdout:4/353: symlink d0/dc/d30/l67 0 2024-03-23T11:47:32.356 INFO:tasks.workunit.client.0.smithi063.stdout:9/310: creat d4/d4c/d44/f6b x:0 0 0 2024-03-23T11:47:32.357 INFO:tasks.workunit.client.1.smithi111.stdout:2/425: creat d2/d11/d17/d3e/d6b/f97 x:0 0 0 2024-03-23T11:47:32.358 INFO:tasks.workunit.client.1.smithi111.stdout:0/363: mkdir d5/db/d1f/d47/d6f 0 2024-03-23T11:47:32.358 INFO:tasks.workunit.client.1.smithi111.stdout:7/337: dwrite d14/d1b/d1e/f39 [0,4194304] 0 2024-03-23T11:47:32.358 INFO:tasks.workunit.client.0.smithi063.stdout:8/604: read d6/d1f/d54/d63/d90/f49 [28732,107119] 0 2024-03-23T11:47:32.360 INFO:tasks.workunit.client.1.smithi111.stdout:1/377: mkdir d0/d5/dd/d1e/d54/d81 0 2024-03-23T11:47:32.361 INFO:tasks.workunit.client.1.smithi111.stdout:6/407: dread d2/f57 [0,4194304] 0 2024-03-23T11:47:32.362 INFO:tasks.workunit.client.1.smithi111.stdout:3/324: truncate d0/f1f 139468 0 2024-03-23T11:47:32.362 INFO:tasks.workunit.client.0.smithi063.stdout:5/251: write d2/d7/f14 [185742,82855] 0 2024-03-23T11:47:32.364 INFO:tasks.workunit.client.0.smithi063.stdout:9/311: write d4/d4c/f30 [1516767,113847] 0 2024-03-23T11:47:32.366 INFO:tasks.workunit.client.1.smithi111.stdout:8/313: creat d5/d26/d31/d3b/d41/f66 x:0 0 0 2024-03-23T11:47:32.368 INFO:tasks.workunit.client.1.smithi111.stdout:8/314: stat d5/d1b/c25 0 2024-03-23T11:47:32.368 INFO:tasks.workunit.client.1.smithi111.stdout:2/426: mkdir d2/d32/d98 0 2024-03-23T11:47:32.369 INFO:tasks.workunit.client.0.smithi063.stdout:3/445: dwrite d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:32.370 INFO:tasks.workunit.client.1.smithi111.stdout:2/427: fdatasync d2/fc 0 2024-03-23T11:47:32.371 INFO:tasks.workunit.client.1.smithi111.stdout:0/364: symlink d5/db/d4d/l70 0 2024-03-23T11:47:32.372 INFO:tasks.workunit.client.0.smithi063.stdout:9/312: write d4/d31/d43/d49/f66 [550522,49816] 0 2024-03-23T11:47:32.373 INFO:tasks.workunit.client.1.smithi111.stdout:5/416: dwrite d1/d3/d6/d8b/d59/d74/f21 [0,4194304] 0 2024-03-23T11:47:32.373 INFO:tasks.workunit.client.0.smithi063.stdout:8/605: creat d6/dd/d59/d7f/dae/fd1 x:0 0 0 2024-03-23T11:47:32.376 INFO:tasks.workunit.client.0.smithi063.stdout:8/606: chown d6/dd/d59/d7f 754788 1 2024-03-23T11:47:32.376 INFO:tasks.workunit.client.0.smithi063.stdout:9/313: truncate d4/f1e 8498038 0 2024-03-23T11:47:32.377 INFO:tasks.workunit.client.1.smithi111.stdout:1/378: rename d0/d5/l7 to d0/d2b/d37/d6a/l82 0 2024-03-23T11:47:32.377 INFO:tasks.workunit.client.1.smithi111.stdout:6/408: unlink d2/d1a/d1c/d2c/d51/f61 0 2024-03-23T11:47:32.378 INFO:tasks.workunit.client.0.smithi063.stdout:5/252: fsync d2/f5 0 2024-03-23T11:47:32.378 INFO:tasks.workunit.client.1.smithi111.stdout:6/409: dread - d2/f3b zero size 2024-03-23T11:47:32.381 INFO:tasks.workunit.client.1.smithi111.stdout:4/354: mknod d0/dc/d4c/d4b/d4f/c68 0 2024-03-23T11:47:32.383 INFO:tasks.workunit.client.0.smithi063.stdout:3/446: symlink d13/d15/l89 0 2024-03-23T11:47:32.384 INFO:tasks.workunit.client.1.smithi111.stdout:4/355: write d0/d7/f1b [359872,91876] 0 2024-03-23T11:47:32.384 INFO:tasks.workunit.client.1.smithi111.stdout:7/338: getdents d14/d37/d54 0 2024-03-23T11:47:32.387 INFO:tasks.workunit.client.1.smithi111.stdout:9/424: link d0/d4/d1c/d36/l8 d0/d4/d1c/d2d/d75/d7a/l8d 0 2024-03-23T11:47:32.387 INFO:tasks.workunit.client.0.smithi063.stdout:2/263: sync 2024-03-23T11:47:32.387 INFO:tasks.workunit.client.0.smithi063.stdout:4/306: sync 2024-03-23T11:47:32.388 INFO:tasks.workunit.client.0.smithi063.stdout:3/447: truncate d13/d22/d2d/f38 3017217 0 2024-03-23T11:47:32.388 INFO:tasks.workunit.client.1.smithi111.stdout:7/339: write d14/d1b/d1e/f59 [656884,81241] 0 2024-03-23T11:47:32.390 INFO:tasks.workunit.client.1.smithi111.stdout:7/340: readlink l5 0 2024-03-23T11:47:32.390 INFO:tasks.workunit.client.0.smithi063.stdout:4/307: dread - d5/d1e/d2b/f35 zero size 2024-03-23T11:47:32.390 INFO:tasks.workunit.client.0.smithi063.stdout:8/607: symlink d6/d1f/d54/d63/d90/d5d/d7d/d9a/ld2 0 2024-03-23T11:47:32.391 INFO:tasks.workunit.client.1.smithi111.stdout:2/428: dwrite d2/f6 [0,4194304] 0 2024-03-23T11:47:32.394 INFO:tasks.workunit.client.1.smithi111.stdout:0/365: creat d5/db/d1f/d48/f71 x:0 0 0 2024-03-23T11:47:32.394 INFO:tasks.workunit.client.1.smithi111.stdout:1/379: dwrite d0/d2b/f3e [0,4194304] 0 2024-03-23T11:47:32.396 INFO:tasks.workunit.client.0.smithi063.stdout:5/253: stat d2/d7/fd 0 2024-03-23T11:47:32.397 INFO:tasks.workunit.client.1.smithi111.stdout:5/417: symlink d1/d3/d6/d7/d63/d2c/l8f 0 2024-03-23T11:47:32.400 INFO:tasks.workunit.client.1.smithi111.stdout:5/418: write d1/d3/d6/f3b [398387,56271] 0 2024-03-23T11:47:32.405 INFO:tasks.workunit.client.1.smithi111.stdout:6/410: mknod d2/d1f/c88 0 2024-03-23T11:47:32.408 INFO:tasks.workunit.client.1.smithi111.stdout:6/411: chown d2/d1f/d45/f62 14 1 2024-03-23T11:47:32.408 INFO:tasks.workunit.client.0.smithi063.stdout:8/608: stat d6/d1f/d54/d63/d90/d9e/fa8 0 2024-03-23T11:47:32.408 INFO:tasks.workunit.client.0.smithi063.stdout:3/448: truncate d13/d22/f3b 3793756 0 2024-03-23T11:47:32.409 INFO:tasks.workunit.client.1.smithi111.stdout:6/412: dread - d2/d1f/d45/d49/f6d zero size 2024-03-23T11:47:32.411 INFO:tasks.workunit.client.1.smithi111.stdout:6/413: chown d2/d1a/d1c/f6b 0 1 2024-03-23T11:47:32.411 INFO:tasks.workunit.client.0.smithi063.stdout:4/308: creat d5/da/d20/d4b/d16/d37/f57 x:0 0 0 2024-03-23T11:47:32.413 INFO:tasks.workunit.client.1.smithi111.stdout:8/315: symlink d5/d26/d31/d5e/l67 0 2024-03-23T11:47:32.416 INFO:tasks.workunit.client.0.smithi063.stdout:2/264: dread d2/f21 [0,4194304] 0 2024-03-23T11:47:32.417 INFO:tasks.workunit.client.0.smithi063.stdout:3/449: write d13/d22/d2d/d3f/d5b/f77 [786107,80564] 0 2024-03-23T11:47:32.417 INFO:tasks.workunit.client.1.smithi111.stdout:9/425: rmdir d0/d4/d10 39 2024-03-23T11:47:32.420 INFO:tasks.workunit.client.1.smithi111.stdout:9/426: write d0/d4/d1c/d2d/f2e [7579510,40297] 0 2024-03-23T11:47:32.421 INFO:tasks.workunit.client.1.smithi111.stdout:7/341: mknod d14/d1b/d20/c5c 0 2024-03-23T11:47:32.424 INFO:tasks.workunit.client.0.smithi063.stdout:9/314: dwrite d4/d4c/f40 [0,4194304] 0 2024-03-23T11:47:32.426 INFO:tasks.workunit.client.1.smithi111.stdout:0/366: mkdir d5/db/d1f/d48/d72 0 2024-03-23T11:47:32.428 INFO:tasks.workunit.client.1.smithi111.stdout:0/367: fsync d5/db/d1f/d39/f62 0 2024-03-23T11:47:32.428 INFO:tasks.workunit.client.1.smithi111.stdout:1/380: mkdir d0/d6/d48/d4a/d6c/d83 0 2024-03-23T11:47:32.429 INFO:tasks.workunit.client.1.smithi111.stdout:1/381: chown d0/d6/f27 94480671 1 2024-03-23T11:47:32.430 INFO:tasks.workunit.client.1.smithi111.stdout:0/368: read d5/db/d1f/d32/f3e [1386365,120523] 0 2024-03-23T11:47:32.433 INFO:tasks.workunit.client.0.smithi063.stdout:3/450: rename d13/f47 to d13/d22/d2d/d3f/f8a 0 2024-03-23T11:47:32.439 INFO:tasks.workunit.client.1.smithi111.stdout:8/316: write d5/fa [4666225,94824] 0 2024-03-23T11:47:32.441 INFO:tasks.workunit.client.1.smithi111.stdout:8/317: chown d5/fa 31 1 2024-03-23T11:47:32.442 INFO:tasks.workunit.client.0.smithi063.stdout:9/315: rename d4/d4c/d44/l53 to d4/d31/d43/l6c 0 2024-03-23T11:47:32.443 INFO:tasks.workunit.client.1.smithi111.stdout:8/318: chown d5/d26/d31/d3b/d55 8872429 1 2024-03-23T11:47:32.444 INFO:tasks.workunit.client.0.smithi063.stdout:3/451: mknod d13/d22/d2d/d3f/d58/d6e/c8b 0 2024-03-23T11:47:32.446 INFO:tasks.workunit.client.0.smithi063.stdout:8/609: dwrite d6/d1c/d27/f8d [0,4194304] 0 2024-03-23T11:47:32.449 INFO:tasks.workunit.client.0.smithi063.stdout:8/610: chown d6/f89 53336 1 2024-03-23T11:47:32.450 INFO:tasks.workunit.client.0.smithi063.stdout:9/316: mknod d4/d2c/c6d 0 2024-03-23T11:47:32.452 INFO:tasks.workunit.client.0.smithi063.stdout:3/452: fsync d13/d22/d2d/d3f/d58/f76 0 2024-03-23T11:47:32.454 INFO:tasks.workunit.client.0.smithi063.stdout:4/309: dwrite d5/da/d20/d4b/d1f/f43 [0,4194304] 0 2024-03-23T11:47:32.455 INFO:tasks.workunit.client.1.smithi111.stdout:3/325: getdents d0/d1a/d1d 0 2024-03-23T11:47:32.457 INFO:tasks.workunit.client.1.smithi111.stdout:4/356: dwrite d0/dc/d4c/d4b/d4d/f51 [0,4194304] 0 2024-03-23T11:47:32.458 INFO:tasks.workunit.client.0.smithi063.stdout:3/453: write d13/d22/d2d/d3f/d67/f84 [979183,41823] 0 2024-03-23T11:47:32.460 INFO:tasks.workunit.client.1.smithi111.stdout:8/319: dwrite d5/f6 [0,4194304] 0 2024-03-23T11:47:32.461 INFO:tasks.workunit.client.0.smithi063.stdout:8/611: symlink d6/dd/d59/d7f/ld3 0 2024-03-23T11:47:32.462 INFO:tasks.workunit.client.0.smithi063.stdout:3/454: chown d13/d22/d2d/l1d 85562 1 2024-03-23T11:47:32.465 INFO:tasks.workunit.client.1.smithi111.stdout:8/320: write d5/d26/d31/d5e/f3f [767938,7548] 0 2024-03-23T11:47:32.465 INFO:tasks.workunit.client.0.smithi063.stdout:8/612: fdatasync d6/d1f/d54/d63/d90/fb3 0 2024-03-23T11:47:32.469 INFO:tasks.workunit.client.0.smithi063.stdout:3/455: write d13/d22/f6f [130046,72401] 0 2024-03-23T11:47:32.479 INFO:tasks.workunit.client.0.smithi063.stdout:8/613: dread d6/dd/df/f4b [0,4194304] 0 2024-03-23T11:47:32.480 INFO:tasks.workunit.client.1.smithi111.stdout:9/427: mkdir d0/d4/d10/d29/d7f/d8e 0 2024-03-23T11:47:32.482 INFO:tasks.workunit.client.1.smithi111.stdout:3/326: chown d0/l2 3994913 1 2024-03-23T11:47:32.482 INFO:tasks.workunit.client.0.smithi063.stdout:3/456: chown d13/d22/l7f 582311 1 2024-03-23T11:47:32.484 INFO:tasks.workunit.client.1.smithi111.stdout:4/357: write d0/f45 [3301706,1951] 0 2024-03-23T11:47:32.487 INFO:tasks.workunit.client.0.smithi063.stdout:9/317: dwrite d4/d2c/f3f [0,4194304] 0 2024-03-23T11:47:32.488 INFO:tasks.workunit.client.1.smithi111.stdout:8/321: creat d5/d1b/f68 x:0 0 0 2024-03-23T11:47:32.491 INFO:tasks.workunit.client.0.smithi063.stdout:2/265: dread d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:32.491 INFO:tasks.workunit.client.0.smithi063.stdout:8/614: chown d6/d1f/d54/d5c/d66/f72 7299494 1 2024-03-23T11:47:32.494 INFO:tasks.workunit.client.0.smithi063.stdout:4/310: dwrite d5/da/f2a [0,4194304] 0 2024-03-23T11:47:32.496 INFO:tasks.workunit.client.0.smithi063.stdout:2/266: dread - d2/f3a zero size 2024-03-23T11:47:32.496 INFO:tasks.workunit.client.0.smithi063.stdout:9/318: truncate d4/f5 5004346 0 2024-03-23T11:47:32.497 INFO:tasks.workunit.client.1.smithi111.stdout:5/419: rename d1/d3/d6/d8b/f5f to d1/d20/f90 0 2024-03-23T11:47:32.498 INFO:tasks.workunit.client.1.smithi111.stdout:2/429: getdents d2/d11/d17/d3e/d6b/d2e/d3f 0 2024-03-23T11:47:32.498 INFO:tasks.workunit.client.1.smithi111.stdout:6/414: creat d2/f89 x:0 0 0 2024-03-23T11:47:32.499 INFO:tasks.workunit.client.0.smithi063.stdout:9/319: dread - d4/d31/d43/f50 zero size 2024-03-23T11:47:32.499 INFO:tasks.workunit.client.1.smithi111.stdout:3/327: creat d0/d1a/d1d/d26/f63 x:0 0 0 2024-03-23T11:47:32.500 INFO:tasks.workunit.client.1.smithi111.stdout:0/369: link d5/c41 d5/db/d1f/d39/d64/c73 0 2024-03-23T11:47:32.500 INFO:tasks.workunit.client.0.smithi063.stdout:9/320: stat d4/d2c/f45 0 2024-03-23T11:47:32.501 INFO:tasks.workunit.client.1.smithi111.stdout:3/328: chown d0/le 468086 1 2024-03-23T11:47:32.502 INFO:tasks.workunit.client.1.smithi111.stdout:5/420: read d1/d20/d5c/f7e [1937995,41212] 0 2024-03-23T11:47:32.502 INFO:tasks.workunit.client.1.smithi111.stdout:9/428: getdents d0/d4/d6a/d89 0 2024-03-23T11:47:32.503 INFO:tasks.workunit.client.1.smithi111.stdout:2/430: truncate d2/d11/d17/d3e/d6b/f97 24501 0 2024-03-23T11:47:32.504 INFO:tasks.workunit.client.0.smithi063.stdout:9/321: creat d4/d31/f6e x:0 0 0 2024-03-23T11:47:32.506 INFO:tasks.workunit.client.1.smithi111.stdout:7/342: rename d14/d37/d43/l4c to d14/d37/d54/l5d 0 2024-03-23T11:47:32.507 INFO:tasks.workunit.client.1.smithi111.stdout:8/322: creat d5/d26/d2f/d42/d63/f69 x:0 0 0 2024-03-23T11:47:32.507 INFO:tasks.workunit.client.1.smithi111.stdout:0/370: fdatasync d5/db/d1f/d39/f3d 0 2024-03-23T11:47:32.508 INFO:tasks.workunit.client.0.smithi063.stdout:9/322: symlink d4/d4c/d19/l6f 0 2024-03-23T11:47:32.509 INFO:tasks.workunit.client.1.smithi111.stdout:3/329: rmdir d0/d1a/d2b 39 2024-03-23T11:47:32.510 INFO:tasks.workunit.client.1.smithi111.stdout:5/421: mknod d1/d3/d6/d8b/d59/c91 0 2024-03-23T11:47:32.515 INFO:tasks.workunit.client.0.smithi063.stdout:9/323: mkdir d4/d31/d70 0 2024-03-23T11:47:32.516 INFO:tasks.workunit.client.1.smithi111.stdout:2/431: fsync d2/d11/d17/d3e/d6b/f2f 0 2024-03-23T11:47:32.518 INFO:tasks.workunit.client.1.smithi111.stdout:7/343: write d14/d1b/f2d [3522367,97669] 0 2024-03-23T11:47:32.522 INFO:tasks.workunit.client.1.smithi111.stdout:5/422: rename d1/d3/d6/d8b/d59/d74/f54 to d1/f92 0 2024-03-23T11:47:32.523 INFO:tasks.workunit.client.0.smithi063.stdout:9/324: fsync d4/d4c/d19/f27 0 2024-03-23T11:47:32.523 INFO:tasks.workunit.client.1.smithi111.stdout:7/344: write d14/d1b/d1e/d2c/f31 [5158391,83646] 0 2024-03-23T11:47:32.524 INFO:tasks.workunit.client.0.smithi063.stdout:2/267: dwrite d2/d23/d2e/d52/f4c [0,4194304] 0 2024-03-23T11:47:32.526 INFO:tasks.workunit.client.1.smithi111.stdout:1/382: dwrite d0/d5/dd/d1e/f31 [0,4194304] 0 2024-03-23T11:47:32.527 INFO:tasks.workunit.client.1.smithi111.stdout:5/423: fdatasync d1/d3/d6/d7/d8c/f8e 0 2024-03-23T11:47:32.528 INFO:tasks.workunit.client.1.smithi111.stdout:2/432: mknod d2/d11/d17/d3e/c99 0 2024-03-23T11:47:32.530 INFO:tasks.workunit.client.1.smithi111.stdout:0/371: rmdir d5/db/d1f/d48/d72 0 2024-03-23T11:47:32.531 INFO:tasks.workunit.client.0.smithi063.stdout:9/325: rename d4/d31/c36 to d4/d31/d43/c71 0 2024-03-23T11:47:32.534 INFO:tasks.workunit.client.1.smithi111.stdout:9/429: getdents d0/d4/d1c/d36/d28 0 2024-03-23T11:47:32.536 INFO:tasks.workunit.client.1.smithi111.stdout:1/383: creat d0/d5/dd/d1e/d54/f84 x:0 0 0 2024-03-23T11:47:32.536 INFO:tasks.workunit.client.1.smithi111.stdout:2/433: creat d2/de/d12/d70/f9a x:0 0 0 2024-03-23T11:47:32.536 INFO:tasks.workunit.client.1.smithi111.stdout:0/372: stat d5/db/f51 0 2024-03-23T11:47:32.539 INFO:tasks.workunit.client.1.smithi111.stdout:7/345: chown d14/d1b/l24 2062729743 1 2024-03-23T11:47:32.539 INFO:tasks.workunit.client.1.smithi111.stdout:5/424: truncate d1/d3/d6/d7/f69 307160 0 2024-03-23T11:47:32.541 INFO:tasks.workunit.client.0.smithi063.stdout:9/326: creat d4/d4c/f72 x:0 0 0 2024-03-23T11:47:32.542 INFO:tasks.workunit.client.1.smithi111.stdout:9/430: readlink d0/d4/d10/l1f 0 2024-03-23T11:47:32.543 INFO:tasks.workunit.client.0.smithi063.stdout:3/457: write d13/f50 [8236262,52940] 0 2024-03-23T11:47:32.546 INFO:tasks.workunit.client.0.smithi063.stdout:9/327: rmdir d4/d4c/d19 39 2024-03-23T11:47:32.548 INFO:tasks.workunit.client.1.smithi111.stdout:1/384: symlink d0/d2b/d37/l85 0 2024-03-23T11:47:32.548 INFO:tasks.workunit.client.0.smithi063.stdout:9/328: fdatasync d4/d4c/f69 0 2024-03-23T11:47:32.550 INFO:tasks.workunit.client.1.smithi111.stdout:3/330: dread d0/d7/f47 [0,4194304] 0 2024-03-23T11:47:32.550 INFO:tasks.workunit.client.0.smithi063.stdout:4/311: dread d5/da/d20/d4b/d1f/f26 [0,4194304] 0 2024-03-23T11:47:32.551 INFO:tasks.workunit.client.1.smithi111.stdout:9/431: mkdir d0/d31/d41/d71/d8f 0 2024-03-23T11:47:32.553 INFO:tasks.workunit.client.0.smithi063.stdout:4/312: read d5/da/f2a [679655,53068] 0 2024-03-23T11:47:32.554 INFO:tasks.workunit.client.0.smithi063.stdout:2/268: truncate d2/d23/d2e/d52/f33 2912498 0 2024-03-23T11:47:32.560 INFO:tasks.workunit.client.0.smithi063.stdout:2/269: creat d2/d23/d56/f57 x:0 0 0 2024-03-23T11:47:32.562 INFO:tasks.workunit.client.0.smithi063.stdout:2/270: dread - d2/de/d22/f4a zero size 2024-03-23T11:47:32.565 INFO:tasks.workunit.client.0.smithi063.stdout:9/329: link d4/c3e d4/d4c/c73 0 2024-03-23T11:47:32.566 INFO:tasks.workunit.client.1.smithi111.stdout:6/415: dwrite d2/d1a/f1d [0,4194304] 0 2024-03-23T11:47:32.567 INFO:tasks.workunit.client.1.smithi111.stdout:3/331: mkdir d0/d1a/d1d/d26/d5a/d64 0 2024-03-23T11:47:32.568 INFO:tasks.workunit.client.0.smithi063.stdout:2/271: write d2/f1e [1355782,44857] 0 2024-03-23T11:47:32.568 INFO:tasks.workunit.client.0.smithi063.stdout:3/458: write d13/d15/f3c [847093,17371] 0 2024-03-23T11:47:32.569 INFO:tasks.workunit.client.1.smithi111.stdout:7/346: dwrite d14/d1b/d1e/d2c/d42/f45 [0,4194304] 0 2024-03-23T11:47:32.570 INFO:tasks.workunit.client.0.smithi063.stdout:9/330: chown d4/ff 1036706272 1 2024-03-23T11:47:32.573 INFO:tasks.workunit.client.1.smithi111.stdout:2/434: dwrite d2/d11/d17/d3e/f40 [0,4194304] 0 2024-03-23T11:47:32.573 INFO:tasks.workunit.client.1.smithi111.stdout:6/416: fsync d2/d1a/d1c/d2c/f4c 0 2024-03-23T11:47:32.575 INFO:tasks.workunit.client.1.smithi111.stdout:4/358: dread d0/dc/f1c [0,4194304] 0 2024-03-23T11:47:32.575 INFO:tasks.workunit.client.1.smithi111.stdout:8/323: dread d5/fe [0,4194304] 0 2024-03-23T11:47:32.576 INFO:tasks.workunit.client.0.smithi063.stdout:3/459: mknod d13/d15/d7e/c8c 0 2024-03-23T11:47:32.581 INFO:tasks.workunit.client.0.smithi063.stdout:9/331: creat d4/f74 x:0 0 0 2024-03-23T11:47:32.583 INFO:tasks.workunit.client.0.smithi063.stdout:4/313: dwrite d5/da/f19 [0,4194304] 0 2024-03-23T11:47:32.583 INFO:tasks.workunit.client.0.smithi063.stdout:2/272: symlink d2/d23/d2e/d52/d44/d4d/l58 0 2024-03-23T11:47:32.587 INFO:tasks.workunit.client.1.smithi111.stdout:0/373: write d5/db/f26 [7957704,6700] 0 2024-03-23T11:47:32.591 INFO:tasks.workunit.client.0.smithi063.stdout:2/273: rmdir d2/d1c 39 2024-03-23T11:47:32.591 INFO:tasks.workunit.client.0.smithi063.stdout:4/314: creat d5/d1e/d2b/f58 x:0 0 0 2024-03-23T11:47:32.594 INFO:tasks.workunit.client.1.smithi111.stdout:0/374: stat d5/db/d4d/c54 0 2024-03-23T11:47:32.595 INFO:tasks.workunit.client.0.smithi063.stdout:9/332: mkdir d4/d4c/d19/d75 0 2024-03-23T11:47:32.596 INFO:tasks.workunit.client.0.smithi063.stdout:9/333: readlink d4/d31/d43/l52 0 2024-03-23T11:47:32.599 INFO:tasks.workunit.client.0.smithi063.stdout:3/460: rename d13/d15/d48/d71 to d13/d22/d2d/d3f/d58/d6e/d8d 0 2024-03-23T11:47:32.600 INFO:tasks.workunit.client.1.smithi111.stdout:4/359: unlink d0/c8 0 2024-03-23T11:47:32.600 INFO:tasks.workunit.client.1.smithi111.stdout:9/432: dwrite d0/d31/d41/d44/f88 [0,4194304] 0 2024-03-23T11:47:32.602 INFO:tasks.workunit.client.0.smithi063.stdout:4/315: truncate d5/da/d20/d4b/d1f/f3f 523732 0 2024-03-23T11:47:32.603 INFO:tasks.workunit.client.0.smithi063.stdout:9/334: fdatasync d4/f47 0 2024-03-23T11:47:32.604 INFO:tasks.workunit.client.1.smithi111.stdout:5/425: dwrite d1/d3/d6/d7/d63/d2c/f34 [0,4194304] 0 2024-03-23T11:47:32.606 INFO:tasks.workunit.client.1.smithi111.stdout:2/435: creat d2/d32/d4b/d52/f9b x:0 0 0 2024-03-23T11:47:32.608 INFO:tasks.workunit.client.0.smithi063.stdout:3/461: rename d13/d22/d2d/d3f/d67/c82 to d13/d22/d2d/d7d/c8e 0 2024-03-23T11:47:32.612 INFO:tasks.workunit.client.1.smithi111.stdout:0/375: mknod d5/db/d1f/d47/c74 0 2024-03-23T11:47:32.612 INFO:tasks.workunit.client.1.smithi111.stdout:1/385: dread d0/d5/dd/f25 [0,4194304] 0 2024-03-23T11:47:32.612 INFO:tasks.workunit.client.1.smithi111.stdout:8/324: mkdir d5/d26/d31/d5e/d46/d4b/d51/d5a/d6a 0 2024-03-23T11:47:32.613 INFO:tasks.workunit.client.1.smithi111.stdout:9/433: write d0/d4/d1c/d36/f11 [3044625,119901] 0 2024-03-23T11:47:32.614 INFO:tasks.workunit.client.0.smithi063.stdout:2/274: mknod d2/d23/d2e/d52/c59 0 2024-03-23T11:47:32.619 INFO:tasks.workunit.client.1.smithi111.stdout:3/332: truncate d0/d1a/d1d/f24 623916 0 2024-03-23T11:47:32.619 INFO:tasks.workunit.client.1.smithi111.stdout:5/426: symlink d1/d3/d6/d7/d63/d2c/d82/d66/d77/l93 0 2024-03-23T11:47:32.623 INFO:tasks.workunit.client.1.smithi111.stdout:8/325: creat d5/d26/d31/d5e/d46/f6b x:0 0 0 2024-03-23T11:47:32.623 INFO:tasks.workunit.client.1.smithi111.stdout:1/386: chown d0/d2b/f5a 0 1 2024-03-23T11:47:32.624 INFO:tasks.workunit.client.1.smithi111.stdout:0/376: truncate d5/db/d1f/d32/f3b 4518620 0 2024-03-23T11:47:32.626 INFO:tasks.workunit.client.1.smithi111.stdout:8/326: fdatasync d5/d1b/f68 0 2024-03-23T11:47:32.628 INFO:tasks.workunit.client.0.smithi063.stdout:2/275: creat d2/d23/d56/f5a x:0 0 0 2024-03-23T11:47:32.628 INFO:tasks.workunit.client.1.smithi111.stdout:3/333: mknod d0/d7/c65 0 2024-03-23T11:47:32.630 INFO:tasks.workunit.client.0.smithi063.stdout:4/316: dwrite d5/da/d20/f48 [0,4194304] 0 2024-03-23T11:47:32.632 INFO:tasks.workunit.client.1.smithi111.stdout:5/427: mkdir d1/d3/d6/d7/d8c/d94 0 2024-03-23T11:47:32.637 INFO:tasks.workunit.client.0.smithi063.stdout:4/317: getdents d5/da/d20/d4b/d16/d50 0 2024-03-23T11:47:32.649 INFO:tasks.workunit.client.0.smithi063.stdout:4/318: link d5/da/d20/d4b/d1f/f43 d5/f59 0 2024-03-23T11:47:32.651 INFO:tasks.workunit.client.0.smithi063.stdout:3/462: rmdir d13/d22/d2d/d3f/d58 39 2024-03-23T11:47:32.651 INFO:tasks.workunit.client.1.smithi111.stdout:6/417: dwrite d2/f57 [0,4194304] 0 2024-03-23T11:47:32.655 INFO:tasks.workunit.client.0.smithi063.stdout:2/276: dwrite d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:32.655 INFO:tasks.workunit.client.1.smithi111.stdout:8/327: chown d5/d26/d31/d5e/d46/d4b/d51/d5a 102785143 1 2024-03-23T11:47:32.661 INFO:tasks.workunit.client.0.smithi063.stdout:3/463: rmdir d13/d15/d7e 39 2024-03-23T11:47:32.662 INFO:tasks.workunit.client.1.smithi111.stdout:0/377: mknod d5/db/d1f/d47/d6f/c75 0 2024-03-23T11:47:32.666 INFO:tasks.workunit.client.0.smithi063.stdout:9/335: truncate d4/d4c/f30 1000654 0 2024-03-23T11:47:32.666 INFO:tasks.workunit.client.1.smithi111.stdout:6/418: readlink d2/d35/d70/l82 0 2024-03-23T11:47:32.666 INFO:tasks.workunit.client.1.smithi111.stdout:5/428: mkdir d1/d3/d6/d7/d63/d2c/d43/d95 0 2024-03-23T11:47:32.668 INFO:tasks.workunit.client.1.smithi111.stdout:4/360: dwrite d0/dc/d4c/d2a/f40 [0,4194304] 0 2024-03-23T11:47:32.671 INFO:tasks.workunit.client.1.smithi111.stdout:7/347: dread d14/d1b/d1e/d2c/f38 [0,4194304] 0 2024-03-23T11:47:32.672 INFO:tasks.workunit.client.0.smithi063.stdout:4/319: getdents d5/da/d20/d4b/d16/d32/d56 0 2024-03-23T11:47:32.673 INFO:tasks.workunit.client.0.smithi063.stdout:3/464: creat d13/d15/d7e/f8f x:0 0 0 2024-03-23T11:47:32.678 INFO:tasks.workunit.client.0.smithi063.stdout:3/465: chown d13/d15/l89 1698 1 2024-03-23T11:47:32.679 INFO:tasks.workunit.client.1.smithi111.stdout:2/436: dwrite d2/d32/d91/f6d [0,4194304] 0 2024-03-23T11:47:32.683 INFO:tasks.workunit.client.1.smithi111.stdout:6/419: mknod d2/d1f/d45/d65/c8a 0 2024-03-23T11:47:32.684 INFO:tasks.workunit.client.1.smithi111.stdout:9/434: dwrite d0/d31/d35/f4d [0,4194304] 0 2024-03-23T11:47:32.684 INFO:tasks.workunit.client.1.smithi111.stdout:4/361: creat d0/d7/f69 x:0 0 0 2024-03-23T11:47:32.688 INFO:tasks.workunit.client.0.smithi063.stdout:3/466: write ff [5964116,78170] 0 2024-03-23T11:47:32.688 INFO:tasks.workunit.client.1.smithi111.stdout:3/334: dwrite d0/d1a/d1d/d26/f46 [0,4194304] 0 2024-03-23T11:47:32.689 INFO:tasks.workunit.client.1.smithi111.stdout:7/348: mkdir d14/d1b/d1e/d2c/d42/d5e 0 2024-03-23T11:47:32.693 INFO:tasks.workunit.client.1.smithi111.stdout:4/362: fdatasync d0/dc/d4c/d4b/d4d/f51 0 2024-03-23T11:47:32.695 INFO:tasks.workunit.client.1.smithi111.stdout:9/435: readlink d0/d4/d1c/d36/d28/l32 0 2024-03-23T11:47:32.695 INFO:tasks.workunit.client.1.smithi111.stdout:4/363: chown d0/dc/f1c 18700377 1 2024-03-23T11:47:32.698 INFO:tasks.workunit.client.1.smithi111.stdout:3/335: write d0/d1a/d1d/f43 [1441706,66584] 0 2024-03-23T11:47:32.701 INFO:tasks.workunit.client.0.smithi063.stdout:2/277: dwrite d2/f45 [0,4194304] 0 2024-03-23T11:47:32.703 INFO:tasks.workunit.client.1.smithi111.stdout:2/437: write d2/d7/da/f3b [909736,47102] 0 2024-03-23T11:47:32.705 INFO:tasks.workunit.client.1.smithi111.stdout:7/349: mkdir d14/d1b/d20/d5f 0 2024-03-23T11:47:32.706 INFO:tasks.workunit.client.0.smithi063.stdout:9/336: dwrite d4/d31/d43/f50 [0,4194304] 0 2024-03-23T11:47:32.708 INFO:tasks.workunit.client.1.smithi111.stdout:0/378: dwrite d5/f13 [8388608,4194304] 0 2024-03-23T11:47:32.709 INFO:tasks.workunit.client.1.smithi111.stdout:4/364: creat d0/dc/d4c/d4b/d4f/f6a x:0 0 0 2024-03-23T11:47:32.712 INFO:tasks.workunit.client.0.smithi063.stdout:2/278: dread - d2/de/d22/f4a zero size 2024-03-23T11:47:32.712 INFO:tasks.workunit.client.1.smithi111.stdout:9/436: symlink d0/d4/l90 0 2024-03-23T11:47:32.714 INFO:tasks.workunit.client.0.smithi063.stdout:3/467: symlink d13/d22/d2d/d3f/d58/d86/l90 0 2024-03-23T11:47:32.715 INFO:tasks.workunit.client.1.smithi111.stdout:7/350: read d14/d1b/f2d [3502316,64185] 0 2024-03-23T11:47:32.715 INFO:tasks.workunit.client.0.smithi063.stdout:4/320: dwrite d5/da/d20/d4b/d16/d37/f53 [0,4194304] 0 2024-03-23T11:47:32.715 INFO:tasks.workunit.client.1.smithi111.stdout:4/365: write d0/dc/f14 [5728331,53370] 0 2024-03-23T11:47:32.716 INFO:tasks.workunit.client.1.smithi111.stdout:3/336: getdents d0/d1a/d1d/d26/d5a/d64 0 2024-03-23T11:47:32.720 INFO:tasks.workunit.client.0.smithi063.stdout:9/337: unlink d4/d31/d43/c5b 0 2024-03-23T11:47:32.721 INFO:tasks.workunit.client.1.smithi111.stdout:0/379: creat d5/db/d1f/d39/f76 x:0 0 0 2024-03-23T11:47:32.721 INFO:tasks.workunit.client.1.smithi111.stdout:9/437: chown d0/d4/d1c/d2d/c54 2340 1 2024-03-23T11:47:32.723 INFO:tasks.workunit.client.0.smithi063.stdout:3/468: symlink d13/d22/d2d/d3f/d67/l91 0 2024-03-23T11:47:32.725 INFO:tasks.workunit.client.0.smithi063.stdout:3/469: stat d13/d22/d2d/d3e 0 2024-03-23T11:47:32.726 INFO:tasks.workunit.client.0.smithi063.stdout:9/338: fdatasync d4/f2a 0 2024-03-23T11:47:32.728 INFO:tasks.workunit.client.0.smithi063.stdout:9/339: write d4/d4c/f72 [595923,29592] 0 2024-03-23T11:47:32.730 INFO:tasks.workunit.client.1.smithi111.stdout:6/420: dwrite d2/d17/f55 [0,4194304] 0 2024-03-23T11:47:32.732 INFO:tasks.workunit.client.0.smithi063.stdout:3/470: chown d13/d22/d2d/l1d 24104202 1 2024-03-23T11:47:32.737 INFO:tasks.workunit.client.1.smithi111.stdout:3/337: symlink d0/d1a/d1d/d26/d5a/l66 0 2024-03-23T11:47:32.737 INFO:tasks.workunit.client.1.smithi111.stdout:6/421: read d2/d1a/f5e [478205,32076] 0 2024-03-23T11:47:32.743 INFO:tasks.workunit.client.1.smithi111.stdout:3/338: symlink d0/d7/d19/d5e/l67 0 2024-03-23T11:47:32.744 INFO:tasks.workunit.client.0.smithi063.stdout:2/279: link d2/d1c/f3c d2/de/d22/f5b 0 2024-03-23T11:47:32.745 INFO:tasks.workunit.client.0.smithi063.stdout:9/340: chown d4/d4c/f41 0 1 2024-03-23T11:47:32.746 INFO:tasks.workunit.client.0.smithi063.stdout:4/321: dwrite d5/da/d20/d4b/d29/d38/f55 [0,4194304] 0 2024-03-23T11:47:32.747 INFO:tasks.workunit.client.1.smithi111.stdout:3/339: write d0/f9 [4752681,36950] 0 2024-03-23T11:47:32.755 INFO:tasks.workunit.client.1.smithi111.stdout:8/328: dread d5/d26/d31/d3b/f3d [0,4194304] 0 2024-03-23T11:47:32.756 INFO:tasks.workunit.client.0.smithi063.stdout:2/280: rename d2/d1c/d51 to d2/d1c/d5c 0 2024-03-23T11:47:32.758 INFO:tasks.workunit.client.1.smithi111.stdout:4/366: dwrite d0/dc/d30/ff [0,4194304] 0 2024-03-23T11:47:32.760 INFO:tasks.workunit.client.1.smithi111.stdout:7/351: dwrite d14/d1b/d1e/d2c/d2e/d33/f3d [0,4194304] 0 2024-03-23T11:47:32.764 INFO:tasks.workunit.client.1.smithi111.stdout:8/329: write d5/d26/d31/d4d/f59 [1067225,16205] 0 2024-03-23T11:47:32.764 INFO:tasks.workunit.client.1.smithi111.stdout:4/367: read d0/d7/f1b [1848634,19554] 0 2024-03-23T11:47:32.770 INFO:tasks.workunit.client.1.smithi111.stdout:7/352: truncate d14/d1b/d1e/f59 1434618 0 2024-03-23T11:47:32.771 INFO:tasks.workunit.client.1.smithi111.stdout:4/368: creat d0/d17/f6b x:0 0 0 2024-03-23T11:47:32.774 INFO:tasks.workunit.client.1.smithi111.stdout:7/353: symlink d14/d1b/d1e/d2c/d2e/d33/l60 0 2024-03-23T11:47:32.774 INFO:tasks.workunit.client.0.smithi063.stdout:2/281: mknod d2/d23/d2e/d52/d44/c5d 0 2024-03-23T11:47:32.777 INFO:tasks.workunit.client.1.smithi111.stdout:4/369: write d0/dc/d4c/d4b/d4d/f54 [601283,29127] 0 2024-03-23T11:47:32.781 INFO:tasks.workunit.client.1.smithi111.stdout:7/354: creat d14/d1b/d20/f61 x:0 0 0 2024-03-23T11:47:32.785 INFO:tasks.workunit.client.0.smithi063.stdout:9/341: link d4/d2c/l68 d4/d31/d43/l76 0 2024-03-23T11:47:32.785 INFO:tasks.workunit.client.1.smithi111.stdout:0/380: truncate d5/db/f38 1566994 0 2024-03-23T11:47:32.788 INFO:tasks.workunit.client.0.smithi063.stdout:9/342: write d4/f47 [4226660,16420] 0 2024-03-23T11:47:32.789 INFO:tasks.workunit.client.1.smithi111.stdout:7/355: chown d14/d1b/d1e/f59 840895 1 2024-03-23T11:47:32.795 INFO:tasks.workunit.client.1.smithi111.stdout:8/330: dwrite d5/f17 [4194304,4194304] 0 2024-03-23T11:47:32.795 INFO:tasks.workunit.client.1.smithi111.stdout:0/381: stat d5/db/d1f/d19/f4c 0 2024-03-23T11:47:32.797 INFO:tasks.workunit.client.1.smithi111.stdout:7/356: creat d14/d37/d50/f62 x:0 0 0 2024-03-23T11:47:32.799 INFO:tasks.workunit.client.0.smithi063.stdout:4/322: dread d5/da/d20/d4b/d1f/f21 [0,4194304] 0 2024-03-23T11:47:32.807 INFO:tasks.workunit.client.1.smithi111.stdout:1/387: sync 2024-03-23T11:47:32.807 INFO:tasks.workunit.client.1.smithi111.stdout:8/331: rename d5/fa to d5/d26/d31/d4d/f6c 0 2024-03-23T11:47:32.810 INFO:tasks.workunit.client.1.smithi111.stdout:4/370: dwrite d0/dc/f55 [0,4194304] 0 2024-03-23T11:47:32.812 INFO:tasks.workunit.client.0.smithi063.stdout:9/343: dwrite d4/d2c/f3f [0,4194304] 0 2024-03-23T11:47:32.813 INFO:tasks.workunit.client.1.smithi111.stdout:3/340: dwrite d0/f16 [0,4194304] 0 2024-03-23T11:47:32.814 INFO:tasks.workunit.client.1.smithi111.stdout:6/422: dwrite d2/d35/f6f [0,4194304] 0 2024-03-23T11:47:32.819 INFO:tasks.workunit.client.1.smithi111.stdout:3/341: chown d0/d7/c65 0 1 2024-03-23T11:47:32.820 INFO:tasks.workunit.client.1.smithi111.stdout:4/371: fdatasync d0/dc/d4c/d2a/f38 0 2024-03-23T11:47:32.821 INFO:tasks.workunit.client.0.smithi063.stdout:9/344: mknod d4/d31/d70/c77 0 2024-03-23T11:47:32.821 INFO:tasks.workunit.client.1.smithi111.stdout:9/438: dread d0/d4/d1c/d36/f18 [0,4194304] 0 2024-03-23T11:47:32.822 INFO:tasks.workunit.client.0.smithi063.stdout:2/282: dwrite d2/de/d22/f5b [0,4194304] 0 2024-03-23T11:47:32.823 INFO:tasks.workunit.client.0.smithi063.stdout:4/323: dread d5/da/d20/d4b/d29/d38/f55 [0,4194304] 0 2024-03-23T11:47:32.826 INFO:tasks.workunit.client.1.smithi111.stdout:8/332: write d5/d1b/f68 [1035810,9696] 0 2024-03-23T11:47:32.826 INFO:tasks.workunit.client.1.smithi111.stdout:4/372: chown d0/d7/c20 960877 1 2024-03-23T11:47:32.827 INFO:tasks.workunit.client.1.smithi111.stdout:0/382: getdents d5/db/d1f/d19 0 2024-03-23T11:47:32.829 INFO:tasks.workunit.client.0.smithi063.stdout:9/345: stat d4/d4c/d19/l2b 0 2024-03-23T11:47:32.829 INFO:tasks.workunit.client.0.smithi063.stdout:2/283: chown d2/d23/d2e/d52/c59 202 1 2024-03-23T11:47:32.836 INFO:tasks.workunit.client.0.smithi063.stdout:4/324: creat d5/da/d20/d4b/f5a x:0 0 0 2024-03-23T11:47:32.837 INFO:tasks.workunit.client.1.smithi111.stdout:9/439: mkdir d0/d4/d10/d91 0 2024-03-23T11:47:32.837 INFO:tasks.workunit.client.1.smithi111.stdout:4/373: symlink d0/dc/d4c/d4b/d4d/l6c 0 2024-03-23T11:47:32.839 INFO:tasks.workunit.client.1.smithi111.stdout:8/333: mknod d5/d26/d2f/d42/c6d 0 2024-03-23T11:47:32.844 INFO:tasks.workunit.client.1.smithi111.stdout:1/388: dwrite d0/d5/f2f [0,4194304] 0 2024-03-23T11:47:32.847 INFO:tasks.workunit.client.1.smithi111.stdout:4/374: truncate d0/d7/da/f5b 382620 0 2024-03-23T11:47:32.847 INFO:tasks.workunit.client.1.smithi111.stdout:6/423: dwrite d2/d1a/f72 [0,4194304] 0 2024-03-23T11:47:32.848 INFO:tasks.workunit.client.1.smithi111.stdout:9/440: unlink d0/d31/d41/d44/f88 0 2024-03-23T11:47:32.849 INFO:tasks.workunit.client.1.smithi111.stdout:5/429: sync 2024-03-23T11:47:32.849 INFO:tasks.workunit.client.1.smithi111.stdout:2/438: sync 2024-03-23T11:47:32.851 INFO:tasks.workunit.client.1.smithi111.stdout:0/383: rename d5/db/d1f/d19/c1e to d5/c77 0 2024-03-23T11:47:32.853 INFO:tasks.workunit.client.0.smithi063.stdout:2/284: truncate d2/de/f2b 199733 0 2024-03-23T11:47:32.862 INFO:tasks.workunit.client.0.smithi063.stdout:4/325: dwrite d5/d1e/d2b/f58 [0,4194304] 0 2024-03-23T11:47:32.863 INFO:tasks.workunit.client.1.smithi111.stdout:7/357: truncate d14/d1b/d22/f4d 536856 0 2024-03-23T11:47:32.863 INFO:tasks.workunit.client.0.smithi063.stdout:2/285: creat d2/d23/d2e/f5e x:0 0 0 2024-03-23T11:47:32.874 INFO:tasks.workunit.client.1.smithi111.stdout:6/424: dread d2/d17/f55 [0,4194304] 0 2024-03-23T11:47:32.874 INFO:tasks.workunit.client.1.smithi111.stdout:3/342: rename d0/d1a/d2b/d38/l3d to d0/d1a/l68 0 2024-03-23T11:47:32.878 INFO:tasks.workunit.client.0.smithi063.stdout:4/326: link d5/d1e/d2b/c3b d5/da/d20/d4b/d16/c5b 0 2024-03-23T11:47:32.879 INFO:tasks.workunit.client.1.smithi111.stdout:8/334: dwrite d5/d26/d31/d5e/f49 [0,4194304] 0 2024-03-23T11:47:32.880 INFO:tasks.workunit.client.1.smithi111.stdout:5/430: getdents d1/d3/d6 0 2024-03-23T11:47:32.883 INFO:tasks.workunit.client.1.smithi111.stdout:4/375: dwrite d0/dc/f55 [0,4194304] 0 2024-03-23T11:47:32.884 INFO:tasks.workunit.client.1.smithi111.stdout:3/343: readlink d0/le 0 2024-03-23T11:47:32.886 INFO:tasks.workunit.client.1.smithi111.stdout:7/358: creat d14/d37/d50/f63 x:0 0 0 2024-03-23T11:47:32.887 INFO:tasks.workunit.client.1.smithi111.stdout:1/389: rename d0/l32 to d0/d2b/d69/l86 0 2024-03-23T11:47:32.889 INFO:tasks.workunit.client.0.smithi063.stdout:4/327: creat d5/da/d20/d4b/d16/f5c x:0 0 0 2024-03-23T11:47:32.889 INFO:tasks.workunit.client.0.smithi063.stdout:4/328: mkdir d5/da/d20/d4b/d16/d50/d5d 0 2024-03-23T11:47:32.892 INFO:tasks.workunit.client.1.smithi111.stdout:3/344: chown d0/fd 27743228 1 2024-03-23T11:47:32.893 INFO:tasks.workunit.client.0.smithi063.stdout:7/305: sync 2024-03-23T11:47:32.894 INFO:tasks.workunit.client.1.smithi111.stdout:0/384: dwrite d5/db/d1f/d47/f5a [0,4194304] 0 2024-03-23T11:47:32.894 INFO:tasks.workunit.client.1.smithi111.stdout:9/441: dwrite d0/d31/d35/f4d [0,4194304] 0 2024-03-23T11:47:32.895 INFO:tasks.workunit.client.0.smithi063.stdout:4/329: creat d5/da/d20/d4b/f5e x:0 0 0 2024-03-23T11:47:32.896 INFO:tasks.workunit.client.1.smithi111.stdout:6/425: symlink d2/d73/l8b 0 2024-03-23T11:47:32.897 INFO:tasks.workunit.client.0.smithi063.stdout:2/286: chown d2/de/f2b 667 1 2024-03-23T11:47:32.897 INFO:tasks.workunit.client.0.smithi063.stdout:2/287: chown d2/de/d22 31716 1 2024-03-23T11:47:32.898 INFO:tasks.workunit.client.0.smithi063.stdout:4/330: read d5/da/d20/f48 [1526683,94662] 0 2024-03-23T11:47:32.901 INFO:tasks.workunit.client.0.smithi063.stdout:4/331: fdatasync d5/d1e/d2b/f35 0 2024-03-23T11:47:32.902 INFO:tasks.workunit.client.1.smithi111.stdout:8/335: creat d5/d26/d31/d3b/f6e x:0 0 0 2024-03-23T11:47:32.902 INFO:tasks.workunit.client.1.smithi111.stdout:1/390: readlink d0/l2a 0 2024-03-23T11:47:32.904 INFO:tasks.workunit.client.1.smithi111.stdout:2/439: dwrite d2/d11/d17/d1f/f20 [0,4194304] 0 2024-03-23T11:47:32.905 INFO:tasks.workunit.client.1.smithi111.stdout:3/345: fdatasync d0/d1a/d1d/f49 0 2024-03-23T11:47:32.905 INFO:tasks.workunit.client.1.smithi111.stdout:6/426: unlink d2/f27 0 2024-03-23T11:47:32.907 INFO:tasks.workunit.client.1.smithi111.stdout:7/359: symlink d14/d1b/d1e/d2c/d2e/d33/l64 0 2024-03-23T11:47:32.910 INFO:tasks.workunit.client.1.smithi111.stdout:1/391: chown d0/d5/dd/d1e/d54/l68 0 1 2024-03-23T11:47:32.911 INFO:tasks.workunit.client.0.smithi063.stdout:4/332: creat d5/da/d20/d4b/d16/d50/f5f x:0 0 0 2024-03-23T11:47:32.914 INFO:tasks.workunit.client.1.smithi111.stdout:7/360: write d14/f36 [354849,78641] 0 2024-03-23T11:47:32.914 INFO:tasks.workunit.client.1.smithi111.stdout:9/442: mkdir d0/d4/d1c/d2d/d3b/d42/d92 0 2024-03-23T11:47:32.918 INFO:tasks.workunit.client.1.smithi111.stdout:2/440: creat d2/d11/d17/d3e/d6b/f9c x:0 0 0 2024-03-23T11:47:32.923 INFO:tasks.workunit.client.1.smithi111.stdout:1/392: truncate d0/f1 5230940 0 2024-03-23T11:47:32.924 INFO:tasks.workunit.client.1.smithi111.stdout:5/431: dread d1/d3/d6/d8b/d59/d74/f21 [0,4194304] 0 2024-03-23T11:47:32.926 INFO:tasks.workunit.client.0.smithi063.stdout:4/333: dread d5/da/d20/d4b/d1f/f41 [0,4194304] 0 2024-03-23T11:47:32.928 INFO:tasks.workunit.client.1.smithi111.stdout:1/393: write d0/d2b/d37/f7d [497300,90143] 0 2024-03-23T11:47:32.933 INFO:tasks.workunit.client.1.smithi111.stdout:0/385: dwrite d5/db/d1f/f37 [0,4194304] 0 2024-03-23T11:47:32.935 INFO:tasks.workunit.client.1.smithi111.stdout:1/394: chown d0/d5/f16 3 1 2024-03-23T11:47:32.936 INFO:tasks.workunit.client.0.smithi063.stdout:7/306: dread d5/d22/f24 [0,4194304] 0 2024-03-23T11:47:32.938 INFO:tasks.workunit.client.1.smithi111.stdout:5/432: creat d1/d3/d6/d7/d8c/f96 x:0 0 0 2024-03-23T11:47:32.940 INFO:tasks.workunit.client.1.smithi111.stdout:0/386: chown d5/db/d1f/d39/c60 14 1 2024-03-23T11:47:32.940 INFO:tasks.workunit.client.1.smithi111.stdout:7/361: rmdir d14/d51 0 2024-03-23T11:47:32.943 INFO:tasks.workunit.client.1.smithi111.stdout:1/395: write d0/d6/f27 [4366908,59158] 0 2024-03-23T11:47:32.946 INFO:tasks.workunit.client.0.smithi063.stdout:2/288: truncate d2/de/f2b 46566 0 2024-03-23T11:47:32.946 INFO:tasks.workunit.client.0.smithi063.stdout:7/307: dread d5/d6/d19/d1f/d37/d20/f53 [0,4194304] 0 2024-03-23T11:47:32.947 INFO:tasks.workunit.client.1.smithi111.stdout:6/427: getdents d2/d1a 0 2024-03-23T11:47:32.948 INFO:tasks.workunit.client.0.smithi063.stdout:2/289: fdatasync d2/d23/d56/f57 0 2024-03-23T11:47:32.948 INFO:tasks.workunit.client.1.smithi111.stdout:8/336: dwrite d5/d26/d31/d3b/f60 [0,4194304] 0 2024-03-23T11:47:32.948 INFO:tasks.workunit.client.1.smithi111.stdout:9/443: dwrite d0/d4/f47 [0,4194304] 0 2024-03-23T11:47:32.949 INFO:tasks.workunit.client.0.smithi063.stdout:4/334: dwrite d5/da/d20/d4b/f42 [0,4194304] 0 2024-03-23T11:47:32.952 INFO:tasks.workunit.client.0.smithi063.stdout:7/308: chown d5/d6/d19/d1f/d37/f3b 8555 1 2024-03-23T11:47:32.953 INFO:tasks.workunit.client.1.smithi111.stdout:5/433: write d1/d3/d6/f2b [811394,102329] 0 2024-03-23T11:47:32.953 INFO:tasks.workunit.client.1.smithi111.stdout:4/376: truncate d0/dc/d30/ff 1353214 0 2024-03-23T11:47:32.954 INFO:tasks.workunit.client.1.smithi111.stdout:8/337: read d5/fe [449851,75674] 0 2024-03-23T11:47:32.955 INFO:tasks.workunit.client.1.smithi111.stdout:3/346: truncate d0/d7/f47 2909937 0 2024-03-23T11:47:32.956 INFO:tasks.workunit.client.0.smithi063.stdout:4/335: creat d5/da/d20/d4b/d29/d38/f60 x:0 0 0 2024-03-23T11:47:32.956 INFO:tasks.workunit.client.1.smithi111.stdout:7/362: write d14/d1b/d20/f61 [1045731,95512] 0 2024-03-23T11:47:32.957 INFO:tasks.workunit.client.1.smithi111.stdout:2/441: dwrite d2/d5c/f90 [0,4194304] 0 2024-03-23T11:47:32.957 INFO:tasks.workunit.client.1.smithi111.stdout:0/387: mknod d5/db/d4d/c78 0 2024-03-23T11:47:32.959 INFO:tasks.workunit.client.0.smithi063.stdout:7/309: creat d5/d6/d19/d1f/d47/d4f/d5b/f66 x:0 0 0 2024-03-23T11:47:32.963 INFO:tasks.workunit.client.1.smithi111.stdout:4/377: truncate d0/d7/da/f46 1263082 0 2024-03-23T11:47:32.964 INFO:tasks.workunit.client.0.smithi063.stdout:7/310: chown d5/d6/d39/f62 65 1 2024-03-23T11:47:32.965 INFO:tasks.workunit.client.0.smithi063.stdout:4/336: unlink d5/da/d20/d4b/d29/c40 0 2024-03-23T11:47:32.967 INFO:tasks.workunit.client.1.smithi111.stdout:7/363: symlink d14/d44/l65 0 2024-03-23T11:47:32.967 INFO:tasks.workunit.client.1.smithi111.stdout:4/378: truncate d0/dc/d4c/d4b/d4d/f51 4917033 0 2024-03-23T11:47:32.967 INFO:tasks.workunit.client.0.smithi063.stdout:4/337: chown d5/da/d20/d4b/d29/d38/f55 614939 1 2024-03-23T11:47:32.971 INFO:tasks.workunit.client.1.smithi111.stdout:6/428: rename d2/d17/f4d to d2/d1a/d1c/d2c/f8c 0 2024-03-23T11:47:32.972 INFO:tasks.workunit.client.1.smithi111.stdout:3/347: symlink d0/l69 0 2024-03-23T11:47:32.973 INFO:tasks.workunit.client.0.smithi063.stdout:4/338: rmdir d5/da/d20/d4b/d16 39 2024-03-23T11:47:32.977 INFO:tasks.workunit.client.0.smithi063.stdout:4/339: mknod d5/da/d20/d4b/d16/c61 0 2024-03-23T11:47:32.979 INFO:tasks.workunit.client.1.smithi111.stdout:8/338: creat d5/d26/d2f/d62/f6f x:0 0 0 2024-03-23T11:47:32.981 INFO:tasks.workunit.client.1.smithi111.stdout:2/442: rename d2/d32/d4b to d2/d11/d17/d3e/d6b/d9d 0 2024-03-23T11:47:32.984 INFO:tasks.workunit.client.0.smithi063.stdout:4/340: rmdir d5/da/d20/d4b/d29/d38 39 2024-03-23T11:47:32.986 INFO:tasks.workunit.client.0.smithi063.stdout:4/341: write d5/fc [2893983,75421] 0 2024-03-23T11:47:32.987 INFO:tasks.workunit.client.1.smithi111.stdout:9/444: dwrite d0/d4/f83 [0,4194304] 0 2024-03-23T11:47:32.988 INFO:tasks.workunit.client.0.smithi063.stdout:4/342: fsync d5/da/d20/d4b/f5e 0 2024-03-23T11:47:32.989 INFO:tasks.workunit.client.1.smithi111.stdout:9/445: stat d0/d31/d41/c4e 0 2024-03-23T11:47:32.991 INFO:tasks.workunit.client.0.smithi063.stdout:4/343: write d5/da/d20/d4b/d1f/f26 [569727,42007] 0 2024-03-23T11:47:32.997 INFO:tasks.workunit.client.1.smithi111.stdout:6/429: dwrite d2/f57 [0,4194304] 0 2024-03-23T11:47:32.999 INFO:tasks.workunit.client.0.smithi063.stdout:4/344: link d5/da/d20/d4b/d16/d32/l4a d5/da/d20/d4b/d29/d38/l62 0 2024-03-23T11:47:32.999 INFO:tasks.workunit.client.1.smithi111.stdout:6/430: dread - d2/f3b zero size 2024-03-23T11:47:33.001 INFO:tasks.workunit.client.1.smithi111.stdout:7/364: mknod d14/d44/d4b/c66 0 2024-03-23T11:47:33.004 INFO:tasks.workunit.client.0.smithi063.stdout:4/345: truncate d5/da/d20/f48 3902039 0 2024-03-23T11:47:33.004 INFO:tasks.workunit.client.1.smithi111.stdout:0/388: creat d5/db/d1f/f79 x:0 0 0 2024-03-23T11:47:33.004 INFO:tasks.workunit.client.1.smithi111.stdout:5/434: dread d1/d3/fc [0,4194304] 0 2024-03-23T11:47:33.004 INFO:tasks.workunit.client.1.smithi111.stdout:8/339: creat d5/d26/d31/d3b/f70 x:0 0 0 2024-03-23T11:47:33.005 INFO:tasks.workunit.client.1.smithi111.stdout:5/435: readlink d1/d3/d6/l12 0 2024-03-23T11:47:33.005 INFO:tasks.workunit.client.1.smithi111.stdout:3/348: symlink d0/d1a/d1d/d26/d5a/l6a 0 2024-03-23T11:47:33.007 INFO:tasks.workunit.client.0.smithi063.stdout:4/346: chown d5/da/d20/d4b/d16/c61 52 1 2024-03-23T11:47:33.007 INFO:tasks.workunit.client.1.smithi111.stdout:2/443: unlink d2/d32/d91/f6d 0 2024-03-23T11:47:33.013 INFO:tasks.workunit.client.1.smithi111.stdout:7/365: mknod d14/d1b/d1e/d2c/c67 0 2024-03-23T11:47:33.016 INFO:tasks.workunit.client.1.smithi111.stdout:8/340: write d5/d26/d31/d3b/f3d [869903,115100] 0 2024-03-23T11:47:33.016 INFO:tasks.workunit.client.1.smithi111.stdout:4/379: creat d0/dc/d4c/f6d x:0 0 0 2024-03-23T11:47:33.017 INFO:tasks.workunit.client.1.smithi111.stdout:7/366: write d14/d1b/d1e/d2c/f31 [901017,104083] 0 2024-03-23T11:47:33.018 INFO:tasks.workunit.client.0.smithi063.stdout:4/347: symlink d5/d1e/d2b/l63 0 2024-03-23T11:47:33.019 INFO:tasks.workunit.client.1.smithi111.stdout:3/349: mknod d0/d55/c6b 0 2024-03-23T11:47:33.021 INFO:tasks.workunit.client.1.smithi111.stdout:2/444: creat d2/d11/d17/d3e/d6b/d2e/d3f/f9e x:0 0 0 2024-03-23T11:47:33.023 INFO:tasks.workunit.client.0.smithi063.stdout:4/348: creat d5/da/d20/d4b/d16/d50/f64 x:0 0 0 2024-03-23T11:47:33.028 INFO:tasks.workunit.client.0.smithi063.stdout:4/349: rename d5/d1e/d2b/l63 to d5/d1e/l65 0 2024-03-23T11:47:33.029 INFO:tasks.workunit.client.1.smithi111.stdout:8/341: dread d5/f11 [0,4194304] 0 2024-03-23T11:47:33.029 INFO:tasks.workunit.client.1.smithi111.stdout:0/389: creat d5/f7a x:0 0 0 2024-03-23T11:47:33.031 INFO:tasks.workunit.client.1.smithi111.stdout:3/350: rename d0/d1a/d1d/d26/f63 to d0/d1a/d2b/f6c 0 2024-03-23T11:47:33.032 INFO:tasks.workunit.client.1.smithi111.stdout:8/342: dread - d5/d26/d31/d3b/f70 zero size 2024-03-23T11:47:33.033 INFO:tasks.workunit.client.1.smithi111.stdout:7/367: creat d14/d1b/d20/d5f/f68 x:0 0 0 2024-03-23T11:47:33.035 INFO:tasks.workunit.client.1.smithi111.stdout:8/343: read d5/f11 [87876,28657] 0 2024-03-23T11:47:33.036 INFO:tasks.workunit.client.1.smithi111.stdout:0/390: symlink d5/db/d4d/l7b 0 2024-03-23T11:47:33.036 INFO:tasks.workunit.client.1.smithi111.stdout:5/436: dread d1/d3/d6/f3b [0,4194304] 0 2024-03-23T11:47:33.040 INFO:tasks.workunit.client.1.smithi111.stdout:8/344: write d5/f1a [303085,56189] 0 2024-03-23T11:47:33.040 INFO:tasks.workunit.client.1.smithi111.stdout:5/437: chown d1/d3/d6/d7/d7b 1849165 1 2024-03-23T11:47:33.041 INFO:tasks.workunit.client.1.smithi111.stdout:6/431: dwrite d2/d1a/d1c/d2c/d51/f5f [0,4194304] 0 2024-03-23T11:47:33.041 INFO:tasks.workunit.client.1.smithi111.stdout:3/351: symlink d0/d1a/d1d/l6d 0 2024-03-23T11:47:33.045 INFO:tasks.workunit.client.0.smithi063.stdout:4/350: dwrite d5/da/d20/d4b/d1f/f39 [0,4194304] 0 2024-03-23T11:47:33.048 INFO:tasks.workunit.client.1.smithi111.stdout:2/445: rename d2/de/d6c/f81 to d2/f9f 0 2024-03-23T11:47:33.049 INFO:tasks.workunit.client.0.smithi063.stdout:4/351: write d5/da/d20/f3c [1032890,126617] 0 2024-03-23T11:47:33.053 INFO:tasks.workunit.client.1.smithi111.stdout:9/446: dwrite d0/d4/f83 [4194304,4194304] 0 2024-03-23T11:47:33.060 INFO:tasks.workunit.client.1.smithi111.stdout:1/396: sync 2024-03-23T11:47:33.060 INFO:tasks.workunit.client.1.smithi111.stdout:4/380: truncate d0/dc/d4c/d2a/f40 3488343 0 2024-03-23T11:47:33.065 INFO:tasks.workunit.client.1.smithi111.stdout:7/368: dwrite d14/d1b/d20/f57 [0,4194304] 0 2024-03-23T11:47:33.065 INFO:tasks.workunit.client.1.smithi111.stdout:0/391: getdents d5/db/d1f 0 2024-03-23T11:47:33.066 INFO:tasks.workunit.client.1.smithi111.stdout:8/345: mkdir d5/d26/d31/d5e/d46/d4b/d51/d5a/d6a/d71 0 2024-03-23T11:47:33.068 INFO:tasks.workunit.client.1.smithi111.stdout:2/446: mkdir d2/d11/d17/d1f/da0 0 2024-03-23T11:47:33.071 INFO:tasks.workunit.client.1.smithi111.stdout:9/447: creat d0/d31/d41/d71/f93 x:0 0 0 2024-03-23T11:47:33.071 INFO:tasks.workunit.client.1.smithi111.stdout:2/447: chown d2/d11/d17/d3e/d6b/f28 10442 1 2024-03-23T11:47:33.072 INFO:tasks.workunit.client.1.smithi111.stdout:4/381: rmdir d0/d17 39 2024-03-23T11:47:33.072 INFO:tasks.workunit.client.1.smithi111.stdout:5/438: write d1/d3/d6/d7/f10 [2216073,42239] 0 2024-03-23T11:47:33.074 INFO:tasks.workunit.client.1.smithi111.stdout:7/369: rmdir d14/d1b/d1e/d2c 39 2024-03-23T11:47:33.075 INFO:tasks.workunit.client.1.smithi111.stdout:1/397: mkdir d0/d5/dd/d1e/d54/d87 0 2024-03-23T11:47:33.076 INFO:tasks.workunit.client.1.smithi111.stdout:8/346: rename d5/c20 to d5/d26/d2f/d42/d63/c72 0 2024-03-23T11:47:33.077 INFO:tasks.workunit.client.0.smithi063.stdout:4/352: dwrite d5/da/d20/d4b/d1f/f43 [0,4194304] 0 2024-03-23T11:47:33.081 INFO:tasks.workunit.client.1.smithi111.stdout:1/398: fsync d0/d5/dd/d1e/f31 0 2024-03-23T11:47:33.082 INFO:tasks.workunit.client.1.smithi111.stdout:7/370: mkdir d14/d44/d4b/d69 0 2024-03-23T11:47:33.083 INFO:tasks.workunit.client.1.smithi111.stdout:2/448: creat d2/d32/d98/fa1 x:0 0 0 2024-03-23T11:47:33.084 INFO:tasks.workunit.client.1.smithi111.stdout:9/448: read d0/d4/d1c/d36/d28/f5e [2037446,120250] 0 2024-03-23T11:47:33.086 INFO:tasks.workunit.client.1.smithi111.stdout:0/392: rename d5/db/f51 to d5/db/d1f/d47/d59/d5b/f7c 0 2024-03-23T11:47:33.087 INFO:tasks.workunit.client.1.smithi111.stdout:5/439: dread d1/d3/d6/f2b [0,4194304] 0 2024-03-23T11:47:33.092 INFO:tasks.workunit.client.1.smithi111.stdout:0/393: chown d5/db/d1f/l5e 463 1 2024-03-23T11:47:33.092 INFO:tasks.workunit.client.1.smithi111.stdout:6/432: dwrite d2/d1f/d45/d49/f5d [0,4194304] 0 2024-03-23T11:47:33.099 INFO:tasks.workunit.client.1.smithi111.stdout:1/399: mknod d0/d5/dd/d1e/d54/c88 0 2024-03-23T11:47:33.101 INFO:tasks.workunit.client.0.smithi063.stdout:4/353: dwrite d5/da/d20/d4b/d1f/f21 [4194304,4194304] 0 2024-03-23T11:47:33.105 INFO:tasks.workunit.client.0.smithi063.stdout:4/354: mknod d5/d1e/c66 0 2024-03-23T11:47:33.107 INFO:tasks.workunit.client.0.smithi063.stdout:4/355: chown d5/d1e/c66 54 1 2024-03-23T11:47:33.108 INFO:tasks.workunit.client.1.smithi111.stdout:2/449: rename d2/d11/d17/d3e/f4e to d2/d32/fa2 0 2024-03-23T11:47:33.109 INFO:tasks.workunit.client.0.smithi063.stdout:4/356: dread - d5/da/d20/d4b/d16/f5c zero size 2024-03-23T11:47:33.111 INFO:tasks.workunit.client.0.smithi063.stdout:6/269: sync 2024-03-23T11:47:33.112 INFO:tasks.workunit.client.0.smithi063.stdout:6/270: chown d1/d15 1620301015 1 2024-03-23T11:47:33.113 INFO:tasks.workunit.client.1.smithi111.stdout:3/352: dread d0/d7/f1e [0,4194304] 0 2024-03-23T11:47:33.113 INFO:tasks.workunit.client.1.smithi111.stdout:6/433: mkdir d2/d1f/d45/d49/d8d 0 2024-03-23T11:47:33.114 INFO:tasks.workunit.client.0.smithi063.stdout:4/357: chown d5/d1e/d2b/l3a 773 1 2024-03-23T11:47:33.115 INFO:tasks.workunit.client.1.smithi111.stdout:4/382: truncate d0/d7/da/f13 2573684 0 2024-03-23T11:47:33.117 INFO:tasks.workunit.client.1.smithi111.stdout:8/347: getdents d5/d26/d31/d4d 0 2024-03-23T11:47:33.120 INFO:tasks.workunit.client.1.smithi111.stdout:5/440: dwrite d1/f17 [4194304,4194304] 0 2024-03-23T11:47:33.120 INFO:tasks.workunit.client.1.smithi111.stdout:2/450: unlink d2/de/c59 0 2024-03-23T11:47:33.120 INFO:tasks.workunit.client.1.smithi111.stdout:1/400: mknod d0/d2b/d69/c89 0 2024-03-23T11:47:33.123 INFO:tasks.workunit.client.1.smithi111.stdout:0/394: dwrite d5/db/d1f/f22 [0,4194304] 0 2024-03-23T11:47:33.124 INFO:tasks.workunit.client.1.smithi111.stdout:3/353: creat d0/d1a/d1d/f6e x:0 0 0 2024-03-23T11:47:33.124 INFO:tasks.workunit.client.0.smithi063.stdout:4/358: chown d5/da/d20/d4b/d16/d37/f57 3 1 2024-03-23T11:47:33.124 INFO:tasks.workunit.client.0.smithi063.stdout:6/271: dread d1/dc/d3a/f3f [4194304,4194304] 0 2024-03-23T11:47:33.124 INFO:tasks.workunit.client.1.smithi111.stdout:4/383: rmdir d0/dc/d1f 39 2024-03-23T11:47:33.125 INFO:tasks.workunit.client.1.smithi111.stdout:8/348: write d5/d26/d31/d5e/d46/d4b/d51/d5a/f48 [982703,106937] 0 2024-03-23T11:47:33.125 INFO:tasks.workunit.client.1.smithi111.stdout:5/441: chown d1/d3/d6/f57 17 1 2024-03-23T11:47:33.126 INFO:tasks.workunit.client.1.smithi111.stdout:6/434: symlink d2/d1a/d1c/d2c/l8e 0 2024-03-23T11:47:33.127 INFO:tasks.workunit.client.1.smithi111.stdout:1/401: chown d0/d6/d48/d4d/c51 1332504 1 2024-03-23T11:47:33.129 INFO:tasks.workunit.client.1.smithi111.stdout:0/395: chown d5/db/d1f/f37 4654 1 2024-03-23T11:47:33.131 INFO:tasks.workunit.client.1.smithi111.stdout:3/354: write d0/d1a/d2b/f33 [933093,123188] 0 2024-03-23T11:47:33.132 INFO:tasks.workunit.client.1.smithi111.stdout:6/435: chown d2/d1f/d45/f62 227671396 1 2024-03-23T11:47:33.133 INFO:tasks.workunit.client.1.smithi111.stdout:0/396: fdatasync d5/db/d1f/d48/f66 0 2024-03-23T11:47:33.134 INFO:tasks.workunit.client.1.smithi111.stdout:4/384: symlink d0/dc/d4c/d4b/d4d/l6e 0 2024-03-23T11:47:33.135 INFO:tasks.workunit.client.1.smithi111.stdout:6/436: readlink d2/d35/d70/l71 0 2024-03-23T11:47:33.138 INFO:tasks.workunit.client.1.smithi111.stdout:1/402: unlink d0/d6/f42 0 2024-03-23T11:47:33.139 INFO:tasks.workunit.client.1.smithi111.stdout:5/442: creat d1/d3/d6/d7/d63/d2c/d82/d66/f97 x:0 0 0 2024-03-23T11:47:33.142 INFO:tasks.workunit.client.1.smithi111.stdout:2/451: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:33.143 INFO:tasks.workunit.client.1.smithi111.stdout:1/403: write d0/d2b/d37/f74 [811778,130662] 0 2024-03-23T11:47:33.144 INFO:tasks.workunit.client.1.smithi111.stdout:0/397: creat d5/db/d1f/d47/d59/f7d x:0 0 0 2024-03-23T11:47:33.148 INFO:tasks.workunit.client.0.smithi063.stdout:4/359: dwrite d5/da/d20/d4b/f5a [0,4194304] 0 2024-03-23T11:47:33.149 INFO:tasks.workunit.client.1.smithi111.stdout:4/385: creat d0/dc/d4c/f6f x:0 0 0 2024-03-23T11:47:33.151 INFO:tasks.workunit.client.1.smithi111.stdout:8/349: dwrite d5/d26/d31/d3b/f70 [0,4194304] 0 2024-03-23T11:47:33.154 INFO:tasks.workunit.client.1.smithi111.stdout:2/452: mkdir d2/d11/d17/d3e/d5d/da3 0 2024-03-23T11:47:33.158 INFO:tasks.workunit.client.1.smithi111.stdout:4/386: creat d0/dc/d4c/d2a/f70 x:0 0 0 2024-03-23T11:47:33.160 INFO:tasks.workunit.client.1.smithi111.stdout:5/443: link d1/d3/d6/d30/c47 d1/d3/d6/c98 0 2024-03-23T11:47:33.170 INFO:tasks.workunit.client.1.smithi111.stdout:2/453: dwrite d2/d7/f7b [0,4194304] 0 2024-03-23T11:47:33.170 INFO:tasks.workunit.client.1.smithi111.stdout:8/350: symlink d5/d32/l73 0 2024-03-23T11:47:33.171 INFO:tasks.workunit.client.1.smithi111.stdout:8/351: stat d5/c2a 0 2024-03-23T11:47:33.172 INFO:tasks.workunit.client.0.smithi063.stdout:6/272: dread d1/f12 [0,4194304] 0 2024-03-23T11:47:33.174 INFO:tasks.workunit.client.1.smithi111.stdout:6/437: dread d2/d17/f5a [0,4194304] 0 2024-03-23T11:47:33.175 INFO:tasks.workunit.client.1.smithi111.stdout:8/352: chown d5/d26/d31/d5e/d46/d4b/d51/d5a/c28 3 1 2024-03-23T11:47:33.176 INFO:tasks.workunit.client.1.smithi111.stdout:2/454: write d2/d7/da/d19/f33 [489583,50993] 0 2024-03-23T11:47:33.179 INFO:tasks.workunit.client.1.smithi111.stdout:8/353: write d5/d1b/f68 [1481227,110291] 0 2024-03-23T11:47:33.179 INFO:tasks.workunit.client.1.smithi111.stdout:4/387: mkdir d0/dc/d4c/d71 0 2024-03-23T11:47:33.180 INFO:tasks.workunit.client.0.smithi063.stdout:6/273: write d1/dc/d3a/f3f [1930258,67212] 0 2024-03-23T11:47:33.185 INFO:tasks.workunit.client.0.smithi063.stdout:6/274: write d1/dc/f32 [513568,95620] 0 2024-03-23T11:47:33.194 INFO:tasks.workunit.client.0.smithi063.stdout:6/275: rename d1/d15/c20 to d1/d2a/d41/c50 0 2024-03-23T11:47:33.195 INFO:tasks.workunit.client.1.smithi111.stdout:4/388: mkdir d0/dc/d4c/d4b/d4f/d72 0 2024-03-23T11:47:33.195 INFO:tasks.workunit.client.0.smithi063.stdout:4/360: dwrite d5/da/d20/f48 [0,4194304] 0 2024-03-23T11:47:33.196 INFO:tasks.workunit.client.1.smithi111.stdout:5/444: dwrite d1/d3/d6/d8b/f26 [0,4194304] 0 2024-03-23T11:47:33.197 INFO:tasks.workunit.client.1.smithi111.stdout:1/404: truncate d0/d6/f34 3416918 0 2024-03-23T11:47:33.198 INFO:tasks.workunit.client.1.smithi111.stdout:5/445: chown d1/d20/f60 24 1 2024-03-23T11:47:33.202 INFO:tasks.workunit.client.1.smithi111.stdout:4/389: rename d0/f3a to d0/dc/d4c/d2a/f73 0 2024-03-23T11:47:33.204 INFO:tasks.workunit.client.0.smithi063.stdout:4/361: mknod d5/da/c67 0 2024-03-23T11:47:33.205 INFO:tasks.workunit.client.1.smithi111.stdout:6/438: creat d2/d1a/d1c/d2c/d51/f8f x:0 0 0 2024-03-23T11:47:33.205 INFO:tasks.workunit.client.0.smithi063.stdout:6/276: mknod d1/dc/d21/d49/c51 0 2024-03-23T11:47:33.208 INFO:tasks.workunit.client.0.smithi063.stdout:4/362: read d5/da/d20/d4b/d1f/f41 [1005661,8348] 0 2024-03-23T11:47:33.212 INFO:tasks.workunit.client.0.smithi063.stdout:6/277: read d1/dc/d3a/f3f [2207404,78533] 0 2024-03-23T11:47:33.215 INFO:tasks.workunit.client.0.smithi063.stdout:6/278: dread - d1/d2a/f4a zero size 2024-03-23T11:47:33.215 INFO:tasks.workunit.client.1.smithi111.stdout:8/354: write d5/d26/d31/f39 [3017258,37653] 0 2024-03-23T11:47:33.218 INFO:tasks.workunit.client.1.smithi111.stdout:1/405: dwrite d0/d5/dd/d1e/f36 [4194304,4194304] 0 2024-03-23T11:47:33.221 INFO:tasks.workunit.client.1.smithi111.stdout:5/446: dread d1/d3/d6/d7/f24 [0,4194304] 0 2024-03-23T11:47:33.224 INFO:tasks.workunit.client.1.smithi111.stdout:5/447: chown d1/d3/d6/d7/d63/d2c 413237 1 2024-03-23T11:47:33.224 INFO:tasks.workunit.client.1.smithi111.stdout:1/406: truncate d0/d5/dd/f6e 197266 0 2024-03-23T11:47:33.225 INFO:tasks.workunit.client.1.smithi111.stdout:2/455: truncate d2/d11/d17/d3e/d6b/f87 463164 0 2024-03-23T11:47:33.228 INFO:tasks.workunit.client.1.smithi111.stdout:1/407: chown d0/d6/d48/d4a/d6c/f7b 389780425 1 2024-03-23T11:47:33.230 INFO:tasks.workunit.client.1.smithi111.stdout:4/390: read - d0/dc/d4c/d2a/f73 zero size 2024-03-23T11:47:33.232 INFO:tasks.workunit.client.1.smithi111.stdout:2/456: creat d2/d11/d17/d3e/d5d/fa4 x:0 0 0 2024-03-23T11:47:33.235 INFO:tasks.workunit.client.1.smithi111.stdout:1/408: rmdir d0/d2b/d37 39 2024-03-23T11:47:33.238 INFO:tasks.workunit.client.1.smithi111.stdout:1/409: readlink d0/d5/dd/d1e/d63/l6d 0 2024-03-23T11:47:33.238 INFO:tasks.workunit.client.1.smithi111.stdout:4/391: getdents d0/dc/d4c/d4b/d4f/d72 0 2024-03-23T11:47:33.239 INFO:tasks.workunit.client.1.smithi111.stdout:2/457: mknod d2/d11/d17/d3e/d5d/d8c/ca5 0 2024-03-23T11:47:33.240 INFO:tasks.workunit.client.0.smithi063.stdout:6/279: dwrite d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:33.243 INFO:tasks.workunit.client.1.smithi111.stdout:4/392: rmdir d0/d17 39 2024-03-23T11:47:33.247 INFO:tasks.workunit.client.1.smithi111.stdout:4/393: chown d0/dc/d4c/f6d 61788 1 2024-03-23T11:47:33.248 INFO:tasks.workunit.client.1.smithi111.stdout:6/439: dwrite d2/fb [4194304,4194304] 0 2024-03-23T11:47:33.248 INFO:tasks.workunit.client.1.smithi111.stdout:5/448: dwrite d1/d3/d6/d8b/d59/d74/f5e [0,4194304] 0 2024-03-23T11:47:33.248 INFO:tasks.workunit.client.0.smithi063.stdout:1/264: sync 2024-03-23T11:47:33.249 INFO:tasks.workunit.client.1.smithi111.stdout:7/371: sync 2024-03-23T11:47:33.249 INFO:tasks.workunit.client.0.smithi063.stdout:4/363: dwrite d5/da/f2a [0,4194304] 0 2024-03-23T11:47:33.251 INFO:tasks.workunit.client.0.smithi063.stdout:1/265: write d3/f10 [3461320,60867] 0 2024-03-23T11:47:33.251 INFO:tasks.workunit.client.1.smithi111.stdout:2/458: rename d2/d11/d17/d1f/d30/f41 to d2/d11/d17/d3e/d5d/fa6 0 2024-03-23T11:47:33.252 INFO:tasks.workunit.client.0.smithi063.stdout:0/328: sync 2024-03-23T11:47:33.258 INFO:tasks.workunit.client.1.smithi111.stdout:5/449: unlink d1/d3/d6/d7/d63/d2c/d82/d66/f87 0 2024-03-23T11:47:33.260 INFO:tasks.workunit.client.1.smithi111.stdout:7/372: fsync d14/d1b/f52 0 2024-03-23T11:47:33.263 INFO:tasks.workunit.client.0.smithi063.stdout:6/280: getdents d1 0 2024-03-23T11:47:33.266 INFO:tasks.workunit.client.1.smithi111.stdout:5/450: mkdir d1/d20/d5c/d99 0 2024-03-23T11:47:33.266 INFO:tasks.workunit.client.0.smithi063.stdout:1/266: rmdir d3/df/d31 39 2024-03-23T11:47:33.269 INFO:tasks.workunit.client.1.smithi111.stdout:2/459: creat d2/d11/d17/d3e/d6b/d2e/d76/fa7 x:0 0 0 2024-03-23T11:47:33.270 INFO:tasks.workunit.client.0.smithi063.stdout:6/281: rmdir d1/dc/d3a 39 2024-03-23T11:47:33.273 INFO:tasks.workunit.client.0.smithi063.stdout:1/267: mkdir d3/d3f/d5c 0 2024-03-23T11:47:33.275 INFO:tasks.workunit.client.1.smithi111.stdout:6/440: dwrite d2/d1f/d45/f63 [0,4194304] 0 2024-03-23T11:47:33.277 INFO:tasks.workunit.client.0.smithi063.stdout:6/282: chown d1/dc/d3a/f30 11162064 1 2024-03-23T11:47:33.277 INFO:tasks.workunit.client.1.smithi111.stdout:2/460: mknod d2/d11/d17/d1f/ca8 0 2024-03-23T11:47:33.278 INFO:tasks.workunit.client.1.smithi111.stdout:5/451: getdents d1/d3/d6/d7/d63/d2c/d43/d95 0 2024-03-23T11:47:33.280 INFO:tasks.workunit.client.0.smithi063.stdout:6/283: write d1/dc/f4e [493141,83498] 0 2024-03-23T11:47:33.282 INFO:tasks.workunit.client.1.smithi111.stdout:6/441: creat d2/d1f/f90 x:0 0 0 2024-03-23T11:47:33.284 INFO:tasks.workunit.client.1.smithi111.stdout:7/373: dwrite f1 [4194304,4194304] 0 2024-03-23T11:47:33.289 INFO:tasks.workunit.client.1.smithi111.stdout:2/461: mknod d2/d11/d17/d3e/d6b/ca9 0 2024-03-23T11:47:33.293 INFO:tasks.workunit.client.1.smithi111.stdout:5/452: symlink d1/d20/d5c/l9a 0 2024-03-23T11:47:33.295 INFO:tasks.workunit.client.1.smithi111.stdout:2/462: readlink d2/d11/d17/d3e/d6b/d2e/d76/l89 0 2024-03-23T11:47:33.299 INFO:tasks.workunit.client.0.smithi063.stdout:1/268: dread d3/df/f30 [0,4194304] 0 2024-03-23T11:47:33.301 INFO:tasks.workunit.client.1.smithi111.stdout:4/394: dwrite d0/d17/f2c [0,4194304] 0 2024-03-23T11:47:33.301 INFO:tasks.workunit.client.1.smithi111.stdout:2/463: creat d2/d11/d17/d3e/d6b/d2e/d3f/faa x:0 0 0 2024-03-23T11:47:33.302 INFO:tasks.workunit.client.0.smithi063.stdout:1/269: creat d3/f5d x:0 0 0 2024-03-23T11:47:33.303 INFO:tasks.workunit.client.0.smithi063.stdout:1/270: chown d3/d3f 569146 1 2024-03-23T11:47:33.306 INFO:tasks.workunit.client.1.smithi111.stdout:5/453: creat d1/d3/d6/d7/d63/d2c/d82/d66/f9b x:0 0 0 2024-03-23T11:47:33.310 INFO:tasks.workunit.client.1.smithi111.stdout:5/454: fdatasync d1/d3/d6/d7/f53 0 2024-03-23T11:47:33.310 INFO:tasks.workunit.client.0.smithi063.stdout:0/329: dread d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:33.313 INFO:tasks.workunit.client.1.smithi111.stdout:2/464: rename d2/d7/da/ff to d2/d11/d17/d3e/d6b/d2e/d76/fab 0 2024-03-23T11:47:33.313 INFO:tasks.workunit.client.0.smithi063.stdout:6/284: truncate d1/f12 1215605 0 2024-03-23T11:47:33.315 INFO:tasks.workunit.client.1.smithi111.stdout:2/465: stat d2/d11/d17/d3e/d6b/d9d/d52/l6e 0 2024-03-23T11:47:33.316 INFO:tasks.workunit.client.1.smithi111.stdout:5/455: getdents d1/d20/d5c/d99 0 2024-03-23T11:47:33.317 INFO:tasks.workunit.client.1.smithi111.stdout:4/395: dread d0/dc/d4c/d2a/f40 [0,4194304] 0 2024-03-23T11:47:33.317 INFO:tasks.workunit.client.1.smithi111.stdout:6/442: truncate d2/d1a/d1c/d2c/d51/f5f 2930458 0 2024-03-23T11:47:33.318 INFO:tasks.workunit.client.1.smithi111.stdout:4/396: chown d0/d7/da 96564 1 2024-03-23T11:47:33.319 INFO:tasks.workunit.client.0.smithi063.stdout:0/330: symlink d2/d14/d36/l75 0 2024-03-23T11:47:33.319 INFO:tasks.workunit.client.1.smithi111.stdout:7/374: truncate d14/d37/f3f 1088918 0 2024-03-23T11:47:33.320 INFO:tasks.workunit.client.1.smithi111.stdout:5/456: write d1/d3/d6/d7/d63/d2c/f51 [332539,125781] 0 2024-03-23T11:47:33.321 INFO:tasks.workunit.client.1.smithi111.stdout:6/443: write d2/d1a/d1c/d2c/f83 [93462,42177] 0 2024-03-23T11:47:33.322 INFO:tasks.workunit.client.1.smithi111.stdout:4/397: write d0/dc/d4c/f6d [328380,26268] 0 2024-03-23T11:47:33.322 INFO:tasks.workunit.client.0.smithi063.stdout:6/285: rename d1/dc/f4e to d1/d13/f52 0 2024-03-23T11:47:33.323 INFO:tasks.workunit.client.0.smithi063.stdout:1/271: dread d3/df/f18 [0,4194304] 0 2024-03-23T11:47:33.325 INFO:tasks.workunit.client.1.smithi111.stdout:4/398: fdatasync d0/dc/f55 0 2024-03-23T11:47:33.328 INFO:tasks.workunit.client.1.smithi111.stdout:5/457: symlink d1/d3/d6/d7/d63/d2c/d82/d66/d77/l9c 0 2024-03-23T11:47:33.329 INFO:tasks.workunit.client.1.smithi111.stdout:7/375: getdents d14/d37/d50 0 2024-03-23T11:47:33.329 INFO:tasks.workunit.client.0.smithi063.stdout:6/286: rename d1/dc/f32 to d1/d13/f53 0 2024-03-23T11:47:33.330 INFO:tasks.workunit.client.0.smithi063.stdout:1/272: creat d3/d35/f5e x:0 0 0 2024-03-23T11:47:33.331 INFO:tasks.workunit.client.1.smithi111.stdout:5/458: write d1/d20/d5c/f7e [340415,85530] 0 2024-03-23T11:47:33.333 INFO:tasks.workunit.client.0.smithi063.stdout:0/331: link d2/l3b d2/d14/l76 0 2024-03-23T11:47:33.333 INFO:tasks.workunit.client.0.smithi063.stdout:1/273: chown d3/df/f36 983 1 2024-03-23T11:47:33.334 INFO:tasks.workunit.client.0.smithi063.stdout:6/287: mknod d1/d2a/d41/d42/c54 0 2024-03-23T11:47:33.335 INFO:tasks.workunit.client.1.smithi111.stdout:4/399: symlink d0/dc/l74 0 2024-03-23T11:47:33.335 INFO:tasks.workunit.client.0.smithi063.stdout:0/332: chown d2/d47 0 1 2024-03-23T11:47:33.337 INFO:tasks.workunit.client.1.smithi111.stdout:5/459: read d1/d3/d6/d8b/d59/d6c/f76 [28705,93733] 0 2024-03-23T11:47:33.338 INFO:tasks.workunit.client.1.smithi111.stdout:4/400: write d0/dc/d4c/d2a/f38 [5235279,53324] 0 2024-03-23T11:47:33.339 INFO:tasks.workunit.client.0.smithi063.stdout:1/274: creat d3/df/d31/f5f x:0 0 0 2024-03-23T11:47:33.342 INFO:tasks.workunit.client.1.smithi111.stdout:5/460: fsync d1/d3/d6/d7/d63/d2c/d82/d66/f97 0 2024-03-23T11:47:33.343 INFO:tasks.workunit.client.0.smithi063.stdout:0/333: rename d2/d47 to d2/d14/d36/d77 0 2024-03-23T11:47:33.345 INFO:tasks.workunit.client.1.smithi111.stdout:7/376: link d14/d1b/d1e/l28 d14/d1b/d1e/d2c/d2e/d33/l6a 0 2024-03-23T11:47:33.345 INFO:tasks.workunit.client.0.smithi063.stdout:1/275: truncate d3/d3f/d4d/f5b 352743 0 2024-03-23T11:47:33.346 INFO:tasks.workunit.client.1.smithi111.stdout:4/401: creat d0/dc/d4c/f75 x:0 0 0 2024-03-23T11:47:33.347 INFO:tasks.workunit.client.1.smithi111.stdout:2/466: dwrite d2/d11/d17/d3e/d6b/f79 [0,4194304] 0 2024-03-23T11:47:33.348 INFO:tasks.workunit.client.0.smithi063.stdout:1/276: write d3/df/f1a [3991596,119729] 0 2024-03-23T11:47:33.351 INFO:tasks.workunit.client.1.smithi111.stdout:2/467: truncate d2/d32/f7d 49973 0 2024-03-23T11:47:33.352 INFO:tasks.workunit.client.1.smithi111.stdout:5/461: mkdir d1/d3/d6/d7/d63/d9d 0 2024-03-23T11:47:33.357 INFO:tasks.workunit.client.0.smithi063.stdout:1/277: mknod d3/d49/c60 0 2024-03-23T11:47:33.359 INFO:tasks.workunit.client.1.smithi111.stdout:6/444: dread d2/d1f/d45/d49/f6c [0,4194304] 0 2024-03-23T11:47:33.364 INFO:tasks.workunit.client.1.smithi111.stdout:2/468: mknod d2/d11/d17/d3e/cac 0 2024-03-23T11:47:33.366 INFO:tasks.workunit.client.0.smithi063.stdout:6/288: dwrite d1/dc/d3a/f40 [0,4194304] 0 2024-03-23T11:47:33.367 INFO:tasks.workunit.client.1.smithi111.stdout:4/402: getdents d0/dc/d1f/d5e 0 2024-03-23T11:47:33.368 INFO:tasks.workunit.client.1.smithi111.stdout:2/469: symlink d2/de/d6c/lad 0 2024-03-23T11:47:33.370 INFO:tasks.workunit.client.1.smithi111.stdout:5/462: dwrite d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:33.370 INFO:tasks.workunit.client.1.smithi111.stdout:2/470: fsync d2/d32/d91/f85 0 2024-03-23T11:47:33.372 INFO:tasks.workunit.client.0.smithi063.stdout:6/289: unlink d1/f35 0 2024-03-23T11:47:33.376 INFO:tasks.workunit.client.1.smithi111.stdout:5/463: mkdir d1/d3/d6/d7/d8c/d9e 0 2024-03-23T11:47:33.378 INFO:tasks.workunit.client.1.smithi111.stdout:4/403: write d0/d7/f16 [3547750,61492] 0 2024-03-23T11:47:33.378 INFO:tasks.workunit.client.1.smithi111.stdout:2/471: unlink d2/d11/d17/d3e/d6b/d9d/f66 0 2024-03-23T11:47:33.379 INFO:tasks.workunit.client.1.smithi111.stdout:5/464: dread - d1/d3/d6/d8b/d59/d74/f6b zero size 2024-03-23T11:47:33.382 INFO:tasks.workunit.client.0.smithi063.stdout:0/334: truncate d2/d4/f5e 2439558 0 2024-03-23T11:47:33.385 INFO:tasks.workunit.client.1.smithi111.stdout:2/472: creat d2/d11/d17/d3e/d6b/d9d/fae x:0 0 0 2024-03-23T11:47:33.386 INFO:tasks.workunit.client.1.smithi111.stdout:4/404: rmdir d0/dc/d4c 39 2024-03-23T11:47:33.389 INFO:tasks.workunit.client.0.smithi063.stdout:7/311: sync 2024-03-23T11:47:33.389 INFO:tasks.workunit.client.0.smithi063.stdout:5/254: sync 2024-03-23T11:47:33.389 INFO:tasks.workunit.client.0.smithi063.stdout:8/615: sync 2024-03-23T11:47:33.389 INFO:tasks.workunit.client.1.smithi111.stdout:7/377: dread d14/d1b/d20/f2b [0,4194304] 0 2024-03-23T11:47:33.389 INFO:tasks.workunit.client.1.smithi111.stdout:2/473: mkdir d2/d11/d17/daf 0 2024-03-23T11:47:33.390 INFO:tasks.workunit.client.1.smithi111.stdout:4/405: write d0/dc/f35 [2685766,3936] 0 2024-03-23T11:47:33.390 INFO:tasks.workunit.client.0.smithi063.stdout:2/290: sync 2024-03-23T11:47:33.390 INFO:tasks.workunit.client.0.smithi063.stdout:9/346: sync 2024-03-23T11:47:33.390 INFO:tasks.workunit.client.0.smithi063.stdout:3/471: sync 2024-03-23T11:47:33.392 INFO:tasks.workunit.client.0.smithi063.stdout:0/335: rename d2/d4/dd/c49 to d2/d4/dd/d25/c78 0 2024-03-23T11:47:33.393 INFO:tasks.workunit.client.1.smithi111.stdout:6/445: dwrite d2/f31 [4194304,4194304] 0 2024-03-23T11:47:33.394 INFO:tasks.workunit.client.1.smithi111.stdout:9/449: sync 2024-03-23T11:47:33.395 INFO:tasks.workunit.client.0.smithi063.stdout:2/291: chown d2/d23/d2e/d52/d44/f50 124554 1 2024-03-23T11:47:33.396 INFO:tasks.workunit.client.1.smithi111.stdout:6/446: write d2/d35/f40 [4813472,95754] 0 2024-03-23T11:47:33.398 INFO:tasks.workunit.client.1.smithi111.stdout:2/474: mknod d2/d7/da/d19/cb0 0 2024-03-23T11:47:33.399 INFO:tasks.workunit.client.1.smithi111.stdout:7/378: rename d14/d1b/c30 to d14/d1b/d1e/d2c/d42/c6b 0 2024-03-23T11:47:33.400 INFO:tasks.workunit.client.0.smithi063.stdout:8/616: read d6/d1f/d54/d5c/d66/f83 [295394,127181] 0 2024-03-23T11:47:33.400 INFO:tasks.workunit.client.1.smithi111.stdout:4/406: getdents d0/dc/d1f/d5e 0 2024-03-23T11:47:33.401 INFO:tasks.workunit.client.1.smithi111.stdout:9/450: truncate d0/d4/d10/f2c 1515883 0 2024-03-23T11:47:33.401 INFO:tasks.workunit.client.1.smithi111.stdout:6/447: truncate d2/d1a/d1c/d2c/f7c 282845 0 2024-03-23T11:47:33.403 INFO:tasks.workunit.client.0.smithi063.stdout:7/312: symlink d5/d6/d19/l67 0 2024-03-23T11:47:33.403 INFO:tasks.workunit.client.0.smithi063.stdout:8/617: readlink d6/d1c/d32/d37/l5b 0 2024-03-23T11:47:33.403 INFO:tasks.workunit.client.1.smithi111.stdout:6/448: chown d2/d35/d70/c75 209 1 2024-03-23T11:47:33.406 INFO:tasks.workunit.client.1.smithi111.stdout:4/407: creat d0/dc/d1f/f76 x:0 0 0 2024-03-23T11:47:33.406 INFO:tasks.workunit.client.0.smithi063.stdout:0/336: creat d2/d14/f79 x:0 0 0 2024-03-23T11:47:33.407 INFO:tasks.workunit.client.0.smithi063.stdout:9/347: creat d4/d31/d43/d49/f78 x:0 0 0 2024-03-23T11:47:33.407 INFO:tasks.workunit.client.1.smithi111.stdout:4/408: stat d0/f1 0 2024-03-23T11:47:33.407 INFO:tasks.workunit.client.1.smithi111.stdout:2/475: rmdir d2/d11/d17/d3e/d6b/d2e/d76 39 2024-03-23T11:47:33.409 INFO:tasks.workunit.client.1.smithi111.stdout:9/451: chown d0/d4/d1c/d36/l8 149396406 1 2024-03-23T11:47:33.409 INFO:tasks.workunit.client.0.smithi063.stdout:2/292: fsync d2/f2f 0 2024-03-23T11:47:33.411 INFO:tasks.workunit.client.1.smithi111.stdout:4/409: truncate d0/dc/d1f/f57 963149 0 2024-03-23T11:47:33.411 INFO:tasks.workunit.client.0.smithi063.stdout:6/290: mknod d1/d2a/d41/c55 0 2024-03-23T11:47:33.411 INFO:tasks.workunit.client.0.smithi063.stdout:3/472: getdents d13/d22/d2d/d5e 0 2024-03-23T11:47:33.412 INFO:tasks.workunit.client.1.smithi111.stdout:2/476: mknod d2/d11/d17/d3e/cb1 0 2024-03-23T11:47:33.412 INFO:tasks.workunit.client.1.smithi111.stdout:9/452: chown d0/d4/d1c/d2d/f2e 11 1 2024-03-23T11:47:33.417 INFO:tasks.workunit.client.0.smithi063.stdout:0/337: stat d2/d12/f44 0 2024-03-23T11:47:33.417 INFO:tasks.workunit.client.1.smithi111.stdout:7/379: getdents d14/d37/d54 0 2024-03-23T11:47:33.419 INFO:tasks.workunit.client.0.smithi063.stdout:9/348: readlink d4/d2c/l68 0 2024-03-23T11:47:33.420 INFO:tasks.workunit.client.1.smithi111.stdout:5/465: dread d1/f17 [0,4194304] 0 2024-03-23T11:47:33.423 INFO:tasks.workunit.client.0.smithi063.stdout:2/293: creat d2/d1c/f5f x:0 0 0 2024-03-23T11:47:33.431 INFO:tasks.workunit.client.0.smithi063.stdout:5/255: dwrite d2/d28/f4f [0,4194304] 0 2024-03-23T11:47:33.434 INFO:tasks.workunit.client.1.smithi111.stdout:4/410: truncate d0/dc/d30/ff 813115 0 2024-03-23T11:47:33.434 INFO:tasks.workunit.client.1.smithi111.stdout:2/477: symlink d2/d11/d17/d3e/d6b/d2e/lb2 0 2024-03-23T11:47:33.435 INFO:tasks.workunit.client.1.smithi111.stdout:5/466: write d1/d20/f50 [4935902,29178] 0 2024-03-23T11:47:33.435 INFO:tasks.workunit.client.0.smithi063.stdout:1/278: dread d3/f10 [0,4194304] 0 2024-03-23T11:47:33.435 INFO:tasks.workunit.client.0.smithi063.stdout:6/291: chown d1/d13/f52 2047 1 2024-03-23T11:47:33.438 INFO:tasks.workunit.client.1.smithi111.stdout:6/449: link d2/c21 d2/d1f/d45/d49/c91 0 2024-03-23T11:47:33.443 INFO:tasks.workunit.client.0.smithi063.stdout:1/279: chown d3/f2a 667366 1 2024-03-23T11:47:33.444 INFO:tasks.workunit.client.1.smithi111.stdout:7/380: creat d14/d1b/d1e/d2c/d42/d5e/f6c x:0 0 0 2024-03-23T11:47:33.447 INFO:tasks.workunit.client.1.smithi111.stdout:9/453: write d0/d4/fe [970801,29921] 0 2024-03-23T11:47:33.449 INFO:tasks.workunit.client.1.smithi111.stdout:2/478: creat d2/d11/d17/d3e/d6b/fb3 x:0 0 0 2024-03-23T11:47:33.450 INFO:tasks.workunit.client.1.smithi111.stdout:9/454: fsync d0/d4/d10/f8b 0 2024-03-23T11:47:33.450 INFO:tasks.workunit.client.0.smithi063.stdout:9/349: rmdir d4/d4c/d19/d75 0 2024-03-23T11:47:33.450 INFO:tasks.workunit.client.1.smithi111.stdout:4/411: dwrite d0/dc/f1c [4194304,4194304] 0 2024-03-23T11:47:33.451 INFO:tasks.workunit.client.1.smithi111.stdout:3/355: sync 2024-03-23T11:47:33.451 INFO:tasks.workunit.client.1.smithi111.stdout:2/479: write d2/d11/d17/f21 [460732,90368] 0 2024-03-23T11:47:33.452 INFO:tasks.workunit.client.1.smithi111.stdout:6/450: creat d2/d1f/f92 x:0 0 0 2024-03-23T11:47:33.453 INFO:tasks.workunit.client.0.smithi063.stdout:4/364: sync 2024-03-23T11:47:33.453 INFO:tasks.workunit.client.1.smithi111.stdout:9/455: readlink d0/d4/d1c/d36/l8 0 2024-03-23T11:47:33.454 INFO:tasks.workunit.client.1.smithi111.stdout:5/467: rename d1/d3/d6/d8b/d59/d74/f22 to d1/d3/d6/d8b/d59/f9f 0 2024-03-23T11:47:33.457 INFO:tasks.workunit.client.1.smithi111.stdout:7/381: rmdir d14 39 2024-03-23T11:47:33.457 INFO:tasks.workunit.client.0.smithi063.stdout:5/256: creat d2/d7/d8/de/d2a/d3b/d43/f5a x:0 0 0 2024-03-23T11:47:33.458 INFO:tasks.workunit.client.1.smithi111.stdout:3/356: dread - d0/d1a/d1d/d26/d5a/f5f zero size 2024-03-23T11:47:33.458 INFO:tasks.workunit.client.0.smithi063.stdout:6/292: creat d1/dc/d21/d49/f56 x:0 0 0 2024-03-23T11:47:33.462 INFO:tasks.workunit.client.1.smithi111.stdout:4/412: readlink d0/dc/d4c/d4b/d4f/l5f 0 2024-03-23T11:47:33.463 INFO:tasks.workunit.client.0.smithi063.stdout:1/280: mknod d3/df/d2f/c61 0 2024-03-23T11:47:33.464 INFO:tasks.workunit.client.1.smithi111.stdout:4/413: truncate d0/dc/d1f/f57 1412470 0 2024-03-23T11:47:33.466 INFO:tasks.workunit.client.1.smithi111.stdout:2/480: dread - d2/d11/d17/d3e/d6b/d2e/f82 zero size 2024-03-23T11:47:33.471 INFO:tasks.workunit.client.1.smithi111.stdout:3/357: rename d0/c17 to d0/d7/d19/d5e/c6f 0 2024-03-23T11:47:33.472 INFO:tasks.workunit.client.1.smithi111.stdout:6/451: dread d2/d17/f5a [0,4194304] 0 2024-03-23T11:47:33.473 INFO:tasks.workunit.client.1.smithi111.stdout:7/382: unlink d14/d1b/d20/f41 0 2024-03-23T11:47:33.475 INFO:tasks.workunit.client.1.smithi111.stdout:7/383: chown d14/l18 655180 1 2024-03-23T11:47:33.475 INFO:tasks.workunit.client.1.smithi111.stdout:6/452: read d2/d35/f40 [2974443,51491] 0 2024-03-23T11:47:33.475 INFO:tasks.workunit.client.0.smithi063.stdout:8/618: dwrite d6/d1f/d54/d63/d90/d5d/f87 [0,4194304] 0 2024-03-23T11:47:33.476 INFO:tasks.workunit.client.1.smithi111.stdout:7/384: fsync d14/d37/d50/f62 0 2024-03-23T11:47:33.477 INFO:tasks.workunit.client.0.smithi063.stdout:5/257: write d2/f2c [309389,84852] 0 2024-03-23T11:47:33.477 INFO:tasks.workunit.client.1.smithi111.stdout:2/481: unlink d2/d32/c93 0 2024-03-23T11:47:33.479 INFO:tasks.workunit.client.1.smithi111.stdout:3/358: rmdir d0/d7/d19/d5e 39 2024-03-23T11:47:33.482 INFO:tasks.workunit.client.1.smithi111.stdout:2/482: fdatasync d2/d32/f7d 0 2024-03-23T11:47:33.482 INFO:tasks.workunit.client.0.smithi063.stdout:9/350: mknod d4/d4c/d44/c79 0 2024-03-23T11:47:33.483 INFO:tasks.workunit.client.1.smithi111.stdout:9/456: dwrite d0/d4/d1c/d36/d28/f5e [4194304,4194304] 0 2024-03-23T11:47:33.484 INFO:tasks.workunit.client.1.smithi111.stdout:4/414: mknod d0/c77 0 2024-03-23T11:47:33.486 INFO:tasks.workunit.client.1.smithi111.stdout:4/415: stat d0/dc/f14 0 2024-03-23T11:47:33.487 INFO:tasks.workunit.client.0.smithi063.stdout:1/281: truncate d3/f4 4928769 0 2024-03-23T11:47:33.487 INFO:tasks.workunit.client.0.smithi063.stdout:3/473: dwrite d13/d15/f3c [0,4194304] 0 2024-03-23T11:47:33.489 INFO:tasks.workunit.client.0.smithi063.stdout:5/258: chown d2/d7/d8/de 421035 1 2024-03-23T11:47:33.491 INFO:tasks.workunit.client.0.smithi063.stdout:2/294: dwrite d2/d23/d2e/f4b [0,4194304] 0 2024-03-23T11:47:33.493 INFO:tasks.workunit.client.0.smithi063.stdout:9/351: dread - d4/d31/f59 zero size 2024-03-23T11:47:33.497 INFO:tasks.workunit.client.0.smithi063.stdout:7/313: dwrite d5/f12 [0,4194304] 0 2024-03-23T11:47:33.498 INFO:tasks.workunit.client.0.smithi063.stdout:4/365: creat d5/da/d20/d4b/f68 x:0 0 0 2024-03-23T11:47:33.504 INFO:tasks.workunit.client.1.smithi111.stdout:9/457: dread d0/f5f [0,4194304] 0 2024-03-23T11:47:33.505 INFO:tasks.workunit.client.1.smithi111.stdout:2/483: dwrite d2/d32/d98/fa1 [0,4194304] 0 2024-03-23T11:47:33.505 INFO:tasks.workunit.client.1.smithi111.stdout:3/359: chown d0/d1a/c3f 11 1 2024-03-23T11:47:33.507 INFO:tasks.workunit.client.0.smithi063.stdout:1/282: mkdir d3/d62 0 2024-03-23T11:47:33.507 INFO:tasks.workunit.client.0.smithi063.stdout:2/295: mknod d2/d23/d2e/c60 0 2024-03-23T11:47:33.508 INFO:tasks.workunit.client.1.smithi111.stdout:8/355: sync 2024-03-23T11:47:33.508 INFO:tasks.workunit.client.1.smithi111.stdout:0/398: sync 2024-03-23T11:47:33.509 INFO:tasks.workunit.client.1.smithi111.stdout:1/410: sync 2024-03-23T11:47:33.514 INFO:tasks.workunit.client.0.smithi063.stdout:3/474: creat d13/d22/d2d/d78/f92 x:0 0 0 2024-03-23T11:47:33.518 INFO:tasks.workunit.client.0.smithi063.stdout:7/314: symlink d5/d6/d19/d1f/l68 0 2024-03-23T11:47:33.518 INFO:tasks.workunit.client.0.smithi063.stdout:5/259: mknod d2/d7/d27/c5b 0 2024-03-23T11:47:33.518 INFO:tasks.workunit.client.1.smithi111.stdout:7/385: dread d14/d1b/d1e/d2c/d2e/d33/f3d [0,4194304] 0 2024-03-23T11:47:33.520 INFO:tasks.workunit.client.1.smithi111.stdout:7/386: stat d14/d1b/d20/f3a 0 2024-03-23T11:47:33.521 INFO:tasks.workunit.client.0.smithi063.stdout:4/366: write d5/da/d20/d4b/d1f/f26 [102826,18699] 0 2024-03-23T11:47:33.522 INFO:tasks.workunit.client.0.smithi063.stdout:2/296: write d2/d23/d2e/f4b [3693358,124645] 0 2024-03-23T11:47:33.522 INFO:tasks.workunit.client.0.smithi063.stdout:9/352: mknod d4/d4c/c7a 0 2024-03-23T11:47:33.523 INFO:tasks.workunit.client.0.smithi063.stdout:6/293: dread d1/dc/f46 [0,4194304] 0 2024-03-23T11:47:33.526 INFO:tasks.workunit.client.0.smithi063.stdout:0/338: dread d2/d14/f23 [0,4194304] 0 2024-03-23T11:47:33.530 INFO:tasks.workunit.client.0.smithi063.stdout:3/475: creat d13/d22/d2d/d3f/d58/d86/f93 x:0 0 0 2024-03-23T11:47:33.532 INFO:tasks.workunit.client.0.smithi063.stdout:2/297: mknod d2/d1c/d5c/c61 0 2024-03-23T11:47:33.533 INFO:tasks.workunit.client.0.smithi063.stdout:5/260: write d2/f6 [486315,81278] 0 2024-03-23T11:47:33.534 INFO:tasks.workunit.client.0.smithi063.stdout:6/294: mkdir d1/dc/d3a/d57 0 2024-03-23T11:47:33.536 INFO:tasks.workunit.client.1.smithi111.stdout:2/484: unlink d2/d11/d17/d3e/d6b/d2e/lb2 0 2024-03-23T11:47:33.537 INFO:tasks.workunit.client.0.smithi063.stdout:7/315: write d5/f12 [4214679,35251] 0 2024-03-23T11:47:33.538 INFO:tasks.workunit.client.0.smithi063.stdout:4/367: truncate d5/d1e/d2b/f45 1359070 0 2024-03-23T11:47:33.539 INFO:tasks.workunit.client.1.smithi111.stdout:2/485: chown d2/d32/d98/fa1 315 1 2024-03-23T11:47:33.540 INFO:tasks.workunit.client.1.smithi111.stdout:7/387: dwrite d14/d1b/d20/f57 [0,4194304] 0 2024-03-23T11:47:33.540 INFO:tasks.workunit.client.1.smithi111.stdout:5/468: write d1/d3/d6/d7/f24 [196372,100780] 0 2024-03-23T11:47:33.540 INFO:tasks.workunit.client.0.smithi063.stdout:9/353: dread - d4/d4c/f5c zero size 2024-03-23T11:47:33.544 INFO:tasks.workunit.client.0.smithi063.stdout:2/298: write d2/d23/d56/f5a [797669,6363] 0 2024-03-23T11:47:33.547 INFO:tasks.workunit.client.1.smithi111.stdout:6/453: creat d2/d1a/d1c/f93 x:0 0 0 2024-03-23T11:47:33.548 INFO:tasks.workunit.client.1.smithi111.stdout:0/399: mknod d5/db/d1f/d48/c7e 0 2024-03-23T11:47:33.550 INFO:tasks.workunit.client.0.smithi063.stdout:0/339: creat d2/d4/f7a x:0 0 0 2024-03-23T11:47:33.551 INFO:tasks.workunit.client.0.smithi063.stdout:7/316: write d5/d6/d19/d1f/d37/f3c [1414975,95057] 0 2024-03-23T11:47:33.553 INFO:tasks.workunit.client.0.smithi063.stdout:5/261: mkdir d2/d7/d8/de/d2a/d3b/d43/d5c 0 2024-03-23T11:47:33.554 INFO:tasks.workunit.client.1.smithi111.stdout:9/458: symlink d0/d4/l94 0 2024-03-23T11:47:33.555 INFO:tasks.workunit.client.0.smithi063.stdout:6/295: write d1/dc/d3a/f45 [858483,111559] 0 2024-03-23T11:47:33.559 INFO:tasks.workunit.client.0.smithi063.stdout:1/283: dwrite d3/f2a [0,4194304] 0 2024-03-23T11:47:33.559 INFO:tasks.workunit.client.1.smithi111.stdout:2/486: creat d2/d7/fb4 x:0 0 0 2024-03-23T11:47:33.559 INFO:tasks.workunit.client.0.smithi063.stdout:8/619: dread d6/d1c/f52 [0,4194304] 0 2024-03-23T11:47:33.560 INFO:tasks.workunit.client.0.smithi063.stdout:9/354: write d4/d4c/f69 [646101,106147] 0 2024-03-23T11:47:33.562 INFO:tasks.workunit.client.1.smithi111.stdout:7/388: dwrite d14/f25 [0,4194304] 0 2024-03-23T11:47:33.568 INFO:tasks.workunit.client.1.smithi111.stdout:0/400: mknod d5/db/d1f/d47/c7f 0 2024-03-23T11:47:33.568 INFO:tasks.workunit.client.1.smithi111.stdout:1/411: creat d0/d5/dd/d1e/d54/d87/f8a x:0 0 0 2024-03-23T11:47:33.573 INFO:tasks.workunit.client.1.smithi111.stdout:9/459: truncate d0/d4/fe 885081 0 2024-03-23T11:47:33.574 INFO:tasks.workunit.client.1.smithi111.stdout:4/416: getdents d0/dc/d4c/d4b/d4d 0 2024-03-23T11:47:33.574 INFO:tasks.workunit.client.0.smithi063.stdout:5/262: readlink d2/d7/d8/l11 0 2024-03-23T11:47:33.575 INFO:tasks.workunit.client.0.smithi063.stdout:4/368: creat d5/da/d20/d4b/d16/d50/f69 x:0 0 0 2024-03-23T11:47:33.576 INFO:tasks.workunit.client.0.smithi063.stdout:1/284: chown d3/d1c/c29 20 1 2024-03-23T11:47:33.581 INFO:tasks.workunit.client.1.smithi111.stdout:2/487: creat d2/de/d12/d70/fb5 x:0 0 0 2024-03-23T11:47:33.582 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:33 smithi063 ceph-mon[35189]: pgmap v36: 65 pgs: 65 active+clean; 1.4 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 30 MiB/s rd, 197 MiB/s wr, 376 op/s 2024-03-23T11:47:33.583 INFO:tasks.workunit.client.1.smithi111.stdout:6/454: creat d2/d1f/d45/d49/d8d/f94 x:0 0 0 2024-03-23T11:47:33.583 INFO:tasks.workunit.client.0.smithi063.stdout:0/340: symlink d2/d4/d63/l7b 0 2024-03-23T11:47:33.586 INFO:tasks.workunit.client.0.smithi063.stdout:8/620: symlink d6/d1f/ld4 0 2024-03-23T11:47:33.587 INFO:tasks.workunit.client.0.smithi063.stdout:2/299: write d2/d23/d2e/f36 [2068043,104565] 0 2024-03-23T11:47:33.587 INFO:tasks.workunit.client.0.smithi063.stdout:6/296: dread d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:33.588 INFO:tasks.workunit.client.0.smithi063.stdout:3/476: dwrite d13/d22/f2c [0,4194304] 0 2024-03-23T11:47:33.590 INFO:tasks.workunit.client.1.smithi111.stdout:8/356: dread d5/d26/d31/d4d/f6c [0,4194304] 0 2024-03-23T11:47:33.594 INFO:tasks.workunit.client.1.smithi111.stdout:5/469: rename d1/d3/d6/l1c to d1/d3/d6/d8b/la0 0 2024-03-23T11:47:33.596 INFO:tasks.workunit.client.1.smithi111.stdout:4/417: mkdir d0/dc/d78 0 2024-03-23T11:47:33.597 INFO:tasks.workunit.client.0.smithi063.stdout:0/341: dread - d2/d4/dd/d28/d46/d51/f65 zero size 2024-03-23T11:47:33.598 INFO:tasks.workunit.client.1.smithi111.stdout:3/360: dwrite d0/d7/f47 [0,4194304] 0 2024-03-23T11:47:33.599 INFO:tasks.workunit.client.1.smithi111.stdout:6/455: symlink d2/l95 0 2024-03-23T11:47:33.599 INFO:tasks.workunit.client.0.smithi063.stdout:9/355: creat d4/f7b x:0 0 0 2024-03-23T11:47:33.599 INFO:tasks.workunit.client.0.smithi063.stdout:2/300: write d2/d23/d2e/f5e [905715,4397] 0 2024-03-23T11:47:33.601 INFO:tasks.workunit.client.0.smithi063.stdout:1/285: mknod d3/df/c63 0 2024-03-23T11:47:33.603 INFO:tasks.workunit.client.1.smithi111.stdout:9/460: rename d0/d4/d10/d29/d5b to d0/d4/d1c/d2d/d3b/d42/d4f/d95 0 2024-03-23T11:47:33.604 INFO:tasks.workunit.client.0.smithi063.stdout:8/621: rmdir d6/d1f/d54/d63/d90/d9e 39 2024-03-23T11:47:33.608 INFO:tasks.workunit.client.0.smithi063.stdout:5/263: dread d2/d28/f2d [0,4194304] 0 2024-03-23T11:47:33.608 INFO:tasks.workunit.client.1.smithi111.stdout:5/470: symlink d1/d3/d3a/la1 0 2024-03-23T11:47:33.609 INFO:tasks.workunit.client.1.smithi111.stdout:2/488: dwrite d2/d32/f7d [0,4194304] 0 2024-03-23T11:47:33.614 INFO:tasks.workunit.client.0.smithi063.stdout:4/369: dread d5/da/d20/d4b/d16/d37/f53 [0,4194304] 0 2024-03-23T11:47:33.614 INFO:tasks.workunit.client.0.smithi063.stdout:0/342: unlink d2/d4/f7a 0 2024-03-23T11:47:33.614 INFO:tasks.workunit.client.1.smithi111.stdout:7/389: dwrite fe [4194304,4194304] 0 2024-03-23T11:47:33.615 INFO:tasks.workunit.client.1.smithi111.stdout:9/461: symlink d0/d31/d41/l96 0 2024-03-23T11:47:33.616 INFO:tasks.workunit.client.0.smithi063.stdout:1/286: creat d3/df/d2f/f64 x:0 0 0 2024-03-23T11:47:33.616 INFO:tasks.workunit.client.0.smithi063.stdout:0/343: chown d2/d14/c1d 12 1 2024-03-23T11:47:33.618 INFO:tasks.workunit.client.0.smithi063.stdout:4/370: chown d5/l44 146353 1 2024-03-23T11:47:33.618 INFO:tasks.workunit.client.1.smithi111.stdout:5/471: unlink d1/f92 0 2024-03-23T11:47:33.621 INFO:tasks.workunit.client.1.smithi111.stdout:6/456: dwrite d2/d1f/d45/d49/f5d [0,4194304] 0 2024-03-23T11:47:33.622 INFO:tasks.workunit.client.1.smithi111.stdout:2/489: creat d2/de/d12/d70/fb6 x:0 0 0 2024-03-23T11:47:33.623 INFO:tasks.workunit.client.0.smithi063.stdout:7/317: getdents d5/d6/d19/d1f/d37/da 0 2024-03-23T11:47:33.624 INFO:tasks.workunit.client.1.smithi111.stdout:7/390: rename d14/d1b/d1e/d2c/c67 to d14/d1b/d22/c6d 0 2024-03-23T11:47:33.625 INFO:tasks.workunit.client.1.smithi111.stdout:0/401: dread d5/db/d1f/d32/f3e [0,4194304] 0 2024-03-23T11:47:33.626 INFO:tasks.workunit.client.1.smithi111.stdout:9/462: rename d0/d4/d6a to d0/d4/d6a/d89/d97 22 2024-03-23T11:47:33.629 INFO:tasks.workunit.client.0.smithi063.stdout:3/477: link d13/f56 d13/d22/d2d/d7d/f94 0 2024-03-23T11:47:33.631 INFO:tasks.workunit.client.1.smithi111.stdout:9/463: dread - d0/d4/d10/d29/f67 zero size 2024-03-23T11:47:33.631 INFO:tasks.workunit.client.1.smithi111.stdout:0/402: chown d5/db/d1f/d47/f5a 69368774 1 2024-03-23T11:47:33.632 INFO:tasks.workunit.client.0.smithi063.stdout:7/318: truncate d5/d6/d19/d1f/d47/d4f/d5b/f66 581257 0 2024-03-23T11:47:33.635 INFO:tasks.workunit.client.0.smithi063.stdout:9/356: dwrite d4/d4c/f69 [0,4194304] 0 2024-03-23T11:47:33.637 INFO:tasks.workunit.client.0.smithi063.stdout:6/297: rename d1/d13/c19 to d1/d15/d4b/c58 0 2024-03-23T11:47:33.641 INFO:tasks.workunit.client.0.smithi063.stdout:2/301: chown d2/d46/l4e 10237731 1 2024-03-23T11:47:33.644 INFO:tasks.workunit.client.0.smithi063.stdout:0/344: creat d2/d14/d36/d77/f7c x:0 0 0 2024-03-23T11:47:33.645 INFO:tasks.workunit.client.1.smithi111.stdout:6/457: rename d2/d17/f5a to d2/d80/f96 0 2024-03-23T11:47:33.647 INFO:tasks.workunit.client.1.smithi111.stdout:0/403: rmdir d5/db/d1f/d47/d59/d5b 39 2024-03-23T11:47:33.647 INFO:tasks.workunit.client.0.smithi063.stdout:4/371: symlink d5/da/d20/d4b/d16/d32/l6a 0 2024-03-23T11:47:33.647 INFO:tasks.workunit.client.0.smithi063.stdout:3/478: dread - d13/d22/f2f zero size 2024-03-23T11:47:33.653 INFO:tasks.workunit.client.0.smithi063.stdout:1/287: rename d3/d35/d54 to d3/d49/d56/d65 0 2024-03-23T11:47:33.655 INFO:tasks.workunit.client.0.smithi063.stdout:6/298: unlink d1/dc/c11 0 2024-03-23T11:47:33.655 INFO:tasks.workunit.client.0.smithi063.stdout:2/302: mknod d2/d23/d2e/d52/c62 0 2024-03-23T11:47:33.657 INFO:tasks.workunit.client.1.smithi111.stdout:6/458: dread d2/d17/f33 [0,4194304] 0 2024-03-23T11:47:33.658 INFO:tasks.workunit.client.1.smithi111.stdout:6/459: chown d2/d1a/d1c 1127172845 1 2024-03-23T11:47:33.658 INFO:tasks.workunit.client.0.smithi063.stdout:4/372: write d5/da/d20/d4b/d16/f5c [569989,4715] 0 2024-03-23T11:47:33.659 INFO:tasks.workunit.client.1.smithi111.stdout:6/460: dread - d2/d1a/d1c/d2c/d51/f8f zero size 2024-03-23T11:47:33.661 INFO:tasks.workunit.client.0.smithi063.stdout:3/479: creat d13/d22/d2d/d3f/d58/d6e/d8d/f95 x:0 0 0 2024-03-23T11:47:33.661 INFO:tasks.workunit.client.0.smithi063.stdout:9/357: creat d4/f7c x:0 0 0 2024-03-23T11:47:33.662 INFO:tasks.workunit.client.0.smithi063.stdout:2/303: write d2/d23/d2e/f35 [186316,111106] 0 2024-03-23T11:47:33.663 INFO:tasks.workunit.client.0.smithi063.stdout:4/373: stat d5/d1e 0 2024-03-23T11:47:33.664 INFO:tasks.workunit.client.0.smithi063.stdout:5/264: getdents d2/d7/d8/de/d2a/d3b 0 2024-03-23T11:47:33.664 INFO:tasks.workunit.client.1.smithi111.stdout:2/490: truncate d2/de/d12/f29 252658 0 2024-03-23T11:47:33.666 INFO:tasks.workunit.client.1.smithi111.stdout:2/491: dread - d2/d11/d17/d3e/d6b/d2e/f82 zero size 2024-03-23T11:47:33.668 INFO:tasks.workunit.client.1.smithi111.stdout:0/404: creat d5/db/d1f/d48/f80 x:0 0 0 2024-03-23T11:47:33.671 INFO:tasks.workunit.client.0.smithi063.stdout:1/288: write d3/df/d2f/f64 [585877,117847] 0 2024-03-23T11:47:33.674 INFO:tasks.workunit.client.0.smithi063.stdout:7/319: dwrite d5/d6/d19/d1f/d37/d20/f31 [0,4194304] 0 2024-03-23T11:47:33.676 INFO:tasks.workunit.client.1.smithi111.stdout:9/464: getdents d0/d4/d10/d29/d7f/d8e 0 2024-03-23T11:47:33.680 INFO:tasks.workunit.client.1.smithi111.stdout:2/492: dwrite d2/d11/d17/d3e/d6b/f79 [0,4194304] 0 2024-03-23T11:47:33.681 INFO:tasks.workunit.client.0.smithi063.stdout:2/304: rename d2/f45 to d2/d1c/d5c/f63 0 2024-03-23T11:47:33.683 INFO:tasks.workunit.client.1.smithi111.stdout:0/405: symlink d5/db/d1f/l81 0 2024-03-23T11:47:33.685 INFO:tasks.workunit.client.1.smithi111.stdout:2/493: write d2/d11/d17/d3e/d6b/d2e/d3f/f4d [3133570,103823] 0 2024-03-23T11:47:33.686 INFO:tasks.workunit.client.1.smithi111.stdout:7/391: getdents d14 0 2024-03-23T11:47:33.688 INFO:tasks.workunit.client.0.smithi063.stdout:2/305: dread - d2/de/d22/f4a zero size 2024-03-23T11:47:33.688 INFO:tasks.workunit.client.0.smithi063.stdout:0/345: dwrite d2/d14/f31 [4194304,4194304] 0 2024-03-23T11:47:33.688 INFO:tasks.workunit.client.0.smithi063.stdout:4/374: symlink d5/da/d20/d4b/d29/d38/l6b 0 2024-03-23T11:47:33.690 INFO:tasks.workunit.client.1.smithi111.stdout:6/461: unlink d2/d1f/d45/d49/f5d 0 2024-03-23T11:47:33.691 INFO:tasks.workunit.client.1.smithi111.stdout:7/392: stat d14/d1b/d1e/d2c/d2e/d33/l64 0 2024-03-23T11:47:33.693 INFO:tasks.workunit.client.1.smithi111.stdout:9/465: symlink d0/d4/d1c/d2d/d3b/d42/d4f/d95/l98 0 2024-03-23T11:47:33.698 INFO:tasks.workunit.client.0.smithi063.stdout:7/320: write d5/d22/f24 [134793,129959] 0 2024-03-23T11:47:33.706 INFO:tasks.workunit.client.0.smithi063.stdout:4/375: mknod d5/da/d20/d4b/d1f/c6c 0 2024-03-23T11:47:33.706 INFO:tasks.workunit.client.0.smithi063.stdout:2/306: dread d2/d23/d2e/f4b [0,4194304] 0 2024-03-23T11:47:33.708 INFO:tasks.workunit.client.1.smithi111.stdout:0/406: truncate d5/db/f38 515512 0 2024-03-23T11:47:33.710 INFO:tasks.workunit.client.1.smithi111.stdout:5/472: getdents d1/d3/d6/d7/d63/d2c/d82 0 2024-03-23T11:47:33.711 INFO:tasks.workunit.client.1.smithi111.stdout:9/466: dwrite d0/d4/f47 [0,4194304] 0 2024-03-23T11:47:33.716 INFO:tasks.workunit.client.1.smithi111.stdout:2/494: rename d2/d7/da/d19/f6a to d2/d11/fb7 0 2024-03-23T11:47:33.717 INFO:tasks.workunit.client.1.smithi111.stdout:7/393: dread d14/f36 [0,4194304] 0 2024-03-23T11:47:33.718 INFO:tasks.workunit.client.1.smithi111.stdout:1/412: sync 2024-03-23T11:47:33.721 INFO:tasks.workunit.client.0.smithi063.stdout:0/346: mknod d2/d4/dd/d28/c7d 0 2024-03-23T11:47:33.725 INFO:tasks.workunit.client.1.smithi111.stdout:1/413: write d0/d5/dd/d1e/d54/f60 [3583482,89644] 0 2024-03-23T11:47:33.726 INFO:tasks.workunit.client.0.smithi063.stdout:8/622: truncate d6/d1c/f6c 426963 0 2024-03-23T11:47:33.727 INFO:tasks.workunit.client.1.smithi111.stdout:7/394: symlink d14/d1b/d20/d5f/l6e 0 2024-03-23T11:47:33.728 INFO:tasks.workunit.client.0.smithi063.stdout:4/376: mknod d5/d1e/d2b/c6d 0 2024-03-23T11:47:33.731 INFO:tasks.workunit.client.1.smithi111.stdout:7/395: write d14/d1b/d20/f57 [634202,34446] 0 2024-03-23T11:47:33.731 INFO:tasks.workunit.client.1.smithi111.stdout:0/407: mkdir d5/db/d1f/d82 0 2024-03-23T11:47:33.733 INFO:tasks.workunit.client.0.smithi063.stdout:6/299: getdents d1/dc 0 2024-03-23T11:47:33.733 INFO:tasks.workunit.client.0.smithi063.stdout:4/377: fsync d5/da/d20/f3c 0 2024-03-23T11:47:33.735 INFO:tasks.workunit.client.0.smithi063.stdout:2/307: creat d2/d23/d2e/f64 x:0 0 0 2024-03-23T11:47:33.739 INFO:tasks.workunit.client.1.smithi111.stdout:2/495: dwrite d2/d11/d17/d3e/d6b/d2e/d3f/f9e [0,4194304] 0 2024-03-23T11:47:33.739 INFO:tasks.workunit.client.1.smithi111.stdout:1/414: mkdir d0/d2b/d7c/d8b 0 2024-03-23T11:47:33.740 INFO:tasks.workunit.client.0.smithi063.stdout:1/289: getdents d3/df/d2f 0 2024-03-23T11:47:33.741 INFO:tasks.workunit.client.1.smithi111.stdout:2/496: dread - d2/d11/d17/d3e/d5d/fa4 zero size 2024-03-23T11:47:33.742 INFO:tasks.workunit.client.1.smithi111.stdout:2/497: read - d2/f74 zero size 2024-03-23T11:47:33.743 INFO:tasks.workunit.client.0.smithi063.stdout:3/480: rename d13/d22/d2d/d3f/d67/c74 to d13/d22/d2d/d3f/c96 0 2024-03-23T11:47:33.744 INFO:tasks.workunit.client.1.smithi111.stdout:7/396: unlink d14/d1b/d1e/d2c/d2e/d33/l35 0 2024-03-23T11:47:33.746 INFO:tasks.workunit.client.0.smithi063.stdout:0/347: dread d2/d14/f23 [0,4194304] 0 2024-03-23T11:47:33.753 INFO:tasks.workunit.client.1.smithi111.stdout:6/462: rename d2/d1f/d45/c69 to d2/d1a/d1c/c97 0 2024-03-23T11:47:33.753 INFO:tasks.workunit.client.1.smithi111.stdout:7/397: dread fe [4194304,4194304] 0 2024-03-23T11:47:33.754 INFO:tasks.workunit.client.0.smithi063.stdout:8/623: dwrite d6/d1c/d32/f57 [0,4194304] 0 2024-03-23T11:47:33.755 INFO:tasks.workunit.client.1.smithi111.stdout:7/398: read - d14/d1b/d1e/d2c/d42/d5e/f6c zero size 2024-03-23T11:47:33.758 INFO:tasks.workunit.client.1.smithi111.stdout:7/399: write d14/d1b/d20/f57 [3587302,121785] 0 2024-03-23T11:47:33.760 INFO:tasks.workunit.client.1.smithi111.stdout:7/400: read - d14/d37/d50/f62 zero size 2024-03-23T11:47:33.763 INFO:tasks.workunit.client.1.smithi111.stdout:7/401: write d14/d1b/d1e/d2c/d2e/f34 [8721827,25714] 0 2024-03-23T11:47:33.764 INFO:tasks.workunit.client.1.smithi111.stdout:6/463: mkdir d2/d35/d70/d98 0 2024-03-23T11:47:33.765 INFO:tasks.workunit.client.0.smithi063.stdout:9/358: truncate d4/d4c/f40 2101486 0 2024-03-23T11:47:33.766 INFO:tasks.workunit.client.0.smithi063.stdout:8/624: write d6/d1c/d32/d37/fc0 [838826,115595] 0 2024-03-23T11:47:33.767 INFO:tasks.workunit.client.1.smithi111.stdout:9/467: dread d0/d4/d1c/f45 [0,4194304] 0 2024-03-23T11:47:33.769 INFO:tasks.workunit.client.0.smithi063.stdout:9/359: read d4/f47 [2864392,17448] 0 2024-03-23T11:47:33.769 INFO:tasks.workunit.client.0.smithi063.stdout:6/300: chown d1/f12 55188951 1 2024-03-23T11:47:33.769 INFO:tasks.workunit.client.1.smithi111.stdout:9/468: readlink d0/d4/d1c/d36/d28/l58 0 2024-03-23T11:47:33.771 INFO:tasks.workunit.client.0.smithi063.stdout:2/308: read d2/f21 [342625,91315] 0 2024-03-23T11:47:33.772 INFO:tasks.workunit.client.0.smithi063.stdout:1/290: creat d3/d1c/f66 x:0 0 0 2024-03-23T11:47:33.777 INFO:tasks.workunit.client.1.smithi111.stdout:5/473: rename d1/d3/d6/c98 to d1/d3/d6/d7/d63/d2c/d43/d95/ca2 0 2024-03-23T11:47:33.778 INFO:tasks.workunit.client.1.smithi111.stdout:7/402: fsync d14/d1b/d1e/d2c/d2e/d33/f3d 0 2024-03-23T11:47:33.779 INFO:tasks.workunit.client.1.smithi111.stdout:2/498: link d2/d11/d17/d3e/c83 d2/d11/d17/d3e/d6b/d2e/d3f/cb8 0 2024-03-23T11:47:33.780 INFO:tasks.workunit.client.0.smithi063.stdout:9/360: write d4/d4c/d19/f1b [113823,41846] 0 2024-03-23T11:47:33.780 INFO:tasks.workunit.client.0.smithi063.stdout:4/378: rename d5/da/d20/d4b/d16/d32/c36 to d5/da/d20/d4b/d16/d37/c6e 0 2024-03-23T11:47:33.781 INFO:tasks.workunit.client.1.smithi111.stdout:6/464: dwrite d2/f31 [4194304,4194304] 0 2024-03-23T11:47:33.782 INFO:tasks.workunit.client.1.smithi111.stdout:3/361: sync 2024-03-23T11:47:33.782 INFO:tasks.workunit.client.1.smithi111.stdout:4/418: sync 2024-03-23T11:47:33.782 INFO:tasks.workunit.client.1.smithi111.stdout:8/357: sync 2024-03-23T11:47:33.784 INFO:tasks.workunit.client.1.smithi111.stdout:0/408: rename d5/db/d1f/d47/d59/l6e to d5/db/d1f/d19/l83 0 2024-03-23T11:47:33.785 INFO:tasks.workunit.client.0.smithi063.stdout:3/481: mkdir d13/d22/d2d/d3f/d58/d97 0 2024-03-23T11:47:33.786 INFO:tasks.workunit.client.1.smithi111.stdout:5/474: unlink d1/d3/f65 0 2024-03-23T11:47:33.786 INFO:tasks.workunit.client.1.smithi111.stdout:8/358: chown d5/d26/d2f 19 1 2024-03-23T11:47:33.786 INFO:tasks.workunit.client.0.smithi063.stdout:9/361: readlink d4/d31/l4f 0 2024-03-23T11:47:33.787 INFO:tasks.workunit.client.0.smithi063.stdout:7/321: getdents d5/d6 0 2024-03-23T11:47:33.787 INFO:tasks.workunit.client.1.smithi111.stdout:6/465: write d2/d1a/d1c/d2c/f48 [919118,125869] 0 2024-03-23T11:47:33.788 INFO:tasks.workunit.client.0.smithi063.stdout:4/379: readlink d5/da/d20/d4b/d29/l30 0 2024-03-23T11:47:33.789 INFO:tasks.workunit.client.0.smithi063.stdout:6/301: symlink d1/d2a/d41/l59 0 2024-03-23T11:47:33.789 INFO:tasks.workunit.client.0.smithi063.stdout:3/482: chown f12 430 1 2024-03-23T11:47:33.793 INFO:tasks.workunit.client.1.smithi111.stdout:9/469: dwrite d0/d4/d10/d29/f67 [0,4194304] 0 2024-03-23T11:47:33.803 INFO:tasks.workunit.client.1.smithi111.stdout:4/419: getdents d0/dc/d4c/d4b/d4d 0 2024-03-23T11:47:33.803 INFO:tasks.workunit.client.0.smithi063.stdout:1/291: dwrite d3/df/f36 [0,4194304] 0 2024-03-23T11:47:33.804 INFO:tasks.workunit.client.0.smithi063.stdout:6/302: write d1/d2a/d41/d42/f48 [126864,27677] 0 2024-03-23T11:47:33.806 INFO:tasks.workunit.client.1.smithi111.stdout:8/359: creat d5/d26/f74 x:0 0 0 2024-03-23T11:47:33.806 INFO:tasks.workunit.client.1.smithi111.stdout:2/499: dwrite d2/de/d12/d70/fb6 [0,4194304] 0 2024-03-23T11:47:33.810 INFO:tasks.workunit.client.0.smithi063.stdout:1/292: read - d3/d1c/f66 zero size 2024-03-23T11:47:33.812 INFO:tasks.workunit.client.1.smithi111.stdout:4/420: readlink d0/dc/d30/l12 0 2024-03-23T11:47:33.815 INFO:tasks.workunit.client.1.smithi111.stdout:8/360: dread - d5/d26/d31/d3b/f6e zero size 2024-03-23T11:47:33.815 INFO:tasks.workunit.client.0.smithi063.stdout:6/303: write d1/dc/d3a/f45 [1438030,61510] 0 2024-03-23T11:47:33.815 INFO:tasks.workunit.client.1.smithi111.stdout:0/409: dread d5/db/d1f/f22 [0,4194304] 0 2024-03-23T11:47:33.815 INFO:tasks.workunit.client.1.smithi111.stdout:7/403: dread d14/d1b/d22/f29 [0,4194304] 0 2024-03-23T11:47:33.818 INFO:tasks.workunit.client.0.smithi063.stdout:4/380: dwrite d5/da/f19 [0,4194304] 0 2024-03-23T11:47:33.818 INFO:tasks.workunit.client.1.smithi111.stdout:3/362: dwrite d0/d1a/d1d/f36 [0,4194304] 0 2024-03-23T11:47:33.818 INFO:tasks.workunit.client.1.smithi111.stdout:2/500: chown d2/de/d6c/f65 94765957 1 2024-03-23T11:47:33.821 INFO:tasks.workunit.client.1.smithi111.stdout:0/410: write d5/f13 [94583,35849] 0 2024-03-23T11:47:33.821 INFO:tasks.workunit.client.0.smithi063.stdout:7/322: dwrite d5/f12 [0,4194304] 0 2024-03-23T11:47:33.821 INFO:tasks.workunit.client.0.smithi063.stdout:6/304: readlink d1/d2a/d41/l59 0 2024-03-23T11:47:33.822 INFO:tasks.workunit.client.1.smithi111.stdout:7/404: readlink d14/d1b/d1e/d2c/d2e/d33/l48 0 2024-03-23T11:47:33.828 INFO:tasks.workunit.client.1.smithi111.stdout:7/405: chown d14/d1b/l24 37 1 2024-03-23T11:47:33.840 INFO:tasks.workunit.client.0.smithi063.stdout:4/381: dread d5/d1e/d2b/f58 [0,4194304] 0 2024-03-23T11:47:33.842 INFO:tasks.workunit.client.1.smithi111.stdout:0/411: creat d5/db/d1f/d19/d45/d4b/f84 x:0 0 0 2024-03-23T11:47:33.844 INFO:tasks.workunit.client.1.smithi111.stdout:0/412: readlink d5/db/l33 0 2024-03-23T11:47:33.846 INFO:tasks.workunit.client.0.smithi063.stdout:9/362: unlink d4/d31/d43/d49/f66 0 2024-03-23T11:47:33.847 INFO:tasks.workunit.client.1.smithi111.stdout:2/501: dwrite d2/d11/d17/f21 [0,4194304] 0 2024-03-23T11:47:33.849 INFO:tasks.workunit.client.1.smithi111.stdout:4/421: rename d0/dc/d4c/d4b/d4f/l5f to d0/dc/d4c/l79 0 2024-03-23T11:47:33.851 INFO:tasks.workunit.client.0.smithi063.stdout:3/483: mknod d13/d15/d48/c98 0 2024-03-23T11:47:33.852 INFO:tasks.workunit.client.0.smithi063.stdout:9/363: dread - d4/f7b zero size 2024-03-23T11:47:33.853 INFO:tasks.workunit.client.1.smithi111.stdout:4/422: readlink d0/dc/d4c/d4b/d4d/l6e 0 2024-03-23T11:47:33.855 INFO:tasks.workunit.client.1.smithi111.stdout:7/406: dwrite f1 [4194304,4194304] 0 2024-03-23T11:47:33.855 INFO:tasks.workunit.client.1.smithi111.stdout:4/423: chown d0/c2 7841 1 2024-03-23T11:47:33.858 INFO:tasks.workunit.client.1.smithi111.stdout:2/502: creat d2/d32/d91/fb9 x:0 0 0 2024-03-23T11:47:33.860 INFO:tasks.workunit.client.0.smithi063.stdout:1/293: creat d3/d49/d56/f67 x:0 0 0 2024-03-23T11:47:33.861 INFO:tasks.workunit.client.1.smithi111.stdout:8/361: rename d5/d26/d31/d5e/d46 to d5/d26/d2f/d62/d75 0 2024-03-23T11:47:33.862 INFO:tasks.workunit.client.1.smithi111.stdout:6/466: dread d2/d1a/f72 [0,4194304] 0 2024-03-23T11:47:33.864 INFO:tasks.workunit.client.1.smithi111.stdout:1/415: dread d0/d6/f34 [0,4194304] 0 2024-03-23T11:47:33.864 INFO:tasks.workunit.client.1.smithi111.stdout:3/363: link d0/d1a/d2b/d38/f51 d0/d1a/d4f/f70 0 2024-03-23T11:47:33.867 INFO:tasks.workunit.client.1.smithi111.stdout:7/407: mknod d14/d44/c6f 0 2024-03-23T11:47:33.869 INFO:tasks.workunit.client.1.smithi111.stdout:0/413: truncate d5/f30 1027698 0 2024-03-23T11:47:33.871 INFO:tasks.workunit.client.0.smithi063.stdout:4/382: readlink d5/da/l4d 0 2024-03-23T11:47:33.874 INFO:tasks.workunit.client.1.smithi111.stdout:3/364: creat d0/d1a/d2b/f71 x:0 0 0 2024-03-23T11:47:33.875 INFO:tasks.workunit.client.1.smithi111.stdout:4/424: mkdir d0/dc/d7a 0 2024-03-23T11:47:33.876 INFO:tasks.workunit.client.1.smithi111.stdout:7/408: rmdir d14/d37/d54 39 2024-03-23T11:47:33.877 INFO:tasks.workunit.client.0.smithi063.stdout:8/625: creat d6/d1c/fd5 x:0 0 0 2024-03-23T11:47:33.879 INFO:tasks.workunit.client.0.smithi063.stdout:1/294: creat d3/d1c/f68 x:0 0 0 2024-03-23T11:47:33.881 INFO:tasks.workunit.client.1.smithi111.stdout:4/425: fdatasync d0/d7/da/f13 0 2024-03-23T11:47:33.881 INFO:tasks.workunit.client.1.smithi111.stdout:8/362: mkdir d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/d76 0 2024-03-23T11:47:33.883 INFO:tasks.workunit.client.0.smithi063.stdout:9/364: chown d4/c3e 41454 1 2024-03-23T11:47:33.883 INFO:tasks.workunit.client.1.smithi111.stdout:0/414: chown d5/db/d1f/d47/d59/d5b/f7c 2510686 1 2024-03-23T11:47:33.884 INFO:tasks.workunit.client.1.smithi111.stdout:2/503: dwrite d2/de/d12/d70/fb5 [0,4194304] 0 2024-03-23T11:47:33.887 INFO:tasks.workunit.client.1.smithi111.stdout:9/470: dwrite d0/f5f [0,4194304] 0 2024-03-23T11:47:33.887 INFO:tasks.workunit.client.1.smithi111.stdout:6/467: link d2/d17/f4a d2/d80/f99 0 2024-03-23T11:47:33.888 INFO:tasks.workunit.client.0.smithi063.stdout:4/383: mkdir d5/da/d20/d4b/d16/d32/d6f 0 2024-03-23T11:47:33.889 INFO:tasks.workunit.client.1.smithi111.stdout:7/409: mkdir d14/d44/d70 0 2024-03-23T11:47:33.890 INFO:tasks.workunit.client.0.smithi063.stdout:7/323: dread d5/d6/d19/d1f/d37/d20/f28 [0,4194304] 0 2024-03-23T11:47:33.890 INFO:tasks.workunit.client.1.smithi111.stdout:1/416: dwrite d0/d5/dd/d1e/d54/d87/f8a [0,4194304] 0 2024-03-23T11:47:33.891 INFO:tasks.workunit.client.0.smithi063.stdout:4/384: read d5/da/d20/d4b/d1f/f39 [159484,70379] 0 2024-03-23T11:47:33.891 INFO:tasks.workunit.client.1.smithi111.stdout:5/475: creat d1/d3/d6/d8b/fa3 x:0 0 0 2024-03-23T11:47:33.896 INFO:tasks.workunit.client.1.smithi111.stdout:0/415: truncate d5/db/d1f/d19/f4c 1033906 0 2024-03-23T11:47:33.897 INFO:tasks.workunit.client.0.smithi063.stdout:8/626: unlink d6/d1f/l3d 0 2024-03-23T11:47:33.898 INFO:tasks.workunit.client.1.smithi111.stdout:8/363: creat d5/d26/d2f/f77 x:0 0 0 2024-03-23T11:47:33.899 INFO:tasks.workunit.client.1.smithi111.stdout:0/416: stat d5/db/l3a 0 2024-03-23T11:47:33.900 INFO:tasks.workunit.client.1.smithi111.stdout:3/365: mkdir d0/d72 0 2024-03-23T11:47:33.903 INFO:tasks.workunit.client.0.smithi063.stdout:4/385: dread d5/da/d20/d4b/d1f/f26 [0,4194304] 0 2024-03-23T11:47:33.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:33 smithi111 ceph-mon[44005]: pgmap v36: 65 pgs: 65 active+clean; 1.4 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 30 MiB/s rd, 197 MiB/s wr, 376 op/s 2024-03-23T11:47:33.904 INFO:tasks.workunit.client.1.smithi111.stdout:6/468: mkdir d2/d1f/d45/d9a 0 2024-03-23T11:47:33.905 INFO:tasks.workunit.client.1.smithi111.stdout:3/366: fdatasync d0/d1a/d4f/f4e 0 2024-03-23T11:47:33.905 INFO:tasks.workunit.client.1.smithi111.stdout:1/417: read d0/d2b/f45 [3820832,16725] 0 2024-03-23T11:47:33.905 INFO:tasks.workunit.client.0.smithi063.stdout:3/484: creat d13/d15/f99 x:0 0 0 2024-03-23T11:47:33.906 INFO:tasks.workunit.client.1.smithi111.stdout:8/364: dread d5/f11 [0,4194304] 0 2024-03-23T11:47:33.910 INFO:tasks.workunit.client.0.smithi063.stdout:7/324: rmdir d5/d6/d19/d1f/d37/da 39 2024-03-23T11:47:33.915 INFO:tasks.workunit.client.0.smithi063.stdout:0/348: rename d2/d4/dd/d1f/f20 to d2/d12/f7e 0 2024-03-23T11:47:33.916 INFO:tasks.workunit.client.1.smithi111.stdout:5/476: dread d1/d3/d6/d8b/d59/d74/f21 [4194304,4194304] 0 2024-03-23T11:47:33.917 INFO:tasks.workunit.client.0.smithi063.stdout:4/386: creat d5/da/d20/d4b/d16/d50/f70 x:0 0 0 2024-03-23T11:47:33.920 INFO:tasks.workunit.client.1.smithi111.stdout:1/418: creat d0/d2b/d69/f8c x:0 0 0 2024-03-23T11:47:33.921 INFO:tasks.workunit.client.1.smithi111.stdout:4/426: link d0/c66 d0/dc/d1f/d52/c7b 0 2024-03-23T11:47:33.922 INFO:tasks.workunit.client.0.smithi063.stdout:7/325: creat d5/d6/d19/d1f/d47/f69 x:0 0 0 2024-03-23T11:47:33.924 INFO:tasks.workunit.client.1.smithi111.stdout:3/367: mknod d0/d1a/d1d/c73 0 2024-03-23T11:47:33.927 INFO:tasks.workunit.client.1.smithi111.stdout:0/417: dwrite d5/db/d1f/d32/f3b [0,4194304] 0 2024-03-23T11:47:33.931 INFO:tasks.workunit.client.1.smithi111.stdout:6/469: rename d2/d1a/d1c/d2c/f4c to d2/d80/f9b 0 2024-03-23T11:47:33.932 INFO:tasks.workunit.client.1.smithi111.stdout:8/365: rename d5/d26/d31 to d5/d26/d31/d3b/d41/d78 22 2024-03-23T11:47:33.934 INFO:tasks.workunit.client.0.smithi063.stdout:2/309: dwrite d2/f21 [0,4194304] 0 2024-03-23T11:47:33.937 INFO:tasks.workunit.client.0.smithi063.stdout:8/627: creat d6/fd6 x:0 0 0 2024-03-23T11:47:33.937 INFO:tasks.workunit.client.0.smithi063.stdout:7/326: rename d5/f8 to d5/d6/d39/f6a 0 2024-03-23T11:47:33.938 INFO:tasks.workunit.client.0.smithi063.stdout:8/628: readlink d6/d1c/d32/l68 0 2024-03-23T11:47:33.939 INFO:tasks.workunit.client.1.smithi111.stdout:7/410: rmdir d14/d44/d4b/d69 0 2024-03-23T11:47:33.943 INFO:tasks.workunit.client.1.smithi111.stdout:2/504: getdents d2/d11/d17/d1f/d30 0 2024-03-23T11:47:33.945 INFO:tasks.workunit.client.1.smithi111.stdout:2/505: readlink d2/de/l1a 0 2024-03-23T11:47:33.945 INFO:tasks.workunit.client.0.smithi063.stdout:4/387: symlink d5/da/d20/d4b/d16/d32/d56/l71 0 2024-03-23T11:47:33.946 INFO:tasks.workunit.client.1.smithi111.stdout:1/419: write d0/d6/d48/d4a/d6c/f7b [590526,11797] 0 2024-03-23T11:47:33.947 INFO:tasks.workunit.client.1.smithi111.stdout:9/471: getdents d0/d4/d1c/d36 0 2024-03-23T11:47:33.947 INFO:tasks.workunit.client.0.smithi063.stdout:7/327: mknod d5/d6/d39/c6b 0 2024-03-23T11:47:33.947 INFO:tasks.workunit.client.0.smithi063.stdout:4/388: fdatasync d5/da/d20/f3c 0 2024-03-23T11:47:33.948 INFO:tasks.workunit.client.1.smithi111.stdout:4/427: stat d0/dc/d30/f15 0 2024-03-23T11:47:33.950 INFO:tasks.workunit.client.1.smithi111.stdout:3/368: creat d0/d1a/d1d/d26/d5a/f74 x:0 0 0 2024-03-23T11:47:33.951 INFO:tasks.workunit.client.1.smithi111.stdout:9/472: truncate d0/f21 5054396 0 2024-03-23T11:47:33.952 INFO:tasks.workunit.client.1.smithi111.stdout:4/428: chown d0/dc/d1f/f4a 185 1 2024-03-23T11:47:33.955 INFO:tasks.workunit.client.1.smithi111.stdout:7/411: mknod d14/d1b/d22/c71 0 2024-03-23T11:47:33.955 INFO:tasks.workunit.client.0.smithi063.stdout:0/349: rmdir d2/d4/dd/d25/d6f 0 2024-03-23T11:47:33.956 INFO:tasks.workunit.client.1.smithi111.stdout:4/429: truncate d0/d7/da/f46 1651946 0 2024-03-23T11:47:33.956 INFO:tasks.workunit.client.1.smithi111.stdout:7/412: chown d14/d1b/d1e/d2c/d42 1379675 1 2024-03-23T11:47:33.961 INFO:tasks.workunit.client.1.smithi111.stdout:3/369: symlink d0/d53/l75 0 2024-03-23T11:47:33.962 INFO:tasks.workunit.client.1.smithi111.stdout:8/366: mkdir d5/d32/d79 0 2024-03-23T11:47:33.962 INFO:tasks.workunit.client.1.smithi111.stdout:8/367: readlink d5/l2e 0 2024-03-23T11:47:33.963 INFO:tasks.workunit.client.0.smithi063.stdout:7/328: unlink d5/c3f 0 2024-03-23T11:47:33.963 INFO:tasks.workunit.client.0.smithi063.stdout:2/310: dread d2/f18 [0,4194304] 0 2024-03-23T11:47:33.967 INFO:tasks.workunit.client.0.smithi063.stdout:8/629: creat d6/d1f/d54/d63/d90/d9e/fd7 x:0 0 0 2024-03-23T11:47:33.969 INFO:tasks.workunit.client.0.smithi063.stdout:2/311: readlink d2/de/d22/l48 0 2024-03-23T11:47:33.971 INFO:tasks.workunit.client.1.smithi111.stdout:6/470: dread d2/d1a/f1d [0,4194304] 0 2024-03-23T11:47:33.972 INFO:tasks.workunit.client.1.smithi111.stdout:6/471: fdatasync d2/d1f/f59 0 2024-03-23T11:47:33.973 INFO:tasks.workunit.client.1.smithi111.stdout:2/506: mkdir d2/d11/d17/d3e/d6b/d9d/d52/dba 0 2024-03-23T11:47:33.977 INFO:tasks.workunit.client.1.smithi111.stdout:0/418: creat d5/f85 x:0 0 0 2024-03-23T11:47:33.980 INFO:tasks.workunit.client.0.smithi063.stdout:7/329: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:33.980 INFO:tasks.workunit.client.0.smithi063.stdout:6/305: mknod d1/dc/c5a 0 2024-03-23T11:47:33.982 INFO:tasks.workunit.client.0.smithi063.stdout:4/389: getdents d5/da/d20/d4b/d16/d37 0 2024-03-23T11:47:33.983 INFO:tasks.workunit.client.1.smithi111.stdout:9/473: symlink d0/d4/d1c/d2d/d3b/d42/d92/l99 0 2024-03-23T11:47:33.984 INFO:tasks.workunit.client.1.smithi111.stdout:9/474: chown d0/d31/l38 29 1 2024-03-23T11:47:33.985 INFO:tasks.workunit.client.0.smithi063.stdout:6/306: fdatasync d1/dc/d3a/f4f 0 2024-03-23T11:47:33.986 INFO:tasks.workunit.client.0.smithi063.stdout:1/295: write d3/f38 [128865,79661] 0 2024-03-23T11:47:33.989 INFO:tasks.workunit.client.0.smithi063.stdout:1/296: stat d3/f9 0 2024-03-23T11:47:33.989 INFO:tasks.workunit.client.0.smithi063.stdout:9/365: write d4/f16 [2667634,69607] 0 2024-03-23T11:47:33.992 INFO:tasks.workunit.client.1.smithi111.stdout:6/472: creat d2/d35/f9c x:0 0 0 2024-03-23T11:47:33.994 INFO:tasks.workunit.client.0.smithi063.stdout:2/312: dread d2/d23/f28 [4194304,4194304] 0 2024-03-23T11:47:33.998 INFO:tasks.workunit.client.0.smithi063.stdout:1/297: creat d3/d3f/d4d/f69 x:0 0 0 2024-03-23T11:47:33.999 INFO:tasks.workunit.client.1.smithi111.stdout:4/430: getdents d0/dc/d4c/d4b/d4f/d72 0 2024-03-23T11:47:34.000 INFO:tasks.workunit.client.0.smithi063.stdout:1/298: read - d3/d3f/d4d/f69 zero size 2024-03-23T11:47:34.002 INFO:tasks.workunit.client.0.smithi063.stdout:1/299: chown d3/df/d31/f41 1560581 1 2024-03-23T11:47:34.004 INFO:tasks.workunit.client.1.smithi111.stdout:5/477: truncate d1/d3/d6/d7/f64 1486621 0 2024-03-23T11:47:34.008 INFO:tasks.workunit.client.0.smithi063.stdout:2/313: rmdir d2/d23/d2e/d52/d44/d4d 39 2024-03-23T11:47:34.011 INFO:tasks.workunit.client.1.smithi111.stdout:1/420: dwrite d0/d5/f23 [0,4194304] 0 2024-03-23T11:47:34.015 INFO:tasks.workunit.client.0.smithi063.stdout:9/366: rename d4/d4c/f3c to d4/d31/d43/f7d 0 2024-03-23T11:47:34.015 INFO:tasks.workunit.client.1.smithi111.stdout:7/413: write d14/d1b/d22/f4d [1237294,46227] 0 2024-03-23T11:47:34.023 INFO:tasks.workunit.client.1.smithi111.stdout:8/368: mknod d5/d26/d2f/d62/d75/d4b/c7a 0 2024-03-23T11:47:34.025 INFO:tasks.workunit.client.1.smithi111.stdout:0/419: symlink d5/db/d1f/d32/l86 0 2024-03-23T11:47:34.025 INFO:tasks.workunit.client.1.smithi111.stdout:9/475: mkdir d0/d4/d6a/d9a 0 2024-03-23T11:47:34.025 INFO:tasks.workunit.client.0.smithi063.stdout:1/300: mknod d3/c6a 0 2024-03-23T11:47:34.026 INFO:tasks.workunit.client.1.smithi111.stdout:8/369: truncate d5/d26/d31/d5e/f3e 173264 0 2024-03-23T11:47:34.026 INFO:tasks.workunit.client.1.smithi111.stdout:0/420: chown d5/db/f26 1611448 1 2024-03-23T11:47:34.027 INFO:tasks.workunit.client.1.smithi111.stdout:8/370: readlink d5/d26/d31/d5e/l5f 0 2024-03-23T11:47:34.028 INFO:tasks.workunit.client.0.smithi063.stdout:1/301: write d3/d35/d40/f4c [1257371,115049] 0 2024-03-23T11:47:34.028 INFO:tasks.workunit.client.0.smithi063.stdout:4/390: getdents d5/da 0 2024-03-23T11:47:34.032 INFO:tasks.workunit.client.0.smithi063.stdout:1/302: write d3/df/f26 [2412199,88870] 0 2024-03-23T11:47:34.035 INFO:tasks.workunit.client.1.smithi111.stdout:7/414: rmdir d14/d1b/d1e/d2c 39 2024-03-23T11:47:34.036 INFO:tasks.workunit.client.0.smithi063.stdout:2/314: rmdir d2/d23/d56 39 2024-03-23T11:47:34.038 INFO:tasks.workunit.client.1.smithi111.stdout:7/415: write d14/d1b/d1e/f47 [873531,3941] 0 2024-03-23T11:47:34.041 INFO:tasks.workunit.client.0.smithi063.stdout:9/367: dwrite d4/d4c/d19/f33 [4194304,4194304] 0 2024-03-23T11:47:34.042 INFO:tasks.workunit.client.1.smithi111.stdout:8/371: dwrite d5/d26/d31/d5e/d4a/f52 [0,4194304] 0 2024-03-23T11:47:34.043 INFO:tasks.workunit.client.1.smithi111.stdout:0/421: mkdir d5/db/d1f/d47/d87 0 2024-03-23T11:47:34.044 INFO:tasks.workunit.client.0.smithi063.stdout:4/391: mkdir d5/da/d72 0 2024-03-23T11:47:34.045 INFO:tasks.workunit.client.1.smithi111.stdout:2/507: link d2/d11/d17/d3e/d6b/l31 d2/d11/d17/d3e/d5d/d8c/lbb 0 2024-03-23T11:47:34.052 INFO:tasks.workunit.client.0.smithi063.stdout:0/350: truncate d2/d14/f64 7940224 0 2024-03-23T11:47:34.056 INFO:tasks.workunit.client.1.smithi111.stdout:3/370: getdents d0/d7 0 2024-03-23T11:47:34.058 INFO:tasks.workunit.client.1.smithi111.stdout:1/421: dread d0/d5/dd/d1e/f21 [0,4194304] 0 2024-03-23T11:47:34.058 INFO:tasks.workunit.client.0.smithi063.stdout:2/315: write d2/d1c/d5c/f63 [249574,118120] 0 2024-03-23T11:47:34.062 INFO:tasks.workunit.client.1.smithi111.stdout:5/478: creat d1/d3/d6/d7/d7b/fa4 x:0 0 0 2024-03-23T11:47:34.064 INFO:tasks.workunit.client.0.smithi063.stdout:9/368: creat d4/d4c/d44/f7e x:0 0 0 2024-03-23T11:47:34.069 INFO:tasks.workunit.client.0.smithi063.stdout:6/307: truncate d1/dc/d21/f29 18501 0 2024-03-23T11:47:34.070 INFO:tasks.workunit.client.0.smithi063.stdout:4/392: dwrite d5/da/d20/d4b/d1f/f43 [0,4194304] 0 2024-03-23T11:47:34.072 INFO:tasks.workunit.client.1.smithi111.stdout:2/508: mknod d2/d7/cbc 0 2024-03-23T11:47:34.075 INFO:tasks.workunit.client.1.smithi111.stdout:3/371: rename d0/d1a/d2b/f5c to d0/d7/f76 0 2024-03-23T11:47:34.076 INFO:tasks.workunit.client.0.smithi063.stdout:1/303: mknod d3/d62/c6b 0 2024-03-23T11:47:34.080 INFO:tasks.workunit.client.1.smithi111.stdout:4/431: creat d0/f7c x:0 0 0 2024-03-23T11:47:34.081 INFO:tasks.workunit.client.1.smithi111.stdout:5/479: dwrite d1/d3/d6/d7/f24 [0,4194304] 0 2024-03-23T11:47:34.081 INFO:tasks.workunit.client.1.smithi111.stdout:6/473: getdents d2/d1a/d1c 0 2024-03-23T11:47:34.081 INFO:tasks.workunit.client.0.smithi063.stdout:2/316: mknod d2/de/d22/c65 0 2024-03-23T11:47:34.083 INFO:tasks.workunit.client.1.smithi111.stdout:4/432: write d0/f7c [400638,4395] 0 2024-03-23T11:47:34.084 INFO:tasks.workunit.client.1.smithi111.stdout:7/416: mknod d14/d1b/d1e/d2c/c72 0 2024-03-23T11:47:34.085 INFO:tasks.workunit.client.0.smithi063.stdout:9/369: creat d4/d2c/f7f x:0 0 0 2024-03-23T11:47:34.089 INFO:tasks.workunit.client.0.smithi063.stdout:9/370: truncate d4/d4c/f5c 162060 0 2024-03-23T11:47:34.089 INFO:tasks.workunit.client.1.smithi111.stdout:0/422: rename d5/db/d1f/d48/c56 to d5/db/d1f/d19/d45/d4b/c88 0 2024-03-23T11:47:34.091 INFO:tasks.workunit.client.0.smithi063.stdout:4/393: unlink d5/da/l1a 0 2024-03-23T11:47:34.092 INFO:tasks.workunit.client.1.smithi111.stdout:1/422: creat d0/d2b/d37/d6a/f8d x:0 0 0 2024-03-23T11:47:34.093 INFO:tasks.workunit.client.0.smithi063.stdout:9/371: write d4/d4c/f5c [1089074,104126] 0 2024-03-23T11:47:34.095 INFO:tasks.workunit.client.1.smithi111.stdout:0/423: write d5/db/d1f/d48/f66 [915618,20276] 0 2024-03-23T11:47:34.095 INFO:tasks.workunit.client.1.smithi111.stdout:9/476: getdents d0/d4/d1c/d2d 0 2024-03-23T11:47:34.096 INFO:tasks.workunit.client.1.smithi111.stdout:6/474: creat d2/d1f/d45/d49/d8d/f9d x:0 0 0 2024-03-23T11:47:34.097 INFO:tasks.workunit.client.1.smithi111.stdout:8/372: rmdir d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/d76 0 2024-03-23T11:47:34.101 INFO:tasks.workunit.client.1.smithi111.stdout:7/417: mkdir d14/d1b/d1e/d2c/d2e/d73 0 2024-03-23T11:47:34.102 INFO:tasks.workunit.client.0.smithi063.stdout:1/304: unlink d3/df/f26 0 2024-03-23T11:47:34.103 INFO:tasks.workunit.client.0.smithi063.stdout:4/394: unlink d5/da/d20/d4b/d29/l30 0 2024-03-23T11:47:34.103 INFO:tasks.workunit.client.1.smithi111.stdout:4/433: dread d0/d7/da/f13 [0,4194304] 0 2024-03-23T11:47:34.103 INFO:tasks.workunit.client.1.smithi111.stdout:2/509: write d2/d7/f38 [4049525,34677] 0 2024-03-23T11:47:34.104 INFO:tasks.workunit.client.0.smithi063.stdout:2/317: rmdir d2/d23/d56 39 2024-03-23T11:47:34.108 INFO:tasks.workunit.client.1.smithi111.stdout:7/418: write d14/d1b/f5a [4476236,105424] 0 2024-03-23T11:47:34.109 INFO:tasks.workunit.client.0.smithi063.stdout:6/308: rename d1/dc/d3a/f34 to d1/dc/f5b 0 2024-03-23T11:47:34.111 INFO:tasks.workunit.client.1.smithi111.stdout:2/510: rmdir d2/d32/d98 39 2024-03-23T11:47:34.111 INFO:tasks.workunit.client.1.smithi111.stdout:6/475: rmdir d2/d1f/d45/d49 39 2024-03-23T11:47:34.113 INFO:tasks.workunit.client.0.smithi063.stdout:0/351: getdents d2/d4/d63 0 2024-03-23T11:47:34.117 INFO:tasks.workunit.client.1.smithi111.stdout:5/480: truncate d1/d20/d5c/f7e 1582230 0 2024-03-23T11:47:34.117 INFO:tasks.workunit.client.1.smithi111.stdout:7/419: unlink d14/d1b/d1e/d2c/d2e/d33/l48 0 2024-03-23T11:47:34.117 INFO:tasks.workunit.client.1.smithi111.stdout:0/424: getdents d5/db/d1f/d19/d45 0 2024-03-23T11:47:34.118 INFO:tasks.workunit.client.0.smithi063.stdout:0/352: stat d2/d14/c54 0 2024-03-23T11:47:34.118 INFO:tasks.workunit.client.1.smithi111.stdout:3/372: dread d0/d55/f56 [0,4194304] 0 2024-03-23T11:47:34.120 INFO:tasks.workunit.client.1.smithi111.stdout:1/423: getdents d0/d6/d48/d4a/d6c 0 2024-03-23T11:47:34.121 INFO:tasks.workunit.client.1.smithi111.stdout:2/511: chown d2/d11/d17/d3e/d6b/d2e/d3f/cb8 13623479 1 2024-03-23T11:47:34.122 INFO:tasks.workunit.client.0.smithi063.stdout:4/395: rmdir d5/da/d20/d4b/d29/d38 39 2024-03-23T11:47:34.122 INFO:tasks.workunit.client.1.smithi111.stdout:9/477: truncate d0/f21 2975264 0 2024-03-23T11:47:34.125 INFO:tasks.workunit.client.1.smithi111.stdout:0/425: symlink d5/db/d4d/l89 0 2024-03-23T11:47:34.125 INFO:tasks.workunit.client.0.smithi063.stdout:1/305: dread d3/f10 [0,4194304] 0 2024-03-23T11:47:34.126 INFO:tasks.workunit.client.0.smithi063.stdout:9/372: rename d4/d2c/f3f to d4/d4c/d44/d64/f80 0 2024-03-23T11:47:34.129 INFO:tasks.workunit.client.0.smithi063.stdout:9/373: chown d4/d4c/d19/l29 3321 1 2024-03-23T11:47:34.129 INFO:tasks.workunit.client.1.smithi111.stdout:4/434: dwrite d0/dc/d4c/d4b/d4d/f54 [0,4194304] 0 2024-03-23T11:47:34.132 INFO:tasks.workunit.client.1.smithi111.stdout:3/373: readlink d0/d7/d19/d5e/l67 0 2024-03-23T11:47:34.138 INFO:tasks.workunit.client.0.smithi063.stdout:4/396: dread d5/da/d20/d4b/f5a [0,4194304] 0 2024-03-23T11:47:34.142 INFO:tasks.workunit.client.1.smithi111.stdout:9/478: link d0/d4/d1c/d2d/d3b/d42/d4f/f61 d0/f9b 0 2024-03-23T11:47:34.144 INFO:tasks.workunit.client.1.smithi111.stdout:5/481: dwrite d1/f49 [0,4194304] 0 2024-03-23T11:47:34.146 INFO:tasks.workunit.client.1.smithi111.stdout:9/479: write d0/f5f [3053540,125590] 0 2024-03-23T11:47:34.148 INFO:tasks.workunit.client.1.smithi111.stdout:6/476: dread d2/ff [0,4194304] 0 2024-03-23T11:47:34.150 INFO:tasks.workunit.client.1.smithi111.stdout:6/477: readlink d2/d17/l19 0 2024-03-23T11:47:34.151 INFO:tasks.workunit.client.1.smithi111.stdout:2/512: dwrite d2/d11/f1b [0,4194304] 0 2024-03-23T11:47:34.151 INFO:tasks.workunit.client.1.smithi111.stdout:9/480: chown d0/d4/d1c/d2d/d75/d7a 12875 1 2024-03-23T11:47:34.153 INFO:tasks.workunit.client.0.smithi063.stdout:1/306: rmdir d3/d49/d56 39 2024-03-23T11:47:34.154 INFO:tasks.workunit.client.1.smithi111.stdout:0/426: link d5/db/c29 d5/db/d1f/d47/d6f/c8a 0 2024-03-23T11:47:34.157 INFO:tasks.workunit.client.0.smithi063.stdout:1/307: write d3/d3f/d4d/f5b [117559,100384] 0 2024-03-23T11:47:34.158 INFO:tasks.workunit.client.1.smithi111.stdout:5/482: readlink d1/d3/d6/d7/d63/d2c/d82/d66/d77/l93 0 2024-03-23T11:47:34.159 INFO:tasks.workunit.client.1.smithi111.stdout:9/481: read d0/f48 [3560370,125780] 0 2024-03-23T11:47:34.164 INFO:tasks.workunit.client.0.smithi063.stdout:9/374: read d4/f21 [220897,36841] 0 2024-03-23T11:47:34.164 INFO:tasks.workunit.client.1.smithi111.stdout:7/420: truncate d14/d1b/d1e/d2c/d2e/f34 6288712 0 2024-03-23T11:47:34.165 INFO:tasks.workunit.client.1.smithi111.stdout:3/374: dread d0/d1a/d2b/f33 [0,4194304] 0 2024-03-23T11:47:34.166 INFO:tasks.workunit.client.1.smithi111.stdout:5/483: dread - d1/d3/d6/d7/d63/d2c/d82/f88 zero size 2024-03-23T11:47:34.168 INFO:tasks.workunit.client.0.smithi063.stdout:0/353: dread d2/d12/f7e [0,4194304] 0 2024-03-23T11:47:34.171 INFO:tasks.workunit.client.0.smithi063.stdout:2/318: getdents d2/de 0 2024-03-23T11:47:34.171 INFO:tasks.workunit.client.1.smithi111.stdout:6/478: link d2/d1f/f90 d2/d1f/d45/d49/f9e 0 2024-03-23T11:47:34.172 INFO:tasks.workunit.client.1.smithi111.stdout:0/427: mkdir d5/db/d1f/d39/d8b 0 2024-03-23T11:47:34.172 INFO:tasks.workunit.client.1.smithi111.stdout:4/435: dwrite d0/d7/f1b [4194304,4194304] 0 2024-03-23T11:47:34.172 INFO:tasks.workunit.client.1.smithi111.stdout:1/424: truncate d0/d2b/d37/f74 245092 0 2024-03-23T11:47:34.173 INFO:tasks.workunit.client.1.smithi111.stdout:9/482: creat d0/d31/d35/f9c x:0 0 0 2024-03-23T11:47:34.176 INFO:tasks.workunit.client.1.smithi111.stdout:0/428: readlink d5/db/d4d/l70 0 2024-03-23T11:47:34.176 INFO:tasks.workunit.client.0.smithi063.stdout:1/308: symlink d3/d3f/l6c 0 2024-03-23T11:47:34.178 INFO:tasks.workunit.client.1.smithi111.stdout:5/484: truncate d1/d3/d6/d7/d63/d2c/d82/d66/f9b 1012257 0 2024-03-23T11:47:34.179 INFO:tasks.workunit.client.0.smithi063.stdout:0/354: creat d2/d48/f7f x:0 0 0 2024-03-23T11:47:34.179 INFO:tasks.workunit.client.1.smithi111.stdout:9/483: chown d0/d4/d1c/d2d/d6e 0 1 2024-03-23T11:47:34.180 INFO:tasks.workunit.client.0.smithi063.stdout:2/319: creat d2/d1c/f66 x:0 0 0 2024-03-23T11:47:34.180 INFO:tasks.workunit.client.1.smithi111.stdout:2/513: dread d2/d11/d17/d3e/d6b/d2e/d76/fab [0,4194304] 0 2024-03-23T11:47:34.182 INFO:tasks.workunit.client.0.smithi063.stdout:1/309: chown d3/df/l14 4 1 2024-03-23T11:47:34.183 INFO:tasks.workunit.client.0.smithi063.stdout:6/309: truncate d1/dc/d3a/f30 4100566 0 2024-03-23T11:47:34.184 INFO:tasks.workunit.client.1.smithi111.stdout:1/425: mkdir d0/d5/dd/d1e/d54/d87/d8e 0 2024-03-23T11:47:34.185 INFO:tasks.workunit.client.1.smithi111.stdout:4/436: symlink d0/dc/d7a/l7d 0 2024-03-23T11:47:34.190 INFO:tasks.workunit.client.0.smithi063.stdout:0/355: creat d2/d14/d36/d77/f80 x:0 0 0 2024-03-23T11:47:34.190 INFO:tasks.workunit.client.0.smithi063.stdout:1/310: write d3/df/d2f/f64 [402743,106794] 0 2024-03-23T11:47:34.191 INFO:tasks.workunit.client.0.smithi063.stdout:9/375: dwrite d4/f2a [0,4194304] 0 2024-03-23T11:47:34.193 INFO:tasks.workunit.client.0.smithi063.stdout:0/356: readlink d2/d14/d36/l69 0 2024-03-23T11:47:34.196 INFO:tasks.workunit.client.1.smithi111.stdout:6/479: dwrite d2/d1a/d1c/f6b [0,4194304] 0 2024-03-23T11:47:34.196 INFO:tasks.workunit.client.0.smithi063.stdout:6/310: creat d1/dc/d3a/d57/f5c x:0 0 0 2024-03-23T11:47:34.197 INFO:tasks.workunit.client.0.smithi063.stdout:0/357: chown d2/d4/dd/d1f/d4f/l52 1751582 1 2024-03-23T11:47:34.201 INFO:tasks.workunit.client.0.smithi063.stdout:6/311: rmdir d1/d2a 39 2024-03-23T11:47:34.203 INFO:tasks.workunit.client.0.smithi063.stdout:0/358: creat d2/d12/d35/f81 x:0 0 0 2024-03-23T11:47:34.205 INFO:tasks.workunit.client.1.smithi111.stdout:2/514: mkdir d2/d7/da/dbd 0 2024-03-23T11:47:34.214 INFO:tasks.workunit.client.1.smithi111.stdout:4/437: dwrite d0/d17/f6b [0,4194304] 0 2024-03-23T11:47:34.214 INFO:tasks.workunit.client.1.smithi111.stdout:1/426: mknod d0/d2b/c8f 0 2024-03-23T11:47:34.215 INFO:tasks.workunit.client.0.smithi063.stdout:4/397: rename d5/da/d20/d4b/d29/d38/c4f to d5/da/d20/d4b/d16/d32/d6f/c73 0 2024-03-23T11:47:34.216 INFO:tasks.workunit.client.0.smithi063.stdout:6/312: read d1/dc/d3a/f3e [1982591,26472] 0 2024-03-23T11:47:34.220 INFO:tasks.workunit.client.0.smithi063.stdout:4/398: chown d5/da/d20/d4b/d16/d37/l3e 2540 1 2024-03-23T11:47:34.220 INFO:tasks.workunit.client.0.smithi063.stdout:6/313: chown d1/dc/d3a/f3f 16 1 2024-03-23T11:47:34.229 INFO:tasks.workunit.client.1.smithi111.stdout:6/480: creat d2/d1f/d45/d86/f9f x:0 0 0 2024-03-23T11:47:34.230 INFO:tasks.workunit.client.0.smithi063.stdout:4/399: mknod d5/da/d20/d4b/d1f/c74 0 2024-03-23T11:47:34.231 INFO:tasks.workunit.client.1.smithi111.stdout:5/485: write d1/d3/d6/f19 [616946,56343] 0 2024-03-23T11:47:34.231 INFO:tasks.workunit.client.1.smithi111.stdout:1/427: creat d0/d5/dd/d1e/d54/d87/f90 x:0 0 0 2024-03-23T11:47:34.237 INFO:tasks.workunit.client.0.smithi063.stdout:6/314: rename d1/d13/l14 to d1/d2a/d41/d42/l5d 0 2024-03-23T11:47:34.239 INFO:tasks.workunit.client.0.smithi063.stdout:0/359: dwrite d2/d12/d32/f6c [0,4194304] 0 2024-03-23T11:47:34.241 INFO:tasks.workunit.client.1.smithi111.stdout:7/421: dwrite d14/d1b/d1e/d2c/f38 [0,4194304] 0 2024-03-23T11:47:34.244 INFO:tasks.workunit.client.1.smithi111.stdout:8/373: sync 2024-03-23T11:47:34.244 INFO:tasks.workunit.client.1.smithi111.stdout:5/486: dread - d1/d3/d6/d7/f5a zero size 2024-03-23T11:47:34.249 INFO:tasks.workunit.client.0.smithi063.stdout:0/360: creat d2/d4/dd/d28/d46/d51/f82 x:0 0 0 2024-03-23T11:47:34.250 INFO:tasks.workunit.client.1.smithi111.stdout:6/481: unlink d2/d35/d70/l71 0 2024-03-23T11:47:34.250 INFO:tasks.workunit.client.1.smithi111.stdout:3/375: dwrite d0/f1f [0,4194304] 0 2024-03-23T11:47:34.250 INFO:tasks.workunit.client.1.smithi111.stdout:1/428: mkdir d0/d5/dd/d91 0 2024-03-23T11:47:34.252 INFO:tasks.workunit.client.0.smithi063.stdout:6/315: dread d1/d2a/d41/d42/f48 [0,4194304] 0 2024-03-23T11:47:34.253 INFO:tasks.workunit.client.1.smithi111.stdout:5/487: readlink d1/d3/d3a/la1 0 2024-03-23T11:47:34.253 INFO:tasks.workunit.client.0.smithi063.stdout:0/361: chown d2/d14/d36/l75 7680414 1 2024-03-23T11:47:34.256 INFO:tasks.workunit.client.1.smithi111.stdout:3/376: fsync d0/d7/d19/f29 0 2024-03-23T11:47:34.258 INFO:tasks.workunit.client.0.smithi063.stdout:4/400: dwrite d5/d1e/d2b/f54 [0,4194304] 0 2024-03-23T11:47:34.258 INFO:tasks.workunit.client.0.smithi063.stdout:6/316: chown d1/d15/d4b 889 1 2024-03-23T11:47:34.258 INFO:tasks.workunit.client.1.smithi111.stdout:2/515: dwrite d2/d11/f37 [0,4194304] 0 2024-03-23T11:47:34.259 INFO:tasks.workunit.client.1.smithi111.stdout:5/488: readlink d1/d3/d6/d8b/la0 0 2024-03-23T11:47:34.261 INFO:tasks.workunit.client.0.smithi063.stdout:9/376: dread d4/f14 [8388608,4194304] 0 2024-03-23T11:47:34.261 INFO:tasks.workunit.client.1.smithi111.stdout:6/482: mknod d2/d35/d70/ca0 0 2024-03-23T11:47:34.261 INFO:tasks.workunit.client.1.smithi111.stdout:9/484: dwrite d0/d4/d10/f2c [0,4194304] 0 2024-03-23T11:47:34.265 INFO:tasks.workunit.client.1.smithi111.stdout:1/429: mkdir d0/d5/dd/d1e/d54/d87/d92 0 2024-03-23T11:47:34.265 INFO:tasks.workunit.client.1.smithi111.stdout:7/422: truncate d14/d1b/d22/f29 1529705 0 2024-03-23T11:47:34.266 INFO:tasks.workunit.client.0.smithi063.stdout:6/317: unlink d1/d2a/d41/d42/c54 0 2024-03-23T11:47:34.266 INFO:tasks.workunit.client.0.smithi063.stdout:4/401: creat d5/d1e/f75 x:0 0 0 2024-03-23T11:47:34.268 INFO:tasks.workunit.client.1.smithi111.stdout:6/483: fsync d2/f31 0 2024-03-23T11:47:34.272 INFO:tasks.workunit.client.1.smithi111.stdout:1/430: write d0/d5/dd/d1e/d54/d87/f8a [3899103,76205] 0 2024-03-23T11:47:34.272 INFO:tasks.workunit.client.0.smithi063.stdout:6/318: dread - d1/dc/d21/d49/f56 zero size 2024-03-23T11:47:34.273 INFO:tasks.workunit.client.1.smithi111.stdout:7/423: mkdir d14/d37/d50/d74 0 2024-03-23T11:47:34.273 INFO:tasks.workunit.client.0.smithi063.stdout:4/402: write d5/da/d20/d4b/d1f/f21 [8376996,61671] 0 2024-03-23T11:47:34.275 INFO:tasks.workunit.client.1.smithi111.stdout:9/485: fdatasync d0/d4/d10/d29/d7f/f8a 0 2024-03-23T11:47:34.280 INFO:tasks.workunit.client.1.smithi111.stdout:6/484: rename d2/d1a/f5e to d2/d1f/d45/d49/fa1 0 2024-03-23T11:47:34.281 INFO:tasks.workunit.client.0.smithi063.stdout:9/377: dread d4/d4c/f40 [0,4194304] 0 2024-03-23T11:47:34.282 INFO:tasks.workunit.client.0.smithi063.stdout:4/403: symlink d5/da/d20/d4b/d16/d50/l76 0 2024-03-23T11:47:34.283 INFO:tasks.workunit.client.0.smithi063.stdout:6/319: creat d1/d2a/d41/f5e x:0 0 0 2024-03-23T11:47:34.283 INFO:tasks.workunit.client.1.smithi111.stdout:9/486: chown d0/d4/d1c/d36/f18 240492344 1 2024-03-23T11:47:34.285 INFO:tasks.workunit.client.1.smithi111.stdout:6/485: fsync d2/d1f/d45/f63 0 2024-03-23T11:47:34.286 INFO:tasks.workunit.client.0.smithi063.stdout:6/320: fdatasync d1/dc/f1d 0 2024-03-23T11:47:34.293 INFO:tasks.workunit.client.0.smithi063.stdout:6/321: write d1/dc/d3a/d57/f5c [994000,12267] 0 2024-03-23T11:47:34.294 INFO:tasks.workunit.client.0.smithi063.stdout:6/322: readlink d1/d13/l23 0 2024-03-23T11:47:34.295 INFO:tasks.workunit.client.0.smithi063.stdout:4/404: link d5/da/d20/d4b/d16/f5c d5/da/d20/f77 0 2024-03-23T11:47:34.295 INFO:tasks.workunit.client.1.smithi111.stdout:2/516: dwrite d2/d11/d17/d3e/f40 [4194304,4194304] 0 2024-03-23T11:47:34.296 INFO:tasks.workunit.client.1.smithi111.stdout:4/438: dread d0/dc/d30/f15 [0,4194304] 0 2024-03-23T11:47:34.300 INFO:tasks.workunit.client.1.smithi111.stdout:5/489: dwrite d1/f49 [0,4194304] 0 2024-03-23T11:47:34.302 INFO:tasks.workunit.client.0.smithi063.stdout:6/323: chown d1/d2a/d41/l59 72 1 2024-03-23T11:47:34.303 INFO:tasks.workunit.client.0.smithi063.stdout:4/405: dread - d5/da/d20/d4b/d16/d50/f69 zero size 2024-03-23T11:47:34.305 INFO:tasks.workunit.client.0.smithi063.stdout:0/362: dread d2/d14/f34 [0,4194304] 0 2024-03-23T11:47:34.305 INFO:tasks.workunit.client.1.smithi111.stdout:0/429: truncate d5/db/d1f/f1a 1070778 0 2024-03-23T11:47:34.308 INFO:tasks.workunit.client.1.smithi111.stdout:9/487: mkdir d0/d31/d41/d9d 0 2024-03-23T11:47:34.310 INFO:tasks.workunit.client.0.smithi063.stdout:9/378: dwrite d4/ff [0,4194304] 0 2024-03-23T11:47:34.310 INFO:tasks.workunit.client.1.smithi111.stdout:1/431: dwrite d0/f20 [0,4194304] 0 2024-03-23T11:47:34.311 INFO:tasks.workunit.client.1.smithi111.stdout:7/424: dwrite d14/d1b/f5a [0,4194304] 0 2024-03-23T11:47:34.313 INFO:tasks.workunit.client.1.smithi111.stdout:6/486: write d2/d80/f96 [2621664,90461] 0 2024-03-23T11:47:34.315 INFO:tasks.workunit.client.1.smithi111.stdout:4/439: rmdir d0/dc/d4c 39 2024-03-23T11:47:34.315 INFO:tasks.workunit.client.0.smithi063.stdout:6/324: truncate d1/d2a/d41/f5e 905918 0 2024-03-23T11:47:34.317 INFO:tasks.workunit.client.1.smithi111.stdout:5/490: truncate d1/d3/d6/d7/d63/d2c/f51 1129268 0 2024-03-23T11:47:34.320 INFO:tasks.workunit.client.0.smithi063.stdout:9/379: write d4/d2c/f4a [1240010,108265] 0 2024-03-23T11:47:34.321 INFO:tasks.workunit.client.0.smithi063.stdout:0/363: creat d2/d4/dd/d28/d46/f83 x:0 0 0 2024-03-23T11:47:34.323 INFO:tasks.workunit.client.1.smithi111.stdout:9/488: dread - d0/d4/d1c/d36/d28/f5d zero size 2024-03-23T11:47:34.325 INFO:tasks.workunit.client.0.smithi063.stdout:0/364: mknod d2/d48/c84 0 2024-03-23T11:47:34.326 INFO:tasks.workunit.client.0.smithi063.stdout:9/380: write f2 [9956678,34414] 0 2024-03-23T11:47:34.330 INFO:tasks.workunit.client.1.smithi111.stdout:7/425: unlink d14/d1b/d1e/d2c/c56 0 2024-03-23T11:47:34.331 INFO:tasks.workunit.client.0.smithi063.stdout:0/365: symlink d2/d12/d32/l85 0 2024-03-23T11:47:34.332 INFO:tasks.workunit.client.0.smithi063.stdout:4/406: dread d5/da/d20/d4b/d1f/f21 [4194304,4194304] 0 2024-03-23T11:47:34.336 INFO:tasks.workunit.client.1.smithi111.stdout:6/487: dwrite d2/d80/f96 [0,4194304] 0 2024-03-23T11:47:34.340 INFO:tasks.workunit.client.0.smithi063.stdout:0/366: creat d2/d4/dd/d28/d46/d51/f86 x:0 0 0 2024-03-23T11:47:34.341 INFO:tasks.workunit.client.1.smithi111.stdout:5/491: dwrite d1/d3/d6/d7/d63/d2c/d82/d66/f9b [0,4194304] 0 2024-03-23T11:47:34.343 INFO:tasks.workunit.client.0.smithi063.stdout:0/367: chown d2/d12/d35/f6a 27960946 1 2024-03-23T11:47:34.343 INFO:tasks.workunit.client.1.smithi111.stdout:4/440: mkdir d0/dc/d1f/d7e 0 2024-03-23T11:47:34.350 INFO:tasks.workunit.client.1.smithi111.stdout:9/489: rename d0/d4/d1c/d36/f6b to d0/d4/d10/d29/d7f/d8e/f9e 0 2024-03-23T11:47:34.351 INFO:tasks.workunit.client.1.smithi111.stdout:3/377: dread d0/d7/f23 [0,4194304] 0 2024-03-23T11:47:34.354 INFO:tasks.workunit.client.0.smithi063.stdout:4/407: mknod d5/da/c78 0 2024-03-23T11:47:34.355 INFO:tasks.workunit.client.1.smithi111.stdout:9/490: write d0/d4/d10/d29/f67 [954856,103611] 0 2024-03-23T11:47:34.357 INFO:tasks.workunit.client.1.smithi111.stdout:5/492: unlink l0 0 2024-03-23T11:47:34.357 INFO:tasks.workunit.client.0.smithi063.stdout:9/381: dwrite d4/d4c/f25 [0,4194304] 0 2024-03-23T11:47:34.357 INFO:tasks.workunit.client.0.smithi063.stdout:4/408: write d5/d1e/d2b/f45 [1973250,88036] 0 2024-03-23T11:47:34.358 INFO:tasks.workunit.client.1.smithi111.stdout:3/378: write d0/f31 [522387,78609] 0 2024-03-23T11:47:34.362 INFO:tasks.workunit.client.1.smithi111.stdout:3/379: write d0/d1a/d1d/f6e [696751,16310] 0 2024-03-23T11:47:34.365 INFO:tasks.workunit.client.1.smithi111.stdout:1/432: dread d0/d5/dd/d1e/d54/f60 [0,4194304] 0 2024-03-23T11:47:34.370 INFO:tasks.workunit.client.1.smithi111.stdout:8/374: sync 2024-03-23T11:47:34.375 INFO:tasks.workunit.client.0.smithi063.stdout:0/368: creat d2/d14/f87 x:0 0 0 2024-03-23T11:47:34.375 INFO:tasks.workunit.client.0.smithi063.stdout:2/320: truncate d2/d23/d2e/f35 319198 0 2024-03-23T11:47:34.378 INFO:tasks.workunit.client.0.smithi063.stdout:9/382: write d4/d4c/d19/f42 [5096188,112130] 0 2024-03-23T11:47:34.379 INFO:tasks.workunit.client.1.smithi111.stdout:2/517: getdents d2/d11/d17/d3e/d6b/d2e/d76 0 2024-03-23T11:47:34.380 INFO:tasks.workunit.client.0.smithi063.stdout:9/383: chown d4/f5a 5830 1 2024-03-23T11:47:34.382 INFO:tasks.workunit.client.0.smithi063.stdout:9/384: readlink d4/d31/d43/d49/l57 0 2024-03-23T11:47:34.394 INFO:tasks.workunit.client.1.smithi111.stdout:1/433: mknod d0/d5/dd/d1e/d54/c93 0 2024-03-23T11:47:34.395 INFO:tasks.workunit.client.1.smithi111.stdout:1/434: chown d0/d2b/d37 281242763 1 2024-03-23T11:47:34.397 INFO:tasks.workunit.client.1.smithi111.stdout:9/491: getdents d0/d31/d41/d71/d8f 0 2024-03-23T11:47:34.398 INFO:tasks.workunit.client.1.smithi111.stdout:5/493: dread d1/d20/f37 [0,4194304] 0 2024-03-23T11:47:34.400 INFO:tasks.workunit.client.1.smithi111.stdout:5/494: stat d1/d3/d6/d7 0 2024-03-23T11:47:34.400 INFO:tasks.workunit.client.1.smithi111.stdout:2/518: truncate d2/d11/d17/d3e/d6b/d2e/f6f 542815 0 2024-03-23T11:47:34.401 INFO:tasks.workunit.client.1.smithi111.stdout:0/430: dwrite d5/db/d1f/f18 [0,4194304] 0 2024-03-23T11:47:34.402 INFO:tasks.workunit.client.1.smithi111.stdout:5/495: readlink d1/d3/d3a/la1 0 2024-03-23T11:47:34.402 INFO:tasks.workunit.client.1.smithi111.stdout:3/380: rename d0/d1a/d2b/d38/l4a to d0/d72/l77 0 2024-03-23T11:47:34.404 INFO:tasks.workunit.client.1.smithi111.stdout:2/519: truncate d2/d7/fb4 506975 0 2024-03-23T11:47:34.406 INFO:tasks.workunit.client.1.smithi111.stdout:1/435: dread d0/d2b/d37/f4f [0,4194304] 0 2024-03-23T11:47:34.406 INFO:tasks.workunit.client.1.smithi111.stdout:0/431: fdatasync d5/db/d1f/d47/f5a 0 2024-03-23T11:47:34.408 INFO:tasks.workunit.client.1.smithi111.stdout:5/496: dread d1/d3/d6/d7/f69 [0,4194304] 0 2024-03-23T11:47:34.409 INFO:tasks.workunit.client.0.smithi063.stdout:6/325: fdatasync d1/dc/d3a/f30 0 2024-03-23T11:47:34.409 INFO:tasks.workunit.client.1.smithi111.stdout:2/520: chown d2/d11/d17/d3e/d6b/f2f 135608732 1 2024-03-23T11:47:34.412 INFO:tasks.workunit.client.1.smithi111.stdout:0/432: stat d5/db/d1f/l68 0 2024-03-23T11:47:34.413 INFO:tasks.workunit.client.0.smithi063.stdout:4/409: rename d5/d1e/d2b/f49 to d5/da/d72/f79 0 2024-03-23T11:47:34.414 INFO:tasks.workunit.client.1.smithi111.stdout:3/381: symlink d0/d1a/d2b/d38/l78 0 2024-03-23T11:47:34.414 INFO:tasks.workunit.client.0.smithi063.stdout:0/369: symlink d2/d4/l88 0 2024-03-23T11:47:34.415 INFO:tasks.workunit.client.1.smithi111.stdout:2/521: fdatasync d2/d11/f13 0 2024-03-23T11:47:34.416 INFO:tasks.workunit.client.1.smithi111.stdout:0/433: write d5/f7a [918549,34068] 0 2024-03-23T11:47:34.416 INFO:tasks.workunit.client.1.smithi111.stdout:9/492: creat d0/d4/f9f x:0 0 0 2024-03-23T11:47:34.417 INFO:tasks.workunit.client.0.smithi063.stdout:0/370: stat d2/d14/d36/l53 0 2024-03-23T11:47:34.418 INFO:tasks.workunit.client.0.smithi063.stdout:2/321: mknod d2/d23/d2e/d52/d44/d4d/c67 0 2024-03-23T11:47:34.419 INFO:tasks.workunit.client.1.smithi111.stdout:7/426: dwrite d14/d1b/d1e/d2c/d2e/f34 [4194304,4194304] 0 2024-03-23T11:47:34.420 INFO:tasks.workunit.client.0.smithi063.stdout:6/326: unlink d1/dc/d3a/f3e 0 2024-03-23T11:47:34.421 INFO:tasks.workunit.client.1.smithi111.stdout:1/436: mknod d0/d5/dd/d1e/d54/d81/c94 0 2024-03-23T11:47:34.422 INFO:tasks.workunit.client.0.smithi063.stdout:4/410: mkdir d5/da/d20/d4b/d29/d38/d7a 0 2024-03-23T11:47:34.422 INFO:tasks.workunit.client.1.smithi111.stdout:8/375: truncate d5/d26/d31/d5e/f3f 1511347 0 2024-03-23T11:47:34.423 INFO:tasks.workunit.client.0.smithi063.stdout:9/385: creat d4/d4c/f81 x:0 0 0 2024-03-23T11:47:34.424 INFO:tasks.workunit.client.1.smithi111.stdout:3/382: truncate d0/d1a/d2b/d38/f51 885755 0 2024-03-23T11:47:34.425 INFO:tasks.workunit.client.0.smithi063.stdout:0/371: creat d2/d12/d32/f89 x:0 0 0 2024-03-23T11:47:34.426 INFO:tasks.workunit.client.1.smithi111.stdout:2/522: symlink d2/d32/lbe 0 2024-03-23T11:47:34.426 INFO:tasks.workunit.client.1.smithi111.stdout:6/488: dwrite d2/d1a/d1c/d2c/d51/f5f [0,4194304] 0 2024-03-23T11:47:34.426 INFO:tasks.workunit.client.1.smithi111.stdout:8/376: truncate f1 4287954 0 2024-03-23T11:47:34.428 INFO:tasks.workunit.client.0.smithi063.stdout:2/322: write d2/f29 [4094184,29175] 0 2024-03-23T11:47:34.430 INFO:tasks.workunit.client.1.smithi111.stdout:2/523: chown d2/f4 1 1 2024-03-23T11:47:34.432 INFO:tasks.workunit.client.0.smithi063.stdout:0/372: write d2/d12/d32/f5c [1319823,11638] 0 2024-03-23T11:47:34.434 INFO:tasks.workunit.client.1.smithi111.stdout:7/427: read d14/d1b/d1e/d2c/d42/f45 [567103,119320] 0 2024-03-23T11:47:34.434 INFO:tasks.workunit.client.1.smithi111.stdout:4/441: dwrite d0/dc/d4c/d2a/f73 [0,4194304] 0 2024-03-23T11:47:34.434 INFO:tasks.workunit.client.0.smithi063.stdout:6/327: write d1/dc/f1f [1885545,57129] 0 2024-03-23T11:47:34.435 INFO:tasks.workunit.client.0.smithi063.stdout:2/323: stat d2/f1e 0 2024-03-23T11:47:34.436 INFO:tasks.workunit.client.1.smithi111.stdout:7/428: truncate d14/f36 1352167 0 2024-03-23T11:47:34.437 INFO:tasks.workunit.client.0.smithi063.stdout:9/386: mknod d4/d4c/c82 0 2024-03-23T11:47:34.439 INFO:tasks.workunit.client.1.smithi111.stdout:9/493: mkdir d0/d31/da0 0 2024-03-23T11:47:34.441 INFO:tasks.workunit.client.0.smithi063.stdout:9/387: readlink d4/d31/d43/l76 0 2024-03-23T11:47:34.441 INFO:tasks.workunit.client.0.smithi063.stdout:6/328: stat d1/dc/f1f 0 2024-03-23T11:47:34.441 INFO:tasks.workunit.client.1.smithi111.stdout:0/434: mknod d5/db/d1f/d39/d8b/c8c 0 2024-03-23T11:47:34.442 INFO:tasks.workunit.client.1.smithi111.stdout:9/494: write d0/d4/d1c/f66 [1003635,13312] 0 2024-03-23T11:47:34.447 INFO:tasks.workunit.client.1.smithi111.stdout:3/383: mknod d0/d1a/d1d/c79 0 2024-03-23T11:47:34.448 INFO:tasks.workunit.client.0.smithi063.stdout:4/411: rename d5/da/d20/d4b/d29/d38/l6b to d5/da/d20/d4b/d16/d37/l7b 0 2024-03-23T11:47:34.448 INFO:tasks.workunit.client.0.smithi063.stdout:6/329: write d1/dc/f1d [2914144,2398] 0 2024-03-23T11:47:34.449 INFO:tasks.workunit.client.1.smithi111.stdout:1/437: write d0/d6/f34 [2462817,16033] 0 2024-03-23T11:47:34.451 INFO:tasks.workunit.client.1.smithi111.stdout:8/377: mknod d5/d26/d2f/d42/c7b 0 2024-03-23T11:47:34.453 INFO:tasks.workunit.client.1.smithi111.stdout:8/378: readlink d5/l15 0 2024-03-23T11:47:34.453 INFO:tasks.workunit.client.1.smithi111.stdout:4/442: creat d0/d7/f7f x:0 0 0 2024-03-23T11:47:34.454 INFO:tasks.workunit.client.1.smithi111.stdout:4/443: fdatasync d0/d7/f16 0 2024-03-23T11:47:34.455 INFO:tasks.workunit.client.1.smithi111.stdout:5/497: rename d1/f49 to d1/d3/d6/d8b/d59/d74/fa5 0 2024-03-23T11:47:34.455 INFO:tasks.workunit.client.1.smithi111.stdout:8/379: fsync d5/d26/d31/d3b/f6e 0 2024-03-23T11:47:34.456 INFO:tasks.workunit.client.0.smithi063.stdout:4/412: write d5/da/d20/d4b/f42 [2073882,34519] 0 2024-03-23T11:47:34.458 INFO:tasks.workunit.client.0.smithi063.stdout:6/330: dread d1/dc/d3a/f3f [4194304,4194304] 0 2024-03-23T11:47:34.458 INFO:tasks.workunit.client.0.smithi063.stdout:0/373: dread d2/d12/f50 [0,4194304] 0 2024-03-23T11:47:34.458 INFO:tasks.workunit.client.1.smithi111.stdout:7/429: readlink d14/d1b/d1e/d2c/d2e/d33/l6a 0 2024-03-23T11:47:34.458 INFO:tasks.workunit.client.1.smithi111.stdout:0/435: readlink d5/db/d1f/l5e 0 2024-03-23T11:47:34.460 INFO:tasks.workunit.client.0.smithi063.stdout:9/388: chown d4/d4c/c26 0 1 2024-03-23T11:47:34.460 INFO:tasks.workunit.client.1.smithi111.stdout:1/438: symlink d0/d2b/d37/l95 0 2024-03-23T11:47:34.461 INFO:tasks.workunit.client.1.smithi111.stdout:0/436: dread - d5/db/d1f/d39/f62 zero size 2024-03-23T11:47:34.464 INFO:tasks.workunit.client.1.smithi111.stdout:4/444: rename d0/d17/c60 to d0/d17/c80 0 2024-03-23T11:47:34.465 INFO:tasks.workunit.client.1.smithi111.stdout:7/430: read f9 [2538714,79409] 0 2024-03-23T11:47:34.467 INFO:tasks.workunit.client.1.smithi111.stdout:8/380: creat d5/d26/d2f/d42/d58/f7c x:0 0 0 2024-03-23T11:47:34.467 INFO:tasks.workunit.client.1.smithi111.stdout:6/489: creat d2/fa2 x:0 0 0 2024-03-23T11:47:34.469 INFO:tasks.workunit.client.0.smithi063.stdout:2/324: unlink d2/d23/d56/f5a 0 2024-03-23T11:47:34.471 INFO:tasks.workunit.client.1.smithi111.stdout:8/381: fdatasync d5/d26/d31/f39 0 2024-03-23T11:47:34.473 INFO:tasks.workunit.client.1.smithi111.stdout:8/382: write d5/d26/d2f/f77 [407357,100880] 0 2024-03-23T11:47:34.474 INFO:tasks.workunit.client.1.smithi111.stdout:4/445: creat d0/dc/d4c/d4b/d4f/f81 x:0 0 0 2024-03-23T11:47:34.476 INFO:tasks.workunit.client.1.smithi111.stdout:3/384: link d0/d7/d19/l3a d0/d7/l7a 0 2024-03-23T11:47:34.476 INFO:tasks.workunit.client.1.smithi111.stdout:7/431: rmdir d14/d1b/d1e/d2c/d2e 39 2024-03-23T11:47:34.477 INFO:tasks.workunit.client.1.smithi111.stdout:6/490: symlink d2/d35/la3 0 2024-03-23T11:47:34.479 INFO:tasks.workunit.client.1.smithi111.stdout:8/383: read d5/d26/d31/d4d/f59 [538447,119740] 0 2024-03-23T11:47:34.480 INFO:tasks.workunit.client.1.smithi111.stdout:9/495: write d0/d4/d1c/d2d/d3b/d42/d4f/f61 [4046928,69626] 0 2024-03-23T11:47:34.481 INFO:tasks.workunit.client.1.smithi111.stdout:3/385: mkdir d0/d72/d7b 0 2024-03-23T11:47:34.482 INFO:tasks.workunit.client.0.smithi063.stdout:4/413: unlink d5/da/f2a 0 2024-03-23T11:47:34.483 INFO:tasks.workunit.client.0.smithi063.stdout:4/414: chown d5/d1e 104759945 1 2024-03-23T11:47:34.483 INFO:tasks.workunit.client.1.smithi111.stdout:9/496: readlink d0/d4/l94 0 2024-03-23T11:47:34.485 INFO:tasks.workunit.client.0.smithi063.stdout:6/331: write d1/f5 [658063,27239] 0 2024-03-23T11:47:34.485 INFO:tasks.workunit.client.0.smithi063.stdout:2/325: dread d2/f37 [0,4194304] 0 2024-03-23T11:47:34.485 INFO:tasks.workunit.client.1.smithi111.stdout:3/386: dread - d0/d1a/d1d/d26/d5a/f74 zero size 2024-03-23T11:47:34.487 INFO:tasks.workunit.client.1.smithi111.stdout:2/524: dwrite d2/f5b [0,4194304] 0 2024-03-23T11:47:34.488 INFO:tasks.workunit.client.0.smithi063.stdout:9/389: mknod d4/c83 0 2024-03-23T11:47:34.489 INFO:tasks.workunit.client.0.smithi063.stdout:9/390: readlink d4/d4c/d44/l54 0 2024-03-23T11:47:34.490 INFO:tasks.workunit.client.1.smithi111.stdout:0/437: link d5/db/d1f/c31 d5/db/d1f/d39/c8d 0 2024-03-23T11:47:34.491 INFO:tasks.workunit.client.1.smithi111.stdout:3/387: write d0/d1a/f28 [1586456,87096] 0 2024-03-23T11:47:34.493 INFO:tasks.workunit.client.1.smithi111.stdout:8/384: mkdir d5/d32/d7d 0 2024-03-23T11:47:34.494 INFO:tasks.workunit.client.1.smithi111.stdout:4/446: dread d0/d17/f2c [0,4194304] 0 2024-03-23T11:47:34.494 INFO:tasks.workunit.client.0.smithi063.stdout:0/374: creat d2/d4/f8a x:0 0 0 2024-03-23T11:47:34.495 INFO:tasks.workunit.client.1.smithi111.stdout:7/432: dread d14/f15 [0,4194304] 0 2024-03-23T11:47:34.497 INFO:tasks.workunit.client.1.smithi111.stdout:8/385: dread - d5/d26/d2f/d42/d58/f7c zero size 2024-03-23T11:47:34.497 INFO:tasks.workunit.client.0.smithi063.stdout:4/415: write d5/da/d20/d4b/f5a [4704913,22507] 0 2024-03-23T11:47:34.501 INFO:tasks.workunit.client.0.smithi063.stdout:0/375: truncate d2/d4/dd/d28/d46/d51/f82 282051 0 2024-03-23T11:47:34.502 INFO:tasks.workunit.client.1.smithi111.stdout:4/447: truncate d0/dc/d4c/d2a/f70 783539 0 2024-03-23T11:47:34.506 INFO:tasks.workunit.client.0.smithi063.stdout:9/391: creat d4/d4c/d44/f84 x:0 0 0 2024-03-23T11:47:34.506 INFO:tasks.workunit.client.1.smithi111.stdout:6/491: dwrite d2/d80/f9b [0,4194304] 0 2024-03-23T11:47:34.506 INFO:tasks.workunit.client.1.smithi111.stdout:9/497: link d0/f9b d0/d31/d35/fa1 0 2024-03-23T11:47:34.508 INFO:tasks.workunit.client.0.smithi063.stdout:2/326: creat d2/d46/d54/f68 x:0 0 0 2024-03-23T11:47:34.510 INFO:tasks.workunit.client.1.smithi111.stdout:0/438: rmdir d5/db/d1f/d19/d45/d4b 39 2024-03-23T11:47:34.511 INFO:tasks.workunit.client.0.smithi063.stdout:0/376: symlink d2/d14/d36/d77/l8b 0 2024-03-23T11:47:34.512 INFO:tasks.workunit.client.0.smithi063.stdout:2/327: write d2/d23/d2e/f55 [449940,87457] 0 2024-03-23T11:47:34.514 INFO:tasks.workunit.client.1.smithi111.stdout:6/492: truncate d2/f28 783199 0 2024-03-23T11:47:34.515 INFO:tasks.workunit.client.1.smithi111.stdout:5/498: truncate d1/d3/d6/d7/d63/d2c/d82/d66/f9b 280755 0 2024-03-23T11:47:34.516 INFO:tasks.workunit.client.0.smithi063.stdout:9/392: mknod d4/d4c/d44/c85 0 2024-03-23T11:47:34.518 INFO:tasks.workunit.client.1.smithi111.stdout:9/498: symlink d0/d4/d10/d29/la2 0 2024-03-23T11:47:34.519 INFO:tasks.workunit.client.1.smithi111.stdout:2/525: dwrite d2/de/d12/d70/fb6 [0,4194304] 0 2024-03-23T11:47:34.520 INFO:tasks.workunit.client.0.smithi063.stdout:6/332: getdents d1 0 2024-03-23T11:47:34.520 INFO:tasks.workunit.client.1.smithi111.stdout:6/493: write d2/f57 [3301612,75139] 0 2024-03-23T11:47:34.520 INFO:tasks.workunit.client.0.smithi063.stdout:4/416: dread f2 [0,4194304] 0 2024-03-23T11:47:34.521 INFO:tasks.workunit.client.0.smithi063.stdout:0/377: mkdir d2/d6e/d8c 0 2024-03-23T11:47:34.523 INFO:tasks.workunit.client.0.smithi063.stdout:0/378: stat d2/d12/l2a 0 2024-03-23T11:47:34.523 INFO:tasks.workunit.client.1.smithi111.stdout:3/388: dwrite d0/d1a/d1d/d26/f46 [4194304,4194304] 0 2024-03-23T11:47:34.523 INFO:tasks.workunit.client.1.smithi111.stdout:5/499: chown d1/d3/d6/d8b/d59/d74/f4b 159 1 2024-03-23T11:47:34.524 INFO:tasks.workunit.client.1.smithi111.stdout:7/433: truncate d14/d1b/d20/f3a 1155913 0 2024-03-23T11:47:34.525 INFO:tasks.workunit.client.1.smithi111.stdout:5/500: fsync d1/d3/d6/f19 0 2024-03-23T11:47:34.526 INFO:tasks.workunit.client.0.smithi063.stdout:5/265: sync 2024-03-23T11:47:34.526 INFO:tasks.workunit.client.1.smithi111.stdout:5/501: chown d1/d20/c78 69346743 1 2024-03-23T11:47:34.527 INFO:tasks.workunit.client.0.smithi063.stdout:3/485: sync 2024-03-23T11:47:34.528 INFO:tasks.workunit.client.0.smithi063.stdout:3/486: chown d13/d22/d2d/c61 1753 1 2024-03-23T11:47:34.529 INFO:tasks.workunit.client.0.smithi063.stdout:2/328: creat d2/d23/f69 x:0 0 0 2024-03-23T11:47:34.529 INFO:tasks.workunit.client.1.smithi111.stdout:0/439: symlink d5/db/d1f/d47/d6f/l8e 0 2024-03-23T11:47:34.533 INFO:tasks.workunit.client.1.smithi111.stdout:9/499: creat d0/d4/d10/d29/fa3 x:0 0 0 2024-03-23T11:47:34.534 INFO:tasks.workunit.client.0.smithi063.stdout:4/417: fdatasync d5/da/d20/d4b/d16/f5c 0 2024-03-23T11:47:34.536 INFO:tasks.workunit.client.1.smithi111.stdout:9/500: write d0/d4/f83 [8060870,89077] 0 2024-03-23T11:47:34.538 INFO:tasks.workunit.client.0.smithi063.stdout:5/266: rmdir d2/d28/d3a 39 2024-03-23T11:47:34.542 INFO:tasks.workunit.client.0.smithi063.stdout:2/329: creat d2/d23/d2e/d52/f6a x:0 0 0 2024-03-23T11:47:34.544 INFO:tasks.workunit.client.1.smithi111.stdout:3/389: dwrite d0/d1a/d2b/f71 [0,4194304] 0 2024-03-23T11:47:34.545 INFO:tasks.workunit.client.1.smithi111.stdout:5/502: symlink d1/d3/d6/d7/d63/d2c/d43/la6 0 2024-03-23T11:47:34.546 INFO:tasks.workunit.client.0.smithi063.stdout:4/418: creat d5/d1e/d2b/f7c x:0 0 0 2024-03-23T11:47:34.547 INFO:tasks.workunit.client.1.smithi111.stdout:8/386: getdents d5/d1b 0 2024-03-23T11:47:34.550 INFO:tasks.workunit.client.1.smithi111.stdout:6/494: dwrite d2/d1a/f64 [0,4194304] 0 2024-03-23T11:47:34.550 INFO:tasks.workunit.client.1.smithi111.stdout:3/390: truncate d0/d1a/d2b/f4d 852733 0 2024-03-23T11:47:34.551 INFO:tasks.workunit.client.1.smithi111.stdout:8/387: chown d5/d1b/f68 406235289 1 2024-03-23T11:47:34.553 INFO:tasks.workunit.client.0.smithi063.stdout:4/419: write d5/d1e/f51 [1004913,86142] 0 2024-03-23T11:47:34.554 INFO:tasks.workunit.client.0.smithi063.stdout:3/487: dwrite d13/d22/d2d/d78/f92 [0,4194304] 0 2024-03-23T11:47:34.557 INFO:tasks.workunit.client.0.smithi063.stdout:3/488: chown d13/d22/f2f 91821 1 2024-03-23T11:47:34.561 INFO:tasks.workunit.client.1.smithi111.stdout:4/448: getdents d0/dc/d1f/d52 0 2024-03-23T11:47:34.561 INFO:tasks.workunit.client.1.smithi111.stdout:3/391: symlink d0/d7/d19/d5e/l7c 0 2024-03-23T11:47:34.561 INFO:tasks.workunit.client.0.smithi063.stdout:6/333: write d1/f12 [755401,1538] 0 2024-03-23T11:47:34.563 INFO:tasks.workunit.client.1.smithi111.stdout:2/526: rename d2/de/d12/d70/f9a to d2/fbf 0 2024-03-23T11:47:34.565 INFO:tasks.workunit.client.1.smithi111.stdout:9/501: mknod d0/d4/d6a/d89/ca4 0 2024-03-23T11:47:34.567 INFO:tasks.workunit.client.1.smithi111.stdout:4/449: dread - d0/dc/d1f/f76 zero size 2024-03-23T11:47:34.571 INFO:tasks.workunit.client.0.smithi063.stdout:0/379: getdents d2 0 2024-03-23T11:47:34.573 INFO:tasks.workunit.client.1.smithi111.stdout:0/440: write d5/db/d1f/d39/f3f [1036384,57859] 0 2024-03-23T11:47:34.574 INFO:tasks.workunit.client.0.smithi063.stdout:9/393: truncate d4/f14 5051974 0 2024-03-23T11:47:34.574 INFO:tasks.workunit.client.1.smithi111.stdout:2/527: write d2/de/d12/f2c [5044387,40336] 0 2024-03-23T11:47:34.575 INFO:tasks.workunit.client.0.smithi063.stdout:4/420: rmdir d5 39 2024-03-23T11:47:34.578 INFO:tasks.workunit.client.1.smithi111.stdout:1/439: sync 2024-03-23T11:47:34.580 INFO:tasks.workunit.client.0.smithi063.stdout:3/489: chown d13/d22/l41 31181885 1 2024-03-23T11:47:34.580 INFO:tasks.workunit.client.1.smithi111.stdout:6/495: dwrite d2/d1f/d45/f62 [0,4194304] 0 2024-03-23T11:47:34.581 INFO:tasks.workunit.client.0.smithi063.stdout:6/334: symlink d1/d2a/d41/d42/l5f 0 2024-03-23T11:47:34.583 INFO:tasks.workunit.client.0.smithi063.stdout:3/490: chown d13/d22/d2d/d3f/d5b 31709 1 2024-03-23T11:47:34.584 INFO:tasks.workunit.client.1.smithi111.stdout:5/503: dread d1/d3/d6/d8b/d59/f9f [0,4194304] 0 2024-03-23T11:47:34.587 INFO:tasks.workunit.client.1.smithi111.stdout:6/496: truncate d2/d1a/f1d 4990373 0 2024-03-23T11:47:34.588 INFO:tasks.workunit.client.1.smithi111.stdout:0/441: write d5/db/d1f/d19/d45/d4b/f61 [230967,29865] 0 2024-03-23T11:47:34.588 INFO:tasks.workunit.client.1.smithi111.stdout:4/450: creat d0/dc/d4c/f82 x:0 0 0 2024-03-23T11:47:34.590 INFO:tasks.workunit.client.0.smithi063.stdout:5/267: truncate d2/f6 788 0 2024-03-23T11:47:34.590 INFO:tasks.workunit.client.0.smithi063.stdout:3/491: read d13/d22/d2d/f3a [26696,75088] 0 2024-03-23T11:47:34.591 INFO:tasks.workunit.client.0.smithi063.stdout:2/330: dread d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:34.591 INFO:tasks.workunit.client.0.smithi063.stdout:4/421: dread - d5/da/d20/d4b/d16/d50/f70 zero size 2024-03-23T11:47:34.595 INFO:tasks.workunit.client.1.smithi111.stdout:3/392: dwrite d0/d1a/d2b/f33 [0,4194304] 0 2024-03-23T11:47:34.598 INFO:tasks.workunit.client.1.smithi111.stdout:3/393: write d0/f31 [3314654,15985] 0 2024-03-23T11:47:34.602 INFO:tasks.workunit.client.1.smithi111.stdout:2/528: getdents d2/d11/d17/d3e 0 2024-03-23T11:47:34.603 INFO:tasks.workunit.client.0.smithi063.stdout:5/268: unlink d2/l3c 0 2024-03-23T11:47:34.604 INFO:tasks.workunit.client.0.smithi063.stdout:0/380: dwrite d2/d4/dd/d28/d46/d51/f86 [0,4194304] 0 2024-03-23T11:47:34.605 INFO:tasks.workunit.client.1.smithi111.stdout:1/440: getdents d0/d2b 0 2024-03-23T11:47:34.606 INFO:tasks.workunit.client.1.smithi111.stdout:0/442: rename d5/f7a to d5/db/d1f/d39/f8f 0 2024-03-23T11:47:34.607 INFO:tasks.workunit.client.0.smithi063.stdout:2/331: unlink d2/f47 0 2024-03-23T11:47:34.611 INFO:tasks.workunit.client.0.smithi063.stdout:0/381: truncate d2/d4/d63/f6b 1020666 0 2024-03-23T11:47:34.617 INFO:tasks.workunit.client.0.smithi063.stdout:3/492: creat d13/d22/d2d/d3f/f9a x:0 0 0 2024-03-23T11:47:34.620 INFO:tasks.workunit.client.1.smithi111.stdout:5/504: symlink d1/d20/d5c/d99/la7 0 2024-03-23T11:47:34.620 INFO:tasks.workunit.client.0.smithi063.stdout:3/493: fdatasync d13/d22/f3d 0 2024-03-23T11:47:34.622 INFO:tasks.workunit.client.1.smithi111.stdout:7/434: sync 2024-03-23T11:47:34.625 INFO:tasks.workunit.client.1.smithi111.stdout:6/497: dwrite d2/d1a/d1c/d2c/f8c [0,4194304] 0 2024-03-23T11:47:34.626 INFO:tasks.workunit.client.1.smithi111.stdout:8/388: dread d5/f19 [0,4194304] 0 2024-03-23T11:47:34.627 INFO:tasks.workunit.client.0.smithi063.stdout:0/382: dread d2/d12/d32/f5c [0,4194304] 0 2024-03-23T11:47:34.630 INFO:tasks.workunit.client.1.smithi111.stdout:9/502: write d0/d4/d1c/d36/f7e [725888,71692] 0 2024-03-23T11:47:34.631 INFO:tasks.workunit.client.0.smithi063.stdout:4/422: rename d5/d1e/l65 to d5/l7d 0 2024-03-23T11:47:34.631 INFO:tasks.workunit.client.1.smithi111.stdout:3/394: mkdir d0/d7/d19/d7d 0 2024-03-23T11:47:34.633 INFO:tasks.workunit.client.0.smithi063.stdout:9/394: dwrite d4/fc [0,4194304] 0 2024-03-23T11:47:34.634 INFO:tasks.workunit.client.1.smithi111.stdout:2/529: mkdir d2/d11/d17/d3e/d5d/dc0 0 2024-03-23T11:47:34.635 INFO:tasks.workunit.client.1.smithi111.stdout:3/395: stat d0/d53 0 2024-03-23T11:47:34.636 INFO:tasks.workunit.client.1.smithi111.stdout:1/441: mkdir d0/d6/d48/d4a/d6c/d96 0 2024-03-23T11:47:34.638 INFO:tasks.workunit.client.1.smithi111.stdout:4/451: dwrite d0/dc/d4c/d4b/d4f/f81 [0,4194304] 0 2024-03-23T11:47:34.639 INFO:tasks.workunit.client.0.smithi063.stdout:4/423: read d5/da/d20/f77 [253779,52805] 0 2024-03-23T11:47:34.640 INFO:tasks.workunit.client.1.smithi111.stdout:5/505: symlink d1/d3/d6/d7/d63/d9d/la8 0 2024-03-23T11:47:34.644 INFO:tasks.workunit.client.0.smithi063.stdout:4/424: chown d5/d1e/d2b/f58 32653 1 2024-03-23T11:47:34.647 INFO:tasks.workunit.client.1.smithi111.stdout:7/435: read d14/d1b/d22/f4d [568128,122604] 0 2024-03-23T11:47:34.647 INFO:tasks.workunit.client.1.smithi111.stdout:3/396: creat d0/d1a/d1d/d26/f7e x:0 0 0 2024-03-23T11:47:34.648 INFO:tasks.workunit.client.0.smithi063.stdout:4/425: dread - d5/d1e/d2b/f7c zero size 2024-03-23T11:47:34.649 INFO:tasks.workunit.client.1.smithi111.stdout:9/503: mknod d0/d4/d1c/d2d/d6e/ca5 0 2024-03-23T11:47:34.650 INFO:tasks.workunit.client.1.smithi111.stdout:2/530: unlink d2/d11/d17/d1f/d30/f35 0 2024-03-23T11:47:34.650 INFO:tasks.workunit.client.1.smithi111.stdout:1/442: symlink d0/d6/d48/d4a/d6c/l97 0 2024-03-23T11:47:34.650 INFO:tasks.workunit.client.1.smithi111.stdout:0/443: dwrite d5/db/f26 [0,4194304] 0 2024-03-23T11:47:34.651 INFO:tasks.workunit.client.1.smithi111.stdout:8/389: mkdir d5/d26/d2f/d62/d7e 0 2024-03-23T11:47:34.653 INFO:tasks.workunit.client.1.smithi111.stdout:1/443: readlink d0/d5/l1c 0 2024-03-23T11:47:34.654 INFO:tasks.workunit.client.1.smithi111.stdout:6/498: creat d2/d35/d70/d98/fa4 x:0 0 0 2024-03-23T11:47:34.657 INFO:tasks.workunit.client.1.smithi111.stdout:9/504: truncate d0/f5f 4952888 0 2024-03-23T11:47:34.660 INFO:tasks.workunit.client.1.smithi111.stdout:5/506: fsync d1/d20/d5c/f7e 0 2024-03-23T11:47:34.660 INFO:tasks.workunit.client.1.smithi111.stdout:6/499: read - d2/d1f/d45/d49/f9e zero size 2024-03-23T11:47:34.663 INFO:tasks.workunit.client.0.smithi063.stdout:2/332: dwrite d2/de/d22/f5b [0,4194304] 0 2024-03-23T11:47:34.667 INFO:tasks.workunit.client.0.smithi063.stdout:0/383: link d2/d14/d36/l69 d2/d4/l8d 0 2024-03-23T11:47:34.669 INFO:tasks.workunit.client.0.smithi063.stdout:9/395: dwrite d4/d4c/d19/f42 [4194304,4194304] 0 2024-03-23T11:47:34.670 INFO:tasks.workunit.client.1.smithi111.stdout:3/397: mkdir d0/d1a/d2b/d38/d7f 0 2024-03-23T11:47:34.671 INFO:tasks.workunit.client.0.smithi063.stdout:0/384: truncate d2/d14/f87 51147 0 2024-03-23T11:47:34.681 INFO:tasks.workunit.client.0.smithi063.stdout:9/396: read d4/f5 [2437376,110394] 0 2024-03-23T11:47:34.683 INFO:tasks.workunit.client.1.smithi111.stdout:2/531: rename d2/c1c to d2/de/d12/d70/d96/cc1 0 2024-03-23T11:47:34.684 INFO:tasks.workunit.client.1.smithi111.stdout:4/452: dread d0/dc/d4c/d2a/f40 [0,4194304] 0 2024-03-23T11:47:34.685 INFO:tasks.workunit.client.0.smithi063.stdout:9/397: chown d4/d4c/f25 7638 1 2024-03-23T11:47:34.685 INFO:tasks.workunit.client.0.smithi063.stdout:0/385: rename d2/d4/f6 to d2/d4/d63/f8e 0 2024-03-23T11:47:34.685 INFO:tasks.workunit.client.1.smithi111.stdout:2/532: fdatasync d2/d7/f56 0 2024-03-23T11:47:34.685 INFO:tasks.workunit.client.1.smithi111.stdout:7/436: symlink d14/d1b/d1e/d2c/l75 0 2024-03-23T11:47:34.686 INFO:tasks.workunit.client.1.smithi111.stdout:6/500: creat d2/d1a/d1c/d2c/fa5 x:0 0 0 2024-03-23T11:47:34.687 INFO:tasks.workunit.client.1.smithi111.stdout:0/444: mknod d5/c90 0 2024-03-23T11:47:34.688 INFO:tasks.workunit.client.1.smithi111.stdout:1/444: dwrite d0/d2b/f45 [4194304,4194304] 0 2024-03-23T11:47:34.688 INFO:tasks.workunit.client.0.smithi063.stdout:5/269: dread d2/d7/d8/de/f41 [0,4194304] 0 2024-03-23T11:47:34.689 INFO:tasks.workunit.client.0.smithi063.stdout:0/386: write d2/d4/f30 [2276693,6427] 0 2024-03-23T11:47:34.690 INFO:tasks.workunit.client.1.smithi111.stdout:9/505: mknod d0/d31/d41/d9d/ca6 0 2024-03-23T11:47:34.690 INFO:tasks.workunit.client.1.smithi111.stdout:5/507: mkdir d1/d20/d41/da9 0 2024-03-23T11:47:34.691 INFO:tasks.workunit.client.0.smithi063.stdout:2/333: creat d2/f6b x:0 0 0 2024-03-23T11:47:34.693 INFO:tasks.workunit.client.0.smithi063.stdout:2/334: stat d2/d23/d2e/d52/d44/c49 0 2024-03-23T11:47:34.695 INFO:tasks.workunit.client.0.smithi063.stdout:4/426: write d5/da/d20/f77 [1030443,125672] 0 2024-03-23T11:47:34.695 INFO:tasks.workunit.client.1.smithi111.stdout:0/445: readlink d5/l65 0 2024-03-23T11:47:34.698 INFO:tasks.workunit.client.1.smithi111.stdout:2/533: mkdir d2/d11/d17/d3e/d5d/dc2 0 2024-03-23T11:47:34.698 INFO:tasks.workunit.client.0.smithi063.stdout:4/427: stat d5/d1e/d2b/f54 0 2024-03-23T11:47:34.702 INFO:tasks.workunit.client.1.smithi111.stdout:6/501: creat d2/d1f/fa6 x:0 0 0 2024-03-23T11:47:34.704 INFO:tasks.workunit.client.1.smithi111.stdout:9/506: mkdir d0/d4/d6a/da7 0 2024-03-23T11:47:34.704 INFO:tasks.workunit.client.1.smithi111.stdout:6/502: chown d2/d1f/l3c 36004212 1 2024-03-23T11:47:34.706 INFO:tasks.workunit.client.1.smithi111.stdout:4/453: mkdir d0/dc/d4c/d83 0 2024-03-23T11:47:34.707 INFO:tasks.workunit.client.0.smithi063.stdout:0/387: creat d2/d48/f8f x:0 0 0 2024-03-23T11:47:34.710 INFO:tasks.workunit.client.0.smithi063.stdout:2/335: readlink d2/l31 0 2024-03-23T11:47:34.711 INFO:tasks.workunit.client.1.smithi111.stdout:3/398: dread d0/d7/f47 [0,4194304] 0 2024-03-23T11:47:34.712 INFO:tasks.workunit.client.1.smithi111.stdout:3/399: chown d0/d1a/d4f 90 1 2024-03-23T11:47:34.714 INFO:tasks.workunit.client.1.smithi111.stdout:3/400: stat d0/d1a/d1d/d26/d5a/l66 0 2024-03-23T11:47:34.715 INFO:tasks.workunit.client.0.smithi063.stdout:0/388: symlink d2/d14/d36/l90 0 2024-03-23T11:47:34.715 INFO:tasks.workunit.client.0.smithi063.stdout:9/398: dwrite d4/ff [4194304,4194304] 0 2024-03-23T11:47:34.718 INFO:tasks.workunit.client.0.smithi063.stdout:0/389: chown d2/d48 0 1 2024-03-23T11:47:34.718 INFO:tasks.workunit.client.0.smithi063.stdout:3/494: dread d13/d22/d2d/d3f/f8a [0,4194304] 0 2024-03-23T11:47:34.719 INFO:tasks.workunit.client.1.smithi111.stdout:3/401: read d0/d7/d19/f29 [739969,107558] 0 2024-03-23T11:47:34.720 INFO:tasks.workunit.client.0.smithi063.stdout:0/390: fsync d2/d14/d36/d77/f7c 0 2024-03-23T11:47:34.721 INFO:tasks.workunit.client.0.smithi063.stdout:3/495: chown d13/d22/d2d/d3f/d5b 496 1 2024-03-23T11:47:34.723 INFO:tasks.workunit.client.1.smithi111.stdout:4/454: dread d0/f7c [0,4194304] 0 2024-03-23T11:47:34.724 INFO:tasks.workunit.client.0.smithi063.stdout:9/399: truncate d4/f1e 9407745 0 2024-03-23T11:47:34.725 INFO:tasks.workunit.client.1.smithi111.stdout:7/437: mknod d14/d37/d50/d74/c76 0 2024-03-23T11:47:34.725 INFO:tasks.workunit.client.1.smithi111.stdout:8/390: link d5/d26/d31/d5e/f49 d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f7f 0 2024-03-23T11:47:34.727 INFO:tasks.workunit.client.1.smithi111.stdout:2/534: creat d2/de/d12/d70/fc3 x:0 0 0 2024-03-23T11:47:34.727 INFO:tasks.workunit.client.1.smithi111.stdout:1/445: mkdir d0/d5/dd/d98 0 2024-03-23T11:47:34.728 INFO:tasks.workunit.client.0.smithi063.stdout:0/391: creat d2/d12/d32/f91 x:0 0 0 2024-03-23T11:47:34.728 INFO:tasks.workunit.client.0.smithi063.stdout:4/428: dread d5/da/d20/d4b/d16/f5c [0,4194304] 0 2024-03-23T11:47:34.728 INFO:tasks.workunit.client.1.smithi111.stdout:6/503: mkdir d2/d35/da7 0 2024-03-23T11:47:34.728 INFO:tasks.workunit.client.1.smithi111.stdout:5/508: mknod d1/d3/d6/d7/d86/caa 0 2024-03-23T11:47:34.732 INFO:tasks.workunit.client.1.smithi111.stdout:2/535: read - d2/de/d12/d70/f7e zero size 2024-03-23T11:47:34.732 INFO:tasks.workunit.client.1.smithi111.stdout:0/446: dwrite d5/db/d1f/f79 [0,4194304] 0 2024-03-23T11:47:34.733 INFO:tasks.workunit.client.1.smithi111.stdout:1/446: readlink d0/l2a 0 2024-03-23T11:47:34.733 INFO:tasks.workunit.client.0.smithi063.stdout:3/496: mknod d13/d22/d2d/d3f/d5b/c9b 0 2024-03-23T11:47:34.738 INFO:tasks.workunit.client.0.smithi063.stdout:3/497: write d13/d66/f75 [381238,50664] 0 2024-03-23T11:47:34.740 INFO:tasks.workunit.client.1.smithi111.stdout:4/455: rename d0/d7/f16 to d0/dc/d4c/d71/f84 0 2024-03-23T11:47:34.740 INFO:tasks.workunit.client.1.smithi111.stdout:9/507: dread d0/d4/d10/f51 [0,4194304] 0 2024-03-23T11:47:34.742 INFO:tasks.workunit.client.1.smithi111.stdout:4/456: fdatasync d0/dc/d4c/d2a/f73 0 2024-03-23T11:47:34.743 INFO:tasks.workunit.client.0.smithi063.stdout:4/429: mkdir d5/da/d7e 0 2024-03-23T11:47:34.745 INFO:tasks.workunit.client.0.smithi063.stdout:2/336: dwrite d2/d23/d2e/f5e [0,4194304] 0 2024-03-23T11:47:34.747 INFO:tasks.workunit.client.1.smithi111.stdout:3/402: fsync d0/d1a/d2b/d38/f3c 0 2024-03-23T11:47:34.749 INFO:tasks.workunit.client.1.smithi111.stdout:7/438: dwrite d14/d1b/d20/f57 [4194304,4194304] 0 2024-03-23T11:47:34.751 INFO:tasks.workunit.client.0.smithi063.stdout:5/270: dread d2/d7/fd [0,4194304] 0 2024-03-23T11:47:34.752 INFO:tasks.workunit.client.0.smithi063.stdout:3/498: creat d13/f9c x:0 0 0 2024-03-23T11:47:34.752 INFO:tasks.workunit.client.1.smithi111.stdout:1/447: write d0/d5/dd/d1e/d54/f60 [3601558,127436] 0 2024-03-23T11:47:34.754 INFO:tasks.workunit.client.0.smithi063.stdout:3/499: chown d13/d22/d2d/c69 45 1 2024-03-23T11:47:34.757 INFO:tasks.workunit.client.0.smithi063.stdout:2/337: rmdir d2/de 39 2024-03-23T11:47:34.757 INFO:tasks.workunit.client.1.smithi111.stdout:0/447: dwrite d5/db/d1f/d48/f6a [0,4194304] 0 2024-03-23T11:47:34.761 INFO:tasks.workunit.client.0.smithi063.stdout:3/500: dread d13/d66/f75 [0,4194304] 0 2024-03-23T11:47:34.761 INFO:tasks.workunit.client.1.smithi111.stdout:0/448: readlink d5/db/d1f/l81 0 2024-03-23T11:47:34.762 INFO:tasks.workunit.client.0.smithi063.stdout:4/430: readlink d5/da/d20/d4b/d29/d38/l62 0 2024-03-23T11:47:34.764 INFO:tasks.workunit.client.1.smithi111.stdout:6/504: mknod d2/d1a/d1c/d2c/d51/ca8 0 2024-03-23T11:47:34.765 INFO:tasks.workunit.client.0.smithi063.stdout:3/501: chown d13/d22/d2d/d78/f92 13154 1 2024-03-23T11:47:34.765 INFO:tasks.workunit.client.0.smithi063.stdout:2/338: write d2/d1c/f30 [5194738,19710] 0 2024-03-23T11:47:34.767 INFO:tasks.workunit.client.1.smithi111.stdout:5/509: rename d1/d3/d6/d30/d6f to d1/d3/d6/d7/d63/d2c/d82/dab 0 2024-03-23T11:47:34.769 INFO:tasks.workunit.client.0.smithi063.stdout:4/431: creat d5/da/d20/d4b/d16/d32/d6f/f7f x:0 0 0 2024-03-23T11:47:34.771 INFO:tasks.workunit.client.1.smithi111.stdout:7/439: creat d14/d44/d4b/f77 x:0 0 0 2024-03-23T11:47:34.771 INFO:tasks.workunit.client.0.smithi063.stdout:2/339: truncate d2/d23/d56/f57 939985 0 2024-03-23T11:47:34.774 INFO:tasks.workunit.client.0.smithi063.stdout:3/502: truncate d13/d22/d2d/d7d/f94 1036766 0 2024-03-23T11:47:34.775 INFO:tasks.workunit.client.1.smithi111.stdout:2/536: creat d2/d11/d17/d3e/d5d/dc2/fc4 x:0 0 0 2024-03-23T11:47:34.775 INFO:tasks.workunit.client.1.smithi111.stdout:1/448: mknod d0/d5/dd/d1e/d54/d81/c99 0 2024-03-23T11:47:34.776 INFO:tasks.workunit.client.1.smithi111.stdout:4/457: creat d0/dc/d78/f85 x:0 0 0 2024-03-23T11:47:34.777 INFO:tasks.workunit.client.0.smithi063.stdout:0/392: dread d2/d4/f9 [0,4194304] 0 2024-03-23T11:47:34.778 INFO:tasks.workunit.client.1.smithi111.stdout:1/449: chown d0/d5/c14 14288515 1 2024-03-23T11:47:34.780 INFO:tasks.workunit.client.1.smithi111.stdout:4/458: read d0/dc/f55 [4185043,25118] 0 2024-03-23T11:47:34.784 INFO:tasks.workunit.client.1.smithi111.stdout:6/505: dwrite d2/fb [0,4194304] 0 2024-03-23T11:47:34.784 INFO:tasks.workunit.client.1.smithi111.stdout:5/510: creat d1/d3/d6/d7/d63/fac x:0 0 0 2024-03-23T11:47:34.786 INFO:tasks.workunit.client.1.smithi111.stdout:8/391: getdents d5/d26/d2f/d62 0 2024-03-23T11:47:34.787 INFO:tasks.workunit.client.1.smithi111.stdout:5/511: dread - d1/d3/f7d zero size 2024-03-23T11:47:34.789 INFO:tasks.workunit.client.0.smithi063.stdout:5/271: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:34.791 INFO:tasks.workunit.client.1.smithi111.stdout:2/537: creat d2/d11/d17/d3e/fc5 x:0 0 0 2024-03-23T11:47:34.791 INFO:tasks.workunit.client.0.smithi063.stdout:8/630: sync 2024-03-23T11:47:34.793 INFO:tasks.workunit.client.0.smithi063.stdout:4/432: getdents d5/da/d20/d4b 0 2024-03-23T11:47:34.796 INFO:tasks.workunit.client.1.smithi111.stdout:1/450: unlink d0/d5/dd/d1e/d54/l72 0 2024-03-23T11:47:34.798 INFO:tasks.workunit.client.1.smithi111.stdout:4/459: rename d0/dc/d1f/d52 to d0/d17/d86 0 2024-03-23T11:47:34.798 INFO:tasks.workunit.client.0.smithi063.stdout:4/433: read d5/fc [302217,56013] 0 2024-03-23T11:47:34.801 INFO:tasks.workunit.client.1.smithi111.stdout:7/440: symlink d14/d1b/d1e/d2c/d2e/d33/l78 0 2024-03-23T11:47:34.804 INFO:tasks.workunit.client.1.smithi111.stdout:5/512: creat d1/d3/d6/d7/d63/d2c/d43/fad x:0 0 0 2024-03-23T11:47:34.805 INFO:tasks.workunit.client.0.smithi063.stdout:4/434: fdatasync d5/da/d20/d4b/d16/d32/d6f/f7f 0 2024-03-23T11:47:34.806 INFO:tasks.workunit.client.1.smithi111.stdout:8/392: dwrite d5/d26/d2f/d42/d58/f7c [0,4194304] 0 2024-03-23T11:47:34.807 INFO:tasks.workunit.client.0.smithi063.stdout:8/631: write d6/d1f/d54/d5c/f6d [1262098,130293] 0 2024-03-23T11:47:34.807 INFO:tasks.workunit.client.1.smithi111.stdout:6/506: dwrite d2/d35/f9c [0,4194304] 0 2024-03-23T11:47:34.807 INFO:tasks.workunit.client.0.smithi063.stdout:9/400: write d4/f5 [1702793,28997] 0 2024-03-23T11:47:34.820 INFO:tasks.workunit.client.0.smithi063.stdout:8/632: unlink d6/dd/df/f18 0 2024-03-23T11:47:34.822 INFO:tasks.workunit.client.1.smithi111.stdout:2/538: fdatasync d2/d11/fb7 0 2024-03-23T11:47:34.824 INFO:tasks.workunit.client.0.smithi063.stdout:9/401: creat d4/d31/d43/f86 x:0 0 0 2024-03-23T11:47:34.824 INFO:tasks.workunit.client.1.smithi111.stdout:3/403: dwrite d0/d1a/d2b/d38/f3c [0,4194304] 0 2024-03-23T11:47:34.824 INFO:tasks.workunit.client.1.smithi111.stdout:2/539: read d2/d11/fb7 [385802,44625] 0 2024-03-23T11:47:34.825 INFO:tasks.workunit.client.0.smithi063.stdout:0/393: truncate d2/d4/d63/f8e 1577935 0 2024-03-23T11:47:34.826 INFO:tasks.workunit.client.0.smithi063.stdout:8/633: readlink d6/d1c/d32/d37/l5b 0 2024-03-23T11:47:34.827 INFO:tasks.workunit.client.0.smithi063.stdout:3/503: dread d13/f50 [0,4194304] 0 2024-03-23T11:47:34.827 INFO:tasks.workunit.client.0.smithi063.stdout:5/272: dwrite d2/d7/f14 [0,4194304] 0 2024-03-23T11:47:34.828 INFO:tasks.workunit.client.1.smithi111.stdout:2/540: write d2/d11/d17/d3e/d6b/d2e/f82 [751175,95794] 0 2024-03-23T11:47:34.829 INFO:tasks.workunit.client.1.smithi111.stdout:8/393: rename f3 to d5/d26/d31/d3b/f80 0 2024-03-23T11:47:34.832 INFO:tasks.workunit.client.0.smithi063.stdout:9/402: creat d4/d2c/f87 x:0 0 0 2024-03-23T11:47:34.832 INFO:tasks.workunit.client.1.smithi111.stdout:1/451: symlink d0/d5/dd/d98/l9a 0 2024-03-23T11:47:34.832 INFO:tasks.workunit.client.1.smithi111.stdout:0/449: truncate d5/db/d4d/f52 2689236 0 2024-03-23T11:47:34.832 INFO:tasks.workunit.client.0.smithi063.stdout:5/273: chown d2/d28/c4c 679056 1 2024-03-23T11:47:34.833 INFO:tasks.workunit.client.0.smithi063.stdout:0/394: fdatasync d2/d12/d35/f81 0 2024-03-23T11:47:34.839 INFO:tasks.workunit.client.0.smithi063.stdout:4/435: dread d5/ff [8388608,4194304] 0 2024-03-23T11:47:34.841 INFO:tasks.workunit.client.1.smithi111.stdout:2/541: mkdir d2/d11/d17/d3e/d6b/dc6 0 2024-03-23T11:47:34.842 INFO:tasks.workunit.client.0.smithi063.stdout:9/403: chown d4/d4c/d19/f27 0 1 2024-03-23T11:47:34.842 INFO:tasks.workunit.client.0.smithi063.stdout:5/274: chown d2/d28/f2d 133617347 1 2024-03-23T11:47:34.842 INFO:tasks.workunit.client.1.smithi111.stdout:8/394: dwrite d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:34.842 INFO:tasks.workunit.client.0.smithi063.stdout:2/340: dread d2/de/ff [0,4194304] 0 2024-03-23T11:47:34.844 INFO:tasks.workunit.client.1.smithi111.stdout:2/542: write d2/d7/f56 [325351,32750] 0 2024-03-23T11:47:34.844 INFO:tasks.workunit.client.0.smithi063.stdout:8/634: rename d6/fd6 to d6/d1f/d54/d5c/d76/d85/fd8 0 2024-03-23T11:47:34.849 INFO:tasks.workunit.client.0.smithi063.stdout:4/436: stat d5/d1e/d2b/c3b 0 2024-03-23T11:47:34.855 INFO:tasks.workunit.client.0.smithi063.stdout:2/341: mkdir d2/d23/d2e/d52/d6c 0 2024-03-23T11:47:34.856 INFO:tasks.workunit.client.1.smithi111.stdout:1/452: creat d0/d5/dd/d1e/d54/d81/f9b x:0 0 0 2024-03-23T11:47:34.859 INFO:tasks.workunit.client.0.smithi063.stdout:5/275: creat d2/d7/d8/de/d49/f5d x:0 0 0 2024-03-23T11:47:34.860 INFO:tasks.workunit.client.0.smithi063.stdout:2/342: rmdir d2/d46/d54 39 2024-03-23T11:47:34.861 INFO:tasks.workunit.client.1.smithi111.stdout:1/453: dread d0/d6/d48/d4a/d6c/f7b [0,4194304] 0 2024-03-23T11:47:34.865 INFO:tasks.workunit.client.1.smithi111.stdout:7/441: rmdir d14/d1b/d1e 39 2024-03-23T11:47:34.866 INFO:tasks.workunit.client.0.smithi063.stdout:4/437: symlink d5/da/d20/l80 0 2024-03-23T11:47:34.868 INFO:tasks.workunit.client.0.smithi063.stdout:9/404: mknod d4/d4c/c88 0 2024-03-23T11:47:34.869 INFO:tasks.workunit.client.1.smithi111.stdout:2/543: symlink d2/d11/d17/d1f/d30/lc7 0 2024-03-23T11:47:34.870 INFO:tasks.workunit.client.1.smithi111.stdout:0/450: mkdir d5/db/d1f/d39/d64/d91 0 2024-03-23T11:47:34.879 INFO:tasks.workunit.client.0.smithi063.stdout:0/395: dwrite d2/d14/d36/f61 [0,4194304] 0 2024-03-23T11:47:34.879 INFO:tasks.workunit.client.0.smithi063.stdout:4/438: chown d5/da/d20/d4b/f5e 5 1 2024-03-23T11:47:34.879 INFO:tasks.workunit.client.0.smithi063.stdout:2/343: creat d2/d23/d2e/d52/d44/d4d/f6d x:0 0 0 2024-03-23T11:47:34.880 INFO:tasks.workunit.client.1.smithi111.stdout:4/460: link d0/d17/c80 d0/dc/d4c/d83/c87 0 2024-03-23T11:47:34.882 INFO:tasks.workunit.client.1.smithi111.stdout:3/404: link d0/d53/l75 d0/d72/d7b/l80 0 2024-03-23T11:47:34.883 INFO:tasks.workunit.client.0.smithi063.stdout:3/504: dwrite d13/d22/d2d/d3f/d5b/f5d [0,4194304] 0 2024-03-23T11:47:34.883 INFO:tasks.workunit.client.0.smithi063.stdout:4/439: read d5/f59 [235779,105263] 0 2024-03-23T11:47:34.886 INFO:tasks.workunit.client.1.smithi111.stdout:6/507: link d2/d35/f6f d2/d1a/d1c/d2c/d51/fa9 0 2024-03-23T11:47:34.891 INFO:tasks.workunit.client.0.smithi063.stdout:0/396: dread - d2/d12/f68 zero size 2024-03-23T11:47:34.891 INFO:tasks.workunit.client.0.smithi063.stdout:8/635: dwrite d6/d1f/d54/d63/d90/d5d/d7d/fd0 [0,4194304] 0 2024-03-23T11:47:34.891 INFO:tasks.workunit.client.0.smithi063.stdout:3/505: readlink d13/d22/d2d/d3f/d67/l81 0 2024-03-23T11:47:34.892 INFO:tasks.workunit.client.0.smithi063.stdout:5/276: rename d2/d7/c15 to d2/d28/c5e 0 2024-03-23T11:47:34.893 INFO:tasks.workunit.client.1.smithi111.stdout:7/442: fsync d14/d1b/f2d 0 2024-03-23T11:47:34.893 INFO:tasks.workunit.client.1.smithi111.stdout:2/544: rename d2/d32/c86 to d2/d11/d17/d3e/d5d/dc0/cc8 0 2024-03-23T11:47:34.893 INFO:tasks.workunit.client.0.smithi063.stdout:5/277: chown d2/d7/d8 33087984 1 2024-03-23T11:47:34.898 INFO:tasks.workunit.client.1.smithi111.stdout:4/461: rmdir d0/dc/d4c/d4b 39 2024-03-23T11:47:34.899 INFO:tasks.workunit.client.0.smithi063.stdout:8/636: write d6/dd/df/f60 [3041695,98203] 0 2024-03-23T11:47:34.899 INFO:tasks.workunit.client.0.smithi063.stdout:5/278: dread - d2/d2f/f37 zero size 2024-03-23T11:47:34.899 INFO:tasks.workunit.client.1.smithi111.stdout:6/508: unlink d2/d1f/f90 0 2024-03-23T11:47:34.901 INFO:tasks.workunit.client.1.smithi111.stdout:4/462: dread - d0/dc/d4c/f39 zero size 2024-03-23T11:47:34.901 INFO:tasks.workunit.client.0.smithi063.stdout:3/506: chown d13/d22/d2d/c4a 12765 1 2024-03-23T11:47:34.902 INFO:tasks.workunit.client.1.smithi111.stdout:1/454: creat d0/d5/f9c x:0 0 0 2024-03-23T11:47:34.906 INFO:tasks.workunit.client.1.smithi111.stdout:3/405: dwrite d0/d1a/d2b/f4d [0,4194304] 0 2024-03-23T11:47:34.906 INFO:tasks.workunit.client.0.smithi063.stdout:2/344: mknod d2/de/d22/c6e 0 2024-03-23T11:47:34.907 INFO:tasks.workunit.client.0.smithi063.stdout:3/507: chown d13/d15/d48/c98 112030 1 2024-03-23T11:47:34.907 INFO:tasks.workunit.client.1.smithi111.stdout:1/455: write d0/d6/d48/d4a/d6c/f77 [886802,109183] 0 2024-03-23T11:47:34.911 INFO:tasks.workunit.client.0.smithi063.stdout:0/397: mknod d2/d4/dd/d28/d46/c92 0 2024-03-23T11:47:34.911 INFO:tasks.workunit.client.0.smithi063.stdout:3/508: read d13/d22/d2d/f53 [1393947,67013] 0 2024-03-23T11:47:34.911 INFO:tasks.workunit.client.1.smithi111.stdout:9/508: sync 2024-03-23T11:47:34.913 INFO:tasks.workunit.client.0.smithi063.stdout:4/440: dwrite d5/d1e/d2b/f7c [0,4194304] 0 2024-03-23T11:47:34.914 INFO:tasks.workunit.client.0.smithi063.stdout:7/330: sync 2024-03-23T11:47:34.914 INFO:tasks.workunit.client.0.smithi063.stdout:1/311: sync 2024-03-23T11:47:34.922 INFO:tasks.workunit.client.1.smithi111.stdout:9/509: dread d0/d4/d10/f74 [0,4194304] 0 2024-03-23T11:47:34.923 INFO:tasks.workunit.client.0.smithi063.stdout:8/637: mknod d6/d1f/d54/d63/d90/d5d/cd9 0 2024-03-23T11:47:34.926 INFO:tasks.workunit.client.0.smithi063.stdout:2/345: rename d2/de/ff to d2/d23/d2e/d52/f6f 0 2024-03-23T11:47:34.928 INFO:tasks.workunit.client.0.smithi063.stdout:3/509: mkdir d13/d15/d48/d9d 0 2024-03-23T11:47:34.929 INFO:tasks.workunit.client.1.smithi111.stdout:1/456: creat d0/d5/dd/d91/f9d x:0 0 0 2024-03-23T11:47:34.930 INFO:tasks.workunit.client.0.smithi063.stdout:0/398: mkdir d2/d4/dd/d25/d93 0 2024-03-23T11:47:34.931 INFO:tasks.workunit.client.1.smithi111.stdout:5/513: write d1/d3/d6/d7/d63/d2c/d82/d66/f9b [610452,48411] 0 2024-03-23T11:47:34.931 INFO:tasks.workunit.client.0.smithi063.stdout:1/312: creat d3/d1c/f6d x:0 0 0 2024-03-23T11:47:34.932 INFO:tasks.workunit.client.0.smithi063.stdout:7/331: mkdir d5/d6/d19/d1f/d6c 0 2024-03-23T11:47:34.935 INFO:tasks.workunit.client.0.smithi063.stdout:3/510: write d13/d15/f87 [672934,11233] 0 2024-03-23T11:47:34.935 INFO:tasks.workunit.client.1.smithi111.stdout:3/406: dwrite d0/fd [4194304,4194304] 0 2024-03-23T11:47:34.935 INFO:tasks.workunit.client.1.smithi111.stdout:1/457: write d0/f59 [3779328,125297] 0 2024-03-23T11:47:34.936 INFO:tasks.workunit.client.0.smithi063.stdout:5/279: mkdir d2/d17/d4d/d5f 0 2024-03-23T11:47:34.937 INFO:tasks.workunit.client.0.smithi063.stdout:9/405: link d4/c3e d4/d31/c89 0 2024-03-23T11:47:34.938 INFO:tasks.workunit.client.1.smithi111.stdout:7/443: dwrite d14/f15 [0,4194304] 0 2024-03-23T11:47:34.940 INFO:tasks.workunit.client.1.smithi111.stdout:4/463: link d0/c36 d0/dc/d4c/c88 0 2024-03-23T11:47:34.940 INFO:tasks.workunit.client.1.smithi111.stdout:6/509: dwrite d2/d1f/d45/f76 [0,4194304] 0 2024-03-23T11:47:34.944 INFO:tasks.workunit.client.1.smithi111.stdout:6/510: read - d2/d1f/d45/d49/d8d/f9d zero size 2024-03-23T11:47:34.948 INFO:tasks.workunit.client.0.smithi063.stdout:8/638: rmdir d6/d1f/d54 39 2024-03-23T11:47:34.951 INFO:tasks.workunit.client.1.smithi111.stdout:3/407: creat d0/d55/f81 x:0 0 0 2024-03-23T11:47:34.951 INFO:tasks.workunit.client.0.smithi063.stdout:2/346: rmdir d2/d1c/d5c 39 2024-03-23T11:47:34.954 INFO:tasks.workunit.client.1.smithi111.stdout:4/464: chown d0/dc/d4c/l5c 105 1 2024-03-23T11:47:34.954 INFO:tasks.workunit.client.1.smithi111.stdout:7/444: write d14/d1b/d1e/f39 [3165039,34485] 0 2024-03-23T11:47:34.957 INFO:tasks.workunit.client.1.smithi111.stdout:1/458: rename d0/d5/l1c to d0/d6/d48/d4a/d6c/d96/l9e 0 2024-03-23T11:47:34.957 INFO:tasks.workunit.client.1.smithi111.stdout:4/465: write d0/d7/f1b [5156804,39981] 0 2024-03-23T11:47:34.958 INFO:tasks.workunit.client.0.smithi063.stdout:0/399: dwrite d2/d12/d35/f6a [0,4194304] 0 2024-03-23T11:47:34.958 INFO:tasks.workunit.client.1.smithi111.stdout:2/545: dread d2/d7/f7b [0,4194304] 0 2024-03-23T11:47:34.960 INFO:tasks.workunit.client.0.smithi063.stdout:5/280: creat d2/d7/d8/de/d35/f60 x:0 0 0 2024-03-23T11:47:34.960 INFO:tasks.workunit.client.1.smithi111.stdout:1/459: write d0/d5/dd/d1e/f36 [2764813,39772] 0 2024-03-23T11:47:34.961 INFO:tasks.workunit.client.1.smithi111.stdout:5/514: symlink d1/d3/d6/d8b/lae 0 2024-03-23T11:47:34.963 INFO:tasks.workunit.client.0.smithi063.stdout:4/441: link d5/da/d20/d4b/d29/d38/f60 d5/da/d20/d4b/d29/f81 0 2024-03-23T11:47:34.964 INFO:tasks.workunit.client.0.smithi063.stdout:6/335: sync 2024-03-23T11:47:34.967 INFO:tasks.workunit.client.1.smithi111.stdout:0/451: dread d5/db/d1f/f28 [0,4194304] 0 2024-03-23T11:47:34.968 INFO:tasks.workunit.client.1.smithi111.stdout:0/452: read d5/db/d4d/f52 [1105447,117297] 0 2024-03-23T11:47:34.969 INFO:tasks.workunit.client.0.smithi063.stdout:7/332: dread f1 [0,4194304] 0 2024-03-23T11:47:34.969 INFO:tasks.workunit.client.0.smithi063.stdout:0/400: dread - d2/d14/f66 zero size 2024-03-23T11:47:34.971 INFO:tasks.workunit.client.1.smithi111.stdout:0/453: write d5/db/d1f/f18 [4116073,61884] 0 2024-03-23T11:47:34.972 INFO:tasks.workunit.client.0.smithi063.stdout:2/347: unlink d2/d23/d2e/f64 0 2024-03-23T11:47:34.972 INFO:tasks.workunit.client.1.smithi111.stdout:4/466: rmdir d0/d17 39 2024-03-23T11:47:34.973 INFO:tasks.workunit.client.1.smithi111.stdout:0/454: chown d5/db/d1f/d82 1556156 1 2024-03-23T11:47:34.974 INFO:tasks.workunit.client.0.smithi063.stdout:0/401: dread - d2/d12/d35/f81 zero size 2024-03-23T11:47:34.975 INFO:tasks.workunit.client.0.smithi063.stdout:2/348: read d2/d23/d2e/f5e [1036367,21858] 0 2024-03-23T11:47:34.975 INFO:tasks.workunit.client.1.smithi111.stdout:4/467: chown d0/c43 57685177 1 2024-03-23T11:47:34.976 INFO:tasks.workunit.client.0.smithi063.stdout:8/639: unlink d6/d30/l8a 0 2024-03-23T11:47:34.978 INFO:tasks.workunit.client.1.smithi111.stdout:2/546: dread d2/d11/f37 [0,4194304] 0 2024-03-23T11:47:34.979 INFO:tasks.workunit.client.0.smithi063.stdout:1/313: rmdir d3/d3f/d4d/d52/d58 0 2024-03-23T11:47:34.979 INFO:tasks.workunit.client.0.smithi063.stdout:9/406: link d4/d4c/c7a d4/d4c/d44/c8a 0 2024-03-23T11:47:34.980 INFO:tasks.workunit.client.1.smithi111.stdout:9/510: creat d0/d4/d1c/d2d/d3b/fa8 x:0 0 0 2024-03-23T11:47:34.984 INFO:tasks.workunit.client.0.smithi063.stdout:7/333: creat d5/d6/d19/d1f/f6d x:0 0 0 2024-03-23T11:47:34.985 INFO:tasks.workunit.client.0.smithi063.stdout:2/349: truncate d2/f3a 533402 0 2024-03-23T11:47:34.987 INFO:tasks.workunit.client.0.smithi063.stdout:7/334: stat d5/d6/d19/d1f/d37 0 2024-03-23T11:47:34.988 INFO:tasks.workunit.client.0.smithi063.stdout:0/402: mkdir d2/d14/d36/d77/d94 0 2024-03-23T11:47:34.989 INFO:tasks.workunit.client.1.smithi111.stdout:0/455: dread d5/f40 [0,4194304] 0 2024-03-23T11:47:34.990 INFO:tasks.workunit.client.0.smithi063.stdout:5/281: mkdir d2/d28/d3a/d46/d61 0 2024-03-23T11:47:34.990 INFO:tasks.workunit.client.0.smithi063.stdout:4/442: link d5/da/d72/f79 d5/d1e/f82 0 2024-03-23T11:47:34.992 INFO:tasks.workunit.client.0.smithi063.stdout:0/403: fsync d2/d4/f30 0 2024-03-23T11:47:34.998 INFO:tasks.workunit.client.0.smithi063.stdout:6/336: mknod d1/dc/c60 0 2024-03-23T11:47:35.002 INFO:tasks.workunit.client.1.smithi111.stdout:1/460: read d0/d6/fc [2398441,58531] 0 2024-03-23T11:47:35.003 INFO:tasks.workunit.client.0.smithi063.stdout:3/511: dread d13/d15/d48/f62 [0,4194304] 0 2024-03-23T11:47:35.004 INFO:tasks.workunit.client.0.smithi063.stdout:8/640: chown d6/d1f/d54/d63/d90/c7e 60 1 2024-03-23T11:47:35.005 INFO:tasks.workunit.client.0.smithi063.stdout:7/335: unlink d5/d6/d19/d1f/d37/d20/f28 0 2024-03-23T11:47:35.005 INFO:tasks.workunit.client.1.smithi111.stdout:5/515: rmdir d1/d3/d6/d7/d63/d2c 39 2024-03-23T11:47:35.006 INFO:tasks.workunit.client.0.smithi063.stdout:4/443: dread d5/da/d20/d4b/d16/f5c [0,4194304] 0 2024-03-23T11:47:35.006 INFO:tasks.workunit.client.1.smithi111.stdout:5/516: read - d1/d20/f83 zero size 2024-03-23T11:47:35.010 INFO:tasks.workunit.client.0.smithi063.stdout:9/407: stat d4/d4c/d44/c8a 0 2024-03-23T11:47:35.011 INFO:tasks.workunit.client.0.smithi063.stdout:1/314: dwrite d3/d1c/f6d [0,4194304] 0 2024-03-23T11:47:35.012 INFO:tasks.workunit.client.1.smithi111.stdout:1/461: read d0/d6/f22 [2739301,23837] 0 2024-03-23T11:47:35.013 INFO:tasks.workunit.client.0.smithi063.stdout:4/444: stat d5/da/d20/d4b/d29/d38/f55 0 2024-03-23T11:47:35.013 INFO:tasks.workunit.client.0.smithi063.stdout:5/282: rename d2/f4 to d2/d7/d8/de/d2a/d3b/d43/d5c/f62 0 2024-03-23T11:47:35.014 INFO:tasks.workunit.client.1.smithi111.stdout:1/462: chown d0/c28 820011 1 2024-03-23T11:47:35.015 INFO:tasks.workunit.client.1.smithi111.stdout:1/463: chown d0/d6/d48/d4a/d6c/f77 2263 1 2024-03-23T11:47:35.022 INFO:tasks.workunit.client.0.smithi063.stdout:1/315: read d3/d35/d40/f42 [791265,97791] 0 2024-03-23T11:47:35.024 INFO:tasks.workunit.client.0.smithi063.stdout:6/337: creat d1/dc/d3a/f61 x:0 0 0 2024-03-23T11:47:35.032 INFO:tasks.workunit.client.0.smithi063.stdout:7/336: mkdir d5/d6/d19/d1f/d47/d6e 0 2024-03-23T11:47:35.035 INFO:tasks.workunit.client.0.smithi063.stdout:8/641: symlink d6/d1f/d54/d63/d90/d9e/lda 0 2024-03-23T11:47:35.040 INFO:tasks.workunit.client.0.smithi063.stdout:0/404: symlink d2/d6e/d8c/l95 0 2024-03-23T11:47:35.043 INFO:tasks.workunit.client.0.smithi063.stdout:0/405: readlink d2/l29 0 2024-03-23T11:47:35.044 INFO:tasks.workunit.client.1.smithi111.stdout:5/517: dread d1/d3/d6/d30/f70 [0,4194304] 0 2024-03-23T11:47:35.044 INFO:tasks.workunit.client.0.smithi063.stdout:5/283: creat d2/d7/d8/de/d49/f63 x:0 0 0 2024-03-23T11:47:35.044 INFO:tasks.workunit.client.0.smithi063.stdout:2/350: creat d2/d46/f70 x:0 0 0 2024-03-23T11:47:35.046 INFO:tasks.workunit.client.0.smithi063.stdout:9/408: unlink d4/c3e 0 2024-03-23T11:47:35.046 INFO:tasks.workunit.client.1.smithi111.stdout:7/445: mknod d14/d1b/d1e/d2c/d2e/d73/c79 0 2024-03-23T11:47:35.047 INFO:tasks.workunit.client.1.smithi111.stdout:0/456: dread - d5/db/d1f/d19/d45/f49 zero size 2024-03-23T11:47:35.049 INFO:tasks.workunit.client.0.smithi063.stdout:1/316: dwrite d3/df/d2f/f64 [0,4194304] 0 2024-03-23T11:47:35.050 INFO:tasks.workunit.client.1.smithi111.stdout:3/408: creat d0/d7/f82 x:0 0 0 2024-03-23T11:47:35.050 INFO:tasks.workunit.client.1.smithi111.stdout:8/395: sync 2024-03-23T11:47:35.052 INFO:tasks.workunit.client.0.smithi063.stdout:8/642: write d6/d1f/d54/d5c/d76/d85/fd8 [445524,63078] 0 2024-03-23T11:47:35.052 INFO:tasks.workunit.client.0.smithi063.stdout:6/338: dwrite d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:35.052 INFO:tasks.workunit.client.1.smithi111.stdout:4/468: mkdir d0/dc/d89 0 2024-03-23T11:47:35.054 INFO:tasks.workunit.client.1.smithi111.stdout:9/511: creat d0/d31/d41/d71/d8f/fa9 x:0 0 0 2024-03-23T11:47:35.054 INFO:tasks.workunit.client.0.smithi063.stdout:0/406: symlink d2/d4/dd/d28/d46/d51/l96 0 2024-03-23T11:47:35.055 INFO:tasks.workunit.client.0.smithi063.stdout:8/643: stat d6/dd/df/l46 0 2024-03-23T11:47:35.056 INFO:tasks.workunit.client.1.smithi111.stdout:9/512: read - d0/d4/d10/d29/fa3 zero size 2024-03-23T11:47:35.056 INFO:tasks.workunit.client.1.smithi111.stdout:1/464: creat d0/d2b/d7c/f9f x:0 0 0 2024-03-23T11:47:35.058 INFO:tasks.workunit.client.1.smithi111.stdout:9/513: chown d0/d4/d10/f2c 0 1 2024-03-23T11:47:35.059 INFO:tasks.workunit.client.1.smithi111.stdout:7/446: symlink d14/d1b/d1e/d2c/d2e/d73/l7a 0 2024-03-23T11:47:35.062 INFO:tasks.workunit.client.0.smithi063.stdout:0/407: dread - d2/d4/dd/d28/d46/d51/f65 zero size 2024-03-23T11:47:35.062 INFO:tasks.workunit.client.0.smithi063.stdout:1/317: rmdir d3/d3f 39 2024-03-23T11:47:35.063 INFO:tasks.workunit.client.1.smithi111.stdout:8/396: rmdir d5/d26/d31/d5e 39 2024-03-23T11:47:35.065 INFO:tasks.workunit.client.0.smithi063.stdout:3/512: write f6 [2668550,11214] 0 2024-03-23T11:47:35.067 INFO:tasks.workunit.client.1.smithi111.stdout:3/409: dwrite d0/d1a/d2b/f33 [0,4194304] 0 2024-03-23T11:47:35.068 INFO:tasks.workunit.client.1.smithi111.stdout:5/518: symlink d1/d20/d41/da9/laf 0 2024-03-23T11:47:35.069 INFO:tasks.workunit.client.0.smithi063.stdout:0/408: write d2/d4/dd/d28/d46/f5b [321684,109568] 0 2024-03-23T11:47:35.072 INFO:tasks.workunit.client.1.smithi111.stdout:3/410: truncate d0/d1a/d2b/f71 4744818 0 2024-03-23T11:47:35.073 INFO:tasks.workunit.client.0.smithi063.stdout:2/351: dread d2/f2f [0,4194304] 0 2024-03-23T11:47:35.074 INFO:tasks.workunit.client.1.smithi111.stdout:1/465: unlink d0/f3d 0 2024-03-23T11:47:35.077 INFO:tasks.workunit.client.0.smithi063.stdout:8/644: mknod d6/dd/d59/d7f/db5/cdb 0 2024-03-23T11:47:35.082 INFO:tasks.workunit.client.1.smithi111.stdout:7/447: dwrite d14/f15 [0,4194304] 0 2024-03-23T11:47:35.084 INFO:tasks.workunit.client.0.smithi063.stdout:2/352: stat d2/de/d22/f5b 0 2024-03-23T11:47:35.088 INFO:tasks.workunit.client.0.smithi063.stdout:0/409: symlink d2/d12/l97 0 2024-03-23T11:47:35.089 INFO:tasks.workunit.client.1.smithi111.stdout:5/519: dwrite d1/d3/d6/d8b/d59/d74/f5e [4194304,4194304] 0 2024-03-23T11:47:35.092 INFO:tasks.workunit.client.1.smithi111.stdout:2/547: getdents d2/d11/d17/d3e/d5d/dc0 0 2024-03-23T11:47:35.093 INFO:tasks.workunit.client.0.smithi063.stdout:1/318: read d3/df/f19 [750350,85305] 0 2024-03-23T11:47:35.093 INFO:tasks.workunit.client.0.smithi063.stdout:8/645: unlink d6/d1c/d32/d37/c78 0 2024-03-23T11:47:35.096 INFO:tasks.workunit.client.0.smithi063.stdout:5/284: dwrite d2/d7/d8/de/d49/f5d [0,4194304] 0 2024-03-23T11:47:35.096 INFO:tasks.workunit.client.1.smithi111.stdout:8/397: creat d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 x:0 0 0 2024-03-23T11:47:35.099 INFO:tasks.workunit.client.0.smithi063.stdout:6/339: dread d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:35.101 INFO:tasks.workunit.client.1.smithi111.stdout:7/448: dwrite d14/d1b/d1e/d2c/d42/f4a [4194304,4194304] 0 2024-03-23T11:47:35.102 INFO:tasks.workunit.client.0.smithi063.stdout:2/353: fdatasync d2/d23/d2e/d52/d44/d4d/f6d 0 2024-03-23T11:47:35.104 INFO:tasks.workunit.client.1.smithi111.stdout:7/449: write d14/d1b/d1e/d2c/d42/d5e/f6c [116982,80095] 0 2024-03-23T11:47:35.104 INFO:tasks.workunit.client.0.smithi063.stdout:7/337: dread d5/d22/f24 [0,4194304] 0 2024-03-23T11:47:35.109 INFO:tasks.workunit.client.1.smithi111.stdout:9/514: creat d0/d4/d6a/d9a/faa x:0 0 0 2024-03-23T11:47:35.110 INFO:tasks.workunit.client.0.smithi063.stdout:3/513: rename d13/d22/d2d/f53 to d13/d22/d2d/d3f/f9e 0 2024-03-23T11:47:35.115 INFO:tasks.workunit.client.0.smithi063.stdout:7/338: chown d5/d6/d19/d1f/f6d 63 1 2024-03-23T11:47:35.115 INFO:tasks.workunit.client.1.smithi111.stdout:0/457: rename d5/db/d4d/f5f to d5/db/f92 0 2024-03-23T11:47:35.117 INFO:tasks.workunit.client.1.smithi111.stdout:0/458: dread - d5/db/d1f/d39/f62 zero size 2024-03-23T11:47:35.117 INFO:tasks.workunit.client.0.smithi063.stdout:9/409: dwrite d4/fe [0,4194304] 0 2024-03-23T11:47:35.117 INFO:tasks.workunit.client.0.smithi063.stdout:5/285: chown d2/d7/d8/de/d2a/d3b/d43/f5a 799396 1 2024-03-23T11:47:35.121 INFO:tasks.workunit.client.0.smithi063.stdout:6/340: creat d1/d2a/f62 x:0 0 0 2024-03-23T11:47:35.125 INFO:tasks.workunit.client.0.smithi063.stdout:8/646: creat d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/fdc x:0 0 0 2024-03-23T11:47:35.125 INFO:tasks.workunit.client.0.smithi063.stdout:2/354: creat d2/d23/d2e/d52/f71 x:0 0 0 2024-03-23T11:47:35.126 INFO:tasks.workunit.client.1.smithi111.stdout:5/520: dread - d1/d20/f60 zero size 2024-03-23T11:47:35.128 INFO:tasks.workunit.client.0.smithi063.stdout:8/647: chown d6/dd/d59/d7f/f97 189034 1 2024-03-23T11:47:35.131 INFO:tasks.workunit.client.0.smithi063.stdout:9/410: dread d4/f5a [0,4194304] 0 2024-03-23T11:47:35.132 INFO:tasks.workunit.client.0.smithi063.stdout:0/410: creat d2/d4/dd/f98 x:0 0 0 2024-03-23T11:47:35.133 INFO:tasks.workunit.client.0.smithi063.stdout:5/286: fdatasync d2/f6 0 2024-03-23T11:47:35.134 INFO:tasks.workunit.client.1.smithi111.stdout:2/548: unlink d2/d7/da/f3b 0 2024-03-23T11:47:35.135 INFO:tasks.workunit.client.0.smithi063.stdout:3/514: write d13/d22/d2d/d3f/f8a [3938254,56763] 0 2024-03-23T11:47:35.135 INFO:tasks.workunit.client.0.smithi063.stdout:7/339: fsync d5/d6/d19/d1f/d37/d20/f53 0 2024-03-23T11:47:35.137 INFO:tasks.workunit.client.0.smithi063.stdout:6/341: mknod d1/dc/d3a/d57/c63 0 2024-03-23T11:47:35.139 INFO:tasks.workunit.client.1.smithi111.stdout:6/511: sync 2024-03-23T11:47:35.141 INFO:tasks.workunit.client.0.smithi063.stdout:0/411: truncate d2/d12/d32/f89 321941 0 2024-03-23T11:47:35.144 INFO:tasks.workunit.client.0.smithi063.stdout:9/411: rename d4/d4c/d19/l6f to d4/d4c/d19/l8b 0 2024-03-23T11:47:35.147 INFO:tasks.workunit.client.1.smithi111.stdout:0/459: mkdir d5/db/d1f/d48/d93 0 2024-03-23T11:47:35.148 INFO:tasks.workunit.client.1.smithi111.stdout:5/521: stat d1/d3/d6/d8b/d59/d74/c4f 0 2024-03-23T11:47:35.150 INFO:tasks.workunit.client.0.smithi063.stdout:1/319: rename d3/df/f30 to d3/df/f6e 0 2024-03-23T11:47:35.154 INFO:tasks.workunit.client.0.smithi063.stdout:3/515: dwrite d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:35.155 INFO:tasks.workunit.client.0.smithi063.stdout:2/355: mknod d2/d23/c72 0 2024-03-23T11:47:35.157 INFO:tasks.workunit.client.0.smithi063.stdout:2/356: truncate d2/f1e 2067002 0 2024-03-23T11:47:35.157 INFO:tasks.workunit.client.1.smithi111.stdout:4/469: getdents d0/dc/d1f 0 2024-03-23T11:47:35.157 INFO:tasks.workunit.client.0.smithi063.stdout:2/357: fdatasync d2/f3a 0 2024-03-23T11:47:35.161 INFO:tasks.workunit.client.0.smithi063.stdout:3/516: read d13/d22/f6f [152466,93965] 0 2024-03-23T11:47:35.164 INFO:tasks.workunit.client.1.smithi111.stdout:1/466: rmdir d0/d5/dd/d1e/d54/d87/d92 0 2024-03-23T11:47:35.164 INFO:tasks.workunit.client.0.smithi063.stdout:8/648: mkdir d6/d1f/d54/d63/d90/ddd 0 2024-03-23T11:47:35.165 INFO:tasks.workunit.client.1.smithi111.stdout:6/512: dread d2/f16 [0,4194304] 0 2024-03-23T11:47:35.165 INFO:tasks.workunit.client.0.smithi063.stdout:3/517: truncate d13/d22/d2d/d3f/d67/f84 1519218 0 2024-03-23T11:47:35.166 INFO:tasks.workunit.client.0.smithi063.stdout:2/358: dread d2/d23/d56/f57 [0,4194304] 0 2024-03-23T11:47:35.170 INFO:tasks.workunit.client.1.smithi111.stdout:7/450: mkdir d14/d44/d70/d7b 0 2024-03-23T11:47:35.170 INFO:tasks.workunit.client.0.smithi063.stdout:2/359: fdatasync d2/d23/d2e/f36 0 2024-03-23T11:47:35.173 INFO:tasks.workunit.client.1.smithi111.stdout:9/515: symlink d0/d31/lab 0 2024-03-23T11:47:35.174 INFO:tasks.workunit.client.1.smithi111.stdout:9/516: chown d0/d4/f83 767656216 1 2024-03-23T11:47:35.178 INFO:tasks.workunit.client.1.smithi111.stdout:0/460: truncate d5/db/f4a 77054 0 2024-03-23T11:47:35.179 INFO:tasks.workunit.client.1.smithi111.stdout:5/522: truncate d1/d3/d6/d8b/d59/d6c/f76 974602 0 2024-03-23T11:47:35.180 INFO:tasks.workunit.client.1.smithi111.stdout:2/549: symlink d2/d11/d17/d3e/d6b/d9d/d52/dba/lc9 0 2024-03-23T11:47:35.183 INFO:tasks.workunit.client.1.smithi111.stdout:4/470: creat d0/dc/d1f/f8a x:0 0 0 2024-03-23T11:47:35.185 INFO:tasks.workunit.client.1.smithi111.stdout:7/451: read d14/f25 [2088811,27394] 0 2024-03-23T11:47:35.185 INFO:tasks.workunit.client.1.smithi111.stdout:1/467: unlink d0/d5/dd/d1e/f36 0 2024-03-23T11:47:35.186 INFO:tasks.workunit.client.0.smithi063.stdout:0/412: write d2/d4/dd/f33 [691551,69752] 0 2024-03-23T11:47:35.186 INFO:tasks.workunit.client.0.smithi063.stdout:6/342: dwrite d1/d13/f52 [0,4194304] 0 2024-03-23T11:47:35.186 INFO:tasks.workunit.client.1.smithi111.stdout:7/452: fdatasync d14/d1b/f5a 0 2024-03-23T11:47:35.187 INFO:tasks.workunit.client.1.smithi111.stdout:7/453: chown d14/d37 8 1 2024-03-23T11:47:35.190 INFO:tasks.workunit.client.0.smithi063.stdout:0/413: fdatasync d2/d12/d35/f6a 0 2024-03-23T11:47:35.190 INFO:tasks.workunit.client.1.smithi111.stdout:9/517: unlink d0/d4/d1c/f26 0 2024-03-23T11:47:35.191 INFO:tasks.workunit.client.0.smithi063.stdout:6/343: dread - d1/dc/f5b zero size 2024-03-23T11:47:35.191 INFO:tasks.workunit.client.1.smithi111.stdout:6/513: dread d2/d1a/d1c/f24 [0,4194304] 0 2024-03-23T11:47:35.191 INFO:tasks.workunit.client.1.smithi111.stdout:9/518: dread - d0/d4/d10/d29/fa3 zero size 2024-03-23T11:47:35.192 INFO:tasks.workunit.client.0.smithi063.stdout:3/518: creat d13/d22/d2d/d3f/d58/d6e/f9f x:0 0 0 2024-03-23T11:47:35.193 INFO:tasks.workunit.client.1.smithi111.stdout:9/519: readlink d0/d4/d10/d29/l79 0 2024-03-23T11:47:35.193 INFO:tasks.workunit.client.0.smithi063.stdout:0/414: read - d2/d12/d35/f81 zero size 2024-03-23T11:47:35.193 INFO:tasks.workunit.client.1.smithi111.stdout:0/461: unlink d5/db/d4d/c54 0 2024-03-23T11:47:35.195 INFO:tasks.workunit.client.1.smithi111.stdout:5/523: chown d1/d3/d6/d7/d63/d2c/f34 760780 1 2024-03-23T11:47:35.197 INFO:tasks.workunit.client.0.smithi063.stdout:0/415: stat d2/d14/d36/d77/f7c 0 2024-03-23T11:47:35.197 INFO:tasks.workunit.client.0.smithi063.stdout:7/340: dwrite d5/d6/f27 [0,4194304] 0 2024-03-23T11:47:35.200 INFO:tasks.workunit.client.0.smithi063.stdout:0/416: write d2/d14/d36/d77/f7c [490829,39524] 0 2024-03-23T11:47:35.203 INFO:tasks.workunit.client.0.smithi063.stdout:0/417: truncate d2/d14/f87 988041 0 2024-03-23T11:47:35.203 INFO:tasks.workunit.client.1.smithi111.stdout:1/468: unlink d0/d2b/l2d 0 2024-03-23T11:47:35.208 INFO:tasks.workunit.client.0.smithi063.stdout:3/519: symlink d13/d22/d2d/d7d/la0 0 2024-03-23T11:47:35.208 INFO:tasks.workunit.client.1.smithi111.stdout:4/471: dwrite d0/dc/d4c/d4b/d4d/f5a [0,4194304] 0 2024-03-23T11:47:35.209 INFO:tasks.workunit.client.0.smithi063.stdout:9/412: link d4/d4c/d19/f22 d4/d4c/f8c 0 2024-03-23T11:47:35.209 INFO:tasks.workunit.client.1.smithi111.stdout:7/454: creat d14/d44/d70/f7c x:0 0 0 2024-03-23T11:47:35.211 INFO:tasks.workunit.client.0.smithi063.stdout:5/287: getdents d2/d17 0 2024-03-23T11:47:35.212 INFO:tasks.workunit.client.0.smithi063.stdout:7/341: fdatasync d5/d6/d39/f62 0 2024-03-23T11:47:35.215 INFO:tasks.workunit.client.0.smithi063.stdout:8/649: creat d6/dd/fde x:0 0 0 2024-03-23T11:47:35.215 INFO:tasks.workunit.client.1.smithi111.stdout:7/455: dread d14/d1b/d1e/d2c/d42/d5e/f6c [0,4194304] 0 2024-03-23T11:47:35.218 INFO:tasks.workunit.client.1.smithi111.stdout:4/472: creat d0/dc/d4c/d2a/f8b x:0 0 0 2024-03-23T11:47:35.219 INFO:tasks.workunit.client.0.smithi063.stdout:8/650: write d6/f43 [3000483,128258] 0 2024-03-23T11:47:35.219 INFO:tasks.workunit.client.0.smithi063.stdout:1/320: link d3/d49/l4a d3/l6f 0 2024-03-23T11:47:35.224 INFO:tasks.workunit.client.0.smithi063.stdout:2/360: getdents d2/de 0 2024-03-23T11:47:35.226 INFO:tasks.workunit.client.0.smithi063.stdout:0/418: symlink d2/d4/l99 0 2024-03-23T11:47:35.230 INFO:tasks.workunit.client.1.smithi111.stdout:2/550: creat d2/d11/d17/d3e/d6b/d2e/fca x:0 0 0 2024-03-23T11:47:35.230 INFO:tasks.workunit.client.0.smithi063.stdout:7/342: mkdir d5/d6/d19/d1f/d47/d4f/d5b/d6f 0 2024-03-23T11:47:35.232 INFO:tasks.workunit.client.1.smithi111.stdout:4/473: dwrite d0/d7/f1b [4194304,4194304] 0 2024-03-23T11:47:35.235 INFO:tasks.workunit.client.0.smithi063.stdout:7/343: write d5/d6/d19/d1f/d37/f3c [2353948,88579] 0 2024-03-23T11:47:35.237 INFO:tasks.workunit.client.0.smithi063.stdout:5/288: mkdir d2/d7/d64 0 2024-03-23T11:47:35.239 INFO:tasks.workunit.client.0.smithi063.stdout:3/520: rename d13/d22/d2d/d3f/d67/f6d to d13/d22/d2d/fa1 0 2024-03-23T11:47:35.243 INFO:tasks.workunit.client.0.smithi063.stdout:2/361: fsync d2/d23/d2e/d52/f3e 0 2024-03-23T11:47:35.244 INFO:tasks.workunit.client.0.smithi063.stdout:6/344: getdents d1/dc/d21/d49 0 2024-03-23T11:47:35.244 INFO:tasks.workunit.client.0.smithi063.stdout:8/651: dwrite d6/d1c/d27/f4a [0,4194304] 0 2024-03-23T11:47:35.246 INFO:tasks.workunit.client.1.smithi111.stdout:5/524: link d1/f17 d1/d3/fb0 0 2024-03-23T11:47:35.247 INFO:tasks.workunit.client.0.smithi063.stdout:6/345: dread - d1/d15/d4b/f4c zero size 2024-03-23T11:47:35.251 INFO:tasks.workunit.client.0.smithi063.stdout:7/344: mknod d5/d26/d32/c70 0 2024-03-23T11:47:35.253 INFO:tasks.workunit.client.1.smithi111.stdout:4/474: creat d0/d17/f8c x:0 0 0 2024-03-23T11:47:35.253 INFO:tasks.workunit.client.0.smithi063.stdout:0/419: rename d2/d48/c84 to d2/d4/dd/d1f/c9a 0 2024-03-23T11:47:35.254 INFO:tasks.workunit.client.0.smithi063.stdout:5/289: readlink d2/d7/d8/l29 0 2024-03-23T11:47:35.254 INFO:tasks.workunit.client.1.smithi111.stdout:7/456: link d14/d44/d4b/c66 d14/d1b/d20/d5f/c7d 0 2024-03-23T11:47:35.255 INFO:tasks.workunit.client.0.smithi063.stdout:3/521: rename d13/d22/d2d/d3f/d58 to d13/d22/d2d/d3f/d58/d86/da2 22 2024-03-23T11:47:35.256 INFO:tasks.workunit.client.0.smithi063.stdout:5/290: dread - d2/f52 zero size 2024-03-23T11:47:35.259 INFO:tasks.workunit.client.1.smithi111.stdout:9/520: dread d0/d4/d1c/f45 [0,4194304] 0 2024-03-23T11:47:35.260 INFO:tasks.workunit.client.0.smithi063.stdout:5/291: fdatasync d2/d28/d3a/f4e 0 2024-03-23T11:47:35.261 INFO:tasks.workunit.client.0.smithi063.stdout:5/292: stat d2/d7/d8/l11 0 2024-03-23T11:47:35.264 INFO:tasks.workunit.client.1.smithi111.stdout:4/475: creat d0/dc/d4c/d4b/d4d/f8d x:0 0 0 2024-03-23T11:47:35.265 INFO:tasks.workunit.client.0.smithi063.stdout:5/293: read d2/d7/d8/f13 [681883,30326] 0 2024-03-23T11:47:35.270 INFO:tasks.workunit.client.0.smithi063.stdout:9/413: getdents d4/d4c/d44 0 2024-03-23T11:47:35.271 INFO:tasks.workunit.client.0.smithi063.stdout:3/522: rmdir d13/d15/d48 39 2024-03-23T11:47:35.272 INFO:tasks.workunit.client.1.smithi111.stdout:3/411: sync 2024-03-23T11:47:35.272 INFO:tasks.workunit.client.0.smithi063.stdout:9/414: write d4/d4c/d19/f2d [3852872,86356] 0 2024-03-23T11:47:35.273 INFO:tasks.workunit.client.0.smithi063.stdout:6/346: mknod d1/dc/c64 0 2024-03-23T11:47:35.273 INFO:tasks.workunit.client.1.smithi111.stdout:5/525: symlink d1/d3/d6/d8b/d59/d74/lb1 0 2024-03-23T11:47:35.273 INFO:tasks.workunit.client.1.smithi111.stdout:4/476: mknod d0/dc/d1f/c8e 0 2024-03-23T11:47:35.275 INFO:tasks.workunit.client.1.smithi111.stdout:3/412: rmdir d0/d1a 39 2024-03-23T11:47:35.275 INFO:tasks.workunit.client.0.smithi063.stdout:9/415: chown d4/f5 1036110 1 2024-03-23T11:47:35.276 INFO:tasks.workunit.client.1.smithi111.stdout:5/526: stat d1/d3/d6/d7/d63/d9d 0 2024-03-23T11:47:35.276 INFO:tasks.workunit.client.0.smithi063.stdout:2/362: write d2/d23/f41 [1012315,58687] 0 2024-03-23T11:47:35.277 INFO:tasks.workunit.client.1.smithi111.stdout:1/469: dread d0/d2b/f75 [0,4194304] 0 2024-03-23T11:47:35.279 INFO:tasks.workunit.client.1.smithi111.stdout:4/477: write d0/dc/d4c/d4b/d4f/f6a [182448,130418] 0 2024-03-23T11:47:35.279 INFO:tasks.workunit.client.1.smithi111.stdout:6/514: write d2/fd [4246466,83336] 0 2024-03-23T11:47:35.279 INFO:tasks.workunit.client.1.smithi111.stdout:3/413: read d0/f39 [269372,92023] 0 2024-03-23T11:47:35.281 INFO:tasks.workunit.client.1.smithi111.stdout:5/527: fsync d1/d3/d6/d7/d8c/f8e 0 2024-03-23T11:47:35.281 INFO:tasks.workunit.client.1.smithi111.stdout:6/515: chown d2/d73 302398 1 2024-03-23T11:47:35.282 INFO:tasks.workunit.client.0.smithi063.stdout:2/363: stat d2/f21 0 2024-03-23T11:47:35.284 INFO:tasks.workunit.client.0.smithi063.stdout:3/523: creat d13/d22/d2d/d3f/d67/fa3 x:0 0 0 2024-03-23T11:47:35.285 INFO:tasks.workunit.client.0.smithi063.stdout:6/347: rename d1/ce to d1/d2a/c65 0 2024-03-23T11:47:35.288 INFO:tasks.workunit.client.1.smithi111.stdout:1/470: creat d0/d5/dd/d1e/fa0 x:0 0 0 2024-03-23T11:47:35.289 INFO:tasks.workunit.client.0.smithi063.stdout:7/345: creat d5/d6/d19/f71 x:0 0 0 2024-03-23T11:47:35.291 INFO:tasks.workunit.client.1.smithi111.stdout:6/516: creat d2/d1f/d45/d86/faa x:0 0 0 2024-03-23T11:47:35.292 INFO:tasks.workunit.client.1.smithi111.stdout:6/517: dread - d2/d1f/fa6 zero size 2024-03-23T11:47:35.294 INFO:tasks.workunit.client.0.smithi063.stdout:2/364: dread d2/de/f15 [0,4194304] 0 2024-03-23T11:47:35.294 INFO:tasks.workunit.client.0.smithi063.stdout:8/652: dwrite d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:35.294 INFO:tasks.workunit.client.0.smithi063.stdout:0/420: getdents d2/d48 0 2024-03-23T11:47:35.296 INFO:tasks.workunit.client.0.smithi063.stdout:7/346: creat d5/d26/f72 x:0 0 0 2024-03-23T11:47:35.297 INFO:tasks.workunit.client.1.smithi111.stdout:4/478: symlink d0/dc/d30/l8f 0 2024-03-23T11:47:35.299 INFO:tasks.workunit.client.1.smithi111.stdout:7/457: truncate d14/f36 438263 0 2024-03-23T11:47:35.303 INFO:tasks.workunit.client.1.smithi111.stdout:5/528: rename d1/d3/d6/d7/d63/d2c/d82/dab to d1/d3/d6/d30/db2 0 2024-03-23T11:47:35.304 INFO:tasks.workunit.client.1.smithi111.stdout:7/458: mkdir d14/d44/d70/d7e 0 2024-03-23T11:47:35.305 INFO:tasks.workunit.client.0.smithi063.stdout:2/365: fdatasync d2/d23/d2e/f55 0 2024-03-23T11:47:35.306 INFO:tasks.workunit.client.1.smithi111.stdout:7/459: chown d14/d37/d50/f62 163 1 2024-03-23T11:47:35.308 INFO:tasks.workunit.client.1.smithi111.stdout:3/414: rename d0/l69 to d0/d1a/d2d/l83 0 2024-03-23T11:47:35.308 INFO:tasks.workunit.client.0.smithi063.stdout:9/416: dwrite d4/d4c/d19/f23 [0,4194304] 0 2024-03-23T11:47:35.311 INFO:tasks.workunit.client.1.smithi111.stdout:3/415: chown d0/d7/d19/l42 14050634 1 2024-03-23T11:47:35.313 INFO:tasks.workunit.client.1.smithi111.stdout:1/471: rename d0/d6/d48/f52 to d0/d5/dd/d91/fa1 0 2024-03-23T11:47:35.314 INFO:tasks.workunit.client.0.smithi063.stdout:0/421: creat d2/d4/dd/d1f/f9b x:0 0 0 2024-03-23T11:47:35.315 INFO:tasks.workunit.client.0.smithi063.stdout:8/653: unlink d6/d1f/d54/d5c/d76/d85/fd8 0 2024-03-23T11:47:35.318 INFO:tasks.workunit.client.1.smithi111.stdout:9/521: getdents d0/d4/d1c/d2d/d6e 0 2024-03-23T11:47:35.318 INFO:tasks.workunit.client.0.smithi063.stdout:7/347: rmdir d5/d6/d19/d1f/d47/d4f/d5b 39 2024-03-23T11:47:35.320 INFO:tasks.workunit.client.0.smithi063.stdout:3/524: dwrite f12 [4194304,4194304] 0 2024-03-23T11:47:35.324 INFO:tasks.workunit.client.1.smithi111.stdout:5/529: dread d1/d3/d6/f2b [0,4194304] 0 2024-03-23T11:47:35.327 INFO:tasks.workunit.client.1.smithi111.stdout:5/530: chown d1/d3/d6/d8b/d59/f6a 107 1 2024-03-23T11:47:35.331 INFO:tasks.workunit.client.1.smithi111.stdout:1/472: mknod d0/d6/ca2 0 2024-03-23T11:47:35.332 INFO:tasks.workunit.client.0.smithi063.stdout:6/348: creat d1/d15/d4b/f66 x:0 0 0 2024-03-23T11:47:35.332 INFO:tasks.workunit.client.0.smithi063.stdout:0/422: creat d2/d12/d32/d6d/f9c x:0 0 0 2024-03-23T11:47:35.334 INFO:tasks.workunit.client.1.smithi111.stdout:4/479: dread d0/dc/d4c/d71/f84 [4194304,4194304] 0 2024-03-23T11:47:35.335 INFO:tasks.workunit.client.1.smithi111.stdout:6/518: truncate d2/fd 2834871 0 2024-03-23T11:47:35.336 INFO:tasks.workunit.client.0.smithi063.stdout:3/525: symlink d13/d22/d2d/d78/la4 0 2024-03-23T11:47:35.337 INFO:tasks.workunit.client.1.smithi111.stdout:6/519: truncate d2/d1f/d45/f62 4766119 0 2024-03-23T11:47:35.339 INFO:tasks.workunit.client.1.smithi111.stdout:3/416: mkdir d0/d7/d19/d84 0 2024-03-23T11:47:35.341 INFO:tasks.workunit.client.0.smithi063.stdout:7/348: rename d5/d6/d19/d1f/d37 to d5/d5c/d2f/d73 0 2024-03-23T11:47:35.344 INFO:tasks.workunit.client.0.smithi063.stdout:0/423: creat d2/d4/dd/d28/d46/d51/f9d x:0 0 0 2024-03-23T11:47:35.346 INFO:tasks.workunit.client.1.smithi111.stdout:7/460: link d14/c26 d14/d1b/d20/d5f/c7f 0 2024-03-23T11:47:35.346 INFO:tasks.workunit.client.0.smithi063.stdout:9/417: getdents d4/d4c/d44 0 2024-03-23T11:47:35.347 INFO:tasks.workunit.client.1.smithi111.stdout:6/520: creat d2/d1a/d1c/d2c/fab x:0 0 0 2024-03-23T11:47:35.350 INFO:tasks.workunit.client.0.smithi063.stdout:8/654: dwrite d6/f89 [0,4194304] 0 2024-03-23T11:47:35.351 INFO:tasks.workunit.client.1.smithi111.stdout:5/531: symlink d1/d3/d6/d7/d8c/d9e/lb3 0 2024-03-23T11:47:35.354 INFO:tasks.workunit.client.1.smithi111.stdout:9/522: creat d0/d4/d10/d29/fac x:0 0 0 2024-03-23T11:47:35.354 INFO:tasks.workunit.client.1.smithi111.stdout:8/398: sync 2024-03-23T11:47:35.355 INFO:tasks.workunit.client.1.smithi111.stdout:6/521: rename d2/d1a/f1d to d2/d17/fac 0 2024-03-23T11:47:35.358 INFO:tasks.workunit.client.1.smithi111.stdout:6/522: write d2/d1f/f7a [740072,4030] 0 2024-03-23T11:47:35.359 INFO:tasks.workunit.client.1.smithi111.stdout:1/473: creat d0/d2b/d7c/d8b/fa3 x:0 0 0 2024-03-23T11:47:35.359 INFO:tasks.workunit.client.0.smithi063.stdout:7/349: fdatasync d5/d22/f44 0 2024-03-23T11:47:35.359 INFO:tasks.workunit.client.0.smithi063.stdout:3/526: rmdir d13 39 2024-03-23T11:47:35.361 INFO:tasks.workunit.client.0.smithi063.stdout:9/418: rmdir d4/d31/d43/d49 39 2024-03-23T11:47:35.362 INFO:tasks.workunit.client.1.smithi111.stdout:7/461: mknod d14/d1b/c80 0 2024-03-23T11:47:35.363 INFO:tasks.workunit.client.1.smithi111.stdout:7/462: chown d14/d37/f49 291 1 2024-03-23T11:47:35.365 INFO:tasks.workunit.client.0.smithi063.stdout:7/350: creat d5/d26/f74 x:0 0 0 2024-03-23T11:47:35.366 INFO:tasks.workunit.client.1.smithi111.stdout:5/532: creat d1/d20/d41/fb4 x:0 0 0 2024-03-23T11:47:35.366 INFO:tasks.workunit.client.1.smithi111.stdout:4/480: dread d0/dc/f55 [0,4194304] 0 2024-03-23T11:47:35.366 INFO:tasks.workunit.client.1.smithi111.stdout:5/533: chown d1/d20/f61 33479766 1 2024-03-23T11:47:35.366 INFO:tasks.workunit.client.0.smithi063.stdout:6/349: rename d1/dc/d21/f2f to d1/f67 0 2024-03-23T11:47:35.367 INFO:tasks.workunit.client.1.smithi111.stdout:9/523: unlink d0/d4/d1c/d2d/d3b/d42/d4f/d95/l6c 0 2024-03-23T11:47:35.367 INFO:tasks.workunit.client.0.smithi063.stdout:8/655: rmdir d6/d1f/d54/d63/d90/ddd 0 2024-03-23T11:47:35.368 INFO:tasks.workunit.client.1.smithi111.stdout:5/534: write d1/d3/d6/d7/d63/fac [466205,73850] 0 2024-03-23T11:47:35.368 INFO:tasks.workunit.client.0.smithi063.stdout:9/419: mkdir d4/d31/d70/d8d 0 2024-03-23T11:47:35.369 INFO:tasks.workunit.client.1.smithi111.stdout:1/474: rmdir d0/d6/d48/d4a/d6c/d96 39 2024-03-23T11:47:35.369 INFO:tasks.workunit.client.1.smithi111.stdout:9/524: dread - d0/d4/d10/d29/d7f/f8a zero size 2024-03-23T11:47:35.377 INFO:tasks.workunit.client.1.smithi111.stdout:8/399: symlink d5/d26/d31/d5e/d4a/l82 0 2024-03-23T11:47:35.379 INFO:tasks.workunit.client.1.smithi111.stdout:8/400: write d5/d26/d2f/d62/d75/f6b [101750,71097] 0 2024-03-23T11:47:35.379 INFO:tasks.workunit.client.0.smithi063.stdout:6/350: mknod d1/c68 0 2024-03-23T11:47:35.381 INFO:tasks.workunit.client.1.smithi111.stdout:8/401: truncate d5/d26/f74 4813612 0 2024-03-23T11:47:35.386 INFO:tasks.workunit.client.1.smithi111.stdout:3/417: write d0/d1a/f1c [934595,65748] 0 2024-03-23T11:47:35.388 INFO:tasks.workunit.client.0.smithi063.stdout:0/424: dwrite d2/d4/dd/d28/d46/d51/f65 [0,4194304] 0 2024-03-23T11:47:35.389 INFO:tasks.workunit.client.1.smithi111.stdout:3/418: write d0/d1a/d2b/d38/f3c [2481136,10807] 0 2024-03-23T11:47:35.390 INFO:tasks.workunit.client.0.smithi063.stdout:0/425: readlink d2/d4/l99 0 2024-03-23T11:47:35.391 INFO:tasks.workunit.client.0.smithi063.stdout:3/527: dread d13/f50 [4194304,4194304] 0 2024-03-23T11:47:35.395 INFO:tasks.workunit.client.1.smithi111.stdout:5/535: dwrite d1/d20/f90 [0,4194304] 0 2024-03-23T11:47:35.397 INFO:tasks.workunit.client.1.smithi111.stdout:1/475: truncate d0/d5/dd/f10 1945310 0 2024-03-23T11:47:35.398 INFO:tasks.workunit.client.1.smithi111.stdout:2/551: sync 2024-03-23T11:47:35.398 INFO:tasks.workunit.client.1.smithi111.stdout:0/462: sync 2024-03-23T11:47:35.399 INFO:tasks.workunit.client.1.smithi111.stdout:4/481: write d0/dc/d4c/d2a/f40 [2295861,106652] 0 2024-03-23T11:47:35.402 INFO:tasks.workunit.client.0.smithi063.stdout:9/420: mkdir d4/d31/d43/d49/d8e 0 2024-03-23T11:47:35.403 INFO:tasks.workunit.client.0.smithi063.stdout:7/351: rename d5/d6/d19/d1f/d47/f69 to d5/d6/d19/d1f/d47/d4f/d5b/f75 0 2024-03-23T11:47:35.403 INFO:tasks.workunit.client.0.smithi063.stdout:8/656: creat d6/dd/d59/daf/dba/fdf x:0 0 0 2024-03-23T11:47:35.404 INFO:tasks.workunit.client.1.smithi111.stdout:1/476: truncate d0/d6/f34 4119806 0 2024-03-23T11:47:35.405 INFO:tasks.workunit.client.1.smithi111.stdout:0/463: write d5/db/f16 [3504271,9214] 0 2024-03-23T11:47:35.407 INFO:tasks.workunit.client.1.smithi111.stdout:7/463: write d14/d37/f3f [1300133,113216] 0 2024-03-23T11:47:35.409 INFO:tasks.workunit.client.1.smithi111.stdout:4/482: symlink d0/dc/d78/l90 0 2024-03-23T11:47:35.409 INFO:tasks.workunit.client.1.smithi111.stdout:5/536: write d1/d3/d6/d7/d63/d2c/d82/f88 [887050,65135] 0 2024-03-23T11:47:35.410 INFO:tasks.workunit.client.1.smithi111.stdout:6/523: truncate d2/d1a/d1c/d2c/f48 806872 0 2024-03-23T11:47:35.412 INFO:tasks.workunit.client.1.smithi111.stdout:0/464: chown d5/db/d1f/d39/f8f 9 1 2024-03-23T11:47:35.414 INFO:tasks.workunit.client.1.smithi111.stdout:9/525: dwrite d0/d4/d10/f74 [0,4194304] 0 2024-03-23T11:47:35.416 INFO:tasks.workunit.client.0.smithi063.stdout:7/352: rmdir d5/d5c 39 2024-03-23T11:47:35.417 INFO:tasks.workunit.client.1.smithi111.stdout:9/526: readlink d0/d4/d1c/d2d/d3b/l3c 0 2024-03-23T11:47:35.418 INFO:tasks.workunit.client.1.smithi111.stdout:8/402: truncate d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f7f 1492136 0 2024-03-23T11:47:35.420 INFO:tasks.workunit.client.1.smithi111.stdout:7/464: mknod d14/c81 0 2024-03-23T11:47:35.423 INFO:tasks.workunit.client.1.smithi111.stdout:3/419: getdents d0/d1a/d1d 0 2024-03-23T11:47:35.423 INFO:tasks.workunit.client.0.smithi063.stdout:9/421: dread d4/f2a [0,4194304] 0 2024-03-23T11:47:35.425 INFO:tasks.workunit.client.0.smithi063.stdout:8/657: symlink d6/d1f/d54/d5c/d76/le0 0 2024-03-23T11:47:35.426 INFO:tasks.workunit.client.1.smithi111.stdout:6/524: creat d2/d1a/fad x:0 0 0 2024-03-23T11:47:35.427 INFO:tasks.workunit.client.0.smithi063.stdout:9/422: read - d4/d31/d43/d49/f78 zero size 2024-03-23T11:47:35.428 INFO:tasks.workunit.client.1.smithi111.stdout:5/537: symlink d1/d3/d6/d7/d86/lb5 0 2024-03-23T11:47:35.431 INFO:tasks.workunit.client.1.smithi111.stdout:1/477: dwrite d0/d5/dd/d1e/d54/f84 [0,4194304] 0 2024-03-23T11:47:35.431 INFO:tasks.workunit.client.1.smithi111.stdout:2/552: creat d2/d7/da/d19/fcb x:0 0 0 2024-03-23T11:47:35.435 INFO:tasks.workunit.client.1.smithi111.stdout:8/403: unlink d5/d26/d31/d3b/d41/f66 0 2024-03-23T11:47:35.440 INFO:tasks.workunit.client.1.smithi111.stdout:4/483: symlink d0/dc/d4c/l91 0 2024-03-23T11:47:35.441 INFO:tasks.workunit.client.0.smithi063.stdout:3/528: dread d13/d22/d2d/f45 [0,4194304] 0 2024-03-23T11:47:35.441 INFO:tasks.workunit.client.0.smithi063.stdout:0/426: dread d2/d14/f15 [0,4194304] 0 2024-03-23T11:47:35.442 INFO:tasks.workunit.client.1.smithi111.stdout:4/484: chown d0/dc/d4c/d2a/c48 583 1 2024-03-23T11:47:35.445 INFO:tasks.workunit.client.0.smithi063.stdout:0/427: chown d2/d14/f23 165095194 1 2024-03-23T11:47:35.447 INFO:tasks.workunit.client.0.smithi063.stdout:9/423: symlink d4/d2c/l8f 0 2024-03-23T11:47:35.448 INFO:tasks.workunit.client.0.smithi063.stdout:6/351: rename d1/d15/c17 to d1/d15/c69 0 2024-03-23T11:47:35.449 INFO:tasks.workunit.client.1.smithi111.stdout:3/420: mknod d0/d7/d19/d5e/c85 0 2024-03-23T11:47:35.454 INFO:tasks.workunit.client.0.smithi063.stdout:8/658: rename d6/d1f/d54/la7 to d6/dd/d59/db2/dbe/le1 0 2024-03-23T11:47:35.456 INFO:tasks.workunit.client.0.smithi063.stdout:7/353: chown d5/d5c/d2f/d73/da/l11 2 1 2024-03-23T11:47:35.456 INFO:tasks.workunit.client.0.smithi063.stdout:3/529: creat d13/d22/d2d/d3f/d58/d6e/fa5 x:0 0 0 2024-03-23T11:47:35.457 INFO:tasks.workunit.client.1.smithi111.stdout:4/485: dwrite d0/dc/f14 [0,4194304] 0 2024-03-23T11:47:35.460 INFO:tasks.workunit.client.0.smithi063.stdout:9/424: mkdir d4/d31/d70/d8d/d90 0 2024-03-23T11:47:35.461 INFO:tasks.workunit.client.1.smithi111.stdout:1/478: mknod d0/d5/dd/d1e/d54/d87/ca4 0 2024-03-23T11:47:35.462 INFO:tasks.workunit.client.0.smithi063.stdout:7/354: mkdir d5/d26/d76 0 2024-03-23T11:47:35.462 INFO:tasks.workunit.client.0.smithi063.stdout:3/530: chown d13/d22/d2d/l35 14684219 1 2024-03-23T11:47:35.464 INFO:tasks.workunit.client.1.smithi111.stdout:8/404: mknod d5/d26/d2f/d42/d63/c83 0 2024-03-23T11:47:35.465 INFO:tasks.workunit.client.1.smithi111.stdout:2/553: dread d2/d11/d17/d3e/d6b/d2e/d3f/f4d [0,4194304] 0 2024-03-23T11:47:35.466 INFO:tasks.workunit.client.0.smithi063.stdout:7/355: stat d5/d22/f24 0 2024-03-23T11:47:35.467 INFO:tasks.workunit.client.0.smithi063.stdout:9/425: write d4/d31/f59 [356635,9190] 0 2024-03-23T11:47:35.467 INFO:tasks.workunit.client.0.smithi063.stdout:8/659: fsync d6/d1c/f6c 0 2024-03-23T11:47:35.470 INFO:tasks.workunit.client.1.smithi111.stdout:3/421: creat d0/d72/f86 x:0 0 0 2024-03-23T11:47:35.475 INFO:tasks.workunit.client.1.smithi111.stdout:0/465: dwrite d5/f2d [0,4194304] 0 2024-03-23T11:47:35.475 INFO:tasks.workunit.client.0.smithi063.stdout:9/426: chown d4/c56 10820849 1 2024-03-23T11:47:35.485 INFO:tasks.workunit.client.0.smithi063.stdout:6/352: dwrite d1/d15/d4b/f4c [0,4194304] 0 2024-03-23T11:47:35.487 INFO:tasks.workunit.client.1.smithi111.stdout:8/405: rmdir d5/d26/d2f/d62/d75/d4b/d51/d5a 39 2024-03-23T11:47:35.487 INFO:tasks.workunit.client.0.smithi063.stdout:0/428: dwrite d2/d12/f50 [0,4194304] 0 2024-03-23T11:47:35.488 INFO:tasks.workunit.client.1.smithi111.stdout:2/554: dread d2/d7/da/d19/f33 [0,4194304] 0 2024-03-23T11:47:35.489 INFO:tasks.workunit.client.1.smithi111.stdout:7/465: dwrite d14/d1b/d1e/f59 [0,4194304] 0 2024-03-23T11:47:35.490 INFO:tasks.workunit.client.1.smithi111.stdout:0/466: dread d5/db/d1f/d19/f23 [0,4194304] 0 2024-03-23T11:47:35.494 INFO:tasks.workunit.client.1.smithi111.stdout:6/525: creat d2/d1f/d45/d49/d8d/fae x:0 0 0 2024-03-23T11:47:35.494 INFO:tasks.workunit.client.1.smithi111.stdout:9/527: link d0/d4/l25 d0/d31/d41/d71/d8f/lad 0 2024-03-23T11:47:35.496 INFO:tasks.workunit.client.1.smithi111.stdout:1/479: fdatasync d0/d2b/f2e 0 2024-03-23T11:47:35.497 INFO:tasks.workunit.client.0.smithi063.stdout:8/660: creat d6/dd/d59/db2/fe2 x:0 0 0 2024-03-23T11:47:35.502 INFO:tasks.workunit.client.0.smithi063.stdout:3/531: dwrite d13/d15/d48/f68 [0,4194304] 0 2024-03-23T11:47:35.502 INFO:tasks.workunit.client.1.smithi111.stdout:3/422: mknod d0/d1a/c87 0 2024-03-23T11:47:35.505 INFO:tasks.workunit.client.0.smithi063.stdout:7/356: dwrite d5/d26/f72 [0,4194304] 0 2024-03-23T11:47:35.505 INFO:tasks.workunit.client.1.smithi111.stdout:7/466: chown d14/d44/d70/f7c 9377 1 2024-03-23T11:47:35.506 INFO:tasks.workunit.client.0.smithi063.stdout:0/429: rmdir d2/d4/dd/d28 39 2024-03-23T11:47:35.514 INFO:tasks.workunit.client.0.smithi063.stdout:3/532: creat d13/d15/d48/fa6 x:0 0 0 2024-03-23T11:47:35.518 INFO:tasks.workunit.client.0.smithi063.stdout:6/353: creat d1/dc/d21/d49/f6a x:0 0 0 2024-03-23T11:47:35.518 INFO:tasks.workunit.client.0.smithi063.stdout:7/357: unlink d5/d6/d19/d1f/d47/d4f/d5b/l5d 0 2024-03-23T11:47:35.521 INFO:tasks.workunit.client.0.smithi063.stdout:9/427: write d4/f47 [739596,130593] 0 2024-03-23T11:47:35.529 INFO:tasks.workunit.client.1.smithi111.stdout:6/526: dwrite d2/d1a/d1c/d2c/d51/f8f [0,4194304] 0 2024-03-23T11:47:35.532 INFO:tasks.workunit.client.1.smithi111.stdout:8/406: dread d5/f1f [0,4194304] 0 2024-03-23T11:47:35.532 INFO:tasks.workunit.client.0.smithi063.stdout:6/354: creat d1/d15/d4b/f6b x:0 0 0 2024-03-23T11:47:35.533 INFO:tasks.workunit.client.1.smithi111.stdout:1/480: mknod d0/d5/dd/d1e/d54/d87/d8e/ca5 0 2024-03-23T11:47:35.533 INFO:tasks.workunit.client.0.smithi063.stdout:8/661: creat d6/fe3 x:0 0 0 2024-03-23T11:47:35.534 INFO:tasks.workunit.client.1.smithi111.stdout:4/486: getdents d0/d7 0 2024-03-23T11:47:35.534 INFO:tasks.workunit.client.1.smithi111.stdout:3/423: mknod d0/d7/d19/d84/c88 0 2024-03-23T11:47:35.534 INFO:tasks.workunit.client.0.smithi063.stdout:9/428: mknod d4/d31/d43/d49/c91 0 2024-03-23T11:47:35.535 INFO:tasks.workunit.client.1.smithi111.stdout:8/407: readlink d5/l54 0 2024-03-23T11:47:35.536 INFO:tasks.workunit.client.0.smithi063.stdout:0/430: getdents d2/d4/dd/d25 0 2024-03-23T11:47:35.536 INFO:tasks.workunit.client.1.smithi111.stdout:6/527: chown d2/d1a/f22 2732 1 2024-03-23T11:47:35.537 INFO:tasks.workunit.client.1.smithi111.stdout:7/467: symlink d14/d44/d70/d7e/l82 0 2024-03-23T11:47:35.537 INFO:tasks.workunit.client.1.smithi111.stdout:4/487: stat d0/d7/c10 0 2024-03-23T11:47:35.539 INFO:tasks.workunit.client.0.smithi063.stdout:7/358: getdents d5/d6 0 2024-03-23T11:47:35.540 INFO:tasks.workunit.client.0.smithi063.stdout:6/355: symlink d1/dc/d21/l6c 0 2024-03-23T11:47:35.542 INFO:tasks.workunit.client.1.smithi111.stdout:5/538: sync 2024-03-23T11:47:35.542 INFO:tasks.workunit.client.1.smithi111.stdout:2/555: dread d2/d11/f1b [0,4194304] 0 2024-03-23T11:47:35.542 INFO:tasks.workunit.client.1.smithi111.stdout:1/481: dread d0/d5/dd/f6e [0,4194304] 0 2024-03-23T11:47:35.543 INFO:tasks.workunit.client.1.smithi111.stdout:3/424: chown d0/d7/f2a 247 1 2024-03-23T11:47:35.546 INFO:tasks.workunit.client.1.smithi111.stdout:6/528: creat d2/d1f/faf x:0 0 0 2024-03-23T11:47:35.548 INFO:tasks.workunit.client.1.smithi111.stdout:6/529: dread - d2/d1a/d1c/d2c/fab zero size 2024-03-23T11:47:35.548 INFO:tasks.workunit.client.1.smithi111.stdout:1/482: creat d0/d5/dd/d1e/d54/d87/fa6 x:0 0 0 2024-03-23T11:47:35.551 INFO:tasks.workunit.client.1.smithi111.stdout:7/468: rename d14/d1b/d22/l4e to d14/d1b/d1e/d2c/d42/l83 0 2024-03-23T11:47:35.551 INFO:tasks.workunit.client.0.smithi063.stdout:6/356: write d1/dc/d21/d49/f6a [825608,117564] 0 2024-03-23T11:47:35.557 INFO:tasks.workunit.client.0.smithi063.stdout:9/429: rename d4/d4c/d19/l29 to d4/d4c/d44/d64/l92 0 2024-03-23T11:47:35.557 INFO:tasks.workunit.client.1.smithi111.stdout:8/408: mknod d5/d32/c84 0 2024-03-23T11:47:35.558 INFO:tasks.workunit.client.1.smithi111.stdout:4/488: dwrite d0/d17/f2c [0,4194304] 0 2024-03-23T11:47:35.560 INFO:tasks.workunit.client.1.smithi111.stdout:1/483: symlink d0/d2b/d7c/la7 0 2024-03-23T11:47:35.560 INFO:tasks.workunit.client.1.smithi111.stdout:8/409: chown d5/d26/f74 31499 1 2024-03-23T11:47:35.561 INFO:tasks.workunit.client.0.smithi063.stdout:0/431: dread d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:35.561 INFO:tasks.workunit.client.0.smithi063.stdout:3/533: dwrite d13/d22/d2d/d3f/d5b/f77 [0,4194304] 0 2024-03-23T11:47:35.563 INFO:tasks.workunit.client.0.smithi063.stdout:6/357: read d1/f8 [2318789,7168] 0 2024-03-23T11:47:35.563 INFO:tasks.workunit.client.1.smithi111.stdout:6/530: creat d2/d1f/d45/d9a/fb0 x:0 0 0 2024-03-23T11:47:35.565 INFO:tasks.workunit.client.1.smithi111.stdout:4/489: creat d0/d7/f92 x:0 0 0 2024-03-23T11:47:35.565 INFO:tasks.workunit.client.0.smithi063.stdout:7/359: link d5/d6/d19/d1f/l68 d5/d6/l77 0 2024-03-23T11:47:35.567 INFO:tasks.workunit.client.0.smithi063.stdout:6/358: fdatasync d1/dc/d3a/f40 0 2024-03-23T11:47:35.568 INFO:tasks.workunit.client.1.smithi111.stdout:8/410: write d5/d26/d31/d5e/d4a/f52 [2930340,114344] 0 2024-03-23T11:47:35.570 INFO:tasks.workunit.client.1.smithi111.stdout:9/528: dread d0/d4/d1c/f46 [0,4194304] 0 2024-03-23T11:47:35.570 INFO:tasks.workunit.client.1.smithi111.stdout:8/411: chown d5/d26/d2f/d62/d75/f6b 536162771 1 2024-03-23T11:47:35.571 INFO:tasks.workunit.client.0.smithi063.stdout:6/359: write d1/d15/d4b/f4c [3703724,18385] 0 2024-03-23T11:47:35.572 INFO:tasks.workunit.client.0.smithi063.stdout:0/432: creat d2/d6e/d8c/f9e x:0 0 0 2024-03-23T11:47:35.572 INFO:tasks.workunit.client.1.smithi111.stdout:8/412: fsync d5/f17 0 2024-03-23T11:47:35.574 INFO:tasks.workunit.client.1.smithi111.stdout:8/413: chown d5/d26/d31/d5e 99394 1 2024-03-23T11:47:35.578 INFO:tasks.workunit.client.0.smithi063.stdout:9/430: truncate d4/d31/f3d 70287 0 2024-03-23T11:47:35.580 INFO:tasks.workunit.client.1.smithi111.stdout:6/531: fsync d2/d1a/f2d 0 2024-03-23T11:47:35.581 INFO:tasks.workunit.client.1.smithi111.stdout:0/467: write d5/db/d1f/d32/f3e [2597251,79164] 0 2024-03-23T11:47:35.581 INFO:tasks.workunit.client.0.smithi063.stdout:9/431: write d4/d4c/d44/f7e [431392,130093] 0 2024-03-23T11:47:35.582 INFO:tasks.workunit.client.1.smithi111.stdout:3/425: link d0/d53/l75 d0/d1a/d1d/d26/d5a/l89 0 2024-03-23T11:47:35.585 INFO:tasks.workunit.client.0.smithi063.stdout:0/433: readlink d2/d14/l76 0 2024-03-23T11:47:35.587 INFO:tasks.workunit.client.1.smithi111.stdout:6/532: fdatasync d2/d1f/d45/f56 0 2024-03-23T11:47:35.589 INFO:tasks.workunit.client.1.smithi111.stdout:6/533: fdatasync d2/d17/f55 0 2024-03-23T11:47:35.589 INFO:tasks.workunit.client.1.smithi111.stdout:8/414: mkdir d5/d26/d2f/d62/d7e/d85 0 2024-03-23T11:47:35.591 INFO:tasks.workunit.client.1.smithi111.stdout:0/468: rename d5/db/d1f/d19/d45/d4b/f84 to d5/db/d1f/d39/f94 0 2024-03-23T11:47:35.592 INFO:tasks.workunit.client.1.smithi111.stdout:1/484: getdents d0/d5/dd/d1e/d70 0 2024-03-23T11:47:35.594 INFO:tasks.workunit.client.0.smithi063.stdout:8/662: write d6/d1f/d54/d63/d90/f35 [1059472,47590] 0 2024-03-23T11:47:35.595 INFO:tasks.workunit.client.0.smithi063.stdout:4/445: sync 2024-03-23T11:47:35.597 INFO:tasks.workunit.client.0.smithi063.stdout:9/432: creat d4/d31/d43/d49/d8e/f93 x:0 0 0 2024-03-23T11:47:35.598 INFO:tasks.workunit.client.1.smithi111.stdout:6/534: creat d2/d80/fb1 x:0 0 0 2024-03-23T11:47:35.600 INFO:tasks.workunit.client.0.smithi063.stdout:6/360: getdents d1/dc/d3a/d57 0 2024-03-23T11:47:35.600 INFO:tasks.workunit.client.0.smithi063.stdout:3/534: getdents d13/d22/d2d/d3f/d67 0 2024-03-23T11:47:35.601 INFO:tasks.workunit.client.0.smithi063.stdout:4/446: mkdir d5/da/d20/d4b/d16/d83 0 2024-03-23T11:47:35.602 INFO:tasks.workunit.client.1.smithi111.stdout:0/469: mknod d5/db/d1f/d48/c95 0 2024-03-23T11:47:35.602 INFO:tasks.workunit.client.1.smithi111.stdout:8/415: dread - d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 zero size 2024-03-23T11:47:35.602 INFO:tasks.workunit.client.1.smithi111.stdout:9/529: link d0/d4/d10/l1f d0/d4/d6a/d9a/lae 0 2024-03-23T11:47:35.603 INFO:tasks.workunit.client.0.smithi063.stdout:9/433: symlink d4/d31/d43/d49/l94 0 2024-03-23T11:47:35.603 INFO:tasks.workunit.client.1.smithi111.stdout:4/490: dread d0/d17/f6b [0,4194304] 0 2024-03-23T11:47:35.604 INFO:tasks.workunit.client.0.smithi063.stdout:6/361: chown d1/dc/f1f 1012411 1 2024-03-23T11:47:35.605 INFO:tasks.workunit.client.1.smithi111.stdout:6/535: creat d2/d1f/d45/d86/fb2 x:0 0 0 2024-03-23T11:47:35.606 INFO:tasks.workunit.client.0.smithi063.stdout:8/663: mkdir d6/dd/d59/d7f/de4 0 2024-03-23T11:47:35.609 INFO:tasks.workunit.client.1.smithi111.stdout:8/416: unlink d5/f17 0 2024-03-23T11:47:35.610 INFO:tasks.workunit.client.1.smithi111.stdout:4/491: write d0/d17/f8c [244899,91510] 0 2024-03-23T11:47:35.613 INFO:tasks.workunit.client.0.smithi063.stdout:6/362: mkdir d1/dc/d3a/d57/d6d 0 2024-03-23T11:47:35.615 INFO:tasks.workunit.client.1.smithi111.stdout:7/469: truncate d14/d1b/d20/f57 7392405 0 2024-03-23T11:47:35.615 INFO:tasks.workunit.client.0.smithi063.stdout:9/434: creat d4/d4c/d44/d64/f95 x:0 0 0 2024-03-23T11:47:35.616 INFO:tasks.workunit.client.1.smithi111.stdout:4/492: dread d0/dc/d4c/d2a/f70 [0,4194304] 0 2024-03-23T11:47:35.617 INFO:tasks.workunit.client.1.smithi111.stdout:7/470: fdatasync d14/d1b/f5a 0 2024-03-23T11:47:35.618 INFO:tasks.workunit.client.1.smithi111.stdout:9/530: rename d0/d4/d1c/d2d/c76 to d0/d4/d1c/caf 0 2024-03-23T11:47:35.618 INFO:tasks.workunit.client.1.smithi111.stdout:3/426: dread d0/d7/f2a [0,4194304] 0 2024-03-23T11:47:35.619 INFO:tasks.workunit.client.1.smithi111.stdout:7/471: read - d14/d1b/f53 zero size 2024-03-23T11:47:35.620 INFO:tasks.workunit.client.0.smithi063.stdout:9/435: read d4/d4c/d19/f42 [208338,22451] 0 2024-03-23T11:47:35.622 INFO:tasks.workunit.client.1.smithi111.stdout:3/427: read - d0/d1a/d1d/d26/d5a/f5f zero size 2024-03-23T11:47:35.623 INFO:tasks.workunit.client.0.smithi063.stdout:0/434: dwrite d2/d4/dd/d1f/f2f [0,4194304] 0 2024-03-23T11:47:35.624 INFO:tasks.workunit.client.1.smithi111.stdout:6/536: dwrite d2/d1f/faf [0,4194304] 0 2024-03-23T11:47:35.631 INFO:tasks.workunit.client.0.smithi063.stdout:0/435: dread - d2/d14/f66 zero size 2024-03-23T11:47:35.631 INFO:tasks.workunit.client.1.smithi111.stdout:9/531: mkdir d0/d31/d41/d9d/db0 0 2024-03-23T11:47:35.640 INFO:tasks.workunit.client.0.smithi063.stdout:3/535: dread f6 [0,4194304] 0 2024-03-23T11:47:35.640 INFO:tasks.workunit.client.0.smithi063.stdout:9/436: truncate d4/f14 458178 0 2024-03-23T11:47:35.644 INFO:tasks.workunit.client.1.smithi111.stdout:0/470: rename d5/db/d1f/d19/l46 to d5/db/d1f/d47/d59/l96 0 2024-03-23T11:47:35.646 INFO:tasks.workunit.client.1.smithi111.stdout:9/532: dread d0/d4/d1c/d36/f7e [0,4194304] 0 2024-03-23T11:47:35.646 INFO:tasks.workunit.client.1.smithi111.stdout:1/485: dwrite d0/d6/d48/f5d [4194304,4194304] 0 2024-03-23T11:47:35.650 INFO:tasks.workunit.client.0.smithi063.stdout:4/447: dwrite d5/da/d20/d4b/d16/d50/f69 [0,4194304] 0 2024-03-23T11:47:35.650 INFO:tasks.workunit.client.1.smithi111.stdout:6/537: dwrite d2/d1a/f64 [4194304,4194304] 0 2024-03-23T11:47:35.650 INFO:tasks.workunit.client.0.smithi063.stdout:8/664: dwrite d6/d1c/fd5 [0,4194304] 0 2024-03-23T11:47:35.651 INFO:tasks.workunit.client.1.smithi111.stdout:7/472: link c6 d14/d44/d70/d7e/c84 0 2024-03-23T11:47:35.653 INFO:tasks.workunit.client.0.smithi063.stdout:9/437: write d4/d31/d43/f50 [4875043,13794] 0 2024-03-23T11:47:35.656 INFO:tasks.workunit.client.1.smithi111.stdout:3/428: rename d0/f31 to d0/d1a/d2b/d38/d7f/f8a 0 2024-03-23T11:47:35.657 INFO:tasks.workunit.client.0.smithi063.stdout:4/448: chown d5/da/d20/d4b/d16/d50/f69 37419318 1 2024-03-23T11:47:35.658 INFO:tasks.workunit.client.1.smithi111.stdout:8/417: dwrite d5/f2d [4194304,4194304] 0 2024-03-23T11:47:35.660 INFO:tasks.workunit.client.0.smithi063.stdout:3/536: creat d13/d22/d2d/fa7 x:0 0 0 2024-03-23T11:47:35.661 INFO:tasks.workunit.client.1.smithi111.stdout:9/533: chown d0/d4/d1c/d2d/d3b/d42/d4f/d95 31 1 2024-03-23T11:47:35.663 INFO:tasks.workunit.client.1.smithi111.stdout:0/471: symlink d5/db/d1f/d19/d45/d4b/l97 0 2024-03-23T11:47:35.664 INFO:tasks.workunit.client.1.smithi111.stdout:0/472: read - d5/db/f92 zero size 2024-03-23T11:47:35.664 INFO:tasks.workunit.client.1.smithi111.stdout:8/418: chown d5/d26/d31/d3b/d55 2 1 2024-03-23T11:47:35.667 INFO:tasks.workunit.client.1.smithi111.stdout:0/473: fsync d5/db/d1f/d47/f5a 0 2024-03-23T11:47:35.674 INFO:tasks.workunit.client.0.smithi063.stdout:6/363: dwrite d1/d2a/f4a [0,4194304] 0 2024-03-23T11:47:35.675 INFO:tasks.workunit.client.1.smithi111.stdout:6/538: symlink d2/d1f/d45/d49/lb3 0 2024-03-23T11:47:35.676 INFO:tasks.workunit.client.1.smithi111.stdout:3/429: write d0/d1a/d2b/f6c [793624,82666] 0 2024-03-23T11:47:35.677 INFO:tasks.workunit.client.1.smithi111.stdout:8/419: creat d5/d26/d2f/d62/f86 x:0 0 0 2024-03-23T11:47:35.677 INFO:tasks.workunit.client.0.smithi063.stdout:8/665: symlink d6/d1f/d54/d63/d90/d5d/d7d/db4/le5 0 2024-03-23T11:47:35.680 INFO:tasks.workunit.client.1.smithi111.stdout:8/420: stat d5/d26/d31/d5e 0 2024-03-23T11:47:35.683 INFO:tasks.workunit.client.1.smithi111.stdout:0/474: rmdir d5/db/d1f/d48/d93 0 2024-03-23T11:47:35.683 INFO:tasks.workunit.client.1.smithi111.stdout:6/539: creat d2/d1f/d45/d9a/fb4 x:0 0 0 2024-03-23T11:47:35.684 INFO:tasks.workunit.client.1.smithi111.stdout:8/421: fsync d5/d26/d31/d5e/d4a/f52 0 2024-03-23T11:47:35.684 INFO:tasks.workunit.client.0.smithi063.stdout:3/537: dread d13/d22/d2d/f45 [0,4194304] 0 2024-03-23T11:47:35.688 INFO:tasks.workunit.client.1.smithi111.stdout:0/475: chown d5/db/d1f/d47/l55 102419 1 2024-03-23T11:47:35.689 INFO:tasks.workunit.client.0.smithi063.stdout:3/538: chown d13/f56 97 1 2024-03-23T11:47:35.691 INFO:tasks.workunit.client.1.smithi111.stdout:8/422: stat d5/d1b/c29 0 2024-03-23T11:47:35.693 INFO:tasks.workunit.client.0.smithi063.stdout:6/364: creat d1/f6e x:0 0 0 2024-03-23T11:47:35.694 INFO:tasks.workunit.client.0.smithi063.stdout:9/438: dwrite d4/d31/f59 [0,4194304] 0 2024-03-23T11:47:35.695 INFO:tasks.workunit.client.1.smithi111.stdout:6/540: rename d2/d1a/d1c/d2c/f83 to d2/d35/fb5 0 2024-03-23T11:47:35.696 INFO:tasks.workunit.client.0.smithi063.stdout:3/539: symlink d13/d22/d2d/d3f/d58/d86/la8 0 2024-03-23T11:47:35.697 INFO:tasks.workunit.client.0.smithi063.stdout:9/439: fdatasync d4/f1e 0 2024-03-23T11:47:35.698 INFO:tasks.workunit.client.0.smithi063.stdout:6/365: chown d1/dc/f1d 1072 1 2024-03-23T11:47:35.698 INFO:tasks.workunit.client.0.smithi063.stdout:8/666: dread d6/d1f/d54/d63/fc2 [0,4194304] 0 2024-03-23T11:47:35.700 INFO:tasks.workunit.client.1.smithi111.stdout:0/476: mknod d5/c98 0 2024-03-23T11:47:35.701 INFO:tasks.workunit.client.0.smithi063.stdout:3/540: read f12 [2934057,49960] 0 2024-03-23T11:47:35.702 INFO:tasks.workunit.client.1.smithi111.stdout:0/477: readlink d5/db/d4d/l6d 0 2024-03-23T11:47:35.703 INFO:tasks.workunit.client.1.smithi111.stdout:8/423: write d5/d1b/f2b [984838,82237] 0 2024-03-23T11:47:35.704 INFO:tasks.workunit.client.0.smithi063.stdout:9/440: chown d4/d4c/d19/f33 10454318 1 2024-03-23T11:47:35.706 INFO:tasks.workunit.client.0.smithi063.stdout:8/667: rmdir d6/dd/d59/daf/dba 39 2024-03-23T11:47:35.708 INFO:tasks.workunit.client.0.smithi063.stdout:8/668: chown d6/d1f/l29 3 1 2024-03-23T11:47:35.709 INFO:tasks.workunit.client.1.smithi111.stdout:6/541: rename d2/d1a/d1c/d2c/l8e to d2/d1f/d45/d49/lb6 0 2024-03-23T11:47:35.712 INFO:tasks.workunit.client.1.smithi111.stdout:5/539: sync 2024-03-23T11:47:35.713 INFO:tasks.workunit.client.0.smithi063.stdout:9/441: symlink d4/d31/d43/l96 0 2024-03-23T11:47:35.716 INFO:tasks.workunit.client.0.smithi063.stdout:9/442: write d4/d4c/f25 [1382928,23084] 0 2024-03-23T11:47:35.719 INFO:tasks.workunit.client.1.smithi111.stdout:8/424: link d5/d26/d31/d3b/f70 d5/d26/d2f/d42/d58/f87 0 2024-03-23T11:47:35.719 INFO:tasks.workunit.client.0.smithi063.stdout:9/443: chown d4/f2a 0 1 2024-03-23T11:47:35.721 INFO:tasks.workunit.client.1.smithi111.stdout:8/425: stat d5/d26/d2f/d62/d75 0 2024-03-23T11:47:35.723 INFO:tasks.workunit.client.0.smithi063.stdout:8/669: symlink d6/d1f/d54/d63/d90/d15/d9d/le6 0 2024-03-23T11:47:35.725 INFO:tasks.workunit.client.1.smithi111.stdout:5/540: symlink d1/d3/d6/d30/db2/lb6 0 2024-03-23T11:47:35.727 INFO:tasks.workunit.client.0.smithi063.stdout:9/444: rename d4/d4c/c88 to d4/d31/d70/c97 0 2024-03-23T11:47:35.730 INFO:tasks.workunit.client.1.smithi111.stdout:5/541: mknod d1/d3/d3a/cb7 0 2024-03-23T11:47:35.730 INFO:tasks.workunit.client.0.smithi063.stdout:8/670: mknod d6/d1f/d54/d5c/d76/ce7 0 2024-03-23T11:47:35.732 INFO:tasks.workunit.client.1.smithi111.stdout:5/542: chown d1/d3/d6/d7/d63/d2c/f51 699 1 2024-03-23T11:47:35.733 INFO:tasks.workunit.client.0.smithi063.stdout:9/445: mknod d4/d31/d43/d49/c98 0 2024-03-23T11:47:35.734 INFO:tasks.workunit.client.1.smithi111.stdout:8/426: rename d5/d26/d31/d5e/f3f to d5/d40/f88 0 2024-03-23T11:47:35.734 INFO:tasks.workunit.client.0.smithi063.stdout:8/671: chown d6/d1c/f1d 35526 1 2024-03-23T11:47:35.735 INFO:tasks.workunit.client.0.smithi063.stdout:0/436: write d2/d14/f64 [2639153,61933] 0 2024-03-23T11:47:35.740 INFO:tasks.workunit.client.1.smithi111.stdout:0/478: rmdir d5/db/d1f/d47/d59 39 2024-03-23T11:47:35.741 INFO:tasks.workunit.client.1.smithi111.stdout:8/427: creat d5/d26/d31/d3b/f89 x:0 0 0 2024-03-23T11:47:35.742 INFO:tasks.workunit.client.0.smithi063.stdout:9/446: symlink d4/d31/d70/l99 0 2024-03-23T11:47:35.743 INFO:tasks.workunit.client.0.smithi063.stdout:8/672: creat d6/d1f/d54/d63/d90/d5d/d7d/fe8 x:0 0 0 2024-03-23T11:47:35.743 INFO:tasks.workunit.client.1.smithi111.stdout:0/479: write d5/db/d1f/d48/f66 [660523,69174] 0 2024-03-23T11:47:35.743 INFO:tasks.workunit.client.1.smithi111.stdout:5/543: symlink d1/d3/d6/d7/d63/d2c/d43/lb8 0 2024-03-23T11:47:35.746 INFO:tasks.workunit.client.0.smithi063.stdout:0/437: creat d2/d4/d63/f9f x:0 0 0 2024-03-23T11:47:35.746 INFO:tasks.workunit.client.1.smithi111.stdout:0/480: write d5/db/d1f/d19/f4c [1131087,73257] 0 2024-03-23T11:47:35.747 INFO:tasks.workunit.client.1.smithi111.stdout:5/544: write d1/d3/d6/d8b/d59/d74/f4d [392094,66802] 0 2024-03-23T11:47:35.749 INFO:tasks.workunit.client.1.smithi111.stdout:9/534: dread d0/d4/f15 [0,4194304] 0 2024-03-23T11:47:35.749 INFO:tasks.workunit.client.0.smithi063.stdout:0/438: chown d2/d12/d32 76325 1 2024-03-23T11:47:35.750 INFO:tasks.workunit.client.0.smithi063.stdout:9/447: truncate d4/d4c/d44/f84 228904 0 2024-03-23T11:47:35.751 INFO:tasks.workunit.client.0.smithi063.stdout:4/449: dwrite d5/da/d20/d4b/f2d [4194304,4194304] 0 2024-03-23T11:47:35.754 INFO:tasks.workunit.client.1.smithi111.stdout:1/486: truncate d0/d5/f35 211994 0 2024-03-23T11:47:35.754 INFO:tasks.workunit.client.0.smithi063.stdout:6/366: truncate d1/dc/d3a/f3f 1348055 0 2024-03-23T11:47:35.755 INFO:tasks.workunit.client.1.smithi111.stdout:1/487: fdatasync d0/d5/f23 0 2024-03-23T11:47:35.755 INFO:tasks.workunit.client.0.smithi063.stdout:3/541: truncate d13/d15/d48/f68 1436727 0 2024-03-23T11:47:35.757 INFO:tasks.workunit.client.0.smithi063.stdout:4/450: write d5/da/d20/d4b/d16/d50/f70 [144036,12181] 0 2024-03-23T11:47:35.757 INFO:tasks.workunit.client.1.smithi111.stdout:8/428: rename d5/c4f to d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/c8a 0 2024-03-23T11:47:35.761 INFO:tasks.workunit.client.1.smithi111.stdout:8/429: read d5/f2d [2151843,90749] 0 2024-03-23T11:47:35.765 INFO:tasks.workunit.client.0.smithi063.stdout:9/448: creat d4/d31/d43/d49/f9a x:0 0 0 2024-03-23T11:47:35.767 INFO:tasks.workunit.client.1.smithi111.stdout:1/488: chown d0/d2b/l46 23362941 1 2024-03-23T11:47:35.768 INFO:tasks.workunit.client.1.smithi111.stdout:5/545: dread d1/d3/d6/d7/f10 [0,4194304] 0 2024-03-23T11:47:35.769 INFO:tasks.workunit.client.0.smithi063.stdout:0/439: rename d2/d14/d36/c37 to d2/d48/ca0 0 2024-03-23T11:47:35.769 INFO:tasks.workunit.client.1.smithi111.stdout:1/489: readlink d0/d6/d48/d4d/l61 0 2024-03-23T11:47:35.775 INFO:tasks.workunit.client.0.smithi063.stdout:6/367: creat d1/d2a/d41/d42/f6f x:0 0 0 2024-03-23T11:47:35.775 INFO:tasks.workunit.client.1.smithi111.stdout:5/546: truncate d1/d3/d6/d7/d63/d2c/d82/d66/f9b 1524924 0 2024-03-23T11:47:35.776 INFO:tasks.workunit.client.1.smithi111.stdout:3/430: dwrite d0/d1a/d1d/f24 [0,4194304] 0 2024-03-23T11:47:35.776 INFO:tasks.workunit.client.0.smithi063.stdout:3/542: unlink d13/d22/d2d/d3f/c79 0 2024-03-23T11:47:35.778 INFO:tasks.workunit.client.1.smithi111.stdout:9/535: mkdir d0/d31/d41/d9d/db0/db1 0 2024-03-23T11:47:35.779 INFO:tasks.workunit.client.0.smithi063.stdout:9/449: dread d4/d4c/d19/f22 [0,4194304] 0 2024-03-23T11:47:35.779 INFO:tasks.workunit.client.0.smithi063.stdout:8/673: creat d6/dd/d59/daf/dc4/fe9 x:0 0 0 2024-03-23T11:47:35.783 INFO:tasks.workunit.client.0.smithi063.stdout:4/451: rename d5/d1e/d2b/f45 to d5/da/d20/d4b/d16/d50/f84 0 2024-03-23T11:47:35.784 INFO:tasks.workunit.client.1.smithi111.stdout:1/490: creat d0/d6/d48/fa8 x:0 0 0 2024-03-23T11:47:35.786 INFO:tasks.workunit.client.0.smithi063.stdout:3/543: truncate d13/d22/d2d/fa7 579769 0 2024-03-23T11:47:35.786 INFO:tasks.workunit.client.0.smithi063.stdout:0/440: mkdir d2/d12/da1 0 2024-03-23T11:47:35.786 INFO:tasks.workunit.client.0.smithi063.stdout:4/452: chown d5/da/d20/d4b/d29/f81 4722541 1 2024-03-23T11:47:35.791 INFO:tasks.workunit.client.0.smithi063.stdout:8/674: unlink d6/d1f/d54/d63/d90/c48 0 2024-03-23T11:47:35.801 INFO:tasks.workunit.client.0.smithi063.stdout:9/450: dwrite d4/f7b [0,4194304] 0 2024-03-23T11:47:35.802 INFO:tasks.workunit.client.0.smithi063.stdout:8/675: mkdir d6/d1f/d54/d63/d90/d15/d9d/dea 0 2024-03-23T11:47:35.804 INFO:tasks.workunit.client.0.smithi063.stdout:0/441: dread d2/d4/dd/f33 [0,4194304] 0 2024-03-23T11:47:35.806 INFO:tasks.workunit.client.0.smithi063.stdout:3/544: getdents d13/d22/d2d/d3f/d67/d7b 0 2024-03-23T11:47:35.807 INFO:tasks.workunit.client.1.smithi111.stdout:8/430: rename d5/d26/d31/d5e/l5f to d5/d26/l8b 0 2024-03-23T11:47:35.808 INFO:tasks.workunit.client.0.smithi063.stdout:8/676: chown d6/d1c/d27/d3f/lcd 2 1 2024-03-23T11:47:35.808 INFO:tasks.workunit.client.0.smithi063.stdout:3/545: stat d13/l16 0 2024-03-23T11:47:35.811 INFO:tasks.workunit.client.0.smithi063.stdout:9/451: symlink d4/d31/d43/d49/d8e/l9b 0 2024-03-23T11:47:35.811 INFO:tasks.workunit.client.0.smithi063.stdout:8/677: read - d6/dd/d59/d7f/dae/fd1 zero size 2024-03-23T11:47:35.812 INFO:tasks.workunit.client.1.smithi111.stdout:1/491: mkdir d0/d2b/d7c/da9 0 2024-03-23T11:47:35.814 INFO:tasks.workunit.client.1.smithi111.stdout:0/481: getdents d5/db/d1f/d32 0 2024-03-23T11:47:35.817 INFO:tasks.workunit.client.1.smithi111.stdout:0/482: chown d5/db/d1f/f22 1 1 2024-03-23T11:47:35.822 INFO:tasks.workunit.client.1.smithi111.stdout:8/431: dwrite d5/d26/d31/d3b/d41/f4e [0,4194304] 0 2024-03-23T11:47:35.824 INFO:tasks.workunit.client.1.smithi111.stdout:1/492: dread d0/f1a [0,4194304] 0 2024-03-23T11:47:35.826 INFO:tasks.workunit.client.0.smithi063.stdout:0/442: rename d2/d4/dd/d28/d46/d51/f86 to d2/d48/fa2 0 2024-03-23T11:47:35.826 INFO:tasks.workunit.client.0.smithi063.stdout:9/452: mkdir d4/d31/d9c 0 2024-03-23T11:47:35.837 INFO:tasks.workunit.client.1.smithi111.stdout:2/556: sync 2024-03-23T11:47:35.842 INFO:tasks.workunit.client.0.smithi063.stdout:0/443: rename d2/d4/dd/d28/f60 to d2/d4/dd/d28/fa3 0 2024-03-23T11:47:35.853 INFO:tasks.workunit.client.0.smithi063.stdout:0/444: creat d2/d6e/d8c/fa4 x:0 0 0 2024-03-23T11:47:35.856 INFO:tasks.workunit.client.0.smithi063.stdout:9/453: link d4/d4c/d44/d64/l92 d4/d31/d9c/l9d 0 2024-03-23T11:47:35.863 INFO:tasks.workunit.client.0.smithi063.stdout:0/445: mkdir d2/d48/da5 0 2024-03-23T11:47:35.868 INFO:tasks.workunit.client.0.smithi063.stdout:9/454: mkdir d4/d31/d9c/d9e 0 2024-03-23T11:47:35.870 INFO:tasks.workunit.client.0.smithi063.stdout:9/455: fdatasync d4/d31/d43/f50 0 2024-03-23T11:47:35.877 INFO:tasks.workunit.client.1.smithi111.stdout:5/547: write d1/d3/d6/d8b/d59/d74/f4b [1737159,60330] 0 2024-03-23T11:47:35.877 INFO:tasks.workunit.client.0.smithi063.stdout:1/321: sync 2024-03-23T11:47:35.880 INFO:tasks.workunit.client.0.smithi063.stdout:9/456: creat d4/d2c/f9f x:0 0 0 2024-03-23T11:47:35.881 INFO:tasks.workunit.client.0.smithi063.stdout:1/322: write d3/d1c/f66 [615742,62787] 0 2024-03-23T11:47:35.881 INFO:tasks.workunit.client.0.smithi063.stdout:3/546: dwrite d13/d22/d2d/f45 [0,4194304] 0 2024-03-23T11:47:35.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:35 smithi063 ceph-mon[35189]: pgmap v37: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 43 MiB/s rd, 208 MiB/s wr, 354 op/s 2024-03-23T11:47:35.886 INFO:tasks.workunit.client.0.smithi063.stdout:1/323: chown d3/df/l13 4034 1 2024-03-23T11:47:35.888 INFO:tasks.workunit.client.0.smithi063.stdout:0/446: rename d2/d4/dd to d2/d14/d36/d77/d94/da6 0 2024-03-23T11:47:35.892 INFO:tasks.workunit.client.0.smithi063.stdout:0/447: read d2/d14/f1c [1113777,28452] 0 2024-03-23T11:47:35.896 INFO:tasks.workunit.client.0.smithi063.stdout:8/678: dwrite d6/d1f/d3e/f8c [0,4194304] 0 2024-03-23T11:47:35.896 INFO:tasks.workunit.client.1.smithi111.stdout:0/483: write d5/db/d1f/f37 [2877678,52381] 0 2024-03-23T11:47:35.898 INFO:tasks.workunit.client.0.smithi063.stdout:9/457: mkdir d4/d31/d70/d8d/da0 0 2024-03-23T11:47:35.902 INFO:tasks.workunit.client.1.smithi111.stdout:8/432: symlink d5/d26/d2f/d62/d75/d4b/d51/d5a/l8c 0 2024-03-23T11:47:35.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:35 smithi111 ceph-mon[44005]: pgmap v37: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 43 MiB/s rd, 208 MiB/s wr, 354 op/s 2024-03-23T11:47:35.904 INFO:tasks.workunit.client.0.smithi063.stdout:3/547: mkdir d13/d22/da9 0 2024-03-23T11:47:35.907 INFO:tasks.workunit.client.0.smithi063.stdout:9/458: creat d4/d31/d43/d49/d8e/fa1 x:0 0 0 2024-03-23T11:47:35.909 INFO:tasks.workunit.client.1.smithi111.stdout:2/557: rmdir d2/d11/d17/d1f/d30 39 2024-03-23T11:47:35.909 INFO:tasks.workunit.client.0.smithi063.stdout:0/448: mkdir d2/d48/da5/da7 0 2024-03-23T11:47:35.915 INFO:tasks.workunit.client.1.smithi111.stdout:7/473: sync 2024-03-23T11:47:35.915 INFO:tasks.workunit.client.1.smithi111.stdout:4/493: sync 2024-03-23T11:47:35.916 INFO:tasks.workunit.client.1.smithi111.stdout:3/431: rmdir d0/d7/d19/d7d 0 2024-03-23T11:47:35.916 INFO:tasks.workunit.client.0.smithi063.stdout:8/679: getdents d6/d1f/d54/d63/d90 0 2024-03-23T11:47:35.916 INFO:tasks.workunit.client.0.smithi063.stdout:9/459: fdatasync d4/d4c/f30 0 2024-03-23T11:47:35.921 INFO:tasks.workunit.client.0.smithi063.stdout:9/460: write d4/fc [3926638,55192] 0 2024-03-23T11:47:35.925 INFO:tasks.workunit.client.0.smithi063.stdout:8/680: creat d6/dd/d59/d7f/dae/feb x:0 0 0 2024-03-23T11:47:35.929 INFO:tasks.workunit.client.0.smithi063.stdout:1/324: dread d3/df/f6e [0,4194304] 0 2024-03-23T11:47:35.931 INFO:tasks.workunit.client.0.smithi063.stdout:9/461: mknod d4/d4c/d19/ca2 0 2024-03-23T11:47:35.932 INFO:tasks.workunit.client.0.smithi063.stdout:3/548: write d13/d22/d2d/f44 [2846138,118019] 0 2024-03-23T11:47:35.934 INFO:tasks.workunit.client.1.smithi111.stdout:8/433: read d5/d26/d31/d4d/f59 [1069768,48214] 0 2024-03-23T11:47:35.935 INFO:tasks.workunit.client.1.smithi111.stdout:2/558: creat d2/d11/d17/d3e/d6b/d2e/d76/fcc x:0 0 0 2024-03-23T11:47:35.937 INFO:tasks.workunit.client.0.smithi063.stdout:3/549: rmdir d13/d66 39 2024-03-23T11:47:35.937 INFO:tasks.workunit.client.0.smithi063.stdout:8/681: getdents d6/d1c/d27/d3f 0 2024-03-23T11:47:35.937 INFO:tasks.workunit.client.1.smithi111.stdout:2/559: chown d2/d11/d17/c25 32125 1 2024-03-23T11:47:35.939 INFO:tasks.workunit.client.0.smithi063.stdout:1/325: dread d3/df/f18 [0,4194304] 0 2024-03-23T11:47:35.939 INFO:tasks.workunit.client.1.smithi111.stdout:4/494: symlink d0/dc/d4c/d2a/l93 0 2024-03-23T11:47:35.940 INFO:tasks.workunit.client.0.smithi063.stdout:1/326: stat d3/f39 0 2024-03-23T11:47:35.941 INFO:tasks.workunit.client.1.smithi111.stdout:7/474: creat d14/d1b/d1e/d2c/d42/d5e/f85 x:0 0 0 2024-03-23T11:47:35.943 INFO:tasks.workunit.client.1.smithi111.stdout:5/548: mknod d1/d3/d6/d8b/d59/d74/cb9 0 2024-03-23T11:47:35.950 INFO:tasks.workunit.client.1.smithi111.stdout:4/495: dread d0/dc/d4c/d4b/d4d/f5a [0,4194304] 0 2024-03-23T11:47:35.954 INFO:tasks.workunit.client.0.smithi063.stdout:1/327: dread d3/f39 [0,4194304] 0 2024-03-23T11:47:35.955 INFO:tasks.workunit.client.1.smithi111.stdout:2/560: symlink d2/d7/lcd 0 2024-03-23T11:47:35.957 INFO:tasks.workunit.client.1.smithi111.stdout:2/561: read d2/d32/d91/f85 [41136,24138] 0 2024-03-23T11:47:35.957 INFO:tasks.workunit.client.0.smithi063.stdout:0/449: dwrite d2/d12/f42 [0,4194304] 0 2024-03-23T11:47:35.957 INFO:tasks.workunit.client.0.smithi063.stdout:3/550: dread f3 [0,4194304] 0 2024-03-23T11:47:35.961 INFO:tasks.workunit.client.0.smithi063.stdout:8/682: link d6/d1f/d54/d63/d90/d9e/cc6 d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/cec 0 2024-03-23T11:47:35.963 INFO:tasks.workunit.client.1.smithi111.stdout:5/549: symlink d1/d3/d6/d7/d63/d2c/d43/d95/lba 0 2024-03-23T11:47:35.963 INFO:tasks.workunit.client.0.smithi063.stdout:0/450: rmdir d2/d14/d36/d77/d94 39 2024-03-23T11:47:35.965 INFO:tasks.workunit.client.0.smithi063.stdout:0/451: readlink d2/d14/d36/l75 0 2024-03-23T11:47:35.966 INFO:tasks.workunit.client.0.smithi063.stdout:9/462: dread d4/d4c/d19/f2d [0,4194304] 0 2024-03-23T11:47:35.967 INFO:tasks.workunit.client.0.smithi063.stdout:3/551: symlink d13/d22/d2d/d3f/d58/d6e/laa 0 2024-03-23T11:47:35.968 INFO:tasks.workunit.client.1.smithi111.stdout:1/493: getdents d0 0 2024-03-23T11:47:35.970 INFO:tasks.workunit.client.0.smithi063.stdout:1/328: getdents d3/d3f 0 2024-03-23T11:47:35.970 INFO:tasks.workunit.client.0.smithi063.stdout:9/463: write d4/d4c/f69 [2957967,47958] 0 2024-03-23T11:47:35.971 INFO:tasks.workunit.client.1.smithi111.stdout:7/475: dwrite d14/f15 [0,4194304] 0 2024-03-23T11:47:35.973 INFO:tasks.workunit.client.0.smithi063.stdout:1/329: fsync d3/f11 0 2024-03-23T11:47:35.978 INFO:tasks.workunit.client.0.smithi063.stdout:1/330: chown d3/d3f/d4d 4 1 2024-03-23T11:47:35.981 INFO:tasks.workunit.client.0.smithi063.stdout:1/331: fsync d3/f2a 0 2024-03-23T11:47:35.981 INFO:tasks.workunit.client.0.smithi063.stdout:9/464: write d4/d4c/d44/d64/f80 [3077417,110055] 0 2024-03-23T11:47:35.982 INFO:tasks.workunit.client.1.smithi111.stdout:2/562: creat d2/de/d12/d70/d96/fce x:0 0 0 2024-03-23T11:47:35.985 INFO:tasks.workunit.client.1.smithi111.stdout:0/484: creat d5/db/d1f/d47/d59/f99 x:0 0 0 2024-03-23T11:47:35.986 INFO:tasks.workunit.client.0.smithi063.stdout:1/332: dread d3/f39 [0,4194304] 0 2024-03-23T11:47:35.987 INFO:tasks.workunit.client.0.smithi063.stdout:0/452: dread d2/d12/f42 [0,4194304] 0 2024-03-23T11:47:35.988 INFO:tasks.workunit.client.1.smithi111.stdout:0/485: truncate d5/db/d1f/d47/d59/f7d 560124 0 2024-03-23T11:47:35.992 INFO:tasks.workunit.client.0.smithi063.stdout:0/453: creat d2/d12/d35/fa8 x:0 0 0 2024-03-23T11:47:35.992 INFO:tasks.workunit.client.1.smithi111.stdout:5/550: truncate d1/d3/d6/d7/f69 78377 0 2024-03-23T11:47:35.993 INFO:tasks.workunit.client.0.smithi063.stdout:8/683: write d6/d1f/d54/d63/d90/f28 [3147685,70335] 0 2024-03-23T11:47:35.994 INFO:tasks.workunit.client.0.smithi063.stdout:0/454: stat d2/d14/d36/f61 0 2024-03-23T11:47:35.996 INFO:tasks.workunit.client.0.smithi063.stdout:0/455: write d2/d14/f79 [416046,5604] 0 2024-03-23T11:47:35.999 INFO:tasks.workunit.client.0.smithi063.stdout:8/684: creat d6/dd/d59/daf/dc4/fed x:0 0 0 2024-03-23T11:47:36.001 INFO:tasks.workunit.client.1.smithi111.stdout:3/432: getdents d0/d7 0 2024-03-23T11:47:36.003 INFO:tasks.workunit.client.1.smithi111.stdout:3/433: read d0/d1a/d1d/f36 [1659263,97645] 0 2024-03-23T11:47:36.004 INFO:tasks.workunit.client.1.smithi111.stdout:6/542: sync 2024-03-23T11:47:36.006 INFO:tasks.workunit.client.1.smithi111.stdout:6/543: fsync d2/d1a/f2d 0 2024-03-23T11:47:36.006 INFO:tasks.workunit.client.1.smithi111.stdout:4/496: link d0/d7/f92 d0/dc/d4c/d4b/d4d/f94 0 2024-03-23T11:47:36.007 INFO:tasks.workunit.client.0.smithi063.stdout:9/465: dwrite d4/f16 [0,4194304] 0 2024-03-23T11:47:36.010 INFO:tasks.workunit.client.0.smithi063.stdout:9/466: dread - d4/d2c/f7f zero size 2024-03-23T11:47:36.011 INFO:tasks.workunit.client.1.smithi111.stdout:2/563: creat d2/d11/d17/fcf x:0 0 0 2024-03-23T11:47:36.012 INFO:tasks.workunit.client.0.smithi063.stdout:8/685: rmdir d6/d1f/d54/d5c/d66/da4 39 2024-03-23T11:47:36.013 INFO:tasks.workunit.client.1.smithi111.stdout:2/564: write d2/d11/d17/f21 [1933160,38546] 0 2024-03-23T11:47:36.015 INFO:tasks.workunit.client.0.smithi063.stdout:3/552: dread ff [4194304,4194304] 0 2024-03-23T11:47:36.016 INFO:tasks.workunit.client.0.smithi063.stdout:0/456: unlink d2/d14/d36/d77/d94/da6/d28/d46/f83 0 2024-03-23T11:47:36.017 INFO:tasks.workunit.client.0.smithi063.stdout:3/553: chown d13/d22/d2d/d3f/d67 627 1 2024-03-23T11:47:36.021 INFO:tasks.workunit.client.0.smithi063.stdout:9/467: mknod d4/d2c/ca3 0 2024-03-23T11:47:36.023 INFO:tasks.workunit.client.0.smithi063.stdout:3/554: dread - d13/d22/d2d/d3f/d58/d6e/fa5 zero size 2024-03-23T11:47:36.025 INFO:tasks.workunit.client.0.smithi063.stdout:8/686: fdatasync d6/f71 0 2024-03-23T11:47:36.030 INFO:tasks.workunit.client.0.smithi063.stdout:3/555: read d13/d15/d48/f68 [323510,29784] 0 2024-03-23T11:47:36.032 INFO:tasks.workunit.client.1.smithi111.stdout:8/434: sync 2024-03-23T11:47:36.032 INFO:tasks.workunit.client.0.smithi063.stdout:0/457: getdents d2/d14/d36/d77/d94/da6/d25/d93 0 2024-03-23T11:47:36.033 INFO:tasks.workunit.client.0.smithi063.stdout:3/556: readlink d13/d22/l51 0 2024-03-23T11:47:36.033 INFO:tasks.workunit.client.1.smithi111.stdout:8/435: write d5/d1b/f2b [2479176,121181] 0 2024-03-23T11:47:36.036 INFO:tasks.workunit.client.0.smithi063.stdout:8/687: rmdir d6/d1f/d54/d63/d90/d5d 39 2024-03-23T11:47:36.037 INFO:tasks.workunit.client.0.smithi063.stdout:0/458: symlink d2/d4/d63/la9 0 2024-03-23T11:47:36.038 INFO:tasks.workunit.client.0.smithi063.stdout:3/557: truncate d13/d22/d2d/d3f/d58/f76 1040573 0 2024-03-23T11:47:36.042 INFO:tasks.workunit.client.0.smithi063.stdout:8/688: write d6/d1f/d54/f6a [916564,82143] 0 2024-03-23T11:47:36.045 INFO:tasks.workunit.client.1.smithi111.stdout:1/494: mknod d0/d6/d48/d4a/d6c/d96/caa 0 2024-03-23T11:47:36.047 INFO:tasks.workunit.client.0.smithi063.stdout:3/558: truncate d13/d22/d2d/d3f/d58/d86/f93 611592 0 2024-03-23T11:47:36.049 INFO:tasks.workunit.client.1.smithi111.stdout:9/536: truncate d0/d4/d1c/d36/f11 5106298 0 2024-03-23T11:47:36.049 INFO:tasks.workunit.client.0.smithi063.stdout:3/559: stat d13/d22/d2d/c69 0 2024-03-23T11:47:36.050 INFO:tasks.workunit.client.1.smithi111.stdout:7/476: fsync d14/d37/f49 0 2024-03-23T11:47:36.051 INFO:tasks.workunit.client.0.smithi063.stdout:9/468: dwrite d4/d31/d43/d49/f67 [4194304,4194304] 0 2024-03-23T11:47:36.052 INFO:tasks.workunit.client.0.smithi063.stdout:0/459: creat d2/d48/da5/da7/faa x:0 0 0 2024-03-23T11:47:36.055 INFO:tasks.workunit.client.1.smithi111.stdout:0/486: dwrite d5/fc [0,4194304] 0 2024-03-23T11:47:36.055 INFO:tasks.workunit.client.0.smithi063.stdout:3/560: write d13/d22/d2d/d3f/d58/d6e/fa5 [181334,22173] 0 2024-03-23T11:47:36.057 INFO:tasks.workunit.client.0.smithi063.stdout:9/469: write d4/d4c/d44/d64/f80 [7167638,57863] 0 2024-03-23T11:47:36.057 INFO:tasks.workunit.client.1.smithi111.stdout:0/487: write d5/db/d1f/f79 [166032,76104] 0 2024-03-23T11:47:36.059 INFO:tasks.workunit.client.0.smithi063.stdout:3/561: fsync d13/d22/d2d/d3f/d5b/f77 0 2024-03-23T11:47:36.063 INFO:tasks.workunit.client.0.smithi063.stdout:0/460: symlink d2/d48/lab 0 2024-03-23T11:47:36.066 INFO:tasks.workunit.client.0.smithi063.stdout:8/689: fsync d6/d1f/d54/d63/d90/d5d/f87 0 2024-03-23T11:47:36.067 INFO:tasks.workunit.client.0.smithi063.stdout:9/470: readlink d4/d31/d9c/l9d 0 2024-03-23T11:47:36.067 INFO:tasks.workunit.client.1.smithi111.stdout:7/477: dwrite d14/d1b/d1e/f47 [0,4194304] 0 2024-03-23T11:47:36.070 INFO:tasks.workunit.client.1.smithi111.stdout:2/565: chown d2/d32/d98 1410 1 2024-03-23T11:47:36.073 INFO:tasks.workunit.client.0.smithi063.stdout:0/461: unlink d2/d4/f5e 0 2024-03-23T11:47:36.076 INFO:tasks.workunit.client.0.smithi063.stdout:9/471: creat d4/d31/d43/d49/fa4 x:0 0 0 2024-03-23T11:47:36.076 INFO:tasks.workunit.client.1.smithi111.stdout:8/436: mkdir d5/d26/d2f/d62/d75/d8d 0 2024-03-23T11:47:36.078 INFO:tasks.workunit.client.1.smithi111.stdout:1/495: mkdir d0/d6/d48/d4d/dab 0 2024-03-23T11:47:36.079 INFO:tasks.workunit.client.0.smithi063.stdout:8/690: write d6/dd/df/f11 [4745845,94766] 0 2024-03-23T11:47:36.081 INFO:tasks.workunit.client.0.smithi063.stdout:8/691: fsync d6/dd/d59/db2/fe2 0 2024-03-23T11:47:36.085 INFO:tasks.workunit.client.0.smithi063.stdout:0/462: truncate d2/d14/d36/d77/d94/da6/d28/d46/d51/f82 319329 0 2024-03-23T11:47:36.086 INFO:tasks.workunit.client.1.smithi111.stdout:3/434: symlink d0/d72/d7b/l8b 0 2024-03-23T11:47:36.086 INFO:tasks.workunit.client.1.smithi111.stdout:2/566: dwrite d2/d11/d17/d3e/f40 [4194304,4194304] 0 2024-03-23T11:47:36.087 INFO:tasks.workunit.client.0.smithi063.stdout:0/463: write d2/d14/f66 [786915,14348] 0 2024-03-23T11:47:36.089 INFO:tasks.workunit.client.0.smithi063.stdout:0/464: stat d2/d14/d36/l53 0 2024-03-23T11:47:36.093 INFO:tasks.workunit.client.1.smithi111.stdout:0/488: symlink d5/db/d1f/d39/d8b/l9a 0 2024-03-23T11:47:36.095 INFO:tasks.workunit.client.1.smithi111.stdout:4/497: mkdir d0/d95 0 2024-03-23T11:47:36.097 INFO:tasks.workunit.client.1.smithi111.stdout:0/489: chown d5/db/d1f/d32/f3b 0 1 2024-03-23T11:47:36.100 INFO:tasks.workunit.client.1.smithi111.stdout:0/490: chown d5/db/d1f/d47/c7f 8162 1 2024-03-23T11:47:36.100 INFO:tasks.workunit.client.1.smithi111.stdout:8/437: fdatasync d5/f6 0 2024-03-23T11:47:36.102 INFO:tasks.workunit.client.0.smithi063.stdout:0/465: mknod d2/d14/d36/d77/d94/da6/d28/d46/cac 0 2024-03-23T11:47:36.102 INFO:tasks.workunit.client.0.smithi063.stdout:8/692: dread d6/d1f/d54/d63/d90/f5f [0,4194304] 0 2024-03-23T11:47:36.103 INFO:tasks.workunit.client.1.smithi111.stdout:3/435: creat d0/d53/f8c x:0 0 0 2024-03-23T11:47:36.103 INFO:tasks.workunit.client.1.smithi111.stdout:9/537: dread d0/d4/d1c/d2d/d3b/d42/d4f/f5c [0,4194304] 0 2024-03-23T11:47:36.105 INFO:tasks.workunit.client.1.smithi111.stdout:8/438: write d5/d26/d31/d3b/d41/f4e [80561,36176] 0 2024-03-23T11:47:36.105 INFO:tasks.workunit.client.1.smithi111.stdout:2/567: stat d2/d11/d17/d1f/d30 0 2024-03-23T11:47:36.106 INFO:tasks.workunit.client.1.smithi111.stdout:3/436: write d0/d7/f82 [926373,105995] 0 2024-03-23T11:47:36.107 INFO:tasks.workunit.client.0.smithi063.stdout:8/693: mknod d6/d1f/d54/d63/cee 0 2024-03-23T11:47:36.109 INFO:tasks.workunit.client.1.smithi111.stdout:6/544: creat d2/d1f/d45/d49/fb7 x:0 0 0 2024-03-23T11:47:36.110 INFO:tasks.workunit.client.0.smithi063.stdout:3/562: dread f8 [4194304,4194304] 0 2024-03-23T11:47:36.112 INFO:tasks.workunit.client.0.smithi063.stdout:0/466: dread d2/d14/d36/d77/d94/da6/d28/fa3 [0,4194304] 0 2024-03-23T11:47:36.115 INFO:tasks.workunit.client.1.smithi111.stdout:4/498: dwrite d0/d17/f2c [4194304,4194304] 0 2024-03-23T11:47:36.117 INFO:tasks.workunit.client.1.smithi111.stdout:7/478: mkdir d14/d1b/d1e/d86 0 2024-03-23T11:47:36.123 INFO:tasks.workunit.client.1.smithi111.stdout:3/437: dwrite d0/d1a/d2b/f33 [0,4194304] 0 2024-03-23T11:47:36.124 INFO:tasks.workunit.client.0.smithi063.stdout:8/694: mknod d6/d1f/d54/d63/d90/d5d/cef 0 2024-03-23T11:47:36.124 INFO:tasks.workunit.client.1.smithi111.stdout:2/568: creat d2/de/d12/fd0 x:0 0 0 2024-03-23T11:47:36.125 INFO:tasks.workunit.client.1.smithi111.stdout:3/438: chown d0/d1a/d1d/d26/d5a 1 1 2024-03-23T11:47:36.128 INFO:tasks.workunit.client.0.smithi063.stdout:8/695: readlink d6/d1f/d54/d5c/d76/le0 0 2024-03-23T11:47:36.129 INFO:tasks.workunit.client.0.smithi063.stdout:0/467: mknod d2/d14/d36/d77/d94/da6/d28/d46/d51/cad 0 2024-03-23T11:47:36.131 INFO:tasks.workunit.client.1.smithi111.stdout:4/499: unlink d0/dc/d4c/d4b/d4d/f5a 0 2024-03-23T11:47:36.132 INFO:tasks.workunit.client.1.smithi111.stdout:3/439: dread d0/d1a/d2b/f6c [0,4194304] 0 2024-03-23T11:47:36.133 INFO:tasks.workunit.client.1.smithi111.stdout:4/500: chown d0/dc/l23 3972 1 2024-03-23T11:47:36.135 INFO:tasks.workunit.client.1.smithi111.stdout:7/479: dread d14/f15 [4194304,4194304] 0 2024-03-23T11:47:36.140 INFO:tasks.workunit.client.1.smithi111.stdout:9/538: rename d0/f13 to d0/d31/d41/d65/fb2 0 2024-03-23T11:47:36.141 INFO:tasks.workunit.client.0.smithi063.stdout:8/696: symlink d6/d1f/d54/d63/d90/d15/d9d/lf0 0 2024-03-23T11:47:36.144 INFO:tasks.workunit.client.0.smithi063.stdout:3/563: dwrite d13/d22/d2d/d3f/d58/f76 [0,4194304] 0 2024-03-23T11:47:36.144 INFO:tasks.workunit.client.1.smithi111.stdout:0/491: creat d5/db/d1f/f9b x:0 0 0 2024-03-23T11:47:36.145 INFO:tasks.workunit.client.1.smithi111.stdout:3/440: symlink d0/d72/l8d 0 2024-03-23T11:47:36.150 INFO:tasks.workunit.client.1.smithi111.stdout:4/501: dwrite d0/dc/d1f/f76 [0,4194304] 0 2024-03-23T11:47:36.152 INFO:tasks.workunit.client.1.smithi111.stdout:4/502: chown d0/dc/l33 1725191725 1 2024-03-23T11:47:36.157 INFO:tasks.workunit.client.1.smithi111.stdout:4/503: write d0/dc/d4c/f39 [47644,38303] 0 2024-03-23T11:47:36.157 INFO:tasks.workunit.client.1.smithi111.stdout:9/539: fsync d0/d4/d6a/f39 0 2024-03-23T11:47:36.160 INFO:tasks.workunit.client.0.smithi063.stdout:0/468: dwrite d2/d12/f68 [0,4194304] 0 2024-03-23T11:47:36.161 INFO:tasks.workunit.client.1.smithi111.stdout:1/496: dread d0/d6/f27 [0,4194304] 0 2024-03-23T11:47:36.164 INFO:tasks.workunit.client.0.smithi063.stdout:8/697: mkdir d6/d1f/d54/d5c/d66/df1 0 2024-03-23T11:47:36.168 INFO:tasks.workunit.client.1.smithi111.stdout:3/441: creat d0/d1a/d1d/d26/d5a/f8e x:0 0 0 2024-03-23T11:47:36.170 INFO:tasks.workunit.client.0.smithi063.stdout:3/564: link d13/d22/d2d/d3f/f72 d13/d22/d2d/d3f/d67/d7b/fab 0 2024-03-23T11:47:36.171 INFO:tasks.workunit.client.1.smithi111.stdout:3/442: fdatasync d0/d1a/f1c 0 2024-03-23T11:47:36.175 INFO:tasks.workunit.client.1.smithi111.stdout:9/540: mknod d0/d4/d10/d29/cb3 0 2024-03-23T11:47:36.178 INFO:tasks.workunit.client.1.smithi111.stdout:5/551: dwrite d1/d3/d6/d30/f70 [0,4194304] 0 2024-03-23T11:47:36.178 INFO:tasks.workunit.client.0.smithi063.stdout:8/698: rename d6/d1f/d54/d5c/lb0 to d6/d30/lf2 0 2024-03-23T11:47:36.184 INFO:tasks.workunit.client.1.smithi111.stdout:1/497: dread d0/d6/d48/f5d [4194304,4194304] 0 2024-03-23T11:47:36.192 INFO:tasks.workunit.client.1.smithi111.stdout:3/443: mknod d0/d1a/d1d/d26/d5a/d64/c8f 0 2024-03-23T11:47:36.194 INFO:tasks.workunit.client.1.smithi111.stdout:3/444: write d0/f5d [626334,36655] 0 2024-03-23T11:47:36.196 INFO:tasks.workunit.client.1.smithi111.stdout:1/498: write d0/d5/dd/d91/fa1 [939801,57105] 0 2024-03-23T11:47:36.197 INFO:tasks.workunit.client.1.smithi111.stdout:1/499: chown d0 5 1 2024-03-23T11:47:36.198 INFO:tasks.workunit.client.1.smithi111.stdout:6/545: dread d2/d1a/f72 [0,4194304] 0 2024-03-23T11:47:36.200 INFO:tasks.workunit.client.0.smithi063.stdout:3/565: read d13/d15/f33 [383742,80395] 0 2024-03-23T11:47:36.200 INFO:tasks.workunit.client.1.smithi111.stdout:7/480: dread d14/d1b/d1e/f39 [0,4194304] 0 2024-03-23T11:47:36.202 INFO:tasks.workunit.client.1.smithi111.stdout:7/481: write f11 [2833835,111626] 0 2024-03-23T11:47:36.204 INFO:tasks.workunit.client.1.smithi111.stdout:5/552: creat d1/d3/d6/d7/d8c/d94/fbb x:0 0 0 2024-03-23T11:47:36.206 INFO:tasks.workunit.client.1.smithi111.stdout:0/492: dread d5/db/f15 [0,4194304] 0 2024-03-23T11:47:36.213 INFO:tasks.workunit.client.1.smithi111.stdout:6/546: unlink d2/d1f/l3c 0 2024-03-23T11:47:36.214 INFO:tasks.workunit.client.1.smithi111.stdout:6/547: readlink d2/d73/l78 0 2024-03-23T11:47:36.215 INFO:tasks.workunit.client.0.smithi063.stdout:8/699: truncate d6/dd/df/f11 991138 0 2024-03-23T11:47:36.217 INFO:tasks.workunit.client.0.smithi063.stdout:3/566: write d13/d22/f2f [724151,71059] 0 2024-03-23T11:47:36.218 INFO:tasks.workunit.client.0.smithi063.stdout:8/700: write d6/f43 [2916421,85481] 0 2024-03-23T11:47:36.220 INFO:tasks.workunit.client.0.smithi063.stdout:3/567: dread - d13/d22/d2d/f4f zero size 2024-03-23T11:47:36.222 INFO:tasks.workunit.client.0.smithi063.stdout:8/701: chown d6/d1f/d54/d63/d90/d15/d9d/dea 260 1 2024-03-23T11:47:36.222 INFO:tasks.workunit.client.0.smithi063.stdout:3/568: chown lb 13 1 2024-03-23T11:47:36.222 INFO:tasks.workunit.client.1.smithi111.stdout:8/439: dread d5/f2d [0,4194304] 0 2024-03-23T11:47:36.225 INFO:tasks.workunit.client.1.smithi111.stdout:6/548: fdatasync d2/d35/f77 0 2024-03-23T11:47:36.226 INFO:tasks.workunit.client.1.smithi111.stdout:6/549: dread - d2/d1f/d45/d9a/fb4 zero size 2024-03-23T11:47:36.228 INFO:tasks.workunit.client.1.smithi111.stdout:6/550: stat d2/d1f/d45/d49/fb7 0 2024-03-23T11:47:36.229 INFO:tasks.workunit.client.1.smithi111.stdout:8/440: creat d5/d26/d31/d5e/d4a/f8e x:0 0 0 2024-03-23T11:47:36.230 INFO:tasks.workunit.client.0.smithi063.stdout:3/569: truncate f5 5699464 0 2024-03-23T11:47:36.231 INFO:tasks.workunit.client.1.smithi111.stdout:2/569: truncate d2/d32/fa2 3189148 0 2024-03-23T11:47:36.233 INFO:tasks.workunit.client.1.smithi111.stdout:8/441: mkdir d5/d26/d2f/d8f 0 2024-03-23T11:47:36.233 INFO:tasks.workunit.client.1.smithi111.stdout:5/553: dread d1/d3/d6/d8b/d59/d74/f21 [0,4194304] 0 2024-03-23T11:47:36.234 INFO:tasks.workunit.client.0.smithi063.stdout:3/570: truncate d13/d15/d48/fa6 652658 0 2024-03-23T11:47:36.234 INFO:tasks.workunit.client.1.smithi111.stdout:6/551: unlink d2/d17/f4a 0 2024-03-23T11:47:36.236 INFO:tasks.workunit.client.1.smithi111.stdout:2/570: dread d2/d11/d17/d3e/d6b/d2e/f6f [0,4194304] 0 2024-03-23T11:47:36.237 INFO:tasks.workunit.client.0.smithi063.stdout:3/571: read d13/d22/f6f [52687,109350] 0 2024-03-23T11:47:36.238 INFO:tasks.workunit.client.1.smithi111.stdout:4/504: write d0/dc/d4c/d2a/f70 [777201,105974] 0 2024-03-23T11:47:36.238 INFO:tasks.workunit.client.1.smithi111.stdout:2/571: dread - d2/d11/d17/d3e/d6b/fb3 zero size 2024-03-23T11:47:36.240 INFO:tasks.workunit.client.1.smithi111.stdout:8/442: rename d5/d32/l37 to d5/d26/d2f/d42/l90 0 2024-03-23T11:47:36.241 INFO:tasks.workunit.client.1.smithi111.stdout:6/552: mknod d2/d1a/d1c/d2c/cb8 0 2024-03-23T11:47:36.242 INFO:tasks.workunit.client.1.smithi111.stdout:2/572: rmdir d2/d11/d17/d3e 39 2024-03-23T11:47:36.243 INFO:tasks.workunit.client.1.smithi111.stdout:8/443: mkdir d5/d26/d2f/d62/d75/d91 0 2024-03-23T11:47:36.245 INFO:tasks.workunit.client.1.smithi111.stdout:4/505: rename d0/dc/d4c/d2a/f40 to d0/dc/d4c/d4b/d4f/f96 0 2024-03-23T11:47:36.247 INFO:tasks.workunit.client.1.smithi111.stdout:9/541: truncate d0/d4/d1c/f45 271679 0 2024-03-23T11:47:36.248 INFO:tasks.workunit.client.1.smithi111.stdout:2/573: mkdir d2/de/d6c/dd1 0 2024-03-23T11:47:36.250 INFO:tasks.workunit.client.1.smithi111.stdout:3/445: write d0/d7/f2a [3324985,125795] 0 2024-03-23T11:47:36.250 INFO:tasks.workunit.client.1.smithi111.stdout:4/506: fdatasync d0/dc/f1c 0 2024-03-23T11:47:36.251 INFO:tasks.workunit.client.1.smithi111.stdout:2/574: dread d2/d32/d91/f85 [0,4194304] 0 2024-03-23T11:47:36.252 INFO:tasks.workunit.client.1.smithi111.stdout:9/542: creat d0/d4/d10/d80/fb4 x:0 0 0 2024-03-23T11:47:36.254 INFO:tasks.workunit.client.0.smithi063.stdout:3/572: dwrite d13/d15/f2a [4194304,4194304] 0 2024-03-23T11:47:36.254 INFO:tasks.workunit.client.1.smithi111.stdout:8/444: mknod d5/d32/c92 0 2024-03-23T11:47:36.255 INFO:tasks.workunit.client.1.smithi111.stdout:3/446: mkdir d0/d1a/d4f/d90 0 2024-03-23T11:47:36.256 INFO:tasks.workunit.client.1.smithi111.stdout:1/500: write d0/d2b/f5c [315843,75000] 0 2024-03-23T11:47:36.258 INFO:tasks.workunit.client.1.smithi111.stdout:2/575: mkdir d2/d11/d17/d1f/da0/dd2 0 2024-03-23T11:47:36.261 INFO:tasks.workunit.client.1.smithi111.stdout:9/543: chown d0/d4/d10/d80/l86 37240 1 2024-03-23T11:47:36.261 INFO:tasks.workunit.client.1.smithi111.stdout:8/445: getdents d5/d32/d7d 0 2024-03-23T11:47:36.264 INFO:tasks.workunit.client.1.smithi111.stdout:3/447: chown d0/d72/l77 43517858 1 2024-03-23T11:47:36.265 INFO:tasks.workunit.client.1.smithi111.stdout:7/482: dwrite f10 [4194304,4194304] 0 2024-03-23T11:47:36.265 INFO:tasks.workunit.client.1.smithi111.stdout:1/501: rename d0/f20 to d0/d5/dd/d91/fac 0 2024-03-23T11:47:36.266 INFO:tasks.workunit.client.1.smithi111.stdout:9/544: rmdir d0/d4/d1c/d2d/d6e 39 2024-03-23T11:47:36.270 INFO:tasks.workunit.client.1.smithi111.stdout:2/576: mkdir d2/d11/d17/d3e/d6b/d9d/d52/dba/dd3 0 2024-03-23T11:47:36.271 INFO:tasks.workunit.client.0.smithi063.stdout:3/573: dwrite d13/f4d [4194304,4194304] 0 2024-03-23T11:47:36.276 INFO:tasks.workunit.client.1.smithi111.stdout:7/483: mkdir d14/d1b/d1e/d87 0 2024-03-23T11:47:36.276 INFO:tasks.workunit.client.0.smithi063.stdout:3/574: fsync d13/d22/d2d/d3f/d5b/f65 0 2024-03-23T11:47:36.276 INFO:tasks.workunit.client.1.smithi111.stdout:8/446: rmdir d5/d26/d2f/d62/d75/d91 0 2024-03-23T11:47:36.277 INFO:tasks.workunit.client.1.smithi111.stdout:2/577: creat d2/d11/d17/d1f/da0/fd4 x:0 0 0 2024-03-23T11:47:36.280 INFO:tasks.workunit.client.0.smithi063.stdout:3/575: mkdir d13/d22/d2d/d3f/d58/d97/dac 0 2024-03-23T11:47:36.283 INFO:tasks.workunit.client.1.smithi111.stdout:4/507: dread d0/f45 [0,4194304] 0 2024-03-23T11:47:36.284 INFO:tasks.workunit.client.1.smithi111.stdout:7/484: rename d14/d1b/d20 to d14/d1b/d1e/d2c/d42/d5e/d88 0 2024-03-23T11:47:36.284 INFO:tasks.workunit.client.1.smithi111.stdout:4/508: dread - d0/dc/d4c/f82 zero size 2024-03-23T11:47:36.285 INFO:tasks.workunit.client.0.smithi063.stdout:3/576: mkdir d13/d22/d2d/d3f/d58/d86/dad 0 2024-03-23T11:47:36.288 INFO:tasks.workunit.client.1.smithi111.stdout:7/485: creat d14/d1b/d1e/d86/f89 x:0 0 0 2024-03-23T11:47:36.288 INFO:tasks.workunit.client.0.smithi063.stdout:3/577: creat d13/d22/d2d/d3f/d58/d97/fae x:0 0 0 2024-03-23T11:47:36.291 INFO:tasks.workunit.client.1.smithi111.stdout:7/486: mkdir d14/d1b/d22/d8a 0 2024-03-23T11:47:36.291 INFO:tasks.workunit.client.0.smithi063.stdout:3/578: symlink d13/d15/laf 0 2024-03-23T11:47:36.295 INFO:tasks.workunit.client.0.smithi063.stdout:3/579: mkdir d13/d22/d2d/d3f/d67/d7b/db0 0 2024-03-23T11:47:36.296 INFO:tasks.workunit.client.0.smithi063.stdout:3/580: fdatasync d13/d15/f3c 0 2024-03-23T11:47:36.297 INFO:tasks.workunit.client.0.smithi063.stdout:3/581: dread - d13/d15/f99 zero size 2024-03-23T11:47:36.300 INFO:tasks.workunit.client.0.smithi063.stdout:5/294: sync 2024-03-23T11:47:36.301 INFO:tasks.workunit.client.0.smithi063.stdout:5/295: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:36.304 INFO:tasks.workunit.client.0.smithi063.stdout:5/296: symlink d2/l65 0 2024-03-23T11:47:36.305 INFO:tasks.workunit.client.0.smithi063.stdout:5/297: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:36.308 INFO:tasks.workunit.client.1.smithi111.stdout:0/493: sync 2024-03-23T11:47:36.309 INFO:tasks.workunit.client.1.smithi111.stdout:1/502: dread d0/d5/f35 [0,4194304] 0 2024-03-23T11:47:36.311 INFO:tasks.workunit.client.0.smithi063.stdout:5/298: symlink d2/d28/d3a/d46/l66 0 2024-03-23T11:47:36.312 INFO:tasks.workunit.client.1.smithi111.stdout:0/494: unlink d5/db/d1f/d32/f3b 0 2024-03-23T11:47:36.313 INFO:tasks.workunit.client.1.smithi111.stdout:0/495: stat d5/db/d1f/d39/f62 0 2024-03-23T11:47:36.317 INFO:tasks.workunit.client.1.smithi111.stdout:1/503: mknod d0/d2b/d69/cad 0 2024-03-23T11:47:36.318 INFO:tasks.workunit.client.1.smithi111.stdout:1/504: fsync d0/d5/dd/d91/fa1 0 2024-03-23T11:47:36.325 INFO:tasks.workunit.client.1.smithi111.stdout:5/554: write d1/d3/d6/d7/f7c [690202,17005] 0 2024-03-23T11:47:36.330 INFO:tasks.workunit.client.0.smithi063.stdout:8/702: dread d6/dd/df/f61 [0,4194304] 0 2024-03-23T11:47:36.337 INFO:tasks.workunit.client.0.smithi063.stdout:8/703: readlink d6/d1f/d54/d63/d90/d5d/d7d/db4/le5 0 2024-03-23T11:47:36.337 INFO:tasks.workunit.client.0.smithi063.stdout:5/299: dwrite d2/d7/d8/d1d/f58 [0,4194304] 0 2024-03-23T11:47:36.337 INFO:tasks.workunit.client.0.smithi063.stdout:3/582: dwrite f12 [0,4194304] 0 2024-03-23T11:47:36.337 INFO:tasks.workunit.client.1.smithi111.stdout:1/505: dwrite d0/d2b/d37/d6a/f8d [0,4194304] 0 2024-03-23T11:47:36.340 INFO:tasks.workunit.client.1.smithi111.stdout:4/509: rename d0/dc/d4c/d4b to d0/dc/d4c/d2a/d97 0 2024-03-23T11:47:36.340 INFO:tasks.workunit.client.1.smithi111.stdout:6/553: write d2/d1a/d1c/d2c/d51/fa9 [1800169,58353] 0 2024-03-23T11:47:36.342 INFO:tasks.workunit.client.0.smithi063.stdout:3/583: symlink d13/d22/d2d/d3f/d5b/lb1 0 2024-03-23T11:47:36.348 INFO:tasks.workunit.client.0.smithi063.stdout:5/300: write d2/d7/d8/de/f41 [2286877,54748] 0 2024-03-23T11:47:36.353 INFO:tasks.workunit.client.0.smithi063.stdout:8/704: rename d6/d1c/c2b to d6/d1f/d54/d63/d90/d5d/cf3 0 2024-03-23T11:47:36.354 INFO:tasks.workunit.client.0.smithi063.stdout:3/584: creat d13/d22/d2d/d3f/d67/d7b/fb2 x:0 0 0 2024-03-23T11:47:36.355 INFO:tasks.workunit.client.1.smithi111.stdout:4/510: creat d0/dc/d78/f98 x:0 0 0 2024-03-23T11:47:36.355 INFO:tasks.workunit.client.1.smithi111.stdout:6/554: creat d2/d1a/d1c/d2c/fb9 x:0 0 0 2024-03-23T11:47:36.356 INFO:tasks.workunit.client.1.smithi111.stdout:3/448: write d0/d1a/d1d/f36 [5109889,109243] 0 2024-03-23T11:47:36.358 INFO:tasks.workunit.client.1.smithi111.stdout:1/506: creat d0/d5/fae x:0 0 0 2024-03-23T11:47:36.359 INFO:tasks.workunit.client.1.smithi111.stdout:9/545: write d0/d4/d6a/f72 [1233203,12065] 0 2024-03-23T11:47:36.360 INFO:tasks.workunit.client.1.smithi111.stdout:6/555: dread - d2/d1f/d45/d49/d8d/f9d zero size 2024-03-23T11:47:36.360 INFO:tasks.workunit.client.1.smithi111.stdout:3/449: chown d0/f39 890 1 2024-03-23T11:47:36.360 INFO:tasks.workunit.client.1.smithi111.stdout:4/511: chown d0/d7/da/f5b 105 1 2024-03-23T11:47:36.362 INFO:tasks.workunit.client.0.smithi063.stdout:5/301: rename d2/c31 to d2/d7/d8/de/d2a/d3b/d43/c67 0 2024-03-23T11:47:36.363 INFO:tasks.workunit.client.1.smithi111.stdout:1/507: write d0/d5/dd/d1e/d54/d87/f90 [842888,99476] 0 2024-03-23T11:47:36.365 INFO:tasks.workunit.client.1.smithi111.stdout:8/447: write d5/d40/f88 [608351,120681] 0 2024-03-23T11:47:36.366 INFO:tasks.workunit.client.1.smithi111.stdout:1/508: stat d0/d5/dd/d1e/d54/d87/f8a 0 2024-03-23T11:47:36.366 INFO:tasks.workunit.client.1.smithi111.stdout:7/487: write d14/d1b/d1e/d2c/d42/d5e/d88/f57 [4298615,123182] 0 2024-03-23T11:47:36.368 INFO:tasks.workunit.client.0.smithi063.stdout:8/705: getdents d6/dd/d59/d7f/dc8 0 2024-03-23T11:47:36.369 INFO:tasks.workunit.client.0.smithi063.stdout:3/585: dread d13/d15/f1f [0,4194304] 0 2024-03-23T11:47:36.369 INFO:tasks.workunit.client.1.smithi111.stdout:1/509: readlink d0/d6/l56 0 2024-03-23T11:47:36.372 INFO:tasks.workunit.client.1.smithi111.stdout:3/450: mkdir d0/d7/d19/d91 0 2024-03-23T11:47:36.375 INFO:tasks.workunit.client.1.smithi111.stdout:4/512: link d0/dc/d4c/d2a/f38 d0/dc/d89/f99 0 2024-03-23T11:47:36.375 INFO:tasks.workunit.client.0.smithi063.stdout:5/302: creat d2/d7/d8/f68 x:0 0 0 2024-03-23T11:47:36.377 INFO:tasks.workunit.client.0.smithi063.stdout:8/706: rename d6/d1f/d54/d63/d90/d15/f65 to d6/d1c/d32/d37/ff4 0 2024-03-23T11:47:36.377 INFO:tasks.workunit.client.1.smithi111.stdout:8/448: rename d5/c8 to d5/d26/d31/d3b/c93 0 2024-03-23T11:47:36.377 INFO:tasks.workunit.client.1.smithi111.stdout:2/578: dwrite d2/d11/d17/d3e/f72 [0,4194304] 0 2024-03-23T11:47:36.378 INFO:tasks.workunit.client.1.smithi111.stdout:0/496: write d5/db/d1f/d48/f58 [225901,98490] 0 2024-03-23T11:47:36.381 INFO:tasks.workunit.client.1.smithi111.stdout:2/579: chown d2/d11/d17/d3e/d5d/dc2 1 1 2024-03-23T11:47:36.385 INFO:tasks.workunit.client.0.smithi063.stdout:3/586: creat d13/d22/d2d/d3f/fb3 x:0 0 0 2024-03-23T11:47:36.386 INFO:tasks.workunit.client.0.smithi063.stdout:8/707: chown d6/fe3 137366091 1 2024-03-23T11:47:36.390 INFO:tasks.workunit.client.1.smithi111.stdout:8/449: write d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [344996,36681] 0 2024-03-23T11:47:36.391 INFO:tasks.workunit.client.1.smithi111.stdout:7/488: mkdir d14/d1b/d1e/d2c/d8b 0 2024-03-23T11:47:36.391 INFO:tasks.workunit.client.1.smithi111.stdout:4/513: chown d0/d7/da/c34 352346618 1 2024-03-23T11:47:36.392 INFO:tasks.workunit.client.1.smithi111.stdout:1/510: mknod d0/d6/d48/d4d/dab/caf 0 2024-03-23T11:47:36.393 INFO:tasks.workunit.client.1.smithi111.stdout:0/497: rename d5/db/d1f/l68 to d5/db/d1f/d39/d64/l9c 0 2024-03-23T11:47:36.398 INFO:tasks.workunit.client.0.smithi063.stdout:3/587: creat d13/d15/d7e/fb4 x:0 0 0 2024-03-23T11:47:36.399 INFO:tasks.workunit.client.1.smithi111.stdout:4/514: chown d0/dc/l47 0 1 2024-03-23T11:47:36.399 INFO:tasks.workunit.client.1.smithi111.stdout:1/511: readlink d0/d5/dd/l39 0 2024-03-23T11:47:36.399 INFO:tasks.workunit.client.0.smithi063.stdout:8/708: write d6/d1f/d54/d63/d90/f35 [3570792,42838] 0 2024-03-23T11:47:36.400 INFO:tasks.workunit.client.1.smithi111.stdout:6/556: dread d2/f31 [4194304,4194304] 0 2024-03-23T11:47:36.401 INFO:tasks.workunit.client.1.smithi111.stdout:0/498: write d5/db/d1f/d47/d59/f99 [727383,54613] 0 2024-03-23T11:47:36.401 INFO:tasks.workunit.client.1.smithi111.stdout:7/489: truncate d14/d1b/d1e/d2c/f31 6028477 0 2024-03-23T11:47:36.405 INFO:tasks.workunit.client.1.smithi111.stdout:3/451: dwrite d0/d1a/d2b/f33 [4194304,4194304] 0 2024-03-23T11:47:36.406 INFO:tasks.workunit.client.1.smithi111.stdout:1/512: rmdir d0/d5/dd/d1e 39 2024-03-23T11:47:36.408 INFO:tasks.workunit.client.0.smithi063.stdout:8/709: rename d6/d1f/d54/d5c/l8f to d6/dd/df/lf5 0 2024-03-23T11:47:36.409 INFO:tasks.workunit.client.1.smithi111.stdout:4/515: write d0/d17/f2c [7163121,121983] 0 2024-03-23T11:47:36.411 INFO:tasks.workunit.client.1.smithi111.stdout:7/490: read d14/d1b/d1e/d2c/f31 [4165615,52307] 0 2024-03-23T11:47:36.412 INFO:tasks.workunit.client.1.smithi111.stdout:9/546: dwrite d0/d31/d41/d65/f6f [0,4194304] 0 2024-03-23T11:47:36.413 INFO:tasks.workunit.client.1.smithi111.stdout:3/452: dread - d0/d1a/d1d/f49 zero size 2024-03-23T11:47:36.415 INFO:tasks.workunit.client.1.smithi111.stdout:0/499: getdents d5/db/d1f/d82 0 2024-03-23T11:47:36.420 INFO:tasks.workunit.client.1.smithi111.stdout:7/491: mknod d14/d37/c8c 0 2024-03-23T11:47:36.420 INFO:tasks.workunit.client.1.smithi111.stdout:0/500: stat d5/db/d1f/d48/c95 0 2024-03-23T11:47:36.422 INFO:tasks.workunit.client.1.smithi111.stdout:9/547: rmdir d0/d4/d10/d29/d7f/d8e 39 2024-03-23T11:47:36.423 INFO:tasks.workunit.client.1.smithi111.stdout:4/516: truncate d0/dc/d4c/d2a/d97/d4d/f8d 148833 0 2024-03-23T11:47:36.423 INFO:tasks.workunit.client.1.smithi111.stdout:2/580: dwrite d2/d11/d17/d3e/d6b/d2e/f6f [0,4194304] 0 2024-03-23T11:47:36.425 INFO:tasks.workunit.client.1.smithi111.stdout:6/557: read d2/d1f/d45/f56 [177647,44439] 0 2024-03-23T11:47:36.426 INFO:tasks.workunit.client.1.smithi111.stdout:0/501: chown d5/db/d1f/d47/l55 1 1 2024-03-23T11:47:36.427 INFO:tasks.workunit.client.1.smithi111.stdout:8/450: dwrite d5/d26/d31/d3b/f80 [8388608,4194304] 0 2024-03-23T11:47:36.431 INFO:tasks.workunit.client.1.smithi111.stdout:7/492: mknod d14/d1b/d1e/d2c/d2e/c8d 0 2024-03-23T11:47:36.431 INFO:tasks.workunit.client.1.smithi111.stdout:7/493: fsync d14/d37/f49 0 2024-03-23T11:47:36.434 INFO:tasks.workunit.client.1.smithi111.stdout:4/517: mkdir d0/dc/d78/d9a 0 2024-03-23T11:47:36.435 INFO:tasks.workunit.client.1.smithi111.stdout:1/513: link d0/d5/dd/l17 d0/d6/d48/d4a/d6c/d83/lb0 0 2024-03-23T11:47:36.435 INFO:tasks.workunit.client.1.smithi111.stdout:5/555: sync 2024-03-23T11:47:36.435 INFO:tasks.workunit.client.1.smithi111.stdout:7/494: chown d14/d1b/d1e/d86/f89 79 1 2024-03-23T11:47:36.437 INFO:tasks.workunit.client.1.smithi111.stdout:8/451: readlink d5/d40/l5b 0 2024-03-23T11:47:36.441 INFO:tasks.workunit.client.0.smithi063.stdout:3/588: dwrite d13/f56 [0,4194304] 0 2024-03-23T11:47:36.445 INFO:tasks.workunit.client.0.smithi063.stdout:8/710: dwrite d6/d1c/f6c [0,4194304] 0 2024-03-23T11:47:36.445 INFO:tasks.workunit.client.1.smithi111.stdout:2/581: creat d2/d11/d17/d3e/d5d/da3/fd5 x:0 0 0 2024-03-23T11:47:36.446 INFO:tasks.workunit.client.0.smithi063.stdout:3/589: chown d13/d22/d2d/d3f/d5b/f77 1360894002 1 2024-03-23T11:47:36.449 INFO:tasks.workunit.client.1.smithi111.stdout:0/502: creat d5/db/d1f/d47/d87/f9d x:0 0 0 2024-03-23T11:47:36.450 INFO:tasks.workunit.client.1.smithi111.stdout:3/453: getdents d0/d1a/d1d/d26/d5a/d64 0 2024-03-23T11:47:36.450 INFO:tasks.workunit.client.0.smithi063.stdout:8/711: mkdir d6/d1f/d3e/df6 0 2024-03-23T11:47:36.451 INFO:tasks.workunit.client.1.smithi111.stdout:4/518: mkdir d0/d95/d9b 0 2024-03-23T11:47:36.453 INFO:tasks.workunit.client.1.smithi111.stdout:6/558: dwrite d2/d1f/d45/f76 [0,4194304] 0 2024-03-23T11:47:36.458 INFO:tasks.workunit.client.1.smithi111.stdout:2/582: dread d2/de/d6c/f65 [0,4194304] 0 2024-03-23T11:47:36.458 INFO:tasks.workunit.client.1.smithi111.stdout:5/556: dread d1/d3/d6/d30/f70 [0,4194304] 0 2024-03-23T11:47:36.461 INFO:tasks.workunit.client.0.smithi063.stdout:8/712: write d6/d1f/f74 [983955,76673] 0 2024-03-23T11:47:36.461 INFO:tasks.workunit.client.1.smithi111.stdout:7/495: rename d14/d37/c46 to d14/d1b/d1e/d86/c8e 0 2024-03-23T11:47:36.461 INFO:tasks.workunit.client.0.smithi063.stdout:3/590: truncate d13/d22/d2d/fa7 442616 0 2024-03-23T11:47:36.466 INFO:tasks.workunit.client.0.smithi063.stdout:8/713: write d6/d1f/d54/d63/d90/f35 [2082370,27026] 0 2024-03-23T11:47:36.469 INFO:tasks.workunit.client.1.smithi111.stdout:8/452: creat d5/d26/d2f/d62/d75/d8d/f94 x:0 0 0 2024-03-23T11:47:36.470 INFO:tasks.workunit.client.1.smithi111.stdout:3/454: write d0/d7/f47 [4195783,2190] 0 2024-03-23T11:47:36.471 INFO:tasks.workunit.client.1.smithi111.stdout:1/514: dwrite d0/d5/f16 [0,4194304] 0 2024-03-23T11:47:36.476 INFO:tasks.workunit.client.1.smithi111.stdout:7/496: write d14/d1b/d1e/d2c/d2e/f34 [1435568,67771] 0 2024-03-23T11:47:36.478 INFO:tasks.workunit.client.1.smithi111.stdout:5/557: rmdir d1/d3/d6/d7/d8c/d9e 39 2024-03-23T11:47:36.478 INFO:tasks.workunit.client.1.smithi111.stdout:4/519: creat d0/dc/d4c/d2a/d97/d4d/f9c x:0 0 0 2024-03-23T11:47:36.480 INFO:tasks.workunit.client.1.smithi111.stdout:8/453: creat d5/d1b/f95 x:0 0 0 2024-03-23T11:47:36.480 INFO:tasks.workunit.client.0.smithi063.stdout:3/591: dread d13/d22/d2d/d3f/f72 [0,4194304] 0 2024-03-23T11:47:36.482 INFO:tasks.workunit.client.1.smithi111.stdout:4/520: write d0/d17/f8c [358582,45754] 0 2024-03-23T11:47:36.484 INFO:tasks.workunit.client.1.smithi111.stdout:5/558: chown d1/d3/d6/d7/d63/d2c/d82/f88 1008182 1 2024-03-23T11:47:36.486 INFO:tasks.workunit.client.1.smithi111.stdout:6/559: rename d2/d80/fb1 to d2/d1f/d45/d49/d8d/fba 0 2024-03-23T11:47:36.487 INFO:tasks.workunit.client.1.smithi111.stdout:3/455: symlink d0/d7/d19/l92 0 2024-03-23T11:47:36.487 INFO:tasks.workunit.client.0.smithi063.stdout:3/592: link d13/d22/d2d/d3f/f72 d13/d22/d2d/d7d/fb5 0 2024-03-23T11:47:36.488 INFO:tasks.workunit.client.1.smithi111.stdout:4/521: rmdir d0/dc/d4c/d83 39 2024-03-23T11:47:36.490 INFO:tasks.workunit.client.0.smithi063.stdout:3/593: read d13/d22/d2d/d3f/d58/d86/f93 [323495,79978] 0 2024-03-23T11:47:36.493 INFO:tasks.workunit.client.1.smithi111.stdout:6/560: mknod d2/d1f/d45/cbb 0 2024-03-23T11:47:36.494 INFO:tasks.workunit.client.1.smithi111.stdout:3/456: chown d0/d7/d19/l3a 124562 1 2024-03-23T11:47:36.496 INFO:tasks.workunit.client.1.smithi111.stdout:4/522: creat d0/d95/f9d x:0 0 0 2024-03-23T11:47:36.497 INFO:tasks.workunit.client.1.smithi111.stdout:6/561: fdatasync d2/d1f/faf 0 2024-03-23T11:47:36.499 INFO:tasks.workunit.client.1.smithi111.stdout:1/515: rename d0/d5/dd/d91/fac to d0/d5/dd/fb1 0 2024-03-23T11:47:36.500 INFO:tasks.workunit.client.1.smithi111.stdout:5/559: getdents d1/d3/d6/d7/d8c 0 2024-03-23T11:47:36.502 INFO:tasks.workunit.client.0.smithi063.stdout:3/594: write d13/d22/d2d/fa1 [3345252,97811] 0 2024-03-23T11:47:36.506 INFO:tasks.workunit.client.1.smithi111.stdout:7/497: dwrite d14/d37/f3f [0,4194304] 0 2024-03-23T11:47:36.506 INFO:tasks.workunit.client.1.smithi111.stdout:1/516: read - d0/d5/dd/d91/f9d zero size 2024-03-23T11:47:36.506 INFO:tasks.workunit.client.0.smithi063.stdout:3/595: rmdir d13/d22/d2d/d3f/d5b 39 2024-03-23T11:47:36.509 INFO:tasks.workunit.client.1.smithi111.stdout:1/517: stat d0/d6/fc 0 2024-03-23T11:47:36.510 INFO:tasks.workunit.client.0.smithi063.stdout:3/596: creat d13/d22/d2d/d3f/d58/d86/dad/fb6 x:0 0 0 2024-03-23T11:47:36.515 INFO:tasks.workunit.client.1.smithi111.stdout:0/503: write d5/db/d1f/f43 [1102044,90685] 0 2024-03-23T11:47:36.515 INFO:tasks.workunit.client.0.smithi063.stdout:2/366: sync 2024-03-23T11:47:36.516 INFO:tasks.workunit.client.1.smithi111.stdout:7/498: fdatasync d14/d1b/f52 0 2024-03-23T11:47:36.516 INFO:tasks.workunit.client.1.smithi111.stdout:6/562: rename d2/d1f/d45/d86/f9f to d2/d1f/d45/fbc 0 2024-03-23T11:47:36.517 INFO:tasks.workunit.client.1.smithi111.stdout:1/518: mkdir d0/d6/d48/db2 0 2024-03-23T11:47:36.519 INFO:tasks.workunit.client.1.smithi111.stdout:7/499: dread - d14/d1b/f53 zero size 2024-03-23T11:47:36.521 INFO:tasks.workunit.client.0.smithi063.stdout:3/597: write f9 [6447660,16062] 0 2024-03-23T11:47:36.521 INFO:tasks.workunit.client.1.smithi111.stdout:6/563: write d2/fb [6023146,35411] 0 2024-03-23T11:47:36.524 INFO:tasks.workunit.client.1.smithi111.stdout:1/519: unlink d0/d6/d48/d4a/c6b 0 2024-03-23T11:47:36.525 INFO:tasks.workunit.client.0.smithi063.stdout:3/598: creat d13/d22/d2d/fb7 x:0 0 0 2024-03-23T11:47:36.526 INFO:tasks.workunit.client.1.smithi111.stdout:3/457: dwrite d0/d7/f47 [0,4194304] 0 2024-03-23T11:47:36.527 INFO:tasks.workunit.client.0.smithi063.stdout:3/599: write d13/d15/f99 [287865,70210] 0 2024-03-23T11:47:36.527 INFO:tasks.workunit.client.1.smithi111.stdout:7/500: chown d14/d1b/d1e/d2c/d2e/c8d 545908256 1 2024-03-23T11:47:36.530 INFO:tasks.workunit.client.1.smithi111.stdout:3/458: readlink d0/d1a/d1d/d26/l32 0 2024-03-23T11:47:36.532 INFO:tasks.workunit.client.0.smithi063.stdout:3/600: mknod d13/d22/d2d/d3f/d5b/cb8 0 2024-03-23T11:47:36.535 INFO:tasks.workunit.client.1.smithi111.stdout:7/501: fsync d14/d1b/d1e/d2c/d42/d5e/d88/f61 0 2024-03-23T11:47:36.535 INFO:tasks.workunit.client.0.smithi063.stdout:2/367: dwrite d2/d1c/f20 [0,4194304] 0 2024-03-23T11:47:36.535 INFO:tasks.workunit.client.0.smithi063.stdout:3/601: unlink d13/d15/f99 0 2024-03-23T11:47:36.536 INFO:tasks.workunit.client.1.smithi111.stdout:3/459: unlink d0/d1a/d1d/f36 0 2024-03-23T11:47:36.536 INFO:tasks.workunit.client.1.smithi111.stdout:6/564: rename d2/d1a/d1c/d2c/f7c to d2/d35/d70/d98/fbd 0 2024-03-23T11:47:36.538 INFO:tasks.workunit.client.0.smithi063.stdout:2/368: stat d2/l32 0 2024-03-23T11:47:36.540 INFO:tasks.workunit.client.1.smithi111.stdout:7/502: fsync d14/d44/d70/f7c 0 2024-03-23T11:47:36.540 INFO:tasks.workunit.client.1.smithi111.stdout:4/523: dread d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:36.540 INFO:tasks.workunit.client.1.smithi111.stdout:5/560: dwrite d1/d3/d6/d7/f1f [4194304,4194304] 0 2024-03-23T11:47:36.541 INFO:tasks.workunit.client.1.smithi111.stdout:6/565: readlink d2/d73/l78 0 2024-03-23T11:47:36.542 INFO:tasks.workunit.client.0.smithi063.stdout:3/602: getdents d13/d22/d2d/d3f/d67/d7b/db0 0 2024-03-23T11:47:36.546 INFO:tasks.workunit.client.1.smithi111.stdout:6/566: chown d2/d1f/c2b 3 1 2024-03-23T11:47:36.547 INFO:tasks.workunit.client.0.smithi063.stdout:2/369: rename d2/de/c53 to d2/d23/d2e/d52/d6c/c73 0 2024-03-23T11:47:36.550 INFO:tasks.workunit.client.1.smithi111.stdout:6/567: fdatasync d2/f3b 0 2024-03-23T11:47:36.553 INFO:tasks.workunit.client.1.smithi111.stdout:7/503: write d14/d1b/d1e/d2c/d42/f4a [6622551,14189] 0 2024-03-23T11:47:36.556 INFO:tasks.workunit.client.1.smithi111.stdout:5/561: write d1/d3/d6/f2b [1489091,115405] 0 2024-03-23T11:47:36.563 INFO:tasks.workunit.client.1.smithi111.stdout:6/568: write d2/d1f/f92 [76192,108108] 0 2024-03-23T11:47:36.563 INFO:tasks.workunit.client.1.smithi111.stdout:0/504: dwrite d5/db/d4d/f52 [0,4194304] 0 2024-03-23T11:47:36.566 INFO:tasks.workunit.client.1.smithi111.stdout:1/520: dwrite d0/f1 [4194304,4194304] 0 2024-03-23T11:47:36.567 INFO:tasks.workunit.client.1.smithi111.stdout:3/460: write d0/d1a/d2b/d38/f51 [1634128,81480] 0 2024-03-23T11:47:36.568 INFO:tasks.workunit.client.0.smithi063.stdout:2/370: truncate d2/d23/d2e/d52/f6f 2973641 0 2024-03-23T11:47:36.569 INFO:tasks.workunit.client.1.smithi111.stdout:4/524: link d0/c3 d0/dc/d1f/d5e/c9e 0 2024-03-23T11:47:36.571 INFO:tasks.workunit.client.1.smithi111.stdout:7/504: mkdir d14/d1b/d1e/d2c/d42/d5e/d8f 0 2024-03-23T11:47:36.573 INFO:tasks.workunit.client.1.smithi111.stdout:7/505: stat d14/d1b/d1e 0 2024-03-23T11:47:36.574 INFO:tasks.workunit.client.1.smithi111.stdout:1/521: fdatasync d0/d6/f22 0 2024-03-23T11:47:36.574 INFO:tasks.workunit.client.1.smithi111.stdout:5/562: creat d1/d3/d6/d7/d8c/d9e/fbc x:0 0 0 2024-03-23T11:47:36.574 INFO:tasks.workunit.client.1.smithi111.stdout:6/569: symlink d2/d1a/d1c/d2c/lbe 0 2024-03-23T11:47:36.575 INFO:tasks.workunit.client.0.smithi063.stdout:2/371: creat d2/de/d22/f74 x:0 0 0 2024-03-23T11:47:36.575 INFO:tasks.workunit.client.1.smithi111.stdout:0/505: mknod d5/db/d1f/d47/d59/d5b/c9e 0 2024-03-23T11:47:36.577 INFO:tasks.workunit.client.1.smithi111.stdout:7/506: chown d14/d37/c8c 570 1 2024-03-23T11:47:36.578 INFO:tasks.workunit.client.1.smithi111.stdout:3/461: rename d0/d1a/d1d/d26/d5a/d64/c8f to d0/d1a/d2d/c93 0 2024-03-23T11:47:36.580 INFO:tasks.workunit.client.1.smithi111.stdout:0/506: mknod d5/db/d1f/d48/c9f 0 2024-03-23T11:47:36.581 INFO:tasks.workunit.client.1.smithi111.stdout:3/462: truncate d0/d55/f81 848856 0 2024-03-23T11:47:36.581 INFO:tasks.workunit.client.0.smithi063.stdout:2/372: unlink d2/f21 0 2024-03-23T11:47:36.583 INFO:tasks.workunit.client.1.smithi111.stdout:9/548: sync 2024-03-23T11:47:36.584 INFO:tasks.workunit.client.1.smithi111.stdout:4/525: chown d0/dc/d1f/d5e/c9e 25212278 1 2024-03-23T11:47:36.584 INFO:tasks.workunit.client.0.smithi063.stdout:2/373: fsync d2/de/f1b 0 2024-03-23T11:47:36.586 INFO:tasks.workunit.client.1.smithi111.stdout:9/549: truncate d0/d31/d35/f9c 61624 0 2024-03-23T11:47:36.588 INFO:tasks.workunit.client.1.smithi111.stdout:4/526: creat d0/dc/d1f/f9f x:0 0 0 2024-03-23T11:47:36.589 INFO:tasks.workunit.client.0.smithi063.stdout:2/374: unlink d2/d23/d2e/f5e 0 2024-03-23T11:47:36.589 INFO:tasks.workunit.client.1.smithi111.stdout:1/522: creat d0/d5/dd/d1e/d54/fb3 x:0 0 0 2024-03-23T11:47:36.590 INFO:tasks.workunit.client.0.smithi063.stdout:3/603: dread d13/d22/d2d/d78/f92 [0,4194304] 0 2024-03-23T11:47:36.592 INFO:tasks.workunit.client.0.smithi063.stdout:2/375: creat d2/d23/d56/f75 x:0 0 0 2024-03-23T11:47:36.592 INFO:tasks.workunit.client.0.smithi063.stdout:3/604: readlink d13/d22/l7f 0 2024-03-23T11:47:36.592 INFO:tasks.workunit.client.1.smithi111.stdout:3/463: symlink d0/d1a/d2b/d38/l94 0 2024-03-23T11:47:36.593 INFO:tasks.workunit.client.1.smithi111.stdout:4/527: read - d0/dc/d4c/d2a/d97/d4d/f94 zero size 2024-03-23T11:47:36.597 INFO:tasks.workunit.client.1.smithi111.stdout:4/528: chown d0/dc/d30/l12 76707730 1 2024-03-23T11:47:36.598 INFO:tasks.workunit.client.0.smithi063.stdout:3/605: dread d13/d15/d48/fa6 [0,4194304] 0 2024-03-23T11:47:36.601 INFO:tasks.workunit.client.1.smithi111.stdout:0/507: dwrite d5/db/d1f/f79 [4194304,4194304] 0 2024-03-23T11:47:36.601 INFO:tasks.workunit.client.1.smithi111.stdout:1/523: symlink d0/d2b/lb4 0 2024-03-23T11:47:36.603 INFO:tasks.workunit.client.0.smithi063.stdout:2/376: truncate d2/f3a 54563 0 2024-03-23T11:47:36.607 INFO:tasks.workunit.client.0.smithi063.stdout:2/377: write d2/d23/d2e/d52/f71 [932746,80010] 0 2024-03-23T11:47:36.607 INFO:tasks.workunit.client.1.smithi111.stdout:3/464: mknod d0/d1a/d2b/d38/c95 0 2024-03-23T11:47:36.608 INFO:tasks.workunit.client.1.smithi111.stdout:6/570: dwrite d2/d1f/d45/d49/d8d/fba [0,4194304] 0 2024-03-23T11:47:36.608 INFO:tasks.workunit.client.1.smithi111.stdout:5/563: write d1/d3/d6/d8b/f75 [458642,97689] 0 2024-03-23T11:47:36.610 INFO:tasks.workunit.client.1.smithi111.stdout:0/508: mknod d5/db/d1f/d39/d64/ca0 0 2024-03-23T11:47:36.611 INFO:tasks.workunit.client.1.smithi111.stdout:7/507: write f9 [1706683,26385] 0 2024-03-23T11:47:36.611 INFO:tasks.workunit.client.0.smithi063.stdout:3/606: link d13/d22/d2d/d3f/d67/d7b/l83 d13/d22/d2d/d7d/lb9 0 2024-03-23T11:47:36.612 INFO:tasks.workunit.client.1.smithi111.stdout:5/564: chown d1/d3/d6/d8b/d59 2029631 1 2024-03-23T11:47:36.614 INFO:tasks.workunit.client.0.smithi063.stdout:2/378: mknod d2/d23/d2e/d52/d6c/c76 0 2024-03-23T11:47:36.614 INFO:tasks.workunit.client.0.smithi063.stdout:3/607: chown d13/d22/d2d/f3a 52 1 2024-03-23T11:47:36.617 INFO:tasks.workunit.client.0.smithi063.stdout:3/608: write d13/d15/f2a [5603588,1356] 0 2024-03-23T11:47:36.620 INFO:tasks.workunit.client.0.smithi063.stdout:2/379: truncate d2/d23/d56/f75 401664 0 2024-03-23T11:47:36.620 INFO:tasks.workunit.client.0.smithi063.stdout:2/380: fdatasync d2/d23/f41 0 2024-03-23T11:47:36.622 INFO:tasks.workunit.client.0.smithi063.stdout:2/381: fdatasync d2/d46/f70 0 2024-03-23T11:47:36.626 INFO:tasks.workunit.client.1.smithi111.stdout:4/529: link d0/dc/l61 d0/dc/d78/d9a/la0 0 2024-03-23T11:47:36.626 INFO:tasks.workunit.client.1.smithi111.stdout:3/465: rename d0/d1a/d2b/f71 to d0/d1a/d1d/f96 0 2024-03-23T11:47:36.627 INFO:tasks.workunit.client.0.smithi063.stdout:2/382: write d2/f2f [823433,121131] 0 2024-03-23T11:47:36.627 INFO:tasks.workunit.client.1.smithi111.stdout:6/571: dwrite d2/f16 [0,4194304] 0 2024-03-23T11:47:36.630 INFO:tasks.workunit.client.1.smithi111.stdout:0/509: mkdir d5/db/d1f/d19/d45/d4b/da1 0 2024-03-23T11:47:36.631 INFO:tasks.workunit.client.0.smithi063.stdout:2/383: write d2/f29 [525651,82456] 0 2024-03-23T11:47:36.633 INFO:tasks.workunit.client.0.smithi063.stdout:2/384: chown d2/de/l26 6518 1 2024-03-23T11:47:36.641 INFO:tasks.workunit.client.0.smithi063.stdout:3/609: dread d13/d22/d2d/d3f/d5b/f77 [0,4194304] 0 2024-03-23T11:47:36.642 INFO:tasks.workunit.client.1.smithi111.stdout:7/508: creat d14/d1b/d1e/d2c/d2e/d33/f90 x:0 0 0 2024-03-23T11:47:36.642 INFO:tasks.workunit.client.1.smithi111.stdout:1/524: write d0/d5/f5e [775797,893] 0 2024-03-23T11:47:36.642 INFO:tasks.workunit.client.1.smithi111.stdout:2/583: sync 2024-03-23T11:47:36.642 INFO:tasks.workunit.client.1.smithi111.stdout:6/572: creat d2/d1a/fbf x:0 0 0 2024-03-23T11:47:36.645 INFO:tasks.workunit.client.1.smithi111.stdout:9/550: dread d0/f9b [0,4194304] 0 2024-03-23T11:47:36.646 INFO:tasks.workunit.client.1.smithi111.stdout:0/510: mknod d5/db/ca2 0 2024-03-23T11:47:36.647 INFO:tasks.workunit.client.1.smithi111.stdout:7/509: chown d14/d1b/d1e/d2c/d2e/c8d 7 1 2024-03-23T11:47:36.650 INFO:tasks.workunit.client.1.smithi111.stdout:5/565: dread d1/d3/d6/d8b/d59/d6c/f76 [0,4194304] 0 2024-03-23T11:47:36.650 INFO:tasks.workunit.client.1.smithi111.stdout:1/525: chown d0/f1a 3118209 1 2024-03-23T11:47:36.651 INFO:tasks.workunit.client.1.smithi111.stdout:3/466: getdents d0/d1a/d1d/d26 0 2024-03-23T11:47:36.651 INFO:tasks.workunit.client.1.smithi111.stdout:2/584: mkdir d2/de/d6c/dd1/dd6 0 2024-03-23T11:47:36.653 INFO:tasks.workunit.client.0.smithi063.stdout:2/385: dwrite d2/d23/d2e/d52/d44/d4d/f6d [0,4194304] 0 2024-03-23T11:47:36.655 INFO:tasks.workunit.client.1.smithi111.stdout:9/551: fsync d0/d4/d6a/d9a/faa 0 2024-03-23T11:47:36.657 INFO:tasks.workunit.client.0.smithi063.stdout:2/386: dread - d2/de/d22/f4a zero size 2024-03-23T11:47:36.658 INFO:tasks.workunit.client.1.smithi111.stdout:1/526: symlink d0/d2b/lb5 0 2024-03-23T11:47:36.661 INFO:tasks.workunit.client.1.smithi111.stdout:0/511: rename d5/db/d1f/d19/l5d to d5/db/d1f/d19/la3 0 2024-03-23T11:47:36.662 INFO:tasks.workunit.client.0.smithi063.stdout:2/387: dread - d2/d46/d54/f68 zero size 2024-03-23T11:47:36.662 INFO:tasks.workunit.client.1.smithi111.stdout:9/552: chown d0/d4/d10/f74 581170980 1 2024-03-23T11:47:36.663 INFO:tasks.workunit.client.1.smithi111.stdout:2/585: stat d2/d11/d17/d3e/cb1 0 2024-03-23T11:47:36.664 INFO:tasks.workunit.client.1.smithi111.stdout:7/510: write d14/d1b/d1e/d2c/d2e/d33/f90 [785733,7680] 0 2024-03-23T11:47:36.664 INFO:tasks.workunit.client.1.smithi111.stdout:5/566: write d1/d3/d6/d8b/d59/d6c/f81 [501441,74769] 0 2024-03-23T11:47:36.664 INFO:tasks.workunit.client.0.smithi063.stdout:3/610: dwrite d13/d22/d2d/d3f/d58/f70 [0,4194304] 0 2024-03-23T11:47:36.667 INFO:tasks.workunit.client.1.smithi111.stdout:0/512: readlink d5/db/l3a 0 2024-03-23T11:47:36.668 INFO:tasks.workunit.client.0.smithi063.stdout:2/388: mkdir d2/d1c/d5c/d77 0 2024-03-23T11:47:36.669 INFO:tasks.workunit.client.1.smithi111.stdout:6/573: link d2/d1a/d1c/d2c/d51/c39 d2/d35/da7/cc0 0 2024-03-23T11:47:36.669 INFO:tasks.workunit.client.1.smithi111.stdout:4/530: dwrite d0/d17/f6b [0,4194304] 0 2024-03-23T11:47:36.670 INFO:tasks.workunit.client.0.smithi063.stdout:3/611: symlink d13/d22/d2d/d78/lba 0 2024-03-23T11:47:36.671 INFO:tasks.workunit.client.0.smithi063.stdout:7/360: sync 2024-03-23T11:47:36.671 INFO:tasks.workunit.client.0.smithi063.stdout:6/368: sync 2024-03-23T11:47:36.672 INFO:tasks.workunit.client.1.smithi111.stdout:7/511: chown d14/d1b/l24 22 1 2024-03-23T11:47:36.673 INFO:tasks.workunit.client.1.smithi111.stdout:9/553: chown d0/d4/cc 11 1 2024-03-23T11:47:36.673 INFO:tasks.workunit.client.1.smithi111.stdout:5/567: unlink d1/d20/l45 0 2024-03-23T11:47:36.673 INFO:tasks.workunit.client.0.smithi063.stdout:7/361: read d5/d6/f27 [3347729,8874] 0 2024-03-23T11:47:36.674 INFO:tasks.workunit.client.1.smithi111.stdout:2/586: dread - d2/d11/d17/d3e/d6b/f9c zero size 2024-03-23T11:47:36.675 INFO:tasks.workunit.client.1.smithi111.stdout:8/454: sync 2024-03-23T11:47:36.677 INFO:tasks.workunit.client.0.smithi063.stdout:2/389: rename d2/f1e to d2/de/d22/f78 0 2024-03-23T11:47:36.677 INFO:tasks.workunit.client.0.smithi063.stdout:7/362: fsync d5/d5c/d2f/d73/f3c 0 2024-03-23T11:47:36.679 INFO:tasks.workunit.client.0.smithi063.stdout:3/612: mknod d13/d22/d2d/d78/cbb 0 2024-03-23T11:47:36.683 INFO:tasks.workunit.client.1.smithi111.stdout:0/513: dread - d5/f85 zero size 2024-03-23T11:47:36.683 INFO:tasks.workunit.client.1.smithi111.stdout:2/587: chown d2/d32/d98 47139236 1 2024-03-23T11:47:36.684 INFO:tasks.workunit.client.1.smithi111.stdout:9/554: write d0/d4/d10/d29/fa3 [641824,15478] 0 2024-03-23T11:47:36.686 INFO:tasks.workunit.client.1.smithi111.stdout:4/531: chown d0/dc/d4c/c44 180025 1 2024-03-23T11:47:36.689 INFO:tasks.workunit.client.1.smithi111.stdout:1/527: rename d0/d6/d48/d4a/d6c/f77 to d0/d5/dd/d1e/d63/fb6 0 2024-03-23T11:47:36.689 INFO:tasks.workunit.client.0.smithi063.stdout:3/613: dread d13/d22/d2d/fa7 [0,4194304] 0 2024-03-23T11:47:36.691 INFO:tasks.workunit.client.0.smithi063.stdout:7/363: rename d5/d6/f5a to d5/d26/d76/f78 0 2024-03-23T11:47:36.691 INFO:tasks.workunit.client.1.smithi111.stdout:4/532: readlink d0/dc/d4c/l91 0 2024-03-23T11:47:36.691 INFO:tasks.workunit.client.1.smithi111.stdout:3/467: dread d0/d1a/d1d/f43 [0,4194304] 0 2024-03-23T11:47:36.698 INFO:tasks.workunit.client.0.smithi063.stdout:3/614: creat d13/d22/d2d/d3f/d58/d6e/fbc x:0 0 0 2024-03-23T11:47:36.699 INFO:tasks.workunit.client.1.smithi111.stdout:0/514: creat d5/db/d1f/d47/d87/fa4 x:0 0 0 2024-03-23T11:47:36.700 INFO:tasks.workunit.client.0.smithi063.stdout:3/615: chown d13/d22/d2d/d3f/d67 7134 1 2024-03-23T11:47:36.702 INFO:tasks.workunit.client.0.smithi063.stdout:7/364: rename d5/d6/d19/f71 to d5/d5c/d2f/f79 0 2024-03-23T11:47:36.703 INFO:tasks.workunit.client.1.smithi111.stdout:2/588: creat d2/d7/da/dbd/fd7 x:0 0 0 2024-03-23T11:47:36.703 INFO:tasks.workunit.client.0.smithi063.stdout:3/616: chown d13/d22/d2d/d5e 49044757 1 2024-03-23T11:47:36.708 INFO:tasks.workunit.client.1.smithi111.stdout:8/455: dwrite d5/d1b/f68 [0,4194304] 0 2024-03-23T11:47:36.709 INFO:tasks.workunit.client.0.smithi063.stdout:3/617: symlink d13/d22/d2d/d78/lbd 0 2024-03-23T11:47:36.709 INFO:tasks.workunit.client.1.smithi111.stdout:6/574: truncate d2/d1a/f64 6313520 0 2024-03-23T11:47:36.710 INFO:tasks.workunit.client.0.smithi063.stdout:2/390: dread d2/d23/d2e/d52/f6f [0,4194304] 0 2024-03-23T11:47:36.711 INFO:tasks.workunit.client.1.smithi111.stdout:9/555: creat d0/d4/d10/d29/d7f/d8e/fb5 x:0 0 0 2024-03-23T11:47:36.712 INFO:tasks.workunit.client.0.smithi063.stdout:7/365: unlink d5/d6/d39/f62 0 2024-03-23T11:47:36.713 INFO:tasks.workunit.client.1.smithi111.stdout:0/515: creat d5/db/d1f/d39/fa5 x:0 0 0 2024-03-23T11:47:36.713 INFO:tasks.workunit.client.0.smithi063.stdout:3/618: read - d13/d22/d2d/f4f zero size 2024-03-23T11:47:36.713 INFO:tasks.workunit.client.0.smithi063.stdout:6/369: dread d1/dc/d3a/d57/f5c [0,4194304] 0 2024-03-23T11:47:36.715 INFO:tasks.workunit.client.1.smithi111.stdout:7/512: dwrite d14/d37/f3f [0,4194304] 0 2024-03-23T11:47:36.717 INFO:tasks.workunit.client.0.smithi063.stdout:7/366: creat d5/d6/f7a x:0 0 0 2024-03-23T11:47:36.718 INFO:tasks.workunit.client.1.smithi111.stdout:9/556: chown d0/d4/d1c/d2d/d3b/d42/d4f/d95 188349 1 2024-03-23T11:47:36.721 INFO:tasks.workunit.client.1.smithi111.stdout:9/557: write d0/d31/d41/d71/d8f/fa9 [92138,55189] 0 2024-03-23T11:47:36.722 INFO:tasks.workunit.client.1.smithi111.stdout:5/568: rename d1/d3/d6/d7/l27 to d1/d3/d6/d7/d63/d2c/d82/lbd 0 2024-03-23T11:47:36.723 INFO:tasks.workunit.client.0.smithi063.stdout:6/370: creat d1/d15/f70 x:0 0 0 2024-03-23T11:47:36.723 INFO:tasks.workunit.client.1.smithi111.stdout:0/516: mkdir d5/db/d4d/da6 0 2024-03-23T11:47:36.724 INFO:tasks.workunit.client.1.smithi111.stdout:9/558: stat d0/d4/d1c/d2d/d3b 0 2024-03-23T11:47:36.725 INFO:tasks.workunit.client.1.smithi111.stdout:4/533: dread d0/d17/f6b [0,4194304] 0 2024-03-23T11:47:36.725 INFO:tasks.workunit.client.0.smithi063.stdout:7/367: creat d5/d6/d19/d1f/d47/d4f/d5b/f7b x:0 0 0 2024-03-23T11:47:36.726 INFO:tasks.workunit.client.1.smithi111.stdout:2/589: dread d2/f4f [0,4194304] 0 2024-03-23T11:47:36.727 INFO:tasks.workunit.client.0.smithi063.stdout:7/368: chown d5/l14 1011 1 2024-03-23T11:47:36.727 INFO:tasks.workunit.client.1.smithi111.stdout:9/559: readlink d0/l27 0 2024-03-23T11:47:36.731 INFO:tasks.workunit.client.1.smithi111.stdout:3/468: rename d0/d53/f8c to d0/d55/f97 0 2024-03-23T11:47:36.732 INFO:tasks.workunit.client.0.smithi063.stdout:7/369: truncate d5/f12 157958 0 2024-03-23T11:47:36.737 INFO:tasks.workunit.client.1.smithi111.stdout:5/569: mknod d1/d3/d6/d7/d7b/cbe 0 2024-03-23T11:47:36.739 INFO:tasks.workunit.client.1.smithi111.stdout:9/560: readlink d0/d4/d6a/d9a/lae 0 2024-03-23T11:47:36.739 INFO:tasks.workunit.client.0.smithi063.stdout:7/370: creat d5/d5c/d2f/d73/f7c x:0 0 0 2024-03-23T11:47:36.740 INFO:tasks.workunit.client.1.smithi111.stdout:5/570: chown d1/d3/d6/d7/d63/d2c/d43/c4a 27701146 1 2024-03-23T11:47:36.740 INFO:tasks.workunit.client.1.smithi111.stdout:9/561: stat d0/d4/d10/d29 0 2024-03-23T11:47:36.741 INFO:tasks.workunit.client.1.smithi111.stdout:2/590: creat d2/d7/da/dbd/fd8 x:0 0 0 2024-03-23T11:47:36.745 INFO:tasks.workunit.client.0.smithi063.stdout:7/371: creat d5/d6/d19/f7d x:0 0 0 2024-03-23T11:47:36.746 INFO:tasks.workunit.client.1.smithi111.stdout:0/517: rename d5/db/d1f/d19/d45/d4b/da1 to d5/db/d1f/d19/d45/d4b/da7 0 2024-03-23T11:47:36.746 INFO:tasks.workunit.client.1.smithi111.stdout:4/534: symlink d0/la1 0 2024-03-23T11:47:36.746 INFO:tasks.workunit.client.1.smithi111.stdout:2/591: chown d2/d11/d17/d3e/d6b/d2e/f6f 28 1 2024-03-23T11:47:36.746 INFO:tasks.workunit.client.0.smithi063.stdout:3/619: rmdir d13/d22/d2d/d3f/d58/d6e 39 2024-03-23T11:47:36.747 INFO:tasks.workunit.client.1.smithi111.stdout:9/562: fdatasync d0/d4/d10/d29/d7f/d8e/fb5 0 2024-03-23T11:47:36.751 INFO:tasks.workunit.client.1.smithi111.stdout:0/518: write d5/db/d1f/d39/f8f [1781935,10537] 0 2024-03-23T11:47:36.753 INFO:tasks.workunit.client.1.smithi111.stdout:1/528: dread d0/d5/f23 [0,4194304] 0 2024-03-23T11:47:36.754 INFO:tasks.workunit.client.0.smithi063.stdout:6/371: write d1/f8 [44541,2217] 0 2024-03-23T11:47:36.756 INFO:tasks.workunit.client.1.smithi111.stdout:4/535: symlink d0/dc/d1f/la2 0 2024-03-23T11:47:36.757 INFO:tasks.workunit.client.1.smithi111.stdout:1/529: write d0/f1 [44804,59018] 0 2024-03-23T11:47:36.759 INFO:tasks.workunit.client.1.smithi111.stdout:9/563: creat d0/d4/d10/d80/fb6 x:0 0 0 2024-03-23T11:47:36.759 INFO:tasks.workunit.client.0.smithi063.stdout:6/372: rmdir d1/dc/d21/d49 39 2024-03-23T11:47:36.761 INFO:tasks.workunit.client.0.smithi063.stdout:6/373: read - d1/f6e zero size 2024-03-23T11:47:36.766 INFO:tasks.workunit.client.1.smithi111.stdout:1/530: rmdir d0/d5/dd 39 2024-03-23T11:47:36.766 INFO:tasks.workunit.client.1.smithi111.stdout:5/571: dwrite d1/d3/fc [0,4194304] 0 2024-03-23T11:47:36.768 INFO:tasks.workunit.client.0.smithi063.stdout:7/372: dread d5/d6/f27 [0,4194304] 0 2024-03-23T11:47:36.768 INFO:tasks.workunit.client.0.smithi063.stdout:6/374: symlink d1/d13/l71 0 2024-03-23T11:47:36.772 INFO:tasks.workunit.client.1.smithi111.stdout:9/564: creat d0/d4/d1c/d2d/fb7 x:0 0 0 2024-03-23T11:47:36.777 INFO:tasks.workunit.client.0.smithi063.stdout:3/620: dwrite d13/d22/f63 [0,4194304] 0 2024-03-23T11:47:36.779 INFO:tasks.workunit.client.1.smithi111.stdout:9/565: fdatasync d0/d4/d10/d29/d7f/f8a 0 2024-03-23T11:47:36.779 INFO:tasks.workunit.client.1.smithi111.stdout:3/469: dread d0/d7/f1e [0,4194304] 0 2024-03-23T11:47:36.780 INFO:tasks.workunit.client.1.smithi111.stdout:2/592: write d2/d32/fa2 [3167515,34374] 0 2024-03-23T11:47:36.788 INFO:tasks.workunit.client.0.smithi063.stdout:3/621: symlink d13/d15/d48/d9d/lbe 0 2024-03-23T11:47:36.789 INFO:tasks.workunit.client.1.smithi111.stdout:3/470: rename d0/f39 to d0/d72/d7b/f98 0 2024-03-23T11:47:36.790 INFO:tasks.workunit.client.1.smithi111.stdout:9/566: truncate d0/d31/d41/d71/f93 694163 0 2024-03-23T11:47:36.793 INFO:tasks.workunit.client.1.smithi111.stdout:5/572: getdents d1/d3/d6/d7 0 2024-03-23T11:47:36.795 INFO:tasks.workunit.client.0.smithi063.stdout:7/373: dwrite d5/d6/d19/d1f/d47/d4f/d5b/f7b [0,4194304] 0 2024-03-23T11:47:36.796 INFO:tasks.workunit.client.1.smithi111.stdout:4/536: dwrite d0/dc/d4c/d2a/d97/d4d/f94 [0,4194304] 0 2024-03-23T11:47:36.797 INFO:tasks.workunit.client.0.smithi063.stdout:3/622: creat d13/d22/d2d/d3f/d58/d6e/fbf x:0 0 0 2024-03-23T11:47:36.797 INFO:tasks.workunit.client.0.smithi063.stdout:7/374: readlink d5/d5c/l59 0 2024-03-23T11:47:36.798 INFO:tasks.workunit.client.1.smithi111.stdout:1/531: dwrite d0/d5/f16 [4194304,4194304] 0 2024-03-23T11:47:36.799 INFO:tasks.workunit.client.1.smithi111.stdout:2/593: rename d2/f74 to d2/d11/d17/d1f/fd9 0 2024-03-23T11:47:36.800 INFO:tasks.workunit.client.0.smithi063.stdout:7/375: fsync d5/d26/f74 0 2024-03-23T11:47:36.801 INFO:tasks.workunit.client.0.smithi063.stdout:3/623: truncate d13/d22/d2d/fb7 406186 0 2024-03-23T11:47:36.803 INFO:tasks.workunit.client.0.smithi063.stdout:7/376: write d5/d6/d19/d1f/f6d [1036725,52311] 0 2024-03-23T11:47:36.804 INFO:tasks.workunit.client.1.smithi111.stdout:0/519: dread d5/f30 [0,4194304] 0 2024-03-23T11:47:36.805 INFO:tasks.workunit.client.1.smithi111.stdout:5/573: mkdir d1/d20/d5c/d99/dbf 0 2024-03-23T11:47:36.809 INFO:tasks.workunit.client.1.smithi111.stdout:1/532: chown d0/d5/dd/d1e/d54/l68 37847 1 2024-03-23T11:47:36.810 INFO:tasks.workunit.client.0.smithi063.stdout:3/624: symlink d13/d22/d2d/d3f/d58/d86/dad/lc0 0 2024-03-23T11:47:36.811 INFO:tasks.workunit.client.0.smithi063.stdout:7/377: read d5/d6/d19/d1f/d47/d4f/d5b/f66 [347018,8335] 0 2024-03-23T11:47:36.813 INFO:tasks.workunit.client.1.smithi111.stdout:2/594: unlink d2/d11/d17/d3e/d5d/l67 0 2024-03-23T11:47:36.815 INFO:tasks.workunit.client.1.smithi111.stdout:0/520: creat d5/db/d1f/d39/d8b/fa8 x:0 0 0 2024-03-23T11:47:36.816 INFO:tasks.workunit.client.0.smithi063.stdout:3/625: creat d13/d22/d2d/d3f/d67/d7b/fc1 x:0 0 0 2024-03-23T11:47:36.822 INFO:tasks.workunit.client.1.smithi111.stdout:3/471: rename d0/f5d to d0/d1a/d2b/d38/f99 0 2024-03-23T11:47:36.823 INFO:tasks.workunit.client.0.smithi063.stdout:7/378: link d5/d5c/d2f/d73/da/l11 d5/l7e 0 2024-03-23T11:47:36.824 INFO:tasks.workunit.client.1.smithi111.stdout:1/533: fdatasync d0/d2b/f75 0 2024-03-23T11:47:36.826 INFO:tasks.workunit.client.1.smithi111.stdout:3/472: chown d0/d1a/d1d/d26/d5a/d60 11682 1 2024-03-23T11:47:36.827 INFO:tasks.workunit.client.0.smithi063.stdout:3/626: rename d13/d66/f75 to d13/d22/da9/fc2 0 2024-03-23T11:47:36.828 INFO:tasks.workunit.client.0.smithi063.stdout:7/379: read - d5/d6/d19/d1f/d47/d4f/d5b/f75 zero size 2024-03-23T11:47:36.829 INFO:tasks.workunit.client.1.smithi111.stdout:9/567: dread d0/d4/fe [0,4194304] 0 2024-03-23T11:47:36.833 INFO:tasks.workunit.client.1.smithi111.stdout:3/473: symlink d0/d1a/d2b/d38/l9a 0 2024-03-23T11:47:36.833 INFO:tasks.workunit.client.1.smithi111.stdout:0/521: dwrite d5/db/d1f/d32/f3e [0,4194304] 0 2024-03-23T11:47:36.834 INFO:tasks.workunit.client.0.smithi063.stdout:7/380: mknod d5/d6/d19/d1f/d47/c7f 0 2024-03-23T11:47:36.835 INFO:tasks.workunit.client.1.smithi111.stdout:1/534: link d0/l15 d0/d6/d48/d4a/d6c/d96/lb7 0 2024-03-23T11:47:36.837 INFO:tasks.workunit.client.1.smithi111.stdout:3/474: fsync d0/d1a/d1d/d26/f46 0 2024-03-23T11:47:36.837 INFO:tasks.workunit.client.0.smithi063.stdout:7/381: chown d5/d6/d19/d1f/d6c 63206 1 2024-03-23T11:47:36.842 INFO:tasks.workunit.client.1.smithi111.stdout:9/568: truncate d0/d4/d1c/d36/f11 2531499 0 2024-03-23T11:47:36.845 INFO:tasks.workunit.client.1.smithi111.stdout:5/574: write d1/d3/d6/f57 [1706039,94244] 0 2024-03-23T11:47:36.846 INFO:tasks.workunit.client.0.smithi063.stdout:1/333: sync 2024-03-23T11:47:36.846 INFO:tasks.workunit.client.0.smithi063.stdout:9/472: sync 2024-03-23T11:47:36.846 INFO:tasks.workunit.client.0.smithi063.stdout:4/453: sync 2024-03-23T11:47:36.847 INFO:tasks.workunit.client.0.smithi063.stdout:8/714: sync 2024-03-23T11:47:36.847 INFO:tasks.workunit.client.0.smithi063.stdout:0/469: sync 2024-03-23T11:47:36.847 INFO:tasks.workunit.client.0.smithi063.stdout:5/303: sync 2024-03-23T11:47:36.847 INFO:tasks.workunit.client.1.smithi111.stdout:1/535: write d0/d5/dd/d1e/d63/fb6 [1920863,95199] 0 2024-03-23T11:47:36.847 INFO:tasks.workunit.client.1.smithi111.stdout:4/537: dread d0/dc/f35 [0,4194304] 0 2024-03-23T11:47:36.848 INFO:tasks.workunit.client.0.smithi063.stdout:3/627: dwrite f12 [4194304,4194304] 0 2024-03-23T11:47:36.848 INFO:tasks.workunit.client.0.smithi063.stdout:0/470: readlink d2/l3b 0 2024-03-23T11:47:36.848 INFO:tasks.workunit.client.0.smithi063.stdout:8/715: stat d6/d1f/d54/d5c/f6d 0 2024-03-23T11:47:36.851 INFO:tasks.workunit.client.1.smithi111.stdout:6/575: sync 2024-03-23T11:47:36.851 INFO:tasks.workunit.client.0.smithi063.stdout:4/454: stat d5/l7d 0 2024-03-23T11:47:36.853 INFO:tasks.workunit.client.1.smithi111.stdout:0/522: dread d5/db/d1f/d39/f3f [0,4194304] 0 2024-03-23T11:47:36.857 INFO:tasks.workunit.client.1.smithi111.stdout:5/575: write d1/d3/d6/d7/d7b/fa4 [385309,28352] 0 2024-03-23T11:47:36.859 INFO:tasks.workunit.client.1.smithi111.stdout:9/569: symlink d0/d4/d1c/d2d/d3b/lb8 0 2024-03-23T11:47:36.865 INFO:tasks.workunit.client.0.smithi063.stdout:7/382: dwrite d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:36.867 INFO:tasks.workunit.client.1.smithi111.stdout:3/475: dwrite d0/d7/f23 [0,4194304] 0 2024-03-23T11:47:36.868 INFO:tasks.workunit.client.1.smithi111.stdout:9/570: creat d0/d4/d1c/d2d/d3b/d42/d4f/d95/fb9 x:0 0 0 2024-03-23T11:47:36.869 INFO:tasks.workunit.client.1.smithi111.stdout:4/538: creat d0/dc/d1f/d7e/fa3 x:0 0 0 2024-03-23T11:47:36.870 INFO:tasks.workunit.client.1.smithi111.stdout:5/576: truncate d1/d20/d5c/f7e 2628397 0 2024-03-23T11:47:36.870 INFO:tasks.workunit.client.0.smithi063.stdout:3/628: rename d13/d15/d48/f68 to d13/d22/d2d/d7d/fc3 0 2024-03-23T11:47:36.874 INFO:tasks.workunit.client.0.smithi063.stdout:4/455: creat d5/da/d20/d4b/d16/d32/d56/f85 x:0 0 0 2024-03-23T11:47:36.879 INFO:tasks.workunit.client.1.smithi111.stdout:3/476: creat d0/d1a/d2b/d38/d7f/f9b x:0 0 0 2024-03-23T11:47:36.879 INFO:tasks.workunit.client.1.smithi111.stdout:9/571: rmdir d0/d4/d1c/d2d/d3b/d42/d4f 39 2024-03-23T11:47:36.880 INFO:tasks.workunit.client.0.smithi063.stdout:8/716: symlink d6/d1f/d54/d63/d90/d5d/d7d/d9a/lf7 0 2024-03-23T11:47:36.881 INFO:tasks.workunit.client.1.smithi111.stdout:5/577: mknod d1/d3/d6/d8b/d59/d6c/cc0 0 2024-03-23T11:47:36.881 INFO:tasks.workunit.client.1.smithi111.stdout:0/523: dwrite d5/f13 [4194304,4194304] 0 2024-03-23T11:47:36.882 INFO:tasks.workunit.client.0.smithi063.stdout:0/471: dwrite d2/d12/f7e [0,4194304] 0 2024-03-23T11:47:36.884 INFO:tasks.workunit.client.0.smithi063.stdout:8/717: chown d6/dd/d59/daf/dc4/fe9 5944937 1 2024-03-23T11:47:36.888 INFO:tasks.workunit.client.1.smithi111.stdout:6/576: creat d2/d1f/d45/fc1 x:0 0 0 2024-03-23T11:47:36.888 INFO:tasks.workunit.client.1.smithi111.stdout:3/477: creat d0/d1a/d2b/d38/d7f/f9c x:0 0 0 2024-03-23T11:47:36.889 INFO:tasks.workunit.client.0.smithi063.stdout:0/472: chown d2/d12/d32/f89 120 1 2024-03-23T11:47:36.889 INFO:tasks.workunit.client.0.smithi063.stdout:9/473: link d4/d31/l46 d4/d4c/d44/la5 0 2024-03-23T11:47:36.901 INFO:tasks.workunit.client.1.smithi111.stdout:3/478: rename d0/d1a/d2b to d0/d1a/d2b/d9d 22 2024-03-23T11:47:36.901 INFO:tasks.workunit.client.1.smithi111.stdout:9/572: symlink d0/d4/d1c/d2d/d3b/d42/d92/lba 0 2024-03-23T11:47:36.901 INFO:tasks.workunit.client.1.smithi111.stdout:8/456: sync 2024-03-23T11:47:36.901 INFO:tasks.workunit.client.1.smithi111.stdout:7/513: sync 2024-03-23T11:47:36.901 INFO:tasks.workunit.client.1.smithi111.stdout:1/536: dwrite d0/d6/f3c [0,4194304] 0 2024-03-23T11:47:36.905 INFO:tasks.workunit.client.1.smithi111.stdout:2/595: dread d2/de/d12/f2c [0,4194304] 0 2024-03-23T11:47:36.907 INFO:tasks.workunit.client.1.smithi111.stdout:5/578: truncate d1/d3/d6/d7/f64 2347535 0 2024-03-23T11:47:36.908 INFO:tasks.workunit.client.0.smithi063.stdout:8/718: mknod d6/d1f/d54/d63/d90/d5d/d7d/d9a/cf8 0 2024-03-23T11:47:36.910 INFO:tasks.workunit.client.1.smithi111.stdout:4/539: dwrite d0/dc/d4c/d2a/d97/d4d/f51 [0,4194304] 0 2024-03-23T11:47:36.911 INFO:tasks.workunit.client.1.smithi111.stdout:7/514: symlink d14/d1b/d1e/d86/l91 0 2024-03-23T11:47:36.911 INFO:tasks.workunit.client.0.smithi063.stdout:0/473: creat d2/d6e/d8c/fae x:0 0 0 2024-03-23T11:47:36.911 INFO:tasks.workunit.client.0.smithi063.stdout:4/456: dwrite d5/da/d20/f3c [0,4194304] 0 2024-03-23T11:47:36.912 INFO:tasks.workunit.client.1.smithi111.stdout:2/596: write d2/d7/da/dbd/fd8 [597408,83760] 0 2024-03-23T11:47:36.913 INFO:tasks.workunit.client.1.smithi111.stdout:1/537: mknod d0/d6/d48/cb8 0 2024-03-23T11:47:36.914 INFO:tasks.workunit.client.1.smithi111.stdout:3/479: symlink d0/d1a/d4f/d90/l9e 0 2024-03-23T11:47:36.915 INFO:tasks.workunit.client.1.smithi111.stdout:0/524: link d5/db/d1f/d19/l1b d5/db/d1f/d32/la9 0 2024-03-23T11:47:36.916 INFO:tasks.workunit.client.1.smithi111.stdout:6/577: getdents d2/d35/d70/d98 0 2024-03-23T11:47:36.919 INFO:tasks.workunit.client.0.smithi063.stdout:4/457: chown d5/d1e/d2b/f54 17653228 1 2024-03-23T11:47:36.920 INFO:tasks.workunit.client.1.smithi111.stdout:1/538: write d0/f59 [4760088,125625] 0 2024-03-23T11:47:36.921 INFO:tasks.workunit.client.0.smithi063.stdout:0/474: mkdir d2/d14/d36/d77/d94/daf 0 2024-03-23T11:47:36.922 INFO:tasks.workunit.client.1.smithi111.stdout:5/579: read d1/d3/d6/d7/d63/d2c/d82/d66/f71 [412430,84458] 0 2024-03-23T11:47:36.925 INFO:tasks.workunit.client.0.smithi063.stdout:5/304: dread d2/d17/f19 [0,4194304] 0 2024-03-23T11:47:36.926 INFO:tasks.workunit.client.1.smithi111.stdout:5/580: fdatasync d1/d3/d6/f57 0 2024-03-23T11:47:36.931 INFO:tasks.workunit.client.1.smithi111.stdout:2/597: mknod d2/de/cda 0 2024-03-23T11:47:36.932 INFO:tasks.workunit.client.0.smithi063.stdout:9/474: dread f2 [0,4194304] 0 2024-03-23T11:47:36.932 INFO:tasks.workunit.client.1.smithi111.stdout:0/525: symlink d5/db/d1f/d19/d45/d4b/laa 0 2024-03-23T11:47:36.933 INFO:tasks.workunit.client.0.smithi063.stdout:4/458: mkdir d5/da/d20/d4b/d16/d50/d86 0 2024-03-23T11:47:36.935 INFO:tasks.workunit.client.0.smithi063.stdout:7/383: write d5/f12 [853260,9280] 0 2024-03-23T11:47:36.937 INFO:tasks.workunit.client.0.smithi063.stdout:3/629: dread d13/d22/f3b [0,4194304] 0 2024-03-23T11:47:36.939 INFO:tasks.workunit.client.1.smithi111.stdout:8/457: dwrite f1 [0,4194304] 0 2024-03-23T11:47:36.943 INFO:tasks.workunit.client.1.smithi111.stdout:1/539: unlink d0/d2b/d69/l76 0 2024-03-23T11:47:36.945 INFO:tasks.workunit.client.0.smithi063.stdout:5/305: symlink d2/d7/d8/de/d47/l69 0 2024-03-23T11:47:36.946 INFO:tasks.workunit.client.1.smithi111.stdout:9/573: truncate d0/d4/d1c/d2d/d3b/d42/d4f/f5c 1393503 0 2024-03-23T11:47:36.946 INFO:tasks.workunit.client.0.smithi063.stdout:9/475: write d4/d4c/d44/f7e [1042433,4577] 0 2024-03-23T11:47:36.948 INFO:tasks.workunit.client.0.smithi063.stdout:7/384: dread - d5/d6/d19/d1f/d47/d4f/d5b/f75 zero size 2024-03-23T11:47:36.950 INFO:tasks.workunit.client.0.smithi063.stdout:8/719: dwrite d6/dd/d59/daf/dc4/fe9 [0,4194304] 0 2024-03-23T11:47:36.950 INFO:tasks.workunit.client.0.smithi063.stdout:0/475: link d2/d4/l99 d2/d14/d36/d77/d94/da6/d28/d46/lb0 0 2024-03-23T11:47:36.951 INFO:tasks.workunit.client.1.smithi111.stdout:7/515: link d14/d37/d54/l5d d14/d1b/d1e/d2c/d42/d5e/l92 0 2024-03-23T11:47:36.951 INFO:tasks.workunit.client.1.smithi111.stdout:4/540: dread d0/dc/d30/ff [0,4194304] 0 2024-03-23T11:47:36.952 INFO:tasks.workunit.client.1.smithi111.stdout:2/598: write d2/d7/da/d19/f77 [2163536,120295] 0 2024-03-23T11:47:36.952 INFO:tasks.workunit.client.1.smithi111.stdout:0/526: readlink d5/db/d1f/l2f 0 2024-03-23T11:47:36.954 INFO:tasks.workunit.client.0.smithi063.stdout:3/630: mkdir d13/d22/d2d/d3f/d58/d6e/dc4 0 2024-03-23T11:47:36.955 INFO:tasks.workunit.client.1.smithi111.stdout:3/480: dwrite d0/d1a/d2b/d38/d7f/f9c [0,4194304] 0 2024-03-23T11:47:36.959 INFO:tasks.workunit.client.1.smithi111.stdout:5/581: dwrite d1/d3/f4 [4194304,4194304] 0 2024-03-23T11:47:36.959 INFO:tasks.workunit.client.0.smithi063.stdout:4/459: truncate d5/da/d20/d4b/d1f/f39 3421215 0 2024-03-23T11:47:36.959 INFO:tasks.workunit.client.0.smithi063.stdout:8/720: chown d6/dd/d59/d7f/db5/cdb 419010 1 2024-03-23T11:47:36.959 INFO:tasks.workunit.client.0.smithi063.stdout:0/476: mkdir d2/d12/d32/db1 0 2024-03-23T11:47:36.960 INFO:tasks.workunit.client.0.smithi063.stdout:0/477: chown d2/d12/d32 113271 1 2024-03-23T11:47:36.962 INFO:tasks.workunit.client.1.smithi111.stdout:8/458: creat d5/d26/d2f/d62/d7e/d85/f96 x:0 0 0 2024-03-23T11:47:36.967 INFO:tasks.workunit.client.1.smithi111.stdout:2/599: unlink d2/d32/l5f 0 2024-03-23T11:47:36.967 INFO:tasks.workunit.client.1.smithi111.stdout:7/516: creat d14/d1b/d1e/d86/f93 x:0 0 0 2024-03-23T11:47:36.969 INFO:tasks.workunit.client.1.smithi111.stdout:1/540: creat d0/d6/d48/d4a/d6c/fb9 x:0 0 0 2024-03-23T11:47:36.974 INFO:tasks.workunit.client.1.smithi111.stdout:2/600: read - d2/f9f zero size 2024-03-23T11:47:36.974 INFO:tasks.workunit.client.0.smithi063.stdout:0/478: mkdir d2/d14/d36/d77/d94/da6/d25/d93/db2 0 2024-03-23T11:47:36.974 INFO:tasks.workunit.client.1.smithi111.stdout:9/574: rename d0/d4/d1c/d36/d28/f5e to d0/d4/d10/fbb 0 2024-03-23T11:47:36.978 INFO:tasks.workunit.client.1.smithi111.stdout:9/575: stat d0/d31/d41/l96 0 2024-03-23T11:47:36.979 INFO:tasks.workunit.client.0.smithi063.stdout:4/460: link d5/da/c2f d5/d1e/d2b/c87 0 2024-03-23T11:47:36.983 INFO:tasks.workunit.client.0.smithi063.stdout:7/385: read d5/d22/f44 [173108,29167] 0 2024-03-23T11:47:36.984 INFO:tasks.workunit.client.1.smithi111.stdout:7/517: unlink d14/d1b/f53 0 2024-03-23T11:47:36.991 INFO:tasks.workunit.client.1.smithi111.stdout:3/481: dwrite d0/d1a/d2b/f4d [0,4194304] 0 2024-03-23T11:47:36.994 INFO:tasks.workunit.client.0.smithi063.stdout:0/479: read d2/d14/d36/d77/d94/da6/d28/d46/d51/f82 [59507,67366] 0 2024-03-23T11:47:36.997 INFO:tasks.workunit.client.0.smithi063.stdout:4/461: creat d5/da/d72/f88 x:0 0 0 2024-03-23T11:47:37.004 INFO:tasks.workunit.client.1.smithi111.stdout:4/541: dread d0/dc/d89/f99 [4194304,4194304] 0 2024-03-23T11:47:37.004 INFO:tasks.workunit.client.1.smithi111.stdout:7/518: write d14/d37/d50/f62 [57834,66036] 0 2024-03-23T11:47:37.005 INFO:tasks.workunit.client.0.smithi063.stdout:0/480: mknod d2/d6e/d8c/cb3 0 2024-03-23T11:47:37.008 INFO:tasks.workunit.client.1.smithi111.stdout:8/459: dwrite d5/d26/f74 [4194304,4194304] 0 2024-03-23T11:47:37.009 INFO:tasks.workunit.client.1.smithi111.stdout:0/527: dwrite d5/db/d1f/d48/f66 [0,4194304] 0 2024-03-23T11:47:37.009 INFO:tasks.workunit.client.0.smithi063.stdout:9/476: dread d4/f5 [0,4194304] 0 2024-03-23T11:47:37.011 INFO:tasks.workunit.client.0.smithi063.stdout:7/386: link d5/d6/d19/d1f/f6d d5/d6/d19/d1f/d6c/f80 0 2024-03-23T11:47:37.014 INFO:tasks.workunit.client.0.smithi063.stdout:9/477: write d4/d2c/f4a [391452,128341] 0 2024-03-23T11:47:37.014 INFO:tasks.workunit.client.1.smithi111.stdout:6/578: write d2/d35/f44 [3799101,118675] 0 2024-03-23T11:47:37.015 INFO:tasks.workunit.client.0.smithi063.stdout:4/462: creat d5/da/d20/d4b/d16/d50/d86/f89 x:0 0 0 2024-03-23T11:47:37.019 INFO:tasks.workunit.client.1.smithi111.stdout:0/528: stat d5/db/d1f/d32 0 2024-03-23T11:47:37.021 INFO:tasks.workunit.client.0.smithi063.stdout:0/481: unlink d2/d14/d36/d77/d94/da6/d28/f73 0 2024-03-23T11:47:37.022 INFO:tasks.workunit.client.0.smithi063.stdout:3/631: dread d13/d15/f3c [0,4194304] 0 2024-03-23T11:47:37.023 INFO:tasks.workunit.client.1.smithi111.stdout:7/519: write d14/d37/d50/f63 [374798,24114] 0 2024-03-23T11:47:37.025 INFO:tasks.workunit.client.0.smithi063.stdout:3/632: readlink d13/d22/d2d/d3f/d67/l91 0 2024-03-23T11:47:37.027 INFO:tasks.workunit.client.1.smithi111.stdout:5/582: dwrite d1/d3/d6/d8b/d59/d74/fa5 [0,4194304] 0 2024-03-23T11:47:37.032 INFO:tasks.workunit.client.1.smithi111.stdout:3/482: rename d0/d1a/d2b/d38/f3c to d0/d1a/d4f/d90/f9f 0 2024-03-23T11:47:37.034 INFO:tasks.workunit.client.0.smithi063.stdout:9/478: rename d4/d2c to d4/d4c/d44/d64/da6 0 2024-03-23T11:47:37.034 INFO:tasks.workunit.client.1.smithi111.stdout:6/579: fsync d2/d80/f96 0 2024-03-23T11:47:37.034 INFO:tasks.workunit.client.1.smithi111.stdout:4/542: mkdir d0/dc/d89/da4 0 2024-03-23T11:47:37.036 INFO:tasks.workunit.client.1.smithi111.stdout:9/576: dwrite d0/f21 [0,4194304] 0 2024-03-23T11:47:37.038 INFO:tasks.workunit.client.0.smithi063.stdout:4/463: write d5/da/d20/d4b/d29/d38/f55 [3702686,6145] 0 2024-03-23T11:47:37.039 INFO:tasks.workunit.client.1.smithi111.stdout:9/577: stat d0/f5f 0 2024-03-23T11:47:37.039 INFO:tasks.workunit.client.1.smithi111.stdout:0/529: symlink d5/db/d4d/da6/lab 0 2024-03-23T11:47:37.039 INFO:tasks.workunit.client.1.smithi111.stdout:6/580: chown d2/d1f/d45/d49/d8d/fae 475248538 1 2024-03-23T11:47:37.041 INFO:tasks.workunit.client.0.smithi063.stdout:7/387: rename d5/d6/d19/d1f/d47/d4f/d5b/f75 to d5/d6/d19/d1f/d47/d6e/f81 0 2024-03-23T11:47:37.041 INFO:tasks.workunit.client.1.smithi111.stdout:1/541: link d0/d2b/d37/l95 d0/lba 0 2024-03-23T11:47:37.042 INFO:tasks.workunit.client.0.smithi063.stdout:5/306: dwrite d2/d28/f3d [0,4194304] 0 2024-03-23T11:47:37.042 INFO:tasks.workunit.client.1.smithi111.stdout:6/581: chown d2/d1a/d1c/d2c/c7b 174 1 2024-03-23T11:47:37.043 INFO:tasks.workunit.client.0.smithi063.stdout:7/388: dread - d5/d5c/d2f/f79 zero size 2024-03-23T11:47:37.050 INFO:tasks.workunit.client.1.smithi111.stdout:3/483: symlink d0/d1a/d1d/d26/la0 0 2024-03-23T11:47:37.051 INFO:tasks.workunit.client.1.smithi111.stdout:1/542: readlink d0/d2b/l65 0 2024-03-23T11:47:37.053 INFO:tasks.workunit.client.1.smithi111.stdout:2/601: dread d2/de/d12/f43 [0,4194304] 0 2024-03-23T11:47:37.053 INFO:tasks.workunit.client.0.smithi063.stdout:9/479: mknod d4/d4c/d19/ca7 0 2024-03-23T11:47:37.054 INFO:tasks.workunit.client.0.smithi063.stdout:5/307: mknod d2/d7/d2b/c6a 0 2024-03-23T11:47:37.057 INFO:tasks.workunit.client.0.smithi063.stdout:7/389: creat d5/d5c/d2f/d73/d20/f82 x:0 0 0 2024-03-23T11:47:37.058 INFO:tasks.workunit.client.1.smithi111.stdout:4/543: mkdir d0/dc/d78/d9a/da5 0 2024-03-23T11:47:37.059 INFO:tasks.workunit.client.1.smithi111.stdout:3/484: creat d0/d1a/d2b/fa1 x:0 0 0 2024-03-23T11:47:37.060 INFO:tasks.workunit.client.1.smithi111.stdout:0/530: creat d5/db/d1f/d82/fac x:0 0 0 2024-03-23T11:47:37.062 INFO:tasks.workunit.client.1.smithi111.stdout:2/602: truncate d2/d11/d17/d1f/da0/fd4 269668 0 2024-03-23T11:47:37.062 INFO:tasks.workunit.client.0.smithi063.stdout:3/633: link d13/d22/d2d/l46 d13/d22/d2d/d7d/lc5 0 2024-03-23T11:47:37.063 INFO:tasks.workunit.client.1.smithi111.stdout:6/582: rename d2/d80/f9b to d2/d35/da7/fc2 0 2024-03-23T11:47:37.067 INFO:tasks.workunit.client.0.smithi063.stdout:9/480: mknod d4/d31/d9c/ca8 0 2024-03-23T11:47:37.068 INFO:tasks.workunit.client.1.smithi111.stdout:4/544: read d0/dc/d4c/d2a/f70 [689823,34102] 0 2024-03-23T11:47:37.071 INFO:tasks.workunit.client.1.smithi111.stdout:4/545: fsync d0/dc/d1f/f9f 0 2024-03-23T11:47:37.074 INFO:tasks.workunit.client.1.smithi111.stdout:4/546: fdatasync d0/d95/f9d 0 2024-03-23T11:47:37.074 INFO:tasks.workunit.client.1.smithi111.stdout:8/460: dread d5/d26/d31/d3b/f80 [0,4194304] 0 2024-03-23T11:47:37.077 INFO:tasks.workunit.client.1.smithi111.stdout:8/461: chown d5/d40/l5b 3 1 2024-03-23T11:47:37.079 INFO:tasks.workunit.client.1.smithi111.stdout:7/520: dread d14/d1b/d1e/f39 [0,4194304] 0 2024-03-23T11:47:37.081 INFO:tasks.workunit.client.1.smithi111.stdout:8/462: mkdir d5/d40/d97 0 2024-03-23T11:47:37.083 INFO:tasks.workunit.client.1.smithi111.stdout:3/485: dread d0/d72/d7b/f98 [0,4194304] 0 2024-03-23T11:47:37.085 INFO:tasks.workunit.client.0.smithi063.stdout:4/464: truncate d5/da/d20/d4b/d29/d38/f55 1484805 0 2024-03-23T11:47:37.085 INFO:tasks.workunit.client.1.smithi111.stdout:7/521: write d14/d1b/f5a [6564195,41921] 0 2024-03-23T11:47:37.088 INFO:tasks.workunit.client.1.smithi111.stdout:0/531: dread d5/db/d1f/f43 [0,4194304] 0 2024-03-23T11:47:37.088 INFO:tasks.workunit.client.1.smithi111.stdout:8/463: readlink d5/d26/d2f/d62/d75/d4b/d51/d5a/l50 0 2024-03-23T11:47:37.088 INFO:tasks.workunit.client.1.smithi111.stdout:1/543: truncate d0/d6/f3c 3035148 0 2024-03-23T11:47:37.089 INFO:tasks.workunit.client.0.smithi063.stdout:3/634: dwrite d13/d15/d7e/f8f [0,4194304] 0 2024-03-23T11:47:37.090 INFO:tasks.workunit.client.1.smithi111.stdout:6/583: truncate d2/d35/d70/d98/fbd 158852 0 2024-03-23T11:47:37.092 INFO:tasks.workunit.client.0.smithi063.stdout:9/481: rename d4/d4c/d44/d64 to d4/d31/d43/da9 0 2024-03-23T11:47:37.094 INFO:tasks.workunit.client.1.smithi111.stdout:3/486: creat d0/d1a/fa2 x:0 0 0 2024-03-23T11:47:37.095 INFO:tasks.workunit.client.0.smithi063.stdout:4/465: truncate d5/da/d20/d4b/d29/d38/f60 734249 0 2024-03-23T11:47:37.096 INFO:tasks.workunit.client.1.smithi111.stdout:7/522: chown d14/d1b/d1e/d2c/d2e/d33/f3d 910 1 2024-03-23T11:47:37.096 INFO:tasks.workunit.client.1.smithi111.stdout:8/464: mknod d5/d26/d2f/d62/d7e/c98 0 2024-03-23T11:47:37.097 INFO:tasks.workunit.client.1.smithi111.stdout:4/547: dwrite d0/d7/f69 [0,4194304] 0 2024-03-23T11:47:37.097 INFO:tasks.workunit.client.0.smithi063.stdout:4/466: dread - d5/d1e/f75 zero size 2024-03-23T11:47:37.098 INFO:tasks.workunit.client.1.smithi111.stdout:1/544: mkdir d0/d5/dd/d1e/d54/dbb 0 2024-03-23T11:47:37.098 INFO:tasks.workunit.client.1.smithi111.stdout:8/465: readlink d5/d40/l5b 0 2024-03-23T11:47:37.099 INFO:tasks.workunit.client.1.smithi111.stdout:3/487: chown d0/d1a/d1d/d26/f46 0 1 2024-03-23T11:47:37.100 INFO:tasks.workunit.client.1.smithi111.stdout:8/466: chown d5/f22 15542734 1 2024-03-23T11:47:37.101 INFO:tasks.workunit.client.1.smithi111.stdout:6/584: write d2/d1f/d45/fbc [688254,77297] 0 2024-03-23T11:47:37.101 INFO:tasks.workunit.client.1.smithi111.stdout:3/488: fsync d0/d1a/f28 0 2024-03-23T11:47:37.101 INFO:tasks.workunit.client.0.smithi063.stdout:9/482: truncate d4/d31/d43/da9/da6/f45 72820 0 2024-03-23T11:47:37.102 INFO:tasks.workunit.client.0.smithi063.stdout:7/390: link d5/d6/d19/d1f/l68 d5/d6/d19/d1f/d47/d4f/l83 0 2024-03-23T11:47:37.103 INFO:tasks.workunit.client.0.smithi063.stdout:4/467: dread - d5/da/d20/d4b/d16/d50/f64 zero size 2024-03-23T11:47:37.104 INFO:tasks.workunit.client.1.smithi111.stdout:1/545: creat d0/d2b/d69/fbc x:0 0 0 2024-03-23T11:47:37.104 INFO:tasks.workunit.client.1.smithi111.stdout:2/603: dread d2/d7/fd [0,4194304] 0 2024-03-23T11:47:37.104 INFO:tasks.workunit.client.1.smithi111.stdout:4/548: mknod d0/dc/ca6 0 2024-03-23T11:47:37.107 INFO:tasks.workunit.client.1.smithi111.stdout:8/467: symlink d5/d26/d2f/d62/l99 0 2024-03-23T11:47:37.109 INFO:tasks.workunit.client.1.smithi111.stdout:7/523: rename d14/d1b/d1e/d2c/d42/d5e/d88/f61 to d14/d1b/d1e/d2c/d42/d5e/d88/f94 0 2024-03-23T11:47:37.110 INFO:tasks.workunit.client.1.smithi111.stdout:8/468: readlink d5/d26/d31/d5e/d4a/l82 0 2024-03-23T11:47:37.113 INFO:tasks.workunit.client.0.smithi063.stdout:5/308: dread d2/f33 [0,4194304] 0 2024-03-23T11:47:37.113 INFO:tasks.workunit.client.0.smithi063.stdout:4/468: write d5/d1e/d2b/f54 [3180302,40999] 0 2024-03-23T11:47:37.114 INFO:tasks.workunit.client.1.smithi111.stdout:8/469: chown d5/d26/d2f/d62/d75/d4b/d51/d5a/f36 3062 1 2024-03-23T11:47:37.118 INFO:tasks.workunit.client.1.smithi111.stdout:6/585: dwrite d2/d1a/f2d [0,4194304] 0 2024-03-23T11:47:37.125 INFO:tasks.workunit.client.0.smithi063.stdout:9/483: link d4/d4c/d44/la5 d4/d31/d43/da9/laa 0 2024-03-23T11:47:37.128 INFO:tasks.workunit.client.0.smithi063.stdout:3/635: dwrite f8 [0,4194304] 0 2024-03-23T11:47:37.129 INFO:tasks.workunit.client.1.smithi111.stdout:1/546: mkdir d0/d6/d48/d4d/dab/dbd 0 2024-03-23T11:47:37.130 INFO:tasks.workunit.client.1.smithi111.stdout:2/604: dread d2/d11/d17/d3e/d6b/f87 [0,4194304] 0 2024-03-23T11:47:37.131 INFO:tasks.workunit.client.0.smithi063.stdout:5/309: mkdir d2/d7/d64/d6b 0 2024-03-23T11:47:37.132 INFO:tasks.workunit.client.1.smithi111.stdout:4/549: creat d0/dc/d78/d9a/fa7 x:0 0 0 2024-03-23T11:47:37.133 INFO:tasks.workunit.client.0.smithi063.stdout:7/391: dwrite d5/d5c/d2f/d73/d20/f82 [0,4194304] 0 2024-03-23T11:47:37.134 INFO:tasks.workunit.client.1.smithi111.stdout:4/550: write d0/dc/d1f/f57 [1957578,54390] 0 2024-03-23T11:47:37.136 INFO:tasks.workunit.client.0.smithi063.stdout:4/469: dread d5/d1e/f82 [0,4194304] 0 2024-03-23T11:47:37.139 INFO:tasks.workunit.client.1.smithi111.stdout:8/470: dwrite d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [0,4194304] 0 2024-03-23T11:47:37.141 INFO:tasks.workunit.client.0.smithi063.stdout:3/636: mkdir d13/d22/d2d/dc6 0 2024-03-23T11:47:37.144 INFO:tasks.workunit.client.0.smithi063.stdout:9/484: mkdir d4/d4c/dab 0 2024-03-23T11:47:37.145 INFO:tasks.workunit.client.0.smithi063.stdout:3/637: fsync d13/d22/d2d/f38 0 2024-03-23T11:47:37.145 INFO:tasks.workunit.client.1.smithi111.stdout:8/471: write d5/d26/d31/d3b/f60 [3333592,93866] 0 2024-03-23T11:47:37.146 INFO:tasks.workunit.client.0.smithi063.stdout:7/392: mknod d5/d6/d39/c84 0 2024-03-23T11:47:37.146 INFO:tasks.workunit.client.0.smithi063.stdout:5/310: creat d2/d7/d8/de/d35/f6c x:0 0 0 2024-03-23T11:47:37.148 INFO:tasks.workunit.client.1.smithi111.stdout:6/586: symlink d2/d73/lc3 0 2024-03-23T11:47:37.149 INFO:tasks.workunit.client.0.smithi063.stdout:3/638: chown d13/d22/d2d/d3f/d58/d86/l90 1 1 2024-03-23T11:47:37.151 INFO:tasks.workunit.client.0.smithi063.stdout:9/485: chown d4/d31/c89 720135 1 2024-03-23T11:47:37.153 INFO:tasks.workunit.client.1.smithi111.stdout:2/605: write d2/d7/da/d19/f49 [672303,61264] 0 2024-03-23T11:47:37.154 INFO:tasks.workunit.client.0.smithi063.stdout:3/639: truncate d13/f9c 250992 0 2024-03-23T11:47:37.157 INFO:tasks.workunit.client.0.smithi063.stdout:3/640: write ff [4408037,120684] 0 2024-03-23T11:47:37.157 INFO:tasks.workunit.client.1.smithi111.stdout:0/532: dread d5/db/d1f/d19/f4c [0,4194304] 0 2024-03-23T11:47:37.161 INFO:tasks.workunit.client.1.smithi111.stdout:0/533: dread - d5/db/d1f/d39/fa5 zero size 2024-03-23T11:47:37.162 INFO:tasks.workunit.client.0.smithi063.stdout:9/486: link d4/f7b d4/d31/d70/d8d/fac 0 2024-03-23T11:47:37.163 INFO:tasks.workunit.client.1.smithi111.stdout:2/606: symlink d2/d11/d17/d3e/d5d/dc2/ldb 0 2024-03-23T11:47:37.163 INFO:tasks.workunit.client.0.smithi063.stdout:3/641: dread - d13/d22/d2d/d3f/d58/d6e/fbf zero size 2024-03-23T11:47:37.167 INFO:tasks.workunit.client.1.smithi111.stdout:1/547: dread d0/d2b/d37/d6a/f8d [0,4194304] 0 2024-03-23T11:47:37.169 INFO:tasks.workunit.client.1.smithi111.stdout:4/551: link d0/dc/d4c/f6f d0/fa8 0 2024-03-23T11:47:37.169 INFO:tasks.workunit.client.1.smithi111.stdout:1/548: write d0/d5/f23 [2874098,36247] 0 2024-03-23T11:47:37.171 INFO:tasks.workunit.client.0.smithi063.stdout:9/487: getdents d4/d31/d9c/d9e 0 2024-03-23T11:47:37.171 INFO:tasks.workunit.client.1.smithi111.stdout:2/607: rename d2/d11/d17/d3e/c83 to d2/d5c/cdc 0 2024-03-23T11:47:37.172 INFO:tasks.workunit.client.1.smithi111.stdout:6/587: getdents d2/d17 0 2024-03-23T11:47:37.173 INFO:tasks.workunit.client.1.smithi111.stdout:4/552: creat d0/d95/fa9 x:0 0 0 2024-03-23T11:47:37.176 INFO:tasks.workunit.client.0.smithi063.stdout:3/642: dread d13/d22/d2d/d7d/fc3 [0,4194304] 0 2024-03-23T11:47:37.176 INFO:tasks.workunit.client.1.smithi111.stdout:0/534: creat d5/db/d1f/d19/d45/fad x:0 0 0 2024-03-23T11:47:37.180 INFO:tasks.workunit.client.0.smithi063.stdout:3/643: dread - d13/d22/d2d/f4f zero size 2024-03-23T11:47:37.180 INFO:tasks.workunit.client.1.smithi111.stdout:6/588: link d2/d35/da7/fc2 d2/d35/d70/fc4 0 2024-03-23T11:47:37.181 INFO:tasks.workunit.client.0.smithi063.stdout:4/470: dread d5/d1e/d2b/f7c [0,4194304] 0 2024-03-23T11:47:37.185 INFO:tasks.workunit.client.0.smithi063.stdout:4/471: write d5/da/d20/d4b/d16/d50/f5f [303173,78049] 0 2024-03-23T11:47:37.186 INFO:tasks.workunit.client.1.smithi111.stdout:2/608: link d2/d7/da/dbd/fd8 d2/d11/d17/d1f/da0/fdd 0 2024-03-23T11:47:37.187 INFO:tasks.workunit.client.1.smithi111.stdout:0/535: symlink d5/db/d1f/d48/lae 0 2024-03-23T11:47:37.188 INFO:tasks.workunit.client.1.smithi111.stdout:2/609: write d2/d11/d17/d3e/d6b/d2e/fca [766678,52399] 0 2024-03-23T11:47:37.188 INFO:tasks.workunit.client.1.smithi111.stdout:0/536: fsync d5/db/f16 0 2024-03-23T11:47:37.190 INFO:tasks.workunit.client.0.smithi063.stdout:9/488: getdents d4/d31/d70/d8d/d90 0 2024-03-23T11:47:37.190 INFO:tasks.workunit.client.1.smithi111.stdout:2/610: write d2/d11/f78 [4983081,24990] 0 2024-03-23T11:47:37.192 INFO:tasks.workunit.client.1.smithi111.stdout:2/611: stat d2/d11/d17/d3e/d6b 0 2024-03-23T11:47:37.192 INFO:tasks.workunit.client.0.smithi063.stdout:3/644: creat d13/d22/d2d/d3f/d67/d7b/fc7 x:0 0 0 2024-03-23T11:47:37.193 INFO:tasks.workunit.client.1.smithi111.stdout:6/589: readlink d2/d1f/d45/d49/lb6 0 2024-03-23T11:47:37.195 INFO:tasks.workunit.client.0.smithi063.stdout:3/645: readlink d13/d22/d2d/l57 0 2024-03-23T11:47:37.196 INFO:tasks.workunit.client.0.smithi063.stdout:7/393: dread d5/d5c/d2f/d73/f3c [0,4194304] 0 2024-03-23T11:47:37.197 INFO:tasks.workunit.client.0.smithi063.stdout:9/489: creat d4/fad x:0 0 0 2024-03-23T11:47:37.199 INFO:tasks.workunit.client.1.smithi111.stdout:0/537: write d5/f30 [4415625,93655] 0 2024-03-23T11:47:37.201 INFO:tasks.workunit.client.1.smithi111.stdout:6/590: mkdir d2/d35/dc5 0 2024-03-23T11:47:37.201 INFO:tasks.workunit.client.0.smithi063.stdout:7/394: symlink d5/d5c/d2f/d73/da/l85 0 2024-03-23T11:47:37.204 INFO:tasks.workunit.client.1.smithi111.stdout:2/612: rename d2/d7/fb4 to d2/d11/d17/d3e/d5d/dc0/fde 0 2024-03-23T11:47:37.205 INFO:tasks.workunit.client.0.smithi063.stdout:7/395: mknod d5/c86 0 2024-03-23T11:47:37.205 INFO:tasks.workunit.client.1.smithi111.stdout:3/489: truncate d0/d7/f2a 1512367 0 2024-03-23T11:47:37.206 INFO:tasks.workunit.client.0.smithi063.stdout:7/396: stat d5/d26/l2b 0 2024-03-23T11:47:37.208 INFO:tasks.workunit.client.1.smithi111.stdout:2/613: dread d2/d11/d17/d3e/d6b/f28 [0,4194304] 0 2024-03-23T11:47:37.209 INFO:tasks.workunit.client.1.smithi111.stdout:0/538: mknod d5/db/d4d/caf 0 2024-03-23T11:47:37.211 INFO:tasks.workunit.client.0.smithi063.stdout:4/472: dwrite d5/da/f19 [0,4194304] 0 2024-03-23T11:47:37.211 INFO:tasks.workunit.client.1.smithi111.stdout:0/539: chown d5/db/d1f/f43 3726 1 2024-03-23T11:47:37.217 INFO:tasks.workunit.client.0.smithi063.stdout:3/646: dwrite d13/d22/d2d/d3f/d67/fa3 [0,4194304] 0 2024-03-23T11:47:37.217 INFO:tasks.workunit.client.1.smithi111.stdout:8/472: truncate d5/d26/d31/d5e/f49 993548 0 2024-03-23T11:47:37.219 INFO:tasks.workunit.client.1.smithi111.stdout:6/591: dwrite d2/d1f/d45/d49/d8d/fae [0,4194304] 0 2024-03-23T11:47:37.219 INFO:tasks.workunit.client.1.smithi111.stdout:0/540: rmdir d5/db/d1f/d47 39 2024-03-23T11:47:37.225 INFO:tasks.workunit.client.1.smithi111.stdout:0/541: unlink d5/f13 0 2024-03-23T11:47:37.227 INFO:tasks.workunit.client.0.smithi063.stdout:4/473: rename d5/da/d72/f79 to d5/da/d20/d4b/d29/d38/d7a/f8a 0 2024-03-23T11:47:37.228 INFO:tasks.workunit.client.1.smithi111.stdout:6/592: readlink d2/d1a/d1c/d2c/d51/l47 0 2024-03-23T11:47:37.231 INFO:tasks.workunit.client.1.smithi111.stdout:0/542: creat d5/db/d1f/d48/fb0 x:0 0 0 2024-03-23T11:47:37.232 INFO:tasks.workunit.client.1.smithi111.stdout:6/593: truncate d2/d1a/d1c/f93 417864 0 2024-03-23T11:47:37.232 INFO:tasks.workunit.client.1.smithi111.stdout:3/490: dread d0/d1a/d1d/d26/f46 [4194304,4194304] 0 2024-03-23T11:47:37.232 INFO:tasks.workunit.client.1.smithi111.stdout:4/553: truncate d0/dc/d4c/d2a/d97/d4d/f94 2051645 0 2024-03-23T11:47:37.232 INFO:tasks.workunit.client.0.smithi063.stdout:7/397: dwrite d5/d5c/d2f/d73/d20/f31 [0,4194304] 0 2024-03-23T11:47:37.235 INFO:tasks.workunit.client.1.smithi111.stdout:4/554: stat d0/dc/d1f/d7e 0 2024-03-23T11:47:37.235 INFO:tasks.workunit.client.1.smithi111.stdout:3/491: dread - d0/d1a/d1d/d26/f7e zero size 2024-03-23T11:47:37.239 INFO:tasks.workunit.client.1.smithi111.stdout:6/594: creat d2/d1f/fc6 x:0 0 0 2024-03-23T11:47:37.239 INFO:tasks.workunit.client.1.smithi111.stdout:5/583: sync 2024-03-23T11:47:37.239 INFO:tasks.workunit.client.1.smithi111.stdout:9/578: sync 2024-03-23T11:47:37.241 INFO:tasks.workunit.client.0.smithi063.stdout:7/398: rmdir d5/d6/d19/d1f/d47/d6e 39 2024-03-23T11:47:37.245 INFO:tasks.workunit.client.1.smithi111.stdout:3/492: mkdir d0/d7/d19/d91/da3 0 2024-03-23T11:47:37.246 INFO:tasks.workunit.client.0.smithi063.stdout:3/647: dwrite f5 [0,4194304] 0 2024-03-23T11:47:37.246 INFO:tasks.workunit.client.1.smithi111.stdout:5/584: truncate d1/d3/d6/d7/d8c/d9e/fbc 377783 0 2024-03-23T11:47:37.246 INFO:tasks.workunit.client.1.smithi111.stdout:6/595: creat d2/d17/fc7 x:0 0 0 2024-03-23T11:47:37.247 INFO:tasks.workunit.client.1.smithi111.stdout:2/614: write d2/d11/d17/d3e/d6b/f2f [587262,64903] 0 2024-03-23T11:47:37.250 INFO:tasks.workunit.client.1.smithi111.stdout:9/579: link d0/d31/d41/d71/d8f/fa9 d0/fbc 0 2024-03-23T11:47:37.253 INFO:tasks.workunit.client.0.smithi063.stdout:3/648: creat d13/d15/d48/d9d/fc8 x:0 0 0 2024-03-23T11:47:37.254 INFO:tasks.workunit.client.0.smithi063.stdout:7/399: mknod d5/d6/d19/d1f/d47/d4f/d5b/d6f/c87 0 2024-03-23T11:47:37.255 INFO:tasks.workunit.client.1.smithi111.stdout:3/493: truncate d0/d1a/d1d/f43 2244787 0 2024-03-23T11:47:37.255 INFO:tasks.workunit.client.1.smithi111.stdout:6/596: chown d2/d1a/d1c/l3a 9598463 1 2024-03-23T11:47:37.257 INFO:tasks.workunit.client.1.smithi111.stdout:0/543: dwrite d5/db/f26 [4194304,4194304] 0 2024-03-23T11:47:37.257 INFO:tasks.workunit.client.1.smithi111.stdout:3/494: stat d0/d7/d19 0 2024-03-23T11:47:37.262 INFO:tasks.workunit.client.0.smithi063.stdout:7/400: dread - d5/d6/d19/d1f/d47/d6e/f81 zero size 2024-03-23T11:47:37.262 INFO:tasks.workunit.client.1.smithi111.stdout:6/597: truncate d2/d1a/d1c/d2c/fa5 71695 0 2024-03-23T11:47:37.267 INFO:tasks.workunit.client.1.smithi111.stdout:5/585: truncate d1/d3/d6/d7/f64 1652798 0 2024-03-23T11:47:37.268 INFO:tasks.workunit.client.0.smithi063.stdout:3/649: getdents d13/d22 0 2024-03-23T11:47:37.268 INFO:tasks.workunit.client.1.smithi111.stdout:0/544: fsync d5/db/d1f/d47/d59/f7d 0 2024-03-23T11:47:37.269 INFO:tasks.workunit.client.1.smithi111.stdout:3/495: symlink d0/d7/d19/la4 0 2024-03-23T11:47:37.269 INFO:tasks.workunit.client.0.smithi063.stdout:3/650: readlink d13/d15/l1e 0 2024-03-23T11:47:37.271 INFO:tasks.workunit.client.0.smithi063.stdout:4/474: dread d5/da/d20/d4b/d16/f5c [0,4194304] 0 2024-03-23T11:47:37.272 INFO:tasks.workunit.client.0.smithi063.stdout:3/651: chown d13/d22/d2d/d3f/d58/d6e/f7a 3495 1 2024-03-23T11:47:37.273 INFO:tasks.workunit.client.1.smithi111.stdout:2/615: fdatasync d2/d11/d17/d3e/d6b/f2f 0 2024-03-23T11:47:37.275 INFO:tasks.workunit.client.1.smithi111.stdout:5/586: unlink d1/d3/d6/d30/db2/l79 0 2024-03-23T11:47:37.278 INFO:tasks.workunit.client.1.smithi111.stdout:0/545: symlink d5/db/d4d/da6/lb1 0 2024-03-23T11:47:37.280 INFO:tasks.workunit.client.1.smithi111.stdout:0/546: write d5/db/d1f/d47/d59/f99 [1768037,67656] 0 2024-03-23T11:47:37.281 INFO:tasks.workunit.client.0.smithi063.stdout:3/652: symlink d13/d22/d2d/d3f/d58/d86/lc9 0 2024-03-23T11:47:37.283 INFO:tasks.workunit.client.1.smithi111.stdout:5/587: creat d1/d3/d6/d30/db2/fc1 x:0 0 0 2024-03-23T11:47:37.284 INFO:tasks.workunit.client.1.smithi111.stdout:0/547: creat d5/db/d4d/fb2 x:0 0 0 2024-03-23T11:47:37.287 INFO:tasks.workunit.client.0.smithi063.stdout:3/653: unlink d13/d22/d2d/d3f/f8a 0 2024-03-23T11:47:37.287 INFO:tasks.workunit.client.1.smithi111.stdout:5/588: mkdir d1/d3/d6/d8b/dc2 0 2024-03-23T11:47:37.289 INFO:tasks.workunit.client.0.smithi063.stdout:3/654: write d13/f9c [651525,117177] 0 2024-03-23T11:47:37.293 INFO:tasks.workunit.client.1.smithi111.stdout:5/589: mkdir d1/d20/d5c/d99/dbf/dc3 0 2024-03-23T11:47:37.296 INFO:tasks.workunit.client.0.smithi063.stdout:7/401: rename d5/d6/d19/d1f/d47/d4f/d5b/d6f to d5/d6/d39/d88 0 2024-03-23T11:47:37.296 INFO:tasks.workunit.client.1.smithi111.stdout:9/580: write d0/d4/d1c/d2d/f63 [473747,86865] 0 2024-03-23T11:47:37.298 INFO:tasks.workunit.client.1.smithi111.stdout:5/590: creat d1/d3/d6/d30/fc4 x:0 0 0 2024-03-23T11:47:37.300 INFO:tasks.workunit.client.0.smithi063.stdout:7/402: creat d5/d5c/d2f/d73/da/f89 x:0 0 0 2024-03-23T11:47:37.302 INFO:tasks.workunit.client.1.smithi111.stdout:4/555: dread d0/d7/da/f13 [4194304,4194304] 0 2024-03-23T11:47:37.303 INFO:tasks.workunit.client.1.smithi111.stdout:9/581: creat d0/d4/fbd x:0 0 0 2024-03-23T11:47:37.304 INFO:tasks.workunit.client.1.smithi111.stdout:0/548: dwrite d5/db/d1f/d48/f58 [0,4194304] 0 2024-03-23T11:47:37.304 INFO:tasks.workunit.client.0.smithi063.stdout:7/403: chown d5/d5c/d2f/d73/da/c52 2535 1 2024-03-23T11:47:37.305 INFO:tasks.workunit.client.1.smithi111.stdout:2/616: dread d2/d7/f38 [0,4194304] 0 2024-03-23T11:47:37.305 INFO:tasks.workunit.client.1.smithi111.stdout:4/556: creat d0/dc/d78/d9a/faa x:0 0 0 2024-03-23T11:47:37.306 INFO:tasks.workunit.client.0.smithi063.stdout:4/475: dwrite d5/da/d20/d4b/d1f/f39 [0,4194304] 0 2024-03-23T11:47:37.306 INFO:tasks.workunit.client.0.smithi063.stdout:3/655: truncate d13/d22/d2d/d3f/d67/fa3 598482 0 2024-03-23T11:47:37.307 INFO:tasks.workunit.client.1.smithi111.stdout:0/549: write d5/db/d1f/d39/f8f [1832817,66967] 0 2024-03-23T11:47:37.310 INFO:tasks.workunit.client.1.smithi111.stdout:9/582: symlink d0/d4/d1c/d2d/d3b/d42/d92/lbe 0 2024-03-23T11:47:37.311 INFO:tasks.workunit.client.0.smithi063.stdout:4/476: write d5/da/d20/d4b/d16/d37/f57 [786239,53394] 0 2024-03-23T11:47:37.312 INFO:tasks.workunit.client.1.smithi111.stdout:9/583: readlink d0/d4/d10/l1f 0 2024-03-23T11:47:37.314 INFO:tasks.workunit.client.0.smithi063.stdout:7/404: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:37.316 INFO:tasks.workunit.client.1.smithi111.stdout:9/584: unlink d0/f5f 0 2024-03-23T11:47:37.319 INFO:tasks.workunit.client.0.smithi063.stdout:4/477: creat d5/da/d20/d4b/f8b x:0 0 0 2024-03-23T11:47:37.319 INFO:tasks.workunit.client.1.smithi111.stdout:4/557: link d0/dc/f14 d0/dc/d30/fab 0 2024-03-23T11:47:37.319 INFO:tasks.workunit.client.0.smithi063.stdout:3/656: rename d13/d22/f63 to d13/d15/fca 0 2024-03-23T11:47:37.320 INFO:tasks.workunit.client.0.smithi063.stdout:7/405: fsync d5/d6/f27 0 2024-03-23T11:47:37.321 INFO:tasks.workunit.client.1.smithi111.stdout:9/585: creat d0/d31/d35/fbf x:0 0 0 2024-03-23T11:47:37.323 INFO:tasks.workunit.client.1.smithi111.stdout:5/591: dwrite d1/d3/d6/d7/f53 [0,4194304] 0 2024-03-23T11:47:37.323 INFO:tasks.workunit.client.1.smithi111.stdout:0/550: link d5/db/d1f/d19/c36 d5/db/d1f/d39/d8b/cb3 0 2024-03-23T11:47:37.324 INFO:tasks.workunit.client.1.smithi111.stdout:2/617: dread d2/d7/f38 [0,4194304] 0 2024-03-23T11:47:37.327 INFO:tasks.workunit.client.1.smithi111.stdout:0/551: chown d5/db/d1f/d48/f66 214 1 2024-03-23T11:47:37.329 INFO:tasks.workunit.client.1.smithi111.stdout:9/586: symlink d0/d4/d10/d29/d7f/d8e/lc0 0 2024-03-23T11:47:37.331 INFO:tasks.workunit.client.1.smithi111.stdout:9/587: chown d0/d31/d35/l6d 52504400 1 2024-03-23T11:47:37.335 INFO:tasks.workunit.client.1.smithi111.stdout:2/618: creat d2/d32/fdf x:0 0 0 2024-03-23T11:47:37.337 INFO:tasks.workunit.client.1.smithi111.stdout:2/619: mkdir d2/de0 0 2024-03-23T11:47:37.340 INFO:tasks.workunit.client.1.smithi111.stdout:2/620: chown d2/d7/cbc 2 1 2024-03-23T11:47:37.342 INFO:tasks.workunit.client.0.smithi063.stdout:7/406: dwrite d5/d5c/d2f/d73/d20/f53 [0,4194304] 0 2024-03-23T11:47:37.343 INFO:tasks.workunit.client.1.smithi111.stdout:4/558: dwrite d0/f45 [0,4194304] 0 2024-03-23T11:47:37.345 INFO:tasks.workunit.client.0.smithi063.stdout:4/478: symlink d5/da/d20/d4b/d29/d38/l8c 0 2024-03-23T11:47:37.345 INFO:tasks.workunit.client.0.smithi063.stdout:3/657: mkdir d13/d22/d2d/d3f/d58/dcb 0 2024-03-23T11:47:37.346 INFO:tasks.workunit.client.0.smithi063.stdout:7/407: read d5/d5c/d2f/d73/f3c [2445701,6473] 0 2024-03-23T11:47:37.347 INFO:tasks.workunit.client.1.smithi111.stdout:2/621: chown d2/d11/d17/d1f/da0/fdd 6852320 1 2024-03-23T11:47:37.351 INFO:tasks.workunit.client.0.smithi063.stdout:7/408: fsync d5/d5c/d2f/f79 0 2024-03-23T11:47:37.352 INFO:tasks.workunit.client.0.smithi063.stdout:4/479: read d5/da/d20/d4b/d16/d50/f70 [8315,8669] 0 2024-03-23T11:47:37.355 INFO:tasks.workunit.client.0.smithi063.stdout:7/409: dread - d5/d5c/d2f/f79 zero size 2024-03-23T11:47:37.355 INFO:tasks.workunit.client.1.smithi111.stdout:5/592: dwrite d1/d3/d6/f57 [0,4194304] 0 2024-03-23T11:47:37.357 INFO:tasks.workunit.client.1.smithi111.stdout:7/524: sync 2024-03-23T11:47:37.359 INFO:tasks.workunit.client.1.smithi111.stdout:2/622: dread - d2/d11/d17/d3e/d6b/f9c zero size 2024-03-23T11:47:37.359 INFO:tasks.workunit.client.1.smithi111.stdout:0/552: dwrite d5/db/d1f/d47/d59/f7d [0,4194304] 0 2024-03-23T11:47:37.360 INFO:tasks.workunit.client.1.smithi111.stdout:4/559: rename d0/dc/f1c to d0/dc/d7a/fac 0 2024-03-23T11:47:37.361 INFO:tasks.workunit.client.0.smithi063.stdout:7/410: fdatasync d5/d6/d19/d1f/d6c/f80 0 2024-03-23T11:47:37.361 INFO:tasks.workunit.client.1.smithi111.stdout:9/588: dwrite d0/d31/d35/fa1 [0,4194304] 0 2024-03-23T11:47:37.363 INFO:tasks.workunit.client.1.smithi111.stdout:5/593: mkdir d1/d20/d5c/dc5 0 2024-03-23T11:47:37.365 INFO:tasks.workunit.client.0.smithi063.stdout:7/411: stat d5/d26/l2b 0 2024-03-23T11:47:37.368 INFO:tasks.workunit.client.1.smithi111.stdout:7/525: mkdir d14/d1b/d1e/d2c/d2e/d95 0 2024-03-23T11:47:37.368 INFO:tasks.workunit.client.0.smithi063.stdout:7/412: stat d5/d5c/d2f/d73 0 2024-03-23T11:47:37.369 INFO:tasks.workunit.client.1.smithi111.stdout:4/560: rmdir d0/d17 39 2024-03-23T11:47:37.369 INFO:tasks.workunit.client.0.smithi063.stdout:7/413: chown d5/c13 372846 1 2024-03-23T11:47:37.371 INFO:tasks.workunit.client.0.smithi063.stdout:4/480: rmdir d5/da/d20/d4b/d16/d83 0 2024-03-23T11:47:37.372 INFO:tasks.workunit.client.0.smithi063.stdout:7/414: fdatasync d5/d26/f74 0 2024-03-23T11:47:37.374 INFO:tasks.workunit.client.1.smithi111.stdout:2/623: rename d2/f4 to d2/d32/d98/fe1 0 2024-03-23T11:47:37.376 INFO:tasks.workunit.client.0.smithi063.stdout:3/658: dwrite d13/d22/d2d/d78/f92 [4194304,4194304] 0 2024-03-23T11:47:37.379 INFO:tasks.workunit.client.1.smithi111.stdout:9/589: mknod d0/d31/d41/d9d/cc1 0 2024-03-23T11:47:37.382 INFO:tasks.workunit.client.1.smithi111.stdout:9/590: fsync d0/d4/d1c/d2d/f2e 0 2024-03-23T11:47:37.382 INFO:tasks.workunit.client.1.smithi111.stdout:4/561: dwrite d0/dc/d4c/f39 [0,4194304] 0 2024-03-23T11:47:37.384 INFO:tasks.workunit.client.1.smithi111.stdout:5/594: rmdir d1/d3/d6/d7/d63/d2c/d82/d66 39 2024-03-23T11:47:37.384 INFO:tasks.workunit.client.1.smithi111.stdout:4/562: stat d0/dc/d78/d9a 0 2024-03-23T11:47:37.384 INFO:tasks.workunit.client.1.smithi111.stdout:9/591: dread - d0/d4/d10/d80/fb6 zero size 2024-03-23T11:47:37.396 INFO:tasks.workunit.client.1.smithi111.stdout:7/526: mknod d14/d1b/d1e/d2c/d42/d5e/c96 0 2024-03-23T11:47:37.396 INFO:tasks.workunit.client.0.smithi063.stdout:3/659: rename d13/d22/d2d/dc6 to d13/d22/d2d/d78/dcc 0 2024-03-23T11:47:37.397 INFO:tasks.workunit.client.0.smithi063.stdout:3/660: stat d13/d15/l89 0 2024-03-23T11:47:37.400 INFO:tasks.workunit.client.1.smithi111.stdout:2/624: rmdir d2/d11 39 2024-03-23T11:47:37.403 INFO:tasks.workunit.client.1.smithi111.stdout:9/592: dwrite d0/d4/d1c/d36/d28/f68 [0,4194304] 0 2024-03-23T11:47:37.409 INFO:tasks.workunit.client.0.smithi063.stdout:4/481: dread d5/da/d20/d4b/d1f/f3f [0,4194304] 0 2024-03-23T11:47:37.411 INFO:tasks.workunit.client.0.smithi063.stdout:7/415: dread f1 [0,4194304] 0 2024-03-23T11:47:37.426 INFO:tasks.workunit.client.0.smithi063.stdout:3/661: mknod d13/d15/d48/d9d/ccd 0 2024-03-23T11:47:37.428 INFO:tasks.workunit.client.0.smithi063.stdout:7/416: stat d5/d6/d19/d1f/c2d 0 2024-03-23T11:47:37.429 INFO:tasks.workunit.client.1.smithi111.stdout:4/563: mkdir d0/dc/d4c/dad 0 2024-03-23T11:47:37.430 INFO:tasks.workunit.client.0.smithi063.stdout:4/482: link d5/l44 d5/da/d20/d4b/d16/d50/d5d/l8d 0 2024-03-23T11:47:37.432 INFO:tasks.workunit.client.0.smithi063.stdout:7/417: creat d5/d5c/d2f/d73/da/f8a x:0 0 0 2024-03-23T11:47:37.434 INFO:tasks.workunit.client.1.smithi111.stdout:0/553: truncate d5/db/d1f/d19/d45/d4b/f69 464782 0 2024-03-23T11:47:37.435 INFO:tasks.workunit.client.1.smithi111.stdout:7/527: symlink d14/d1b/l97 0 2024-03-23T11:47:37.437 INFO:tasks.workunit.client.1.smithi111.stdout:7/528: fsync d14/d44/d4b/f77 0 2024-03-23T11:47:37.440 INFO:tasks.workunit.client.1.smithi111.stdout:4/564: dread d0/f7c [0,4194304] 0 2024-03-23T11:47:37.440 INFO:tasks.workunit.client.1.smithi111.stdout:7/529: chown d14/d1b/d1e/d2c/d42/d5e/d8f 28 1 2024-03-23T11:47:37.441 INFO:tasks.workunit.client.1.smithi111.stdout:0/554: rename d5/f2d to d5/db/d1f/d47/d59/d5b/fb4 0 2024-03-23T11:47:37.447 INFO:tasks.workunit.client.1.smithi111.stdout:7/530: mkdir d14/d1b/d1e/d2c/d2e/d33/d98 0 2024-03-23T11:47:37.450 INFO:tasks.workunit.client.1.smithi111.stdout:0/555: link d5/db/d1f/f79 d5/db/d1f/d47/d59/d5b/fb5 0 2024-03-23T11:47:37.452 INFO:tasks.workunit.client.1.smithi111.stdout:5/595: dwrite d1/d3/d6/d7/d63/d2c/d82/d66/f71 [0,4194304] 0 2024-03-23T11:47:37.452 INFO:tasks.workunit.client.0.smithi063.stdout:3/662: write d13/d22/d2d/d3f/d67/fa3 [1412492,20556] 0 2024-03-23T11:47:37.452 INFO:tasks.workunit.client.0.smithi063.stdout:7/418: write d5/d5c/d2f/d73/d20/f35 [567303,79687] 0 2024-03-23T11:47:37.454 INFO:tasks.workunit.client.0.smithi063.stdout:4/483: dwrite d5/d1e/d2b/f7c [0,4194304] 0 2024-03-23T11:47:37.455 INFO:tasks.workunit.client.1.smithi111.stdout:4/565: getdents d0/dc/d4c/d2a/d97/d4f 0 2024-03-23T11:47:37.459 INFO:tasks.workunit.client.0.smithi063.stdout:7/419: fsync d5/d5c/d2f/d73/f3b 0 2024-03-23T11:47:37.461 INFO:tasks.workunit.client.1.smithi111.stdout:4/566: write d0/dc/d4c/d2a/d97/d4d/f54 [3038982,27018] 0 2024-03-23T11:47:37.461 INFO:tasks.workunit.client.1.smithi111.stdout:0/556: mkdir d5/db/d1f/d47/d59/db6 0 2024-03-23T11:47:37.461 INFO:tasks.workunit.client.0.smithi063.stdout:3/663: rename d13/d15/d48 to d13/d22/d2d/d3e/dce 0 2024-03-23T11:47:37.464 INFO:tasks.workunit.client.0.smithi063.stdout:4/484: mkdir d5/da/d20/d4b/d16/d50/d8e 0 2024-03-23T11:47:37.466 INFO:tasks.workunit.client.1.smithi111.stdout:4/567: write d0/dc/d1f/f8a [1038116,5847] 0 2024-03-23T11:47:37.467 INFO:tasks.workunit.client.1.smithi111.stdout:0/557: write d5/db/f26 [223290,69440] 0 2024-03-23T11:47:37.470 INFO:tasks.workunit.client.0.smithi063.stdout:3/664: fsync d13/d22/d2d/d3e/dce/fa6 0 2024-03-23T11:47:37.472 INFO:tasks.workunit.client.0.smithi063.stdout:3/665: stat d13/d22/d2d/d7d/fc3 0 2024-03-23T11:47:37.472 INFO:tasks.workunit.client.1.smithi111.stdout:4/568: chown d0/dc/d30/l67 3491195 1 2024-03-23T11:47:37.473 INFO:tasks.workunit.client.1.smithi111.stdout:9/593: dread d0/f48 [0,4194304] 0 2024-03-23T11:47:37.473 INFO:tasks.workunit.client.1.smithi111.stdout:7/531: dwrite d14/d37/f3f [0,4194304] 0 2024-03-23T11:47:37.475 INFO:tasks.workunit.client.1.smithi111.stdout:0/558: chown d5/db/d1f/d48/f6a 2188 1 2024-03-23T11:47:37.479 INFO:tasks.workunit.client.0.smithi063.stdout:7/420: getdents d5/d6/d19/d1f/d47/d4f/d5b 0 2024-03-23T11:47:37.480 INFO:tasks.workunit.client.0.smithi063.stdout:3/666: mkdir d13/d22/d2d/d78/dcc/dcf 0 2024-03-23T11:47:37.482 INFO:tasks.workunit.client.1.smithi111.stdout:4/569: rmdir d0/dc/d4c/d2a/d97/d4d 39 2024-03-23T11:47:37.483 INFO:tasks.workunit.client.1.smithi111.stdout:9/594: symlink d0/d31/d41/d71/lc2 0 2024-03-23T11:47:37.484 INFO:tasks.workunit.client.1.smithi111.stdout:0/559: mknod d5/db/d1f/d47/d87/cb7 0 2024-03-23T11:47:37.488 INFO:tasks.workunit.client.1.smithi111.stdout:4/570: symlink d0/d7/da/lae 0 2024-03-23T11:47:37.490 INFO:tasks.workunit.client.1.smithi111.stdout:9/595: mkdir d0/d4/d10/d29/d7f/d8e/dc3 0 2024-03-23T11:47:37.491 INFO:tasks.workunit.client.1.smithi111.stdout:0/560: rmdir d5/db/d1f/d47/d59/d5b 39 2024-03-23T11:47:37.492 INFO:tasks.workunit.client.0.smithi063.stdout:3/667: rename d13/d15/f1c to d13/d22/d2d/d78/dcc/dcf/fd0 0 2024-03-23T11:47:37.492 INFO:tasks.workunit.client.0.smithi063.stdout:4/485: write d5/da/d20/d4b/d29/d38/f55 [1480156,44399] 0 2024-03-23T11:47:37.493 INFO:tasks.workunit.client.1.smithi111.stdout:0/561: write d5/db/d1f/f9b [262805,67002] 0 2024-03-23T11:47:37.494 INFO:tasks.workunit.client.1.smithi111.stdout:0/562: readlink d5/db/d1f/l81 0 2024-03-23T11:47:37.498 INFO:tasks.workunit.client.0.smithi063.stdout:4/486: read d5/da/d20/d4b/d16/d37/f57 [268915,24978] 0 2024-03-23T11:47:37.499 INFO:tasks.workunit.client.1.smithi111.stdout:5/596: dread d1/d20/d5c/f7e [0,4194304] 0 2024-03-23T11:47:37.501 INFO:tasks.workunit.client.1.smithi111.stdout:9/596: read - d0/d4/f9f zero size 2024-03-23T11:47:37.501 INFO:tasks.workunit.client.0.smithi063.stdout:4/487: read - d5/da/d20/d4b/d16/d50/d86/f89 zero size 2024-03-23T11:47:37.502 INFO:tasks.workunit.client.1.smithi111.stdout:7/532: creat d14/d1b/f99 x:0 0 0 2024-03-23T11:47:37.503 INFO:tasks.workunit.client.1.smithi111.stdout:4/571: creat d0/d95/d9b/faf x:0 0 0 2024-03-23T11:47:37.506 INFO:tasks.workunit.client.0.smithi063.stdout:3/668: dread d13/d22/d2d/d3f/d67/f84 [0,4194304] 0 2024-03-23T11:47:37.509 INFO:tasks.workunit.client.1.smithi111.stdout:4/572: unlink d0/f7c 0 2024-03-23T11:47:37.511 INFO:tasks.workunit.client.1.smithi111.stdout:7/533: write d14/d1b/d1e/d2c/d42/d5e/d88/f94 [1079920,4390] 0 2024-03-23T11:47:37.512 INFO:tasks.workunit.client.1.smithi111.stdout:1/549: sync 2024-03-23T11:47:37.512 INFO:tasks.workunit.client.1.smithi111.stdout:8/473: sync 2024-03-23T11:47:37.513 INFO:tasks.workunit.client.0.smithi063.stdout:4/488: write d5/da/d20/d4b/d29/d38/f60 [313879,101566] 0 2024-03-23T11:47:37.514 INFO:tasks.workunit.client.0.smithi063.stdout:3/669: write d13/d22/d2d/d7d/fb5 [547457,98379] 0 2024-03-23T11:47:37.516 INFO:tasks.workunit.client.1.smithi111.stdout:7/534: creat d14/d44/d4b/f9a x:0 0 0 2024-03-23T11:47:37.518 INFO:tasks.workunit.client.1.smithi111.stdout:8/474: symlink d5/d26/d31/d4d/l9a 0 2024-03-23T11:47:37.519 INFO:tasks.workunit.client.0.smithi063.stdout:4/489: mkdir d5/da/d20/d4b/d29/d38/d7a/d8f 0 2024-03-23T11:47:37.519 INFO:tasks.workunit.client.1.smithi111.stdout:5/597: getdents d1/d3/d6/d7/d86 0 2024-03-23T11:47:37.520 INFO:tasks.workunit.client.1.smithi111.stdout:0/563: truncate d5/f30 3722221 0 2024-03-23T11:47:37.521 INFO:tasks.workunit.client.0.smithi063.stdout:3/670: write d13/d22/d2d/fa7 [368043,119169] 0 2024-03-23T11:47:37.522 INFO:tasks.workunit.client.1.smithi111.stdout:4/573: dread d0/d7/da/f13 [4194304,4194304] 0 2024-03-23T11:47:37.522 INFO:tasks.workunit.client.1.smithi111.stdout:7/535: symlink d14/d1b/l9b 0 2024-03-23T11:47:37.522 INFO:tasks.workunit.client.1.smithi111.stdout:9/597: write d0/d4/d1c/d36/f11 [2142174,8000] 0 2024-03-23T11:47:37.524 INFO:tasks.workunit.client.0.smithi063.stdout:3/671: chown d13/d22/d2d/d78/cbb 2002080 1 2024-03-23T11:47:37.525 INFO:tasks.workunit.client.1.smithi111.stdout:5/598: creat d1/d3/d6/d7/fc6 x:0 0 0 2024-03-23T11:47:37.526 INFO:tasks.workunit.client.0.smithi063.stdout:3/672: chown d13/d22/d2d/l1d 1062217 1 2024-03-23T11:47:37.527 INFO:tasks.workunit.client.1.smithi111.stdout:0/564: mknod d5/db/d1f/d47/d59/cb8 0 2024-03-23T11:47:37.527 INFO:tasks.workunit.client.1.smithi111.stdout:1/550: dread d0/d2b/d37/d6a/f8d [0,4194304] 0 2024-03-23T11:47:37.528 INFO:tasks.workunit.client.1.smithi111.stdout:5/599: dread - d1/d3/d6/d8b/d59/d74/f6b zero size 2024-03-23T11:47:37.528 INFO:tasks.workunit.client.1.smithi111.stdout:7/536: mkdir d14/d1b/d1e/d2c/d42/d9c 0 2024-03-23T11:47:37.529 INFO:tasks.workunit.client.0.smithi063.stdout:4/490: write d5/da/d20/d4b/d1f/f3f [530374,104469] 0 2024-03-23T11:47:37.529 INFO:tasks.workunit.client.0.smithi063.stdout:3/673: readlink d13/d22/d2d/d78/lba 0 2024-03-23T11:47:37.530 INFO:tasks.workunit.client.1.smithi111.stdout:0/565: write d5/db/d1f/d19/f23 [661885,52675] 0 2024-03-23T11:47:37.531 INFO:tasks.workunit.client.0.smithi063.stdout:4/491: getdents d5/da/d7e 0 2024-03-23T11:47:37.533 INFO:tasks.workunit.client.1.smithi111.stdout:4/574: rename d0/dc/d78/d9a to d0/dc/d1f/db0 0 2024-03-23T11:47:37.534 INFO:tasks.workunit.client.0.smithi063.stdout:3/674: creat d13/d22/d2d/d3f/d67/fd1 x:0 0 0 2024-03-23T11:47:37.534 INFO:tasks.workunit.client.0.smithi063.stdout:4/492: stat d5/da/d20/d4b/d29/d38/l62 0 2024-03-23T11:47:37.536 INFO:tasks.workunit.client.0.smithi063.stdout:4/493: dread - d5/da/d20/d4b/f8b zero size 2024-03-23T11:47:37.537 INFO:tasks.workunit.client.0.smithi063.stdout:4/494: fdatasync d5/da/f19 0 2024-03-23T11:47:37.538 INFO:tasks.workunit.client.0.smithi063.stdout:3/675: truncate d13/d22/d2d/d3f/f9a 309414 0 2024-03-23T11:47:37.540 INFO:tasks.workunit.client.0.smithi063.stdout:3/676: stat d13/d22/d2d/d3e/c85 0 2024-03-23T11:47:37.543 INFO:tasks.workunit.client.0.smithi063.stdout:4/495: creat d5/f90 x:0 0 0 2024-03-23T11:47:37.544 INFO:tasks.workunit.client.1.smithi111.stdout:5/600: dread d1/d3/d6/f2b [0,4194304] 0 2024-03-23T11:47:37.545 INFO:tasks.workunit.client.1.smithi111.stdout:0/566: fdatasync d5/db/d1f/f1a 0 2024-03-23T11:47:37.547 INFO:tasks.workunit.client.0.smithi063.stdout:3/677: rename d13/d22/d2d/f4f to d13/d15/d7e/fd2 0 2024-03-23T11:47:37.549 INFO:tasks.workunit.client.1.smithi111.stdout:7/537: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/f57 [0,4194304] 0 2024-03-23T11:47:37.552 INFO:tasks.workunit.client.1.smithi111.stdout:0/567: unlink d5/db/d1f/c34 0 2024-03-23T11:47:37.562 INFO:tasks.workunit.client.0.smithi063.stdout:4/496: dwrite d5/da/d20/d4b/d16/d50/f64 [0,4194304] 0 2024-03-23T11:47:37.565 INFO:tasks.workunit.client.1.smithi111.stdout:9/598: dwrite d0/d4/d1c/d36/f59 [4194304,4194304] 0 2024-03-23T11:47:37.573 INFO:tasks.workunit.client.1.smithi111.stdout:0/568: rename d5/db/d1f/l5e to d5/db/d1f/d47/d87/lb9 0 2024-03-23T11:47:37.577 INFO:tasks.workunit.client.1.smithi111.stdout:1/551: dwrite d0/d6/f3c [0,4194304] 0 2024-03-23T11:47:37.579 INFO:tasks.workunit.client.1.smithi111.stdout:9/599: fsync d0/d4/f83 0 2024-03-23T11:47:37.582 INFO:tasks.workunit.client.1.smithi111.stdout:4/575: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:37.582 INFO:tasks.workunit.client.1.smithi111.stdout:0/569: stat d5/db/d1f/d47/d59 0 2024-03-23T11:47:37.585 INFO:tasks.workunit.client.1.smithi111.stdout:5/601: dwrite d1/d3/d6/d7/d63/d2c/d82/d66/d77/f7a [8388608,4194304] 0 2024-03-23T11:47:37.585 INFO:tasks.workunit.client.1.smithi111.stdout:1/552: stat d0/d5/dd/d1e/d63/l6d 0 2024-03-23T11:47:37.586 INFO:tasks.workunit.client.0.smithi063.stdout:4/497: dwrite d5/d1e/d2b/f7c [0,4194304] 0 2024-03-23T11:47:37.588 INFO:tasks.workunit.client.0.smithi063.stdout:3/678: dwrite d13/d22/d2d/d3f/d67/f84 [0,4194304] 0 2024-03-23T11:47:37.588 INFO:tasks.workunit.client.1.smithi111.stdout:4/576: fsync d0/dc/d4c/f39 0 2024-03-23T11:47:37.589 INFO:tasks.workunit.client.1.smithi111.stdout:7/538: dwrite d14/f15 [0,4194304] 0 2024-03-23T11:47:37.592 INFO:tasks.workunit.client.0.smithi063.stdout:4/498: symlink d5/da/d20/d4b/d16/d32/l91 0 2024-03-23T11:47:37.592 INFO:tasks.workunit.client.1.smithi111.stdout:9/600: write d0/d4/d10/d80/fb4 [796033,26920] 0 2024-03-23T11:47:37.594 INFO:tasks.workunit.client.1.smithi111.stdout:5/602: unlink d1/d20/f50 0 2024-03-23T11:47:37.594 INFO:tasks.workunit.client.1.smithi111.stdout:1/553: readlink d0/d2b/d7c/la7 0 2024-03-23T11:47:37.595 INFO:tasks.workunit.client.1.smithi111.stdout:7/539: chown d14/l27 88485 1 2024-03-23T11:47:37.595 INFO:tasks.workunit.client.0.smithi063.stdout:4/499: write d5/d1e/f51 [1973633,20409] 0 2024-03-23T11:47:37.598 INFO:tasks.workunit.client.0.smithi063.stdout:4/500: chown d5/da/d20/d4b/d1f/f39 4 1 2024-03-23T11:47:37.599 INFO:tasks.workunit.client.1.smithi111.stdout:1/554: chown d0/d5/dd/d1e/d54/c88 3699301 1 2024-03-23T11:47:37.600 INFO:tasks.workunit.client.0.smithi063.stdout:4/501: creat d5/da/d72/f92 x:0 0 0 2024-03-23T11:47:37.602 INFO:tasks.workunit.client.1.smithi111.stdout:5/603: write d1/d3/d6/d30/db2/fc1 [758553,38069] 0 2024-03-23T11:47:37.603 INFO:tasks.workunit.client.1.smithi111.stdout:4/577: dread d0/d7/f92 [0,4194304] 0 2024-03-23T11:47:37.603 INFO:tasks.workunit.client.0.smithi063.stdout:4/502: mkdir d5/da/d20/d4b/d29/d38/d93 0 2024-03-23T11:47:37.605 INFO:tasks.workunit.client.0.smithi063.stdout:1/334: sync 2024-03-23T11:47:37.606 INFO:tasks.workunit.client.0.smithi063.stdout:2/391: sync 2024-03-23T11:47:37.606 INFO:tasks.workunit.client.0.smithi063.stdout:6/375: sync 2024-03-23T11:47:37.617 INFO:tasks.workunit.client.1.smithi111.stdout:4/578: fdatasync d0/d17/f8c 0 2024-03-23T11:47:37.617 INFO:tasks.workunit.client.0.smithi063.stdout:4/503: truncate d5/f1c 548097 0 2024-03-23T11:47:37.618 INFO:tasks.workunit.client.0.smithi063.stdout:6/376: truncate d1/f3b 691675 0 2024-03-23T11:47:37.620 INFO:tasks.workunit.client.0.smithi063.stdout:1/335: mknod d3/df/d2f/c70 0 2024-03-23T11:47:37.622 INFO:tasks.workunit.client.0.smithi063.stdout:1/336: readlink d3/d3f/d4d/l55 0 2024-03-23T11:47:37.623 INFO:tasks.workunit.client.0.smithi063.stdout:4/504: mkdir d5/da/d20/d4b/d16/d32/d6f/d94 0 2024-03-23T11:47:37.625 INFO:tasks.workunit.client.0.smithi063.stdout:6/377: mknod d1/c72 0 2024-03-23T11:47:37.626 INFO:tasks.workunit.client.1.smithi111.stdout:4/579: symlink d0/dc/d1f/d7e/lb1 0 2024-03-23T11:47:37.626 INFO:tasks.workunit.client.1.smithi111.stdout:7/540: dwrite f9 [0,4194304] 0 2024-03-23T11:47:37.626 INFO:tasks.workunit.client.0.smithi063.stdout:1/337: unlink d3/d1c/f27 0 2024-03-23T11:47:37.629 INFO:tasks.workunit.client.1.smithi111.stdout:7/541: readlink d14/d1b/l97 0 2024-03-23T11:47:37.633 INFO:tasks.workunit.client.0.smithi063.stdout:4/505: rmdir d5/da/d20/d4b/d29/d38/d7a 39 2024-03-23T11:47:37.634 INFO:tasks.workunit.client.0.smithi063.stdout:3/679: write d13/d22/d2d/f3a [2083956,110227] 0 2024-03-23T11:47:37.637 INFO:tasks.workunit.client.1.smithi111.stdout:5/604: dwrite d1/d3/d6/d7/d8c/d94/fbb [0,4194304] 0 2024-03-23T11:47:37.638 INFO:tasks.workunit.client.0.smithi063.stdout:4/506: chown d5/da/d20/d4b/d16/d37/f53 4 1 2024-03-23T11:47:37.641 INFO:tasks.workunit.client.1.smithi111.stdout:5/605: chown d1/d3/d6/d7/d63/d2c/f51 653291146 1 2024-03-23T11:47:37.642 INFO:tasks.workunit.client.1.smithi111.stdout:9/601: dwrite d0/d4/d1c/d36/f18 [0,4194304] 0 2024-03-23T11:47:37.643 INFO:tasks.workunit.client.0.smithi063.stdout:3/680: stat d13/d22/d2d/l2b 0 2024-03-23T11:47:37.645 INFO:tasks.workunit.client.0.smithi063.stdout:4/507: write d5/da/d20/d4b/d16/d50/d86/f89 [352183,39489] 0 2024-03-23T11:47:37.652 INFO:tasks.workunit.client.0.smithi063.stdout:3/681: read f6 [4395406,74778] 0 2024-03-23T11:47:37.654 INFO:tasks.workunit.client.1.smithi111.stdout:4/580: dwrite d0/d7/f7f [0,4194304] 0 2024-03-23T11:47:37.659 INFO:tasks.workunit.client.0.smithi063.stdout:6/378: dwrite d1/dc/d21/d49/f56 [0,4194304] 0 2024-03-23T11:47:37.665 INFO:tasks.workunit.client.1.smithi111.stdout:5/606: rename d1/d20/c32 to d1/d3/d6/d7/d7b/cc7 0 2024-03-23T11:47:37.666 INFO:tasks.workunit.client.1.smithi111.stdout:9/602: dwrite d0/d4/d10/f51 [0,4194304] 0 2024-03-23T11:47:37.668 INFO:tasks.workunit.client.1.smithi111.stdout:0/570: dwrite d5/db/d1f/f1a [0,4194304] 0 2024-03-23T11:47:37.669 INFO:tasks.workunit.client.1.smithi111.stdout:5/607: truncate d1/d3/d6/d8b/fa3 818641 0 2024-03-23T11:47:37.672 INFO:tasks.workunit.client.0.smithi063.stdout:6/379: mknod d1/dc/d3a/d57/d6d/c73 0 2024-03-23T11:47:37.677 INFO:tasks.workunit.client.0.smithi063.stdout:1/338: dread d3/f11 [0,4194304] 0 2024-03-23T11:47:37.679 INFO:tasks.workunit.client.0.smithi063.stdout:2/392: dread d2/de/d22/f5b [4194304,4194304] 0 2024-03-23T11:47:37.680 INFO:tasks.workunit.client.0.smithi063.stdout:1/339: chown d3/d62/c6b 4836228 1 2024-03-23T11:47:37.682 INFO:tasks.workunit.client.1.smithi111.stdout:4/581: rename d0/d17/c80 to d0/dc/d4c/dad/cb2 0 2024-03-23T11:47:37.683 INFO:tasks.workunit.client.1.smithi111.stdout:4/582: dread - d0/fa8 zero size 2024-03-23T11:47:37.686 INFO:tasks.workunit.client.1.smithi111.stdout:5/608: creat d1/d3/d6/d8b/d59/d6c/fc8 x:0 0 0 2024-03-23T11:47:37.687 INFO:tasks.workunit.client.1.smithi111.stdout:6/598: sync 2024-03-23T11:47:37.690 INFO:tasks.workunit.client.0.smithi063.stdout:1/340: read d3/d1c/f1e [644664,111859] 0 2024-03-23T11:47:37.690 INFO:tasks.workunit.client.0.smithi063.stdout:4/508: write d5/da/d20/d4b/d1f/f26 [21730,126151] 0 2024-03-23T11:47:37.691 INFO:tasks.workunit.client.1.smithi111.stdout:7/542: dwrite d14/d37/d43/f55 [0,4194304] 0 2024-03-23T11:47:37.691 INFO:tasks.workunit.client.1.smithi111.stdout:0/571: mknod d5/db/d1f/d32/cba 0 2024-03-23T11:47:37.691 INFO:tasks.workunit.client.0.smithi063.stdout:2/393: creat d2/d23/d2e/d52/d44/d4d/f79 x:0 0 0 2024-03-23T11:47:37.694 INFO:tasks.workunit.client.0.smithi063.stdout:6/380: getdents d1/d13 0 2024-03-23T11:47:37.694 INFO:tasks.workunit.client.1.smithi111.stdout:0/572: truncate d5/db/d1f/d39/d8b/fa8 510186 0 2024-03-23T11:47:37.696 INFO:tasks.workunit.client.1.smithi111.stdout:0/573: fdatasync d5/db/d1f/d48/fb0 0 2024-03-23T11:47:37.699 INFO:tasks.workunit.client.0.smithi063.stdout:1/341: rename d3/f5a to d3/d1c/d4b/f71 0 2024-03-23T11:47:37.701 INFO:tasks.workunit.client.1.smithi111.stdout:4/583: mkdir d0/d95/d9b/db3 0 2024-03-23T11:47:37.703 INFO:tasks.workunit.client.1.smithi111.stdout:9/603: symlink d0/d4/d1c/d2d/d3b/lc4 0 2024-03-23T11:47:37.704 INFO:tasks.workunit.client.0.smithi063.stdout:2/394: rename d2/d1c/d5c to d2/de/d7a 0 2024-03-23T11:47:37.705 INFO:tasks.workunit.client.0.smithi063.stdout:1/342: chown d3/df/d2f/c46 51692 1 2024-03-23T11:47:37.706 INFO:tasks.workunit.client.0.smithi063.stdout:4/509: dread d5/da/d20/d4b/d16/f5c [0,4194304] 0 2024-03-23T11:47:37.710 INFO:tasks.workunit.client.0.smithi063.stdout:1/343: write d3/f10 [4330044,81921] 0 2024-03-23T11:47:37.710 INFO:tasks.workunit.client.0.smithi063.stdout:2/395: rmdir d2/d23/d2e/d52/d44 39 2024-03-23T11:47:37.711 INFO:tasks.workunit.client.1.smithi111.stdout:2/625: sync 2024-03-23T11:47:37.711 INFO:tasks.workunit.client.1.smithi111.stdout:3/496: sync 2024-03-23T11:47:37.711 INFO:tasks.workunit.client.1.smithi111.stdout:8/475: sync 2024-03-23T11:47:37.712 INFO:tasks.workunit.client.1.smithi111.stdout:4/584: symlink d0/dc/d4c/d83/lb4 0 2024-03-23T11:47:37.713 INFO:tasks.workunit.client.0.smithi063.stdout:1/344: chown d3/cb 177320 1 2024-03-23T11:47:37.713 INFO:tasks.workunit.client.0.smithi063.stdout:4/510: write d5/da/d20/d4b/d1f/f41 [3788975,99800] 0 2024-03-23T11:47:37.713 INFO:tasks.workunit.client.1.smithi111.stdout:9/604: rmdir d0/d4/d1c/d2d/d75/d7a 39 2024-03-23T11:47:37.714 INFO:tasks.workunit.client.1.smithi111.stdout:8/476: stat d5/d1b/f68 0 2024-03-23T11:47:37.714 INFO:tasks.workunit.client.1.smithi111.stdout:0/574: mknod d5/db/d1f/d47/d59/db6/cbb 0 2024-03-23T11:47:37.716 INFO:tasks.workunit.client.1.smithi111.stdout:7/543: creat d14/d37/f9d x:0 0 0 2024-03-23T11:47:37.716 INFO:tasks.workunit.client.1.smithi111.stdout:0/575: read d5/db/d1f/f43 [1143265,19336] 0 2024-03-23T11:47:37.717 INFO:tasks.workunit.client.1.smithi111.stdout:2/626: chown d2/d11/d17/d3e/d6b/d2e/d76 208 1 2024-03-23T11:47:37.719 INFO:tasks.workunit.client.1.smithi111.stdout:7/544: dread - d14/d1b/d1e/d2c/d42/d5e/d88/d5f/f68 zero size 2024-03-23T11:47:37.722 INFO:tasks.workunit.client.1.smithi111.stdout:4/585: rmdir d0/dc/d1f 39 2024-03-23T11:47:37.722 INFO:tasks.workunit.client.1.smithi111.stdout:7/545: write d14/d37/f3f [1387123,82916] 0 2024-03-23T11:47:37.722 INFO:tasks.workunit.client.0.smithi063.stdout:2/396: mknod d2/d23/d2e/d52/d44/d4d/c7b 0 2024-03-23T11:47:37.725 INFO:tasks.workunit.client.1.smithi111.stdout:3/497: mknod d0/d7/d19/ca5 0 2024-03-23T11:47:37.729 INFO:tasks.workunit.client.1.smithi111.stdout:0/576: symlink d5/db/d1f/d19/d45/d4b/da7/lbc 0 2024-03-23T11:47:37.730 INFO:tasks.workunit.client.1.smithi111.stdout:2/627: creat d2/de/d6c/dd1/fe2 x:0 0 0 2024-03-23T11:47:37.731 INFO:tasks.workunit.client.1.smithi111.stdout:2/628: chown d2/d11/fb7 0 1 2024-03-23T11:47:37.732 INFO:tasks.workunit.client.1.smithi111.stdout:6/599: dread d2/d35/f40 [0,4194304] 0 2024-03-23T11:47:37.732 INFO:tasks.workunit.client.0.smithi063.stdout:4/511: dwrite d5/da/d20/d4b/f42 [0,4194304] 0 2024-03-23T11:47:37.733 INFO:tasks.workunit.client.1.smithi111.stdout:4/586: mkdir d0/d95/db5 0 2024-03-23T11:47:37.733 INFO:tasks.workunit.client.1.smithi111.stdout:8/477: mkdir d5/d32/d9b 0 2024-03-23T11:47:37.733 INFO:tasks.workunit.client.1.smithi111.stdout:9/605: creat d0/d4/d1c/d2d/d75/d7a/fc5 x:0 0 0 2024-03-23T11:47:37.735 INFO:tasks.workunit.client.0.smithi063.stdout:4/512: truncate d5/da/d72/f88 577575 0 2024-03-23T11:47:37.736 INFO:tasks.workunit.client.1.smithi111.stdout:7/546: mkdir d14/d1b/d1e/d2c/d42/d5e/d9e 0 2024-03-23T11:47:37.738 INFO:tasks.workunit.client.0.smithi063.stdout:4/513: write d5/da/d20/f48 [2794066,10702] 0 2024-03-23T11:47:37.739 INFO:tasks.workunit.client.1.smithi111.stdout:0/577: dread - d5/db/d1f/d47/d59/d5b/f7c zero size 2024-03-23T11:47:37.743 INFO:tasks.workunit.client.1.smithi111.stdout:7/547: write d14/d1b/f99 [473372,32910] 0 2024-03-23T11:47:37.745 INFO:tasks.workunit.client.0.smithi063.stdout:2/397: getdents d2/d23/d2e/d52/d6c 0 2024-03-23T11:47:37.745 INFO:tasks.workunit.client.1.smithi111.stdout:8/478: write d5/d26/d2f/d62/d7e/d85/f96 [603361,45805] 0 2024-03-23T11:47:37.746 INFO:tasks.workunit.client.1.smithi111.stdout:9/606: rename d0/d31/c4a to d0/cc6 0 2024-03-23T11:47:37.748 INFO:tasks.workunit.client.1.smithi111.stdout:4/587: write d0/dc/d4c/d2a/d97/d4d/f54 [1289698,4792] 0 2024-03-23T11:47:37.749 INFO:tasks.workunit.client.1.smithi111.stdout:7/548: chown d14/d44/d4b 110048577 1 2024-03-23T11:47:37.749 INFO:tasks.workunit.client.0.smithi063.stdout:4/514: creat d5/da/d20/d4b/d16/d37/f95 x:0 0 0 2024-03-23T11:47:37.750 INFO:tasks.workunit.client.0.smithi063.stdout:1/345: write d3/df/f12 [3509966,47488] 0 2024-03-23T11:47:37.752 INFO:tasks.workunit.client.0.smithi063.stdout:2/398: creat d2/d23/d2e/d52/d6c/f7c x:0 0 0 2024-03-23T11:47:37.754 INFO:tasks.workunit.client.1.smithi111.stdout:1/555: sync 2024-03-23T11:47:37.755 INFO:tasks.workunit.client.1.smithi111.stdout:2/629: dread d2/d11/f51 [0,4194304] 0 2024-03-23T11:47:37.756 INFO:tasks.workunit.client.0.smithi063.stdout:2/399: fdatasync d2/f6b 0 2024-03-23T11:47:37.757 INFO:tasks.workunit.client.1.smithi111.stdout:3/498: dwrite d0/d1a/d1d/d26/d5a/f8e [0,4194304] 0 2024-03-23T11:47:37.757 INFO:tasks.workunit.client.1.smithi111.stdout:4/588: unlink d0/d95/d9b/faf 0 2024-03-23T11:47:37.757 INFO:tasks.workunit.client.0.smithi063.stdout:1/346: symlink d3/d35/d40/l72 0 2024-03-23T11:47:37.758 INFO:tasks.workunit.client.0.smithi063.stdout:1/347: chown d3/d49 1 1 2024-03-23T11:47:37.758 INFO:tasks.workunit.client.0.smithi063.stdout:2/400: dread - d2/de/d22/f4a zero size 2024-03-23T11:47:37.760 INFO:tasks.workunit.client.1.smithi111.stdout:0/578: rename d5/db/d1f/d39 to d5/db/d1f/d19/dbd 0 2024-03-23T11:47:37.763 INFO:tasks.workunit.client.1.smithi111.stdout:1/556: write d0/d2b/d7c/f9f [169309,65063] 0 2024-03-23T11:47:37.764 INFO:tasks.workunit.client.0.smithi063.stdout:2/401: creat d2/d23/d2e/d52/f7d x:0 0 0 2024-03-23T11:47:37.767 INFO:tasks.workunit.client.0.smithi063.stdout:2/402: write d2/d23/d2e/f36 [1903384,54386] 0 2024-03-23T11:47:37.769 INFO:tasks.workunit.client.1.smithi111.stdout:7/549: truncate d14/d37/d50/f63 1052634 0 2024-03-23T11:47:37.769 INFO:tasks.workunit.client.1.smithi111.stdout:3/499: mkdir d0/d7/d19/d5e/da6 0 2024-03-23T11:47:37.770 INFO:tasks.workunit.client.0.smithi063.stdout:4/515: dwrite d5/da/d20/f3c [0,4194304] 0 2024-03-23T11:47:37.770 INFO:tasks.workunit.client.1.smithi111.stdout:1/557: symlink d0/d5/dd/d98/lbe 0 2024-03-23T11:47:37.771 INFO:tasks.workunit.client.0.smithi063.stdout:2/403: symlink d2/de/d22/l7e 0 2024-03-23T11:47:37.772 INFO:tasks.workunit.client.1.smithi111.stdout:6/600: dwrite d2/d1f/f7a [0,4194304] 0 2024-03-23T11:47:37.774 INFO:tasks.workunit.client.0.smithi063.stdout:4/516: mkdir d5/da/d20/d4b/d16/d50/d8e/d96 0 2024-03-23T11:47:37.775 INFO:tasks.workunit.client.1.smithi111.stdout:7/550: dread - d14/d1b/d1e/d86/f89 zero size 2024-03-23T11:47:37.776 INFO:tasks.workunit.client.0.smithi063.stdout:1/348: dread d3/d35/d40/f53 [0,4194304] 0 2024-03-23T11:47:37.777 INFO:tasks.workunit.client.1.smithi111.stdout:3/500: creat d0/d7/fa7 x:0 0 0 2024-03-23T11:47:37.780 INFO:tasks.workunit.client.0.smithi063.stdout:1/349: write d3/df/f1a [7974393,100472] 0 2024-03-23T11:47:37.781 INFO:tasks.workunit.client.1.smithi111.stdout:1/558: truncate d0/d5/fae 901641 0 2024-03-23T11:47:37.782 INFO:tasks.workunit.client.0.smithi063.stdout:1/350: dread - d3/d1c/f68 zero size 2024-03-23T11:47:37.784 INFO:tasks.workunit.client.0.smithi063.stdout:4/517: link d5/da/d20/d4b/d1f/f43 d5/da/d20/d4b/d16/d50/d8e/f97 0 2024-03-23T11:47:37.785 INFO:tasks.workunit.client.1.smithi111.stdout:8/479: dread d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [0,4194304] 0 2024-03-23T11:47:37.786 INFO:tasks.workunit.client.1.smithi111.stdout:6/601: symlink d2/d1a/d1c/d2c/d51/lc8 0 2024-03-23T11:47:37.787 INFO:tasks.workunit.client.1.smithi111.stdout:6/602: readlink d2/l66 0 2024-03-23T11:47:37.789 INFO:tasks.workunit.client.0.smithi063.stdout:4/518: fsync d5/da/d20/d4b/d16/d50/f84 0 2024-03-23T11:47:37.791 INFO:tasks.workunit.client.1.smithi111.stdout:6/603: mknod d2/d1f/cc9 0 2024-03-23T11:47:37.792 INFO:tasks.workunit.client.1.smithi111.stdout:4/589: dwrite d0/d7/da/f46 [0,4194304] 0 2024-03-23T11:47:37.797 INFO:tasks.workunit.client.0.smithi063.stdout:4/519: rename d5/da/d20/d4b/d29/d38/f60 to d5/da/d20/d4b/f98 0 2024-03-23T11:47:37.798 INFO:tasks.workunit.client.1.smithi111.stdout:5/609: dread d1/d3/fb0 [0,4194304] 0 2024-03-23T11:47:37.799 INFO:tasks.workunit.client.1.smithi111.stdout:8/480: dread d5/d26/d31/d4d/f59 [0,4194304] 0 2024-03-23T11:47:37.801 INFO:tasks.workunit.client.1.smithi111.stdout:5/610: write d1/d3/d6/d7/d63/fac [326171,86849] 0 2024-03-23T11:47:37.804 INFO:tasks.workunit.client.0.smithi063.stdout:4/520: chown d5/da/d20/d4b/d16/d37/l7b 63 1 2024-03-23T11:47:37.806 INFO:tasks.workunit.client.1.smithi111.stdout:5/611: creat d1/d20/fc9 x:0 0 0 2024-03-23T11:47:37.807 INFO:tasks.workunit.client.0.smithi063.stdout:4/521: stat d5/da/d20/d4b/d29/f81 0 2024-03-23T11:47:37.808 INFO:tasks.workunit.client.1.smithi111.stdout:6/604: dwrite d2/d1a/d1c/d2c/d51/f8f [4194304,4194304] 0 2024-03-23T11:47:37.808 INFO:tasks.workunit.client.1.smithi111.stdout:5/612: chown d1/d20/d41/da9/laf 558 1 2024-03-23T11:47:37.812 INFO:tasks.workunit.client.0.smithi063.stdout:4/522: getdents d5/da/d20/d4b/d1f 0 2024-03-23T11:47:37.814 INFO:tasks.workunit.client.1.smithi111.stdout:5/613: mknod d1/d20/d41/da9/cca 0 2024-03-23T11:47:37.817 INFO:tasks.workunit.client.0.smithi063.stdout:1/351: dread d3/d1c/f6d [0,4194304] 0 2024-03-23T11:47:37.818 INFO:tasks.workunit.client.1.smithi111.stdout:8/481: dwrite d5/d26/d31/d5e/d4a/f52 [0,4194304] 0 2024-03-23T11:47:37.818 INFO:tasks.workunit.client.0.smithi063.stdout:4/523: dread d5/da/d20/d4b/d16/d50/f5f [0,4194304] 0 2024-03-23T11:47:37.820 INFO:tasks.workunit.client.0.smithi063.stdout:1/352: symlink d3/d49/l73 0 2024-03-23T11:47:37.822 INFO:tasks.workunit.client.1.smithi111.stdout:8/482: creat d5/d40/d97/f9c x:0 0 0 2024-03-23T11:47:37.823 INFO:tasks.workunit.client.0.smithi063.stdout:1/353: symlink d3/d35/d40/l74 0 2024-03-23T11:47:37.823 INFO:tasks.workunit.client.0.smithi063.stdout:0/482: sync 2024-03-23T11:47:37.823 INFO:tasks.workunit.client.0.smithi063.stdout:8/721: sync 2024-03-23T11:47:37.825 INFO:tasks.workunit.client.1.smithi111.stdout:8/483: symlink d5/d26/d2f/d62/d75/d8d/l9d 0 2024-03-23T11:47:37.826 INFO:tasks.workunit.client.0.smithi063.stdout:4/524: getdents d5/da/d20/d4b/d16/d32/d6f 0 2024-03-23T11:47:37.826 INFO:tasks.workunit.client.0.smithi063.stdout:0/483: chown d2/d14/d36/d77/d94/da6/d1f/d4f/l52 7 1 2024-03-23T11:47:37.827 INFO:tasks.workunit.client.0.smithi063.stdout:1/354: rename d3/d35/l37 to d3/d3f/d4d/d52/l75 0 2024-03-23T11:47:37.828 INFO:tasks.workunit.client.1.smithi111.stdout:8/484: mknod d5/d26/d2f/d62/d75/d4b/d51/c9e 0 2024-03-23T11:47:37.830 INFO:tasks.workunit.client.0.smithi063.stdout:4/525: dread d5/da/d20/d4b/d16/d50/d86/f89 [0,4194304] 0 2024-03-23T11:47:37.835 INFO:tasks.workunit.client.0.smithi063.stdout:8/722: creat d6/d1c/ff9 x:0 0 0 2024-03-23T11:47:37.837 INFO:tasks.workunit.client.0.smithi063.stdout:4/526: fsync d5/da/d20/d4b/d16/d32/d56/f85 0 2024-03-23T11:47:37.843 INFO:tasks.workunit.client.0.smithi063.stdout:4/527: mknod d5/d1e/c99 0 2024-03-23T11:47:37.844 INFO:tasks.workunit.client.0.smithi063.stdout:8/723: mknod d6/d1f/d54/d5c/dc5/cfa 0 2024-03-23T11:47:37.847 INFO:tasks.workunit.client.0.smithi063.stdout:0/484: dwrite d2/d12/d32/f5c [0,4194304] 0 2024-03-23T11:47:37.856 INFO:tasks.workunit.client.0.smithi063.stdout:0/485: truncate d2/d12/d35/fa8 863733 0 2024-03-23T11:47:37.869 INFO:tasks.workunit.client.0.smithi063.stdout:0/486: stat d2/d14/d36/d77/d94/da6/d25/c78 0 2024-03-23T11:47:37.871 INFO:tasks.workunit.client.0.smithi063.stdout:0/487: mkdir d2/d14/d36/d77/d94/db4 0 2024-03-23T11:47:37.874 INFO:tasks.workunit.client.0.smithi063.stdout:1/355: dread d3/f7 [4194304,4194304] 0 2024-03-23T11:47:37.876 INFO:tasks.workunit.client.0.smithi063.stdout:8/724: dwrite d6/d1c/f1d [0,4194304] 0 2024-03-23T11:47:37.876 INFO:tasks.workunit.client.0.smithi063.stdout:4/528: dwrite d5/da/d20/d4b/d29/d38/f55 [0,4194304] 0 2024-03-23T11:47:37.877 INFO:tasks.workunit.client.1.smithi111.stdout:0/579: rename d5/db/d1f/d19/d45/d4b to d5/db/dbe 0 2024-03-23T11:47:37.878 INFO:tasks.workunit.client.0.smithi063.stdout:1/356: write d3/df/f1a [720777,85614] 0 2024-03-23T11:47:37.879 INFO:tasks.workunit.client.0.smithi063.stdout:8/725: chown d6/dd/d59/db2 3308683 1 2024-03-23T11:47:37.880 INFO:tasks.workunit.client.0.smithi063.stdout:8/726: readlink d6/dd/l53 0 2024-03-23T11:47:37.882 INFO:tasks.workunit.client.0.smithi063.stdout:0/488: creat d2/d14/d36/d77/d94/da6/d1f/fb5 x:0 0 0 2024-03-23T11:47:37.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:37 smithi063 ceph-mon[35189]: pgmap v38: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 43 MiB/s rd, 208 MiB/s wr, 354 op/s 2024-03-23T11:47:37.884 INFO:tasks.workunit.client.0.smithi063.stdout:8/727: readlink d6/d1c/d27/d3f/lcd 0 2024-03-23T11:47:37.886 INFO:tasks.workunit.client.0.smithi063.stdout:1/357: symlink d3/d3f/d5c/l76 0 2024-03-23T11:47:37.888 INFO:tasks.workunit.client.1.smithi111.stdout:1/559: rename d0/d2b/d7c/d8b/fa3 to d0/d6/d48/d4a/d6c/d96/fbf 0 2024-03-23T11:47:37.889 INFO:tasks.workunit.client.0.smithi063.stdout:1/358: chown d3/c44 6223 1 2024-03-23T11:47:37.893 INFO:tasks.workunit.client.1.smithi111.stdout:4/590: rename d0/dc/d1f/d7e to d0/d95/d9b/db3/db6 0 2024-03-23T11:47:37.899 INFO:tasks.workunit.client.1.smithi111.stdout:1/560: write d0/d2b/d37/d6a/f8d [1630286,56528] 0 2024-03-23T11:47:37.901 INFO:tasks.workunit.client.1.smithi111.stdout:4/591: write d0/d7/da/f13 [1237165,48771] 0 2024-03-23T11:47:37.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:37 smithi111 ceph-mon[44005]: pgmap v38: 65 pgs: 65 active+clean; 1.6 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 43 MiB/s rd, 208 MiB/s wr, 354 op/s 2024-03-23T11:47:37.904 INFO:tasks.workunit.client.1.smithi111.stdout:1/561: rmdir d0 39 2024-03-23T11:47:37.904 INFO:tasks.workunit.client.0.smithi063.stdout:1/359: creat d3/d3f/f77 x:0 0 0 2024-03-23T11:47:37.907 INFO:tasks.workunit.client.1.smithi111.stdout:4/592: symlink d0/d95/lb7 0 2024-03-23T11:47:37.911 INFO:tasks.workunit.client.0.smithi063.stdout:0/489: dwrite d2/d14/f87 [0,4194304] 0 2024-03-23T11:47:37.911 INFO:tasks.workunit.client.0.smithi063.stdout:1/360: write d3/d3f/d4d/f69 [358163,107090] 0 2024-03-23T11:47:37.912 INFO:tasks.workunit.client.1.smithi111.stdout:4/593: fsync d0/d95/d9b/db3/db6/fa3 0 2024-03-23T11:47:37.912 INFO:tasks.workunit.client.1.smithi111.stdout:7/551: write d14/f25 [96154,49936] 0 2024-03-23T11:47:37.913 INFO:tasks.workunit.client.1.smithi111.stdout:2/630: dwrite d2/de/d6c/f65 [0,4194304] 0 2024-03-23T11:47:37.915 INFO:tasks.workunit.client.0.smithi063.stdout:8/728: dread d6/d1f/dbb/fc3 [0,4194304] 0 2024-03-23T11:47:37.915 INFO:tasks.workunit.client.1.smithi111.stdout:7/552: truncate d14/d37/f9d 939526 0 2024-03-23T11:47:37.918 INFO:tasks.workunit.client.0.smithi063.stdout:2/404: dread d2/de/d22/f78 [0,4194304] 0 2024-03-23T11:47:37.918 INFO:tasks.workunit.client.0.smithi063.stdout:0/490: chown d2/d4/d63/f6b 3 1 2024-03-23T11:47:37.920 INFO:tasks.workunit.client.0.smithi063.stdout:4/529: link d5/da/d20/d4b/d1f/f41 d5/da/d20/d4b/d29/d38/d7a/d8f/f9a 0 2024-03-23T11:47:37.923 INFO:tasks.workunit.client.0.smithi063.stdout:1/361: truncate d3/df/f19 585420 0 2024-03-23T11:47:37.923 INFO:tasks.workunit.client.1.smithi111.stdout:8/485: sync 2024-03-23T11:47:37.924 INFO:tasks.workunit.client.1.smithi111.stdout:7/553: rmdir d14/d37/d54 39 2024-03-23T11:47:37.927 INFO:tasks.workunit.client.1.smithi111.stdout:2/631: rename d2/d11/d17/d3e/d6b/f28 to d2/de/d6c/dd1/fe3 0 2024-03-23T11:47:37.930 INFO:tasks.workunit.client.0.smithi063.stdout:0/491: creat d2/d4/d63/fb6 x:0 0 0 2024-03-23T11:47:37.931 INFO:tasks.workunit.client.0.smithi063.stdout:4/530: mknod d5/da/d72/c9b 0 2024-03-23T11:47:37.934 INFO:tasks.workunit.client.1.smithi111.stdout:6/605: write d2/d80/f99 [301328,91714] 0 2024-03-23T11:47:37.936 INFO:tasks.workunit.client.1.smithi111.stdout:9/607: dread d0/d4/d1c/d2d/f2e [0,4194304] 0 2024-03-23T11:47:37.938 INFO:tasks.workunit.client.1.smithi111.stdout:9/608: write d0/d4/d10/f51 [741170,12455] 0 2024-03-23T11:47:37.941 INFO:tasks.workunit.client.1.smithi111.stdout:1/562: creat d0/d5/dd/fc0 x:0 0 0 2024-03-23T11:47:37.943 INFO:tasks.workunit.client.1.smithi111.stdout:9/609: dread - d0/d4/d10/d80/fb6 zero size 2024-03-23T11:47:37.949 INFO:tasks.workunit.client.1.smithi111.stdout:9/610: fdatasync d0/d4/d1c/d2d/f63 0 2024-03-23T11:47:37.949 INFO:tasks.workunit.client.1.smithi111.stdout:5/614: dread d1/d3/d6/d8b/d59/d74/f5e [4194304,4194304] 0 2024-03-23T11:47:37.949 INFO:tasks.workunit.client.1.smithi111.stdout:3/501: dread d0/fd [0,4194304] 0 2024-03-23T11:47:37.950 INFO:tasks.workunit.client.0.smithi063.stdout:4/531: symlink d5/d1e/l9c 0 2024-03-23T11:47:37.951 INFO:tasks.workunit.client.1.smithi111.stdout:3/502: rename d0/d1a to d0/d1a/d1d/da8 22 2024-03-23T11:47:37.952 INFO:tasks.workunit.client.1.smithi111.stdout:4/594: link d0/dc/d30/f15 d0/d95/fb8 0 2024-03-23T11:47:37.953 INFO:tasks.workunit.client.1.smithi111.stdout:8/486: mknod d5/d26/c9f 0 2024-03-23T11:47:37.957 INFO:tasks.workunit.client.1.smithi111.stdout:4/595: write d0/dc/d1f/db0/fa7 [1024540,92672] 0 2024-03-23T11:47:37.958 INFO:tasks.workunit.client.1.smithi111.stdout:9/611: read d0/d4/d6a/f72 [594018,63095] 0 2024-03-23T11:47:37.961 INFO:tasks.workunit.client.0.smithi063.stdout:4/532: write d5/da/d20/d4b/d16/d37/f57 [429492,78091] 0 2024-03-23T11:47:37.961 INFO:tasks.workunit.client.0.smithi063.stdout:8/729: dread d6/d1f/d54/d63/d90/f35 [0,4194304] 0 2024-03-23T11:47:37.963 INFO:tasks.workunit.client.1.smithi111.stdout:1/563: dwrite d0/d5/dd/d1e/d54/d87/f8a [0,4194304] 0 2024-03-23T11:47:37.967 INFO:tasks.workunit.client.0.smithi063.stdout:4/533: read d5/da/d20/d4b/d1f/f21 [3019634,26486] 0 2024-03-23T11:47:37.969 INFO:tasks.workunit.client.1.smithi111.stdout:5/615: symlink d1/d3/d6/d7/d7b/lcb 0 2024-03-23T11:47:37.969 INFO:tasks.workunit.client.1.smithi111.stdout:2/632: symlink d2/le4 0 2024-03-23T11:47:37.970 INFO:tasks.workunit.client.0.smithi063.stdout:0/492: link d2/d14/d36/d77/d94/da6/d1f/c9a d2/d14/d36/d77/d94/da6/cb7 0 2024-03-23T11:47:37.972 INFO:tasks.workunit.client.1.smithi111.stdout:4/596: creat d0/dc/d4c/d2a/fb9 x:0 0 0 2024-03-23T11:47:37.973 INFO:tasks.workunit.client.1.smithi111.stdout:7/554: dwrite d14/d1b/d1e/d2c/d42/d5e/f6c [0,4194304] 0 2024-03-23T11:47:37.974 INFO:tasks.workunit.client.0.smithi063.stdout:2/405: write d2/f25 [3047250,127698] 0 2024-03-23T11:47:37.975 INFO:tasks.workunit.client.0.smithi063.stdout:0/493: mknod d2/cb8 0 2024-03-23T11:47:37.978 INFO:tasks.workunit.client.0.smithi063.stdout:4/534: symlink d5/da/d20/l9d 0 2024-03-23T11:47:37.981 INFO:tasks.workunit.client.1.smithi111.stdout:9/612: fsync d0/d4/d1c/d2d/d3b/d42/d4f/d95/f7b 0 2024-03-23T11:47:37.981 INFO:tasks.workunit.client.1.smithi111.stdout:6/606: dwrite d2/ff [0,4194304] 0 2024-03-23T11:47:37.981 INFO:tasks.workunit.client.0.smithi063.stdout:4/535: readlink d5/da/d20/l9d 0 2024-03-23T11:47:37.983 INFO:tasks.workunit.client.1.smithi111.stdout:8/487: mkdir d5/d32/d7d/da0 0 2024-03-23T11:47:37.983 INFO:tasks.workunit.client.1.smithi111.stdout:0/580: rename d5/db/d1f/d32/la9 to d5/db/d1f/d47/d6f/lbf 0 2024-03-23T11:47:37.983 INFO:tasks.workunit.client.1.smithi111.stdout:7/555: dread - d14/d44/d70/f7c zero size 2024-03-23T11:47:37.990 INFO:tasks.workunit.client.1.smithi111.stdout:5/616: mkdir d1/d3/d6/d30/db2/dcc 0 2024-03-23T11:47:37.992 INFO:tasks.workunit.client.0.smithi063.stdout:9/490: sync 2024-03-23T11:47:37.992 INFO:tasks.workunit.client.0.smithi063.stdout:5/311: sync 2024-03-23T11:47:37.992 INFO:tasks.workunit.client.0.smithi063.stdout:1/362: dwrite d3/d1c/d4b/f71 [0,4194304] 0 2024-03-23T11:47:37.993 INFO:tasks.workunit.client.1.smithi111.stdout:6/607: chown d2/f28 57261041 1 2024-03-23T11:47:37.994 INFO:tasks.workunit.client.1.smithi111.stdout:0/581: stat d5/db/d1f/d19/dbd/f3f 0 2024-03-23T11:47:37.994 INFO:tasks.workunit.client.0.smithi063.stdout:1/363: stat d3/df 0 2024-03-23T11:47:37.994 INFO:tasks.workunit.client.0.smithi063.stdout:2/406: rename d2/d23/d2e/d52/c59 to d2/de/d7a/d77/c7f 0 2024-03-23T11:47:37.995 INFO:tasks.workunit.client.0.smithi063.stdout:0/494: symlink d2/d14/d36/d77/d94/daf/lb9 0 2024-03-23T11:47:37.995 INFO:tasks.workunit.client.1.smithi111.stdout:3/503: dread d0/f16 [0,4194304] 0 2024-03-23T11:47:37.997 INFO:tasks.workunit.client.0.smithi063.stdout:1/364: chown d3/d62 2534007 1 2024-03-23T11:47:37.997 INFO:tasks.workunit.client.0.smithi063.stdout:4/536: rmdir d5/da/d20/d4b/d16/d50 39 2024-03-23T11:47:38.000 INFO:tasks.workunit.client.0.smithi063.stdout:8/730: dwrite d6/dd/df/f61 [0,4194304] 0 2024-03-23T11:47:38.001 INFO:tasks.workunit.client.1.smithi111.stdout:1/564: dwrite d0/d2b/d69/fbc [0,4194304] 0 2024-03-23T11:47:38.009 INFO:tasks.workunit.client.1.smithi111.stdout:1/565: stat d0/d2b/f75 0 2024-03-23T11:47:38.012 INFO:tasks.workunit.client.1.smithi111.stdout:3/504: mkdir d0/d72/da9 0 2024-03-23T11:47:38.012 INFO:tasks.workunit.client.1.smithi111.stdout:6/608: fdatasync d2/d1a/f72 0 2024-03-23T11:47:38.013 INFO:tasks.workunit.client.0.smithi063.stdout:0/495: creat d2/d14/d36/d77/d94/da6/d1f/fba x:0 0 0 2024-03-23T11:47:38.015 INFO:tasks.workunit.client.1.smithi111.stdout:3/505: readlink d0/l58 0 2024-03-23T11:47:38.022 INFO:tasks.workunit.client.0.smithi063.stdout:9/491: rename d4/d31/d43/da9/da6/f9f to d4/fae 0 2024-03-23T11:47:38.023 INFO:tasks.workunit.client.1.smithi111.stdout:3/506: write d0/d55/f81 [896212,1883] 0 2024-03-23T11:47:38.024 INFO:tasks.workunit.client.1.smithi111.stdout:3/507: stat d0/c3e 0 2024-03-23T11:47:38.026 INFO:tasks.workunit.client.1.smithi111.stdout:8/488: dwrite d5/d26/d2f/d42/d63/f69 [0,4194304] 0 2024-03-23T11:47:38.027 INFO:tasks.workunit.client.1.smithi111.stdout:0/582: dwrite d5/db/d1f/d48/f6a [0,4194304] 0 2024-03-23T11:47:38.027 INFO:tasks.workunit.client.0.smithi063.stdout:9/492: chown d4/d31/d43/d49/f67 857046 1 2024-03-23T11:47:38.033 INFO:tasks.workunit.client.0.smithi063.stdout:9/493: fsync d4/d31/d43/da9/da6/f4a 0 2024-03-23T11:47:38.036 INFO:tasks.workunit.client.0.smithi063.stdout:9/494: stat d4/ff 0 2024-03-23T11:47:38.040 INFO:tasks.workunit.client.0.smithi063.stdout:8/731: rename d6/dd/d59/daf/dc4 to d6/dd/d59/db2/dbe/dfb 0 2024-03-23T11:47:38.043 INFO:tasks.workunit.client.0.smithi063.stdout:8/732: chown d6/d1f/d54/d63/d90/d15 1277954593 1 2024-03-23T11:47:38.044 INFO:tasks.workunit.client.0.smithi063.stdout:0/496: dwrite d2/d48/da5/da7/faa [0,4194304] 0 2024-03-23T11:47:38.044 INFO:tasks.workunit.client.1.smithi111.stdout:9/613: dread d0/d4/d10/d29/f67 [0,4194304] 0 2024-03-23T11:47:38.047 INFO:tasks.workunit.client.1.smithi111.stdout:8/489: dwrite d5/d26/d2f/f77 [0,4194304] 0 2024-03-23T11:47:38.050 INFO:tasks.workunit.client.1.smithi111.stdout:8/490: truncate d5/d26/d31/d5e/d4a/f8e 33612 0 2024-03-23T11:47:38.051 INFO:tasks.workunit.client.0.smithi063.stdout:4/537: dwrite d5/da/d20/d4b/d16/d50/f5f [0,4194304] 0 2024-03-23T11:47:38.053 INFO:tasks.workunit.client.0.smithi063.stdout:1/365: dread d3/df/d2f/f64 [0,4194304] 0 2024-03-23T11:47:38.063 INFO:tasks.workunit.client.1.smithi111.stdout:4/597: dread d0/dc/d1f/f76 [0,4194304] 0 2024-03-23T11:47:38.064 INFO:tasks.workunit.client.1.smithi111.stdout:9/614: dwrite d0/d4/d10/f8b [0,4194304] 0 2024-03-23T11:47:38.065 INFO:tasks.workunit.client.0.smithi063.stdout:5/312: write d2/f5 [1739431,13254] 0 2024-03-23T11:47:38.066 INFO:tasks.workunit.client.1.smithi111.stdout:2/633: write d2/d11/d17/d1f/da0/fdd [322609,61965] 0 2024-03-23T11:47:38.068 INFO:tasks.workunit.client.1.smithi111.stdout:9/615: dread - d0/d4/d1c/d2d/d3b/fa8 zero size 2024-03-23T11:47:38.069 INFO:tasks.workunit.client.0.smithi063.stdout:9/495: dwrite d4/d4c/d44/f84 [0,4194304] 0 2024-03-23T11:47:38.070 INFO:tasks.workunit.client.0.smithi063.stdout:0/497: mknod d2/d4/d63/cbb 0 2024-03-23T11:47:38.076 INFO:tasks.workunit.client.0.smithi063.stdout:0/498: dread - d2/d4/d63/fb6 zero size 2024-03-23T11:47:38.077 INFO:tasks.workunit.client.0.smithi063.stdout:0/499: readlink d2/l29 0 2024-03-23T11:47:38.080 INFO:tasks.workunit.client.0.smithi063.stdout:4/538: write d5/da/d20/f77 [233593,128538] 0 2024-03-23T11:47:38.084 INFO:tasks.workunit.client.0.smithi063.stdout:5/313: symlink d2/d28/d3a/l6d 0 2024-03-23T11:47:38.085 INFO:tasks.workunit.client.0.smithi063.stdout:4/539: write d5/da/d20/f77 [1400576,36805] 0 2024-03-23T11:47:38.087 INFO:tasks.workunit.client.1.smithi111.stdout:7/556: dwrite d14/f1c [4194304,4194304] 0 2024-03-23T11:47:38.087 INFO:tasks.workunit.client.0.smithi063.stdout:9/496: symlink d4/d31/laf 0 2024-03-23T11:47:38.089 INFO:tasks.workunit.client.0.smithi063.stdout:0/500: dread d2/d4/d63/f6b [0,4194304] 0 2024-03-23T11:47:38.093 INFO:tasks.workunit.client.1.smithi111.stdout:6/609: chown d2/d1f/d45/d49/f9e 613205448 1 2024-03-23T11:47:38.098 INFO:tasks.workunit.client.1.smithi111.stdout:4/598: mkdir d0/dc/d4c/d2a/d97/d4f/dba 0 2024-03-23T11:47:38.100 INFO:tasks.workunit.client.1.smithi111.stdout:5/617: rename d1/d3/d6/d8b/d59/f67 to d1/d3/fcd 0 2024-03-23T11:47:38.102 INFO:tasks.workunit.client.0.smithi063.stdout:5/314: mkdir d2/d17/d4d/d6e 0 2024-03-23T11:47:38.104 INFO:tasks.workunit.client.1.smithi111.stdout:1/566: creat d0/d5/dd/d1e/d54/dbb/fc1 x:0 0 0 2024-03-23T11:47:38.104 INFO:tasks.workunit.client.0.smithi063.stdout:4/540: fsync d5/d1e/d2b/f58 0 2024-03-23T11:47:38.108 INFO:tasks.workunit.client.0.smithi063.stdout:1/366: creat d3/df/f78 x:0 0 0 2024-03-23T11:47:38.110 INFO:tasks.workunit.client.0.smithi063.stdout:5/315: readlink d2/d28/d3a/l6d 0 2024-03-23T11:47:38.116 INFO:tasks.workunit.client.1.smithi111.stdout:3/508: rename d0/d1a/d1d/d26/d5a/f5f to d0/d1a/d2b/d38/d7f/faa 0 2024-03-23T11:47:38.118 INFO:tasks.workunit.client.0.smithi063.stdout:0/501: dwrite d2/d14/f64 [4194304,4194304] 0 2024-03-23T11:47:38.120 INFO:tasks.workunit.client.0.smithi063.stdout:0/502: read - d2/d12/d32/f91 zero size 2024-03-23T11:47:38.122 INFO:tasks.workunit.client.0.smithi063.stdout:4/541: creat d5/da/d20/d4b/d16/d32/d56/f9e x:0 0 0 2024-03-23T11:47:38.127 INFO:tasks.workunit.client.0.smithi063.stdout:4/542: fsync d5/da/d20/d4b/d16/d32/d6f/f7f 0 2024-03-23T11:47:38.128 INFO:tasks.workunit.client.1.smithi111.stdout:0/583: creat d5/db/d1f/fc0 x:0 0 0 2024-03-23T11:47:38.131 INFO:tasks.workunit.client.1.smithi111.stdout:1/567: dread d0/d2b/f5c [0,4194304] 0 2024-03-23T11:47:38.132 INFO:tasks.workunit.client.0.smithi063.stdout:4/543: fsync d5/da/d20/d4b/d29/d38/f55 0 2024-03-23T11:47:38.134 INFO:tasks.workunit.client.0.smithi063.stdout:5/316: creat d2/d7/d64/f6f x:0 0 0 2024-03-23T11:47:38.137 INFO:tasks.workunit.client.1.smithi111.stdout:8/491: rename d5/d26/d2f/d62/d75/d4b/d51/d5a/l8c to d5/d26/d2f/d62/d75/d8d/la1 0 2024-03-23T11:47:38.138 INFO:tasks.workunit.client.0.smithi063.stdout:1/367: creat d3/d49/d56/d65/f79 x:0 0 0 2024-03-23T11:47:38.139 INFO:tasks.workunit.client.1.smithi111.stdout:8/492: fsync d5/d26/d2f/d62/d75/f6b 0 2024-03-23T11:47:38.142 INFO:tasks.workunit.client.1.smithi111.stdout:2/634: getdents d2/d11/d17/d3e/d5d/dc0 0 2024-03-23T11:47:38.143 INFO:tasks.workunit.client.0.smithi063.stdout:8/733: dwrite d6/d1c/f2d [0,4194304] 0 2024-03-23T11:47:38.143 INFO:tasks.workunit.client.0.smithi063.stdout:5/317: write d2/d28/f4f [2543431,90338] 0 2024-03-23T11:47:38.143 INFO:tasks.workunit.client.0.smithi063.stdout:0/503: mknod d2/d14/d36/d77/d94/da6/cbc 0 2024-03-23T11:47:38.145 INFO:tasks.workunit.client.0.smithi063.stdout:4/544: rmdir d5/da/d72 39 2024-03-23T11:47:38.146 INFO:tasks.workunit.client.1.smithi111.stdout:3/509: write d0/d7/f1e [811080,116776] 0 2024-03-23T11:47:38.146 INFO:tasks.workunit.client.0.smithi063.stdout:8/734: chown d6/d1f/d54/d63/d90/d15/d9d 1062811 1 2024-03-23T11:47:38.148 INFO:tasks.workunit.client.1.smithi111.stdout:3/510: readlink d0/d72/l77 0 2024-03-23T11:47:38.148 INFO:tasks.workunit.client.1.smithi111.stdout:1/568: creat d0/d5/dd/d1e/d54/fc2 x:0 0 0 2024-03-23T11:47:38.151 INFO:tasks.workunit.client.1.smithi111.stdout:1/569: fsync d0/d6/f34 0 2024-03-23T11:47:38.152 INFO:tasks.workunit.client.1.smithi111.stdout:9/616: rename d0/d4/d10/d80/fb6 to d0/d4/d1c/d2d/d3b/d42/d92/fc7 0 2024-03-23T11:47:38.153 INFO:tasks.workunit.client.0.smithi063.stdout:9/497: dread d4/d31/d43/d49/f67 [4194304,4194304] 0 2024-03-23T11:47:38.157 INFO:tasks.workunit.client.1.smithi111.stdout:8/493: dwrite d5/d26/d31/d4d/f6c [0,4194304] 0 2024-03-23T11:47:38.157 INFO:tasks.workunit.client.1.smithi111.stdout:9/617: read d0/d31/d41/d65/f6f [279779,59873] 0 2024-03-23T11:47:38.157 INFO:tasks.workunit.client.0.smithi063.stdout:0/504: dread d2/d14/f79 [0,4194304] 0 2024-03-23T11:47:38.158 INFO:tasks.workunit.client.0.smithi063.stdout:0/505: chown d2/d4/d63 200735 1 2024-03-23T11:47:38.159 INFO:tasks.workunit.client.0.smithi063.stdout:3/682: sync 2024-03-23T11:47:38.160 INFO:tasks.workunit.client.0.smithi063.stdout:6/381: sync 2024-03-23T11:47:38.160 INFO:tasks.workunit.client.0.smithi063.stdout:7/421: sync 2024-03-23T11:47:38.160 INFO:tasks.workunit.client.1.smithi111.stdout:0/584: unlink d5/db/d1f/d19/l1b 0 2024-03-23T11:47:38.162 INFO:tasks.workunit.client.0.smithi063.stdout:4/545: symlink d5/da/d20/d4b/d16/d32/d56/l9f 0 2024-03-23T11:47:38.165 INFO:tasks.workunit.client.0.smithi063.stdout:8/735: truncate d6/dd/f56 190503 0 2024-03-23T11:47:38.168 INFO:tasks.workunit.client.0.smithi063.stdout:5/318: link d2/d7/d8/de/d35/f6c d2/d2f/f70 0 2024-03-23T11:47:38.168 INFO:tasks.workunit.client.0.smithi063.stdout:8/736: chown d6/dd/l2f 13980023 1 2024-03-23T11:47:38.170 INFO:tasks.workunit.client.0.smithi063.stdout:9/498: readlink d4/d31/d43/l6c 0 2024-03-23T11:47:38.171 INFO:tasks.workunit.client.1.smithi111.stdout:6/610: rename d2/d1f/fa6 to d2/d1f/d45/d65/fca 0 2024-03-23T11:47:38.172 INFO:tasks.workunit.client.0.smithi063.stdout:3/683: symlink d13/d66/ld3 0 2024-03-23T11:47:38.180 INFO:tasks.workunit.client.0.smithi063.stdout:6/382: fsync d1/dc/d21/f3c 0 2024-03-23T11:47:38.181 INFO:tasks.workunit.client.0.smithi063.stdout:8/737: truncate d6/dd/d59/db2/dbe/dfb/fed 371009 0 2024-03-23T11:47:38.182 INFO:tasks.workunit.client.0.smithi063.stdout:4/546: creat d5/da/d20/d4b/d16/d32/d56/fa0 x:0 0 0 2024-03-23T11:47:38.183 INFO:tasks.workunit.client.1.smithi111.stdout:9/618: creat d0/d31/d41/d9d/fc8 x:0 0 0 2024-03-23T11:47:38.184 INFO:tasks.workunit.client.1.smithi111.stdout:7/557: write d14/d1b/d1e/f47 [4729904,56510] 0 2024-03-23T11:47:38.184 INFO:tasks.workunit.client.0.smithi063.stdout:1/368: getdents d3/d49/d56/d65 0 2024-03-23T11:47:38.187 INFO:tasks.workunit.client.1.smithi111.stdout:8/494: fdatasync d5/d26/d2f/d42/f57 0 2024-03-23T11:47:38.189 INFO:tasks.workunit.client.1.smithi111.stdout:3/511: symlink d0/lab 0 2024-03-23T11:47:38.191 INFO:tasks.workunit.client.1.smithi111.stdout:6/611: truncate d2/d1a/d1c/f24 101555 0 2024-03-23T11:47:38.193 INFO:tasks.workunit.client.0.smithi063.stdout:9/499: creat d4/d31/d43/da9/da6/fb0 x:0 0 0 2024-03-23T11:47:38.194 INFO:tasks.workunit.client.1.smithi111.stdout:2/635: creat d2/d11/fe5 x:0 0 0 2024-03-23T11:47:38.195 INFO:tasks.workunit.client.0.smithi063.stdout:4/547: fsync d5/da/d20/d4b/d16/d32/d56/f85 0 2024-03-23T11:47:38.195 INFO:tasks.workunit.client.1.smithi111.stdout:5/618: truncate d1/d3/d6/d7/d63/d2c/d82/d66/f9b 600225 0 2024-03-23T11:47:38.198 INFO:tasks.workunit.client.1.smithi111.stdout:5/619: chown d1/d3/d6/d7/d63/d2c/d43/la6 856437699 1 2024-03-23T11:47:38.202 INFO:tasks.workunit.client.0.smithi063.stdout:7/422: dwrite d5/d5c/d2f/d73/da/f89 [0,4194304] 0 2024-03-23T11:47:38.203 INFO:tasks.workunit.client.1.smithi111.stdout:7/558: read d14/d1b/d1e/d2c/f31 [368292,118683] 0 2024-03-23T11:47:38.207 INFO:tasks.workunit.client.1.smithi111.stdout:0/585: mknod d5/db/d1f/d19/dbd/d64/d91/cc1 0 2024-03-23T11:47:38.208 INFO:tasks.workunit.client.0.smithi063.stdout:0/506: dread d2/d14/f23 [0,4194304] 0 2024-03-23T11:47:38.209 INFO:tasks.workunit.client.0.smithi063.stdout:8/738: mkdir d6/dd/d59/d7f/dc8/dfc 0 2024-03-23T11:47:38.211 INFO:tasks.workunit.client.1.smithi111.stdout:6/612: dread d2/d35/d70/fc4 [0,4194304] 0 2024-03-23T11:47:38.214 INFO:tasks.workunit.client.0.smithi063.stdout:5/319: dwrite d2/d28/f3d [0,4194304] 0 2024-03-23T11:47:38.217 INFO:tasks.workunit.client.1.smithi111.stdout:2/636: rename d2/de/d12/fd0 to d2/de/d6c/dd1/fe6 0 2024-03-23T11:47:38.218 INFO:tasks.workunit.client.0.smithi063.stdout:3/684: dwrite f12 [0,4194304] 0 2024-03-23T11:47:38.218 INFO:tasks.workunit.client.0.smithi063.stdout:7/423: fdatasync d5/d5c/d2f/d73/f7c 0 2024-03-23T11:47:38.219 INFO:tasks.workunit.client.0.smithi063.stdout:8/739: chown d6/dd/d59/db2/dbe/dfb/fe9 500709 1 2024-03-23T11:47:38.220 INFO:tasks.workunit.client.1.smithi111.stdout:1/570: dwrite d0/d5/f2f [4194304,4194304] 0 2024-03-23T11:47:38.221 INFO:tasks.workunit.client.0.smithi063.stdout:1/369: write d3/df/d2f/f64 [3707455,35857] 0 2024-03-23T11:47:38.226 INFO:tasks.workunit.client.0.smithi063.stdout:1/370: chown d3/d3f/d5c 8 1 2024-03-23T11:47:38.227 INFO:tasks.workunit.client.1.smithi111.stdout:3/512: creat d0/d1a/d1d/d26/d5a/d60/fac x:0 0 0 2024-03-23T11:47:38.228 INFO:tasks.workunit.client.0.smithi063.stdout:1/371: write d3/d35/d40/f4c [193045,20522] 0 2024-03-23T11:47:38.229 INFO:tasks.workunit.client.1.smithi111.stdout:6/613: mknod d2/d1f/d45/d49/d8d/ccb 0 2024-03-23T11:47:38.231 INFO:tasks.workunit.client.1.smithi111.stdout:2/637: symlink d2/d11/d17/d3e/d5d/le7 0 2024-03-23T11:47:38.232 INFO:tasks.workunit.client.0.smithi063.stdout:9/500: creat d4/fb1 x:0 0 0 2024-03-23T11:47:38.232 INFO:tasks.workunit.client.0.smithi063.stdout:4/548: creat d5/da/fa1 x:0 0 0 2024-03-23T11:47:38.233 INFO:tasks.workunit.client.0.smithi063.stdout:6/383: dwrite d1/d15/d4b/f4c [0,4194304] 0 2024-03-23T11:47:38.233 INFO:tasks.workunit.client.1.smithi111.stdout:6/614: write d2/d1f/d45/d9a/fb0 [185753,22721] 0 2024-03-23T11:47:38.234 INFO:tasks.workunit.client.0.smithi063.stdout:1/372: dread - d3/df/f78 zero size 2024-03-23T11:47:38.234 INFO:tasks.workunit.client.1.smithi111.stdout:1/571: symlink d0/d6/d48/d4d/dab/lc3 0 2024-03-23T11:47:38.235 INFO:tasks.workunit.client.0.smithi063.stdout:5/320: mkdir d2/d28/d3a/d46/d71 0 2024-03-23T11:47:38.238 INFO:tasks.workunit.client.0.smithi063.stdout:9/501: fsync d4/d31/d43/da9/da6/fb0 0 2024-03-23T11:47:38.240 INFO:tasks.workunit.client.0.smithi063.stdout:7/424: unlink d5/c48 0 2024-03-23T11:47:38.241 INFO:tasks.workunit.client.1.smithi111.stdout:2/638: symlink d2/d7/da/dbd/le8 0 2024-03-23T11:47:38.242 INFO:tasks.workunit.client.1.smithi111.stdout:6/615: creat d2/d35/fcc x:0 0 0 2024-03-23T11:47:38.242 INFO:tasks.workunit.client.1.smithi111.stdout:7/559: dread d14/f25 [0,4194304] 0 2024-03-23T11:47:38.244 INFO:tasks.workunit.client.1.smithi111.stdout:6/616: chown d2/d1f/d45/f76 45568 1 2024-03-23T11:47:38.244 INFO:tasks.workunit.client.1.smithi111.stdout:7/560: stat d14/d44/d70/d7e 0 2024-03-23T11:47:38.246 INFO:tasks.workunit.client.1.smithi111.stdout:6/617: read - d2/d1a/d1c/d2c/fb9 zero size 2024-03-23T11:47:38.247 INFO:tasks.workunit.client.1.smithi111.stdout:5/620: rename d1/d3/d6/d7/d8c/d94 to d1/d20/dce 0 2024-03-23T11:47:38.248 INFO:tasks.workunit.client.1.smithi111.stdout:7/561: dread - d14/d1b/d1e/d2c/d42/d5e/d88/d5f/f68 zero size 2024-03-23T11:47:38.250 INFO:tasks.workunit.client.1.smithi111.stdout:5/621: write d1/d3/fc [11819344,121188] 0 2024-03-23T11:47:38.250 INFO:tasks.workunit.client.0.smithi063.stdout:0/507: rmdir d2/d14/d36/d77/d94/db4 0 2024-03-23T11:47:38.251 INFO:tasks.workunit.client.1.smithi111.stdout:7/562: chown d14/d44/d70/d7e 1818 1 2024-03-23T11:47:38.251 INFO:tasks.workunit.client.1.smithi111.stdout:3/513: mknod d0/d7/d19/d5e/cad 0 2024-03-23T11:47:38.252 INFO:tasks.workunit.client.0.smithi063.stdout:5/321: symlink d2/d17/l72 0 2024-03-23T11:47:38.254 INFO:tasks.workunit.client.1.smithi111.stdout:1/572: symlink d0/d5/lc4 0 2024-03-23T11:47:38.254 INFO:tasks.workunit.client.0.smithi063.stdout:2/407: sync 2024-03-23T11:47:38.258 INFO:tasks.workunit.client.1.smithi111.stdout:2/639: dread d2/d11/d17/d1f/da0/fd4 [0,4194304] 0 2024-03-23T11:47:38.261 INFO:tasks.workunit.client.0.smithi063.stdout:7/425: mkdir d5/d5c/d2f/d8b 0 2024-03-23T11:47:38.261 INFO:tasks.workunit.client.0.smithi063.stdout:3/685: mkdir d13/d22/d2d/d3f/dd4 0 2024-03-23T11:47:38.263 INFO:tasks.workunit.client.0.smithi063.stdout:0/508: rmdir d2/d48/da5/da7 39 2024-03-23T11:47:38.265 INFO:tasks.workunit.client.1.smithi111.stdout:9/619: write d0/d4/d10/d29/d7f/d8e/f9e [990520,56108] 0 2024-03-23T11:47:38.267 INFO:tasks.workunit.client.1.smithi111.stdout:6/618: fdatasync d2/d1f/f26 0 2024-03-23T11:47:38.267 INFO:tasks.workunit.client.0.smithi063.stdout:9/502: mknod d4/d31/d70/d8d/d90/cb2 0 2024-03-23T11:47:38.268 INFO:tasks.workunit.client.1.smithi111.stdout:9/620: read - d0/d4/d1c/d2d/d3b/d42/d92/fc7 zero size 2024-03-23T11:47:38.269 INFO:tasks.workunit.client.1.smithi111.stdout:7/563: fsync d14/d1b/d1e/f39 0 2024-03-23T11:47:38.269 INFO:tasks.workunit.client.0.smithi063.stdout:7/426: chown d5/d5c/d2f/d73/c15 3372736 1 2024-03-23T11:47:38.270 INFO:tasks.workunit.client.1.smithi111.stdout:7/564: fdatasync d14/d37/d50/f63 0 2024-03-23T11:47:38.272 INFO:tasks.workunit.client.0.smithi063.stdout:4/549: getdents d5/da/d20/d4b 0 2024-03-23T11:47:38.273 INFO:tasks.workunit.client.1.smithi111.stdout:1/573: rename d0/d5/dd/d1e/d54/d87/f8a to d0/d6/fc5 0 2024-03-23T11:47:38.274 INFO:tasks.workunit.client.1.smithi111.stdout:2/640: rmdir d2/d5c 39 2024-03-23T11:47:38.275 INFO:tasks.workunit.client.1.smithi111.stdout:1/574: chown d0/d5/f2f 309590962 1 2024-03-23T11:47:38.275 INFO:tasks.workunit.client.1.smithi111.stdout:1/575: stat d0/d6/d48/d4a 0 2024-03-23T11:47:38.277 INFO:tasks.workunit.client.0.smithi063.stdout:1/373: dwrite d3/d3f/d4d/f69 [0,4194304] 0 2024-03-23T11:47:38.281 INFO:tasks.workunit.client.0.smithi063.stdout:0/509: write d2/d4/f9 [1267291,60641] 0 2024-03-23T11:47:38.282 INFO:tasks.workunit.client.0.smithi063.stdout:6/384: getdents d1/dc/d3a 0 2024-03-23T11:47:38.283 INFO:tasks.workunit.client.1.smithi111.stdout:6/619: creat d2/d35/fcd x:0 0 0 2024-03-23T11:47:38.284 INFO:tasks.workunit.client.0.smithi063.stdout:5/322: link d2/d7/fd d2/d7/d8/de/d2a/d3b/f73 0 2024-03-23T11:47:38.288 INFO:tasks.workunit.client.0.smithi063.stdout:2/408: creat d2/d23/d2e/d52/d44/f80 x:0 0 0 2024-03-23T11:47:38.288 INFO:tasks.workunit.client.1.smithi111.stdout:0/586: write d5/f30 [497991,32467] 0 2024-03-23T11:47:38.291 INFO:tasks.workunit.client.0.smithi063.stdout:4/550: mkdir d5/da/d20/d4b/d16/d50/d86/da2 0 2024-03-23T11:47:38.295 INFO:tasks.workunit.client.0.smithi063.stdout:4/551: truncate d5/f90 150299 0 2024-03-23T11:47:38.295 INFO:tasks.workunit.client.0.smithi063.stdout:8/740: write d6/d1c/d27/f34 [5212299,102417] 0 2024-03-23T11:47:38.298 INFO:tasks.workunit.client.0.smithi063.stdout:7/427: dwrite d5/d6/f7a [0,4194304] 0 2024-03-23T11:47:38.300 INFO:tasks.workunit.client.1.smithi111.stdout:4/599: sync 2024-03-23T11:47:38.303 INFO:tasks.workunit.client.0.smithi063.stdout:0/510: dread d2/f56 [4194304,4194304] 0 2024-03-23T11:47:38.309 INFO:tasks.workunit.client.1.smithi111.stdout:2/641: mknod d2/d11/d17/d3e/d6b/d9d/d52/dba/ce9 0 2024-03-23T11:47:38.312 INFO:tasks.workunit.client.0.smithi063.stdout:2/409: mknod d2/d23/d2e/d52/d44/d4d/c81 0 2024-03-23T11:47:38.313 INFO:tasks.workunit.client.0.smithi063.stdout:1/374: symlink d3/d49/l7a 0 2024-03-23T11:47:38.315 INFO:tasks.workunit.client.1.smithi111.stdout:9/621: mknod d0/d4/d6a/da7/cc9 0 2024-03-23T11:47:38.316 INFO:tasks.workunit.client.1.smithi111.stdout:4/600: mknod d0/dc/d4c/d71/cbb 0 2024-03-23T11:47:38.317 INFO:tasks.workunit.client.1.smithi111.stdout:6/620: dread d2/d1a/d1c/d2c/f48 [0,4194304] 0 2024-03-23T11:47:38.318 INFO:tasks.workunit.client.0.smithi063.stdout:8/741: rmdir d6/dd/d59/d7f/db5 39 2024-03-23T11:47:38.318 INFO:tasks.workunit.client.1.smithi111.stdout:7/565: creat d14/d1b/d1e/d2c/d42/d5e/d9e/f9f x:0 0 0 2024-03-23T11:47:38.320 INFO:tasks.workunit.client.1.smithi111.stdout:1/576: mkdir d0/d5/dd/dc6 0 2024-03-23T11:47:38.321 INFO:tasks.workunit.client.1.smithi111.stdout:1/577: chown d0/d5/dd/dc6 634 1 2024-03-23T11:47:38.323 INFO:tasks.workunit.client.1.smithi111.stdout:2/642: creat d2/d7/fea x:0 0 0 2024-03-23T11:47:38.324 INFO:tasks.workunit.client.0.smithi063.stdout:6/385: rename d1/d15/d4b/c58 to d1/dc/d3a/d57/d6d/c74 0 2024-03-23T11:47:38.325 INFO:tasks.workunit.client.0.smithi063.stdout:0/511: truncate d2/d12/d32/d6d/f74 41310 0 2024-03-23T11:47:38.327 INFO:tasks.workunit.client.1.smithi111.stdout:3/514: truncate d0/f16 3400022 0 2024-03-23T11:47:38.328 INFO:tasks.workunit.client.1.smithi111.stdout:7/566: readlink d14/l18 0 2024-03-23T11:47:38.329 INFO:tasks.workunit.client.0.smithi063.stdout:1/375: rmdir d3/d35/d40 39 2024-03-23T11:47:38.334 INFO:tasks.workunit.client.0.smithi063.stdout:9/503: getdents d4/d4c/d19 0 2024-03-23T11:47:38.334 INFO:tasks.workunit.client.1.smithi111.stdout:3/515: creat d0/d1a/d4f/d90/fae x:0 0 0 2024-03-23T11:47:38.338 INFO:tasks.workunit.client.1.smithi111.stdout:2/643: link d2/d11/d17/f21 d2/de/d6c/dd1/dd6/feb 0 2024-03-23T11:47:38.340 INFO:tasks.workunit.client.1.smithi111.stdout:2/644: readlink d2/d11/d17/d1f/d30/lc7 0 2024-03-23T11:47:38.341 INFO:tasks.workunit.client.1.smithi111.stdout:7/567: dwrite d14/f1c [4194304,4194304] 0 2024-03-23T11:47:38.343 INFO:tasks.workunit.client.1.smithi111.stdout:0/587: link d5/db/d4d/fb2 d5/db/dbe/fc2 0 2024-03-23T11:47:38.343 INFO:tasks.workunit.client.1.smithi111.stdout:9/622: link d0/fbc d0/d31/d41/d44/fca 0 2024-03-23T11:47:38.344 INFO:tasks.workunit.client.1.smithi111.stdout:3/516: creat d0/d53/faf x:0 0 0 2024-03-23T11:47:38.345 INFO:tasks.workunit.client.0.smithi063.stdout:4/552: dread d5/fc [0,4194304] 0 2024-03-23T11:47:38.346 INFO:tasks.workunit.client.0.smithi063.stdout:5/323: truncate d2/d7/d8/de/f41 901779 0 2024-03-23T11:47:38.346 INFO:tasks.workunit.client.1.smithi111.stdout:7/568: chown d14/d1b/d1e/d2c/d42/d5e/d8f 61 1 2024-03-23T11:47:38.348 INFO:tasks.workunit.client.0.smithi063.stdout:8/742: dread d6/d1c/f3c [0,4194304] 0 2024-03-23T11:47:38.350 INFO:tasks.workunit.client.1.smithi111.stdout:0/588: mkdir d5/dc3 0 2024-03-23T11:47:38.350 INFO:tasks.workunit.client.0.smithi063.stdout:9/504: readlink d4/d31/l46 0 2024-03-23T11:47:38.351 INFO:tasks.workunit.client.0.smithi063.stdout:2/410: creat d2/d23/d2e/d52/d44/f82 x:0 0 0 2024-03-23T11:47:38.353 INFO:tasks.workunit.client.1.smithi111.stdout:7/569: fdatasync d14/d1b/f52 0 2024-03-23T11:47:38.353 INFO:tasks.workunit.client.0.smithi063.stdout:7/428: getdents d5/d6/d19 0 2024-03-23T11:47:38.355 INFO:tasks.workunit.client.0.smithi063.stdout:0/512: creat d2/d14/d36/d77/fbd x:0 0 0 2024-03-23T11:47:38.357 INFO:tasks.workunit.client.0.smithi063.stdout:5/324: unlink d2/d7/d8/de/c1b 0 2024-03-23T11:47:38.361 INFO:tasks.workunit.client.0.smithi063.stdout:8/743: stat d6/dd/f56 0 2024-03-23T11:47:38.362 INFO:tasks.workunit.client.1.smithi111.stdout:2/645: dwrite d2/d11/d17/d3e/d6b/d2e/d3f/f8e [0,4194304] 0 2024-03-23T11:47:38.363 INFO:tasks.workunit.client.0.smithi063.stdout:9/505: mkdir d4/d31/d70/d8d/db3 0 2024-03-23T11:47:38.363 INFO:tasks.workunit.client.0.smithi063.stdout:0/513: mknod d2/d14/d36/cbe 0 2024-03-23T11:47:38.364 INFO:tasks.workunit.client.1.smithi111.stdout:7/570: chown d14/d44/d70/d7e/c84 24244 1 2024-03-23T11:47:38.364 INFO:tasks.workunit.client.1.smithi111.stdout:2/646: fdatasync d2/d32/fa2 0 2024-03-23T11:47:38.364 INFO:tasks.workunit.client.0.smithi063.stdout:7/429: mkdir d5/d5c/d2f/d8c 0 2024-03-23T11:47:38.365 INFO:tasks.workunit.client.0.smithi063.stdout:6/386: getdents d1/d2a/d41 0 2024-03-23T11:47:38.367 INFO:tasks.workunit.client.1.smithi111.stdout:7/571: write d14/d1b/d1e/d2c/d2e/d33/f90 [266983,66092] 0 2024-03-23T11:47:38.368 INFO:tasks.workunit.client.0.smithi063.stdout:6/387: dread - d1/f67 zero size 2024-03-23T11:47:38.370 INFO:tasks.workunit.client.0.smithi063.stdout:9/506: write d4/d4c/f81 [378956,68709] 0 2024-03-23T11:47:38.371 INFO:tasks.workunit.client.0.smithi063.stdout:7/430: fsync d5/d6/d19/f7d 0 2024-03-23T11:47:38.371 INFO:tasks.workunit.client.0.smithi063.stdout:8/744: write d6/d1f/d3e/f8c [1383265,120764] 0 2024-03-23T11:47:38.372 INFO:tasks.workunit.client.1.smithi111.stdout:6/621: write d2/d35/d70/d98/fbd [855902,110030] 0 2024-03-23T11:47:38.372 INFO:tasks.workunit.client.0.smithi063.stdout:5/325: creat d2/d7/d8/d1d/f74 x:0 0 0 2024-03-23T11:47:38.373 INFO:tasks.workunit.client.0.smithi063.stdout:1/376: truncate d3/d3f/d4d/f5b 324680 0 2024-03-23T11:47:38.376 INFO:tasks.workunit.client.1.smithi111.stdout:4/601: truncate d0/d7/da/f13 6458233 0 2024-03-23T11:47:38.377 INFO:tasks.workunit.client.0.smithi063.stdout:4/553: creat d5/da/d20/d4b/d16/fa3 x:0 0 0 2024-03-23T11:47:38.378 INFO:tasks.workunit.client.0.smithi063.stdout:1/377: dread - d3/d35/f5e zero size 2024-03-23T11:47:38.378 INFO:tasks.workunit.client.0.smithi063.stdout:8/745: stat d6/fe3 0 2024-03-23T11:47:38.378 INFO:tasks.workunit.client.1.smithi111.stdout:4/602: readlink d0/d95/lb7 0 2024-03-23T11:47:38.378 INFO:tasks.workunit.client.1.smithi111.stdout:9/623: creat d0/d4/d1c/d2d/fcb x:0 0 0 2024-03-23T11:47:38.379 INFO:tasks.workunit.client.0.smithi063.stdout:9/507: write d4/d31/d43/f50 [249914,122165] 0 2024-03-23T11:47:38.380 INFO:tasks.workunit.client.0.smithi063.stdout:0/514: fsync d2/d12/d32/f43 0 2024-03-23T11:47:38.382 INFO:tasks.workunit.client.1.smithi111.stdout:9/624: read d0/d4/d10/f2c [2726019,117469] 0 2024-03-23T11:47:38.382 INFO:tasks.workunit.client.0.smithi063.stdout:4/554: chown d5/d1e 0 1 2024-03-23T11:47:38.383 INFO:tasks.workunit.client.0.smithi063.stdout:6/388: rmdir d1/d13 39 2024-03-23T11:47:38.384 INFO:tasks.workunit.client.1.smithi111.stdout:8/495: sync 2024-03-23T11:47:38.384 INFO:tasks.workunit.client.0.smithi063.stdout:1/378: dread - d3/df/f78 zero size 2024-03-23T11:47:38.387 INFO:tasks.workunit.client.0.smithi063.stdout:5/326: write d2/d7/d8/de/d35/f60 [580076,122445] 0 2024-03-23T11:47:38.387 INFO:tasks.workunit.client.0.smithi063.stdout:0/515: chown d2/d12/d32/f43 14 1 2024-03-23T11:47:38.389 INFO:tasks.workunit.client.0.smithi063.stdout:2/411: dwrite d2/d1c/f20 [0,4194304] 0 2024-03-23T11:47:38.389 INFO:tasks.workunit.client.1.smithi111.stdout:3/517: link d0/d1a/d1d/c2e d0/d72/d7b/cb0 0 2024-03-23T11:47:38.391 INFO:tasks.workunit.client.1.smithi111.stdout:1/578: dwrite d0/d2b/f2e [0,4194304] 0 2024-03-23T11:47:38.393 INFO:tasks.workunit.client.1.smithi111.stdout:7/572: mknod d14/d44/d4b/ca0 0 2024-03-23T11:47:38.394 INFO:tasks.workunit.client.1.smithi111.stdout:7/573: write d14/d37/d50/f63 [304302,38525] 0 2024-03-23T11:47:38.405 INFO:tasks.workunit.client.1.smithi111.stdout:4/603: truncate d0/dc/d1f/f4a 4278113 0 2024-03-23T11:47:38.406 INFO:tasks.workunit.client.0.smithi063.stdout:4/555: rmdir d5/da 39 2024-03-23T11:47:38.407 INFO:tasks.workunit.client.0.smithi063.stdout:0/516: fdatasync d2/d14/f1c 0 2024-03-23T11:47:38.407 INFO:tasks.workunit.client.1.smithi111.stdout:4/604: read d0/dc/d4c/d2a/d97/d4d/f8d [40989,89650] 0 2024-03-23T11:47:38.409 INFO:tasks.workunit.client.1.smithi111.stdout:2/647: symlink d2/lec 0 2024-03-23T11:47:38.410 INFO:tasks.workunit.client.0.smithi063.stdout:9/508: symlink d4/lb4 0 2024-03-23T11:47:38.410 INFO:tasks.workunit.client.0.smithi063.stdout:0/517: write d2/d12/d32/f6c [4610834,103180] 0 2024-03-23T11:47:38.411 INFO:tasks.workunit.client.1.smithi111.stdout:8/496: mknod d5/d26/d2f/d62/d75/d4b/d51/d5a/ca2 0 2024-03-23T11:47:38.412 INFO:tasks.workunit.client.1.smithi111.stdout:3/518: creat d0/d1a/d2b/fb1 x:0 0 0 2024-03-23T11:47:38.418 INFO:tasks.workunit.client.1.smithi111.stdout:0/589: getdents d5/db/d4d 0 2024-03-23T11:47:38.419 INFO:tasks.workunit.client.1.smithi111.stdout:9/625: symlink d0/d4/d1c/d2d/d3b/lcc 0 2024-03-23T11:47:38.422 INFO:tasks.workunit.client.0.smithi063.stdout:1/379: mkdir d3/df/d7b 0 2024-03-23T11:47:38.422 INFO:tasks.workunit.client.1.smithi111.stdout:4/605: rename d0/l1a to d0/dc/d4c/d71/lbc 0 2024-03-23T11:47:38.423 INFO:tasks.workunit.client.0.smithi063.stdout:7/431: creat d5/d5c/d2f/d73/f8d x:0 0 0 2024-03-23T11:47:38.425 INFO:tasks.workunit.client.1.smithi111.stdout:3/519: creat d0/d1a/d4f/d90/fb2 x:0 0 0 2024-03-23T11:47:38.427 INFO:tasks.workunit.client.1.smithi111.stdout:0/590: unlink d5/db/d4d/f52 0 2024-03-23T11:47:38.428 INFO:tasks.workunit.client.0.smithi063.stdout:5/327: creat d2/d28/d3a/d46/d61/f75 x:0 0 0 2024-03-23T11:47:38.431 INFO:tasks.workunit.client.0.smithi063.stdout:5/328: dread - d2/d28/d3a/d46/d61/f75 zero size 2024-03-23T11:47:38.438 INFO:tasks.workunit.client.1.smithi111.stdout:2/648: rmdir d2/d11/d17/daf 0 2024-03-23T11:47:38.438 INFO:tasks.workunit.client.0.smithi063.stdout:1/380: creat d3/d49/d56/d65/f7c x:0 0 0 2024-03-23T11:47:38.439 INFO:tasks.workunit.client.0.smithi063.stdout:2/412: creat d2/d23/f83 x:0 0 0 2024-03-23T11:47:38.440 INFO:tasks.workunit.client.1.smithi111.stdout:4/606: link d0/dc/d4c/d2a/d97/d4d/f8d d0/dc/d89/da4/fbd 0 2024-03-23T11:47:38.440 INFO:tasks.workunit.client.0.smithi063.stdout:8/746: getdents d6/d1f/d54/d5c 0 2024-03-23T11:47:38.441 INFO:tasks.workunit.client.1.smithi111.stdout:8/497: dread d5/f1a [0,4194304] 0 2024-03-23T11:47:38.441 INFO:tasks.workunit.client.1.smithi111.stdout:7/574: getdents d14/d1b/d1e/d2c/d42/d5e 0 2024-03-23T11:47:38.442 INFO:tasks.workunit.client.0.smithi063.stdout:5/329: rmdir d2/d7/d64 39 2024-03-23T11:47:38.446 INFO:tasks.workunit.client.1.smithi111.stdout:3/520: dwrite d0/d1a/d2b/d38/d7f/f8a [0,4194304] 0 2024-03-23T11:47:38.446 INFO:tasks.workunit.client.0.smithi063.stdout:6/389: getdents d1/dc 0 2024-03-23T11:47:38.448 INFO:tasks.workunit.client.1.smithi111.stdout:7/575: mknod d14/d1b/d1e/d2c/d42/ca1 0 2024-03-23T11:47:38.452 INFO:tasks.workunit.client.0.smithi063.stdout:0/518: rmdir d2/d14/d36/d77/d94/da6/d25/d93/db2 0 2024-03-23T11:47:38.452 INFO:tasks.workunit.client.0.smithi063.stdout:1/381: write d3/f5d [282783,15040] 0 2024-03-23T11:47:38.454 INFO:tasks.workunit.client.1.smithi111.stdout:8/498: rename d5/l15 to d5/d26/d2f/d62/d75/la3 0 2024-03-23T11:47:38.454 INFO:tasks.workunit.client.0.smithi063.stdout:6/390: stat d1/d2a/d41/f5e 0 2024-03-23T11:47:38.456 INFO:tasks.workunit.client.1.smithi111.stdout:8/499: write d5/d26/d31/d5e/f3e [950350,60623] 0 2024-03-23T11:47:38.456 INFO:tasks.workunit.client.1.smithi111.stdout:0/591: link d5/db/d1f/d48/c95 d5/db/d1f/d47/d59/cc4 0 2024-03-23T11:47:38.458 INFO:tasks.workunit.client.1.smithi111.stdout:7/576: read d14/d1b/d1e/d2c/d42/f45 [1388011,23262] 0 2024-03-23T11:47:38.459 INFO:tasks.workunit.client.0.smithi063.stdout:1/382: dread - d3/d3f/f77 zero size 2024-03-23T11:47:38.459 INFO:tasks.workunit.client.0.smithi063.stdout:2/413: mkdir d2/d84 0 2024-03-23T11:47:38.462 INFO:tasks.workunit.client.0.smithi063.stdout:4/556: rename d5/f59 to d5/da/fa4 0 2024-03-23T11:47:38.465 INFO:tasks.workunit.client.0.smithi063.stdout:0/519: chown d2/d14/d36/d77/d94/da6/d1f/fb5 208436400 1 2024-03-23T11:47:38.466 INFO:tasks.workunit.client.0.smithi063.stdout:7/432: dwrite d5/d5c/d2f/d73/da/f8a [0,4194304] 0 2024-03-23T11:47:38.470 INFO:tasks.workunit.client.0.smithi063.stdout:9/509: dwrite d4/f74 [0,4194304] 0 2024-03-23T11:47:38.473 INFO:tasks.workunit.client.0.smithi063.stdout:6/391: dread d1/d2a/d41/f5e [0,4194304] 0 2024-03-23T11:47:38.474 INFO:tasks.workunit.client.1.smithi111.stdout:7/577: creat d14/d1b/d1e/d2c/d42/d5e/d9e/fa2 x:0 0 0 2024-03-23T11:47:38.477 INFO:tasks.workunit.client.1.smithi111.stdout:6/622: write d2/d1a/d1c/d2c/f48 [630774,109873] 0 2024-03-23T11:47:38.478 INFO:tasks.workunit.client.0.smithi063.stdout:9/510: write d4/d4c/d19/f23 [3582615,96702] 0 2024-03-23T11:47:38.482 INFO:tasks.workunit.client.0.smithi063.stdout:9/511: fdatasync d4/d31/d43/da9/f95 0 2024-03-23T11:47:38.484 INFO:tasks.workunit.client.1.smithi111.stdout:6/623: mkdir d2/d1f/dce 0 2024-03-23T11:47:38.489 INFO:tasks.workunit.client.1.smithi111.stdout:6/624: write d2/d1a/d1c/d2c/d51/f8f [4880600,87370] 0 2024-03-23T11:47:38.492 INFO:tasks.workunit.client.0.smithi063.stdout:8/747: link d6/d1f/d54/d63/c88 d6/cfd 0 2024-03-23T11:47:38.493 INFO:tasks.workunit.client.1.smithi111.stdout:6/625: write d2/d1f/d45/fbc [1069849,80152] 0 2024-03-23T11:47:38.494 INFO:tasks.workunit.client.1.smithi111.stdout:7/578: dwrite d14/d1b/d1e/d2c/f38 [4194304,4194304] 0 2024-03-23T11:47:38.498 INFO:tasks.workunit.client.1.smithi111.stdout:7/579: write f9 [4640362,77757] 0 2024-03-23T11:47:38.498 INFO:tasks.workunit.client.0.smithi063.stdout:6/392: mknod d1/dc/d21/d49/c75 0 2024-03-23T11:47:38.499 INFO:tasks.workunit.client.0.smithi063.stdout:2/414: creat d2/f85 x:0 0 0 2024-03-23T11:47:38.500 INFO:tasks.workunit.client.1.smithi111.stdout:7/580: write d14/d37/d50/f63 [683305,13201] 0 2024-03-23T11:47:38.503 INFO:tasks.workunit.client.1.smithi111.stdout:0/592: dread d5/db/d1f/d19/f2a [0,4194304] 0 2024-03-23T11:47:38.504 INFO:tasks.workunit.client.1.smithi111.stdout:6/626: chown d2/cc 0 1 2024-03-23T11:47:38.506 INFO:tasks.workunit.client.1.smithi111.stdout:6/627: stat d2/d1a/c5b 0 2024-03-23T11:47:38.506 INFO:tasks.workunit.client.0.smithi063.stdout:4/557: creat d5/da/d20/d4b/fa5 x:0 0 0 2024-03-23T11:47:38.507 INFO:tasks.workunit.client.1.smithi111.stdout:6/628: stat d2/d17 0 2024-03-23T11:47:38.507 INFO:tasks.workunit.client.1.smithi111.stdout:5/622: sync 2024-03-23T11:47:38.507 INFO:tasks.workunit.client.0.smithi063.stdout:1/383: dwrite d3/d49/d56/d65/f7c [0,4194304] 0 2024-03-23T11:47:38.507 INFO:tasks.workunit.client.0.smithi063.stdout:0/520: creat d2/d14/d36/d77/d94/da6/d25/d93/fbf x:0 0 0 2024-03-23T11:47:38.508 INFO:tasks.workunit.client.1.smithi111.stdout:7/581: fsync fe 0 2024-03-23T11:47:38.512 INFO:tasks.workunit.client.1.smithi111.stdout:7/582: chown d14/d1b/c80 200592 1 2024-03-23T11:47:38.515 INFO:tasks.workunit.client.1.smithi111.stdout:9/626: truncate d0/d4/d1c/d36/f59 6518094 0 2024-03-23T11:47:38.520 INFO:tasks.workunit.client.0.smithi063.stdout:9/512: dwrite d4/d4c/f5c [0,4194304] 0 2024-03-23T11:47:38.521 INFO:tasks.workunit.client.0.smithi063.stdout:0/521: creat d2/d12/d32/fc0 x:0 0 0 2024-03-23T11:47:38.530 INFO:tasks.workunit.client.1.smithi111.stdout:4/607: write d0/fa8 [718054,129115] 0 2024-03-23T11:47:38.531 INFO:tasks.workunit.client.1.smithi111.stdout:7/583: rmdir d14/d1b/d22 39 2024-03-23T11:47:38.534 INFO:tasks.workunit.client.1.smithi111.stdout:9/627: creat d0/d31/d41/d65/fcd x:0 0 0 2024-03-23T11:47:38.535 INFO:tasks.workunit.client.1.smithi111.stdout:7/584: truncate d14/d37/f9d 1922670 0 2024-03-23T11:47:38.537 INFO:tasks.workunit.client.1.smithi111.stdout:4/608: creat d0/dc/d4c/d2a/d97/d4d/fbe x:0 0 0 2024-03-23T11:47:38.539 INFO:tasks.workunit.client.0.smithi063.stdout:2/415: mkdir d2/d23/d2e/d52/d86 0 2024-03-23T11:47:38.543 INFO:tasks.workunit.client.1.smithi111.stdout:3/521: write d0/d72/d7b/f98 [1134243,6844] 0 2024-03-23T11:47:38.544 INFO:tasks.workunit.client.1.smithi111.stdout:0/593: rename d5/db/d1f/d19/dbd/l6c to d5/db/lc5 0 2024-03-23T11:47:38.546 INFO:tasks.workunit.client.1.smithi111.stdout:3/522: chown d0/d1a/d2b/d38 37 1 2024-03-23T11:47:38.547 INFO:tasks.workunit.client.1.smithi111.stdout:4/609: fdatasync d0/dc/d4c/d2a/f38 0 2024-03-23T11:47:38.549 INFO:tasks.workunit.client.0.smithi063.stdout:0/522: creat d2/d14/d36/d77/d94/da6/d1f/d4f/fc1 x:0 0 0 2024-03-23T11:47:38.549 INFO:tasks.workunit.client.0.smithi063.stdout:6/393: dwrite d1/d15/f70 [0,4194304] 0 2024-03-23T11:47:38.551 INFO:tasks.workunit.client.1.smithi111.stdout:5/623: getdents d1 0 2024-03-23T11:47:38.551 INFO:tasks.workunit.client.0.smithi063.stdout:7/433: link d5/l7e d5/d6/d19/d1f/d47/l8e 0 2024-03-23T11:47:38.551 INFO:tasks.workunit.client.1.smithi111.stdout:9/628: dwrite d0/f21 [4194304,4194304] 0 2024-03-23T11:47:38.552 INFO:tasks.workunit.client.0.smithi063.stdout:5/330: dread d2/d7/d8/de/d49/f5d [0,4194304] 0 2024-03-23T11:47:38.553 INFO:tasks.workunit.client.1.smithi111.stdout:9/629: stat d0/d4/d10/d29/l79 0 2024-03-23T11:47:38.554 INFO:tasks.workunit.client.1.smithi111.stdout:3/523: fdatasync d0/fd 0 2024-03-23T11:47:38.560 INFO:tasks.workunit.client.1.smithi111.stdout:4/610: unlink d0/dc/d4c/d2a/d97/d4d/l6c 0 2024-03-23T11:47:38.562 INFO:tasks.workunit.client.0.smithi063.stdout:1/384: dwrite d3/df/f18 [0,4194304] 0 2024-03-23T11:47:38.564 INFO:tasks.workunit.client.0.smithi063.stdout:7/434: truncate d5/d6/f25 4926534 0 2024-03-23T11:47:38.569 INFO:tasks.workunit.client.0.smithi063.stdout:4/558: symlink d5/da/d20/d4b/d16/d50/la6 0 2024-03-23T11:47:38.570 INFO:tasks.workunit.client.0.smithi063.stdout:9/513: dwrite d4/d31/d43/f50 [0,4194304] 0 2024-03-23T11:47:38.570 INFO:tasks.workunit.client.0.smithi063.stdout:7/435: fsync d5/d6/f7a 0 2024-03-23T11:47:38.574 INFO:tasks.workunit.client.0.smithi063.stdout:9/514: readlink d4/d31/laf 0 2024-03-23T11:47:38.575 INFO:tasks.workunit.client.0.smithi063.stdout:6/394: dread d1/d2a/d41/f5e [0,4194304] 0 2024-03-23T11:47:38.576 INFO:tasks.workunit.client.0.smithi063.stdout:8/748: write d6/d1f/d54/d63/d90/f13 [157139,67614] 0 2024-03-23T11:47:38.579 INFO:tasks.workunit.client.0.smithi063.stdout:6/395: write d1/d15/f70 [3590404,12320] 0 2024-03-23T11:47:38.582 INFO:tasks.workunit.client.0.smithi063.stdout:7/436: fsync d5/d6/d19/d1f/d47/d4f/d5b/f7b 0 2024-03-23T11:47:38.583 INFO:tasks.workunit.client.0.smithi063.stdout:8/749: chown d6/dd/df/f61 0 1 2024-03-23T11:47:38.584 INFO:tasks.workunit.client.0.smithi063.stdout:6/396: chown d1/d2a/d41/d42 1 1 2024-03-23T11:47:38.585 INFO:tasks.workunit.client.0.smithi063.stdout:0/523: rename d2/d12/d32/c39 to d2/d14/d36/d77/d94/da6/cc2 0 2024-03-23T11:47:38.590 INFO:tasks.workunit.client.1.smithi111.stdout:4/611: symlink d0/dc/d4c/d71/lbf 0 2024-03-23T11:47:38.592 INFO:tasks.workunit.client.1.smithi111.stdout:6/629: dread d2/d1a/f22 [0,4194304] 0 2024-03-23T11:47:38.593 INFO:tasks.workunit.client.0.smithi063.stdout:7/437: mknod d5/d6/d19/d1f/d47/d6e/c8f 0 2024-03-23T11:47:38.596 INFO:tasks.workunit.client.0.smithi063.stdout:6/397: write d1/d2a/d41/d42/f6f [273632,65824] 0 2024-03-23T11:47:38.596 INFO:tasks.workunit.client.0.smithi063.stdout:8/750: fdatasync d6/d1f/d54/d63/d90/d15/f24 0 2024-03-23T11:47:38.599 INFO:tasks.workunit.client.0.smithi063.stdout:0/524: chown d2/d14/d36/d77/d94/da6/cbc 41809093 1 2024-03-23T11:47:38.605 INFO:tasks.workunit.client.0.smithi063.stdout:9/515: truncate d4/d31/f3d 935314 0 2024-03-23T11:47:38.609 INFO:tasks.workunit.client.0.smithi063.stdout:5/331: write d2/d17/f19 [1851760,85484] 0 2024-03-23T11:47:38.612 INFO:tasks.workunit.client.0.smithi063.stdout:6/398: creat d1/dc/d21/d49/f76 x:0 0 0 2024-03-23T11:47:38.612 INFO:tasks.workunit.client.1.smithi111.stdout:4/612: dwrite d0/dc/d1f/db0/fa7 [0,4194304] 0 2024-03-23T11:47:38.615 INFO:tasks.workunit.client.0.smithi063.stdout:1/385: dwrite d3/df/f6e [0,4194304] 0 2024-03-23T11:47:38.624 INFO:tasks.workunit.client.1.smithi111.stdout:7/585: dread f11 [0,4194304] 0 2024-03-23T11:47:38.625 INFO:tasks.workunit.client.1.smithi111.stdout:7/586: stat d14/d37/d43 0 2024-03-23T11:47:38.626 INFO:tasks.workunit.client.0.smithi063.stdout:0/525: dwrite d2/d12/d32/f89 [0,4194304] 0 2024-03-23T11:47:38.626 INFO:tasks.workunit.client.1.smithi111.stdout:7/587: chown d14/d37/d43 586 1 2024-03-23T11:47:38.626 INFO:tasks.workunit.client.0.smithi063.stdout:8/751: mkdir d6/dd/d59/d7f/db5/dfe 0 2024-03-23T11:47:38.628 INFO:tasks.workunit.client.0.smithi063.stdout:0/526: stat d2/d4/d63/cbb 0 2024-03-23T11:47:38.636 INFO:tasks.workunit.client.0.smithi063.stdout:4/559: truncate d5/d1e/d2b/f7c 2056187 0 2024-03-23T11:47:38.636 INFO:tasks.workunit.client.1.smithi111.stdout:6/630: creat d2/d1f/fcf x:0 0 0 2024-03-23T11:47:38.641 INFO:tasks.workunit.client.0.smithi063.stdout:5/332: creat d2/d17/d4d/d6e/f76 x:0 0 0 2024-03-23T11:47:38.641 INFO:tasks.workunit.client.0.smithi063.stdout:8/752: dread - d6/dd/d59/d7f/f97 zero size 2024-03-23T11:47:38.648 INFO:tasks.workunit.client.0.smithi063.stdout:4/560: creat d5/da/d20/d4b/d16/d50/d8e/fa7 x:0 0 0 2024-03-23T11:47:38.648 INFO:tasks.workunit.client.1.smithi111.stdout:6/631: symlink d2/d1f/d45/d9a/ld0 0 2024-03-23T11:47:38.650 INFO:tasks.workunit.client.0.smithi063.stdout:5/333: symlink d2/d7/d8/d1d/l77 0 2024-03-23T11:47:38.652 INFO:tasks.workunit.client.1.smithi111.stdout:5/624: dwrite d1/d3/d3a/f3d [0,4194304] 0 2024-03-23T11:47:38.652 INFO:tasks.workunit.client.1.smithi111.stdout:6/632: write d2/d17/fac [550104,10393] 0 2024-03-23T11:47:38.653 INFO:tasks.workunit.client.1.smithi111.stdout:7/588: creat d14/d1b/fa3 x:0 0 0 2024-03-23T11:47:38.653 INFO:tasks.workunit.client.1.smithi111.stdout:3/524: write d0/d7/f12 [2591387,36043] 0 2024-03-23T11:47:38.656 INFO:tasks.workunit.client.0.smithi063.stdout:9/516: write d4/f35 [3436338,92731] 0 2024-03-23T11:47:38.656 INFO:tasks.workunit.client.0.smithi063.stdout:5/334: dread - d2/d28/d3a/f4e zero size 2024-03-23T11:47:38.658 INFO:tasks.workunit.client.0.smithi063.stdout:4/561: rmdir d5/da/d20/d4b/d16/d37 39 2024-03-23T11:47:38.658 INFO:tasks.workunit.client.1.smithi111.stdout:6/633: fdatasync d2/d1f/d45/d49/d8d/f94 0 2024-03-23T11:47:38.666 INFO:tasks.workunit.client.0.smithi063.stdout:9/517: unlink d4/f1e 0 2024-03-23T11:47:38.669 INFO:tasks.workunit.client.0.smithi063.stdout:1/386: write d3/d1c/f1e [463881,6244] 0 2024-03-23T11:47:38.672 INFO:tasks.workunit.client.1.smithi111.stdout:9/630: dwrite d0/fbc [0,4194304] 0 2024-03-23T11:47:38.673 INFO:tasks.workunit.client.1.smithi111.stdout:7/589: getdents d14/d37/d43 0 2024-03-23T11:47:38.677 INFO:tasks.workunit.client.0.smithi063.stdout:5/335: creat d2/d28/d3a/d46/d71/f78 x:0 0 0 2024-03-23T11:47:38.678 INFO:tasks.workunit.client.1.smithi111.stdout:9/631: read - d0/d31/d41/d9d/fc8 zero size 2024-03-23T11:47:38.678 INFO:tasks.workunit.client.1.smithi111.stdout:7/590: read fe [4100532,59117] 0 2024-03-23T11:47:38.678 INFO:tasks.workunit.client.0.smithi063.stdout:0/527: dread d2/d12/d35/f6a [0,4194304] 0 2024-03-23T11:47:38.681 INFO:tasks.workunit.client.0.smithi063.stdout:6/399: dwrite d1/dc/fd [4194304,4194304] 0 2024-03-23T11:47:38.682 INFO:tasks.workunit.client.1.smithi111.stdout:3/525: dwrite d0/fd [0,4194304] 0 2024-03-23T11:47:38.685 INFO:tasks.workunit.client.1.smithi111.stdout:9/632: read d0/d31/d41/d44/fca [463720,55482] 0 2024-03-23T11:47:38.686 INFO:tasks.workunit.client.0.smithi063.stdout:7/438: dwrite d5/d6/d19/d1f/f6d [0,4194304] 0 2024-03-23T11:47:38.691 INFO:tasks.workunit.client.0.smithi063.stdout:1/387: creat d3/df/f7d x:0 0 0 2024-03-23T11:47:38.693 INFO:tasks.workunit.client.0.smithi063.stdout:6/400: write d1/d13/f52 [4081105,39340] 0 2024-03-23T11:47:38.694 INFO:tasks.workunit.client.0.smithi063.stdout:0/528: unlink d2/d14/f31 0 2024-03-23T11:47:38.694 INFO:tasks.workunit.client.1.smithi111.stdout:7/591: symlink d14/d37/d50/la4 0 2024-03-23T11:47:38.696 INFO:tasks.workunit.client.1.smithi111.stdout:4/613: dread d0/dc/d4c/f39 [0,4194304] 0 2024-03-23T11:47:38.696 INFO:tasks.workunit.client.0.smithi063.stdout:5/336: unlink d2/d28/d3a/d46/d61/f75 0 2024-03-23T11:47:38.699 INFO:tasks.workunit.client.0.smithi063.stdout:4/562: dread d5/da/d20/d4b/d29/d38/d7a/f8a [0,4194304] 0 2024-03-23T11:47:38.699 INFO:tasks.workunit.client.1.smithi111.stdout:9/633: write d0/d4/fbd [247427,116054] 0 2024-03-23T11:47:38.702 INFO:tasks.workunit.client.1.smithi111.stdout:5/625: dwrite d1/d3/d6/d7/d63/d2c/f34 [0,4194304] 0 2024-03-23T11:47:38.703 INFO:tasks.workunit.client.0.smithi063.stdout:7/439: mkdir d5/d5c/d2f/d73/da/d90 0 2024-03-23T11:47:38.703 INFO:tasks.workunit.client.0.smithi063.stdout:6/401: fsync d1/d13/f52 0 2024-03-23T11:47:38.703 INFO:tasks.workunit.client.0.smithi063.stdout:1/388: creat d3/d49/d56/d65/f7e x:0 0 0 2024-03-23T11:47:38.706 INFO:tasks.workunit.client.1.smithi111.stdout:3/526: creat d0/d1a/fb3 x:0 0 0 2024-03-23T11:47:38.706 INFO:tasks.workunit.client.0.smithi063.stdout:1/389: write d3/f11 [4306927,129221] 0 2024-03-23T11:47:38.707 INFO:tasks.workunit.client.0.smithi063.stdout:0/529: mkdir d2/d14/d36/dc3 0 2024-03-23T11:47:38.710 INFO:tasks.workunit.client.0.smithi063.stdout:5/337: mkdir d2/d28/d3a/d46/d79 0 2024-03-23T11:47:38.713 INFO:tasks.workunit.client.0.smithi063.stdout:4/563: unlink d5/da/d20/d4b/d16/d50/f70 0 2024-03-23T11:47:38.714 INFO:tasks.workunit.client.1.smithi111.stdout:6/634: dwrite d2/d1f/d45/d49/f6d [0,4194304] 0 2024-03-23T11:47:38.714 INFO:tasks.workunit.client.0.smithi063.stdout:5/338: dread - d2/d7/d8/de/d2a/d3b/d43/f45 zero size 2024-03-23T11:47:38.715 INFO:tasks.workunit.client.0.smithi063.stdout:6/402: mkdir d1/dc/d3a/d57/d77 0 2024-03-23T11:47:38.718 INFO:tasks.workunit.client.1.smithi111.stdout:8/500: sync 2024-03-23T11:47:38.718 INFO:tasks.workunit.client.1.smithi111.stdout:2/649: sync 2024-03-23T11:47:38.718 INFO:tasks.workunit.client.1.smithi111.stdout:1/579: sync 2024-03-23T11:47:38.719 INFO:tasks.workunit.client.1.smithi111.stdout:7/592: creat d14/d1b/d1e/fa5 x:0 0 0 2024-03-23T11:47:38.724 INFO:tasks.workunit.client.0.smithi063.stdout:6/403: chown d1/d13/c33 392889 1 2024-03-23T11:47:38.724 INFO:tasks.workunit.client.1.smithi111.stdout:7/593: dread - d14/d1b/fa3 zero size 2024-03-23T11:47:38.727 INFO:tasks.workunit.client.0.smithi063.stdout:4/564: chown d5/da/d20/d4b/d16/d32/l4a 21984162 1 2024-03-23T11:47:38.728 INFO:tasks.workunit.client.0.smithi063.stdout:7/440: creat d5/d5c/d2f/d73/f91 x:0 0 0 2024-03-23T11:47:38.734 INFO:tasks.workunit.client.1.smithi111.stdout:4/614: link d0/dc/d4c/d2a/d97/d4d/f51 d0/dc/d78/fc0 0 2024-03-23T11:47:38.736 INFO:tasks.workunit.client.0.smithi063.stdout:0/530: rmdir d2/d12/d32/db1 0 2024-03-23T11:47:38.737 INFO:tasks.workunit.client.0.smithi063.stdout:6/404: symlink d1/dc/d3a/l78 0 2024-03-23T11:47:38.740 INFO:tasks.workunit.client.0.smithi063.stdout:1/390: dwrite d3/df/f78 [0,4194304] 0 2024-03-23T11:47:38.742 INFO:tasks.workunit.client.1.smithi111.stdout:1/580: creat d0/d2b/fc7 x:0 0 0 2024-03-23T11:47:38.742 INFO:tasks.workunit.client.1.smithi111.stdout:2/650: creat d2/de/d12/d70/fed x:0 0 0 2024-03-23T11:47:38.743 INFO:tasks.workunit.client.1.smithi111.stdout:3/527: dwrite d0/d1a/d2b/f33 [0,4194304] 0 2024-03-23T11:47:38.744 INFO:tasks.workunit.client.0.smithi063.stdout:9/518: dread d4/d31/d43/da9/f80 [4194304,4194304] 0 2024-03-23T11:47:38.747 INFO:tasks.workunit.client.1.smithi111.stdout:7/594: symlink d14/d1b/d1e/d2c/d2e/d33/d98/la6 0 2024-03-23T11:47:38.749 INFO:tasks.workunit.client.1.smithi111.stdout:9/634: dwrite d0/d31/d41/d71/f93 [0,4194304] 0 2024-03-23T11:47:38.749 INFO:tasks.workunit.client.0.smithi063.stdout:7/441: creat d5/d6/d19/f92 x:0 0 0 2024-03-23T11:47:38.751 INFO:tasks.workunit.client.0.smithi063.stdout:5/339: dread d2/d7/d8/de/f41 [0,4194304] 0 2024-03-23T11:47:38.752 INFO:tasks.workunit.client.1.smithi111.stdout:1/581: unlink d0/d6/d48/f55 0 2024-03-23T11:47:38.758 INFO:tasks.workunit.client.1.smithi111.stdout:8/501: dread d5/d26/d2f/f77 [0,4194304] 0 2024-03-23T11:47:38.758 INFO:tasks.workunit.client.1.smithi111.stdout:5/626: write d1/d3/d6/d8b/f26 [391620,64225] 0 2024-03-23T11:47:38.759 INFO:tasks.workunit.client.0.smithi063.stdout:0/531: mknod d2/d12/d32/cc4 0 2024-03-23T11:47:38.761 INFO:tasks.workunit.client.0.smithi063.stdout:9/519: mkdir d4/d31/d43/da9/da6/db5 0 2024-03-23T11:47:38.764 INFO:tasks.workunit.client.1.smithi111.stdout:6/635: getdents d2 0 2024-03-23T11:47:38.765 INFO:tasks.workunit.client.1.smithi111.stdout:5/627: stat d1/d3/d6/d7/fc6 0 2024-03-23T11:47:38.768 INFO:tasks.workunit.client.1.smithi111.stdout:7/595: rmdir d14/d37/d54 39 2024-03-23T11:47:38.768 INFO:tasks.workunit.client.1.smithi111.stdout:2/651: truncate d2/d7/da/d19/f34 1692202 0 2024-03-23T11:47:38.769 INFO:tasks.workunit.client.1.smithi111.stdout:1/582: rename d0/d2b/lb5 to d0/d6/d48/d4d/lc8 0 2024-03-23T11:47:38.771 INFO:tasks.workunit.client.1.smithi111.stdout:0/594: sync 2024-03-23T11:47:38.771 INFO:tasks.workunit.client.1.smithi111.stdout:6/636: creat d2/d1f/d45/d86/fd1 x:0 0 0 2024-03-23T11:47:38.771 INFO:tasks.workunit.client.0.smithi063.stdout:5/340: fsync f1 0 2024-03-23T11:47:38.772 INFO:tasks.workunit.client.1.smithi111.stdout:5/628: mkdir d1/dcf 0 2024-03-23T11:47:38.774 INFO:tasks.workunit.client.1.smithi111.stdout:9/635: creat d0/d4/d1c/d2d/d75/fce x:0 0 0 2024-03-23T11:47:38.777 INFO:tasks.workunit.client.0.smithi063.stdout:5/341: write d2/d7/d8/f68 [913248,26216] 0 2024-03-23T11:47:38.777 INFO:tasks.workunit.client.1.smithi111.stdout:0/595: read - d5/db/d1f/d48/f80 zero size 2024-03-23T11:47:38.778 INFO:tasks.workunit.client.0.smithi063.stdout:5/342: dread - d2/d17/d4d/d6e/f76 zero size 2024-03-23T11:47:38.779 INFO:tasks.workunit.client.1.smithi111.stdout:3/528: dwrite d0/d1a/d2b/d38/d7f/f9b [0,4194304] 0 2024-03-23T11:47:38.781 INFO:tasks.workunit.client.1.smithi111.stdout:2/652: unlink d2/d11/fb7 0 2024-03-23T11:47:38.783 INFO:tasks.workunit.client.1.smithi111.stdout:3/529: chown d0/d7/d19/d5e/da6 0 1 2024-03-23T11:47:38.784 INFO:tasks.workunit.client.0.smithi063.stdout:5/343: fsync f1 0 2024-03-23T11:47:38.784 INFO:tasks.workunit.client.1.smithi111.stdout:5/629: symlink d1/d20/dce/ld0 0 2024-03-23T11:47:38.785 INFO:tasks.workunit.client.1.smithi111.stdout:4/615: dread d0/d7/f1b [0,4194304] 0 2024-03-23T11:47:38.787 INFO:tasks.workunit.client.1.smithi111.stdout:5/630: fsync d1/d3/d6/d30/fc4 0 2024-03-23T11:47:38.789 INFO:tasks.workunit.client.1.smithi111.stdout:8/502: rename d5/d40 to d5/d26/d2f/d62/d75/d4b/d51/da4 0 2024-03-23T11:47:38.793 INFO:tasks.workunit.client.0.smithi063.stdout:1/391: dwrite d3/d1c/f6d [4194304,4194304] 0 2024-03-23T11:47:38.793 INFO:tasks.workunit.client.1.smithi111.stdout:8/503: chown d5/d26/d2f/d62/d75/d4b/d51/da4/l5b 13975107 1 2024-03-23T11:47:38.794 INFO:tasks.workunit.client.1.smithi111.stdout:3/530: fsync d0/d1a/d2d/f45 0 2024-03-23T11:47:38.798 INFO:tasks.workunit.client.0.smithi063.stdout:1/392: chown d3/df/l14 22739275 1 2024-03-23T11:47:38.800 INFO:tasks.workunit.client.0.smithi063.stdout:6/405: dwrite d1/dc/d21/f3c [0,4194304] 0 2024-03-23T11:47:38.803 INFO:tasks.workunit.client.1.smithi111.stdout:4/616: dwrite d0/dc/d4c/d2a/d97/d4d/fbe [0,4194304] 0 2024-03-23T11:47:38.804 INFO:tasks.workunit.client.0.smithi063.stdout:6/406: fsync d1/d13/f52 0 2024-03-23T11:47:38.806 INFO:tasks.workunit.client.1.smithi111.stdout:1/583: rename d0/c28 to d0/d5/dd/d1e/d70/cc9 0 2024-03-23T11:47:38.806 INFO:tasks.workunit.client.0.smithi063.stdout:1/393: chown d3/c44 0 1 2024-03-23T11:47:38.807 INFO:tasks.workunit.client.1.smithi111.stdout:4/617: truncate d0/dc/d78/f98 291277 0 2024-03-23T11:47:38.808 INFO:tasks.workunit.client.0.smithi063.stdout:1/394: stat d3/d3f/f77 0 2024-03-23T11:47:38.809 INFO:tasks.workunit.client.1.smithi111.stdout:1/584: read - d0/d5/dd/d1e/d54/fc2 zero size 2024-03-23T11:47:38.809 INFO:tasks.workunit.client.1.smithi111.stdout:4/618: dread - d0/d95/fa9 zero size 2024-03-23T11:47:38.816 INFO:tasks.workunit.client.1.smithi111.stdout:7/596: rmdir d14/d1b/d22/d8a 0 2024-03-23T11:47:38.818 INFO:tasks.workunit.client.1.smithi111.stdout:7/597: write d14/f15 [2978752,20980] 0 2024-03-23T11:47:38.821 INFO:tasks.workunit.client.1.smithi111.stdout:8/504: mknod d5/d26/d2f/d62/d75/d4b/d51/da4/d97/ca5 0 2024-03-23T11:47:38.823 INFO:tasks.workunit.client.1.smithi111.stdout:5/631: creat d1/d20/d5c/dc5/fd1 x:0 0 0 2024-03-23T11:47:38.826 INFO:tasks.workunit.client.1.smithi111.stdout:2/653: dread d2/d11/d17/d3e/d6b/f79 [0,4194304] 0 2024-03-23T11:47:38.828 INFO:tasks.workunit.client.1.smithi111.stdout:6/637: rename d2/d1a/d1c/d2c/fab to d2/d1a/fd2 0 2024-03-23T11:47:38.829 INFO:tasks.workunit.client.0.smithi063.stdout:3/686: sync 2024-03-23T11:47:38.829 INFO:tasks.workunit.client.1.smithi111.stdout:6/638: fdatasync d2/f57 0 2024-03-23T11:47:38.830 INFO:tasks.workunit.client.1.smithi111.stdout:4/619: creat d0/d95/d9b/fc1 x:0 0 0 2024-03-23T11:47:38.830 INFO:tasks.workunit.client.1.smithi111.stdout:4/620: stat d0/d17/c3e 0 2024-03-23T11:47:38.831 INFO:tasks.workunit.client.1.smithi111.stdout:3/531: unlink d0/d7/d19/d5e/c6f 0 2024-03-23T11:47:38.833 INFO:tasks.workunit.client.1.smithi111.stdout:7/598: creat d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fa7 x:0 0 0 2024-03-23T11:47:38.834 INFO:tasks.workunit.client.1.smithi111.stdout:3/532: dread - d0/d1a/d1d/d26/d5a/f74 zero size 2024-03-23T11:47:38.837 INFO:tasks.workunit.client.1.smithi111.stdout:2/654: dread d2/d11/d17/d1f/da0/fd4 [0,4194304] 0 2024-03-23T11:47:38.838 INFO:tasks.workunit.client.1.smithi111.stdout:9/636: rename d0/d4/d1c/f46 to d0/d4/d6a/d89/fcf 0 2024-03-23T11:47:38.839 INFO:tasks.workunit.client.1.smithi111.stdout:2/655: chown d2/d11/d17/d3e/d6b/d9d/fae 44379 1 2024-03-23T11:47:38.840 INFO:tasks.workunit.client.1.smithi111.stdout:9/637: write d0/d31/d41/d71/f93 [2370864,84298] 0 2024-03-23T11:47:38.842 INFO:tasks.workunit.client.1.smithi111.stdout:5/632: mknod d1/d3/d6/d8b/dc2/cd2 0 2024-03-23T11:47:38.845 INFO:tasks.workunit.client.1.smithi111.stdout:3/533: symlink d0/d1a/d1d/lb4 0 2024-03-23T11:47:38.846 INFO:tasks.workunit.client.1.smithi111.stdout:9/638: dread - d0/d4/d1c/d2d/fb7 zero size 2024-03-23T11:47:38.848 INFO:tasks.workunit.client.1.smithi111.stdout:2/656: mkdir d2/d11/d17/d3e/dee 0 2024-03-23T11:47:38.853 INFO:tasks.workunit.client.1.smithi111.stdout:7/599: creat d14/d44/d70/d7b/fa8 x:0 0 0 2024-03-23T11:47:38.853 INFO:tasks.workunit.client.1.smithi111.stdout:3/534: creat d0/d1a/d1d/d26/fb5 x:0 0 0 2024-03-23T11:47:38.854 INFO:tasks.workunit.client.1.smithi111.stdout:9/639: creat d0/d4/d6a/fd0 x:0 0 0 2024-03-23T11:47:38.855 INFO:tasks.workunit.client.1.smithi111.stdout:3/535: dread - d0/d1a/fb3 zero size 2024-03-23T11:47:38.859 INFO:tasks.workunit.client.1.smithi111.stdout:6/639: link d2/l4 d2/d1f/d45/d9a/ld3 0 2024-03-23T11:47:38.861 INFO:tasks.workunit.client.0.smithi063.stdout:4/565: write d5/da/d20/d4b/d1f/f41 [148837,66237] 0 2024-03-23T11:47:38.865 INFO:tasks.workunit.client.1.smithi111.stdout:5/633: dwrite d1/d3/d6/d7/d63/d2c/d82/d66/f97 [0,4194304] 0 2024-03-23T11:47:38.865 INFO:tasks.workunit.client.0.smithi063.stdout:7/442: truncate d5/d5c/d2f/d73/da/f8a 840185 0 2024-03-23T11:47:38.866 INFO:tasks.workunit.client.1.smithi111.stdout:0/596: truncate d5/db/f4a 741868 0 2024-03-23T11:47:38.870 INFO:tasks.workunit.client.1.smithi111.stdout:1/585: write d0/fe [4228753,52479] 0 2024-03-23T11:47:38.873 INFO:tasks.workunit.client.1.smithi111.stdout:9/640: symlink d0/d31/ld1 0 2024-03-23T11:47:38.876 INFO:tasks.workunit.client.1.smithi111.stdout:8/505: write d5/f11 [519852,6320] 0 2024-03-23T11:47:38.877 INFO:tasks.workunit.client.0.smithi063.stdout:9/520: dwrite d4/d4c/d44/f6a [0,4194304] 0 2024-03-23T11:47:38.879 INFO:tasks.workunit.client.1.smithi111.stdout:8/506: stat d5/d26/d31/d4d 0 2024-03-23T11:47:38.880 INFO:tasks.workunit.client.1.smithi111.stdout:5/634: mkdir d1/d20/d5c/dc5/dd3 0 2024-03-23T11:47:38.881 INFO:tasks.workunit.client.0.smithi063.stdout:1/395: readlink d3/df/l48 0 2024-03-23T11:47:38.882 INFO:tasks.workunit.client.1.smithi111.stdout:5/635: chown d1/d3/d6/d7/l1b 192 1 2024-03-23T11:47:38.884 INFO:tasks.workunit.client.1.smithi111.stdout:1/586: rmdir d0/d6/d48/d4d 39 2024-03-23T11:47:38.886 INFO:tasks.workunit.client.0.smithi063.stdout:1/396: write d3/f11 [3348507,64446] 0 2024-03-23T11:47:38.887 INFO:tasks.workunit.client.1.smithi111.stdout:1/587: dread - d0/d5/dd/fc0 zero size 2024-03-23T11:47:38.887 INFO:tasks.workunit.client.1.smithi111.stdout:9/641: write d0/d4/f15 [198406,102768] 0 2024-03-23T11:47:38.888 INFO:tasks.workunit.client.1.smithi111.stdout:6/640: link d2/d1f/d45/d86/faa d2/d1f/d45/d9a/fd4 0 2024-03-23T11:47:38.891 INFO:tasks.workunit.client.1.smithi111.stdout:0/597: creat d5/db/d4d/fc6 x:0 0 0 2024-03-23T11:47:38.893 INFO:tasks.workunit.client.0.smithi063.stdout:0/532: dwrite d2/d4/d63/f9f [0,4194304] 0 2024-03-23T11:47:38.894 INFO:tasks.workunit.client.1.smithi111.stdout:3/536: dread d0/d7/f1e [0,4194304] 0 2024-03-23T11:47:38.894 INFO:tasks.workunit.client.1.smithi111.stdout:4/621: dwrite d0/dc/d78/fc0 [0,4194304] 0 2024-03-23T11:47:38.896 INFO:tasks.workunit.client.1.smithi111.stdout:2/657: dread d2/d11/d17/d3e/f72 [0,4194304] 0 2024-03-23T11:47:38.896 INFO:tasks.workunit.client.1.smithi111.stdout:8/507: creat d5/d32/d7d/fa6 x:0 0 0 2024-03-23T11:47:38.897 INFO:tasks.workunit.client.1.smithi111.stdout:2/658: readlink d2/lec 0 2024-03-23T11:47:38.898 INFO:tasks.workunit.client.0.smithi063.stdout:5/344: mknod d2/d28/d3a/d46/d79/c7a 0 2024-03-23T11:47:38.902 INFO:tasks.workunit.client.0.smithi063.stdout:6/407: mkdir d1/dc/d3a/d57/d77/d79 0 2024-03-23T11:47:38.902 INFO:tasks.workunit.client.1.smithi111.stdout:1/588: mkdir d0/d2b/d7c/dca 0 2024-03-23T11:47:38.903 INFO:tasks.workunit.client.1.smithi111.stdout:1/589: readlink d0/d6/l67 0 2024-03-23T11:47:38.905 INFO:tasks.workunit.client.1.smithi111.stdout:1/590: dread - d0/d6/d48/fa8 zero size 2024-03-23T11:47:38.906 INFO:tasks.workunit.client.0.smithi063.stdout:9/521: fdatasync d4/d31/d70/d8d/fac 0 2024-03-23T11:47:38.907 INFO:tasks.workunit.client.1.smithi111.stdout:9/642: creat d0/d4/d1c/d36/d28/d69/fd2 x:0 0 0 2024-03-23T11:47:38.908 INFO:tasks.workunit.client.0.smithi063.stdout:9/522: readlink d4/d31/d43/d49/l61 0 2024-03-23T11:47:38.913 INFO:tasks.workunit.client.0.smithi063.stdout:3/687: creat d13/d22/d2d/d3f/d58/dcb/fd5 x:0 0 0 2024-03-23T11:47:38.914 INFO:tasks.workunit.client.1.smithi111.stdout:4/622: fsync d0/dc/d1f/f76 0 2024-03-23T11:47:38.914 INFO:tasks.workunit.client.1.smithi111.stdout:8/508: rmdir d5/d26/d2f/d62/d75/d4b/d51 39 2024-03-23T11:47:38.916 INFO:tasks.workunit.client.1.smithi111.stdout:8/509: fsync d5/d1b/f95 0 2024-03-23T11:47:38.917 INFO:tasks.workunit.client.1.smithi111.stdout:7/600: truncate d14/f1c 3195490 0 2024-03-23T11:47:38.922 INFO:tasks.workunit.client.1.smithi111.stdout:8/510: fsync d5/d26/f74 0 2024-03-23T11:47:38.925 INFO:tasks.workunit.client.1.smithi111.stdout:1/591: dwrite d0/d6/f3c [4194304,4194304] 0 2024-03-23T11:47:38.925 INFO:tasks.workunit.client.0.smithi063.stdout:6/408: dread d1/d15/d4b/f4c [0,4194304] 0 2024-03-23T11:47:38.926 INFO:tasks.workunit.client.1.smithi111.stdout:9/643: mknod d0/d4/d1c/d2d/d75/d7a/cd3 0 2024-03-23T11:47:38.926 INFO:tasks.workunit.client.0.smithi063.stdout:7/443: read - d5/d6/d19/f7d zero size 2024-03-23T11:47:38.930 INFO:tasks.workunit.client.0.smithi063.stdout:7/444: stat d5/d6/d19/d1f/d47/d4f/l5f 0 2024-03-23T11:47:38.932 INFO:tasks.workunit.client.1.smithi111.stdout:5/636: creat d1/d3/d6/d7/d63/d2c/d82/fd4 x:0 0 0 2024-03-23T11:47:38.932 INFO:tasks.workunit.client.1.smithi111.stdout:4/623: dread d0/d7/da/f5b [0,4194304] 0 2024-03-23T11:47:38.934 INFO:tasks.workunit.client.1.smithi111.stdout:4/624: chown d0/fa8 247386 1 2024-03-23T11:47:38.934 INFO:tasks.workunit.client.0.smithi063.stdout:5/345: mknod d2/d7/d27/c7b 0 2024-03-23T11:47:38.938 INFO:tasks.workunit.client.1.smithi111.stdout:4/625: truncate d0/dc/d1f/db0/faa 867599 0 2024-03-23T11:47:38.939 INFO:tasks.workunit.client.0.smithi063.stdout:0/533: creat d2/d14/d36/d77/d94/da6/d28/fc5 x:0 0 0 2024-03-23T11:47:38.940 INFO:tasks.workunit.client.1.smithi111.stdout:4/626: fsync d0/dc/d1f/db0/fa7 0 2024-03-23T11:47:38.941 INFO:tasks.workunit.client.1.smithi111.stdout:9/644: mkdir d0/d31/d41/d9d/dd4 0 2024-03-23T11:47:38.943 INFO:tasks.workunit.client.1.smithi111.stdout:6/641: getdents d2/d1f/d45/d49/d8d 0 2024-03-23T11:47:38.949 INFO:tasks.workunit.client.0.smithi063.stdout:9/523: write d4/d31/f63 [632272,73488] 0 2024-03-23T11:47:38.951 INFO:tasks.workunit.client.1.smithi111.stdout:3/537: rename d0/d72/d7b/cb0 to d0/d7/d19/cb6 0 2024-03-23T11:47:38.951 INFO:tasks.workunit.client.1.smithi111.stdout:5/637: dwrite d1/d3/d6/d8b/d59/d74/f4b [0,4194304] 0 2024-03-23T11:47:38.955 INFO:tasks.workunit.client.0.smithi063.stdout:6/409: unlink d1/dc/d3a/d57/c63 0 2024-03-23T11:47:38.955 INFO:tasks.workunit.client.1.smithi111.stdout:3/538: chown d0/d53/faf 406624 1 2024-03-23T11:47:38.957 INFO:tasks.workunit.client.0.smithi063.stdout:7/445: fdatasync d5/d5c/d2f/d73/da/f4e 0 2024-03-23T11:47:38.959 INFO:tasks.workunit.client.1.smithi111.stdout:6/642: mknod d2/d35/d70/cd5 0 2024-03-23T11:47:38.960 INFO:tasks.workunit.client.0.smithi063.stdout:6/410: read d1/dc/d3a/f3f [1082329,24463] 0 2024-03-23T11:47:38.962 INFO:tasks.workunit.client.1.smithi111.stdout:8/511: rename d5/f1a to d5/d26/d2f/d62/d7e/fa7 0 2024-03-23T11:47:38.962 INFO:tasks.workunit.client.1.smithi111.stdout:4/627: dwrite d0/dc/d1f/f8a [0,4194304] 0 2024-03-23T11:47:38.964 INFO:tasks.workunit.client.1.smithi111.stdout:7/601: getdents d14/d44/d70 0 2024-03-23T11:47:38.966 INFO:tasks.workunit.client.0.smithi063.stdout:9/524: rmdir d4/d4c/d44 39 2024-03-23T11:47:38.970 INFO:tasks.workunit.client.1.smithi111.stdout:3/539: rmdir d0/d7/d19/d5e 39 2024-03-23T11:47:38.970 INFO:tasks.workunit.client.0.smithi063.stdout:9/525: chown d4/d4c/d19/f27 7 1 2024-03-23T11:47:38.971 INFO:tasks.workunit.client.0.smithi063.stdout:5/346: dread d2/d7/d8/d1d/f58 [0,4194304] 0 2024-03-23T11:47:38.973 INFO:tasks.workunit.client.1.smithi111.stdout:9/645: truncate d0/d4/d1c/f45 947795 0 2024-03-23T11:47:38.973 INFO:tasks.workunit.client.1.smithi111.stdout:3/540: stat d0/l3b 0 2024-03-23T11:47:38.974 INFO:tasks.workunit.client.0.smithi063.stdout:6/411: chown d1/l36 0 1 2024-03-23T11:47:38.975 INFO:tasks.workunit.client.0.smithi063.stdout:9/526: fdatasync d4/d31/d43/da9/da6/f4a 0 2024-03-23T11:47:38.977 INFO:tasks.workunit.client.1.smithi111.stdout:5/638: dread d1/d3/d6/d8b/f75 [0,4194304] 0 2024-03-23T11:47:38.978 INFO:tasks.workunit.client.0.smithi063.stdout:6/412: write d1/d15/f70 [4176393,103019] 0 2024-03-23T11:47:38.981 INFO:tasks.workunit.client.1.smithi111.stdout:2/659: dwrite d2/de/d6c/dd1/dd6/feb [4194304,4194304] 0 2024-03-23T11:47:38.984 INFO:tasks.workunit.client.0.smithi063.stdout:0/534: rmdir d2/d14/d36/d77/d94/da6/d1f/d4f 39 2024-03-23T11:47:38.984 INFO:tasks.workunit.client.1.smithi111.stdout:3/541: chown d0/d1a/d1d/c73 367096772 1 2024-03-23T11:47:38.986 INFO:tasks.workunit.client.1.smithi111.stdout:3/542: fsync d0/d1a/f28 0 2024-03-23T11:47:38.988 INFO:tasks.workunit.client.1.smithi111.stdout:0/598: dwrite d5/db/f38 [0,4194304] 0 2024-03-23T11:47:38.989 INFO:tasks.workunit.client.1.smithi111.stdout:8/512: creat d5/d26/d2f/d62/d75/d4b/d51/da4/d97/fa8 x:0 0 0 2024-03-23T11:47:38.989 INFO:tasks.workunit.client.1.smithi111.stdout:1/592: write d0/d6/f27 [1363545,127377] 0 2024-03-23T11:47:38.991 INFO:tasks.workunit.client.1.smithi111.stdout:6/643: rename d2/d1a/f64 to d2/d35/da7/fd6 0 2024-03-23T11:47:38.998 INFO:tasks.workunit.client.1.smithi111.stdout:2/660: fsync d2/d11/d17/d3e/f72 0 2024-03-23T11:47:38.999 INFO:tasks.workunit.client.1.smithi111.stdout:1/593: read - d0/d5/dd/d1e/d54/fc2 zero size 2024-03-23T11:47:39.000 INFO:tasks.workunit.client.0.smithi063.stdout:6/413: symlink d1/d15/l7a 0 2024-03-23T11:47:39.002 INFO:tasks.workunit.client.0.smithi063.stdout:5/347: getdents d2/d7/d8/de/d47 0 2024-03-23T11:47:39.003 INFO:tasks.workunit.client.0.smithi063.stdout:9/527: dwrite d4/f47 [0,4194304] 0 2024-03-23T11:47:39.004 INFO:tasks.workunit.client.1.smithi111.stdout:3/543: readlink d0/d1a/d2d/l83 0 2024-03-23T11:47:39.004 INFO:tasks.workunit.client.1.smithi111.stdout:2/661: write d2/d11/d17/d1f/da0/fdd [1558554,26347] 0 2024-03-23T11:47:39.007 INFO:tasks.workunit.client.0.smithi063.stdout:6/414: chown d1/dc/d3a/f30 13 1 2024-03-23T11:47:39.014 INFO:tasks.workunit.client.0.smithi063.stdout:5/348: write d2/d7/d8/de/d49/f5d [1105186,5407] 0 2024-03-23T11:47:39.015 INFO:tasks.workunit.client.1.smithi111.stdout:2/662: chown d2/d11/d17/d3e/d6b/d2e/f82 6352 1 2024-03-23T11:47:39.017 INFO:tasks.workunit.client.1.smithi111.stdout:0/599: readlink d5/db/dbe/l97 0 2024-03-23T11:47:39.018 INFO:tasks.workunit.client.0.smithi063.stdout:6/415: stat d1/d15/c69 0 2024-03-23T11:47:39.019 INFO:tasks.workunit.client.0.smithi063.stdout:9/528: fdatasync d4/fe 0 2024-03-23T11:47:39.020 INFO:tasks.workunit.client.1.smithi111.stdout:0/600: stat d5/db/d1f/d19/dbd/d8b/l9a 0 2024-03-23T11:47:39.021 INFO:tasks.workunit.client.0.smithi063.stdout:5/349: fsync d2/d7/d8/f20 0 2024-03-23T11:47:39.024 INFO:tasks.workunit.client.1.smithi111.stdout:3/544: rmdir d0/d1a/d1d 39 2024-03-23T11:47:39.026 INFO:tasks.workunit.client.0.smithi063.stdout:5/350: fdatasync d2/f5 0 2024-03-23T11:47:39.027 INFO:tasks.workunit.client.1.smithi111.stdout:5/639: rename d1/d3/d6/d7/d63/d2c/d82/l8a to d1/d3/d6/d8b/d59/ld5 0 2024-03-23T11:47:39.028 INFO:tasks.workunit.client.1.smithi111.stdout:7/602: getdents d14/d44 0 2024-03-23T11:47:39.028 INFO:tasks.workunit.client.1.smithi111.stdout:4/628: dread d0/d7/da/f46 [0,4194304] 0 2024-03-23T11:47:39.035 INFO:tasks.workunit.client.0.smithi063.stdout:0/535: dread d2/d12/f68 [0,4194304] 0 2024-03-23T11:47:39.037 INFO:tasks.workunit.client.0.smithi063.stdout:9/529: mknod d4/d31/d43/da9/da6/cb6 0 2024-03-23T11:47:39.040 INFO:tasks.workunit.client.0.smithi063.stdout:5/351: mkdir d2/d7/d8/de/d47/d7c 0 2024-03-23T11:47:39.044 INFO:tasks.workunit.client.1.smithi111.stdout:7/603: dwrite d14/d1b/d1e/d2c/d42/f4a [8388608,4194304] 0 2024-03-23T11:47:39.044 INFO:tasks.workunit.client.0.smithi063.stdout:9/530: symlink d4/d4c/d19/lb7 0 2024-03-23T11:47:39.057 INFO:tasks.workunit.client.1.smithi111.stdout:0/601: mkdir d5/db/d1f/d19/dbd/d64/dc7 0 2024-03-23T11:47:39.057 INFO:tasks.workunit.client.1.smithi111.stdout:8/513: rename d5/d26/d2f/d62/f86 to d5/d26/d31/d3b/fa9 0 2024-03-23T11:47:39.058 INFO:tasks.workunit.client.1.smithi111.stdout:8/514: dread - d5/d32/d7d/fa6 zero size 2024-03-23T11:47:39.058 INFO:tasks.workunit.client.1.smithi111.stdout:9/646: dread d0/f48 [0,4194304] 0 2024-03-23T11:47:39.058 INFO:tasks.workunit.client.1.smithi111.stdout:5/640: rename d1/d3/d3a/f3d to d1/d3/d3a/fd6 0 2024-03-23T11:47:39.059 INFO:tasks.workunit.client.0.smithi063.stdout:0/536: dwrite d2/d48/f7f [0,4194304] 0 2024-03-23T11:47:39.060 INFO:tasks.workunit.client.0.smithi063.stdout:2/416: sync 2024-03-23T11:47:39.060 INFO:tasks.workunit.client.1.smithi111.stdout:4/629: symlink d0/dc/d4c/d71/lc2 0 2024-03-23T11:47:39.061 INFO:tasks.workunit.client.1.smithi111.stdout:5/641: dread - d1/d20/fc9 zero size 2024-03-23T11:47:39.061 INFO:tasks.workunit.client.0.smithi063.stdout:1/397: write d3/d3f/d4d/f69 [5107759,109597] 0 2024-03-23T11:47:39.062 INFO:tasks.workunit.client.1.smithi111.stdout:7/604: creat d14/d37/d50/d74/fa9 x:0 0 0 2024-03-23T11:47:39.065 INFO:tasks.workunit.client.0.smithi063.stdout:2/417: chown d2/d23/d2e/d52/d44/f82 26 1 2024-03-23T11:47:39.067 INFO:tasks.workunit.client.1.smithi111.stdout:5/642: truncate d1/d3/d6/d7/d63/d2c/d82/f88 1069833 0 2024-03-23T11:47:39.069 INFO:tasks.workunit.client.1.smithi111.stdout:5/643: fsync d1/d3/fc 0 2024-03-23T11:47:39.074 INFO:tasks.workunit.client.0.smithi063.stdout:0/537: link d2/d14/d36/d77/d94/da6/f33 d2/d6e/fc6 0 2024-03-23T11:47:39.075 INFO:tasks.workunit.client.1.smithi111.stdout:9/647: write d0/d4/d1c/d2d/d3b/d42/d92/fc7 [345770,62528] 0 2024-03-23T11:47:39.076 INFO:tasks.workunit.client.0.smithi063.stdout:3/688: mknod d13/d22/d2d/d3f/d58/cd6 0 2024-03-23T11:47:39.078 INFO:tasks.workunit.client.0.smithi063.stdout:5/352: unlink d2/l65 0 2024-03-23T11:47:39.079 INFO:tasks.workunit.client.0.smithi063.stdout:5/353: chown d2 220483 1 2024-03-23T11:47:39.080 INFO:tasks.workunit.client.0.smithi063.stdout:3/689: write f5 [5038927,28886] 0 2024-03-23T11:47:39.081 INFO:tasks.workunit.client.1.smithi111.stdout:1/594: write d0/d2b/f3e [1983643,26527] 0 2024-03-23T11:47:39.083 INFO:tasks.workunit.client.1.smithi111.stdout:4/630: dread d0/dc/d4c/d2a/f70 [0,4194304] 0 2024-03-23T11:47:39.087 INFO:tasks.workunit.client.1.smithi111.stdout:5/644: dwrite d1/d3/d6/d7/d63/d2c/f51 [0,4194304] 0 2024-03-23T11:47:39.088 INFO:tasks.workunit.client.0.smithi063.stdout:3/690: rmdir d13/d22/d2d/d3f/d58 39 2024-03-23T11:47:39.089 INFO:tasks.workunit.client.1.smithi111.stdout:3/545: creat d0/d1a/fb7 x:0 0 0 2024-03-23T11:47:39.090 INFO:tasks.workunit.client.1.smithi111.stdout:5/645: write d1/d3/d6/d8b/fa3 [1451394,56002] 0 2024-03-23T11:47:39.091 INFO:tasks.workunit.client.1.smithi111.stdout:6/644: dwrite d2/d1f/d45/d49/fa1 [0,4194304] 0 2024-03-23T11:47:39.091 INFO:tasks.workunit.client.0.smithi063.stdout:1/398: dwrite d3/d1c/f1e [0,4194304] 0 2024-03-23T11:47:39.092 INFO:tasks.workunit.client.0.smithi063.stdout:6/416: rename d1/d2a/c65 to d1/dc/d21/c7b 0 2024-03-23T11:47:39.095 INFO:tasks.workunit.client.0.smithi063.stdout:5/354: creat d2/d7/d8/de/d2a/d3b/d43/d5c/f7d x:0 0 0 2024-03-23T11:47:39.098 INFO:tasks.workunit.client.1.smithi111.stdout:9/648: unlink d0/d31/d41/d9d/fc8 0 2024-03-23T11:47:39.099 INFO:tasks.workunit.client.1.smithi111.stdout:2/663: truncate d2/d11/d17/d3e/d5d/dc0/fde 374523 0 2024-03-23T11:47:39.099 INFO:tasks.workunit.client.0.smithi063.stdout:5/355: chown d2/d7/d8/f13 4027 1 2024-03-23T11:47:39.102 INFO:tasks.workunit.client.0.smithi063.stdout:2/418: rename d2/d23/d2e/d52/d44 to d2/d23/d2e/d87 0 2024-03-23T11:47:39.105 INFO:tasks.workunit.client.0.smithi063.stdout:1/399: unlink d3/d49/d56/d65/f7e 0 2024-03-23T11:47:39.110 INFO:tasks.workunit.client.1.smithi111.stdout:1/595: unlink d0/d5/dd/d1e/d54/d87/fa6 0 2024-03-23T11:47:39.112 INFO:tasks.workunit.client.1.smithi111.stdout:6/645: mknod d2/d1f/cd7 0 2024-03-23T11:47:39.112 INFO:tasks.workunit.client.0.smithi063.stdout:2/419: read d2/d1c/f3c [1973378,105789] 0 2024-03-23T11:47:39.113 INFO:tasks.workunit.client.1.smithi111.stdout:5/646: chown d1/d3/d6/f3b 140045251 1 2024-03-23T11:47:39.115 INFO:tasks.workunit.client.0.smithi063.stdout:0/538: rename d2/d14/d36/d77/d94/da6/cbc to d2/d48/cc7 0 2024-03-23T11:47:39.115 INFO:tasks.workunit.client.1.smithi111.stdout:2/664: creat d2/d11/d17/d3e/fef x:0 0 0 2024-03-23T11:47:39.116 INFO:tasks.workunit.client.1.smithi111.stdout:9/649: creat d0/d4/d6a/da7/fd5 x:0 0 0 2024-03-23T11:47:39.117 INFO:tasks.workunit.client.1.smithi111.stdout:5/647: chown d1/d3/d6/d7/d63/d2c/d82 15430 1 2024-03-23T11:47:39.117 INFO:tasks.workunit.client.0.smithi063.stdout:9/531: truncate d4/f16 3364878 0 2024-03-23T11:47:39.120 INFO:tasks.workunit.client.1.smithi111.stdout:8/515: truncate d5/f1f 2781475 0 2024-03-23T11:47:39.120 INFO:tasks.workunit.client.1.smithi111.stdout:5/648: chown d1/d3/d6/d7/l6e 55 1 2024-03-23T11:47:39.123 INFO:tasks.workunit.client.1.smithi111.stdout:8/516: read d5/d26/d31/d4d/f59 [272087,48061] 0 2024-03-23T11:47:39.125 INFO:tasks.workunit.client.1.smithi111.stdout:3/546: creat d0/d1a/d1d/fb8 x:0 0 0 2024-03-23T11:47:39.126 INFO:tasks.workunit.client.0.smithi063.stdout:5/356: creat d2/d28/d3a/d46/d79/f7e x:0 0 0 2024-03-23T11:47:39.127 INFO:tasks.workunit.client.1.smithi111.stdout:1/596: dwrite d0/d6/f27 [4194304,4194304] 0 2024-03-23T11:47:39.127 INFO:tasks.workunit.client.0.smithi063.stdout:7/446: dwrite d5/d5c/d2f/d73/da/f8a [0,4194304] 0 2024-03-23T11:47:39.127 INFO:tasks.workunit.client.1.smithi111.stdout:6/646: rmdir d2/d73 39 2024-03-23T11:47:39.128 INFO:tasks.workunit.client.1.smithi111.stdout:7/605: write d14/d1b/d1e/d2c/d42/d5e/d88/f3a [1063045,94300] 0 2024-03-23T11:47:39.129 INFO:tasks.workunit.client.1.smithi111.stdout:0/602: link d5/db/d1f/f79 d5/db/d1f/d47/d6f/fc8 0 2024-03-23T11:47:39.131 INFO:tasks.workunit.client.1.smithi111.stdout:4/631: creat d0/fc3 x:0 0 0 2024-03-23T11:47:39.138 INFO:tasks.workunit.client.1.smithi111.stdout:5/649: creat d1/d3/d6/d7/d8c/fd7 x:0 0 0 2024-03-23T11:47:39.139 INFO:tasks.workunit.client.1.smithi111.stdout:8/517: fsync d5/d26/d2f/f77 0 2024-03-23T11:47:39.141 INFO:tasks.workunit.client.1.smithi111.stdout:1/597: creat d0/d2b/d69/fcb x:0 0 0 2024-03-23T11:47:39.145 INFO:tasks.workunit.client.1.smithi111.stdout:7/606: rename d14/l27 to d14/d1b/d1e/d86/laa 0 2024-03-23T11:47:39.147 INFO:tasks.workunit.client.1.smithi111.stdout:3/547: mkdir d0/db9 0 2024-03-23T11:47:39.149 INFO:tasks.workunit.client.1.smithi111.stdout:8/518: creat d5/d32/d7d/faa x:0 0 0 2024-03-23T11:47:39.152 INFO:tasks.workunit.client.0.smithi063.stdout:1/400: dwrite d3/d1c/f6d [0,4194304] 0 2024-03-23T11:47:39.152 INFO:tasks.workunit.client.1.smithi111.stdout:4/632: dread d0/fa8 [0,4194304] 0 2024-03-23T11:47:39.154 INFO:tasks.workunit.client.1.smithi111.stdout:8/519: read d5/d26/d31/d3b/f60 [999030,7268] 0 2024-03-23T11:47:39.155 INFO:tasks.workunit.client.0.smithi063.stdout:6/417: link d1/d13/c33 d1/c7c 0 2024-03-23T11:47:39.156 INFO:tasks.workunit.client.0.smithi063.stdout:5/357: truncate d2/d7/f10 2946929 0 2024-03-23T11:47:39.158 INFO:tasks.workunit.client.0.smithi063.stdout:3/691: rename d13/d22/d2d/d7d/lc5 to d13/d22/d2d/d3f/d58/d97/dac/ld7 0 2024-03-23T11:47:39.160 INFO:tasks.workunit.client.0.smithi063.stdout:2/420: dwrite d2/d23/d2e/d87/d4d/f6d [0,4194304] 0 2024-03-23T11:47:39.161 INFO:tasks.workunit.client.0.smithi063.stdout:0/539: dread d2/d12/d32/f5c [0,4194304] 0 2024-03-23T11:47:39.164 INFO:tasks.workunit.client.1.smithi111.stdout:6/647: mknod d2/d35/cd8 0 2024-03-23T11:47:39.169 INFO:tasks.workunit.client.1.smithi111.stdout:8/520: dwrite d5/d26/d31/d3b/f6e [0,4194304] 0 2024-03-23T11:47:39.170 INFO:tasks.workunit.client.0.smithi063.stdout:7/447: rmdir d5/d5c/d2f/d73/da 39 2024-03-23T11:47:39.176 INFO:tasks.workunit.client.1.smithi111.stdout:4/633: creat d0/d7/fc4 x:0 0 0 2024-03-23T11:47:39.177 INFO:tasks.workunit.client.0.smithi063.stdout:7/448: chown d5/d6/d19/d1f/d47/d4f/d5b/c5e 502471135 1 2024-03-23T11:47:39.180 INFO:tasks.workunit.client.1.smithi111.stdout:9/650: write d0/d4/d1c/d2d/d3b/d42/d4f/d95/f7b [247965,102298] 0 2024-03-23T11:47:39.182 INFO:tasks.workunit.client.1.smithi111.stdout:9/651: write d0/d4/fbd [56862,96883] 0 2024-03-23T11:47:39.184 INFO:tasks.workunit.client.1.smithi111.stdout:9/652: truncate d0/d4/d6a/fd0 117139 0 2024-03-23T11:47:39.185 INFO:tasks.workunit.client.1.smithi111.stdout:3/548: link d0/d1a/d2b/d38/d7f/faa d0/d1a/d1d/d26/d5a/d64/fba 0 2024-03-23T11:47:39.185 INFO:tasks.workunit.client.1.smithi111.stdout:8/521: creat d5/d26/d2f/d42/d63/fab x:0 0 0 2024-03-23T11:47:39.187 INFO:tasks.workunit.client.1.smithi111.stdout:2/665: link d2/d11/d17/d3e/d6b/d9d/d52/l6e d2/de/d12/d70/lf0 0 2024-03-23T11:47:39.188 INFO:tasks.workunit.client.1.smithi111.stdout:9/653: write d0/d31/d35/fa1 [3074508,86251] 0 2024-03-23T11:47:39.189 INFO:tasks.workunit.client.1.smithi111.stdout:3/549: write d0/d53/faf [956496,9216] 0 2024-03-23T11:47:39.200 INFO:tasks.workunit.client.1.smithi111.stdout:0/603: getdents d5/db 0 2024-03-23T11:47:39.204 INFO:tasks.workunit.client.1.smithi111.stdout:5/650: getdents d1/d3/d6/d8b/d59/d6c 0 2024-03-23T11:47:39.204 INFO:tasks.workunit.client.0.smithi063.stdout:6/418: dwrite d1/d2a/f62 [0,4194304] 0 2024-03-23T11:47:39.204 INFO:tasks.workunit.client.0.smithi063.stdout:2/421: rename d2/d23/f28 to d2/d23/d2e/d52/d6c/f88 0 2024-03-23T11:47:39.209 INFO:tasks.workunit.client.1.smithi111.stdout:2/666: chown d2/d32/l5a 28 1 2024-03-23T11:47:39.211 INFO:tasks.workunit.client.0.smithi063.stdout:1/401: dwrite d3/df/f18 [0,4194304] 0 2024-03-23T11:47:39.213 INFO:tasks.workunit.client.1.smithi111.stdout:9/654: symlink d0/d4/d6a/ld6 0 2024-03-23T11:47:39.214 INFO:tasks.workunit.client.0.smithi063.stdout:3/692: dwrite d13/f56 [0,4194304] 0 2024-03-23T11:47:39.215 INFO:tasks.workunit.client.1.smithi111.stdout:3/550: chown d0/d7/l7a 14069 1 2024-03-23T11:47:39.217 INFO:tasks.workunit.client.0.smithi063.stdout:5/358: dread d2/fc [0,4194304] 0 2024-03-23T11:47:39.220 INFO:tasks.workunit.client.1.smithi111.stdout:7/607: dread - d14/d44/d4b/f9a zero size 2024-03-23T11:47:39.221 INFO:tasks.workunit.client.1.smithi111.stdout:0/604: creat d5/db/dbe/da7/fc9 x:0 0 0 2024-03-23T11:47:39.221 INFO:tasks.workunit.client.1.smithi111.stdout:5/651: mknod d1/d20/d5c/dc5/cd8 0 2024-03-23T11:47:39.223 INFO:tasks.workunit.client.1.smithi111.stdout:0/605: fdatasync d5/db/d1f/d82/fac 0 2024-03-23T11:47:39.224 INFO:tasks.workunit.client.1.smithi111.stdout:6/648: rename d2/d1a/d1c/d2c/d51/l3e to d2/d35/dc5/ld9 0 2024-03-23T11:47:39.225 INFO:tasks.workunit.client.0.smithi063.stdout:9/532: dwrite d4/d4c/f72 [0,4194304] 0 2024-03-23T11:47:39.225 INFO:tasks.workunit.client.0.smithi063.stdout:0/540: symlink d2/d14/d36/d77/d94/da6/lc8 0 2024-03-23T11:47:39.227 INFO:tasks.workunit.client.1.smithi111.stdout:8/522: mknod d5/d26/d2f/d8f/cac 0 2024-03-23T11:47:39.228 INFO:tasks.workunit.client.1.smithi111.stdout:8/523: fdatasync f1 0 2024-03-23T11:47:39.229 INFO:tasks.workunit.client.0.smithi063.stdout:9/533: read - d4/d31/d43/d49/fa4 zero size 2024-03-23T11:47:39.231 INFO:tasks.workunit.client.1.smithi111.stdout:4/634: link d0/dc/d4c/d2a/f70 d0/d17/fc5 0 2024-03-23T11:47:39.231 INFO:tasks.workunit.client.1.smithi111.stdout:8/524: stat d5/d1b/f68 0 2024-03-23T11:47:39.232 INFO:tasks.workunit.client.0.smithi063.stdout:2/422: fsync d2/d23/d56/f57 0 2024-03-23T11:47:39.234 INFO:tasks.workunit.client.0.smithi063.stdout:1/402: creat d3/d49/d56/f7f x:0 0 0 2024-03-23T11:47:39.235 INFO:tasks.workunit.client.0.smithi063.stdout:8/753: sync 2024-03-23T11:47:39.237 INFO:tasks.workunit.client.1.smithi111.stdout:0/606: creat d5/db/d1f/d47/d59/db6/fca x:0 0 0 2024-03-23T11:47:39.238 INFO:tasks.workunit.client.1.smithi111.stdout:7/608: chown d14/d1b/d1e/d2c/d2e/d33/l60 0 1 2024-03-23T11:47:39.240 INFO:tasks.workunit.client.0.smithi063.stdout:2/423: write d2/d23/f83 [150915,87227] 0 2024-03-23T11:47:39.240 INFO:tasks.workunit.client.1.smithi111.stdout:5/652: mkdir d1/d3/d6/d7/d63/dd9 0 2024-03-23T11:47:39.241 INFO:tasks.workunit.client.1.smithi111.stdout:6/649: dread d2/d35/d70/fc4 [0,4194304] 0 2024-03-23T11:47:39.243 INFO:tasks.workunit.client.1.smithi111.stdout:5/653: write d1/d20/f83 [1047696,72813] 0 2024-03-23T11:47:39.244 INFO:tasks.workunit.client.1.smithi111.stdout:3/551: truncate d0/f16 4124144 0 2024-03-23T11:47:39.245 INFO:tasks.workunit.client.1.smithi111.stdout:8/525: unlink d5/d26/d2f/d62/d75/d4b/d51/da4/l5b 0 2024-03-23T11:47:39.245 INFO:tasks.workunit.client.1.smithi111.stdout:3/552: fdatasync d0/fd 0 2024-03-23T11:47:39.246 INFO:tasks.workunit.client.1.smithi111.stdout:5/654: dread - d1/d20/fc9 zero size 2024-03-23T11:47:39.247 INFO:tasks.workunit.client.1.smithi111.stdout:5/655: chown d1/d20/f72 0 1 2024-03-23T11:47:39.248 INFO:tasks.workunit.client.1.smithi111.stdout:8/526: write d5/d26/f74 [2534633,110430] 0 2024-03-23T11:47:39.248 INFO:tasks.workunit.client.1.smithi111.stdout:3/553: write d0/d1a/d1d/f6e [982996,14775] 0 2024-03-23T11:47:39.249 INFO:tasks.workunit.client.1.smithi111.stdout:2/667: creat d2/d11/d17/d3e/d6b/d2e/ff1 x:0 0 0 2024-03-23T11:47:39.250 INFO:tasks.workunit.client.0.smithi063.stdout:0/541: write d2/d6e/fc6 [39803,114078] 0 2024-03-23T11:47:39.255 INFO:tasks.workunit.client.0.smithi063.stdout:5/359: creat d2/f7f x:0 0 0 2024-03-23T11:47:39.256 INFO:tasks.workunit.client.1.smithi111.stdout:7/609: unlink d14/d1b/d1e/f59 0 2024-03-23T11:47:39.258 INFO:tasks.workunit.client.1.smithi111.stdout:7/610: chown d14/d1b/d1e/d2c/d42/d5e/d88/d5f 0 1 2024-03-23T11:47:39.259 INFO:tasks.workunit.client.1.smithi111.stdout:6/650: fdatasync d2/d1f/d45/d49/d8d/f9d 0 2024-03-23T11:47:39.265 INFO:tasks.workunit.client.0.smithi063.stdout:2/424: creat d2/d84/f89 x:0 0 0 2024-03-23T11:47:39.265 INFO:tasks.workunit.client.0.smithi063.stdout:3/693: getdents d13/d22/d2d/d3f/d58/d86/dad 0 2024-03-23T11:47:39.266 INFO:tasks.workunit.client.1.smithi111.stdout:3/554: creat d0/d1a/d2d/fbb x:0 0 0 2024-03-23T11:47:39.267 INFO:tasks.workunit.client.1.smithi111.stdout:2/668: chown d2/d7/da/d19/f34 15470834 1 2024-03-23T11:47:39.268 INFO:tasks.workunit.client.0.smithi063.stdout:7/449: dwrite d5/d26/d76/f78 [0,4194304] 0 2024-03-23T11:47:39.270 INFO:tasks.workunit.client.1.smithi111.stdout:4/635: creat d0/fc6 x:0 0 0 2024-03-23T11:47:39.271 INFO:tasks.workunit.client.1.smithi111.stdout:9/655: rename d0/d4/d1c/d36/c16 to d0/d4/d1c/cd7 0 2024-03-23T11:47:39.273 INFO:tasks.workunit.client.1.smithi111.stdout:7/611: mknod d14/d1b/d1e/d2c/d42/d5e/d88/cab 0 2024-03-23T11:47:39.273 INFO:tasks.workunit.client.0.smithi063.stdout:2/425: readlink d2/d23/l24 0 2024-03-23T11:47:39.274 INFO:tasks.workunit.client.0.smithi063.stdout:8/754: rename d6/d1f/d54/d63/d90/d15/d9d/dea to d6/d1c/dff 0 2024-03-23T11:47:39.275 INFO:tasks.workunit.client.0.smithi063.stdout:1/403: write d3/d3f/d4d/f5b [238868,10761] 0 2024-03-23T11:47:39.276 INFO:tasks.workunit.client.1.smithi111.stdout:4/636: chown d0/dc/d4c/d2a/d97/d4f/c50 35 1 2024-03-23T11:47:39.277 INFO:tasks.workunit.client.1.smithi111.stdout:8/527: dread d5/d26/d31/d3b/d41/f4e [0,4194304] 0 2024-03-23T11:47:39.279 INFO:tasks.workunit.client.1.smithi111.stdout:6/651: dwrite d2/d1a/d1c/d2c/fa5 [0,4194304] 0 2024-03-23T11:47:39.279 INFO:tasks.workunit.client.1.smithi111.stdout:3/555: mknod d0/d1a/d1d/cbc 0 2024-03-23T11:47:39.279 INFO:tasks.workunit.client.0.smithi063.stdout:3/694: mknod d13/d22/d2d/d3f/d58/d97/dac/cd8 0 2024-03-23T11:47:39.281 INFO:tasks.workunit.client.0.smithi063.stdout:0/542: rename d2/d12/da1 to d2/d14/d36/d77/d94/da6/d25/d93/dc9 0 2024-03-23T11:47:39.284 INFO:tasks.workunit.client.1.smithi111.stdout:6/652: write d2/d1a/fbf [107445,35347] 0 2024-03-23T11:47:39.286 INFO:tasks.workunit.client.0.smithi063.stdout:6/419: dread d1/dc/d21/d49/f56 [0,4194304] 0 2024-03-23T11:47:39.286 INFO:tasks.workunit.client.1.smithi111.stdout:6/653: stat d2/d1f/d45/d65/fca 0 2024-03-23T11:47:39.289 INFO:tasks.workunit.client.1.smithi111.stdout:9/656: creat d0/d4/d10/d80/fd8 x:0 0 0 2024-03-23T11:47:39.290 INFO:tasks.workunit.client.1.smithi111.stdout:6/654: chown d2/d1a/f22 3951428 1 2024-03-23T11:47:39.292 INFO:tasks.workunit.client.0.smithi063.stdout:1/404: symlink d3/d1c/d4b/l80 0 2024-03-23T11:47:39.293 INFO:tasks.workunit.client.1.smithi111.stdout:4/637: mkdir d0/d95/db5/dc7 0 2024-03-23T11:47:39.293 INFO:tasks.workunit.client.1.smithi111.stdout:5/656: write d1/d20/f60 [729890,30923] 0 2024-03-23T11:47:39.295 INFO:tasks.workunit.client.0.smithi063.stdout:7/450: creat d5/d6/d19/d1f/d47/d4f/f93 x:0 0 0 2024-03-23T11:47:39.295 INFO:tasks.workunit.client.0.smithi063.stdout:6/420: mknod d1/d2a/d41/d42/c7d 0 2024-03-23T11:47:39.297 INFO:tasks.workunit.client.0.smithi063.stdout:4/566: sync 2024-03-23T11:47:39.298 INFO:tasks.workunit.client.0.smithi063.stdout:9/534: dwrite d4/f16 [0,4194304] 0 2024-03-23T11:47:39.299 INFO:tasks.workunit.client.1.smithi111.stdout:6/655: stat d2/d1a/d1c/d2c/c7b 0 2024-03-23T11:47:39.306 INFO:tasks.workunit.client.0.smithi063.stdout:0/543: mknod d2/d14/d36/d77/d94/da6/d28/d46/d51/cca 0 2024-03-23T11:47:39.308 INFO:tasks.workunit.client.0.smithi063.stdout:6/421: symlink d1/d2a/l7e 0 2024-03-23T11:47:39.311 INFO:tasks.workunit.client.1.smithi111.stdout:7/612: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fa7 [0,4194304] 0 2024-03-23T11:47:39.311 INFO:tasks.workunit.client.1.smithi111.stdout:9/657: fsync d0/d4/d1c/d36/f7e 0 2024-03-23T11:47:39.312 INFO:tasks.workunit.client.0.smithi063.stdout:5/360: dwrite d2/d7/d8/de/d2a/d3b/d43/d5c/f62 [0,4194304] 0 2024-03-23T11:47:39.313 INFO:tasks.workunit.client.0.smithi063.stdout:9/535: chown d4/d4c/d44/f84 0 1 2024-03-23T11:47:39.314 INFO:tasks.workunit.client.0.smithi063.stdout:6/422: write d1/f6e [960592,127966] 0 2024-03-23T11:47:39.316 INFO:tasks.workunit.client.0.smithi063.stdout:7/451: rename d5/d5c/d2f/d73/f91 to d5/d22/f94 0 2024-03-23T11:47:39.318 INFO:tasks.workunit.client.0.smithi063.stdout:1/405: read - d3/df/d31/f5f zero size 2024-03-23T11:47:39.323 INFO:tasks.workunit.client.1.smithi111.stdout:0/607: dwrite d5/db/d1f/d19/f2a [0,4194304] 0 2024-03-23T11:47:39.323 INFO:tasks.workunit.client.1.smithi111.stdout:1/598: sync 2024-03-23T11:47:39.324 INFO:tasks.workunit.client.1.smithi111.stdout:5/657: creat d1/d20/d5c/d99/dbf/dc3/fda x:0 0 0 2024-03-23T11:47:39.326 INFO:tasks.workunit.client.1.smithi111.stdout:3/556: dwrite d0/d1a/d2d/fbb [0,4194304] 0 2024-03-23T11:47:39.328 INFO:tasks.workunit.client.0.smithi063.stdout:5/361: creat d2/d28/d3a/d46/d79/f80 x:0 0 0 2024-03-23T11:47:39.331 INFO:tasks.workunit.client.1.smithi111.stdout:7/613: readlink d14/d1b/d1e/d2c/d42/d5e/l92 0 2024-03-23T11:47:39.332 INFO:tasks.workunit.client.0.smithi063.stdout:0/544: link d2/d14/d36/d77/f7c d2/d48/fcb 0 2024-03-23T11:47:39.335 INFO:tasks.workunit.client.1.smithi111.stdout:3/557: dread - d0/d55/f97 zero size 2024-03-23T11:47:39.335 INFO:tasks.workunit.client.1.smithi111.stdout:1/599: chown d0/d5/f16 319658628 1 2024-03-23T11:47:39.336 INFO:tasks.workunit.client.1.smithi111.stdout:0/608: fsync d5/db/d1f/d19/dbd/f76 0 2024-03-23T11:47:39.337 INFO:tasks.workunit.client.1.smithi111.stdout:2/669: dwrite d2/d11/f8a [0,4194304] 0 2024-03-23T11:47:39.339 INFO:tasks.workunit.client.1.smithi111.stdout:5/658: fsync d1/d3/d6/d8b/d59/d74/f4b 0 2024-03-23T11:47:39.340 INFO:tasks.workunit.client.0.smithi063.stdout:9/536: mknod d4/d31/d43/d49/cb8 0 2024-03-23T11:47:39.343 INFO:tasks.workunit.client.1.smithi111.stdout:4/638: dread d0/d7/da/f46 [0,4194304] 0 2024-03-23T11:47:39.344 INFO:tasks.workunit.client.1.smithi111.stdout:2/670: write d2/d7/da/dbd/fd8 [409598,19613] 0 2024-03-23T11:47:39.346 INFO:tasks.workunit.client.1.smithi111.stdout:2/671: chown d2/d32/d98 365201718 1 2024-03-23T11:47:39.346 INFO:tasks.workunit.client.1.smithi111.stdout:5/659: chown d1/d20/d41/da9 66281 1 2024-03-23T11:47:39.347 INFO:tasks.workunit.client.1.smithi111.stdout:3/558: symlink d0/d1a/d2d/lbd 0 2024-03-23T11:47:39.347 INFO:tasks.workunit.client.1.smithi111.stdout:8/528: dread d5/d26/d2f/d42/d58/f87 [0,4194304] 0 2024-03-23T11:47:39.350 INFO:tasks.workunit.client.0.smithi063.stdout:6/423: truncate d1/d13/f16 3325048 0 2024-03-23T11:47:39.353 INFO:tasks.workunit.client.1.smithi111.stdout:0/609: mknod d5/db/d1f/d47/ccb 0 2024-03-23T11:47:39.358 INFO:tasks.workunit.client.1.smithi111.stdout:3/559: rmdir d0/d55 39 2024-03-23T11:47:39.358 INFO:tasks.workunit.client.1.smithi111.stdout:8/529: mkdir d5/d26/d31/d5e/d4a/dad 0 2024-03-23T11:47:39.360 INFO:tasks.workunit.client.1.smithi111.stdout:4/639: creat d0/dc/fc8 x:0 0 0 2024-03-23T11:47:39.361 INFO:tasks.workunit.client.1.smithi111.stdout:3/560: unlink d0/d1a/d2d/l83 0 2024-03-23T11:47:39.362 INFO:tasks.workunit.client.0.smithi063.stdout:9/537: rename d4/d31/d43/d49/d8e/fa1 to d4/d31/fb9 0 2024-03-23T11:47:39.362 INFO:tasks.workunit.client.1.smithi111.stdout:3/561: fdatasync d0/d1a/d2b/fb1 0 2024-03-23T11:47:39.365 INFO:tasks.workunit.client.1.smithi111.stdout:1/600: rename d0/d5/dd/d1e/d54/d81/c99 to d0/d6/d48/d4d/dab/dbd/ccc 0 2024-03-23T11:47:39.369 INFO:tasks.workunit.client.0.smithi063.stdout:8/755: dwrite d6/dd/df/f11 [0,4194304] 0 2024-03-23T11:47:39.370 INFO:tasks.workunit.client.0.smithi063.stdout:1/406: creat d3/f81 x:0 0 0 2024-03-23T11:47:39.370 INFO:tasks.workunit.client.1.smithi111.stdout:3/562: readlink d0/d7/d19/d5e/l67 0 2024-03-23T11:47:39.375 INFO:tasks.workunit.client.0.smithi063.stdout:6/424: fdatasync d1/dc/d3a/d57/f5c 0 2024-03-23T11:47:39.377 INFO:tasks.workunit.client.1.smithi111.stdout:1/601: creat d0/d5/dd/d1e/d54/d87/d8e/fcd x:0 0 0 2024-03-23T11:47:39.378 INFO:tasks.workunit.client.1.smithi111.stdout:2/672: link d2/d5c/f63 d2/d32/ff2 0 2024-03-23T11:47:39.382 INFO:tasks.workunit.client.0.smithi063.stdout:2/426: truncate d2/d23/d2e/d87/d4d/f6d 2914376 0 2024-03-23T11:47:39.384 INFO:tasks.workunit.client.1.smithi111.stdout:6/656: dread d2/d1f/f7a [0,4194304] 0 2024-03-23T11:47:39.385 INFO:tasks.workunit.client.1.smithi111.stdout:8/530: getdents d5/d26 0 2024-03-23T11:47:39.386 INFO:tasks.workunit.client.0.smithi063.stdout:5/362: unlink d2/d7/d64/f6f 0 2024-03-23T11:47:39.387 INFO:tasks.workunit.client.0.smithi063.stdout:2/427: dread - d2/d23/f69 zero size 2024-03-23T11:47:39.387 INFO:tasks.workunit.client.1.smithi111.stdout:9/658: dread d0/d4/d1c/d2d/d3b/d42/d4f/f61 [0,4194304] 0 2024-03-23T11:47:39.390 INFO:tasks.workunit.client.1.smithi111.stdout:6/657: mknod d2/d35/cda 0 2024-03-23T11:47:39.391 INFO:tasks.workunit.client.0.smithi063.stdout:9/538: fdatasync d4/d31/d43/f7d 0 2024-03-23T11:47:39.393 INFO:tasks.workunit.client.1.smithi111.stdout:6/658: truncate d2/d1f/d45/d9a/fb4 903542 0 2024-03-23T11:47:39.394 INFO:tasks.workunit.client.0.smithi063.stdout:3/695: dwrite d13/d22/d2d/d3f/d58/d86/f93 [0,4194304] 0 2024-03-23T11:47:39.395 INFO:tasks.workunit.client.1.smithi111.stdout:0/610: write d5/db/d1f/f28 [2762714,60171] 0 2024-03-23T11:47:39.397 INFO:tasks.workunit.client.1.smithi111.stdout:6/659: truncate d2/d1a/d1c/d2c/fb9 848063 0 2024-03-23T11:47:39.397 INFO:tasks.workunit.client.0.smithi063.stdout:1/407: rmdir d3/d49 39 2024-03-23T11:47:39.398 INFO:tasks.workunit.client.1.smithi111.stdout:1/602: truncate d0/d2b/d37/f74 934061 0 2024-03-23T11:47:39.398 INFO:tasks.workunit.client.1.smithi111.stdout:3/563: getdents d0/d1a/d4f 0 2024-03-23T11:47:39.399 INFO:tasks.workunit.client.1.smithi111.stdout:6/660: chown d2/d35/fcc 26 1 2024-03-23T11:47:39.401 INFO:tasks.workunit.client.1.smithi111.stdout:3/564: chown d0/d1a/d1d/d26/d5a/d60/fac 1 1 2024-03-23T11:47:39.403 INFO:tasks.workunit.client.0.smithi063.stdout:7/452: dread d5/d5c/d2f/d73/d20/f53 [0,4194304] 0 2024-03-23T11:47:39.404 INFO:tasks.workunit.client.1.smithi111.stdout:8/531: mkdir d5/d26/d2f/d62/d7e/dae 0 2024-03-23T11:47:39.404 INFO:tasks.workunit.client.1.smithi111.stdout:5/660: truncate d1/d3/d6/d30/db2/fc1 471756 0 2024-03-23T11:47:39.406 INFO:tasks.workunit.client.1.smithi111.stdout:8/532: dread - d5/d32/d7d/faa zero size 2024-03-23T11:47:39.406 INFO:tasks.workunit.client.1.smithi111.stdout:5/661: chown d1/d3/d6/d7/d8c/d9e 307037683 1 2024-03-23T11:47:39.407 INFO:tasks.workunit.client.1.smithi111.stdout:9/659: rename d0/d4/d1c/d36/d28/l32 to d0/d31/d41/d9d/ld9 0 2024-03-23T11:47:39.407 INFO:tasks.workunit.client.1.smithi111.stdout:4/640: write d0/d7/da/f13 [2987231,67043] 0 2024-03-23T11:47:39.409 INFO:tasks.workunit.client.1.smithi111.stdout:0/611: mkdir d5/db/d4d/da6/dcc 0 2024-03-23T11:47:39.411 INFO:tasks.workunit.client.1.smithi111.stdout:2/673: truncate d2/d11/d17/d3e/d6b/d2e/fca 287779 0 2024-03-23T11:47:39.413 INFO:tasks.workunit.client.1.smithi111.stdout:8/533: creat d5/d26/d31/d4d/faf x:0 0 0 2024-03-23T11:47:39.415 INFO:tasks.workunit.client.1.smithi111.stdout:9/660: fsync d0/d31/d41/d71/d8f/fa9 0 2024-03-23T11:47:39.416 INFO:tasks.workunit.client.1.smithi111.stdout:3/565: creat d0/fbe x:0 0 0 2024-03-23T11:47:39.417 INFO:tasks.workunit.client.0.smithi063.stdout:6/425: mknod d1/d2a/c7f 0 2024-03-23T11:47:39.418 INFO:tasks.workunit.client.1.smithi111.stdout:9/661: write d0/d4/d1c/d2d/d75/d7a/fc5 [842574,31044] 0 2024-03-23T11:47:39.424 INFO:tasks.workunit.client.0.smithi063.stdout:0/545: dwrite d2/d12/d32/f5c [0,4194304] 0 2024-03-23T11:47:39.426 INFO:tasks.workunit.client.0.smithi063.stdout:5/363: mkdir d2/d7/d64/d81 0 2024-03-23T11:47:39.427 INFO:tasks.workunit.client.1.smithi111.stdout:1/603: rename d0/d5/dd/l17 to d0/d5/lce 0 2024-03-23T11:47:39.428 INFO:tasks.workunit.client.1.smithi111.stdout:1/604: write d0/d5/dd/fc0 [491104,3911] 0 2024-03-23T11:47:39.429 INFO:tasks.workunit.client.0.smithi063.stdout:4/567: dwrite d5/ff [8388608,4194304] 0 2024-03-23T11:47:39.429 INFO:tasks.workunit.client.1.smithi111.stdout:8/534: symlink d5/d32/lb0 0 2024-03-23T11:47:39.431 INFO:tasks.workunit.client.1.smithi111.stdout:3/566: symlink d0/d1a/d1d/d26/d5a/d60/lbf 0 2024-03-23T11:47:39.432 INFO:tasks.workunit.client.1.smithi111.stdout:3/567: chown d0/d72/da9 75 1 2024-03-23T11:47:39.437 INFO:tasks.workunit.client.0.smithi063.stdout:1/408: symlink d3/d62/l82 0 2024-03-23T11:47:39.441 INFO:tasks.workunit.client.0.smithi063.stdout:7/453: mknod d5/d26/d32/c95 0 2024-03-23T11:47:39.441 INFO:tasks.workunit.client.1.smithi111.stdout:4/641: dread d0/dc/d1f/f76 [0,4194304] 0 2024-03-23T11:47:39.444 INFO:tasks.workunit.client.1.smithi111.stdout:2/674: link d2/d7/f7b d2/d11/d17/d3e/d6b/d2e/ff3 0 2024-03-23T11:47:39.448 INFO:tasks.workunit.client.0.smithi063.stdout:3/696: dwrite f9 [0,4194304] 0 2024-03-23T11:47:39.448 INFO:tasks.workunit.client.0.smithi063.stdout:5/364: unlink d2/d7/d8/de/d49/f5d 0 2024-03-23T11:47:39.452 INFO:tasks.workunit.client.0.smithi063.stdout:9/539: creat d4/fba x:0 0 0 2024-03-23T11:47:39.455 INFO:tasks.workunit.client.0.smithi063.stdout:2/428: dwrite d2/d1c/f66 [0,4194304] 0 2024-03-23T11:47:39.456 INFO:tasks.workunit.client.0.smithi063.stdout:8/756: getdents d6/dd/d59/d7f/dc8 0 2024-03-23T11:47:39.456 INFO:tasks.workunit.client.1.smithi111.stdout:1/605: unlink d0/d2b/d37/l38 0 2024-03-23T11:47:39.459 INFO:tasks.workunit.client.1.smithi111.stdout:4/642: dwrite d0/dc/d89/f99 [0,4194304] 0 2024-03-23T11:47:39.459 INFO:tasks.workunit.client.1.smithi111.stdout:3/568: creat d0/d53/fc0 x:0 0 0 2024-03-23T11:47:39.460 INFO:tasks.workunit.client.0.smithi063.stdout:7/454: read d5/d6/d19/d1f/d47/d4f/d5b/f66 [118512,3373] 0 2024-03-23T11:47:39.460 INFO:tasks.workunit.client.1.smithi111.stdout:3/569: fdatasync d0/fbe 0 2024-03-23T11:47:39.462 INFO:tasks.workunit.client.1.smithi111.stdout:5/662: truncate d1/d3/f4 2173643 0 2024-03-23T11:47:39.462 INFO:tasks.workunit.client.1.smithi111.stdout:0/612: link d5/c35 d5/db/d1f/d47/ccd 0 2024-03-23T11:47:39.463 INFO:tasks.workunit.client.0.smithi063.stdout:6/426: creat d1/f80 x:0 0 0 2024-03-23T11:47:39.463 INFO:tasks.workunit.client.0.smithi063.stdout:1/409: rename d3/df/l2b to d3/d1c/d4b/l83 0 2024-03-23T11:47:39.464 INFO:tasks.workunit.client.0.smithi063.stdout:3/697: write d13/d22/d2d/d3f/f72 [963961,81208] 0 2024-03-23T11:47:39.465 INFO:tasks.workunit.client.1.smithi111.stdout:7/614: sync 2024-03-23T11:47:39.468 INFO:tasks.workunit.client.0.smithi063.stdout:5/365: mknod d2/d7/d8/de/d2a/c82 0 2024-03-23T11:47:39.471 INFO:tasks.workunit.client.0.smithi063.stdout:5/366: stat d2/d17/d4d/d6e 0 2024-03-23T11:47:39.473 INFO:tasks.workunit.client.1.smithi111.stdout:9/662: creat d0/d4/d10/d91/fda x:0 0 0 2024-03-23T11:47:39.473 INFO:tasks.workunit.client.0.smithi063.stdout:7/455: fdatasync d5/d6/d19/d1f/d47/d4f/f93 0 2024-03-23T11:47:39.474 INFO:tasks.workunit.client.1.smithi111.stdout:6/661: rename d2/c21 to d2/d1f/d45/d49/d8d/cdb 0 2024-03-23T11:47:39.476 INFO:tasks.workunit.client.0.smithi063.stdout:6/427: creat d1/f81 x:0 0 0 2024-03-23T11:47:39.477 INFO:tasks.workunit.client.0.smithi063.stdout:3/698: fdatasync d13/d22/d2d/d3e/dce/d9d/fc8 0 2024-03-23T11:47:39.478 INFO:tasks.workunit.client.1.smithi111.stdout:6/662: write d2/d35/fcc [742030,1516] 0 2024-03-23T11:47:39.479 INFO:tasks.workunit.client.0.smithi063.stdout:4/568: chown d5/da/d20/d4b/f68 5595 1 2024-03-23T11:47:39.481 INFO:tasks.workunit.client.1.smithi111.stdout:0/613: dwrite d5/f40 [0,4194304] 0 2024-03-23T11:47:39.482 INFO:tasks.workunit.client.0.smithi063.stdout:2/429: creat d2/d23/d2e/d87/f8a x:0 0 0 2024-03-23T11:47:39.483 INFO:tasks.workunit.client.1.smithi111.stdout:5/663: write d1/d3/d6/d30/f70 [3576248,72935] 0 2024-03-23T11:47:39.485 INFO:tasks.workunit.client.0.smithi063.stdout:7/456: rmdir d5/d5c 39 2024-03-23T11:47:39.486 INFO:tasks.workunit.client.0.smithi063.stdout:4/569: write d5/da/d20/d4b/f8b [1010403,120849] 0 2024-03-23T11:47:39.487 INFO:tasks.workunit.client.1.smithi111.stdout:5/664: write d1/d3/d6/d7/f1f [5700076,125761] 0 2024-03-23T11:47:39.489 INFO:tasks.workunit.client.0.smithi063.stdout:2/430: dread - d2/d23/d2e/d52/f3e zero size 2024-03-23T11:47:39.490 INFO:tasks.workunit.client.1.smithi111.stdout:5/665: chown d1/d3/d6/d7/d63/d2c/d82/d66 2729 1 2024-03-23T11:47:39.492 INFO:tasks.workunit.client.0.smithi063.stdout:6/428: chown d1/f12 6597 1 2024-03-23T11:47:39.493 INFO:tasks.workunit.client.0.smithi063.stdout:2/431: readlink d2/d23/d2e/l42 0 2024-03-23T11:47:39.494 INFO:tasks.workunit.client.0.smithi063.stdout:8/757: creat d6/d1f/d54/d63/d90/f100 x:0 0 0 2024-03-23T11:47:39.496 INFO:tasks.workunit.client.0.smithi063.stdout:3/699: mkdir d13/d22/d2d/d3f/d58/d97/dac/dd9 0 2024-03-23T11:47:39.498 INFO:tasks.workunit.client.1.smithi111.stdout:0/614: dwrite d5/db/d1f/d19/f2a [0,4194304] 0 2024-03-23T11:47:39.501 INFO:tasks.workunit.client.1.smithi111.stdout:8/535: creat d5/d26/d2f/d62/d75/d4b/fb1 x:0 0 0 2024-03-23T11:47:39.504 INFO:tasks.workunit.client.0.smithi063.stdout:8/758: chown d6/d1f/d54/d5c/d66/c4d 15484 1 2024-03-23T11:47:39.508 INFO:tasks.workunit.client.1.smithi111.stdout:3/570: rename d0/d1a/d1d/d26/la0 to d0/d53/lc1 0 2024-03-23T11:47:39.510 INFO:tasks.workunit.client.0.smithi063.stdout:3/700: write d13/d22/d2d/d3f/d58/dcb/fd5 [713511,37172] 0 2024-03-23T11:47:39.510 INFO:tasks.workunit.client.1.smithi111.stdout:9/663: symlink d0/d4/d6a/d89/ldb 0 2024-03-23T11:47:39.511 INFO:tasks.workunit.client.0.smithi063.stdout:0/546: truncate d2/f56 1214819 0 2024-03-23T11:47:39.514 INFO:tasks.workunit.client.0.smithi063.stdout:5/367: dwrite d2/d28/d3a/f4e [0,4194304] 0 2024-03-23T11:47:39.520 INFO:tasks.workunit.client.1.smithi111.stdout:4/643: fsync d0/dc/d4c/d2a/f38 0 2024-03-23T11:47:39.520 INFO:tasks.workunit.client.0.smithi063.stdout:0/547: read - d2/d14/d36/d77/fbd zero size 2024-03-23T11:47:39.522 INFO:tasks.workunit.client.1.smithi111.stdout:4/644: fdatasync d0/dc/d1f/db0/faa 0 2024-03-23T11:47:39.524 INFO:tasks.workunit.client.1.smithi111.stdout:4/645: write d0/d17/f2c [1506665,61784] 0 2024-03-23T11:47:39.527 INFO:tasks.workunit.client.0.smithi063.stdout:5/368: dread - d2/d28/d3a/d46/d79/f7e zero size 2024-03-23T11:47:39.529 INFO:tasks.workunit.client.0.smithi063.stdout:2/432: rmdir d2/d23 39 2024-03-23T11:47:39.531 INFO:tasks.workunit.client.1.smithi111.stdout:0/615: symlink d5/db/d1f/d47/d87/lce 0 2024-03-23T11:47:39.535 INFO:tasks.workunit.client.0.smithi063.stdout:0/548: read d2/d14/d36/d77/f7c [272989,84828] 0 2024-03-23T11:47:39.539 INFO:tasks.workunit.client.0.smithi063.stdout:3/701: rmdir d13/d22/d2d/d3f 39 2024-03-23T11:47:39.540 INFO:tasks.workunit.client.1.smithi111.stdout:6/663: rename d2/d35/d70/cd5 to d2/d1f/d45/d65/cdc 0 2024-03-23T11:47:39.541 INFO:tasks.workunit.client.0.smithi063.stdout:7/457: mknod d5/d5c/d2f/d73/da/c96 0 2024-03-23T11:47:39.541 INFO:tasks.workunit.client.1.smithi111.stdout:7/615: write fe [1592363,99309] 0 2024-03-23T11:47:39.542 INFO:tasks.workunit.client.1.smithi111.stdout:6/664: write d2/d1f/fcf [831909,73630] 0 2024-03-23T11:47:39.543 INFO:tasks.workunit.client.1.smithi111.stdout:1/606: truncate d0/d5/dd/d1e/d54/f84 2259589 0 2024-03-23T11:47:39.544 INFO:tasks.workunit.client.0.smithi063.stdout:6/429: write d1/dc/d3a/f3f [1187533,11803] 0 2024-03-23T11:47:39.547 INFO:tasks.workunit.client.0.smithi063.stdout:1/410: dread d3/f11 [0,4194304] 0 2024-03-23T11:47:39.549 INFO:tasks.workunit.client.0.smithi063.stdout:9/540: dwrite d4/d31/f59 [4194304,4194304] 0 2024-03-23T11:47:39.557 INFO:tasks.workunit.client.0.smithi063.stdout:2/433: symlink d2/d46/l8b 0 2024-03-23T11:47:39.562 INFO:tasks.workunit.client.0.smithi063.stdout:4/570: dwrite d5/da/d20/d4b/d16/d50/d86/f89 [0,4194304] 0 2024-03-23T11:47:39.563 INFO:tasks.workunit.client.0.smithi063.stdout:9/541: write d4/d31/d43/da9/da6/f87 [547312,67834] 0 2024-03-23T11:47:39.565 INFO:tasks.workunit.client.1.smithi111.stdout:4/646: mkdir d0/dc/d89/dc9 0 2024-03-23T11:47:39.566 INFO:tasks.workunit.client.1.smithi111.stdout:2/675: getdents d2/d5c 0 2024-03-23T11:47:39.568 INFO:tasks.workunit.client.1.smithi111.stdout:8/536: creat d5/d32/d9b/fb2 x:0 0 0 2024-03-23T11:47:39.569 INFO:tasks.workunit.client.1.smithi111.stdout:4/647: truncate d0/dc/d78/f85 44116 0 2024-03-23T11:47:39.570 INFO:tasks.workunit.client.1.smithi111.stdout:3/571: mkdir d0/d1a/d2b/d38/dc2 0 2024-03-23T11:47:39.573 INFO:tasks.workunit.client.0.smithi063.stdout:8/759: dwrite d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/fdc [0,4194304] 0 2024-03-23T11:47:39.577 INFO:tasks.workunit.client.1.smithi111.stdout:6/665: mkdir d2/d1f/d45/d9a/ddd 0 2024-03-23T11:47:39.578 INFO:tasks.workunit.client.0.smithi063.stdout:8/760: write d6/d1f/f74 [2039132,59392] 0 2024-03-23T11:47:39.580 INFO:tasks.workunit.client.1.smithi111.stdout:6/666: write d2/d1a/d1c/d2c/d51/f2f [1224648,17430] 0 2024-03-23T11:47:39.581 INFO:tasks.workunit.client.0.smithi063.stdout:1/411: mkdir d3/d49/d56/d65/d84 0 2024-03-23T11:47:39.583 INFO:tasks.workunit.client.1.smithi111.stdout:5/666: link d1/l42 d1/d3/d6/d7/d63/d2c/d43/ldb 0 2024-03-23T11:47:39.585 INFO:tasks.workunit.client.1.smithi111.stdout:5/667: read - d1/d20/d41/fb4 zero size 2024-03-23T11:47:39.585 INFO:tasks.workunit.client.0.smithi063.stdout:7/458: creat d5/d6/d19/d1f/d47/f97 x:0 0 0 2024-03-23T11:47:39.590 INFO:tasks.workunit.client.0.smithi063.stdout:6/430: getdents d1 0 2024-03-23T11:47:39.593 INFO:tasks.workunit.client.0.smithi063.stdout:8/761: mknod d6/d1f/d3e/df6/c101 0 2024-03-23T11:47:39.595 INFO:tasks.workunit.client.0.smithi063.stdout:2/434: rename d2/c12 to d2/c8c 0 2024-03-23T11:47:39.597 INFO:tasks.workunit.client.1.smithi111.stdout:6/667: dwrite d2/d1a/f72 [0,4194304] 0 2024-03-23T11:47:39.597 INFO:tasks.workunit.client.0.smithi063.stdout:7/459: readlink d5/d6/d19/d1f/d47/d4f/l83 0 2024-03-23T11:47:39.598 INFO:tasks.workunit.client.1.smithi111.stdout:2/676: dread d2/d11/d17/d3e/d6b/d9d/d52/f73 [0,4194304] 0 2024-03-23T11:47:39.598 INFO:tasks.workunit.client.1.smithi111.stdout:9/664: creat d0/d4/d1c/d36/fdc x:0 0 0 2024-03-23T11:47:39.600 INFO:tasks.workunit.client.1.smithi111.stdout:9/665: readlink d0/d31/d35/l57 0 2024-03-23T11:47:39.601 INFO:tasks.workunit.client.1.smithi111.stdout:5/668: dread d1/d3/d6/d7/d63/d2c/f34 [0,4194304] 0 2024-03-23T11:47:39.603 INFO:tasks.workunit.client.0.smithi063.stdout:2/435: fsync d2/de/d22/f4a 0 2024-03-23T11:47:39.605 INFO:tasks.workunit.client.1.smithi111.stdout:1/607: link d0/d5/dd/fc0 d0/d5/dd/dc6/fcf 0 2024-03-23T11:47:39.605 INFO:tasks.workunit.client.1.smithi111.stdout:8/537: creat d5/d32/d79/fb3 x:0 0 0 2024-03-23T11:47:39.606 INFO:tasks.workunit.client.0.smithi063.stdout:7/460: mkdir d5/d6/d19/d1f/d47/d6e/d98 0 2024-03-23T11:47:39.607 INFO:tasks.workunit.client.1.smithi111.stdout:8/538: write d5/d32/d7d/faa [438903,92561] 0 2024-03-23T11:47:39.607 INFO:tasks.workunit.client.1.smithi111.stdout:1/608: write d0/d5/dd/d91/fa1 [1259915,112446] 0 2024-03-23T11:47:39.608 INFO:tasks.workunit.client.1.smithi111.stdout:4/648: symlink d0/dc/d89/dc9/lca 0 2024-03-23T11:47:39.609 INFO:tasks.workunit.client.0.smithi063.stdout:0/549: write d2/d14/f23 [2454440,30487] 0 2024-03-23T11:47:39.610 INFO:tasks.workunit.client.0.smithi063.stdout:8/762: truncate d6/d1f/d54/d5c/d66/f83 1274932 0 2024-03-23T11:47:39.610 INFO:tasks.workunit.client.0.smithi063.stdout:7/461: write d5/d6/f7a [4562348,17608] 0 2024-03-23T11:47:39.612 INFO:tasks.workunit.client.1.smithi111.stdout:1/609: read d0/d5/f5e [434033,97731] 0 2024-03-23T11:47:39.612 INFO:tasks.workunit.client.1.smithi111.stdout:6/668: fsync d2/d35/d70/fc4 0 2024-03-23T11:47:39.615 INFO:tasks.workunit.client.1.smithi111.stdout:2/677: write d2/d11/d17/d3e/d6b/d2e/d76/fab [101761,1682] 0 2024-03-23T11:47:39.617 INFO:tasks.workunit.client.1.smithi111.stdout:5/669: chown d1/d3/d6/l1a 55 1 2024-03-23T11:47:39.618 INFO:tasks.workunit.client.1.smithi111.stdout:9/666: dwrite d0/d31/d41/d71/d8f/fa9 [0,4194304] 0 2024-03-23T11:47:39.618 INFO:tasks.workunit.client.0.smithi063.stdout:2/436: mkdir d2/de/d7a/d8d 0 2024-03-23T11:47:39.619 INFO:tasks.workunit.client.0.smithi063.stdout:1/412: dread d3/df/d2f/f64 [0,4194304] 0 2024-03-23T11:47:39.619 INFO:tasks.workunit.client.1.smithi111.stdout:0/616: getdents d5/db/d1f/d19/dbd/d64 0 2024-03-23T11:47:39.621 INFO:tasks.workunit.client.1.smithi111.stdout:4/649: mkdir d0/d95/dcb 0 2024-03-23T11:47:39.626 INFO:tasks.workunit.client.0.smithi063.stdout:2/437: write d2/f85 [101201,112017] 0 2024-03-23T11:47:39.629 INFO:tasks.workunit.client.1.smithi111.stdout:8/539: mkdir d5/d26/d31/d5e/d4a/db4 0 2024-03-23T11:47:39.630 INFO:tasks.workunit.client.1.smithi111.stdout:8/540: fdatasync d5/d32/d7d/fa6 0 2024-03-23T11:47:39.631 INFO:tasks.workunit.client.0.smithi063.stdout:6/431: getdents d1/dc/d21 0 2024-03-23T11:47:39.635 INFO:tasks.workunit.client.0.smithi063.stdout:1/413: symlink d3/d1c/d4b/l85 0 2024-03-23T11:47:39.637 INFO:tasks.workunit.client.0.smithi063.stdout:8/763: dread d6/d1f/d54/f6a [0,4194304] 0 2024-03-23T11:47:39.639 INFO:tasks.workunit.client.1.smithi111.stdout:1/610: creat d0/d6/d48/d4a/fd0 x:0 0 0 2024-03-23T11:47:39.640 INFO:tasks.workunit.client.0.smithi063.stdout:6/432: mkdir d1/dc/d3a/d82 0 2024-03-23T11:47:39.640 INFO:tasks.workunit.client.0.smithi063.stdout:1/414: stat d3/d1c/c1f 0 2024-03-23T11:47:39.642 INFO:tasks.workunit.client.1.smithi111.stdout:6/669: mkdir d2/d1f/d45/d9a/dde 0 2024-03-23T11:47:39.645 INFO:tasks.workunit.client.1.smithi111.stdout:2/678: creat d2/de/d12/d70/ff4 x:0 0 0 2024-03-23T11:47:39.646 INFO:tasks.workunit.client.0.smithi063.stdout:3/702: dwrite d13/d22/d2d/d3e/dce/fa6 [0,4194304] 0 2024-03-23T11:47:39.646 INFO:tasks.workunit.client.0.smithi063.stdout:1/415: creat d3/d49/d56/f86 x:0 0 0 2024-03-23T11:47:39.646 INFO:tasks.workunit.client.0.smithi063.stdout:0/550: dread d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:39.657 INFO:tasks.workunit.client.0.smithi063.stdout:8/764: getdents d6/d1f/d54/d63/d90/d5d 0 2024-03-23T11:47:39.660 INFO:tasks.workunit.client.0.smithi063.stdout:9/542: dwrite d4/d4c/d44/f6b [0,4194304] 0 2024-03-23T11:47:39.660 INFO:tasks.workunit.client.1.smithi111.stdout:4/650: mkdir d0/d95/d9b/db3/dcc 0 2024-03-23T11:47:39.663 INFO:tasks.workunit.client.1.smithi111.stdout:8/541: rename d5/d26/d2f/d42/d58 to d5/d26/d2f/d62/db5 0 2024-03-23T11:47:39.665 INFO:tasks.workunit.client.1.smithi111.stdout:6/670: stat d2/f3b 0 2024-03-23T11:47:39.666 INFO:tasks.workunit.client.1.smithi111.stdout:8/542: write d5/d32/d9b/fb2 [331132,42263] 0 2024-03-23T11:47:39.667 INFO:tasks.workunit.client.1.smithi111.stdout:6/671: fsync d2/d1f/d45/d49/fb7 0 2024-03-23T11:47:39.669 INFO:tasks.workunit.client.0.smithi063.stdout:6/433: dwrite d1/d15/f70 [0,4194304] 0 2024-03-23T11:47:39.669 INFO:tasks.workunit.client.1.smithi111.stdout:3/572: write d0/d1a/d4f/f4e [1585190,73997] 0 2024-03-23T11:47:39.672 INFO:tasks.workunit.client.1.smithi111.stdout:2/679: rename d2/de/d12/d70/fc3 to d2/de/d12/d70/d96/ff5 0 2024-03-23T11:47:39.673 INFO:tasks.workunit.client.0.smithi063.stdout:3/703: rmdir d13/d15/d7e 39 2024-03-23T11:47:39.673 INFO:tasks.workunit.client.0.smithi063.stdout:6/434: write d1/f6e [588502,14221] 0 2024-03-23T11:47:39.673 INFO:tasks.workunit.client.1.smithi111.stdout:2/680: fdatasync d2/d11/f8a 0 2024-03-23T11:47:39.676 INFO:tasks.workunit.client.1.smithi111.stdout:7/616: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/f2b [0,4194304] 0 2024-03-23T11:47:39.677 INFO:tasks.workunit.client.1.smithi111.stdout:9/667: rmdir d0/d4/d1c 39 2024-03-23T11:47:39.681 INFO:tasks.workunit.client.0.smithi063.stdout:9/543: creat d4/d31/d70/d8d/fbb x:0 0 0 2024-03-23T11:47:39.682 INFO:tasks.workunit.client.1.smithi111.stdout:1/611: link d0/d5/dd/d1e/d54/d87/d8e/fcd d0/d2b/d7c/dca/fd1 0 2024-03-23T11:47:39.682 INFO:tasks.workunit.client.0.smithi063.stdout:8/765: mknod d6/d1f/d54/d5c/d76/c102 0 2024-03-23T11:47:39.682 INFO:tasks.workunit.client.0.smithi063.stdout:7/462: truncate d5/d6/d19/d1f/f6d 2873393 0 2024-03-23T11:47:39.683 INFO:tasks.workunit.client.0.smithi063.stdout:0/551: dwrite d2/d48/f7f [0,4194304] 0 2024-03-23T11:47:39.688 INFO:tasks.workunit.client.0.smithi063.stdout:0/552: chown d2/d48/lab 0 1 2024-03-23T11:47:39.689 INFO:tasks.workunit.client.1.smithi111.stdout:8/543: dwrite d5/d26/d31/d5e/d4a/f52 [0,4194304] 0 2024-03-23T11:47:39.690 INFO:tasks.workunit.client.1.smithi111.stdout:6/672: mkdir d2/d1a/d1c/ddf 0 2024-03-23T11:47:39.694 INFO:tasks.workunit.client.0.smithi063.stdout:3/704: write d13/d22/d2d/d3f/d67/fa3 [33157,70024] 0 2024-03-23T11:47:39.695 INFO:tasks.workunit.client.1.smithi111.stdout:3/573: write d0/d7/d19/f29 [5170862,74706] 0 2024-03-23T11:47:39.696 INFO:tasks.workunit.client.0.smithi063.stdout:6/435: read d1/d15/d4b/f4c [1168060,58617] 0 2024-03-23T11:47:39.697 INFO:tasks.workunit.client.0.smithi063.stdout:9/544: rmdir d4/d31/d43/d49/d8e 39 2024-03-23T11:47:39.700 INFO:tasks.workunit.client.1.smithi111.stdout:5/670: link d1/d3/d6/d7/d63/d2c/d43/ldb d1/d3/d6/d7/d63/d9d/ldc 0 2024-03-23T11:47:39.700 INFO:tasks.workunit.client.1.smithi111.stdout:4/651: link d0/dc/d4c/d71/lbf d0/dc/d1f/d5e/lcd 0 2024-03-23T11:47:39.702 INFO:tasks.workunit.client.1.smithi111.stdout:0/617: write d5/db/f15 [2708139,43598] 0 2024-03-23T11:47:39.705 INFO:tasks.workunit.client.1.smithi111.stdout:0/618: chown d5/db/f16 24 1 2024-03-23T11:47:39.705 INFO:tasks.workunit.client.1.smithi111.stdout:5/671: fdatasync d1/d3/d6/d7/d63/d2c/d82/d66/f71 0 2024-03-23T11:47:39.708 INFO:tasks.workunit.client.1.smithi111.stdout:6/673: creat d2/d1a/fe0 x:0 0 0 2024-03-23T11:47:39.709 INFO:tasks.workunit.client.0.smithi063.stdout:2/438: dwrite d2/de/f15 [0,4194304] 0 2024-03-23T11:47:39.711 INFO:tasks.workunit.client.0.smithi063.stdout:8/766: mknod d6/d1f/dbb/c103 0 2024-03-23T11:47:39.718 INFO:tasks.workunit.client.0.smithi063.stdout:1/416: link d3/d1c/c3a d3/d49/d56/d65/d84/c87 0 2024-03-23T11:47:39.719 INFO:tasks.workunit.client.1.smithi111.stdout:4/652: readlink d0/dc/d4c/d71/lbc 0 2024-03-23T11:47:39.721 INFO:tasks.workunit.client.1.smithi111.stdout:1/612: dread d0/f1a [0,4194304] 0 2024-03-23T11:47:39.725 INFO:tasks.workunit.client.0.smithi063.stdout:7/463: dwrite d5/d6/f7a [0,4194304] 0 2024-03-23T11:47:39.729 INFO:tasks.workunit.client.1.smithi111.stdout:2/681: write d2/d7/fd [3781526,125520] 0 2024-03-23T11:47:39.732 INFO:tasks.workunit.client.1.smithi111.stdout:6/674: creat d2/d1f/dce/fe1 x:0 0 0 2024-03-23T11:47:39.733 INFO:tasks.workunit.client.1.smithi111.stdout:1/613: symlink d0/d5/dd/d1e/ld2 0 2024-03-23T11:47:39.737 INFO:tasks.workunit.client.0.smithi063.stdout:1/417: write d3/d3f/d4d/f69 [5949283,61653] 0 2024-03-23T11:47:39.742 INFO:tasks.workunit.client.1.smithi111.stdout:5/672: truncate d1/d3/d6/d8b/d59/d74/f4b 986245 0 2024-03-23T11:47:39.743 INFO:tasks.workunit.client.1.smithi111.stdout:8/544: rename d5/d1b/f68 to d5/d26/d31/fb6 0 2024-03-23T11:47:39.744 INFO:tasks.workunit.client.1.smithi111.stdout:2/682: fdatasync d2/f4f 0 2024-03-23T11:47:39.745 INFO:tasks.workunit.client.0.smithi063.stdout:6/436: dread d1/d13/f53 [0,4194304] 0 2024-03-23T11:47:39.746 INFO:tasks.workunit.client.1.smithi111.stdout:5/673: fdatasync d1/d3/d6/d7/d7b/fa4 0 2024-03-23T11:47:39.746 INFO:tasks.workunit.client.1.smithi111.stdout:9/668: write d0/d4/d1c/d36/f7e [426315,82331] 0 2024-03-23T11:47:39.747 INFO:tasks.workunit.client.1.smithi111.stdout:3/574: dread d0/d1a/d2d/f45 [0,4194304] 0 2024-03-23T11:47:39.749 INFO:tasks.workunit.client.0.smithi063.stdout:3/705: dwrite d13/d22/d2d/d3f/d58/d86/f93 [0,4194304] 0 2024-03-23T11:47:39.750 INFO:tasks.workunit.client.1.smithi111.stdout:6/675: write d2/ff [1847352,123969] 0 2024-03-23T11:47:39.750 INFO:tasks.workunit.client.0.smithi063.stdout:9/545: truncate d4/d31/d43/d49/d8e/f93 187196 0 2024-03-23T11:47:39.750 INFO:tasks.workunit.client.0.smithi063.stdout:1/418: write d3/f2a [4936236,38626] 0 2024-03-23T11:47:39.751 INFO:tasks.workunit.client.1.smithi111.stdout:2/683: truncate d2/de/d12/d70/f7e 90908 0 2024-03-23T11:47:39.753 INFO:tasks.workunit.client.1.smithi111.stdout:3/575: truncate d0/d1a/d1d/f49 1023778 0 2024-03-23T11:47:39.753 INFO:tasks.workunit.client.1.smithi111.stdout:5/674: write d1/d3/d6/d7/d63/d2c/d82/d66/d77/f7a [9502997,124628] 0 2024-03-23T11:47:39.755 INFO:tasks.workunit.client.1.smithi111.stdout:5/675: chown d1/d20/d5c 0 1 2024-03-23T11:47:39.757 INFO:tasks.workunit.client.1.smithi111.stdout:4/653: rename d0/dc/d7a/fac to d0/d95/d9b/db3/db6/fce 0 2024-03-23T11:47:39.758 INFO:tasks.workunit.client.0.smithi063.stdout:0/553: dwrite d2/d12/d32/d6d/f9c [0,4194304] 0 2024-03-23T11:47:39.758 INFO:tasks.workunit.client.0.smithi063.stdout:6/437: dread - d1/dc/d3a/f61 zero size 2024-03-23T11:47:39.759 INFO:tasks.workunit.client.0.smithi063.stdout:8/767: dread d6/d1f/f74 [0,4194304] 0 2024-03-23T11:47:39.761 INFO:tasks.workunit.client.1.smithi111.stdout:1/614: symlink d0/d2b/d7c/dca/ld3 0 2024-03-23T11:47:39.761 INFO:tasks.workunit.client.1.smithi111.stdout:4/654: truncate d0/dc/d4c/d2a/d97/d4d/f9c 585812 0 2024-03-23T11:47:39.762 INFO:tasks.workunit.client.1.smithi111.stdout:1/615: fdatasync d0/d5/f2f 0 2024-03-23T11:47:39.763 INFO:tasks.workunit.client.1.smithi111.stdout:1/616: readlink d0/l2a 0 2024-03-23T11:47:39.763 INFO:tasks.workunit.client.1.smithi111.stdout:4/655: readlink d0/dc/d4c/d83/lb4 0 2024-03-23T11:47:39.766 INFO:tasks.workunit.client.0.smithi063.stdout:7/464: getdents d5/d26 0 2024-03-23T11:47:39.767 INFO:tasks.workunit.client.0.smithi063.stdout:1/419: rmdir d3/df/d2f 39 2024-03-23T11:47:39.767 INFO:tasks.workunit.client.1.smithi111.stdout:0/619: write d5/db/d1f/d19/f53 [389821,43820] 0 2024-03-23T11:47:39.768 INFO:tasks.workunit.client.1.smithi111.stdout:4/656: chown d0/dc/d78/f98 14 1 2024-03-23T11:47:39.771 INFO:tasks.workunit.client.0.smithi063.stdout:2/439: dwrite d2/d23/f41 [0,4194304] 0 2024-03-23T11:47:39.773 INFO:tasks.workunit.client.1.smithi111.stdout:4/657: fdatasync d0/fc6 0 2024-03-23T11:47:39.773 INFO:tasks.workunit.client.1.smithi111.stdout:2/684: symlink d2/d11/d17/d1f/da0/dd2/lf6 0 2024-03-23T11:47:39.774 INFO:tasks.workunit.client.1.smithi111.stdout:2/685: chown d2/d32/d91/fb9 413846 1 2024-03-23T11:47:39.777 INFO:tasks.workunit.client.1.smithi111.stdout:1/617: rename d0/d6/c47 to d0/d5/dd/d1e/d70/cd4 0 2024-03-23T11:47:39.778 INFO:tasks.workunit.client.0.smithi063.stdout:8/768: fsync d6/dd/d59/db2/dbe/dfb/fed 0 2024-03-23T11:47:39.780 INFO:tasks.workunit.client.0.smithi063.stdout:6/438: creat d1/d2a/f83 x:0 0 0 2024-03-23T11:47:39.782 INFO:tasks.workunit.client.1.smithi111.stdout:9/669: getdents d0/d4/d6a/d89 0 2024-03-23T11:47:39.784 INFO:tasks.workunit.client.1.smithi111.stdout:4/658: truncate d0/dc/d89/da4/fbd 1039429 0 2024-03-23T11:47:39.789 INFO:tasks.workunit.client.0.smithi063.stdout:2/440: read - d2/d84/f89 zero size 2024-03-23T11:47:39.790 INFO:tasks.workunit.client.0.smithi063.stdout:7/465: write d5/d26/f74 [3141039,118521] 0 2024-03-23T11:47:39.790 INFO:tasks.workunit.client.0.smithi063.stdout:0/554: chown d2/d14/d36/d77/d94/da6/d28/d46/d51/l96 6863 1 2024-03-23T11:47:39.791 INFO:tasks.workunit.client.0.smithi063.stdout:9/546: link d4/d31/d43/da9/da6/f87 d4/d31/d43/d49/d8e/fbc 0 2024-03-23T11:47:39.793 INFO:tasks.workunit.client.1.smithi111.stdout:1/618: read d0/d2b/f45 [2622184,46302] 0 2024-03-23T11:47:39.793 INFO:tasks.workunit.client.1.smithi111.stdout:9/670: chown d0/d4/d1c/d2d/d3b/d42/d4f/c7d 194 1 2024-03-23T11:47:39.794 INFO:tasks.workunit.client.1.smithi111.stdout:5/676: dwrite d1/d3/d6/d7/f10 [0,4194304] 0 2024-03-23T11:47:39.796 INFO:tasks.workunit.client.0.smithi063.stdout:8/769: getdents d6/dd 0 2024-03-23T11:47:39.796 INFO:tasks.workunit.client.1.smithi111.stdout:9/671: chown d0/d4/d10 95353562 1 2024-03-23T11:47:39.796 INFO:tasks.workunit.client.1.smithi111.stdout:3/576: dwrite d0/d55/f81 [0,4194304] 0 2024-03-23T11:47:39.801 INFO:tasks.workunit.client.0.smithi063.stdout:0/555: readlink d2/d14/d36/d77/d94/da6/d28/l4b 0 2024-03-23T11:47:39.803 INFO:tasks.workunit.client.0.smithi063.stdout:6/439: fsync d1/d2a/f83 0 2024-03-23T11:47:39.804 INFO:tasks.workunit.client.1.smithi111.stdout:1/619: mkdir d0/d5/dd5 0 2024-03-23T11:47:39.805 INFO:tasks.workunit.client.0.smithi063.stdout:7/466: write d5/d5c/d2f/d73/d20/f82 [4390515,58261] 0 2024-03-23T11:47:39.809 INFO:tasks.workunit.client.0.smithi063.stdout:9/547: symlink d4/d31/d43/da9/lbd 0 2024-03-23T11:47:39.812 INFO:tasks.workunit.client.1.smithi111.stdout:1/620: rmdir d0/d6/d48 39 2024-03-23T11:47:39.812 INFO:tasks.workunit.client.0.smithi063.stdout:3/706: dwrite d13/d22/d2d/f38 [0,4194304] 0 2024-03-23T11:47:39.812 INFO:tasks.workunit.client.0.smithi063.stdout:0/556: write d2/d12/d32/fc0 [712901,120741] 0 2024-03-23T11:47:39.813 INFO:tasks.workunit.client.1.smithi111.stdout:4/659: dwrite d0/dc/d78/fc0 [4194304,4194304] 0 2024-03-23T11:47:39.815 INFO:tasks.workunit.client.1.smithi111.stdout:2/686: write d2/d11/d17/d1f/f20 [5179725,31870] 0 2024-03-23T11:47:39.816 INFO:tasks.workunit.client.0.smithi063.stdout:8/770: unlink d6/d1f/d54/d63/d90/d5d/d7d/d9a/ld2 0 2024-03-23T11:47:39.818 INFO:tasks.workunit.client.1.smithi111.stdout:9/672: symlink d0/d4/d1c/d2d/d3b/d42/ldd 0 2024-03-23T11:47:39.818 INFO:tasks.workunit.client.0.smithi063.stdout:1/420: dwrite d3/d1c/f6d [4194304,4194304] 0 2024-03-23T11:47:39.824 INFO:tasks.workunit.client.1.smithi111.stdout:2/687: fsync d2/de/f84 0 2024-03-23T11:47:39.824 INFO:tasks.workunit.client.1.smithi111.stdout:6/676: dread d2/d1f/d45/d49/f6c [0,4194304] 0 2024-03-23T11:47:39.824 INFO:tasks.workunit.client.0.smithi063.stdout:9/548: write d4/d31/d43/da9/da6/fb0 [704988,105348] 0 2024-03-23T11:47:39.826 INFO:tasks.workunit.client.1.smithi111.stdout:6/677: readlink d2/d35/d70/l84 0 2024-03-23T11:47:39.827 INFO:tasks.workunit.client.1.smithi111.stdout:3/577: dwrite d0/d53/faf [0,4194304] 0 2024-03-23T11:47:39.828 INFO:tasks.workunit.client.0.smithi063.stdout:1/421: fdatasync d3/d49/d56/d65/f7c 0 2024-03-23T11:47:39.830 INFO:tasks.workunit.client.1.smithi111.stdout:4/660: creat d0/dc/d89/da4/fcf x:0 0 0 2024-03-23T11:47:39.833 INFO:tasks.workunit.client.0.smithi063.stdout:8/771: truncate d6/d1c/d32/d37/fc0 847198 0 2024-03-23T11:47:39.835 INFO:tasks.workunit.client.1.smithi111.stdout:2/688: mkdir d2/d11/d17/d3e/d5d/dc0/df7 0 2024-03-23T11:47:39.836 INFO:tasks.workunit.client.1.smithi111.stdout:9/673: rename d0/d4/l90 to d0/d31/da0/lde 0 2024-03-23T11:47:39.837 INFO:tasks.workunit.client.0.smithi063.stdout:0/557: creat d2/d14/d36/d77/d94/da6/d1f/d4f/fcc x:0 0 0 2024-03-23T11:47:39.840 INFO:tasks.workunit.client.0.smithi063.stdout:9/549: mkdir d4/d31/d43/da9/da6/dbe 0 2024-03-23T11:47:39.840 INFO:tasks.workunit.client.0.smithi063.stdout:3/707: mkdir d13/d22/d2d/d3f/d58/d6e/d8d/dda 0 2024-03-23T11:47:39.842 INFO:tasks.workunit.client.1.smithi111.stdout:4/661: rmdir d0/dc/d30 39 2024-03-23T11:47:39.842 INFO:tasks.workunit.client.0.smithi063.stdout:3/708: write d13/d15/f2a [109835,52523] 0 2024-03-23T11:47:39.842 INFO:tasks.workunit.client.1.smithi111.stdout:3/578: creat d0/d1a/d2b/d38/d7f/fc3 x:0 0 0 2024-03-23T11:47:39.843 INFO:tasks.workunit.client.1.smithi111.stdout:2/689: symlink d2/d11/d17/d3e/d5d/lf8 0 2024-03-23T11:47:39.845 INFO:tasks.workunit.client.1.smithi111.stdout:3/579: chown d0/d7/d19 1 1 2024-03-23T11:47:39.845 INFO:tasks.workunit.client.0.smithi063.stdout:9/550: write d4/d4c/f72 [4864952,107811] 0 2024-03-23T11:47:39.846 INFO:tasks.workunit.client.0.smithi063.stdout:8/772: mkdir d6/d1f/d54/d63/d90/d5d/d7d/d9a/d104 0 2024-03-23T11:47:39.848 INFO:tasks.workunit.client.1.smithi111.stdout:2/690: unlink d2/d32/fdf 0 2024-03-23T11:47:39.848 INFO:tasks.workunit.client.1.smithi111.stdout:6/678: truncate d2/d1f/d45/f62 2644949 0 2024-03-23T11:47:39.848 INFO:tasks.workunit.client.1.smithi111.stdout:3/580: readlink d0/d7/d19/la4 0 2024-03-23T11:47:39.848 INFO:tasks.workunit.client.1.smithi111.stdout:4/662: truncate d0/fc6 190437 0 2024-03-23T11:47:39.850 INFO:tasks.workunit.client.0.smithi063.stdout:1/422: dread d3/d35/d40/f42 [0,4194304] 0 2024-03-23T11:47:39.853 INFO:tasks.workunit.client.1.smithi111.stdout:3/581: mknod d0/d1a/d2d/cc4 0 2024-03-23T11:47:39.855 INFO:tasks.workunit.client.1.smithi111.stdout:6/679: write d2/d1f/d45/d9a/fb0 [262893,62671] 0 2024-03-23T11:47:39.855 INFO:tasks.workunit.client.1.smithi111.stdout:2/691: symlink d2/d11/d17/d1f/da0/lf9 0 2024-03-23T11:47:39.856 INFO:tasks.workunit.client.0.smithi063.stdout:6/440: dread d1/dc/d3a/f3f [0,4194304] 0 2024-03-23T11:47:39.858 INFO:tasks.workunit.client.0.smithi063.stdout:8/773: symlink d6/d1f/d54/d5c/dc5/l105 0 2024-03-23T11:47:39.859 INFO:tasks.workunit.client.1.smithi111.stdout:1/621: dwrite d0/d2b/f5a [4194304,4194304] 0 2024-03-23T11:47:39.861 INFO:tasks.workunit.client.0.smithi063.stdout:8/774: chown d6/d1f/c21 3646 1 2024-03-23T11:47:39.863 INFO:tasks.workunit.client.1.smithi111.stdout:5/677: write d1/d3/d6/d7/d8c/f96 [325192,22422] 0 2024-03-23T11:47:39.865 INFO:tasks.workunit.client.1.smithi111.stdout:3/582: rmdir d0/d1a/d1d/d26/d5a 39 2024-03-23T11:47:39.867 INFO:tasks.workunit.client.1.smithi111.stdout:4/663: rename d0/dc/d4c/d2a/l93 to d0/dc/d4c/d2a/d97/d4f/dba/ld0 0 2024-03-23T11:47:39.867 INFO:tasks.workunit.client.0.smithi063.stdout:3/709: creat d13/d22/d2d/d3f/d58/d6e/d8d/dda/fdb x:0 0 0 2024-03-23T11:47:39.867 INFO:tasks.workunit.client.0.smithi063.stdout:0/558: link d2/d14/d36/d77/c58 d2/d12/d32/ccd 0 2024-03-23T11:47:39.870 INFO:tasks.workunit.client.0.smithi063.stdout:8/775: write d6/d1f/d54/d63/d90/f28 [4141338,105318] 0 2024-03-23T11:47:39.870 INFO:tasks.workunit.client.0.smithi063.stdout:0/559: chown d2/c1e 189597554 1 2024-03-23T11:47:39.870 INFO:tasks.workunit.client.1.smithi111.stdout:6/680: creat d2/d1f/d45/d86/fe2 x:0 0 0 2024-03-23T11:47:39.871 INFO:tasks.workunit.client.1.smithi111.stdout:9/674: dwrite d0/d4/d10/d29/fa3 [0,4194304] 0 2024-03-23T11:47:39.872 INFO:tasks.workunit.client.0.smithi063.stdout:3/710: chown d13/d15/f3c 0 1 2024-03-23T11:47:39.873 INFO:tasks.workunit.client.0.smithi063.stdout:0/560: chown d2/d6e/d8c/f9e 2 1 2024-03-23T11:47:39.877 INFO:tasks.workunit.client.1.smithi111.stdout:7/617: sync 2024-03-23T11:47:39.877 INFO:tasks.workunit.client.1.smithi111.stdout:4/664: creat d0/dc/d78/fd1 x:0 0 0 2024-03-23T11:47:39.877 INFO:tasks.workunit.client.1.smithi111.stdout:5/678: fsync d1/d3/fcd 0 2024-03-23T11:47:39.880 INFO:tasks.workunit.client.0.smithi063.stdout:3/711: creat d13/d22/d2d/d78/dcc/fdc x:0 0 0 2024-03-23T11:47:39.881 INFO:tasks.workunit.client.1.smithi111.stdout:4/665: chown d0/dc/d4c/f6f 11 1 2024-03-23T11:47:39.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:39 smithi063 ceph-mon[35189]: pgmap v39: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 85 MiB/s rd, 317 MiB/s wr, 574 op/s 2024-03-23T11:47:39.886 INFO:tasks.workunit.client.1.smithi111.stdout:5/679: dread - d1/d20/d5c/dc5/fd1 zero size 2024-03-23T11:47:39.888 INFO:tasks.workunit.client.1.smithi111.stdout:3/583: getdents d0/d7/d19 0 2024-03-23T11:47:39.889 INFO:tasks.workunit.client.1.smithi111.stdout:7/618: unlink d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fa7 0 2024-03-23T11:47:39.889 INFO:tasks.workunit.client.1.smithi111.stdout:1/622: dwrite d0/d5/dd/d1e/f21 [0,4194304] 0 2024-03-23T11:47:39.891 INFO:tasks.workunit.client.1.smithi111.stdout:2/692: rename d2/d5c/f63 to d2/d11/d17/d3e/d6b/d9d/d52/dba/dd3/ffa 0 2024-03-23T11:47:39.895 INFO:tasks.workunit.client.1.smithi111.stdout:5/680: write d1/d20/f60 [970147,66769] 0 2024-03-23T11:47:39.897 INFO:tasks.workunit.client.0.smithi063.stdout:3/712: rmdir d13/d22/d2d/d3f/d58/d97/dac/dd9 0 2024-03-23T11:47:39.899 INFO:tasks.workunit.client.0.smithi063.stdout:1/423: write d3/f9 [4579983,63897] 0 2024-03-23T11:47:39.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:39 smithi111 ceph-mon[44005]: pgmap v39: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 85 MiB/s rd, 317 MiB/s wr, 574 op/s 2024-03-23T11:47:39.903 INFO:tasks.workunit.client.0.smithi063.stdout:9/551: truncate d4/d4c/f69 4184266 0 2024-03-23T11:47:39.906 INFO:tasks.workunit.client.1.smithi111.stdout:2/693: dread d2/de/d12/f43 [4194304,4194304] 0 2024-03-23T11:47:39.906 INFO:tasks.workunit.client.0.smithi063.stdout:8/776: write d6/d1f/d54/d63/d90/d15/f24 [4878008,13028] 0 2024-03-23T11:47:39.907 INFO:tasks.workunit.client.1.smithi111.stdout:9/675: creat d0/d4/d1c/d2d/d3b/d42/fdf x:0 0 0 2024-03-23T11:47:39.908 INFO:tasks.workunit.client.0.smithi063.stdout:6/441: dread d1/f8 [0,4194304] 0 2024-03-23T11:47:39.909 INFO:tasks.workunit.client.0.smithi063.stdout:3/713: symlink d13/d22/d2d/d3f/d58/d86/dad/ldd 0 2024-03-23T11:47:39.916 INFO:tasks.workunit.client.1.smithi111.stdout:1/623: symlink d0/d2b/d7c/ld6 0 2024-03-23T11:47:39.917 INFO:tasks.workunit.client.0.smithi063.stdout:6/442: unlink d1/d13/f52 0 2024-03-23T11:47:39.918 INFO:tasks.workunit.client.1.smithi111.stdout:3/584: rmdir d0/d1a 39 2024-03-23T11:47:39.918 INFO:tasks.workunit.client.0.smithi063.stdout:3/714: chown d13/d22/d2d/d7d/f94 1023398 1 2024-03-23T11:47:39.923 INFO:tasks.workunit.client.1.smithi111.stdout:4/666: rename d0/dc/d30/l67 to d0/dc/d4c/d2a/d97/d4f/dba/ld2 0 2024-03-23T11:47:39.923 INFO:tasks.workunit.client.0.smithi063.stdout:3/715: chown d13/d22/d2d/l18 124488 1 2024-03-23T11:47:39.924 INFO:tasks.workunit.client.1.smithi111.stdout:4/667: chown d0/d95/d9b/fc1 268941649 1 2024-03-23T11:47:39.931 INFO:tasks.workunit.client.0.smithi063.stdout:8/777: dwrite d6/d1c/d27/f34 [0,4194304] 0 2024-03-23T11:47:39.932 INFO:tasks.workunit.client.1.smithi111.stdout:5/681: creat d1/d20/d5c/d99/dbf/fdd x:0 0 0 2024-03-23T11:47:39.933 INFO:tasks.workunit.client.0.smithi063.stdout:8/778: chown d6/c58 279 1 2024-03-23T11:47:39.934 INFO:tasks.workunit.client.1.smithi111.stdout:7/619: truncate d14/f36 1406571 0 2024-03-23T11:47:39.936 INFO:tasks.workunit.client.1.smithi111.stdout:5/682: readlink d1/d3/d6/d7/d63/d2c/d82/d66/d77/l93 0 2024-03-23T11:47:39.936 INFO:tasks.workunit.client.1.smithi111.stdout:3/585: dread d0/d7/d19/f52 [0,4194304] 0 2024-03-23T11:47:39.937 INFO:tasks.workunit.client.0.smithi063.stdout:0/561: dread d2/d4/f9 [0,4194304] 0 2024-03-23T11:47:39.938 INFO:tasks.workunit.client.1.smithi111.stdout:4/668: dwrite d0/d17/f2c [0,4194304] 0 2024-03-23T11:47:39.940 INFO:tasks.workunit.client.0.smithi063.stdout:1/424: dread d3/f9 [0,4194304] 0 2024-03-23T11:47:39.942 INFO:tasks.workunit.client.1.smithi111.stdout:2/694: creat d2/de0/ffb x:0 0 0 2024-03-23T11:47:39.943 INFO:tasks.workunit.client.0.smithi063.stdout:1/425: stat d3/d1c/d4b/l85 0 2024-03-23T11:47:39.945 INFO:tasks.workunit.client.0.smithi063.stdout:8/779: mkdir d6/d1f/d106 0 2024-03-23T11:47:39.945 INFO:tasks.workunit.client.0.smithi063.stdout:3/716: dwrite d13/d22/d2d/d3f/d58/f76 [0,4194304] 0 2024-03-23T11:47:39.945 INFO:tasks.workunit.client.1.smithi111.stdout:9/676: mkdir d0/d31/d41/d9d/dd4/de0 0 2024-03-23T11:47:39.947 INFO:tasks.workunit.client.0.smithi063.stdout:3/717: read f5 [1263775,28424] 0 2024-03-23T11:47:39.948 INFO:tasks.workunit.client.0.smithi063.stdout:1/426: mkdir d3/d49/d56/d65/d88 0 2024-03-23T11:47:39.954 INFO:tasks.workunit.client.0.smithi063.stdout:3/718: truncate d13/d22/d2d/d3e/dce/fa6 4474690 0 2024-03-23T11:47:39.955 INFO:tasks.workunit.client.1.smithi111.stdout:4/669: creat d0/dc/d89/da4/fd3 x:0 0 0 2024-03-23T11:47:39.955 INFO:tasks.workunit.client.1.smithi111.stdout:9/677: rmdir d0/d4/d1c/d2d 39 2024-03-23T11:47:39.956 INFO:tasks.workunit.client.1.smithi111.stdout:7/620: creat d14/fac x:0 0 0 2024-03-23T11:47:39.959 INFO:tasks.workunit.client.1.smithi111.stdout:3/586: rmdir d0/d72/da9 0 2024-03-23T11:47:39.960 INFO:tasks.workunit.client.0.smithi063.stdout:1/427: dread d3/f39 [0,4194304] 0 2024-03-23T11:47:39.961 INFO:tasks.workunit.client.1.smithi111.stdout:2/695: creat d2/d32/ffc x:0 0 0 2024-03-23T11:47:39.962 INFO:tasks.workunit.client.1.smithi111.stdout:9/678: chown d0/d31/d41/d9d/dd4 760842217 1 2024-03-23T11:47:39.963 INFO:tasks.workunit.client.0.smithi063.stdout:3/719: mkdir d13/d66/dde 0 2024-03-23T11:47:39.963 INFO:tasks.workunit.client.1.smithi111.stdout:4/670: mknod d0/dc/d4c/d2a/d97/d4f/d72/cd4 0 2024-03-23T11:47:39.965 INFO:tasks.workunit.client.1.smithi111.stdout:1/624: dread d0/d5/f5e [0,4194304] 0 2024-03-23T11:47:39.966 INFO:tasks.workunit.client.0.smithi063.stdout:1/428: mknod d3/df/d7b/c89 0 2024-03-23T11:47:39.966 INFO:tasks.workunit.client.1.smithi111.stdout:3/587: chown d0/d1a/d4f/f70 1316520939 1 2024-03-23T11:47:39.966 INFO:tasks.workunit.client.0.smithi063.stdout:0/562: dwrite d2/d14/d36/d77/d94/da6/d25/d93/fbf [0,4194304] 0 2024-03-23T11:47:39.967 INFO:tasks.workunit.client.1.smithi111.stdout:5/683: dwrite d1/d3/fb0 [4194304,4194304] 0 2024-03-23T11:47:39.969 INFO:tasks.workunit.client.1.smithi111.stdout:7/621: mknod d14/d1b/d1e/d2c/d2e/d33/cad 0 2024-03-23T11:47:39.970 INFO:tasks.workunit.client.1.smithi111.stdout:4/671: readlink d0/dc/d1f/db0/la0 0 2024-03-23T11:47:39.971 INFO:tasks.workunit.client.1.smithi111.stdout:7/622: dread - d14/d1b/d1e/d86/f93 zero size 2024-03-23T11:47:39.972 INFO:tasks.workunit.client.0.smithi063.stdout:1/429: rmdir d3/d3f/d4d 39 2024-03-23T11:47:39.973 INFO:tasks.workunit.client.1.smithi111.stdout:5/684: symlink d1/d3/d6/d30/lde 0 2024-03-23T11:47:39.973 INFO:tasks.workunit.client.1.smithi111.stdout:3/588: readlink d0/d1a/d2b/d38/l5b 0 2024-03-23T11:47:39.974 INFO:tasks.workunit.client.1.smithi111.stdout:4/672: write d0/dc/d1f/f57 [2830824,121698] 0 2024-03-23T11:47:39.975 INFO:tasks.workunit.client.1.smithi111.stdout:2/696: creat d2/d32/ffd x:0 0 0 2024-03-23T11:47:39.981 INFO:tasks.workunit.client.1.smithi111.stdout:7/623: unlink d14/d1b/d1e/d2c/d2e/c8d 0 2024-03-23T11:47:39.982 INFO:tasks.workunit.client.1.smithi111.stdout:5/685: creat d1/d3/d6/fdf x:0 0 0 2024-03-23T11:47:39.985 INFO:tasks.workunit.client.1.smithi111.stdout:5/686: dread - d1/d20/d5c/d99/dbf/fdd zero size 2024-03-23T11:47:39.986 INFO:tasks.workunit.client.1.smithi111.stdout:2/697: truncate d2/d11/f8a 3423438 0 2024-03-23T11:47:39.988 INFO:tasks.workunit.client.1.smithi111.stdout:3/589: dread d0/d55/f81 [0,4194304] 0 2024-03-23T11:47:39.995 INFO:tasks.workunit.client.0.smithi063.stdout:3/720: dwrite d13/d22/d2d/d3f/d67/d7b/fc7 [0,4194304] 0 2024-03-23T11:47:39.996 INFO:tasks.workunit.client.1.smithi111.stdout:3/590: mkdir d0/d1a/d1d/dc5 0 2024-03-23T11:47:39.996 INFO:tasks.workunit.client.1.smithi111.stdout:7/624: rename d14/d37/d50 to d14/d1b/d1e/d2c/d2e/d73/dae 0 2024-03-23T11:47:39.997 INFO:tasks.workunit.client.1.smithi111.stdout:5/687: creat d1/d3/d6/d7/d63/d2c/d43/d95/fe0 x:0 0 0 2024-03-23T11:47:39.998 INFO:tasks.workunit.client.0.smithi063.stdout:0/563: truncate d2/d12/f7e 2433478 0 2024-03-23T11:47:39.998 INFO:tasks.workunit.client.1.smithi111.stdout:8/545: sync 2024-03-23T11:47:39.999 INFO:tasks.workunit.client.0.smithi063.stdout:1/430: dwrite d3/f10 [4194304,4194304] 0 2024-03-23T11:47:40.000 INFO:tasks.workunit.client.1.smithi111.stdout:2/698: creat d2/de/d6c/dd1/dd6/ffe x:0 0 0 2024-03-23T11:47:40.001 INFO:tasks.workunit.client.1.smithi111.stdout:8/546: fdatasync d5/d26/d31/d4d/faf 0 2024-03-23T11:47:40.002 INFO:tasks.workunit.client.0.smithi063.stdout:0/564: chown d2/d14 11 1 2024-03-23T11:47:40.002 INFO:tasks.workunit.client.0.smithi063.stdout:3/721: write d13/d22/d2d/f44 [4671068,106169] 0 2024-03-23T11:47:40.003 INFO:tasks.workunit.client.0.smithi063.stdout:1/431: chown d3/cb 13 1 2024-03-23T11:47:40.007 INFO:tasks.workunit.client.1.smithi111.stdout:1/625: dread d0/d2b/f49 [4194304,4194304] 0 2024-03-23T11:47:40.007 INFO:tasks.workunit.client.1.smithi111.stdout:8/547: truncate d5/d32/d9b/fb2 959181 0 2024-03-23T11:47:40.007 INFO:tasks.workunit.client.1.smithi111.stdout:7/625: mkdir d14/d44/d70/daf 0 2024-03-23T11:47:40.008 INFO:tasks.workunit.client.1.smithi111.stdout:3/591: creat d0/d1a/d2b/d38/dc2/fc6 x:0 0 0 2024-03-23T11:47:40.010 INFO:tasks.workunit.client.1.smithi111.stdout:1/626: dread - d0/d2b/d7c/dca/fd1 zero size 2024-03-23T11:47:40.010 INFO:tasks.workunit.client.1.smithi111.stdout:2/699: rename d2/d5c/c64 to d2/d11/d17/d3e/d6b/d2e/d3f/cff 0 2024-03-23T11:47:40.013 INFO:tasks.workunit.client.1.smithi111.stdout:5/688: creat d1/d3/d6/d8b/dc2/fe1 x:0 0 0 2024-03-23T11:47:40.014 INFO:tasks.workunit.client.0.smithi063.stdout:8/780: dread d6/d1c/d27/f31 [0,4194304] 0 2024-03-23T11:47:40.014 INFO:tasks.workunit.client.0.smithi063.stdout:3/722: creat d13/d22/d2d/d3e/dce/fdf x:0 0 0 2024-03-23T11:47:40.020 INFO:tasks.workunit.client.0.smithi063.stdout:8/781: write d6/d1c/d27/f34 [3356046,48594] 0 2024-03-23T11:47:40.023 INFO:tasks.workunit.client.0.smithi063.stdout:3/723: mkdir d13/d22/d2d/d3f/de0 0 2024-03-23T11:47:40.024 INFO:tasks.workunit.client.1.smithi111.stdout:8/548: creat d5/d26/d2f/d8f/fb7 x:0 0 0 2024-03-23T11:47:40.024 INFO:tasks.workunit.client.1.smithi111.stdout:7/626: creat d14/d1b/d1e/d2c/d42/d5e/fb0 x:0 0 0 2024-03-23T11:47:40.026 INFO:tasks.workunit.client.1.smithi111.stdout:7/627: chown d14/d1b/d1e/d2c/c72 23 1 2024-03-23T11:47:40.029 INFO:tasks.workunit.client.1.smithi111.stdout:7/628: fsync d14/d1b/d1e/d2c/d42/d5e/d88/f94 0 2024-03-23T11:47:40.029 INFO:tasks.workunit.client.0.smithi063.stdout:0/565: dwrite d2/d14/f34 [0,4194304] 0 2024-03-23T11:47:40.031 INFO:tasks.workunit.client.1.smithi111.stdout:2/700: creat d2/d11/d17/d3e/d5d/dc2/f100 x:0 0 0 2024-03-23T11:47:40.032 INFO:tasks.workunit.client.1.smithi111.stdout:6/681: sync 2024-03-23T11:47:40.032 INFO:tasks.workunit.client.1.smithi111.stdout:0/620: sync 2024-03-23T11:47:40.033 INFO:tasks.workunit.client.1.smithi111.stdout:3/592: dread d0/d1a/d2b/f6c [0,4194304] 0 2024-03-23T11:47:40.034 INFO:tasks.workunit.client.1.smithi111.stdout:3/593: dread - d0/d1a/d2b/d38/dc2/fc6 zero size 2024-03-23T11:47:40.035 INFO:tasks.workunit.client.0.smithi063.stdout:8/782: creat d6/d1f/d54/d5c/d66/df1/f107 x:0 0 0 2024-03-23T11:47:40.038 INFO:tasks.workunit.client.0.smithi063.stdout:1/432: dwrite d3/d49/d56/d65/f7c [4194304,4194304] 0 2024-03-23T11:47:40.040 INFO:tasks.workunit.client.1.smithi111.stdout:9/679: write d0/d4/d1c/d36/f59 [2475777,71513] 0 2024-03-23T11:47:40.042 INFO:tasks.workunit.client.0.smithi063.stdout:0/566: mknod d2/d4/cce 0 2024-03-23T11:47:40.047 INFO:tasks.workunit.client.1.smithi111.stdout:4/673: dread d0/dc/d4c/d71/f84 [0,4194304] 0 2024-03-23T11:47:40.049 INFO:tasks.workunit.client.0.smithi063.stdout:1/433: truncate d3/d49/d56/f7f 1025110 0 2024-03-23T11:47:40.050 INFO:tasks.workunit.client.0.smithi063.stdout:0/567: mkdir d2/d12/d35/dcf 0 2024-03-23T11:47:40.055 INFO:tasks.workunit.client.0.smithi063.stdout:3/724: dwrite d13/f4d [4194304,4194304] 0 2024-03-23T11:47:40.055 INFO:tasks.workunit.client.0.smithi063.stdout:1/434: stat d3/df/l14 0 2024-03-23T11:47:40.055 INFO:tasks.workunit.client.1.smithi111.stdout:4/674: write d0/dc/d4c/f39 [1262959,106680] 0 2024-03-23T11:47:40.057 INFO:tasks.workunit.client.0.smithi063.stdout:1/435: dread - d3/d3f/f77 zero size 2024-03-23T11:47:40.061 INFO:tasks.workunit.client.0.smithi063.stdout:1/436: dread d3/f39 [0,4194304] 0 2024-03-23T11:47:40.063 INFO:tasks.workunit.client.1.smithi111.stdout:3/594: dwrite d0/d1a/d2b/d38/f51 [0,4194304] 0 2024-03-23T11:47:40.068 INFO:tasks.workunit.client.0.smithi063.stdout:1/437: rmdir d3 39 2024-03-23T11:47:40.076 INFO:tasks.workunit.client.1.smithi111.stdout:1/627: dread d0/d5/f16 [4194304,4194304] 0 2024-03-23T11:47:40.078 INFO:tasks.workunit.client.0.smithi063.stdout:3/725: getdents d13/d22/d2d/d7d 0 2024-03-23T11:47:40.078 INFO:tasks.workunit.client.0.smithi063.stdout:0/568: rmdir d2/d12/d35 39 2024-03-23T11:47:40.079 INFO:tasks.workunit.client.0.smithi063.stdout:1/438: stat d3/d49 0 2024-03-23T11:47:40.079 INFO:tasks.workunit.client.1.smithi111.stdout:8/549: creat d5/d26/d2f/d62/d75/d8d/fb8 x:0 0 0 2024-03-23T11:47:40.082 INFO:tasks.workunit.client.0.smithi063.stdout:1/439: chown d3/d49/d56/f7f 51464 1 2024-03-23T11:47:40.082 INFO:tasks.workunit.client.0.smithi063.stdout:3/726: readlink d13/d22/d2d/l46 0 2024-03-23T11:47:40.083 INFO:tasks.workunit.client.1.smithi111.stdout:7/629: mkdir d14/d37/d43/db1 0 2024-03-23T11:47:40.085 INFO:tasks.workunit.client.1.smithi111.stdout:7/630: chown d14/d1b/l24 582 1 2024-03-23T11:47:40.086 INFO:tasks.workunit.client.1.smithi111.stdout:2/701: creat d2/d7/da/f101 x:0 0 0 2024-03-23T11:47:40.088 INFO:tasks.workunit.client.1.smithi111.stdout:2/702: stat d2/d7/da/d19/f34 0 2024-03-23T11:47:40.090 INFO:tasks.workunit.client.0.smithi063.stdout:0/569: truncate d2/d14/d36/d77/d94/da6/f98 613615 0 2024-03-23T11:47:40.092 INFO:tasks.workunit.client.1.smithi111.stdout:7/631: dread d14/d1b/d1e/d2c/d2e/d73/dae/f63 [0,4194304] 0 2024-03-23T11:47:40.095 INFO:tasks.workunit.client.0.smithi063.stdout:0/570: creat d2/d12/d32/d6d/fd0 x:0 0 0 2024-03-23T11:47:40.097 INFO:tasks.workunit.client.1.smithi111.stdout:0/621: rename d5/db/d1f/d47/d87/fa4 to d5/db/d1f/d19/dbd/d64/dc7/fcf 0 2024-03-23T11:47:40.102 INFO:tasks.workunit.client.0.smithi063.stdout:1/440: dread d3/df/d2f/f64 [0,4194304] 0 2024-03-23T11:47:40.105 INFO:tasks.workunit.client.1.smithi111.stdout:9/680: write d0/d4/d1c/d2d/d3b/d42/d4f/d95/fb9 [1019843,96792] 0 2024-03-23T11:47:40.105 INFO:tasks.workunit.client.0.smithi063.stdout:0/571: mknod d2/d14/d36/d77/d94/da6/d25/d93/cd1 0 2024-03-23T11:47:40.107 INFO:tasks.workunit.client.1.smithi111.stdout:4/675: stat d0/dc/d4c/d83/c87 0 2024-03-23T11:47:40.108 INFO:tasks.workunit.client.0.smithi063.stdout:1/441: creat d3/d1c/d4b/f8a x:0 0 0 2024-03-23T11:47:40.109 INFO:tasks.workunit.client.0.smithi063.stdout:3/727: dwrite d13/d22/d2d/f45 [0,4194304] 0 2024-03-23T11:47:40.110 INFO:tasks.workunit.client.1.smithi111.stdout:1/628: symlink d0/d5/dd/d98/ld7 0 2024-03-23T11:47:40.111 INFO:tasks.workunit.client.1.smithi111.stdout:8/550: rmdir d5/d26/d2f 39 2024-03-23T11:47:40.111 INFO:tasks.workunit.client.0.smithi063.stdout:1/442: fsync d3/d49/d56/d65/f7c 0 2024-03-23T11:47:40.112 INFO:tasks.workunit.client.1.smithi111.stdout:8/551: chown d5/d32 842339249 1 2024-03-23T11:47:40.127 INFO:tasks.workunit.client.1.smithi111.stdout:3/595: dread d0/d1a/d2b/f33 [0,4194304] 0 2024-03-23T11:47:40.132 INFO:tasks.workunit.client.0.smithi063.stdout:0/572: write d2/d12/d35/f62 [462595,42538] 0 2024-03-23T11:47:40.134 INFO:tasks.workunit.client.1.smithi111.stdout:7/632: mkdir d14/d37/d54/db2 0 2024-03-23T11:47:40.134 INFO:tasks.workunit.client.1.smithi111.stdout:7/633: readlink l5 0 2024-03-23T11:47:40.134 INFO:tasks.workunit.client.0.smithi063.stdout:0/573: readlink d2/d12/l22 0 2024-03-23T11:47:40.138 INFO:tasks.workunit.client.0.smithi063.stdout:0/574: chown d2/d4/l99 6505088 1 2024-03-23T11:47:40.139 INFO:tasks.workunit.client.1.smithi111.stdout:8/552: stat d5/d26/d2f/d62/db5/f87 0 2024-03-23T11:47:40.141 INFO:tasks.workunit.client.0.smithi063.stdout:0/575: fdatasync d2/d48/f7f 0 2024-03-23T11:47:40.141 INFO:tasks.workunit.client.1.smithi111.stdout:8/553: truncate d5/d1b/f2b 5693106 0 2024-03-23T11:47:40.142 INFO:tasks.workunit.client.1.smithi111.stdout:6/682: getdents d2/d1f 0 2024-03-23T11:47:40.143 INFO:tasks.workunit.client.0.smithi063.stdout:0/576: chown d2/c1e 703 1 2024-03-23T11:47:40.144 INFO:tasks.workunit.client.0.smithi063.stdout:3/728: dwrite d13/d22/d2d/d3f/f9e [0,4194304] 0 2024-03-23T11:47:40.146 INFO:tasks.workunit.client.1.smithi111.stdout:3/596: rename d0/d53/c62 to d0/d7/d19/d5e/cc7 0 2024-03-23T11:47:40.147 INFO:tasks.workunit.client.1.smithi111.stdout:9/681: dread d0/d4/d10/fbb [4194304,4194304] 0 2024-03-23T11:47:40.148 INFO:tasks.workunit.client.1.smithi111.stdout:3/597: chown d0/d1a/d2b/d38/d7f/faa 29659229 1 2024-03-23T11:47:40.150 INFO:tasks.workunit.client.0.smithi063.stdout:3/729: read d13/d22/d2d/fa7 [359044,32514] 0 2024-03-23T11:47:40.150 INFO:tasks.workunit.client.0.smithi063.stdout:0/577: creat d2/d14/d36/d77/d94/daf/fd2 x:0 0 0 2024-03-23T11:47:40.151 INFO:tasks.workunit.client.1.smithi111.stdout:6/683: dread d2/d1f/fcf [0,4194304] 0 2024-03-23T11:47:40.153 INFO:tasks.workunit.client.0.smithi063.stdout:0/578: dread - d2/d12/d32/d6d/fd0 zero size 2024-03-23T11:47:40.153 INFO:tasks.workunit.client.0.smithi063.stdout:3/730: fdatasync d13/d22/d2d/d3f/d67/fd1 0 2024-03-23T11:47:40.156 INFO:tasks.workunit.client.1.smithi111.stdout:1/629: creat d0/d6/fd8 x:0 0 0 2024-03-23T11:47:40.157 INFO:tasks.workunit.client.1.smithi111.stdout:2/703: getdents d2/d32 0 2024-03-23T11:47:40.158 INFO:tasks.workunit.client.0.smithi063.stdout:0/579: mkdir d2/d14/d36/d77/d94/da6/d25/d93/dd3 0 2024-03-23T11:47:40.158 INFO:tasks.workunit.client.1.smithi111.stdout:1/630: write d0/d5/dd/d1e/f21 [1548206,95952] 0 2024-03-23T11:47:40.160 INFO:tasks.workunit.client.0.smithi063.stdout:0/580: chown d2/d14/d36/d77/d94/da6/lc8 444979572 1 2024-03-23T11:47:40.161 INFO:tasks.workunit.client.1.smithi111.stdout:8/554: read d5/d26/d31/d5e/d4a/f8e [27137,79674] 0 2024-03-23T11:47:40.165 INFO:tasks.workunit.client.1.smithi111.stdout:9/682: symlink d0/d4/d1c/d36/d28/d69/le1 0 2024-03-23T11:47:40.167 INFO:tasks.workunit.client.1.smithi111.stdout:6/684: rename d2/d1f/d45/d65 to d2/d35/de3 0 2024-03-23T11:47:40.169 INFO:tasks.workunit.client.1.smithi111.stdout:5/689: dwrite d1/d3/d6/d8b/d59/f6a [0,4194304] 0 2024-03-23T11:47:40.171 INFO:tasks.workunit.client.1.smithi111.stdout:5/690: readlink d1/d3/d6/d7/l2f 0 2024-03-23T11:47:40.179 INFO:tasks.workunit.client.1.smithi111.stdout:8/555: rmdir d5/d26/d31/d5e/d4a 39 2024-03-23T11:47:40.179 INFO:tasks.workunit.client.1.smithi111.stdout:7/634: link d14/d1b/d1e/d2c/d2e/d73/l7a d14/d44/d4b/lb3 0 2024-03-23T11:47:40.180 INFO:tasks.workunit.client.1.smithi111.stdout:0/622: truncate d5/fc 1980185 0 2024-03-23T11:47:40.180 INFO:tasks.workunit.client.0.smithi063.stdout:3/731: dread f9 [4194304,4194304] 0 2024-03-23T11:47:40.182 INFO:tasks.workunit.client.1.smithi111.stdout:9/683: mknod d0/d31/d41/d71/d8f/ce2 0 2024-03-23T11:47:40.182 INFO:tasks.workunit.client.1.smithi111.stdout:6/685: mknod d2/d35/da7/ce4 0 2024-03-23T11:47:40.184 INFO:tasks.workunit.client.0.smithi063.stdout:0/581: dwrite d2/d12/d35/f81 [0,4194304] 0 2024-03-23T11:47:40.186 INFO:tasks.workunit.client.1.smithi111.stdout:9/684: readlink d0/d4/d10/d80/l85 0 2024-03-23T11:47:40.187 INFO:tasks.workunit.client.1.smithi111.stdout:2/704: dread d2/d11/f1b [0,4194304] 0 2024-03-23T11:47:40.187 INFO:tasks.workunit.client.1.smithi111.stdout:4/676: dwrite d0/dc/f14 [4194304,4194304] 0 2024-03-23T11:47:40.189 INFO:tasks.workunit.client.0.smithi063.stdout:3/732: mkdir d13/d22/d2d/d3f/d58/d6e/d8d/dda/de1 0 2024-03-23T11:47:40.190 INFO:tasks.workunit.client.1.smithi111.stdout:8/556: readlink l4 0 2024-03-23T11:47:40.191 INFO:tasks.workunit.client.0.smithi063.stdout:0/582: rename d2/d4/l99 to d2/d14/d36/d77/d94/da6/d25/ld4 0 2024-03-23T11:47:40.193 INFO:tasks.workunit.client.0.smithi063.stdout:0/583: dread - d2/d6e/d8c/fae zero size 2024-03-23T11:47:40.194 INFO:tasks.workunit.client.1.smithi111.stdout:2/705: write d2/d7/da/dbd/fd7 [80919,92145] 0 2024-03-23T11:47:40.196 INFO:tasks.workunit.client.0.smithi063.stdout:0/584: readlink d2/d4/d63/l7b 0 2024-03-23T11:47:40.201 INFO:tasks.workunit.client.1.smithi111.stdout:7/635: creat d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb4 x:0 0 0 2024-03-23T11:47:40.201 INFO:tasks.workunit.client.1.smithi111.stdout:6/686: fdatasync d2/d1f/f7a 0 2024-03-23T11:47:40.202 INFO:tasks.workunit.client.0.smithi063.stdout:0/585: creat d2/d14/d36/d77/d94/fd5 x:0 0 0 2024-03-23T11:47:40.203 INFO:tasks.workunit.client.1.smithi111.stdout:5/691: mknod d1/d3/d6/d30/db2/ce2 0 2024-03-23T11:47:40.204 INFO:tasks.workunit.client.1.smithi111.stdout:0/623: dwrite d5/db/d1f/d47/d59/f7d [4194304,4194304] 0 2024-03-23T11:47:40.205 INFO:tasks.workunit.client.1.smithi111.stdout:7/636: fsync d14/d1b/d1e/d2c/d2e/d73/dae/f62 0 2024-03-23T11:47:40.206 INFO:tasks.workunit.client.0.smithi063.stdout:0/586: fdatasync d2/f56 0 2024-03-23T11:47:40.209 INFO:tasks.workunit.client.0.smithi063.stdout:0/587: write d2/d4/f9 [1985966,127028] 0 2024-03-23T11:47:40.213 INFO:tasks.workunit.client.1.smithi111.stdout:2/706: mknod d2/d11/d17/d3e/d6b/d9d/d52/dba/dd3/c102 0 2024-03-23T11:47:40.215 INFO:tasks.workunit.client.1.smithi111.stdout:1/631: dwrite d0/d5/dd/dc6/fcf [0,4194304] 0 2024-03-23T11:47:40.217 INFO:tasks.workunit.client.1.smithi111.stdout:5/692: creat d1/d3/d6/d7/d7b/fe3 x:0 0 0 2024-03-23T11:47:40.218 INFO:tasks.workunit.client.0.smithi063.stdout:0/588: symlink d2/d6e/d8c/ld6 0 2024-03-23T11:47:40.220 INFO:tasks.workunit.client.0.smithi063.stdout:3/733: dread d13/d22/d2d/d7d/fc3 [0,4194304] 0 2024-03-23T11:47:40.220 INFO:tasks.workunit.client.1.smithi111.stdout:5/693: write d1/d3/d6/d7/d7b/fa4 [705560,127902] 0 2024-03-23T11:47:40.221 INFO:tasks.workunit.client.0.smithi063.stdout:0/589: write d2/d14/f23 [2609108,86073] 0 2024-03-23T11:47:40.224 INFO:tasks.workunit.client.1.smithi111.stdout:0/624: mkdir d5/db/d1f/d19/dbd/d8b/dd0 0 2024-03-23T11:47:40.225 INFO:tasks.workunit.client.1.smithi111.stdout:4/677: rename d0/dc/d1f/d5e/lcd to d0/dc/d4c/d2a/ld5 0 2024-03-23T11:47:40.226 INFO:tasks.workunit.client.1.smithi111.stdout:8/557: truncate d5/d26/d31/d5e/d4a/f8e 5980 0 2024-03-23T11:47:40.226 INFO:tasks.workunit.client.1.smithi111.stdout:0/625: fdatasync d5/f85 0 2024-03-23T11:47:40.227 INFO:tasks.workunit.client.1.smithi111.stdout:2/707: truncate d2/d11/d17/d3e/d5d/fa4 129920 0 2024-03-23T11:47:40.227 INFO:tasks.workunit.client.1.smithi111.stdout:6/687: chown d2/d1a/d1c/d2c/d51 221262013 1 2024-03-23T11:47:40.227 INFO:tasks.workunit.client.0.smithi063.stdout:0/590: chown d2/d14/d36/d77/d94/da6/d25/d93/dc9 104 1 2024-03-23T11:47:40.228 INFO:tasks.workunit.client.0.smithi063.stdout:3/734: unlink d13/d22/d2d/c69 0 2024-03-23T11:47:40.228 INFO:tasks.workunit.client.0.smithi063.stdout:0/591: stat d2/d6e 0 2024-03-23T11:47:40.229 INFO:tasks.workunit.client.1.smithi111.stdout:7/637: dread d14/d1b/d1e/d2c/d42/d5e/d88/f94 [0,4194304] 0 2024-03-23T11:47:40.229 INFO:tasks.workunit.client.1.smithi111.stdout:1/632: truncate d0/d6/fc 1723230 0 2024-03-23T11:47:40.230 INFO:tasks.workunit.client.1.smithi111.stdout:0/626: dread - d5/f85 zero size 2024-03-23T11:47:40.230 INFO:tasks.workunit.client.0.smithi063.stdout:3/735: chown d13/d15/c37 41920 1 2024-03-23T11:47:40.232 INFO:tasks.workunit.client.1.smithi111.stdout:5/694: write d1/d3/d6/f57 [2237828,73718] 0 2024-03-23T11:47:40.233 INFO:tasks.workunit.client.1.smithi111.stdout:9/685: creat d0/d4/d1c/fe3 x:0 0 0 2024-03-23T11:47:40.234 INFO:tasks.workunit.client.0.smithi063.stdout:0/592: read d2/d14/d36/d77/d94/da6/d1f/f2d [84726,127347] 0 2024-03-23T11:47:40.235 INFO:tasks.workunit.client.1.smithi111.stdout:5/695: chown d1/d20/d5c/dc5/dd3 14902302 1 2024-03-23T11:47:40.238 INFO:tasks.workunit.client.0.smithi063.stdout:3/736: read d13/d22/d2d/d3e/dce/f62 [2591057,33425] 0 2024-03-23T11:47:40.238 INFO:tasks.workunit.client.1.smithi111.stdout:4/678: mknod d0/d17/d86/cd6 0 2024-03-23T11:47:40.239 INFO:tasks.workunit.client.0.smithi063.stdout:0/593: symlink d2/d12/d35/ld7 0 2024-03-23T11:47:40.241 INFO:tasks.workunit.client.1.smithi111.stdout:2/708: creat d2/d7/da/f103 x:0 0 0 2024-03-23T11:47:40.242 INFO:tasks.workunit.client.0.smithi063.stdout:3/737: fdatasync d13/d22/f3d 0 2024-03-23T11:47:40.246 INFO:tasks.workunit.client.0.smithi063.stdout:0/594: rename d2/d12/d35/fa8 to d2/d4/fd8 0 2024-03-23T11:47:40.246 INFO:tasks.workunit.client.1.smithi111.stdout:8/558: mkdir d5/d26/d2f/d62/d75/d4b/d51/da4/d97/db9 0 2024-03-23T11:47:40.248 INFO:tasks.workunit.client.0.smithi063.stdout:3/738: write d13/d22/d2d/d3f/d5b/f65 [4816769,13288] 0 2024-03-23T11:47:40.249 INFO:tasks.workunit.client.1.smithi111.stdout:6/688: symlink d2/d1a/d1c/le5 0 2024-03-23T11:47:40.251 INFO:tasks.workunit.client.0.smithi063.stdout:0/595: write d2/d14/f15 [746370,117348] 0 2024-03-23T11:47:40.251 INFO:tasks.workunit.client.1.smithi111.stdout:5/696: creat d1/d20/dce/fe4 x:0 0 0 2024-03-23T11:47:40.254 INFO:tasks.workunit.client.0.smithi063.stdout:3/739: write d13/d22/d2d/d3f/d58/d97/fae [73293,95339] 0 2024-03-23T11:47:40.257 INFO:tasks.workunit.client.1.smithi111.stdout:7/638: unlink d14/c16 0 2024-03-23T11:47:40.259 INFO:tasks.workunit.client.1.smithi111.stdout:4/679: creat d0/dc/d1f/db0/fd7 x:0 0 0 2024-03-23T11:47:40.261 INFO:tasks.workunit.client.1.smithi111.stdout:2/709: unlink d2/d7/da/dbd/le8 0 2024-03-23T11:47:40.261 INFO:tasks.workunit.client.0.smithi063.stdout:0/596: fsync d2/d12/f7e 0 2024-03-23T11:47:40.262 INFO:tasks.workunit.client.1.smithi111.stdout:0/627: readlink d5/db/d1f/d47/d6f/lbf 0 2024-03-23T11:47:40.263 INFO:tasks.workunit.client.1.smithi111.stdout:2/710: truncate d2/f9f 714820 0 2024-03-23T11:47:40.263 INFO:tasks.workunit.client.0.smithi063.stdout:3/740: rename d13/d22/d2d/d3f/d5b/cb8 to d13/d22/d2d/d7d/ce2 0 2024-03-23T11:47:40.263 INFO:tasks.workunit.client.1.smithi111.stdout:9/686: dread d0/d4/fe [0,4194304] 0 2024-03-23T11:47:40.265 INFO:tasks.workunit.client.1.smithi111.stdout:1/633: truncate d0/d2b/d37/f74 1529178 0 2024-03-23T11:47:40.270 INFO:tasks.workunit.client.0.smithi063.stdout:0/597: mknod d2/d48/da5/da7/cd9 0 2024-03-23T11:47:40.272 INFO:tasks.workunit.client.0.smithi063.stdout:3/741: mknod d13/d22/d2d/d3f/d58/ce3 0 2024-03-23T11:47:40.273 INFO:tasks.workunit.client.1.smithi111.stdout:5/697: unlink d1/d20/c89 0 2024-03-23T11:47:40.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.289+0000 7f0d94c00700 1 -- 172.21.15.63:0/1451320441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d900708f0 msgr2=0x7f0d90070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:40.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.289+0000 7f0d94c00700 1 --2- 172.21.15.63:0/1451320441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d900708f0 0x7f0d90070d60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c004880 tx=0x7f0d7c009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:40.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.289+0000 7f0d94c00700 1 -- 172.21.15.63:0/1451320441 shutdown_connections 2024-03-23T11:47:40.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.289+0000 7f0d94c00700 1 --2- 172.21.15.63:0/1451320441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d900708f0 0x7f0d90070d60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.289+0000 7f0d94c00700 1 --2- 172.21.15.63:0/1451320441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90071ea0 0x7f0d900703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.289+0000 7f0d94c00700 1 -- 172.21.15.63:0/1451320441 >> 172.21.15.63:0/1451320441 conn(0x7f0d900f8560 msgr2=0x7f0d900fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:40.275 INFO:tasks.workunit.client.0.smithi063.stdout:3/742: stat d13/d22/d2d/d3f/d58/d86 0 2024-03-23T11:47:40.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.290+0000 7f0d94c00700 1 -- 172.21.15.63:0/1451320441 shutdown_connections 2024-03-23T11:47:40.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.290+0000 7f0d94c00700 1 -- 172.21.15.63:0/1451320441 wait complete. 2024-03-23T11:47:40.276 INFO:tasks.workunit.client.1.smithi111.stdout:4/680: rmdir d0/dc/d78 39 2024-03-23T11:47:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.291+0000 7f0d94c00700 1 Processor -- start 2024-03-23T11:47:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.292+0000 7f0d94c00700 1 -- start start 2024-03-23T11:47:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.292+0000 7f0d94c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d900708f0 0x7f0d9006e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.292+0000 7f0d94c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90071ea0 0x7f0d9006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.292+0000 7f0d94c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d90070080 con 0x7f0d900708f0 2024-03-23T11:47:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.292+0000 7f0d94c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9006eec0 con 0x7f0d90071ea0 2024-03-23T11:47:40.278 INFO:tasks.workunit.client.1.smithi111.stdout:7/639: symlink d14/d1b/d1e/d2c/d42/d5e/d9e/lb5 0 2024-03-23T11:47:40.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.293+0000 7f0d8de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d900708f0 0x7f0d9006e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:40.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.293+0000 7f0d8d400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90071ea0 0x7f0d9006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:40.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.293+0000 7f0d8d400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90071ea0 0x7f0d9006e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:37372/0 (socket says 172.21.15.63:37372) 2024-03-23T11:47:40.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.293+0000 7f0d8d400700 1 -- 172.21.15.63:0/2360215244 learned_addr learned my addr 172.21.15.63:0/2360215244 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:40.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.294+0000 7f0d8d400700 1 -- 172.21.15.63:0/2360215244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d900708f0 msgr2=0x7f0d9006e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.294+0000 7f0d8d400700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d900708f0 0x7f0d9006e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.294+0000 7f0d8d400700 1 -- 172.21.15.63:0/2360215244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c005040 con 0x7f0d90071ea0 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.294+0000 7f0d8de00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d900708f0 0x7f0d9006e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.294+0000 7f0d8d400700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90071ea0 0x7f0d9006e980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c013420 tx=0x7f0d7c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.295+0000 7f0d86000700 1 -- 172.21.15.63:0/2360215244 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c00e030 con 0x7f0d90071ea0 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.295+0000 7f0d86000700 1 -- 172.21.15.63:0/2360215244 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d7c013c30 con 0x7f0d90071ea0 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.295+0000 7f0d86000700 1 -- 172.21.15.63:0/2360215244 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c01ce90 con 0x7f0d90071ea0 2024-03-23T11:47:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.295+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9006f110 con 0x7f0d90071ea0 2024-03-23T11:47:40.281 INFO:tasks.workunit.client.1.smithi111.stdout:6/689: dread d2/d1f/d45/d49/d8d/fba [0,4194304] 0 2024-03-23T11:47:40.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.295+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d90122430 con 0x7f0d90071ea0 2024-03-23T11:47:40.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.297+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9005f5f0 con 0x7f0d90071ea0 2024-03-23T11:47:40.283 INFO:tasks.workunit.client.1.smithi111.stdout:6/690: readlink d2/d1a/d1c/d2c/d51/lc8 0 2024-03-23T11:47:40.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.299+0000 7f0d86000700 1 -- 172.21.15.63:0/2360215244 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0d7c01c400 con 0x7f0d90071ea0 2024-03-23T11:47:40.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.299+0000 7f0d86000700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f0d7406ad60 0x7f0d7406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:40.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.300+0000 7f0d86000700 1 -- 172.21.15.63:0/2360215244 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f0d7c08ab90 con 0x7f0d90071ea0 2024-03-23T11:47:40.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.300+0000 7f0d8de00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f0d7406ad60 0x7f0d7406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:40.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.301+0000 7f0d8de00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f0d7406ad60 0x7f0d7406d210 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d9006fd00 tx=0x7f0d8000b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:40.287 INFO:tasks.workunit.client.1.smithi111.stdout:2/711: dread d2/d7/da/d19/f49 [0,4194304] 0 2024-03-23T11:47:40.289 INFO:tasks.workunit.client.1.smithi111.stdout:8/559: write d5/f2d [4410000,48387] 0 2024-03-23T11:47:40.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.306+0000 7f0d86000700 1 -- 172.21.15.63:0/2360215244 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0d7c055400 con 0x7f0d90071ea0 2024-03-23T11:47:40.291 INFO:tasks.workunit.client.0.smithi063.stdout:0/598: write d2/d14/d36/d77/f80 [983227,46378] 0 2024-03-23T11:47:40.291 INFO:tasks.workunit.client.1.smithi111.stdout:2/712: write d2/d11/fe5 [183430,42804] 0 2024-03-23T11:47:40.292 INFO:tasks.workunit.client.1.smithi111.stdout:5/698: creat d1/d20/d41/fe5 x:0 0 0 2024-03-23T11:47:40.293 INFO:tasks.workunit.client.0.smithi063.stdout:0/599: readlink d2/d14/d36/l75 0 2024-03-23T11:47:40.296 INFO:tasks.workunit.client.1.smithi111.stdout:1/634: stat d0/d6/d48/d4d/lc8 0 2024-03-23T11:47:40.296 INFO:tasks.workunit.client.1.smithi111.stdout:2/713: write d2/d7/da/f101 [588214,15489] 0 2024-03-23T11:47:40.297 INFO:tasks.workunit.client.1.smithi111.stdout:9/687: creat d0/d31/d41/d9d/dd4/de0/fe4 x:0 0 0 2024-03-23T11:47:40.298 INFO:tasks.workunit.client.1.smithi111.stdout:4/681: dwrite d0/dc/d4c/f82 [0,4194304] 0 2024-03-23T11:47:40.300 INFO:tasks.workunit.client.1.smithi111.stdout:3/598: sync 2024-03-23T11:47:40.301 INFO:tasks.workunit.client.0.smithi063.stdout:0/600: fsync d2/d14/d36/d77/d94/da6/f33 0 2024-03-23T11:47:40.302 INFO:tasks.workunit.client.1.smithi111.stdout:3/599: chown d0/d1a/d1d 12 1 2024-03-23T11:47:40.304 INFO:tasks.workunit.client.1.smithi111.stdout:5/699: fdatasync d1/d20/f37 0 2024-03-23T11:47:40.305 INFO:tasks.workunit.client.1.smithi111.stdout:0/628: getdents d5/db/d1f 0 2024-03-23T11:47:40.306 INFO:tasks.workunit.client.0.smithi063.stdout:0/601: fsync d2/d14/d36/d77/d94/da6/f98 0 2024-03-23T11:47:40.306 INFO:tasks.workunit.client.1.smithi111.stdout:9/688: dread d0/d31/d35/f9c [0,4194304] 0 2024-03-23T11:47:40.307 INFO:tasks.workunit.client.1.smithi111.stdout:5/700: stat d1/d3/fcd 0 2024-03-23T11:47:40.307 INFO:tasks.workunit.client.1.smithi111.stdout:2/714: dread d2/d11/d17/d3e/d6b/d9d/d52/f73 [0,4194304] 0 2024-03-23T11:47:40.309 INFO:tasks.workunit.client.1.smithi111.stdout:0/629: dread - d5/db/d1f/d19/dbd/f94 zero size 2024-03-23T11:47:40.309 INFO:tasks.workunit.client.1.smithi111.stdout:9/689: write d0/d4/f15 [3926696,35819] 0 2024-03-23T11:47:40.310 INFO:tasks.workunit.client.1.smithi111.stdout:5/701: write d1/d20/f90 [2514789,99997] 0 2024-03-23T11:47:40.310 INFO:tasks.workunit.client.1.smithi111.stdout:4/682: mknod d0/d95/d9b/cd8 0 2024-03-23T11:47:40.311 INFO:tasks.workunit.client.0.smithi063.stdout:3/743: dwrite d13/d22/d2d/d3f/d58/d6e/f7a [0,4194304] 0 2024-03-23T11:47:40.311 INFO:tasks.workunit.client.1.smithi111.stdout:1/635: mkdir d0/d5/dd/d1e/d54/d87/dd9 0 2024-03-23T11:47:40.311 INFO:tasks.workunit.client.0.smithi063.stdout:0/602: creat d2/d14/d36/fda x:0 0 0 2024-03-23T11:47:40.312 INFO:tasks.workunit.client.1.smithi111.stdout:6/691: dread d2/d1f/d45/d49/fa1 [0,4194304] 0 2024-03-23T11:47:40.318 INFO:tasks.workunit.client.0.smithi063.stdout:0/603: read d2/d14/d36/d77/d94/da6/d25/d93/fbf [1279041,113793] 0 2024-03-23T11:47:40.321 INFO:tasks.workunit.client.0.smithi063.stdout:3/744: dread d13/f9c [0,4194304] 0 2024-03-23T11:47:40.324 INFO:tasks.workunit.client.0.smithi063.stdout:0/604: mkdir d2/d4/d63/ddb 0 2024-03-23T11:47:40.325 INFO:tasks.workunit.client.0.smithi063.stdout:3/745: chown d13/d22/d2d/d3f/d67 47882 1 2024-03-23T11:47:40.329 INFO:tasks.workunit.client.1.smithi111.stdout:5/702: dwrite d1/d3/d6/d8b/d59/d74/f21 [4194304,4194304] 0 2024-03-23T11:47:40.329 INFO:tasks.workunit.client.0.smithi063.stdout:3/746: write d13/d15/fca [2389564,28535] 0 2024-03-23T11:47:40.330 INFO:tasks.workunit.client.0.smithi063.stdout:0/605: symlink d2/d14/d36/d77/d94/da6/d1f/d4f/ldc 0 2024-03-23T11:47:40.334 INFO:tasks.workunit.client.1.smithi111.stdout:0/630: rmdir d5 39 2024-03-23T11:47:40.338 INFO:tasks.workunit.client.1.smithi111.stdout:1/636: mkdir d0/dda 0 2024-03-23T11:47:40.339 INFO:tasks.workunit.client.0.smithi063.stdout:0/606: mkdir d2/d14/d36/d77/d94/da6/d25/d93/dc9/ddd 0 2024-03-23T11:47:40.341 INFO:tasks.workunit.client.1.smithi111.stdout:2/715: unlink d2/d11/d17/d3e/d6b/f87 0 2024-03-23T11:47:40.344 INFO:tasks.workunit.client.1.smithi111.stdout:5/703: dwrite d1/d3/d6/d8b/d59/d74/fa5 [0,4194304] 0 2024-03-23T11:47:40.344 INFO:tasks.workunit.client.0.smithi063.stdout:0/607: write d2/d14/d36/d77/d94/da6/d28/d46/d51/f65 [4143079,46494] 0 2024-03-23T11:47:40.349 INFO:tasks.workunit.client.1.smithi111.stdout:9/690: stat d0/d4/d1c/d36/l8 0 2024-03-23T11:47:40.350 INFO:tasks.workunit.client.1.smithi111.stdout:9/691: readlink d0/d31/d35/l6d 0 2024-03-23T11:47:40.350 INFO:tasks.workunit.client.1.smithi111.stdout:6/692: rmdir d2/d1a 39 2024-03-23T11:47:40.352 INFO:tasks.workunit.client.1.smithi111.stdout:6/693: read - d2/d1f/d45/d86/fb2 zero size 2024-03-23T11:47:40.356 INFO:tasks.workunit.client.1.smithi111.stdout:8/560: truncate d5/d26/d31/d5e/d4a/f8e 659822 0 2024-03-23T11:47:40.358 INFO:tasks.workunit.client.1.smithi111.stdout:3/600: mkdir d0/d7/d19/d91/da3/dc8 0 2024-03-23T11:47:40.359 INFO:tasks.workunit.client.0.smithi063.stdout:0/608: link d2/d14/d36/d77/d94/da6/d28/d46/d51/l96 d2/d6e/d8c/lde 0 2024-03-23T11:47:40.361 INFO:tasks.workunit.client.1.smithi111.stdout:7/640: link d14/d1b/d22/c6d d14/d1b/d1e/d2c/d2e/cb6 0 2024-03-23T11:47:40.364 INFO:tasks.workunit.client.0.smithi063.stdout:3/747: dwrite d13/d22/d2d/d3f/d67/fd1 [0,4194304] 0 2024-03-23T11:47:40.367 INFO:tasks.workunit.client.1.smithi111.stdout:1/637: mkdir d0/d6/d48/d4d/dab/ddb 0 2024-03-23T11:47:40.369 INFO:tasks.workunit.client.1.smithi111.stdout:4/683: mknod d0/dc/d78/cd9 0 2024-03-23T11:47:40.370 INFO:tasks.workunit.client.1.smithi111.stdout:1/638: dread - d0/d5/dd/d1e/d54/fc2 zero size 2024-03-23T11:47:40.370 INFO:tasks.workunit.client.1.smithi111.stdout:8/561: dwrite d5/d1b/f2b [0,4194304] 0 2024-03-23T11:47:40.370 INFO:tasks.workunit.client.0.smithi063.stdout:0/609: creat d2/d14/d36/d77/d94/da6/d28/fdf x:0 0 0 2024-03-23T11:47:40.370 INFO:tasks.workunit.client.1.smithi111.stdout:9/692: rmdir d0/d31/d41/d44 39 2024-03-23T11:47:40.373 INFO:tasks.workunit.client.1.smithi111.stdout:3/601: symlink d0/d1a/d2d/lc9 0 2024-03-23T11:47:40.376 INFO:tasks.workunit.client.0.smithi063.stdout:0/610: readlink d2/d14/d36/d77/d94/da6/d1f/d4f/ldc 0 2024-03-23T11:47:40.376 INFO:tasks.workunit.client.1.smithi111.stdout:0/631: symlink d5/db/d4d/ld1 0 2024-03-23T11:47:40.378 INFO:tasks.workunit.client.1.smithi111.stdout:9/693: dread - d0/d4/d10/d91/fda zero size 2024-03-23T11:47:40.380 INFO:tasks.workunit.client.1.smithi111.stdout:4/684: write d0/d7/da/f46 [1399287,73088] 0 2024-03-23T11:47:40.382 INFO:tasks.workunit.client.0.smithi063.stdout:0/611: mknod d2/d4/ce0 0 2024-03-23T11:47:40.383 INFO:tasks.workunit.client.1.smithi111.stdout:7/641: dwrite d14/d1b/d1e/d2c/d42/d5e/fb0 [0,4194304] 0 2024-03-23T11:47:40.383 INFO:tasks.workunit.client.1.smithi111.stdout:1/639: dread d0/d6/f19 [0,4194304] 0 2024-03-23T11:47:40.386 INFO:tasks.workunit.client.0.smithi063.stdout:0/612: write d2/d14/d36/d77/d94/da6/d1f/fba [646274,81042] 0 2024-03-23T11:47:40.388 INFO:tasks.workunit.client.1.smithi111.stdout:2/716: rename d2/de/l1a to d2/d11/d17/d1f/l104 0 2024-03-23T11:47:40.393 INFO:tasks.workunit.client.1.smithi111.stdout:7/642: creat d14/d37/d54/fb7 x:0 0 0 2024-03-23T11:47:40.395 INFO:tasks.workunit.client.1.smithi111.stdout:5/704: getdents d1/d3/d6/d7/d63/d2c/d43/d95 0 2024-03-23T11:47:40.395 INFO:tasks.workunit.client.1.smithi111.stdout:0/632: mknod d5/dc3/cd2 0 2024-03-23T11:47:40.397 INFO:tasks.workunit.client.1.smithi111.stdout:9/694: rename d0/d4/d6a/fd0 to d0/fe5 0 2024-03-23T11:47:40.397 INFO:tasks.workunit.client.1.smithi111.stdout:2/717: mknod d2/de/d12/c105 0 2024-03-23T11:47:40.398 INFO:tasks.workunit.client.1.smithi111.stdout:0/633: write d5/db/d1f/d48/fb0 [76852,19544] 0 2024-03-23T11:47:40.398 INFO:tasks.workunit.client.1.smithi111.stdout:9/695: chown d0/d4 488999 1 2024-03-23T11:47:40.398 INFO:tasks.workunit.client.1.smithi111.stdout:7/643: rmdir d14/d44/d70 39 2024-03-23T11:47:40.401 INFO:tasks.workunit.client.1.smithi111.stdout:1/640: link d0/fe d0/d5/dd/dc6/fdc 0 2024-03-23T11:47:40.401 INFO:tasks.workunit.client.1.smithi111.stdout:7/644: chown d14/d1b/fa3 8291437 1 2024-03-23T11:47:40.402 INFO:tasks.workunit.client.1.smithi111.stdout:0/634: write d5/db/d1f/d48/f66 [177563,45748] 0 2024-03-23T11:47:40.402 INFO:tasks.workunit.client.0.smithi063.stdout:0/613: dread d2/d48/fcb [0,4194304] 0 2024-03-23T11:47:40.403 INFO:tasks.workunit.client.1.smithi111.stdout:2/718: truncate d2/de/d6c/dd1/fe6 490719 0 2024-03-23T11:47:40.405 INFO:tasks.workunit.client.1.smithi111.stdout:5/705: mkdir d1/d3/d6/d30/de6 0 2024-03-23T11:47:40.405 INFO:tasks.workunit.client.1.smithi111.stdout:2/719: stat d2/d11/d17/d3e/d5d/dc2/f100 0 2024-03-23T11:47:40.406 INFO:tasks.workunit.client.0.smithi063.stdout:0/614: read d2/d14/f79 [257097,116011] 0 2024-03-23T11:47:40.411 INFO:tasks.workunit.client.0.smithi063.stdout:3/748: dwrite d13/d15/f33 [0,4194304] 0 2024-03-23T11:47:40.412 INFO:tasks.workunit.client.1.smithi111.stdout:6/694: getdents d2/d1a/d1c 0 2024-03-23T11:47:40.412 INFO:tasks.workunit.client.1.smithi111.stdout:8/562: getdents d5/d26/d2f/d62/db5 0 2024-03-23T11:47:40.415 INFO:tasks.workunit.client.1.smithi111.stdout:9/696: mkdir d0/d4/d1c/d36/d28/de6 0 2024-03-23T11:47:40.416 INFO:tasks.workunit.client.1.smithi111.stdout:7/645: creat d14/d1b/d1e/d2c/d42/d5e/d9e/fb8 x:0 0 0 2024-03-23T11:47:40.418 INFO:tasks.workunit.client.1.smithi111.stdout:0/635: creat d5/db/d1f/d47/d59/db6/fd3 x:0 0 0 2024-03-23T11:47:40.419 INFO:tasks.workunit.client.1.smithi111.stdout:0/636: chown d5/db/f15 101053 1 2024-03-23T11:47:40.420 INFO:tasks.workunit.client.1.smithi111.stdout:0/637: fdatasync d5/db/d1f/f28 0 2024-03-23T11:47:40.426 INFO:tasks.workunit.client.0.smithi063.stdout:0/615: symlink d2/d14/d36/le1 0 2024-03-23T11:47:40.426 INFO:tasks.workunit.client.1.smithi111.stdout:5/706: creat d1/d3/d6/d7/d7b/fe7 x:0 0 0 2024-03-23T11:47:40.429 INFO:tasks.workunit.client.1.smithi111.stdout:3/602: getdents d0/d7/d19/d91 0 2024-03-23T11:47:40.430 INFO:tasks.workunit.client.1.smithi111.stdout:6/695: mknod d2/d1a/d1c/d2c/ce6 0 2024-03-23T11:47:40.431 INFO:tasks.workunit.client.1.smithi111.stdout:3/603: fsync d0/d72/f86 0 2024-03-23T11:47:40.433 INFO:tasks.workunit.client.1.smithi111.stdout:8/563: unlink d5/d26/d2f/d62/d75/d8d/f94 0 2024-03-23T11:47:40.438 INFO:tasks.workunit.client.1.smithi111.stdout:7/646: creat d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb9 x:0 0 0 2024-03-23T11:47:40.441 INFO:tasks.workunit.client.1.smithi111.stdout:2/720: dread d2/d11/f13 [0,4194304] 0 2024-03-23T11:47:40.442 INFO:tasks.workunit.client.1.smithi111.stdout:7/647: read - d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb9 zero size 2024-03-23T11:47:40.442 INFO:tasks.workunit.client.1.smithi111.stdout:3/604: rename d0/d1a/d4f/d90/fae to d0/d1a/d2b/fca 0 2024-03-23T11:47:40.442 INFO:tasks.workunit.client.1.smithi111.stdout:5/707: dwrite d1/d3/d6/d8b/d59/d74/f4d [0,4194304] 0 2024-03-23T11:47:40.442 INFO:tasks.workunit.client.0.smithi063.stdout:3/749: dwrite d13/d22/d2d/d78/dcc/dcf/fd0 [0,4194304] 0 2024-03-23T11:47:40.444 INFO:tasks.workunit.client.1.smithi111.stdout:7/648: chown d14/d1b/d1e/d2c/d2e/d73 15 1 2024-03-23T11:47:40.445 INFO:tasks.workunit.client.1.smithi111.stdout:8/564: symlink d5/d1b/lba 0 2024-03-23T11:47:40.448 INFO:tasks.workunit.client.1.smithi111.stdout:7/649: write d14/d1b/d1e/d2c/d2e/f34 [5976147,33274] 0 2024-03-23T11:47:40.450 INFO:tasks.workunit.client.0.smithi063.stdout:0/616: dwrite d2/d12/d35/f81 [0,4194304] 0 2024-03-23T11:47:40.457 INFO:tasks.workunit.client.1.smithi111.stdout:3/605: mknod d0/d1a/d2b/d38/dc2/ccb 0 2024-03-23T11:47:40.457 INFO:tasks.workunit.client.0.smithi063.stdout:3/750: symlink d13/d66/dde/le4 0 2024-03-23T11:47:40.460 INFO:tasks.workunit.client.1.smithi111.stdout:9/697: write d0/d4/fe [2893068,79498] 0 2024-03-23T11:47:40.462 INFO:tasks.workunit.client.1.smithi111.stdout:8/565: dread - d5/d26/d31/f64 zero size 2024-03-23T11:47:40.464 INFO:tasks.workunit.client.1.smithi111.stdout:7/650: fdatasync d14/d1b/d22/f29 0 2024-03-23T11:47:40.474 INFO:tasks.workunit.client.1.smithi111.stdout:3/606: unlink d0/d1a/d1d/d26/d5a/l89 0 2024-03-23T11:47:40.479 INFO:tasks.workunit.client.1.smithi111.stdout:7/651: rename d14/d1b/d1e/d2c/d42/d5e/d88/f3a to d14/fba 0 2024-03-23T11:47:40.480 INFO:tasks.workunit.client.0.smithi063.stdout:0/617: dwrite d2/d14/f87 [0,4194304] 0 2024-03-23T11:47:40.481 INFO:tasks.workunit.client.1.smithi111.stdout:3/607: creat d0/d1a/d2b/d38/dc2/fcc x:0 0 0 2024-03-23T11:47:40.482 INFO:tasks.workunit.client.0.smithi063.stdout:0/618: dread - d2/d6e/d8c/fae zero size 2024-03-23T11:47:40.482 INFO:tasks.workunit.client.1.smithi111.stdout:8/566: link d5/d26/d31/d3b/l4c d5/d26/d2f/d62/d75/d4b/lbb 0 2024-03-23T11:47:40.484 INFO:tasks.workunit.client.1.smithi111.stdout:7/652: mknod d14/d1b/d1e/d2c/d2e/d33/cbb 0 2024-03-23T11:47:40.488 INFO:tasks.workunit.client.0.smithi063.stdout:0/619: mkdir d2/d48/de2 0 2024-03-23T11:47:40.489 INFO:tasks.workunit.client.1.smithi111.stdout:5/708: write d1/d3/d6/d7/f5a [534006,100305] 0 2024-03-23T11:47:40.490 INFO:tasks.workunit.client.1.smithi111.stdout:3/608: mkdir d0/d7/dcd 0 2024-03-23T11:47:40.492 INFO:tasks.workunit.client.1.smithi111.stdout:5/709: chown d1/d20/d5c/d99/dbf 223 1 2024-03-23T11:47:40.493 INFO:tasks.workunit.client.1.smithi111.stdout:2/721: write d2/d11/d17/d3e/d5d/fa4 [60473,2877] 0 2024-03-23T11:47:40.493 INFO:tasks.workunit.client.1.smithi111.stdout:8/567: getdents d5/d26/d31/d3b/d55 0 2024-03-23T11:47:40.495 INFO:tasks.workunit.client.1.smithi111.stdout:7/653: mkdir d14/d44/d70/d7e/dbc 0 2024-03-23T11:47:40.498 INFO:tasks.workunit.client.1.smithi111.stdout:3/609: dread d0/d1a/d2b/f6c [0,4194304] 0 2024-03-23T11:47:40.500 INFO:tasks.workunit.client.1.smithi111.stdout:2/722: chown d2/d11/d17/d1f/da0/fd4 2770876 1 2024-03-23T11:47:40.504 INFO:tasks.workunit.client.0.smithi063.stdout:3/751: truncate f6 3225326 0 2024-03-23T11:47:40.505 INFO:tasks.workunit.client.1.smithi111.stdout:8/568: fdatasync d5/d26/d31/d3b/fa9 0 2024-03-23T11:47:40.505 INFO:tasks.workunit.client.0.smithi063.stdout:0/620: truncate d2/d12/d32/f89 2692140 0 2024-03-23T11:47:40.506 INFO:tasks.workunit.client.1.smithi111.stdout:2/723: rmdir d2/d11/d17/d3e/d6b/d2e/d3f 39 2024-03-23T11:47:40.506 INFO:tasks.workunit.client.1.smithi111.stdout:9/698: dwrite d0/d31/d35/f9c [0,4194304] 0 2024-03-23T11:47:40.506 INFO:tasks.workunit.client.0.smithi063.stdout:3/752: chown d13/d22/d2d/d78/lba 2421240 1 2024-03-23T11:47:40.508 INFO:tasks.workunit.client.1.smithi111.stdout:5/710: creat d1/d3/d6/d7/d63/fe8 x:0 0 0 2024-03-23T11:47:40.509 INFO:tasks.workunit.client.0.smithi063.stdout:3/753: dread - d13/d22/d2d/d3f/d58/d6e/fbf zero size 2024-03-23T11:47:40.509 INFO:tasks.workunit.client.1.smithi111.stdout:3/610: link d0/d1a/d1d/f24 d0/d1a/d4f/d90/fce 0 2024-03-23T11:47:40.510 INFO:tasks.workunit.client.0.smithi063.stdout:0/621: truncate d2/d14/d36/d77/d94/da6/d28/fc5 729263 0 2024-03-23T11:47:40.510 INFO:tasks.workunit.client.1.smithi111.stdout:5/711: dread - d1/d20/dce/fe4 zero size 2024-03-23T11:47:40.510 INFO:tasks.workunit.client.1.smithi111.stdout:8/569: mkdir d5/d1b/dbc 0 2024-03-23T11:47:40.514 INFO:tasks.workunit.client.1.smithi111.stdout:2/724: unlink d2/d11/f13 0 2024-03-23T11:47:40.514 INFO:tasks.workunit.client.1.smithi111.stdout:7/654: dwrite fe [0,4194304] 0 2024-03-23T11:47:40.516 INFO:tasks.workunit.client.1.smithi111.stdout:9/699: symlink d0/d4/d10/le7 0 2024-03-23T11:47:40.517 INFO:tasks.workunit.client.0.smithi063.stdout:0/622: symlink d2/d14/d36/d77/d94/da6/d28/d46/d51/le3 0 2024-03-23T11:47:40.518 INFO:tasks.workunit.client.0.smithi063.stdout:5/369: sync 2024-03-23T11:47:40.518 INFO:tasks.workunit.client.0.smithi063.stdout:0/623: chown d2/d12/d35/dcf 340096 1 2024-03-23T11:47:40.520 INFO:tasks.workunit.client.0.smithi063.stdout:5/370: chown d2/d7/d8 3 1 2024-03-23T11:47:40.521 INFO:tasks.workunit.client.0.smithi063.stdout:5/371: chown d2/d28/f4b 16546 1 2024-03-23T11:47:40.524 INFO:tasks.workunit.client.1.smithi111.stdout:4/685: sync 2024-03-23T11:47:40.526 INFO:tasks.workunit.client.1.smithi111.stdout:4/686: dread - d0/dc/d4c/d2a/f8b zero size 2024-03-23T11:47:40.526 INFO:tasks.workunit.client.1.smithi111.stdout:5/712: rmdir d1/d20/d5c 39 2024-03-23T11:47:40.530 INFO:tasks.workunit.client.1.smithi111.stdout:2/725: write d2/d11/d17/d3e/d6b/d2e/d3f/f8e [2701766,129553] 0 2024-03-23T11:47:40.532 INFO:tasks.workunit.client.0.smithi063.stdout:5/372: rmdir d2/d7/d64/d81 0 2024-03-23T11:47:40.534 INFO:tasks.workunit.client.1.smithi111.stdout:7/655: creat d14/d1b/d1e/d2c/d2e/d73/dae/d74/fbd x:0 0 0 2024-03-23T11:47:40.535 INFO:tasks.workunit.client.1.smithi111.stdout:8/570: mknod d5/d26/cbd 0 2024-03-23T11:47:40.535 INFO:tasks.workunit.client.1.smithi111.stdout:9/700: rmdir d0/d4/d1c/d2d/d3b/d42/d4f/d95 39 2024-03-23T11:47:40.536 INFO:tasks.workunit.client.0.smithi063.stdout:3/754: dread d13/d22/d2d/d3f/d5b/f77 [0,4194304] 0 2024-03-23T11:47:40.538 INFO:tasks.workunit.client.1.smithi111.stdout:9/701: write d0/d4/d1c/d2d/d3b/d42/d92/fc7 [390893,110102] 0 2024-03-23T11:47:40.539 INFO:tasks.workunit.client.0.smithi063.stdout:0/624: getdents d2/d14/d36/d77/d94/da6/d28/d46/d51 0 2024-03-23T11:47:40.541 INFO:tasks.workunit.client.0.smithi063.stdout:3/755: write d13/d22/d2d/d78/f92 [412868,27070] 0 2024-03-23T11:47:40.541 INFO:tasks.workunit.client.1.smithi111.stdout:4/687: creat d0/d17/d86/fda x:0 0 0 2024-03-23T11:47:40.543 INFO:tasks.workunit.client.0.smithi063.stdout:0/625: write d2/d12/d32/fc0 [1000857,47195] 0 2024-03-23T11:47:40.544 INFO:tasks.workunit.client.1.smithi111.stdout:5/713: creat d1/d3/d3a/fe9 x:0 0 0 2024-03-23T11:47:40.547 INFO:tasks.workunit.client.0.smithi063.stdout:3/756: read d13/d22/d2d/d3f/d58/d97/fae [87073,68559] 0 2024-03-23T11:47:40.548 INFO:tasks.workunit.client.1.smithi111.stdout:7/656: unlink d14/d1b/d1e/f39 0 2024-03-23T11:47:40.548 INFO:tasks.workunit.client.1.smithi111.stdout:8/571: mkdir d5/d26/d2f/d62/d75/d4b/d51/d5a/dbe 0 2024-03-23T11:47:40.549 INFO:tasks.workunit.client.1.smithi111.stdout:9/702: rmdir d0/d4/d1c/d2d/d3b 39 2024-03-23T11:47:40.550 INFO:tasks.workunit.client.0.smithi063.stdout:3/757: dread - d13/d22/d2d/d3e/dce/fdf zero size 2024-03-23T11:47:40.550 INFO:tasks.workunit.client.1.smithi111.stdout:2/726: dwrite d2/d7/fd [0,4194304] 0 2024-03-23T11:47:40.551 INFO:tasks.workunit.client.0.smithi063.stdout:0/626: link d2/d4/fd8 d2/d6e/d8c/fe4 0 2024-03-23T11:47:40.554 INFO:tasks.workunit.client.0.smithi063.stdout:3/758: mkdir d13/de5 0 2024-03-23T11:47:40.559 INFO:tasks.workunit.client.1.smithi111.stdout:9/703: symlink d0/d4/d10/d80/le8 0 2024-03-23T11:47:40.560 INFO:tasks.workunit.client.1.smithi111.stdout:2/727: write d2/de/d6c/dd1/fe6 [1241699,78172] 0 2024-03-23T11:47:40.562 INFO:tasks.workunit.client.1.smithi111.stdout:4/688: dread d0/d7/da/f46 [0,4194304] 0 2024-03-23T11:47:40.562 INFO:tasks.workunit.client.0.smithi063.stdout:3/759: rename d13/d15/f25 to d13/d22/fe6 0 2024-03-23T11:47:40.567 INFO:tasks.workunit.client.1.smithi111.stdout:4/689: mkdir d0/dc/d1f/ddb 0 2024-03-23T11:47:40.568 INFO:tasks.workunit.client.1.smithi111.stdout:8/572: getdents d5/d26/d31/d5e 0 2024-03-23T11:47:40.573 INFO:tasks.workunit.client.1.smithi111.stdout:4/690: creat d0/d95/db5/dc7/fdc x:0 0 0 2024-03-23T11:47:40.576 INFO:tasks.workunit.client.1.smithi111.stdout:8/573: getdents d5/d26/d2f/d62/d75/d4b/d51/da4/d97/db9 0 2024-03-23T11:47:40.577 INFO:tasks.workunit.client.1.smithi111.stdout:5/714: truncate d1/d3/d6/d7/f64 1626293 0 2024-03-23T11:47:40.579 INFO:tasks.workunit.client.1.smithi111.stdout:9/704: truncate d0/d31/d41/d71/f93 1986187 0 2024-03-23T11:47:40.581 INFO:tasks.workunit.client.1.smithi111.stdout:8/574: truncate d5/d26/d31/f64 548038 0 2024-03-23T11:47:40.582 INFO:tasks.workunit.client.0.smithi063.stdout:0/627: dwrite d2/d12/d32/fc0 [0,4194304] 0 2024-03-23T11:47:40.583 INFO:tasks.workunit.client.1.smithi111.stdout:4/691: getdents d0/dc/d4c/d83 0 2024-03-23T11:47:40.586 INFO:tasks.workunit.client.1.smithi111.stdout:4/692: creat d0/dc/d1f/db0/fdd x:0 0 0 2024-03-23T11:47:40.587 INFO:tasks.workunit.client.0.smithi063.stdout:0/628: dread - d2/d14/d36/d77/d94/da6/d28/d46/d51/f9d zero size 2024-03-23T11:47:40.589 INFO:tasks.workunit.client.0.smithi063.stdout:3/760: dwrite d13/d22/d2d/d3f/d58/d6e/f7a [4194304,4194304] 0 2024-03-23T11:47:40.593 INFO:tasks.workunit.client.0.smithi063.stdout:0/629: creat d2/d14/d36/d77/fe5 x:0 0 0 2024-03-23T11:47:40.594 INFO:tasks.workunit.client.1.smithi111.stdout:4/693: getdents d0/dc/d78 0 2024-03-23T11:47:40.596 INFO:tasks.workunit.client.0.smithi063.stdout:0/630: chown d2/d48/fcb 22762 1 2024-03-23T11:47:40.597 INFO:tasks.workunit.client.0.smithi063.stdout:3/761: mknod d13/d22/d2d/d3f/d58/d6e/d8d/dda/de1/ce7 0 2024-03-23T11:47:40.603 INFO:tasks.workunit.client.0.smithi063.stdout:0/631: truncate d2/d14/d36/d77/d94/da6/d1f/d4f/fc1 293681 0 2024-03-23T11:47:40.603 INFO:tasks.workunit.client.0.smithi063.stdout:3/762: rename d13/d22/d2d/d3f/d58 to d13/d22/d2d/d3f/d58/d6e/d8d/dda/de1/de8 22 2024-03-23T11:47:40.604 INFO:tasks.workunit.client.1.smithi111.stdout:9/705: dwrite d0/d4/d10/d29/d7f/d8e/fb5 [0,4194304] 0 2024-03-23T11:47:40.605 INFO:tasks.workunit.client.0.smithi063.stdout:0/632: stat d2/d12/f57 0 2024-03-23T11:47:40.605 INFO:tasks.workunit.client.1.smithi111.stdout:8/575: dwrite d5/d26/d31/d5e/d4a/f52 [0,4194304] 0 2024-03-23T11:47:40.608 INFO:tasks.workunit.client.1.smithi111.stdout:9/706: truncate d0/d4/fbd 1386210 0 2024-03-23T11:47:40.610 INFO:tasks.workunit.client.0.smithi063.stdout:0/633: creat d2/d12/d32/fe6 x:0 0 0 2024-03-23T11:47:40.612 INFO:tasks.workunit.client.0.smithi063.stdout:0/634: chown d2/d14/c2e 4216597 1 2024-03-23T11:47:40.622 INFO:tasks.workunit.client.1.smithi111.stdout:4/694: write d0/d95/fb8 [1860987,112025] 0 2024-03-23T11:47:40.624 INFO:tasks.workunit.client.0.smithi063.stdout:3/763: dread d13/d22/d2d/d3f/d58/f76 [0,4194304] 0 2024-03-23T11:47:40.627 INFO:tasks.workunit.client.1.smithi111.stdout:4/695: creat d0/d17/fde x:0 0 0 2024-03-23T11:47:40.628 INFO:tasks.workunit.client.1.smithi111.stdout:9/707: dwrite d0/d4/d1c/fe3 [0,4194304] 0 2024-03-23T11:47:40.629 INFO:tasks.workunit.client.1.smithi111.stdout:0/638: sync 2024-03-23T11:47:40.629 INFO:tasks.workunit.client.1.smithi111.stdout:1/641: sync 2024-03-23T11:47:40.634 INFO:tasks.workunit.client.1.smithi111.stdout:4/696: mknod d0/d95/db5/cdf 0 2024-03-23T11:47:40.635 INFO:tasks.workunit.client.1.smithi111.stdout:0/639: creat d5/db/dbe/fd4 x:0 0 0 2024-03-23T11:47:40.637 INFO:tasks.workunit.client.1.smithi111.stdout:1/642: creat d0/fdd x:0 0 0 2024-03-23T11:47:40.640 INFO:tasks.workunit.client.1.smithi111.stdout:4/697: symlink d0/d17/d86/le0 0 2024-03-23T11:47:40.641 INFO:tasks.workunit.client.1.smithi111.stdout:0/640: mkdir d5/db/d1f/d19/dbd/d64/d91/dd5 0 2024-03-23T11:47:40.642 INFO:tasks.workunit.client.1.smithi111.stdout:8/576: write d5/d26/d31/d5e/d4a/f8e [787961,26822] 0 2024-03-23T11:47:40.643 INFO:tasks.workunit.client.1.smithi111.stdout:4/698: write d0/d95/fa9 [278411,92557] 0 2024-03-23T11:47:40.646 INFO:tasks.workunit.client.1.smithi111.stdout:0/641: write d5/db/d1f/f9b [1131462,77061] 0 2024-03-23T11:47:40.647 INFO:tasks.workunit.client.0.smithi063.stdout:0/635: write d2/d4/fd8 [1193353,52774] 0 2024-03-23T11:47:40.649 INFO:tasks.workunit.client.1.smithi111.stdout:9/708: symlink d0/d4/d1c/d2d/d3b/d42/d4f/d95/le9 0 2024-03-23T11:47:40.652 INFO:tasks.workunit.client.1.smithi111.stdout:8/577: unlink d5/d26/d31/d5e/c5c 0 2024-03-23T11:47:40.654 INFO:tasks.workunit.client.1.smithi111.stdout:8/578: chown d5/d26/d2f/d62/d75/d4b/c7a 30436 1 2024-03-23T11:47:40.654 INFO:tasks.workunit.client.0.smithi063.stdout:0/636: rename d2/d14/d36/d77/d94/da6/d28/d46/c92 to d2/d12/d32/d6d/ce7 0 2024-03-23T11:47:40.656 INFO:tasks.workunit.client.0.smithi063.stdout:3/764: dwrite f6 [0,4194304] 0 2024-03-23T11:47:40.656 INFO:tasks.workunit.client.1.smithi111.stdout:4/699: creat d0/dc/fe1 x:0 0 0 2024-03-23T11:47:40.660 INFO:tasks.workunit.client.1.smithi111.stdout:0/642: creat d5/fd6 x:0 0 0 2024-03-23T11:47:40.662 INFO:tasks.workunit.client.1.smithi111.stdout:0/643: fdatasync d5/db/d1f/d48/f66 0 2024-03-23T11:47:40.663 INFO:tasks.workunit.client.1.smithi111.stdout:9/709: rename d0/d31/l38 to d0/d31/lea 0 2024-03-23T11:47:40.664 INFO:tasks.workunit.client.1.smithi111.stdout:9/710: write d0/fbc [4098616,23617] 0 2024-03-23T11:47:40.665 INFO:tasks.workunit.client.1.smithi111.stdout:4/700: getdents d0/dc/d78 0 2024-03-23T11:47:40.666 INFO:tasks.workunit.client.0.smithi063.stdout:3/765: getdents d13/d22/d2d/d3f/d58/d97 0 2024-03-23T11:47:40.670 INFO:tasks.workunit.client.1.smithi111.stdout:4/701: rename d0/dc/d78/fd1 to d0/d17/d86/fe2 0 2024-03-23T11:47:40.673 INFO:tasks.workunit.client.0.smithi063.stdout:3/766: creat d13/d22/d2d/d3f/de0/fe9 x:0 0 0 2024-03-23T11:47:40.675 INFO:tasks.workunit.client.1.smithi111.stdout:9/711: rename d0/d4/d1c/d2d/d3b/d42/d4f/c7d to d0/d4/d1c/d2d/d3b/d42/d92/ceb 0 2024-03-23T11:47:40.675 INFO:tasks.workunit.client.1.smithi111.stdout:4/702: mkdir d0/dc/d4c/d2a/d97/d4d/de3 0 2024-03-23T11:47:40.676 INFO:tasks.workunit.client.1.smithi111.stdout:0/644: link d5/db/d1f/d32/l86 d5/db/d1f/d47/d59/ld7 0 2024-03-23T11:47:40.679 INFO:tasks.workunit.client.1.smithi111.stdout:9/712: symlink d0/d31/d41/d65/lec 0 2024-03-23T11:47:40.681 INFO:tasks.workunit.client.1.smithi111.stdout:4/703: unlink d0/dc/d4c/d2a/d97/d4f/f62 0 2024-03-23T11:47:40.683 INFO:tasks.workunit.client.1.smithi111.stdout:9/713: creat d0/d31/fed x:0 0 0 2024-03-23T11:47:40.685 INFO:tasks.workunit.client.1.smithi111.stdout:4/704: mknod d0/dc/d1f/d5e/ce4 0 2024-03-23T11:47:40.687 INFO:tasks.workunit.client.0.smithi063.stdout:3/767: dread d13/d22/d2d/d3f/d5b/f77 [0,4194304] 0 2024-03-23T11:47:40.690 INFO:tasks.workunit.client.1.smithi111.stdout:9/714: rmdir d0/d4/d1c/d2d/d3b/d42/d4f/d95 39 2024-03-23T11:47:40.693 INFO:tasks.workunit.client.1.smithi111.stdout:8/579: write d5/d1b/f3a [4350364,16776] 0 2024-03-23T11:47:40.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.712+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d900ffa40 con 0x7f0d7406ad60 2024-03-23T11:47:40.697 INFO:tasks.workunit.client.0.smithi063.stdout:3/768: read d13/d22/d2d/d3f/d58/f70 [3822186,112023] 0 2024-03-23T11:47:40.698 INFO:tasks.workunit.client.1.smithi111.stdout:4/705: rename d0/d7/fc4 to d0/dc/d4c/d2a/d97/fe5 0 2024-03-23T11:47:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.715+0000 7f0d86000700 1 -- 172.21.15.63:0/2360215244 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f0d900ffa40 con 0x7f0d7406ad60 2024-03-23T11:47:40.700 INFO:tasks.workunit.client.1.smithi111.stdout:8/580: fdatasync d5/d26/d31/fb6 0 2024-03-23T11:47:40.701 INFO:tasks.workunit.client.1.smithi111.stdout:0/645: dwrite d5/db/d1f/d47/d59/d5b/f7c [0,4194304] 0 2024-03-23T11:47:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f0d7406ad60 msgr2=0x7f0d7406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f0d7406ad60 0x7f0d7406d210 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d9006fd00 tx=0x7f0d8000b040 comp rx=0 tx=0).stop 2024-03-23T11:47:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90071ea0 msgr2=0x7f0d9006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90071ea0 0x7f0d9006e980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c013420 tx=0x7f0d7c009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 shutdown_connections 2024-03-23T11:47:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d900708f0 0x7f0d9006e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f0d7406ad60 0x7f0d7406d210 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 --2- 172.21.15.63:0/2360215244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90071ea0 0x7f0d9006e980 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c013420 tx=0x7f0d7c009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.718+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 >> 172.21.15.63:0/2360215244 conn(0x7f0d900f8560 msgr2=0x7f0d900f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.720+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 shutdown_connections 2024-03-23T11:47:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.720+0000 7f0d94c00700 1 -- 172.21.15.63:0/2360215244 wait complete. 2024-03-23T11:47:40.706 INFO:tasks.workunit.client.0.smithi063.stdout:3/769: dread d13/d22/da9/fc2 [0,4194304] 0 2024-03-23T11:47:40.708 INFO:tasks.workunit.client.1.smithi111.stdout:8/581: mknod d5/d26/d2f/d62/d7e/d85/cbf 0 2024-03-23T11:47:40.710 INFO:tasks.workunit.client.1.smithi111.stdout:1/643: dread d0/d2b/d69/fbc [0,4194304] 0 2024-03-23T11:47:40.711 INFO:tasks.workunit.client.0.smithi063.stdout:3/770: creat d13/d22/d2d/d3f/d58/d97/fea x:0 0 0 2024-03-23T11:47:40.712 INFO:tasks.workunit.client.1.smithi111.stdout:8/582: symlink d5/d1b/lc0 0 2024-03-23T11:47:40.715 INFO:tasks.workunit.client.1.smithi111.stdout:1/644: fsync d0/d2b/d37/f4c 0 2024-03-23T11:47:40.717 INFO:tasks.workunit.client.1.smithi111.stdout:4/706: write d0/dc/d89/da4/fbd [417268,103013] 0 2024-03-23T11:47:40.718 INFO:tasks.workunit.client.1.smithi111.stdout:8/583: symlink d5/d26/d2f/d62/d75/d4b/lc1 0 2024-03-23T11:47:40.720 INFO:tasks.workunit.client.1.smithi111.stdout:0/646: write d5/db/d1f/d19/dbd/f3d [1737281,37916] 0 2024-03-23T11:47:40.721 INFO:tasks.workunit.client.1.smithi111.stdout:1/645: mknod d0/d5/dd/d1e/d70/cde 0 2024-03-23T11:47:40.722 INFO:tasks.workunit.client.1.smithi111.stdout:4/707: creat d0/d95/fe6 x:0 0 0 2024-03-23T11:47:40.726 INFO:tasks.workunit.client.1.smithi111.stdout:0/647: symlink d5/db/d1f/d19/dbd/d8b/ld8 0 2024-03-23T11:47:40.727 INFO:tasks.workunit.client.1.smithi111.stdout:1/646: write d0/d6/d48/f5d [899891,30351] 0 2024-03-23T11:47:40.727 INFO:tasks.workunit.client.1.smithi111.stdout:4/708: write d0/dc/d1f/f8a [3433,14260] 0 2024-03-23T11:47:40.727 INFO:tasks.workunit.client.1.smithi111.stdout:8/584: symlink d5/d26/d2f/d62/d75/lc2 0 2024-03-23T11:47:40.728 INFO:tasks.workunit.client.1.smithi111.stdout:9/715: dwrite d0/d4/f43 [0,4194304] 0 2024-03-23T11:47:40.729 INFO:tasks.workunit.client.0.smithi063.stdout:3/771: dwrite ff [0,4194304] 0 2024-03-23T11:47:40.730 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:47:40.732 INFO:tasks.workunit.client.1.smithi111.stdout:0/648: dread - d5/db/d1f/d19/d45/fad zero size 2024-03-23T11:47:40.733 INFO:tasks.workunit.client.1.smithi111.stdout:4/709: symlink d0/dc/d78/le7 0 2024-03-23T11:47:40.735 INFO:tasks.workunit.client.1.smithi111.stdout:9/716: chown d0/d4/d1c/f45 4777 1 2024-03-23T11:47:40.738 INFO:tasks.workunit.client.1.smithi111.stdout:9/717: dread - d0/d4/d1c/d2d/d75/fce zero size 2024-03-23T11:47:40.738 INFO:tasks.workunit.client.1.smithi111.stdout:4/710: truncate d0/dc/d4c/f6d 422837 0 2024-03-23T11:47:40.738 INFO:tasks.workunit.client.0.smithi063.stdout:3/772: readlink d13/d22/l41 0 2024-03-23T11:47:40.741 INFO:tasks.workunit.client.1.smithi111.stdout:9/718: chown d0/d31/d41/d65/l77 8 1 2024-03-23T11:47:40.743 INFO:tasks.workunit.client.1.smithi111.stdout:1/647: rename d0/d6/d48/d4a/d6c/fb9 to d0/d2b/fdf 0 2024-03-23T11:47:40.744 INFO:tasks.workunit.client.1.smithi111.stdout:6/696: sync 2024-03-23T11:47:40.745 INFO:tasks.workunit.client.1.smithi111.stdout:4/711: symlink d0/d95/db5/dc7/le8 0 2024-03-23T11:47:40.747 INFO:tasks.workunit.client.1.smithi111.stdout:1/648: mknod d0/ce0 0 2024-03-23T11:47:40.747 INFO:tasks.workunit.client.0.smithi063.stdout:3/773: rename d13/d66/ld3 to d13/d22/d2d/d3f/d58/leb 0 2024-03-23T11:47:40.747 INFO:tasks.workunit.client.1.smithi111.stdout:4/712: chown d0/dc/d78/l90 29772137 1 2024-03-23T11:47:40.748 INFO:tasks.workunit.client.1.smithi111.stdout:1/649: fsync d0/d6/f34 0 2024-03-23T11:47:40.749 INFO:tasks.workunit.client.1.smithi111.stdout:6/697: truncate d2/d1a/d1c/d2c/fa5 4598496 0 2024-03-23T11:47:40.749 INFO:tasks.workunit.client.1.smithi111.stdout:8/585: dwrite d5/d26/d31/d3b/fa9 [0,4194304] 0 2024-03-23T11:47:40.750 INFO:tasks.workunit.client.0.smithi063.stdout:3/774: creat d13/de5/fec x:0 0 0 2024-03-23T11:47:40.751 INFO:tasks.workunit.client.1.smithi111.stdout:1/650: write d0/d2b/f5a [1934085,40484] 0 2024-03-23T11:47:40.753 INFO:tasks.workunit.client.1.smithi111.stdout:6/698: symlink d2/d1a/le7 0 2024-03-23T11:47:40.753 INFO:tasks.workunit.client.1.smithi111.stdout:1/651: chown d0/d2b/d7c/d8b 417 1 2024-03-23T11:47:40.754 INFO:tasks.workunit.client.1.smithi111.stdout:8/586: mkdir d5/d1b/dbc/dc3 0 2024-03-23T11:47:40.757 INFO:tasks.workunit.client.1.smithi111.stdout:1/652: write d0/d5/dd/d1e/d54/fb3 [1000284,107477] 0 2024-03-23T11:47:40.758 INFO:tasks.workunit.client.0.smithi063.stdout:3/775: link d13/d22/d2d/d3f/de0/fe9 d13/d22/d2d/fed 0 2024-03-23T11:47:40.759 INFO:tasks.workunit.client.1.smithi111.stdout:1/653: chown d0/d5/c14 15352 1 2024-03-23T11:47:40.759 INFO:tasks.workunit.client.1.smithi111.stdout:8/587: read d5/d26/d31/f39 [2887087,108504] 0 2024-03-23T11:47:40.761 INFO:tasks.workunit.client.1.smithi111.stdout:1/654: chown d0/d5/dd/d98/l9a 52195 1 2024-03-23T11:47:40.761 INFO:tasks.workunit.client.1.smithi111.stdout:6/699: rename d2/c15 to d2/d1f/d45/d49/d8d/ce8 0 2024-03-23T11:47:40.767 INFO:tasks.workunit.client.1.smithi111.stdout:0/649: dread d5/f30 [0,4194304] 0 2024-03-23T11:47:40.768 INFO:tasks.workunit.client.1.smithi111.stdout:1/655: dread d0/d6/f19 [0,4194304] 0 2024-03-23T11:47:40.772 INFO:tasks.workunit.client.0.smithi063.stdout:3/776: dwrite d13/d22/d2d/d3f/d5b/f65 [0,4194304] 0 2024-03-23T11:47:40.772 INFO:tasks.workunit.client.1.smithi111.stdout:1/656: write d0/d5/dd/fb1 [4556166,88423] 0 2024-03-23T11:47:40.773 INFO:tasks.workunit.client.0.smithi063.stdout:3/777: fdatasync fc 0 2024-03-23T11:47:40.773 INFO:tasks.workunit.client.0.smithi063.stdout:4/571: sync 2024-03-23T11:47:40.775 INFO:tasks.workunit.client.1.smithi111.stdout:8/588: dread d5/d26/d31/d4d/f6c [0,4194304] 0 2024-03-23T11:47:40.775 INFO:tasks.workunit.client.1.smithi111.stdout:0/650: link d5/db/d1f/d48/c57 d5/db/d1f/d19/dbd/d64/cd9 0 2024-03-23T11:47:40.777 INFO:tasks.workunit.client.1.smithi111.stdout:1/657: read - d0/d5/dd/d1e/f40 zero size 2024-03-23T11:47:40.778 INFO:tasks.workunit.client.1.smithi111.stdout:8/589: chown d5/d26/d31/d4d/faf 650 1 2024-03-23T11:47:40.778 INFO:tasks.workunit.client.1.smithi111.stdout:0/651: write d5/db/d1f/d82/fac [154493,66513] 0 2024-03-23T11:47:40.780 INFO:tasks.workunit.client.0.smithi063.stdout:4/572: creat d5/da/d72/fa8 x:0 0 0 2024-03-23T11:47:40.783 INFO:tasks.workunit.client.0.smithi063.stdout:4/573: write d5/da/d20/f3c [4127647,41337] 0 2024-03-23T11:47:40.783 INFO:tasks.workunit.client.1.smithi111.stdout:1/658: getdents d0/d6/d48 0 2024-03-23T11:47:40.784 INFO:tasks.workunit.client.0.smithi063.stdout:3/778: mkdir d13/d22/d2d/d3f/d58/d6e/d8d/dda/dee 0 2024-03-23T11:47:40.785 INFO:tasks.workunit.client.1.smithi111.stdout:1/659: chown d0/d2b/d37/d6a 12361740 1 2024-03-23T11:47:40.787 INFO:tasks.workunit.client.0.smithi063.stdout:3/779: write d13/d22/d2d/d7d/f94 [1746563,86442] 0 2024-03-23T11:47:40.787 INFO:tasks.workunit.client.1.smithi111.stdout:0/652: creat d5/db/d1f/d19/dbd/d8b/dd0/fda x:0 0 0 2024-03-23T11:47:40.791 INFO:tasks.workunit.client.1.smithi111.stdout:6/700: dread d2/d1f/d45/f63 [0,4194304] 0 2024-03-23T11:47:40.792 INFO:tasks.workunit.client.0.smithi063.stdout:3/780: creat d13/d22/d2d/d3f/d58/d86/dad/fef x:0 0 0 2024-03-23T11:47:40.793 INFO:tasks.workunit.client.1.smithi111.stdout:0/653: creat d5/db/d4d/da6/fdb x:0 0 0 2024-03-23T11:47:40.794 INFO:tasks.workunit.client.0.smithi063.stdout:3/781: chown d13/d22/d2d/d78/lba 25 1 2024-03-23T11:47:40.796 INFO:tasks.workunit.client.0.smithi063.stdout:3/782: read d13/f9c [515575,80172] 0 2024-03-23T11:47:40.798 INFO:tasks.workunit.client.1.smithi111.stdout:8/590: dwrite d5/d26/d2f/d62/d75/d4b/fb1 [0,4194304] 0 2024-03-23T11:47:40.799 INFO:tasks.workunit.client.1.smithi111.stdout:8/591: write f1 [645363,18705] 0 2024-03-23T11:47:40.804 INFO:tasks.workunit.client.0.smithi063.stdout:4/574: write d5/f1c [1193461,75366] 0 2024-03-23T11:47:40.804 INFO:tasks.workunit.client.1.smithi111.stdout:8/592: readlink d5/d26/d31/d3b/l4c 0 2024-03-23T11:47:40.807 INFO:tasks.workunit.client.1.smithi111.stdout:8/593: truncate d5/d26/d31/d3b/f89 144546 0 2024-03-23T11:47:40.809 INFO:tasks.workunit.client.1.smithi111.stdout:6/701: write d2/d35/d70/fc4 [1820514,60187] 0 2024-03-23T11:47:40.810 INFO:tasks.workunit.client.1.smithi111.stdout:0/654: dwrite d5/db/d1f/d19/dbd/f8f [0,4194304] 0 2024-03-23T11:47:40.811 INFO:tasks.workunit.client.0.smithi063.stdout:3/783: link d13/d22/d2d/l57 d13/d22/d2d/d3f/d58/lf0 0 2024-03-23T11:47:40.814 INFO:tasks.workunit.client.0.smithi063.stdout:4/575: creat d5/da/d20/d4b/fa9 x:0 0 0 2024-03-23T11:47:40.817 INFO:tasks.workunit.client.0.smithi063.stdout:3/784: write d13/d22/d2d/d3f/d67/fa3 [1527013,125644] 0 2024-03-23T11:47:40.821 INFO:tasks.workunit.client.0.smithi063.stdout:3/785: read d13/d22/d2d/f45 [337545,91294] 0 2024-03-23T11:47:40.828 INFO:tasks.workunit.client.1.smithi111.stdout:6/702: dwrite d2/d35/da7/fc2 [0,4194304] 0 2024-03-23T11:47:40.828 INFO:tasks.workunit.client.1.smithi111.stdout:8/594: dwrite d5/d26/d2f/d62/d75/f6b [0,4194304] 0 2024-03-23T11:47:40.832 INFO:tasks.workunit.client.0.smithi063.stdout:3/786: read d13/d15/fca [3172293,109409] 0 2024-03-23T11:47:40.833 INFO:tasks.workunit.client.0.smithi063.stdout:4/576: getdents d5/da/d20/d4b/d16/d32 0 2024-03-23T11:47:40.838 INFO:tasks.workunit.client.0.smithi063.stdout:3/787: chown d13/d22/d2d/d7d/la0 14818 1 2024-03-23T11:47:40.843 INFO:tasks.workunit.client.0.smithi063.stdout:3/788: write d13/d22/d2d/d3f/f9a [596150,28829] 0 2024-03-23T11:47:40.846 INFO:tasks.workunit.client.1.smithi111.stdout:3/611: sync 2024-03-23T11:47:40.847 INFO:tasks.workunit.client.1.smithi111.stdout:7/657: sync 2024-03-23T11:47:40.847 INFO:tasks.workunit.client.0.smithi063.stdout:3/789: mkdir d13/d22/d2d/d78/df1 0 2024-03-23T11:47:40.848 INFO:tasks.workunit.client.1.smithi111.stdout:1/660: dread d0/d6/f27 [4194304,4194304] 0 2024-03-23T11:47:40.850 INFO:tasks.workunit.client.0.smithi063.stdout:3/790: chown d13/d22/d2d/fed 2527 1 2024-03-23T11:47:40.857 INFO:tasks.workunit.client.0.smithi063.stdout:3/791: rename f3 to d13/de5/ff2 0 2024-03-23T11:47:40.866 INFO:tasks.workunit.client.0.smithi063.stdout:3/792: rename d13/d22/d2d/d3f/d58/f76 to d13/d66/ff3 0 2024-03-23T11:47:40.871 INFO:tasks.workunit.client.0.smithi063.stdout:3/793: write d13/d22/d2d/d3f/f72 [1089922,83804] 0 2024-03-23T11:47:40.875 INFO:tasks.workunit.client.0.smithi063.stdout:3/794: mknod d13/d22/da9/cf4 0 2024-03-23T11:47:40.878 INFO:tasks.workunit.client.0.smithi063.stdout:4/577: dread d5/da/d20/d4b/d16/d50/f5f [0,4194304] 0 2024-03-23T11:47:40.879 INFO:tasks.workunit.client.0.smithi063.stdout:3/795: rmdir d13/d22/d2d/d3f/d58/d86 39 2024-03-23T11:47:40.882 INFO:tasks.workunit.client.0.smithi063.stdout:3/796: read d13/d22/d2d/d7d/fb5 [42564,10116] 0 2024-03-23T11:47:40.883 INFO:tasks.workunit.client.0.smithi063.stdout:4/578: dread - d5/da/d20/d4b/f5e zero size 2024-03-23T11:47:40.885 INFO:tasks.workunit.client.0.smithi063.stdout:3/797: write d13/d22/d2d/d3f/d5b/f65 [3821239,79743] 0 2024-03-23T11:47:40.901 INFO:tasks.workunit.client.0.smithi063.stdout:4/579: dwrite d5/da/d20/d4b/d16/d32/d56/f85 [0,4194304] 0 2024-03-23T11:47:40.905 INFO:tasks.workunit.client.0.smithi063.stdout:3/798: dread d13/d15/f87 [0,4194304] 0 2024-03-23T11:47:40.910 INFO:tasks.workunit.client.1.smithi111.stdout:2/728: sync 2024-03-23T11:47:40.910 INFO:tasks.workunit.client.1.smithi111.stdout:5/715: sync 2024-03-23T11:47:40.910 INFO:tasks.workunit.client.1.smithi111.stdout:9/719: sync 2024-03-23T11:47:40.919 INFO:tasks.workunit.client.0.smithi063.stdout:3/799: truncate d13/d22/f3b 982085 0 2024-03-23T11:47:40.923 INFO:tasks.workunit.client.0.smithi063.stdout:3/800: mkdir d13/d66/dde/df5 0 2024-03-23T11:47:40.927 INFO:tasks.workunit.client.0.smithi063.stdout:3/801: mkdir d13/d22/d2d/d3f/df6 0 2024-03-23T11:47:40.928 INFO:tasks.workunit.client.0.smithi063.stdout:4/580: fsync d5/da/d20/d4b/d16/d32/d56/f85 0 2024-03-23T11:47:40.931 INFO:tasks.workunit.client.0.smithi063.stdout:4/581: dread - d5/da/d20/d4b/f5e zero size 2024-03-23T11:47:40.933 INFO:tasks.workunit.client.0.smithi063.stdout:3/802: mkdir d13/d22/d2d/d3f/d58/d6e/d8d/dda/df7 0 2024-03-23T11:47:40.935 INFO:tasks.workunit.client.0.smithi063.stdout:4/582: mkdir d5/d1e/d2b/daa 0 2024-03-23T11:47:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 -- 172.21.15.63:0/3591399067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74071e40 msgr2=0x7efd74070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 --2- 172.21.15.63:0/3591399067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74071e40 0x7efd74070350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efd680048b0 tx=0x7efd68009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 -- 172.21.15.63:0/3591399067 shutdown_connections 2024-03-23T11:47:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 --2- 172.21.15.63:0/3591399067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd74070890 0x7efd74070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 --2- 172.21.15.63:0/3591399067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74071e40 0x7efd74070350 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 -- 172.21.15.63:0/3591399067 >> 172.21.15.63:0/3591399067 conn(0x7efd740f8540 msgr2=0x7efd740fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 -- 172.21.15.63:0/3591399067 shutdown_connections 2024-03-23T11:47:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.953+0000 7efd7c800700 1 -- 172.21.15.63:0/3591399067 wait complete. 2024-03-23T11:47:40.939 INFO:tasks.workunit.client.0.smithi063.stdout:3/803: dread d13/d22/da9/fc2 [0,4194304] 0 2024-03-23T11:47:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.954+0000 7efd7c800700 1 Processor -- start 2024-03-23T11:47:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.955+0000 7efd7c800700 1 -- start start 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.955+0000 7efd7c800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74070890 0x7efd7410fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.955+0000 7efd7c800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd74071e40 0x7efd74110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.955+0000 7efd7c800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74110640 con 0x7efd74071e40 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.955+0000 7efd7c800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74110e90 con 0x7efd74070890 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.956+0000 7efd79e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74070890 0x7efd7410fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.956+0000 7efd79e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74070890 0x7efd7410fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:37396/0 (socket says 172.21.15.63:37396) 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.956+0000 7efd79e00700 1 -- 172.21.15.63:0/1755346460 learned_addr learned my addr 172.21.15.63:0/1755346460 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.956+0000 7efd79400700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd74071e40 0x7efd74110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:40.942 INFO:tasks.workunit.client.0.smithi063.stdout:3/804: dread - d13/de5/fec zero size 2024-03-23T11:47:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.957+0000 7efd79e00700 1 -- 172.21.15.63:0/1755346460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd74071e40 msgr2=0x7efd74110040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.957+0000 7efd79e00700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd74071e40 0x7efd74110040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.957+0000 7efd79e00700 1 -- 172.21.15.63:0/1755346460 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68005040 con 0x7efd74070890 2024-03-23T11:47:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.957+0000 7efd79e00700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74070890 0x7efd7410fb00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd68000c00 tx=0x7efd68009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.958+0000 7efd6e000700 1 -- 172.21.15.63:0/1755346460 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6800e040 con 0x7efd74070890 2024-03-23T11:47:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.958+0000 7efd6e000700 1 -- 172.21.15.63:0/1755346460 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd68014c30 con 0x7efd74070890 2024-03-23T11:47:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.958+0000 7efd6e000700 1 -- 172.21.15.63:0/1755346460 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd68012600 con 0x7efd74070890 2024-03-23T11:47:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.958+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd74111090 con 0x7efd74070890 2024-03-23T11:47:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.958+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd741115f0 con 0x7efd74070890 2024-03-23T11:47:40.944 INFO:tasks.workunit.client.1.smithi111.stdout:3/612: rename d0/d1a/fa2 to d0/d53/fcf 0 2024-03-23T11:47:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.961+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7405f5f0 con 0x7efd74070890 2024-03-23T11:47:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.962+0000 7efd6e000700 1 -- 172.21.15.63:0/1755346460 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efd6801c9e0 con 0x7efd74070890 2024-03-23T11:47:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.963+0000 7efd6e000700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7efd5c06ad60 0x7efd5c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.963+0000 7efd6e000700 1 -- 172.21.15.63:0/1755346460 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7efd6808a130 con 0x7efd74070890 2024-03-23T11:47:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.964+0000 7efd79400700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7efd5c06ad60 0x7efd5c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:40.949 INFO:tasks.workunit.client.0.smithi063.stdout:3/805: mkdir d13/d22/d2d/d3f/d58/d97/df8 0 2024-03-23T11:47:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.965+0000 7efd79400700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7efd5c06ad60 0x7efd5c06d210 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efd64009b80 tx=0x7efd6400b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:40.953 INFO:tasks.workunit.client.0.smithi063.stdout:3/806: mknod d13/d66/dde/cf9 0 2024-03-23T11:47:40.953 INFO:tasks.workunit.client.1.smithi111.stdout:1/661: unlink d0/d6/d48/d4d/c58 0 2024-03-23T11:47:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:40.969+0000 7efd6e000700 1 -- 172.21.15.63:0/1755346460 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7efd68058310 con 0x7efd74070890 2024-03-23T11:47:40.955 INFO:tasks.workunit.client.1.smithi111.stdout:1/662: write d0/d5/f2f [8910599,7978] 0 2024-03-23T11:47:40.957 INFO:tasks.workunit.client.0.smithi063.stdout:3/807: dread - d13/d22/d2d/d3f/d58/d6e/d8d/f95 zero size 2024-03-23T11:47:40.958 INFO:tasks.workunit.client.0.smithi063.stdout:3/808: chown ff 337 1 2024-03-23T11:47:40.959 INFO:tasks.workunit.client.1.smithi111.stdout:4/713: sync 2024-03-23T11:47:40.961 INFO:tasks.workunit.client.1.smithi111.stdout:2/729: stat d2/de/d6c/l7f 0 2024-03-23T11:47:40.963 INFO:tasks.workunit.client.0.smithi063.stdout:3/809: symlink d13/d66/dde/df5/lfa 0 2024-03-23T11:47:40.963 INFO:tasks.workunit.client.1.smithi111.stdout:3/613: dread d0/d1a/d2d/f45 [0,4194304] 0 2024-03-23T11:47:40.966 INFO:tasks.workunit.client.1.smithi111.stdout:0/655: link d5/c12 d5/db/d1f/d19/dbd/cdc 0 2024-03-23T11:47:40.966 INFO:tasks.workunit.client.0.smithi063.stdout:4/583: dread d5/ff [0,4194304] 0 2024-03-23T11:47:40.974 INFO:tasks.workunit.client.1.smithi111.stdout:1/663: dwrite d0/d5/dd/d1e/f21 [4194304,4194304] 0 2024-03-23T11:47:40.974 INFO:tasks.workunit.client.1.smithi111.stdout:9/720: dwrite d0/fe5 [0,4194304] 0 2024-03-23T11:47:40.980 INFO:tasks.workunit.client.1.smithi111.stdout:7/658: rename d14/d1b/d1e/d2c/f31 to d14/d1b/d1e/d86/fbe 0 2024-03-23T11:47:40.981 INFO:tasks.workunit.client.0.smithi063.stdout:3/810: dread ff [0,4194304] 0 2024-03-23T11:47:40.981 INFO:tasks.workunit.client.1.smithi111.stdout:4/714: symlink d0/d17/d86/le9 0 2024-03-23T11:47:40.981 INFO:tasks.workunit.client.1.smithi111.stdout:5/716: mknod d1/d3/d6/d30/de6/cea 0 2024-03-23T11:47:40.983 INFO:tasks.workunit.client.0.smithi063.stdout:4/584: write d5/da/d20/d4b/f2d [7375730,53227] 0 2024-03-23T11:47:40.985 INFO:tasks.workunit.client.1.smithi111.stdout:2/730: unlink d2/d11/d17/d1f/da0/fdd 0 2024-03-23T11:47:40.985 INFO:tasks.workunit.client.1.smithi111.stdout:3/614: mkdir d0/d1a/d2b/d38/d7f/dd0 0 2024-03-23T11:47:40.986 INFO:tasks.workunit.client.1.smithi111.stdout:6/703: link d2/d1a/d1c/f24 d2/d1f/d45/d49/d8d/fe9 0 2024-03-23T11:47:40.986 INFO:tasks.workunit.client.0.smithi063.stdout:3/811: write d13/d22/d2d/d3f/fb3 [110966,83343] 0 2024-03-23T11:47:40.989 INFO:tasks.workunit.client.1.smithi111.stdout:7/659: fdatasync d14/d1b/d1e/d2c/d42/f4a 0 2024-03-23T11:47:40.990 INFO:tasks.workunit.client.0.smithi063.stdout:4/585: read d5/da/f19 [1979902,74418] 0 2024-03-23T11:47:40.990 INFO:tasks.workunit.client.1.smithi111.stdout:9/721: dread d0/d4/d1c/d2d/f63 [0,4194304] 0 2024-03-23T11:47:40.991 INFO:tasks.workunit.client.1.smithi111.stdout:1/664: creat d0/d6/d48/d4a/d6c/d96/fe1 x:0 0 0 2024-03-23T11:47:40.993 INFO:tasks.workunit.client.0.smithi063.stdout:3/812: mkdir d13/d22/d2d/d3e/dce/dfb 0 2024-03-23T11:47:40.996 INFO:tasks.workunit.client.1.smithi111.stdout:0/656: dwrite d5/db/d1f/d47/d87/f9d [0,4194304] 0 2024-03-23T11:47:40.998 INFO:tasks.workunit.client.1.smithi111.stdout:5/717: mknod d1/d3/d6/d30/db2/ceb 0 2024-03-23T11:47:41.000 INFO:tasks.workunit.client.0.smithi063.stdout:4/586: chown d5/da/l11 39 1 2024-03-23T11:47:41.006 INFO:tasks.workunit.client.0.smithi063.stdout:4/587: fsync d5/da/d20/d4b/d16/d50/d8e/f97 0 2024-03-23T11:47:41.013 INFO:tasks.workunit.client.1.smithi111.stdout:7/660: dwrite d14/d37/f49 [0,4194304] 0 2024-03-23T11:47:41.014 INFO:tasks.workunit.client.1.smithi111.stdout:1/665: dwrite d0/d6/d48/fa8 [0,4194304] 0 2024-03-23T11:47:41.015 INFO:tasks.workunit.client.1.smithi111.stdout:7/661: readlink d14/d1b/l1d 0 2024-03-23T11:47:41.017 INFO:tasks.workunit.client.0.smithi063.stdout:4/588: mknod d5/da/d20/d4b/d29/d38/d7a/cab 0 2024-03-23T11:47:41.018 INFO:tasks.workunit.client.1.smithi111.stdout:3/615: unlink d0/d1a/d1d/d26/l32 0 2024-03-23T11:47:41.021 INFO:tasks.workunit.client.0.smithi063.stdout:4/589: symlink d5/da/d20/d4b/d29/d38/d7a/d8f/lac 0 2024-03-23T11:47:41.031 INFO:tasks.workunit.client.0.smithi063.stdout:3/813: truncate d13/d22/d2d/d3f/d5b/f65 4471501 0 2024-03-23T11:47:41.032 INFO:tasks.workunit.client.1.smithi111.stdout:0/657: fsync d5/db/f92 0 2024-03-23T11:47:41.032 INFO:tasks.workunit.client.1.smithi111.stdout:2/731: truncate d2/d11/d17/d3e/d6b/d2e/d76/fa7 618588 0 2024-03-23T11:47:41.033 INFO:tasks.workunit.client.1.smithi111.stdout:7/662: dread d14/d37/f9d [0,4194304] 0 2024-03-23T11:47:41.035 INFO:tasks.workunit.client.0.smithi063.stdout:3/814: symlink d13/d22/d2d/d7d/lfc 0 2024-03-23T11:47:41.036 INFO:tasks.workunit.client.1.smithi111.stdout:9/722: dread d0/d4/d1c/d2d/d3b/d42/d4f/d95/fb9 [0,4194304] 0 2024-03-23T11:47:41.039 INFO:tasks.workunit.client.0.smithi063.stdout:3/815: write d13/d22/d2d/d78/f92 [8735628,114733] 0 2024-03-23T11:47:41.041 INFO:tasks.workunit.client.1.smithi111.stdout:9/723: chown d0/d4/d6a/f39 0 1 2024-03-23T11:47:41.045 INFO:tasks.workunit.client.0.smithi063.stdout:3/816: fdatasync d13/d22/d2d/d3f/d58/d6e/d8d/f95 0 2024-03-23T11:47:41.046 INFO:tasks.workunit.client.1.smithi111.stdout:3/616: dwrite d0/fbe [0,4194304] 0 2024-03-23T11:47:41.061 INFO:tasks.workunit.client.1.smithi111.stdout:7/663: dread d14/d1b/f52 [0,4194304] 0 2024-03-23T11:47:41.063 INFO:tasks.workunit.client.1.smithi111.stdout:7/664: chown d14/d1b/d1e/d2c/l75 405 1 2024-03-23T11:47:41.064 INFO:tasks.workunit.client.1.smithi111.stdout:7/665: readlink d14/d1b/l1d 0 2024-03-23T11:47:41.066 INFO:tasks.workunit.client.1.smithi111.stdout:7/666: dread - d14/d37/d54/fb7 zero size 2024-03-23T11:47:41.069 INFO:tasks.workunit.client.0.smithi063.stdout:3/817: dread f9 [0,4194304] 0 2024-03-23T11:47:41.072 INFO:tasks.workunit.client.1.smithi111.stdout:8/595: truncate d5/d26/d2f/d62/d75/f6b 501704 0 2024-03-23T11:47:41.072 INFO:tasks.workunit.client.1.smithi111.stdout:8/596: stat d5/d32/lb0 0 2024-03-23T11:47:41.074 INFO:tasks.workunit.client.0.smithi063.stdout:3/818: rmdir d13/d22/d2d/d5e 0 2024-03-23T11:47:41.076 INFO:tasks.workunit.client.0.smithi063.stdout:4/590: dread d5/da/d20/d4b/d1f/f21 [0,4194304] 0 2024-03-23T11:47:41.079 INFO:tasks.workunit.client.0.smithi063.stdout:4/591: chown d5/da/d20/d4b/d16/d50/d86/da2 24629 1 2024-03-23T11:47:41.082 INFO:tasks.workunit.client.0.smithi063.stdout:2/441: sync 2024-03-23T11:47:41.088 INFO:tasks.workunit.client.1.smithi111.stdout:0/658: mkdir d5/db/d1f/d19/dbd/d64/d91/ddd 0 2024-03-23T11:47:41.089 INFO:tasks.workunit.client.0.smithi063.stdout:4/592: fsync d5/da/d20/d4b/d29/d38/d7a/f8a 0 2024-03-23T11:47:41.092 INFO:tasks.workunit.client.0.smithi063.stdout:2/442: write d2/d1c/f3c [6413866,91455] 0 2024-03-23T11:47:41.095 INFO:tasks.workunit.client.1.smithi111.stdout:9/724: symlink d0/d4/d1c/d2d/d3b/d42/d4f/d95/lee 0 2024-03-23T11:47:41.098 INFO:tasks.workunit.client.0.smithi063.stdout:2/443: mkdir d2/d23/d56/d8e 0 2024-03-23T11:47:41.100 INFO:tasks.workunit.client.1.smithi111.stdout:7/667: creat d14/d44/d4b/fbf x:0 0 0 2024-03-23T11:47:41.100 INFO:tasks.workunit.client.1.smithi111.stdout:5/718: link d1/d3/d6/d7/d63/d2c/f34 d1/d3/d6/d7/d7b/fec 0 2024-03-23T11:47:41.100 INFO:tasks.workunit.client.0.smithi063.stdout:4/593: mkdir d5/da/d20/d4b/d16/d50/d8e/dad 0 2024-03-23T11:47:41.101 INFO:tasks.workunit.client.0.smithi063.stdout:3/819: dwrite d13/d22/fe6 [0,4194304] 0 2024-03-23T11:47:41.101 INFO:tasks.workunit.client.1.smithi111.stdout:8/597: truncate d5/fe 2009782 0 2024-03-23T11:47:41.105 INFO:tasks.workunit.client.1.smithi111.stdout:0/659: creat d5/db/d4d/da6/fde x:0 0 0 2024-03-23T11:47:41.106 INFO:tasks.workunit.client.0.smithi063.stdout:2/444: mknod d2/d1c/c8f 0 2024-03-23T11:47:41.109 INFO:tasks.workunit.client.0.smithi063.stdout:3/820: rmdir d13/d22/d2d/d3f/de0 39 2024-03-23T11:47:41.111 INFO:tasks.workunit.client.1.smithi111.stdout:0/660: read d5/db/d1f/d48/f6a [484847,17295] 0 2024-03-23T11:47:41.120 INFO:tasks.workunit.client.1.smithi111.stdout:4/715: rename d0/d7 to d0/d95/d9b/db3/dcc/dea 0 2024-03-23T11:47:41.120 INFO:tasks.workunit.client.0.smithi063.stdout:3/821: creat d13/de5/ffd x:0 0 0 2024-03-23T11:47:41.123 INFO:tasks.workunit.client.1.smithi111.stdout:7/668: symlink d14/d1b/d1e/d87/lc0 0 2024-03-23T11:47:41.124 INFO:tasks.workunit.client.1.smithi111.stdout:9/725: dwrite d0/d4/d1c/d2d/d3b/d42/d4f/f61 [0,4194304] 0 2024-03-23T11:47:41.124 INFO:tasks.workunit.client.1.smithi111.stdout:0/661: mkdir d5/db/d1f/d47/d87/ddf 0 2024-03-23T11:47:41.127 INFO:tasks.workunit.client.1.smithi111.stdout:0/662: chown d5/db/d1f/d48/c9f 19 1 2024-03-23T11:47:41.129 INFO:tasks.workunit.client.1.smithi111.stdout:5/719: creat d1/d20/d5c/dc5/dd3/fed x:0 0 0 2024-03-23T11:47:41.131 INFO:tasks.workunit.client.1.smithi111.stdout:6/704: rename d2/d1f/d45/f76 to d2/d35/fea 0 2024-03-23T11:47:41.132 INFO:tasks.workunit.client.0.smithi063.stdout:4/594: dwrite d5/d1e/d2b/f7c [0,4194304] 0 2024-03-23T11:47:41.136 INFO:tasks.workunit.client.0.smithi063.stdout:2/445: dwrite d2/d23/d2e/f36 [0,4194304] 0 2024-03-23T11:47:41.137 INFO:tasks.workunit.client.1.smithi111.stdout:7/669: truncate d14/d1b/f52 5482930 0 2024-03-23T11:47:41.138 INFO:tasks.workunit.client.0.smithi063.stdout:3/822: rename d13/d22/d2d/d3f/d58/d6e/fbc to d13/d22/d2d/d3f/d67/d7b/ffe 0 2024-03-23T11:47:41.138 INFO:tasks.workunit.client.1.smithi111.stdout:6/705: dread d2/d1a/d1c/d2c/fb9 [0,4194304] 0 2024-03-23T11:47:41.138 INFO:tasks.workunit.client.1.smithi111.stdout:5/720: dread d1/d3/d6/d8b/fa3 [0,4194304] 0 2024-03-23T11:47:41.141 INFO:tasks.workunit.client.1.smithi111.stdout:6/706: chown d2/d35/de3/c6e 280603 1 2024-03-23T11:47:41.141 INFO:tasks.workunit.client.1.smithi111.stdout:1/666: write d0/d5/dd/d1e/d54/d87/d8e/fcd [126342,19270] 0 2024-03-23T11:47:41.141 INFO:tasks.workunit.client.0.smithi063.stdout:2/446: read d2/d23/d2e/f36 [2441929,53560] 0 2024-03-23T11:47:41.142 INFO:tasks.workunit.client.0.smithi063.stdout:3/823: stat d13/d22/d2d/d3f/d58/d6e/d8d/f95 0 2024-03-23T11:47:41.143 INFO:tasks.workunit.client.1.smithi111.stdout:9/726: mkdir d0/d31/d41/d71/d8f/def 0 2024-03-23T11:47:41.144 INFO:tasks.workunit.client.1.smithi111.stdout:7/670: chown d14/d1b/d1e/d2c/d42/d5e/d9e/lb5 602013 1 2024-03-23T11:47:41.145 INFO:tasks.workunit.client.1.smithi111.stdout:9/727: dread - d0/d4/d1c/d2d/d3b/d42/fdf zero size 2024-03-23T11:47:41.147 INFO:tasks.workunit.client.1.smithi111.stdout:7/671: truncate d14/d44/d70/f7c 273190 0 2024-03-23T11:47:41.148 INFO:tasks.workunit.client.1.smithi111.stdout:3/617: rename d0/d1a/d2d/lbd to d0/d1a/d2b/ld1 0 2024-03-23T11:47:41.156 INFO:tasks.workunit.client.1.smithi111.stdout:5/721: mkdir d1/d3/d6/d7/dee 0 2024-03-23T11:47:41.157 INFO:tasks.workunit.client.1.smithi111.stdout:3/618: truncate d0/d1a/d2b/d38/dc2/fcc 839936 0 2024-03-23T11:47:41.158 INFO:tasks.workunit.client.0.smithi063.stdout:3/824: link d13/d22/d2d/d3e/dce/f62 d13/d22/d2d/d3f/d58/fff 0 2024-03-23T11:47:41.158 INFO:tasks.workunit.client.0.smithi063.stdout:2/447: getdents d2/de/d7a/d77 0 2024-03-23T11:47:41.164 INFO:tasks.workunit.client.1.smithi111.stdout:5/722: dread d1/d3/d6/d7/f5a [0,4194304] 0 2024-03-23T11:47:41.164 INFO:tasks.workunit.client.0.smithi063.stdout:2/448: truncate d2/de/f1b 1853377 0 2024-03-23T11:47:41.167 INFO:tasks.workunit.client.0.smithi063.stdout:3/825: getdents d13/d22/d2d 0 2024-03-23T11:47:41.168 INFO:tasks.workunit.client.1.smithi111.stdout:7/672: dwrite d14/d37/d43/f55 [4194304,4194304] 0 2024-03-23T11:47:41.169 INFO:tasks.workunit.client.1.smithi111.stdout:9/728: creat d0/d4/d1c/d2d/d3b/d42/d4f/d95/ff0 x:0 0 0 2024-03-23T11:47:41.169 INFO:tasks.workunit.client.0.smithi063.stdout:3/826: chown d13/d15/c37 613 1 2024-03-23T11:47:41.172 INFO:tasks.workunit.client.0.smithi063.stdout:3/827: chown d13/d22/d2d/d3f/d58/d6e/f7a 61725 1 2024-03-23T11:47:41.173 INFO:tasks.workunit.client.1.smithi111.stdout:9/729: readlink d0/d4/d1c/d2d/d3b/d42/d4f/d95/l98 0 2024-03-23T11:47:41.175 INFO:tasks.workunit.client.1.smithi111.stdout:1/667: dwrite d0/d6/f22 [4194304,4194304] 0 2024-03-23T11:47:41.176 INFO:tasks.workunit.client.1.smithi111.stdout:9/730: stat d0/d4/d1c/d2d/f63 0 2024-03-23T11:47:41.182 INFO:tasks.workunit.client.1.smithi111.stdout:2/732: write d2/d32/d91/fb9 [954695,115749] 0 2024-03-23T11:47:41.182 INFO:tasks.workunit.client.1.smithi111.stdout:6/707: creat d2/feb x:0 0 0 2024-03-23T11:47:41.184 INFO:tasks.workunit.client.0.smithi063.stdout:3/828: dread d13/d22/d2d/fa7 [0,4194304] 0 2024-03-23T11:47:41.189 INFO:tasks.workunit.client.1.smithi111.stdout:1/668: chown d0/d2b/c7f 155 1 2024-03-23T11:47:41.190 INFO:tasks.workunit.client.0.smithi063.stdout:3/829: mknod d13/d22/d2d/d78/df1/c100 0 2024-03-23T11:47:41.190 INFO:tasks.workunit.client.1.smithi111.stdout:7/673: dwrite d14/d1b/d1e/fa5 [0,4194304] 0 2024-03-23T11:47:41.199 INFO:tasks.workunit.client.1.smithi111.stdout:3/619: dwrite d0/d1a/d4f/d90/f9f [0,4194304] 0 2024-03-23T11:47:41.201 INFO:tasks.workunit.client.1.smithi111.stdout:1/669: fsync d0/d6/f19 0 2024-03-23T11:47:41.201 INFO:tasks.workunit.client.1.smithi111.stdout:3/620: dread - d0/d1a/d4f/d90/fb2 zero size 2024-03-23T11:47:41.203 INFO:tasks.workunit.client.1.smithi111.stdout:1/670: truncate d0/d5/fae 1350419 0 2024-03-23T11:47:41.204 INFO:tasks.workunit.client.1.smithi111.stdout:9/731: creat d0/d4/d1c/d2d/d6e/ff1 x:0 0 0 2024-03-23T11:47:41.208 INFO:tasks.workunit.client.1.smithi111.stdout:4/716: getdents d0 0 2024-03-23T11:47:41.208 INFO:tasks.workunit.client.1.smithi111.stdout:7/674: mkdir d14/d1b/d1e/d2c/d42/dc1 0 2024-03-23T11:47:41.208 INFO:tasks.workunit.client.1.smithi111.stdout:9/732: chown d0/d4/d6a/d89/ca4 62 1 2024-03-23T11:47:41.211 INFO:tasks.workunit.client.1.smithi111.stdout:6/708: link d2/d35/d70/fc4 d2/d1f/dce/fec 0 2024-03-23T11:47:41.214 INFO:tasks.workunit.client.1.smithi111.stdout:8/598: dwrite d5/d26/d31/d3b/f3d [4194304,4194304] 0 2024-03-23T11:47:41.217 INFO:tasks.workunit.client.1.smithi111.stdout:3/621: chown d0/d7/f23 756 1 2024-03-23T11:47:41.221 INFO:tasks.workunit.client.1.smithi111.stdout:1/671: truncate d0/f1a 203165 0 2024-03-23T11:47:41.221 INFO:tasks.workunit.client.1.smithi111.stdout:3/622: write d0/d1a/d1d/d26/f46 [6892898,42365] 0 2024-03-23T11:47:41.225 INFO:tasks.workunit.client.1.smithi111.stdout:1/672: chown d0/d5/dd/fc0 6 1 2024-03-23T11:47:41.228 INFO:tasks.workunit.client.1.smithi111.stdout:9/733: creat d0/d4/d10/ff2 x:0 0 0 2024-03-23T11:47:41.231 INFO:tasks.workunit.client.1.smithi111.stdout:7/675: creat d14/d1b/d1e/d2c/d2e/d73/fc2 x:0 0 0 2024-03-23T11:47:41.232 INFO:tasks.workunit.client.1.smithi111.stdout:6/709: unlink d2/d1a/c79 0 2024-03-23T11:47:41.233 INFO:tasks.workunit.client.0.smithi063.stdout:2/449: dwrite d2/d23/d2e/d87/d4d/f6d [0,4194304] 0 2024-03-23T11:47:41.235 INFO:tasks.workunit.client.0.smithi063.stdout:3/830: dwrite d13/d22/d2d/d7d/fc3 [0,4194304] 0 2024-03-23T11:47:41.239 INFO:tasks.workunit.client.1.smithi111.stdout:4/717: fdatasync d0/dc/d4c/f6d 0 2024-03-23T11:47:41.239 INFO:tasks.workunit.client.0.smithi063.stdout:3/831: write d13/d22/d2d/fb7 [1299297,53745] 0 2024-03-23T11:47:41.242 INFO:tasks.workunit.client.0.smithi063.stdout:2/450: dread - d2/d23/d2e/d52/d6c/f7c zero size 2024-03-23T11:47:41.243 INFO:tasks.workunit.client.1.smithi111.stdout:1/673: fdatasync d0/d5/dd/d1e/d54/f60 0 2024-03-23T11:47:41.244 INFO:tasks.workunit.client.0.smithi063.stdout:2/451: fdatasync d2/d23/d2e/d52/d6c/f88 0 2024-03-23T11:47:41.247 INFO:tasks.workunit.client.1.smithi111.stdout:9/734: readlink d0/d31/da0/lde 0 2024-03-23T11:47:41.247 INFO:tasks.workunit.client.1.smithi111.stdout:0/663: truncate d5/db/d1f/d82/fac 171586 0 2024-03-23T11:47:41.250 INFO:tasks.workunit.client.1.smithi111.stdout:1/674: dread d0/d2b/d7c/f9f [0,4194304] 0 2024-03-23T11:47:41.252 INFO:tasks.workunit.client.1.smithi111.stdout:9/735: dread d0/f48 [4194304,4194304] 0 2024-03-23T11:47:41.253 INFO:tasks.workunit.client.0.smithi063.stdout:2/452: symlink d2/l90 0 2024-03-23T11:47:41.259 INFO:tasks.workunit.client.0.smithi063.stdout:2/453: mkdir d2/de/d22/d91 0 2024-03-23T11:47:41.260 INFO:tasks.workunit.client.1.smithi111.stdout:3/623: getdents d0/d1a/d2b/d38/d7f/dd0 0 2024-03-23T11:47:41.263 INFO:tasks.workunit.client.1.smithi111.stdout:2/733: dread d2/d32/f7d [0,4194304] 0 2024-03-23T11:47:41.266 INFO:tasks.workunit.client.1.smithi111.stdout:3/624: write d0/d1a/d2b/d38/d7f/fc3 [167037,124035] 0 2024-03-23T11:47:41.267 INFO:tasks.workunit.client.1.smithi111.stdout:9/736: unlink d0/d31/da0/lde 0 2024-03-23T11:47:41.271 INFO:tasks.workunit.client.1.smithi111.stdout:5/723: dwrite d1/d3/d6/d8b/d59/d74/f73 [0,4194304] 0 2024-03-23T11:47:41.272 INFO:tasks.workunit.client.1.smithi111.stdout:1/675: truncate d0/d5/f5e 1645303 0 2024-03-23T11:47:41.274 INFO:tasks.workunit.client.0.smithi063.stdout:3/832: dwrite d13/d22/d2d/f45 [4194304,4194304] 0 2024-03-23T11:47:41.283 INFO:tasks.workunit.client.1.smithi111.stdout:4/718: link d0/d95/f9d d0/dc/d4c/d2a/d97/d4f/dba/feb 0 2024-03-23T11:47:41.283 INFO:tasks.workunit.client.0.smithi063.stdout:2/454: dread d2/d23/d2e/d52/f6f [0,4194304] 0 2024-03-23T11:47:41.284 INFO:tasks.workunit.client.1.smithi111.stdout:4/719: fsync d0/d17/fde 0 2024-03-23T11:47:41.287 INFO:tasks.workunit.client.1.smithi111.stdout:3/625: dwrite d0/d1a/d1d/f6e [0,4194304] 0 2024-03-23T11:47:41.291 INFO:tasks.workunit.client.0.smithi063.stdout:2/455: mkdir d2/d84/d92 0 2024-03-23T11:47:41.293 INFO:tasks.workunit.client.0.smithi063.stdout:3/833: link d13/d22/d2d/d3f/d58/d86/dad/fb6 d13/d66/dde/f101 0 2024-03-23T11:47:41.296 INFO:tasks.workunit.client.0.smithi063.stdout:2/456: creat d2/de/d7a/d77/f93 x:0 0 0 2024-03-23T11:47:41.297 INFO:tasks.workunit.client.1.smithi111.stdout:9/737: mkdir d0/d4/d1c/d36/d28/df3 0 2024-03-23T11:47:41.297 INFO:tasks.workunit.client.0.smithi063.stdout:2/457: fsync d2/f2f 0 2024-03-23T11:47:41.299 INFO:tasks.workunit.client.1.smithi111.stdout:9/738: chown d0/d4/d10/l1f 2363065 1 2024-03-23T11:47:41.300 INFO:tasks.workunit.client.0.smithi063.stdout:3/834: truncate d13/d22/d2d/d3f/d58/f70 4599376 0 2024-03-23T11:47:41.301 INFO:tasks.workunit.client.1.smithi111.stdout:9/739: truncate d0/d4/d1c/d36/d28/f68 5007465 0 2024-03-23T11:47:41.303 INFO:tasks.workunit.client.1.smithi111.stdout:9/740: write d0/d4/d10/d29/d7f/d8e/fb5 [3784455,99213] 0 2024-03-23T11:47:41.307 INFO:tasks.workunit.client.1.smithi111.stdout:4/720: mknod d0/cec 0 2024-03-23T11:47:41.311 INFO:tasks.workunit.client.1.smithi111.stdout:0/664: rename d5/db/d1f/d82 to d5/db/d1f/d47/de0 0 2024-03-23T11:47:41.312 INFO:tasks.workunit.client.1.smithi111.stdout:6/710: dwrite d2/d1a/d1c/d2c/fb9 [0,4194304] 0 2024-03-23T11:47:41.312 INFO:tasks.workunit.client.0.smithi063.stdout:2/458: getdents d2/d23/d2e 0 2024-03-23T11:47:41.317 INFO:tasks.workunit.client.0.smithi063.stdout:2/459: mkdir d2/d84/d94 0 2024-03-23T11:47:41.318 INFO:tasks.workunit.client.1.smithi111.stdout:7/676: truncate f1 1464672 0 2024-03-23T11:47:41.320 INFO:tasks.workunit.client.0.smithi063.stdout:2/460: symlink d2/d23/d2e/d87/d4d/l95 0 2024-03-23T11:47:41.321 INFO:tasks.workunit.client.0.smithi063.stdout:9/552: sync 2024-03-23T11:47:41.321 INFO:tasks.workunit.client.0.smithi063.stdout:7/467: sync 2024-03-23T11:47:41.326 INFO:tasks.workunit.client.1.smithi111.stdout:2/734: dread d2/d11/d17/f21 [0,4194304] 0 2024-03-23T11:47:41.330 INFO:tasks.workunit.client.0.smithi063.stdout:7/468: symlink d5/d6/d39/l99 0 2024-03-23T11:47:41.331 INFO:tasks.workunit.client.1.smithi111.stdout:0/665: unlink d5/f40 0 2024-03-23T11:47:41.337 INFO:tasks.workunit.client.1.smithi111.stdout:7/677: mknod d14/d1b/d1e/d2c/d2e/d73/cc3 0 2024-03-23T11:47:41.338 INFO:tasks.workunit.client.1.smithi111.stdout:9/741: mkdir d0/d31/d41/df4 0 2024-03-23T11:47:41.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.353+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd741119e0 con 0x7efd5c06ad60 2024-03-23T11:47:41.340 INFO:tasks.workunit.client.1.smithi111.stdout:9/742: chown d0/d4/d1c 37 1 2024-03-23T11:47:41.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.355+0000 7efd6e000700 1 -- 172.21.15.63:0/1755346460 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7efd741119e0 con 0x7efd5c06ad60 2024-03-23T11:47:41.340 INFO:tasks.workunit.client.1.smithi111.stdout:4/721: dwrite d0/dc/d4c/f39 [4194304,4194304] 0 2024-03-23T11:47:41.342 INFO:tasks.workunit.client.1.smithi111.stdout:2/735: read d2/de/d12/d70/fb6 [1802853,3833] 0 2024-03-23T11:47:41.343 INFO:tasks.workunit.client.1.smithi111.stdout:9/743: write d0/d31/d41/d65/fcd [128996,121230] 0 2024-03-23T11:47:41.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.359+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7efd5c06ad60 msgr2=0x7efd5c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:41.344 INFO:tasks.workunit.client.0.smithi063.stdout:3/835: dwrite d13/d22/d2d/d3f/d67/d7b/fab [0,4194304] 0 2024-03-23T11:47:41.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.359+0000 7efd7c800700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7efd5c06ad60 0x7efd5c06d210 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efd64009b80 tx=0x7efd6400b040 comp rx=0 tx=0).stop 2024-03-23T11:47:41.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.359+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74070890 msgr2=0x7efd7410fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:41.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.359+0000 7efd7c800700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74070890 0x7efd7410fb00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd68000c00 tx=0x7efd68009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:41.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.360+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 shutdown_connections 2024-03-23T11:47:41.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.360+0000 7efd7c800700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7efd5c06ad60 0x7efd5c06d210 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:41.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.360+0000 7efd7c800700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd74070890 0x7efd7410fb00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:41.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.360+0000 7efd7c800700 1 --2- 172.21.15.63:0/1755346460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd74071e40 0x7efd74110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:41.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.360+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 >> 172.21.15.63:0/1755346460 conn(0x7efd740f8540 msgr2=0x7efd740f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:41.346 INFO:tasks.workunit.client.1.smithi111.stdout:3/626: dread d0/d1a/d2b/f33 [0,4194304] 0 2024-03-23T11:47:41.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.360+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 shutdown_connections 2024-03-23T11:47:41.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.361+0000 7efd7c800700 1 -- 172.21.15.63:0/1755346460 wait complete. 2024-03-23T11:47:41.349 INFO:tasks.workunit.client.0.smithi063.stdout:7/469: symlink d5/d6/d19/d65/l9a 0 2024-03-23T11:47:41.352 INFO:tasks.workunit.client.0.smithi063.stdout:9/553: write d4/d31/d70/d8d/fac [3819546,43579] 0 2024-03-23T11:47:41.358 INFO:tasks.workunit.client.1.smithi111.stdout:5/724: dwrite d1/d20/f72 [0,4194304] 0 2024-03-23T11:47:41.360 INFO:tasks.workunit.client.0.smithi063.stdout:9/554: creat d4/d4c/d19/fbf x:0 0 0 2024-03-23T11:47:41.361 INFO:tasks.workunit.client.0.smithi063.stdout:2/461: dwrite d2/d23/d2e/f4b [0,4194304] 0 2024-03-23T11:47:41.363 INFO:tasks.workunit.client.1.smithi111.stdout:2/736: symlink d2/d32/d98/l106 0 2024-03-23T11:47:41.366 INFO:tasks.workunit.client.0.smithi063.stdout:9/555: dread - d4/fb1 zero size 2024-03-23T11:47:41.368 INFO:tasks.workunit.client.0.smithi063.stdout:7/470: creat d5/d5c/d2f/d73/da/d90/f9b x:0 0 0 2024-03-23T11:47:41.368 INFO:tasks.workunit.client.1.smithi111.stdout:3/627: mknod d0/d53/cd2 0 2024-03-23T11:47:41.368 INFO:tasks.workunit.client.1.smithi111.stdout:1/676: truncate d0/d2b/d69/fbc 1867312 0 2024-03-23T11:47:41.370 INFO:tasks.workunit.client.1.smithi111.stdout:2/737: dread d2/d11/d17/d3e/d5d/fa4 [0,4194304] 0 2024-03-23T11:47:41.372 INFO:tasks.workunit.client.1.smithi111.stdout:7/678: symlink d14/lc4 0 2024-03-23T11:47:41.374 INFO:tasks.workunit.client.0.smithi063.stdout:2/462: truncate d2/d23/d2e/d52/f7d 223207 0 2024-03-23T11:47:41.374 INFO:tasks.workunit.client.0.smithi063.stdout:2/463: chown d2/d23/d2e/d52/f4c 0 1 2024-03-23T11:47:41.377 INFO:tasks.workunit.client.0.smithi063.stdout:3/836: link d13/d66/dde/df5/lfa d13/d22/d2d/d3f/d67/d7b/db0/l102 0 2024-03-23T11:47:41.378 INFO:tasks.workunit.client.1.smithi111.stdout:9/744: rename d0/d31/d35/fbf to d0/d31/d41/d71/d8f/def/ff5 0 2024-03-23T11:47:41.386 INFO:tasks.workunit.client.0.smithi063.stdout:9/556: creat d4/fc0 x:0 0 0 2024-03-23T11:47:41.388 INFO:tasks.workunit.client.1.smithi111.stdout:7/679: mkdir d14/d1b/d1e/d2c/d42/d5e/dc5 0 2024-03-23T11:47:41.390 INFO:tasks.workunit.client.0.smithi063.stdout:9/557: truncate d4/d31/d43/da9/f95 295205 0 2024-03-23T11:47:41.392 INFO:tasks.workunit.client.0.smithi063.stdout:3/837: mkdir d13/d22/d2d/d3f/d103 0 2024-03-23T11:47:41.395 INFO:tasks.workunit.client.1.smithi111.stdout:4/722: dread d0/dc/d30/fab [0,4194304] 0 2024-03-23T11:47:41.396 INFO:tasks.workunit.client.0.smithi063.stdout:6/443: sync 2024-03-23T11:47:41.396 INFO:tasks.workunit.client.0.smithi063.stdout:0/637: sync 2024-03-23T11:47:41.396 INFO:tasks.workunit.client.0.smithi063.stdout:8/783: sync 2024-03-23T11:47:41.396 INFO:tasks.workunit.client.0.smithi063.stdout:1/443: sync 2024-03-23T11:47:41.396 INFO:tasks.workunit.client.0.smithi063.stdout:5/373: sync 2024-03-23T11:47:41.398 INFO:tasks.workunit.client.1.smithi111.stdout:3/628: link d0/d1a/d1d/f6e d0/d1a/d2b/d38/d7f/dd0/fd3 0 2024-03-23T11:47:41.401 INFO:tasks.workunit.client.0.smithi063.stdout:7/471: write d5/d6/d19/d1f/d47/d4f/d5b/f66 [158442,110088] 0 2024-03-23T11:47:41.402 INFO:tasks.workunit.client.0.smithi063.stdout:9/558: dread d4/d31/d43/da9/da6/f45 [0,4194304] 0 2024-03-23T11:47:41.405 INFO:tasks.workunit.client.1.smithi111.stdout:7/680: unlink d14/d1b/fa3 0 2024-03-23T11:47:41.406 INFO:tasks.workunit.client.1.smithi111.stdout:9/745: dread d0/d4/d10/d29/fa3 [0,4194304] 0 2024-03-23T11:47:41.407 INFO:tasks.workunit.client.0.smithi063.stdout:8/784: fsync d6/d1f/d54/d63/d90/f100 0 2024-03-23T11:47:41.407 INFO:tasks.workunit.client.1.smithi111.stdout:5/725: creat d1/d3/d6/d7/d63/d2c/d82/d66/fef x:0 0 0 2024-03-23T11:47:41.408 INFO:tasks.workunit.client.1.smithi111.stdout:7/681: write d14/d1b/f99 [269877,97329] 0 2024-03-23T11:47:41.411 INFO:tasks.workunit.client.1.smithi111.stdout:7/682: dread - d14/d44/d4b/fbf zero size 2024-03-23T11:47:41.411 INFO:tasks.workunit.client.1.smithi111.stdout:4/723: fdatasync d0/d95/db5/dc7/fdc 0 2024-03-23T11:47:41.411 INFO:tasks.workunit.client.1.smithi111.stdout:9/746: read d0/d4/fe [2295064,120463] 0 2024-03-23T11:47:41.412 INFO:tasks.workunit.client.0.smithi063.stdout:3/838: dread f9 [4194304,4194304] 0 2024-03-23T11:47:41.413 INFO:tasks.workunit.client.0.smithi063.stdout:0/638: rename d2/d4/d63 to d2/d14/de8 0 2024-03-23T11:47:41.414 INFO:tasks.workunit.client.1.smithi111.stdout:9/747: write d0/d4/d1c/d2d/d3b/d42/d4f/f61 [715703,102284] 0 2024-03-23T11:47:41.417 INFO:tasks.workunit.client.1.smithi111.stdout:3/629: mkdir d0/d1a/d2b/d38/dd4 0 2024-03-23T11:47:41.418 INFO:tasks.workunit.client.1.smithi111.stdout:3/630: dread - d0/d1a/d2b/fb1 zero size 2024-03-23T11:47:41.423 INFO:tasks.workunit.client.1.smithi111.stdout:5/726: symlink d1/d3/d6/d30/db2/lf0 0 2024-03-23T11:47:41.425 INFO:tasks.workunit.client.1.smithi111.stdout:4/724: stat d0/dc/d4c/d2a/f70 0 2024-03-23T11:47:41.426 INFO:tasks.workunit.client.1.smithi111.stdout:7/683: symlink d14/d1b/d1e/d2c/d2e/lc6 0 2024-03-23T11:47:41.431 INFO:tasks.workunit.client.0.smithi063.stdout:2/464: truncate d2/de/f15 2041665 0 2024-03-23T11:47:41.432 INFO:tasks.workunit.client.0.smithi063.stdout:0/639: symlink d2/d14/d36/d77/d94/da6/d28/d46/le9 0 2024-03-23T11:47:41.433 INFO:tasks.workunit.client.0.smithi063.stdout:2/465: chown d2/l90 7839781 1 2024-03-23T11:47:41.433 INFO:tasks.workunit.client.1.smithi111.stdout:3/631: creat d0/d1a/d2d/fd5 x:0 0 0 2024-03-23T11:47:41.434 INFO:tasks.workunit.client.1.smithi111.stdout:9/748: dwrite d0/d4/d6a/d9a/faa [0,4194304] 0 2024-03-23T11:47:41.437 INFO:tasks.workunit.client.1.smithi111.stdout:1/677: dwrite d0/d5/dd/d1e/f40 [0,4194304] 0 2024-03-23T11:47:41.439 INFO:tasks.workunit.client.1.smithi111.stdout:0/666: dread d5/db/d1f/f79 [4194304,4194304] 0 2024-03-23T11:47:41.441 INFO:tasks.workunit.client.1.smithi111.stdout:5/727: rename d1/d3/d6/d7/d63/d2c/d43/d95/fe0 to d1/d20/d5c/d99/dbf/ff1 0 2024-03-23T11:47:41.442 INFO:tasks.workunit.client.1.smithi111.stdout:2/738: truncate d2/d11/d17/d3e/f40 1404809 0 2024-03-23T11:47:41.445 INFO:tasks.workunit.client.1.smithi111.stdout:5/728: truncate d1/d3/d6/d8b/d59/d6c/fc8 352152 0 2024-03-23T11:47:41.446 INFO:tasks.workunit.client.1.smithi111.stdout:8/599: sync 2024-03-23T11:47:41.447 INFO:tasks.workunit.client.1.smithi111.stdout:8/600: stat d5/d1b/dbc/dc3 0 2024-03-23T11:47:41.448 INFO:tasks.workunit.client.0.smithi063.stdout:6/444: dread d1/dc/fd [4194304,4194304] 0 2024-03-23T11:47:41.450 INFO:tasks.workunit.client.1.smithi111.stdout:7/684: read d14/d1b/d1e/d2c/d2e/d73/dae/f63 [208615,74565] 0 2024-03-23T11:47:41.450 INFO:tasks.workunit.client.1.smithi111.stdout:8/601: chown d5/d26/d2f/d62/db5/f87 28 1 2024-03-23T11:47:41.450 INFO:tasks.workunit.client.1.smithi111.stdout:5/729: truncate d1/d20/d5c/dc5/dd3/fed 1017743 0 2024-03-23T11:47:41.451 INFO:tasks.workunit.client.0.smithi063.stdout:6/445: chown d1/d2a/d41/d42/c7d 180669524 1 2024-03-23T11:47:41.452 INFO:tasks.workunit.client.1.smithi111.stdout:3/632: symlink d0/d1a/d2b/d38/dc2/ld6 0 2024-03-23T11:47:41.452 INFO:tasks.workunit.client.1.smithi111.stdout:5/730: chown d1/d3/d6/d8b/d59/d6c 0 1 2024-03-23T11:47:41.454 INFO:tasks.workunit.client.0.smithi063.stdout:1/444: truncate d3/f2a 2929994 0 2024-03-23T11:47:41.454 INFO:tasks.workunit.client.1.smithi111.stdout:3/633: dread - d0/d1a/d2b/d38/dc2/fc6 zero size 2024-03-23T11:47:41.456 INFO:tasks.workunit.client.1.smithi111.stdout:9/749: mknod d0/d31/d41/d71/cf6 0 2024-03-23T11:47:41.456 INFO:tasks.workunit.client.1.smithi111.stdout:3/634: fsync d0/d1a/d1d/d26/d5a/f74 0 2024-03-23T11:47:41.461 INFO:tasks.workunit.client.0.smithi063.stdout:5/374: write d2/d7/d8/de/d2a/d3b/f73 [1652646,48718] 0 2024-03-23T11:47:41.468 INFO:tasks.workunit.client.1.smithi111.stdout:2/739: creat d2/de/d12/f107 x:0 0 0 2024-03-23T11:47:41.469 INFO:tasks.workunit.client.1.smithi111.stdout:9/750: dwrite d0/d4/d6a/da7/fd5 [0,4194304] 0 2024-03-23T11:47:41.469 INFO:tasks.workunit.client.0.smithi063.stdout:4/595: sync 2024-03-23T11:47:41.470 INFO:tasks.workunit.client.0.smithi063.stdout:4/596: chown f2 17 1 2024-03-23T11:47:41.471 INFO:tasks.workunit.client.1.smithi111.stdout:4/725: creat d0/dc/d30/fed x:0 0 0 2024-03-23T11:47:41.474 INFO:tasks.workunit.client.1.smithi111.stdout:7/685: rename d14/d1b/d1e/d2c/d42/d5e/d9e/fb8 to d14/d44/d70/d7e/dbc/fc7 0 2024-03-23T11:47:41.477 INFO:tasks.workunit.client.1.smithi111.stdout:3/635: creat d0/d1a/d1d/fd7 x:0 0 0 2024-03-23T11:47:41.482 INFO:tasks.workunit.client.1.smithi111.stdout:4/726: unlink d0/dc/d4c/d2a/f38 0 2024-03-23T11:47:41.487 INFO:tasks.workunit.client.1.smithi111.stdout:2/740: mknod d2/d11/d17/c108 0 2024-03-23T11:47:41.489 INFO:tasks.workunit.client.1.smithi111.stdout:2/741: chown d2/de/d12/d70 3498026 1 2024-03-23T11:47:41.490 INFO:tasks.workunit.client.1.smithi111.stdout:1/678: getdents d0/d5/dd/d1e/d54/d87/d8e 0 2024-03-23T11:47:41.494 INFO:tasks.workunit.client.1.smithi111.stdout:3/636: dwrite d0/d1a/d2d/fbb [0,4194304] 0 2024-03-23T11:47:41.494 INFO:tasks.workunit.client.1.smithi111.stdout:0/667: getdents d5/db/d1f/d19/dbd/d8b 0 2024-03-23T11:47:41.495 INFO:tasks.workunit.client.1.smithi111.stdout:0/668: dread - d5/f85 zero size 2024-03-23T11:47:41.497 INFO:tasks.workunit.client.0.smithi063.stdout:9/559: creat d4/d31/d43/da9/da6/db5/fc1 x:0 0 0 2024-03-23T11:47:41.497 INFO:tasks.workunit.client.1.smithi111.stdout:1/679: unlink d0/d5/dd/d1e/d54/l7e 0 2024-03-23T11:47:41.497 INFO:tasks.workunit.client.1.smithi111.stdout:4/727: dread d0/d95/d9b/db3/dcc/dea/da/f46 [0,4194304] 0 2024-03-23T11:47:41.501 INFO:tasks.workunit.client.1.smithi111.stdout:2/742: mkdir d2/d11/d17/d109 0 2024-03-23T11:47:41.502 INFO:tasks.workunit.client.0.smithi063.stdout:7/472: mkdir d5/d6/d19/d1f/d6c/d9c 0 2024-03-23T11:47:41.504 INFO:tasks.workunit.client.1.smithi111.stdout:4/728: truncate d0/d95/d9b/db3/db6/fa3 30637 0 2024-03-23T11:47:41.504 INFO:tasks.workunit.client.1.smithi111.stdout:8/602: write d5/d26/d2f/d62/f6f [421439,92737] 0 2024-03-23T11:47:41.505 INFO:tasks.workunit.client.0.smithi063.stdout:8/785: creat d6/f108 x:0 0 0 2024-03-23T11:47:41.507 INFO:tasks.workunit.client.0.smithi063.stdout:3/839: rename d13/d22/d2d/d3f/d67/d7b/fb2 to d13/d22/d2d/d3f/de0/f104 0 2024-03-23T11:47:41.509 INFO:tasks.workunit.client.1.smithi111.stdout:2/743: dread - d2/d32/ff2 zero size 2024-03-23T11:47:41.511 INFO:tasks.workunit.client.1.smithi111.stdout:1/680: symlink d0/d5/dd/d1e/d70/le2 0 2024-03-23T11:47:41.514 INFO:tasks.workunit.client.0.smithi063.stdout:7/473: write d5/d6/d19/d1f/d47/d4f/d5b/f66 [224504,96799] 0 2024-03-23T11:47:41.515 INFO:tasks.workunit.client.1.smithi111.stdout:3/637: dwrite d0/d1a/d1d/d26/f7e [0,4194304] 0 2024-03-23T11:47:41.519 INFO:tasks.workunit.client.1.smithi111.stdout:5/731: dwrite d1/d3/d6/d7/d7b/fec [0,4194304] 0 2024-03-23T11:47:41.522 INFO:tasks.workunit.client.0.smithi063.stdout:5/375: creat d2/d7/d27/f83 x:0 0 0 2024-03-23T11:47:41.526 INFO:tasks.workunit.client.1.smithi111.stdout:8/603: creat d5/d26/d31/d4d/fc4 x:0 0 0 2024-03-23T11:47:41.530 INFO:tasks.workunit.client.0.smithi063.stdout:3/840: fdatasync d13/d22/f3d 0 2024-03-23T11:47:41.531 INFO:tasks.workunit.client.0.smithi063.stdout:2/466: creat d2/d1c/f96 x:0 0 0 2024-03-23T11:47:41.531 INFO:tasks.workunit.client.0.smithi063.stdout:4/597: mknod d5/da/d20/d4b/d16/d50/d8e/dad/cae 0 2024-03-23T11:47:41.532 INFO:tasks.workunit.client.0.smithi063.stdout:5/376: dread - d2/d28/d3a/d46/d79/f80 zero size 2024-03-23T11:47:41.534 INFO:tasks.workunit.client.0.smithi063.stdout:0/640: getdents d2/d14/d36/d77/d94 0 2024-03-23T11:47:41.535 INFO:tasks.workunit.client.1.smithi111.stdout:2/744: creat d2/d11/d17/d3e/d5d/d8c/f10a x:0 0 0 2024-03-23T11:47:41.535 INFO:tasks.workunit.client.1.smithi111.stdout:0/669: creat d5/fe1 x:0 0 0 2024-03-23T11:47:41.535 INFO:tasks.workunit.client.1.smithi111.stdout:3/638: unlink d0/d55/c6b 0 2024-03-23T11:47:41.539 INFO:tasks.workunit.client.1.smithi111.stdout:4/729: dwrite d0/d17/f8c [0,4194304] 0 2024-03-23T11:47:41.543 INFO:tasks.workunit.client.1.smithi111.stdout:7/686: dread f1 [0,4194304] 0 2024-03-23T11:47:41.554 INFO:tasks.workunit.client.1.smithi111.stdout:4/730: creat d0/d95/d9b/db3/dcc/dea/fee x:0 0 0 2024-03-23T11:47:41.555 INFO:tasks.workunit.client.1.smithi111.stdout:5/732: mkdir d1/d3/d6/d7/d63/dd9/df2 0 2024-03-23T11:47:41.556 INFO:tasks.workunit.client.0.smithi063.stdout:9/560: dwrite d4/d31/d43/da9/f80 [4194304,4194304] 0 2024-03-23T11:47:41.556 INFO:tasks.workunit.client.0.smithi063.stdout:8/786: getdents d6/d1f/d54 0 2024-03-23T11:47:41.559 INFO:tasks.workunit.client.0.smithi063.stdout:8/787: chown d6/c58 2767995 1 2024-03-23T11:47:41.562 INFO:tasks.workunit.client.1.smithi111.stdout:7/687: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb4 [0,4194304] 0 2024-03-23T11:47:41.562 INFO:tasks.workunit.client.1.smithi111.stdout:8/604: fdatasync d5/d26/d31/f64 0 2024-03-23T11:47:41.565 INFO:tasks.workunit.client.1.smithi111.stdout:3/639: unlink d0/d7/f47 0 2024-03-23T11:47:41.569 INFO:tasks.workunit.client.1.smithi111.stdout:6/711: sync 2024-03-23T11:47:41.573 INFO:tasks.workunit.client.0.smithi063.stdout:0/641: dwrite d2/d12/f50 [0,4194304] 0 2024-03-23T11:47:41.576 INFO:tasks.workunit.client.0.smithi063.stdout:3/841: link lb d13/d22/d2d/l105 0 2024-03-23T11:47:41.579 INFO:tasks.workunit.client.0.smithi063.stdout:8/788: fdatasync d6/d1f/d54/d5c/d76/f92 0 2024-03-23T11:47:41.579 INFO:tasks.workunit.client.0.smithi063.stdout:5/377: creat d2/f84 x:0 0 0 2024-03-23T11:47:41.582 INFO:tasks.workunit.client.0.smithi063.stdout:0/642: read d2/d6e/d8c/fe4 [570335,9196] 0 2024-03-23T11:47:41.582 INFO:tasks.workunit.client.1.smithi111.stdout:5/733: dwrite d1/d3/d3a/fd6 [0,4194304] 0 2024-03-23T11:47:41.589 INFO:tasks.workunit.client.0.smithi063.stdout:2/467: getdents d2/d46 0 2024-03-23T11:47:41.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.608+0000 7fa529a00700 1 -- 172.21.15.63:0/2550532577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5240662e0 msgr2=0x7fa524101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:41.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.608+0000 7fa529a00700 1 --2- 172.21.15.63:0/2550532577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5240662e0 0x7fa524101dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009ce0 tx=0x7fa51c004d10 comp rx=0 tx=0).stop 2024-03-23T11:47:41.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.609+0000 7fa529a00700 1 -- 172.21.15.63:0/2550532577 shutdown_connections 2024-03-23T11:47:41.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.609+0000 7fa529a00700 1 --2- 172.21.15.63:0/2550532577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 0x7fa5241046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:41.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.609+0000 7fa529a00700 1 --2- 172.21.15.63:0/2550532577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5240662e0 0x7fa524101dc0 secure :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009ce0 tx=0x7fa51c004d10 comp rx=0 tx=0).stop 2024-03-23T11:47:41.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.609+0000 7fa529a00700 1 -- 172.21.15.63:0/2550532577 >> 172.21.15.63:0/2550532577 conn(0x7fa5240f7430 msgr2=0x7fa5240f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:41.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.609+0000 7fa529a00700 1 -- 172.21.15.63:0/2550532577 shutdown_connections 2024-03-23T11:47:41.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.609+0000 7fa529a00700 1 -- 172.21.15.63:0/2550532577 wait complete. 2024-03-23T11:47:41.595 INFO:tasks.workunit.client.1.smithi111.stdout:8/605: chown d5/c45 64786 1 2024-03-23T11:47:41.595 INFO:tasks.workunit.client.1.smithi111.stdout:9/751: sync 2024-03-23T11:47:41.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.610+0000 7fa529a00700 1 Processor -- start 2024-03-23T11:47:41.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.611+0000 7fa529a00700 1 -- start start 2024-03-23T11:47:41.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.611+0000 7fa529a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 0x7fa52410ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:41.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.611+0000 7fa529a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa524110470 0x7fa524111250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:41.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.611+0000 7fa529a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5241109c0 con 0x7fa524110470 2024-03-23T11:47:41.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.611+0000 7fa529a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa524110b00 con 0x7fa524102300 2024-03-23T11:47:41.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.612+0000 7fa522a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 0x7fa52410ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:41.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.612+0000 7fa522a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 0x7fa52410ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:32982/0 (socket says 172.21.15.63:32982) 2024-03-23T11:47:41.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.612+0000 7fa522a00700 1 -- 172.21.15.63:0/2890936963 learned_addr learned my addr 172.21.15.63:0/2890936963 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:41.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.613+0000 7fa522000700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa524110470 0x7fa524111250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:41.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.613+0000 7fa522a00700 1 -- 172.21.15.63:0/2890936963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa524110470 msgr2=0x7fa524111250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:41.599 INFO:tasks.workunit.client.1.smithi111.stdout:7/688: rename d14/d1b/d1e/d2c/d2e/cb6 to d14/d37/d43/cc8 0 2024-03-23T11:47:41.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.614+0000 7fa522a00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa524110470 0x7fa524111250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:41.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.614+0000 7fa522a00700 1 -- 172.21.15.63:0/2890936963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c009940 con 0x7fa524102300 2024-03-23T11:47:41.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.614+0000 7fa522000700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa524110470 0x7fa524111250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:47:41.600 INFO:tasks.workunit.client.0.smithi063.stdout:9/561: mknod d4/d31/d9c/d9e/cc2 0 2024-03-23T11:47:41.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.616+0000 7fa522a00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 0x7fa52410ff30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa51c000f80 tx=0x7fa51c004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:41.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.617+0000 7fa51aa00700 1 -- 172.21.15.63:0/2890936963 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51c00e040 con 0x7fa524102300 2024-03-23T11:47:41.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.617+0000 7fa51aa00700 1 -- 172.21.15.63:0/2890936963 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa51c007e90 con 0x7fa524102300 2024-03-23T11:47:41.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.617+0000 7fa51aa00700 1 -- 172.21.15.63:0/2890936963 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51c01c6d0 con 0x7fa524102300 2024-03-23T11:47:41.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.618+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5241118d0 con 0x7fa524102300 2024-03-23T11:47:41.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.618+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa524111e90 con 0x7fa524102300 2024-03-23T11:47:41.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.620+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52405f5f0 con 0x7fa524102300 2024-03-23T11:47:41.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.621+0000 7fa51aa00700 1 -- 172.21.15.63:0/2890936963 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa51c012ac0 con 0x7fa524102300 2024-03-23T11:47:41.606 INFO:tasks.workunit.client.1.smithi111.stdout:0/670: creat d5/db/d1f/d19/d45/fe2 x:0 0 0 2024-03-23T11:47:41.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.622+0000 7fa51aa00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa50806ace0 0x7fa50806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:41.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.622+0000 7fa51aa00700 1 -- 172.21.15.63:0/2890936963 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fa51c08aae0 con 0x7fa524102300 2024-03-23T11:47:41.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.622+0000 7fa522000700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa50806ace0 0x7fa50806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:41.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.623+0000 7fa522000700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa50806ace0 0x7fa50806d190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510008040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:41.608 INFO:tasks.workunit.client.1.smithi111.stdout:0/671: stat d5/db/d1f/d19/dbd/c60 0 2024-03-23T11:47:41.609 INFO:tasks.workunit.client.1.smithi111.stdout:0/672: chown d5/db/d1f 147941807 1 2024-03-23T11:47:41.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:41.627+0000 7fa51aa00700 1 -- 172.21.15.63:0/2890936963 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa51c058b80 con 0x7fa524102300 2024-03-23T11:47:41.615 INFO:tasks.workunit.client.0.smithi063.stdout:6/446: write d1/d13/f1c [833575,43916] 0 2024-03-23T11:47:41.618 INFO:tasks.workunit.client.0.smithi063.stdout:5/378: mkdir d2/d7/d8/de/d2a/d3b/d85 0 2024-03-23T11:47:41.620 INFO:tasks.workunit.client.0.smithi063.stdout:5/379: chown d2/d28/f4b 23 1 2024-03-23T11:47:41.621 INFO:tasks.workunit.client.1.smithi111.stdout:1/681: truncate d0/d5/dd/fb1 646648 0 2024-03-23T11:47:41.621 INFO:tasks.workunit.client.0.smithi063.stdout:0/643: mknod d2/d14/d36/d77/d94/da6/d25/cea 0 2024-03-23T11:47:41.622 INFO:tasks.workunit.client.1.smithi111.stdout:5/734: fsync d1/d3/d6/d7/d8c/f8e 0 2024-03-23T11:47:41.624 INFO:tasks.workunit.client.0.smithi063.stdout:2/468: symlink d2/d23/d56/l97 0 2024-03-23T11:47:41.629 INFO:tasks.workunit.client.1.smithi111.stdout:5/735: read d1/d3/d6/d7/d63/fac [247928,99526] 0 2024-03-23T11:47:41.629 INFO:tasks.workunit.client.1.smithi111.stdout:2/745: dwrite d2/d11/d17/d3e/d6b/d9d/fae [0,4194304] 0 2024-03-23T11:47:41.629 INFO:tasks.workunit.client.0.smithi063.stdout:0/644: read - d2/d14/d36/d77/d94/da6/d28/fdf zero size 2024-03-23T11:47:41.630 INFO:tasks.workunit.client.1.smithi111.stdout:5/736: dread - d1/d20/dce/fe4 zero size 2024-03-23T11:47:41.631 INFO:tasks.workunit.client.1.smithi111.stdout:9/752: unlink d0/d4/d10/d29/fac 0 2024-03-23T11:47:41.631 INFO:tasks.workunit.client.0.smithi063.stdout:2/469: dread - d2/d23/d2e/d87/f82 zero size 2024-03-23T11:47:41.635 INFO:tasks.workunit.client.1.smithi111.stdout:1/682: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:41.636 INFO:tasks.workunit.client.0.smithi063.stdout:1/445: dwrite d3/d35/d40/f42 [0,4194304] 0 2024-03-23T11:47:41.638 INFO:tasks.workunit.client.1.smithi111.stdout:8/606: rename d5/d26/d2f/d62/d75/d4b/lbb to d5/d26/d2f/d62/d75/d8d/lc5 0 2024-03-23T11:47:41.641 INFO:tasks.workunit.client.1.smithi111.stdout:3/640: write d0/d1a/d4f/d90/fce [1062978,95867] 0 2024-03-23T11:47:41.646 INFO:tasks.workunit.client.1.smithi111.stdout:0/673: mknod d5/db/d1f/d19/dbd/d64/dc7/ce3 0 2024-03-23T11:47:41.650 INFO:tasks.workunit.client.1.smithi111.stdout:4/731: truncate d0/dc/d4c/d2a/d97/d4d/fbe 399866 0 2024-03-23T11:47:41.652 INFO:tasks.workunit.client.0.smithi063.stdout:2/470: mkdir d2/d23/d2e/d98 0 2024-03-23T11:47:41.652 INFO:tasks.workunit.client.0.smithi063.stdout:8/789: write d6/d1f/d54/fa6 [1999652,112770] 0 2024-03-23T11:47:41.657 INFO:tasks.workunit.client.0.smithi063.stdout:2/471: chown d2/de/d7a/d77/c7f 1104446 1 2024-03-23T11:47:41.659 INFO:tasks.workunit.client.1.smithi111.stdout:9/753: write d0/d4/d10/d29/fa3 [1309674,53123] 0 2024-03-23T11:47:41.659 INFO:tasks.workunit.client.1.smithi111.stdout:5/737: mkdir d1/d3/d6/d8b/d59/d6c/df3 0 2024-03-23T11:47:41.660 INFO:tasks.workunit.client.0.smithi063.stdout:5/380: rename d2/d7/d8/d1d/f58 to d2/d28/d3a/d46/f86 0 2024-03-23T11:47:41.661 INFO:tasks.workunit.client.0.smithi063.stdout:6/447: mkdir d1/d15/d84 0 2024-03-23T11:47:41.666 INFO:tasks.workunit.client.0.smithi063.stdout:1/446: creat d3/d1c/d4b/f8b x:0 0 0 2024-03-23T11:47:41.671 INFO:tasks.workunit.client.0.smithi063.stdout:5/381: dread - d2/d7/d8/d1d/f74 zero size 2024-03-23T11:47:41.673 INFO:tasks.workunit.client.0.smithi063.stdout:0/645: dread d2/d12/d32/f5c [0,4194304] 0 2024-03-23T11:47:41.674 INFO:tasks.workunit.client.1.smithi111.stdout:4/732: dread d0/dc/d1f/db0/fa7 [0,4194304] 0 2024-03-23T11:47:41.675 INFO:tasks.workunit.client.1.smithi111.stdout:4/733: stat d0/dc/d4c/c44 0 2024-03-23T11:47:41.676 INFO:tasks.workunit.client.0.smithi063.stdout:1/447: readlink d3/l6f 0 2024-03-23T11:47:41.680 INFO:tasks.workunit.client.0.smithi063.stdout:4/598: dwrite d5/da/d20/d4b/f68 [0,4194304] 0 2024-03-23T11:47:41.680 INFO:tasks.workunit.client.1.smithi111.stdout:4/734: read d0/d95/d9b/db3/dcc/dea/f7f [3317258,119912] 0 2024-03-23T11:47:41.682 INFO:tasks.workunit.client.1.smithi111.stdout:2/746: creat d2/d11/d17/d3e/dee/f10b x:0 0 0 2024-03-23T11:47:41.683 INFO:tasks.workunit.client.1.smithi111.stdout:1/683: mkdir d0/d2b/de3 0 2024-03-23T11:47:41.685 INFO:tasks.workunit.client.1.smithi111.stdout:2/747: chown d2/d11/d17/d1f/da0/dd2 39 1 2024-03-23T11:47:41.685 INFO:tasks.workunit.client.0.smithi063.stdout:3/842: mkdir d13/d22/d2d/d78/dcc/d106 0 2024-03-23T11:47:41.696 INFO:tasks.workunit.client.0.smithi063.stdout:5/382: creat d2/d7/d64/f87 x:0 0 0 2024-03-23T11:47:41.696 INFO:tasks.workunit.client.0.smithi063.stdout:1/448: readlink d3/d1c/l4e 0 2024-03-23T11:47:41.698 INFO:tasks.workunit.client.1.smithi111.stdout:6/712: truncate d2/d1a/d1c/d2c/d51/f2f 5341326 0 2024-03-23T11:47:41.699 INFO:tasks.workunit.client.0.smithi063.stdout:1/449: chown d3/f11 285402 1 2024-03-23T11:47:41.705 INFO:tasks.workunit.client.1.smithi111.stdout:7/689: rename d14/f25 to d14/d1b/d1e/d2c/d42/fc9 0 2024-03-23T11:47:41.705 INFO:tasks.workunit.client.0.smithi063.stdout:2/472: rename d2/d23/d56/d8e to d2/d23/d2e/d52/d99 0 2024-03-23T11:47:41.707 INFO:tasks.workunit.client.0.smithi063.stdout:6/448: mkdir d1/dc/d3a/d82/d85 0 2024-03-23T11:47:41.708 INFO:tasks.workunit.client.1.smithi111.stdout:9/754: dread d0/d4/fe [0,4194304] 0 2024-03-23T11:47:41.709 INFO:tasks.workunit.client.0.smithi063.stdout:4/599: dwrite d5/da/d20/d4b/d29/d38/f55 [0,4194304] 0 2024-03-23T11:47:41.711 INFO:tasks.workunit.client.1.smithi111.stdout:9/755: read d0/d4/d1c/d2d/d75/d7a/fc5 [546783,3054] 0 2024-03-23T11:47:41.712 INFO:tasks.workunit.client.1.smithi111.stdout:3/641: symlink d0/d7/d19/d91/da3/ld8 0 2024-03-23T11:47:41.714 INFO:tasks.workunit.client.0.smithi063.stdout:7/474: sync 2024-03-23T11:47:41.714 INFO:tasks.workunit.client.0.smithi063.stdout:0/646: mkdir d2/d14/d36/d77/deb 0 2024-03-23T11:47:41.716 INFO:tasks.workunit.client.0.smithi063.stdout:0/647: write d2/d48/f7f [3899639,62705] 0 2024-03-23T11:47:41.720 INFO:tasks.workunit.client.0.smithi063.stdout:5/383: creat d2/d7/d8/d1d/f88 x:0 0 0 2024-03-23T11:47:41.720 INFO:tasks.workunit.client.1.smithi111.stdout:4/735: mknod d0/dc/d4c/dad/cef 0 2024-03-23T11:47:41.722 INFO:tasks.workunit.client.1.smithi111.stdout:5/738: fdatasync d1/d3/d6/d8b/f26 0 2024-03-23T11:47:41.726 INFO:tasks.workunit.client.1.smithi111.stdout:5/739: fsync d1/d3/d6/d7/d63/d2c/f34 0 2024-03-23T11:47:41.733 INFO:tasks.workunit.client.0.smithi063.stdout:9/562: dwrite d4/d31/d43/d49/f9a [0,4194304] 0 2024-03-23T11:47:41.734 INFO:tasks.workunit.client.0.smithi063.stdout:3/843: fdatasync d13/d22/d2d/d3f/d58/d97/fae 0 2024-03-23T11:47:41.737 INFO:tasks.workunit.client.1.smithi111.stdout:9/756: unlink d0/d4/fbd 0 2024-03-23T11:47:41.737 INFO:tasks.workunit.client.1.smithi111.stdout:3/642: mknod d0/d1a/d1d/d26/d5a/d64/cd9 0 2024-03-23T11:47:41.743 INFO:tasks.workunit.client.0.smithi063.stdout:0/648: dread d2/d4/f9 [0,4194304] 0 2024-03-23T11:47:41.744 INFO:tasks.workunit.client.1.smithi111.stdout:9/757: dread d0/d4/d1c/d2d/d3b/d42/d92/fc7 [0,4194304] 0 2024-03-23T11:47:41.746 INFO:tasks.workunit.client.1.smithi111.stdout:8/607: dread d5/d26/d31/d5e/f49 [0,4194304] 0 2024-03-23T11:47:41.748 INFO:tasks.workunit.client.1.smithi111.stdout:5/740: creat d1/d3/d6/d30/db2/ff4 x:0 0 0 2024-03-23T11:47:41.748 INFO:tasks.workunit.client.0.smithi063.stdout:2/473: chown d2/de/f15 19741610 1 2024-03-23T11:47:41.750 INFO:tasks.workunit.client.0.smithi063.stdout:8/790: write d6/dd/df/f2c [3798149,85814] 0 2024-03-23T11:47:41.752 INFO:tasks.workunit.client.0.smithi063.stdout:2/474: fsync d2/d84/f89 0 2024-03-23T11:47:41.755 INFO:tasks.workunit.client.1.smithi111.stdout:3/643: dwrite d0/d1a/d4f/f4e [0,4194304] 0 2024-03-23T11:47:41.758 INFO:tasks.workunit.client.1.smithi111.stdout:3/644: write d0/d1a/f28 [77397,122652] 0 2024-03-23T11:47:41.760 INFO:tasks.workunit.client.1.smithi111.stdout:3/645: write d0/d1a/d2b/f4d [4729007,12264] 0 2024-03-23T11:47:41.767 INFO:tasks.workunit.client.0.smithi063.stdout:3/844: fsync d13/d15/f1b 0 2024-03-23T11:47:41.767 INFO:tasks.workunit.client.0.smithi063.stdout:9/563: creat d4/d31/d70/d8d/fc3 x:0 0 0 2024-03-23T11:47:41.768 INFO:tasks.workunit.client.0.smithi063.stdout:0/649: dwrite d2/d12/d32/f91 [0,4194304] 0 2024-03-23T11:47:41.776 INFO:tasks.workunit.client.1.smithi111.stdout:0/674: rename d5/db/dbe to d5/db/d1f/de4 0 2024-03-23T11:47:41.778 INFO:tasks.workunit.client.0.smithi063.stdout:6/449: rename d1/c7c to d1/d15/d4b/c86 0 2024-03-23T11:47:41.785 INFO:tasks.workunit.client.1.smithi111.stdout:8/608: symlink d5/d32/d7d/lc6 0 2024-03-23T11:47:41.785 INFO:tasks.workunit.client.1.smithi111.stdout:7/690: link d14/d44/l65 d14/d44/d70/d7e/lca 0 2024-03-23T11:47:41.789 INFO:tasks.workunit.client.0.smithi063.stdout:8/791: mkdir d6/dd/d109 0 2024-03-23T11:47:41.790 INFO:tasks.workunit.client.1.smithi111.stdout:3/646: dread d0/d1a/d2b/d38/d7f/f9c [0,4194304] 0 2024-03-23T11:47:41.794 INFO:tasks.workunit.client.1.smithi111.stdout:2/748: rename d2/d11/d17/d3e/d6b/d2e/f6f to d2/de/d12/d70/f10c 0 2024-03-23T11:47:41.794 INFO:tasks.workunit.client.0.smithi063.stdout:9/564: unlink d4/d4c/c26 0 2024-03-23T11:47:41.797 INFO:tasks.workunit.client.1.smithi111.stdout:0/675: mkdir d5/db/d1f/de4/de5 0 2024-03-23T11:47:41.798 INFO:tasks.workunit.client.0.smithi063.stdout:3/845: creat d13/d22/d2d/d3f/d58/d86/dad/f107 x:0 0 0 2024-03-23T11:47:41.801 INFO:tasks.workunit.client.1.smithi111.stdout:1/684: sync 2024-03-23T11:47:41.801 INFO:tasks.workunit.client.1.smithi111.stdout:8/609: fsync d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 0 2024-03-23T11:47:41.802 INFO:tasks.workunit.client.1.smithi111.stdout:7/691: write d14/f15 [833218,2280] 0 2024-03-23T11:47:41.803 INFO:tasks.workunit.client.1.smithi111.stdout:8/610: stat d5/d26/d2f/d62 0 2024-03-23T11:47:41.806 INFO:tasks.workunit.client.0.smithi063.stdout:6/450: rename d1/dc/d21/d49/f56 to d1/dc/d3a/d57/d77/f87 0 2024-03-23T11:47:41.812 INFO:tasks.workunit.client.1.smithi111.stdout:3/647: rename d0/l3b to d0/d72/d7b/lda 0 2024-03-23T11:47:41.813 INFO:tasks.workunit.client.1.smithi111.stdout:9/758: link d0/d4/d1c/f45 d0/d31/d41/d9d/dd4/ff7 0 2024-03-23T11:47:41.815 INFO:tasks.workunit.client.1.smithi111.stdout:3/648: write d0/d1a/d2b/d38/dc2/fcc [117668,25230] 0 2024-03-23T11:47:41.817 INFO:tasks.workunit.client.1.smithi111.stdout:3/649: write d0/d53/faf [3689581,117172] 0 2024-03-23T11:47:41.819 INFO:tasks.workunit.client.1.smithi111.stdout:1/685: creat d0/d2b/d37/d6a/fe4 x:0 0 0 2024-03-23T11:47:41.821 INFO:tasks.workunit.client.1.smithi111.stdout:1/686: fdatasync d0/d2b/f3e 0 2024-03-23T11:47:41.823 INFO:tasks.workunit.client.1.smithi111.stdout:2/749: rename d2/d32/lbe to d2/de/d6c/dd1/l10d 0 2024-03-23T11:47:41.825 INFO:tasks.workunit.client.0.smithi063.stdout:6/451: read d1/dc/d3a/f40 [3872778,23141] 0 2024-03-23T11:47:41.828 INFO:tasks.workunit.client.1.smithi111.stdout:5/741: dread d1/d3/d6/d30/f70 [0,4194304] 0 2024-03-23T11:47:41.829 INFO:tasks.workunit.client.0.smithi063.stdout:3/846: mknod d13/d22/d2d/d3f/dd4/c108 0 2024-03-23T11:47:41.831 INFO:tasks.workunit.client.0.smithi063.stdout:1/450: dwrite d3/f39 [0,4194304] 0 2024-03-23T11:47:41.832 INFO:tasks.workunit.client.0.smithi063.stdout:9/565: truncate d4/d31/f3d 1592743 0 2024-03-23T11:47:41.833 INFO:tasks.workunit.client.0.smithi063.stdout:5/384: write d2/d7/d8/de/d2a/d3b/d43/f5a [249733,10891] 0 2024-03-23T11:47:41.842 INFO:tasks.workunit.client.1.smithi111.stdout:0/676: chown d5/db/d1f/d32/l86 112311724 1 2024-03-23T11:47:41.844 INFO:tasks.workunit.client.0.smithi063.stdout:9/566: write d4/d31/d43/da9/f95 [954484,57098] 0 2024-03-23T11:47:41.845 INFO:tasks.workunit.client.0.smithi063.stdout:6/452: fdatasync d1/dc/f46 0 2024-03-23T11:47:41.846 INFO:tasks.workunit.client.0.smithi063.stdout:2/475: write d2/d1c/f30 [4803628,66358] 0 2024-03-23T11:47:41.846 INFO:tasks.workunit.client.0.smithi063.stdout:7/475: write d5/d6/d19/d1f/f6d [2897421,44155] 0 2024-03-23T11:47:41.846 INFO:tasks.workunit.client.1.smithi111.stdout:9/759: mkdir d0/d31/d41/d9d/dd4/de0/df8 0 2024-03-23T11:47:41.848 INFO:tasks.workunit.client.1.smithi111.stdout:6/713: write d2/d1a/d1c/f24 [234299,1883] 0 2024-03-23T11:47:41.850 INFO:tasks.workunit.client.0.smithi063.stdout:3/847: write d13/d22/d2d/fb7 [810650,32873] 0 2024-03-23T11:47:41.853 INFO:tasks.workunit.client.0.smithi063.stdout:6/453: stat d1/d2a/d41/f5e 0 2024-03-23T11:47:41.856 INFO:tasks.workunit.client.0.smithi063.stdout:7/476: dread - d5/d6/d19/d1f/d47/d4f/f93 zero size 2024-03-23T11:47:41.856 INFO:tasks.workunit.client.0.smithi063.stdout:6/454: readlink d1/d15/l7a 0 2024-03-23T11:47:41.857 INFO:tasks.workunit.client.0.smithi063.stdout:8/792: chown d6/dd/d59/fad 30098482 1 2024-03-23T11:47:41.861 INFO:tasks.workunit.client.0.smithi063.stdout:1/451: creat d3/d3f/d4d/f8c x:0 0 0 2024-03-23T11:47:41.865 INFO:tasks.workunit.client.0.smithi063.stdout:0/650: link d2/d14/d36/d77/d94/da6/d1f/c9a d2/d48/da5/cec 0 2024-03-23T11:47:41.866 INFO:tasks.workunit.client.0.smithi063.stdout:3/848: creat d13/d22/d2d/d3f/d58/d6e/d8d/f109 x:0 0 0 2024-03-23T11:47:41.867 INFO:tasks.workunit.client.0.smithi063.stdout:5/385: write d2/f2c [160273,81694] 0 2024-03-23T11:47:41.879 INFO:tasks.workunit.client.0.smithi063.stdout:2/476: symlink d2/l9a 0 2024-03-23T11:47:41.882 INFO:tasks.workunit.client.0.smithi063.stdout:2/477: chown d2/de/d22/c2a 59734833 1 2024-03-23T11:47:41.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:41 smithi063 ceph-mon[35189]: pgmap v40: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 77 MiB/s rd, 223 MiB/s wr, 435 op/s 2024-03-23T11:47:41.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:41 smithi063 ceph-mon[35189]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:41.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:41 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:41.887 INFO:tasks.workunit.client.0.smithi063.stdout:7/477: dread d5/d5c/d2f/d73/f3c [0,4194304] 0 2024-03-23T11:47:41.894 INFO:tasks.workunit.client.0.smithi063.stdout:9/567: creat d4/fc4 x:0 0 0 2024-03-23T11:47:41.898 INFO:tasks.workunit.client.0.smithi063.stdout:5/386: truncate d2/d7/d8/d1d/f57 1017280 0 2024-03-23T11:47:41.898 INFO:tasks.workunit.client.0.smithi063.stdout:9/568: read d4/d31/d43/da9/f95 [409970,62812] 0 2024-03-23T11:47:41.899 INFO:tasks.workunit.client.0.smithi063.stdout:6/455: write d1/dc/d3a/f4f [272781,92949] 0 2024-03-23T11:47:41.900 INFO:tasks.workunit.client.0.smithi063.stdout:8/793: write d6/d1c/d32/d37/fcf [777792,125973] 0 2024-03-23T11:47:41.903 INFO:tasks.workunit.client.0.smithi063.stdout:9/569: chown d4/d4c/d19/f23 1084206798 1 2024-03-23T11:47:41.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:41 smithi111 ceph-mon[44005]: pgmap v40: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 77 MiB/s rd, 223 MiB/s wr, 435 op/s 2024-03-23T11:47:41.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:41 smithi111 ceph-mon[44005]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:41.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:41 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:41.906 INFO:tasks.workunit.client.0.smithi063.stdout:2/478: mknod d2/de/d7a/d77/c9b 0 2024-03-23T11:47:41.908 INFO:tasks.workunit.client.0.smithi063.stdout:8/794: chown d6/d1f/d54/fa6 0 1 2024-03-23T11:47:41.910 INFO:tasks.workunit.client.0.smithi063.stdout:7/478: creat d5/d22/f9d x:0 0 0 2024-03-23T11:47:41.912 INFO:tasks.workunit.client.0.smithi063.stdout:1/452: truncate d3/df/f1a 7400659 0 2024-03-23T11:47:41.912 INFO:tasks.workunit.client.1.smithi111.stdout:4/736: truncate d0/d17/f2c 435276 0 2024-03-23T11:47:41.915 INFO:tasks.workunit.client.1.smithi111.stdout:5/742: creat d1/d3/d3a/ff5 x:0 0 0 2024-03-23T11:47:41.916 INFO:tasks.workunit.client.0.smithi063.stdout:1/453: write d3/f10 [4426915,100131] 0 2024-03-23T11:47:41.916 INFO:tasks.workunit.client.1.smithi111.stdout:4/737: write d0/dc/d30/f15 [111817,13658] 0 2024-03-23T11:47:41.918 INFO:tasks.workunit.client.0.smithi063.stdout:0/651: dwrite d2/d14/f66 [0,4194304] 0 2024-03-23T11:47:41.921 INFO:tasks.workunit.client.1.smithi111.stdout:2/750: mkdir d2/d11/d17/d3e/d5d/dc0/df7/d10e 0 2024-03-23T11:47:41.922 INFO:tasks.workunit.client.1.smithi111.stdout:6/714: dwrite d2/d35/fcc [0,4194304] 0 2024-03-23T11:47:41.923 INFO:tasks.workunit.client.1.smithi111.stdout:4/738: rmdir d0 39 2024-03-23T11:47:41.925 INFO:tasks.workunit.client.1.smithi111.stdout:3/650: dwrite d0/d1a/d1d/f96 [0,4194304] 0 2024-03-23T11:47:41.926 INFO:tasks.workunit.client.1.smithi111.stdout:8/611: getdents d5/d26/d2f/d62/d75 0 2024-03-23T11:47:41.926 INFO:tasks.workunit.client.0.smithi063.stdout:2/479: symlink d2/d23/l9c 0 2024-03-23T11:47:41.929 INFO:tasks.workunit.client.1.smithi111.stdout:3/651: chown d0/db9 268335302 1 2024-03-23T11:47:41.931 INFO:tasks.workunit.client.0.smithi063.stdout:2/480: chown d2/de/d7a/d77/c9b 1468 1 2024-03-23T11:47:41.936 INFO:tasks.workunit.client.0.smithi063.stdout:9/570: link d4/fc4 d4/d31/d9c/d9e/fc5 0 2024-03-23T11:47:41.936 INFO:tasks.workunit.client.1.smithi111.stdout:5/743: mknod d1/d3/d6/d7/d63/dd9/df2/cf6 0 2024-03-23T11:47:41.936 INFO:tasks.workunit.client.1.smithi111.stdout:0/677: creat d5/db/d1f/d47/d59/fe6 x:0 0 0 2024-03-23T11:47:41.940 INFO:tasks.workunit.client.1.smithi111.stdout:4/739: rmdir d0/dc/d1f/db0 39 2024-03-23T11:47:41.941 INFO:tasks.workunit.client.0.smithi063.stdout:3/849: link d13/d22/f2f d13/d22/d2d/d3f/d5b/f10a 0 2024-03-23T11:47:41.941 INFO:tasks.workunit.client.1.smithi111.stdout:5/744: symlink d1/d20/d5c/d99/dbf/lf7 0 2024-03-23T11:47:41.944 INFO:tasks.workunit.client.1.smithi111.stdout:8/612: creat d5/d26/d31/fc7 x:0 0 0 2024-03-23T11:47:41.944 INFO:tasks.workunit.client.1.smithi111.stdout:4/740: mknod d0/d95/db5/dc7/cf0 0 2024-03-23T11:47:41.945 INFO:tasks.workunit.client.0.smithi063.stdout:0/652: stat d2/d14/d36/d77/d94/da6/d25/ld4 0 2024-03-23T11:47:41.949 INFO:tasks.workunit.client.1.smithi111.stdout:0/678: rename d5/db/d1f/de4/fc2 to d5/db/d1f/d47/d6f/fe7 0 2024-03-23T11:47:41.950 INFO:tasks.workunit.client.1.smithi111.stdout:0/679: readlink d5/db/d4d/l6d 0 2024-03-23T11:47:41.951 INFO:tasks.workunit.client.0.smithi063.stdout:6/456: creat d1/d15/f88 x:0 0 0 2024-03-23T11:47:41.953 INFO:tasks.workunit.client.1.smithi111.stdout:4/741: symlink d0/dc/d4c/d2a/d97/d4f/dba/lf1 0 2024-03-23T11:47:41.955 INFO:tasks.workunit.client.1.smithi111.stdout:0/680: creat d5/db/d1f/d32/fe8 x:0 0 0 2024-03-23T11:47:41.956 INFO:tasks.workunit.client.0.smithi063.stdout:5/387: link d2/d7/d27/c5b d2/d28/d3a/d46/d61/c89 0 2024-03-23T11:47:41.957 INFO:tasks.workunit.client.0.smithi063.stdout:2/481: mknod d2/d46/d54/c9d 0 2024-03-23T11:47:41.957 INFO:tasks.workunit.client.1.smithi111.stdout:8/613: creat d5/d26/d31/d5e/fc8 x:0 0 0 2024-03-23T11:47:41.961 INFO:tasks.workunit.client.1.smithi111.stdout:5/745: dwrite d1/d3/d6/d7/d63/fe8 [0,4194304] 0 2024-03-23T11:47:41.961 INFO:tasks.workunit.client.1.smithi111.stdout:9/760: dread d0/d4/d6a/f72 [0,4194304] 0 2024-03-23T11:47:41.963 INFO:tasks.workunit.client.1.smithi111.stdout:4/742: write d0/dc/d1f/db0/fdd [804880,97710] 0 2024-03-23T11:47:41.968 INFO:tasks.workunit.client.1.smithi111.stdout:5/746: truncate d1/d3/d6/d7/d63/d2c/d82/fd4 500298 0 2024-03-23T11:47:41.971 INFO:tasks.workunit.client.1.smithi111.stdout:8/614: truncate d5/f1f 1536124 0 2024-03-23T11:47:41.972 INFO:tasks.workunit.client.1.smithi111.stdout:7/692: sync 2024-03-23T11:47:41.972 INFO:tasks.workunit.client.1.smithi111.stdout:1/687: sync 2024-03-23T11:47:41.973 INFO:tasks.workunit.client.0.smithi063.stdout:7/479: dread d5/d26/d76/f78 [0,4194304] 0 2024-03-23T11:47:41.974 INFO:tasks.workunit.client.1.smithi111.stdout:1/688: truncate d0/d2b/fc7 723763 0 2024-03-23T11:47:41.977 INFO:tasks.workunit.client.1.smithi111.stdout:0/681: truncate d5/db/d1f/f28 4522847 0 2024-03-23T11:47:41.978 INFO:tasks.workunit.client.0.smithi063.stdout:9/571: rename d4/d4c/f72 to d4/d31/d70/d8d/fc6 0 2024-03-23T11:47:41.980 INFO:tasks.workunit.client.1.smithi111.stdout:4/743: mkdir d0/dc/d4c/d2a/df2 0 2024-03-23T11:47:41.982 INFO:tasks.workunit.client.1.smithi111.stdout:4/744: write d0/dc/d1f/db0/fa7 [3001916,30835] 0 2024-03-23T11:47:41.984 INFO:tasks.workunit.client.1.smithi111.stdout:9/761: mknod d0/d4/d10/d29/d7f/cf9 0 2024-03-23T11:47:41.986 INFO:tasks.workunit.client.1.smithi111.stdout:9/762: chown d0/d4/d10/d29/la2 11 1 2024-03-23T11:47:41.987 INFO:tasks.workunit.client.1.smithi111.stdout:9/763: stat d0/d4/d1c/d2d/f2e 0 2024-03-23T11:47:41.987 INFO:tasks.workunit.client.1.smithi111.stdout:8/615: rmdir d5/d1b/dbc 39 2024-03-23T11:47:41.989 INFO:tasks.workunit.client.1.smithi111.stdout:8/616: dread - d5/d26/d2f/d42/f57 zero size 2024-03-23T11:47:41.989 INFO:tasks.workunit.client.1.smithi111.stdout:7/693: symlink d14/d1b/d1e/d2c/d42/d5e/d9e/lcb 0 2024-03-23T11:47:41.991 INFO:tasks.workunit.client.1.smithi111.stdout:8/617: chown d5/d26/d31/d4d/l9a 31 1 2024-03-23T11:47:41.991 INFO:tasks.workunit.client.1.smithi111.stdout:1/689: stat d0/d2b/f5c 0 2024-03-23T11:47:41.993 INFO:tasks.workunit.client.1.smithi111.stdout:6/715: write d2/d1a/d1c/d2c/d51/f5f [2188188,71098] 0 2024-03-23T11:47:41.995 INFO:tasks.workunit.client.1.smithi111.stdout:4/745: symlink d0/d95/lf3 0 2024-03-23T11:47:41.996 INFO:tasks.workunit.client.1.smithi111.stdout:5/747: mknod d1/d3/d6/d7/d63/d2c/d82/cf8 0 2024-03-23T11:47:41.997 INFO:tasks.workunit.client.1.smithi111.stdout:9/764: fsync d0/d4/d1c/d2d/d75/d7a/fc5 0 2024-03-23T11:47:42.000 INFO:tasks.workunit.client.0.smithi063.stdout:1/454: write d3/f2a [2811750,54493] 0 2024-03-23T11:47:42.001 INFO:tasks.workunit.client.1.smithi111.stdout:1/690: mkdir d0/d6/d48/d4a/de5 0 2024-03-23T11:47:42.003 INFO:tasks.workunit.client.0.smithi063.stdout:2/482: creat d2/de/d7a/d8d/f9e x:0 0 0 2024-03-23T11:47:42.004 INFO:tasks.workunit.client.1.smithi111.stdout:9/765: write d0/f21 [2910722,109741] 0 2024-03-23T11:47:42.005 INFO:tasks.workunit.client.0.smithi063.stdout:1/455: read d3/df/f6e [2774764,78523] 0 2024-03-23T11:47:42.007 INFO:tasks.workunit.client.1.smithi111.stdout:2/751: dwrite d2/d11/d17/d3e/d5d/fa4 [0,4194304] 0 2024-03-23T11:47:42.007 INFO:tasks.workunit.client.0.smithi063.stdout:5/388: getdents d2/d7/d8/de/d47/d7c 0 2024-03-23T11:47:42.008 INFO:tasks.workunit.client.0.smithi063.stdout:7/480: symlink d5/d22/d57/d64/l9e 0 2024-03-23T11:47:42.008 INFO:tasks.workunit.client.0.smithi063.stdout:0/653: rename d2/d12/d32/f89 to d2/d14/d36/d77/d94/da6/d1f/d4f/fed 0 2024-03-23T11:47:42.010 INFO:tasks.workunit.client.0.smithi063.stdout:1/456: rmdir d3/df/d31 39 2024-03-23T11:47:42.015 INFO:tasks.workunit.client.1.smithi111.stdout:3/652: dread d0/d1a/d1d/f24 [0,4194304] 0 2024-03-23T11:47:42.016 INFO:tasks.workunit.client.0.smithi063.stdout:9/572: creat d4/d4c/fc7 x:0 0 0 2024-03-23T11:47:42.019 INFO:tasks.workunit.client.0.smithi063.stdout:1/457: creat d3/d1c/d4b/f8d x:0 0 0 2024-03-23T11:47:42.020 INFO:tasks.workunit.client.1.smithi111.stdout:0/682: dwrite d5/db/d1f/d19/dbd/f3d [0,4194304] 0 2024-03-23T11:47:42.021 INFO:tasks.workunit.client.0.smithi063.stdout:0/654: truncate d2/d6e/d8c/fa4 364573 0 2024-03-23T11:47:42.022 INFO:tasks.workunit.client.1.smithi111.stdout:3/653: write d0/fbe [1207805,33364] 0 2024-03-23T11:47:42.024 INFO:tasks.workunit.client.1.smithi111.stdout:4/746: dwrite d0/d95/d9b/db3/db6/fa3 [0,4194304] 0 2024-03-23T11:47:42.027 INFO:tasks.workunit.client.0.smithi063.stdout:2/483: dread d2/d23/d2e/f4b [0,4194304] 0 2024-03-23T11:47:42.030 INFO:tasks.workunit.client.0.smithi063.stdout:0/655: dread - d2/d12/d32/d6d/fd0 zero size 2024-03-23T11:47:42.031 INFO:tasks.workunit.client.1.smithi111.stdout:0/683: truncate d5/db/d1f/d19/dbd/f94 188179 0 2024-03-23T11:47:42.038 INFO:tasks.workunit.client.0.smithi063.stdout:3/850: truncate d13/d22/d2d/d3f/d67/d7b/fab 3867838 0 2024-03-23T11:47:42.041 INFO:tasks.workunit.client.1.smithi111.stdout:8/618: getdents d5/d26/d31/d3b/d55 0 2024-03-23T11:47:42.041 INFO:tasks.workunit.client.0.smithi063.stdout:0/656: write d2/d14/d36/d77/d94/da6/d28/d46/d51/f65 [2706295,57486] 0 2024-03-23T11:47:42.042 INFO:tasks.workunit.client.0.smithi063.stdout:4/600: sync 2024-03-23T11:47:42.044 INFO:tasks.workunit.client.1.smithi111.stdout:1/691: dread d0/d2b/f5c [0,4194304] 0 2024-03-23T11:47:42.044 INFO:tasks.workunit.client.0.smithi063.stdout:0/657: chown d2/d48/fa2 1 1 2024-03-23T11:47:42.045 INFO:tasks.workunit.client.1.smithi111.stdout:5/748: truncate d1/d3/d6/d7/d63/d2c/f52 492477 0 2024-03-23T11:47:42.046 INFO:tasks.workunit.client.1.smithi111.stdout:1/692: fdatasync d0/d2b/f5a 0 2024-03-23T11:47:42.052 INFO:tasks.workunit.client.1.smithi111.stdout:2/752: creat d2/d11/d17/d3e/d6b/f10f x:0 0 0 2024-03-23T11:47:42.061 INFO:tasks.workunit.client.1.smithi111.stdout:6/716: dread d2/d35/f44 [0,4194304] 0 2024-03-23T11:47:42.064 INFO:tasks.workunit.client.0.smithi063.stdout:7/481: dread d5/d6/d19/d1f/d6c/f80 [0,4194304] 0 2024-03-23T11:47:42.065 INFO:tasks.workunit.client.1.smithi111.stdout:6/717: read d2/d1f/d45/d9a/fb4 [233710,100645] 0 2024-03-23T11:47:42.066 INFO:tasks.workunit.client.1.smithi111.stdout:3/654: creat d0/d7/d19/d91/fdb x:0 0 0 2024-03-23T11:47:42.067 INFO:tasks.workunit.client.1.smithi111.stdout:3/655: readlink d0/l2 0 2024-03-23T11:47:42.067 INFO:tasks.workunit.client.1.smithi111.stdout:2/753: dwrite d2/de0/ffb [0,4194304] 0 2024-03-23T11:47:42.079 INFO:tasks.workunit.client.0.smithi063.stdout:4/601: dwrite d5/da/d20/f48 [4194304,4194304] 0 2024-03-23T11:47:42.082 INFO:tasks.workunit.client.0.smithi063.stdout:0/658: dread d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:42.084 INFO:tasks.workunit.client.0.smithi063.stdout:1/458: dread d3/d3f/d4d/f69 [0,4194304] 0 2024-03-23T11:47:42.085 INFO:tasks.workunit.client.0.smithi063.stdout:4/602: fsync d5/da/d20/d4b/d1f/f3f 0 2024-03-23T11:47:42.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.104+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa524103c60 con 0x7fa50806ace0 2024-03-23T11:47:42.089 INFO:tasks.workunit.client.0.smithi063.stdout:3/851: link d13/d22/d2d/l18 d13/d22/d2d/d3f/d58/d97/df8/l10b 0 2024-03-23T11:47:42.095 INFO:tasks.workunit.client.0.smithi063.stdout:4/603: write d5/da/d20/d4b/d16/d50/d86/f89 [1767377,41543] 0 2024-03-23T11:47:42.095 INFO:tasks.workunit.client.1.smithi111.stdout:9/766: dwrite d0/d31/d41/d65/f6f [0,4194304] 0 2024-03-23T11:47:42.096 INFO:tasks.workunit.client.0.smithi063.stdout:0/659: write d2/d14/d36/d77/d94/da6/d1f/fba [1675209,127110] 0 2024-03-23T11:47:42.103 INFO:tasks.workunit.client.0.smithi063.stdout:7/482: dwrite d5/d5c/d2f/d73/d20/f31 [0,4194304] 0 2024-03-23T11:47:42.104 INFO:tasks.workunit.client.0.smithi063.stdout:7/483: readlink d5/l10 0 2024-03-23T11:47:42.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.120+0000 7fa51aa00700 1 -- 172.21.15.63:0/2890936963 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fa524103c60 con 0x7fa50806ace0 2024-03-23T11:47:42.105 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:47:42.105 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 starting - - - - 2024-03-23T11:47:42.105 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (10m) 65s ago 10m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:47:42.105 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (9m) 62s ago 9m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:47:42.105 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (8m) 65s ago 9m 74.1M - 9.4.7 954c08fa6188 d05e188def00 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (3m) 65s ago 3m 70.5M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (3m) 65s ago 3m 42.0M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (3m) 62s ago 3m 41.1M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (3m) 62s ago 3m 31.6M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:9283 running (11m) 65s ago 11m 156M - 17.2.7-726-g9c4e0c34 837124da804f 63833d132315 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (88s) 62s ago 9m 478M - 18.2.2-820-gb630d6cd 881b71eac2fe 278e3d19ec5d 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (11m) 65s ago 11m 48.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (8m) 62s ago 8m 37.0M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 starting - - - - 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (9m) 62s ago 9m 7889k - 1.5.0 0da6a335fe13 68f97d563d92 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (8m) 65s ago 8m 73.2M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (7m) 65s ago 7m 78.9M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (6m) 65s ago 6m 67.7M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:47:42.106 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (6m) 62s ago 6m 78.8M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:47:42.107 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (5m) 62s ago 5m 74.8M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:47:42.107 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (5m) 62s ago 5m 71.7M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:47:42.107 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (8m) 65s ago 9m 47.5M - 2.43.0 a07b618ecd1d 328fbe2c4719 2024-03-23T11:47:42.108 INFO:tasks.workunit.client.0.smithi063.stdout:0/660: read d2/d14/d36/d77/d94/da6/d28/d46/d51/f82 [14885,33996] 0 2024-03-23T11:47:42.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.123+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa50806ace0 msgr2=0x7fa50806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:42.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.123+0000 7fa529a00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa50806ace0 0x7fa50806d190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510008040 comp rx=0 tx=0).stop 2024-03-23T11:47:42.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.123+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 msgr2=0x7fa52410ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:42.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.123+0000 7fa529a00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 0x7fa52410ff30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa51c000f80 tx=0x7fa51c004d10 comp rx=0 tx=0).stop 2024-03-23T11:47:42.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.124+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 shutdown_connections 2024-03-23T11:47:42.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.124+0000 7fa529a00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fa50806ace0 0x7fa50806d190 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.124+0000 7fa529a00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa524102300 0x7fa52410ff30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.124+0000 7fa529a00700 1 --2- 172.21.15.63:0/2890936963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa524110470 0x7fa524111250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.124+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 >> 172.21.15.63:0/2890936963 conn(0x7fa5240f7430 msgr2=0x7fa5240f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:42.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.124+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 shutdown_connections 2024-03-23T11:47:42.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.124+0000 7fa529a00700 1 -- 172.21.15.63:0/2890936963 wait complete. 2024-03-23T11:47:42.115 INFO:tasks.workunit.client.0.smithi063.stdout:9/573: write d4/d31/d43/da9/da6/f87 [311516,37132] 0 2024-03-23T11:47:42.115 INFO:tasks.workunit.client.0.smithi063.stdout:2/484: truncate d2/d1c/f30 4077541 0 2024-03-23T11:47:42.117 INFO:tasks.workunit.client.0.smithi063.stdout:1/459: creat d3/d49/d56/d65/d84/f8e x:0 0 0 2024-03-23T11:47:42.117 INFO:tasks.workunit.client.1.smithi111.stdout:7/694: truncate d14/d1b/d1e/d2c/d42/d5e/d88/f2b 2286834 0 2024-03-23T11:47:42.119 INFO:tasks.workunit.client.1.smithi111.stdout:7/695: write d14/d1b/d1e/f47 [5127136,12750] 0 2024-03-23T11:47:42.120 INFO:tasks.workunit.client.0.smithi063.stdout:1/460: chown d3/df/c47 3705440 1 2024-03-23T11:47:42.122 INFO:tasks.workunit.client.0.smithi063.stdout:7/484: mknod d5/d5c/d2f/d8b/c9f 0 2024-03-23T11:47:42.124 INFO:tasks.workunit.client.0.smithi063.stdout:1/461: chown d3/d49/d56/f7f 10121658 1 2024-03-23T11:47:42.126 INFO:tasks.workunit.client.0.smithi063.stdout:9/574: rmdir d4/d31/d43/d49/d8e 39 2024-03-23T11:47:42.130 INFO:tasks.workunit.client.0.smithi063.stdout:1/462: chown d3/df/d31 5 1 2024-03-23T11:47:42.132 INFO:tasks.workunit.client.0.smithi063.stdout:9/575: write d4/d31/d70/d8d/fbb [568782,129058] 0 2024-03-23T11:47:42.133 INFO:tasks.workunit.client.0.smithi063.stdout:0/661: dread d2/d4/fd8 [0,4194304] 0 2024-03-23T11:47:42.140 INFO:tasks.workunit.client.0.smithi063.stdout:1/463: rename d3/d49/l4a to d3/df/d7b/l8f 0 2024-03-23T11:47:42.143 INFO:tasks.workunit.client.0.smithi063.stdout:0/662: truncate d2/d48/f8f 419540 0 2024-03-23T11:47:42.146 INFO:tasks.workunit.client.0.smithi063.stdout:4/604: write f2 [2128380,822] 0 2024-03-23T11:47:42.148 INFO:tasks.workunit.client.0.smithi063.stdout:0/663: dread - d2/d14/d36/fda zero size 2024-03-23T11:47:42.150 INFO:tasks.workunit.client.0.smithi063.stdout:3/852: write d13/d22/d2d/d3f/d5b/f10a [1785545,35073] 0 2024-03-23T11:47:42.155 INFO:tasks.workunit.client.0.smithi063.stdout:4/605: fsync d5/da/d20/d4b/d1f/f26 0 2024-03-23T11:47:42.155 INFO:tasks.workunit.client.1.smithi111.stdout:5/749: sync 2024-03-23T11:47:42.158 INFO:tasks.workunit.client.0.smithi063.stdout:9/576: dwrite d4/f21 [0,4194304] 0 2024-03-23T11:47:42.158 INFO:tasks.workunit.client.0.smithi063.stdout:4/606: chown d5/da/d20/d4b/d16/fa3 167065238 1 2024-03-23T11:47:42.162 INFO:tasks.workunit.client.1.smithi111.stdout:3/656: unlink d0/f9 0 2024-03-23T11:47:42.164 INFO:tasks.workunit.client.1.smithi111.stdout:2/754: read - d2/d11/d17/d3e/d6b/d2e/d76/fcc zero size 2024-03-23T11:47:42.165 INFO:tasks.workunit.client.0.smithi063.stdout:0/664: creat d2/d14/d36/d77/deb/fee x:0 0 0 2024-03-23T11:47:42.165 INFO:tasks.workunit.client.0.smithi063.stdout:4/607: creat d5/d1e/d2b/faf x:0 0 0 2024-03-23T11:47:42.170 INFO:tasks.workunit.client.1.smithi111.stdout:5/750: dwrite d1/d20/dce/fe4 [0,4194304] 0 2024-03-23T11:47:42.171 INFO:tasks.workunit.client.0.smithi063.stdout:0/665: symlink d2/lef 0 2024-03-23T11:47:42.173 INFO:tasks.workunit.client.0.smithi063.stdout:9/577: creat d4/d31/d70/d8d/da0/fc8 x:0 0 0 2024-03-23T11:47:42.174 INFO:tasks.workunit.client.1.smithi111.stdout:9/767: unlink d0/d31/d35/fa1 0 2024-03-23T11:47:42.175 INFO:tasks.workunit.client.1.smithi111.stdout:9/768: dread - d0/d4/f9f zero size 2024-03-23T11:47:42.176 INFO:tasks.workunit.client.0.smithi063.stdout:9/578: stat d4/c56 0 2024-03-23T11:47:42.176 INFO:tasks.workunit.client.0.smithi063.stdout:4/608: read d5/da/d20/d4b/d29/d38/d7a/f8a [330012,51588] 0 2024-03-23T11:47:42.178 INFO:tasks.workunit.client.1.smithi111.stdout:9/769: write d0/d4/d1c/d2d/fb7 [549256,59140] 0 2024-03-23T11:47:42.180 INFO:tasks.workunit.client.0.smithi063.stdout:9/579: chown d4/d31/fb9 7558952 1 2024-03-23T11:47:42.181 INFO:tasks.workunit.client.0.smithi063.stdout:7/485: dread d5/d6/d39/f6a [0,4194304] 0 2024-03-23T11:47:42.181 INFO:tasks.workunit.client.0.smithi063.stdout:0/666: rmdir d2/d14/d36/d77/d94/da6/d28 39 2024-03-23T11:47:42.182 INFO:tasks.workunit.client.1.smithi111.stdout:9/770: chown d0/d4/d1c/d36/d28/d69/f87 1521460083 1 2024-03-23T11:47:42.185 INFO:tasks.workunit.client.0.smithi063.stdout:7/486: mknod d5/d26/d32/ca0 0 2024-03-23T11:47:42.186 INFO:tasks.workunit.client.0.smithi063.stdout:9/580: mknod d4/d31/d43/d49/cc9 0 2024-03-23T11:47:42.190 INFO:tasks.workunit.client.0.smithi063.stdout:7/487: dread - d5/d5c/d2f/f79 zero size 2024-03-23T11:47:42.192 INFO:tasks.workunit.client.1.smithi111.stdout:7/696: creat d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fcc x:0 0 0 2024-03-23T11:47:42.192 INFO:tasks.workunit.client.0.smithi063.stdout:4/609: link d5/da/d20/d4b/d16/d32/d56/fa0 d5/da/d20/d4b/d16/d32/fb0 0 2024-03-23T11:47:42.194 INFO:tasks.workunit.client.0.smithi063.stdout:0/667: readlink d2/d14/d36/d77/d94/da6/d28/d46/d51/le3 0 2024-03-23T11:47:42.195 INFO:tasks.workunit.client.0.smithi063.stdout:7/488: write d5/d5c/d2f/d73/d20/f31 [3309521,93984] 0 2024-03-23T11:47:42.198 INFO:tasks.workunit.client.0.smithi063.stdout:9/581: rmdir d4/d31/d43/da9/da6 39 2024-03-23T11:47:42.198 INFO:tasks.workunit.client.1.smithi111.stdout:8/619: write d5/d26/d31/f39 [2152486,43900] 0 2024-03-23T11:47:42.204 INFO:tasks.workunit.client.0.smithi063.stdout:0/668: creat d2/d14/ff0 x:0 0 0 2024-03-23T11:47:42.205 INFO:tasks.workunit.client.1.smithi111.stdout:6/718: symlink d2/d1f/d45/d9a/dde/led 0 2024-03-23T11:47:42.207 INFO:tasks.workunit.client.1.smithi111.stdout:3/657: mknod d0/d1a/d1d/cdc 0 2024-03-23T11:47:42.209 INFO:tasks.workunit.client.0.smithi063.stdout:1/464: dread d3/d1c/f6d [4194304,4194304] 0 2024-03-23T11:47:42.212 INFO:tasks.workunit.client.1.smithi111.stdout:2/755: rename d2/d11/d17/d3e/d6b/d9d/d52/dba/dd3 to d2/de0/d110 0 2024-03-23T11:47:42.212 INFO:tasks.workunit.client.0.smithi063.stdout:0/669: creat d2/d6e/d8c/ff1 x:0 0 0 2024-03-23T11:47:42.215 INFO:tasks.workunit.client.0.smithi063.stdout:7/489: mkdir d5/d6/d19/d1f/d47/d6e/d98/da1 0 2024-03-23T11:47:42.217 INFO:tasks.workunit.client.0.smithi063.stdout:4/610: getdents d5/da/d20/d4b/d16/d50/d86 0 2024-03-23T11:47:42.218 INFO:tasks.workunit.client.1.smithi111.stdout:5/751: rmdir d1/d3/d6/d30/db2 39 2024-03-23T11:47:42.219 INFO:tasks.workunit.client.1.smithi111.stdout:7/697: dread d14/d37/f49 [0,4194304] 0 2024-03-23T11:47:42.220 INFO:tasks.workunit.client.1.smithi111.stdout:7/698: stat d14/d1b/d1e/d87 0 2024-03-23T11:47:42.220 INFO:tasks.workunit.client.0.smithi063.stdout:9/582: creat d4/d4c/fca x:0 0 0 2024-03-23T11:47:42.221 INFO:tasks.workunit.client.0.smithi063.stdout:7/490: write d5/d6/d19/d1f/d47/d4f/d5b/f7b [885417,16487] 0 2024-03-23T11:47:42.222 INFO:tasks.workunit.client.1.smithi111.stdout:7/699: readlink d14/d1b/d1e/d2c/d42/d5e/d88/d5f/l6e 0 2024-03-23T11:47:42.222 INFO:tasks.workunit.client.1.smithi111.stdout:9/771: dread d0/d4/d1c/d2d/d3b/d42/d92/fc7 [0,4194304] 0 2024-03-23T11:47:42.224 INFO:tasks.workunit.client.1.smithi111.stdout:7/700: fsync d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb9 0 2024-03-23T11:47:42.228 INFO:tasks.workunit.client.0.smithi063.stdout:1/465: creat d3/d35/f90 x:0 0 0 2024-03-23T11:47:42.229 INFO:tasks.workunit.client.1.smithi111.stdout:7/701: truncate d14/d1b/d1e/d2c/d42/d5e/d88/f94 1451130 0 2024-03-23T11:47:42.229 INFO:tasks.workunit.client.1.smithi111.stdout:0/684: creat d5/db/d1f/d47/d6f/fe9 x:0 0 0 2024-03-23T11:47:42.230 INFO:tasks.workunit.client.0.smithi063.stdout:0/670: fdatasync d2/d14/de8/f8e 0 2024-03-23T11:47:42.232 INFO:tasks.workunit.client.0.smithi063.stdout:9/583: truncate d4/d31/d43/d49/d8e/fbc 1329535 0 2024-03-23T11:47:42.239 INFO:tasks.workunit.client.0.smithi063.stdout:4/611: rename d5/da/d20/d4b/d16/d32/d56/fa0 to d5/da/d72/fb1 0 2024-03-23T11:47:42.239 INFO:tasks.workunit.client.0.smithi063.stdout:7/491: rmdir d5/d6/d19/d1f/d47/d6e 39 2024-03-23T11:47:42.240 INFO:tasks.workunit.client.0.smithi063.stdout:1/466: mknod d3/df/d7b/c91 0 2024-03-23T11:47:42.243 INFO:tasks.workunit.client.1.smithi111.stdout:3/658: truncate d0/d1a/f1c 3344345 0 2024-03-23T11:47:42.246 INFO:tasks.workunit.client.0.smithi063.stdout:8/795: sync 2024-03-23T11:47:42.249 INFO:tasks.workunit.client.1.smithi111.stdout:2/756: mkdir d2/de/d6c/dd1/d111 0 2024-03-23T11:47:42.249 INFO:tasks.workunit.client.0.smithi063.stdout:8/796: chown d6/dd/d59/d7f/dc8/dfc 201257985 1 2024-03-23T11:47:42.251 INFO:tasks.workunit.client.0.smithi063.stdout:1/467: mkdir d3/d49/d56/d65/d88/d92 0 2024-03-23T11:47:42.253 INFO:tasks.workunit.client.1.smithi111.stdout:9/772: creat d0/d4/d10/ffa x:0 0 0 2024-03-23T11:47:42.254 INFO:tasks.workunit.client.1.smithi111.stdout:4/747: getdents d0/dc/d4c/d2a/d97/d4f/d72 0 2024-03-23T11:47:42.255 INFO:tasks.workunit.client.0.smithi063.stdout:7/492: mkdir d5/d6/d19/d1f/d47/d6e/da2 0 2024-03-23T11:47:42.256 INFO:tasks.workunit.client.0.smithi063.stdout:0/671: link d2/d12/d32/d6d/ce7 d2/d14/de8/cf2 0 2024-03-23T11:47:42.257 INFO:tasks.workunit.client.0.smithi063.stdout:7/493: readlink d5/d22/d57/d64/l9e 0 2024-03-23T11:47:42.258 INFO:tasks.workunit.client.0.smithi063.stdout:1/468: rmdir d3/d62 39 2024-03-23T11:47:42.259 INFO:tasks.workunit.client.1.smithi111.stdout:1/693: getdents d0/d2b 0 2024-03-23T11:47:42.263 INFO:tasks.workunit.client.1.smithi111.stdout:3/659: symlink d0/d72/ldd 0 2024-03-23T11:47:42.264 INFO:tasks.workunit.client.1.smithi111.stdout:3/660: stat d0/d1a/d2b/f4d 0 2024-03-23T11:47:42.264 INFO:tasks.workunit.client.0.smithi063.stdout:8/797: rename d6/dd/d59/d7f to d6/d1c/d32/d37/d10a 0 2024-03-23T11:47:42.269 INFO:tasks.workunit.client.1.smithi111.stdout:2/757: mkdir d2/de0/d110/d112 0 2024-03-23T11:47:42.269 INFO:tasks.workunit.client.0.smithi063.stdout:4/612: dwrite d5/da/d20/d4b/d1f/f39 [0,4194304] 0 2024-03-23T11:47:42.271 INFO:tasks.workunit.client.0.smithi063.stdout:0/672: creat d2/d14/d36/d77/d94/da6/d25/d93/ff3 x:0 0 0 2024-03-23T11:47:42.271 INFO:tasks.workunit.client.1.smithi111.stdout:2/758: write d2/d7/da/f103 [445931,8808] 0 2024-03-23T11:47:42.273 INFO:tasks.workunit.client.1.smithi111.stdout:9/773: truncate d0/d4/fe 1693308 0 2024-03-23T11:47:42.275 INFO:tasks.workunit.client.1.smithi111.stdout:2/759: read d2/de/d6c/dd1/fe6 [766913,117009] 0 2024-03-23T11:47:42.277 INFO:tasks.workunit.client.1.smithi111.stdout:4/748: symlink d0/dc/d1f/d5e/lf4 0 2024-03-23T11:47:42.278 INFO:tasks.workunit.client.1.smithi111.stdout:6/719: truncate d2/d35/f44 1512060 0 2024-03-23T11:47:42.280 INFO:tasks.workunit.client.0.smithi063.stdout:1/469: creat d3/d35/f93 x:0 0 0 2024-03-23T11:47:42.281 INFO:tasks.workunit.client.0.smithi063.stdout:8/798: mknod d6/dd/d59/daf/dba/c10b 0 2024-03-23T11:47:42.282 INFO:tasks.workunit.client.0.smithi063.stdout:7/494: dwrite d5/d6/d19/d1f/d47/d4f/d5b/f7b [0,4194304] 0 2024-03-23T11:47:42.284 INFO:tasks.workunit.client.0.smithi063.stdout:1/470: chown d3/d1c/l4e 241 1 2024-03-23T11:47:42.285 INFO:tasks.workunit.client.1.smithi111.stdout:3/661: dread d0/d1a/d2d/f45 [0,4194304] 0 2024-03-23T11:47:42.286 INFO:tasks.workunit.client.1.smithi111.stdout:1/694: creat d0/d5/dd/d1e/d54/d87/d8e/fe6 x:0 0 0 2024-03-23T11:47:42.288 INFO:tasks.workunit.client.1.smithi111.stdout:5/752: truncate d1/d20/f60 473907 0 2024-03-23T11:47:42.289 INFO:tasks.workunit.client.0.smithi063.stdout:7/495: chown d5/d5c/d2f/d73/d20/c55 14278816 1 2024-03-23T11:47:42.295 INFO:tasks.workunit.client.0.smithi063.stdout:1/471: truncate d3/d49/d56/f67 519276 0 2024-03-23T11:47:42.299 INFO:tasks.workunit.client.1.smithi111.stdout:8/620: rename d5/d26/d2f/d62/f6f to d5/d1b/fc9 0 2024-03-23T11:47:42.300 INFO:tasks.workunit.client.0.smithi063.stdout:1/472: chown d3/cc 255304 1 2024-03-23T11:47:42.301 INFO:tasks.workunit.client.0.smithi063.stdout:0/673: dwrite d2/d12/d32/d6d/f9c [0,4194304] 0 2024-03-23T11:47:42.309 INFO:tasks.workunit.client.0.smithi063.stdout:0/674: creat d2/d14/d36/ff4 x:0 0 0 2024-03-23T11:47:42.315 INFO:tasks.workunit.client.0.smithi063.stdout:0/675: mknod d2/d14/d36/d77/d94/da6/d1f/d4f/cf5 0 2024-03-23T11:47:42.316 INFO:tasks.workunit.client.1.smithi111.stdout:4/749: dread d0/fc6 [0,4194304] 0 2024-03-23T11:47:42.319 INFO:tasks.workunit.client.1.smithi111.stdout:0/685: dwrite d5/db/d1f/f79 [4194304,4194304] 0 2024-03-23T11:47:42.319 INFO:tasks.workunit.client.0.smithi063.stdout:4/613: dwrite d5/da/d20/d4b/f98 [0,4194304] 0 2024-03-23T11:47:42.324 INFO:tasks.workunit.client.0.smithi063.stdout:0/676: creat d2/d14/de8/ddb/ff6 x:0 0 0 2024-03-23T11:47:42.328 INFO:tasks.workunit.client.0.smithi063.stdout:4/614: write d5/da/d20/d4b/fa5 [350906,15314] 0 2024-03-23T11:47:42.329 INFO:tasks.workunit.client.1.smithi111.stdout:2/760: truncate d2/d11/d17/d3e/d5d/da3/fd5 392398 0 2024-03-23T11:47:42.332 INFO:tasks.workunit.client.0.smithi063.stdout:7/496: dwrite d5/d5c/d2f/d73/da/d90/f9b [0,4194304] 0 2024-03-23T11:47:42.339 INFO:tasks.workunit.client.1.smithi111.stdout:0/686: dwrite d5/db/d1f/d47/d59/db6/fca [0,4194304] 0 2024-03-23T11:47:42.341 INFO:tasks.workunit.client.1.smithi111.stdout:0/687: write d5/db/f16 [3294217,2973] 0 2024-03-23T11:47:42.342 INFO:tasks.workunit.client.1.smithi111.stdout:8/621: symlink d5/d26/d2f/d42/lca 0 2024-03-23T11:47:42.346 INFO:tasks.workunit.client.0.smithi063.stdout:1/473: dwrite d3/d49/d56/d65/f7c [4194304,4194304] 0 2024-03-23T11:47:42.347 INFO:tasks.workunit.client.1.smithi111.stdout:5/753: dread d1/d20/dce/fbb [0,4194304] 0 2024-03-23T11:47:42.349 INFO:tasks.workunit.client.1.smithi111.stdout:4/750: mknod d0/dc/d4c/dad/cf5 0 2024-03-23T11:47:42.352 INFO:tasks.workunit.client.0.smithi063.stdout:8/799: dwrite d6/d1f/dbb/fc3 [0,4194304] 0 2024-03-23T11:47:42.354 INFO:tasks.workunit.client.1.smithi111.stdout:6/720: unlink d2/l95 0 2024-03-23T11:47:42.362 INFO:tasks.workunit.client.1.smithi111.stdout:1/695: mkdir d0/d2b/de3/de7 0 2024-03-23T11:47:42.365 INFO:tasks.workunit.client.1.smithi111.stdout:7/702: sync 2024-03-23T11:47:42.365 INFO:tasks.workunit.client.0.smithi063.stdout:1/474: creat d3/d35/f94 x:0 0 0 2024-03-23T11:47:42.365 INFO:tasks.workunit.client.1.smithi111.stdout:0/688: dwrite d5/db/d1f/d19/dbd/d8b/fa8 [0,4194304] 0 2024-03-23T11:47:42.368 INFO:tasks.workunit.client.1.smithi111.stdout:0/689: dread - d5/f85 zero size 2024-03-23T11:47:42.370 INFO:tasks.workunit.client.1.smithi111.stdout:9/774: link d0/d31/d41/c4e d0/cfb 0 2024-03-23T11:47:42.372 INFO:tasks.workunit.client.0.smithi063.stdout:1/475: creat d3/d3f/d4d/f95 x:0 0 0 2024-03-23T11:47:42.373 INFO:tasks.workunit.client.1.smithi111.stdout:9/775: fdatasync d0/d4/d10/ff2 0 2024-03-23T11:47:42.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.389+0000 7f87f9600700 1 -- 172.21.15.63:0/4057951413 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87e800e070 con 0x7f87fc06eb00 2024-03-23T11:47:42.374 INFO:tasks.workunit.client.0.smithi063.stdout:4/615: write d5/da/d20/d4b/d16/d50/f69 [1472975,51893] 0 2024-03-23T11:47:42.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.389+0000 7f8801e00700 1 -- 172.21.15.63:0/4057951413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06eb00 msgr2=0x7f87fc06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:42.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.389+0000 7f8801e00700 1 --2- 172.21.15.63:0/4057951413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06eb00 0x7f87fc06ef50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f87e8004940 tx=0x7f87e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:42.375 INFO:tasks.workunit.client.1.smithi111.stdout:7/703: read d14/f15 [3532096,29181] 0 2024-03-23T11:47:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.392+0000 7f8801e00700 1 -- 172.21.15.63:0/4057951413 shutdown_connections 2024-03-23T11:47:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.392+0000 7f8801e00700 1 --2- 172.21.15.63:0/4057951413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06eb00 0x7f87fc06ef50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.392+0000 7f8801e00700 1 --2- 172.21.15.63:0/4057951413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87fc06fda0 0x7f87fc06e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.392+0000 7f8801e00700 1 -- 172.21.15.63:0/4057951413 >> 172.21.15.63:0/4057951413 conn(0x7f87fc06a380 msgr2=0x7f87fc06c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.392+0000 7f8801e00700 1 -- 172.21.15.63:0/4057951413 shutdown_connections 2024-03-23T11:47:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.392+0000 7f8801e00700 1 -- 172.21.15.63:0/4057951413 wait complete. 2024-03-23T11:47:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.394+0000 7f8801e00700 1 Processor -- start 2024-03-23T11:47:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.395+0000 7f8801e00700 1 -- start start 2024-03-23T11:47:42.381 INFO:tasks.workunit.client.0.smithi063.stdout:8/800: symlink d6/d1f/d54/d5c/d76/d85/l10c 0 2024-03-23T11:47:42.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.397+0000 7f8801e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87fc06eb00 0x7f87fc11c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.397+0000 7f8801e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06fda0 0x7f87fc11cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.397+0000 7f8801e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc11d260 con 0x7f87fc06fda0 2024-03-23T11:47:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.397+0000 7f8801e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc121c00 con 0x7f87fc06eb00 2024-03-23T11:47:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.397+0000 7f87fa000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06fda0 0x7f87fc11cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.398+0000 7f87fa000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06fda0 0x7f87fc11cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54270/0 (socket says 172.21.15.63:54270) 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.398+0000 7f87fa000700 1 -- 172.21.15.63:0/3826751852 learned_addr learned my addr 172.21.15.63:0/3826751852 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.398+0000 7f87fa000700 1 -- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87fc06eb00 msgr2=0x7f87fc11c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.398+0000 7f87fa000700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87fc06eb00 0x7f87fc11c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.398+0000 7f87fa000700 1 -- 172.21.15.63:0/3826751852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e8005040 con 0x7f87fc06fda0 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.399+0000 7f87fa000700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06fda0 0x7f87fc11cbd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f87e80043d0 tx=0x7f87e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.399+0000 7f87f2a00700 1 -- 172.21.15.63:0/3826751852 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87e800e040 con 0x7f87fc06fda0 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.399+0000 7f87f2a00700 1 -- 172.21.15.63:0/3826751852 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87e800bd50 con 0x7f87fc06fda0 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.399+0000 7f87f2a00700 1 -- 172.21.15.63:0/3826751852 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87e801c630 con 0x7f87fc06fda0 2024-03-23T11:47:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.400+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc121e00 con 0x7f87fc06fda0 2024-03-23T11:47:42.386 INFO:tasks.workunit.client.1.smithi111.stdout:7/704: read d14/f15 [1706909,61326] 0 2024-03-23T11:47:42.390 INFO:tasks.workunit.client.1.smithi111.stdout:8/622: dread d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [0,4194304] 0 2024-03-23T11:47:42.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.405+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc1222c0 con 0x7f87fc06fda0 2024-03-23T11:47:42.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.406+0000 7f87f2a00700 1 -- 172.21.15.63:0/3826751852 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f87e8013a60 con 0x7f87fc06fda0 2024-03-23T11:47:42.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.407+0000 7f87f2a00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f87e006aca0 0x7f87e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:42.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.408+0000 7f87faa00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f87e006aca0 0x7f87e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:42.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.408+0000 7f87f2a00700 1 -- 172.21.15.63:0/3826751852 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f87e8058eb0 con 0x7f87fc06fda0 2024-03-23T11:47:42.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.408+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc108870 con 0x7f87fc06fda0 2024-03-23T11:47:42.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.411+0000 7f87faa00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f87e006aca0 0x7f87e006d150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87ec006fd0 tx=0x7f87ec006d30 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:42.397 INFO:tasks.workunit.client.1.smithi111.stdout:3/662: write d0/d7/f82 [255781,111235] 0 2024-03-23T11:47:42.399 INFO:tasks.workunit.client.0.smithi063.stdout:7/497: dwrite d5/d6/f25 [4194304,4194304] 0 2024-03-23T11:47:42.399 INFO:tasks.workunit.client.0.smithi063.stdout:8/801: rename d6/d1c/d27/cac to d6/d1f/d106/c10d 0 2024-03-23T11:47:42.401 INFO:tasks.workunit.client.1.smithi111.stdout:3/663: truncate d0/d1a/d1d/d26/d5a/d60/fac 813143 0 2024-03-23T11:47:42.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.417+0000 7f87f2a00700 1 -- 172.21.15.63:0/3826751852 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f87e8055280 con 0x7f87fc06fda0 2024-03-23T11:47:42.408 INFO:tasks.workunit.client.1.smithi111.stdout:6/721: fsync d2/d17/f60 0 2024-03-23T11:47:42.410 INFO:tasks.workunit.client.0.smithi063.stdout:8/802: read d6/d1f/d54/d63/d90/d15/f17 [362511,31641] 0 2024-03-23T11:47:42.414 INFO:tasks.workunit.client.1.smithi111.stdout:7/705: creat d14/d1b/d1e/d2c/d2e/d33/fcd x:0 0 0 2024-03-23T11:47:42.416 INFO:tasks.workunit.client.1.smithi111.stdout:7/706: dread - d14/d1b/d1e/d2c/d2e/d73/dae/d74/fa9 zero size 2024-03-23T11:47:42.417 INFO:tasks.workunit.client.1.smithi111.stdout:1/696: truncate d0/d2b/d37/f74 943949 0 2024-03-23T11:47:42.420 INFO:tasks.workunit.client.0.smithi063.stdout:1/476: dwrite d3/f4 [0,4194304] 0 2024-03-23T11:47:42.422 INFO:tasks.workunit.client.1.smithi111.stdout:5/754: write d1/d20/f61 [7736914,20227] 0 2024-03-23T11:47:42.425 INFO:tasks.workunit.client.0.smithi063.stdout:5/389: sync 2024-03-23T11:47:42.425 INFO:tasks.workunit.client.0.smithi063.stdout:6/457: sync 2024-03-23T11:47:42.426 INFO:tasks.workunit.client.1.smithi111.stdout:3/664: unlink d0/d1a/d2d/lc9 0 2024-03-23T11:47:42.434 INFO:tasks.workunit.client.1.smithi111.stdout:7/707: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fcc [0,4194304] 0 2024-03-23T11:47:42.436 INFO:tasks.workunit.client.1.smithi111.stdout:2/761: creat d2/d11/d17/d1f/f113 x:0 0 0 2024-03-23T11:47:42.436 INFO:tasks.workunit.client.1.smithi111.stdout:4/751: link d0/dc/d30/fab d0/dc/d30/ff6 0 2024-03-23T11:47:42.441 INFO:tasks.workunit.client.1.smithi111.stdout:1/697: symlink d0/d5/dd/d1e/d54/d87/d8e/le8 0 2024-03-23T11:47:42.442 INFO:tasks.workunit.client.0.smithi063.stdout:1/477: truncate d3/df/d31/f5f 177938 0 2024-03-23T11:47:42.444 INFO:tasks.workunit.client.1.smithi111.stdout:1/698: write d0/d5/dd/d1e/d54/fc2 [93027,32007] 0 2024-03-23T11:47:42.446 INFO:tasks.workunit.client.1.smithi111.stdout:1/699: chown d0/d5/l1b 5 1 2024-03-23T11:47:42.447 INFO:tasks.workunit.client.1.smithi111.stdout:1/700: chown d0/d6/f19 13791 1 2024-03-23T11:47:42.449 INFO:tasks.workunit.client.0.smithi063.stdout:4/616: dread d5/da/d20/f3c [0,4194304] 0 2024-03-23T11:47:42.450 INFO:tasks.workunit.client.0.smithi063.stdout:8/803: dwrite d6/d1f/f74 [0,4194304] 0 2024-03-23T11:47:42.453 INFO:tasks.workunit.client.1.smithi111.stdout:7/708: creat d14/d1b/d1e/d2c/d2e/d73/fce x:0 0 0 2024-03-23T11:47:42.454 INFO:tasks.workunit.client.0.smithi063.stdout:5/390: unlink d2/d7/d8/de/d47/l69 0 2024-03-23T11:47:42.454 INFO:tasks.workunit.client.0.smithi063.stdout:6/458: mknod d1/dc/d3a/d57/d6d/c89 0 2024-03-23T11:47:42.460 INFO:tasks.workunit.client.1.smithi111.stdout:5/755: dread d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:42.460 INFO:tasks.workunit.client.1.smithi111.stdout:9/776: rename d0/d4/d1c/d2d/d3b/d42/d92/l99 to d0/d4/d1c/d2d/d75/lfc 0 2024-03-23T11:47:42.460 INFO:tasks.workunit.client.0.smithi063.stdout:6/459: symlink d1/dc/d3a/l8a 0 2024-03-23T11:47:42.460 INFO:tasks.workunit.client.0.smithi063.stdout:8/804: stat d6/dd/df/f4b 0 2024-03-23T11:47:42.462 INFO:tasks.workunit.client.0.smithi063.stdout:7/498: dread d5/d5c/d2f/d73/da/f17 [0,4194304] 0 2024-03-23T11:47:42.468 INFO:tasks.workunit.client.0.smithi063.stdout:5/391: rename d2/d7/d8/l29 to d2/d7/d8/de/l8a 0 2024-03-23T11:47:42.473 INFO:tasks.workunit.client.1.smithi111.stdout:0/690: rename d5/db/d1f/d48/lae to d5/db/d1f/de4/lea 0 2024-03-23T11:47:42.476 INFO:tasks.workunit.client.0.smithi063.stdout:6/460: getdents d1/d2a 0 2024-03-23T11:47:42.477 INFO:tasks.workunit.client.1.smithi111.stdout:5/756: mknod d1/d3/d6/d8b/d59/cf9 0 2024-03-23T11:47:42.478 INFO:tasks.workunit.client.0.smithi063.stdout:7/499: getdents d5/d6/d19/d1f/d47/d4f/d5b 0 2024-03-23T11:47:42.478 INFO:tasks.workunit.client.0.smithi063.stdout:8/805: rename d6/d1f/d54/d63/d90/d15/d9d/fa5 to d6/d1f/d54/d63/d90/d5d/d7d/db4/f10e 0 2024-03-23T11:47:42.483 INFO:tasks.workunit.client.0.smithi063.stdout:7/500: mkdir d5/d22/da3 0 2024-03-23T11:47:42.486 INFO:tasks.workunit.client.0.smithi063.stdout:8/806: creat d6/d1f/d54/d63/d90/d5d/d7d/d9a/f10f x:0 0 0 2024-03-23T11:47:42.491 INFO:tasks.workunit.client.1.smithi111.stdout:5/757: truncate d1/d3/d6/d8b/d59/d6c/f76 1435753 0 2024-03-23T11:47:42.494 INFO:tasks.workunit.client.1.smithi111.stdout:0/691: mkdir d5/deb 0 2024-03-23T11:47:42.495 INFO:tasks.workunit.client.0.smithi063.stdout:5/392: dwrite d2/d28/d3a/d46/d79/f80 [0,4194304] 0 2024-03-23T11:47:42.498 INFO:tasks.workunit.client.0.smithi063.stdout:7/501: mknod d5/d6/d19/d1f/d47/ca4 0 2024-03-23T11:47:42.500 INFO:tasks.workunit.client.1.smithi111.stdout:2/762: write d2/de/f84 [1691263,118781] 0 2024-03-23T11:47:42.501 INFO:tasks.workunit.client.1.smithi111.stdout:3/665: truncate d0/d7/d19/f29 3694629 0 2024-03-23T11:47:42.501 INFO:tasks.workunit.client.0.smithi063.stdout:5/393: write d2/d28/d3a/d46/d79/f7e [698797,39818] 0 2024-03-23T11:47:42.503 INFO:tasks.workunit.client.0.smithi063.stdout:8/807: mkdir d6/d1c/d110 0 2024-03-23T11:47:42.513 INFO:tasks.workunit.client.1.smithi111.stdout:5/758: rename d1/d3/d6/d7/d63/d2c/f34 to d1/d3/d6/d7/dee/ffa 0 2024-03-23T11:47:42.514 INFO:tasks.workunit.client.1.smithi111.stdout:1/701: dwrite d0/d5/dd/f6e [0,4194304] 0 2024-03-23T11:47:42.514 INFO:tasks.workunit.client.1.smithi111.stdout:4/752: dwrite d0/dc/d4c/f6d [0,4194304] 0 2024-03-23T11:47:42.517 INFO:tasks.workunit.client.1.smithi111.stdout:9/777: dwrite d0/d4/d1c/d2d/d3b/d42/d4f/f61 [0,4194304] 0 2024-03-23T11:47:42.522 INFO:tasks.workunit.client.0.smithi063.stdout:7/502: dread f1 [0,4194304] 0 2024-03-23T11:47:42.524 INFO:tasks.workunit.client.0.smithi063.stdout:6/461: dread d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:42.531 INFO:tasks.workunit.client.1.smithi111.stdout:1/702: rmdir d0/d5/dd/d1e 39 2024-03-23T11:47:42.536 INFO:tasks.workunit.client.1.smithi111.stdout:8/623: sync 2024-03-23T11:47:42.538 INFO:tasks.workunit.client.1.smithi111.stdout:8/624: write d5/d26/d31/d4d/faf [1047685,43829] 0 2024-03-23T11:47:42.540 INFO:tasks.workunit.client.1.smithi111.stdout:3/666: mkdir d0/d1a/d2b/d38/dd4/dde 0 2024-03-23T11:47:42.541 INFO:tasks.workunit.client.1.smithi111.stdout:0/692: dread d5/db/d1f/d47/f5a [0,4194304] 0 2024-03-23T11:47:42.542 INFO:tasks.workunit.client.0.smithi063.stdout:5/394: dwrite d2/d28/f3d [0,4194304] 0 2024-03-23T11:47:42.556 INFO:tasks.workunit.client.1.smithi111.stdout:0/693: chown d5/db/d1f/d47/d59/cc4 10483441 1 2024-03-23T11:47:42.556 INFO:tasks.workunit.client.0.smithi063.stdout:7/503: truncate d5/d6/d39/f6a 2182337 0 2024-03-23T11:47:42.556 INFO:tasks.workunit.client.0.smithi063.stdout:6/462: truncate d1/d15/d4b/f4c 2436489 0 2024-03-23T11:47:42.560 INFO:tasks.workunit.client.0.smithi063.stdout:6/463: stat d1/dc/d21/d49/c75 0 2024-03-23T11:47:42.561 INFO:tasks.workunit.client.0.smithi063.stdout:5/395: rename d2/d28/d3a/d46/d61 to d2/d7/d2b/d8b 0 2024-03-23T11:47:42.562 INFO:tasks.workunit.client.0.smithi063.stdout:7/504: mknod d5/d26/d32/ca5 0 2024-03-23T11:47:42.564 INFO:tasks.workunit.client.1.smithi111.stdout:2/763: write d2/d11/d17/d3e/d5d/dc2/fc4 [280953,15375] 0 2024-03-23T11:47:42.569 INFO:tasks.workunit.client.0.smithi063.stdout:6/464: mkdir d1/d15/d8b 0 2024-03-23T11:47:42.569 INFO:tasks.workunit.client.0.smithi063.stdout:8/808: dread d6/d1f/d54/d63/d90/f28 [0,4194304] 0 2024-03-23T11:47:42.571 INFO:tasks.workunit.client.0.smithi063.stdout:7/505: creat d5/d5c/d2f/d73/da/d60/fa6 x:0 0 0 2024-03-23T11:47:42.574 INFO:tasks.workunit.client.0.smithi063.stdout:6/465: read d1/d13/f1c [120174,120472] 0 2024-03-23T11:47:42.576 INFO:tasks.workunit.client.0.smithi063.stdout:7/506: read d5/d6/d19/d1f/f6d [2239051,97812] 0 2024-03-23T11:47:42.580 INFO:tasks.workunit.client.1.smithi111.stdout:5/759: getdents d1/d20/d5c/d99/dbf/dc3 0 2024-03-23T11:47:42.584 INFO:tasks.workunit.client.1.smithi111.stdout:0/694: mknod d5/db/d1f/d47/d59/d5b/cec 0 2024-03-23T11:47:42.588 INFO:tasks.workunit.client.1.smithi111.stdout:4/753: getdents d0/d95/d9b/db3 0 2024-03-23T11:47:42.590 INFO:tasks.workunit.client.1.smithi111.stdout:9/778: link d0/d4/d10/d29/cb3 d0/d4/d1c/d2d/d3b/d42/cfd 0 2024-03-23T11:47:42.595 INFO:tasks.workunit.client.1.smithi111.stdout:5/760: rename d1/d3/d3a/ff5 to d1/dcf/ffb 0 2024-03-23T11:47:42.597 INFO:tasks.workunit.client.1.smithi111.stdout:2/764: symlink d2/d11/d17/d1f/l114 0 2024-03-23T11:47:42.598 INFO:tasks.workunit.client.0.smithi063.stdout:6/466: dwrite d1/d2a/d41/d42/f6f [0,4194304] 0 2024-03-23T11:47:42.601 INFO:tasks.workunit.client.1.smithi111.stdout:4/754: mknod d0/dc/d78/cf7 0 2024-03-23T11:47:42.602 INFO:tasks.workunit.client.1.smithi111.stdout:4/755: readlink d0/dc/d7a/l7d 0 2024-03-23T11:47:42.603 INFO:tasks.workunit.client.1.smithi111.stdout:1/703: truncate d0/d5/dd/d1e/f21 1148462 0 2024-03-23T11:47:42.610 INFO:tasks.workunit.client.0.smithi063.stdout:7/507: dwrite d5/d6/d19/d1f/d47/d4f/f93 [0,4194304] 0 2024-03-23T11:47:42.611 INFO:tasks.workunit.client.0.smithi063.stdout:6/467: mkdir d1/dc/d8c 0 2024-03-23T11:47:42.614 INFO:tasks.workunit.client.1.smithi111.stdout:1/704: mknod d0/d5/dd/d1e/d63/ce9 0 2024-03-23T11:47:42.616 INFO:tasks.workunit.client.0.smithi063.stdout:6/468: truncate d1/d15/f88 970595 0 2024-03-23T11:47:42.617 INFO:tasks.workunit.client.0.smithi063.stdout:8/809: dread d6/d1f/d54/d63/d90/d5d/d7d/db4/f10e [0,4194304] 0 2024-03-23T11:47:42.618 INFO:tasks.workunit.client.1.smithi111.stdout:2/765: dwrite d2/d11/d17/d3e/d5d/d8c/f10a [0,4194304] 0 2024-03-23T11:47:42.621 INFO:tasks.workunit.client.0.smithi063.stdout:8/810: write d6/d1f/d54/d63/d90/f13 [131422,62828] 0 2024-03-23T11:47:42.621 INFO:tasks.workunit.client.0.smithi063.stdout:7/508: unlink d5/d5c/d2f/d73/f7c 0 2024-03-23T11:47:42.623 INFO:tasks.workunit.client.1.smithi111.stdout:7/709: sync 2024-03-23T11:47:42.623 INFO:tasks.workunit.client.1.smithi111.stdout:6/722: sync 2024-03-23T11:47:42.625 INFO:tasks.workunit.client.1.smithi111.stdout:1/705: creat d0/d6/d48/d4a/de5/fea x:0 0 0 2024-03-23T11:47:42.626 INFO:tasks.workunit.client.1.smithi111.stdout:9/779: dread d0/d4/d1c/d36/d28/d69/f87 [0,4194304] 0 2024-03-23T11:47:42.626 INFO:tasks.workunit.client.0.smithi063.stdout:6/469: fdatasync d1/f67 0 2024-03-23T11:47:42.628 INFO:tasks.workunit.client.1.smithi111.stdout:2/766: dread - d2/d11/d17/d3e/d6b/d2e/d3f/faa zero size 2024-03-23T11:47:42.631 INFO:tasks.workunit.client.0.smithi063.stdout:8/811: mknod d6/dd/df/c111 0 2024-03-23T11:47:42.634 INFO:tasks.workunit.client.0.smithi063.stdout:6/470: symlink d1/dc/d3a/d57/d6d/l8d 0 2024-03-23T11:47:42.636 INFO:tasks.workunit.client.0.smithi063.stdout:8/812: mknod d6/d1c/dff/c112 0 2024-03-23T11:47:42.637 INFO:tasks.workunit.client.1.smithi111.stdout:9/780: dread - d0/d4/d1c/d36/fdc zero size 2024-03-23T11:47:42.637 INFO:tasks.workunit.client.0.smithi063.stdout:7/509: dread d5/d5c/d2f/d73/da/f17 [4194304,4194304] 0 2024-03-23T11:47:42.639 INFO:tasks.workunit.client.1.smithi111.stdout:0/695: write d5/db/d1f/f18 [3032469,114952] 0 2024-03-23T11:47:42.640 INFO:tasks.workunit.client.1.smithi111.stdout:1/706: symlink d0/d6/d48/d4a/d6c/leb 0 2024-03-23T11:47:42.641 INFO:tasks.workunit.client.1.smithi111.stdout:2/767: symlink d2/d32/l115 0 2024-03-23T11:47:42.644 INFO:tasks.workunit.client.1.smithi111.stdout:9/781: mkdir d0/d4/d1c/dfe 0 2024-03-23T11:47:42.645 INFO:tasks.workunit.client.0.smithi063.stdout:8/813: mkdir d6/d1f/d54/d5c/d66/da4/d113 0 2024-03-23T11:47:42.645 INFO:tasks.workunit.client.0.smithi063.stdout:7/510: write d5/d22/f94 [255170,19225] 0 2024-03-23T11:47:42.646 INFO:tasks.workunit.client.1.smithi111.stdout:5/761: dwrite d1/d20/d5c/f7e [0,4194304] 0 2024-03-23T11:47:42.647 INFO:tasks.workunit.client.0.smithi063.stdout:8/814: stat d6/d1f/d54/l91 0 2024-03-23T11:47:42.648 INFO:tasks.workunit.client.1.smithi111.stdout:0/696: mknod d5/db/ced 0 2024-03-23T11:47:42.649 INFO:tasks.workunit.client.0.smithi063.stdout:7/511: symlink d5/d6/d39/d88/la7 0 2024-03-23T11:47:42.650 INFO:tasks.workunit.client.0.smithi063.stdout:8/815: chown d6/dd/c7a 0 1 2024-03-23T11:47:42.653 INFO:tasks.workunit.client.1.smithi111.stdout:1/707: rmdir d0/d5/dd/d1e/d54/dbb 39 2024-03-23T11:47:42.656 INFO:tasks.workunit.client.1.smithi111.stdout:9/782: mknod d0/d4/d10/d29/d7f/d8e/dc3/cff 0 2024-03-23T11:47:42.658 INFO:tasks.workunit.client.1.smithi111.stdout:2/768: read d2/de/d6c/dd1/fe3 [400761,81041] 0 2024-03-23T11:47:42.658 INFO:tasks.workunit.client.1.smithi111.stdout:6/723: dwrite d2/d1f/fc6 [0,4194304] 0 2024-03-23T11:47:42.660 INFO:tasks.workunit.client.0.smithi063.stdout:2/485: sync 2024-03-23T11:47:42.660 INFO:tasks.workunit.client.0.smithi063.stdout:3/853: sync 2024-03-23T11:47:42.660 INFO:tasks.workunit.client.1.smithi111.stdout:1/708: write d0/d2b/d7c/dca/fd1 [5158,30312] 0 2024-03-23T11:47:42.661 INFO:tasks.workunit.client.0.smithi063.stdout:2/486: readlink d2/de/d22/l7e 0 2024-03-23T11:47:42.663 INFO:tasks.workunit.client.1.smithi111.stdout:6/724: rmdir d2/d1f 39 2024-03-23T11:47:42.664 INFO:tasks.workunit.client.1.smithi111.stdout:4/756: dread d0/dc/d1f/f8a [0,4194304] 0 2024-03-23T11:47:42.665 INFO:tasks.workunit.client.0.smithi063.stdout:2/487: fdatasync d2/de/d7a/d8d/f9e 0 2024-03-23T11:47:42.665 INFO:tasks.workunit.client.1.smithi111.stdout:5/762: rename d1/d20/d41/da9 to d1/d3/d6/d7/d63/d2c/d43/dfc 0 2024-03-23T11:47:42.667 INFO:tasks.workunit.client.1.smithi111.stdout:2/769: mknod d2/d5c/c116 0 2024-03-23T11:47:42.667 INFO:tasks.workunit.client.1.smithi111.stdout:1/709: chown d0/d5/dd/d1e/d54/c93 489 1 2024-03-23T11:47:42.667 INFO:tasks.workunit.client.1.smithi111.stdout:9/783: mknod d0/d4/d1c/c100 0 2024-03-23T11:47:42.668 INFO:tasks.workunit.client.0.smithi063.stdout:2/488: truncate d2/d23/d2e/f36 4657074 0 2024-03-23T11:47:42.669 INFO:tasks.workunit.client.1.smithi111.stdout:0/697: dread d5/db/d1f/d19/f53 [0,4194304] 0 2024-03-23T11:47:42.670 INFO:tasks.workunit.client.1.smithi111.stdout:6/725: chown d2/d1f/d45/d49/f9e 3 1 2024-03-23T11:47:42.671 INFO:tasks.workunit.client.1.smithi111.stdout:5/763: stat d1/d3/d6/d7/d63/d9d/la8 0 2024-03-23T11:47:42.672 INFO:tasks.workunit.client.1.smithi111.stdout:7/710: dwrite d14/d1b/d22/f29 [0,4194304] 0 2024-03-23T11:47:42.673 INFO:tasks.workunit.client.1.smithi111.stdout:0/698: chown d5/db/l2b 42942902 1 2024-03-23T11:47:42.675 INFO:tasks.workunit.client.1.smithi111.stdout:5/764: chown d1/d20/d41/fe5 7198 1 2024-03-23T11:47:42.677 INFO:tasks.workunit.client.1.smithi111.stdout:2/770: truncate d2/de/d6c/dd1/dd6/ffe 518415 0 2024-03-23T11:47:42.679 INFO:tasks.workunit.client.0.smithi063.stdout:2/489: symlink d2/d23/d2e/d52/d86/l9f 0 2024-03-23T11:47:42.680 INFO:tasks.workunit.client.1.smithi111.stdout:7/711: fdatasync d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb4 0 2024-03-23T11:47:42.682 INFO:tasks.workunit.client.1.smithi111.stdout:4/757: dread d0/d17/f2c [0,4194304] 0 2024-03-23T11:47:42.682 INFO:tasks.workunit.client.1.smithi111.stdout:7/712: dread - d14/d1b/d1e/d86/f93 zero size 2024-03-23T11:47:42.684 INFO:tasks.workunit.client.1.smithi111.stdout:7/713: fdatasync d14/d1b/d1e/d2c/d2e/d73/dae/f62 0 2024-03-23T11:47:42.685 INFO:tasks.workunit.client.1.smithi111.stdout:0/699: creat d5/db/d1f/de4/fee x:0 0 0 2024-03-23T11:47:42.687 INFO:tasks.workunit.client.1.smithi111.stdout:9/784: truncate d0/d4/d10/d29/d7f/f8a 1003967 0 2024-03-23T11:47:42.691 INFO:tasks.workunit.client.0.smithi063.stdout:2/490: mkdir d2/d23/d2e/d52/d86/da0 0 2024-03-23T11:47:42.692 INFO:tasks.workunit.client.1.smithi111.stdout:6/726: getdents d2/d1f 0 2024-03-23T11:47:42.694 INFO:tasks.workunit.client.0.smithi063.stdout:7/512: truncate d5/d6/d19/d1f/d47/d4f/d5b/f66 56298 0 2024-03-23T11:47:42.696 INFO:tasks.workunit.client.1.smithi111.stdout:3/667: sync 2024-03-23T11:47:42.696 INFO:tasks.workunit.client.1.smithi111.stdout:8/625: sync 2024-03-23T11:47:42.698 INFO:tasks.workunit.client.0.smithi063.stdout:2/491: truncate d2/d23/d2e/f55 244696 0 2024-03-23T11:47:42.698 INFO:tasks.workunit.client.1.smithi111.stdout:1/710: link d0/l9 d0/d2b/d37/lec 0 2024-03-23T11:47:42.700 INFO:tasks.workunit.client.1.smithi111.stdout:6/727: write d2/d35/fb5 [484398,80407] 0 2024-03-23T11:47:42.702 INFO:tasks.workunit.client.1.smithi111.stdout:7/714: unlink d14/d1b/d22/c6d 0 2024-03-23T11:47:42.705 INFO:tasks.workunit.client.1.smithi111.stdout:2/771: creat d2/d11/d17/d3e/d6b/d2e/f117 x:0 0 0 2024-03-23T11:47:42.706 INFO:tasks.workunit.client.0.smithi063.stdout:7/513: write d5/d6/d19/d1f/d6c/f80 [1494996,8693] 0 2024-03-23T11:47:42.706 INFO:tasks.workunit.client.1.smithi111.stdout:1/711: readlink d0/d6/d48/d4a/d6c/d96/lb7 0 2024-03-23T11:47:42.706 INFO:tasks.workunit.client.1.smithi111.stdout:0/700: write d5/db/d1f/d32/f3e [2931993,74288] 0 2024-03-23T11:47:42.711 INFO:tasks.workunit.client.0.smithi063.stdout:8/816: dread d6/d1f/f55 [0,4194304] 0 2024-03-23T11:47:42.713 INFO:tasks.workunit.client.1.smithi111.stdout:5/765: write d1/d3/d6/d30/db2/fc1 [1261714,7301] 0 2024-03-23T11:47:42.714 INFO:tasks.workunit.client.1.smithi111.stdout:9/785: dwrite d0/fbc [0,4194304] 0 2024-03-23T11:47:42.715 INFO:tasks.workunit.client.1.smithi111.stdout:7/715: rmdir d14/d1b/d1e/d2c/d2e/d33 39 2024-03-23T11:47:42.716 INFO:tasks.workunit.client.1.smithi111.stdout:5/766: readlink d1/l14 0 2024-03-23T11:47:42.717 INFO:tasks.workunit.client.1.smithi111.stdout:2/772: mkdir d2/d7/da/d118 0 2024-03-23T11:47:42.719 INFO:tasks.workunit.client.1.smithi111.stdout:5/767: stat d1/d3/d6/d7/d7b/fe7 0 2024-03-23T11:47:42.721 INFO:tasks.workunit.client.0.smithi063.stdout:8/817: fsync d6/d1c/d27/f99 0 2024-03-23T11:47:42.721 INFO:tasks.workunit.client.0.smithi063.stdout:3/854: dread d13/f50 [0,4194304] 0 2024-03-23T11:47:42.722 INFO:tasks.workunit.client.1.smithi111.stdout:1/712: mknod d0/d2b/d69/ced 0 2024-03-23T11:47:42.725 INFO:tasks.workunit.client.0.smithi063.stdout:2/492: dwrite d2/d23/f83 [0,4194304] 0 2024-03-23T11:47:42.728 INFO:tasks.workunit.client.1.smithi111.stdout:0/701: dread d5/db/d1f/d19/f23 [0,4194304] 0 2024-03-23T11:47:42.728 INFO:tasks.workunit.client.1.smithi111.stdout:5/768: rmdir d1/d3/d3a 39 2024-03-23T11:47:42.731 INFO:tasks.workunit.client.0.smithi063.stdout:8/818: creat d6/d1f/d54/d63/d90/f114 x:0 0 0 2024-03-23T11:47:42.732 INFO:tasks.workunit.client.1.smithi111.stdout:3/668: dwrite d0/d1a/d2b/d38/d7f/f9b [0,4194304] 0 2024-03-23T11:47:42.733 INFO:tasks.workunit.client.0.smithi063.stdout:2/493: chown d2/d23/d2e/d87/d4d/f6d 30 1 2024-03-23T11:47:42.736 INFO:tasks.workunit.client.0.smithi063.stdout:2/494: write d2/de/d7a/d77/f93 [87184,18856] 0 2024-03-23T11:47:42.736 INFO:tasks.workunit.client.1.smithi111.stdout:0/702: write d5/db/d1f/de4/da7/fc9 [186529,111072] 0 2024-03-23T11:47:42.737 INFO:tasks.workunit.client.1.smithi111.stdout:0/703: stat d5/db/ced 0 2024-03-23T11:47:42.740 INFO:tasks.workunit.client.1.smithi111.stdout:9/786: dread d0/f9b [0,4194304] 0 2024-03-23T11:47:42.742 INFO:tasks.workunit.client.1.smithi111.stdout:5/769: symlink d1/d3/d6/d7/d63/d2c/d43/d95/lfd 0 2024-03-23T11:47:42.744 INFO:tasks.workunit.client.1.smithi111.stdout:3/669: creat d0/d7/d19/d91/fdf x:0 0 0 2024-03-23T11:47:42.746 INFO:tasks.workunit.client.1.smithi111.stdout:1/713: link d0/d2b/d37/lec d0/d6/d48/lee 0 2024-03-23T11:47:42.747 INFO:tasks.workunit.client.1.smithi111.stdout:2/773: getdents d2/d11/d17/d3e/d6b/d9d/d52/dba 0 2024-03-23T11:47:42.748 INFO:tasks.workunit.client.1.smithi111.stdout:3/670: stat d0/d1a/d4f 0 2024-03-23T11:47:42.748 INFO:tasks.workunit.client.1.smithi111.stdout:7/716: write d14/d37/d43/f55 [5587932,19025] 0 2024-03-23T11:47:42.753 INFO:tasks.workunit.client.0.smithi063.stdout:2/495: creat d2/d84/d94/fa1 x:0 0 0 2024-03-23T11:47:42.754 INFO:tasks.workunit.client.1.smithi111.stdout:7/717: fdatasync d14/d1b/d1e/d2c/d42/d5e/d88/f94 0 2024-03-23T11:47:42.755 INFO:tasks.workunit.client.0.smithi063.stdout:7/514: truncate d5/d5c/d2f/d73/d20/f53 3610655 0 2024-03-23T11:47:42.758 INFO:tasks.workunit.client.0.smithi063.stdout:7/515: readlink d5/d5c/d2f/d73/da/l63 0 2024-03-23T11:47:42.758 INFO:tasks.workunit.client.1.smithi111.stdout:5/770: fsync d1/d3/d3a/fd6 0 2024-03-23T11:47:42.760 INFO:tasks.workunit.client.1.smithi111.stdout:1/714: mkdir d0/def 0 2024-03-23T11:47:42.760 INFO:tasks.workunit.client.1.smithi111.stdout:0/704: dread d5/db/d1f/de4/f69 [0,4194304] 0 2024-03-23T11:47:42.765 INFO:tasks.workunit.client.0.smithi063.stdout:3/855: write d13/d22/f3b [901888,80228] 0 2024-03-23T11:47:42.766 INFO:tasks.workunit.client.1.smithi111.stdout:1/715: readlink d0/d5/dd/d1e/d54/l68 0 2024-03-23T11:47:42.766 INFO:tasks.workunit.client.1.smithi111.stdout:6/728: dread d2/d1a/d1c/d2c/d51/f2f [0,4194304] 0 2024-03-23T11:47:42.767 INFO:tasks.workunit.client.1.smithi111.stdout:1/716: chown d0/c18 88829 1 2024-03-23T11:47:42.768 INFO:tasks.workunit.client.0.smithi063.stdout:3/856: stat d13/d15/f1b 0 2024-03-23T11:47:42.769 INFO:tasks.workunit.client.0.smithi063.stdout:7/516: creat d5/d6/d19/fa8 x:0 0 0 2024-03-23T11:47:42.770 INFO:tasks.workunit.client.1.smithi111.stdout:2/774: creat d2/d11/d17/d1f/da0/f119 x:0 0 0 2024-03-23T11:47:42.770 INFO:tasks.workunit.client.1.smithi111.stdout:3/671: mknod d0/d1a/d1d/d26/d5a/ce0 0 2024-03-23T11:47:42.771 INFO:tasks.workunit.client.1.smithi111.stdout:5/771: creat d1/d3/d6/d8b/d59/ffe x:0 0 0 2024-03-23T11:47:42.772 INFO:tasks.workunit.client.1.smithi111.stdout:7/718: mknod d14/d1b/d1e/d2c/d42/d5e/d88/d5f/ccf 0 2024-03-23T11:47:42.773 INFO:tasks.workunit.client.1.smithi111.stdout:1/717: creat d0/ff0 x:0 0 0 2024-03-23T11:47:42.773 INFO:tasks.workunit.client.1.smithi111.stdout:6/729: write d2/d1a/d1c/d2c/fb9 [3336302,54829] 0 2024-03-23T11:47:42.776 INFO:tasks.workunit.client.0.smithi063.stdout:7/517: readlink d5/d6/d19/d1f/d47/d4f/l5f 0 2024-03-23T11:47:42.778 INFO:tasks.workunit.client.1.smithi111.stdout:9/787: write d0/d31/d41/d9d/dd4/ff7 [1148294,99027] 0 2024-03-23T11:47:42.779 INFO:tasks.workunit.client.1.smithi111.stdout:7/719: write d14/d1b/d1e/d2c/d2e/d73/fc2 [526786,107693] 0 2024-03-23T11:47:42.779 INFO:tasks.workunit.client.0.smithi063.stdout:3/857: unlink lb 0 2024-03-23T11:47:42.781 INFO:tasks.workunit.client.1.smithi111.stdout:1/718: unlink d0/d5/f2f 0 2024-03-23T11:47:42.781 INFO:tasks.workunit.client.1.smithi111.stdout:6/730: chown d2/d1f/d45/d49/f9e 3861791 1 2024-03-23T11:47:42.783 INFO:tasks.workunit.client.0.smithi063.stdout:2/496: dwrite d2/d23/d2e/d87/f82 [0,4194304] 0 2024-03-23T11:47:42.783 INFO:tasks.workunit.client.0.smithi063.stdout:3/858: fdatasync d13/d22/d2d/d3f/d67/fd1 0 2024-03-23T11:47:42.784 INFO:tasks.workunit.client.1.smithi111.stdout:6/731: chown d2/d35/f77 105277 1 2024-03-23T11:47:42.785 INFO:tasks.workunit.client.1.smithi111.stdout:0/705: creat d5/fef x:0 0 0 2024-03-23T11:47:42.785 INFO:tasks.workunit.client.0.smithi063.stdout:8/819: dwrite d6/d1c/d32/d37/d10a/dae/fd1 [0,4194304] 0 2024-03-23T11:47:42.786 INFO:tasks.workunit.client.1.smithi111.stdout:5/772: dread d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:42.789 INFO:tasks.workunit.client.1.smithi111.stdout:1/719: creat d0/ff1 x:0 0 0 2024-03-23T11:47:42.790 INFO:tasks.workunit.client.1.smithi111.stdout:9/788: mknod d0/d4/d1c/d2d/d3b/d42/d92/c101 0 2024-03-23T11:47:42.791 INFO:tasks.workunit.client.1.smithi111.stdout:0/706: write d5/f30 [1122471,71913] 0 2024-03-23T11:47:42.792 INFO:tasks.workunit.client.1.smithi111.stdout:1/720: chown d0/d2b/d37/l85 13668375 1 2024-03-23T11:47:42.794 INFO:tasks.workunit.client.0.smithi063.stdout:2/497: fdatasync d2/d23/d2e/f4b 0 2024-03-23T11:47:42.795 INFO:tasks.workunit.client.1.smithi111.stdout:0/707: write d5/db/d4d/fc6 [181705,68564] 0 2024-03-23T11:47:42.795 INFO:tasks.workunit.client.1.smithi111.stdout:1/721: stat d0/d2b/d69/ced 0 2024-03-23T11:47:42.798 INFO:tasks.workunit.client.0.smithi063.stdout:3/859: creat d13/d22/d2d/d3f/d67/d7b/f10c x:0 0 0 2024-03-23T11:47:42.799 INFO:tasks.workunit.client.1.smithi111.stdout:7/720: mknod d14/d1b/d1e/d2c/d2e/d95/cd0 0 2024-03-23T11:47:42.800 INFO:tasks.workunit.client.1.smithi111.stdout:0/708: chown d5/db/d1f/d47/d59/cc4 44733436 1 2024-03-23T11:47:42.801 INFO:tasks.workunit.client.0.smithi063.stdout:2/498: rmdir d2/d23/d2e/d87 39 2024-03-23T11:47:42.802 INFO:tasks.workunit.client.0.smithi063.stdout:8/820: creat d6/d1c/d32/d37/d10a/dc8/dfc/f115 x:0 0 0 2024-03-23T11:47:42.803 INFO:tasks.workunit.client.1.smithi111.stdout:5/773: rename d1/d3/d6/d7/d63/d2c/d82/d66/fef to d1/d20/dce/fff 0 2024-03-23T11:47:42.806 INFO:tasks.workunit.client.0.smithi063.stdout:3/860: rmdir d13/d22/d2d/d78 39 2024-03-23T11:47:42.807 INFO:tasks.workunit.client.0.smithi063.stdout:2/499: mkdir d2/de/d7a/da2 0 2024-03-23T11:47:42.808 INFO:tasks.workunit.client.1.smithi111.stdout:3/672: link d0/d7/l25 d0/d72/d7b/le1 0 2024-03-23T11:47:42.809 INFO:tasks.workunit.client.1.smithi111.stdout:0/709: rmdir d5/db/d1f/d19 39 2024-03-23T11:47:42.809 INFO:tasks.workunit.client.1.smithi111.stdout:3/673: chown d0/fbe 967764 1 2024-03-23T11:47:42.809 INFO:tasks.workunit.client.0.smithi063.stdout:8/821: fdatasync d6/d1f/d54/d63/fc2 0 2024-03-23T11:47:42.810 INFO:tasks.workunit.client.0.smithi063.stdout:7/518: write d5/d5c/d2f/d73/f3b [2133431,17228] 0 2024-03-23T11:47:42.811 INFO:tasks.workunit.client.1.smithi111.stdout:5/774: unlink d1/d3/fc 0 2024-03-23T11:47:42.812 INFO:tasks.workunit.client.1.smithi111.stdout:6/732: creat d2/d1a/d1c/d2c/d51/fee x:0 0 0 2024-03-23T11:47:42.818 INFO:tasks.workunit.client.1.smithi111.stdout:0/710: creat d5/db/d1f/de4/da7/ff0 x:0 0 0 2024-03-23T11:47:42.822 INFO:tasks.workunit.client.1.smithi111.stdout:9/789: dwrite d0/d4/d10/fbb [0,4194304] 0 2024-03-23T11:47:42.823 INFO:tasks.workunit.client.0.smithi063.stdout:2/500: getdents d2/de/d22/d91 0 2024-03-23T11:47:42.825 INFO:tasks.workunit.client.1.smithi111.stdout:9/790: write d0/d4/d1c/d36/d28/d69/fd2 [738899,118734] 0 2024-03-23T11:47:42.832 INFO:tasks.workunit.client.0.smithi063.stdout:3/861: getdents d13/d22/d2d/d3f/d58/d97/dac 0 2024-03-23T11:47:42.832 INFO:tasks.workunit.client.1.smithi111.stdout:1/722: rename d0/d5/dd/fc0 to d0/d6/d48/d4a/ff2 0 2024-03-23T11:47:42.834 INFO:tasks.workunit.client.0.smithi063.stdout:7/519: dread d5/d5c/d2f/d73/da/d90/f9b [0,4194304] 0 2024-03-23T11:47:42.840 INFO:tasks.workunit.client.0.smithi063.stdout:8/822: dwrite d6/d1c/ff9 [0,4194304] 0 2024-03-23T11:47:42.846 INFO:tasks.workunit.client.1.smithi111.stdout:7/721: dwrite d14/d1b/d22/f4d [0,4194304] 0 2024-03-23T11:47:42.848 INFO:tasks.workunit.client.0.smithi063.stdout:8/823: mknod d6/d1c/d32/d37/d10a/dae/c116 0 2024-03-23T11:47:42.850 INFO:tasks.workunit.client.1.smithi111.stdout:5/775: link d1/d3/d6/d8b/d59/d74/cb9 d1/d3/d6/d30/db2/dcc/c100 0 2024-03-23T11:47:42.853 INFO:tasks.workunit.client.1.smithi111.stdout:1/723: mknod d0/d5/dd/dc6/cf3 0 2024-03-23T11:47:42.854 INFO:tasks.workunit.client.1.smithi111.stdout:3/674: dread d0/d1a/d1d/d26/f46 [0,4194304] 0 2024-03-23T11:47:42.855 INFO:tasks.workunit.client.0.smithi063.stdout:3/862: dread - d13/de5/ffd zero size 2024-03-23T11:47:42.855 INFO:tasks.workunit.client.1.smithi111.stdout:6/733: getdents d2/d1a/d1c 0 2024-03-23T11:47:42.856 INFO:tasks.workunit.client.1.smithi111.stdout:1/724: write d0/f1 [3408463,18604] 0 2024-03-23T11:47:42.856 INFO:tasks.workunit.client.0.smithi063.stdout:8/824: truncate d6/dd/d59/db2/fe2 327826 0 2024-03-23T11:47:42.857 INFO:tasks.workunit.client.1.smithi111.stdout:3/675: dread - d0/d55/f97 zero size 2024-03-23T11:47:42.862 INFO:tasks.workunit.client.0.smithi063.stdout:7/520: dwrite d5/d6/f7a [0,4194304] 0 2024-03-23T11:47:42.867 INFO:tasks.workunit.client.0.smithi063.stdout:7/521: chown d5/d6/d39/c84 7754399 1 2024-03-23T11:47:42.867 INFO:tasks.workunit.client.1.smithi111.stdout:5/776: creat d1/d20/dce/f101 x:0 0 0 2024-03-23T11:47:42.867 INFO:tasks.workunit.client.1.smithi111.stdout:0/711: dread d5/db/d1f/d48/f6a [0,4194304] 0 2024-03-23T11:47:42.869 INFO:tasks.workunit.client.1.smithi111.stdout:9/791: truncate d0/d4/d1c/d2d/fb7 323191 0 2024-03-23T11:47:42.870 INFO:tasks.workunit.client.0.smithi063.stdout:3/863: mkdir d13/d22/d2d/d3f/d58/d10d 0 2024-03-23T11:47:42.872 INFO:tasks.workunit.client.1.smithi111.stdout:1/725: symlink d0/d5/dd/d1e/lf4 0 2024-03-23T11:47:42.873 INFO:tasks.workunit.client.0.smithi063.stdout:3/864: write d13/d22/fe6 [2017280,9813] 0 2024-03-23T11:47:42.876 INFO:tasks.workunit.client.0.smithi063.stdout:3/865: stat d13/d22/d2d/d3f/dd4 0 2024-03-23T11:47:42.876 INFO:tasks.workunit.client.0.smithi063.stdout:7/522: getdents d5/d6/d19/d65 0 2024-03-23T11:47:42.877 INFO:tasks.workunit.client.0.smithi063.stdout:3/866: chown d13/d22 478570909 1 2024-03-23T11:47:42.881 INFO:tasks.workunit.client.0.smithi063.stdout:3/867: chown d13/d66/dde/cf9 326 1 2024-03-23T11:47:42.883 INFO:tasks.workunit.client.1.smithi111.stdout:7/722: dwrite d14/d1b/d1e/d2c/d42/f45 [4194304,4194304] 0 2024-03-23T11:47:42.883 INFO:tasks.workunit.client.0.smithi063.stdout:7/523: truncate d5/d5c/d2f/f79 176279 0 2024-03-23T11:47:42.884 INFO:tasks.workunit.client.1.smithi111.stdout:3/676: dwrite d0/d1a/d1d/f49 [0,4194304] 0 2024-03-23T11:47:42.885 INFO:tasks.workunit.client.0.smithi063.stdout:7/524: write d5/d5c/d2f/f79 [717803,706] 0 2024-03-23T11:47:42.890 INFO:tasks.workunit.client.0.smithi063.stdout:3/868: mknod d13/d22/d2d/c10e 0 2024-03-23T11:47:42.892 INFO:tasks.workunit.client.1.smithi111.stdout:3/677: write d0/d1a/d2b/d38/dc2/fcc [1764117,127345] 0 2024-03-23T11:47:42.892 INFO:tasks.workunit.client.1.smithi111.stdout:5/777: creat d1/d3/d6/d30/f102 x:0 0 0 2024-03-23T11:47:42.892 INFO:tasks.workunit.client.1.smithi111.stdout:4/758: sync 2024-03-23T11:47:42.894 INFO:tasks.workunit.client.1.smithi111.stdout:8/626: sync 2024-03-23T11:47:42.894 INFO:tasks.workunit.client.1.smithi111.stdout:7/723: fsync d14/d1b/d1e/d2c/d42/d5e/f6c 0 2024-03-23T11:47:42.895 INFO:tasks.workunit.client.0.smithi063.stdout:7/525: fdatasync d5/d5c/d2f/d73/da/f4e 0 2024-03-23T11:47:42.895 INFO:tasks.workunit.client.0.smithi063.stdout:3/869: rmdir d13/d22/d2d/d3f 39 2024-03-23T11:47:42.896 INFO:tasks.workunit.client.0.smithi063.stdout:7/526: readlink d5/d6/d19/d1f/l68 0 2024-03-23T11:47:42.896 INFO:tasks.workunit.client.0.smithi063.stdout:9/584: sync 2024-03-23T11:47:42.896 INFO:tasks.workunit.client.0.smithi063.stdout:0/677: sync 2024-03-23T11:47:42.897 INFO:tasks.workunit.client.1.smithi111.stdout:0/712: dread d5/db/d1f/d47/d59/f99 [0,4194304] 0 2024-03-23T11:47:42.902 INFO:tasks.workunit.client.1.smithi111.stdout:3/678: mkdir d0/d55/de2 0 2024-03-23T11:47:42.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:42 smithi111 ceph-mon[44005]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:42.903 INFO:tasks.workunit.client.0.smithi063.stdout:0/678: chown d2/d14/d36/dc3 23 1 2024-03-23T11:47:42.904 INFO:tasks.workunit.client.0.smithi063.stdout:3/870: stat d13/d15/f2a 0 2024-03-23T11:47:42.905 INFO:tasks.workunit.client.1.smithi111.stdout:5/778: symlink d1/d3/d6/d7/dee/l103 0 2024-03-23T11:47:42.905 INFO:tasks.workunit.client.1.smithi111.stdout:1/726: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:42.908 INFO:tasks.workunit.client.1.smithi111.stdout:7/724: readlink d14/d1b/d1e/d2c/d2e/d33/l6a 0 2024-03-23T11:47:42.909 INFO:tasks.workunit.client.0.smithi063.stdout:8/825: dwrite d6/dd/df/f60 [0,4194304] 0 2024-03-23T11:47:42.914 INFO:tasks.workunit.client.0.smithi063.stdout:8/826: stat d6/d30 0 2024-03-23T11:47:42.917 INFO:tasks.workunit.client.0.smithi063.stdout:9/585: rename d4/d4c/fc7 to d4/d31/d70/d8d/fcb 0 2024-03-23T11:47:42.920 INFO:tasks.workunit.client.1.smithi111.stdout:9/792: dread d0/d4/d1c/d2d/d3b/d42/d4f/f5c [0,4194304] 0 2024-03-23T11:47:42.922 INFO:tasks.workunit.client.1.smithi111.stdout:6/734: dwrite d2/f41 [0,4194304] 0 2024-03-23T11:47:42.922 INFO:tasks.workunit.client.0.smithi063.stdout:7/527: dread d5/d6/d19/d1f/f6d [0,4194304] 0 2024-03-23T11:47:42.922 INFO:tasks.workunit.client.0.smithi063.stdout:9/586: truncate d4/d31/d43/d49/d8e/f93 365035 0 2024-03-23T11:47:42.929 INFO:tasks.workunit.client.1.smithi111.stdout:5/779: mkdir d1/d3/d6/d7/d63/d9d/d104 0 2024-03-23T11:47:42.930 INFO:tasks.workunit.client.0.smithi063.stdout:0/679: creat d2/d14/d36/d77/d94/da6/d25/d93/ff7 x:0 0 0 2024-03-23T11:47:42.931 INFO:tasks.workunit.client.1.smithi111.stdout:4/759: link d0/dc/d4c/d2a/f8b d0/d95/db5/dc7/ff8 0 2024-03-23T11:47:42.936 INFO:tasks.workunit.client.1.smithi111.stdout:7/725: rename d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb4 to d14/d1b/d1e/d2c/d2e/d33/fd1 0 2024-03-23T11:47:42.938 INFO:tasks.workunit.client.1.smithi111.stdout:8/627: creat d5/d26/d2f/d62/d7e/dae/fcb x:0 0 0 2024-03-23T11:47:42.940 INFO:tasks.workunit.client.1.smithi111.stdout:8/628: stat d5/f19 0 2024-03-23T11:47:42.940 INFO:tasks.workunit.client.1.smithi111.stdout:3/679: dwrite d0/d53/fcf [0,4194304] 0 2024-03-23T11:47:42.944 INFO:tasks.workunit.client.1.smithi111.stdout:2/775: sync 2024-03-23T11:47:42.944 INFO:tasks.workunit.client.1.smithi111.stdout:0/713: mknod d5/db/d1f/d19/d45/cf1 0 2024-03-23T11:47:42.944 INFO:tasks.workunit.client.1.smithi111.stdout:6/735: dwrite d2/d1f/d45/d86/fe2 [0,4194304] 0 2024-03-23T11:47:42.946 INFO:tasks.workunit.client.1.smithi111.stdout:2/776: dread - d2/d11/d17/d1f/fd9 zero size 2024-03-23T11:47:42.947 INFO:tasks.workunit.client.1.smithi111.stdout:4/760: chown d0/c36 344512 1 2024-03-23T11:47:42.948 INFO:tasks.workunit.client.0.smithi063.stdout:0/680: unlink d2/d14/f15 0 2024-03-23T11:47:42.949 INFO:tasks.workunit.client.0.smithi063.stdout:9/587: symlink d4/d31/d43/da9/lcc 0 2024-03-23T11:47:42.952 INFO:tasks.workunit.client.1.smithi111.stdout:7/726: mkdir d14/d1b/d1e/d2c/d2e/d33/d98/dd2 0 2024-03-23T11:47:42.954 INFO:tasks.workunit.client.0.smithi063.stdout:0/681: dread - d2/d14/d36/d77/d94/da6/d28/fdf zero size 2024-03-23T11:47:42.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.970+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f87fc106900 con 0x7f87fc06fda0 2024-03-23T11:47:42.956 INFO:tasks.workunit.client.1.smithi111.stdout:8/629: mkdir d5/d26/d2f/d62/db5/dcc 0 2024-03-23T11:47:42.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.971+0000 7f87f2a00700 1 -- 172.21.15.63:0/3826751852 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+848 (secure 0 0 0) 0x7f87e8055420 con 0x7f87fc06fda0 2024-03-23T11:47:42.956 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:47:42.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:47:42.958 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:47:42.958 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 13, 2024-03-23T11:47:42.958 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:47:42.958 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:47:42.958 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:47:42.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f87e006aca0 msgr2=0x7f87e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:42.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f87e006aca0 0x7f87e006d150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87ec006fd0 tx=0x7f87ec006d30 comp rx=0 tx=0).stop 2024-03-23T11:47:42.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06fda0 msgr2=0x7f87fc11cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:42.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06fda0 0x7f87fc11cbd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f87e80043d0 tx=0x7f87e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:42.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 shutdown_connections 2024-03-23T11:47:42.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f87e006aca0 0x7f87e006d150 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87ec006fd0 tx=0x7f87ec006d30 comp rx=0 tx=0).stop 2024-03-23T11:47:42.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87fc06eb00 0x7f87fc11c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 --2- 172.21.15.63:0/3826751852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87fc06fda0 0x7f87fc11cbd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:42.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.975+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 >> 172.21.15.63:0/3826751852 conn(0x7f87fc06a380 msgr2=0x7f87fc06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:42.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.976+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 shutdown_connections 2024-03-23T11:47:42.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:42.976+0000 7f8801e00700 1 -- 172.21.15.63:0/3826751852 wait complete. 2024-03-23T11:47:42.962 INFO:tasks.workunit.client.1.smithi111.stdout:0/714: mkdir d5/db/d1f/d19/dbd/d64/df2 0 2024-03-23T11:47:42.962 INFO:tasks.workunit.client.0.smithi063.stdout:0/682: unlink d2/d12/d32/f6c 0 2024-03-23T11:47:42.963 INFO:tasks.workunit.client.0.smithi063.stdout:9/588: mkdir d4/dcd 0 2024-03-23T11:47:42.964 INFO:tasks.workunit.client.1.smithi111.stdout:0/715: fsync d5/db/d1f/d47/d59/d5b/fb5 0 2024-03-23T11:47:42.969 INFO:tasks.workunit.client.1.smithi111.stdout:4/761: symlink d0/d95/d9b/lf9 0 2024-03-23T11:47:42.972 INFO:tasks.workunit.client.1.smithi111.stdout:3/680: mknod d0/d1a/d1d/d26/ce3 0 2024-03-23T11:47:42.973 INFO:tasks.workunit.client.0.smithi063.stdout:9/589: unlink d4/d31/d70/d8d/fcb 0 2024-03-23T11:47:42.975 INFO:tasks.workunit.client.0.smithi063.stdout:0/683: mknod d2/d14/d36/d77/d94/da6/d25/d93/cf8 0 2024-03-23T11:47:42.975 INFO:tasks.workunit.client.1.smithi111.stdout:8/630: mknod d5/d26/d2f/d62/d75/d8d/ccd 0 2024-03-23T11:47:42.977 INFO:tasks.workunit.client.1.smithi111.stdout:1/727: getdents d0/d5/dd/d1e 0 2024-03-23T11:47:42.979 INFO:tasks.workunit.client.1.smithi111.stdout:6/736: mkdir d2/d1a/d1c/d2c/d51/def 0 2024-03-23T11:47:42.979 INFO:tasks.workunit.client.1.smithi111.stdout:7/727: dwrite d14/d1b/d22/f4d [4194304,4194304] 0 2024-03-23T11:47:42.984 INFO:tasks.workunit.client.0.smithi063.stdout:0/684: mknod d2/d14/d36/d77/d94/da6/cf9 0 2024-03-23T11:47:42.986 INFO:tasks.workunit.client.1.smithi111.stdout:2/777: dread d2/d11/d17/d3e/f72 [0,4194304] 0 2024-03-23T11:47:42.986 INFO:tasks.workunit.client.1.smithi111.stdout:6/737: readlink d2/d1a/d1c/l3a 0 2024-03-23T11:47:42.988 INFO:tasks.workunit.client.1.smithi111.stdout:9/793: getdents d0 0 2024-03-23T11:47:42.991 INFO:tasks.workunit.client.0.smithi063.stdout:8/827: dwrite d6/dd/fde [0,4194304] 0 2024-03-23T11:47:42.991 INFO:tasks.workunit.client.0.smithi063.stdout:9/590: dread d4/d31/d43/da9/f95 [0,4194304] 0 2024-03-23T11:47:42.991 INFO:tasks.workunit.client.0.smithi063.stdout:7/528: dwrite f1 [0,4194304] 0 2024-03-23T11:47:42.992 INFO:tasks.workunit.client.1.smithi111.stdout:8/631: mkdir d5/d26/d2f/d62/db5/dce 0 2024-03-23T11:47:42.992 INFO:tasks.workunit.client.1.smithi111.stdout:5/780: truncate d1/d3/d6/f57 914573 0 2024-03-23T11:47:42.994 INFO:tasks.workunit.client.1.smithi111.stdout:7/728: rmdir d14/d1b/d1e/d2c/d42/d5e/d9e 39 2024-03-23T11:47:42.995 INFO:tasks.workunit.client.1.smithi111.stdout:6/738: creat d2/d1f/ff0 x:0 0 0 2024-03-23T11:47:42.998 INFO:tasks.workunit.client.1.smithi111.stdout:2/778: write d2/d11/f1b [3524733,26176] 0 2024-03-23T11:47:42.998 INFO:tasks.workunit.client.1.smithi111.stdout:9/794: mkdir d0/d31/d41/d71/d8f/d102 0 2024-03-23T11:47:43.000 INFO:tasks.workunit.client.1.smithi111.stdout:6/739: chown d2/d17/c50 217 1 2024-03-23T11:47:43.001 INFO:tasks.workunit.client.1.smithi111.stdout:3/681: fsync d0/d7/d19/f29 0 2024-03-23T11:47:43.006 INFO:tasks.workunit.client.1.smithi111.stdout:5/781: dread d1/d3/d6/d8b/d59/d6c/fc8 [0,4194304] 0 2024-03-23T11:47:43.007 INFO:tasks.workunit.client.1.smithi111.stdout:0/716: write d5/db/d1f/d48/f58 [4964703,112736] 0 2024-03-23T11:47:43.010 INFO:tasks.workunit.client.1.smithi111.stdout:1/728: link d0/d5/cf d0/d5/dd/d1e/d70/cf5 0 2024-03-23T11:47:43.012 INFO:tasks.workunit.client.0.smithi063.stdout:9/591: link d4/d4c/d19/l1d d4/d31/lce 0 2024-03-23T11:47:43.015 INFO:tasks.workunit.client.0.smithi063.stdout:8/828: write d6/fbf [30066,126797] 0 2024-03-23T11:47:43.015 INFO:tasks.workunit.client.0.smithi063.stdout:7/529: getdents d5/d22/d57/d64 0 2024-03-23T11:47:43.016 INFO:tasks.workunit.client.1.smithi111.stdout:9/795: write d0/d4/d10/d29/f67 [4442616,73662] 0 2024-03-23T11:47:43.016 INFO:tasks.workunit.client.1.smithi111.stdout:6/740: dread - d2/fa2 zero size 2024-03-23T11:47:43.016 INFO:tasks.workunit.client.1.smithi111.stdout:7/729: mkdir d14/d1b/d1e/d2c/d42/d9c/dd3 0 2024-03-23T11:47:43.021 INFO:tasks.workunit.client.1.smithi111.stdout:2/779: chown d2/d11/d17/d3e/d6b/d2e/d3f/f44 1829479119 1 2024-03-23T11:47:43.024 INFO:tasks.workunit.client.1.smithi111.stdout:4/762: dwrite d0/fc3 [0,4194304] 0 2024-03-23T11:47:43.026 INFO:tasks.workunit.client.0.smithi063.stdout:9/592: creat d4/d31/d43/d49/fcf x:0 0 0 2024-03-23T11:47:43.028 INFO:tasks.workunit.client.1.smithi111.stdout:0/717: rename d5/db/f4a to d5/db/ff3 0 2024-03-23T11:47:43.030 INFO:tasks.workunit.client.1.smithi111.stdout:7/730: read - d14/d1b/d1e/d2c/d2e/d73/dae/d74/fbd zero size 2024-03-23T11:47:43.034 INFO:tasks.workunit.client.1.smithi111.stdout:8/632: rename d5/d26/d31/d5e/d4a to d5/d32/dcf 0 2024-03-23T11:47:43.040 INFO:tasks.workunit.client.1.smithi111.stdout:6/741: unlink d2/d1a/d1c/f24 0 2024-03-23T11:47:43.040 INFO:tasks.workunit.client.1.smithi111.stdout:3/682: dread d0/d1a/d1d/f24 [0,4194304] 0 2024-03-23T11:47:43.040 INFO:tasks.workunit.client.1.smithi111.stdout:8/633: write d5/d26/d2f/f77 [2959577,76521] 0 2024-03-23T11:47:43.041 INFO:tasks.workunit.client.1.smithi111.stdout:9/796: unlink d0/d4/c12 0 2024-03-23T11:47:43.048 INFO:tasks.workunit.client.0.smithi063.stdout:7/530: dwrite d5/d5c/d2f/d73/d20/f31 [4194304,4194304] 0 2024-03-23T11:47:43.051 INFO:tasks.workunit.client.1.smithi111.stdout:4/763: creat d0/d95/db5/dc7/ffa x:0 0 0 2024-03-23T11:47:43.051 INFO:tasks.workunit.client.1.smithi111.stdout:6/742: fsync d2/d1a/d1c/d2c/d51/f5f 0 2024-03-23T11:47:43.052 INFO:tasks.workunit.client.1.smithi111.stdout:2/780: write d2/f5b [1984167,51267] 0 2024-03-23T11:47:43.054 INFO:tasks.workunit.client.1.smithi111.stdout:8/634: write d5/d26/d31/f39 [2538636,83258] 0 2024-03-23T11:47:43.056 INFO:tasks.workunit.client.0.smithi063.stdout:0/685: dread d2/d12/f44 [0,4194304] 0 2024-03-23T11:47:43.062 INFO:tasks.workunit.client.0.smithi063.stdout:8/829: dwrite d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/fdc [0,4194304] 0 2024-03-23T11:47:43.063 INFO:tasks.workunit.client.1.smithi111.stdout:0/718: dwrite d5/db/d1f/f79 [4194304,4194304] 0 2024-03-23T11:47:43.065 INFO:tasks.workunit.client.0.smithi063.stdout:7/531: rename d5/d5c/d2f/d73/d20/f53 to d5/d6/d39/d88/fa9 0 2024-03-23T11:47:43.065 INFO:tasks.workunit.client.0.smithi063.stdout:9/593: dwrite d4/d4c/d44/f84 [0,4194304] 0 2024-03-23T11:47:43.072 INFO:tasks.workunit.client.0.smithi063.stdout:9/594: dread - d4/d31/d70/d8d/fc3 zero size 2024-03-23T11:47:43.076 INFO:tasks.workunit.client.0.smithi063.stdout:0/686: unlink d2/d14/d36/d77/d94/da6/d28/d46/d51/cca 0 2024-03-23T11:47:43.082 INFO:tasks.workunit.client.0.smithi063.stdout:7/532: mkdir d5/d5c/d2f/d8b/daa 0 2024-03-23T11:47:43.086 INFO:tasks.workunit.client.0.smithi063.stdout:9/595: rmdir d4/d31/d43/d49 39 2024-03-23T11:47:43.090 INFO:tasks.workunit.client.1.smithi111.stdout:1/729: rename d0/d6/d48/db2 to d0/d5/dd5/df6 0 2024-03-23T11:47:43.093 INFO:tasks.workunit.client.0.smithi063.stdout:0/687: rename d2/d14/ff0 to d2/d14/d36/d77/d94/da6/d1f/ffa 0 2024-03-23T11:47:43.095 INFO:tasks.workunit.client.1.smithi111.stdout:4/764: mkdir d0/d95/db5/dfb 0 2024-03-23T11:47:43.095 INFO:tasks.workunit.client.1.smithi111.stdout:0/719: mkdir d5/db/d1f/d19/dbd/d64/dc7/df4 0 2024-03-23T11:47:43.097 INFO:tasks.workunit.client.1.smithi111.stdout:9/797: mkdir d0/d31/d41/df4/d103 0 2024-03-23T11:47:43.100 INFO:tasks.workunit.client.1.smithi111.stdout:7/731: creat d14/d1b/d1e/d2c/fd4 x:0 0 0 2024-03-23T11:47:43.101 INFO:tasks.workunit.client.1.smithi111.stdout:3/683: rename d0/d1a/fb3 to d0/d7/d19/d84/fe4 0 2024-03-23T11:47:43.102 INFO:tasks.workunit.client.1.smithi111.stdout:7/732: fdatasync d14/d1b/d1e/d2c/d2e/d73/fc2 0 2024-03-23T11:47:43.103 INFO:tasks.workunit.client.1.smithi111.stdout:4/765: rmdir d0/dc/d4c/d83 39 2024-03-23T11:47:43.104 INFO:tasks.workunit.client.1.smithi111.stdout:5/782: getdents d1/d3/d6/d7/d63/dd9/df2 0 2024-03-23T11:47:43.105 INFO:tasks.workunit.client.1.smithi111.stdout:7/733: fsync d14/d1b/d22/f29 0 2024-03-23T11:47:43.106 INFO:tasks.workunit.client.1.smithi111.stdout:9/798: mknod d0/d4/d6a/c104 0 2024-03-23T11:47:43.107 INFO:tasks.workunit.client.1.smithi111.stdout:5/783: fsync d1/d3/d6/d7/d8c/fd7 0 2024-03-23T11:47:43.107 INFO:tasks.workunit.client.0.smithi063.stdout:8/830: rename d6/d1f/d54/d63/d90/d9e/fd7 to d6/d1f/d54/d63/d90/d15/d9d/f117 0 2024-03-23T11:47:43.108 INFO:tasks.workunit.client.1.smithi111.stdout:6/743: rename d2/d1f/d45/d86/fe2 to d2/d17/ff1 0 2024-03-23T11:47:43.110 INFO:tasks.workunit.client.1.smithi111.stdout:3/684: unlink d0/d72/l77 0 2024-03-23T11:47:43.111 INFO:tasks.workunit.client.0.smithi063.stdout:1/478: sync 2024-03-23T11:47:43.111 INFO:tasks.workunit.client.1.smithi111.stdout:5/784: truncate d1/d3/d6/d7/d8c/fd7 824154 0 2024-03-23T11:47:43.112 INFO:tasks.workunit.client.1.smithi111.stdout:0/720: creat d5/db/d1f/d19/dbd/d64/df2/ff5 x:0 0 0 2024-03-23T11:47:43.114 INFO:tasks.workunit.client.1.smithi111.stdout:9/799: readlink d0/d31/d41/d9d/ld9 0 2024-03-23T11:47:43.115 INFO:tasks.workunit.client.1.smithi111.stdout:9/800: fsync d0/fbc 0 2024-03-23T11:47:43.116 INFO:tasks.workunit.client.1.smithi111.stdout:8/635: getdents d5/d26/d2f/d62/d7e/d85 0 2024-03-23T11:47:43.116 INFO:tasks.workunit.client.1.smithi111.stdout:0/721: dread - d5/db/d1f/d47/d59/fe6 zero size 2024-03-23T11:47:43.116 INFO:tasks.workunit.client.0.smithi063.stdout:7/533: rename d5/d26/d32/c70 to d5/d5c/d2f/d8c/cab 0 2024-03-23T11:47:43.116 INFO:tasks.workunit.client.0.smithi063.stdout:1/479: readlink d3/df/d2f/l45 0 2024-03-23T11:47:43.118 INFO:tasks.workunit.client.1.smithi111.stdout:9/801: fdatasync d0/d4/f9f 0 2024-03-23T11:47:43.118 INFO:tasks.workunit.client.1.smithi111.stdout:8/636: chown d5/d26/d2f/d8f 2558380 1 2024-03-23T11:47:43.121 INFO:tasks.workunit.client.1.smithi111.stdout:9/802: chown d0/d4/d6a/c53 5 1 2024-03-23T11:47:43.121 INFO:tasks.workunit.client.1.smithi111.stdout:0/722: truncate d5/db/d1f/d47/d6f/fc8 9359769 0 2024-03-23T11:47:43.122 INFO:tasks.workunit.client.0.smithi063.stdout:8/831: dread d6/d1f/d54/f6a [0,4194304] 0 2024-03-23T11:47:43.124 INFO:tasks.workunit.client.0.smithi063.stdout:1/480: dread - d3/f81 zero size 2024-03-23T11:47:43.125 INFO:tasks.workunit.client.1.smithi111.stdout:4/766: mkdir d0/dc/d4c/d2a/df2/dfc 0 2024-03-23T11:47:43.128 INFO:tasks.workunit.client.0.smithi063.stdout:8/832: readlink d6/d1f/l81 0 2024-03-23T11:47:43.129 INFO:tasks.workunit.client.0.smithi063.stdout:1/481: fsync d3/d1c/f6d 0 2024-03-23T11:47:43.130 INFO:tasks.workunit.client.0.smithi063.stdout:9/596: dwrite d4/d4c/d44/f7e [0,4194304] 0 2024-03-23T11:47:43.130 INFO:tasks.workunit.client.1.smithi111.stdout:2/781: rename d2/d11/d17/d3e/d6b/d9d/d52/c75 to d2/d7/c11a 0 2024-03-23T11:47:43.132 INFO:tasks.workunit.client.0.smithi063.stdout:8/833: chown d6/d1c/d27/d3f 168 1 2024-03-23T11:47:43.133 INFO:tasks.workunit.client.1.smithi111.stdout:2/782: chown d2/d11/d17/d3e/d6b/d9d/d52/f73 7824 1 2024-03-23T11:47:43.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:42 smithi063 ceph-mon[35189]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:43.135 INFO:tasks.workunit.client.1.smithi111.stdout:7/734: dread f11 [0,4194304] 0 2024-03-23T11:47:43.136 INFO:tasks.workunit.client.0.smithi063.stdout:0/688: dwrite d2/d12/d32/f91 [4194304,4194304] 0 2024-03-23T11:47:43.136 INFO:tasks.workunit.client.0.smithi063.stdout:1/482: fsync d3/df/f78 0 2024-03-23T11:47:43.137 INFO:tasks.workunit.client.0.smithi063.stdout:7/534: truncate d5/d6/d19/d1f/d47/d6e/f81 50731 0 2024-03-23T11:47:43.137 INFO:tasks.workunit.client.1.smithi111.stdout:7/735: readlink d14/d1b/d1e/d2c/d2e/d73/dae/la4 0 2024-03-23T11:47:43.139 INFO:tasks.workunit.client.0.smithi063.stdout:9/597: symlink d4/d31/d9c/d9e/ld0 0 2024-03-23T11:47:43.142 INFO:tasks.workunit.client.1.smithi111.stdout:9/803: mknod d0/d31/da0/c105 0 2024-03-23T11:47:43.144 INFO:tasks.workunit.client.0.smithi063.stdout:1/483: symlink d3/d1c/d4b/l96 0 2024-03-23T11:47:43.146 INFO:tasks.workunit.client.1.smithi111.stdout:1/730: link d0/d2b/d69/l86 d0/d5/dd/d1e/lf7 0 2024-03-23T11:47:43.149 INFO:tasks.workunit.client.1.smithi111.stdout:4/767: dread - d0/dc/d4c/d2a/fb9 zero size 2024-03-23T11:47:43.153 INFO:tasks.workunit.client.0.smithi063.stdout:1/484: creat d3/df/d2f/f97 x:0 0 0 2024-03-23T11:47:43.153 INFO:tasks.workunit.client.0.smithi063.stdout:8/834: dread d6/dd/f56 [0,4194304] 0 2024-03-23T11:47:43.158 INFO:tasks.workunit.client.1.smithi111.stdout:5/785: rename d1/d3/d6/d7/d86/caa to d1/c105 0 2024-03-23T11:47:43.158 INFO:tasks.workunit.client.1.smithi111.stdout:2/783: creat d2/d11/d17/d3e/d5d/dc2/f11b x:0 0 0 2024-03-23T11:47:43.159 INFO:tasks.workunit.client.0.smithi063.stdout:1/485: write d3/d1c/f6d [9408632,38768] 0 2024-03-23T11:47:43.160 INFO:tasks.workunit.client.1.smithi111.stdout:7/736: creat d14/d44/d70/d7b/fd5 x:0 0 0 2024-03-23T11:47:43.162 INFO:tasks.workunit.client.1.smithi111.stdout:7/737: readlink d14/d1b/d1e/d2c/d2e/d73/dae/la4 0 2024-03-23T11:47:43.164 INFO:tasks.workunit.client.1.smithi111.stdout:6/744: creat d2/ff2 x:0 0 0 2024-03-23T11:47:43.170 INFO:tasks.workunit.client.0.smithi063.stdout:8/835: rmdir d6/d1f/d54/d63/d90/d5d/d7d/d9a/d104 0 2024-03-23T11:47:43.176 INFO:tasks.workunit.client.0.smithi063.stdout:0/689: dwrite d2/d14/d36/d77/d94/da6/d1f/fba [0,4194304] 0 2024-03-23T11:47:43.177 INFO:tasks.workunit.client.0.smithi063.stdout:9/598: dwrite d4/d31/d43/d49/f9a [4194304,4194304] 0 2024-03-23T11:47:43.179 INFO:tasks.workunit.client.0.smithi063.stdout:1/486: rename d3/d3f/d4d/f69 to d3/d49/d56/d65/d84/f98 0 2024-03-23T11:47:43.184 INFO:tasks.workunit.client.1.smithi111.stdout:8/637: rename d5/d26/d31/d3b/d41 to d5/d26/d31/d3b/d55/dd0 0 2024-03-23T11:47:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.207+0000 7f21c7600700 1 -- 172.21.15.63:0/3948086847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fce80 msgr2=0x7f21c00fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.207+0000 7f21c7600700 1 --2- 172.21.15.63:0/3948086847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fce80 0x7f21c00fd290 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f21b4004940 tx=0x7f21b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.207+0000 7f21c7600700 1 -- 172.21.15.63:0/3948086847 shutdown_connections 2024-03-23T11:47:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.207+0000 7f21c7600700 1 --2- 172.21.15.63:0/3948086847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c00fdff0 0x7f21c00fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.207+0000 7f21c7600700 1 --2- 172.21.15.63:0/3948086847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fce80 0x7f21c00fd290 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.207+0000 7f21c7600700 1 -- 172.21.15.63:0/3948086847 >> 172.21.15.63:0/3948086847 conn(0x7f21c00f8540 msgr2=0x7f21c00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.208+0000 7f21c7600700 1 -- 172.21.15.63:0/3948086847 shutdown_connections 2024-03-23T11:47:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.208+0000 7f21c7600700 1 -- 172.21.15.63:0/3948086847 wait complete. 2024-03-23T11:47:43.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.209+0000 7f21c7600700 1 Processor -- start 2024-03-23T11:47:43.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.210+0000 7f21c7600700 1 -- start start 2024-03-23T11:47:43.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.210+0000 7f21c7600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c00fce80 0x7f21c006e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:43.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.210+0000 7f21c7600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fdff0 0x7f21c006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.210+0000 7f21c7600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00700a0 con 0x7f21c00fdff0 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.210+0000 7f21c7600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00701e0 con 0x7f21c00fce80 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.211+0000 7f21bfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fdff0 0x7f21c006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.211+0000 7f21bfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fdff0 0x7f21c006e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54294/0 (socket says 172.21.15.63:54294) 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.211+0000 7f21bfe00700 1 -- 172.21.15.63:0/2226006985 learned_addr learned my addr 172.21.15.63:0/2226006985 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.212+0000 7f21c4c00700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c00fce80 0x7f21c006e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.212+0000 7f21bfe00700 1 -- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c00fce80 msgr2=0x7f21c006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.212+0000 7f21bfe00700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c00fce80 0x7f21c006e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:43.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.212+0000 7f21bfe00700 1 -- 172.21.15.63:0/2226006985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4005040 con 0x7f21c00fdff0 2024-03-23T11:47:43.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.212+0000 7f21bfe00700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fdff0 0x7f21c006e9a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00b3b0 tx=0x7f21ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:43.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.213+0000 7f21bcc00700 1 -- 172.21.15.63:0/2226006985 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21ac019070 con 0x7f21c00fdff0 2024-03-23T11:47:43.199 INFO:tasks.workunit.client.0.smithi063.stdout:1/487: truncate d3/df/d31/f41 896042 0 2024-03-23T11:47:43.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.213+0000 7f21bcc00700 1 -- 172.21.15.63:0/2226006985 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21ac007c90 con 0x7f21c00fdff0 2024-03-23T11:47:43.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.214+0000 7f21bcc00700 1 -- 172.21.15.63:0/2226006985 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21ac01f7d0 con 0x7f21c00fdff0 2024-03-23T11:47:43.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.214+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c0070460 con 0x7f21c00fdff0 2024-03-23T11:47:43.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.214+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c0070a20 con 0x7f21c00fdff0 2024-03-23T11:47:43.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.215+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c005f5f0 con 0x7f21c00fdff0 2024-03-23T11:47:43.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.218+0000 7f21bcc00700 1 -- 172.21.15.63:0/2226006985 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f21ac0090d0 con 0x7f21c00fdff0 2024-03-23T11:47:43.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.219+0000 7f21bcc00700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f21a806adb0 0x7f21a806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:47:43.205 INFO:tasks.workunit.client.1.smithi111.stdout:7/738: mknod d14/d37/cd6 0 2024-03-23T11:47:43.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.220+0000 7f21bcc00700 1 -- 172.21.15.63:0/2226006985 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f21ac092070 con 0x7f21c00fdff0 2024-03-23T11:47:43.206 INFO:tasks.workunit.client.0.smithi063.stdout:0/690: dwrite d2/d14/d36/d77/d94/da6/f98 [0,4194304] 0 2024-03-23T11:47:43.206 INFO:tasks.workunit.client.1.smithi111.stdout:6/745: truncate d2/d35/d70/fc4 2992781 0 2024-03-23T11:47:43.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.221+0000 7f21c4c00700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f21a806adb0 0x7f21a806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:47:43.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.223+0000 7f21c4c00700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f21a806adb0 0x7f21a806d260 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f21b400b0d0 tx=0x7f21b4009e30 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:47:43.210 INFO:tasks.workunit.client.1.smithi111.stdout:8/638: mkdir d5/d26/d2f/d62/d75/d4b/d51/dd1 0 2024-03-23T11:47:43.210 INFO:tasks.workunit.client.1.smithi111.stdout:5/786: link d1/d3/fcd d1/d3/d6/d30/db2/f106 0 2024-03-23T11:47:43.213 INFO:tasks.workunit.client.1.smithi111.stdout:5/787: write d1/d3/d6/d8b/d59/d74/f73 [1102295,9625] 0 2024-03-23T11:47:43.215 INFO:tasks.workunit.client.0.smithi063.stdout:7/535: dwrite d5/d6/d19/d1f/d47/d4f/d5b/f66 [0,4194304] 0 2024-03-23T11:47:43.218 INFO:tasks.workunit.client.1.smithi111.stdout:6/746: dread d2/d1f/d45/d9a/fb4 [0,4194304] 0 2024-03-23T11:47:43.218 INFO:tasks.workunit.client.1.smithi111.stdout:8/639: mknod d5/d26/d2f/d62/d75/d4b/d51/cd2 0 2024-03-23T11:47:43.219 INFO:tasks.workunit.client.0.smithi063.stdout:7/536: stat d5/d6/f27 0 2024-03-23T11:47:43.221 INFO:tasks.workunit.client.1.smithi111.stdout:7/739: rename d14/d37/d54/fb7 to d14/d37/d43/db1/fd7 0 2024-03-23T11:47:43.222 INFO:tasks.workunit.client.0.smithi063.stdout:0/691: truncate d2/d14/d36/d77/d94/da6/d1f/fb5 303193 0 2024-03-23T11:47:43.223 INFO:tasks.workunit.client.1.smithi111.stdout:5/788: creat d1/d3/d6/d8b/d59/d6c/f107 x:0 0 0 2024-03-23T11:47:43.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.238+0000 7f21bcc00700 1 -- 172.21.15.63:0/2226006985 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f21ac05c700 con 0x7f21c00fdff0 2024-03-23T11:47:43.224 INFO:tasks.workunit.client.1.smithi111.stdout:7/740: fsync d14/d1b/d1e/f47 0 2024-03-23T11:47:43.226 INFO:tasks.workunit.client.0.smithi063.stdout:0/692: chown d2/d14/d36/d77/d94/fd5 734 1 2024-03-23T11:47:43.228 INFO:tasks.workunit.client.0.smithi063.stdout:7/537: rmdir d5/d5c/d2f 39 2024-03-23T11:47:43.229 INFO:tasks.workunit.client.0.smithi063.stdout:8/836: write d6/d1c/f52 [1757263,110988] 0 2024-03-23T11:47:43.231 INFO:tasks.workunit.client.1.smithi111.stdout:8/640: mkdir d5/d26/d2f/d42/dd3 0 2024-03-23T11:47:43.235 INFO:tasks.workunit.client.0.smithi063.stdout:7/538: fdatasync d5/d5c/d2f/d73/f3b 0 2024-03-23T11:47:43.237 INFO:tasks.workunit.client.1.smithi111.stdout:7/741: dread - d14/d1b/d1e/d86/f89 zero size 2024-03-23T11:47:43.237 INFO:tasks.workunit.client.1.smithi111.stdout:7/742: chown d14/d44 53 1 2024-03-23T11:47:43.238 INFO:tasks.workunit.client.1.smithi111.stdout:7/743: stat d14/d37/f9d 0 2024-03-23T11:47:43.239 INFO:tasks.workunit.client.0.smithi063.stdout:7/539: dread - d5/d5c/d2f/d73/da/d60/fa6 zero size 2024-03-23T11:47:43.240 INFO:tasks.workunit.client.0.smithi063.stdout:8/837: mkdir d6/dd/d118 0 2024-03-23T11:47:43.240 INFO:tasks.workunit.client.1.smithi111.stdout:7/744: write d14/d37/d43/f55 [7900278,48421] 0 2024-03-23T11:47:43.249 INFO:tasks.workunit.client.1.smithi111.stdout:8/641: unlink d5/d26/d31/f39 0 2024-03-23T11:47:43.250 INFO:tasks.workunit.client.0.smithi063.stdout:1/488: write d3/df/d31/f41 [1027984,39074] 0 2024-03-23T11:47:43.252 INFO:tasks.workunit.client.1.smithi111.stdout:0/723: write d5/db/ff3 [1461609,3324] 0 2024-03-23T11:47:43.255 INFO:tasks.workunit.client.1.smithi111.stdout:7/745: fdatasync d14/d1b/f5a 0 2024-03-23T11:47:43.255 INFO:tasks.workunit.client.1.smithi111.stdout:1/731: write d0/d2b/d69/f8c [703671,13785] 0 2024-03-23T11:47:43.255 INFO:tasks.workunit.client.1.smithi111.stdout:3/685: truncate d0/d7/f12 2208865 0 2024-03-23T11:47:43.256 INFO:tasks.workunit.client.0.smithi063.stdout:7/540: symlink d5/d5c/d2f/d8b/daa/lac 0 2024-03-23T11:47:43.258 INFO:tasks.workunit.client.1.smithi111.stdout:3/686: dread - d0/d7/d19/d91/fdb zero size 2024-03-23T11:47:43.262 INFO:tasks.workunit.client.0.smithi063.stdout:7/541: mkdir d5/d6/dad 0 2024-03-23T11:47:43.262 INFO:tasks.workunit.client.0.smithi063.stdout:8/838: getdents d6/d1f/d54/d63/d90/d5d 0 2024-03-23T11:47:43.262 INFO:tasks.workunit.client.1.smithi111.stdout:2/784: truncate d2/d11/d17/d1f/f20 574797 0 2024-03-23T11:47:43.264 INFO:tasks.workunit.client.1.smithi111.stdout:8/642: creat d5/d26/d31/d4d/fd4 x:0 0 0 2024-03-23T11:47:43.267 INFO:tasks.workunit.client.0.smithi063.stdout:0/693: dread d2/f56 [0,4194304] 0 2024-03-23T11:47:43.270 INFO:tasks.workunit.client.0.smithi063.stdout:8/839: mkdir d6/d1f/d54/d63/d119 0 2024-03-23T11:47:43.271 INFO:tasks.workunit.client.1.smithi111.stdout:6/747: link d2/d35/da7/fd6 d2/d1a/d1c/ddf/ff3 0 2024-03-23T11:47:43.273 INFO:tasks.workunit.client.1.smithi111.stdout:3/687: chown d0/d7/l4c 25 1 2024-03-23T11:47:43.273 INFO:tasks.workunit.client.1.smithi111.stdout:0/724: dread d5/db/d1f/d47/d6f/fc8 [4194304,4194304] 0 2024-03-23T11:47:43.275 INFO:tasks.workunit.client.0.smithi063.stdout:1/489: dwrite d3/df/d2f/f97 [0,4194304] 0 2024-03-23T11:47:43.277 INFO:tasks.workunit.client.1.smithi111.stdout:5/789: dread d1/d20/f61 [4194304,4194304] 0 2024-03-23T11:47:43.277 INFO:tasks.workunit.client.1.smithi111.stdout:1/732: rename d0/d5/f16 to d0/d2b/d37/d6a/ff8 0 2024-03-23T11:47:43.278 INFO:tasks.workunit.client.0.smithi063.stdout:8/840: stat d6/dd/c7a 0 2024-03-23T11:47:43.280 INFO:tasks.workunit.client.0.smithi063.stdout:0/694: creat d2/d14/d36/d77/d94/daf/ffb x:0 0 0 2024-03-23T11:47:43.281 INFO:tasks.workunit.client.1.smithi111.stdout:9/804: dwrite d0/d4/d10/d29/d7f/f8a [0,4194304] 0 2024-03-23T11:47:43.283 INFO:tasks.workunit.client.1.smithi111.stdout:3/688: symlink d0/d7/d19/d5e/le5 0 2024-03-23T11:47:43.284 INFO:tasks.workunit.client.1.smithi111.stdout:0/725: fdatasync d5/db/d1f/d19/dbd/f3f 0 2024-03-23T11:47:43.285 INFO:tasks.workunit.client.0.smithi063.stdout:7/542: link d5/d5c/d2f/d73/d20/f82 d5/d5c/d2f/d8c/fae 0 2024-03-23T11:47:43.287 INFO:tasks.workunit.client.0.smithi063.stdout:7/543: dread - d5/d5c/d2f/d73/da/d60/fa6 zero size 2024-03-23T11:47:43.288 INFO:tasks.workunit.client.0.smithi063.stdout:8/841: stat d6/dd/df/f11 0 2024-03-23T11:47:43.291 INFO:tasks.workunit.client.1.smithi111.stdout:0/726: write d5/db/d1f/d19/f2a [47974,96718] 0 2024-03-23T11:47:43.293 INFO:tasks.workunit.client.1.smithi111.stdout:5/790: mknod d1/d3/d6/d30/db2/c108 0 2024-03-23T11:47:43.294 INFO:tasks.workunit.client.0.smithi063.stdout:7/544: creat d5/d26/d32/faf x:0 0 0 2024-03-23T11:47:43.295 INFO:tasks.workunit.client.0.smithi063.stdout:8/842: dread - d6/dd/df/f50 zero size 2024-03-23T11:47:43.296 INFO:tasks.workunit.client.1.smithi111.stdout:7/746: dread d14/d1b/d1e/d2c/d42/d5e/fb0 [0,4194304] 0 2024-03-23T11:47:43.298 INFO:tasks.workunit.client.1.smithi111.stdout:7/747: read - d14/d44/d4b/f9a zero size 2024-03-23T11:47:43.300 INFO:tasks.workunit.client.0.smithi063.stdout:7/545: dread - d5/d5c/d2f/d73/da/d60/fa6 zero size 2024-03-23T11:47:43.300 INFO:tasks.workunit.client.1.smithi111.stdout:7/748: write d14/d1b/d1e/f47 [6100580,15046] 0 2024-03-23T11:47:43.303 INFO:tasks.workunit.client.0.smithi063.stdout:7/546: fsync d5/d6/d19/d1f/d47/f97 0 2024-03-23T11:47:43.304 INFO:tasks.workunit.client.1.smithi111.stdout:9/805: unlink d0/cc6 0 2024-03-23T11:47:43.305 INFO:tasks.workunit.client.1.smithi111.stdout:8/643: dread d5/d26/d2f/d62/db5/f7c [0,4194304] 0 2024-03-23T11:47:43.306 INFO:tasks.workunit.client.0.smithi063.stdout:7/547: dread - d5/d6/d19/d1f/d47/f97 zero size 2024-03-23T11:47:43.306 INFO:tasks.workunit.client.1.smithi111.stdout:3/689: rmdir d0/d1a/d2b/d38 39 2024-03-23T11:47:43.308 INFO:tasks.workunit.client.1.smithi111.stdout:2/785: link d2/de/d12/f43 d2/de/d6c/dd1/d111/f11c 0 2024-03-23T11:47:43.309 INFO:tasks.workunit.client.0.smithi063.stdout:7/548: creat d5/d5c/d2f/d8c/fb0 x:0 0 0 2024-03-23T11:47:43.313 INFO:tasks.workunit.client.0.smithi063.stdout:7/549: mknod d5/d5c/d2f/cb1 0 2024-03-23T11:47:43.315 INFO:tasks.workunit.client.1.smithi111.stdout:5/791: unlink d1/d3/d6/d30/db2/fc1 0 2024-03-23T11:47:43.316 INFO:tasks.workunit.client.0.smithi063.stdout:1/490: truncate d3/d1c/d4b/f71 1285687 0 2024-03-23T11:47:43.316 INFO:tasks.workunit.client.0.smithi063.stdout:7/550: truncate d5/d26/d32/faf 494909 0 2024-03-23T11:47:43.317 INFO:tasks.workunit.client.0.smithi063.stdout:8/843: write d6/d1f/d54/d5c/d66/f4c [1488985,4462] 0 2024-03-23T11:47:43.318 INFO:tasks.workunit.client.1.smithi111.stdout:9/806: mknod d0/d4/d10/d91/c106 0 2024-03-23T11:47:43.323 INFO:tasks.workunit.client.0.smithi063.stdout:1/491: creat d3/d49/d56/f99 x:0 0 0 2024-03-23T11:47:43.329 INFO:tasks.workunit.client.0.smithi063.stdout:1/492: mkdir d3/df/d2f/d9a 0 2024-03-23T11:47:43.329 INFO:tasks.workunit.client.0.smithi063.stdout:0/695: dwrite d2/f56 [0,4194304] 0 2024-03-23T11:47:43.332 INFO:tasks.workunit.client.1.smithi111.stdout:0/727: mkdir d5/db/d1f/d19/dbd/d64/d91/ddd/df6 0 2024-03-23T11:47:43.333 INFO:tasks.workunit.client.0.smithi063.stdout:8/844: rename d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/fce to d6/d1f/d54/d5c/dc5/f11a 0 2024-03-23T11:47:43.334 INFO:tasks.workunit.client.1.smithi111.stdout:0/728: chown d5/db/d1f/d47/l55 34465 1 2024-03-23T11:47:43.335 INFO:tasks.workunit.client.1.smithi111.stdout:2/786: dread d2/d7/da/dbd/fd8 [0,4194304] 0 2024-03-23T11:47:43.337 INFO:tasks.workunit.client.0.smithi063.stdout:7/551: dread d5/d6/d19/d1f/f6d [0,4194304] 0 2024-03-23T11:47:43.337 INFO:tasks.workunit.client.0.smithi063.stdout:8/845: dread - d6/d1f/d54/d63/d90/f100 zero size 2024-03-23T11:47:43.339 INFO:tasks.workunit.client.0.smithi063.stdout:8/846: chown d6/d1f/d54/d63 204072 1 2024-03-23T11:47:43.342 INFO:tasks.workunit.client.0.smithi063.stdout:0/696: write d2/d14/d36/d77/d94/da6/d1f/ffa [335484,103377] 0 2024-03-23T11:47:43.348 INFO:tasks.workunit.client.0.smithi063.stdout:7/552: unlink d5/d22/f44 0 2024-03-23T11:47:43.351 INFO:tasks.workunit.client.1.smithi111.stdout:1/733: link d0/d5/dd/d1e/d70/cf5 d0/d5/dd/d1e/d54/d87/d8e/cf9 0 2024-03-23T11:47:43.353 INFO:tasks.workunit.client.1.smithi111.stdout:1/734: write d0/d2b/f2e [2804692,18675] 0 2024-03-23T11:47:43.354 INFO:tasks.workunit.client.1.smithi111.stdout:3/690: dwrite d0/d1a/d1d/d26/d5a/d64/fba [0,4194304] 0 2024-03-23T11:47:43.357 INFO:tasks.workunit.client.1.smithi111.stdout:2/787: truncate d2/f6 2606588 0 2024-03-23T11:47:43.358 INFO:tasks.workunit.client.1.smithi111.stdout:7/749: creat d14/d1b/d1e/d2c/fd8 x:0 0 0 2024-03-23T11:47:43.358 INFO:tasks.workunit.client.1.smithi111.stdout:9/807: mknod d0/d4/d1c/c107 0 2024-03-23T11:47:43.360 INFO:tasks.workunit.client.1.smithi111.stdout:9/808: write d0/d4/d10/d29/d7f/d8e/f9e [749005,51625] 0 2024-03-23T11:47:43.362 INFO:tasks.workunit.client.0.smithi063.stdout:8/847: dwrite d6/d1f/d54/d5c/d66/df1/f107 [0,4194304] 0 2024-03-23T11:47:43.362 INFO:tasks.workunit.client.0.smithi063.stdout:1/493: rename d3/d3f/l6c to d3/l9b 0 2024-03-23T11:47:43.364 INFO:tasks.workunit.client.1.smithi111.stdout:4/768: sync 2024-03-23T11:47:43.368 INFO:tasks.workunit.client.1.smithi111.stdout:8/644: link d5/d32/c33 d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/cd5 0 2024-03-23T11:47:43.368 INFO:tasks.workunit.client.0.smithi063.stdout:7/553: creat d5/d22/d57/d64/fb2 x:0 0 0 2024-03-23T11:47:43.370 INFO:tasks.workunit.client.1.smithi111.stdout:1/735: creat d0/d5/dd/d1e/d63/ffa x:0 0 0 2024-03-23T11:47:43.372 INFO:tasks.workunit.client.0.smithi063.stdout:1/494: creat d3/d49/d56/d65/d88/f9c x:0 0 0 2024-03-23T11:47:43.373 INFO:tasks.workunit.client.0.smithi063.stdout:8/848: creat d6/d1f/d54/d5c/d66/df1/f11b x:0 0 0 2024-03-23T11:47:43.375 INFO:tasks.workunit.client.1.smithi111.stdout:3/691: creat d0/d7/d19/d84/fe6 x:0 0 0 2024-03-23T11:47:43.379 INFO:tasks.workunit.client.0.smithi063.stdout:8/849: write d6/dd/df/f61 [831704,56502] 0 2024-03-23T11:47:43.383 INFO:tasks.workunit.client.0.smithi063.stdout:0/697: dwrite d2/d4/f8a [0,4194304] 0 2024-03-23T11:47:43.385 INFO:tasks.workunit.client.1.smithi111.stdout:0/729: dwrite d5/db/d1f/d19/dbd/fa5 [0,4194304] 0 2024-03-23T11:47:43.387 INFO:tasks.workunit.client.0.smithi063.stdout:4/617: sync 2024-03-23T11:47:43.387 INFO:tasks.workunit.client.0.smithi063.stdout:5/396: sync 2024-03-23T11:47:43.387 INFO:tasks.workunit.client.1.smithi111.stdout:2/788: unlink d2/d7/l2b 0 2024-03-23T11:47:43.388 INFO:tasks.workunit.client.0.smithi063.stdout:5/397: chown d2/d28/f2d 7590 1 2024-03-23T11:47:43.394 INFO:tasks.workunit.client.1.smithi111.stdout:9/809: rename d0/d4/d10/fbb to d0/d31/d41/d9d/dd4/f108 0 2024-03-23T11:47:43.397 INFO:tasks.workunit.client.1.smithi111.stdout:9/810: chown d0/d4/d1c/d2d/d3b/d42/d92/lbe 0 1 2024-03-23T11:47:43.397 INFO:tasks.workunit.client.0.smithi063.stdout:8/850: rename d6/d1f/d3e/fcb to d6/d1f/d54/d5c/d66/df1/f11c 0 2024-03-23T11:47:43.398 INFO:tasks.workunit.client.1.smithi111.stdout:4/769: mknod d0/dc/d4c/d2a/d97/d4d/cfd 0 2024-03-23T11:47:43.399 INFO:tasks.workunit.client.0.smithi063.stdout:0/698: link d2/lef d2/d48/de2/lfc 0 2024-03-23T11:47:43.399 INFO:tasks.workunit.client.1.smithi111.stdout:9/811: readlink d0/d4/d10/d29/d7f/d8e/lc0 0 2024-03-23T11:47:43.407 INFO:tasks.workunit.client.0.smithi063.stdout:0/699: write d2/d14/d36/d77/d94/da6/d25/d93/ff3 [542887,23964] 0 2024-03-23T11:47:43.408 INFO:tasks.workunit.client.1.smithi111.stdout:5/792: getdents d1/d3/d6/d7/d7b 0 2024-03-23T11:47:43.408 INFO:tasks.workunit.client.0.smithi063.stdout:7/554: truncate f1 3383018 0 2024-03-23T11:47:43.410 INFO:tasks.workunit.client.0.smithi063.stdout:8/851: rename d6/d1f/d54/d5c/dc5/f11a to d6/d1f/d54/d5c/f11d 0 2024-03-23T11:47:43.415 INFO:tasks.workunit.client.1.smithi111.stdout:0/730: read d5/db/d1f/d47/d59/d5b/fb5 [604142,50171] 0 2024-03-23T11:47:43.418 INFO:tasks.workunit.client.0.smithi063.stdout:0/700: creat d2/d14/d36/d77/d94/daf/ffd x:0 0 0 2024-03-23T11:47:43.418 INFO:tasks.workunit.client.1.smithi111.stdout:2/789: creat d2/d11/d17/d3e/d5d/f11d x:0 0 0 2024-03-23T11:47:43.419 INFO:tasks.workunit.client.1.smithi111.stdout:6/748: sync 2024-03-23T11:47:43.420 INFO:tasks.workunit.client.0.smithi063.stdout:7/555: dread - d5/d6/d19/f7d zero size 2024-03-23T11:47:43.421 INFO:tasks.workunit.client.1.smithi111.stdout:7/750: dwrite d14/d44/d4b/f77 [0,4194304] 0 2024-03-23T11:47:43.427 INFO:tasks.workunit.client.0.smithi063.stdout:4/618: truncate d5/da/d20/d4b/d16/d50/f64 1686545 0 2024-03-23T11:47:43.428 INFO:tasks.workunit.client.1.smithi111.stdout:1/736: rename d0/d6/d48/d4a/d6c/d96/fe1 to d0/d5/dd/d1e/ffb 0 2024-03-23T11:47:43.428 INFO:tasks.workunit.client.1.smithi111.stdout:5/793: dwrite d1/d3/d6/d30/db2/ff4 [0,4194304] 0 2024-03-23T11:47:43.429 INFO:tasks.workunit.client.1.smithi111.stdout:1/737: chown d0/d5/dd5 2106580141 1 2024-03-23T11:47:43.429 INFO:tasks.workunit.client.1.smithi111.stdout:0/731: dread d5/db/d1f/d19/dbd/f3d [0,4194304] 0 2024-03-23T11:47:43.434 INFO:tasks.workunit.client.0.smithi063.stdout:1/495: dwrite d3/d1c/d4b/f71 [0,4194304] 0 2024-03-23T11:47:43.434 INFO:tasks.workunit.client.1.smithi111.stdout:8/645: fsync d5/d26/d31/d5e/f65 0 2024-03-23T11:47:43.435 INFO:tasks.workunit.client.0.smithi063.stdout:0/701: read d2/d14/de8/f9f [1175783,27878] 0 2024-03-23T11:47:43.436 INFO:tasks.workunit.client.1.smithi111.stdout:9/812: creat d0/d31/d35/f109 x:0 0 0 2024-03-23T11:47:43.437 INFO:tasks.workunit.client.0.smithi063.stdout:7/556: write d5/d5c/d2f/d73/d20/f31 [3362455,95834] 0 2024-03-23T11:47:43.442 INFO:tasks.workunit.client.1.smithi111.stdout:4/770: mknod d0/dc/d4c/d2a/d97/d4f/d72/cfe 0 2024-03-23T11:47:43.444 INFO:tasks.workunit.client.1.smithi111.stdout:4/771: stat d0/dc/d4c/dad/cf5 0 2024-03-23T11:47:43.445 INFO:tasks.workunit.client.1.smithi111.stdout:5/794: dread d1/d3/d6/d30/db2/ff4 [0,4194304] 0 2024-03-23T11:47:43.445 INFO:tasks.workunit.client.1.smithi111.stdout:4/772: chown d0/dc/l74 54784 1 2024-03-23T11:47:43.455 INFO:tasks.workunit.client.1.smithi111.stdout:8/646: dwrite d5/d26/d31/d3b/f6e [0,4194304] 0 2024-03-23T11:47:43.459 INFO:tasks.workunit.client.0.smithi063.stdout:1/496: fdatasync d3/f39 0 2024-03-23T11:47:43.462 INFO:tasks.workunit.client.1.smithi111.stdout:6/749: creat d2/d1a/d1c/d2c/ff4 x:0 0 0 2024-03-23T11:47:43.463 INFO:tasks.workunit.client.0.smithi063.stdout:5/398: dwrite d2/d7/f14 [0,4194304] 0 2024-03-23T11:47:43.463 INFO:tasks.workunit.client.1.smithi111.stdout:7/751: rmdir d14/d37/d43/db1 39 2024-03-23T11:47:43.469 INFO:tasks.workunit.client.0.smithi063.stdout:5/399: dread - d2/d17/d4d/d6e/f76 zero size 2024-03-23T11:47:43.471 INFO:tasks.workunit.client.1.smithi111.stdout:1/738: rename d0/d5/dd/d1e/d54/c88 to d0/cfc 0 2024-03-23T11:47:43.475 INFO:tasks.workunit.client.1.smithi111.stdout:2/790: write d2/d11/d17/fcf [607893,76831] 0 2024-03-23T11:47:43.479 INFO:tasks.workunit.client.1.smithi111.stdout:9/813: fsync d0/d4/d1c/d2d/d3b/d42/d4f/d95/fb9 0 2024-03-23T11:47:43.481 INFO:tasks.workunit.client.1.smithi111.stdout:3/692: creat d0/fe7 x:0 0 0 2024-03-23T11:47:43.481 INFO:tasks.workunit.client.1.smithi111.stdout:7/752: dwrite d14/d44/d70/d7b/fa8 [0,4194304] 0 2024-03-23T11:47:43.482 INFO:tasks.workunit.client.1.smithi111.stdout:4/773: symlink d0/dc/d78/lff 0 2024-03-23T11:47:43.483 INFO:tasks.workunit.client.0.smithi063.stdout:8/852: dread d6/f89 [0,4194304] 0 2024-03-23T11:47:43.484 INFO:tasks.workunit.client.1.smithi111.stdout:4/774: chown d0/dc/d1f/c58 38 1 2024-03-23T11:47:43.484 INFO:tasks.workunit.client.1.smithi111.stdout:4/775: stat d0/dc/d4c/d2a/f8b 0 2024-03-23T11:47:43.496 INFO:tasks.workunit.client.1.smithi111.stdout:6/750: truncate d2/d17/f33 839204 0 2024-03-23T11:47:43.510 INFO:tasks.workunit.client.1.smithi111.stdout:7/753: symlink d14/d1b/d1e/d87/ld9 0 2024-03-23T11:47:43.514 INFO:tasks.workunit.client.0.smithi063.stdout:0/702: symlink d2/d48/da5/da7/lfe 0 2024-03-23T11:47:43.515 INFO:tasks.workunit.client.0.smithi063.stdout:1/497: fsync d3/df/f12 0 2024-03-23T11:47:43.519 INFO:tasks.workunit.client.0.smithi063.stdout:5/400: mknod d2/d17/d4d/d6e/c8c 0 2024-03-23T11:47:43.522 INFO:tasks.workunit.client.0.smithi063.stdout:8/853: symlink d6/d1f/d54/d63/d90/d5d/d7d/d9a/l11e 0 2024-03-23T11:47:43.522 INFO:tasks.workunit.client.1.smithi111.stdout:2/791: symlink d2/d11/d17/d3e/d5d/dc0/df7/d10e/l11e 0 2024-03-23T11:47:43.523 INFO:tasks.workunit.client.0.smithi063.stdout:4/619: mknod d5/da/d20/d4b/d29/d38/d93/cb2 0 2024-03-23T11:47:43.523 INFO:tasks.workunit.client.0.smithi063.stdout:8/854: stat d6/d1f/ld4 0 2024-03-23T11:47:43.524 INFO:tasks.workunit.client.1.smithi111.stdout:2/792: write d2/f5b [4914491,27699] 0 2024-03-23T11:47:43.524 INFO:tasks.workunit.client.1.smithi111.stdout:1/739: truncate d0/d5/dd/d91/fa1 3352419 0 2024-03-23T11:47:43.525 INFO:tasks.workunit.client.0.smithi063.stdout:0/703: creat d2/d12/d32/d6d/fff x:0 0 0 2024-03-23T11:47:43.529 INFO:tasks.workunit.client.0.smithi063.stdout:1/498: mknod d3/df/d2f/c9d 0 2024-03-23T11:47:43.529 INFO:tasks.workunit.client.0.smithi063.stdout:0/704: write d2/d14/d36/d77/fbd [553813,36518] 0 2024-03-23T11:47:43.531 INFO:tasks.workunit.client.0.smithi063.stdout:7/557: creat d5/d6/d19/d1f/d47/fb3 x:0 0 0 2024-03-23T11:47:43.533 INFO:tasks.workunit.client.1.smithi111.stdout:7/754: unlink d14/d44/d4b/f9a 0 2024-03-23T11:47:43.534 INFO:tasks.workunit.client.1.smithi111.stdout:4/776: mknod d0/dc/d1f/c100 0 2024-03-23T11:47:43.537 INFO:tasks.workunit.client.0.smithi063.stdout:0/705: chown d2/d14/d36/d77/d94/da6/d25/d93/dc9 187479 1 2024-03-23T11:47:43.537 INFO:tasks.workunit.client.0.smithi063.stdout:8/855: rmdir d6/d1f/d54/d63 39 2024-03-23T11:47:43.540 INFO:tasks.workunit.client.1.smithi111.stdout:8/647: dread d5/d26/d2f/d62/d75/d4b/fb1 [0,4194304] 0 2024-03-23T11:47:43.541 INFO:tasks.workunit.client.0.smithi063.stdout:0/706: truncate d2/d14/f64 5109593 0 2024-03-23T11:47:43.541 INFO:tasks.workunit.client.1.smithi111.stdout:3/693: dwrite d0/d7/f76 [0,4194304] 0 2024-03-23T11:47:43.543 INFO:tasks.workunit.client.0.smithi063.stdout:7/558: symlink d5/d5c/d2f/d73/da/d60/lb4 0 2024-03-23T11:47:43.546 INFO:tasks.workunit.client.1.smithi111.stdout:9/814: truncate d0/d4/d10/f8b 3391983 0 2024-03-23T11:47:43.552 INFO:tasks.workunit.client.1.smithi111.stdout:4/777: unlink d0/dc/d4c/d71/cbb 0 2024-03-23T11:47:43.553 INFO:tasks.workunit.client.0.smithi063.stdout:1/499: mkdir d3/d49/d56/d65/d88/d92/d9e 0 2024-03-23T11:47:43.554 INFO:tasks.workunit.client.0.smithi063.stdout:1/500: fsync d3/f2a 0 2024-03-23T11:47:43.556 INFO:tasks.workunit.client.1.smithi111.stdout:3/694: mknod d0/d1a/d1d/d26/d5a/d60/ce8 0 2024-03-23T11:47:43.556 INFO:tasks.workunit.client.0.smithi063.stdout:1/501: chown d3/d1c/f68 2 1 2024-03-23T11:47:43.558 INFO:tasks.workunit.client.1.smithi111.stdout:2/793: dread d2/d11/d17/d3e/d6b/d2e/d76/fab [0,4194304] 0 2024-03-23T11:47:43.558 INFO:tasks.workunit.client.1.smithi111.stdout:7/755: dread d14/d37/f49 [0,4194304] 0 2024-03-23T11:47:43.558 INFO:tasks.workunit.client.1.smithi111.stdout:1/740: truncate d0/d2b/f5a 1709519 0 2024-03-23T11:47:43.559 INFO:tasks.workunit.client.0.smithi063.stdout:6/471: sync 2024-03-23T11:47:43.559 INFO:tasks.workunit.client.0.smithi063.stdout:9/599: sync 2024-03-23T11:47:43.559 INFO:tasks.workunit.client.0.smithi063.stdout:2/501: sync 2024-03-23T11:47:43.559 INFO:tasks.workunit.client.0.smithi063.stdout:3/871: sync 2024-03-23T11:47:43.563 INFO:tasks.workunit.client.0.smithi063.stdout:8/856: creat d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/f11f x:0 0 0 2024-03-23T11:47:43.565 INFO:tasks.workunit.client.1.smithi111.stdout:3/695: mknod d0/d55/ce9 0 2024-03-23T11:47:43.569 INFO:tasks.workunit.client.1.smithi111.stdout:3/696: dread - d0/d1a/d1d/d26/fb5 zero size 2024-03-23T11:47:43.569 INFO:tasks.workunit.client.1.smithi111.stdout:6/751: dwrite d2/f89 [0,4194304] 0 2024-03-23T11:47:43.571 INFO:tasks.workunit.client.1.smithi111.stdout:1/741: creat d0/d5/dd/d1e/d54/d81/ffd x:0 0 0 2024-03-23T11:47:43.573 INFO:tasks.workunit.client.1.smithi111.stdout:4/778: truncate d0/f1 4910217 0 2024-03-23T11:47:43.575 INFO:tasks.workunit.client.1.smithi111.stdout:9/815: dwrite d0/d4/d1c/d36/f11 [0,4194304] 0 2024-03-23T11:47:43.576 INFO:tasks.workunit.client.0.smithi063.stdout:8/857: rmdir d6/d1f 39 2024-03-23T11:47:43.577 INFO:tasks.workunit.client.1.smithi111.stdout:4/779: write d0/dc/d78/f85 [59625,27287] 0 2024-03-23T11:47:43.581 INFO:tasks.workunit.client.1.smithi111.stdout:4/780: write d0/dc/d78/fc0 [4592595,58652] 0 2024-03-23T11:47:43.583 INFO:tasks.workunit.client.0.smithi063.stdout:3/872: chown d13/d22/d2d/d3f/d58/d97/dac/cd8 1372 1 2024-03-23T11:47:43.584 INFO:tasks.workunit.client.0.smithi063.stdout:8/858: read - d6/f108 zero size 2024-03-23T11:47:43.584 INFO:tasks.workunit.client.1.smithi111.stdout:7/756: creat d14/d37/fda x:0 0 0 2024-03-23T11:47:43.585 INFO:tasks.workunit.client.1.smithi111.stdout:2/794: creat d2/d11/d17/d3e/d6b/d9d/d52/f11f x:0 0 0 2024-03-23T11:47:43.586 INFO:tasks.workunit.client.0.smithi063.stdout:2/502: read d2/de/d22/f78 [1669920,6189] 0 2024-03-23T11:47:43.590 INFO:tasks.workunit.client.0.smithi063.stdout:1/502: symlink d3/df/d31/l9f 0 2024-03-23T11:47:43.591 INFO:tasks.workunit.client.0.smithi063.stdout:6/472: dwrite d1/dc/d3a/f61 [0,4194304] 0 2024-03-23T11:47:43.591 INFO:tasks.workunit.client.1.smithi111.stdout:9/816: creat d0/d31/d41/d9d/dd4/f10a x:0 0 0 2024-03-23T11:47:43.595 INFO:tasks.workunit.client.1.smithi111.stdout:6/752: truncate d2/d35/da7/fd6 2723351 0 2024-03-23T11:47:43.595 INFO:tasks.workunit.client.1.smithi111.stdout:1/742: dwrite d0/d2b/d69/f8c [0,4194304] 0 2024-03-23T11:47:43.597 INFO:tasks.workunit.client.1.smithi111.stdout:6/753: chown d2/d1f/d45/d49/fb7 348430 1 2024-03-23T11:47:43.598 INFO:tasks.workunit.client.1.smithi111.stdout:1/743: dread - d0/d6/d48/d4a/de5/fea zero size 2024-03-23T11:47:43.599 INFO:tasks.workunit.client.0.smithi063.stdout:1/503: readlink d3/d1c/d4b/l80 0 2024-03-23T11:47:43.599 INFO:tasks.workunit.client.0.smithi063.stdout:6/473: readlink d1/dc/d3a/l8a 0 2024-03-23T11:47:43.599 INFO:tasks.workunit.client.0.smithi063.stdout:3/873: symlink d13/d22/d2d/d3f/d58/d97/l10f 0 2024-03-23T11:47:43.601 INFO:tasks.workunit.client.1.smithi111.stdout:4/781: write d0/dc/d4c/d2a/d97/fe5 [219589,316] 0 2024-03-23T11:47:43.601 INFO:tasks.workunit.client.0.smithi063.stdout:8/859: read d6/d1c/f3c [637176,93176] 0 2024-03-23T11:47:43.603 INFO:tasks.workunit.client.0.smithi063.stdout:5/401: write d2/f3e [804003,91145] 0 2024-03-23T11:47:43.603 INFO:tasks.workunit.client.1.smithi111.stdout:3/697: dwrite d0/d1a/d2b/d38/d7f/f9c [0,4194304] 0 2024-03-23T11:47:43.606 INFO:tasks.workunit.client.1.smithi111.stdout:1/744: dread d0/d5/fae [0,4194304] 0 2024-03-23T11:47:43.606 INFO:tasks.workunit.client.1.smithi111.stdout:7/757: dread - d14/d1b/d1e/d2c/d42/d5e/d9e/fa2 zero size 2024-03-23T11:47:43.607 INFO:tasks.workunit.client.0.smithi063.stdout:9/600: dread d4/f74 [0,4194304] 0 2024-03-23T11:47:43.608 INFO:tasks.workunit.client.1.smithi111.stdout:4/782: dread d0/d95/d9b/db3/dcc/dea/da/f5b [0,4194304] 0 2024-03-23T11:47:43.608 INFO:tasks.workunit.client.0.smithi063.stdout:6/474: truncate d1/d2a/f4a 4847608 0 2024-03-23T11:47:43.609 INFO:tasks.workunit.client.1.smithi111.stdout:1/745: dread - d0/d5/dd/d1e/d63/ffa zero size 2024-03-23T11:47:43.611 INFO:tasks.workunit.client.1.smithi111.stdout:7/758: fdatasync d14/d44/d4b/fbf 0 2024-03-23T11:47:43.611 INFO:tasks.workunit.client.1.smithi111.stdout:9/817: creat d0/d4/d6a/da7/f10b x:0 0 0 2024-03-23T11:47:43.614 INFO:tasks.workunit.client.1.smithi111.stdout:2/795: symlink d2/d11/d17/l120 0 2024-03-23T11:47:43.616 INFO:tasks.workunit.client.1.smithi111.stdout:3/698: mkdir d0/d7/d19/d84/dea 0 2024-03-23T11:47:43.616 INFO:tasks.workunit.client.1.smithi111.stdout:2/796: readlink d2/d11/d17/l46 0 2024-03-23T11:47:43.617 INFO:tasks.workunit.client.1.smithi111.stdout:4/783: unlink d0/dc/d1f/c21 0 2024-03-23T11:47:43.617 INFO:tasks.workunit.client.0.smithi063.stdout:2/503: dread d2/d23/d2e/f55 [0,4194304] 0 2024-03-23T11:47:43.618 INFO:tasks.workunit.client.0.smithi063.stdout:6/475: read d1/d2a/d41/d42/f6f [1326144,114055] 0 2024-03-23T11:47:43.619 INFO:tasks.workunit.client.1.smithi111.stdout:3/699: chown d0/d1a/d2d/fbb 43 1 2024-03-23T11:47:43.619 INFO:tasks.workunit.client.1.smithi111.stdout:9/818: read d0/d31/d41/d65/f6f [2668103,98547] 0 2024-03-23T11:47:43.621 INFO:tasks.workunit.client.0.smithi063.stdout:1/504: symlink d3/d35/la0 0 2024-03-23T11:47:43.621 INFO:tasks.workunit.client.0.smithi063.stdout:8/860: write d6/dd/df/f2c [71495,85551] 0 2024-03-23T11:47:43.623 INFO:tasks.workunit.client.0.smithi063.stdout:1/505: readlink l1 0 2024-03-23T11:47:43.625 INFO:tasks.workunit.client.1.smithi111.stdout:6/754: rmdir d2/d1f/d45 39 2024-03-23T11:47:43.627 INFO:tasks.workunit.client.0.smithi063.stdout:6/476: chown d1/d15/f70 28626 1 2024-03-23T11:47:43.627 INFO:tasks.workunit.client.1.smithi111.stdout:6/755: write d2/d1a/fe0 [306171,39190] 0 2024-03-23T11:47:43.629 INFO:tasks.workunit.client.1.smithi111.stdout:1/746: mkdir d0/d5/dd/dfe 0 2024-03-23T11:47:43.630 INFO:tasks.workunit.client.1.smithi111.stdout:2/797: symlink d2/d11/d17/d3e/d6b/d2e/d3f/l121 0 2024-03-23T11:47:43.630 INFO:tasks.workunit.client.0.smithi063.stdout:2/504: write d2/d1c/f96 [346474,72649] 0 2024-03-23T11:47:43.633 INFO:tasks.workunit.client.0.smithi063.stdout:1/506: rmdir d3/d3f 39 2024-03-23T11:47:43.634 INFO:tasks.workunit.client.0.smithi063.stdout:4/620: dwrite d5/da/d20/d4b/f5a [0,4194304] 0 2024-03-23T11:47:43.637 INFO:tasks.workunit.client.0.smithi063.stdout:7/559: dwrite d5/d5c/d2f/d73/da/f4e [0,4194304] 0 2024-03-23T11:47:43.638 INFO:tasks.workunit.client.0.smithi063.stdout:1/507: dread - d3/d1c/d4b/f8d zero size 2024-03-23T11:47:43.639 INFO:tasks.workunit.client.1.smithi111.stdout:3/700: dwrite d0/d1a/d2b/d38/d7f/f8a [4194304,4194304] 0 2024-03-23T11:47:43.642 INFO:tasks.workunit.client.0.smithi063.stdout:3/874: getdents d13/d22/d2d/d3f/d5b 0 2024-03-23T11:47:43.644 INFO:tasks.workunit.client.1.smithi111.stdout:3/701: chown d0/d1a/d1d/d26/d5a 155 1 2024-03-23T11:47:43.645 INFO:tasks.workunit.client.0.smithi063.stdout:8/861: fsync d6/d1f/d54/d63/d90/f28 0 2024-03-23T11:47:43.646 INFO:tasks.workunit.client.0.smithi063.stdout:8/862: chown d6/d1f/d54/d63 3 1 2024-03-23T11:47:43.647 INFO:tasks.workunit.client.0.smithi063.stdout:2/505: creat d2/d46/d54/fa3 x:0 0 0 2024-03-23T11:47:43.648 INFO:tasks.workunit.client.0.smithi063.stdout:8/863: write d6/dd/df/f61 [1717213,116025] 0 2024-03-23T11:47:43.650 INFO:tasks.workunit.client.1.smithi111.stdout:9/819: dread d0/d4/d10/d80/fb4 [0,4194304] 0 2024-03-23T11:47:43.651 INFO:tasks.workunit.client.0.smithi063.stdout:6/477: mknod d1/dc/d3a/c8e 0 2024-03-23T11:47:43.652 INFO:tasks.workunit.client.0.smithi063.stdout:1/508: mkdir d3/d1c/da1 0 2024-03-23T11:47:43.653 INFO:tasks.workunit.client.1.smithi111.stdout:9/820: write d0/d31/fed [724426,29438] 0 2024-03-23T11:47:43.655 INFO:tasks.workunit.client.0.smithi063.stdout:8/864: write d6/d1f/d54/d63/d90/d15/f24 [2729073,58139] 0 2024-03-23T11:47:43.655 INFO:tasks.workunit.client.0.smithi063.stdout:3/875: symlink d13/d22/d2d/d3f/d58/d6e/d8d/l110 0 2024-03-23T11:47:43.656 INFO:tasks.workunit.client.1.smithi111.stdout:6/756: rename d2/d1a/d1c/l1e to d2/d35/de3/lf5 0 2024-03-23T11:47:43.659 INFO:tasks.workunit.client.1.smithi111.stdout:9/821: truncate d0/d31/d41/d9d/dd4/ff7 1899341 0 2024-03-23T11:47:43.659 INFO:tasks.workunit.client.1.smithi111.stdout:7/759: link l5 d14/d1b/d1e/d2c/d2e/d33/d98/dd2/ldb 0 2024-03-23T11:47:43.661 INFO:tasks.workunit.client.1.smithi111.stdout:9/822: dread - d0/d4/d1c/d2d/fcb zero size 2024-03-23T11:47:43.662 INFO:tasks.workunit.client.1.smithi111.stdout:3/702: mkdir d0/d7/d19/d5e/da6/deb 0 2024-03-23T11:47:43.665 INFO:tasks.workunit.client.1.smithi111.stdout:6/757: creat d2/d1f/ff6 x:0 0 0 2024-03-23T11:47:43.667 INFO:tasks.workunit.client.1.smithi111.stdout:3/703: chown d0/d7/d19/d91/fdf 3 1 2024-03-23T11:47:43.668 INFO:tasks.workunit.client.0.smithi063.stdout:8/865: readlink d6/dd/l2f 0 2024-03-23T11:47:43.668 INFO:tasks.workunit.client.1.smithi111.stdout:7/760: mkdir d14/d1b/d1e/d2c/d42/d5e/ddc 0 2024-03-23T11:47:43.669 INFO:tasks.workunit.client.1.smithi111.stdout:0/732: sync 2024-03-23T11:47:43.670 INFO:tasks.workunit.client.1.smithi111.stdout:9/823: symlink d0/d31/d41/d71/d8f/def/l10c 0 2024-03-23T11:47:43.670 INFO:tasks.workunit.client.1.smithi111.stdout:5/795: sync 2024-03-23T11:47:43.674 INFO:tasks.workunit.client.1.smithi111.stdout:4/784: dwrite d0/d95/d9b/db3/dcc/dea/f7f [4194304,4194304] 0 2024-03-23T11:47:43.675 INFO:tasks.workunit.client.1.smithi111.stdout:3/704: dread - d0/d1a/d2b/fca zero size 2024-03-23T11:47:43.676 INFO:tasks.workunit.client.0.smithi063.stdout:9/601: dread d4/d4c/f41 [0,4194304] 0 2024-03-23T11:47:43.679 INFO:tasks.workunit.client.0.smithi063.stdout:1/509: dwrite d3/f5d [0,4194304] 0 2024-03-23T11:47:43.679 INFO:tasks.workunit.client.1.smithi111.stdout:1/747: dwrite d0/d6/fc5 [0,4194304] 0 2024-03-23T11:47:43.680 INFO:tasks.workunit.client.0.smithi063.stdout:1/510: readlink d3/d35/la0 0 2024-03-23T11:47:43.681 INFO:tasks.workunit.client.0.smithi063.stdout:9/602: chown d4/d31/d70/d8d/d90 580304714 1 2024-03-23T11:47:43.683 INFO:tasks.workunit.client.1.smithi111.stdout:2/798: write d2/d11/d17/d3e/d6b/fb3 [958537,76137] 0 2024-03-23T11:47:43.684 INFO:tasks.workunit.client.0.smithi063.stdout:1/511: write d3/d1c/d4b/f8d [458499,60759] 0 2024-03-23T11:47:43.686 INFO:tasks.workunit.client.0.smithi063.stdout:6/478: getdents d1/dc/d3a/d82 0 2024-03-23T11:47:43.687 INFO:tasks.workunit.client.1.smithi111.stdout:5/796: creat d1/d3/d6/d30/db2/f109 x:0 0 0 2024-03-23T11:47:43.689 INFO:tasks.workunit.client.1.smithi111.stdout:2/799: chown d2/d11/d17/d3e/d5d/dc2/f11b 5657990 1 2024-03-23T11:47:43.689 INFO:tasks.workunit.client.1.smithi111.stdout:4/785: write d0/dc/d1f/f8a [3062929,45583] 0 2024-03-23T11:47:43.690 INFO:tasks.workunit.client.1.smithi111.stdout:6/758: write d2/d17/f60 [3097112,55674] 0 2024-03-23T11:47:43.692 INFO:tasks.workunit.client.0.smithi063.stdout:9/603: chown d4/d31/d43/da9/da6/dbe 21680 1 2024-03-23T11:47:43.693 INFO:tasks.workunit.client.1.smithi111.stdout:7/761: mkdir d14/d1b/d1e/d2c/d8b/ddd 0 2024-03-23T11:47:43.696 INFO:tasks.workunit.client.1.smithi111.stdout:5/797: mkdir d1/d3/d6/d7/d7b/d10a 0 2024-03-23T11:47:43.697 INFO:tasks.workunit.client.0.smithi063.stdout:3/876: getdents d13/de5 0 2024-03-23T11:47:43.698 INFO:tasks.workunit.client.0.smithi063.stdout:6/479: creat d1/dc/d3a/f8f x:0 0 0 2024-03-23T11:47:43.702 INFO:tasks.workunit.client.1.smithi111.stdout:8/648: sync 2024-03-23T11:47:43.702 INFO:tasks.workunit.client.0.smithi063.stdout:9/604: symlink d4/d31/d43/d49/ld1 0 2024-03-23T11:47:43.702 INFO:tasks.workunit.client.1.smithi111.stdout:9/824: dwrite d0/d4/d10/f2c [0,4194304] 0 2024-03-23T11:47:43.704 INFO:tasks.workunit.client.0.smithi063.stdout:8/866: dread d6/dd/fde [0,4194304] 0 2024-03-23T11:47:43.704 INFO:tasks.workunit.client.1.smithi111.stdout:9/825: dread - d0/d4/d10/ff2 zero size 2024-03-23T11:47:43.705 INFO:tasks.workunit.client.1.smithi111.stdout:9/826: stat d0/d31/ld1 0 2024-03-23T11:47:43.706 INFO:tasks.workunit.client.1.smithi111.stdout:9/827: chown d0/d31/d35/l57 29704 1 2024-03-23T11:47:43.707 INFO:tasks.workunit.client.1.smithi111.stdout:9/828: readlink d0/d4/d10/d29/d7f/l82 0 2024-03-23T11:47:43.708 INFO:tasks.workunit.client.1.smithi111.stdout:3/705: rename d0/d1a/d1d/cdc to d0/d1a/cec 0 2024-03-23T11:47:43.709 INFO:tasks.workunit.client.0.smithi063.stdout:9/605: dread - d4/d31/d43/f86 zero size 2024-03-23T11:47:43.709 INFO:tasks.workunit.client.1.smithi111.stdout:3/706: write d0/d1a/d1d/fb8 [272848,84190] 0 2024-03-23T11:47:43.711 INFO:tasks.workunit.client.1.smithi111.stdout:1/748: symlink d0/d6/d48/d4d/dab/ddb/lff 0 2024-03-23T11:47:43.711 INFO:tasks.workunit.client.1.smithi111.stdout:2/800: write d2/d11/d17/d3e/d6b/d2e/d3f/faa [451727,107341] 0 2024-03-23T11:47:43.712 INFO:tasks.workunit.client.1.smithi111.stdout:4/786: write d0/d95/d9b/db3/dcc/dea/f69 [2786540,98538] 0 2024-03-23T11:47:43.712 INFO:tasks.workunit.client.0.smithi063.stdout:3/877: creat d13/d22/f111 x:0 0 0 2024-03-23T11:47:43.714 INFO:tasks.workunit.client.1.smithi111.stdout:0/733: getdents d5/db/d1f/de4/da7 0 2024-03-23T11:47:43.714 INFO:tasks.workunit.client.1.smithi111.stdout:2/801: readlink d2/d11/d17/d3e/d5d/lf8 0 2024-03-23T11:47:43.716 INFO:tasks.workunit.client.0.smithi063.stdout:5/402: write d2/d7/d8/de/d2a/d3b/d43/f45 [15231,6351] 0 2024-03-23T11:47:43.718 INFO:tasks.workunit.client.1.smithi111.stdout:7/762: mknod d14/d1b/d1e/d2c/d42/d5e/ddc/cde 0 2024-03-23T11:47:43.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.733+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21c006eff0 con 0x7f21a806adb0 2024-03-23T11:47:43.719 INFO:tasks.workunit.client.0.smithi063.stdout:3/878: mkdir d13/d66/d112 0 2024-03-23T11:47:43.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.737+0000 7f21bcc00700 1 -- 172.21.15.63:0/2226006985 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f21c006eff0 con 0x7f21a806adb0 2024-03-23T11:47:43.725 INFO:tasks.workunit.client.1.smithi111.stdout:5/798: mkdir d1/d3/d6/d30/db2/dcc/d10b 0 2024-03-23T11:47:43.725 INFO:tasks.workunit.client.1.smithi111.stdout:3/707: mknod d0/d7/d19/d91/ced 0 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [], 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "1/2 daemons upgraded", 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "", 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:47:43.726 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:47:43.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f21a806adb0 msgr2=0x7f21a806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:43.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f21a806adb0 0x7f21a806d260 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f21b400b0d0 tx=0x7f21b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:47:43.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fdff0 msgr2=0x7f21c006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:47:43.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fdff0 0x7f21c006e9a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00b3b0 tx=0x7f21ac00b040 comp rx=0 tx=0).stop 2024-03-23T11:47:43.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 shutdown_connections 2024-03-23T11:47:43.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f21a806adb0 0x7f21a806d260 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:43.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c00fce80 0x7f21c006e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:43.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 --2- 172.21.15.63:0/2226006985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21c00fdff0 0x7f21c006e9a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:47:43.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 >> 172.21.15.63:0/2226006985 conn(0x7f21c00f8540 msgr2=0x7f21c00f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:47:43.728 INFO:tasks.workunit.client.1.smithi111.stdout:4/787: creat d0/d95/db5/f101 x:0 0 0 2024-03-23T11:47:43.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.742+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 shutdown_connections 2024-03-23T11:47:43.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:47:43.743+0000 7f21c7600700 1 -- 172.21.15.63:0/2226006985 wait complete. 2024-03-23T11:47:43.729 INFO:tasks.workunit.client.1.smithi111.stdout:5/799: write d1/d20/d5c/dc5/fd1 [146754,5184] 0 2024-03-23T11:47:43.729 INFO:tasks.workunit.client.1.smithi111.stdout:0/734: fsync d5/db/d1f/f79 0 2024-03-23T11:47:43.731 INFO:tasks.workunit.client.1.smithi111.stdout:2/802: creat d2/de/d12/d70/d96/f122 x:0 0 0 2024-03-23T11:47:43.732 INFO:tasks.workunit.client.0.smithi063.stdout:6/480: dwrite d1/dc/d21/d49/f76 [0,4194304] 0 2024-03-23T11:47:43.732 INFO:tasks.workunit.client.1.smithi111.stdout:7/763: mknod d14/d1b/d1e/d2c/d2e/d95/cdf 0 2024-03-23T11:47:43.734 INFO:tasks.workunit.client.1.smithi111.stdout:3/708: mknod d0/d1a/d2b/d38/d7f/cee 0 2024-03-23T11:47:43.735 INFO:tasks.workunit.client.1.smithi111.stdout:6/759: getdents d2/d1f/d45 0 2024-03-23T11:47:43.736 INFO:tasks.workunit.client.1.smithi111.stdout:2/803: dread - d2/de/d12/d70/ff4 zero size 2024-03-23T11:47:43.737 INFO:tasks.workunit.client.1.smithi111.stdout:1/749: dread d0/d2b/d37/f74 [0,4194304] 0 2024-03-23T11:47:43.738 INFO:tasks.workunit.client.0.smithi063.stdout:8/867: getdents d6/d1c/d27 0 2024-03-23T11:47:43.739 INFO:tasks.workunit.client.1.smithi111.stdout:4/788: truncate d0/fc6 1129916 0 2024-03-23T11:47:43.739 INFO:tasks.workunit.client.1.smithi111.stdout:3/709: stat d0/d1a/d4f/d90/fb2 0 2024-03-23T11:47:43.743 INFO:tasks.workunit.client.1.smithi111.stdout:2/804: chown d2/d11/c8b 61 1 2024-03-23T11:47:43.745 INFO:tasks.workunit.client.1.smithi111.stdout:1/750: stat d0/d6/d48/d4a/d6c/l97 0 2024-03-23T11:47:43.747 INFO:tasks.workunit.client.1.smithi111.stdout:2/805: chown d2/d11/d17/d3e/d5d/fa4 1938 1 2024-03-23T11:47:43.747 INFO:tasks.workunit.client.1.smithi111.stdout:9/829: creat d0/d4/d1c/d36/d28/f10d x:0 0 0 2024-03-23T11:47:43.749 INFO:tasks.workunit.client.0.smithi063.stdout:6/481: symlink d1/d2a/d41/d42/l90 0 2024-03-23T11:47:43.749 INFO:tasks.workunit.client.0.smithi063.stdout:5/403: dwrite d2/d7/d8/d1d/f88 [0,4194304] 0 2024-03-23T11:47:43.749 INFO:tasks.workunit.client.1.smithi111.stdout:6/760: truncate d2/d1f/d45/d49/f9e 859759 0 2024-03-23T11:47:43.752 INFO:tasks.workunit.client.0.smithi063.stdout:6/482: write d1/f81 [363533,7155] 0 2024-03-23T11:47:43.753 INFO:tasks.workunit.client.1.smithi111.stdout:1/751: read d0/d6/fc [1165171,114978] 0 2024-03-23T11:47:43.756 INFO:tasks.workunit.client.0.smithi063.stdout:3/879: truncate d13/d22/d2d/d3f/d67/d7b/fab 4403690 0 2024-03-23T11:47:43.756 INFO:tasks.workunit.client.1.smithi111.stdout:4/789: fsync d0/dc/d89/da4/fcf 0 2024-03-23T11:47:43.757 INFO:tasks.workunit.client.0.smithi063.stdout:6/483: chown d1/dc/d3a/d82 62919658 1 2024-03-23T11:47:43.758 INFO:tasks.workunit.client.1.smithi111.stdout:6/761: write d2/d1a/fe0 [494604,109817] 0 2024-03-23T11:47:43.759 INFO:tasks.workunit.client.1.smithi111.stdout:1/752: chown d0/d6/ca 27806395 1 2024-03-23T11:47:43.763 INFO:tasks.workunit.client.1.smithi111.stdout:8/649: dwrite d5/d26/d31/d3b/f60 [0,4194304] 0 2024-03-23T11:47:43.764 INFO:tasks.workunit.client.1.smithi111.stdout:6/762: stat d2/d35/d70/d98 0 2024-03-23T11:47:43.767 INFO:tasks.workunit.client.1.smithi111.stdout:4/790: creat d0/dc/d1f/db0/f102 x:0 0 0 2024-03-23T11:47:43.767 INFO:tasks.workunit.client.1.smithi111.stdout:5/800: dwrite d1/d3/d6/d7/d8c/fd7 [0,4194304] 0 2024-03-23T11:47:43.768 INFO:tasks.workunit.client.0.smithi063.stdout:5/404: rename d2/f50 to d2/d28/d3a/f8d 0 2024-03-23T11:47:43.778 INFO:tasks.workunit.client.1.smithi111.stdout:8/650: symlink d5/d26/d2f/d42/d63/ld6 0 2024-03-23T11:47:43.779 INFO:tasks.workunit.client.0.smithi063.stdout:3/880: creat d13/d22/d2d/d3f/d5b/f113 x:0 0 0 2024-03-23T11:47:43.779 INFO:tasks.workunit.client.1.smithi111.stdout:5/801: fdatasync d1/d3/d6/d7/d63/d2c/f51 0 2024-03-23T11:47:43.779 INFO:tasks.workunit.client.1.smithi111.stdout:6/763: creat d2/d73/ff7 x:0 0 0 2024-03-23T11:47:43.781 INFO:tasks.workunit.client.1.smithi111.stdout:4/791: creat d0/dc/d89/da4/f103 x:0 0 0 2024-03-23T11:47:43.781 INFO:tasks.workunit.client.0.smithi063.stdout:6/484: unlink d1/d15/d4b/c86 0 2024-03-23T11:47:43.785 INFO:tasks.workunit.client.1.smithi111.stdout:3/710: truncate d0/d55/f81 432769 0 2024-03-23T11:47:43.786 INFO:tasks.workunit.client.0.smithi063.stdout:8/868: link d6/d1c/d27/f99 d6/d1f/d3e/f120 0 2024-03-23T11:47:43.789 INFO:tasks.workunit.client.1.smithi111.stdout:4/792: creat d0/d95/db5/dc7/f104 x:0 0 0 2024-03-23T11:47:43.790 INFO:tasks.workunit.client.0.smithi063.stdout:9/606: getdents d4/d31/d43/d49 0 2024-03-23T11:47:43.793 INFO:tasks.workunit.client.1.smithi111.stdout:6/764: fsync d2/f28 0 2024-03-23T11:47:43.795 INFO:tasks.workunit.client.0.smithi063.stdout:1/512: dwrite d3/f7 [8388608,4194304] 0 2024-03-23T11:47:43.795 INFO:tasks.workunit.client.0.smithi063.stdout:6/485: dread - d1/dc/f5b zero size 2024-03-23T11:47:43.800 INFO:tasks.workunit.client.1.smithi111.stdout:8/651: chown d5/d26/d2f/d62/d75/d8d/la1 7047528 1 2024-03-23T11:47:43.804 INFO:tasks.workunit.client.1.smithi111.stdout:4/793: mkdir d0/dc/d4c/d2a/d97/d4d/de3/d105 0 2024-03-23T11:47:43.807 INFO:tasks.workunit.client.1.smithi111.stdout:8/652: dread - d5/d26/d31/d4d/fc4 zero size 2024-03-23T11:47:43.808 INFO:tasks.workunit.client.1.smithi111.stdout:2/806: dwrite d2/d7/da/d19/fcb [0,4194304] 0 2024-03-23T11:47:43.809 INFO:tasks.workunit.client.0.smithi063.stdout:3/881: dwrite d13/de5/ffd [0,4194304] 0 2024-03-23T11:47:43.812 INFO:tasks.workunit.client.0.smithi063.stdout:4/621: dread d5/ff [8388608,4194304] 0 2024-03-23T11:47:43.812 INFO:tasks.workunit.client.1.smithi111.stdout:5/802: truncate d1/d3/d6/d8b/d59/d74/f4b 1882096 0 2024-03-23T11:47:43.813 INFO:tasks.workunit.client.0.smithi063.stdout:8/869: dread d6/d1c/d32/d37/d10a/dae/fd1 [0,4194304] 0 2024-03-23T11:47:43.815 INFO:tasks.workunit.client.0.smithi063.stdout:3/882: chown d13/d22/d2d/d3f/d58/d6e/d8d/dda/dee 100 1 2024-03-23T11:47:43.820 INFO:tasks.workunit.client.1.smithi111.stdout:4/794: creat d0/dc/d4c/d2a/d97/d4f/dba/f106 x:0 0 0 2024-03-23T11:47:43.825 INFO:tasks.workunit.client.1.smithi111.stdout:4/795: fdatasync d0/d95/d9b/db3/db6/fa3 0 2024-03-23T11:47:43.825 INFO:tasks.workunit.client.1.smithi111.stdout:5/803: mknod d1/d3/d6/d7/d63/dd9/c10c 0 2024-03-23T11:47:43.830 INFO:tasks.workunit.client.1.smithi111.stdout:0/735: dread d5/db/d1f/f1a [0,4194304] 0 2024-03-23T11:47:43.831 INFO:tasks.workunit.client.1.smithi111.stdout:7/764: sync 2024-03-23T11:47:43.832 INFO:tasks.workunit.client.1.smithi111.stdout:4/796: creat d0/d95/d9b/f107 x:0 0 0 2024-03-23T11:47:43.838 INFO:tasks.workunit.client.1.smithi111.stdout:1/753: dread d0/d5/f23 [0,4194304] 0 2024-03-23T11:47:43.840 INFO:tasks.workunit.client.1.smithi111.stdout:5/804: mkdir d1/d3/d6/d7/d7b/d10d 0 2024-03-23T11:47:43.844 INFO:tasks.workunit.client.1.smithi111.stdout:8/653: getdents d5/d26/d31/d3b/d55/dd0 0 2024-03-23T11:47:43.846 INFO:tasks.workunit.client.1.smithi111.stdout:5/805: creat d1/f10e x:0 0 0 2024-03-23T11:47:43.850 INFO:tasks.workunit.client.1.smithi111.stdout:0/736: mknod d5/db/d1f/d47/cf7 0 2024-03-23T11:47:43.851 INFO:tasks.workunit.client.1.smithi111.stdout:3/711: write d0/d1a/d1d/d26/d5a/f8e [4511661,35950] 0 2024-03-23T11:47:43.854 INFO:tasks.workunit.client.1.smithi111.stdout:7/765: read d14/d1b/d1e/d2c/d2e/d33/f3d [3225628,123705] 0 2024-03-23T11:47:43.854 INFO:tasks.workunit.client.1.smithi111.stdout:3/712: fsync d0/d53/fcf 0 2024-03-23T11:47:43.859 INFO:tasks.workunit.client.1.smithi111.stdout:9/830: dread d0/d4/d1c/d36/d28/f68 [0,4194304] 0 2024-03-23T11:47:43.860 INFO:tasks.workunit.client.1.smithi111.stdout:0/737: read d5/f30 [3034887,56176] 0 2024-03-23T11:47:43.862 INFO:tasks.workunit.client.1.smithi111.stdout:9/831: fdatasync d0/d4/d6a/d9a/faa 0 2024-03-23T11:47:43.869 INFO:tasks.workunit.client.1.smithi111.stdout:2/807: dread d2/de/d6c/f65 [0,4194304] 0 2024-03-23T11:47:43.871 INFO:tasks.workunit.client.1.smithi111.stdout:2/808: chown d2/de/d6c/l45 386279 1 2024-03-23T11:47:43.877 INFO:tasks.workunit.client.1.smithi111.stdout:1/754: dread d0/d5/dd/d91/fa1 [0,4194304] 0 2024-03-23T11:47:43.878 INFO:tasks.workunit.client.0.smithi063.stdout:5/405: getdents d2/d17/d4d 0 2024-03-23T11:47:43.879 INFO:tasks.workunit.client.0.smithi063.stdout:6/486: truncate d1/dc/f1f 335643 0 2024-03-23T11:47:43.883 INFO:tasks.workunit.client.0.smithi063.stdout:6/487: write d1/d15/f88 [1317342,56892] 0 2024-03-23T11:47:43.885 INFO:tasks.workunit.client.0.smithi063.stdout:5/406: dread - d2/d7/d8/de/d2a/d3b/d43/d5c/f7d zero size 2024-03-23T11:47:43.888 INFO:tasks.workunit.client.0.smithi063.stdout:5/407: stat d2/d7/d8/de/d49 0 2024-03-23T11:47:43.890 INFO:tasks.workunit.client.0.smithi063.stdout:4/622: read d5/da/d20/d4b/d1f/f21 [2192296,95136] 0 2024-03-23T11:47:43.892 INFO:tasks.workunit.client.0.smithi063.stdout:5/408: read d2/d7/d8/de/d2a/d3b/d43/d5c/f62 [4165880,112682] 0 2024-03-23T11:47:43.905 INFO:tasks.workunit.client.0.smithi063.stdout:9/607: creat d4/d31/d43/da9/da6/dbe/fd2 x:0 0 0 2024-03-23T11:47:43.918 INFO:tasks.workunit.client.0.smithi063.stdout:3/883: mkdir d13/d15/d114 0 2024-03-23T11:47:43.926 INFO:tasks.workunit.client.0.smithi063.stdout:8/870: mknod d6/d1c/d32/d37/d10a/db5/dfe/c121 0 2024-03-23T11:47:43.938 INFO:tasks.workunit.client.0.smithi063.stdout:3/884: creat d13/de5/f115 x:0 0 0 2024-03-23T11:47:43.948 INFO:tasks.workunit.client.0.smithi063.stdout:8/871: rmdir d6/d1c/d32/d37/d10a/de4 0 2024-03-23T11:47:43.949 INFO:tasks.workunit.client.0.smithi063.stdout:3/885: dread d13/d22/d2d/d3f/d5b/f10a [0,4194304] 0 2024-03-23T11:47:43.959 INFO:tasks.workunit.client.0.smithi063.stdout:3/886: mkdir d13/d22/da9/d116 0 2024-03-23T11:47:43.961 INFO:tasks.workunit.client.0.smithi063.stdout:8/872: symlink d6/d1f/d54/d63/d90/d5d/d7d/l122 0 2024-03-23T11:47:43.964 INFO:tasks.workunit.client.0.smithi063.stdout:8/873: chown d6/d1f/d54/d63/d90/f5f 3245759 1 2024-03-23T11:47:43.968 INFO:tasks.workunit.client.0.smithi063.stdout:1/513: symlink d3/d49/d56/la2 0 2024-03-23T11:47:43.968 INFO:tasks.workunit.client.0.smithi063.stdout:8/874: chown d6/d1f/d54/d63/d90/d9e/fa8 983337936 1 2024-03-23T11:47:43.971 INFO:tasks.workunit.client.0.smithi063.stdout:8/875: chown d6/d1c/d27/d3f/lcd 13 1 2024-03-23T11:47:43.975 INFO:tasks.workunit.client.0.smithi063.stdout:8/876: write d6/d1f/d54/d63/d90/f13 [818659,66213] 0 2024-03-23T11:47:43.983 INFO:tasks.workunit.client.0.smithi063.stdout:8/877: truncate d6/d1f/f55 3734395 0 2024-03-23T11:47:43.984 INFO:tasks.workunit.client.0.smithi063.stdout:8/878: fdatasync d6/f16 0 2024-03-23T11:47:43.990 INFO:tasks.workunit.client.0.smithi063.stdout:9/608: mknod d4/d31/d43/da9/cd3 0 2024-03-23T11:47:43.991 INFO:tasks.workunit.client.0.smithi063.stdout:8/879: mknod d6/d1f/d54/d5c/dc5/c123 0 2024-03-23T11:47:43.996 INFO:tasks.workunit.client.0.smithi063.stdout:3/887: dread d13/d22/f2c [4194304,4194304] 0 2024-03-23T11:47:43.999 INFO:tasks.workunit.client.0.smithi063.stdout:9/609: creat d4/d31/d9c/d9e/fd4 x:0 0 0 2024-03-23T11:47:44.003 INFO:tasks.workunit.client.0.smithi063.stdout:3/888: write d13/de5/f115 [648465,67501] 0 2024-03-23T11:47:44.004 INFO:tasks.workunit.client.0.smithi063.stdout:6/488: creat d1/dc/f91 x:0 0 0 2024-03-23T11:47:44.006 INFO:tasks.workunit.client.0.smithi063.stdout:5/409: dwrite d2/d7/d8/de/d35/f60 [0,4194304] 0 2024-03-23T11:47:44.010 INFO:tasks.workunit.client.0.smithi063.stdout:9/610: fsync d4/d4c/f25 0 2024-03-23T11:47:44.012 INFO:tasks.workunit.client.0.smithi063.stdout:1/514: dread d3/df/f1a [4194304,4194304] 0 2024-03-23T11:47:44.013 INFO:tasks.workunit.client.0.smithi063.stdout:6/489: dread - d1/d15/d4b/f6b zero size 2024-03-23T11:47:44.019 INFO:tasks.workunit.client.0.smithi063.stdout:1/515: chown d3/df/d2f/c61 522970 1 2024-03-23T11:47:44.022 INFO:tasks.workunit.client.0.smithi063.stdout:8/880: dwrite d6/dd/df/f2c [0,4194304] 0 2024-03-23T11:47:44.023 INFO:tasks.workunit.client.0.smithi063.stdout:5/410: creat d2/d7/d8/de/d49/f8e x:0 0 0 2024-03-23T11:47:44.024 INFO:tasks.workunit.client.0.smithi063.stdout:9/611: mknod d4/d31/d70/d8d/da0/cd5 0 2024-03-23T11:47:44.030 INFO:tasks.workunit.client.0.smithi063.stdout:4/623: link d5/da/d20/d4b/d1f/f26 d5/da/d20/d4b/d16/fb3 0 2024-03-23T11:47:44.030 INFO:tasks.workunit.client.0.smithi063.stdout:9/612: read d4/d31/d43/da9/da6/f87 [1131232,2344] 0 2024-03-23T11:47:44.035 INFO:tasks.workunit.client.0.smithi063.stdout:9/613: stat d4/d31/d70/d8d/fac 0 2024-03-23T11:47:44.039 INFO:tasks.workunit.client.0.smithi063.stdout:3/889: mkdir d13/d22/d2d/d3e/dce/dfb/d117 0 2024-03-23T11:47:44.042 INFO:tasks.workunit.client.0.smithi063.stdout:5/411: mkdir d2/d7/d8/de/d8f 0 2024-03-23T11:47:44.044 INFO:tasks.workunit.client.0.smithi063.stdout:5/412: chown d2/f33 289042022 1 2024-03-23T11:47:44.046 INFO:tasks.workunit.client.0.smithi063.stdout:8/881: rename d6/d1c/d27/f31 to d6/dd/df/f124 0 2024-03-23T11:47:44.048 INFO:tasks.workunit.client.0.smithi063.stdout:6/490: fdatasync d1/d15/d4b/f4c 0 2024-03-23T11:47:44.054 INFO:tasks.workunit.client.0.smithi063.stdout:0/707: sync 2024-03-23T11:47:44.054 INFO:tasks.workunit.client.0.smithi063.stdout:2/506: sync 2024-03-23T11:47:44.055 INFO:tasks.workunit.client.0.smithi063.stdout:7/560: sync 2024-03-23T11:47:44.055 INFO:tasks.workunit.client.0.smithi063.stdout:1/516: readlink d3/d3f/d4d/d52/l75 0 2024-03-23T11:47:44.057 INFO:tasks.workunit.client.0.smithi063.stdout:3/890: symlink d13/d66/dde/df5/l118 0 2024-03-23T11:47:44.059 INFO:tasks.workunit.client.0.smithi063.stdout:5/413: rmdir d2/d17/d4d/d6e 39 2024-03-23T11:47:44.063 INFO:tasks.workunit.client.0.smithi063.stdout:4/624: getdents d5/da/d20/d4b/d16/d50/d86/da2 0 2024-03-23T11:47:44.066 INFO:tasks.workunit.client.0.smithi063.stdout:5/414: truncate d2/d17/f19 4351880 0 2024-03-23T11:47:44.074 INFO:tasks.workunit.client.0.smithi063.stdout:0/708: symlink d2/d12/d32/l100 0 2024-03-23T11:47:44.076 INFO:tasks.workunit.client.0.smithi063.stdout:9/614: write d4/d4c/f30 [1569430,73435] 0 2024-03-23T11:47:44.076 INFO:tasks.workunit.client.0.smithi063.stdout:6/491: dwrite d1/f6e [0,4194304] 0 2024-03-23T11:47:44.076 INFO:tasks.workunit.client.0.smithi063.stdout:8/882: mknod d6/d1f/d54/d63/dc7/c125 0 2024-03-23T11:47:44.081 INFO:tasks.workunit.client.0.smithi063.stdout:4/625: fsync d5/da/fa1 0 2024-03-23T11:47:44.082 INFO:tasks.workunit.client.0.smithi063.stdout:5/415: mknod d2/d28/d3a/d46/c90 0 2024-03-23T11:47:44.087 INFO:tasks.workunit.client.0.smithi063.stdout:7/561: write d5/d26/f72 [1733240,37135] 0 2024-03-23T11:47:44.088 INFO:tasks.workunit.client.0.smithi063.stdout:3/891: rename d13/f9c to d13/d22/d2d/d3f/d58/d97/df8/f119 0 2024-03-23T11:47:44.092 INFO:tasks.workunit.client.0.smithi063.stdout:9/615: creat d4/d4c/d19/fd6 x:0 0 0 2024-03-23T11:47:44.098 INFO:tasks.workunit.client.0.smithi063.stdout:0/709: mkdir d2/d14/d36/d77/d94/da6/d1f/d101 0 2024-03-23T11:47:44.098 INFO:tasks.workunit.client.0.smithi063.stdout:5/416: mknod d2/d7/d27/c91 0 2024-03-23T11:47:44.099 INFO:tasks.workunit.client.0.smithi063.stdout:5/417: chown d2/d7 12043 1 2024-03-23T11:47:44.103 INFO:tasks.workunit.client.0.smithi063.stdout:2/507: dwrite d2/d23/d2e/f36 [0,4194304] 0 2024-03-23T11:47:44.105 INFO:tasks.workunit.client.0.smithi063.stdout:5/418: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:44.110 INFO:tasks.workunit.client.0.smithi063.stdout:1/517: dwrite d3/d49/d56/d65/f79 [0,4194304] 0 2024-03-23T11:47:44.112 INFO:tasks.workunit.client.0.smithi063.stdout:6/492: truncate d1/dc/d21/f29 983770 0 2024-03-23T11:47:44.114 INFO:tasks.workunit.client.1.smithi111.stdout:6/765: write d2/d1f/d45/d86/faa [843019,3666] 0 2024-03-23T11:47:44.114 INFO:tasks.workunit.client.0.smithi063.stdout:4/626: creat d5/da/d20/d4b/d16/fb4 x:0 0 0 2024-03-23T11:47:44.119 INFO:tasks.workunit.client.0.smithi063.stdout:7/562: write d5/d5c/d2f/d73/da/f17 [7817086,26913] 0 2024-03-23T11:47:44.119 INFO:tasks.workunit.client.0.smithi063.stdout:6/493: chown d1/dc/f46 1746947707 1 2024-03-23T11:47:44.124 INFO:tasks.workunit.client.0.smithi063.stdout:8/883: getdents d6 0 2024-03-23T11:47:44.129 INFO:tasks.workunit.client.0.smithi063.stdout:2/508: fsync d2/de/f2b 0 2024-03-23T11:47:44.131 INFO:tasks.workunit.client.0.smithi063.stdout:3/892: dwrite d13/f56 [0,4194304] 0 2024-03-23T11:47:44.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:43 smithi063 ceph-mon[35189]: from='client.24465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:44.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:43 smithi063 ceph-mon[35189]: pgmap v41: 65 pgs: 65 active+clean; 2.5 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 102 MiB/s rd, 278 MiB/s wr, 535 op/s 2024-03-23T11:47:44.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:43 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3826751852' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:47:44.134 INFO:tasks.workunit.client.1.smithi111.stdout:0/738: chown d5/db/d1f/l2f 24242 1 2024-03-23T11:47:44.136 INFO:tasks.workunit.client.0.smithi063.stdout:4/627: symlink d5/da/d20/d4b/d16/d50/lb5 0 2024-03-23T11:47:44.136 INFO:tasks.workunit.client.0.smithi063.stdout:3/893: write d13/f56 [2604460,58228] 0 2024-03-23T11:47:44.138 INFO:tasks.workunit.client.0.smithi063.stdout:6/494: mknod d1/dc/d3a/c92 0 2024-03-23T11:47:44.142 INFO:tasks.workunit.client.1.smithi111.stdout:2/809: creat d2/d11/d17/d1f/da0/dd2/f123 x:0 0 0 2024-03-23T11:47:44.142 INFO:tasks.workunit.client.1.smithi111.stdout:4/797: truncate d0/d95/d9b/db3/dcc/dea/f1b 5481576 0 2024-03-23T11:47:44.143 INFO:tasks.workunit.client.1.smithi111.stdout:1/755: mkdir d0/d2b/d37/d6a/d100 0 2024-03-23T11:47:44.144 INFO:tasks.workunit.client.1.smithi111.stdout:5/806: write d1/d3/d6/d7/f7c [445737,70964] 0 2024-03-23T11:47:44.144 INFO:tasks.workunit.client.0.smithi063.stdout:1/518: dwrite d3/d35/f93 [0,4194304] 0 2024-03-23T11:47:44.145 INFO:tasks.workunit.client.0.smithi063.stdout:5/419: write d2/f44 [1152414,128281] 0 2024-03-23T11:47:44.146 INFO:tasks.workunit.client.0.smithi063.stdout:0/710: dread d2/d14/d36/d77/d94/da6/d28/fa3 [0,4194304] 0 2024-03-23T11:47:44.148 INFO:tasks.workunit.client.1.smithi111.stdout:5/807: stat d1/d3/d6/d30/db2/f109 0 2024-03-23T11:47:44.149 INFO:tasks.workunit.client.0.smithi063.stdout:9/616: link d4/d31/d9c/d9e/cc2 d4/d31/d9c/d9e/cd7 0 2024-03-23T11:47:44.153 INFO:tasks.workunit.client.0.smithi063.stdout:8/884: mkdir d6/d1f/d54/d63/d90/d5d/d7d/d9a/d126 0 2024-03-23T11:47:44.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:43 smithi111 ceph-mon[44005]: from='client.24465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:44.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:43 smithi111 ceph-mon[44005]: pgmap v41: 65 pgs: 65 active+clean; 2.5 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 102 MiB/s rd, 278 MiB/s wr, 535 op/s 2024-03-23T11:47:44.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:43 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3826751852' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:47:44.155 INFO:tasks.workunit.client.0.smithi063.stdout:2/509: creat d2/d23/d56/fa4 x:0 0 0 2024-03-23T11:47:44.157 INFO:tasks.workunit.client.1.smithi111.stdout:7/766: dwrite d14/d1b/f2d [0,4194304] 0 2024-03-23T11:47:44.157 INFO:tasks.workunit.client.0.smithi063.stdout:2/510: chown d2/d23/d2e/d52/f6f 0 1 2024-03-23T11:47:44.159 INFO:tasks.workunit.client.1.smithi111.stdout:7/767: chown d14/d37/fda 417321 1 2024-03-23T11:47:44.161 INFO:tasks.workunit.client.0.smithi063.stdout:3/894: mkdir d13/d22/d2d/d3f/d67/d11a 0 2024-03-23T11:47:44.162 INFO:tasks.workunit.client.0.smithi063.stdout:5/420: stat d2/d17/d4d/d6e 0 2024-03-23T11:47:44.167 INFO:tasks.workunit.client.1.smithi111.stdout:5/808: dwrite d1/d3/d6/d7/d7b/fa4 [0,4194304] 0 2024-03-23T11:47:44.170 INFO:tasks.workunit.client.0.smithi063.stdout:8/885: mknod d6/c127 0 2024-03-23T11:47:44.171 INFO:tasks.workunit.client.0.smithi063.stdout:9/617: symlink d4/d31/d43/da9/da6/db5/ld8 0 2024-03-23T11:47:44.173 INFO:tasks.workunit.client.0.smithi063.stdout:2/511: dread d2/d23/d56/f57 [0,4194304] 0 2024-03-23T11:47:44.173 INFO:tasks.workunit.client.0.smithi063.stdout:9/618: chown d4/c56 8549791 1 2024-03-23T11:47:44.174 INFO:tasks.workunit.client.0.smithi063.stdout:0/711: mknod d2/d14/d36/d77/d94/da6/d25/d93/dc9/ddd/c102 0 2024-03-23T11:47:44.174 INFO:tasks.workunit.client.0.smithi063.stdout:3/895: read f6 [3319727,19387] 0 2024-03-23T11:47:44.178 INFO:tasks.workunit.client.0.smithi063.stdout:1/519: creat d3/d3f/fa3 x:0 0 0 2024-03-23T11:47:44.180 INFO:tasks.workunit.client.0.smithi063.stdout:5/421: fsync d2/d28/f4f 0 2024-03-23T11:47:44.182 INFO:tasks.workunit.client.1.smithi111.stdout:0/739: mkdir d5/db/d4d/df8 0 2024-03-23T11:47:44.184 INFO:tasks.workunit.client.0.smithi063.stdout:4/628: write d5/da/d20/d4b/d16/fa3 [796347,7667] 0 2024-03-23T11:47:44.184 INFO:tasks.workunit.client.1.smithi111.stdout:2/810: creat d2/de/d12/d70/f124 x:0 0 0 2024-03-23T11:47:44.186 INFO:tasks.workunit.client.0.smithi063.stdout:0/712: fdatasync d2/d14/f1c 0 2024-03-23T11:47:44.186 INFO:tasks.workunit.client.0.smithi063.stdout:7/563: truncate d5/d26/d76/f78 3863666 0 2024-03-23T11:47:44.188 INFO:tasks.workunit.client.1.smithi111.stdout:5/809: creat d1/d3/d6/d7/d63/d2c/d43/f10f x:0 0 0 2024-03-23T11:47:44.188 INFO:tasks.workunit.client.0.smithi063.stdout:0/713: readlink d2/d14/l76 0 2024-03-23T11:47:44.189 INFO:tasks.workunit.client.0.smithi063.stdout:9/619: mkdir d4/d31/d70/d8d/d90/dd9 0 2024-03-23T11:47:44.190 INFO:tasks.workunit.client.1.smithi111.stdout:9/832: rmdir d0/d4/d1c/dfe 0 2024-03-23T11:47:44.193 INFO:tasks.workunit.client.1.smithi111.stdout:0/740: creat d5/db/d1f/d47/de0/ff9 x:0 0 0 2024-03-23T11:47:44.193 INFO:tasks.workunit.client.0.smithi063.stdout:9/620: stat d4/d31/d43/d49/d8e/l9b 0 2024-03-23T11:47:44.194 INFO:tasks.workunit.client.0.smithi063.stdout:1/520: mknod d3/df/d2f/ca4 0 2024-03-23T11:47:44.197 INFO:tasks.workunit.client.0.smithi063.stdout:2/512: mknod d2/d1c/ca5 0 2024-03-23T11:47:44.198 INFO:tasks.workunit.client.1.smithi111.stdout:7/768: fdatasync d14/d1b/d1e/d86/fbe 0 2024-03-23T11:47:44.199 INFO:tasks.workunit.client.0.smithi063.stdout:2/513: chown d2/d23/d2e/d52/d99 0 1 2024-03-23T11:47:44.200 INFO:tasks.workunit.client.1.smithi111.stdout:3/713: getdents d0/d1a/d1d/d26 0 2024-03-23T11:47:44.200 INFO:tasks.workunit.client.1.smithi111.stdout:6/766: fdatasync d2/d80/f96 0 2024-03-23T11:47:44.202 INFO:tasks.workunit.client.1.smithi111.stdout:7/769: write d14/d44/d70/d7b/fa8 [2469688,46332] 0 2024-03-23T11:47:44.202 INFO:tasks.workunit.client.0.smithi063.stdout:7/564: dread d5/d6/d19/d1f/d6c/f80 [0,4194304] 0 2024-03-23T11:47:44.202 INFO:tasks.workunit.client.1.smithi111.stdout:5/810: mkdir d1/d110 0 2024-03-23T11:47:44.204 INFO:tasks.workunit.client.0.smithi063.stdout:3/896: creat d13/d15/d114/f11b x:0 0 0 2024-03-23T11:47:44.206 INFO:tasks.workunit.client.1.smithi111.stdout:5/811: write d1/d3/d6/d8b/d59/d74/fa5 [2859845,82538] 0 2024-03-23T11:47:44.206 INFO:tasks.workunit.client.1.smithi111.stdout:7/770: truncate d14/d1b/d1e/d86/f93 296005 0 2024-03-23T11:47:44.207 INFO:tasks.workunit.client.1.smithi111.stdout:9/833: mknod d0/d4/d1c/d2d/d3b/d42/d4f/d95/c10e 0 2024-03-23T11:47:44.209 INFO:tasks.workunit.client.0.smithi063.stdout:3/897: chown d13/d22/d2d/d3f/d5b/f77 1415027 1 2024-03-23T11:47:44.209 INFO:tasks.workunit.client.1.smithi111.stdout:1/756: rename d0/fdd to d0/d5/dd/f101 0 2024-03-23T11:47:44.212 INFO:tasks.workunit.client.0.smithi063.stdout:3/898: chown d13/d22/d2d/d3f/d58/dcb 13710123 1 2024-03-23T11:47:44.213 INFO:tasks.workunit.client.0.smithi063.stdout:1/521: rmdir d3/d3f/d5c 39 2024-03-23T11:47:44.214 INFO:tasks.workunit.client.1.smithi111.stdout:2/811: mknod d2/de0/d110/d112/c125 0 2024-03-23T11:47:44.221 INFO:tasks.workunit.client.1.smithi111.stdout:6/767: creat d2/d1f/dce/ff8 x:0 0 0 2024-03-23T11:47:44.221 INFO:tasks.workunit.client.0.smithi063.stdout:5/422: write f1 [3281386,109031] 0 2024-03-23T11:47:44.222 INFO:tasks.workunit.client.1.smithi111.stdout:7/771: chown lf 122775266 1 2024-03-23T11:47:44.223 INFO:tasks.workunit.client.0.smithi063.stdout:4/629: dread d5/d1e/d2b/f7c [0,4194304] 0 2024-03-23T11:47:44.226 INFO:tasks.workunit.client.0.smithi063.stdout:9/621: symlink d4/lda 0 2024-03-23T11:47:44.226 INFO:tasks.workunit.client.1.smithi111.stdout:5/812: rmdir d1/d3/d3a 39 2024-03-23T11:47:44.229 INFO:tasks.workunit.client.0.smithi063.stdout:1/522: unlink d3/d35/la0 0 2024-03-23T11:47:44.230 INFO:tasks.workunit.client.1.smithi111.stdout:1/757: read - d0/d5/dd/d1e/d54/d81/f9b zero size 2024-03-23T11:47:44.233 INFO:tasks.workunit.client.1.smithi111.stdout:3/714: dread d0/d1a/d2b/d38/d7f/f9b [0,4194304] 0 2024-03-23T11:47:44.233 INFO:tasks.workunit.client.0.smithi063.stdout:0/714: dread d2/d14/de8/f9f [0,4194304] 0 2024-03-23T11:47:44.234 INFO:tasks.workunit.client.1.smithi111.stdout:2/812: dwrite d2/de/d12/d70/f7e [0,4194304] 0 2024-03-23T11:47:44.234 INFO:tasks.workunit.client.0.smithi063.stdout:3/899: mknod d13/d66/d112/c11c 0 2024-03-23T11:47:44.235 INFO:tasks.workunit.client.1.smithi111.stdout:6/768: symlink d2/d1a/d1c/lf9 0 2024-03-23T11:47:44.236 INFO:tasks.workunit.client.0.smithi063.stdout:1/523: read d3/f4 [3405841,83447] 0 2024-03-23T11:47:44.239 INFO:tasks.workunit.client.1.smithi111.stdout:9/834: dread d0/d4/d10/d29/f67 [0,4194304] 0 2024-03-23T11:47:44.240 INFO:tasks.workunit.client.0.smithi063.stdout:8/886: truncate d6/dd/d59/db2/fe2 1140030 0 2024-03-23T11:47:44.240 INFO:tasks.workunit.client.0.smithi063.stdout:0/715: read d2/d14/de8/f8e [652077,120643] 0 2024-03-23T11:47:44.240 INFO:tasks.workunit.client.0.smithi063.stdout:4/630: symlink d5/da/d20/d4b/d16/d32/d6f/d94/lb6 0 2024-03-23T11:47:44.244 INFO:tasks.workunit.client.0.smithi063.stdout:5/423: creat d2/f92 x:0 0 0 2024-03-23T11:47:44.245 INFO:tasks.workunit.client.0.smithi063.stdout:7/565: dread d5/d5c/d2f/d73/da/f17 [4194304,4194304] 0 2024-03-23T11:47:44.252 INFO:tasks.workunit.client.0.smithi063.stdout:5/424: write d2/f3e [5045115,59272] 0 2024-03-23T11:47:44.256 INFO:tasks.workunit.client.0.smithi063.stdout:2/514: dwrite d2/d23/d2e/d52/f3e [0,4194304] 0 2024-03-23T11:47:44.259 INFO:tasks.workunit.client.0.smithi063.stdout:7/566: mkdir d5/d5c/d2f/db5 0 2024-03-23T11:47:44.265 INFO:tasks.workunit.client.0.smithi063.stdout:4/631: creat d5/da/d20/fb7 x:0 0 0 2024-03-23T11:47:44.265 INFO:tasks.workunit.client.0.smithi063.stdout:5/425: fdatasync d2/d17/f51 0 2024-03-23T11:47:44.267 INFO:tasks.workunit.client.0.smithi063.stdout:2/515: creat d2/d84/d94/fa6 x:0 0 0 2024-03-23T11:47:44.273 INFO:tasks.workunit.client.0.smithi063.stdout:1/524: write d3/d35/d40/f53 [677776,70729] 0 2024-03-23T11:47:44.274 INFO:tasks.workunit.client.0.smithi063.stdout:5/426: symlink d2/d17/d4d/d6e/l93 0 2024-03-23T11:47:44.278 INFO:tasks.workunit.client.0.smithi063.stdout:0/716: dwrite d2/d12/d32/fe6 [0,4194304] 0 2024-03-23T11:47:44.279 INFO:tasks.workunit.client.1.smithi111.stdout:4/798: sync 2024-03-23T11:47:44.281 INFO:tasks.workunit.client.0.smithi063.stdout:7/567: rename d5/d26/d32/ca5 to d5/d5c/d2f/d73/da/cb6 0 2024-03-23T11:47:44.282 INFO:tasks.workunit.client.0.smithi063.stdout:2/516: write d2/de/d7a/d77/f93 [311365,120698] 0 2024-03-23T11:47:44.285 INFO:tasks.workunit.client.0.smithi063.stdout:4/632: truncate d5/da/fa1 961624 0 2024-03-23T11:47:44.291 INFO:tasks.workunit.client.0.smithi063.stdout:3/900: truncate d13/d22/d2d/fa1 658930 0 2024-03-23T11:47:44.292 INFO:tasks.workunit.client.0.smithi063.stdout:1/525: write d3/f4 [5343477,87654] 0 2024-03-23T11:47:44.297 INFO:tasks.workunit.client.0.smithi063.stdout:9/622: dwrite d4/d4c/d19/f22 [0,4194304] 0 2024-03-23T11:47:44.297 INFO:tasks.workunit.client.0.smithi063.stdout:4/633: creat d5/d1e/d2b/daa/fb8 x:0 0 0 2024-03-23T11:47:44.305 INFO:tasks.workunit.client.0.smithi063.stdout:8/887: dwrite d6/d1f/d54/d5c/d76/f92 [0,4194304] 0 2024-03-23T11:47:44.305 INFO:tasks.workunit.client.0.smithi063.stdout:5/427: rename d2/d7/d8/d1d/f74 to d2/d7/d27/f94 0 2024-03-23T11:47:44.311 INFO:tasks.workunit.client.0.smithi063.stdout:2/517: mknod d2/de/d7a/da2/ca7 0 2024-03-23T11:47:44.313 INFO:tasks.workunit.client.0.smithi063.stdout:4/634: chown d5/da/d20/d4b/d16/d32/d6f/c73 574875 1 2024-03-23T11:47:44.315 INFO:tasks.workunit.client.0.smithi063.stdout:7/568: getdents d5/d5c 0 2024-03-23T11:47:44.316 INFO:tasks.workunit.client.0.smithi063.stdout:3/901: read - d13/d22/d2d/d3f/d67/d7b/fc1 zero size 2024-03-23T11:47:44.321 INFO:tasks.workunit.client.0.smithi063.stdout:0/717: getdents d2/d14 0 2024-03-23T11:47:44.324 INFO:tasks.workunit.client.0.smithi063.stdout:2/518: symlink d2/d46/d54/la8 0 2024-03-23T11:47:44.351 INFO:tasks.workunit.client.0.smithi063.stdout:0/718: fdatasync d2/d14/d36/d77/d94/da6/d28/fdf 0 2024-03-23T11:47:44.355 INFO:tasks.workunit.client.0.smithi063.stdout:9/623: dread d4/f5a [0,4194304] 0 2024-03-23T11:47:44.362 INFO:tasks.workunit.client.0.smithi063.stdout:2/519: fdatasync d2/d23/d2e/d87/d4d/f6d 0 2024-03-23T11:47:44.365 INFO:tasks.workunit.client.0.smithi063.stdout:3/902: creat d13/d22/d2d/d78/dcc/f11d x:0 0 0 2024-03-23T11:47:44.371 INFO:tasks.workunit.client.0.smithi063.stdout:9/624: mknod d4/d31/d70/d8d/d90/dd9/cdb 0 2024-03-23T11:47:44.375 INFO:tasks.workunit.client.0.smithi063.stdout:0/719: dread d2/f56 [4194304,4194304] 0 2024-03-23T11:47:44.375 INFO:tasks.workunit.client.0.smithi063.stdout:7/569: write d5/d6/d39/f6a [1789231,107376] 0 2024-03-23T11:47:44.377 INFO:tasks.workunit.client.0.smithi063.stdout:1/526: dwrite d3/d1c/f66 [0,4194304] 0 2024-03-23T11:47:44.382 INFO:tasks.workunit.client.0.smithi063.stdout:9/625: write d4/d31/f63 [807183,110170] 0 2024-03-23T11:47:44.387 INFO:tasks.workunit.client.0.smithi063.stdout:7/570: mkdir d5/d26/db7 0 2024-03-23T11:47:44.393 INFO:tasks.workunit.client.0.smithi063.stdout:4/635: dread d5/da/d20/d4b/f68 [0,4194304] 0 2024-03-23T11:47:44.397 INFO:tasks.workunit.client.0.smithi063.stdout:9/626: creat d4/d31/d70/d8d/fdc x:0 0 0 2024-03-23T11:47:44.400 INFO:tasks.workunit.client.0.smithi063.stdout:1/527: fsync d3/f38 0 2024-03-23T11:47:44.401 INFO:tasks.workunit.client.0.smithi063.stdout:8/888: dread d6/d1f/d3e/f8c [0,4194304] 0 2024-03-23T11:47:44.404 INFO:tasks.workunit.client.0.smithi063.stdout:2/520: dwrite d2/d23/d2e/f4b [0,4194304] 0 2024-03-23T11:47:44.404 INFO:tasks.workunit.client.0.smithi063.stdout:7/571: write d5/d5c/d2f/d73/da/f4e [64072,13565] 0 2024-03-23T11:47:44.404 INFO:tasks.workunit.client.0.smithi063.stdout:5/428: dwrite d2/d2f/f70 [0,4194304] 0 2024-03-23T11:47:44.412 INFO:tasks.workunit.client.0.smithi063.stdout:7/572: chown d5/d6/d19/d1f 1516 1 2024-03-23T11:47:44.416 INFO:tasks.workunit.client.0.smithi063.stdout:2/521: stat d2/cd 0 2024-03-23T11:47:44.418 INFO:tasks.workunit.client.0.smithi063.stdout:0/720: rename d2/d14/d36/d77/d94/da6/d28/d46/cac to d2/d4/c103 0 2024-03-23T11:47:44.420 INFO:tasks.workunit.client.0.smithi063.stdout:5/429: symlink d2/d7/d2b/l95 0 2024-03-23T11:47:44.422 INFO:tasks.workunit.client.0.smithi063.stdout:3/903: dwrite d13/d22/d2d/d3f/d58/d86/dad/f107 [0,4194304] 0 2024-03-23T11:47:44.427 INFO:tasks.workunit.client.0.smithi063.stdout:8/889: mkdir d6/d30/d128 0 2024-03-23T11:47:44.431 INFO:tasks.workunit.client.0.smithi063.stdout:7/573: mknod d5/d5c/d2f/d73/da/d90/cb8 0 2024-03-23T11:47:44.433 INFO:tasks.workunit.client.0.smithi063.stdout:1/528: dwrite d3/f10 [4194304,4194304] 0 2024-03-23T11:47:44.437 INFO:tasks.workunit.client.0.smithi063.stdout:0/721: symlink d2/d48/da5/da7/l104 0 2024-03-23T11:47:44.442 INFO:tasks.workunit.client.0.smithi063.stdout:4/636: link d5/da/d20/d4b/d16/d50/d8e/fa7 d5/d1e/fb9 0 2024-03-23T11:47:44.446 INFO:tasks.workunit.client.0.smithi063.stdout:3/904: symlink d13/d22/d2d/d3f/d58/d97/l11e 0 2024-03-23T11:47:44.448 INFO:tasks.workunit.client.0.smithi063.stdout:1/529: read d3/d1c/f6d [2660918,100804] 0 2024-03-23T11:47:44.450 INFO:tasks.workunit.client.0.smithi063.stdout:9/627: rename d4/d4c/d19/f2d to d4/d31/d43/fdd 0 2024-03-23T11:47:44.453 INFO:tasks.workunit.client.0.smithi063.stdout:0/722: fdatasync d2/d12/f21 0 2024-03-23T11:47:44.464 INFO:tasks.workunit.client.0.smithi063.stdout:8/890: rmdir d6/d1c/d32/d37/d10a/dc8/dfc 39 2024-03-23T11:47:44.467 INFO:tasks.workunit.client.0.smithi063.stdout:1/530: mkdir d3/d1c/d4b/da5 0 2024-03-23T11:47:44.468 INFO:tasks.workunit.client.0.smithi063.stdout:9/628: chown d4/d4c/d44/f84 11 1 2024-03-23T11:47:44.469 INFO:tasks.workunit.client.0.smithi063.stdout:3/905: creat d13/d22/d2d/d3f/de0/f11f x:0 0 0 2024-03-23T11:47:44.477 INFO:tasks.workunit.client.0.smithi063.stdout:8/891: write d6/d1f/d54/d63/d90/d5d/d7d/d9a/daa/f11f [27365,99847] 0 2024-03-23T11:47:44.478 INFO:tasks.workunit.client.0.smithi063.stdout:5/430: dread d2/d7/f10 [0,4194304] 0 2024-03-23T11:47:44.482 INFO:tasks.workunit.client.0.smithi063.stdout:2/522: dwrite d2/f29 [0,4194304] 0 2024-03-23T11:47:44.487 INFO:tasks.workunit.client.0.smithi063.stdout:7/574: dwrite d5/f12 [0,4194304] 0 2024-03-23T11:47:44.491 INFO:tasks.workunit.client.0.smithi063.stdout:7/575: fsync d5/d22/f9d 0 2024-03-23T11:47:44.495 INFO:tasks.workunit.client.0.smithi063.stdout:0/723: mknod d2/d14/d36/d77/d94/da6/d25/c105 0 2024-03-23T11:47:44.499 INFO:tasks.workunit.client.0.smithi063.stdout:0/724: write d2/d14/d36/d77/fbd [1242049,100527] 0 2024-03-23T11:47:44.500 INFO:tasks.workunit.client.0.smithi063.stdout:8/892: creat d6/d1f/d106/f129 x:0 0 0 2024-03-23T11:47:44.500 INFO:tasks.workunit.client.0.smithi063.stdout:1/531: write d3/df/f7d [370482,16871] 0 2024-03-23T11:47:44.501 INFO:tasks.workunit.client.0.smithi063.stdout:3/906: rename d13/d15/l88 to d13/d22/d2d/d3f/d58/d6e/d8d/dda/de1/l120 0 2024-03-23T11:47:44.505 INFO:tasks.workunit.client.0.smithi063.stdout:3/907: fsync d13/d22/d2d/d3f/d58/d86/dad/fef 0 2024-03-23T11:47:44.508 INFO:tasks.workunit.client.0.smithi063.stdout:2/523: creat d2/de/d7a/d8d/fa9 x:0 0 0 2024-03-23T11:47:44.512 INFO:tasks.workunit.client.0.smithi063.stdout:3/908: chown d13/d22/d2d/d3f/d58/d86/dad/fb6 4141 1 2024-03-23T11:47:44.514 INFO:tasks.workunit.client.0.smithi063.stdout:2/524: symlink d2/d23/d56/laa 0 2024-03-23T11:47:44.519 INFO:tasks.workunit.client.0.smithi063.stdout:9/629: creat d4/d4c/fde x:0 0 0 2024-03-23T11:47:44.521 INFO:tasks.workunit.client.0.smithi063.stdout:7/576: mkdir d5/d6/d19/d1f/d47/d6e/d98/da1/db9 0 2024-03-23T11:47:44.526 INFO:tasks.workunit.client.0.smithi063.stdout:7/577: write d5/d6/d19/d1f/d47/fb3 [66468,73177] 0 2024-03-23T11:47:44.528 INFO:tasks.workunit.client.0.smithi063.stdout:5/431: dwrite d2/f7f [0,4194304] 0 2024-03-23T11:47:44.529 INFO:tasks.workunit.client.0.smithi063.stdout:7/578: chown d5/d6/d39/c84 15 1 2024-03-23T11:47:44.529 INFO:tasks.workunit.client.0.smithi063.stdout:0/725: dread d2/d14/f64 [0,4194304] 0 2024-03-23T11:47:44.531 INFO:tasks.workunit.client.0.smithi063.stdout:7/579: dread - d5/d6/d19/f7d zero size 2024-03-23T11:47:44.547 INFO:tasks.workunit.client.0.smithi063.stdout:6/495: sync 2024-03-23T11:47:44.550 INFO:tasks.workunit.client.0.smithi063.stdout:2/525: readlink d2/d23/d2e/d52/l38 0 2024-03-23T11:47:44.554 INFO:tasks.workunit.client.0.smithi063.stdout:9/630: fdatasync d4/fc 0 2024-03-23T11:47:44.554 INFO:tasks.workunit.client.0.smithi063.stdout:7/580: dread d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:44.557 INFO:tasks.workunit.client.0.smithi063.stdout:7/581: fsync d5/d6/d19/d1f/d47/fb3 0 2024-03-23T11:47:44.559 INFO:tasks.workunit.client.0.smithi063.stdout:0/726: dread - d2/d6e/d8c/f9e zero size 2024-03-23T11:47:44.562 INFO:tasks.workunit.client.0.smithi063.stdout:0/727: dread - d2/d14/d36/d77/d94/da6/d1f/d4f/fcc zero size 2024-03-23T11:47:44.570 INFO:tasks.workunit.client.0.smithi063.stdout:9/631: write d4/d31/d43/d49/fa4 [832170,110891] 0 2024-03-23T11:47:44.572 INFO:tasks.workunit.client.0.smithi063.stdout:1/532: link d3/l6f d3/d1c/d4b/da5/la6 0 2024-03-23T11:47:44.576 INFO:tasks.workunit.client.0.smithi063.stdout:7/582: creat d5/d6/d19/d65/fba x:0 0 0 2024-03-23T11:47:44.577 INFO:tasks.workunit.client.0.smithi063.stdout:7/583: chown d5/d5c/d2f/d73/da 1336248 1 2024-03-23T11:47:44.578 INFO:tasks.workunit.client.0.smithi063.stdout:0/728: fdatasync d2/d12/f68 0 2024-03-23T11:47:44.581 INFO:tasks.workunit.client.0.smithi063.stdout:1/533: mkdir d3/d49/d56/d65/d84/da7 0 2024-03-23T11:47:44.583 INFO:tasks.workunit.client.0.smithi063.stdout:7/584: symlink d5/d26/lbb 0 2024-03-23T11:47:44.585 INFO:tasks.workunit.client.0.smithi063.stdout:5/432: getdents d2/d7/d8/de/d35 0 2024-03-23T11:47:44.587 INFO:tasks.workunit.client.0.smithi063.stdout:0/729: unlink d2/d14/de8/f6b 0 2024-03-23T11:47:44.592 INFO:tasks.workunit.client.0.smithi063.stdout:7/585: dread d5/d5c/d2f/f79 [0,4194304] 0 2024-03-23T11:47:44.593 INFO:tasks.workunit.client.0.smithi063.stdout:9/632: link d4/lda d4/d31/d43/da9/da6/ldf 0 2024-03-23T11:47:44.596 INFO:tasks.workunit.client.0.smithi063.stdout:7/586: chown d5/d22/l4b 482221 1 2024-03-23T11:47:44.597 INFO:tasks.workunit.client.0.smithi063.stdout:9/633: chown d4/d4c/d19/f22 192 1 2024-03-23T11:47:44.599 INFO:tasks.workunit.client.0.smithi063.stdout:4/637: truncate d5/da/d20/d4b/d16/d50/f69 4190067 0 2024-03-23T11:47:44.601 INFO:tasks.workunit.client.0.smithi063.stdout:1/534: dwrite d3/d3f/f77 [0,4194304] 0 2024-03-23T11:47:44.605 INFO:tasks.workunit.client.0.smithi063.stdout:0/730: symlink d2/d12/d35/dcf/l106 0 2024-03-23T11:47:44.606 INFO:tasks.workunit.client.0.smithi063.stdout:5/433: rmdir d2/d7/d64/d6b 0 2024-03-23T11:47:44.609 INFO:tasks.workunit.client.0.smithi063.stdout:7/587: write d5/d6/d19/d1f/f6d [1352052,92744] 0 2024-03-23T11:47:44.611 INFO:tasks.workunit.client.0.smithi063.stdout:3/909: write d13/d22/d2d/d3f/d67/d7b/fc1 [314321,88253] 0 2024-03-23T11:47:44.611 INFO:tasks.workunit.client.0.smithi063.stdout:1/535: mkdir d3/d1c/da8 0 2024-03-23T11:47:44.611 INFO:tasks.workunit.client.0.smithi063.stdout:0/731: fsync d2/d12/d32/fc0 0 2024-03-23T11:47:44.616 INFO:tasks.workunit.client.0.smithi063.stdout:5/434: read d2/d28/f2d [508713,85607] 0 2024-03-23T11:47:44.622 INFO:tasks.workunit.client.0.smithi063.stdout:4/638: fsync d5/da/d20/d4b/d16/d37/f95 0 2024-03-23T11:47:44.624 INFO:tasks.workunit.client.0.smithi063.stdout:2/526: write d2/d23/d2e/f35 [717206,50219] 0 2024-03-23T11:47:44.625 INFO:tasks.workunit.client.0.smithi063.stdout:3/910: truncate d13/d22/d2d/d3f/d58/d97/fea 690672 0 2024-03-23T11:47:44.632 INFO:tasks.workunit.client.0.smithi063.stdout:9/634: getdents d4/d31/d70/d8d/d90/dd9 0 2024-03-23T11:47:44.636 INFO:tasks.workunit.client.0.smithi063.stdout:5/435: dread - d2/d28/d3a/f8d zero size 2024-03-23T11:47:44.637 INFO:tasks.workunit.client.0.smithi063.stdout:3/911: mkdir d13/d66/d112/d121 0 2024-03-23T11:47:44.640 INFO:tasks.workunit.client.0.smithi063.stdout:6/496: dwrite d1/dc/d21/d49/f6a [0,4194304] 0 2024-03-23T11:47:44.648 INFO:tasks.workunit.client.0.smithi063.stdout:4/639: dwrite f2 [0,4194304] 0 2024-03-23T11:47:44.653 INFO:tasks.workunit.client.0.smithi063.stdout:4/640: chown d5/l6 1 1 2024-03-23T11:47:44.659 INFO:tasks.workunit.client.0.smithi063.stdout:9/635: rename d4/d31/d43/da9/da6/f4a to d4/d31/d70/d8d/db3/fe0 0 2024-03-23T11:47:44.663 INFO:tasks.workunit.client.1.smithi111.stdout:5/813: stat d1/d3/d6/d30/db2/f106 0 2024-03-23T11:47:44.664 INFO:tasks.workunit.client.0.smithi063.stdout:0/732: link d2/d14/d36/d77/d94/da6/d1f/d4f/ldc d2/d14/de8/l107 0 2024-03-23T11:47:44.664 INFO:tasks.workunit.client.0.smithi063.stdout:7/588: getdents d5/d5c/d2f/d8c 0 2024-03-23T11:47:44.672 INFO:tasks.workunit.client.0.smithi063.stdout:6/497: dread d1/dc/d21/f29 [0,4194304] 0 2024-03-23T11:47:44.673 INFO:tasks.workunit.client.0.smithi063.stdout:2/527: write d2/d23/d2e/d52/f71 [1328983,90277] 0 2024-03-23T11:47:44.674 INFO:tasks.workunit.client.0.smithi063.stdout:4/641: rename d5/da/d20/d4b/d1f/f3f to d5/da/d20/d4b/d16/d32/fba 0 2024-03-23T11:47:44.676 INFO:tasks.workunit.client.1.smithi111.stdout:0/741: creat d5/db/d1f/d47/d6f/ffa x:0 0 0 2024-03-23T11:47:44.678 INFO:tasks.workunit.client.1.smithi111.stdout:5/814: dwrite d1/d3/d6/d7/d63/d2c/d82/d66/f71 [0,4194304] 0 2024-03-23T11:47:44.679 INFO:tasks.workunit.client.1.smithi111.stdout:0/742: chown d5/dc3/cd2 23729 1 2024-03-23T11:47:44.679 INFO:tasks.workunit.client.1.smithi111.stdout:5/815: chown d1/d3/f4 16732273 1 2024-03-23T11:47:44.680 INFO:tasks.workunit.client.1.smithi111.stdout:3/715: creat d0/d1a/d1d/d26/d5a/fef x:0 0 0 2024-03-23T11:47:44.686 INFO:tasks.workunit.client.0.smithi063.stdout:1/536: dwrite d3/d35/d40/f4c [4194304,4194304] 0 2024-03-23T11:47:44.687 INFO:tasks.workunit.client.1.smithi111.stdout:2/813: unlink d2/d11/d17/d3e/d5d/lf8 0 2024-03-23T11:47:44.688 INFO:tasks.workunit.client.1.smithi111.stdout:6/769: truncate d2/f3b 633723 0 2024-03-23T11:47:44.689 INFO:tasks.workunit.client.1.smithi111.stdout:6/770: fsync d2/feb 0 2024-03-23T11:47:44.689 INFO:tasks.workunit.client.0.smithi063.stdout:6/498: creat d1/d2a/f93 x:0 0 0 2024-03-23T11:47:44.690 INFO:tasks.workunit.client.0.smithi063.stdout:3/912: write d13/d22/d2d/d3f/d5b/f5d [5185744,90857] 0 2024-03-23T11:47:44.690 INFO:tasks.workunit.client.1.smithi111.stdout:6/771: chown d2/d35/cda 7722 1 2024-03-23T11:47:44.695 INFO:tasks.workunit.client.1.smithi111.stdout:4/799: fdatasync d0/d95/db5/dc7/ff8 0 2024-03-23T11:47:44.697 INFO:tasks.workunit.client.0.smithi063.stdout:9/636: rename d4/d31/f6e to d4/d31/d70/d8d/da0/fe1 0 2024-03-23T11:47:44.698 INFO:tasks.workunit.client.1.smithi111.stdout:7/772: mknod d14/d1b/d1e/d2c/d2e/ce0 0 2024-03-23T11:47:44.698 INFO:tasks.workunit.client.1.smithi111.stdout:4/800: fdatasync d0/d95/d9b/db3/dcc/dea/fee 0 2024-03-23T11:47:44.700 INFO:tasks.workunit.client.1.smithi111.stdout:5/816: fsync d1/d3/f4 0 2024-03-23T11:47:44.701 INFO:tasks.workunit.client.0.smithi063.stdout:6/499: chown d1/d2a/d41/d42/l5f 20900117 1 2024-03-23T11:47:44.705 INFO:tasks.workunit.client.1.smithi111.stdout:5/817: write d1/d3/d6/d8b/d59/d74/f21 [6301967,77127] 0 2024-03-23T11:47:44.709 INFO:tasks.workunit.client.1.smithi111.stdout:6/772: unlink d2/d1a/d1c/d2c/d51/f5f 0 2024-03-23T11:47:44.714 INFO:tasks.workunit.client.1.smithi111.stdout:6/773: chown d2/d1a/d1c/d2c/d51/l47 377340 1 2024-03-23T11:47:44.716 INFO:tasks.workunit.client.0.smithi063.stdout:5/436: dwrite d2/f6 [0,4194304] 0 2024-03-23T11:47:44.717 INFO:tasks.workunit.client.0.smithi063.stdout:0/733: dwrite d2/d14/d36/d77/d94/daf/fd2 [0,4194304] 0 2024-03-23T11:47:44.725 INFO:tasks.workunit.client.1.smithi111.stdout:5/818: dwrite d1/d3/d6/d7/d63/d2c/f51 [0,4194304] 0 2024-03-23T11:47:44.725 INFO:tasks.workunit.client.1.smithi111.stdout:4/801: mkdir d0/d95/db5/d108 0 2024-03-23T11:47:44.729 INFO:tasks.workunit.client.1.smithi111.stdout:0/743: link d5/db/f15 d5/db/d1f/de4/ffb 0 2024-03-23T11:47:44.730 INFO:tasks.workunit.client.0.smithi063.stdout:9/637: dwrite d4/f5 [0,4194304] 0 2024-03-23T11:47:44.733 INFO:tasks.workunit.client.1.smithi111.stdout:0/744: readlink d5/db/d4d/ld1 0 2024-03-23T11:47:44.735 INFO:tasks.workunit.client.1.smithi111.stdout:1/758: dread d0/d6/d48/f5d [4194304,4194304] 0 2024-03-23T11:47:44.737 INFO:tasks.workunit.client.1.smithi111.stdout:0/745: chown d5/db/d1f/d32/f3e 0 1 2024-03-23T11:47:44.740 INFO:tasks.workunit.client.1.smithi111.stdout:0/746: dread - d5/db/d1f/d19/d45/fad zero size 2024-03-23T11:47:44.742 INFO:tasks.workunit.client.1.smithi111.stdout:7/773: dread d14/fba [0,4194304] 0 2024-03-23T11:47:44.744 INFO:tasks.workunit.client.1.smithi111.stdout:0/747: chown d5/db/d1f/de4/l97 12275 1 2024-03-23T11:47:44.749 INFO:tasks.workunit.client.1.smithi111.stdout:9/835: write d0/d4/d1c/d2d/d75/d7a/fc5 [1836197,81433] 0 2024-03-23T11:47:44.750 INFO:tasks.workunit.client.1.smithi111.stdout:4/802: mkdir d0/d95/db5/d108/d109 0 2024-03-23T11:47:44.751 INFO:tasks.workunit.client.1.smithi111.stdout:4/803: chown d0/dc/l61 248 1 2024-03-23T11:47:44.754 INFO:tasks.workunit.client.1.smithi111.stdout:6/774: link d2/d1f/d45/d9a/dde/led d2/d1a/d1c/ddf/lfa 0 2024-03-23T11:47:44.757 INFO:tasks.workunit.client.1.smithi111.stdout:6/775: fsync d2/d35/fb5 0 2024-03-23T11:47:44.761 INFO:tasks.workunit.client.0.smithi063.stdout:9/638: dwrite d4/d31/d43/d49/f9a [4194304,4194304] 0 2024-03-23T11:47:44.763 INFO:tasks.workunit.client.0.smithi063.stdout:9/639: dread - d4/d31/d9c/d9e/fc5 zero size 2024-03-23T11:47:44.767 INFO:tasks.workunit.client.1.smithi111.stdout:7/774: dwrite d14/d1b/d1e/d2c/d2e/d73/dae/f62 [0,4194304] 0 2024-03-23T11:47:44.770 INFO:tasks.workunit.client.1.smithi111.stdout:3/716: dwrite d0/f1f [0,4194304] 0 2024-03-23T11:47:44.776 INFO:tasks.workunit.client.1.smithi111.stdout:1/759: link d0/d5/dd/d1e/d70/cf5 d0/d5/dd/d1e/d54/d81/c102 0 2024-03-23T11:47:44.776 INFO:tasks.workunit.client.1.smithi111.stdout:6/776: symlink d2/lfb 0 2024-03-23T11:47:44.777 INFO:tasks.workunit.client.1.smithi111.stdout:0/748: link d5/db/d1f/d19/dbd/c60 d5/db/d1f/d19/dbd/d64/dc7/df4/cfc 0 2024-03-23T11:47:44.777 INFO:tasks.workunit.client.1.smithi111.stdout:9/836: dwrite d0/d4/d1c/d36/f59 [4194304,4194304] 0 2024-03-23T11:47:44.778 INFO:tasks.workunit.client.1.smithi111.stdout:1/760: chown d0/d5/dd/dc6 170219836 1 2024-03-23T11:47:44.779 INFO:tasks.workunit.client.1.smithi111.stdout:7/775: mkdir d14/d44/d70/d7e/dbc/de1 0 2024-03-23T11:47:44.785 INFO:tasks.workunit.client.1.smithi111.stdout:5/819: dwrite d1/d3/d6/d7/f24 [4194304,4194304] 0 2024-03-23T11:47:44.794 INFO:tasks.workunit.client.1.smithi111.stdout:7/776: rmdir d14/d1b/d1e/d2c/d42/d5e/d88 39 2024-03-23T11:47:44.794 INFO:tasks.workunit.client.1.smithi111.stdout:3/717: creat d0/d1a/d2b/ff0 x:0 0 0 2024-03-23T11:47:44.798 INFO:tasks.workunit.client.1.smithi111.stdout:5/820: unlink d1/d3/d6/d30/l85 0 2024-03-23T11:47:44.801 INFO:tasks.workunit.client.1.smithi111.stdout:5/821: stat d1/d3/d6/d30/db2/c108 0 2024-03-23T11:47:44.805 INFO:tasks.workunit.client.1.smithi111.stdout:0/749: dwrite d5/db/d1f/d48/fb0 [0,4194304] 0 2024-03-23T11:47:44.807 INFO:tasks.workunit.client.1.smithi111.stdout:5/822: write d1/d3/d6/d8b/d59/f6a [1869139,99746] 0 2024-03-23T11:47:44.809 INFO:tasks.workunit.client.1.smithi111.stdout:3/718: creat d0/d1a/d2d/ff1 x:0 0 0 2024-03-23T11:47:44.811 INFO:tasks.workunit.client.1.smithi111.stdout:9/837: getdents d0/d31/d35 0 2024-03-23T11:47:44.814 INFO:tasks.workunit.client.1.smithi111.stdout:4/804: dwrite d0/d95/d9b/db3/dcc/dea/f92 [0,4194304] 0 2024-03-23T11:47:44.816 INFO:tasks.workunit.client.1.smithi111.stdout:5/823: write d1/d20/dce/fe4 [3598972,92313] 0 2024-03-23T11:47:44.816 INFO:tasks.workunit.client.1.smithi111.stdout:7/777: creat d14/d37/d43/fe2 x:0 0 0 2024-03-23T11:47:44.819 INFO:tasks.workunit.client.1.smithi111.stdout:9/838: fsync d0/d4/d6a/d89/fcf 0 2024-03-23T11:47:44.820 INFO:tasks.workunit.client.1.smithi111.stdout:5/824: chown d1/d3 2732575 1 2024-03-23T11:47:44.821 INFO:tasks.workunit.client.1.smithi111.stdout:3/719: write d0/d55/f97 [999749,94379] 0 2024-03-23T11:47:44.822 INFO:tasks.workunit.client.1.smithi111.stdout:7/778: chown d14/d44/d70/d7e/c84 33521930 1 2024-03-23T11:47:44.822 INFO:tasks.workunit.client.1.smithi111.stdout:4/805: rmdir d0/d95/d9b/db3/dcc/dea/da 39 2024-03-23T11:47:44.829 INFO:tasks.workunit.client.1.smithi111.stdout:5/825: read d1/d3/d6/d7/d63/fe8 [3227678,16438] 0 2024-03-23T11:47:44.830 INFO:tasks.workunit.client.1.smithi111.stdout:9/839: read d0/d31/d41/d65/f6f [1744005,93917] 0 2024-03-23T11:47:44.833 INFO:tasks.workunit.client.1.smithi111.stdout:1/761: write d0/d2b/d69/fbc [600969,62743] 0 2024-03-23T11:47:44.834 INFO:tasks.workunit.client.1.smithi111.stdout:4/806: rmdir d0/dc/d1f/db0 39 2024-03-23T11:47:44.836 INFO:tasks.workunit.client.1.smithi111.stdout:7/779: creat d14/d1b/d1e/d2c/d2e/d73/dae/d74/fe3 x:0 0 0 2024-03-23T11:47:44.836 INFO:tasks.workunit.client.1.smithi111.stdout:6/777: write d2/d1f/d45/d49/d8d/f94 [274446,109321] 0 2024-03-23T11:47:44.836 INFO:tasks.workunit.client.1.smithi111.stdout:4/807: readlink d0/dc/d78/le7 0 2024-03-23T11:47:44.837 INFO:tasks.workunit.client.1.smithi111.stdout:9/840: stat d0/d4/f47 0 2024-03-23T11:47:44.838 INFO:tasks.workunit.client.1.smithi111.stdout:2/814: sync 2024-03-23T11:47:44.842 INFO:tasks.workunit.client.1.smithi111.stdout:7/780: creat d14/d37/d54/fe4 x:0 0 0 2024-03-23T11:47:44.843 INFO:tasks.workunit.client.1.smithi111.stdout:1/762: getdents d0/d2b/d7c/dca 0 2024-03-23T11:47:44.845 INFO:tasks.workunit.client.1.smithi111.stdout:3/720: truncate d0/d55/f97 526546 0 2024-03-23T11:47:44.846 INFO:tasks.workunit.client.1.smithi111.stdout:4/808: write d0/dc/f14 [6732259,30456] 0 2024-03-23T11:47:44.846 INFO:tasks.workunit.client.1.smithi111.stdout:9/841: rename d0/d4/d1c/d2d/d3b/d42/fdf to d0/d4/d10/d91/f10f 0 2024-03-23T11:47:44.852 INFO:tasks.workunit.client.1.smithi111.stdout:5/826: dread d1/d3/d6/d30/db2/ff4 [0,4194304] 0 2024-03-23T11:47:44.852 INFO:tasks.workunit.client.1.smithi111.stdout:3/721: creat d0/d72/ff2 x:0 0 0 2024-03-23T11:47:44.853 INFO:tasks.workunit.client.1.smithi111.stdout:1/763: chown d0/d6/f34 19 1 2024-03-23T11:47:44.855 INFO:tasks.workunit.client.1.smithi111.stdout:2/815: getdents d2/de/d6c 0 2024-03-23T11:47:44.860 INFO:tasks.workunit.client.1.smithi111.stdout:1/764: write d0/d6/d48/d4a/fd0 [715729,1908] 0 2024-03-23T11:47:44.861 INFO:tasks.workunit.client.1.smithi111.stdout:0/750: dwrite d5/db/d1f/fc0 [0,4194304] 0 2024-03-23T11:47:44.862 INFO:tasks.workunit.client.1.smithi111.stdout:3/722: write d0/d1a/d2b/d38/d7f/f9c [2058500,78118] 0 2024-03-23T11:47:44.862 INFO:tasks.workunit.client.1.smithi111.stdout:5/827: symlink d1/d3/d6/d7/dee/l111 0 2024-03-23T11:47:44.863 INFO:tasks.workunit.client.1.smithi111.stdout:2/816: mkdir d2/d5c/d126 0 2024-03-23T11:47:44.875 INFO:tasks.workunit.client.1.smithi111.stdout:2/817: stat d2/de/d12/f107 0 2024-03-23T11:47:44.875 INFO:tasks.workunit.client.1.smithi111.stdout:9/842: dread d0/d4/fe [0,4194304] 0 2024-03-23T11:47:44.877 INFO:tasks.workunit.client.1.smithi111.stdout:7/781: truncate d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb9 456796 0 2024-03-23T11:47:44.882 INFO:tasks.workunit.client.1.smithi111.stdout:4/809: dwrite d0/d95/d9b/f107 [0,4194304] 0 2024-03-23T11:47:44.884 INFO:tasks.workunit.client.1.smithi111.stdout:5/828: rename d1/d3/d3a/l6d to d1/d3/d6/d8b/d59/d6c/l112 0 2024-03-23T11:47:44.885 INFO:tasks.workunit.client.1.smithi111.stdout:0/751: getdents d5/db/d1f/d19/dbd/d8b 0 2024-03-23T11:47:44.889 INFO:tasks.workunit.client.1.smithi111.stdout:1/765: write d0/d2b/f45 [2482762,30786] 0 2024-03-23T11:47:44.891 INFO:tasks.workunit.client.1.smithi111.stdout:2/818: creat d2/d7/da/d118/f127 x:0 0 0 2024-03-23T11:47:44.893 INFO:tasks.workunit.client.1.smithi111.stdout:9/843: rename d0/d4/d1c/fe3 to d0/d4/d1c/d2d/d75/d7a/f110 0 2024-03-23T11:47:44.893 INFO:tasks.workunit.client.1.smithi111.stdout:6/778: dread d2/d1f/d45/d49/f6d [0,4194304] 0 2024-03-23T11:47:44.895 INFO:tasks.workunit.client.1.smithi111.stdout:5/829: dread - d1/d3/d6/d7/fc6 zero size 2024-03-23T11:47:44.899 INFO:tasks.workunit.client.1.smithi111.stdout:7/782: truncate d14/d1b/d1e/d2c/d2e/d33/f3d 844829 0 2024-03-23T11:47:44.899 INFO:tasks.workunit.client.1.smithi111.stdout:0/752: symlink d5/db/d1f/d19/dbd/d64/d91/ddd/df6/lfd 0 2024-03-23T11:47:44.900 INFO:tasks.workunit.client.1.smithi111.stdout:9/844: dread - d0/d4/d1c/d2d/d6e/ff1 zero size 2024-03-23T11:47:44.901 INFO:tasks.workunit.client.1.smithi111.stdout:2/819: mkdir d2/d128 0 2024-03-23T11:47:44.902 INFO:tasks.workunit.client.1.smithi111.stdout:6/779: mknod d2/d80/cfc 0 2024-03-23T11:47:44.905 INFO:tasks.workunit.client.1.smithi111.stdout:5/830: fsync d1/d3/fb0 0 2024-03-23T11:47:44.905 INFO:tasks.workunit.client.1.smithi111.stdout:3/723: dwrite d0/d7/f23 [0,4194304] 0 2024-03-23T11:47:44.906 INFO:tasks.workunit.client.1.smithi111.stdout:7/783: dread - d14/d1b/d1e/d2c/d2e/d73/fce zero size 2024-03-23T11:47:44.911 INFO:tasks.workunit.client.1.smithi111.stdout:3/724: creat d0/d7/d19/d5e/da6/ff3 x:0 0 0 2024-03-23T11:47:44.914 INFO:tasks.workunit.client.1.smithi111.stdout:0/753: mknod d5/db/d1f/d19/dbd/d64/d91/cfe 0 2024-03-23T11:47:44.916 INFO:tasks.workunit.client.1.smithi111.stdout:5/831: rmdir d1/d20/d5c/d99 39 2024-03-23T11:47:44.918 INFO:tasks.workunit.client.1.smithi111.stdout:6/780: dread d2/d1f/d45/d49/d8d/f94 [0,4194304] 0 2024-03-23T11:47:44.918 INFO:tasks.workunit.client.1.smithi111.stdout:1/766: rename d0/d5/dd/d1e/d70/cc9 to d0/d2b/d37/d6a/d100/c103 0 2024-03-23T11:47:44.921 INFO:tasks.workunit.client.1.smithi111.stdout:2/820: truncate d2/d7/da/dbd/fd8 1959053 0 2024-03-23T11:47:44.925 INFO:tasks.workunit.client.1.smithi111.stdout:5/832: mknod d1/d3/d6/d8b/dc2/c113 0 2024-03-23T11:47:44.926 INFO:tasks.workunit.client.1.smithi111.stdout:7/784: rename d14/d44/d70/d7b/fd5 to d14/d44/d70/fe5 0 2024-03-23T11:47:44.928 INFO:tasks.workunit.client.1.smithi111.stdout:6/781: fdatasync d2/d1a/d1c/d2c/d51/f8f 0 2024-03-23T11:47:44.931 INFO:tasks.workunit.client.1.smithi111.stdout:5/833: write d1/d3/d6/d7/f7c [747230,113355] 0 2024-03-23T11:47:44.933 INFO:tasks.workunit.client.1.smithi111.stdout:1/767: truncate d0/d2b/d37/d6a/f8d 2784167 0 2024-03-23T11:47:44.934 INFO:tasks.workunit.client.1.smithi111.stdout:7/785: mknod d14/d1b/d1e/d87/ce6 0 2024-03-23T11:47:44.935 INFO:tasks.workunit.client.1.smithi111.stdout:6/782: mknod d2/d1f/dce/cfd 0 2024-03-23T11:47:44.936 INFO:tasks.workunit.client.1.smithi111.stdout:2/821: getdents d2/de/d6c/dd1/dd6 0 2024-03-23T11:47:44.938 INFO:tasks.workunit.client.1.smithi111.stdout:7/786: stat d14/d1b/d1e/d2c/d2e/ce0 0 2024-03-23T11:47:44.938 INFO:tasks.workunit.client.1.smithi111.stdout:6/783: stat d2/d17/fac 0 2024-03-23T11:47:44.939 INFO:tasks.workunit.client.1.smithi111.stdout:9/845: write d0/d4/d1c/d36/d28/f5d [489545,9684] 0 2024-03-23T11:47:44.941 INFO:tasks.workunit.client.1.smithi111.stdout:5/834: creat d1/d3/f114 x:0 0 0 2024-03-23T11:47:44.946 INFO:tasks.workunit.client.1.smithi111.stdout:2/822: getdents d2/d11/d17/d109 0 2024-03-23T11:47:44.946 INFO:tasks.workunit.client.1.smithi111.stdout:3/725: dwrite d0/d1a/d1d/d26/d5a/d64/fba [0,4194304] 0 2024-03-23T11:47:44.949 INFO:tasks.workunit.client.1.smithi111.stdout:9/846: write d0/d4/d10/d29/d7f/d8e/f9e [1923824,25357] 0 2024-03-23T11:47:44.949 INFO:tasks.workunit.client.1.smithi111.stdout:3/726: fsync d0/d1a/d1d/f96 0 2024-03-23T11:47:44.956 INFO:tasks.workunit.client.1.smithi111.stdout:2/823: truncate d2/d32/f7d 758798 0 2024-03-23T11:47:44.956 INFO:tasks.workunit.client.1.smithi111.stdout:6/784: dread d2/d35/fb5 [0,4194304] 0 2024-03-23T11:47:44.957 INFO:tasks.workunit.client.1.smithi111.stdout:3/727: creat d0/d7/d19/d91/ff4 x:0 0 0 2024-03-23T11:47:44.958 INFO:tasks.workunit.client.1.smithi111.stdout:0/754: dread d5/db/f26 [4194304,4194304] 0 2024-03-23T11:47:44.963 INFO:tasks.workunit.client.1.smithi111.stdout:3/728: rmdir d0/d1a/d2d 39 2024-03-23T11:47:44.967 INFO:tasks.workunit.client.1.smithi111.stdout:7/787: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/f94 [0,4194304] 0 2024-03-23T11:47:44.969 INFO:tasks.workunit.client.1.smithi111.stdout:3/729: fdatasync d0/d1a/d1d/fd7 0 2024-03-23T11:47:44.970 INFO:tasks.workunit.client.0.smithi063.stdout:8/893: sync 2024-03-23T11:47:44.974 INFO:tasks.workunit.client.1.smithi111.stdout:6/785: dread d2/d1f/d45/fbc [0,4194304] 0 2024-03-23T11:47:44.976 INFO:tasks.workunit.client.1.smithi111.stdout:9/847: write d0/d4/d1c/d2d/fb7 [256687,116267] 0 2024-03-23T11:47:44.978 INFO:tasks.workunit.client.1.smithi111.stdout:3/730: readlink d0/d1a/d1d/d26/d5a/d60/lbf 0 2024-03-23T11:47:44.979 INFO:tasks.workunit.client.1.smithi111.stdout:0/755: write d5/db/d1f/f79 [3736679,61967] 0 2024-03-23T11:47:44.980 INFO:tasks.workunit.client.1.smithi111.stdout:4/810: sync 2024-03-23T11:47:44.982 INFO:tasks.workunit.client.1.smithi111.stdout:2/824: dwrite d2/d32/ffd [0,4194304] 0 2024-03-23T11:47:44.985 INFO:tasks.workunit.client.1.smithi111.stdout:3/731: readlink d0/d7/d19/d91/da3/ld8 0 2024-03-23T11:47:44.985 INFO:tasks.workunit.client.1.smithi111.stdout:4/811: chown d0/d17/fc5 887002 1 2024-03-23T11:47:44.995 INFO:tasks.workunit.client.1.smithi111.stdout:3/732: mkdir d0/d1a/d2d/df5 0 2024-03-23T11:47:44.995 INFO:tasks.workunit.client.1.smithi111.stdout:5/835: dwrite d1/d3/d6/d8b/d59/d74/f5e [0,4194304] 0 2024-03-23T11:47:44.995 INFO:tasks.workunit.client.1.smithi111.stdout:0/756: dread - d5/db/d1f/d19/d45/f49 zero size 2024-03-23T11:47:44.997 INFO:tasks.workunit.client.1.smithi111.stdout:2/825: rmdir d2/de/d12/d70 39 2024-03-23T11:47:45.003 INFO:tasks.workunit.client.1.smithi111.stdout:6/786: getdents d2/d1a 0 2024-03-23T11:47:45.006 INFO:tasks.workunit.client.1.smithi111.stdout:4/812: mkdir d0/dc/d89/da4/d10a 0 2024-03-23T11:47:45.006 INFO:tasks.workunit.client.1.smithi111.stdout:2/826: stat d2/d32/d91/fb9 0 2024-03-23T11:47:45.010 INFO:tasks.workunit.client.1.smithi111.stdout:0/757: creat d5/db/d1f/d19/dbd/d64/df2/fff x:0 0 0 2024-03-23T11:47:45.011 INFO:tasks.workunit.client.1.smithi111.stdout:6/787: readlink d2/d1f/d45/d49/lb3 0 2024-03-23T11:47:45.011 INFO:tasks.workunit.client.1.smithi111.stdout:9/848: dwrite d0/fe5 [0,4194304] 0 2024-03-23T11:47:45.013 INFO:tasks.workunit.client.1.smithi111.stdout:1/768: dread d0/f59 [0,4194304] 0 2024-03-23T11:47:45.023 INFO:tasks.workunit.client.1.smithi111.stdout:1/769: creat d0/d5/dd5/df6/f104 x:0 0 0 2024-03-23T11:47:45.025 INFO:tasks.workunit.client.1.smithi111.stdout:4/813: rename d0/d95/d9b/db3/dcc/dea/f92 to d0/dc/d4c/d83/f10b 0 2024-03-23T11:47:45.028 INFO:tasks.workunit.client.1.smithi111.stdout:7/788: dwrite d14/d1b/d1e/d2c/d2e/d33/f3d [0,4194304] 0 2024-03-23T11:47:45.028 INFO:tasks.workunit.client.1.smithi111.stdout:2/827: truncate d2/d11/d17/d3e/d6b/f10f 229654 0 2024-03-23T11:47:45.028 INFO:tasks.workunit.client.1.smithi111.stdout:9/849: creat d0/d31/d41/d9d/dd4/de0/df8/f111 x:0 0 0 2024-03-23T11:47:45.030 INFO:tasks.workunit.client.1.smithi111.stdout:1/770: fsync d0/d5/f9c 0 2024-03-23T11:47:45.032 INFO:tasks.workunit.client.1.smithi111.stdout:3/733: write d0/d7/f1e [2367288,121448] 0 2024-03-23T11:47:45.032 INFO:tasks.workunit.client.1.smithi111.stdout:6/788: dread d2/d1f/d45/d9a/fd4 [0,4194304] 0 2024-03-23T11:47:45.036 INFO:tasks.workunit.client.1.smithi111.stdout:5/836: dread d1/d3/d6/d7/d63/d2c/f51 [0,4194304] 0 2024-03-23T11:47:45.037 INFO:tasks.workunit.client.1.smithi111.stdout:3/734: chown d0/d1a/d1d/d26/ce3 1223460 1 2024-03-23T11:47:45.038 INFO:tasks.workunit.client.1.smithi111.stdout:9/850: chown d0/d4/d1c/d36/d28/de6 7 1 2024-03-23T11:47:45.042 INFO:tasks.workunit.client.1.smithi111.stdout:6/789: symlink d2/d35/de3/lfe 0 2024-03-23T11:47:45.044 INFO:tasks.workunit.client.1.smithi111.stdout:2/828: rename d2/d11/d17/d3e/d5d/da3 to d2/d32/d129 0 2024-03-23T11:47:45.045 INFO:tasks.workunit.client.1.smithi111.stdout:4/814: getdents d0/dc/d4c/dad 0 2024-03-23T11:47:45.047 INFO:tasks.workunit.client.1.smithi111.stdout:9/851: unlink d0/d31/fed 0 2024-03-23T11:47:45.048 INFO:tasks.workunit.client.1.smithi111.stdout:9/852: stat d0/d4/d1c/d36/f59 0 2024-03-23T11:47:45.048 INFO:tasks.workunit.client.1.smithi111.stdout:1/771: symlink d0/d5/dd/d1e/d54/l105 0 2024-03-23T11:47:45.052 INFO:tasks.workunit.client.1.smithi111.stdout:6/790: mknod d2/d1a/d1c/d2c/cff 0 2024-03-23T11:47:45.052 INFO:tasks.workunit.client.1.smithi111.stdout:6/791: stat d2/d1a/f22 0 2024-03-23T11:47:45.053 INFO:tasks.workunit.client.1.smithi111.stdout:6/792: fdatasync d2/f89 0 2024-03-23T11:47:45.059 INFO:tasks.workunit.client.1.smithi111.stdout:7/789: rename d14/d1b/d1e/d2c/d42/d5e/d88/c2a to d14/d1b/d1e/d2c/d2e/d73/dae/d74/ce7 0 2024-03-23T11:47:45.060 INFO:tasks.workunit.client.1.smithi111.stdout:4/815: rmdir d0/dc/d4c/d2a/d97 39 2024-03-23T11:47:45.061 INFO:tasks.workunit.client.1.smithi111.stdout:9/853: rmdir d0/d4/d1c/d2d/d3b 39 2024-03-23T11:47:45.061 INFO:tasks.workunit.client.1.smithi111.stdout:5/837: symlink d1/d3/d6/d8b/d59/d6c/df3/l115 0 2024-03-23T11:47:45.064 INFO:tasks.workunit.client.1.smithi111.stdout:7/790: truncate d14/d37/f49 5132055 0 2024-03-23T11:47:45.065 INFO:tasks.workunit.client.1.smithi111.stdout:0/758: dread d5/db/d1f/d32/f3e [0,4194304] 0 2024-03-23T11:47:45.065 INFO:tasks.workunit.client.1.smithi111.stdout:4/816: write d0/d95/db5/dc7/ffa [138920,16168] 0 2024-03-23T11:47:45.073 INFO:tasks.workunit.client.1.smithi111.stdout:3/735: dwrite d0/d1a/d2b/d38/d7f/dd0/fd3 [0,4194304] 0 2024-03-23T11:47:45.075 INFO:tasks.workunit.client.1.smithi111.stdout:3/736: chown d0/d1a/d2b/d38/d7f/dd0/fd3 0 1 2024-03-23T11:47:45.076 INFO:tasks.workunit.client.1.smithi111.stdout:2/829: dwrite d2/d7/da/d19/f33 [0,4194304] 0 2024-03-23T11:47:45.079 INFO:tasks.workunit.client.1.smithi111.stdout:2/830: chown d2/d11/d17/d3e/d6b/d9d/d52/dba/ce9 92585 1 2024-03-23T11:47:45.099 INFO:tasks.workunit.client.1.smithi111.stdout:1/772: rename d0/d5/dd/d1e/d54/d81 to d0/d2b/d7c/da9/d106 0 2024-03-23T11:47:45.099 INFO:tasks.workunit.client.1.smithi111.stdout:5/838: fdatasync d1/d20/dce/fff 0 2024-03-23T11:47:45.105 INFO:tasks.workunit.client.1.smithi111.stdout:3/737: write d0/d1a/d4f/d90/fce [1454902,59302] 0 2024-03-23T11:47:45.115 INFO:tasks.workunit.client.1.smithi111.stdout:4/817: write d0/dc/d4c/d2a/d97/d4d/f9c [834244,39900] 0 2024-03-23T11:47:45.119 INFO:tasks.workunit.client.1.smithi111.stdout:1/773: truncate d0/d2b/d37/d6a/f8d 1296431 0 2024-03-23T11:47:45.122 INFO:tasks.workunit.client.1.smithi111.stdout:9/854: getdents d0 0 2024-03-23T11:47:45.123 INFO:tasks.workunit.client.1.smithi111.stdout:4/818: creat d0/d95/db5/d108/d109/f10c x:0 0 0 2024-03-23T11:47:45.125 INFO:tasks.workunit.client.1.smithi111.stdout:1/774: rename d0/d5/dd/d98/ld7 to d0/d5/dd/d1e/d54/d87/dd9/l107 0 2024-03-23T11:47:45.126 INFO:tasks.workunit.client.1.smithi111.stdout:4/819: unlink d0/dc/l23 0 2024-03-23T11:47:45.129 INFO:tasks.workunit.client.1.smithi111.stdout:0/759: write d5/db/d1f/f1a [2260157,115828] 0 2024-03-23T11:47:45.129 INFO:tasks.workunit.client.1.smithi111.stdout:4/820: mkdir d0/d95/d10d 0 2024-03-23T11:47:45.130 INFO:tasks.workunit.client.1.smithi111.stdout:5/839: write d1/d3/d6/d8b/d59/f9f [2536408,116179] 0 2024-03-23T11:47:45.131 INFO:tasks.workunit.client.1.smithi111.stdout:1/775: mkdir d0/d5/dd/dfe/d108 0 2024-03-23T11:47:45.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:44 smithi063 ceph-mon[35189]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:45.135 INFO:tasks.workunit.client.1.smithi111.stdout:9/855: link d0/d4/d6a/da7/f10b d0/d4/f112 0 2024-03-23T11:47:45.136 INFO:tasks.workunit.client.1.smithi111.stdout:5/840: chown d1/d3/d6/d7/dee 31 1 2024-03-23T11:47:45.136 INFO:tasks.workunit.client.1.smithi111.stdout:2/831: write d2/d7/da/dbd/fd8 [34820,102873] 0 2024-03-23T11:47:45.137 INFO:tasks.workunit.client.1.smithi111.stdout:3/738: write d0/d7/f2a [102055,32669] 0 2024-03-23T11:47:45.137 INFO:tasks.workunit.client.1.smithi111.stdout:4/821: truncate d0/d95/db5/dc7/f104 211186 0 2024-03-23T11:47:45.140 INFO:tasks.workunit.client.1.smithi111.stdout:6/793: dwrite d2/d1a/d1c/d2c/d51/f8f [4194304,4194304] 0 2024-03-23T11:47:45.146 INFO:tasks.workunit.client.1.smithi111.stdout:5/841: fdatasync d1/d20/dce/fbb 0 2024-03-23T11:47:45.146 INFO:tasks.workunit.client.1.smithi111.stdout:9/856: stat d0/d31/d41/d44/fca 0 2024-03-23T11:47:45.147 INFO:tasks.workunit.client.1.smithi111.stdout:0/760: rename d5/db/d1f/d47/d59/db6 to d5/db/d1f/d19/dbd/d64/dc7/d100 0 2024-03-23T11:47:45.147 INFO:tasks.workunit.client.1.smithi111.stdout:4/822: fsync d0/dc/d89/da4/fd3 0 2024-03-23T11:47:45.149 INFO:tasks.workunit.client.1.smithi111.stdout:3/739: mknod d0/d1a/d1d/d26/d5a/d64/cf6 0 2024-03-23T11:47:45.150 INFO:tasks.workunit.client.1.smithi111.stdout:9/857: dread - d0/d4/d10/d91/fda zero size 2024-03-23T11:47:45.152 INFO:tasks.workunit.client.1.smithi111.stdout:2/832: creat d2/d11/d17/d3e/d6b/d2e/d76/f12a x:0 0 0 2024-03-23T11:47:45.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:44 smithi111 ceph-mon[44005]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:47:45.154 INFO:tasks.workunit.client.1.smithi111.stdout:5/842: stat d1/d3/d6/d8b/d59/ld5 0 2024-03-23T11:47:45.155 INFO:tasks.workunit.client.1.smithi111.stdout:6/794: mkdir d2/d80/d100 0 2024-03-23T11:47:45.157 INFO:tasks.workunit.client.1.smithi111.stdout:5/843: chown d1/d3/d3a/c84 1389 1 2024-03-23T11:47:45.158 INFO:tasks.workunit.client.1.smithi111.stdout:0/761: read - d5/db/d1f/d19/dbd/f62 zero size 2024-03-23T11:47:45.159 INFO:tasks.workunit.client.1.smithi111.stdout:4/823: creat d0/dc/d4c/d2a/d97/d4d/f10e x:0 0 0 2024-03-23T11:47:45.160 INFO:tasks.workunit.client.1.smithi111.stdout:3/740: creat d0/d7/d19/d5e/da6/ff7 x:0 0 0 2024-03-23T11:47:45.161 INFO:tasks.workunit.client.1.smithi111.stdout:9/858: symlink d0/d31/d41/d9d/db0/l113 0 2024-03-23T11:47:45.162 INFO:tasks.workunit.client.1.smithi111.stdout:1/776: dwrite d0/d5/dd/d1e/d54/f57 [0,4194304] 0 2024-03-23T11:47:45.163 INFO:tasks.workunit.client.1.smithi111.stdout:4/824: write d0/dc/d1f/f8a [3230883,98227] 0 2024-03-23T11:47:45.163 INFO:tasks.workunit.client.1.smithi111.stdout:2/833: creat d2/d11/d17/d3e/d6b/d2e/d76/f12b x:0 0 0 2024-03-23T11:47:45.170 INFO:tasks.workunit.client.1.smithi111.stdout:3/741: creat d0/d1a/d2b/d38/dd4/ff8 x:0 0 0 2024-03-23T11:47:45.170 INFO:tasks.workunit.client.1.smithi111.stdout:9/859: unlink d0/f48 0 2024-03-23T11:47:45.170 INFO:tasks.workunit.client.1.smithi111.stdout:0/762: mknod d5/db/d1f/d47/d59/c101 0 2024-03-23T11:47:45.175 INFO:tasks.workunit.client.1.smithi111.stdout:5/844: dread d1/d3/d6/d8b/f75 [0,4194304] 0 2024-03-23T11:47:45.176 INFO:tasks.workunit.client.1.smithi111.stdout:3/742: truncate d0/d1a/d2d/fbb 4867105 0 2024-03-23T11:47:45.177 INFO:tasks.workunit.client.1.smithi111.stdout:4/825: dread d0/d95/db5/dc7/ffa [0,4194304] 0 2024-03-23T11:47:45.179 INFO:tasks.workunit.client.1.smithi111.stdout:2/834: dread d2/d11/fe5 [0,4194304] 0 2024-03-23T11:47:45.179 INFO:tasks.workunit.client.1.smithi111.stdout:6/795: write d2/f3b [733100,94347] 0 2024-03-23T11:47:45.181 INFO:tasks.workunit.client.1.smithi111.stdout:9/860: getdents d0/d31/d41/d9d/dd4 0 2024-03-23T11:47:45.183 INFO:tasks.workunit.client.1.smithi111.stdout:5/845: chown d1/d3/d6/d7/d63/d2c/d82/d66/f97 905096606 1 2024-03-23T11:47:45.184 INFO:tasks.workunit.client.1.smithi111.stdout:0/763: write d5/db/d4d/fb2 [164063,40610] 0 2024-03-23T11:47:45.184 INFO:tasks.workunit.client.1.smithi111.stdout:6/796: rmdir d2/d35/de3 39 2024-03-23T11:47:45.185 INFO:tasks.workunit.client.1.smithi111.stdout:3/743: dread d0/d55/f97 [0,4194304] 0 2024-03-23T11:47:45.186 INFO:tasks.workunit.client.1.smithi111.stdout:4/826: symlink d0/dc/d4c/d2a/d97/d4d/de3/d105/l10f 0 2024-03-23T11:47:45.188 INFO:tasks.workunit.client.1.smithi111.stdout:2/835: write d2/d11/d17/d3e/d5d/dc2/fc4 [573997,73716] 0 2024-03-23T11:47:45.190 INFO:tasks.workunit.client.1.smithi111.stdout:5/846: mkdir d1/d20/d5c/dc5/dd3/d116 0 2024-03-23T11:47:45.194 INFO:tasks.workunit.client.1.smithi111.stdout:4/827: fdatasync d0/dc/d1f/db0/f102 0 2024-03-23T11:47:45.194 INFO:tasks.workunit.client.1.smithi111.stdout:6/797: unlink d2/d1a/d1c/d2c/f48 0 2024-03-23T11:47:45.198 INFO:tasks.workunit.client.1.smithi111.stdout:3/744: dread d0/d1a/d2b/d38/f99 [0,4194304] 0 2024-03-23T11:47:45.199 INFO:tasks.workunit.client.1.smithi111.stdout:0/764: truncate d5/db/d1f/d47/d6f/ffa 897610 0 2024-03-23T11:47:45.202 INFO:tasks.workunit.client.1.smithi111.stdout:5/847: dread d1/d20/d5c/dc5/dd3/fed [0,4194304] 0 2024-03-23T11:47:45.202 INFO:tasks.workunit.client.1.smithi111.stdout:1/777: dwrite d0/d5/dd/d1e/f21 [0,4194304] 0 2024-03-23T11:47:45.205 INFO:tasks.workunit.client.1.smithi111.stdout:7/791: sync 2024-03-23T11:47:45.209 INFO:tasks.workunit.client.1.smithi111.stdout:3/745: creat d0/d7/d19/d5e/da6/deb/ff9 x:0 0 0 2024-03-23T11:47:45.210 INFO:tasks.workunit.client.1.smithi111.stdout:9/861: dwrite d0/d4/d10/d80/fd8 [0,4194304] 0 2024-03-23T11:47:45.211 INFO:tasks.workunit.client.1.smithi111.stdout:1/778: symlink d0/d2b/d7c/da9/l109 0 2024-03-23T11:47:45.211 INFO:tasks.workunit.client.1.smithi111.stdout:7/792: write d14/d44/d70/f7c [1180292,18775] 0 2024-03-23T11:47:45.213 INFO:tasks.workunit.client.1.smithi111.stdout:0/765: symlink d5/db/d1f/d19/d45/l102 0 2024-03-23T11:47:45.216 INFO:tasks.workunit.client.1.smithi111.stdout:1/779: dread - d0/d5/dd/d1e/d54/d87/d8e/fe6 zero size 2024-03-23T11:47:45.216 INFO:tasks.workunit.client.1.smithi111.stdout:2/836: dwrite d2/de/d12/f2c [0,4194304] 0 2024-03-23T11:47:45.217 INFO:tasks.workunit.client.1.smithi111.stdout:6/798: rename d2/d80/f96 to d2/d35/f101 0 2024-03-23T11:47:45.219 INFO:tasks.workunit.client.1.smithi111.stdout:0/766: read d5/db/d1f/d47/d6f/ffa [788519,41100] 0 2024-03-23T11:47:45.221 INFO:tasks.workunit.client.1.smithi111.stdout:0/767: write d5/db/d1f/de4/da7/fc9 [622033,72493] 0 2024-03-23T11:47:45.223 INFO:tasks.workunit.client.1.smithi111.stdout:4/828: write d0/d95/d9b/db3/dcc/dea/da/f13 [548531,69769] 0 2024-03-23T11:47:45.224 INFO:tasks.workunit.client.1.smithi111.stdout:5/848: creat d1/d3/d6/d30/de6/f117 x:0 0 0 2024-03-23T11:47:45.225 INFO:tasks.workunit.client.1.smithi111.stdout:3/746: write d0/d1a/d2b/fca [180029,17105] 0 2024-03-23T11:47:45.233 INFO:tasks.workunit.client.1.smithi111.stdout:1/780: rename d0/d5/dd/d1e/d54/d87/d8e/fcd to d0/d2b/d69/f10a 0 2024-03-23T11:47:45.237 INFO:tasks.workunit.client.1.smithi111.stdout:7/793: unlink d14/d1b/d1e/d2c/d2e/d73/l7a 0 2024-03-23T11:47:45.239 INFO:tasks.workunit.client.1.smithi111.stdout:0/768: unlink d5/db/d1f/d48/c9f 0 2024-03-23T11:47:45.241 INFO:tasks.workunit.client.1.smithi111.stdout:0/769: stat d5/db/d1f/d48/f80 0 2024-03-23T11:47:45.242 INFO:tasks.workunit.client.1.smithi111.stdout:3/747: mkdir d0/d53/dfa 0 2024-03-23T11:47:45.243 INFO:tasks.workunit.client.1.smithi111.stdout:3/748: chown d0/l11 0 1 2024-03-23T11:47:45.244 INFO:tasks.workunit.client.1.smithi111.stdout:2/837: write d2/de/d12/d70/d96/ff5 [382315,48015] 0 2024-03-23T11:47:45.246 INFO:tasks.workunit.client.1.smithi111.stdout:1/781: rename d0/d5/f9c to d0/d2b/d69/f10b 0 2024-03-23T11:47:45.246 INFO:tasks.workunit.client.1.smithi111.stdout:2/838: readlink d2/d5c/l7a 0 2024-03-23T11:47:45.249 INFO:tasks.workunit.client.1.smithi111.stdout:0/770: creat d5/dc3/f103 x:0 0 0 2024-03-23T11:47:45.249 INFO:tasks.workunit.client.1.smithi111.stdout:7/794: chown d14/d1b/d1e/d2c/d42/c6b 12334 1 2024-03-23T11:47:45.251 INFO:tasks.workunit.client.1.smithi111.stdout:1/782: readlink d0/d6/d48/d4d/l61 0 2024-03-23T11:47:45.253 INFO:tasks.workunit.client.1.smithi111.stdout:3/749: creat d0/db9/ffb x:0 0 0 2024-03-23T11:47:45.255 INFO:tasks.workunit.client.1.smithi111.stdout:7/795: mknod d14/d1b/d22/ce8 0 2024-03-23T11:47:45.257 INFO:tasks.workunit.client.1.smithi111.stdout:3/750: stat d0/d53/faf 0 2024-03-23T11:47:45.258 INFO:tasks.workunit.client.1.smithi111.stdout:0/771: mknod d5/db/d1f/de4/da7/c104 0 2024-03-23T11:47:45.259 INFO:tasks.workunit.client.1.smithi111.stdout:9/862: dwrite d0/d4/d10/d80/fb4 [0,4194304] 0 2024-03-23T11:47:45.260 INFO:tasks.workunit.client.1.smithi111.stdout:1/783: mknod d0/d5/dd/d1e/c10c 0 2024-03-23T11:47:45.264 INFO:tasks.workunit.client.1.smithi111.stdout:7/796: dread d14/fba [0,4194304] 0 2024-03-23T11:47:45.267 INFO:tasks.workunit.client.0.smithi063.stdout:2/528: creat d2/de/d7a/da2/fab x:0 0 0 2024-03-23T11:47:45.268 INFO:tasks.workunit.client.1.smithi111.stdout:5/849: write d1/d3/d6/d7/f1f [1973944,45517] 0 2024-03-23T11:47:45.269 INFO:tasks.workunit.client.1.smithi111.stdout:0/772: creat d5/db/d1f/d19/f105 x:0 0 0 2024-03-23T11:47:45.273 INFO:tasks.workunit.client.1.smithi111.stdout:6/799: dread d2/f57 [0,4194304] 0 2024-03-23T11:47:45.275 INFO:tasks.workunit.client.1.smithi111.stdout:2/839: write d2/d11/d17/d3e/d6b/d2e/f82 [822704,3590] 0 2024-03-23T11:47:45.281 INFO:tasks.workunit.client.1.smithi111.stdout:4/829: dread d0/dc/f55 [0,4194304] 0 2024-03-23T11:47:45.283 INFO:tasks.workunit.client.1.smithi111.stdout:6/800: dread - d2/d35/d70/d98/fa4 zero size 2024-03-23T11:47:45.285 INFO:tasks.workunit.client.1.smithi111.stdout:3/751: getdents d0/d1a/d2b 0 2024-03-23T11:47:45.287 INFO:tasks.workunit.client.0.smithi063.stdout:3/913: creat d13/d22/d2d/d3f/dd4/f122 x:0 0 0 2024-03-23T11:47:45.290 INFO:tasks.workunit.client.1.smithi111.stdout:0/773: mkdir d5/db/d1f/d47/d106 0 2024-03-23T11:47:45.290 INFO:tasks.workunit.client.1.smithi111.stdout:2/840: write d2/de/d6c/dd1/fe3 [797199,86763] 0 2024-03-23T11:47:45.290 INFO:tasks.workunit.client.1.smithi111.stdout:1/784: rename d0/d6/ca to d0/d5/dd/d1e/d54/c10d 0 2024-03-23T11:47:45.292 INFO:tasks.workunit.client.1.smithi111.stdout:1/785: fdatasync d0/ff1 0 2024-03-23T11:47:45.295 INFO:tasks.workunit.client.1.smithi111.stdout:9/863: getdents d0/d4/d1c 0 2024-03-23T11:47:45.296 INFO:tasks.workunit.client.0.smithi063.stdout:5/437: symlink d2/d7/d8/d1d/l96 0 2024-03-23T11:47:45.298 INFO:tasks.workunit.client.1.smithi111.stdout:5/850: dwrite d1/d20/f72 [0,4194304] 0 2024-03-23T11:47:45.306 INFO:tasks.workunit.client.1.smithi111.stdout:9/864: truncate d0/d31/d35/f109 972892 0 2024-03-23T11:47:45.306 INFO:tasks.workunit.client.1.smithi111.stdout:3/752: creat d0/d1a/d1d/dc5/ffc x:0 0 0 2024-03-23T11:47:45.310 INFO:tasks.workunit.client.1.smithi111.stdout:7/797: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/d5f/f68 [0,4194304] 0 2024-03-23T11:47:45.311 INFO:tasks.workunit.client.0.smithi063.stdout:1/537: mknod d3/df/d2f/d9a/ca9 0 2024-03-23T11:47:45.314 INFO:tasks.workunit.client.1.smithi111.stdout:0/774: mknod d5/db/d1f/c107 0 2024-03-23T11:47:45.315 INFO:tasks.workunit.client.0.smithi063.stdout:1/538: write d3/f39 [1783499,31325] 0 2024-03-23T11:47:45.315 INFO:tasks.workunit.client.1.smithi111.stdout:2/841: creat d2/de/d6c/dd1/dd6/f12c x:0 0 0 2024-03-23T11:47:45.316 INFO:tasks.workunit.client.1.smithi111.stdout:1/786: creat d0/d5/dd/d1e/d54/d87/dd9/f10e x:0 0 0 2024-03-23T11:47:45.316 INFO:tasks.workunit.client.1.smithi111.stdout:6/801: link d2/d35/d70/l84 d2/d35/da7/l102 0 2024-03-23T11:47:45.318 INFO:tasks.workunit.client.0.smithi063.stdout:5/438: rmdir d2/d7/d2b 39 2024-03-23T11:47:45.320 INFO:tasks.workunit.client.0.smithi063.stdout:7/589: getdents d5 0 2024-03-23T11:47:45.321 INFO:tasks.workunit.client.1.smithi111.stdout:3/753: creat d0/d7/d19/d91/da3/ffd x:0 0 0 2024-03-23T11:47:45.321 INFO:tasks.workunit.client.0.smithi063.stdout:3/914: write d13/d15/d7e/fd2 [492597,1392] 0 2024-03-23T11:47:45.322 INFO:tasks.workunit.client.1.smithi111.stdout:4/830: getdents d0/dc/d4c/d2a/df2 0 2024-03-23T11:47:45.324 INFO:tasks.workunit.client.1.smithi111.stdout:0/775: truncate d5/db/f26 287409 0 2024-03-23T11:47:45.324 INFO:tasks.workunit.client.1.smithi111.stdout:9/865: mkdir d0/d4/d1c/d2d/d3b/d42/d92/d114 0 2024-03-23T11:47:45.325 INFO:tasks.workunit.client.0.smithi063.stdout:3/915: chown d13/d22/d2d/d3f/dd4/c108 38470 1 2024-03-23T11:47:45.326 INFO:tasks.workunit.client.1.smithi111.stdout:2/842: read d2/d7/f56 [3506739,33966] 0 2024-03-23T11:47:45.327 INFO:tasks.workunit.client.1.smithi111.stdout:7/798: symlink d14/d37/d43/db1/le9 0 2024-03-23T11:47:45.328 INFO:tasks.workunit.client.1.smithi111.stdout:3/754: read d0/d1a/d1d/d26/d5a/f8e [1818398,109722] 0 2024-03-23T11:47:45.330 INFO:tasks.workunit.client.1.smithi111.stdout:5/851: creat d1/d20/f118 x:0 0 0 2024-03-23T11:47:45.330 INFO:tasks.workunit.client.1.smithi111.stdout:4/831: readlink d0/dc/d4c/d2a/d97/d4f/dba/ld0 0 2024-03-23T11:47:45.330 INFO:tasks.workunit.client.0.smithi063.stdout:7/590: write d5/d26/f72 [1672416,102002] 0 2024-03-23T11:47:45.331 INFO:tasks.workunit.client.1.smithi111.stdout:6/802: truncate d2/d35/da7/fd6 2575952 0 2024-03-23T11:47:45.334 INFO:tasks.workunit.client.1.smithi111.stdout:2/843: stat d2/d5c/f90 0 2024-03-23T11:47:45.334 INFO:tasks.workunit.client.1.smithi111.stdout:0/776: symlink d5/db/d1f/d47/d87/ddf/l108 0 2024-03-23T11:47:45.336 INFO:tasks.workunit.client.1.smithi111.stdout:7/799: write d14/d1b/d1e/d2c/d42/d5e/d88/f94 [3403502,5152] 0 2024-03-23T11:47:45.336 INFO:tasks.workunit.client.1.smithi111.stdout:0/777: chown d5/db/d1f/de4 132 1 2024-03-23T11:47:45.342 INFO:tasks.workunit.client.1.smithi111.stdout:6/803: symlink d2/d1f/d45/d9a/l103 0 2024-03-23T11:47:45.343 INFO:tasks.workunit.client.1.smithi111.stdout:0/778: write d5/fef [1005086,10988] 0 2024-03-23T11:47:45.344 INFO:tasks.workunit.client.0.smithi063.stdout:3/916: read d13/d22/d2d/d3f/f72 [2393655,32526] 0 2024-03-23T11:47:45.344 INFO:tasks.workunit.client.0.smithi063.stdout:5/439: getdents d2/d7/d2b 0 2024-03-23T11:47:45.345 INFO:tasks.workunit.client.1.smithi111.stdout:4/832: rmdir d0/d95/db5/dfb 0 2024-03-23T11:47:45.346 INFO:tasks.workunit.client.1.smithi111.stdout:5/852: symlink d1/d20/d5c/dc5/dd3/d116/l119 0 2024-03-23T11:47:45.348 INFO:tasks.workunit.client.1.smithi111.stdout:7/800: read d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb9 [417477,27945] 0 2024-03-23T11:47:45.349 INFO:tasks.workunit.client.0.smithi063.stdout:1/539: dwrite d3/d35/d40/f53 [0,4194304] 0 2024-03-23T11:47:45.349 INFO:tasks.workunit.client.0.smithi063.stdout:3/917: chown d13/d22/d2d/d3f/d58/d86/dad/fef 6 1 2024-03-23T11:47:45.350 INFO:tasks.workunit.client.0.smithi063.stdout:7/591: dread d5/d6/d19/d1f/f6d [0,4194304] 0 2024-03-23T11:47:45.353 INFO:tasks.workunit.client.1.smithi111.stdout:4/833: symlink d0/d95/db5/l110 0 2024-03-23T11:47:45.354 INFO:tasks.workunit.client.1.smithi111.stdout:5/853: truncate d1/d3/d6/d8b/d59/f9f 3355447 0 2024-03-23T11:47:45.357 INFO:tasks.workunit.client.1.smithi111.stdout:0/779: write d5/db/d1f/d32/f3e [3924174,78166] 0 2024-03-23T11:47:45.358 INFO:tasks.workunit.client.1.smithi111.stdout:9/866: dwrite d0/d4/d10/d29/d7f/d8e/f9e [0,4194304] 0 2024-03-23T11:47:45.362 INFO:tasks.workunit.client.1.smithi111.stdout:7/801: rmdir d14/d1b/d1e/d2c/d42/d5e/d8f 0 2024-03-23T11:47:45.362 INFO:tasks.workunit.client.1.smithi111.stdout:3/755: dread d0/d1a/d2b/d38/d7f/f9b [0,4194304] 0 2024-03-23T11:47:45.364 INFO:tasks.workunit.client.1.smithi111.stdout:4/834: creat d0/d95/d9b/db3/dcc/dea/da/f111 x:0 0 0 2024-03-23T11:47:45.366 INFO:tasks.workunit.client.1.smithi111.stdout:7/802: read - d14/d1b/d1e/d2c/fd8 zero size 2024-03-23T11:47:45.367 INFO:tasks.workunit.client.1.smithi111.stdout:0/780: dread d5/db/d1f/d47/d6f/fe7 [0,4194304] 0 2024-03-23T11:47:45.367 INFO:tasks.workunit.client.0.smithi063.stdout:5/440: dwrite d2/d7/f14 [0,4194304] 0 2024-03-23T11:47:45.372 INFO:tasks.workunit.client.1.smithi111.stdout:5/854: rename d1/d3/d6/d8b/d59/d74/f4b to d1/d20/d5c/dc5/dd3/f11a 0 2024-03-23T11:47:45.373 INFO:tasks.workunit.client.1.smithi111.stdout:3/756: truncate d0/d1a/d2d/f45 312735 0 2024-03-23T11:47:45.374 INFO:tasks.workunit.client.1.smithi111.stdout:3/757: fdatasync d0/d7/f1e 0 2024-03-23T11:47:45.385 INFO:tasks.workunit.client.1.smithi111.stdout:4/835: unlink d0/dc/d1f/f4a 0 2024-03-23T11:47:45.385 INFO:tasks.workunit.client.1.smithi111.stdout:7/803: dwrite d14/d44/d70/f7c [0,4194304] 0 2024-03-23T11:47:45.388 INFO:tasks.workunit.client.1.smithi111.stdout:0/781: dwrite d5/db/d1f/d19/dbd/d64/dc7/d100/fca [0,4194304] 0 2024-03-23T11:47:45.392 INFO:tasks.workunit.client.1.smithi111.stdout:5/855: dread - d1/d3/d6/d7/d63/d2c/d43/fad zero size 2024-03-23T11:47:45.393 INFO:tasks.workunit.client.1.smithi111.stdout:3/758: creat d0/d1a/d2b/d38/dd4/ffe x:0 0 0 2024-03-23T11:47:45.395 INFO:tasks.workunit.client.1.smithi111.stdout:2/844: write d2/f54 [182087,127583] 0 2024-03-23T11:47:45.399 INFO:tasks.workunit.client.1.smithi111.stdout:0/782: creat d5/deb/f109 x:0 0 0 2024-03-23T11:47:45.399 INFO:tasks.workunit.client.1.smithi111.stdout:5/856: read d1/d20/f60 [144520,96821] 0 2024-03-23T11:47:45.404 INFO:tasks.workunit.client.1.smithi111.stdout:4/836: dwrite d0/dc/d4c/f82 [0,4194304] 0 2024-03-23T11:47:45.404 INFO:tasks.workunit.client.1.smithi111.stdout:3/759: creat d0/d1a/d4f/d90/fff x:0 0 0 2024-03-23T11:47:45.412 INFO:tasks.workunit.client.1.smithi111.stdout:6/804: dwrite d2/d1f/d45/d49/f6c [0,4194304] 0 2024-03-23T11:47:45.417 INFO:tasks.workunit.client.1.smithi111.stdout:3/760: rmdir d0/d1a/d2b/d38/d7f/dd0 39 2024-03-23T11:47:45.418 INFO:tasks.workunit.client.1.smithi111.stdout:4/837: symlink d0/d95/d9b/db3/dcc/dea/da/l112 0 2024-03-23T11:47:45.421 INFO:tasks.workunit.client.1.smithi111.stdout:3/761: truncate d0/d1a/d2b/ff0 308457 0 2024-03-23T11:47:45.421 INFO:tasks.workunit.client.1.smithi111.stdout:6/805: read - d2/d17/fc7 zero size 2024-03-23T11:47:45.422 INFO:tasks.workunit.client.1.smithi111.stdout:3/762: chown d0/d7/f23 582966 1 2024-03-23T11:47:45.432 INFO:tasks.workunit.client.1.smithi111.stdout:4/838: stat d0/l11 0 2024-03-23T11:47:45.433 INFO:tasks.workunit.client.1.smithi111.stdout:4/839: dread - d0/dc/d89/da4/fcf zero size 2024-03-23T11:47:45.434 INFO:tasks.workunit.client.1.smithi111.stdout:3/763: creat d0/d1a/d1d/d26/d5a/d64/f100 x:0 0 0 2024-03-23T11:47:45.434 INFO:tasks.workunit.client.1.smithi111.stdout:4/840: chown d0/d95/f9d 2850202 1 2024-03-23T11:47:45.436 INFO:tasks.workunit.client.1.smithi111.stdout:4/841: fsync d0/d95/fa9 0 2024-03-23T11:47:45.438 INFO:tasks.workunit.client.1.smithi111.stdout:5/857: rename d1/d3/d6/d7/d63/d2c/d43/lb8 to d1/d3/d6/d30/l11b 0 2024-03-23T11:47:45.443 INFO:tasks.workunit.client.1.smithi111.stdout:2/845: getdents d2/d11/d17/d3e/d6b/d9d/d52 0 2024-03-23T11:47:45.444 INFO:tasks.workunit.client.1.smithi111.stdout:6/806: mknod d2/d1a/d1c/ddf/c104 0 2024-03-23T11:47:45.444 INFO:tasks.workunit.client.1.smithi111.stdout:0/783: dwrite d5/db/d1f/de4/f61 [0,4194304] 0 2024-03-23T11:47:45.450 INFO:tasks.workunit.client.1.smithi111.stdout:5/858: unlink d1/d3/d6/d7/dee/ffa 0 2024-03-23T11:47:45.450 INFO:tasks.workunit.client.1.smithi111.stdout:3/764: mknod d0/d7/d19/d91/da3/dc8/c101 0 2024-03-23T11:47:45.459 INFO:tasks.workunit.client.1.smithi111.stdout:3/765: mkdir d0/d1a/d1d/d26/d5a/d60/d102 0 2024-03-23T11:47:45.460 INFO:tasks.workunit.client.1.smithi111.stdout:4/842: getdents d0/dc/d4c/d2a/df2 0 2024-03-23T11:47:45.461 INFO:tasks.workunit.client.1.smithi111.stdout:3/766: readlink d0/d1a/d1d/lb4 0 2024-03-23T11:47:45.463 INFO:tasks.workunit.client.1.smithi111.stdout:0/784: link d5/f30 d5/db/d1f/d19/dbd/d64/d91/ddd/df6/f10a 0 2024-03-23T11:47:45.463 INFO:tasks.workunit.client.1.smithi111.stdout:4/843: chown d0/d95/d9b/db3/dcc/dea/da/l2b 1304715026 1 2024-03-23T11:47:45.465 INFO:tasks.workunit.client.1.smithi111.stdout:0/785: dread - d5/db/d1f/de4/fd4 zero size 2024-03-23T11:47:45.474 INFO:tasks.workunit.client.1.smithi111.stdout:3/767: getdents d0/d7 0 2024-03-23T11:47:45.475 INFO:tasks.workunit.client.1.smithi111.stdout:4/844: write d0/dc/fc8 [546552,12778] 0 2024-03-23T11:47:45.475 INFO:tasks.workunit.client.1.smithi111.stdout:2/846: dwrite d2/de/d6c/f65 [0,4194304] 0 2024-03-23T11:47:45.480 INFO:tasks.workunit.client.1.smithi111.stdout:3/768: rmdir d0/d1a/d2b/d38/d7f 39 2024-03-23T11:47:45.481 INFO:tasks.workunit.client.1.smithi111.stdout:3/769: stat d0/d7/d19/f29 0 2024-03-23T11:47:45.482 INFO:tasks.workunit.client.1.smithi111.stdout:4/845: mkdir d0/dc/d113 0 2024-03-23T11:47:45.484 INFO:tasks.workunit.client.1.smithi111.stdout:4/846: dread - d0/d17/fde zero size 2024-03-23T11:47:45.485 INFO:tasks.workunit.client.1.smithi111.stdout:5/859: dread d1/d20/f37 [0,4194304] 0 2024-03-23T11:47:45.487 INFO:tasks.workunit.client.1.smithi111.stdout:5/860: dread - d1/d3/d6/d7/fc6 zero size 2024-03-23T11:47:45.488 INFO:tasks.workunit.client.1.smithi111.stdout:0/786: dwrite d5/db/d1f/d19/dbd/d64/dc7/d100/fca [0,4194304] 0 2024-03-23T11:47:45.491 INFO:tasks.workunit.client.1.smithi111.stdout:3/770: fsync d0/d1a/d2d/f45 0 2024-03-23T11:47:45.492 INFO:tasks.workunit.client.1.smithi111.stdout:1/787: sync 2024-03-23T11:47:45.496 INFO:tasks.workunit.client.1.smithi111.stdout:3/771: fdatasync d0/d1a/d4f/d90/f9f 0 2024-03-23T11:47:45.496 INFO:tasks.workunit.client.1.smithi111.stdout:5/861: creat d1/d20/d5c/dc5/f11c x:0 0 0 2024-03-23T11:47:45.497 INFO:tasks.workunit.client.0.smithi063.stdout:1/540: unlink d3/d49/d56/f99 0 2024-03-23T11:47:45.498 INFO:tasks.workunit.client.1.smithi111.stdout:0/787: chown d5/db/d1f/de4/lea 18758 1 2024-03-23T11:47:45.499 INFO:tasks.workunit.client.1.smithi111.stdout:3/772: dread - d0/d1a/d1d/fd7 zero size 2024-03-23T11:47:45.499 INFO:tasks.workunit.client.0.smithi063.stdout:5/441: dread - d2/d28/f4b zero size 2024-03-23T11:47:45.502 INFO:tasks.workunit.client.1.smithi111.stdout:1/788: rename d0/d6/l67 to d0/d2b/d7c/da9/d106/l10f 0 2024-03-23T11:47:45.503 INFO:tasks.workunit.client.1.smithi111.stdout:0/788: mkdir d5/db/d4d/da6/d10b 0 2024-03-23T11:47:45.505 INFO:tasks.workunit.client.1.smithi111.stdout:0/789: write d5/db/d1f/d19/f105 [481158,1286] 0 2024-03-23T11:47:45.597 INFO:tasks.workunit.client.1.smithi111.stdout:9/867: sync 2024-03-23T11:47:45.646 INFO:tasks.workunit.client.1.smithi111.stdout:7/804: sync 2024-03-23T11:47:45.646 INFO:tasks.workunit.client.1.smithi111.stdout:6/807: sync 2024-03-23T11:47:45.646 INFO:tasks.workunit.client.1.smithi111.stdout:2/847: sync 2024-03-23T11:47:45.687 INFO:tasks.workunit.client.1.smithi111.stdout:2/848: dread d2/d11/d17/d3e/d6b/d9d/fae [0,4194304] 0 2024-03-23T11:47:45.689 INFO:tasks.workunit.client.1.smithi111.stdout:2/849: stat d2/de/d12/f107 0 2024-03-23T11:47:45.973 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:45 smithi063 ceph-mon[35189]: pgmap v42: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 98 MiB/s rd, 243 MiB/s wr, 494 op/s 2024-03-23T11:47:46.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:45 smithi111 ceph-mon[44005]: pgmap v42: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 98 MiB/s rd, 243 MiB/s wr, 494 op/s 2024-03-23T11:47:46.797 INFO:tasks.workunit.client.1.smithi111.stdout:4/847: dwrite d0/dc/d4c/d2a/f70 [0,4194304] 0 2024-03-23T11:47:46.807 INFO:tasks.workunit.client.1.smithi111.stdout:9/868: readlink d0/d4/d1c/d2d/d75/lfc 0 2024-03-23T11:47:46.812 INFO:tasks.workunit.client.1.smithi111.stdout:6/808: dread - d2/d35/de3/fca zero size 2024-03-23T11:47:46.817 INFO:tasks.workunit.client.1.smithi111.stdout:7/805: mkdir d14/d1b/d1e/d2c/d2e/d73/dae/dea 0 2024-03-23T11:47:46.823 INFO:tasks.workunit.client.1.smithi111.stdout:2/850: write d2/d11/d17/d1f/fd9 [628460,128592] 0 2024-03-23T11:47:46.846 INFO:tasks.workunit.client.1.smithi111.stdout:0/790: unlink d5/db/d1f/d19/dbd/d8b/c8c 0 2024-03-23T11:47:46.847 INFO:tasks.workunit.client.1.smithi111.stdout:0/791: chown d5/db/l33 10628677 1 2024-03-23T11:47:46.877 INFO:tasks.workunit.client.0.smithi063.stdout:9/640: write d4/fba [1005371,125992] 0 2024-03-23T11:47:47.199 INFO:tasks.workunit.client.0.smithi063.stdout:2/529: write d2/d23/d56/f75 [378616,38621] 0 2024-03-23T11:47:47.199 INFO:tasks.workunit.client.0.smithi063.stdout:4/642: write d5/da/d20/d4b/d16/d50/f64 [2579697,85746] 0 2024-03-23T11:47:47.199 INFO:tasks.workunit.client.0.smithi063.stdout:6/500: write d1/d15/f26 [2420089,69949] 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:8/894: dwrite d6/d1c/f4e [0,4194304] 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:0/734: dwrite d2/d4/f30 [0,4194304] 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:3/918: dwrite d13/d22/d2d/d3f/d58/d6e/fa5 [0,4194304] 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:7/592: write d5/d5c/d2f/d73/da/f89 [3258934,55910] 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:0/735: chown d2/d14/d36/d77/d94/da6/d1f/f2f 454427 1 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:5/442: creat d2/d7/d2b/f97 x:0 0 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:9/641: creat d4/d31/d70/d8d/da0/fe2 x:0 0 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:4/643: creat d5/da/d20/d4b/d16/d32/d56/fbb x:0 0 0 2024-03-23T11:47:47.200 INFO:tasks.workunit.client.0.smithi063.stdout:8/895: mknod d6/d1f/d54/d63/d90/d15/c12a 0 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:7/593: symlink d5/d22/d57/lbc 0 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:1/541: write d3/f38 [316050,70214] 0 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:2/530: symlink d2/d23/lac 0 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:9/642: symlink d4/d31/d43/d49/d8e/le3 0 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:2/531: write d2/de/d7a/d8d/fa9 [99245,16178] 0 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:4/644: fdatasync d5/da/d72/fa8 0 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:9/643: chown d4/d31/d70/c77 137 1 2024-03-23T11:47:47.201 INFO:tasks.workunit.client.0.smithi063.stdout:2/532: fsync d2/d84/d94/fa6 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:8/896: getdents d6/d1f/d54/d63/d90/d5d/d7d/d9a/d126 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:7/594: symlink d5/d6/d19/d1f/d47/lbd 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:9/644: dwrite d4/d4c/fde [0,4194304] 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:5/443: rmdir d2/d7/d8/de/d47/d7c 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:4/645: creat d5/da/d20/d4b/d29/d38/d93/fbc x:0 0 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:1/542: unlink d3/l9b 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:5/444: readlink d2/d7/d8/d1d/l21 0 2024-03-23T11:47:47.202 INFO:tasks.workunit.client.0.smithi063.stdout:1/543: fsync d3/d49/d56/d65/d88/f9c 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:5/445: fsync d2/d7/d8/de/d2a/d3b/d43/f5a 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:9/645: creat d4/d31/d43/da9/da6/dbe/fe4 x:0 0 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:4/646: mkdir d5/d1e/d2b/dbd 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:4/647: stat d5/da/d20/d4b/d16/d32/d6f/c73 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:9/646: truncate d4/d4c/d44/f6a 5133158 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:8/897: chown d6/d1c/d27/f34 55 1 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:1/544: readlink d3/l6f 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:1/545: stat d3/df 0 2024-03-23T11:47:47.203 INFO:tasks.workunit.client.0.smithi063.stdout:9/647: mknod d4/d31/d70/d8d/db3/ce5 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:4/648: dread d5/da/d20/d4b/f68 [0,4194304] 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:1/546: creat d3/d1c/faa x:0 0 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:1/547: read d3/df/f7d [269943,69704] 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:8/898: mknod d6/dd/d118/c12b 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:4/649: fsync d5/f1c 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:9/648: dread d4/d4c/f81 [0,4194304] 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:4/650: stat d5/da/d20/d4b/d1f 0 2024-03-23T11:47:47.205 INFO:tasks.workunit.client.0.smithi063.stdout:6/501: write d1/d2a/f62 [1642293,94644] 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:4/651: write d5/da/d20/d4b/d16/d32/d56/fbb [584758,75023] 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:1/548: creat d3/d3f/d4d/d52/fab x:0 0 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:1/549: stat d3/d1c/d4b/l85 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:3/919: write d13/d22/d2d/d3f/d67/d7b/fab [3469438,77446] 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:4/652: truncate d5/da/d20/d4b/d16/d50/d8e/fa7 814207 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:0/736: dwrite d2/d14/d36/d77/d94/da6/d28/d46/d51/f82 [0,4194304] 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:3/920: write d13/d22/f2f [2387800,88744] 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:2/533: write d2/d23/f41 [988886,31123] 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:1/550: dwrite d3/d3f/d4d/f8c [0,4194304] 0 2024-03-23T11:47:47.206 INFO:tasks.workunit.client.0.smithi063.stdout:8/899: creat d6/d1c/f12c x:0 0 0 2024-03-23T11:47:47.209 INFO:tasks.workunit.client.0.smithi063.stdout:6/502: rmdir d1/d15/d84 0 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:3/773: mkdir d0/d53/dfa/d103 0 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:3/774: dread - d0/d1a/d2b/fa1 zero size 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:4/848: fdatasync d0/d95/d9b/db3/db6/fce 0 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:3/775: chown d0/d7/d19/d5e/l67 9 1 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:4/849: truncate d0/d95/db5/dc7/f104 1124587 0 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:9/869: mknod d0/d31/d41/d9d/dd4/c115 0 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:3/776: dread - d0/d1a/d1d/dc5/ffc zero size 2024-03-23T11:47:47.210 INFO:tasks.workunit.client.1.smithi111.stdout:4/850: chown d0/dc/d4c/d2a/c48 92308667 1 2024-03-23T11:47:47.211 INFO:tasks.workunit.client.1.smithi111.stdout:4/851: fsync d0/dc/d4c/f82 0 2024-03-23T11:47:47.211 INFO:tasks.workunit.client.1.smithi111.stdout:3/777: write d0/d7/d19/d5e/da6/ff3 [687745,39636] 0 2024-03-23T11:47:47.211 INFO:tasks.workunit.client.1.smithi111.stdout:5/862: link d1/d3/fcd d1/d20/d5c/dc5/dd3/d116/f11d 0 2024-03-23T11:47:47.211 INFO:tasks.workunit.client.1.smithi111.stdout:6/809: symlink d2/d1f/d45/d49/l105 0 2024-03-23T11:47:47.211 INFO:tasks.workunit.client.1.smithi111.stdout:3/778: write d0/d1a/d1d/f96 [1412118,121671] 0 2024-03-23T11:47:47.211 INFO:tasks.workunit.client.1.smithi111.stdout:7/806: mkdir d14/d1b/d1e/d2c/d42/dc1/deb 0 2024-03-23T11:47:47.211 INFO:tasks.workunit.client.1.smithi111.stdout:1/789: getdents d0/d2b/d7c 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:4/852: unlink d0/dc/d78/fc0 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:9/870: dwrite d0/d4/d6a/da7/fd5 [0,4194304] 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:4/853: write d0/dc/d1f/db0/f102 [432208,60392] 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:6/810: truncate d2/d1a/d1c/d2c/d51/f2f 509009 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:0/792: getdents d5/db/d1f/d19/dbd/d64/d91/ddd/df6 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:3/779: rmdir d0/d1a/d1d/dc5 39 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:5/863: dwrite d1/d3/d6/d8b/d59/d74/f5e [4194304,4194304] 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:9/871: creat d0/f116 x:0 0 0 2024-03-23T11:47:47.212 INFO:tasks.workunit.client.1.smithi111.stdout:7/807: getdents d14/d1b/d1e/d2c/d42/d5e/dc5 0 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:0/793: dread - d5/dc3/f103 zero size 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:6/811: read - d2/d1f/dce/fe1 zero size 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:4/854: rename d0/dc/d4c/d2a/d97/d4d/f9c to d0/dc/d89/da4/f114 0 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:3/780: mkdir d0/d72/d104 0 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:0/794: write d5/db/d1f/d47/d6f/fe9 [552759,89935] 0 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:7/808: dread - d14/d1b/d1e/d2c/d2e/d73/fce zero size 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:1/790: dread d0/f59 [0,4194304] 0 2024-03-23T11:47:47.213 INFO:tasks.workunit.client.1.smithi111.stdout:6/812: mkdir d2/d1a/d106 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:3/781: truncate d0/d7/d19/d5e/da6/deb/ff9 8896 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:3/782: write d0/d1a/d2b/fca [852729,127072] 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:0/795: symlink d5/db/d1f/d47/d59/d5b/l10c 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:7/809: rename d14/d1b/d1e/d2c/d2e/d33/cbb to d14/d44/d70/daf/cec 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:5/864: dread d1/d3/d6/d7/f24 [0,4194304] 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:9/872: fdatasync d0/d4/d6a/da7/fd5 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:0/796: creat d5/db/d1f/d19/dbd/d64/d91/ddd/f10d x:0 0 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:4/855: creat d0/dc/d4c/f115 x:0 0 0 2024-03-23T11:47:47.214 INFO:tasks.workunit.client.1.smithi111.stdout:5/865: mkdir d1/d3/d6/d8b/d59/d11e 0 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:7/810: truncate d14/d1b/d1e/d2c/d42/d5e/d9e/f9f 746914 0 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:1/791: dread d0/d2b/d69/f8c [0,4194304] 0 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:0/797: rename d5/deb/f109 to d5/db/d1f/de4/da7/f10e 0 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:3/783: dread d0/d1a/d1d/f24 [0,4194304] 0 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:5/866: mknod d1/d3/d3a/c11f 0 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:7/811: unlink d14/d37/f49 0 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:5/867: dread - d1/d3/d6/d8b/d59/d6c/f107 zero size 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:1/792: rmdir d0/d5/dd/d98 39 2024-03-23T11:47:47.215 INFO:tasks.workunit.client.1.smithi111.stdout:0/798: dwrite d5/db/d1f/de4/f61 [0,4194304] 0 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:6/813: rename d2/l4 to d2/l107 0 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:3/784: write d0/d1a/d2b/d38/d7f/dd0/fd3 [1550887,41626] 0 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:1/793: chown d0/d6/d48/cb8 65084 1 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:5/868: fdatasync d1/d3/d6/d8b/d59/d74/f6b 0 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:1/794: readlink d0/d2b/l65 0 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:6/814: chown d2/d1f/d45/d49/d8d/fae 8153 1 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:4/856: write d0/dc/f35 [4575658,42062] 0 2024-03-23T11:47:47.216 INFO:tasks.workunit.client.1.smithi111.stdout:4/857: chown d0/dc/d4c/d2a/f73 8 1 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:4/858: dwrite d0/dc/d89/da4/f103 [0,4194304] 0 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:7/812: dwrite f1 [0,4194304] 0 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:3/785: rename d0/d55/f97 to d0/d7/d19/d91/f105 0 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:1/795: mknod d0/d5/dd/dc6/c110 0 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:6/815: chown d2/d1f/d45/d49/d8d/cdb 10 1 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:4/859: creat d0/dc/d4c/dad/f116 x:0 0 0 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:7/813: truncate d14/d37/f9d 693053 0 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:4/860: stat d0/dc/d4c/d2a/d97/d4d/l6e 0 2024-03-23T11:47:47.217 INFO:tasks.workunit.client.1.smithi111.stdout:7/814: dread - d14/d37/d43/fe2 zero size 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:7/815: truncate d14/d37/d54/fe4 677052 0 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:3/786: mknod d0/d1a/d2d/c106 0 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:3/787: read d0/d1a/d2b/d38/d7f/f8a [1198375,8280] 0 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:3/788: readlink d0/d7/d19/l42 0 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:6/816: rename d2/d35/de3 to d2/d1a/d106/d108 0 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:7/816: creat d14/d1b/d1e/d86/fed x:0 0 0 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:3/789: dread d0/d7/f82 [0,4194304] 0 2024-03-23T11:47:47.218 INFO:tasks.workunit.client.1.smithi111.stdout:0/799: truncate d5/db/d1f/f18 966033 0 2024-03-23T11:47:47.219 INFO:tasks.workunit.client.1.smithi111.stdout:4/861: write d0/dc/d4c/f6f [1724846,74333] 0 2024-03-23T11:47:47.219 INFO:tasks.workunit.client.1.smithi111.stdout:7/817: link d14/d1b/d1e/d2c/d2e/d95/cd0 d14/d1b/d1e/d2c/d42/d9c/cee 0 2024-03-23T11:47:47.220 INFO:tasks.workunit.client.1.smithi111.stdout:3/790: mknod d0/d7/d19/d91/da3/dc8/c107 0 2024-03-23T11:47:47.220 INFO:tasks.workunit.client.0.smithi063.stdout:4/653: link d5/da/d20/d4b/d1f/f39 d5/da/d20/d4b/d16/d37/fbe 0 2024-03-23T11:47:47.221 INFO:tasks.workunit.client.1.smithi111.stdout:3/791: chown d0/d72 536705943 1 2024-03-23T11:47:47.221 INFO:tasks.workunit.client.1.smithi111.stdout:6/817: write d2/d1a/d1c/d2c/fa5 [492927,28587] 0 2024-03-23T11:47:47.221 INFO:tasks.workunit.client.0.smithi063.stdout:4/654: readlink d5/da/d20/l9d 0 2024-03-23T11:47:47.224 INFO:tasks.workunit.client.1.smithi111.stdout:0/800: mkdir d5/db/d1f/d47/d106/d10f 0 2024-03-23T11:47:47.224 INFO:tasks.workunit.client.1.smithi111.stdout:7/818: fsync d14/d1b/d1e/d2c/d2e/d73/dae/d74/fa9 0 2024-03-23T11:47:47.228 INFO:tasks.workunit.client.1.smithi111.stdout:5/869: dwrite d1/d20/d5c/d99/dbf/dc3/fda [0,4194304] 0 2024-03-23T11:47:47.229 INFO:tasks.workunit.client.1.smithi111.stdout:3/792: mknod d0/db9/c108 0 2024-03-23T11:47:47.231 INFO:tasks.workunit.client.1.smithi111.stdout:3/793: chown d0/d1a/d2b/d38/dc2/ld6 194661 1 2024-03-23T11:47:47.232 INFO:tasks.workunit.client.0.smithi063.stdout:3/921: link d13/d22/d2d/d7d/fb5 d13/d22/d2d/d3f/d58/d6e/d8d/f123 0 2024-03-23T11:47:47.232 INFO:tasks.workunit.client.1.smithi111.stdout:6/818: rmdir d2/d1a/d1c/d2c 39 2024-03-23T11:47:47.236 INFO:tasks.workunit.client.0.smithi063.stdout:9/649: truncate d4/d31/d70/d8d/fc6 1983773 0 2024-03-23T11:47:47.237 INFO:tasks.workunit.client.0.smithi063.stdout:0/737: dread d2/d14/f87 [0,4194304] 0 2024-03-23T11:47:47.246 INFO:tasks.workunit.client.1.smithi111.stdout:1/796: dread d0/d6/f34 [0,4194304] 0 2024-03-23T11:47:47.248 INFO:tasks.workunit.client.1.smithi111.stdout:1/797: fsync d0/d5/dd/f6e 0 2024-03-23T11:47:47.248 INFO:tasks.workunit.client.0.smithi063.stdout:8/900: link d6/d1f/d54/d63/d90/d5d/d7d/d9a/lb8 d6/d30/d128/l12d 0 2024-03-23T11:47:47.251 INFO:tasks.workunit.client.0.smithi063.stdout:1/551: dread d3/d49/d56/d65/d84/f98 [0,4194304] 0 2024-03-23T11:47:47.251 INFO:tasks.workunit.client.0.smithi063.stdout:6/503: getdents d1/d2a/d41/d42 0 2024-03-23T11:47:47.254 INFO:tasks.workunit.client.1.smithi111.stdout:5/870: mkdir d1/d3/d120 0 2024-03-23T11:47:47.256 INFO:tasks.workunit.client.0.smithi063.stdout:3/922: mkdir d13/d66/d112/d124 0 2024-03-23T11:47:47.257 INFO:tasks.workunit.client.0.smithi063.stdout:1/552: write d3/d49/d56/d65/f7c [7247713,116668] 0 2024-03-23T11:47:47.258 INFO:tasks.workunit.client.1.smithi111.stdout:6/819: read d2/d1a/d1c/d2c/f8c [2174501,117235] 0 2024-03-23T11:47:47.259 INFO:tasks.workunit.client.1.smithi111.stdout:0/801: creat d5/f110 x:0 0 0 2024-03-23T11:47:47.261 INFO:tasks.workunit.client.1.smithi111.stdout:4/862: link d0/dc/d30/c31 d0/d95/c117 0 2024-03-23T11:47:47.262 INFO:tasks.workunit.client.0.smithi063.stdout:9/650: rename d4/d4c/d44/l54 to d4/d31/le6 0 2024-03-23T11:47:47.263 INFO:tasks.workunit.client.1.smithi111.stdout:5/871: rename d1/d20/f90 to d1/d20/d5c/dc5/f121 0 2024-03-23T11:47:47.265 INFO:tasks.workunit.client.0.smithi063.stdout:0/738: dwrite d2/d14/d36/ff4 [0,4194304] 0 2024-03-23T11:47:47.265 INFO:tasks.workunit.client.1.smithi111.stdout:6/820: rmdir d2/d35/da7 39 2024-03-23T11:47:47.266 INFO:tasks.workunit.client.0.smithi063.stdout:9/651: stat d4/d31/d43/d49/fa4 0 2024-03-23T11:47:47.266 INFO:tasks.workunit.client.1.smithi111.stdout:7/819: write d14/f1c [4038740,15341] 0 2024-03-23T11:47:47.267 INFO:tasks.workunit.client.1.smithi111.stdout:4/863: creat d0/dc/d4c/d2a/d97/d4f/dba/f118 x:0 0 0 2024-03-23T11:47:47.274 INFO:tasks.workunit.client.1.smithi111.stdout:0/802: rename d5/c98 to d5/dc3/c111 0 2024-03-23T11:47:47.275 INFO:tasks.workunit.client.1.smithi111.stdout:3/794: truncate d0/d1a/d1d/d26/f46 7246665 0 2024-03-23T11:47:47.275 INFO:tasks.workunit.client.0.smithi063.stdout:1/553: dread - d3/d49/d56/f86 zero size 2024-03-23T11:47:47.276 INFO:tasks.workunit.client.1.smithi111.stdout:5/872: truncate d1/d3/d6/d7/f69 1030844 0 2024-03-23T11:47:47.277 INFO:tasks.workunit.client.0.smithi063.stdout:4/655: getdents d5/da/d72 0 2024-03-23T11:47:47.277 INFO:tasks.workunit.client.0.smithi063.stdout:9/652: creat d4/d31/d9c/d9e/fe7 x:0 0 0 2024-03-23T11:47:47.278 INFO:tasks.workunit.client.1.smithi111.stdout:6/821: mknod d2/d1f/d45/d9a/c109 0 2024-03-23T11:47:47.278 INFO:tasks.workunit.client.1.smithi111.stdout:3/795: chown d0/d1a/f28 281530737 1 2024-03-23T11:47:47.279 INFO:tasks.workunit.client.0.smithi063.stdout:0/739: unlink d2/d14/d36/d77/d94/da6/f98 0 2024-03-23T11:47:47.280 INFO:tasks.workunit.client.1.smithi111.stdout:0/803: write d5/db/d1f/d19/dbd/d64/df2/ff5 [584400,102147] 0 2024-03-23T11:47:47.283 INFO:tasks.workunit.client.0.smithi063.stdout:0/740: write d2/d12/d32/f91 [1879303,114229] 0 2024-03-23T11:47:47.283 INFO:tasks.workunit.client.0.smithi063.stdout:3/923: creat d13/d22/d2d/d78/dcc/d106/f125 x:0 0 0 2024-03-23T11:47:47.284 INFO:tasks.workunit.client.1.smithi111.stdout:7/820: symlink d14/d1b/d1e/d2c/d42/d9c/dd3/lef 0 2024-03-23T11:47:47.284 INFO:tasks.workunit.client.1.smithi111.stdout:1/798: getdents d0/d5/dd/d1e/d70 0 2024-03-23T11:47:47.284 INFO:tasks.workunit.client.0.smithi063.stdout:1/554: read - d3/d1c/d4b/f8a zero size 2024-03-23T11:47:47.286 INFO:tasks.workunit.client.1.smithi111.stdout:5/873: rmdir d1/d3/d6/d30/db2 39 2024-03-23T11:47:47.288 INFO:tasks.workunit.client.1.smithi111.stdout:6/822: symlink d2/d35/dc5/l10a 0 2024-03-23T11:47:47.288 INFO:tasks.workunit.client.1.smithi111.stdout:7/821: rename d14/d1b/d1e to d14/d1b/d1e/d2c/d42/d5e/dc5/df0 22 2024-03-23T11:47:47.289 INFO:tasks.workunit.client.1.smithi111.stdout:3/796: mkdir d0/d7/d19/d5e/da6/d109 0 2024-03-23T11:47:47.290 INFO:tasks.workunit.client.1.smithi111.stdout:1/799: symlink d0/d5/dd/d1e/l111 0 2024-03-23T11:47:47.293 INFO:tasks.workunit.client.1.smithi111.stdout:1/800: stat d0/d6/d48/d4d/dab/ddb 0 2024-03-23T11:47:47.296 INFO:tasks.workunit.client.1.smithi111.stdout:4/864: rmdir d0/d95/dcb 0 2024-03-23T11:47:47.300 INFO:tasks.workunit.client.0.smithi063.stdout:3/924: creat d13/d22/d2d/d3f/d58/d6e/d8d/dda/f126 x:0 0 0 2024-03-23T11:47:47.304 INFO:tasks.workunit.client.1.smithi111.stdout:7/822: unlink d14/d1b/d1e/d2c/d2e/ce0 0 2024-03-23T11:47:47.304 INFO:tasks.workunit.client.0.smithi063.stdout:1/555: rename d3/f38 to d3/d49/d56/d65/d84/fac 0 2024-03-23T11:47:47.305 INFO:tasks.workunit.client.0.smithi063.stdout:8/901: write d6/dd/d59/fad [610814,34560] 0 2024-03-23T11:47:47.308 INFO:tasks.workunit.client.1.smithi111.stdout:3/797: dwrite d0/f1f [0,4194304] 0 2024-03-23T11:47:47.310 INFO:tasks.workunit.client.0.smithi063.stdout:9/653: creat d4/d31/d43/fe8 x:0 0 0 2024-03-23T11:47:47.313 INFO:tasks.workunit.client.1.smithi111.stdout:4/865: mknod d0/dc/d4c/d2a/d97/d4f/c119 0 2024-03-23T11:47:47.315 INFO:tasks.workunit.client.1.smithi111.stdout:0/804: truncate d5/db/d1f/de4/f61 3429423 0 2024-03-23T11:47:47.318 INFO:tasks.workunit.client.1.smithi111.stdout:6/823: rename d2/d1f/d45/f62 to d2/d1f/d45/d9a/dde/f10b 0 2024-03-23T11:47:47.320 INFO:tasks.workunit.client.0.smithi063.stdout:9/654: write d4/d31/fb9 [771250,66109] 0 2024-03-23T11:47:47.320 INFO:tasks.workunit.client.1.smithi111.stdout:3/798: mknod d0/d7/d19/d5e/da6/deb/c10a 0 2024-03-23T11:47:47.322 INFO:tasks.workunit.client.1.smithi111.stdout:3/799: dread - d0/d1a/d1d/fd7 zero size 2024-03-23T11:47:47.326 INFO:tasks.workunit.client.0.smithi063.stdout:1/556: dread d3/d35/d40/f53 [0,4194304] 0 2024-03-23T11:47:47.326 INFO:tasks.workunit.client.1.smithi111.stdout:4/866: creat d0/dc/d7a/f11a x:0 0 0 2024-03-23T11:47:47.328 INFO:tasks.workunit.client.1.smithi111.stdout:4/867: chown d0/dc/d78/cf7 3523 1 2024-03-23T11:47:47.328 INFO:tasks.workunit.client.1.smithi111.stdout:0/805: creat d5/db/d1f/de4/f112 x:0 0 0 2024-03-23T11:47:47.329 INFO:tasks.workunit.client.0.smithi063.stdout:0/741: write d2/d12/f44 [4332362,45877] 0 2024-03-23T11:47:47.330 INFO:tasks.workunit.client.1.smithi111.stdout:7/823: mknod d14/d1b/d1e/d2c/d42/d5e/dc5/cf1 0 2024-03-23T11:47:47.331 INFO:tasks.workunit.client.1.smithi111.stdout:6/824: fdatasync d2/d1f/f7a 0 2024-03-23T11:47:47.332 INFO:tasks.workunit.client.1.smithi111.stdout:5/874: link d1/d3/d6/d7/d63/d2c/d43/dfc/laf d1/d3/d6/d7/d86/l122 0 2024-03-23T11:47:47.334 INFO:tasks.workunit.client.0.smithi063.stdout:9/655: fsync d4/d4c/d19/f1b 0 2024-03-23T11:47:47.335 INFO:tasks.workunit.client.0.smithi063.stdout:0/742: chown d2/d14/f34 62366287 1 2024-03-23T11:47:47.337 INFO:tasks.workunit.client.1.smithi111.stdout:3/800: creat d0/d1a/d1d/d26/d5a/f10b x:0 0 0 2024-03-23T11:47:47.337 INFO:tasks.workunit.client.1.smithi111.stdout:6/825: read d2/d1f/fc6 [342105,2446] 0 2024-03-23T11:47:47.337 INFO:tasks.workunit.client.1.smithi111.stdout:1/801: rename d0/d6/d48/l73 to d0/d5/dd/d98/l112 0 2024-03-23T11:47:47.338 INFO:tasks.workunit.client.0.smithi063.stdout:8/902: dread d6/d1f/d54/d5c/f6d [0,4194304] 0 2024-03-23T11:47:47.340 INFO:tasks.workunit.client.0.smithi063.stdout:4/656: dwrite d5/da/d20/d4b/d16/d50/d8e/fa7 [0,4194304] 0 2024-03-23T11:47:47.346 INFO:tasks.workunit.client.1.smithi111.stdout:3/801: write d0/d1a/d1d/f24 [1366050,105936] 0 2024-03-23T11:47:47.348 INFO:tasks.workunit.client.1.smithi111.stdout:5/875: mkdir d1/d3/d6/d8b/d59/d6c/df3/d123 0 2024-03-23T11:47:47.350 INFO:tasks.workunit.client.0.smithi063.stdout:0/743: creat d2/d14/de8/ddb/f108 x:0 0 0 2024-03-23T11:47:47.350 INFO:tasks.workunit.client.0.smithi063.stdout:3/925: write d13/d22/d2d/d3f/f9e [215849,115471] 0 2024-03-23T11:47:47.352 INFO:tasks.workunit.client.0.smithi063.stdout:9/656: symlink d4/d31/d9c/le9 0 2024-03-23T11:47:47.353 INFO:tasks.workunit.client.0.smithi063.stdout:4/657: rmdir d5/da/d20/d4b/d16/d32 39 2024-03-23T11:47:47.354 INFO:tasks.workunit.client.0.smithi063.stdout:1/557: rename d3/d3f/d5c to d3/d62/dad 0 2024-03-23T11:47:47.355 INFO:tasks.workunit.client.1.smithi111.stdout:7/824: dwrite d14/d37/d54/fe4 [0,4194304] 0 2024-03-23T11:47:47.355 INFO:tasks.workunit.client.0.smithi063.stdout:8/903: chown d6/d1f/d54/d63/d90/d15/d9d/f117 153 1 2024-03-23T11:47:47.359 INFO:tasks.workunit.client.1.smithi111.stdout:1/802: unlink d0/d6/d48/cb8 0 2024-03-23T11:47:47.359 INFO:tasks.workunit.client.1.smithi111.stdout:7/825: truncate d14/d37/d43/fe2 477109 0 2024-03-23T11:47:47.360 INFO:tasks.workunit.client.0.smithi063.stdout:4/658: write d5/da/d20/d4b/d16/fb4 [491805,116629] 0 2024-03-23T11:47:47.363 INFO:tasks.workunit.client.0.smithi063.stdout:1/558: readlink d3/d3f/d4d/d52/l75 0 2024-03-23T11:47:47.364 INFO:tasks.workunit.client.1.smithi111.stdout:6/826: fsync d2/d1a/d1c/ddf/ff3 0 2024-03-23T11:47:47.365 INFO:tasks.workunit.client.0.smithi063.stdout:4/659: stat d5/d1e/d2b/daa 0 2024-03-23T11:47:47.367 INFO:tasks.workunit.client.0.smithi063.stdout:9/657: unlink d4/d31/f5d 0 2024-03-23T11:47:47.368 INFO:tasks.workunit.client.1.smithi111.stdout:0/806: dread d5/db/f15 [0,4194304] 0 2024-03-23T11:47:47.371 INFO:tasks.workunit.client.0.smithi063.stdout:1/559: chown d3/df/d2f/c61 7668669 1 2024-03-23T11:47:47.371 INFO:tasks.workunit.client.0.smithi063.stdout:3/926: symlink d13/d22/d2d/d3f/l127 0 2024-03-23T11:47:47.377 INFO:tasks.workunit.client.1.smithi111.stdout:5/876: dwrite d1/d3/d6/d7/d63/d2c/d43/f10f [0,4194304] 0 2024-03-23T11:47:47.379 INFO:tasks.workunit.client.0.smithi063.stdout:9/658: creat d4/d31/d9c/d9e/fea x:0 0 0 2024-03-23T11:47:47.379 INFO:tasks.workunit.client.0.smithi063.stdout:8/904: symlink d6/d30/d128/l12e 0 2024-03-23T11:47:47.382 INFO:tasks.workunit.client.1.smithi111.stdout:4/868: dwrite d0/f1 [4194304,4194304] 0 2024-03-23T11:47:47.383 INFO:tasks.workunit.client.0.smithi063.stdout:4/660: mkdir d5/da/d7e/dbf 0 2024-03-23T11:47:47.383 INFO:tasks.workunit.client.0.smithi063.stdout:1/560: unlink d3/c25 0 2024-03-23T11:47:47.384 INFO:tasks.workunit.client.1.smithi111.stdout:0/807: getdents d5/db/d1f/d19/dbd/d64/d91/ddd/df6 0 2024-03-23T11:47:47.385 INFO:tasks.workunit.client.0.smithi063.stdout:3/927: stat d13/d22/d2d/d3f/d58/d97/df8/l10b 0 2024-03-23T11:47:47.385 INFO:tasks.workunit.client.1.smithi111.stdout:4/869: chown d0/dc/d4c/d2a/d97/d4d 112155604 1 2024-03-23T11:47:47.385 INFO:tasks.workunit.client.1.smithi111.stdout:6/827: link d2/d35/la3 d2/d1f/d45/d9a/l10c 0 2024-03-23T11:47:47.391 INFO:tasks.workunit.client.1.smithi111.stdout:6/828: chown d2/d1f/d45/d49/d8d/f9d 844 1 2024-03-23T11:47:47.393 INFO:tasks.workunit.client.1.smithi111.stdout:0/808: truncate d5/db/d1f/f28 805461 0 2024-03-23T11:47:47.394 INFO:tasks.workunit.client.0.smithi063.stdout:1/561: unlink d3/cb 0 2024-03-23T11:47:47.394 INFO:tasks.workunit.client.1.smithi111.stdout:0/809: stat d5/db/ced 0 2024-03-23T11:47:47.396 INFO:tasks.workunit.client.0.smithi063.stdout:8/905: truncate d6/d1f/d54/d63/d90/d15/f17 1586082 0 2024-03-23T11:47:47.396 INFO:tasks.workunit.client.1.smithi111.stdout:5/877: getdents d1/d3/d6/d7/d7b/d10d 0 2024-03-23T11:47:47.398 INFO:tasks.workunit.client.1.smithi111.stdout:6/829: symlink d2/d1a/d1c/d2c/l10d 0 2024-03-23T11:47:47.401 INFO:tasks.workunit.client.1.smithi111.stdout:0/810: creat d5/db/d1f/d47/d106/f113 x:0 0 0 2024-03-23T11:47:47.410 INFO:tasks.workunit.client.1.smithi111.stdout:5/878: rename d1/d3/d6/d7/fc6 to d1/d3/d6/d7/d8c/d9e/f124 0 2024-03-23T11:47:47.410 INFO:tasks.workunit.client.1.smithi111.stdout:3/802: write d0/d7/fa7 [389948,60576] 0 2024-03-23T11:47:47.410 INFO:tasks.workunit.client.0.smithi063.stdout:0/744: dwrite d2/d12/d35/f6a [4194304,4194304] 0 2024-03-23T11:47:47.410 INFO:tasks.workunit.client.0.smithi063.stdout:8/906: chown d6/d1f/d54/d63/fc2 1 1 2024-03-23T11:47:47.411 INFO:tasks.workunit.client.0.smithi063.stdout:4/661: mkdir d5/da/d20/d4b/d16/d50/d8e/d96/dc0 0 2024-03-23T11:47:47.412 INFO:tasks.workunit.client.1.smithi111.stdout:3/803: truncate d0/d72/ff2 613427 0 2024-03-23T11:47:47.413 INFO:tasks.workunit.client.1.smithi111.stdout:5/879: chown d1/d3/d6/d7/d63/d2c/d43/dfc/cca 7514 1 2024-03-23T11:47:47.415 INFO:tasks.workunit.client.0.smithi063.stdout:9/659: dwrite d4/d4c/fca [0,4194304] 0 2024-03-23T11:47:47.440 INFO:tasks.workunit.client.0.smithi063.stdout:9/660: read d4/d31/d43/da9/da6/fb0 [183143,96137] 0 2024-03-23T11:47:47.443 INFO:tasks.workunit.client.0.smithi063.stdout:3/928: link d13/d22/d2d/d7d/fc3 d13/d22/d2d/d78/dcc/f128 0 2024-03-23T11:47:47.447 INFO:tasks.workunit.client.0.smithi063.stdout:1/562: creat d3/d35/fae x:0 0 0 2024-03-23T11:47:47.448 INFO:tasks.workunit.client.0.smithi063.stdout:3/929: read - d13/d22/d2d/d3f/de0/fe9 zero size 2024-03-23T11:47:47.450 INFO:tasks.workunit.client.0.smithi063.stdout:1/563: fsync d3/df/d2f/f97 0 2024-03-23T11:47:47.453 INFO:tasks.workunit.client.0.smithi063.stdout:3/930: fdatasync d13/d22/d2d/d3f/d58/d6e/d8d/f109 0 2024-03-23T11:47:47.456 INFO:tasks.workunit.client.0.smithi063.stdout:0/745: mkdir d2/d14/d36/d77/d94/da6/d25/d93/dd3/d109 0 2024-03-23T11:47:47.462 INFO:tasks.workunit.client.0.smithi063.stdout:4/662: dread d5/da/d20/d4b/f8b [0,4194304] 0 2024-03-23T11:47:47.467 INFO:tasks.workunit.client.0.smithi063.stdout:9/661: dwrite d4/d31/d43/da9/da6/dbe/fe4 [0,4194304] 0 2024-03-23T11:47:47.470 INFO:tasks.workunit.client.0.smithi063.stdout:8/907: write d6/dd/d59/db2/fe2 [1042105,113654] 0 2024-03-23T11:47:47.506 INFO:tasks.workunit.client.1.smithi111.stdout:0/811: fsync d5/db/d1f/d19/dbd/d64/dc7/d100/fd3 0 2024-03-23T11:47:47.509 INFO:tasks.workunit.client.1.smithi111.stdout:7/826: dread d14/d1b/d1e/d2c/d42/f4a [8388608,4194304] 0 2024-03-23T11:47:47.518 INFO:tasks.workunit.client.1.smithi111.stdout:7/827: mknod d14/d37/d54/cf2 0 2024-03-23T11:47:47.520 INFO:tasks.workunit.client.1.smithi111.stdout:3/804: creat d0/d53/dfa/d103/f10c x:0 0 0 2024-03-23T11:47:47.521 INFO:tasks.workunit.client.0.smithi063.stdout:3/931: dread f6 [0,4194304] 0 2024-03-23T11:47:47.521 INFO:tasks.workunit.client.1.smithi111.stdout:3/805: readlink d0/d1a/d1d/d26/d5a/l66 0 2024-03-23T11:47:47.523 INFO:tasks.workunit.client.1.smithi111.stdout:7/828: symlink d14/d1b/d1e/d2c/d8b/lf3 0 2024-03-23T11:47:47.525 INFO:tasks.workunit.client.1.smithi111.stdout:4/870: dread d0/f1 [0,4194304] 0 2024-03-23T11:47:47.526 INFO:tasks.workunit.client.1.smithi111.stdout:0/812: rmdir d5/db/d4d/df8 0 2024-03-23T11:47:47.531 INFO:tasks.workunit.client.1.smithi111.stdout:4/871: creat d0/dc/d4c/d2a/df2/f11b x:0 0 0 2024-03-23T11:47:47.531 INFO:tasks.workunit.client.0.smithi063.stdout:0/746: symlink d2/d48/de2/l10a 0 2024-03-23T11:47:47.533 INFO:tasks.workunit.client.1.smithi111.stdout:6/830: dread d2/d1f/fc6 [0,4194304] 0 2024-03-23T11:47:47.536 INFO:tasks.workunit.client.1.smithi111.stdout:4/872: chown d0/d95/db5/dc7/cf0 112 1 2024-03-23T11:47:47.539 INFO:tasks.workunit.client.0.smithi063.stdout:9/662: mknod d4/ceb 0 2024-03-23T11:47:47.539 INFO:tasks.workunit.client.0.smithi063.stdout:8/908: truncate d6/d1f/d54/fcc 18444 0 2024-03-23T11:47:47.540 INFO:tasks.workunit.client.1.smithi111.stdout:3/806: dwrite d0/d7/f23 [4194304,4194304] 0 2024-03-23T11:47:47.543 INFO:tasks.workunit.client.0.smithi063.stdout:8/909: read - d6/d1f/d54/d63/d90/f114 zero size 2024-03-23T11:47:47.544 INFO:tasks.workunit.client.1.smithi111.stdout:3/807: chown d0/d1a/d4f/d90/fce 4 1 2024-03-23T11:47:47.548 INFO:tasks.workunit.client.1.smithi111.stdout:5/880: dread d1/f17 [0,4194304] 0 2024-03-23T11:47:47.549 INFO:tasks.workunit.client.1.smithi111.stdout:3/808: dread - d0/d1a/d2d/fd5 zero size 2024-03-23T11:47:47.550 INFO:tasks.workunit.client.0.smithi063.stdout:4/663: getdents d5/da/d20 0 2024-03-23T11:47:47.551 INFO:tasks.workunit.client.1.smithi111.stdout:3/809: stat d0/d1a/d2b/ff0 0 2024-03-23T11:47:47.552 INFO:tasks.workunit.client.1.smithi111.stdout:4/873: dwrite d0/d95/db5/dc7/f104 [0,4194304] 0 2024-03-23T11:47:47.555 INFO:tasks.workunit.client.1.smithi111.stdout:5/881: chown d1/d3/d6/d30/db2 4 1 2024-03-23T11:47:47.556 INFO:tasks.workunit.client.1.smithi111.stdout:0/813: dwrite d5/db/d1f/d19/dbd/d64/dc7/d100/fd3 [0,4194304] 0 2024-03-23T11:47:47.559 INFO:tasks.workunit.client.1.smithi111.stdout:4/874: creat d0/dc/d4c/dad/f11c x:0 0 0 2024-03-23T11:47:47.567 INFO:tasks.workunit.client.0.smithi063.stdout:9/663: dwrite d4/d31/d43/d49/d8e/f93 [0,4194304] 0 2024-03-23T11:47:47.572 INFO:tasks.workunit.client.1.smithi111.stdout:5/882: dwrite d1/d20/f61 [4194304,4194304] 0 2024-03-23T11:47:47.573 INFO:tasks.workunit.client.0.smithi063.stdout:8/910: mkdir d6/dd/d59/db2/dbe/dfb/d12f 0 2024-03-23T11:47:47.578 INFO:tasks.workunit.client.1.smithi111.stdout:3/810: getdents d0/d7/d19/d5e 0 2024-03-23T11:47:47.578 INFO:tasks.workunit.client.1.smithi111.stdout:0/814: truncate d5/db/f16 2158821 0 2024-03-23T11:47:47.578 INFO:tasks.workunit.client.1.smithi111.stdout:4/875: creat d0/dc/d113/f11d x:0 0 0 2024-03-23T11:47:47.583 INFO:tasks.workunit.client.0.smithi063.stdout:1/564: creat d3/d1c/da8/faf x:0 0 0 2024-03-23T11:47:47.584 INFO:tasks.workunit.client.1.smithi111.stdout:4/876: rmdir d0/dc/d4c/d2a/d97 39 2024-03-23T11:47:47.585 INFO:tasks.workunit.client.0.smithi063.stdout:0/747: link d2/d14/d36/d77/d94/da6/lc8 d2/d6e/l10b 0 2024-03-23T11:47:47.585 INFO:tasks.workunit.client.0.smithi063.stdout:4/664: dread d5/da/d20/d4b/d16/fa3 [0,4194304] 0 2024-03-23T11:47:47.587 INFO:tasks.workunit.client.0.smithi063.stdout:0/748: chown d2/d12/d35/dcf 20213 1 2024-03-23T11:47:47.589 INFO:tasks.workunit.client.0.smithi063.stdout:4/665: dread - d5/da/d72/f92 zero size 2024-03-23T11:47:47.591 INFO:tasks.workunit.client.1.smithi111.stdout:7/829: dread d14/d44/d70/d7b/fa8 [0,4194304] 0 2024-03-23T11:47:47.593 INFO:tasks.workunit.client.0.smithi063.stdout:8/911: truncate d6/d1f/d54/d63/d90/d15/d9d/f117 265971 0 2024-03-23T11:47:47.595 INFO:tasks.workunit.client.1.smithi111.stdout:5/883: dread d1/d3/d6/d8b/fa3 [0,4194304] 0 2024-03-23T11:47:47.595 INFO:tasks.workunit.client.1.smithi111.stdout:0/815: fdatasync d5/db/d1f/d47/de0/fac 0 2024-03-23T11:47:47.598 INFO:tasks.workunit.client.1.smithi111.stdout:0/816: stat d5/db/d1f/d48/f66 0 2024-03-23T11:47:47.599 INFO:tasks.workunit.client.1.smithi111.stdout:3/811: dwrite d0/d1a/d1d/d26/d5a/d64/fba [0,4194304] 0 2024-03-23T11:47:47.602 INFO:tasks.workunit.client.1.smithi111.stdout:3/812: write d0/d7/d19/d5e/da6/deb/ff9 [895359,102439] 0 2024-03-23T11:47:47.605 INFO:tasks.workunit.client.0.smithi063.stdout:1/565: chown d3/d1c/f68 859 1 2024-03-23T11:47:47.610 INFO:tasks.workunit.client.0.smithi063.stdout:4/666: rmdir d5/d1e/d2b 39 2024-03-23T11:47:47.613 INFO:tasks.workunit.client.0.smithi063.stdout:3/932: dwrite d13/d22/d2d/d3f/d58/f70 [0,4194304] 0 2024-03-23T11:47:47.616 INFO:tasks.workunit.client.1.smithi111.stdout:4/877: creat d0/dc/d4c/d2a/d97/d4d/f11e x:0 0 0 2024-03-23T11:47:47.621 INFO:tasks.workunit.client.0.smithi063.stdout:8/912: mknod d6/d1f/d54/d63/d90/d5d/d7d/d9a/d126/c130 0 2024-03-23T11:47:47.622 INFO:tasks.workunit.client.1.smithi111.stdout:5/884: creat d1/d3/d6/d7/d63/d2c/d43/dfc/f125 x:0 0 0 2024-03-23T11:47:47.624 INFO:tasks.workunit.client.0.smithi063.stdout:0/749: dread d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:47.629 INFO:tasks.workunit.client.0.smithi063.stdout:1/566: rmdir d3/d49/d56/d65/d84 39 2024-03-23T11:47:47.633 INFO:tasks.workunit.client.0.smithi063.stdout:0/750: fsync d2/d14/f23 0 2024-03-23T11:47:47.637 INFO:tasks.workunit.client.0.smithi063.stdout:3/933: truncate d13/d22/d2d/fb7 424690 0 2024-03-23T11:47:47.638 INFO:tasks.workunit.client.1.smithi111.stdout:7/830: creat d14/d37/d43/ff4 x:0 0 0 2024-03-23T11:47:47.640 INFO:tasks.workunit.client.0.smithi063.stdout:0/751: truncate d2/d14/d36/d77/d94/da6/d25/d93/ff7 769350 0 2024-03-23T11:47:47.645 INFO:tasks.workunit.client.1.smithi111.stdout:5/885: dwrite d1/d3/d6/d8b/d59/d74/f73 [4194304,4194304] 0 2024-03-23T11:47:47.647 INFO:tasks.workunit.client.1.smithi111.stdout:4/878: creat d0/dc/d4c/f11f x:0 0 0 2024-03-23T11:47:47.647 INFO:tasks.workunit.client.1.smithi111.stdout:5/886: fsync d1/dcf/ffb 0 2024-03-23T11:47:47.649 INFO:tasks.workunit.client.1.smithi111.stdout:0/817: dread d5/db/d1f/d19/f4c [0,4194304] 0 2024-03-23T11:47:47.651 INFO:tasks.workunit.client.1.smithi111.stdout:4/879: write d0/d95/d9b/db3/dcc/dea/f69 [1428614,20649] 0 2024-03-23T11:47:47.653 INFO:tasks.workunit.client.0.smithi063.stdout:1/567: truncate d3/d1c/d4b/f8a 757685 0 2024-03-23T11:47:47.657 INFO:tasks.workunit.client.1.smithi111.stdout:4/880: mkdir d0/d95/d9b/db3/d120 0 2024-03-23T11:47:47.659 INFO:tasks.workunit.client.1.smithi111.stdout:3/813: truncate d0/d7/f76 809772 0 2024-03-23T11:47:47.659 INFO:tasks.workunit.client.0.smithi063.stdout:0/752: rename d2/d14/d36/d77/d94/da6/f33 to d2/d48/f10c 0 2024-03-23T11:47:47.660 INFO:tasks.workunit.client.1.smithi111.stdout:5/887: getdents d1/d3/d6/d7/d8c/d9e 0 2024-03-23T11:47:47.661 INFO:tasks.workunit.client.1.smithi111.stdout:3/814: readlink d0/d1a/d1d/lb4 0 2024-03-23T11:47:47.662 INFO:tasks.workunit.client.1.smithi111.stdout:8/654: write d5/d26/d2f/d62/d75/d4b/d51/da4/d97/f9c [376506,23420] 0 2024-03-23T11:47:47.663 INFO:tasks.workunit.client.1.smithi111.stdout:3/815: fsync d0/d1a/d1d/d26/d5a/f10b 0 2024-03-23T11:47:47.664 INFO:tasks.workunit.client.0.smithi063.stdout:1/568: creat d3/d49/d56/d65/d88/d92/fb0 x:0 0 0 2024-03-23T11:47:47.664 INFO:tasks.workunit.client.0.smithi063.stdout:3/934: mknod d13/d22/d2d/d3f/d58/c129 0 2024-03-23T11:47:47.666 INFO:tasks.workunit.client.1.smithi111.stdout:3/816: chown d0/d1a/d1d/lb4 573851 1 2024-03-23T11:47:47.669 INFO:tasks.workunit.client.0.smithi063.stdout:3/935: write d13/d22/d2d/d78/dcc/d106/f125 [164584,111660] 0 2024-03-23T11:47:47.672 INFO:tasks.workunit.client.1.smithi111.stdout:3/817: read d0/d7/f1e [1079542,129872] 0 2024-03-23T11:47:47.674 INFO:tasks.workunit.client.1.smithi111.stdout:5/888: write d1/d20/f37 [3002092,46112] 0 2024-03-23T11:47:47.674 INFO:tasks.workunit.client.1.smithi111.stdout:4/881: link d0/d95/d9b/db3/dcc/dea/da/f111 d0/dc/d4c/f121 0 2024-03-23T11:47:47.677 INFO:tasks.workunit.client.1.smithi111.stdout:0/818: dwrite d5/db/d1f/d19/d45/fe2 [0,4194304] 0 2024-03-23T11:47:47.678 INFO:tasks.workunit.client.0.smithi063.stdout:3/936: creat d13/d22/d2d/d78/df1/f12a x:0 0 0 2024-03-23T11:47:47.680 INFO:tasks.workunit.client.1.smithi111.stdout:4/882: read - d0/d95/fe6 zero size 2024-03-23T11:47:47.688 INFO:tasks.workunit.client.1.smithi111.stdout:3/818: truncate d0/d72/f86 858224 0 2024-03-23T11:47:47.688 INFO:tasks.workunit.client.1.smithi111.stdout:4/883: symlink d0/d95/d9b/db3/dcc/l122 0 2024-03-23T11:47:47.688 INFO:tasks.workunit.client.0.smithi063.stdout:1/569: rmdir d3/d1c/da1 0 2024-03-23T11:47:47.688 INFO:tasks.workunit.client.0.smithi063.stdout:8/913: dwrite d6/d1c/d27/f8d [0,4194304] 0 2024-03-23T11:47:47.689 INFO:tasks.workunit.client.1.smithi111.stdout:5/889: write d1/d20/d41/fe5 [204693,122692] 0 2024-03-23T11:47:47.692 INFO:tasks.workunit.client.1.smithi111.stdout:3/819: creat d0/d1a/d4f/d90/f10d x:0 0 0 2024-03-23T11:47:47.692 INFO:tasks.workunit.client.1.smithi111.stdout:5/890: chown d1/d3/d6/d30/db2/ff4 54670 1 2024-03-23T11:47:47.693 INFO:tasks.workunit.client.0.smithi063.stdout:3/937: readlink d13/d22/d2d/l18 0 2024-03-23T11:47:47.693 INFO:tasks.workunit.client.0.smithi063.stdout:0/753: write d2/d6e/d8c/fe4 [1418971,27415] 0 2024-03-23T11:47:47.696 INFO:tasks.workunit.client.0.smithi063.stdout:8/914: symlink d6/d1c/d32/l131 0 2024-03-23T11:47:47.697 INFO:tasks.workunit.client.1.smithi111.stdout:5/891: stat d1/d3/d6/d7/d63/d2c/d82/f88 0 2024-03-23T11:47:47.699 INFO:tasks.workunit.client.1.smithi111.stdout:4/884: rename d0/d95/d9b/db3/dcc/dea to d0/dc/d4c/d2a/df2/d123 0 2024-03-23T11:47:47.699 INFO:tasks.workunit.client.1.smithi111.stdout:8/655: write d5/f1f [1525648,62618] 0 2024-03-23T11:47:47.700 INFO:tasks.workunit.client.0.smithi063.stdout:3/938: rmdir d13/d15/d114 39 2024-03-23T11:47:47.703 INFO:tasks.workunit.client.1.smithi111.stdout:8/656: readlink d5/d26/d2f/d62/d75/d4b/lc1 0 2024-03-23T11:47:47.703 INFO:tasks.workunit.client.0.smithi063.stdout:0/754: symlink d2/d48/da5/da7/l10d 0 2024-03-23T11:47:47.704 INFO:tasks.workunit.client.1.smithi111.stdout:2/851: sync 2024-03-23T11:47:47.704 INFO:tasks.workunit.client.1.smithi111.stdout:5/892: truncate d1/d20/d5c/d99/dbf/ff1 409873 0 2024-03-23T11:47:47.705 INFO:tasks.workunit.client.0.smithi063.stdout:8/915: symlink d6/d1f/d54/d63/l132 0 2024-03-23T11:47:47.705 INFO:tasks.workunit.client.1.smithi111.stdout:3/820: getdents d0/d7/d19/d91 0 2024-03-23T11:47:47.705 INFO:tasks.workunit.client.1.smithi111.stdout:4/885: truncate d0/dc/d89/da4/fbd 60027 0 2024-03-23T11:47:47.706 INFO:tasks.workunit.client.0.smithi063.stdout:0/755: readlink d2/d14/d36/le1 0 2024-03-23T11:47:47.707 INFO:tasks.workunit.client.1.smithi111.stdout:5/893: readlink d1/d3/d6/d8b/d59/ld5 0 2024-03-23T11:47:47.710 INFO:tasks.workunit.client.1.smithi111.stdout:5/894: stat d1/d3/d6/d7/d63/d2c/l33 0 2024-03-23T11:47:47.715 INFO:tasks.workunit.client.0.smithi063.stdout:0/756: truncate d2/d14/f79 335896 0 2024-03-23T11:47:47.717 INFO:tasks.workunit.client.1.smithi111.stdout:3/821: rename d0/d7 to d0/d1a/d2b/d10e 0 2024-03-23T11:47:47.719 INFO:tasks.workunit.client.1.smithi111.stdout:3/822: readlink d0/d72/d7b/l8b 0 2024-03-23T11:47:47.720 INFO:tasks.workunit.client.0.smithi063.stdout:3/939: write d13/d22/f3b [396580,2087] 0 2024-03-23T11:47:47.720 INFO:tasks.workunit.client.1.smithi111.stdout:2/852: mknod d2/d11/c12d 0 2024-03-23T11:47:47.721 INFO:tasks.workunit.client.0.smithi063.stdout:0/757: chown d2/d14/d36/d77/d94/da6/d28/fdf 8618745 1 2024-03-23T11:47:47.721 INFO:tasks.workunit.client.1.smithi111.stdout:8/657: dwrite d5/d26/d2f/f77 [0,4194304] 0 2024-03-23T11:47:47.724 INFO:tasks.workunit.client.1.smithi111.stdout:4/886: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:47.728 INFO:tasks.workunit.client.1.smithi111.stdout:3/823: mkdir d0/d53/dfa/d10f 0 2024-03-23T11:47:47.735 INFO:tasks.workunit.client.1.smithi111.stdout:2/853: rename d2/d7/lcd to d2/d128/l12e 0 2024-03-23T11:47:47.735 INFO:tasks.workunit.client.0.smithi063.stdout:0/758: creat d2/d14/d36/d77/d94/da6/d28/d46/f10e x:0 0 0 2024-03-23T11:47:47.736 INFO:tasks.workunit.client.1.smithi111.stdout:8/658: symlink d5/d26/d2f/d62/d7e/dae/ld7 0 2024-03-23T11:47:47.741 INFO:tasks.workunit.client.1.smithi111.stdout:8/659: mkdir d5/d26/dd8 0 2024-03-23T11:47:47.748 INFO:tasks.workunit.client.0.smithi063.stdout:0/759: getdents d2/d14/d36/d77/d94/da6/d25/d93/dd3/d109 0 2024-03-23T11:47:47.754 INFO:tasks.workunit.client.0.smithi063.stdout:0/760: creat d2/d12/d35/f10f x:0 0 0 2024-03-23T11:47:47.757 INFO:tasks.workunit.client.1.smithi111.stdout:5/895: creat d1/d3/f126 x:0 0 0 2024-03-23T11:47:47.758 INFO:tasks.workunit.client.0.smithi063.stdout:0/761: mkdir d2/d48/d110 0 2024-03-23T11:47:47.759 INFO:tasks.workunit.client.0.smithi063.stdout:3/940: dwrite d13/d22/d2d/d3e/dce/fa6 [0,4194304] 0 2024-03-23T11:47:47.765 INFO:tasks.workunit.client.1.smithi111.stdout:8/660: write d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f7f [1846678,100678] 0 2024-03-23T11:47:47.766 INFO:tasks.workunit.client.1.smithi111.stdout:8/661: stat d5/d26/d2f/d62/d75/d8d/l9d 0 2024-03-23T11:47:47.769 INFO:tasks.workunit.client.0.smithi063.stdout:0/762: unlink d2/d14/d36/d77/d94/da6/d28/d46/d51/f65 0 2024-03-23T11:47:47.772 INFO:tasks.workunit.client.1.smithi111.stdout:5/896: link d1/d3/d6/d8b/d59/d74/f5e d1/d110/f127 0 2024-03-23T11:47:47.774 INFO:tasks.workunit.client.0.smithi063.stdout:0/763: write d2/d6e/d8c/fe4 [871303,82883] 0 2024-03-23T11:47:47.774 INFO:tasks.workunit.client.0.smithi063.stdout:3/941: link d13/d22/d2d/d3f/d67/d7b/fab d13/d22/d2d/d3f/d58/dcb/f12b 0 2024-03-23T11:47:47.776 INFO:tasks.workunit.client.1.smithi111.stdout:5/897: write d1/d3/d6/d7/d8c/fd7 [968199,101809] 0 2024-03-23T11:47:47.779 INFO:tasks.workunit.client.0.smithi063.stdout:3/942: mkdir d13/d22/d2d/d3e/d12c 0 2024-03-23T11:47:47.782 INFO:tasks.workunit.client.0.smithi063.stdout:0/764: rename d2/d14/d36/d77/d94/da6/d25/cea to d2/d14/d36/d77/deb/c111 0 2024-03-23T11:47:47.783 INFO:tasks.workunit.client.1.smithi111.stdout:5/898: mkdir d1/d3/d128 0 2024-03-23T11:47:47.783 INFO:tasks.workunit.client.1.smithi111.stdout:8/662: dwrite d5/d26/d31/d3b/f70 [0,4194304] 0 2024-03-23T11:47:47.785 INFO:tasks.workunit.client.0.smithi063.stdout:3/943: getdents d13/d22/d2d/d3e/d12c 0 2024-03-23T11:47:47.790 INFO:tasks.workunit.client.0.smithi063.stdout:0/765: creat d2/d14/d36/d77/d94/da6/d1f/f112 x:0 0 0 2024-03-23T11:47:47.790 INFO:tasks.workunit.client.1.smithi111.stdout:5/899: mkdir d1/d20/d41/d129 0 2024-03-23T11:47:47.790 INFO:tasks.workunit.client.0.smithi063.stdout:3/944: symlink d13/d22/d2d/d3f/d67/d7b/l12d 0 2024-03-23T11:47:47.797 INFO:tasks.workunit.client.1.smithi111.stdout:8/663: rename d5/d32/dcf to d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9 0 2024-03-23T11:47:47.799 INFO:tasks.workunit.client.0.smithi063.stdout:0/766: mknod d2/d48/c113 0 2024-03-23T11:47:47.799 INFO:tasks.workunit.client.1.smithi111.stdout:8/664: chown d5/d26/d2f/d62/db5/dce 676968 1 2024-03-23T11:47:47.800 INFO:tasks.workunit.client.1.smithi111.stdout:5/900: creat d1/d3/d6/d30/f12a x:0 0 0 2024-03-23T11:47:47.800 INFO:tasks.workunit.client.1.smithi111.stdout:8/665: dread - d5/d32/d7d/fa6 zero size 2024-03-23T11:47:47.805 INFO:tasks.workunit.client.1.smithi111.stdout:5/901: mkdir d1/d3/d6/d30/d12b 0 2024-03-23T11:47:47.807 INFO:tasks.workunit.client.1.smithi111.stdout:8/666: rename d5/d26/d2f/d62/d75/d4b/d51/da4/d97/ca5 to d5/d26/d2f/d62/d75/cda 0 2024-03-23T11:47:47.809 INFO:tasks.workunit.client.1.smithi111.stdout:5/902: mknod d1/d20/d5c/d99/dbf/c12c 0 2024-03-23T11:47:47.817 INFO:tasks.workunit.client.1.smithi111.stdout:5/903: getdents d1/d3/d6/d30/de6 0 2024-03-23T11:47:47.818 INFO:tasks.workunit.client.1.smithi111.stdout:8/667: creat d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/dad/fdb x:0 0 0 2024-03-23T11:47:47.818 INFO:tasks.workunit.client.1.smithi111.stdout:5/904: chown d1/d3/d6/d7/d63/d2c/d43/dfc/f125 1283324 1 2024-03-23T11:47:47.826 INFO:tasks.workunit.client.1.smithi111.stdout:5/905: rename d1/d3/d6/d7/d7b/cc7 to d1/d3/d6/d8b/d59/d6c/df3/c12d 0 2024-03-23T11:47:47.826 INFO:tasks.workunit.client.0.smithi063.stdout:0/767: write d2/d14/d36/d77/d94/da6/d1f/f9b [637985,104056] 0 2024-03-23T11:47:47.831 INFO:tasks.workunit.client.0.smithi063.stdout:0/768: write d2/d14/d36/ff4 [4919855,104846] 0 2024-03-23T11:47:47.834 INFO:tasks.workunit.client.1.smithi111.stdout:5/906: fdatasync d1/d3/d6/d8b/f26 0 2024-03-23T11:47:47.837 INFO:tasks.workunit.client.0.smithi063.stdout:3/945: dwrite d13/d22/f3d [0,4194304] 0 2024-03-23T11:47:47.838 INFO:tasks.workunit.client.1.smithi111.stdout:8/668: dwrite d5/d26/d31/d4d/f59 [0,4194304] 0 2024-03-23T11:47:47.840 INFO:tasks.workunit.client.1.smithi111.stdout:8/669: stat d5/d26/d2f/d62/d7e 0 2024-03-23T11:47:47.844 INFO:tasks.workunit.client.1.smithi111.stdout:8/670: mknod d5/d26/d2f/d62/d7e/d85/cdc 0 2024-03-23T11:47:47.845 INFO:tasks.workunit.client.1.smithi111.stdout:5/907: stat d1/d3/d6/f19 0 2024-03-23T11:47:47.849 INFO:tasks.workunit.client.1.smithi111.stdout:3/824: read d0/d1a/d2b/d10e/f23 [5494106,38076] 0 2024-03-23T11:47:47.853 INFO:tasks.workunit.client.1.smithi111.stdout:2/854: getdents d2/d128 0 2024-03-23T11:47:47.854 INFO:tasks.workunit.client.1.smithi111.stdout:5/908: mkdir d1/d3/d6/d8b/d59/d12e 0 2024-03-23T11:47:47.865 INFO:tasks.workunit.client.1.smithi111.stdout:4/887: dwrite d0/dc/d4c/d2a/df2/d123/da/f46 [4194304,4194304] 0 2024-03-23T11:47:47.866 INFO:tasks.workunit.client.1.smithi111.stdout:3/825: rename d0/d1a/d1d/d26/ce3 to d0/d1a/d2b/c110 0 2024-03-23T11:47:47.871 INFO:tasks.workunit.client.0.smithi063.stdout:0/769: truncate d2/d14/d36/d77/d94/da6/d28/d46/d51/f82 3722580 0 2024-03-23T11:47:47.872 INFO:tasks.workunit.client.1.smithi111.stdout:2/855: dwrite d2/d7/da/f103 [0,4194304] 0 2024-03-23T11:47:47.874 INFO:tasks.workunit.client.1.smithi111.stdout:5/909: dwrite d1/d20/d5c/f7e [0,4194304] 0 2024-03-23T11:47:47.876 INFO:tasks.workunit.client.0.smithi063.stdout:0/770: write d2/d14/d36/d77/d94/daf/fd2 [2977292,18988] 0 2024-03-23T11:47:47.876 INFO:tasks.workunit.client.1.smithi111.stdout:1/803: sync 2024-03-23T11:47:47.877 INFO:tasks.workunit.client.1.smithi111.stdout:9/873: sync 2024-03-23T11:47:47.877 INFO:tasks.workunit.client.1.smithi111.stdout:4/888: creat d0/d95/d9b/db3/f124 x:0 0 0 2024-03-23T11:47:47.878 INFO:tasks.workunit.client.1.smithi111.stdout:5/910: fdatasync d1/d3/d3a/fd6 0 2024-03-23T11:47:47.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:47 smithi063 ceph-mon[35189]: pgmap v43: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 203 MiB/s wr, 435 op/s 2024-03-23T11:47:47.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:47 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:47.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:47 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:47.884 INFO:tasks.workunit.client.1.smithi111.stdout:9/874: fdatasync d0/d4/d1c/d2d/f2e 0 2024-03-23T11:47:47.890 INFO:tasks.workunit.client.1.smithi111.stdout:5/911: truncate d1/d3/d6/d7/f64 641101 0 2024-03-23T11:47:47.894 INFO:tasks.workunit.client.1.smithi111.stdout:2/856: creat d2/de/d12/f12f x:0 0 0 2024-03-23T11:47:47.894 INFO:tasks.workunit.client.1.smithi111.stdout:5/912: dread - d1/d20/f118 zero size 2024-03-23T11:47:47.896 INFO:tasks.workunit.client.1.smithi111.stdout:2/857: fdatasync d2/de/d6c/f65 0 2024-03-23T11:47:47.899 INFO:tasks.workunit.client.0.smithi063.stdout:0/771: write d2/d14/d36/d77/d94/da6/d28/fc5 [342924,75957] 0 2024-03-23T11:47:47.901 INFO:tasks.workunit.client.0.smithi063.stdout:0/772: readlink d2/l29 0 2024-03-23T11:47:47.903 INFO:tasks.workunit.client.1.smithi111.stdout:5/913: readlink d1/d3/d6/d8b/d59/ld5 0 2024-03-23T11:47:47.903 INFO:tasks.workunit.client.1.smithi111.stdout:4/889: link d0/fc6 d0/dc/d4c/d2a/d97/f125 0 2024-03-23T11:47:47.907 INFO:tasks.workunit.client.1.smithi111.stdout:1/804: write d0/d5/dd/d1e/d63/fb6 [2738468,90520] 0 2024-03-23T11:47:47.909 INFO:tasks.workunit.client.0.smithi063.stdout:0/773: write d2/d14/d36/d77/fe5 [116285,69204] 0 2024-03-23T11:47:47.910 INFO:tasks.workunit.client.1.smithi111.stdout:2/858: link d2/d32/ffc d2/d11/d17/d3e/d5d/d8c/f130 0 2024-03-23T11:47:47.913 INFO:tasks.workunit.client.1.smithi111.stdout:5/914: unlink d1/d3/d6/d7/f10 0 2024-03-23T11:47:47.913 INFO:tasks.workunit.client.1.smithi111.stdout:9/875: dwrite d0/fbc [0,4194304] 0 2024-03-23T11:47:47.914 INFO:tasks.workunit.client.0.smithi063.stdout:0/774: stat d2/d4/c26 0 2024-03-23T11:47:47.914 INFO:tasks.workunit.client.1.smithi111.stdout:1/805: fdatasync d0/d2b/d37/f4f 0 2024-03-23T11:47:47.917 INFO:tasks.workunit.client.1.smithi111.stdout:2/859: chown d2/d11/d17/d3e/d5d/d8c/ca5 50774 1 2024-03-23T11:47:47.917 INFO:tasks.workunit.client.1.smithi111.stdout:3/826: dwrite d0/d1a/d1d/d26/d5a/f10b [0,4194304] 0 2024-03-23T11:47:47.918 INFO:tasks.workunit.client.0.smithi063.stdout:0/775: write d2/d14/d36/d77/d94/da6/d28/fdf [64065,85889] 0 2024-03-23T11:47:47.923 INFO:tasks.workunit.client.0.smithi063.stdout:0/776: dread - d2/d14/d36/d77/d94/da6/d28/d46/f10e zero size 2024-03-23T11:47:47.926 INFO:tasks.workunit.client.1.smithi111.stdout:4/890: getdents d0/d17 0 2024-03-23T11:47:47.926 INFO:tasks.workunit.client.1.smithi111.stdout:1/806: write d0/d5/dd/d1e/d54/fc2 [1064117,53950] 0 2024-03-23T11:47:47.929 INFO:tasks.workunit.client.0.smithi063.stdout:0/777: rmdir d2/d14/d36/d77/d94/da6/d25/d93/dc9 39 2024-03-23T11:47:47.931 INFO:tasks.workunit.client.1.smithi111.stdout:1/807: creat d0/d2b/d7c/f113 x:0 0 0 2024-03-23T11:47:47.932 INFO:tasks.workunit.client.1.smithi111.stdout:2/860: mkdir d2/d7/da/d19/d131 0 2024-03-23T11:47:47.932 INFO:tasks.workunit.client.0.smithi063.stdout:0/778: write d2/d48/fa2 [1531182,62601] 0 2024-03-23T11:47:47.935 INFO:tasks.workunit.client.1.smithi111.stdout:3/827: mknod d0/d1a/d2b/d10e/d19/d91/c111 0 2024-03-23T11:47:47.937 INFO:tasks.workunit.client.1.smithi111.stdout:9/876: dwrite d0/d4/d10/ff2 [0,4194304] 0 2024-03-23T11:47:47.937 INFO:tasks.workunit.client.1.smithi111.stdout:2/861: creat d2/d32/d129/f132 x:0 0 0 2024-03-23T11:47:47.939 INFO:tasks.workunit.client.0.smithi063.stdout:0/779: mknod d2/d14/d36/d77/c114 0 2024-03-23T11:47:47.941 INFO:tasks.workunit.client.1.smithi111.stdout:1/808: rename d0/d2b/c8f to d0/d5/dd/d1e/d54/c114 0 2024-03-23T11:47:47.941 INFO:tasks.workunit.client.0.smithi063.stdout:0/780: chown d2/d12/d35/f81 4140 1 2024-03-23T11:47:47.943 INFO:tasks.workunit.client.1.smithi111.stdout:3/828: mkdir d0/d1a/d4f/d90/d112 0 2024-03-23T11:47:47.945 INFO:tasks.workunit.client.0.smithi063.stdout:0/781: mkdir d2/d6e/d115 0 2024-03-23T11:47:47.946 INFO:tasks.workunit.client.0.smithi063.stdout:0/782: readlink d2/d12/l2a 0 2024-03-23T11:47:47.946 INFO:tasks.workunit.client.1.smithi111.stdout:2/862: mkdir d2/d11/d17/d3e/d5d/dc2/d133 0 2024-03-23T11:47:47.948 INFO:tasks.workunit.client.1.smithi111.stdout:1/809: creat d0/d2b/de3/f115 x:0 0 0 2024-03-23T11:47:47.950 INFO:tasks.workunit.client.1.smithi111.stdout:4/891: dwrite d0/dc/d4c/d2a/d97/d4d/f10e [0,4194304] 0 2024-03-23T11:47:47.951 INFO:tasks.workunit.client.0.smithi063.stdout:0/783: chown d2/d4/l8d 863 1 2024-03-23T11:47:47.953 INFO:tasks.workunit.client.1.smithi111.stdout:2/863: creat d2/de/d12/d70/f134 x:0 0 0 2024-03-23T11:47:47.958 INFO:tasks.workunit.client.1.smithi111.stdout:1/810: mkdir d0/d5/dd/d1e/d63/d116 0 2024-03-23T11:47:47.962 INFO:tasks.workunit.client.1.smithi111.stdout:4/892: rename d0/f1 to d0/dc/d4c/d71/f126 0 2024-03-23T11:47:47.962 INFO:tasks.workunit.client.0.smithi063.stdout:0/784: creat d2/d14/d36/d77/d94/da6/d1f/d4f/f116 x:0 0 0 2024-03-23T11:47:47.968 INFO:tasks.workunit.client.1.smithi111.stdout:4/893: truncate d0/d17/fde 343828 0 2024-03-23T11:47:47.968 INFO:tasks.workunit.client.0.smithi063.stdout:0/785: readlink d2/d14/d36/d77/d94/da6/lc8 0 2024-03-23T11:47:47.971 INFO:tasks.workunit.client.1.smithi111.stdout:3/829: dread d0/d1a/d2b/d38/d7f/f9c [0,4194304] 0 2024-03-23T11:47:47.971 INFO:tasks.workunit.client.1.smithi111.stdout:9/877: dread d0/d4/d10/f51 [0,4194304] 0 2024-03-23T11:47:47.972 INFO:tasks.workunit.client.1.smithi111.stdout:1/811: dread d0/d6/f34 [0,4194304] 0 2024-03-23T11:47:47.974 INFO:tasks.workunit.client.1.smithi111.stdout:9/878: dread - d0/d4/d1c/d2d/d6e/ff1 zero size 2024-03-23T11:47:47.975 INFO:tasks.workunit.client.0.smithi063.stdout:0/786: chown d2/d14/d36/d77/d94/da6/d28/d46/f5b 993 1 2024-03-23T11:47:47.979 INFO:tasks.workunit.client.1.smithi111.stdout:4/894: link d0/dc/d4c/f121 d0/d95/db5/d108/d109/f127 0 2024-03-23T11:47:47.979 INFO:tasks.workunit.client.1.smithi111.stdout:6/831: sync 2024-03-23T11:47:47.980 INFO:tasks.workunit.client.1.smithi111.stdout:1/812: creat d0/d6/d48/d4d/dab/dbd/f117 x:0 0 0 2024-03-23T11:47:47.981 INFO:tasks.workunit.client.0.smithi063.stdout:0/787: write d2/d14/d36/d77/d94/daf/ffb [774829,115534] 0 2024-03-23T11:47:47.981 INFO:tasks.workunit.client.1.smithi111.stdout:4/895: dread - d0/dc/d89/da4/fd3 zero size 2024-03-23T11:47:47.983 INFO:tasks.workunit.client.1.smithi111.stdout:6/832: chown d2/d1f/d45/d86/fd1 24381998 1 2024-03-23T11:47:47.984 INFO:tasks.workunit.client.1.smithi111.stdout:4/896: stat d0/dc/d30/ff 0 2024-03-23T11:47:47.985 INFO:tasks.workunit.client.0.smithi063.stdout:0/788: mkdir d2/d6e/d117 0 2024-03-23T11:47:47.989 INFO:tasks.workunit.client.1.smithi111.stdout:3/830: dwrite d0/d1a/d1d/d26/d5a/fef [0,4194304] 0 2024-03-23T11:47:47.992 INFO:tasks.workunit.client.1.smithi111.stdout:2/864: dwrite d2/d11/f37 [0,4194304] 0 2024-03-23T11:47:47.993 INFO:tasks.workunit.client.1.smithi111.stdout:4/897: creat d0/d95/db5/f128 x:0 0 0 2024-03-23T11:47:48.002 INFO:tasks.workunit.client.1.smithi111.stdout:6/833: truncate d2/d1a/d1c/d2c/d51/f2f 168453 0 2024-03-23T11:47:48.003 INFO:tasks.workunit.client.1.smithi111.stdout:9/879: write d0/d4/d1c/d2d/d3b/d42/d4f/f5c [2057683,33992] 0 2024-03-23T11:47:48.003 INFO:tasks.workunit.client.1.smithi111.stdout:1/813: dwrite d0/d2b/d69/f8c [0,4194304] 0 2024-03-23T11:47:48.004 INFO:tasks.workunit.client.1.smithi111.stdout:1/814: dread - d0/ff0 zero size 2024-03-23T11:47:48.006 INFO:tasks.workunit.client.1.smithi111.stdout:4/898: mknod d0/dc/d4c/dad/c129 0 2024-03-23T11:47:48.007 INFO:tasks.workunit.client.1.smithi111.stdout:4/899: stat d0/dc/l61 0 2024-03-23T11:47:48.011 INFO:tasks.workunit.client.1.smithi111.stdout:2/865: chown d2/d11/d17/d3e/d6b/d2e/d76/fa7 495630642 1 2024-03-23T11:47:48.016 INFO:tasks.workunit.client.0.smithi063.stdout:0/789: dwrite d2/d12/f68 [4194304,4194304] 0 2024-03-23T11:47:48.021 INFO:tasks.workunit.client.1.smithi111.stdout:9/880: symlink d0/d31/d41/d9d/dd4/de0/l117 0 2024-03-23T11:47:48.024 INFO:tasks.workunit.client.1.smithi111.stdout:4/900: chown d0/dc/d4c/d2a/d97/f125 14 1 2024-03-23T11:47:48.024 INFO:tasks.workunit.client.0.smithi063.stdout:0/790: mkdir d2/d14/d36/d77/d94/da6/d28/d46/d51/d118 0 2024-03-23T11:47:48.025 INFO:tasks.workunit.client.0.smithi063.stdout:0/791: chown d2/d4/f8a 85095617 1 2024-03-23T11:47:48.026 INFO:tasks.workunit.client.1.smithi111.stdout:3/831: dwrite d0/d1a/d2b/d38/dc2/fc6 [0,4194304] 0 2024-03-23T11:47:48.027 INFO:tasks.workunit.client.1.smithi111.stdout:2/866: symlink d2/d128/l135 0 2024-03-23T11:47:48.029 INFO:tasks.workunit.client.1.smithi111.stdout:1/815: mkdir d0/d5/dd/d1e/d54/dbb/d118 0 2024-03-23T11:47:48.029 INFO:tasks.workunit.client.1.smithi111.stdout:2/867: truncate d2/fbf 959857 0 2024-03-23T11:47:48.030 INFO:tasks.workunit.client.0.smithi063.stdout:0/792: creat d2/d6e/d115/f119 x:0 0 0 2024-03-23T11:47:48.031 INFO:tasks.workunit.client.1.smithi111.stdout:2/868: readlink d2/d5c/l7a 0 2024-03-23T11:47:48.034 INFO:tasks.workunit.client.1.smithi111.stdout:2/869: truncate d2/d11/d17/d1f/fd9 1173354 0 2024-03-23T11:47:48.035 INFO:tasks.workunit.client.1.smithi111.stdout:6/834: dwrite d2/d1f/dce/fe1 [0,4194304] 0 2024-03-23T11:47:48.040 INFO:tasks.workunit.client.1.smithi111.stdout:3/832: creat d0/db9/f113 x:0 0 0 2024-03-23T11:47:48.041 INFO:tasks.workunit.client.1.smithi111.stdout:1/816: creat d0/d2b/d7c/dca/f119 x:0 0 0 2024-03-23T11:47:48.043 INFO:tasks.workunit.client.1.smithi111.stdout:6/835: symlink d2/d1a/d1c/d2c/l10e 0 2024-03-23T11:47:48.043 INFO:tasks.workunit.client.0.smithi063.stdout:0/793: link d2/d14/d36/d77/d94/da6/d1f/c40 d2/d14/d36/d77/d94/da6/d28/c11a 0 2024-03-23T11:47:48.045 INFO:tasks.workunit.client.1.smithi111.stdout:2/870: creat d2/d11/d17/d3e/d5d/dc2/d133/f136 x:0 0 0 2024-03-23T11:47:48.045 INFO:tasks.workunit.client.1.smithi111.stdout:1/817: chown d0/d6/d48/d4a/d6c 121080 1 2024-03-23T11:47:48.048 INFO:tasks.workunit.client.1.smithi111.stdout:1/818: chown d0/c26 15862177 1 2024-03-23T11:47:48.049 INFO:tasks.workunit.client.1.smithi111.stdout:9/881: creat d0/d31/f118 x:0 0 0 2024-03-23T11:47:48.050 INFO:tasks.workunit.client.1.smithi111.stdout:7/831: sync 2024-03-23T11:47:48.050 INFO:tasks.workunit.client.1.smithi111.stdout:0/819: sync 2024-03-23T11:47:48.055 INFO:tasks.workunit.client.1.smithi111.stdout:2/871: creat d2/d11/d17/d3e/d6b/d9d/d52/dba/f137 x:0 0 0 2024-03-23T11:47:48.055 INFO:tasks.workunit.client.1.smithi111.stdout:1/819: fsync d0/d5/fae 0 2024-03-23T11:47:48.058 INFO:tasks.workunit.client.1.smithi111.stdout:0/820: dread - d5/db/d1f/d32/fe8 zero size 2024-03-23T11:47:48.059 INFO:tasks.workunit.client.1.smithi111.stdout:1/820: readlink d0/d5/lc4 0 2024-03-23T11:47:48.062 INFO:tasks.workunit.client.1.smithi111.stdout:1/821: chown d0/d2b/d37/c53 4195 1 2024-03-23T11:47:48.064 INFO:tasks.workunit.client.1.smithi111.stdout:3/833: dread d0/d1a/d2b/d38/d7f/faa [0,4194304] 0 2024-03-23T11:47:48.065 INFO:tasks.workunit.client.1.smithi111.stdout:0/821: mknod d5/db/d1f/de4/c114 0 2024-03-23T11:47:48.068 INFO:tasks.workunit.client.1.smithi111.stdout:4/901: dwrite d0/dc/d4c/d2a/d97/d4d/f54 [0,4194304] 0 2024-03-23T11:47:48.073 INFO:tasks.workunit.client.1.smithi111.stdout:2/872: truncate d2/d32/ffc 563893 0 2024-03-23T11:47:48.074 INFO:tasks.workunit.client.1.smithi111.stdout:6/836: dread d2/d1f/d45/f56 [0,4194304] 0 2024-03-23T11:47:48.074 INFO:tasks.workunit.client.0.smithi063.stdout:0/794: dwrite d2/d14/d36/d77/d94/da6/d28/d46/d51/f82 [4194304,4194304] 0 2024-03-23T11:47:48.077 INFO:tasks.workunit.client.0.smithi063.stdout:0/795: dread - d2/d6e/d8c/f9e zero size 2024-03-23T11:47:48.077 INFO:tasks.workunit.client.1.smithi111.stdout:4/902: write d0/d95/d9b/db3/db6/fa3 [4568980,63705] 0 2024-03-23T11:47:48.078 INFO:tasks.workunit.client.1.smithi111.stdout:6/837: chown d2/d1f/d45/d49/d8d/f9d 297086 1 2024-03-23T11:47:48.082 INFO:tasks.workunit.client.0.smithi063.stdout:0/796: fsync d2/d14/d36/d77/d94/da6/d28/d46/f5b 0 2024-03-23T11:47:48.085 INFO:tasks.workunit.client.0.smithi063.stdout:0/797: read - d2/d6e/d8c/fae zero size 2024-03-23T11:47:48.088 INFO:tasks.workunit.client.0.smithi063.stdout:0/798: creat d2/d14/d36/f11b x:0 0 0 2024-03-23T11:47:48.092 INFO:tasks.workunit.client.1.smithi111.stdout:3/834: dread d0/d1a/d1d/f43 [0,4194304] 0 2024-03-23T11:47:48.098 INFO:tasks.workunit.client.0.smithi063.stdout:0/799: link d2/d48/de2/lfc d2/d12/d32/d6d/l11c 0 2024-03-23T11:47:48.100 INFO:tasks.workunit.client.1.smithi111.stdout:7/832: dread d14/d37/d43/f55 [0,4194304] 0 2024-03-23T11:47:48.104 INFO:tasks.workunit.client.0.smithi063.stdout:0/800: dread d2/d12/d32/d6d/f74 [0,4194304] 0 2024-03-23T11:47:48.116 INFO:tasks.workunit.client.1.smithi111.stdout:9/882: link d0/d4/d10/d29/c3d d0/d4/d1c/d2d/d75/c119 0 2024-03-23T11:47:48.119 INFO:tasks.workunit.client.1.smithi111.stdout:2/873: mkdir d2/d11/d17/d1f/d30/d138 0 2024-03-23T11:47:48.121 INFO:tasks.workunit.client.1.smithi111.stdout:4/903: read - d0/dc/d4c/d2a/df2/d123/da/f111 zero size 2024-03-23T11:47:48.121 INFO:tasks.workunit.client.1.smithi111.stdout:3/835: mknod d0/d1a/d2b/d10e/d19/d5e/c114 0 2024-03-23T11:47:48.123 INFO:tasks.workunit.client.1.smithi111.stdout:4/904: stat d0/dc/d30/ff6 0 2024-03-23T11:47:48.123 INFO:tasks.workunit.client.1.smithi111.stdout:8/671: sync 2024-03-23T11:47:48.124 INFO:tasks.workunit.client.1.smithi111.stdout:5/915: sync 2024-03-23T11:47:48.124 INFO:tasks.workunit.client.1.smithi111.stdout:3/836: write d0/d1a/d2b/d38/dd4/ffe [530328,112608] 0 2024-03-23T11:47:48.125 INFO:tasks.workunit.client.1.smithi111.stdout:7/833: creat d14/d1b/d1e/d2c/d42/d9c/dd3/ff5 x:0 0 0 2024-03-23T11:47:48.126 INFO:tasks.workunit.client.1.smithi111.stdout:8/672: chown d5/f1f 58974 1 2024-03-23T11:47:48.127 INFO:tasks.workunit.client.1.smithi111.stdout:5/916: chown d1/d3/d6/d8b/d59/d74/f73 32216504 1 2024-03-23T11:47:48.128 INFO:tasks.workunit.client.1.smithi111.stdout:1/822: link d0/d5/dd/d1e/d54/d87/dd9/f10e d0/def/f11a 0 2024-03-23T11:47:48.129 INFO:tasks.workunit.client.1.smithi111.stdout:8/673: mknod d5/d32/d9b/cdd 0 2024-03-23T11:47:48.130 INFO:tasks.workunit.client.0.smithi063.stdout:0/801: dread d2/d12/d32/fe6 [0,4194304] 0 2024-03-23T11:47:48.132 INFO:tasks.workunit.client.0.smithi063.stdout:0/802: dread - d2/d12/f57 zero size 2024-03-23T11:47:48.133 INFO:tasks.workunit.client.1.smithi111.stdout:8/674: truncate d5/d26/d31/d5e/f65 591257 0 2024-03-23T11:47:48.134 INFO:tasks.workunit.client.1.smithi111.stdout:2/874: chown d2/d11/d17/f21 879500 1 2024-03-23T11:47:48.136 INFO:tasks.workunit.client.1.smithi111.stdout:8/675: creat d5/d26/d2f/d42/dd3/fde x:0 0 0 2024-03-23T11:47:48.138 INFO:tasks.workunit.client.1.smithi111.stdout:7/834: write d14/fba [1902140,3595] 0 2024-03-23T11:47:48.141 INFO:tasks.workunit.client.0.smithi063.stdout:0/803: dread d2/d14/d36/d77/d94/da6/d28/fc5 [0,4194304] 0 2024-03-23T11:47:48.142 INFO:tasks.workunit.client.1.smithi111.stdout:3/837: creat d0/d53/dfa/d103/f115 x:0 0 0 2024-03-23T11:47:48.145 INFO:tasks.workunit.client.0.smithi063.stdout:0/804: getdents d2/d6e/d117 0 2024-03-23T11:47:48.147 INFO:tasks.workunit.client.0.smithi063.stdout:0/805: chown d2/d14/d36/le1 83 1 2024-03-23T11:47:48.147 INFO:tasks.workunit.client.1.smithi111.stdout:8/676: dread d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/f8e [0,4194304] 0 2024-03-23T11:47:48.148 INFO:tasks.workunit.client.1.smithi111.stdout:5/917: rename d1/d3/d6/d7/d63/c38 to d1/d110/c12f 0 2024-03-23T11:47:48.151 INFO:tasks.workunit.client.1.smithi111.stdout:2/875: creat d2/d11/d17/d3e/d5d/dc0/df7/d10e/f139 x:0 0 0 2024-03-23T11:47:48.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:47 smithi111 ceph-mon[44005]: pgmap v43: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 203 MiB/s wr, 435 op/s 2024-03-23T11:47:48.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:47 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:48.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:47 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:47:48.157 INFO:tasks.workunit.client.1.smithi111.stdout:5/918: rmdir d1/d3/d6/d7/d8c 39 2024-03-23T11:47:48.157 INFO:tasks.workunit.client.0.smithi063.stdout:0/806: write d2/d12/f7e [3204638,95530] 0 2024-03-23T11:47:48.161 INFO:tasks.workunit.client.1.smithi111.stdout:2/876: rmdir d2/de/d12 39 2024-03-23T11:47:48.162 INFO:tasks.workunit.client.1.smithi111.stdout:8/677: truncate d5/d26/d31/d3b/d55/dd0/f4e 2182545 0 2024-03-23T11:47:48.162 INFO:tasks.workunit.client.0.smithi063.stdout:0/807: read - d2/d14/d36/d77/d94/fd5 zero size 2024-03-23T11:47:48.166 INFO:tasks.workunit.client.1.smithi111.stdout:9/883: getdents d0/d4/d1c/d2d/d75 0 2024-03-23T11:47:48.166 INFO:tasks.workunit.client.1.smithi111.stdout:8/678: creat d5/d26/d2f/d62/d75/d4b/d51/d5a/fdf x:0 0 0 2024-03-23T11:47:48.169 INFO:tasks.workunit.client.1.smithi111.stdout:4/905: link d0/dc/d89/f99 d0/dc/d4c/d2a/df2/dfc/f12a 0 2024-03-23T11:47:48.170 INFO:tasks.workunit.client.1.smithi111.stdout:2/877: mknod d2/d7/da/dbd/c13a 0 2024-03-23T11:47:48.172 INFO:tasks.workunit.client.1.smithi111.stdout:8/679: mkdir d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/de0 0 2024-03-23T11:47:48.173 INFO:tasks.workunit.client.1.smithi111.stdout:3/838: link d0/d1a/d2b/d10e/d19/d5e/le5 d0/d1a/d2b/d10e/d19/d5e/da6/l116 0 2024-03-23T11:47:48.173 INFO:tasks.workunit.client.1.smithi111.stdout:8/680: chown d5/d26/d31/d4d/fc4 19 1 2024-03-23T11:47:48.176 INFO:tasks.workunit.client.1.smithi111.stdout:8/681: write d5/d1b/fc9 [1219379,27132] 0 2024-03-23T11:47:48.177 INFO:tasks.workunit.client.1.smithi111.stdout:9/884: rename d0/d4/d1c/d2d/d3b/lc4 to d0/d4/d10/d29/l11a 0 2024-03-23T11:47:48.179 INFO:tasks.workunit.client.0.smithi063.stdout:0/808: symlink d2/d14/d36/d77/d94/da6/d25/d93/dc9/ddd/l11d 0 2024-03-23T11:47:48.182 INFO:tasks.workunit.client.1.smithi111.stdout:8/682: symlink d5/d26/le1 0 2024-03-23T11:47:48.183 INFO:tasks.workunit.client.1.smithi111.stdout:4/906: rename d0/dc/l74 to d0/d95/db5/dc7/l12b 0 2024-03-23T11:47:48.183 INFO:tasks.workunit.client.1.smithi111.stdout:3/839: read d0/d1a/d2b/d10e/f23 [1064616,89673] 0 2024-03-23T11:47:48.183 INFO:tasks.workunit.client.0.smithi063.stdout:0/809: creat d2/d6e/f11e x:0 0 0 2024-03-23T11:47:48.184 INFO:tasks.workunit.client.1.smithi111.stdout:0/822: write d5/db/d1f/d19/dbd/f62 [169433,79125] 0 2024-03-23T11:47:48.186 INFO:tasks.workunit.client.0.smithi063.stdout:0/810: write d2/d12/d32/f91 [6594658,1793] 0 2024-03-23T11:47:48.187 INFO:tasks.workunit.client.1.smithi111.stdout:8/683: mknod d5/d32/ce2 0 2024-03-23T11:47:48.189 INFO:tasks.workunit.client.1.smithi111.stdout:0/823: dread - d5/db/d1f/d47/de0/ff9 zero size 2024-03-23T11:47:48.189 INFO:tasks.workunit.client.1.smithi111.stdout:4/907: rmdir d0/dc/d4c/d2a/d97/d4d 39 2024-03-23T11:47:48.192 INFO:tasks.workunit.client.1.smithi111.stdout:3/840: symlink d0/d1a/d2b/d10e/l117 0 2024-03-23T11:47:48.193 INFO:tasks.workunit.client.1.smithi111.stdout:8/684: mknod d5/d26/d2f/d62/d75/d4b/ce3 0 2024-03-23T11:47:48.195 INFO:tasks.workunit.client.1.smithi111.stdout:7/835: write d14/d1b/d1e/d2c/f38 [5699425,93176] 0 2024-03-23T11:47:48.196 INFO:tasks.workunit.client.1.smithi111.stdout:1/823: write d0/f1a [840490,36273] 0 2024-03-23T11:47:48.200 INFO:tasks.workunit.client.1.smithi111.stdout:6/838: dwrite d2/d1f/d45/f63 [0,4194304] 0 2024-03-23T11:47:48.202 INFO:tasks.workunit.client.1.smithi111.stdout:8/685: stat d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71 0 2024-03-23T11:47:48.206 INFO:tasks.workunit.client.1.smithi111.stdout:9/885: rename d0/d4/d10/d29/l11a to d0/d4/d1c/d2d/l11b 0 2024-03-23T11:47:48.206 INFO:tasks.workunit.client.1.smithi111.stdout:2/878: dread d2/d7/da/d19/f49 [0,4194304] 0 2024-03-23T11:47:48.210 INFO:tasks.workunit.client.1.smithi111.stdout:6/839: creat d2/d73/f10f x:0 0 0 2024-03-23T11:47:48.211 INFO:tasks.workunit.client.1.smithi111.stdout:7/836: symlink d14/d1b/d1e/d2c/d2e/d73/dae/dea/lf6 0 2024-03-23T11:47:48.211 INFO:tasks.workunit.client.1.smithi111.stdout:1/824: write d0/d6/d48/d4a/ff2 [5104951,105135] 0 2024-03-23T11:47:48.212 INFO:tasks.workunit.client.1.smithi111.stdout:5/919: dwrite d1/d3/d6/d8b/d59/ffe [0,4194304] 0 2024-03-23T11:47:48.212 INFO:tasks.workunit.client.1.smithi111.stdout:8/686: chown d5/d26/f74 6348 1 2024-03-23T11:47:48.215 INFO:tasks.workunit.client.1.smithi111.stdout:6/840: chown d2/lfb 3257906 1 2024-03-23T11:47:48.218 INFO:tasks.workunit.client.1.smithi111.stdout:8/687: chown d5/d26/d2f/d8f 11852 1 2024-03-23T11:47:48.222 INFO:tasks.workunit.client.1.smithi111.stdout:9/886: rename d0/d31/d41/d71/d8f/def/ff5 to d0/d31/d41/f11c 0 2024-03-23T11:47:48.225 INFO:tasks.workunit.client.1.smithi111.stdout:1/825: truncate d0/f1 4144522 0 2024-03-23T11:47:48.229 INFO:tasks.workunit.client.1.smithi111.stdout:6/841: unlink d2/d1f/d45/d9a/l10c 0 2024-03-23T11:47:48.230 INFO:tasks.workunit.client.1.smithi111.stdout:3/841: dwrite d0/d1a/d2b/fb1 [0,4194304] 0 2024-03-23T11:47:48.231 INFO:tasks.workunit.client.1.smithi111.stdout:2/879: truncate d2/d11/d17/d3e/d6b/d9d/d52/f9b 863522 0 2024-03-23T11:47:48.231 INFO:tasks.workunit.client.1.smithi111.stdout:5/920: getdents d1/d3/d6/d8b/d59/d11e 0 2024-03-23T11:47:48.233 INFO:tasks.workunit.client.1.smithi111.stdout:1/826: mknod d0/d2b/de3/de7/c11b 0 2024-03-23T11:47:48.235 INFO:tasks.workunit.client.1.smithi111.stdout:3/842: readlink d0/d1a/d1d/lb4 0 2024-03-23T11:47:48.235 INFO:tasks.workunit.client.1.smithi111.stdout:6/842: chown d2/d1a/d1c/d2c/d51/f8f 12374 1 2024-03-23T11:47:48.238 INFO:tasks.workunit.client.1.smithi111.stdout:1/827: chown d0/d5/l29 87680440 1 2024-03-23T11:47:48.242 INFO:tasks.workunit.client.1.smithi111.stdout:4/908: dread d0/d95/d9b/db3/db6/fce [4194304,4194304] 0 2024-03-23T11:47:48.245 INFO:tasks.workunit.client.1.smithi111.stdout:7/837: truncate d14/d37/d43/f55 1301091 0 2024-03-23T11:47:48.248 INFO:tasks.workunit.client.1.smithi111.stdout:5/921: mkdir d1/d3/d6/d7/d63/d2c/d43/d95/d130 0 2024-03-23T11:47:48.248 INFO:tasks.workunit.client.1.smithi111.stdout:9/887: truncate d0/d4/d10/d80/fd8 3251967 0 2024-03-23T11:47:48.251 INFO:tasks.workunit.client.1.smithi111.stdout:6/843: mkdir d2/d1f/d45/d9a/ddd/d110 0 2024-03-23T11:47:48.255 INFO:tasks.workunit.client.1.smithi111.stdout:1/828: rename d0/fe to d0/d6/f11c 0 2024-03-23T11:47:48.256 INFO:tasks.workunit.client.1.smithi111.stdout:7/838: mknod d14/d1b/d1e/d2c/cf7 0 2024-03-23T11:47:48.262 INFO:tasks.workunit.client.1.smithi111.stdout:9/888: rename d0/d4/d1c/d2d/d3b/d42/d4f/f61 to d0/d31/d35/f11d 0 2024-03-23T11:47:48.265 INFO:tasks.workunit.client.1.smithi111.stdout:4/909: dread d0/dc/d4c/d2a/d97/d4d/fbe [0,4194304] 0 2024-03-23T11:47:48.269 INFO:tasks.workunit.client.1.smithi111.stdout:3/843: getdents d0/d1a/d2b/d10e/d19/d91/da3/dc8 0 2024-03-23T11:47:48.271 INFO:tasks.workunit.client.1.smithi111.stdout:5/922: dread d1/d3/d6/d7/d63/d2c/d43/f10f [0,4194304] 0 2024-03-23T11:47:48.272 INFO:tasks.workunit.client.1.smithi111.stdout:2/880: dwrite d2/d11/d17/d3e/dee/f10b [0,4194304] 0 2024-03-23T11:47:48.273 INFO:tasks.workunit.client.1.smithi111.stdout:3/844: dread - d0/d1a/d2b/d10e/d19/d84/fe4 zero size 2024-03-23T11:47:48.275 INFO:tasks.workunit.client.1.smithi111.stdout:2/881: chown d2/d7/fd 135 1 2024-03-23T11:47:48.275 INFO:tasks.workunit.client.1.smithi111.stdout:6/844: rename d2/d35/f77 to d2/f111 0 2024-03-23T11:47:48.278 INFO:tasks.workunit.client.1.smithi111.stdout:5/923: fsync d1/d3/d6/d7/d63/d2c/d82/fd4 0 2024-03-23T11:47:48.282 INFO:tasks.workunit.client.1.smithi111.stdout:6/845: truncate d2/d73/f10f 258181 0 2024-03-23T11:47:48.283 INFO:tasks.workunit.client.1.smithi111.stdout:7/839: rename d14/d37/d43/cc8 to d14/d1b/d1e/d2c/d42/d5e/d88/d5f/cf8 0 2024-03-23T11:47:48.287 INFO:tasks.workunit.client.1.smithi111.stdout:3/845: dread d0/d1a/d2b/d10e/f82 [0,4194304] 0 2024-03-23T11:47:48.289 INFO:tasks.workunit.client.1.smithi111.stdout:5/924: creat d1/d3/d6/d7/d7b/f131 x:0 0 0 2024-03-23T11:47:48.290 INFO:tasks.workunit.client.1.smithi111.stdout:6/846: readlink d2/d35/la3 0 2024-03-23T11:47:48.293 INFO:tasks.workunit.client.1.smithi111.stdout:1/829: dwrite d0/f1 [0,4194304] 0 2024-03-23T11:47:48.295 INFO:tasks.workunit.client.1.smithi111.stdout:4/910: dread d0/dc/d1f/f57 [0,4194304] 0 2024-03-23T11:47:48.295 INFO:tasks.workunit.client.1.smithi111.stdout:6/847: chown d2/d1f/d45/fc1 6132 1 2024-03-23T11:47:48.298 INFO:tasks.workunit.client.1.smithi111.stdout:6/848: readlink d2/lfb 0 2024-03-23T11:47:48.299 INFO:tasks.workunit.client.1.smithi111.stdout:9/889: dwrite d0/d4/d1c/d2d/d3b/d42/d4f/d95/ff0 [0,4194304] 0 2024-03-23T11:47:48.303 INFO:tasks.workunit.client.1.smithi111.stdout:3/846: dread d0/d72/ff2 [0,4194304] 0 2024-03-23T11:47:48.304 INFO:tasks.workunit.client.1.smithi111.stdout:9/890: write d0/d31/d41/d71/d8f/fa9 [2572832,1386] 0 2024-03-23T11:47:48.306 INFO:tasks.workunit.client.1.smithi111.stdout:5/925: symlink d1/d3/d6/d30/db2/dcc/d10b/l132 0 2024-03-23T11:47:48.306 INFO:tasks.workunit.client.1.smithi111.stdout:6/849: symlink d2/d17/l112 0 2024-03-23T11:47:48.308 INFO:tasks.workunit.client.1.smithi111.stdout:3/847: symlink d0/d72/l118 0 2024-03-23T11:47:48.308 INFO:tasks.workunit.client.1.smithi111.stdout:4/911: mknod d0/dc/d4c/d2a/d97/d4d/de3/d105/c12c 0 2024-03-23T11:47:48.311 INFO:tasks.workunit.client.1.smithi111.stdout:9/891: mknod d0/d4/d10/d80/c11e 0 2024-03-23T11:47:48.315 INFO:tasks.workunit.client.1.smithi111.stdout:5/926: creat d1/d3/d6/d7/d63/d2c/f133 x:0 0 0 2024-03-23T11:47:48.315 INFO:tasks.workunit.client.1.smithi111.stdout:4/912: fdatasync d0/dc/d4c/d2a/d97/d4f/f96 0 2024-03-23T11:47:48.317 INFO:tasks.workunit.client.1.smithi111.stdout:3/848: mknod d0/d1a/d1d/d26/d5a/d64/c119 0 2024-03-23T11:47:48.320 INFO:tasks.workunit.client.1.smithi111.stdout:5/927: chown d1/d3/d6/d30/de6/f117 4319 1 2024-03-23T11:47:48.321 INFO:tasks.workunit.client.1.smithi111.stdout:6/850: write d2/d1a/d1c/d2c/f8c [5008400,27057] 0 2024-03-23T11:47:48.321 INFO:tasks.workunit.client.1.smithi111.stdout:7/840: dread d14/d1b/d1e/fa5 [0,4194304] 0 2024-03-23T11:47:48.322 INFO:tasks.workunit.client.1.smithi111.stdout:1/830: dwrite d0/d5/dd/d1e/f40 [0,4194304] 0 2024-03-23T11:47:48.325 INFO:tasks.workunit.client.1.smithi111.stdout:4/913: rename d0/dc/d4c/dad/cef to d0/dc/d4c/d83/c12d 0 2024-03-23T11:47:48.331 INFO:tasks.workunit.client.1.smithi111.stdout:3/849: truncate d0/d1a/d2b/d10e/d19/d91/da3/ffd 927514 0 2024-03-23T11:47:48.331 INFO:tasks.workunit.client.1.smithi111.stdout:3/850: readlink d0/lab 0 2024-03-23T11:47:48.334 INFO:tasks.workunit.client.1.smithi111.stdout:6/851: mknod d2/d1f/d45/d9a/ddd/d110/c113 0 2024-03-23T11:47:48.338 INFO:tasks.workunit.client.1.smithi111.stdout:7/841: rename d14/d1b/d1e/d2c/f38 to d14/d44/d70/d7e/ff9 0 2024-03-23T11:47:48.340 INFO:tasks.workunit.client.1.smithi111.stdout:6/852: unlink d2/d1f/c6a 0 2024-03-23T11:47:48.340 INFO:tasks.workunit.client.1.smithi111.stdout:5/928: truncate d1/d3/d6/d7/d63/d2c/d82/f88 253475 0 2024-03-23T11:47:48.343 INFO:tasks.workunit.client.1.smithi111.stdout:0/824: sync 2024-03-23T11:47:48.344 INFO:tasks.workunit.client.1.smithi111.stdout:7/842: dread - d14/d1b/d1e/d2c/d2e/d73/dae/d74/fe3 zero size 2024-03-23T11:47:48.345 INFO:tasks.workunit.client.1.smithi111.stdout:0/825: stat d5/db/d1f/d32/fe8 0 2024-03-23T11:47:48.350 INFO:tasks.workunit.client.1.smithi111.stdout:7/843: stat d14/d1b/d1e/d2c/d2e/d73/dae/d74/fbd 0 2024-03-23T11:47:48.351 INFO:tasks.workunit.client.1.smithi111.stdout:5/929: fsync d1/d3/d6/f2b 0 2024-03-23T11:47:48.356 INFO:tasks.workunit.client.1.smithi111.stdout:1/831: dwrite d0/d6/d48/d4a/ff2 [0,4194304] 0 2024-03-23T11:47:48.360 INFO:tasks.workunit.client.1.smithi111.stdout:0/826: mkdir d5/db/d4d/da6/d10b/d115 0 2024-03-23T11:47:48.360 INFO:tasks.workunit.client.1.smithi111.stdout:3/851: dwrite d0/d1a/d4f/d90/f10d [0,4194304] 0 2024-03-23T11:47:48.368 INFO:tasks.workunit.client.1.smithi111.stdout:5/930: chown d1/d3/d6/d8b/d59/d74/f4d 163085 1 2024-03-23T11:47:48.371 INFO:tasks.workunit.client.1.smithi111.stdout:9/892: dread d0/d4/d1c/d2d/f2e [0,4194304] 0 2024-03-23T11:47:48.375 INFO:tasks.workunit.client.1.smithi111.stdout:4/914: dwrite d0/dc/d4c/d2a/d97/d4d/f51 [4194304,4194304] 0 2024-03-23T11:47:48.375 INFO:tasks.workunit.client.1.smithi111.stdout:1/832: rename d0/d5/dd to d0/d6/d48/d4a/d6c/d83/d11d 0 2024-03-23T11:47:48.380 INFO:tasks.workunit.client.1.smithi111.stdout:6/853: dwrite d2/d1a/d106/d108/fca [0,4194304] 0 2024-03-23T11:47:48.386 INFO:tasks.workunit.client.1.smithi111.stdout:9/893: creat d0/d31/d41/df4/f11f x:0 0 0 2024-03-23T11:47:48.386 INFO:tasks.workunit.client.1.smithi111.stdout:5/931: symlink d1/d3/d6/d7/d7b/l134 0 2024-03-23T11:47:48.391 INFO:tasks.workunit.client.1.smithi111.stdout:0/827: link d5/db/d1f/de4/l97 d5/db/d4d/da6/dcc/l116 0 2024-03-23T11:47:48.391 INFO:tasks.workunit.client.1.smithi111.stdout:9/894: readlink d0/d31/d35/l62 0 2024-03-23T11:47:48.391 INFO:tasks.workunit.client.1.smithi111.stdout:5/932: chown d1/d3/d6/d30/db2/ce2 5 1 2024-03-23T11:47:48.391 INFO:tasks.workunit.client.1.smithi111.stdout:1/833: dread d0/d2b/d7c/f9f [0,4194304] 0 2024-03-23T11:47:48.391 INFO:tasks.workunit.client.1.smithi111.stdout:4/915: creat d0/dc/d89/f12e x:0 0 0 2024-03-23T11:47:48.391 INFO:tasks.workunit.client.1.smithi111.stdout:9/895: write d0/d4/d10/f2c [2817860,15477] 0 2024-03-23T11:47:48.393 INFO:tasks.workunit.client.1.smithi111.stdout:7/844: dread d14/d1b/d1e/d2c/d2e/d33/f90 [0,4194304] 0 2024-03-23T11:47:48.398 INFO:tasks.workunit.client.1.smithi111.stdout:4/916: write d0/dc/d4c/d2a/d97/d4d/f54 [1887361,108273] 0 2024-03-23T11:47:48.399 INFO:tasks.workunit.client.1.smithi111.stdout:1/834: fdatasync d0/d6/f34 0 2024-03-23T11:47:48.403 INFO:tasks.workunit.client.1.smithi111.stdout:6/854: dwrite d2/d1a/d1c/f6b [0,4194304] 0 2024-03-23T11:47:48.403 INFO:tasks.workunit.client.1.smithi111.stdout:9/896: mknod d0/d4/d6a/d9a/c120 0 2024-03-23T11:47:48.408 INFO:tasks.workunit.client.1.smithi111.stdout:8/688: sync 2024-03-23T11:47:48.410 INFO:tasks.workunit.client.1.smithi111.stdout:0/828: link d5/db/d1f/d48/f66 d5/db/d1f/d47/d87/f117 0 2024-03-23T11:47:48.413 INFO:tasks.workunit.client.1.smithi111.stdout:4/917: creat d0/dc/d4c/d2a/d97/d4f/d72/f12f x:0 0 0 2024-03-23T11:47:48.416 INFO:tasks.workunit.client.1.smithi111.stdout:5/933: rmdir d1/d3/d6/d7/d63/d2c/d43/d95/d130 0 2024-03-23T11:47:48.417 INFO:tasks.workunit.client.1.smithi111.stdout:6/855: mkdir d2/d1a/d1c/ddf/d114 0 2024-03-23T11:47:48.424 INFO:tasks.workunit.client.1.smithi111.stdout:0/829: readlink d5/db/d4d/da6/dcc/l116 0 2024-03-23T11:47:48.427 INFO:tasks.workunit.client.1.smithi111.stdout:6/856: symlink d2/d35/dc5/l115 0 2024-03-23T11:47:48.427 INFO:tasks.workunit.client.1.smithi111.stdout:3/852: dread d0/d1a/d2b/f4d [0,4194304] 0 2024-03-23T11:47:48.429 INFO:tasks.workunit.client.1.smithi111.stdout:9/897: write d0/d4/d10/f8b [1640534,68709] 0 2024-03-23T11:47:48.431 INFO:tasks.workunit.client.1.smithi111.stdout:7/845: dwrite d14/d1b/d1e/d2c/d42/d9c/dd3/ff5 [0,4194304] 0 2024-03-23T11:47:48.431 INFO:tasks.workunit.client.1.smithi111.stdout:6/857: readlink d2/d35/dc5/l115 0 2024-03-23T11:47:48.434 INFO:tasks.workunit.client.1.smithi111.stdout:5/934: creat d1/d3/d6/d8b/d59/d12e/f135 x:0 0 0 2024-03-23T11:47:48.440 INFO:tasks.workunit.client.1.smithi111.stdout:9/898: chown d0/d31/d41/df4 12 1 2024-03-23T11:47:48.440 INFO:tasks.workunit.client.1.smithi111.stdout:8/689: write d5/d26/d31/d5e/f65 [601769,63830] 0 2024-03-23T11:47:48.444 INFO:tasks.workunit.client.1.smithi111.stdout:1/835: dwrite d0/d2b/d7c/f113 [0,4194304] 0 2024-03-23T11:47:48.447 INFO:tasks.workunit.client.1.smithi111.stdout:6/858: fsync d2/d35/f44 0 2024-03-23T11:47:48.451 INFO:tasks.workunit.client.1.smithi111.stdout:2/882: sync 2024-03-23T11:47:48.456 INFO:tasks.workunit.client.1.smithi111.stdout:5/935: mknod d1/d3/d6/d7/d63/c136 0 2024-03-23T11:47:48.456 INFO:tasks.workunit.client.1.smithi111.stdout:9/899: mkdir d0/d4/d10/d29/d7f/d121 0 2024-03-23T11:47:48.456 INFO:tasks.workunit.client.1.smithi111.stdout:7/846: write d14/d1b/d1e/d2c/d2e/d73/dae/d74/fe3 [42861,28441] 0 2024-03-23T11:47:48.456 INFO:tasks.workunit.client.1.smithi111.stdout:4/918: truncate d0/dc/d4c/d2a/d97/d4d/f94 4106379 0 2024-03-23T11:47:48.459 INFO:tasks.workunit.client.1.smithi111.stdout:2/883: mknod d2/d5c/c13b 0 2024-03-23T11:47:48.460 INFO:tasks.workunit.client.1.smithi111.stdout:6/859: chown d2/d1f/d45/f56 1734 1 2024-03-23T11:47:48.461 INFO:tasks.workunit.client.1.smithi111.stdout:5/936: mkdir d1/d20/dce/d137 0 2024-03-23T11:47:48.464 INFO:tasks.workunit.client.1.smithi111.stdout:3/853: dwrite d0/d1a/d1d/f24 [4194304,4194304] 0 2024-03-23T11:47:48.464 INFO:tasks.workunit.client.1.smithi111.stdout:9/900: mkdir d0/d4/d10/d29/d7f/d121/d122 0 2024-03-23T11:47:48.464 INFO:tasks.workunit.client.1.smithi111.stdout:4/919: stat d0/dc/d4c/d2a/d97/d4d/f54 0 2024-03-23T11:47:48.468 INFO:tasks.workunit.client.1.smithi111.stdout:6/860: chown d2/d1a/d1c/f93 0 1 2024-03-23T11:47:48.469 INFO:tasks.workunit.client.1.smithi111.stdout:8/690: link d5/d26/d2f/d62/d75/d4b/d51/d5a/l50 d5/d26/d2f/d42/le4 0 2024-03-23T11:47:48.472 INFO:tasks.workunit.client.1.smithi111.stdout:3/854: dread - d0/d53/fc0 zero size 2024-03-23T11:47:48.472 INFO:tasks.workunit.client.1.smithi111.stdout:0/830: dwrite d5/db/d1f/d47/d59/f7d [4194304,4194304] 0 2024-03-23T11:47:48.472 INFO:tasks.workunit.client.1.smithi111.stdout:6/861: fdatasync d2/f41 0 2024-03-23T11:47:48.475 INFO:tasks.workunit.client.1.smithi111.stdout:5/937: rmdir d1/d3/d6/d7/d63/d2c/d43/dfc 39 2024-03-23T11:47:48.477 INFO:tasks.workunit.client.1.smithi111.stdout:1/836: dwrite d0/d2b/d37/d6a/fe4 [0,4194304] 0 2024-03-23T11:47:48.480 INFO:tasks.workunit.client.1.smithi111.stdout:9/901: creat d0/d31/d41/d71/d8f/def/f123 x:0 0 0 2024-03-23T11:47:48.481 INFO:tasks.workunit.client.1.smithi111.stdout:3/855: read d0/d1a/d2b/d10e/d19/f29 [2972367,78646] 0 2024-03-23T11:47:48.484 INFO:tasks.workunit.client.1.smithi111.stdout:9/902: chown d0/d4/d10/d29/d7f/f8a 45 1 2024-03-23T11:47:48.484 INFO:tasks.workunit.client.1.smithi111.stdout:2/884: truncate d2/d11/d17/d3e/d6b/d2e/d3f/f44 2080740 0 2024-03-23T11:47:48.485 INFO:tasks.workunit.client.1.smithi111.stdout:3/856: read d0/d1a/d2b/d10e/fa7 [160795,26279] 0 2024-03-23T11:47:48.486 INFO:tasks.workunit.client.1.smithi111.stdout:4/920: creat d0/dc/d89/da4/f130 x:0 0 0 2024-03-23T11:47:48.487 INFO:tasks.workunit.client.1.smithi111.stdout:8/691: write d5/d26/d2f/d62/d75/f6b [588555,38272] 0 2024-03-23T11:47:48.491 INFO:tasks.workunit.client.1.smithi111.stdout:7/847: write d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fb9 [758099,11356] 0 2024-03-23T11:47:48.491 INFO:tasks.workunit.client.1.smithi111.stdout:6/862: mkdir d2/d1a/d1c/ddf/d116 0 2024-03-23T11:47:48.501 INFO:tasks.workunit.client.1.smithi111.stdout:3/857: rmdir d0/d1a/d4f/d90 39 2024-03-23T11:47:48.502 INFO:tasks.workunit.client.1.smithi111.stdout:5/938: chown d1/d3/d6/d8b/d59/d6c/f76 111515 1 2024-03-23T11:47:48.502 INFO:tasks.workunit.client.1.smithi111.stdout:2/885: rmdir d2/d11/d17/d3e/d5d/d8c 39 2024-03-23T11:47:48.503 INFO:tasks.workunit.client.1.smithi111.stdout:9/903: truncate d0/d4/d10/d29/fa3 734576 0 2024-03-23T11:47:48.505 INFO:tasks.workunit.client.1.smithi111.stdout:5/939: write d1/d3/d6/d7/f1f [4480207,125908] 0 2024-03-23T11:47:48.505 INFO:tasks.workunit.client.1.smithi111.stdout:6/863: rmdir d2/d35/d70/d98 39 2024-03-23T11:47:48.505 INFO:tasks.workunit.client.1.smithi111.stdout:7/848: chown d14/f36 0 1 2024-03-23T11:47:48.514 INFO:tasks.workunit.client.1.smithi111.stdout:0/831: dwrite d5/db/d1f/f43 [0,4194304] 0 2024-03-23T11:47:48.514 INFO:tasks.workunit.client.1.smithi111.stdout:1/837: write d0/d6/d48/d4a/d6c/d83/d11d/d91/f9d [240722,84662] 0 2024-03-23T11:47:48.515 INFO:tasks.workunit.client.1.smithi111.stdout:3/858: mkdir d0/d1a/d2d/df5/d11a 0 2024-03-23T11:47:48.517 INFO:tasks.workunit.client.1.smithi111.stdout:1/838: read d0/d2b/d69/f8c [2895125,57243] 0 2024-03-23T11:47:48.517 INFO:tasks.workunit.client.1.smithi111.stdout:9/904: rename d0/ca to d0/d4/d1c/d2d/d3b/d42/d92/c124 0 2024-03-23T11:47:48.519 INFO:tasks.workunit.client.1.smithi111.stdout:0/832: fsync d5/db/d1f/d19/dbd/fa5 0 2024-03-23T11:47:48.521 INFO:tasks.workunit.client.1.smithi111.stdout:8/692: dwrite d5/d1b/f3a [0,4194304] 0 2024-03-23T11:47:48.523 INFO:tasks.workunit.client.1.smithi111.stdout:7/849: fsync d14/d44/d4b/fbf 0 2024-03-23T11:47:48.529 INFO:tasks.workunit.client.1.smithi111.stdout:4/921: truncate d0/dc/d4c/d2a/f70 1423998 0 2024-03-23T11:47:48.529 INFO:tasks.workunit.client.1.smithi111.stdout:3/859: creat d0/d1a/d2b/d10e/d19/f11b x:0 0 0 2024-03-23T11:47:48.533 INFO:tasks.workunit.client.1.smithi111.stdout:6/864: rename d2/d1a/d1c/d2c/c7b to d2/d1f/d45/d9a/ddd/d110/c117 0 2024-03-23T11:47:48.536 INFO:tasks.workunit.client.1.smithi111.stdout:8/693: mkdir d5/d26/d2f/d62/d75/d4b/de5 0 2024-03-23T11:47:48.536 INFO:tasks.workunit.client.1.smithi111.stdout:6/865: write d2/d1f/dce/ff8 [406690,121621] 0 2024-03-23T11:47:48.537 INFO:tasks.workunit.client.1.smithi111.stdout:1/839: symlink d0/d6/d48/d4a/d6c/d96/l11e 0 2024-03-23T11:47:48.537 INFO:tasks.workunit.client.1.smithi111.stdout:5/940: mknod d1/d3/d6/d7/d7b/d10d/c138 0 2024-03-23T11:47:48.541 INFO:tasks.workunit.client.1.smithi111.stdout:0/833: dwrite d5/db/d1f/d19/dbd/d64/d91/ddd/f10d [0,4194304] 0 2024-03-23T11:47:48.546 INFO:tasks.workunit.client.1.smithi111.stdout:8/694: write d5/d26/d2f/d8f/fb7 [253848,64450] 0 2024-03-23T11:47:48.547 INFO:tasks.workunit.client.1.smithi111.stdout:1/840: chown d0/d5/dd5/df6 700906390 1 2024-03-23T11:47:48.551 INFO:tasks.workunit.client.1.smithi111.stdout:3/860: rename d0/d1a/c44 to d0/d1a/d2b/d38/dc2/c11c 0 2024-03-23T11:47:48.551 INFO:tasks.workunit.client.1.smithi111.stdout:9/905: link d0/d31/d41/f11c d0/d31/da0/f125 0 2024-03-23T11:47:48.551 INFO:tasks.workunit.client.1.smithi111.stdout:6/866: truncate d2/d17/fac 5327374 0 2024-03-23T11:47:48.554 INFO:tasks.workunit.client.1.smithi111.stdout:4/922: mkdir d0/dc/d4c/d2a/df2/d123/d131 0 2024-03-23T11:47:48.554 INFO:tasks.workunit.client.1.smithi111.stdout:6/867: fdatasync d2/d1a/d1c/d2c/fa5 0 2024-03-23T11:47:48.555 INFO:tasks.workunit.client.1.smithi111.stdout:6/868: readlink d2/d17/l19 0 2024-03-23T11:47:48.555 INFO:tasks.workunit.client.1.smithi111.stdout:3/861: read d0/d1a/d2d/fbb [3047772,10159] 0 2024-03-23T11:47:48.556 INFO:tasks.workunit.client.1.smithi111.stdout:8/695: creat d5/d26/d31/fe6 x:0 0 0 2024-03-23T11:47:48.558 INFO:tasks.workunit.client.1.smithi111.stdout:7/850: write d14/d37/f3f [2294982,61068] 0 2024-03-23T11:47:48.558 INFO:tasks.workunit.client.1.smithi111.stdout:0/834: mknod d5/db/d1f/d19/dbd/d64/d91/ddd/c118 0 2024-03-23T11:47:48.560 INFO:tasks.workunit.client.1.smithi111.stdout:1/841: creat d0/def/f11f x:0 0 0 2024-03-23T11:47:48.561 INFO:tasks.workunit.client.1.smithi111.stdout:8/696: creat d5/d26/d2f/d42/d63/fe7 x:0 0 0 2024-03-23T11:47:48.563 INFO:tasks.workunit.client.1.smithi111.stdout:9/906: symlink d0/d4/l126 0 2024-03-23T11:47:48.566 INFO:tasks.workunit.client.1.smithi111.stdout:6/869: mkdir d2/d1a/d1c/d2c/d51/def/d118 0 2024-03-23T11:47:48.567 INFO:tasks.workunit.client.1.smithi111.stdout:3/862: symlink d0/d1a/d2b/d38/l11d 0 2024-03-23T11:47:48.568 INFO:tasks.workunit.client.1.smithi111.stdout:7/851: symlink d14/d1b/d1e/d2c/d2e/d73/dae/d74/lfa 0 2024-03-23T11:47:48.578 INFO:tasks.workunit.client.1.smithi111.stdout:4/923: dwrite d0/dc/d89/da4/f114 [0,4194304] 0 2024-03-23T11:47:48.580 INFO:tasks.workunit.client.1.smithi111.stdout:5/941: dread d1/d20/d5c/d99/dbf/dc3/fda [0,4194304] 0 2024-03-23T11:47:48.584 INFO:tasks.workunit.client.1.smithi111.stdout:1/842: dread d0/d5/f23 [0,4194304] 0 2024-03-23T11:47:48.597 INFO:tasks.workunit.client.1.smithi111.stdout:0/835: fdatasync d5/db/d1f/f18 0 2024-03-23T11:47:48.609 INFO:tasks.workunit.client.1.smithi111.stdout:2/886: sync 2024-03-23T11:47:48.614 INFO:tasks.workunit.client.1.smithi111.stdout:6/870: rename d2/d17/fac to d2/d1a/d1c/d2c/d51/def/f119 0 2024-03-23T11:47:48.618 INFO:tasks.workunit.client.1.smithi111.stdout:2/887: symlink d2/d5c/l13c 0 2024-03-23T11:47:48.618 INFO:tasks.workunit.client.1.smithi111.stdout:1/843: truncate d0/d2b/d37/d6a/f8d 777551 0 2024-03-23T11:47:48.620 INFO:tasks.workunit.client.1.smithi111.stdout:9/907: write d0/d4/d1c/d2d/d75/fce [188675,4882] 0 2024-03-23T11:47:48.622 INFO:tasks.workunit.client.1.smithi111.stdout:3/863: write d0/d53/fc0 [556746,88008] 0 2024-03-23T11:47:48.623 INFO:tasks.workunit.client.1.smithi111.stdout:6/871: fsync d2/d35/f6f 0 2024-03-23T11:47:48.623 INFO:tasks.workunit.client.1.smithi111.stdout:7/852: link d14/d1b/d1e/d2c/d2e/d33/l78 d14/d1b/d1e/d2c/d42/d9c/lfb 0 2024-03-23T11:47:48.629 INFO:tasks.workunit.client.1.smithi111.stdout:3/864: stat d0/d1a/d2b/fb1 0 2024-03-23T11:47:48.630 INFO:tasks.workunit.client.1.smithi111.stdout:8/697: dread d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/f52 [0,4194304] 0 2024-03-23T11:47:48.633 INFO:tasks.workunit.client.1.smithi111.stdout:1/844: mknod d0/d6/d48/d4a/d6c/d83/d11d/d91/c120 0 2024-03-23T11:47:48.633 INFO:tasks.workunit.client.1.smithi111.stdout:9/908: write d0/d31/da0/f125 [359760,21146] 0 2024-03-23T11:47:48.634 INFO:tasks.workunit.client.1.smithi111.stdout:2/888: write d2/d11/d17/d3e/d6b/f79 [4375737,69013] 0 2024-03-23T11:47:48.634 INFO:tasks.workunit.client.1.smithi111.stdout:8/698: chown d5/d26 1352 1 2024-03-23T11:47:48.634 INFO:tasks.workunit.client.1.smithi111.stdout:7/853: symlink d14/d1b/d1e/d2c/d42/lfc 0 2024-03-23T11:47:48.640 INFO:tasks.workunit.client.1.smithi111.stdout:7/854: fdatasync d14/fac 0 2024-03-23T11:47:48.642 INFO:tasks.workunit.client.1.smithi111.stdout:1/845: creat d0/dda/f121 x:0 0 0 2024-03-23T11:47:48.643 INFO:tasks.workunit.client.1.smithi111.stdout:3/865: dread d0/d1a/d1d/fb8 [0,4194304] 0 2024-03-23T11:47:48.650 INFO:tasks.workunit.client.1.smithi111.stdout:5/942: dwrite d1/d3/d6/d7/d7b/fe3 [0,4194304] 0 2024-03-23T11:47:48.653 INFO:tasks.workunit.client.1.smithi111.stdout:7/855: rmdir d14/d37/d43 39 2024-03-23T11:47:48.653 INFO:tasks.workunit.client.1.smithi111.stdout:8/699: symlink d5/d26/d31/d3b/d55/le8 0 2024-03-23T11:47:48.659 INFO:tasks.workunit.client.1.smithi111.stdout:3/866: creat d0/d1a/d2d/df5/f11e x:0 0 0 2024-03-23T11:47:48.661 INFO:tasks.workunit.client.1.smithi111.stdout:0/836: dwrite d5/db/d1f/d47/d87/f117 [0,4194304] 0 2024-03-23T11:47:48.664 INFO:tasks.workunit.client.1.smithi111.stdout:7/856: mkdir d14/d1b/d1e/d2c/d2e/d33/d98/dfd 0 2024-03-23T11:47:48.667 INFO:tasks.workunit.client.1.smithi111.stdout:6/872: dwrite d2/f89 [0,4194304] 0 2024-03-23T11:47:48.670 INFO:tasks.workunit.client.1.smithi111.stdout:4/924: dwrite d0/dc/d4c/d2a/f70 [0,4194304] 0 2024-03-23T11:47:48.700 INFO:tasks.workunit.client.1.smithi111.stdout:2/889: dwrite d2/d7/f7b [0,4194304] 0 2024-03-23T11:47:48.704 INFO:tasks.workunit.client.1.smithi111.stdout:6/873: rename d2/d1f/d45/d49/d8d/fae to d2/d1f/d45/f11a 0 2024-03-23T11:47:48.710 INFO:tasks.workunit.client.1.smithi111.stdout:9/909: dwrite d0/d4/d1c/d2d/d75/fce [0,4194304] 0 2024-03-23T11:47:48.710 INFO:tasks.workunit.client.1.smithi111.stdout:2/890: mkdir d2/d13d 0 2024-03-23T11:47:48.716 INFO:tasks.workunit.client.1.smithi111.stdout:7/857: write d14/d1b/d1e/d2c/d42/f4a [10286303,15834] 0 2024-03-23T11:47:48.724 INFO:tasks.workunit.client.1.smithi111.stdout:5/943: dwrite d1/d3/d6/d7/f64 [0,4194304] 0 2024-03-23T11:47:48.727 INFO:tasks.workunit.client.1.smithi111.stdout:8/700: rename d5/d26/d31/d5e/f49 to d5/d26/d31/d3b/d55/dd0/fe9 0 2024-03-23T11:47:48.741 INFO:tasks.workunit.client.1.smithi111.stdout:4/925: dwrite d0/dc/d4c/d2a/d97/d4d/f51 [0,4194304] 0 2024-03-23T11:47:48.741 INFO:tasks.workunit.client.1.smithi111.stdout:1/846: dread d0/d6/d48/d4a/d6c/d83/d11d/dc6/fdc [0,4194304] 0 2024-03-23T11:47:48.743 INFO:tasks.workunit.client.1.smithi111.stdout:5/944: mkdir d1/d3/d6/d7/d63/dd9/d139 0 2024-03-23T11:47:48.747 INFO:tasks.workunit.client.1.smithi111.stdout:0/837: dwrite d5/db/d1f/d19/dbd/f76 [0,4194304] 0 2024-03-23T11:47:48.747 INFO:tasks.workunit.client.1.smithi111.stdout:1/847: fsync d0/def/f11f 0 2024-03-23T11:47:48.749 INFO:tasks.workunit.client.1.smithi111.stdout:3/867: dread d0/d1a/d2b/d10e/f23 [0,4194304] 0 2024-03-23T11:47:48.766 INFO:tasks.workunit.client.1.smithi111.stdout:4/926: write d0/dc/d4c/d2a/f70 [3449802,72928] 0 2024-03-23T11:47:48.769 INFO:tasks.workunit.client.1.smithi111.stdout:7/858: dwrite d14/d1b/d22/f29 [0,4194304] 0 2024-03-23T11:47:48.773 INFO:tasks.workunit.client.1.smithi111.stdout:8/701: rename d5/f1f to d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/dad/fea 0 2024-03-23T11:47:48.774 INFO:tasks.workunit.client.1.smithi111.stdout:5/945: creat d1/d3/d6/d30/de6/f13a x:0 0 0 2024-03-23T11:47:48.775 INFO:tasks.workunit.client.1.smithi111.stdout:0/838: mknod d5/db/d1f/d47/d106/d10f/c119 0 2024-03-23T11:47:48.777 INFO:tasks.workunit.client.1.smithi111.stdout:6/874: dwrite d2/f31 [0,4194304] 0 2024-03-23T11:47:48.781 INFO:tasks.workunit.client.1.smithi111.stdout:3/868: dread d0/d72/ff2 [0,4194304] 0 2024-03-23T11:47:48.781 INFO:tasks.workunit.client.1.smithi111.stdout:2/891: dwrite d2/d11/d17/d3e/d6b/d2e/ff1 [0,4194304] 0 2024-03-23T11:47:48.782 INFO:tasks.workunit.client.1.smithi111.stdout:4/927: creat d0/dc/d89/dc9/f132 x:0 0 0 2024-03-23T11:47:48.784 INFO:tasks.workunit.client.1.smithi111.stdout:7/859: creat d14/d44/d70/d7e/ffe x:0 0 0 2024-03-23T11:47:48.785 INFO:tasks.workunit.client.1.smithi111.stdout:9/910: truncate d0/d4/d10/f8b 2994373 0 2024-03-23T11:47:48.788 INFO:tasks.workunit.client.1.smithi111.stdout:2/892: fsync d2/d11/d17/d3e/d6b/d2e/ff3 0 2024-03-23T11:47:48.791 INFO:tasks.workunit.client.1.smithi111.stdout:5/946: mknod d1/d20/d5c/d99/dbf/c13b 0 2024-03-23T11:47:48.794 INFO:tasks.workunit.client.1.smithi111.stdout:5/947: chown d1/d3/d6/d8b/d59/d6c/df3/d123 505628391 1 2024-03-23T11:47:48.795 INFO:tasks.workunit.client.1.smithi111.stdout:6/875: dread d2/d1a/fe0 [0,4194304] 0 2024-03-23T11:47:48.795 INFO:tasks.workunit.client.1.smithi111.stdout:3/869: symlink d0/d53/dfa/d103/l11f 0 2024-03-23T11:47:48.796 INFO:tasks.workunit.client.1.smithi111.stdout:9/911: creat d0/f127 x:0 0 0 2024-03-23T11:47:48.796 INFO:tasks.workunit.client.1.smithi111.stdout:6/876: chown d2/d35 1021220 1 2024-03-23T11:47:48.800 INFO:tasks.workunit.client.1.smithi111.stdout:2/893: rmdir d2/de0 39 2024-03-23T11:47:48.802 INFO:tasks.workunit.client.1.smithi111.stdout:2/894: stat d2/d11/d17/d1f/d30 0 2024-03-23T11:47:48.810 INFO:tasks.workunit.client.1.smithi111.stdout:7/860: rename d14/d1b/d1e/fa5 to d14/d44/d4b/fff 0 2024-03-23T11:47:48.811 INFO:tasks.workunit.client.1.smithi111.stdout:0/839: getdents d5/db/d1f/d47/d59 0 2024-03-23T11:47:48.817 INFO:tasks.workunit.client.1.smithi111.stdout:6/877: link d2/d17/l19 d2/d80/d100/l11b 0 2024-03-23T11:47:48.818 INFO:tasks.workunit.client.1.smithi111.stdout:7/861: write d14/d37/d43/ff4 [154545,69934] 0 2024-03-23T11:47:48.822 INFO:tasks.workunit.client.1.smithi111.stdout:7/862: stat d14/d1b/d1e/d86/c8e 0 2024-03-23T11:47:48.824 INFO:tasks.workunit.client.1.smithi111.stdout:7/863: fsync d14/d1b/d1e/d2c/d42/d5e/d9e/fa2 0 2024-03-23T11:47:48.825 INFO:tasks.workunit.client.1.smithi111.stdout:9/912: rename d0/d4/d6a/d9a/faa to d0/d4/d10/d29/d7f/f128 0 2024-03-23T11:47:48.827 INFO:tasks.workunit.client.1.smithi111.stdout:9/913: fsync d0/d4/d1c/d36/d28/f10d 0 2024-03-23T11:47:48.830 INFO:tasks.workunit.client.1.smithi111.stdout:3/870: write d0/d72/f86 [547189,7763] 0 2024-03-23T11:47:48.831 INFO:tasks.workunit.client.1.smithi111.stdout:5/948: write d1/d20/d5c/dc5/f121 [2572603,81119] 0 2024-03-23T11:47:48.832 INFO:tasks.workunit.client.1.smithi111.stdout:2/895: write d2/de/d12/d70/d96/fce [739693,50928] 0 2024-03-23T11:47:48.837 INFO:tasks.workunit.client.1.smithi111.stdout:4/928: dwrite d0/dc/d4c/d83/f10b [4194304,4194304] 0 2024-03-23T11:47:48.838 INFO:tasks.workunit.client.1.smithi111.stdout:7/864: read d14/d37/d43/ff4 [85131,130898] 0 2024-03-23T11:47:48.841 INFO:tasks.workunit.client.1.smithi111.stdout:5/949: creat d1/d3/d6/d7/dee/f13c x:0 0 0 2024-03-23T11:47:48.843 INFO:tasks.workunit.client.1.smithi111.stdout:4/929: symlink d0/d95/d9b/db3/dcc/l133 0 2024-03-23T11:47:48.847 INFO:tasks.workunit.client.1.smithi111.stdout:4/930: stat d0/dc/d4c/d2a/d97/d4d/de3/d105 0 2024-03-23T11:47:48.848 INFO:tasks.workunit.client.1.smithi111.stdout:5/950: creat d1/d20/d5c/f13d x:0 0 0 2024-03-23T11:47:48.852 INFO:tasks.workunit.client.1.smithi111.stdout:4/931: read - d0/dc/d89/dc9/f132 zero size 2024-03-23T11:47:48.855 INFO:tasks.workunit.client.1.smithi111.stdout:5/951: write d1/d20/f118 [962083,75382] 0 2024-03-23T11:47:48.855 INFO:tasks.workunit.client.1.smithi111.stdout:2/896: dwrite d2/d11/d17/d3e/d6b/fb3 [0,4194304] 0 2024-03-23T11:47:48.857 INFO:tasks.workunit.client.1.smithi111.stdout:4/932: unlink d0/dc/d78/f98 0 2024-03-23T11:47:48.859 INFO:tasks.workunit.client.1.smithi111.stdout:5/952: chown d1/d20/d5c/dc5/f11c 701188963 1 2024-03-23T11:47:48.859 INFO:tasks.workunit.client.1.smithi111.stdout:4/933: readlink d0/dc/d78/lff 0 2024-03-23T11:47:48.860 INFO:tasks.workunit.client.1.smithi111.stdout:3/871: dwrite d0/d1a/d1d/d26/d5a/d64/fba [4194304,4194304] 0 2024-03-23T11:47:48.864 INFO:tasks.workunit.client.1.smithi111.stdout:5/953: chown d1/d3/d6/d30/db2/dcc 1461147 1 2024-03-23T11:47:48.867 INFO:tasks.workunit.client.1.smithi111.stdout:2/897: creat d2/de/d6c/dd1/dd6/f13e x:0 0 0 2024-03-23T11:47:48.868 INFO:tasks.workunit.client.1.smithi111.stdout:4/934: mkdir d0/dc/d1f/d5e/d134 0 2024-03-23T11:47:48.874 INFO:tasks.workunit.client.1.smithi111.stdout:4/935: write d0/dc/d4c/d2a/d97/d4f/dba/f118 [998387,6614] 0 2024-03-23T11:47:48.875 INFO:tasks.workunit.client.1.smithi111.stdout:2/898: chown d2/de/d6c/dd1/l10d 5359 1 2024-03-23T11:47:48.877 INFO:tasks.workunit.client.1.smithi111.stdout:5/954: creat d1/d3/d6/d7/d8c/f13e x:0 0 0 2024-03-23T11:47:48.888 INFO:tasks.workunit.client.1.smithi111.stdout:5/955: dread d1/d3/d6/d8b/d59/d6c/fc8 [0,4194304] 0 2024-03-23T11:47:48.888 INFO:tasks.workunit.client.1.smithi111.stdout:2/899: link d2/de/d12/f12f d2/de/d12/d70/f13f 0 2024-03-23T11:47:48.888 INFO:tasks.workunit.client.1.smithi111.stdout:1/848: sync 2024-03-23T11:47:48.890 INFO:tasks.workunit.client.1.smithi111.stdout:3/872: dwrite d0/fbe [0,4194304] 0 2024-03-23T11:47:48.895 INFO:tasks.workunit.client.1.smithi111.stdout:3/873: rmdir d0/d55 39 2024-03-23T11:47:48.895 INFO:tasks.workunit.client.1.smithi111.stdout:2/900: chown d2/d7/f7b 1278973115 1 2024-03-23T11:47:48.903 INFO:tasks.workunit.client.1.smithi111.stdout:3/874: write d0/d1a/d1d/d26/d5a/d64/f100 [56559,85078] 0 2024-03-23T11:47:48.904 INFO:tasks.workunit.client.1.smithi111.stdout:7/865: dread d14/d1b/d1e/d2c/d42/f45 [0,4194304] 0 2024-03-23T11:47:48.905 INFO:tasks.workunit.client.1.smithi111.stdout:4/936: dwrite d0/d17/fde [0,4194304] 0 2024-03-23T11:47:48.906 INFO:tasks.workunit.client.1.smithi111.stdout:5/956: chown d1/d3/d6/d7/d63/d2c/d43/dfc/laf 1096734 1 2024-03-23T11:47:48.906 INFO:tasks.workunit.client.1.smithi111.stdout:1/849: creat d0/d6/d48/d4a/d6c/d96/f122 x:0 0 0 2024-03-23T11:47:48.908 INFO:tasks.workunit.client.1.smithi111.stdout:7/866: read - d14/d1b/d1e/d86/f89 zero size 2024-03-23T11:47:48.913 INFO:tasks.workunit.client.1.smithi111.stdout:3/875: creat d0/db9/f120 x:0 0 0 2024-03-23T11:47:48.916 INFO:tasks.workunit.client.1.smithi111.stdout:4/937: unlink d0/dc/d89/da4/fbd 0 2024-03-23T11:47:48.924 INFO:tasks.workunit.client.1.smithi111.stdout:1/850: rename d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/f60 to d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/f123 0 2024-03-23T11:47:48.925 INFO:tasks.workunit.client.1.smithi111.stdout:3/876: mkdir d0/d1a/d2b/d38/d7f/dd0/d121 0 2024-03-23T11:47:48.925 INFO:tasks.workunit.client.1.smithi111.stdout:1/851: stat d0/d2b/d7c/dca/f119 0 2024-03-23T11:47:48.926 INFO:tasks.workunit.client.1.smithi111.stdout:8/702: read d5/f11 [331420,93522] 0 2024-03-23T11:47:48.926 INFO:tasks.workunit.client.1.smithi111.stdout:3/877: chown d0/d72/l118 49 1 2024-03-23T11:47:48.931 INFO:tasks.workunit.client.1.smithi111.stdout:7/867: link d14/d37/d54/fe4 d14/d1b/d1e/d2c/d8b/ddd/f100 0 2024-03-23T11:47:48.932 INFO:tasks.workunit.client.1.smithi111.stdout:2/901: getdents d2/d7/da/d19 0 2024-03-23T11:47:48.934 INFO:tasks.workunit.client.1.smithi111.stdout:5/957: write d1/d3/d6/d30/db2/f106 [1640537,86553] 0 2024-03-23T11:47:48.935 INFO:tasks.workunit.client.1.smithi111.stdout:3/878: symlink d0/db9/l122 0 2024-03-23T11:47:48.935 INFO:tasks.workunit.client.1.smithi111.stdout:5/958: fsync d1/d20/f118 0 2024-03-23T11:47:48.936 INFO:tasks.workunit.client.1.smithi111.stdout:4/938: symlink d0/dc/d1f/d5e/d134/l135 0 2024-03-23T11:47:48.938 INFO:tasks.workunit.client.1.smithi111.stdout:5/959: dread - d1/d3/d6/d7/d7b/f131 zero size 2024-03-23T11:47:48.938 INFO:tasks.workunit.client.1.smithi111.stdout:4/939: readlink d0/d95/db5/dc7/le8 0 2024-03-23T11:47:48.941 INFO:tasks.workunit.client.1.smithi111.stdout:2/902: mknod d2/de0/d110/c140 0 2024-03-23T11:47:48.942 INFO:tasks.workunit.client.1.smithi111.stdout:3/879: mkdir d0/d1a/d1d/d123 0 2024-03-23T11:47:48.943 INFO:tasks.workunit.client.1.smithi111.stdout:7/868: link d14/d1b/d22/ce8 d14/d1b/d1e/d2c/d42/d5e/dc5/c101 0 2024-03-23T11:47:48.943 INFO:tasks.workunit.client.1.smithi111.stdout:4/940: chown d0/d95/c117 2920545 1 2024-03-23T11:47:48.943 INFO:tasks.workunit.client.1.smithi111.stdout:3/880: stat d0/d1a/d2b/d38/d7f/f8a 0 2024-03-23T11:47:48.946 INFO:tasks.workunit.client.1.smithi111.stdout:5/960: creat d1/d20/dce/d137/f13f x:0 0 0 2024-03-23T11:47:48.947 INFO:tasks.workunit.client.1.smithi111.stdout:2/903: symlink d2/l141 0 2024-03-23T11:47:48.949 INFO:tasks.workunit.client.1.smithi111.stdout:2/904: chown d2/d11/d17/d3e/d5d/dc2 0 1 2024-03-23T11:47:48.949 INFO:tasks.workunit.client.1.smithi111.stdout:7/869: creat d14/d1b/d1e/d2c/d2e/d33/f102 x:0 0 0 2024-03-23T11:47:48.949 INFO:tasks.workunit.client.1.smithi111.stdout:1/852: link d0/d6/d48/d4a/d6c/d83/d11d/d1e/lf7 d0/d5/l124 0 2024-03-23T11:47:48.951 INFO:tasks.workunit.client.1.smithi111.stdout:5/961: creat d1/d3/d3a/f140 x:0 0 0 2024-03-23T11:47:48.951 INFO:tasks.workunit.client.1.smithi111.stdout:3/881: mknod d0/d1a/d2b/d38/d7f/dd0/d121/c124 0 2024-03-23T11:47:48.954 INFO:tasks.workunit.client.1.smithi111.stdout:1/853: chown d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/c93 184 1 2024-03-23T11:47:48.954 INFO:tasks.workunit.client.1.smithi111.stdout:7/870: unlink d14/d1b/d1e/d2c/d2e/d73/c79 0 2024-03-23T11:47:48.956 INFO:tasks.workunit.client.1.smithi111.stdout:7/871: chown d14/d44/d70/fe5 1164 1 2024-03-23T11:47:48.957 INFO:tasks.workunit.client.1.smithi111.stdout:2/905: rename d2/de/d12/f2c to d2/f142 0 2024-03-23T11:47:48.958 INFO:tasks.workunit.client.1.smithi111.stdout:3/882: symlink d0/d1a/d1d/d123/l125 0 2024-03-23T11:47:48.959 INFO:tasks.workunit.client.1.smithi111.stdout:7/872: stat d14/d1b/d1e/d2c/d2e/d73/dae/d74/fbd 0 2024-03-23T11:47:48.960 INFO:tasks.workunit.client.1.smithi111.stdout:7/873: fdatasync d14/d44/d70/f7c 0 2024-03-23T11:47:48.961 INFO:tasks.workunit.client.1.smithi111.stdout:5/962: rename d1/d3/d6/d7/dee/f13c to d1/d3/d6/d8b/d59/d6c/f141 0 2024-03-23T11:47:48.966 INFO:tasks.workunit.client.1.smithi111.stdout:1/854: rename d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/dbb/d118 to d0/d6/d48/d4a/de5/d125 0 2024-03-23T11:47:48.968 INFO:tasks.workunit.client.1.smithi111.stdout:3/883: rmdir d0/d72/d104 0 2024-03-23T11:47:48.969 INFO:tasks.workunit.client.1.smithi111.stdout:1/855: write d0/d6/d48/d4a/de5/fea [162589,10528] 0 2024-03-23T11:47:48.969 INFO:tasks.workunit.client.1.smithi111.stdout:7/874: mkdir d14/d1b/d1e/d2c/d42/dc1/deb/d103 0 2024-03-23T11:47:48.971 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:48 smithi063 ceph-mon[35189]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-03-23T11:47:48.971 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:48 smithi063 ceph-mon[35189]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-03-23T11:47:48.974 INFO:tasks.workunit.client.1.smithi111.stdout:1/856: truncate d0/d6/d48/d4a/d6c/d83/d11d/f101 252192 0 2024-03-23T11:47:48.975 INFO:tasks.workunit.client.1.smithi111.stdout:3/884: unlink d0/d1a/d2b/d38/l9a 0 2024-03-23T11:47:48.977 INFO:tasks.workunit.client.1.smithi111.stdout:7/875: fdatasync d14/d1b/d22/f4d 0 2024-03-23T11:47:48.984 INFO:tasks.workunit.client.1.smithi111.stdout:5/963: rename d1/d3/d3a/l3c to d1/d3/d6/d7/d63/d2c/d43/l142 0 2024-03-23T11:47:48.985 INFO:tasks.workunit.client.1.smithi111.stdout:7/876: rmdir d14/d1b/d1e/d2c/d2e/d33/d98 39 2024-03-23T11:47:48.987 INFO:tasks.workunit.client.1.smithi111.stdout:3/885: truncate d0/d1a/d1d/d26/d5a/d60/fac 542395 0 2024-03-23T11:47:48.989 INFO:tasks.workunit.client.1.smithi111.stdout:1/857: dread d0/d2b/d69/f8c [0,4194304] 0 2024-03-23T11:47:48.992 INFO:tasks.workunit.client.1.smithi111.stdout:8/703: truncate d5/d26/d31/d4d/f6c 806689 0 2024-03-23T11:47:48.999 INFO:tasks.workunit.client.1.smithi111.stdout:8/704: unlink d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/f8e 0 2024-03-23T11:47:49.002 INFO:tasks.workunit.client.1.smithi111.stdout:3/886: dwrite d0/d53/dfa/d103/f10c [0,4194304] 0 2024-03-23T11:47:49.003 INFO:tasks.workunit.client.1.smithi111.stdout:5/964: rename d1/d3/d6/d7/d63/dd9 to d1/d3/d6/d8b/d59/d74/d143 0 2024-03-23T11:47:49.003 INFO:tasks.workunit.client.1.smithi111.stdout:2/906: write d2/de/d12/d70/fb6 [5226984,108408] 0 2024-03-23T11:47:49.005 INFO:tasks.workunit.client.1.smithi111.stdout:7/877: getdents d14/d1b/d1e/d2c/d42/d9c 0 2024-03-23T11:47:49.007 INFO:tasks.workunit.client.1.smithi111.stdout:3/887: rename d0/d1a/d1d/d26/d5a to d0/d1a/d1d/d26/d5a/d60/d102/d126 22 2024-03-23T11:47:49.008 INFO:tasks.workunit.client.1.smithi111.stdout:0/840: sync 2024-03-23T11:47:49.011 INFO:tasks.workunit.client.1.smithi111.stdout:7/878: unlink d14/d1b/l1d 0 2024-03-23T11:47:49.018 INFO:tasks.workunit.client.1.smithi111.stdout:3/888: dread d0/d1a/d2b/d10e/fa7 [0,4194304] 0 2024-03-23T11:47:49.019 INFO:tasks.workunit.client.1.smithi111.stdout:7/879: creat d14/d44/d70/d7e/dbc/f104 x:0 0 0 2024-03-23T11:47:49.019 INFO:tasks.workunit.client.1.smithi111.stdout:2/907: dread d2/d11/d17/d3e/d5d/fa6 [0,4194304] 0 2024-03-23T11:47:49.021 INFO:tasks.workunit.client.1.smithi111.stdout:7/880: stat d14/d37/d43/db1/le9 0 2024-03-23T11:47:49.022 INFO:tasks.workunit.client.1.smithi111.stdout:0/841: write d5/db/d1f/d19/dbd/f8f [308768,83935] 0 2024-03-23T11:47:49.022 INFO:tasks.workunit.client.1.smithi111.stdout:2/908: rmdir d2 39 2024-03-23T11:47:49.023 INFO:tasks.workunit.client.1.smithi111.stdout:3/889: fdatasync d0/d1a/d4f/d90/fff 0 2024-03-23T11:47:49.024 INFO:tasks.workunit.client.1.smithi111.stdout:5/965: truncate d1/d3/d6/d8b/d59/ffe 2640511 0 2024-03-23T11:47:49.028 INFO:tasks.workunit.client.1.smithi111.stdout:2/909: chown d2/de/cda 2807 1 2024-03-23T11:47:49.028 INFO:tasks.workunit.client.1.smithi111.stdout:1/858: dwrite d0/d6/d48/d4a/d6c/d83/d11d/f10 [0,4194304] 0 2024-03-23T11:47:49.029 INFO:tasks.workunit.client.1.smithi111.stdout:0/842: creat d5/db/d1f/d48/f11a x:0 0 0 2024-03-23T11:47:49.031 INFO:tasks.workunit.client.1.smithi111.stdout:3/890: fdatasync d0/d1a/d2d/fd5 0 2024-03-23T11:47:49.037 INFO:tasks.workunit.client.1.smithi111.stdout:0/843: creat d5/db/d1f/d19/dbd/d64/f11b x:0 0 0 2024-03-23T11:47:49.041 INFO:tasks.workunit.client.1.smithi111.stdout:1/859: mknod d0/d6/d48/d4a/d6c/d83/d11d/d1e/d70/c126 0 2024-03-23T11:47:49.041 INFO:tasks.workunit.client.1.smithi111.stdout:2/910: truncate d2/d11/d17/d1f/da0/fd4 854506 0 2024-03-23T11:47:49.043 INFO:tasks.workunit.client.1.smithi111.stdout:3/891: symlink d0/d55/l127 0 2024-03-23T11:47:49.046 INFO:tasks.workunit.client.1.smithi111.stdout:7/881: dwrite d14/d1b/d1e/d2c/d42/d5e/d9e/fa2 [0,4194304] 0 2024-03-23T11:47:49.050 INFO:tasks.workunit.client.1.smithi111.stdout:2/911: creat d2/d7/da/d19/f143 x:0 0 0 2024-03-23T11:47:49.050 INFO:tasks.workunit.client.1.smithi111.stdout:1/860: symlink d0/d5/l127 0 2024-03-23T11:47:49.055 INFO:tasks.workunit.client.1.smithi111.stdout:0/844: link d5/db/d1f/d48/f11a d5/db/d1f/d47/f11c 0 2024-03-23T11:47:49.055 INFO:tasks.workunit.client.1.smithi111.stdout:3/892: mknod d0/d1a/d1d/d26/c128 0 2024-03-23T11:47:49.057 INFO:tasks.workunit.client.1.smithi111.stdout:5/966: dwrite d1/d3/d6/d7/d63/d2c/f52 [0,4194304] 0 2024-03-23T11:47:49.057 INFO:tasks.workunit.client.1.smithi111.stdout:1/861: mkdir d0/d5/d128 0 2024-03-23T11:47:49.059 INFO:tasks.workunit.client.1.smithi111.stdout:0/845: readlink d5/db/d1f/d19/dbd/d64/d91/ddd/df6/lfd 0 2024-03-23T11:47:49.061 INFO:tasks.workunit.client.1.smithi111.stdout:1/862: chown d0/d6/d48/d4a/d6c/leb 0 1 2024-03-23T11:47:49.064 INFO:tasks.workunit.client.1.smithi111.stdout:0/846: symlink d5/dc3/l11d 0 2024-03-23T11:47:49.064 INFO:tasks.workunit.client.1.smithi111.stdout:2/912: write d2/d7/da/d19/f143 [85306,103180] 0 2024-03-23T11:47:49.065 INFO:tasks.workunit.client.1.smithi111.stdout:7/882: write d14/d1b/d1e/d2c/d42/d5e/f85 [936057,55848] 0 2024-03-23T11:47:49.070 INFO:tasks.workunit.client.1.smithi111.stdout:3/893: write d0/d1a/d2b/fa1 [714555,16009] 0 2024-03-23T11:47:49.071 INFO:tasks.workunit.client.1.smithi111.stdout:7/883: read d14/d1b/d1e/d2c/d2e/d73/fc2 [580769,78378] 0 2024-03-23T11:47:49.071 INFO:tasks.workunit.client.1.smithi111.stdout:0/847: mknod d5/db/d1f/d47/d106/c11e 0 2024-03-23T11:47:49.080 INFO:tasks.workunit.client.1.smithi111.stdout:9/914: sync 2024-03-23T11:47:49.080 INFO:tasks.workunit.client.1.smithi111.stdout:4/941: sync 2024-03-23T11:47:49.080 INFO:tasks.workunit.client.1.smithi111.stdout:6/878: sync 2024-03-23T11:47:49.082 INFO:tasks.workunit.client.1.smithi111.stdout:5/967: rename d1/d3/d6/d8b/d59/d6c/f76 to d1/d3/d6/f144 0 2024-03-23T11:47:49.082 INFO:tasks.workunit.client.1.smithi111.stdout:3/894: readlink d0/d1a/d2b/d38/dc2/ld6 0 2024-03-23T11:47:49.083 INFO:tasks.workunit.client.1.smithi111.stdout:4/942: chown d0/cec 74 1 2024-03-23T11:47:49.097 INFO:tasks.workunit.client.1.smithi111.stdout:3/895: dread d0/d1a/d2b/d10e/d19/f29 [0,4194304] 0 2024-03-23T11:47:49.099 INFO:tasks.workunit.client.1.smithi111.stdout:7/884: unlink d14/d1b/d22/c71 0 2024-03-23T11:47:49.102 INFO:tasks.workunit.client.1.smithi111.stdout:1/863: rename d0/ce0 to d0/d5/dd5/c129 0 2024-03-23T11:47:49.103 INFO:tasks.workunit.client.1.smithi111.stdout:1/864: dread - d0/d2b/de3/f115 zero size 2024-03-23T11:47:49.105 INFO:tasks.workunit.client.1.smithi111.stdout:4/943: rmdir d0/d95/d9b/db3/db6 39 2024-03-23T11:47:49.106 INFO:tasks.workunit.client.1.smithi111.stdout:0/848: truncate d5/db/d1f/d19/f53 910747 0 2024-03-23T11:47:49.108 INFO:tasks.workunit.client.1.smithi111.stdout:2/913: creat d2/d11/d17/d3e/d5d/f144 x:0 0 0 2024-03-23T11:47:49.110 INFO:tasks.workunit.client.1.smithi111.stdout:9/915: fdatasync d0/f9b 0 2024-03-23T11:47:49.113 INFO:tasks.workunit.client.1.smithi111.stdout:6/879: getdents d2/d1a/d1c/d2c/d51/def/d118 0 2024-03-23T11:47:49.118 INFO:tasks.workunit.client.1.smithi111.stdout:0/849: symlink d5/db/d1f/d19/l11f 0 2024-03-23T11:47:49.119 INFO:tasks.workunit.client.1.smithi111.stdout:3/896: creat d0/d1a/d1d/dc5/f129 x:0 0 0 2024-03-23T11:47:49.119 INFO:tasks.workunit.client.1.smithi111.stdout:2/914: dread - d2/d11/d17/d3e/d6b/f9c zero size 2024-03-23T11:47:49.123 INFO:tasks.workunit.client.1.smithi111.stdout:7/885: truncate d14/d1b/d1e/d2c/d42/d5e/d88/f2b 1620525 0 2024-03-23T11:47:49.124 INFO:tasks.workunit.client.1.smithi111.stdout:0/850: creat d5/db/d1f/de4/da7/f120 x:0 0 0 2024-03-23T11:47:49.124 INFO:tasks.workunit.client.1.smithi111.stdout:3/897: readlink d0/d53/lc1 0 2024-03-23T11:47:49.125 INFO:tasks.workunit.client.1.smithi111.stdout:9/916: symlink d0/d31/l129 0 2024-03-23T11:47:49.127 INFO:tasks.workunit.client.1.smithi111.stdout:2/915: mkdir d2/d11/d17/d1f/da0/dd2/d145 0 2024-03-23T11:47:49.131 INFO:tasks.workunit.client.1.smithi111.stdout:5/968: link d1/d3/d6/d7/d63/d2c/d82/d66/d77/l9c d1/d3/d6/d7/dee/l145 0 2024-03-23T11:47:49.133 INFO:tasks.workunit.client.1.smithi111.stdout:1/865: link d0/d2b/d7c/da9/d106/f9b d0/d5/f12a 0 2024-03-23T11:47:49.134 INFO:tasks.workunit.client.1.smithi111.stdout:2/916: creat d2/de/d12/d70/d96/f146 x:0 0 0 2024-03-23T11:47:49.135 INFO:tasks.workunit.client.1.smithi111.stdout:1/866: chown d0/d6/f22 2118165636 1 2024-03-23T11:47:49.135 INFO:tasks.workunit.client.1.smithi111.stdout:3/898: creat d0/d1a/d2d/df5/d11a/f12a x:0 0 0 2024-03-23T11:47:49.138 INFO:tasks.workunit.client.1.smithi111.stdout:5/969: symlink d1/d3/d6/d7/d63/d2c/d43/l146 0 2024-03-23T11:47:49.139 INFO:tasks.workunit.client.1.smithi111.stdout:3/899: write d0/f1f [3441982,121636] 0 2024-03-23T11:47:49.140 INFO:tasks.workunit.client.1.smithi111.stdout:1/867: mkdir d0/d2b/d37/d6a/d100/d12b 0 2024-03-23T11:47:49.141 INFO:tasks.workunit.client.1.smithi111.stdout:5/970: mknod d1/d3/d3a/c147 0 2024-03-23T11:47:49.143 INFO:tasks.workunit.client.1.smithi111.stdout:2/917: rename d2/d11/d17/d3e/d5d/dc2/d133 to d2/de0/d110/d147 0 2024-03-23T11:47:49.144 INFO:tasks.workunit.client.1.smithi111.stdout:3/900: mkdir d0/d55/de2/d12b 0 2024-03-23T11:47:49.146 INFO:tasks.workunit.client.1.smithi111.stdout:5/971: chown d1/d3/d6/d7/d63/d2c/f133 22648 1 2024-03-23T11:47:49.146 INFO:tasks.workunit.client.1.smithi111.stdout:1/868: unlink d0/d5/dd5/df6/f104 0 2024-03-23T11:47:49.147 INFO:tasks.workunit.client.1.smithi111.stdout:2/918: write d2/de/d6c/dd1/fe3 [933439,79697] 0 2024-03-23T11:47:49.148 INFO:tasks.workunit.client.1.smithi111.stdout:7/886: dwrite d14/d44/d70/d7e/ff9 [4194304,4194304] 0 2024-03-23T11:47:49.153 INFO:tasks.workunit.client.1.smithi111.stdout:1/869: truncate d0/d6/d48/d4a/de5/fea 915292 0 2024-03-23T11:47:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:48 smithi111 ceph-mon[44005]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-03-23T11:47:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:48 smithi111 ceph-mon[44005]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-03-23T11:47:49.154 INFO:tasks.workunit.client.1.smithi111.stdout:3/901: truncate d0/d1a/d2b/d10e/f82 1223549 0 2024-03-23T11:47:49.156 INFO:tasks.workunit.client.1.smithi111.stdout:1/870: chown d0/d5/f35 585410407 1 2024-03-23T11:47:49.159 INFO:tasks.workunit.client.1.smithi111.stdout:2/919: truncate d2/d11/d17/d3e/d5d/f144 402405 0 2024-03-23T11:47:49.163 INFO:tasks.workunit.client.1.smithi111.stdout:4/944: dwrite d0/dc/d4c/f121 [0,4194304] 0 2024-03-23T11:47:49.166 INFO:tasks.workunit.client.1.smithi111.stdout:0/851: dwrite d5/db/d1f/d32/fe8 [0,4194304] 0 2024-03-23T11:47:49.196 INFO:tasks.workunit.client.1.smithi111.stdout:8/705: sync 2024-03-23T11:47:49.198 INFO:tasks.workunit.client.1.smithi111.stdout:8/706: chown d5/d1b/c1c 34 1 2024-03-23T11:47:49.223 INFO:tasks.workunit.client.1.smithi111.stdout:7/887: chown d14/d1b/d1e/d2c/d42/d5e/d88/d5f/cf8 40 1 2024-03-23T11:47:49.226 INFO:tasks.workunit.client.1.smithi111.stdout:3/902: mknod d0/d53/dfa/d103/c12c 0 2024-03-23T11:47:49.227 INFO:tasks.workunit.client.1.smithi111.stdout:7/888: dread - d14/d44/d70/d7e/dbc/fc7 zero size 2024-03-23T11:47:49.229 INFO:tasks.workunit.client.1.smithi111.stdout:0/852: dread d5/db/d1f/d19/dbd/f62 [0,4194304] 0 2024-03-23T11:47:49.230 INFO:tasks.workunit.client.1.smithi111.stdout:4/945: rmdir d0/dc 39 2024-03-23T11:47:49.230 INFO:tasks.workunit.client.1.smithi111.stdout:2/920: readlink d2/d11/d17/d3e/d6b/d2e/l8d 0 2024-03-23T11:47:49.232 INFO:tasks.workunit.client.1.smithi111.stdout:1/871: symlink d0/d6/d48/d4a/d6c/d83/d11d/d1e/d63/d116/l12c 0 2024-03-23T11:47:49.233 INFO:tasks.workunit.client.1.smithi111.stdout:3/903: rmdir d0/d1a/d4f 39 2024-03-23T11:47:49.234 INFO:tasks.workunit.client.1.smithi111.stdout:9/917: dwrite d0/d4/d6a/f72 [0,4194304] 0 2024-03-23T11:47:49.243 INFO:tasks.workunit.client.1.smithi111.stdout:8/707: write d5/d26/f74 [3786343,38075] 0 2024-03-23T11:47:49.244 INFO:tasks.workunit.client.1.smithi111.stdout:2/921: creat d2/de0/d110/f148 x:0 0 0 2024-03-23T11:47:49.250 INFO:tasks.workunit.client.1.smithi111.stdout:7/889: mkdir d14/d1b/d105 0 2024-03-23T11:47:49.251 INFO:tasks.workunit.client.1.smithi111.stdout:3/904: fsync d0/d1a/d2b/d38/d7f/f9c 0 2024-03-23T11:47:49.251 INFO:tasks.workunit.client.1.smithi111.stdout:4/946: mkdir d0/dc/d4c/d2a/d97/d4d/de3/d105/d136 0 2024-03-23T11:47:49.254 INFO:tasks.workunit.client.1.smithi111.stdout:5/972: dwrite d1/d3/f4 [0,4194304] 0 2024-03-23T11:47:49.254 INFO:tasks.workunit.client.1.smithi111.stdout:2/922: symlink d2/d11/d17/d3e/d5d/dc0/l149 0 2024-03-23T11:47:49.254 INFO:tasks.workunit.client.1.smithi111.stdout:6/880: sync 2024-03-23T11:47:49.256 INFO:tasks.workunit.client.1.smithi111.stdout:8/708: getdents d5/d26/d2f/d62/d75/d4b 0 2024-03-23T11:47:49.261 INFO:tasks.workunit.client.1.smithi111.stdout:3/905: rename d0/d53/faf to d0/d72/d7b/f12d 0 2024-03-23T11:47:49.263 INFO:tasks.workunit.client.1.smithi111.stdout:8/709: read - d5/d26/d31/fe6 zero size 2024-03-23T11:47:49.263 INFO:tasks.workunit.client.1.smithi111.stdout:1/872: link d0/d6/d48/d4a/d6c/d83/d11d/d1e/d63/d116/l12c d0/d5/dd5/l12d 0 2024-03-23T11:47:49.264 INFO:tasks.workunit.client.1.smithi111.stdout:5/973: mknod d1/c148 0 2024-03-23T11:47:49.265 INFO:tasks.workunit.client.1.smithi111.stdout:9/918: link d0/d31/d41/d44/c7c d0/d4/d6a/d89/c12a 0 2024-03-23T11:47:49.266 INFO:tasks.workunit.client.1.smithi111.stdout:7/890: write d14/f36 [2215756,117988] 0 2024-03-23T11:47:49.268 INFO:tasks.workunit.client.1.smithi111.stdout:0/853: dwrite d5/db/d4d/fb2 [0,4194304] 0 2024-03-23T11:47:49.269 INFO:tasks.workunit.client.1.smithi111.stdout:2/923: unlink d2/de/d12/d70/lf0 0 2024-03-23T11:47:49.270 INFO:tasks.workunit.client.1.smithi111.stdout:6/881: creat d2/d1a/d1c/d2c/d51/def/d118/f11c x:0 0 0 2024-03-23T11:47:49.274 INFO:tasks.workunit.client.1.smithi111.stdout:3/906: truncate d0/d1a/d2d/df5/d11a/f12a 860105 0 2024-03-23T11:47:49.274 INFO:tasks.workunit.client.1.smithi111.stdout:4/947: write d0/f45 [431809,7112] 0 2024-03-23T11:47:49.280 INFO:tasks.workunit.client.1.smithi111.stdout:0/854: write d5/db/d1f/de4/ffb [847485,40788] 0 2024-03-23T11:47:49.281 INFO:tasks.workunit.client.1.smithi111.stdout:7/891: getdents d14/d1b/d1e 0 2024-03-23T11:47:49.281 INFO:tasks.workunit.client.1.smithi111.stdout:5/974: link d1/d3/d6/d7/d7b/f131 d1/d3/d6/d8b/d59/f149 0 2024-03-23T11:47:49.283 INFO:tasks.workunit.client.1.smithi111.stdout:4/948: read d0/dc/d1f/f8a [2635843,36123] 0 2024-03-23T11:47:49.283 INFO:tasks.workunit.client.1.smithi111.stdout:6/882: creat d2/d1f/f11d x:0 0 0 2024-03-23T11:47:49.284 INFO:tasks.workunit.client.1.smithi111.stdout:2/924: rename d2/d11/d17/d3e/d5d/dc0/df7/d10e/l11e to d2/d7/l14a 0 2024-03-23T11:47:49.285 INFO:tasks.workunit.client.1.smithi111.stdout:3/907: symlink d0/d1a/d4f/l12e 0 2024-03-23T11:47:49.287 INFO:tasks.workunit.client.1.smithi111.stdout:5/975: dread - d1/d20/fc9 zero size 2024-03-23T11:47:49.291 INFO:tasks.workunit.client.1.smithi111.stdout:8/710: truncate d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 1443240 0 2024-03-23T11:47:49.296 INFO:tasks.workunit.client.1.smithi111.stdout:5/976: truncate d1/d20/dce/d137/f13f 1023470 0 2024-03-23T11:47:49.301 INFO:tasks.workunit.client.1.smithi111.stdout:8/711: truncate d5/f19 5152186 0 2024-03-23T11:47:49.302 INFO:tasks.workunit.client.1.smithi111.stdout:9/919: truncate d0/d4/d1c/d2d/d3b/d42/d4f/d95/ff0 1725539 0 2024-03-23T11:47:49.307 INFO:tasks.workunit.client.1.smithi111.stdout:5/977: mknod d1/d3/d6/d8b/d59/d6c/c14a 0 2024-03-23T11:47:49.312 INFO:tasks.workunit.client.1.smithi111.stdout:8/712: symlink d5/d32/d7d/da0/leb 0 2024-03-23T11:47:49.321 INFO:tasks.workunit.client.1.smithi111.stdout:5/978: dread - d1/d3/f126 zero size 2024-03-23T11:47:49.322 INFO:tasks.workunit.client.1.smithi111.stdout:3/908: dwrite d0/d1a/d2d/ff1 [0,4194304] 0 2024-03-23T11:47:49.326 INFO:tasks.workunit.client.1.smithi111.stdout:5/979: dread - d1/d3/d6/d30/f12a zero size 2024-03-23T11:47:49.327 INFO:tasks.workunit.client.1.smithi111.stdout:6/883: dwrite d2/fb [4194304,4194304] 0 2024-03-23T11:47:49.328 INFO:tasks.workunit.client.1.smithi111.stdout:8/713: mkdir d5/d26/d31/dec 0 2024-03-23T11:47:49.328 INFO:tasks.workunit.client.1.smithi111.stdout:0/855: dwrite d5/db/d1f/d19/dbd/d64/dc7/d100/fd3 [0,4194304] 0 2024-03-23T11:47:49.331 INFO:tasks.workunit.client.1.smithi111.stdout:7/892: dwrite d14/d1b/d1e/d2c/d8b/ddd/f100 [4194304,4194304] 0 2024-03-23T11:47:49.334 INFO:tasks.workunit.client.1.smithi111.stdout:4/949: dwrite d0/dc/f55 [0,4194304] 0 2024-03-23T11:47:49.335 INFO:tasks.workunit.client.1.smithi111.stdout:2/925: dread d2/d11/d17/d3e/dee/f10b [0,4194304] 0 2024-03-23T11:47:49.340 INFO:tasks.workunit.client.1.smithi111.stdout:6/884: readlink d2/d1f/d45/d49/lb3 0 2024-03-23T11:47:49.343 INFO:tasks.workunit.client.1.smithi111.stdout:3/909: dread d0/d1a/d2b/ff0 [0,4194304] 0 2024-03-23T11:47:49.344 INFO:tasks.workunit.client.1.smithi111.stdout:8/714: rmdir d5/d32/d9b 39 2024-03-23T11:47:49.347 INFO:tasks.workunit.client.1.smithi111.stdout:3/910: chown d0/db9/f113 7399 1 2024-03-23T11:47:49.350 INFO:tasks.workunit.client.1.smithi111.stdout:0/856: mkdir d5/d121 0 2024-03-23T11:47:49.350 INFO:tasks.workunit.client.1.smithi111.stdout:4/950: chown d0/dc/d4c/c88 978640 1 2024-03-23T11:47:49.353 INFO:tasks.workunit.client.1.smithi111.stdout:5/980: mkdir d1/d20/d5c/dc5/dd3/d116/d14b 0 2024-03-23T11:47:49.355 INFO:tasks.workunit.client.1.smithi111.stdout:2/926: chown d2/d11/d17/d3e/d5d/fa4 958710 1 2024-03-23T11:47:49.355 INFO:tasks.workunit.client.1.smithi111.stdout:6/885: creat d2/d73/f11e x:0 0 0 2024-03-23T11:47:49.356 INFO:tasks.workunit.client.1.smithi111.stdout:9/920: dwrite d0/d4/f83 [8388608,4194304] 0 2024-03-23T11:47:49.361 INFO:tasks.workunit.client.1.smithi111.stdout:0/857: stat d5/db/d1f/d19/dbd/d64/df2/ff5 0 2024-03-23T11:47:49.365 INFO:tasks.workunit.client.1.smithi111.stdout:5/981: stat d1/d3/d6/d8b/d59/d74/d143/df2/cf6 0 2024-03-23T11:47:49.366 INFO:tasks.workunit.client.1.smithi111.stdout:7/893: link d14/d1b/d1e/d2c/d2e/d33/f3d d14/d37/d43/f106 0 2024-03-23T11:47:49.366 INFO:tasks.workunit.client.1.smithi111.stdout:0/858: fsync d5/db/d1f/d47/d6f/fe7 0 2024-03-23T11:47:49.370 INFO:tasks.workunit.client.1.smithi111.stdout:8/715: unlink d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/dad/fdb 0 2024-03-23T11:47:49.373 INFO:tasks.workunit.client.1.smithi111.stdout:2/927: truncate d2/d11/d17/d3e/d6b/d2e/ff1 4266725 0 2024-03-23T11:47:49.378 INFO:tasks.workunit.client.1.smithi111.stdout:5/982: link d1/d3/d6/d30/f12a d1/d20/dce/d137/f14c 0 2024-03-23T11:47:49.379 INFO:tasks.workunit.client.1.smithi111.stdout:7/894: symlink d14/d1b/d1e/d2c/d2e/d33/d98/l107 0 2024-03-23T11:47:49.381 INFO:tasks.workunit.client.1.smithi111.stdout:8/716: creat d5/d26/d2f/d62/db5/dce/fed x:0 0 0 2024-03-23T11:47:49.381 INFO:tasks.workunit.client.1.smithi111.stdout:3/911: write d0/d1a/d2b/d38/d7f/fc3 [821024,72733] 0 2024-03-23T11:47:49.383 INFO:tasks.workunit.client.1.smithi111.stdout:2/928: symlink d2/d7/l14b 0 2024-03-23T11:47:49.385 INFO:tasks.workunit.client.1.smithi111.stdout:6/886: write d2/d1f/d45/d49/fa1 [4561991,48117] 0 2024-03-23T11:47:49.385 INFO:tasks.workunit.client.1.smithi111.stdout:9/921: dread d0/d4/d1c/f45 [0,4194304] 0 2024-03-23T11:47:49.385 INFO:tasks.workunit.client.1.smithi111.stdout:5/983: chown d1/d3/d6/d8b/d59/d6c/cc0 28329 1 2024-03-23T11:47:49.390 INFO:tasks.workunit.client.1.smithi111.stdout:1/873: sync 2024-03-23T11:47:49.390 INFO:tasks.workunit.client.1.smithi111.stdout:7/895: symlink d14/d1b/d1e/d2c/d2e/l108 0 2024-03-23T11:47:49.391 INFO:tasks.workunit.client.1.smithi111.stdout:0/859: write d5/db/d1f/d19/dbd/f94 [1012162,126940] 0 2024-03-23T11:47:49.392 INFO:tasks.workunit.client.1.smithi111.stdout:6/887: symlink d2/d17/l11f 0 2024-03-23T11:47:49.394 INFO:tasks.workunit.client.1.smithi111.stdout:5/984: mkdir d1/d3/d6/d8b/d59/d12e/d14d 0 2024-03-23T11:47:49.397 INFO:tasks.workunit.client.1.smithi111.stdout:6/888: readlink d2/d1a/d1c/d2c/l10d 0 2024-03-23T11:47:49.397 INFO:tasks.workunit.client.1.smithi111.stdout:5/985: fdatasync d1/f17 0 2024-03-23T11:47:49.400 INFO:tasks.workunit.client.1.smithi111.stdout:6/889: stat d2/d73 0 2024-03-23T11:47:49.402 INFO:tasks.workunit.client.1.smithi111.stdout:0/860: mkdir d5/db/d4d/da6/d10b/d122 0 2024-03-23T11:47:49.404 INFO:tasks.workunit.client.1.smithi111.stdout:1/874: fsync d0/d2b/d37/d6a/f8d 0 2024-03-23T11:47:49.405 INFO:tasks.workunit.client.1.smithi111.stdout:4/951: dread d0/dc/d1f/f76 [0,4194304] 0 2024-03-23T11:47:49.406 INFO:tasks.workunit.client.1.smithi111.stdout:9/922: getdents d0/d4/d1c/d2d/d75/d7a 0 2024-03-23T11:47:49.413 INFO:tasks.workunit.client.1.smithi111.stdout:0/861: read - d5/db/d1f/d48/f11a zero size 2024-03-23T11:47:49.413 INFO:tasks.workunit.client.1.smithi111.stdout:3/912: dwrite d0/d1a/d2d/ff1 [0,4194304] 0 2024-03-23T11:47:49.413 INFO:tasks.workunit.client.1.smithi111.stdout:9/923: rmdir d0/d31/d41/d9d 39 2024-03-23T11:47:49.414 INFO:tasks.workunit.client.1.smithi111.stdout:5/986: write d1/d3/d6/d7/d8c/f96 [760479,105250] 0 2024-03-23T11:47:49.418 INFO:tasks.workunit.client.1.smithi111.stdout:4/952: symlink d0/d95/d9b/db3/d120/l137 0 2024-03-23T11:47:49.420 INFO:tasks.workunit.client.1.smithi111.stdout:1/875: fdatasync d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/fb3 0 2024-03-23T11:47:49.420 INFO:tasks.workunit.client.1.smithi111.stdout:5/987: stat d1/d3/d6/d7/d63/d2c/d82 0 2024-03-23T11:47:49.427 INFO:tasks.workunit.client.1.smithi111.stdout:8/717: dread d5/d26/d2f/d62/db5/f87 [0,4194304] 0 2024-03-23T11:47:49.427 INFO:tasks.workunit.client.1.smithi111.stdout:3/913: rename d0/d1a/d4f/f4e to d0/d1a/d4f/d90/d112/f12f 0 2024-03-23T11:47:49.430 INFO:tasks.workunit.client.1.smithi111.stdout:3/914: chown d0/d1a/d2b/d10e/d19 14724209 1 2024-03-23T11:47:49.432 INFO:tasks.workunit.client.1.smithi111.stdout:6/890: dwrite d2/d17/ff1 [4194304,4194304] 0 2024-03-23T11:47:49.434 INFO:tasks.workunit.client.1.smithi111.stdout:4/953: mkdir d0/dc/d4c/d138 0 2024-03-23T11:47:49.434 INFO:tasks.workunit.client.1.smithi111.stdout:3/915: fsync d0/d1a/d2b/d38/d7f/dd0/fd3 0 2024-03-23T11:47:49.439 INFO:tasks.workunit.client.1.smithi111.stdout:8/718: mknod d5/d26/d31/d3b/cee 0 2024-03-23T11:47:49.439 INFO:tasks.workunit.client.1.smithi111.stdout:9/924: getdents d0/d4/d1c/d36/d28/d69 0 2024-03-23T11:47:49.440 INFO:tasks.workunit.client.1.smithi111.stdout:0/862: dwrite d5/db/d1f/d47/d59/f7d [4194304,4194304] 0 2024-03-23T11:47:49.441 INFO:tasks.workunit.client.1.smithi111.stdout:6/891: write d2/f31 [9919,79360] 0 2024-03-23T11:47:49.444 INFO:tasks.workunit.client.1.smithi111.stdout:3/916: mknod d0/d55/de2/c130 0 2024-03-23T11:47:49.444 INFO:tasks.workunit.client.1.smithi111.stdout:4/954: creat d0/dc/d4c/d83/f139 x:0 0 0 2024-03-23T11:47:49.447 INFO:tasks.workunit.client.1.smithi111.stdout:8/719: chown d5/d26/d31/d3b/d55/dd0 199367433 1 2024-03-23T11:47:49.449 INFO:tasks.workunit.client.1.smithi111.stdout:9/925: chown d0/d31/d41/d44/c7c 32 1 2024-03-23T11:47:49.450 INFO:tasks.workunit.client.1.smithi111.stdout:4/955: chown d0/dc/d4c/d2a/d97/d4f/dba/ld0 17646 1 2024-03-23T11:47:49.450 INFO:tasks.workunit.client.1.smithi111.stdout:8/720: mknod d5/d26/d2f/d42/cef 0 2024-03-23T11:47:49.527 INFO:tasks.workunit.client.1.smithi111.stdout:1/876: write d0/d6/d48/fa8 [4636612,16811] 0 2024-03-23T11:47:49.528 INFO:tasks.workunit.client.1.smithi111.stdout:9/926: read d0/d4/d1c/d2d/d3b/d42/d4f/f5c [492243,16484] 0 2024-03-23T11:47:49.529 INFO:tasks.workunit.client.1.smithi111.stdout:5/988: link d1/d3/d6/d8b/d59/ffe d1/d3/d6/d7/d63/d2c/d82/d66/f14e 0 2024-03-23T11:47:49.532 INFO:tasks.workunit.client.1.smithi111.stdout:3/917: dread d0/d1a/d2d/fbb [4194304,4194304] 0 2024-03-23T11:47:49.533 INFO:tasks.workunit.client.1.smithi111.stdout:5/989: chown d1/d20/d41/fb4 9204 1 2024-03-23T11:47:49.534 INFO:tasks.workunit.client.1.smithi111.stdout:1/877: creat d0/d6/d48/d4a/d6c/d83/d11d/d98/f12e x:0 0 0 2024-03-23T11:47:49.538 INFO:tasks.workunit.client.1.smithi111.stdout:5/990: chown d1/d3/d6/d7/f64 27241 1 2024-03-23T11:47:49.540 INFO:tasks.workunit.client.1.smithi111.stdout:0/863: dwrite d5/db/d1f/de4/f69 [0,4194304] 0 2024-03-23T11:47:49.543 INFO:tasks.workunit.client.1.smithi111.stdout:3/918: getdents d0/d1a/d2b/d38/d7f/dd0 0 2024-03-23T11:47:49.549 INFO:tasks.workunit.client.1.smithi111.stdout:8/721: dwrite d5/d26/d2f/d62/d75/d8d/fb8 [0,4194304] 0 2024-03-23T11:47:49.556 INFO:tasks.workunit.client.1.smithi111.stdout:9/927: dwrite d0/d4/d1c/d2d/d3b/d42/d4f/f5c [0,4194304] 0 2024-03-23T11:47:49.558 INFO:tasks.workunit.client.1.smithi111.stdout:1/878: dread d0/d2b/d69/f8c [0,4194304] 0 2024-03-23T11:47:49.559 INFO:tasks.workunit.client.1.smithi111.stdout:9/928: readlink d0/d4/d1c/d2d/d3b/d42/d4f/l52 0 2024-03-23T11:47:49.565 INFO:tasks.workunit.client.1.smithi111.stdout:3/919: read d0/d1a/d1d/d26/f46 [5425383,24632] 0 2024-03-23T11:47:49.567 INFO:tasks.workunit.client.1.smithi111.stdout:8/722: dread d5/d26/d2f/d62/d75/d4b/d51/da4/d97/f9c [0,4194304] 0 2024-03-23T11:47:49.569 INFO:tasks.workunit.client.1.smithi111.stdout:3/920: creat d0/db9/f131 x:0 0 0 2024-03-23T11:47:49.571 INFO:tasks.workunit.client.1.smithi111.stdout:3/921: creat d0/d53/f132 x:0 0 0 2024-03-23T11:47:49.572 INFO:tasks.workunit.client.1.smithi111.stdout:2/929: sync 2024-03-23T11:47:49.575 INFO:tasks.workunit.client.1.smithi111.stdout:3/922: rmdir d0/d1a/d2b/d38/dd4 39 2024-03-23T11:47:49.577 INFO:tasks.workunit.client.1.smithi111.stdout:3/923: dread - d0/db9/f113 zero size 2024-03-23T11:47:49.578 INFO:tasks.workunit.client.1.smithi111.stdout:2/930: mknod d2/de/c14c 0 2024-03-23T11:47:49.578 INFO:tasks.workunit.client.1.smithi111.stdout:3/924: read - d0/d1a/d1d/fd7 zero size 2024-03-23T11:47:49.580 INFO:tasks.workunit.client.1.smithi111.stdout:2/931: creat d2/d11/d17/d3e/dee/f14d x:0 0 0 2024-03-23T11:47:49.581 INFO:tasks.workunit.client.1.smithi111.stdout:3/925: creat d0/d1a/d2d/df5/d11a/f133 x:0 0 0 2024-03-23T11:47:49.584 INFO:tasks.workunit.client.1.smithi111.stdout:3/926: creat d0/d1a/d2b/d10e/d19/d91/f134 x:0 0 0 2024-03-23T11:47:49.587 INFO:tasks.workunit.client.1.smithi111.stdout:9/929: dread d0/d4/d10/d29/f67 [0,4194304] 0 2024-03-23T11:47:49.588 INFO:tasks.workunit.client.1.smithi111.stdout:3/927: truncate d0/d1a/d1d/f49 3163182 0 2024-03-23T11:47:49.592 INFO:tasks.workunit.client.1.smithi111.stdout:3/928: unlink d0/d1a/d1d/d26/f7e 0 2024-03-23T11:47:49.595 INFO:tasks.workunit.client.1.smithi111.stdout:3/929: unlink d0/l2 0 2024-03-23T11:47:49.599 INFO:tasks.workunit.client.1.smithi111.stdout:2/932: dread d2/d7/da/f103 [0,4194304] 0 2024-03-23T11:47:49.600 INFO:tasks.workunit.client.1.smithi111.stdout:2/933: fsync d2/d11/f37 0 2024-03-23T11:47:49.602 INFO:tasks.workunit.client.0.smithi063.stdout:7/595: sync 2024-03-23T11:47:49.603 INFO:tasks.workunit.client.1.smithi111.stdout:2/934: write d2/d7/da/d19/f143 [768815,34173] 0 2024-03-23T11:47:49.604 INFO:tasks.workunit.client.1.smithi111.stdout:5/991: write d1/d3/d6/d7/d7b/f131 [355597,4553] 0 2024-03-23T11:47:49.609 INFO:tasks.workunit.client.0.smithi063.stdout:7/596: truncate d5/d26/d32/faf 331495 0 2024-03-23T11:47:49.611 INFO:tasks.workunit.client.1.smithi111.stdout:5/992: mknod d1/d3/d6/d7/d63/c14f 0 2024-03-23T11:47:49.614 INFO:tasks.workunit.client.1.smithi111.stdout:2/935: rename d2/d11/d17/d3e/d6b/d2e/d3f/faa to d2/d11/d17/d3e/d5d/dc0/f14e 0 2024-03-23T11:47:49.614 INFO:tasks.workunit.client.0.smithi063.stdout:7/597: mknod d5/d22/d57/d64/cbe 0 2024-03-23T11:47:49.617 INFO:tasks.workunit.client.1.smithi111.stdout:0/864: truncate d5/db/d1f/d19/d45/fe2 1818376 0 2024-03-23T11:47:49.619 INFO:tasks.workunit.client.0.smithi063.stdout:7/598: symlink d5/d6/d19/d1f/d47/d6e/lbf 0 2024-03-23T11:47:49.620 INFO:tasks.workunit.client.1.smithi111.stdout:1/879: dwrite d0/d5/f35 [0,4194304] 0 2024-03-23T11:47:49.620 INFO:tasks.workunit.client.1.smithi111.stdout:3/930: dread d0/d1a/d1d/d26/d5a/f10b [0,4194304] 0 2024-03-23T11:47:49.624 INFO:tasks.workunit.client.1.smithi111.stdout:8/723: write d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [2153022,120455] 0 2024-03-23T11:47:49.624 INFO:tasks.workunit.client.1.smithi111.stdout:5/993: write d1/d3/d6/d7/d7b/fec [1304585,21104] 0 2024-03-23T11:47:49.627 INFO:tasks.workunit.client.1.smithi111.stdout:9/930: dread d0/d4/d10/f2c [0,4194304] 0 2024-03-23T11:47:49.627 INFO:tasks.workunit.client.1.smithi111.stdout:3/931: write d0/d1a/d1d/f6e [4118712,124989] 0 2024-03-23T11:47:49.628 INFO:tasks.workunit.client.1.smithi111.stdout:0/865: dread - d5/db/d4d/da6/fdb zero size 2024-03-23T11:47:49.629 INFO:tasks.workunit.client.1.smithi111.stdout:1/880: mknod d0/d2b/d7c/da9/d106/c12f 0 2024-03-23T11:47:49.637 INFO:tasks.workunit.client.1.smithi111.stdout:3/932: creat d0/db9/f135 x:0 0 0 2024-03-23T11:47:49.639 INFO:tasks.workunit.client.1.smithi111.stdout:7/896: sync 2024-03-23T11:47:49.643 INFO:tasks.workunit.client.1.smithi111.stdout:7/897: fsync d14/d1b/d22/f29 0 2024-03-23T11:47:49.644 INFO:tasks.workunit.client.0.smithi063.stdout:7/599: dwrite d5/d22/f94 [0,4194304] 0 2024-03-23T11:47:49.644 INFO:tasks.workunit.client.1.smithi111.stdout:8/724: link f1 d5/d32/ff0 0 2024-03-23T11:47:49.646 INFO:tasks.workunit.client.1.smithi111.stdout:7/898: fsync d14/fba 0 2024-03-23T11:47:49.646 INFO:tasks.workunit.client.0.smithi063.stdout:7/600: chown d5/d5c/d2f/d8b/daa 9117 1 2024-03-23T11:47:49.648 INFO:tasks.workunit.client.1.smithi111.stdout:1/881: mkdir d0/d2b/d37/d6a/d100/d12b/d130 0 2024-03-23T11:47:49.649 INFO:tasks.workunit.client.0.smithi063.stdout:7/601: symlink d5/d6/lc0 0 2024-03-23T11:47:49.650 INFO:tasks.workunit.client.1.smithi111.stdout:9/931: creat d0/d31/d41/d9d/db0/f12b x:0 0 0 2024-03-23T11:47:49.652 INFO:tasks.workunit.client.1.smithi111.stdout:2/936: dwrite d2/f6 [0,4194304] 0 2024-03-23T11:47:49.652 INFO:tasks.workunit.client.1.smithi111.stdout:5/994: creat d1/d3/d6/d7/d63/d2c/d43/f150 x:0 0 0 2024-03-23T11:47:49.655 INFO:tasks.workunit.client.1.smithi111.stdout:0/866: link d5/db/d1f/d47/d6f/fe9 d5/db/d1f/f123 0 2024-03-23T11:47:49.658 INFO:tasks.workunit.client.1.smithi111.stdout:8/725: rmdir d5/d26/d2f/d62/d75/d4b/d51/d5a 39 2024-03-23T11:47:49.660 INFO:tasks.workunit.client.1.smithi111.stdout:1/882: fsync d0/d6/d48/d4a/d6c/d83/d11d/d1e/fa0 0 2024-03-23T11:47:49.661 INFO:tasks.workunit.client.1.smithi111.stdout:3/933: dread d0/d1a/d2b/d38/d7f/f9c [0,4194304] 0 2024-03-23T11:47:49.665 INFO:tasks.workunit.client.1.smithi111.stdout:1/883: dread - d0/ff0 zero size 2024-03-23T11:47:49.666 INFO:tasks.workunit.client.1.smithi111.stdout:8/726: creat d5/d26/d31/d3b/d55/ff1 x:0 0 0 2024-03-23T11:47:49.667 INFO:tasks.workunit.client.1.smithi111.stdout:7/899: rename d14/d1b/d1e/d2c/l75 to d14/d37/l109 0 2024-03-23T11:47:49.670 INFO:tasks.workunit.client.0.smithi063.stdout:7/602: dwrite d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:49.673 INFO:tasks.workunit.client.1.smithi111.stdout:3/934: creat d0/d1a/d2b/f136 x:0 0 0 2024-03-23T11:47:49.674 INFO:tasks.workunit.client.0.smithi063.stdout:7/603: mknod d5/d5c/d2f/d8b/daa/cc1 0 2024-03-23T11:47:49.675 INFO:tasks.workunit.client.1.smithi111.stdout:7/900: fsync d14/d37/d43/ff4 0 2024-03-23T11:47:49.676 INFO:tasks.workunit.client.0.smithi063.stdout:7/604: mkdir d5/d6/d39/d88/dc2 0 2024-03-23T11:47:49.677 INFO:tasks.workunit.client.1.smithi111.stdout:6/892: sync 2024-03-23T11:47:49.677 INFO:tasks.workunit.client.1.smithi111.stdout:4/956: sync 2024-03-23T11:47:49.680 INFO:tasks.workunit.client.1.smithi111.stdout:8/727: dread d5/f6 [4194304,4194304] 0 2024-03-23T11:47:49.682 INFO:tasks.workunit.client.1.smithi111.stdout:4/957: fdatasync d0/dc/f55 0 2024-03-23T11:47:49.686 INFO:tasks.workunit.client.1.smithi111.stdout:7/901: mkdir d14/d10a 0 2024-03-23T11:47:49.687 INFO:tasks.workunit.client.1.smithi111.stdout:0/867: dwrite d5/db/d1f/d47/de0/ff9 [0,4194304] 0 2024-03-23T11:47:49.690 INFO:tasks.workunit.client.1.smithi111.stdout:4/958: unlink d0/d95/d9b/fc1 0 2024-03-23T11:47:49.692 INFO:tasks.workunit.client.1.smithi111.stdout:5/995: dwrite d1/d3/d6/d7/d7b/fa4 [0,4194304] 0 2024-03-23T11:47:49.693 INFO:tasks.workunit.client.1.smithi111.stdout:8/728: mknod d5/d26/d31/d5e/cf2 0 2024-03-23T11:47:49.696 INFO:tasks.workunit.client.0.smithi063.stdout:7/605: dwrite d5/d5c/d2f/d73/f3b [0,4194304] 0 2024-03-23T11:47:49.698 INFO:tasks.workunit.client.1.smithi111.stdout:9/932: dwrite d0/d31/d35/f11d [0,4194304] 0 2024-03-23T11:47:49.699 INFO:tasks.workunit.client.1.smithi111.stdout:6/893: creat d2/d35/d70/d98/f120 x:0 0 0 2024-03-23T11:47:49.706 INFO:tasks.workunit.client.1.smithi111.stdout:9/933: creat d0/d4/d1c/f12c x:0 0 0 2024-03-23T11:47:49.706 INFO:tasks.workunit.client.0.smithi063.stdout:7/606: rmdir d5/d6/dad 0 2024-03-23T11:47:49.708 INFO:tasks.workunit.client.1.smithi111.stdout:8/729: rmdir d5/d32/d9b 39 2024-03-23T11:47:49.711 INFO:tasks.workunit.client.1.smithi111.stdout:9/934: fsync d0/d31/d41/d9d/dd4/de0/fe4 0 2024-03-23T11:47:49.714 INFO:tasks.workunit.client.0.smithi063.stdout:7/607: creat d5/d5c/d2f/fc3 x:0 0 0 2024-03-23T11:47:49.716 INFO:tasks.workunit.client.1.smithi111.stdout:2/937: dread d2/d11/d17/d3e/d5d/d8c/f10a [0,4194304] 0 2024-03-23T11:47:49.718 INFO:tasks.workunit.client.1.smithi111.stdout:9/935: symlink d0/d4/d10/d80/l12d 0 2024-03-23T11:47:49.723 INFO:tasks.workunit.client.0.smithi063.stdout:7/608: write d5/d6/d19/d1f/f6d [1357397,69443] 0 2024-03-23T11:47:49.724 INFO:tasks.workunit.client.1.smithi111.stdout:8/730: rmdir d5/d26/d2f/d62/d75/d4b/d51 39 2024-03-23T11:47:49.727 INFO:tasks.workunit.client.1.smithi111.stdout:7/902: dwrite d14/d1b/d1e/d2c/d2e/d73/dae/d74/fa9 [0,4194304] 0 2024-03-23T11:47:49.729 INFO:tasks.workunit.client.1.smithi111.stdout:4/959: dread d0/dc/d1f/db0/faa [0,4194304] 0 2024-03-23T11:47:49.729 INFO:tasks.workunit.client.1.smithi111.stdout:0/868: dread d5/db/d1f/d19/f1c [0,4194304] 0 2024-03-23T11:47:49.733 INFO:tasks.workunit.client.1.smithi111.stdout:6/894: dwrite d2/d1f/d45/f63 [0,4194304] 0 2024-03-23T11:47:49.736 INFO:tasks.workunit.client.1.smithi111.stdout:9/936: truncate d0/d31/d35/f9c 1210026 0 2024-03-23T11:47:49.737 INFO:tasks.workunit.client.0.smithi063.stdout:7/609: dread d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:49.740 INFO:tasks.workunit.client.1.smithi111.stdout:0/869: mknod d5/db/d1f/d19/dbd/d64/d91/c124 0 2024-03-23T11:47:49.740 INFO:tasks.workunit.client.1.smithi111.stdout:3/935: dread d0/d1a/d2b/d38/d7f/dd0/fd3 [0,4194304] 0 2024-03-23T11:47:49.742 INFO:tasks.workunit.client.0.smithi063.stdout:7/610: rmdir d5/d26/d32 39 2024-03-23T11:47:49.743 INFO:tasks.workunit.client.1.smithi111.stdout:6/895: creat d2/d1a/d1c/ddf/d116/f121 x:0 0 0 2024-03-23T11:47:49.749 INFO:tasks.workunit.client.1.smithi111.stdout:8/731: dread - d5/d26/d2f/d62/d75/d4b/d51/da4/d97/fa8 zero size 2024-03-23T11:47:49.751 INFO:tasks.workunit.client.1.smithi111.stdout:8/732: fdatasync d5/d26/d31/fe6 0 2024-03-23T11:47:49.751 INFO:tasks.workunit.client.1.smithi111.stdout:0/870: creat d5/db/d1f/d19/dbd/d64/d91/ddd/df6/f125 x:0 0 0 2024-03-23T11:47:49.754 INFO:tasks.workunit.client.1.smithi111.stdout:4/960: write d0/fc3 [2029845,58805] 0 2024-03-23T11:47:49.757 INFO:tasks.workunit.client.1.smithi111.stdout:0/871: chown d5/db/d1f/l2f 10 1 2024-03-23T11:47:49.758 INFO:tasks.workunit.client.1.smithi111.stdout:3/936: mkdir d0/d1a/d2b/d10e/dcd/d137 0 2024-03-23T11:47:49.758 INFO:tasks.workunit.client.1.smithi111.stdout:8/733: stat d5/d26/d2f/d62/d75/d4b/d51/da4/f88 0 2024-03-23T11:47:49.761 INFO:tasks.workunit.client.1.smithi111.stdout:2/938: dwrite d2/d7/f56 [0,4194304] 0 2024-03-23T11:47:49.762 INFO:tasks.workunit.client.1.smithi111.stdout:8/734: fdatasync d5/d26/d2f/d42/d63/fab 0 2024-03-23T11:47:49.762 INFO:tasks.workunit.client.1.smithi111.stdout:0/872: dread - d5/db/d1f/d47/f11c zero size 2024-03-23T11:47:49.763 INFO:tasks.workunit.client.0.smithi063.stdout:7/611: dwrite d5/d6/d19/fa8 [0,4194304] 0 2024-03-23T11:47:49.767 INFO:tasks.workunit.client.0.smithi063.stdout:7/612: rmdir d5/d6/d19/d65 39 2024-03-23T11:47:49.769 INFO:tasks.workunit.client.1.smithi111.stdout:3/937: chown d0/d1a/d2b/d10e/d19/l3a 47 1 2024-03-23T11:47:49.777 INFO:tasks.workunit.client.1.smithi111.stdout:3/938: chown d0/d1a/d2b/d10e/d19/d5e/l7c 45958 1 2024-03-23T11:47:49.780 INFO:tasks.workunit.client.1.smithi111.stdout:8/735: write d5/d26/d2f/d62/d75/d4b/d51/da4/f88 [2249645,114842] 0 2024-03-23T11:47:49.783 INFO:tasks.workunit.client.1.smithi111.stdout:0/873: truncate d5/db/d1f/d19/dbd/fa5 366276 0 2024-03-23T11:47:49.784 INFO:tasks.workunit.client.0.smithi063.stdout:7/613: read d5/d26/d76/f78 [3274601,32773] 0 2024-03-23T11:47:49.788 INFO:tasks.workunit.client.1.smithi111.stdout:4/961: write d0/dc/d4c/d2a/d97/f125 [2064144,14019] 0 2024-03-23T11:47:49.794 INFO:tasks.workunit.client.1.smithi111.stdout:3/939: creat d0/d1a/d2b/d38/f138 x:0 0 0 2024-03-23T11:47:49.794 INFO:tasks.workunit.client.1.smithi111.stdout:0/874: read d5/db/d1f/fc0 [113253,56499] 0 2024-03-23T11:47:49.804 INFO:tasks.workunit.client.1.smithi111.stdout:4/962: chown d0/dc/d4c/d2a/df2/d123/da/l112 5 1 2024-03-23T11:47:49.805 INFO:tasks.workunit.client.1.smithi111.stdout:5/996: dread d1/d20/d5c/dc5/f121 [0,4194304] 0 2024-03-23T11:47:49.811 INFO:tasks.workunit.client.0.smithi063.stdout:7/614: write d5/d5c/d2f/d73/d20/f82 [764866,105123] 0 2024-03-23T11:47:49.814 INFO:tasks.workunit.client.1.smithi111.stdout:7/903: dread d14/d1b/f2d [0,4194304] 0 2024-03-23T11:47:49.816 INFO:tasks.workunit.client.0.smithi063.stdout:7/615: symlink d5/d6/d19/d1f/d47/d6e/da2/lc4 0 2024-03-23T11:47:49.818 INFO:tasks.workunit.client.1.smithi111.stdout:0/875: read d5/db/d1f/d48/f66 [2744054,125640] 0 2024-03-23T11:47:49.818 INFO:tasks.workunit.client.1.smithi111.stdout:4/963: link d0/d95/db5/dc7/ff8 d0/d95/f13a 0 2024-03-23T11:47:49.823 INFO:tasks.workunit.client.1.smithi111.stdout:2/939: dread d2/d11/d17/d3e/d6b/d2e/ff3 [0,4194304] 0 2024-03-23T11:47:49.824 INFO:tasks.workunit.client.1.smithi111.stdout:4/964: mkdir d0/dc/d78/d13b 0 2024-03-23T11:47:49.825 INFO:tasks.workunit.client.0.smithi063.stdout:7/616: write d5/d5c/d2f/d8c/fb0 [119306,57885] 0 2024-03-23T11:47:49.826 INFO:tasks.workunit.client.1.smithi111.stdout:7/904: getdents d14/d1b/d1e/d2c/d42 0 2024-03-23T11:47:49.826 INFO:tasks.workunit.client.1.smithi111.stdout:8/736: write d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/f52 [456480,110363] 0 2024-03-23T11:47:49.829 INFO:tasks.workunit.client.1.smithi111.stdout:6/896: dread d2/f31 [0,4194304] 0 2024-03-23T11:47:49.829 INFO:tasks.workunit.client.0.smithi063.stdout:7/617: rmdir d5/d5c/d2f 39 2024-03-23T11:47:49.831 INFO:tasks.workunit.client.0.smithi063.stdout:7/618: chown d5/d6/d19/d1f/d6c/d9c 12047 1 2024-03-23T11:47:49.832 INFO:tasks.workunit.client.1.smithi111.stdout:4/965: mknod d0/dc/d7a/c13c 0 2024-03-23T11:47:49.833 INFO:tasks.workunit.client.1.smithi111.stdout:7/905: mkdir d14/d44/d70/d7e/d10b 0 2024-03-23T11:47:49.834 INFO:tasks.workunit.client.0.smithi063.stdout:7/619: symlink d5/d6/d19/lc5 0 2024-03-23T11:47:49.836 INFO:tasks.workunit.client.1.smithi111.stdout:1/884: sync 2024-03-23T11:47:49.836 INFO:tasks.workunit.client.1.smithi111.stdout:3/940: dwrite d0/d1a/d4f/d90/fb2 [0,4194304] 0 2024-03-23T11:47:49.838 INFO:tasks.workunit.client.0.smithi063.stdout:7/620: rename d5/d6/d19/d1f/d47 to d5/d26/db7/dc6 0 2024-03-23T11:47:49.840 INFO:tasks.workunit.client.1.smithi111.stdout:4/966: symlink d0/d17/l13d 0 2024-03-23T11:47:49.840 INFO:tasks.workunit.client.1.smithi111.stdout:3/941: chown d0/d1a/d2b/d10e/d19/d91/f105 371575 1 2024-03-23T11:47:49.842 INFO:tasks.workunit.client.1.smithi111.stdout:0/876: link d5/db/d1f/d19/c4e d5/db/d1f/d47/d6f/c126 0 2024-03-23T11:47:49.842 INFO:tasks.workunit.client.1.smithi111.stdout:7/906: creat d14/d1b/d1e/d2c/d2e/d33/d98/f10c x:0 0 0 2024-03-23T11:47:49.842 INFO:tasks.workunit.client.0.smithi063.stdout:7/621: truncate d5/d5c/d2f/d73/da/f89 5038562 0 2024-03-23T11:47:49.843 INFO:tasks.workunit.client.1.smithi111.stdout:4/967: readlink d0/dc/d1f/d5e/lf4 0 2024-03-23T11:47:49.844 INFO:tasks.workunit.client.1.smithi111.stdout:5/997: dwrite d1/d3/d6/d8b/d59/d6c/fc8 [0,4194304] 0 2024-03-23T11:47:49.850 INFO:tasks.workunit.client.1.smithi111.stdout:3/942: mknod d0/d1a/d2b/d10e/d19/c139 0 2024-03-23T11:47:49.852 INFO:tasks.workunit.client.1.smithi111.stdout:5/998: readlink d1/d3/d6/d7/d7b/l134 0 2024-03-23T11:47:49.856 INFO:tasks.workunit.client.1.smithi111.stdout:5/999: write d1/d3/d6/d8b/d59/d74/f21 [4935839,42341] 0 2024-03-23T11:47:49.857 INFO:tasks.workunit.client.1.smithi111.stdout:4/968: fdatasync d0/dc/d4c/d2a/f73 0 2024-03-23T11:47:49.861 INFO:tasks.workunit.client.0.smithi063.stdout:7/622: dwrite f1 [0,4194304] 0 2024-03-23T11:47:49.869 INFO:tasks.workunit.client.0.smithi063.stdout:7/623: fdatasync d5/d5c/d2f/f79 0 2024-03-23T11:47:49.886 INFO:tasks.workunit.client.0.smithi063.stdout:7/624: dwrite d5/d22/d57/d64/fb2 [0,4194304] 0 2024-03-23T11:47:49.886 INFO:tasks.workunit.client.0.smithi063.stdout:1/570: sync 2024-03-23T11:47:49.886 INFO:tasks.workunit.client.0.smithi063.stdout:3/946: sync 2024-03-23T11:47:49.886 INFO:tasks.workunit.client.0.smithi063.stdout:8/916: sync 2024-03-23T11:47:49.886 INFO:tasks.workunit.client.0.smithi063.stdout:9/664: sync 2024-03-23T11:47:49.887 INFO:tasks.workunit.client.0.smithi063.stdout:5/446: sync 2024-03-23T11:47:49.887 INFO:tasks.workunit.client.0.smithi063.stdout:4/667: sync 2024-03-23T11:47:49.887 INFO:tasks.workunit.client.0.smithi063.stdout:2/534: sync 2024-03-23T11:47:49.887 INFO:tasks.workunit.client.0.smithi063.stdout:0/811: sync 2024-03-23T11:47:49.887 INFO:tasks.workunit.client.0.smithi063.stdout:6/504: sync 2024-03-23T11:47:49.888 INFO:tasks.workunit.client.0.smithi063.stdout:6/505: readlink d1/lb 0 2024-03-23T11:47:49.888 INFO:tasks.workunit.client.0.smithi063.stdout:4/668: chown d5/da/d72 3623993 1 2024-03-23T11:47:49.893 INFO:tasks.workunit.client.0.smithi063.stdout:6/506: chown d1/dc/c60 49669461 1 2024-03-23T11:47:49.893 INFO:tasks.workunit.client.0.smithi063.stdout:1/571: rmdir d3/d49/d56/d65 39 2024-03-23T11:47:49.897 INFO:tasks.workunit.client.0.smithi063.stdout:6/507: chown d1/d2a/l3d 11054 1 2024-03-23T11:47:49.897 INFO:tasks.workunit.client.0.smithi063.stdout:3/947: write d13/d22/d2d/d3e/dce/fa6 [3189046,20207] 0 2024-03-23T11:47:49.899 INFO:tasks.workunit.client.0.smithi063.stdout:5/447: truncate d2/d7/f1a 130606 0 2024-03-23T11:47:49.899 INFO:tasks.workunit.client.0.smithi063.stdout:6/508: stat d1/dc/d21/l6c 0 2024-03-23T11:47:49.900 INFO:tasks.workunit.client.0.smithi063.stdout:8/917: symlink d6/d1f/d54/d63/d119/l133 0 2024-03-23T11:47:49.901 INFO:tasks.workunit.client.0.smithi063.stdout:4/669: rmdir d5/d1e/d2b 39 2024-03-23T11:47:49.903 INFO:tasks.workunit.client.0.smithi063.stdout:0/812: creat d2/d14/d36/d77/d94/da6/d28/d46/d51/d118/f11f x:0 0 0 2024-03-23T11:47:49.904 INFO:tasks.workunit.client.0.smithi063.stdout:8/918: chown d6/d1f/d106/f129 5243 1 2024-03-23T11:47:49.908 INFO:tasks.workunit.client.0.smithi063.stdout:0/813: fsync d2/d4/fd8 0 2024-03-23T11:47:49.909 INFO:tasks.workunit.client.0.smithi063.stdout:6/509: creat d1/d15/f94 x:0 0 0 2024-03-23T11:47:49.911 INFO:tasks.workunit.client.0.smithi063.stdout:5/448: symlink d2/d7/d8/de/d2a/d3b/d85/l98 0 2024-03-23T11:47:49.914 INFO:tasks.workunit.client.0.smithi063.stdout:6/510: rmdir d1/dc/d21 39 2024-03-23T11:47:49.915 INFO:tasks.workunit.client.0.smithi063.stdout:9/665: dread d4/d31/d70/d8d/fc6 [0,4194304] 0 2024-03-23T11:47:49.918 INFO:tasks.workunit.client.0.smithi063.stdout:8/919: symlink d6/d1f/d54/d63/d90/d5d/d7d/l134 0 2024-03-23T11:47:49.922 INFO:tasks.workunit.client.0.smithi063.stdout:3/948: write d13/d22/d2d/d3f/de0/f11f [29316,36994] 0 2024-03-23T11:47:49.924 INFO:tasks.workunit.client.0.smithi063.stdout:0/814: creat d2/d14/d36/d77/f120 x:0 0 0 2024-03-23T11:47:49.924 INFO:tasks.workunit.client.0.smithi063.stdout:7/625: dwrite d5/d5c/d2f/d73/f8d [0,4194304] 0 2024-03-23T11:47:49.925 INFO:tasks.workunit.client.0.smithi063.stdout:4/670: write d5/da/d20/d4b/d16/f5c [2328211,112348] 0 2024-03-23T11:47:49.928 INFO:tasks.workunit.client.0.smithi063.stdout:1/572: write d3/df/f7d [1025419,114616] 0 2024-03-23T11:47:49.928 INFO:tasks.workunit.client.1.smithi111.stdout:3/943: rename d0/d1a/d2b/d38/d7f/dd0 to d0/d1a/d4f/d13a 0 2024-03-23T11:47:49.932 INFO:tasks.workunit.client.1.smithi111.stdout:8/737: dread d5/f2d [0,4194304] 0 2024-03-23T11:47:49.933 INFO:tasks.workunit.client.1.smithi111.stdout:3/944: readlink d0/d1a/d2b/d38/l5b 0 2024-03-23T11:47:49.936 INFO:tasks.workunit.client.0.smithi063.stdout:6/511: creat d1/dc/d21/d49/f95 x:0 0 0 2024-03-23T11:47:49.939 INFO:tasks.workunit.client.1.smithi111.stdout:6/897: dwrite d2/d1a/d1c/ddf/ff3 [0,4194304] 0 2024-03-23T11:47:49.940 INFO:tasks.workunit.client.1.smithi111.stdout:0/877: dwrite d5/db/d1f/f43 [0,4194304] 0 2024-03-23T11:47:49.942 INFO:tasks.workunit.client.0.smithi063.stdout:7/626: rmdir d5 39 2024-03-23T11:47:49.942 INFO:tasks.workunit.client.1.smithi111.stdout:1/885: dwrite d0/d6/d48/d4a/d6c/d83/d11d/d1e/fa0 [0,4194304] 0 2024-03-23T11:47:49.942 INFO:tasks.workunit.client.1.smithi111.stdout:0/878: chown d5/db/d1f/de4/da7/lbc 504359292 1 2024-03-23T11:47:49.944 INFO:tasks.workunit.client.0.smithi063.stdout:0/815: dread - d2/d14/d36/d77/d94/da6/d28/d46/f10e zero size 2024-03-23T11:47:49.944 INFO:tasks.workunit.client.0.smithi063.stdout:4/671: symlink d5/da/d20/d4b/d29/d38/d93/lc1 0 2024-03-23T11:47:49.945 INFO:tasks.workunit.client.0.smithi063.stdout:1/573: dread - d3/d35/fae zero size 2024-03-23T11:47:49.946 INFO:tasks.workunit.client.1.smithi111.stdout:0/879: read - d5/f110 zero size 2024-03-23T11:47:49.949 INFO:tasks.workunit.client.0.smithi063.stdout:3/949: mkdir d13/d15/d114/d12e 0 2024-03-23T11:47:49.950 INFO:tasks.workunit.client.0.smithi063.stdout:0/816: dread - d2/d14/d36/d77/d94/da6/d1f/f112 zero size 2024-03-23T11:47:49.952 INFO:tasks.workunit.client.1.smithi111.stdout:6/898: rmdir d2/d35 39 2024-03-23T11:47:49.953 INFO:tasks.workunit.client.0.smithi063.stdout:6/512: creat d1/dc/d8c/f96 x:0 0 0 2024-03-23T11:47:49.955 INFO:tasks.workunit.client.1.smithi111.stdout:1/886: unlink d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/f90 0 2024-03-23T11:47:49.956 INFO:tasks.workunit.client.1.smithi111.stdout:9/937: sync 2024-03-23T11:47:49.958 INFO:tasks.workunit.client.0.smithi063.stdout:0/817: write d2/d14/d36/d77/f120 [525716,46464] 0 2024-03-23T11:47:49.958 INFO:tasks.workunit.client.0.smithi063.stdout:6/513: dread - d1/dc/f5b zero size 2024-03-23T11:47:49.959 INFO:tasks.workunit.client.0.smithi063.stdout:0/818: chown d2 9798 1 2024-03-23T11:47:49.965 INFO:tasks.workunit.client.1.smithi111.stdout:8/738: dwrite d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f7f [0,4194304] 0 2024-03-23T11:47:49.966 INFO:tasks.workunit.client.1.smithi111.stdout:9/938: mkdir d0/d31/d41/df4/d12e 0 2024-03-23T11:47:49.968 INFO:tasks.workunit.client.0.smithi063.stdout:5/449: dread d2/d7/d2b/f39 [0,4194304] 0 2024-03-23T11:47:49.973 INFO:tasks.workunit.client.1.smithi111.stdout:4/969: dwrite d0/dc/d1f/db0/fd7 [0,4194304] 0 2024-03-23T11:47:49.975 INFO:tasks.workunit.client.0.smithi063.stdout:6/514: creat d1/dc/d3a/d57/f97 x:0 0 0 2024-03-23T11:47:49.984 INFO:tasks.workunit.client.1.smithi111.stdout:1/887: link d0/d2b/d37/d6a/fe4 d0/d2b/d37/d6a/d100/f131 0 2024-03-23T11:47:49.984 INFO:tasks.workunit.client.1.smithi111.stdout:4/970: mknod d0/dc/d89/dc9/c13e 0 2024-03-23T11:47:49.985 INFO:tasks.workunit.client.0.smithi063.stdout:0/819: dwrite d2/d6e/d8c/fe4 [0,4194304] 0 2024-03-23T11:47:49.991 INFO:tasks.workunit.client.0.smithi063.stdout:8/920: dread d6/f16 [0,4194304] 0 2024-03-23T11:47:49.994 INFO:tasks.workunit.client.1.smithi111.stdout:8/739: rmdir d5/d26/d2f/d62/d75/d4b/d51/da4/d97/db9 0 2024-03-23T11:47:49.996 INFO:tasks.workunit.client.0.smithi063.stdout:0/820: chown d2/d14/d36/d77/d94/da6/d28/d46/le9 595 1 2024-03-23T11:47:49.997 INFO:tasks.workunit.client.1.smithi111.stdout:1/888: mkdir d0/d2b/d7c/da9/d132 0 2024-03-23T11:47:49.997 INFO:tasks.workunit.client.1.smithi111.stdout:6/899: write d2/d17/f55 [3391635,84632] 0 2024-03-23T11:47:49.998 INFO:tasks.workunit.client.0.smithi063.stdout:9/666: dwrite d4/d31/d43/fdd [0,4194304] 0 2024-03-23T11:47:49.998 INFO:tasks.workunit.client.0.smithi063.stdout:7/627: write d5/d5c/d2f/d73/d20/f31 [1792671,126181] 0 2024-03-23T11:47:49.999 INFO:tasks.workunit.client.1.smithi111.stdout:0/880: truncate d5/db/d1f/d19/dbd/d64/df2/ff5 193406 0 2024-03-23T11:47:50.002 INFO:tasks.workunit.client.1.smithi111.stdout:9/939: creat d0/d4/d1c/d2d/f12f x:0 0 0 2024-03-23T11:47:50.003 INFO:tasks.workunit.client.0.smithi063.stdout:1/574: write d3/d49/d56/f67 [58814,117026] 0 2024-03-23T11:47:50.006 INFO:tasks.workunit.client.1.smithi111.stdout:8/740: unlink d5/d1b/c25 0 2024-03-23T11:47:50.007 INFO:tasks.workunit.client.1.smithi111.stdout:1/889: unlink d0/d6/d48/d4a/d6c/d83/d11d/d91/f9d 0 2024-03-23T11:47:50.009 INFO:tasks.workunit.client.1.smithi111.stdout:8/741: chown d5/d26/d31/d4d/fd4 0 1 2024-03-23T11:47:50.010 INFO:tasks.workunit.client.0.smithi063.stdout:5/450: write d2/d7/f23 [768323,47960] 0 2024-03-23T11:47:50.011 INFO:tasks.workunit.client.1.smithi111.stdout:0/881: creat d5/db/d1f/d47/d59/f127 x:0 0 0 2024-03-23T11:47:50.011 INFO:tasks.workunit.client.0.smithi063.stdout:4/672: dwrite d5/d1e/f75 [0,4194304] 0 2024-03-23T11:47:50.012 INFO:tasks.workunit.client.1.smithi111.stdout:8/742: chown d5/d26/d31/d4d 8233197 1 2024-03-23T11:47:50.013 INFO:tasks.workunit.client.0.smithi063.stdout:3/950: getdents d13/d22/d2d/d3f/dd4 0 2024-03-23T11:47:50.017 INFO:tasks.workunit.client.0.smithi063.stdout:0/821: mkdir d2/d12/d35/dcf/d121 0 2024-03-23T11:47:50.021 INFO:tasks.workunit.client.1.smithi111.stdout:1/890: unlink d0/d2b/f5a 0 2024-03-23T11:47:50.022 INFO:tasks.workunit.client.0.smithi063.stdout:8/921: mkdir d6/dd/d59/daf/dba/d135 0 2024-03-23T11:47:50.023 INFO:tasks.workunit.client.0.smithi063.stdout:0/822: fsync d2/d12/f68 0 2024-03-23T11:47:50.024 INFO:tasks.workunit.client.0.smithi063.stdout:9/667: symlink d4/d31/d70/lec 0 2024-03-23T11:47:50.027 INFO:tasks.workunit.client.1.smithi111.stdout:0/882: mknod d5/db/d1f/d19/d45/c128 0 2024-03-23T11:47:50.032 INFO:tasks.workunit.client.1.smithi111.stdout:8/743: mknod d5/d26/d31/cf3 0 2024-03-23T11:47:50.033 INFO:tasks.workunit.client.1.smithi111.stdout:1/891: getdents d0/d5/dd5/df6 0 2024-03-23T11:47:50.037 INFO:tasks.workunit.client.0.smithi063.stdout:5/451: creat d2/d7/d8/de/d2a/d3b/f99 x:0 0 0 2024-03-23T11:47:50.038 INFO:tasks.workunit.client.1.smithi111.stdout:6/900: dread d2/d1a/d1c/d2c/d51/f8f [4194304,4194304] 0 2024-03-23T11:47:50.039 INFO:tasks.workunit.client.1.smithi111.stdout:0/883: getdents d5/db/d1f/d19/dbd/d64/dc7/d100 0 2024-03-23T11:47:50.039 INFO:tasks.workunit.client.0.smithi063.stdout:3/951: rename d13/d22/d2d/d3f/d58/d6e/fbf to d13/d22/da9/f12f 0 2024-03-23T11:47:50.043 INFO:tasks.workunit.client.0.smithi063.stdout:6/515: link d1/f12 d1/dc/d3a/d57/f98 0 2024-03-23T11:47:50.043 INFO:tasks.workunit.client.0.smithi063.stdout:8/922: fdatasync d6/d1f/d54/d63/d90/d9e/fa8 0 2024-03-23T11:47:50.044 INFO:tasks.workunit.client.1.smithi111.stdout:6/901: creat d2/d1f/d45/d9a/ddd/f122 x:0 0 0 2024-03-23T11:47:50.044 INFO:tasks.workunit.client.1.smithi111.stdout:8/744: getdents d5/d32/d7d/da0 0 2024-03-23T11:47:50.045 INFO:tasks.workunit.client.0.smithi063.stdout:5/452: symlink d2/d7/d64/l9a 0 2024-03-23T11:47:50.045 INFO:tasks.workunit.client.1.smithi111.stdout:0/884: mknod d5/db/d1f/d19/dbd/d64/dc7/c129 0 2024-03-23T11:47:50.049 INFO:tasks.workunit.client.0.smithi063.stdout:9/668: rename d4/d31/d43/da9/l92 to d4/d31/d70/d8d/d90/dd9/led 0 2024-03-23T11:47:50.050 INFO:tasks.workunit.client.0.smithi063.stdout:3/952: unlink d13/d22/d2d/d3f/d58/d86/dad/lc0 0 2024-03-23T11:47:50.050 INFO:tasks.workunit.client.1.smithi111.stdout:0/885: rmdir d5/db/d1f/de4 39 2024-03-23T11:47:50.050 INFO:tasks.workunit.client.1.smithi111.stdout:3/945: dread d0/d1a/d1d/f24 [0,4194304] 0 2024-03-23T11:47:50.050 INFO:tasks.workunit.client.1.smithi111.stdout:6/902: dread d2/d1f/d45/d49/d8d/f94 [0,4194304] 0 2024-03-23T11:47:50.053 INFO:tasks.workunit.client.1.smithi111.stdout:8/745: mknod d5/d26/d2f/d62/db5/dce/cf4 0 2024-03-23T11:47:50.055 INFO:tasks.workunit.client.1.smithi111.stdout:0/886: unlink d5/db/d4d/da6/fdb 0 2024-03-23T11:47:50.056 INFO:tasks.workunit.client.0.smithi063.stdout:5/453: creat d2/d28/d3a/f9b x:0 0 0 2024-03-23T11:47:50.057 INFO:tasks.workunit.client.0.smithi063.stdout:4/673: getdents d5/d1e 0 2024-03-23T11:47:50.058 INFO:tasks.workunit.client.0.smithi063.stdout:0/823: rename d2/d14/d36/d77/d94/da6/d1f/d4f/fc1 to d2/d14/d36/d77/d94/da6/d25/f122 0 2024-03-23T11:47:50.058 INFO:tasks.workunit.client.1.smithi111.stdout:6/903: fdatasync d2/d1a/d1c/d2c/d51/fa9 0 2024-03-23T11:47:50.060 INFO:tasks.workunit.client.1.smithi111.stdout:8/746: mkdir d5/d26/d2f/d62/d75/df5 0 2024-03-23T11:47:50.065 INFO:tasks.workunit.client.1.smithi111.stdout:8/747: truncate d5/d26/d2f/d8f/fb7 521196 0 2024-03-23T11:47:50.065 INFO:tasks.workunit.client.1.smithi111.stdout:2/940: sync 2024-03-23T11:47:50.066 INFO:tasks.workunit.client.1.smithi111.stdout:8/748: write d5/d26/f74 [4341426,118477] 0 2024-03-23T11:47:50.066 INFO:tasks.workunit.client.0.smithi063.stdout:9/669: mknod d4/d31/d9c/cee 0 2024-03-23T11:47:50.066 INFO:tasks.workunit.client.0.smithi063.stdout:5/454: chown d2/d7/d8/d1d/l77 1670911 1 2024-03-23T11:47:50.068 INFO:tasks.workunit.client.0.smithi063.stdout:3/953: fsync d13/d15/d7e/f8f 0 2024-03-23T11:47:50.069 INFO:tasks.workunit.client.1.smithi111.stdout:3/946: dwrite d0/d1a/d1d/d26/d5a/f8e [0,4194304] 0 2024-03-23T11:47:50.071 INFO:tasks.workunit.client.0.smithi063.stdout:4/674: fsync d5/da/f19 0 2024-03-23T11:47:50.076 INFO:tasks.workunit.client.1.smithi111.stdout:3/947: write d0/d1a/d1d/dc5/ffc [118543,106787] 0 2024-03-23T11:47:50.076 INFO:tasks.workunit.client.0.smithi063.stdout:0/824: mknod d2/d14/d36/d77/d94/da6/d28/d46/d51/c123 0 2024-03-23T11:47:50.083 INFO:tasks.workunit.client.0.smithi063.stdout:7/628: dwrite d5/d5c/d2f/d73/da/d90/f9b [0,4194304] 0 2024-03-23T11:47:50.088 INFO:tasks.workunit.client.0.smithi063.stdout:1/575: dread d3/f5d [0,4194304] 0 2024-03-23T11:47:50.088 INFO:tasks.workunit.client.1.smithi111.stdout:8/749: mknod d5/d26/d2f/d62/d75/cf6 0 2024-03-23T11:47:50.089 INFO:tasks.workunit.client.1.smithi111.stdout:3/948: dread d0/d1a/d2b/ff0 [0,4194304] 0 2024-03-23T11:47:50.090 INFO:tasks.workunit.client.1.smithi111.stdout:6/904: chown d2/d35/da7/cc0 27269302 1 2024-03-23T11:47:50.097 INFO:tasks.workunit.client.1.smithi111.stdout:8/750: truncate d5/d26/d2f/d62/d75/d4b/d51/da4/d97/fa8 76032 0 2024-03-23T11:47:50.097 INFO:tasks.workunit.client.1.smithi111.stdout:6/905: dread d2/f28 [0,4194304] 0 2024-03-23T11:47:50.097 INFO:tasks.workunit.client.1.smithi111.stdout:9/940: dwrite d0/d31/d41/d9d/dd4/ff7 [0,4194304] 0 2024-03-23T11:47:50.099 INFO:tasks.workunit.client.0.smithi063.stdout:7/629: creat d5/fc7 x:0 0 0 2024-03-23T11:47:50.101 INFO:tasks.workunit.client.0.smithi063.stdout:6/516: write d1/dc/fd [8033396,82410] 0 2024-03-23T11:47:50.105 INFO:tasks.workunit.client.1.smithi111.stdout:0/887: write d5/db/d1f/d47/d6f/fe9 [796025,110688] 0 2024-03-23T11:47:50.111 INFO:tasks.workunit.client.1.smithi111.stdout:1/892: dwrite d0/d2b/d69/f8c [0,4194304] 0 2024-03-23T11:47:50.114 INFO:tasks.workunit.client.1.smithi111.stdout:7/907: sync 2024-03-23T11:47:50.117 INFO:tasks.workunit.client.1.smithi111.stdout:2/941: dwrite d2/de/d6c/dd1/dd6/feb [4194304,4194304] 0 2024-03-23T11:47:50.121 INFO:tasks.workunit.client.0.smithi063.stdout:6/517: mkdir d1/d2a/d41/d42/d99 0 2024-03-23T11:47:50.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:49 smithi063 ceph-mon[35189]: pgmap v44: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 107 MiB/s rd, 261 MiB/s wr, 583 op/s 2024-03-23T11:47:50.136 INFO:tasks.workunit.client.1.smithi111.stdout:8/751: dwrite d5/d26/d2f/d42/d63/f69 [0,4194304] 0 2024-03-23T11:47:50.139 INFO:tasks.workunit.client.0.smithi063.stdout:8/923: dwrite d6/d1c/d32/d37/d10a/dae/feb [0,4194304] 0 2024-03-23T11:47:50.141 INFO:tasks.workunit.client.0.smithi063.stdout:9/670: dwrite d4/d4c/f25 [0,4194304] 0 2024-03-23T11:47:50.146 INFO:tasks.workunit.client.1.smithi111.stdout:8/752: symlink d5/d26/d2f/d62/d75/d4b/d51/d5a/lf7 0 2024-03-23T11:47:50.148 INFO:tasks.workunit.client.1.smithi111.stdout:8/753: chown d5/d26/d31/d3b/f6e 0 1 2024-03-23T11:47:50.150 INFO:tasks.workunit.client.1.smithi111.stdout:9/941: creat d0/d4/d1c/d2d/d6e/f130 x:0 0 0 2024-03-23T11:47:50.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:49 smithi111 ceph-mon[44005]: pgmap v44: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 107 MiB/s rd, 261 MiB/s wr, 583 op/s 2024-03-23T11:47:50.154 INFO:tasks.workunit.client.1.smithi111.stdout:9/942: write d0/d4/d1c/d2d/d6e/f130 [476835,15312] 0 2024-03-23T11:47:50.158 INFO:tasks.workunit.client.1.smithi111.stdout:1/893: mknod d0/d6/d48/d4a/c133 0 2024-03-23T11:47:50.170 INFO:tasks.workunit.client.0.smithi063.stdout:9/671: chown f2 495863 1 2024-03-23T11:47:50.171 INFO:tasks.workunit.client.0.smithi063.stdout:4/675: dwrite d5/d1e/d2b/f54 [0,4194304] 0 2024-03-23T11:47:50.172 INFO:tasks.workunit.client.0.smithi063.stdout:3/954: dwrite d13/d22/f111 [0,4194304] 0 2024-03-23T11:47:50.172 INFO:tasks.workunit.client.1.smithi111.stdout:9/943: mkdir d0/d31/d41/d44/d131 0 2024-03-23T11:47:50.172 INFO:tasks.workunit.client.1.smithi111.stdout:7/908: symlink d14/d1b/d105/l10d 0 2024-03-23T11:47:50.178 INFO:tasks.workunit.client.1.smithi111.stdout:1/894: getdents d0/d6/d48/d4a/d6c/d83/d11d/dfe/d108 0 2024-03-23T11:47:50.181 INFO:tasks.workunit.client.1.smithi111.stdout:8/754: unlink d5/l9 0 2024-03-23T11:47:50.182 INFO:tasks.workunit.client.0.smithi063.stdout:8/924: fsync d6/d1f/d54/fcc 0 2024-03-23T11:47:50.183 INFO:tasks.workunit.client.0.smithi063.stdout:0/825: dwrite d2/d14/de8/fb6 [0,4194304] 0 2024-03-23T11:47:50.183 INFO:tasks.workunit.client.1.smithi111.stdout:3/949: getdents d0/d1a/d2d/df5/d11a 0 2024-03-23T11:47:50.186 INFO:tasks.workunit.client.1.smithi111.stdout:8/755: fdatasync d5/d26/d2f/d42/dd3/fde 0 2024-03-23T11:47:50.188 INFO:tasks.workunit.client.1.smithi111.stdout:1/895: mkdir d0/def/d134 0 2024-03-23T11:47:50.189 INFO:tasks.workunit.client.0.smithi063.stdout:5/455: dwrite d2/d7/d8/d1d/f1f [0,4194304] 0 2024-03-23T11:47:50.190 INFO:tasks.workunit.client.1.smithi111.stdout:3/950: mknod d0/d1a/d2b/d10e/dcd/c13b 0 2024-03-23T11:47:50.194 INFO:tasks.workunit.client.1.smithi111.stdout:7/909: dread f9 [4194304,4194304] 0 2024-03-23T11:47:50.194 INFO:tasks.workunit.client.1.smithi111.stdout:6/906: dread d2/d35/f9c [0,4194304] 0 2024-03-23T11:47:50.195 INFO:tasks.workunit.client.1.smithi111.stdout:1/896: creat d0/d6/d48/d4d/dab/dbd/f135 x:0 0 0 2024-03-23T11:47:50.196 INFO:tasks.workunit.client.1.smithi111.stdout:8/756: rename d5/d26/d2f/d42/l43 to d5/d26/d2f/lf8 0 2024-03-23T11:47:50.196 INFO:tasks.workunit.client.0.smithi063.stdout:1/576: dwrite d3/df/f1a [8388608,4194304] 0 2024-03-23T11:47:50.201 INFO:tasks.workunit.client.0.smithi063.stdout:9/672: read - d4/d31/d43/fe8 zero size 2024-03-23T11:47:50.202 INFO:tasks.workunit.client.0.smithi063.stdout:5/456: chown d2/d28/d3a/d46/c90 10381143 1 2024-03-23T11:47:50.203 INFO:tasks.workunit.client.1.smithi111.stdout:3/951: rmdir d0/d1a/d2b/d10e/d19/d91/da3 39 2024-03-23T11:47:50.204 INFO:tasks.workunit.client.1.smithi111.stdout:8/757: mkdir d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/df9 0 2024-03-23T11:47:50.207 INFO:tasks.workunit.client.0.smithi063.stdout:4/676: chown d5/d1e/d2b/f7c 52525989 1 2024-03-23T11:47:50.208 INFO:tasks.workunit.client.0.smithi063.stdout:7/630: dwrite d5/f12 [0,4194304] 0 2024-03-23T11:47:50.209 INFO:tasks.workunit.client.0.smithi063.stdout:0/826: fsync d2/d12/d32/d6d/fd0 0 2024-03-23T11:47:50.213 INFO:tasks.workunit.client.0.smithi063.stdout:5/457: mknod d2/d7/d64/c9c 0 2024-03-23T11:47:50.218 INFO:tasks.workunit.client.0.smithi063.stdout:4/677: fdatasync d5/da/d20/d4b/d29/d38/d93/fbc 0 2024-03-23T11:47:50.219 INFO:tasks.workunit.client.0.smithi063.stdout:3/955: fsync d13/d22/f111 0 2024-03-23T11:47:50.228 INFO:tasks.workunit.client.0.smithi063.stdout:0/827: rmdir d2/d14/d36/d77/d94/da6/d25 39 2024-03-23T11:47:50.228 INFO:tasks.workunit.client.1.smithi111.stdout:4/971: sync 2024-03-23T11:47:50.231 INFO:tasks.workunit.client.0.smithi063.stdout:5/458: mkdir d2/d28/d3a/d46/d71/d9d 0 2024-03-23T11:47:50.236 INFO:tasks.workunit.client.1.smithi111.stdout:6/907: mkdir d2/d35/d123 0 2024-03-23T11:47:50.241 INFO:tasks.workunit.client.0.smithi063.stdout:8/925: rename d6/d1c/d27/f4a to d6/d1c/f136 0 2024-03-23T11:47:50.243 INFO:tasks.workunit.client.0.smithi063.stdout:7/631: dwrite f1 [4194304,4194304] 0 2024-03-23T11:47:50.244 INFO:tasks.workunit.client.0.smithi063.stdout:5/459: creat d2/d7/d8/de/d2a/d3b/d85/f9e x:0 0 0 2024-03-23T11:47:50.245 INFO:tasks.workunit.client.1.smithi111.stdout:4/972: mknod d0/d95/c13f 0 2024-03-23T11:47:50.245 INFO:tasks.workunit.client.1.smithi111.stdout:3/952: dwrite d0/d1a/d1d/fb8 [0,4194304] 0 2024-03-23T11:47:50.246 INFO:tasks.workunit.client.1.smithi111.stdout:8/758: unlink d5/d26/d2f/lf8 0 2024-03-23T11:47:50.247 INFO:tasks.workunit.client.1.smithi111.stdout:3/953: chown d0/d1a/d1d/l6d 3662 1 2024-03-23T11:47:50.247 INFO:tasks.workunit.client.1.smithi111.stdout:3/954: stat d0 0 2024-03-23T11:47:50.250 INFO:tasks.workunit.client.1.smithi111.stdout:6/908: creat d2/d1a/d1c/d2c/d51/def/f124 x:0 0 0 2024-03-23T11:47:50.251 INFO:tasks.workunit.client.0.smithi063.stdout:3/956: creat d13/d22/d2d/d3f/f130 x:0 0 0 2024-03-23T11:47:50.255 INFO:tasks.workunit.client.0.smithi063.stdout:9/673: dread d4/d31/d43/d49/f9a [0,4194304] 0 2024-03-23T11:47:50.255 INFO:tasks.workunit.client.1.smithi111.stdout:8/759: unlink d5/d26/d2f/d62/d75/d4b/d51/da4/d97/f9c 0 2024-03-23T11:47:50.260 INFO:tasks.workunit.client.0.smithi063.stdout:4/678: rename d5/da/d20/d4b/d16/d50/f69 to d5/da/d7e/dbf/fc2 0 2024-03-23T11:47:50.262 INFO:tasks.workunit.client.1.smithi111.stdout:8/760: symlink d5/d26/d2f/d62/d75/d4b/d51/d5a/dbe/lfa 0 2024-03-23T11:47:50.263 INFO:tasks.workunit.client.0.smithi063.stdout:9/674: mknod d4/d31/d43/da9/da6/db5/cef 0 2024-03-23T11:47:50.264 INFO:tasks.workunit.client.1.smithi111.stdout:8/761: fdatasync d5/d26/d2f/d62/d75/f6b 0 2024-03-23T11:47:50.264 INFO:tasks.workunit.client.1.smithi111.stdout:2/942: sync 2024-03-23T11:47:50.269 INFO:tasks.workunit.client.0.smithi063.stdout:8/926: link d6/d1f/d54/d5c/dc5/c123 d6/d30/c137 0 2024-03-23T11:47:50.269 INFO:tasks.workunit.client.0.smithi063.stdout:1/577: dread d3/df/f36 [0,4194304] 0 2024-03-23T11:47:50.270 INFO:tasks.workunit.client.1.smithi111.stdout:3/955: mknod d0/d1a/d2b/d10e/dcd/c13c 0 2024-03-23T11:47:50.273 INFO:tasks.workunit.client.0.smithi063.stdout:0/828: rename d2/d14/d36/d77/d94/da6 to d2/d12/d32/d6d/d124 0 2024-03-23T11:47:50.274 INFO:tasks.workunit.client.1.smithi111.stdout:4/973: unlink d0/c5 0 2024-03-23T11:47:50.276 INFO:tasks.workunit.client.1.smithi111.stdout:4/974: chown d0/dc/d4c/d2a/df2/d123/da/lae 64 1 2024-03-23T11:47:50.278 INFO:tasks.workunit.client.0.smithi063.stdout:8/927: creat d6/d1f/d3e/f138 x:0 0 0 2024-03-23T11:47:50.278 INFO:tasks.workunit.client.1.smithi111.stdout:6/909: dread d2/d17/f33 [0,4194304] 0 2024-03-23T11:47:50.280 INFO:tasks.workunit.client.1.smithi111.stdout:2/943: link d2/d11/d17/d3e/d6b/d9d/d52/dba/f137 d2/de/d12/d70/d96/f14f 0 2024-03-23T11:47:50.284 INFO:tasks.workunit.client.1.smithi111.stdout:4/975: rename d0/dc/ca6 to d0/dc/d78/d13b/c140 0 2024-03-23T11:47:50.288 INFO:tasks.workunit.client.1.smithi111.stdout:2/944: fdatasync d2/d7/da/f101 0 2024-03-23T11:47:50.288 INFO:tasks.workunit.client.0.smithi063.stdout:1/578: symlink d3/d49/d56/d65/d88/d92/lb1 0 2024-03-23T11:47:50.290 INFO:tasks.workunit.client.1.smithi111.stdout:6/910: symlink d2/d1a/d1c/d2c/d51/l125 0 2024-03-23T11:47:50.291 INFO:tasks.workunit.client.1.smithi111.stdout:6/911: chown d2/d80/d100 6 1 2024-03-23T11:47:50.293 INFO:tasks.workunit.client.0.smithi063.stdout:3/957: rename d13/d22/d2d/d3f/f9a to d13/f131 0 2024-03-23T11:47:50.293 INFO:tasks.workunit.client.0.smithi063.stdout:8/928: truncate d6/d1c/d32/d37/ff4 657922 0 2024-03-23T11:47:50.293 INFO:tasks.workunit.client.1.smithi111.stdout:2/945: truncate d2/d32/d91/f85 564006 0 2024-03-23T11:47:50.295 INFO:tasks.workunit.client.0.smithi063.stdout:3/958: readlink d13/d22/d2d/d7d/lfc 0 2024-03-23T11:47:50.295 INFO:tasks.workunit.client.1.smithi111.stdout:2/946: chown d2/d11/d17/d3e/d6b/d2e/d3f/f4d 3 1 2024-03-23T11:47:50.296 INFO:tasks.workunit.client.1.smithi111.stdout:0/888: write d5/db/f92 [707220,3089] 0 2024-03-23T11:47:50.297 INFO:tasks.workunit.client.0.smithi063.stdout:1/579: dread d3/d49/d56/f7f [0,4194304] 0 2024-03-23T11:47:50.297 INFO:tasks.workunit.client.0.smithi063.stdout:9/675: rename d4/d31/d43/da9/da6/l2f to d4/d31/d9c/d9e/lf0 0 2024-03-23T11:47:50.297 INFO:tasks.workunit.client.1.smithi111.stdout:0/889: write d5/f110 [87986,52695] 0 2024-03-23T11:47:50.298 INFO:tasks.workunit.client.0.smithi063.stdout:8/929: symlink d6/dd/d59/daf/dba/l139 0 2024-03-23T11:47:50.298 INFO:tasks.workunit.client.1.smithi111.stdout:9/944: write d0/d4/f43 [3734206,119245] 0 2024-03-23T11:47:50.300 INFO:tasks.workunit.client.0.smithi063.stdout:9/676: chown d4/d31/d9c/d9e/fc5 0 1 2024-03-23T11:47:50.302 INFO:tasks.workunit.client.1.smithi111.stdout:7/910: write d14/d37/d43/f55 [162790,67658] 0 2024-03-23T11:47:50.303 INFO:tasks.workunit.client.0.smithi063.stdout:3/959: unlink d13/d22/d2d/d3f/d58/d6e/d8d/f95 0 2024-03-23T11:47:50.303 INFO:tasks.workunit.client.0.smithi063.stdout:8/930: symlink d6/d30/l13a 0 2024-03-23T11:47:50.306 INFO:tasks.workunit.client.0.smithi063.stdout:1/580: symlink d3/d49/d56/d65/d84/lb2 0 2024-03-23T11:47:50.307 INFO:tasks.workunit.client.1.smithi111.stdout:0/890: truncate d5/db/d1f/d47/f5a 4930663 0 2024-03-23T11:47:50.309 INFO:tasks.workunit.client.1.smithi111.stdout:1/897: dwrite d0/d6/d48/d4a/d6c/d83/d11d/f25 [0,4194304] 0 2024-03-23T11:47:50.310 INFO:tasks.workunit.client.1.smithi111.stdout:3/956: truncate d0/d1a/d2b/d10e/d19/d5e/da6/deb/ff9 843339 0 2024-03-23T11:47:50.310 INFO:tasks.workunit.client.0.smithi063.stdout:1/581: symlink d3/d1c/lb3 0 2024-03-23T11:47:50.312 INFO:tasks.workunit.client.1.smithi111.stdout:7/911: creat d14/d44/d70/f10e x:0 0 0 2024-03-23T11:47:50.312 INFO:tasks.workunit.client.1.smithi111.stdout:2/947: link d2/d11/d17/d3e/d5d/fa4 d2/d11/d17/d3e/d6b/d2e/d76/f150 0 2024-03-23T11:47:50.314 INFO:tasks.workunit.client.0.smithi063.stdout:3/960: getdents d13/d22/d2d/d3e 0 2024-03-23T11:47:50.317 INFO:tasks.workunit.client.1.smithi111.stdout:9/945: dwrite d0/d4/d6a/da7/fd5 [0,4194304] 0 2024-03-23T11:47:50.319 INFO:tasks.workunit.client.1.smithi111.stdout:7/912: chown d14/d1b/d1e/d2c/c5b 44879575 1 2024-03-23T11:47:50.319 INFO:tasks.workunit.client.0.smithi063.stdout:1/582: mknod d3/d35/cb4 0 2024-03-23T11:47:50.321 INFO:tasks.workunit.client.1.smithi111.stdout:4/976: write d0/dc/d4c/d2a/df2/d123/f1b [5017857,85697] 0 2024-03-23T11:47:50.321 INFO:tasks.workunit.client.1.smithi111.stdout:1/898: creat d0/d6/d48/d4d/dab/dbd/f136 x:0 0 0 2024-03-23T11:47:50.322 INFO:tasks.workunit.client.1.smithi111.stdout:3/957: symlink d0/d1a/d1d/d26/d5a/d60/l13d 0 2024-03-23T11:47:50.324 INFO:tasks.workunit.client.0.smithi063.stdout:1/583: fsync d3/d3f/d4d/d52/fab 0 2024-03-23T11:47:50.325 INFO:tasks.workunit.client.0.smithi063.stdout:8/931: dread d6/d1c/d32/d37/d10a/dae/feb [0,4194304] 0 2024-03-23T11:47:50.325 INFO:tasks.workunit.client.1.smithi111.stdout:9/946: unlink d0/d4/d1c/d2d/d3b/d42/d4f/d95/c10e 0 2024-03-23T11:47:50.326 INFO:tasks.workunit.client.1.smithi111.stdout:7/913: mkdir d14/d37/d43/db1/d10f 0 2024-03-23T11:47:50.331 INFO:tasks.workunit.client.1.smithi111.stdout:1/899: chown d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/c93 38327392 1 2024-03-23T11:47:50.331 INFO:tasks.workunit.client.0.smithi063.stdout:8/932: symlink d6/d1f/d54/d63/d90/d5d/d7d/db4/l13b 0 2024-03-23T11:47:50.334 INFO:tasks.workunit.client.1.smithi111.stdout:0/891: write d5/db/d1f/d48/f6a [2382895,59050] 0 2024-03-23T11:47:50.335 INFO:tasks.workunit.client.0.smithi063.stdout:8/933: readlink d6/d1f/d54/d63/d90/d5d/d7d/d9a/l11e 0 2024-03-23T11:47:50.336 INFO:tasks.workunit.client.1.smithi111.stdout:2/948: dwrite d2/f6 [0,4194304] 0 2024-03-23T11:47:50.337 INFO:tasks.workunit.client.0.smithi063.stdout:8/934: rmdir d6 39 2024-03-23T11:47:50.339 INFO:tasks.workunit.client.0.smithi063.stdout:1/584: dread d3/f5d [0,4194304] 0 2024-03-23T11:47:50.342 INFO:tasks.workunit.client.1.smithi111.stdout:4/977: creat d0/dc/d4c/d2a/d97/d4f/dba/f141 x:0 0 0 2024-03-23T11:47:50.343 INFO:tasks.workunit.client.1.smithi111.stdout:3/958: write d0/d1a/d1d/d26/f46 [55276,110314] 0 2024-03-23T11:47:50.348 INFO:tasks.workunit.client.0.smithi063.stdout:1/585: mkdir d3/d1c/d4b/db5 0 2024-03-23T11:47:50.349 INFO:tasks.workunit.client.1.smithi111.stdout:1/900: mkdir d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d137 0 2024-03-23T11:47:50.350 INFO:tasks.workunit.client.0.smithi063.stdout:8/935: fsync d6/d1f/d54/d63/d90/d15/f24 0 2024-03-23T11:47:50.351 INFO:tasks.workunit.client.1.smithi111.stdout:2/949: write d2/d7/da/f103 [652656,1488] 0 2024-03-23T11:47:50.354 INFO:tasks.workunit.client.1.smithi111.stdout:3/959: dread - d0/fe7 zero size 2024-03-23T11:47:50.358 INFO:tasks.workunit.client.1.smithi111.stdout:9/947: link d0/d4/d1c/c107 d0/d31/d41/d71/d8f/def/c132 0 2024-03-23T11:47:50.361 INFO:tasks.workunit.client.1.smithi111.stdout:0/892: dread d5/db/d1f/d48/f66 [0,4194304] 0 2024-03-23T11:47:50.361 INFO:tasks.workunit.client.0.smithi063.stdout:1/586: read d3/f9 [630786,78146] 0 2024-03-23T11:47:50.363 INFO:tasks.workunit.client.1.smithi111.stdout:2/950: getdents d2/d11/d17/d3e/d6b 0 2024-03-23T11:47:50.364 INFO:tasks.workunit.client.0.smithi063.stdout:8/936: rename d6/dd/d59/db2/dbe/dfb/fed to d6/d1f/d54/d63/d90/f13c 0 2024-03-23T11:47:50.364 INFO:tasks.workunit.client.1.smithi111.stdout:7/914: truncate d14/d1b/d1e/d2c/d42/d5e/d88/d5f/f68 3689355 0 2024-03-23T11:47:50.366 INFO:tasks.workunit.client.1.smithi111.stdout:7/915: stat d14/f1c 0 2024-03-23T11:47:50.367 INFO:tasks.workunit.client.1.smithi111.stdout:3/960: write d0/d1a/d4f/d90/fce [4904207,123853] 0 2024-03-23T11:47:50.369 INFO:tasks.workunit.client.1.smithi111.stdout:2/951: truncate d2/d7/f38 3670688 0 2024-03-23T11:47:50.370 INFO:tasks.workunit.client.0.smithi063.stdout:5/460: write d2/d17/f51 [4229415,122525] 0 2024-03-23T11:47:50.372 INFO:tasks.workunit.client.1.smithi111.stdout:7/916: write d14/d44/d70/f7c [2950584,33255] 0 2024-03-23T11:47:50.374 INFO:tasks.workunit.client.0.smithi063.stdout:8/937: getdents d6/d1c/d32/d37/d10a 0 2024-03-23T11:47:50.377 INFO:tasks.workunit.client.0.smithi063.stdout:5/461: chown d2/d28/c5e 55 1 2024-03-23T11:47:50.384 INFO:tasks.workunit.client.1.smithi111.stdout:3/961: write d0/d1a/f28 [3728875,34254] 0 2024-03-23T11:47:50.386 INFO:tasks.workunit.client.1.smithi111.stdout:1/901: dwrite d0/d6/d48/d4a/d6c/d83/d11d/f6e [4194304,4194304] 0 2024-03-23T11:47:50.388 INFO:tasks.workunit.client.1.smithi111.stdout:4/978: dwrite d0/dc/d1f/f8a [0,4194304] 0 2024-03-23T11:47:50.389 INFO:tasks.workunit.client.1.smithi111.stdout:2/952: mknod d2/d7/da/d19/d131/c151 0 2024-03-23T11:47:50.390 INFO:tasks.workunit.client.1.smithi111.stdout:6/912: sync 2024-03-23T11:47:50.390 INFO:tasks.workunit.client.1.smithi111.stdout:9/948: dwrite d0/d4/d10/f74 [0,4194304] 0 2024-03-23T11:47:50.399 INFO:tasks.workunit.client.1.smithi111.stdout:0/893: dwrite d5/db/d1f/d47/d59/f127 [0,4194304] 0 2024-03-23T11:47:50.401 INFO:tasks.workunit.client.1.smithi111.stdout:1/902: write d0/d6/f11c [2222607,125752] 0 2024-03-23T11:47:50.401 INFO:tasks.workunit.client.1.smithi111.stdout:2/953: fdatasync d2/de/d6c/dd1/fe2 0 2024-03-23T11:47:50.402 INFO:tasks.workunit.client.0.smithi063.stdout:4/679: dwrite d5/da/d72/fb1 [0,4194304] 0 2024-03-23T11:47:50.407 INFO:tasks.workunit.client.0.smithi063.stdout:8/938: rename d6/d1f/d54/d63/d90 to d6/dd/d59/d13d 0 2024-03-23T11:47:50.410 INFO:tasks.workunit.client.0.smithi063.stdout:0/829: dwrite d2/d48/fcb [0,4194304] 0 2024-03-23T11:47:50.411 INFO:tasks.workunit.client.1.smithi111.stdout:9/949: rename d0/d4/d10/d91/fda to d0/d31/d41/d9d/f133 0 2024-03-23T11:47:50.416 INFO:tasks.workunit.client.0.smithi063.stdout:1/587: write d3/df/d31/f41 [709986,96570] 0 2024-03-23T11:47:50.417 INFO:tasks.workunit.client.1.smithi111.stdout:8/762: dwrite d5/d1b/f2b [0,4194304] 0 2024-03-23T11:47:50.423 INFO:tasks.workunit.client.1.smithi111.stdout:6/913: truncate d2/d1f/d45/f56 2348759 0 2024-03-23T11:47:50.427 INFO:tasks.workunit.client.1.smithi111.stdout:0/894: creat d5/db/d1f/d19/dbd/d64/d91/ddd/f12a x:0 0 0 2024-03-23T11:47:50.429 INFO:tasks.workunit.client.1.smithi111.stdout:1/903: rename d0/d6/d48/d4a/d6c/d83/d11d/d1e/d63/l6d to d0/d2b/d37/d6a/l138 0 2024-03-23T11:47:50.431 INFO:tasks.workunit.client.0.smithi063.stdout:3/961: dwrite d13/d22/d2d/d3f/d58/fff [0,4194304] 0 2024-03-23T11:47:50.433 INFO:tasks.workunit.client.1.smithi111.stdout:2/954: creat d2/d11/d17/d1f/da0/dd2/d145/f152 x:0 0 0 2024-03-23T11:47:50.434 INFO:tasks.workunit.client.1.smithi111.stdout:9/950: rename d0/d31/lab to d0/d4/d1c/d36/d28/de6/l134 0 2024-03-23T11:47:50.435 INFO:tasks.workunit.client.1.smithi111.stdout:7/917: dread d14/d1b/d1e/d2c/d42/d5e/f6c [0,4194304] 0 2024-03-23T11:47:50.435 INFO:tasks.workunit.client.1.smithi111.stdout:0/895: mkdir d5/db/d1f/d47/d87/ddf/d12b 0 2024-03-23T11:47:50.435 INFO:tasks.workunit.client.0.smithi063.stdout:0/830: mknod d2/d48/de2/c125 0 2024-03-23T11:47:50.436 INFO:tasks.workunit.client.0.smithi063.stdout:5/462: dwrite d2/d28/d3a/f8d [0,4194304] 0 2024-03-23T11:47:50.437 INFO:tasks.workunit.client.1.smithi111.stdout:6/914: symlink d2/d1a/d1c/l126 0 2024-03-23T11:47:50.438 INFO:tasks.workunit.client.0.smithi063.stdout:9/677: dwrite d4/f14 [0,4194304] 0 2024-03-23T11:47:50.440 INFO:tasks.workunit.client.1.smithi111.stdout:7/918: write d14/d37/d54/fe4 [183736,27058] 0 2024-03-23T11:47:50.440 INFO:tasks.workunit.client.1.smithi111.stdout:4/979: dwrite d0/d95/fb8 [4194304,4194304] 0 2024-03-23T11:47:50.442 INFO:tasks.workunit.client.0.smithi063.stdout:1/588: chown d3/d1c/f68 429328 1 2024-03-23T11:47:50.445 INFO:tasks.workunit.client.1.smithi111.stdout:9/951: symlink d0/d4/d6a/da7/l135 0 2024-03-23T11:47:50.446 INFO:tasks.workunit.client.1.smithi111.stdout:6/915: dread - d2/d1a/d1c/d2c/ff4 zero size 2024-03-23T11:47:50.448 INFO:tasks.workunit.client.0.smithi063.stdout:3/962: creat d13/d66/d112/d124/f132 x:0 0 0 2024-03-23T11:47:50.449 INFO:tasks.workunit.client.1.smithi111.stdout:9/952: chown d0/d4/d10/c70 2002363 1 2024-03-23T11:47:50.451 INFO:tasks.workunit.client.1.smithi111.stdout:6/916: mkdir d2/d1a/d1c/d2c/d127 0 2024-03-23T11:47:50.461 INFO:tasks.workunit.client.1.smithi111.stdout:8/763: write d5/f2d [6823684,47890] 0 2024-03-23T11:47:50.463 INFO:tasks.workunit.client.0.smithi063.stdout:0/831: mkdir d2/d12/d32/d6d/d124/d1f/d4f/d126 0 2024-03-23T11:47:50.464 INFO:tasks.workunit.client.1.smithi111.stdout:6/917: rename d2/d80/d100/l11b to d2/d1f/d45/d9a/ddd/d110/l128 0 2024-03-23T11:47:50.473 INFO:tasks.workunit.client.1.smithi111.stdout:6/918: unlink d2/d1a/c5b 0 2024-03-23T11:47:50.475 INFO:tasks.workunit.client.1.smithi111.stdout:9/953: dwrite d0/d31/d41/d9d/dd4/ff7 [0,4194304] 0 2024-03-23T11:47:50.476 INFO:tasks.workunit.client.0.smithi063.stdout:9/678: link d4/d31/d43/d49/fa4 d4/d31/d70/d8d/d90/ff1 0 2024-03-23T11:47:50.476 INFO:tasks.workunit.client.0.smithi063.stdout:1/589: dwrite d3/d49/d56/d65/f7c [4194304,4194304] 0 2024-03-23T11:47:50.476 INFO:tasks.workunit.client.0.smithi063.stdout:3/963: mkdir d13/d22/d133 0 2024-03-23T11:47:50.477 INFO:tasks.workunit.client.0.smithi063.stdout:4/680: write d5/da/d20/d4b/f98 [3390276,26473] 0 2024-03-23T11:47:50.480 INFO:tasks.workunit.client.1.smithi111.stdout:6/919: unlink d2/d1f/d45/f4e 0 2024-03-23T11:47:50.484 INFO:tasks.workunit.client.1.smithi111.stdout:4/980: write d0/dc/d89/f99 [3123464,21650] 0 2024-03-23T11:47:50.488 INFO:tasks.workunit.client.0.smithi063.stdout:1/590: write d3/d49/d56/d65/f7c [4096050,112556] 0 2024-03-23T11:47:50.489 INFO:tasks.workunit.client.1.smithi111.stdout:9/954: rename d0/d31/d41/d71/cf6 to d0/d4/d1c/d2d/c136 0 2024-03-23T11:47:50.489 INFO:tasks.workunit.client.1.smithi111.stdout:4/981: fdatasync d0/dc/d78/f85 0 2024-03-23T11:47:50.502 INFO:tasks.workunit.client.1.smithi111.stdout:9/955: mknod d0/d31/d41/d71/d8f/d102/c137 0 2024-03-23T11:47:50.503 INFO:tasks.workunit.client.1.smithi111.stdout:4/982: truncate d0/d95/d9b/db3/db6/fa3 4653150 0 2024-03-23T11:47:50.503 INFO:tasks.workunit.client.1.smithi111.stdout:8/764: truncate d5/d26/d2f/d62/d75/d4b/fb1 2355356 0 2024-03-23T11:47:50.504 INFO:tasks.workunit.client.0.smithi063.stdout:1/591: fdatasync d3/df/f1a 0 2024-03-23T11:47:50.504 INFO:tasks.workunit.client.0.smithi063.stdout:4/681: rmdir d5/da/d7e 39 2024-03-23T11:47:50.507 INFO:tasks.workunit.client.1.smithi111.stdout:8/765: chown d5/d26/d31/d3b/f70 3892429 1 2024-03-23T11:47:50.509 INFO:tasks.workunit.client.1.smithi111.stdout:1/904: dread d0/d6/f22 [0,4194304] 0 2024-03-23T11:47:50.512 INFO:tasks.workunit.client.0.smithi063.stdout:8/939: write d6/d1f/d54/d5c/d76/f92 [4011711,66252] 0 2024-03-23T11:47:50.517 INFO:tasks.workunit.client.1.smithi111.stdout:1/905: read d0/f1 [138209,39133] 0 2024-03-23T11:47:50.518 INFO:tasks.workunit.client.1.smithi111.stdout:4/983: rmdir d0/dc/d4c/d2a/d97/d4d/de3/d105/d136 0 2024-03-23T11:47:50.518 INFO:tasks.workunit.client.1.smithi111.stdout:8/766: creat d5/d26/d2f/d62/d75/d4b/d51/d5a/dbe/ffb x:0 0 0 2024-03-23T11:47:50.523 INFO:tasks.workunit.client.0.smithi063.stdout:0/832: write d2/d12/f57 [756765,128180] 0 2024-03-23T11:47:50.525 INFO:tasks.workunit.client.1.smithi111.stdout:8/767: creat d5/d26/d2f/d62/d7e/dae/ffc x:0 0 0 2024-03-23T11:47:50.527 INFO:tasks.workunit.client.1.smithi111.stdout:4/984: read - d0/dc/d4c/d2a/d97/d4f/dba/f106 zero size 2024-03-23T11:47:50.532 INFO:tasks.workunit.client.1.smithi111.stdout:4/985: rmdir d0/d17/d86 39 2024-03-23T11:47:50.534 INFO:tasks.workunit.client.1.smithi111.stdout:3/962: sync 2024-03-23T11:47:50.538 INFO:tasks.workunit.client.0.smithi063.stdout:9/679: unlink d4/lda 0 2024-03-23T11:47:50.538 INFO:tasks.workunit.client.1.smithi111.stdout:4/986: read d0/fa8 [761835,96630] 0 2024-03-23T11:47:50.540 INFO:tasks.workunit.client.1.smithi111.stdout:3/963: mknod d0/d55/c13e 0 2024-03-23T11:47:50.540 INFO:tasks.workunit.client.1.smithi111.stdout:1/906: dwrite d0/d6/d48/d4a/d6c/d83/d11d/f6e [4194304,4194304] 0 2024-03-23T11:47:50.543 INFO:tasks.workunit.client.0.smithi063.stdout:1/592: mkdir d3/d49/d56/d65/db6 0 2024-03-23T11:47:50.543 INFO:tasks.workunit.client.1.smithi111.stdout:3/964: dread - d0/db9/f113 zero size 2024-03-23T11:47:50.543 INFO:tasks.workunit.client.1.smithi111.stdout:8/768: dread d5/d26/d31/d3b/f70 [0,4194304] 0 2024-03-23T11:47:50.547 INFO:tasks.workunit.client.0.smithi063.stdout:9/680: link d4/d31/d43/fe8 d4/d31/d70/ff2 0 2024-03-23T11:47:50.548 INFO:tasks.workunit.client.1.smithi111.stdout:1/907: unlink d0/d2b/d69/fcb 0 2024-03-23T11:47:50.550 INFO:tasks.workunit.client.1.smithi111.stdout:4/987: truncate d0/dc/d4c/d2a/df2/d123/fee 801073 0 2024-03-23T11:47:50.551 INFO:tasks.workunit.client.1.smithi111.stdout:3/965: creat d0/d1a/d4f/d90/f13f x:0 0 0 2024-03-23T11:47:50.555 INFO:tasks.workunit.client.1.smithi111.stdout:1/908: creat d0/d6/d48/d4a/d6c/d83/f139 x:0 0 0 2024-03-23T11:47:50.555 INFO:tasks.workunit.client.1.smithi111.stdout:0/896: dread d5/db/d1f/d47/d59/d5b/fb5 [4194304,4194304] 0 2024-03-23T11:47:50.557 INFO:tasks.workunit.client.1.smithi111.stdout:9/956: dread d0/d4/f15 [0,4194304] 0 2024-03-23T11:47:50.557 INFO:tasks.workunit.client.1.smithi111.stdout:0/897: chown d5/db/d1f/d48/f66 0 1 2024-03-23T11:47:50.558 INFO:tasks.workunit.client.1.smithi111.stdout:4/988: creat d0/dc/d4c/d2a/df2/dfc/f142 x:0 0 0 2024-03-23T11:47:50.559 INFO:tasks.workunit.client.1.smithi111.stdout:1/909: chown d0/d2b/d37/d6a/l138 22 1 2024-03-23T11:47:50.561 INFO:tasks.workunit.client.1.smithi111.stdout:9/957: symlink d0/d31/d41/df4/l138 0 2024-03-23T11:47:50.562 INFO:tasks.workunit.client.0.smithi063.stdout:4/682: dread d5/da/d20/d4b/d16/d32/fb0 [0,4194304] 0 2024-03-23T11:47:50.565 INFO:tasks.workunit.client.1.smithi111.stdout:4/989: mknod d0/dc/d1f/d5e/d134/c143 0 2024-03-23T11:47:50.566 INFO:tasks.workunit.client.1.smithi111.stdout:0/898: creat d5/db/d1f/f12c x:0 0 0 2024-03-23T11:47:50.570 INFO:tasks.workunit.client.1.smithi111.stdout:4/990: creat d0/dc/d4c/d2a/d97/d4f/dba/f144 x:0 0 0 2024-03-23T11:47:50.570 INFO:tasks.workunit.client.0.smithi063.stdout:0/833: dread d2/d12/f42 [0,4194304] 0 2024-03-23T11:47:50.570 INFO:tasks.workunit.client.1.smithi111.stdout:9/958: getdents d0/d4/d1c/d2d/d3b 0 2024-03-23T11:47:50.572 INFO:tasks.workunit.client.1.smithi111.stdout:0/899: creat d5/db/d1f/d47/d59/d5b/f12d x:0 0 0 2024-03-23T11:47:50.573 INFO:tasks.workunit.client.1.smithi111.stdout:4/991: write d0/d95/fb8 [7529166,89676] 0 2024-03-23T11:47:50.573 INFO:tasks.workunit.client.0.smithi063.stdout:8/940: write d6/d1c/d32/d37/d10a/dae/fbc [35036,2699] 0 2024-03-23T11:47:50.576 INFO:tasks.workunit.client.0.smithi063.stdout:2/535: sync 2024-03-23T11:47:50.576 INFO:tasks.workunit.client.1.smithi111.stdout:1/910: dwrite d0/d2b/d7c/f113 [0,4194304] 0 2024-03-23T11:47:50.577 INFO:tasks.workunit.client.1.smithi111.stdout:4/992: write d0/dc/d4c/d2a/df2/f11b [459970,36703] 0 2024-03-23T11:47:50.581 INFO:tasks.workunit.client.0.smithi063.stdout:0/834: mkdir d2/d6e/d8c/d127 0 2024-03-23T11:47:50.584 INFO:tasks.workunit.client.1.smithi111.stdout:1/911: dread d0/d6/d48/d4a/d6c/d83/d11d/f101 [0,4194304] 0 2024-03-23T11:47:50.584 INFO:tasks.workunit.client.1.smithi111.stdout:9/959: mknod d0/d4/d1c/d36/d28/df3/c139 0 2024-03-23T11:47:50.588 INFO:tasks.workunit.client.0.smithi063.stdout:4/683: dwrite d5/da/d20/d4b/d16/fb4 [0,4194304] 0 2024-03-23T11:47:50.588 INFO:tasks.workunit.client.0.smithi063.stdout:8/941: mknod d6/dd/d59/d13d/d5d/d7d/d9a/d126/c13e 0 2024-03-23T11:47:50.594 INFO:tasks.workunit.client.0.smithi063.stdout:9/681: write d4/d4c/d19/f23 [2094322,28338] 0 2024-03-23T11:47:50.598 INFO:tasks.workunit.client.1.smithi111.stdout:9/960: rename d0/d4/d1c/d2d/f2e to d0/d31/d41/d71/d8f/f13a 0 2024-03-23T11:47:50.600 INFO:tasks.workunit.client.0.smithi063.stdout:0/835: chown d2/d6e/d8c/fa4 28 1 2024-03-23T11:47:50.602 INFO:tasks.workunit.client.1.smithi111.stdout:4/993: link d0/dc/d30/l12 d0/dc/d89/da4/l145 0 2024-03-23T11:47:50.602 INFO:tasks.workunit.client.1.smithi111.stdout:0/900: dread d5/db/f38 [0,4194304] 0 2024-03-23T11:47:50.603 INFO:tasks.workunit.client.1.smithi111.stdout:8/769: dwrite d5/d26/d2f/d62/d7e/d85/f96 [0,4194304] 0 2024-03-23T11:47:50.603 INFO:tasks.workunit.client.0.smithi063.stdout:0/836: stat d2/d14/d36/d77/c114 0 2024-03-23T11:47:50.604 INFO:tasks.workunit.client.0.smithi063.stdout:1/593: dwrite d3/f81 [0,4194304] 0 2024-03-23T11:47:50.606 INFO:tasks.workunit.client.1.smithi111.stdout:1/912: dwrite d0/d6/d48/d4a/ff2 [4194304,4194304] 0 2024-03-23T11:47:50.608 INFO:tasks.workunit.client.0.smithi063.stdout:1/594: read d3/d35/d40/f53 [768153,18543] 0 2024-03-23T11:47:50.609 INFO:tasks.workunit.client.1.smithi111.stdout:8/770: write d5/d26/d31/d3b/d55/ff1 [950004,74028] 0 2024-03-23T11:47:50.610 INFO:tasks.workunit.client.0.smithi063.stdout:1/595: write d3/df/f7d [815884,15329] 0 2024-03-23T11:47:50.612 INFO:tasks.workunit.client.0.smithi063.stdout:2/536: write d2/d23/d2e/d87/d4d/f79 [585388,108926] 0 2024-03-23T11:47:50.613 INFO:tasks.workunit.client.1.smithi111.stdout:4/994: truncate d0/dc/d4c/d2a/d97/d4f/dba/f118 1703887 0 2024-03-23T11:47:50.618 INFO:tasks.workunit.client.0.smithi063.stdout:4/684: truncate d5/da/d20/d4b/d29/d38/f55 2493620 0 2024-03-23T11:47:50.622 INFO:tasks.workunit.client.1.smithi111.stdout:9/961: dwrite d0/d4/f43 [4194304,4194304] 0 2024-03-23T11:47:50.623 INFO:tasks.workunit.client.1.smithi111.stdout:0/901: dread d5/db/d1f/f79 [4194304,4194304] 0 2024-03-23T11:47:50.626 INFO:tasks.workunit.client.0.smithi063.stdout:0/837: unlink d2/d12/d32/d6d/d124/d1f/d4f/fcc 0 2024-03-23T11:47:50.626 INFO:tasks.workunit.client.1.smithi111.stdout:9/962: chown d0/d4/d1c/d2d/d3b/d42/d4f/d95/lee 3624 1 2024-03-23T11:47:50.627 INFO:tasks.workunit.client.1.smithi111.stdout:1/913: read d0/d6/d48/d4a/fd0 [712934,45147] 0 2024-03-23T11:47:50.630 INFO:tasks.workunit.client.1.smithi111.stdout:2/955: sync 2024-03-23T11:47:50.630 INFO:tasks.workunit.client.0.smithi063.stdout:2/537: truncate d2/de/d22/f4a 879534 0 2024-03-23T11:47:50.631 INFO:tasks.workunit.client.1.smithi111.stdout:2/956: read - d2/de0/d110/ffa zero size 2024-03-23T11:47:50.633 INFO:tasks.workunit.client.1.smithi111.stdout:2/957: chown d2/de/d6c/dd1/dd6/feb 0 1 2024-03-23T11:47:50.636 INFO:tasks.workunit.client.1.smithi111.stdout:2/958: readlink d2/d32/l115 0 2024-03-23T11:47:50.637 INFO:tasks.workunit.client.0.smithi063.stdout:0/838: fdatasync d2/d12/d32/f43 0 2024-03-23T11:47:50.640 INFO:tasks.workunit.client.1.smithi111.stdout:0/902: rename d5/db/d1f/f123 to d5/db/d1f/d47/f12e 0 2024-03-23T11:47:50.640 INFO:tasks.workunit.client.1.smithi111.stdout:8/771: link d5/d26/d2f/d62/d75/d4b/d51/d5a/l50 d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/df9/lfd 0 2024-03-23T11:47:50.641 INFO:tasks.workunit.client.0.smithi063.stdout:4/685: truncate d5/da/d20/d4b/d1f/f39 2925220 0 2024-03-23T11:47:50.644 INFO:tasks.workunit.client.0.smithi063.stdout:0/839: fsync d2/d14/f64 0 2024-03-23T11:47:50.645 INFO:tasks.workunit.client.0.smithi063.stdout:2/538: read d2/d23/d2e/d52/d6c/f88 [7200605,47554] 0 2024-03-23T11:47:50.646 INFO:tasks.workunit.client.0.smithi063.stdout:1/596: creat d3/d49/d56/fb7 x:0 0 0 2024-03-23T11:47:50.647 INFO:tasks.workunit.client.0.smithi063.stdout:0/840: dread - d2/d6e/d8c/fae zero size 2024-03-23T11:47:50.650 INFO:tasks.workunit.client.0.smithi063.stdout:4/686: dread d5/d1e/f82 [0,4194304] 0 2024-03-23T11:47:50.653 INFO:tasks.workunit.client.1.smithi111.stdout:8/772: mknod d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/df9/cfe 0 2024-03-23T11:47:50.653 INFO:tasks.workunit.client.1.smithi111.stdout:0/903: dread d5/db/d1f/d47/d59/f127 [0,4194304] 0 2024-03-23T11:47:50.655 INFO:tasks.workunit.client.0.smithi063.stdout:1/597: rmdir d3/df/d31 39 2024-03-23T11:47:50.658 INFO:tasks.workunit.client.0.smithi063.stdout:2/539: dread - d2/d23/d2e/d52/d6c/f7c zero size 2024-03-23T11:47:50.661 INFO:tasks.workunit.client.1.smithi111.stdout:2/959: write d2/de/d6c/dd1/dd6/ffe [683006,73270] 0 2024-03-23T11:47:50.664 INFO:tasks.workunit.client.0.smithi063.stdout:9/682: link d4/d31/d43/da9/da6/db5/cef d4/d31/d43/da9/cf3 0 2024-03-23T11:47:50.669 INFO:tasks.workunit.client.0.smithi063.stdout:8/942: dwrite d6/d1c/d32/d37/ff4 [0,4194304] 0 2024-03-23T11:47:50.672 INFO:tasks.workunit.client.0.smithi063.stdout:8/943: dread - d6/dd/d59/d13d/f114 zero size 2024-03-23T11:47:50.673 INFO:tasks.workunit.client.1.smithi111.stdout:0/904: dread d5/db/d1f/d47/d59/f127 [0,4194304] 0 2024-03-23T11:47:50.676 INFO:tasks.workunit.client.0.smithi063.stdout:1/598: mkdir d3/df/db8 0 2024-03-23T11:47:50.677 INFO:tasks.workunit.client.1.smithi111.stdout:9/963: dwrite d0/d4/d10/d29/fa3 [0,4194304] 0 2024-03-23T11:47:50.677 INFO:tasks.workunit.client.1.smithi111.stdout:4/995: dwrite d0/dc/d30/fed [0,4194304] 0 2024-03-23T11:47:50.678 INFO:tasks.workunit.client.1.smithi111.stdout:1/914: dread d0/d6/fc5 [0,4194304] 0 2024-03-23T11:47:50.680 INFO:tasks.workunit.client.1.smithi111.stdout:8/773: dwrite d5/d26/d2f/d62/d75/d4b/d51/da4/f88 [0,4194304] 0 2024-03-23T11:47:50.682 INFO:tasks.workunit.client.0.smithi063.stdout:8/944: write d6/d1c/d32/d37/d10a/dae/feb [1268879,70499] 0 2024-03-23T11:47:50.682 INFO:tasks.workunit.client.1.smithi111.stdout:0/905: rename d5/db/d1f/d48/f11a to d5/db/f12f 0 2024-03-23T11:47:50.682 INFO:tasks.workunit.client.1.smithi111.stdout:9/964: dread - d0/d31/d41/d9d/db0/f12b zero size 2024-03-23T11:47:50.686 INFO:tasks.workunit.client.0.smithi063.stdout:1/599: mknod d3/d62/dad/cb9 0 2024-03-23T11:47:50.689 INFO:tasks.workunit.client.1.smithi111.stdout:8/774: chown d5/d26/d31/d3b/f3d 11765 1 2024-03-23T11:47:50.689 INFO:tasks.workunit.client.1.smithi111.stdout:1/915: readlink d0/d6/d48/d4a/d6c/d83/d11d/d1e/d70/le2 0 2024-03-23T11:47:50.690 INFO:tasks.workunit.client.0.smithi063.stdout:2/540: dwrite d2/de/d7a/d8d/f9e [0,4194304] 0 2024-03-23T11:47:50.691 INFO:tasks.workunit.client.1.smithi111.stdout:2/960: dwrite d2/d7/f38 [0,4194304] 0 2024-03-23T11:47:50.697 INFO:tasks.workunit.client.0.smithi063.stdout:1/600: mknod d3/d3f/cba 0 2024-03-23T11:47:50.697 INFO:tasks.workunit.client.1.smithi111.stdout:0/906: mkdir d5/db/d4d/da6/dcc/d130 0 2024-03-23T11:47:50.702 INFO:tasks.workunit.client.1.smithi111.stdout:8/775: mknod d5/d26/d2f/d62/d75/cff 0 2024-03-23T11:47:50.704 INFO:tasks.workunit.client.1.smithi111.stdout:4/996: rename d0/dc/d4c/d71/lbc to d0/dc/d4c/d2a/df2/d123/l146 0 2024-03-23T11:47:50.705 INFO:tasks.workunit.client.1.smithi111.stdout:8/776: write d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [2171818,16380] 0 2024-03-23T11:47:50.707 INFO:tasks.workunit.client.1.smithi111.stdout:8/777: read d5/d26/d2f/d42/d63/f69 [3773890,67492] 0 2024-03-23T11:47:50.710 INFO:tasks.workunit.client.1.smithi111.stdout:0/907: mknod d5/db/d1f/d47/d106/d10f/c131 0 2024-03-23T11:47:50.714 INFO:tasks.workunit.client.1.smithi111.stdout:2/961: rename d2/d32/d129 to d2/d11/d17/d1f/da0/d153 0 2024-03-23T11:47:50.715 INFO:tasks.workunit.client.1.smithi111.stdout:8/778: truncate d5/d32/ff0 6157657 0 2024-03-23T11:47:50.717 INFO:tasks.workunit.client.1.smithi111.stdout:0/908: creat d5/db/d4d/da6/d10b/d115/f132 x:0 0 0 2024-03-23T11:47:50.718 INFO:tasks.workunit.client.0.smithi063.stdout:2/541: dwrite d2/d23/f41 [0,4194304] 0 2024-03-23T11:47:50.719 INFO:tasks.workunit.client.1.smithi111.stdout:2/962: mknod d2/c154 0 2024-03-23T11:47:50.721 INFO:tasks.workunit.client.1.smithi111.stdout:0/909: mkdir d5/db/d4d/da6/d133 0 2024-03-23T11:47:50.731 INFO:tasks.workunit.client.1.smithi111.stdout:8/779: link d5/d26/d31/d3b/cee d5/d26/d2f/d42/d63/c100 0 2024-03-23T11:47:50.732 INFO:tasks.workunit.client.1.smithi111.stdout:7/919: sync 2024-03-23T11:47:50.733 INFO:tasks.workunit.client.0.smithi063.stdout:8/945: write d6/d1c/d32/d37/d10a/dae/fb1 [534785,104477] 0 2024-03-23T11:47:50.737 INFO:tasks.workunit.client.0.smithi063.stdout:4/687: dread d5/da/d20/d4b/d29/d38/f55 [0,4194304] 0 2024-03-23T11:47:50.738 INFO:tasks.workunit.client.1.smithi111.stdout:8/780: mknod d5/d32/d7d/da0/c101 0 2024-03-23T11:47:50.738 INFO:tasks.workunit.client.1.smithi111.stdout:7/920: mknod d14/d1b/d1e/c110 0 2024-03-23T11:47:50.739 INFO:tasks.workunit.client.1.smithi111.stdout:9/965: write d0/d31/d41/d65/f6f [4665192,85670] 0 2024-03-23T11:47:50.740 INFO:tasks.workunit.client.1.smithi111.stdout:8/781: write d5/d26/d2f/f77 [4489087,3094] 0 2024-03-23T11:47:50.744 INFO:tasks.workunit.client.0.smithi063.stdout:2/542: dread d2/d23/d2e/f35 [0,4194304] 0 2024-03-23T11:47:50.748 INFO:tasks.workunit.client.0.smithi063.stdout:4/688: rmdir d5/da/d20/d4b/d16/d50/d86 39 2024-03-23T11:47:50.750 INFO:tasks.workunit.client.0.smithi063.stdout:2/543: mkdir d2/d23/d56/dad 0 2024-03-23T11:47:50.754 INFO:tasks.workunit.client.1.smithi111.stdout:9/966: chown d0/d4/l56 353644 1 2024-03-23T11:47:50.754 INFO:tasks.workunit.client.0.smithi063.stdout:1/601: dwrite d3/df/d31/f5f [0,4194304] 0 2024-03-23T11:47:50.756 INFO:tasks.workunit.client.0.smithi063.stdout:2/544: symlink d2/de/d7a/da2/lae 0 2024-03-23T11:47:50.761 INFO:tasks.workunit.client.0.smithi063.stdout:1/602: truncate d3/d1c/da8/faf 383362 0 2024-03-23T11:47:50.761 INFO:tasks.workunit.client.0.smithi063.stdout:4/689: mkdir d5/da/dc3 0 2024-03-23T11:47:50.763 INFO:tasks.workunit.client.1.smithi111.stdout:2/963: write d2/d11/d17/d3e/d5d/fa4 [1062392,66090] 0 2024-03-23T11:47:50.766 INFO:tasks.workunit.client.0.smithi063.stdout:1/603: truncate d3/d3f/d4d/d52/fab 681483 0 2024-03-23T11:47:50.770 INFO:tasks.workunit.client.0.smithi063.stdout:8/946: write d6/d1f/d54/d63/fc2 [1174596,96264] 0 2024-03-23T11:47:50.772 INFO:tasks.workunit.client.0.smithi063.stdout:2/545: read d2/f29 [1298829,50010] 0 2024-03-23T11:47:50.774 INFO:tasks.workunit.client.0.smithi063.stdout:1/604: creat d3/d1c/d4b/fbb x:0 0 0 2024-03-23T11:47:50.777 INFO:tasks.workunit.client.0.smithi063.stdout:4/690: mkdir d5/da/d20/d4b/d16/dc4 0 2024-03-23T11:47:50.779 INFO:tasks.workunit.client.0.smithi063.stdout:8/947: mknod d6/d1c/d32/d37/d10a/db5/c13f 0 2024-03-23T11:47:50.780 INFO:tasks.workunit.client.1.smithi111.stdout:2/964: dwrite d2/f6 [4194304,4194304] 0 2024-03-23T11:47:50.781 INFO:tasks.workunit.client.1.smithi111.stdout:8/782: write f1 [280014,63288] 0 2024-03-23T11:47:50.782 INFO:tasks.workunit.client.0.smithi063.stdout:2/546: creat d2/d23/d2e/d98/faf x:0 0 0 2024-03-23T11:47:50.783 INFO:tasks.workunit.client.0.smithi063.stdout:8/948: creat d6/dd/df/f140 x:0 0 0 2024-03-23T11:47:50.784 INFO:tasks.workunit.client.0.smithi063.stdout:4/691: write d5/da/d20/d4b/d16/d32/fb0 [5102823,56626] 0 2024-03-23T11:47:50.785 INFO:tasks.workunit.client.1.smithi111.stdout:2/965: creat d2/d13d/f155 x:0 0 0 2024-03-23T11:47:50.785 INFO:tasks.workunit.client.0.smithi063.stdout:1/605: read d3/d1c/f1e [1208353,65102] 0 2024-03-23T11:47:50.786 INFO:tasks.workunit.client.1.smithi111.stdout:8/783: symlink d5/d32/d7d/da0/l102 0 2024-03-23T11:47:50.786 INFO:tasks.workunit.client.1.smithi111.stdout:7/921: dread d14/d44/d70/d7e/ff9 [4194304,4194304] 0 2024-03-23T11:47:50.792 INFO:tasks.workunit.client.1.smithi111.stdout:2/966: rmdir d2/d128 39 2024-03-23T11:47:50.792 INFO:tasks.workunit.client.0.smithi063.stdout:1/606: truncate d3/d49/d56/d65/d88/f9c 441989 0 2024-03-23T11:47:50.793 INFO:tasks.workunit.client.0.smithi063.stdout:4/692: fsync d5/da/d20/d4b/d1f/f21 0 2024-03-23T11:47:50.799 INFO:tasks.workunit.client.0.smithi063.stdout:8/949: mkdir d6/d1f/d141 0 2024-03-23T11:47:50.800 INFO:tasks.workunit.client.0.smithi063.stdout:4/693: dread d5/d1e/f82 [0,4194304] 0 2024-03-23T11:47:50.802 INFO:tasks.workunit.client.1.smithi111.stdout:6/920: sync 2024-03-23T11:47:50.803 INFO:tasks.workunit.client.1.smithi111.stdout:7/922: write d14/f15 [6270146,40663] 0 2024-03-23T11:47:50.805 INFO:tasks.workunit.client.0.smithi063.stdout:4/694: mknod d5/da/d20/d4b/d1f/cc5 0 2024-03-23T11:47:50.807 INFO:tasks.workunit.client.0.smithi063.stdout:8/950: truncate d6/d1c/d27/f34 3240668 0 2024-03-23T11:47:50.813 INFO:tasks.workunit.client.0.smithi063.stdout:8/951: fsync d6/d1f/d54/d5c/d66/df1/f11b 0 2024-03-23T11:47:50.815 INFO:tasks.workunit.client.1.smithi111.stdout:8/784: dwrite d5/d26/d31/d3b/f60 [4194304,4194304] 0 2024-03-23T11:47:50.817 INFO:tasks.workunit.client.0.smithi063.stdout:4/695: symlink d5/da/d20/d4b/d16/dc4/lc6 0 2024-03-23T11:47:50.817 INFO:tasks.workunit.client.1.smithi111.stdout:2/967: dread d2/d7/fd [0,4194304] 0 2024-03-23T11:47:50.818 INFO:tasks.workunit.client.0.smithi063.stdout:1/607: dwrite d3/d1c/d4b/fbb [0,4194304] 0 2024-03-23T11:47:50.821 INFO:tasks.workunit.client.0.smithi063.stdout:2/547: dwrite d2/d23/f69 [0,4194304] 0 2024-03-23T11:47:50.821 INFO:tasks.workunit.client.1.smithi111.stdout:2/968: unlink d2/de/d12/d70/d96/f122 0 2024-03-23T11:47:50.822 INFO:tasks.workunit.client.0.smithi063.stdout:1/608: chown d3/df/d31/f5f 169 1 2024-03-23T11:47:50.824 INFO:tasks.workunit.client.0.smithi063.stdout:8/952: rmdir d6/d1f/d54/d5c/dc5 39 2024-03-23T11:47:50.826 INFO:tasks.workunit.client.1.smithi111.stdout:2/969: read d2/de/d12/d70/d96/fce [430254,28488] 0 2024-03-23T11:47:50.827 INFO:tasks.workunit.client.0.smithi063.stdout:4/696: truncate d5/da/d20/d4b/d1f/f26 1534422 0 2024-03-23T11:47:50.830 INFO:tasks.workunit.client.0.smithi063.stdout:1/609: mkdir d3/d3f/d4d/dbc 0 2024-03-23T11:47:50.832 INFO:tasks.workunit.client.0.smithi063.stdout:6/518: sync 2024-03-23T11:47:50.835 INFO:tasks.workunit.client.1.smithi111.stdout:2/970: link d2/d5c/c116 d2/de/d6c/dd1/c156 0 2024-03-23T11:47:50.835 INFO:tasks.workunit.client.1.smithi111.stdout:6/921: dread d2/d1a/f2d [0,4194304] 0 2024-03-23T11:47:50.838 INFO:tasks.workunit.client.1.smithi111.stdout:2/971: symlink d2/d5c/d126/l157 0 2024-03-23T11:47:50.839 INFO:tasks.workunit.client.1.smithi111.stdout:2/972: readlink d2/d11/d17/d3e/d5d/dc0/l149 0 2024-03-23T11:47:50.841 INFO:tasks.workunit.client.0.smithi063.stdout:2/548: creat d2/d84/d92/fb0 x:0 0 0 2024-03-23T11:47:50.844 INFO:tasks.workunit.client.0.smithi063.stdout:4/697: mkdir d5/da/d20/d4b/d16/d32/d56/dc7 0 2024-03-23T11:47:50.844 INFO:tasks.workunit.client.0.smithi063.stdout:2/549: write d2/de/d7a/d8d/fa9 [490634,18872] 0 2024-03-23T11:47:50.849 INFO:tasks.workunit.client.0.smithi063.stdout:1/610: rename d3/d62/dad/l76 to d3/d3f/d4d/d52/lbd 0 2024-03-23T11:47:50.850 INFO:tasks.workunit.client.0.smithi063.stdout:6/519: read d1/dc/d3a/d57/f5c [863512,49771] 0 2024-03-23T11:47:50.851 INFO:tasks.workunit.client.0.smithi063.stdout:4/698: symlink d5/da/d20/d4b/d29/d38/d93/lc8 0 2024-03-23T11:47:50.857 INFO:tasks.workunit.client.0.smithi063.stdout:4/699: creat d5/da/d20/d4b/d16/d32/d6f/fc9 x:0 0 0 2024-03-23T11:47:50.863 INFO:tasks.workunit.client.1.smithi111.stdout:3/966: sync 2024-03-23T11:47:50.865 INFO:tasks.workunit.client.1.smithi111.stdout:6/922: write d2/d1f/d45/d9a/fb0 [130616,130685] 0 2024-03-23T11:47:50.868 INFO:tasks.workunit.client.0.smithi063.stdout:6/520: dread d1/d13/f1c [0,4194304] 0 2024-03-23T11:47:50.870 INFO:tasks.workunit.client.1.smithi111.stdout:6/923: creat d2/d80/d100/f129 x:0 0 0 2024-03-23T11:47:50.871 INFO:tasks.workunit.client.1.smithi111.stdout:8/785: dwrite d5/d26/d31/d4d/fc4 [0,4194304] 0 2024-03-23T11:47:50.872 INFO:tasks.workunit.client.1.smithi111.stdout:2/973: dread d2/d7/da/d19/f77 [0,4194304] 0 2024-03-23T11:47:50.873 INFO:tasks.workunit.client.1.smithi111.stdout:8/786: readlink d5/d26/le1 0 2024-03-23T11:47:50.876 INFO:tasks.workunit.client.0.smithi063.stdout:4/700: write d5/da/d20/d4b/f5a [1410912,6893] 0 2024-03-23T11:47:50.876 INFO:tasks.workunit.client.0.smithi063.stdout:1/611: dwrite d3/df/d31/f5f [0,4194304] 0 2024-03-23T11:47:50.877 INFO:tasks.workunit.client.0.smithi063.stdout:8/953: dwrite d6/dd/d59/d13d/f100 [0,4194304] 0 2024-03-23T11:47:50.878 INFO:tasks.workunit.client.1.smithi111.stdout:2/974: mkdir d2/de/d12/d158 0 2024-03-23T11:47:50.880 INFO:tasks.workunit.client.0.smithi063.stdout:2/550: dwrite d2/f18 [0,4194304] 0 2024-03-23T11:47:50.886 INFO:tasks.workunit.client.1.smithi111.stdout:3/967: dwrite d0/d1a/d1d/f43 [0,4194304] 0 2024-03-23T11:47:50.888 INFO:tasks.workunit.client.1.smithi111.stdout:3/968: chown d0/d1a/d2d/ff1 3918733 1 2024-03-23T11:47:50.891 INFO:tasks.workunit.client.0.smithi063.stdout:8/954: mkdir d6/dd/d59/db2/dbe/dfb/d12f/d142 0 2024-03-23T11:47:50.895 INFO:tasks.workunit.client.1.smithi111.stdout:3/969: truncate d0/d1a/d2d/fd5 90209 0 2024-03-23T11:47:50.899 INFO:tasks.workunit.client.0.smithi063.stdout:8/955: creat d6/dd/d59/d13d/d5d/d7d/f143 x:0 0 0 2024-03-23T11:47:50.901 INFO:tasks.workunit.client.1.smithi111.stdout:3/970: creat d0/d1a/d4f/d13a/f140 x:0 0 0 2024-03-23T11:47:50.909 INFO:tasks.workunit.client.1.smithi111.stdout:6/924: dread d2/d1f/dce/fec [0,4194304] 0 2024-03-23T11:47:50.912 INFO:tasks.workunit.client.1.smithi111.stdout:6/925: mkdir d2/d17/d12a 0 2024-03-23T11:47:50.925 INFO:tasks.workunit.client.1.smithi111.stdout:8/787: dread d5/d26/d31/d3b/f3d [4194304,4194304] 0 2024-03-23T11:47:50.926 INFO:tasks.workunit.client.1.smithi111.stdout:8/788: stat d5/d26/d2f/d42/dd3/fde 0 2024-03-23T11:47:50.936 INFO:tasks.workunit.client.1.smithi111.stdout:8/789: dread d5/d1b/f2b [0,4194304] 0 2024-03-23T11:47:50.941 INFO:tasks.workunit.client.1.smithi111.stdout:3/971: dwrite d0/d55/f56 [0,4194304] 0 2024-03-23T11:47:50.941 INFO:tasks.workunit.client.1.smithi111.stdout:8/790: symlink d5/d26/d2f/d62/d75/d4b/d51/dd1/l103 0 2024-03-23T11:47:50.942 INFO:tasks.workunit.client.1.smithi111.stdout:6/926: dwrite d2/fd [0,4194304] 0 2024-03-23T11:47:50.944 INFO:tasks.workunit.client.1.smithi111.stdout:8/791: creat d5/d32/d79/f104 x:0 0 0 2024-03-23T11:47:50.965 INFO:tasks.workunit.client.1.smithi111.stdout:8/792: dwrite d5/f11 [0,4194304] 0 2024-03-23T11:47:50.969 INFO:tasks.workunit.client.0.smithi063.stdout:8/956: chown d6/d1c/d32/d37/c69 27 1 2024-03-23T11:47:50.971 INFO:tasks.workunit.client.0.smithi063.stdout:1/612: dwrite d3/d1c/d4b/f8d [0,4194304] 0 2024-03-23T11:47:50.973 INFO:tasks.workunit.client.0.smithi063.stdout:4/701: dwrite d5/d1e/d2b/f58 [0,4194304] 0 2024-03-23T11:47:50.975 INFO:tasks.workunit.client.1.smithi111.stdout:8/793: getdents d5/d26/d2f/d62/db5 0 2024-03-23T11:47:50.978 INFO:tasks.workunit.client.1.smithi111.stdout:3/972: dread d0/d1a/d2b/d10e/f12 [0,4194304] 0 2024-03-23T11:47:50.978 INFO:tasks.workunit.client.1.smithi111.stdout:6/927: dread d2/d1f/f26 [0,4194304] 0 2024-03-23T11:47:50.980 INFO:tasks.workunit.client.0.smithi063.stdout:1/613: mknod d3/d49/d56/d65/cbe 0 2024-03-23T11:47:50.982 INFO:tasks.workunit.client.1.smithi111.stdout:6/928: symlink d2/d1a/l12b 0 2024-03-23T11:47:50.985 INFO:tasks.workunit.client.1.smithi111.stdout:1/916: sync 2024-03-23T11:47:50.985 INFO:tasks.workunit.client.1.smithi111.stdout:4/997: sync 2024-03-23T11:47:50.985 INFO:tasks.workunit.client.1.smithi111.stdout:0/910: sync 2024-03-23T11:47:50.987 INFO:tasks.workunit.client.1.smithi111.stdout:3/973: getdents d0/d1a 0 2024-03-23T11:47:50.989 INFO:tasks.workunit.client.1.smithi111.stdout:6/929: dread d2/d1f/d45/d49/d8d/f94 [0,4194304] 0 2024-03-23T11:47:50.990 INFO:tasks.workunit.client.1.smithi111.stdout:1/917: read d0/d6/f27 [6761271,33104] 0 2024-03-23T11:47:50.995 INFO:tasks.workunit.client.0.smithi063.stdout:8/957: read d6/dd/d59/d13d/d9e/fa8 [3625331,8175] 0 2024-03-23T11:47:50.997 INFO:tasks.workunit.client.1.smithi111.stdout:8/794: dwrite d5/d26/d2f/d42/d63/fe7 [0,4194304] 0 2024-03-23T11:47:50.998 INFO:tasks.workunit.client.1.smithi111.stdout:0/911: fsync d5/db/d1f/d47/d59/d5b/f12d 0 2024-03-23T11:47:50.999 INFO:tasks.workunit.client.0.smithi063.stdout:4/702: dread d5/da/d72/f88 [0,4194304] 0 2024-03-23T11:47:51.000 INFO:tasks.workunit.client.0.smithi063.stdout:1/614: dwrite d3/d35/d40/f42 [0,4194304] 0 2024-03-23T11:47:51.002 INFO:tasks.workunit.client.0.smithi063.stdout:4/703: readlink d5/d1e/l9c 0 2024-03-23T11:47:51.005 INFO:tasks.workunit.client.1.smithi111.stdout:6/930: fdatasync d2/d35/f44 0 2024-03-23T11:47:51.005 INFO:tasks.workunit.client.0.smithi063.stdout:6/521: dread d1/dc/d3a/d57/d77/f87 [0,4194304] 0 2024-03-23T11:47:51.006 INFO:tasks.workunit.client.0.smithi063.stdout:1/615: unlink d3/d1c/f68 0 2024-03-23T11:47:51.006 INFO:tasks.workunit.client.1.smithi111.stdout:0/912: fsync d5/db/d1f/d48/f71 0 2024-03-23T11:47:51.009 INFO:tasks.workunit.client.1.smithi111.stdout:9/967: sync 2024-03-23T11:47:51.009 INFO:tasks.workunit.client.1.smithi111.stdout:2/975: sync 2024-03-23T11:47:51.009 INFO:tasks.workunit.client.1.smithi111.stdout:7/923: sync 2024-03-23T11:47:51.010 INFO:tasks.workunit.client.0.smithi063.stdout:6/522: dread - d1/d15/d4b/f66 zero size 2024-03-23T11:47:51.011 INFO:tasks.workunit.client.1.smithi111.stdout:3/974: dwrite d0/db9/f135 [0,4194304] 0 2024-03-23T11:47:51.011 INFO:tasks.workunit.client.1.smithi111.stdout:9/968: chown d0/l27 172 1 2024-03-23T11:47:51.014 INFO:tasks.workunit.client.0.smithi063.stdout:8/958: write d6/dd/df/f60 [1778950,39347] 0 2024-03-23T11:47:51.016 INFO:tasks.workunit.client.1.smithi111.stdout:6/931: rename d2/d1f/c88 to d2/d1f/d45/d9a/dde/c12c 0 2024-03-23T11:47:51.020 INFO:tasks.workunit.client.1.smithi111.stdout:8/795: dread d5/d26/d2f/d62/d75/f6b [0,4194304] 0 2024-03-23T11:47:51.021 INFO:tasks.workunit.client.1.smithi111.stdout:3/975: creat d0/db9/f141 x:0 0 0 2024-03-23T11:47:51.022 INFO:tasks.workunit.client.1.smithi111.stdout:0/913: mkdir d5/db/d4d/da6/d10b/d122/d134 0 2024-03-23T11:47:51.022 INFO:tasks.workunit.client.0.smithi063.stdout:8/959: creat d6/d1c/dff/f144 x:0 0 0 2024-03-23T11:47:51.027 INFO:tasks.workunit.client.0.smithi063.stdout:8/960: mkdir d6/d1c/d110/d145 0 2024-03-23T11:47:51.028 INFO:tasks.workunit.client.1.smithi111.stdout:1/918: write d0/d2b/d37/f4c [4657005,72267] 0 2024-03-23T11:47:51.029 INFO:tasks.workunit.client.0.smithi063.stdout:8/961: chown d6/d1c/cca 1312745009 1 2024-03-23T11:47:51.029 INFO:tasks.workunit.client.0.smithi063.stdout:1/616: dwrite d3/d35/d40/f42 [0,4194304] 0 2024-03-23T11:47:51.031 INFO:tasks.workunit.client.0.smithi063.stdout:8/962: readlink d6/dd/d59/daf/dba/l139 0 2024-03-23T11:47:51.033 INFO:tasks.workunit.client.1.smithi111.stdout:2/976: dread d2/d7/fd [0,4194304] 0 2024-03-23T11:47:51.034 INFO:tasks.workunit.client.1.smithi111.stdout:3/976: creat d0/d1a/d2d/df5/d11a/f142 x:0 0 0 2024-03-23T11:47:51.034 INFO:tasks.workunit.client.1.smithi111.stdout:8/796: rmdir d5/d26/d31/d3b/d55 39 2024-03-23T11:47:51.036 INFO:tasks.workunit.client.1.smithi111.stdout:4/998: dwrite d0/d95/d9b/f107 [0,4194304] 0 2024-03-23T11:47:51.040 INFO:tasks.workunit.client.1.smithi111.stdout:8/797: dread - d5/d26/d2f/d42/f57 zero size 2024-03-23T11:47:51.040 INFO:tasks.workunit.client.1.smithi111.stdout:3/977: write d0/d53/dfa/d103/f10c [1172559,24144] 0 2024-03-23T11:47:51.040 INFO:tasks.workunit.client.1.smithi111.stdout:1/919: dread d0/d2b/d7c/dca/fd1 [0,4194304] 0 2024-03-23T11:47:51.044 INFO:tasks.workunit.client.1.smithi111.stdout:1/920: write d0/d2b/de3/f115 [539042,26859] 0 2024-03-23T11:47:51.047 INFO:tasks.workunit.client.1.smithi111.stdout:0/914: rename d5/db/d1f/de4/fee to d5/d121/f135 0 2024-03-23T11:47:51.049 INFO:tasks.workunit.client.1.smithi111.stdout:4/999: creat d0/dc/d1f/db0/f147 x:0 0 0 2024-03-23T11:47:51.052 INFO:tasks.workunit.client.0.smithi063.stdout:6/523: dread d1/f6e [0,4194304] 0 2024-03-23T11:47:51.054 INFO:tasks.workunit.client.0.smithi063.stdout:6/524: chown d1/d15/d8b 4607672 1 2024-03-23T11:47:51.056 INFO:tasks.workunit.client.1.smithi111.stdout:3/978: getdents d0/d1a/d4f/d90/d112 0 2024-03-23T11:47:51.056 INFO:tasks.workunit.client.1.smithi111.stdout:0/915: chown d5/db/d1f/d19/dbd/d8b/cb3 2405914 1 2024-03-23T11:47:51.058 INFO:tasks.workunit.client.0.smithi063.stdout:6/525: truncate d1/d15/d4b/f4c 229046 0 2024-03-23T11:47:51.060 INFO:tasks.workunit.client.0.smithi063.stdout:6/526: creat d1/dc/d3a/d57/d6d/f9a x:0 0 0 2024-03-23T11:47:51.061 INFO:tasks.workunit.client.1.smithi111.stdout:3/979: symlink d0/d1a/d2b/d10e/d19/d91/da3/l143 0 2024-03-23T11:47:51.062 INFO:tasks.workunit.client.1.smithi111.stdout:1/921: link d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/d8e/le8 d0/d6/d48/d4a/d6c/d83/d11d/d91/l13a 0 2024-03-23T11:47:51.064 INFO:tasks.workunit.client.1.smithi111.stdout:3/980: chown d0/d1a/d2b/d38/d7f/f9b 164436511 1 2024-03-23T11:47:51.064 INFO:tasks.workunit.client.0.smithi063.stdout:6/527: dread - d1/d2a/f83 zero size 2024-03-23T11:47:51.064 INFO:tasks.workunit.client.0.smithi063.stdout:7/632: sync 2024-03-23T11:47:51.064 INFO:tasks.workunit.client.0.smithi063.stdout:5/463: sync 2024-03-23T11:47:51.065 INFO:tasks.workunit.client.1.smithi111.stdout:0/916: dread d5/db/d1f/d19/f53 [0,4194304] 0 2024-03-23T11:47:51.067 INFO:tasks.workunit.client.1.smithi111.stdout:3/981: creat d0/d72/f144 x:0 0 0 2024-03-23T11:47:51.071 INFO:tasks.workunit.client.1.smithi111.stdout:3/982: unlink d0/d53/fcf 0 2024-03-23T11:47:51.073 INFO:tasks.workunit.client.1.smithi111.stdout:1/922: dread d0/d6/f27 [4194304,4194304] 0 2024-03-23T11:47:51.073 INFO:tasks.workunit.client.0.smithi063.stdout:6/528: rename d1/dc/f5b to d1/d2a/f9b 0 2024-03-23T11:47:51.073 INFO:tasks.workunit.client.0.smithi063.stdout:1/617: dread d3/df/f18 [0,4194304] 0 2024-03-23T11:47:51.073 INFO:tasks.workunit.client.0.smithi063.stdout:5/464: mkdir d2/d28/d3a/d46/d71/d9f 0 2024-03-23T11:47:51.076 INFO:tasks.workunit.client.1.smithi111.stdout:8/798: dread d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [0,4194304] 0 2024-03-23T11:47:51.078 INFO:tasks.workunit.client.0.smithi063.stdout:5/465: chown d2/d28/f3d 2348 1 2024-03-23T11:47:51.082 INFO:tasks.workunit.client.1.smithi111.stdout:7/924: write d14/d44/d4b/f77 [3147906,21355] 0 2024-03-23T11:47:51.083 INFO:tasks.workunit.client.0.smithi063.stdout:8/963: write d6/d1f/d54/d5c/d66/da4/fab [781208,73653] 0 2024-03-23T11:47:51.085 INFO:tasks.workunit.client.1.smithi111.stdout:0/917: dwrite d5/db/d1f/d48/f66 [0,4194304] 0 2024-03-23T11:47:51.085 INFO:tasks.workunit.client.1.smithi111.stdout:3/983: getdents d0/d1a/d2b/d38 0 2024-03-23T11:47:51.089 INFO:tasks.workunit.client.0.smithi063.stdout:6/529: rmdir d1/dc/d3a/d57/d6d 39 2024-03-23T11:47:51.092 INFO:tasks.workunit.client.0.smithi063.stdout:7/633: write d5/d6/f7a [2822094,102547] 0 2024-03-23T11:47:51.094 INFO:tasks.workunit.client.0.smithi063.stdout:8/964: unlink d6/dd/d59/d13d/d5d/d7d/ca2 0 2024-03-23T11:47:51.095 INFO:tasks.workunit.client.0.smithi063.stdout:7/634: fdatasync d5/d6/d19/d1f/d6c/f80 0 2024-03-23T11:47:51.097 INFO:tasks.workunit.client.0.smithi063.stdout:5/466: creat d2/d7/fa0 x:0 0 0 2024-03-23T11:47:51.098 INFO:tasks.workunit.client.1.smithi111.stdout:8/799: write d5/f19 [2897295,44090] 0 2024-03-23T11:47:51.102 INFO:tasks.workunit.client.1.smithi111.stdout:3/984: stat d0/d1a/d2b/d10e/d19/d91/c111 0 2024-03-23T11:47:51.106 INFO:tasks.workunit.client.0.smithi063.stdout:6/530: symlink d1/dc/d3a/d57/d6d/l9c 0 2024-03-23T11:47:51.107 INFO:tasks.workunit.client.0.smithi063.stdout:5/467: truncate d2/f52 896990 0 2024-03-23T11:47:51.108 INFO:tasks.workunit.client.0.smithi063.stdout:6/531: chown d1/d15/d8b 4246590 1 2024-03-23T11:47:51.111 INFO:tasks.workunit.client.1.smithi111.stdout:2/977: dwrite d2/d11/d17/d3e/f72 [0,4194304] 0 2024-03-23T11:47:51.111 INFO:tasks.workunit.client.1.smithi111.stdout:6/932: dwrite d2/d17/f33 [0,4194304] 0 2024-03-23T11:47:51.112 INFO:tasks.workunit.client.0.smithi063.stdout:1/618: dwrite d3/d3f/fa3 [0,4194304] 0 2024-03-23T11:47:51.113 INFO:tasks.workunit.client.1.smithi111.stdout:9/969: dwrite d0/d31/d35/f9c [0,4194304] 0 2024-03-23T11:47:51.114 INFO:tasks.workunit.client.1.smithi111.stdout:8/800: mkdir d5/d26/d2f/d62/db5/dcc/d105 0 2024-03-23T11:47:51.115 INFO:tasks.workunit.client.0.smithi063.stdout:8/965: write d6/d1c/d32/d37/fcf [1105754,83116] 0 2024-03-23T11:47:51.117 INFO:tasks.workunit.client.1.smithi111.stdout:1/923: dread d0/d2b/d37/d6a/ff8 [0,4194304] 0 2024-03-23T11:47:51.122 INFO:tasks.workunit.client.0.smithi063.stdout:6/532: creat d1/d2a/d41/d42/f9d x:0 0 0 2024-03-23T11:47:51.126 INFO:tasks.workunit.client.0.smithi063.stdout:7/635: rmdir d5/d6/d19/d1f/d6c/d9c 0 2024-03-23T11:47:51.131 INFO:tasks.workunit.client.0.smithi063.stdout:5/468: creat d2/d28/d3a/d46/d71/d9f/fa1 x:0 0 0 2024-03-23T11:47:51.132 INFO:tasks.workunit.client.1.smithi111.stdout:8/801: symlink d5/d26/d2f/d62/db5/l106 0 2024-03-23T11:47:51.139 INFO:tasks.workunit.client.0.smithi063.stdout:6/533: fdatasync d1/d2a/f9b 0 2024-03-23T11:47:51.141 INFO:tasks.workunit.client.1.smithi111.stdout:0/918: symlink d5/db/d4d/l136 0 2024-03-23T11:47:51.141 INFO:tasks.workunit.client.1.smithi111.stdout:8/802: write d5/d26/d2f/d62/db5/f87 [1556282,114281] 0 2024-03-23T11:47:51.144 INFO:tasks.workunit.client.1.smithi111.stdout:1/924: dread d0/d2b/d37/d6a/ff8 [4194304,4194304] 0 2024-03-23T11:47:51.144 INFO:tasks.workunit.client.0.smithi063.stdout:8/966: dwrite d6/d1c/d32/d37/fcf [0,4194304] 0 2024-03-23T11:47:51.146 INFO:tasks.workunit.client.1.smithi111.stdout:1/925: dread - d0/d6/d48/d4d/dab/dbd/f117 zero size 2024-03-23T11:47:51.147 INFO:tasks.workunit.client.1.smithi111.stdout:8/803: read d5/d26/d31/fb6 [1074615,9803] 0 2024-03-23T11:47:51.148 INFO:tasks.workunit.client.0.smithi063.stdout:1/619: creat d3/df/fbf x:0 0 0 2024-03-23T11:47:51.149 INFO:tasks.workunit.client.0.smithi063.stdout:6/534: creat d1/f9e x:0 0 0 2024-03-23T11:47:51.150 INFO:tasks.workunit.client.0.smithi063.stdout:8/967: dread - d6/d1c/dff/f144 zero size 2024-03-23T11:47:51.152 INFO:tasks.workunit.client.1.smithi111.stdout:9/970: creat d0/d4/d1c/d2d/d6e/f13b x:0 0 0 2024-03-23T11:47:51.153 INFO:tasks.workunit.client.0.smithi063.stdout:8/968: stat d6/d1c/dff/f144 0 2024-03-23T11:47:51.153 INFO:tasks.workunit.client.1.smithi111.stdout:2/978: rename d2/d11/d17/d3e/d5d/dc2/f100 to d2/d13d/f159 0 2024-03-23T11:47:51.153 INFO:tasks.workunit.client.0.smithi063.stdout:1/620: write d3/d49/d56/d65/d88/d92/fb0 [917591,32535] 0 2024-03-23T11:47:51.154 INFO:tasks.workunit.client.1.smithi111.stdout:9/971: write d0/d31/d41/d9d/db0/f12b [90982,128693] 0 2024-03-23T11:47:51.155 INFO:tasks.workunit.client.1.smithi111.stdout:1/926: symlink d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/l13b 0 2024-03-23T11:47:51.157 INFO:tasks.workunit.client.1.smithi111.stdout:0/919: dread d5/db/d1f/d47/f12e [0,4194304] 0 2024-03-23T11:47:51.157 INFO:tasks.workunit.client.1.smithi111.stdout:1/927: truncate d0/ff0 432275 0 2024-03-23T11:47:51.161 INFO:tasks.workunit.client.1.smithi111.stdout:8/804: creat d5/d26/d31/d4d/f107 x:0 0 0 2024-03-23T11:47:51.163 INFO:tasks.workunit.client.1.smithi111.stdout:9/972: chown d0/d4/d1c/d2d/d75/c119 1249301935 1 2024-03-23T11:47:51.163 INFO:tasks.workunit.client.1.smithi111.stdout:2/979: read - d2/d11/d17/d3e/d6b/d2e/d76/f12a zero size 2024-03-23T11:47:51.164 INFO:tasks.workunit.client.0.smithi063.stdout:7/636: getdents d5/d6/d19/d1f 0 2024-03-23T11:47:51.165 INFO:tasks.workunit.client.1.smithi111.stdout:0/920: creat d5/dc3/f137 x:0 0 0 2024-03-23T11:47:51.165 INFO:tasks.workunit.client.0.smithi063.stdout:8/969: mknod d6/d30/d128/c146 0 2024-03-23T11:47:51.167 INFO:tasks.workunit.client.1.smithi111.stdout:7/925: rename d14/d1b/d1e/d2c/d2e/d73/dae/la4 to d14/d1b/d1e/d2c/d2e/d33/l111 0 2024-03-23T11:47:51.167 INFO:tasks.workunit.client.1.smithi111.stdout:3/985: dread d0/d1a/d4f/d13a/fd3 [0,4194304] 0 2024-03-23T11:47:51.168 INFO:tasks.workunit.client.0.smithi063.stdout:6/535: symlink d1/dc/l9f 0 2024-03-23T11:47:51.172 INFO:tasks.workunit.client.1.smithi111.stdout:2/980: creat d2/d13d/f15a x:0 0 0 2024-03-23T11:47:51.174 INFO:tasks.workunit.client.1.smithi111.stdout:9/973: symlink d0/d4/d1c/d2d/d3b/d42/d4f/d95/l13c 0 2024-03-23T11:47:51.179 INFO:tasks.workunit.client.1.smithi111.stdout:0/921: mkdir d5/db/d1f/d47/de0/d138 0 2024-03-23T11:47:51.180 INFO:tasks.workunit.client.0.smithi063.stdout:1/621: dwrite d3/d49/d56/fb7 [0,4194304] 0 2024-03-23T11:47:51.180 INFO:tasks.workunit.client.0.smithi063.stdout:7/637: fdatasync d5/d5c/d2f/d73/f3c 0 2024-03-23T11:47:51.181 INFO:tasks.workunit.client.0.smithi063.stdout:8/970: link d6/dd/l2f d6/d1f/l147 0 2024-03-23T11:47:51.181 INFO:tasks.workunit.client.1.smithi111.stdout:2/981: dread d2/d11/d17/d3e/d6b/f2f [0,4194304] 0 2024-03-23T11:47:51.183 INFO:tasks.workunit.client.1.smithi111.stdout:7/926: dwrite d14/d44/d70/d7e/dbc/f104 [0,4194304] 0 2024-03-23T11:47:51.184 INFO:tasks.workunit.client.1.smithi111.stdout:1/928: truncate d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/f123 1198669 0 2024-03-23T11:47:51.189 INFO:tasks.workunit.client.1.smithi111.stdout:9/974: rmdir d0/d4/d10/d80 39 2024-03-23T11:47:51.192 INFO:tasks.workunit.client.0.smithi063.stdout:5/469: dwrite d2/f84 [0,4194304] 0 2024-03-23T11:47:51.198 INFO:tasks.workunit.client.1.smithi111.stdout:3/986: mkdir d0/d1a/d2b/d10e/d19/d84/dea/d145 0 2024-03-23T11:47:51.198 INFO:tasks.workunit.client.1.smithi111.stdout:7/927: mkdir d14/d1b/d105/d112 0 2024-03-23T11:47:51.199 INFO:tasks.workunit.client.1.smithi111.stdout:8/805: rmdir d5/d26/d31/d4d 39 2024-03-23T11:47:51.199 INFO:tasks.workunit.client.1.smithi111.stdout:0/922: read d5/db/d1f/d47/f5a [1768217,78788] 0 2024-03-23T11:47:51.201 INFO:tasks.workunit.client.0.smithi063.stdout:6/536: dwrite d1/dc/d21/d49/f95 [0,4194304] 0 2024-03-23T11:47:51.201 INFO:tasks.workunit.client.0.smithi063.stdout:1/622: mknod d3/cc0 0 2024-03-23T11:47:51.201 INFO:tasks.workunit.client.1.smithi111.stdout:0/923: truncate d5/db/d1f/f12c 169370 0 2024-03-23T11:47:51.203 INFO:tasks.workunit.client.1.smithi111.stdout:8/806: write d5/d26/d31/d4d/f107 [1030523,49091] 0 2024-03-23T11:47:51.205 INFO:tasks.workunit.client.1.smithi111.stdout:1/929: creat d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/f13c x:0 0 0 2024-03-23T11:47:51.207 INFO:tasks.workunit.client.1.smithi111.stdout:8/807: mkdir d5/d26/d2f/d62/db5/d108 0 2024-03-23T11:47:51.212 INFO:tasks.workunit.client.0.smithi063.stdout:5/470: creat d2/d7/d64/fa2 x:0 0 0 2024-03-23T11:47:51.215 INFO:tasks.workunit.client.0.smithi063.stdout:6/537: truncate d1/dc/f1f 270235 0 2024-03-23T11:47:51.219 INFO:tasks.workunit.client.0.smithi063.stdout:5/471: mkdir d2/d17/da3 0 2024-03-23T11:47:51.220 INFO:tasks.workunit.client.1.smithi111.stdout:8/808: creat d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/dad/f109 x:0 0 0 2024-03-23T11:47:51.220 INFO:tasks.workunit.client.1.smithi111.stdout:3/987: fdatasync d0/d1a/d1d/d26/fb5 0 2024-03-23T11:47:51.222 INFO:tasks.workunit.client.0.smithi063.stdout:5/472: chown d2/d7/d8/f68 1225474397 1 2024-03-23T11:47:51.225 INFO:tasks.workunit.client.1.smithi111.stdout:0/924: creat d5/db/d4d/da6/d133/f139 x:0 0 0 2024-03-23T11:47:51.226 INFO:tasks.workunit.client.0.smithi063.stdout:7/638: write d5/d5c/d2f/d73/d20/f35 [1722410,93436] 0 2024-03-23T11:47:51.226 INFO:tasks.workunit.client.0.smithi063.stdout:6/538: creat d1/dc/d3a/d57/d6d/fa0 x:0 0 0 2024-03-23T11:47:51.226 INFO:tasks.workunit.client.1.smithi111.stdout:8/809: creat d5/d26/d2f/d62/d75/d4b/d51/dd1/f10a x:0 0 0 2024-03-23T11:47:51.229 INFO:tasks.workunit.client.0.smithi063.stdout:3/964: sync 2024-03-23T11:47:51.230 INFO:tasks.workunit.client.1.smithi111.stdout:3/988: stat d0/d1a/d2b/d38/dc2/c11c 0 2024-03-23T11:47:51.232 INFO:tasks.workunit.client.1.smithi111.stdout:3/989: readlink d0/d1a/l2f 0 2024-03-23T11:47:51.233 INFO:tasks.workunit.client.0.smithi063.stdout:6/539: symlink d1/dc/d3a/d82/la1 0 2024-03-23T11:47:51.234 INFO:tasks.workunit.client.1.smithi111.stdout:3/990: chown d0/d1a/d1d/f24 0 1 2024-03-23T11:47:51.234 INFO:tasks.workunit.client.1.smithi111.stdout:6/933: write d2/d1f/f92 [347754,53847] 0 2024-03-23T11:47:51.235 INFO:tasks.workunit.client.0.smithi063.stdout:7/639: mkdir d5/d22/d57/dc8 0 2024-03-23T11:47:51.237 INFO:tasks.workunit.client.1.smithi111.stdout:3/991: chown d0/d1a/d2d 766 1 2024-03-23T11:47:51.237 INFO:tasks.workunit.client.0.smithi063.stdout:7/640: readlink d5/d5c/d2f/d73/l21 0 2024-03-23T11:47:51.238 INFO:tasks.workunit.client.1.smithi111.stdout:6/934: write d2/d1a/d1c/d2c/f8c [4665269,29683] 0 2024-03-23T11:47:51.239 INFO:tasks.workunit.client.0.smithi063.stdout:6/540: read d1/f3b [495576,9944] 0 2024-03-23T11:47:51.241 INFO:tasks.workunit.client.1.smithi111.stdout:0/925: dread d5/db/d1f/d47/d59/f127 [0,4194304] 0 2024-03-23T11:47:51.241 INFO:tasks.workunit.client.0.smithi063.stdout:3/965: link d13/d22/d2d/d3f/d67/fd1 d13/d15/d114/f134 0 2024-03-23T11:47:51.243 INFO:tasks.workunit.client.1.smithi111.stdout:8/810: dread d5/d26/d2f/d62/d7e/d85/f96 [0,4194304] 0 2024-03-23T11:47:51.244 INFO:tasks.workunit.client.0.smithi063.stdout:7/641: read d5/d5c/d2f/d73/da/f89 [3904664,68141] 0 2024-03-23T11:47:51.244 INFO:tasks.workunit.client.0.smithi063.stdout:3/966: read - d13/d66/dde/f101 zero size 2024-03-23T11:47:51.245 INFO:tasks.workunit.client.0.smithi063.stdout:1/623: dread d3/f2a [0,4194304] 0 2024-03-23T11:47:51.249 INFO:tasks.workunit.client.0.smithi063.stdout:6/541: rename d1/d2a to d1/dc/d3a/d82/da2 0 2024-03-23T11:47:51.251 INFO:tasks.workunit.client.1.smithi111.stdout:3/992: rename d0/d1a/d2b/d10e/l7a to d0/d1a/d2b/d38/dd4/l146 0 2024-03-23T11:47:51.255 INFO:tasks.workunit.client.1.smithi111.stdout:6/935: dwrite d2/d73/f10f [0,4194304] 0 2024-03-23T11:47:51.256 INFO:tasks.workunit.client.1.smithi111.stdout:8/811: rename d5/d1b/lba to d5/d32/d7d/l10b 0 2024-03-23T11:47:51.257 INFO:tasks.workunit.client.1.smithi111.stdout:9/975: write d0/d4/d1c/d2d/d3b/d42/d4f/d95/ff0 [2507877,57436] 0 2024-03-23T11:47:51.258 INFO:tasks.workunit.client.1.smithi111.stdout:2/982: write d2/d32/d91/f85 [1353337,81479] 0 2024-03-23T11:47:51.262 INFO:tasks.workunit.client.1.smithi111.stdout:2/983: chown d2/d11/d17/c71 35 1 2024-03-23T11:47:51.263 INFO:tasks.workunit.client.1.smithi111.stdout:8/812: truncate d5/d32/d79/fb3 294221 0 2024-03-23T11:47:51.265 INFO:tasks.workunit.client.1.smithi111.stdout:1/930: truncate d0/d6/d48/d4a/de5/fea 220078 0 2024-03-23T11:47:51.265 INFO:tasks.workunit.client.1.smithi111.stdout:8/813: readlink d5/d26/d2f/d62/d75/d8d/l9d 0 2024-03-23T11:47:51.268 INFO:tasks.workunit.client.0.smithi063.stdout:3/967: symlink d13/d22/l135 0 2024-03-23T11:47:51.269 INFO:tasks.workunit.client.1.smithi111.stdout:1/931: truncate d0/dda/f121 1030122 0 2024-03-23T11:47:51.273 INFO:tasks.workunit.client.0.smithi063.stdout:7/642: rename d5/d22/d57 to d5/d26/db7/dc6/d6e/da2/dc9 0 2024-03-23T11:47:51.273 INFO:tasks.workunit.client.1.smithi111.stdout:0/926: write d5/db/d1f/d19/dbd/d64/dc7/d100/fca [800430,25822] 0 2024-03-23T11:47:51.275 INFO:tasks.workunit.client.0.smithi063.stdout:6/542: symlink d1/dc/d3a/d82/da2/d41/d42/d99/la3 0 2024-03-23T11:47:51.277 INFO:tasks.workunit.client.0.smithi063.stdout:6/543: chown d1/dc/d3a/d82/da2/f9b 48787 1 2024-03-23T11:47:51.281 INFO:tasks.workunit.client.0.smithi063.stdout:7/643: rmdir d5/d26/db7/dc6/d4f 39 2024-03-23T11:47:51.282 INFO:tasks.workunit.client.1.smithi111.stdout:2/984: getdents d2/d11/d17/d3e/d5d/dc2 0 2024-03-23T11:47:51.285 INFO:tasks.workunit.client.1.smithi111.stdout:7/928: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/f2b [0,4194304] 0 2024-03-23T11:47:51.286 INFO:tasks.workunit.client.1.smithi111.stdout:6/936: dwrite d2/d1a/d1c/d2c/d51/def/d118/f11c [0,4194304] 0 2024-03-23T11:47:51.289 INFO:tasks.workunit.client.1.smithi111.stdout:3/993: dwrite d0/d1a/d2b/fca [0,4194304] 0 2024-03-23T11:47:51.289 INFO:tasks.workunit.client.1.smithi111.stdout:0/927: mkdir d5/db/d1f/d47/d59/d13a 0 2024-03-23T11:47:51.294 INFO:tasks.workunit.client.0.smithi063.stdout:1/624: dwrite d3/df/f36 [0,4194304] 0 2024-03-23T11:47:51.296 INFO:tasks.workunit.client.1.smithi111.stdout:8/814: dwrite d5/f19 [0,4194304] 0 2024-03-23T11:47:51.298 INFO:tasks.workunit.client.1.smithi111.stdout:2/985: chown d2/d11/d17/d3e/d5d/dc0/fde 44884 1 2024-03-23T11:47:51.303 INFO:tasks.workunit.client.1.smithi111.stdout:8/815: chown d5/d26/d2f/d62/d7e/d85 484995 1 2024-03-23T11:47:51.303 INFO:tasks.workunit.client.0.smithi063.stdout:1/625: fdatasync d3/d1c/d4b/f71 0 2024-03-23T11:47:51.304 INFO:tasks.workunit.client.0.smithi063.stdout:1/626: chown d3/d62/dad 249 1 2024-03-23T11:47:51.308 INFO:tasks.workunit.client.1.smithi111.stdout:7/929: unlink d14/d1b/d1e/d87/lc0 0 2024-03-23T11:47:51.309 INFO:tasks.workunit.client.0.smithi063.stdout:6/544: write d1/dc/d3a/d82/da2/f83 [54504,69005] 0 2024-03-23T11:47:51.311 INFO:tasks.workunit.client.0.smithi063.stdout:3/968: write d13/d22/d2d/d7d/fb5 [4020023,69795] 0 2024-03-23T11:47:51.312 INFO:tasks.workunit.client.1.smithi111.stdout:2/986: symlink d2/de/l15b 0 2024-03-23T11:47:51.318 INFO:tasks.workunit.client.1.smithi111.stdout:6/937: dwrite d2/d73/f11e [0,4194304] 0 2024-03-23T11:47:51.318 INFO:tasks.workunit.client.1.smithi111.stdout:1/932: dwrite d0/d6/d48/d4a/d6c/d83/d11d/f6e [0,4194304] 0 2024-03-23T11:47:51.321 INFO:tasks.workunit.client.1.smithi111.stdout:8/816: symlink d5/d26/d2f/d62/l10c 0 2024-03-23T11:47:51.325 INFO:tasks.workunit.client.1.smithi111.stdout:7/930: write d14/d44/d70/d7b/fa8 [303532,64832] 0 2024-03-23T11:47:51.329 INFO:tasks.workunit.client.1.smithi111.stdout:6/938: creat d2/d1f/d45/d9a/ddd/f12d x:0 0 0 2024-03-23T11:47:51.330 INFO:tasks.workunit.client.1.smithi111.stdout:1/933: fdatasync d0/d6/d48/d4a/d6c/d83/d11d/d1e/f21 0 2024-03-23T11:47:51.332 INFO:tasks.workunit.client.1.smithi111.stdout:9/976: dread d0/d31/d41/d65/f6f [0,4194304] 0 2024-03-23T11:47:51.335 INFO:tasks.workunit.client.0.smithi063.stdout:3/969: rename d13/d15/fca to d13/d22/d2d/d3e/dce/dfb/d117/f136 0 2024-03-23T11:47:51.338 INFO:tasks.workunit.client.1.smithi111.stdout:1/934: dread - d0/d2b/d69/f10b zero size 2024-03-23T11:47:51.339 INFO:tasks.workunit.client.0.smithi063.stdout:7/644: dwrite d5/d26/f72 [0,4194304] 0 2024-03-23T11:47:51.339 INFO:tasks.workunit.client.1.smithi111.stdout:9/977: dread - d0/d31/d41/d9d/dd4/de0/fe4 zero size 2024-03-23T11:47:51.339 INFO:tasks.workunit.client.1.smithi111.stdout:0/928: dwrite d5/db/d1f/d47/f5a [4194304,4194304] 0 2024-03-23T11:47:51.340 INFO:tasks.workunit.client.0.smithi063.stdout:1/627: creat d3/d3f/fc1 x:0 0 0 2024-03-23T11:47:51.341 INFO:tasks.workunit.client.1.smithi111.stdout:8/817: fdatasync d5/d26/d31/d5e/f3e 0 2024-03-23T11:47:51.342 INFO:tasks.workunit.client.0.smithi063.stdout:7/645: chown d5/d6/d39/d88 602473 1 2024-03-23T11:47:51.345 INFO:tasks.workunit.client.0.smithi063.stdout:3/970: mkdir d13/d22/d2d/d3f/d58/d6e/d8d/dda/dee/d137 0 2024-03-23T11:47:51.350 INFO:tasks.workunit.client.1.smithi111.stdout:1/935: mkdir d0/d6/d48/d4d/dab/d13d 0 2024-03-23T11:47:51.353 INFO:tasks.workunit.client.0.smithi063.stdout:1/628: unlink d3/f81 0 2024-03-23T11:47:51.354 INFO:tasks.workunit.client.1.smithi111.stdout:1/936: mkdir d0/d2b/d7c/d13e 0 2024-03-23T11:47:51.356 INFO:tasks.workunit.client.0.smithi063.stdout:7/646: dread - d5/d22/f9d zero size 2024-03-23T11:47:51.359 INFO:tasks.workunit.client.1.smithi111.stdout:0/929: getdents d5/db/d1f/d47/d59/d13a 0 2024-03-23T11:47:51.360 INFO:tasks.workunit.client.1.smithi111.stdout:3/994: dwrite d0/d1a/d2b/d10e/d19/d5e/da6/deb/ff9 [0,4194304] 0 2024-03-23T11:47:51.360 INFO:tasks.workunit.client.0.smithi063.stdout:3/971: symlink d13/d22/d2d/d3f/dd4/l138 0 2024-03-23T11:47:51.361 INFO:tasks.workunit.client.0.smithi063.stdout:1/629: symlink d3/d3f/d4d/d52/lc2 0 2024-03-23T11:47:51.362 INFO:tasks.workunit.client.1.smithi111.stdout:8/818: mkdir d5/d10d 0 2024-03-23T11:47:51.364 INFO:tasks.workunit.client.1.smithi111.stdout:2/987: dwrite d2/d11/d17/d3e/d6b/d2e/d3f/f4d [0,4194304] 0 2024-03-23T11:47:51.366 INFO:tasks.workunit.client.0.smithi063.stdout:1/630: dread - d3/d35/f94 zero size 2024-03-23T11:47:51.368 INFO:tasks.workunit.client.0.smithi063.stdout:3/972: getdents d13/d22/d2d/d3f/d58/d6e/dc4 0 2024-03-23T11:47:51.368 INFO:tasks.workunit.client.0.smithi063.stdout:1/631: readlink d3/d1c/l4e 0 2024-03-23T11:47:51.372 INFO:tasks.workunit.client.0.smithi063.stdout:3/973: chown d13/d22/d2d/d3f/d67/d7b/fab 23227967 1 2024-03-23T11:47:51.376 INFO:tasks.workunit.client.1.smithi111.stdout:2/988: read d2/d11/d17/d3e/d5d/fa6 [209007,15689] 0 2024-03-23T11:47:51.377 INFO:tasks.workunit.client.0.smithi063.stdout:3/974: symlink d13/d66/d112/l139 0 2024-03-23T11:47:51.377 INFO:tasks.workunit.client.1.smithi111.stdout:0/930: fsync d5/db/d1f/d19/dbd/fa5 0 2024-03-23T11:47:51.380 INFO:tasks.workunit.client.1.smithi111.stdout:3/995: link d0/d1a/d1d/d26/d5a/d60/l13d d0/d72/d7b/l147 0 2024-03-23T11:47:51.382 INFO:tasks.workunit.client.1.smithi111.stdout:8/819: symlink d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/dad/l10e 0 2024-03-23T11:47:51.385 INFO:tasks.workunit.client.1.smithi111.stdout:1/937: truncate d0/d2b/de3/f115 335162 0 2024-03-23T11:47:51.385 INFO:tasks.workunit.client.1.smithi111.stdout:7/931: dwrite d14/d1b/d1e/d2c/d2e/d73/dae/f62 [0,4194304] 0 2024-03-23T11:47:51.388 INFO:tasks.workunit.client.1.smithi111.stdout:6/939: dread d2/d1a/d1c/d2c/fa5 [0,4194304] 0 2024-03-23T11:47:51.390 INFO:tasks.workunit.client.0.smithi063.stdout:3/975: truncate d13/d22/d2d/f45 3551869 0 2024-03-23T11:47:51.391 INFO:tasks.workunit.client.1.smithi111.stdout:0/931: rmdir d5/db/d1f/d19/dbd/d64/d91 39 2024-03-23T11:47:51.391 INFO:tasks.workunit.client.1.smithi111.stdout:3/996: dread - d0/d1a/d2b/d38/dd4/ff8 zero size 2024-03-23T11:47:51.394 INFO:tasks.workunit.client.1.smithi111.stdout:2/989: read d2/d11/d17/d3e/d6b/d2e/d3f/f4d [2940842,28814] 0 2024-03-23T11:47:51.396 INFO:tasks.workunit.client.1.smithi111.stdout:7/932: mkdir d14/d44/d70/d7b/d113 0 2024-03-23T11:47:51.397 INFO:tasks.workunit.client.0.smithi063.stdout:3/976: unlink d13/d22/d2d/d7d/ce2 0 2024-03-23T11:47:51.400 INFO:tasks.workunit.client.1.smithi111.stdout:9/978: dwrite d0/d4/f112 [0,4194304] 0 2024-03-23T11:47:51.400 INFO:tasks.workunit.client.1.smithi111.stdout:8/820: rename d5/l54 to d5/d26/d2f/l10f 0 2024-03-23T11:47:51.404 INFO:tasks.workunit.client.0.smithi063.stdout:3/977: fsync d13/d22/d2d/d3f/d58/d97/df8/f119 0 2024-03-23T11:47:51.404 INFO:tasks.workunit.client.0.smithi063.stdout:6/545: dwrite d1/dc/d21/d49/f6a [0,4194304] 0 2024-03-23T11:47:51.405 INFO:tasks.workunit.client.1.smithi111.stdout:2/990: symlink d2/d11/d17/d3e/d6b/d2e/l15c 0 2024-03-23T11:47:51.407 INFO:tasks.workunit.client.0.smithi063.stdout:7/647: write d5/d26/db7/dc6/d4f/d5b/f66 [2291878,116310] 0 2024-03-23T11:47:51.408 INFO:tasks.workunit.client.0.smithi063.stdout:1/632: write d3/d49/d56/f86 [680266,102615] 0 2024-03-23T11:47:51.409 INFO:tasks.workunit.client.0.smithi063.stdout:7/648: chown d5/d6/d39/d88 0 1 2024-03-23T11:47:51.411 INFO:tasks.workunit.client.0.smithi063.stdout:3/978: symlink d13/d22/d2d/d78/l13a 0 2024-03-23T11:47:51.412 INFO:tasks.workunit.client.1.smithi111.stdout:7/933: creat d14/d1b/d1e/d2c/d2e/d73/dae/dea/f114 x:0 0 0 2024-03-23T11:47:51.413 INFO:tasks.workunit.client.1.smithi111.stdout:2/991: truncate d2/d11/d17/d3e/d5d/dc0/df7/d10e/f139 353308 0 2024-03-23T11:47:51.418 INFO:tasks.workunit.client.0.smithi063.stdout:7/649: write d5/d5c/d2f/d73/d20/f35 [734304,87630] 0 2024-03-23T11:47:51.419 INFO:tasks.workunit.client.1.smithi111.stdout:3/997: dwrite d0/d55/f56 [4194304,4194304] 0 2024-03-23T11:47:51.423 INFO:tasks.workunit.client.0.smithi063.stdout:6/546: link d1/f9e d1/d15/fa4 0 2024-03-23T11:47:51.423 INFO:tasks.workunit.client.1.smithi111.stdout:1/938: rename d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/c114 to d0/d2b/c13f 0 2024-03-23T11:47:51.423 INFO:tasks.workunit.client.0.smithi063.stdout:7/650: write d5/d6/d19/d1f/d6c/f80 [2309077,100701] 0 2024-03-23T11:47:51.425 INFO:tasks.workunit.client.1.smithi111.stdout:2/992: chown d2/de/d6c/dd1/fe2 5276571 1 2024-03-23T11:47:51.431 INFO:tasks.workunit.client.1.smithi111.stdout:6/940: dread d2/d80/f99 [0,4194304] 0 2024-03-23T11:47:51.432 INFO:tasks.workunit.client.1.smithi111.stdout:8/821: dwrite d5/d26/d2f/d62/d7e/dae/ffc [0,4194304] 0 2024-03-23T11:47:51.433 INFO:tasks.workunit.client.1.smithi111.stdout:0/932: rename d5/db/d1f/d19/dbd/d64/d91/cfe to d5/deb/c13b 0 2024-03-23T11:47:51.439 INFO:tasks.workunit.client.1.smithi111.stdout:2/993: symlink d2/d11/d17/d1f/da0/dd2/d145/l15d 0 2024-03-23T11:47:51.443 INFO:tasks.workunit.client.1.smithi111.stdout:3/998: rename d0/d1a/d4f/d90/l9e to d0/d1a/d4f/d13a/l148 0 2024-03-23T11:47:51.444 INFO:tasks.workunit.client.1.smithi111.stdout:0/933: mknod d5/d121/c13c 0 2024-03-23T11:47:51.446 INFO:tasks.workunit.client.1.smithi111.stdout:2/994: mkdir d2/de0/d110/d147/d15e 0 2024-03-23T11:47:51.446 INFO:tasks.workunit.client.1.smithi111.stdout:8/822: write d5/d26/d2f/d62/d7e/dae/ffc [353189,52433] 0 2024-03-23T11:47:51.449 INFO:tasks.workunit.client.1.smithi111.stdout:2/995: chown d2/de/d12/l7c 167586222 1 2024-03-23T11:47:51.454 INFO:tasks.workunit.client.1.smithi111.stdout:2/996: rmdir d2/d13d 39 2024-03-23T11:47:51.456 INFO:tasks.workunit.client.1.smithi111.stdout:9/979: dread d0/d4/d1c/d2d/d3b/d42/d4f/d95/fb9 [0,4194304] 0 2024-03-23T11:47:51.459 INFO:tasks.workunit.client.1.smithi111.stdout:0/934: rename d5/db/d1f/d19/f105 to d5/db/d1f/d47/d6f/f13d 0 2024-03-23T11:47:51.459 INFO:tasks.workunit.client.0.smithi063.stdout:1/633: dwrite d3/d35/d40/f4c [8388608,4194304] 0 2024-03-23T11:47:51.460 INFO:tasks.workunit.client.0.smithi063.stdout:6/547: write d1/d15/f70 [177554,41082] 0 2024-03-23T11:47:51.461 INFO:tasks.workunit.client.0.smithi063.stdout:3/979: dwrite d13/de5/ffd [0,4194304] 0 2024-03-23T11:47:51.462 INFO:tasks.workunit.client.1.smithi111.stdout:9/980: rmdir d0/d31/d41/d9d/db0 39 2024-03-23T11:47:51.463 INFO:tasks.workunit.client.0.smithi063.stdout:1/634: write d3/d3f/d4d/f8c [353487,24816] 0 2024-03-23T11:47:51.464 INFO:tasks.workunit.client.1.smithi111.stdout:0/935: mknod d5/db/d1f/d47/d87/ddf/c13e 0 2024-03-23T11:47:51.464 INFO:tasks.workunit.client.0.smithi063.stdout:3/980: dread - d13/d22/d2d/d3f/de0/f104 zero size 2024-03-23T11:47:51.468 INFO:tasks.workunit.client.0.smithi063.stdout:6/548: mkdir d1/dc/d21/da5 0 2024-03-23T11:47:51.471 INFO:tasks.workunit.client.1.smithi111.stdout:0/936: fsync d5/db/d1f/d19/f23 0 2024-03-23T11:47:51.472 INFO:tasks.workunit.client.0.smithi063.stdout:1/635: creat d3/d1c/da8/fc3 x:0 0 0 2024-03-23T11:47:51.473 INFO:tasks.workunit.client.0.smithi063.stdout:3/981: fsync d13/d22/d2d/d3f/d5b/f113 0 2024-03-23T11:47:51.477 INFO:tasks.workunit.client.0.smithi063.stdout:6/549: truncate d1/d15/d4b/f6b 123487 0 2024-03-23T11:47:51.478 INFO:tasks.workunit.client.0.smithi063.stdout:4/704: sync 2024-03-23T11:47:51.478 INFO:tasks.workunit.client.0.smithi063.stdout:0/841: sync 2024-03-23T11:47:51.478 INFO:tasks.workunit.client.0.smithi063.stdout:9/683: sync 2024-03-23T11:47:51.479 INFO:tasks.workunit.client.1.smithi111.stdout:7/934: write d14/d1b/f52 [5760527,110989] 0 2024-03-23T11:47:51.479 INFO:tasks.workunit.client.0.smithi063.stdout:2/551: sync 2024-03-23T11:47:51.479 INFO:tasks.workunit.client.0.smithi063.stdout:1/636: creat d3/df/db8/fc4 x:0 0 0 2024-03-23T11:47:51.482 INFO:tasks.workunit.client.1.smithi111.stdout:3/999: write d0/fd [8469765,71871] 0 2024-03-23T11:47:51.484 INFO:tasks.workunit.client.0.smithi063.stdout:2/552: stat d2/d46/d54/f68 0 2024-03-23T11:47:51.486 INFO:tasks.workunit.client.1.smithi111.stdout:2/997: dwrite d2/d32/d98/fe1 [0,4194304] 0 2024-03-23T11:47:51.490 INFO:tasks.workunit.client.0.smithi063.stdout:0/842: write d2/d12/d32/d6d/d124/d28/fdf [175272,14375] 0 2024-03-23T11:47:51.491 INFO:tasks.workunit.client.1.smithi111.stdout:0/937: truncate d5/db/d1f/d19/d45/fad 821123 0 2024-03-23T11:47:51.491 INFO:tasks.workunit.client.1.smithi111.stdout:1/939: dread d0/d2b/d37/f4c [0,4194304] 0 2024-03-23T11:47:51.492 INFO:tasks.workunit.client.1.smithi111.stdout:8/823: dwrite d5/d26/d2f/d62/d7e/d85/f96 [0,4194304] 0 2024-03-23T11:47:51.493 INFO:tasks.workunit.client.0.smithi063.stdout:1/637: unlink d3/df/l48 0 2024-03-23T11:47:51.495 INFO:tasks.workunit.client.1.smithi111.stdout:6/941: dread d2/d1a/d1c/d2c/d51/fa9 [0,4194304] 0 2024-03-23T11:47:51.495 INFO:tasks.workunit.client.0.smithi063.stdout:9/684: mkdir d4/d4c/df4 0 2024-03-23T11:47:51.496 INFO:tasks.workunit.client.1.smithi111.stdout:7/935: truncate d14/d1b/d1e/d2c/d42/fc9 521239 0 2024-03-23T11:47:51.496 INFO:tasks.workunit.client.0.smithi063.stdout:6/550: rename d1/d15/d4b/l4d to d1/dc/d3a/d82/da2/d41/d42/la6 0 2024-03-23T11:47:51.498 INFO:tasks.workunit.client.1.smithi111.stdout:0/938: rmdir d5/db/d1f/d47/d106/d10f 39 2024-03-23T11:47:51.501 INFO:tasks.workunit.client.1.smithi111.stdout:8/824: fdatasync d5/d26/d31/d3b/d55/ff1 0 2024-03-23T11:47:51.501 INFO:tasks.workunit.client.1.smithi111.stdout:0/939: chown d5/db/d1f/d48/fb0 63 1 2024-03-23T11:47:51.503 INFO:tasks.workunit.client.1.smithi111.stdout:1/940: mknod d0/d6/d48/d4a/d6c/d83/d11d/d91/c140 0 2024-03-23T11:47:51.505 INFO:tasks.workunit.client.1.smithi111.stdout:8/825: chown d5/d26/d2f/d62/d75/d4b/d51/d5a/dbe 23640978 1 2024-03-23T11:47:51.505 INFO:tasks.workunit.client.0.smithi063.stdout:2/553: read d2/d1c/f66 [2563589,51697] 0 2024-03-23T11:47:51.506 INFO:tasks.workunit.client.1.smithi111.stdout:7/936: creat d14/d44/f115 x:0 0 0 2024-03-23T11:47:51.507 INFO:tasks.workunit.client.0.smithi063.stdout:1/638: mkdir d3/d49/d56/d65/db6/dc5 0 2024-03-23T11:47:51.507 INFO:tasks.workunit.client.0.smithi063.stdout:9/685: chown d4/d4c/d19/l37 28485876 1 2024-03-23T11:47:51.507 INFO:tasks.workunit.client.1.smithi111.stdout:0/940: fdatasync d5/db/d1f/d19/dbd/d64/d91/ddd/df6/f125 0 2024-03-23T11:47:51.510 INFO:tasks.workunit.client.0.smithi063.stdout:0/843: getdents d2/d6e/d8c 0 2024-03-23T11:47:51.511 INFO:tasks.workunit.client.1.smithi111.stdout:6/942: dread d2/d35/f6f [0,4194304] 0 2024-03-23T11:47:51.516 INFO:tasks.workunit.client.1.smithi111.stdout:6/943: creat d2/d1f/d45/f12e x:0 0 0 2024-03-23T11:47:51.518 INFO:tasks.workunit.client.0.smithi063.stdout:4/705: dwrite d5/da/d20/d4b/d16/d50/f64 [0,4194304] 0 2024-03-23T11:47:51.519 INFO:tasks.workunit.client.1.smithi111.stdout:6/944: dread - d2/d35/d70/d98/fa4 zero size 2024-03-23T11:47:51.522 INFO:tasks.workunit.client.1.smithi111.stdout:0/941: link d5/db/c3c d5/db/d1f/d19/dbd/d64/d91/ddd/c13f 0 2024-03-23T11:47:51.524 INFO:tasks.workunit.client.1.smithi111.stdout:2/998: dwrite d2/d32/ffc [0,4194304] 0 2024-03-23T11:47:51.526 INFO:tasks.workunit.client.1.smithi111.stdout:8/826: dwrite d5/d26/d2f/d8f/fb7 [0,4194304] 0 2024-03-23T11:47:51.527 INFO:tasks.workunit.client.0.smithi063.stdout:6/551: fdatasync d1/dc/d3a/d82/da2/f4a 0 2024-03-23T11:47:51.529 INFO:tasks.workunit.client.0.smithi063.stdout:2/554: truncate d2/d23/d2e/f55 98555 0 2024-03-23T11:47:51.530 INFO:tasks.workunit.client.1.smithi111.stdout:6/945: rmdir d2/d1a/d1c/d2c/d51/def/d118 39 2024-03-23T11:47:51.531 INFO:tasks.workunit.client.0.smithi063.stdout:6/552: fsync d1/dc/d3a/d82/da2/d41/d42/f9d 0 2024-03-23T11:47:51.536 INFO:tasks.workunit.client.1.smithi111.stdout:0/942: unlink d5/db/d1f/d19/c4e 0 2024-03-23T11:47:51.540 INFO:tasks.workunit.client.1.smithi111.stdout:6/946: creat d2/d1a/d1c/d2c/d127/f12f x:0 0 0 2024-03-23T11:47:51.545 INFO:tasks.workunit.client.1.smithi111.stdout:0/943: creat d5/db/d4d/f140 x:0 0 0 2024-03-23T11:47:51.545 INFO:tasks.workunit.client.0.smithi063.stdout:0/844: mknod d2/d12/d32/d6d/d124/d1f/d4f/c128 0 2024-03-23T11:47:51.547 INFO:tasks.workunit.client.0.smithi063.stdout:3/982: link d13/d22/d2d/f45 d13/d22/d2d/f13b 0 2024-03-23T11:47:51.548 INFO:tasks.workunit.client.1.smithi111.stdout:8/827: dwrite d5/d26/d31/fe6 [0,4194304] 0 2024-03-23T11:47:51.548 INFO:tasks.workunit.client.0.smithi063.stdout:2/555: creat d2/d23/d2e/d87/d4d/fb1 x:0 0 0 2024-03-23T11:47:51.550 INFO:tasks.workunit.client.0.smithi063.stdout:1/639: dwrite d3/d49/d56/d65/d88/d92/fb0 [0,4194304] 0 2024-03-23T11:47:51.551 INFO:tasks.workunit.client.0.smithi063.stdout:6/553: dread - d1/f67 zero size 2024-03-23T11:47:51.553 INFO:tasks.workunit.client.1.smithi111.stdout:8/828: stat d5/d26/d2f/d42/dd3/fde 0 2024-03-23T11:47:51.553 INFO:tasks.workunit.client.0.smithi063.stdout:6/554: stat d1/dc/d3a 0 2024-03-23T11:47:51.554 INFO:tasks.workunit.client.1.smithi111.stdout:0/944: mknod d5/db/d1f/d47/d59/c141 0 2024-03-23T11:47:51.556 INFO:tasks.workunit.client.0.smithi063.stdout:3/983: rmdir d13/d22/d2d 39 2024-03-23T11:47:51.561 INFO:tasks.workunit.client.1.smithi111.stdout:6/947: dread d2/d73/f11e [0,4194304] 0 2024-03-23T11:47:51.569 INFO:tasks.workunit.client.0.smithi063.stdout:0/845: rename d2/d12/d35 to d2/d14/d36/d77/d94/d129 0 2024-03-23T11:47:51.570 INFO:tasks.workunit.client.1.smithi111.stdout:6/948: mkdir d2/d1a/d1c/d2c/d51/d130 0 2024-03-23T11:47:51.572 INFO:tasks.workunit.client.0.smithi063.stdout:9/686: dread d4/f47 [0,4194304] 0 2024-03-23T11:47:51.574 INFO:tasks.workunit.client.1.smithi111.stdout:2/999: dread d2/d5c/f90 [0,4194304] 0 2024-03-23T11:47:51.575 INFO:tasks.workunit.client.0.smithi063.stdout:2/556: unlink d2/d23/d2e/d52/f33 0 2024-03-23T11:47:51.579 INFO:tasks.workunit.client.0.smithi063.stdout:6/555: symlink d1/dc/d3a/d82/da2/d41/la7 0 2024-03-23T11:47:51.582 INFO:tasks.workunit.client.1.smithi111.stdout:0/945: write d5/fef [930929,65540] 0 2024-03-23T11:47:51.583 INFO:tasks.workunit.client.1.smithi111.stdout:8/829: rename d5/d32/d9b/cdd to d5/d32/d9b/c110 0 2024-03-23T11:47:51.585 INFO:tasks.workunit.client.1.smithi111.stdout:8/830: read - d5/d32/d7d/fa6 zero size 2024-03-23T11:47:51.585 INFO:tasks.workunit.client.1.smithi111.stdout:6/949: write d2/d1f/ff0 [175533,102836] 0 2024-03-23T11:47:51.586 INFO:tasks.workunit.client.0.smithi063.stdout:4/706: dwrite d5/fc [0,4194304] 0 2024-03-23T11:47:51.589 INFO:tasks.workunit.client.1.smithi111.stdout:6/950: creat d2/d1f/d45/f131 x:0 0 0 2024-03-23T11:47:51.591 INFO:tasks.workunit.client.1.smithi111.stdout:0/946: write d5/db/d1f/f79 [818094,117564] 0 2024-03-23T11:47:51.594 INFO:tasks.workunit.client.0.smithi063.stdout:2/557: chown d2/c8c 580097 1 2024-03-23T11:47:51.596 INFO:tasks.workunit.client.1.smithi111.stdout:6/951: rename d2/d1f/f7a to d2/d1f/dce/f132 0 2024-03-23T11:47:51.597 INFO:tasks.workunit.client.1.smithi111.stdout:0/947: unlink d5/f30 0 2024-03-23T11:47:51.598 INFO:tasks.workunit.client.1.smithi111.stdout:6/952: chown d2/d1f/dce/fec 40207 1 2024-03-23T11:47:51.598 INFO:tasks.workunit.client.1.smithi111.stdout:0/948: chown d5/db/d4d/da6/d10b 0 1 2024-03-23T11:47:51.601 INFO:tasks.workunit.client.1.smithi111.stdout:0/949: dread - d5/db/d1f/d19/dbd/d64/dc7/fcf zero size 2024-03-23T11:47:51.602 INFO:tasks.workunit.client.0.smithi063.stdout:0/846: symlink d2/d48/d110/l12a 0 2024-03-23T11:47:51.602 INFO:tasks.workunit.client.1.smithi111.stdout:8/831: creat d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/dad/f111 x:0 0 0 2024-03-23T11:47:51.602 INFO:tasks.workunit.client.1.smithi111.stdout:6/953: creat d2/d35/f133 x:0 0 0 2024-03-23T11:47:51.603 INFO:tasks.workunit.client.1.smithi111.stdout:0/950: stat d5/db/ld 0 2024-03-23T11:47:51.604 INFO:tasks.workunit.client.1.smithi111.stdout:8/832: chown d5/f19 3886 1 2024-03-23T11:47:51.607 INFO:tasks.workunit.client.1.smithi111.stdout:0/951: truncate d5/db/d4d/da6/d133/f139 979981 0 2024-03-23T11:47:51.607 INFO:tasks.workunit.client.0.smithi063.stdout:4/707: mkdir d5/da/d20/d4b/d16/d50/d8e/d96/dca 0 2024-03-23T11:47:51.607 INFO:tasks.workunit.client.1.smithi111.stdout:6/954: creat d2/d1a/d1c/f134 x:0 0 0 2024-03-23T11:47:51.608 INFO:tasks.workunit.client.1.smithi111.stdout:0/952: stat d5/c77 0 2024-03-23T11:47:51.608 INFO:tasks.workunit.client.1.smithi111.stdout:6/955: stat d2/d1f/d45/d49/lb6 0 2024-03-23T11:47:51.609 INFO:tasks.workunit.client.1.smithi111.stdout:8/833: unlink d5/d26/d31/d3b/f3d 0 2024-03-23T11:47:51.610 INFO:tasks.workunit.client.0.smithi063.stdout:2/558: truncate d2/d23/d2e/f35 999566 0 2024-03-23T11:47:51.610 INFO:tasks.workunit.client.1.smithi111.stdout:0/953: dread - d5/db/d1f/de4/f112 zero size 2024-03-23T11:47:51.610 INFO:tasks.workunit.client.0.smithi063.stdout:3/984: getdents d13/d22/d2d/d3f/d58/d10d 0 2024-03-23T11:47:51.611 INFO:tasks.workunit.client.0.smithi063.stdout:9/687: getdents d4/d31/d43/d49 0 2024-03-23T11:47:51.612 INFO:tasks.workunit.client.0.smithi063.stdout:1/640: fsync d3/d35/f5e 0 2024-03-23T11:47:51.617 INFO:tasks.workunit.client.0.smithi063.stdout:0/847: symlink d2/d14/d36/dc3/l12b 0 2024-03-23T11:47:51.618 INFO:tasks.workunit.client.1.smithi111.stdout:6/956: dread d2/d1f/dce/fec [0,4194304] 0 2024-03-23T11:47:51.620 INFO:tasks.workunit.client.1.smithi111.stdout:6/957: chown d2/d1a/d1c/d2c/d51/c74 32762610 1 2024-03-23T11:47:51.622 INFO:tasks.workunit.client.0.smithi063.stdout:9/688: stat d4/d31/d43/da9/da6/c6d 0 2024-03-23T11:47:51.625 INFO:tasks.workunit.client.0.smithi063.stdout:2/559: creat d2/de/d22/d91/fb2 x:0 0 0 2024-03-23T11:47:51.625 INFO:tasks.workunit.client.1.smithi111.stdout:6/958: creat d2/d1f/f135 x:0 0 0 2024-03-23T11:47:51.626 INFO:tasks.workunit.client.1.smithi111.stdout:0/954: dwrite d5/db/d1f/fc0 [0,4194304] 0 2024-03-23T11:47:51.627 INFO:tasks.workunit.client.0.smithi063.stdout:2/560: fdatasync d2/de/d22/f5b 0 2024-03-23T11:47:51.627 INFO:tasks.workunit.client.0.smithi063.stdout:8/971: sync 2024-03-23T11:47:51.630 INFO:tasks.workunit.client.0.smithi063.stdout:4/708: write d5/da/d20/fb7 [923693,23331] 0 2024-03-23T11:47:51.631 INFO:tasks.workunit.client.0.smithi063.stdout:3/985: getdents d13/d22/d2d/d3f/df6 0 2024-03-23T11:47:51.633 INFO:tasks.workunit.client.0.smithi063.stdout:6/556: dread d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:51.637 INFO:tasks.workunit.client.0.smithi063.stdout:0/848: write d2/d6e/d8c/ff1 [601126,29112] 0 2024-03-23T11:47:51.639 INFO:tasks.workunit.client.0.smithi063.stdout:1/641: fdatasync d3/f39 0 2024-03-23T11:47:51.640 INFO:tasks.workunit.client.0.smithi063.stdout:2/561: truncate d2/d23/f83 472847 0 2024-03-23T11:47:51.641 INFO:tasks.workunit.client.0.smithi063.stdout:3/986: mknod d13/d22/d2d/d78/dcc/c13c 0 2024-03-23T11:47:51.642 INFO:tasks.workunit.client.0.smithi063.stdout:4/709: fsync d5/da/d20/d4b/d16/d50/d8e/f97 0 2024-03-23T11:47:51.645 INFO:tasks.workunit.client.0.smithi063.stdout:1/642: stat d3/d62/c6b 0 2024-03-23T11:47:51.645 INFO:tasks.workunit.client.1.smithi111.stdout:8/834: dread d5/d26/d31/d3b/fa9 [0,4194304] 0 2024-03-23T11:47:51.645 INFO:tasks.workunit.client.0.smithi063.stdout:4/710: chown d5/da/d20/d4b/d16/dc4 60 1 2024-03-23T11:47:51.646 INFO:tasks.workunit.client.1.smithi111.stdout:6/959: dwrite d2/d17/f55 [0,4194304] 0 2024-03-23T11:47:51.647 INFO:tasks.workunit.client.0.smithi063.stdout:8/972: creat d6/dd/d59/daf/dba/d135/f148 x:0 0 0 2024-03-23T11:47:51.647 INFO:tasks.workunit.client.0.smithi063.stdout:9/689: write d4/d4c/f8c [4376634,99156] 0 2024-03-23T11:47:51.651 INFO:tasks.workunit.client.0.smithi063.stdout:6/557: getdents d1/d15/d8b 0 2024-03-23T11:47:51.653 INFO:tasks.workunit.client.0.smithi063.stdout:9/690: write d4/d31/d70/d8d/da0/fe2 [195387,47674] 0 2024-03-23T11:47:51.656 INFO:tasks.workunit.client.1.smithi111.stdout:0/955: dwrite d5/db/d1f/f9b [0,4194304] 0 2024-03-23T11:47:51.657 INFO:tasks.workunit.client.0.smithi063.stdout:3/987: symlink d13/d22/d2d/d3f/d67/l13d 0 2024-03-23T11:47:51.658 INFO:tasks.workunit.client.0.smithi063.stdout:0/849: mknod d2/d12/d32/d6d/d124/d1f/d101/c12c 0 2024-03-23T11:47:51.660 INFO:tasks.workunit.client.0.smithi063.stdout:0/850: chown d2/d4/f8a 1681134738 1 2024-03-23T11:47:51.660 INFO:tasks.workunit.client.0.smithi063.stdout:3/988: fdatasync d13/d22/d2d/d3f/d58/f70 0 2024-03-23T11:47:51.660 INFO:tasks.workunit.client.1.smithi111.stdout:0/956: symlink d5/db/d1f/de4/l142 0 2024-03-23T11:47:51.662 INFO:tasks.workunit.client.0.smithi063.stdout:7/651: sync 2024-03-23T11:47:51.662 INFO:tasks.workunit.client.0.smithi063.stdout:5/473: sync 2024-03-23T11:47:51.664 INFO:tasks.workunit.client.1.smithi111.stdout:9/981: sync 2024-03-23T11:47:51.664 INFO:tasks.workunit.client.1.smithi111.stdout:7/937: sync 2024-03-23T11:47:51.664 INFO:tasks.workunit.client.1.smithi111.stdout:1/941: sync 2024-03-23T11:47:51.671 INFO:tasks.workunit.client.1.smithi111.stdout:9/982: stat d0/d4/d1c/f45 0 2024-03-23T11:47:51.671 INFO:tasks.workunit.client.1.smithi111.stdout:1/942: truncate d0/d6/fd8 605124 0 2024-03-23T11:47:51.673 INFO:tasks.workunit.client.0.smithi063.stdout:6/558: creat d1/dc/d3a/d57/d6d/fa8 x:0 0 0 2024-03-23T11:47:51.673 INFO:tasks.workunit.client.0.smithi063.stdout:7/652: readlink d5/d5c/d2f/d73/da/l1b 0 2024-03-23T11:47:51.674 INFO:tasks.workunit.client.1.smithi111.stdout:8/835: truncate d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/f52 4011153 0 2024-03-23T11:47:51.676 INFO:tasks.workunit.client.1.smithi111.stdout:0/957: rename d5/db/d1f/d19/dbd/d64/d91/ddd/df6/f10a to d5/f143 0 2024-03-23T11:47:51.676 INFO:tasks.workunit.client.0.smithi063.stdout:3/989: readlink d13/d22/d2d/d3f/d58/d86/l90 0 2024-03-23T11:47:51.676 INFO:tasks.workunit.client.0.smithi063.stdout:1/643: chown d3/d3f/d4d/f8c 1046777432 1 2024-03-23T11:47:51.677 INFO:tasks.workunit.client.1.smithi111.stdout:9/983: mknod d0/d31/d41/d9d/dd4/c13d 0 2024-03-23T11:47:51.679 INFO:tasks.workunit.client.1.smithi111.stdout:9/984: fsync d0/d4/f83 0 2024-03-23T11:47:51.682 INFO:tasks.workunit.client.1.smithi111.stdout:0/958: truncate d5/db/d1f/d19/f53 1004294 0 2024-03-23T11:47:51.684 INFO:tasks.workunit.client.0.smithi063.stdout:8/973: mkdir d6/d1f/d54/d5c/d66/da4/d113/d149 0 2024-03-23T11:47:51.684 INFO:tasks.workunit.client.0.smithi063.stdout:0/851: symlink d2/d12/d32/d6d/d124/d28/d46/l12d 0 2024-03-23T11:47:51.685 INFO:tasks.workunit.client.1.smithi111.stdout:7/938: rename d14/d1b/d1e/d2c/d2e/d95 to d14/d1b/d1e/d116 0 2024-03-23T11:47:51.685 INFO:tasks.workunit.client.1.smithi111.stdout:1/943: dread d0/d6/d48/f5d [0,4194304] 0 2024-03-23T11:47:51.686 INFO:tasks.workunit.client.1.smithi111.stdout:7/939: chown d14/d44/d70/d7e 51923 1 2024-03-23T11:47:51.688 INFO:tasks.workunit.client.1.smithi111.stdout:0/959: fdatasync d5/db/f26 0 2024-03-23T11:47:51.688 INFO:tasks.workunit.client.0.smithi063.stdout:3/990: chown d13/l16 24353 1 2024-03-23T11:47:51.692 INFO:tasks.workunit.client.1.smithi111.stdout:0/960: dread d5/f110 [0,4194304] 0 2024-03-23T11:47:51.693 INFO:tasks.workunit.client.0.smithi063.stdout:8/974: chown d6/d1f/d54/d5c/d76/d85/l9f 49230743 1 2024-03-23T11:47:51.694 INFO:tasks.workunit.client.1.smithi111.stdout:1/944: mkdir d0/def/d141 0 2024-03-23T11:47:51.700 INFO:tasks.workunit.client.1.smithi111.stdout:8/836: write d5/d26/d2f/d62/d7e/dae/fcb [267491,91028] 0 2024-03-23T11:47:51.702 INFO:tasks.workunit.client.0.smithi063.stdout:8/975: dread - d6/d1f/d54/d5c/d66/df1/f11b zero size 2024-03-23T11:47:51.704 INFO:tasks.workunit.client.1.smithi111.stdout:9/985: rename d0/d4/d1c/caf to d0/d4/d1c/d36/d28/df3/c13e 0 2024-03-23T11:47:51.706 INFO:tasks.workunit.client.0.smithi063.stdout:7/653: write d5/d6/f25 [1992556,103013] 0 2024-03-23T11:47:51.706 INFO:tasks.workunit.client.0.smithi063.stdout:2/562: dwrite d2/d1c/f5f [0,4194304] 0 2024-03-23T11:47:51.707 INFO:tasks.workunit.client.0.smithi063.stdout:0/852: symlink d2/d12/l12e 0 2024-03-23T11:47:51.709 INFO:tasks.workunit.client.0.smithi063.stdout:4/711: dread d5/da/d20/d4b/d16/d37/fbe [0,4194304] 0 2024-03-23T11:47:51.712 INFO:tasks.workunit.client.0.smithi063.stdout:3/991: fdatasync d13/d22/d2d/d3f/d58/d97/fea 0 2024-03-23T11:47:51.715 INFO:tasks.workunit.client.1.smithi111.stdout:7/940: rename d14/d1b/d1e/d2c/d2e/d33/d98/dfd to d14/d1b/d1e/d2c/d2e/d73/dae/d74/d117 0 2024-03-23T11:47:51.720 INFO:tasks.workunit.client.0.smithi063.stdout:7/654: dread - d5/d6/d19/f92 zero size 2024-03-23T11:47:51.721 INFO:tasks.workunit.client.0.smithi063.stdout:9/691: link d4/d31/d43/da9/cf3 d4/d31/d43/da9/da6/dbe/cf5 0 2024-03-23T11:47:51.721 INFO:tasks.workunit.client.1.smithi111.stdout:8/837: dread d5/d26/d31/d3b/f70 [0,4194304] 0 2024-03-23T11:47:51.723 INFO:tasks.workunit.client.1.smithi111.stdout:1/945: dwrite d0/d2b/d69/f10a [0,4194304] 0 2024-03-23T11:47:51.726 INFO:tasks.workunit.client.0.smithi063.stdout:5/474: dwrite d2/d7/d8/d1d/f88 [0,4194304] 0 2024-03-23T11:47:51.727 INFO:tasks.workunit.client.1.smithi111.stdout:1/946: chown d0/d6/d48/d4a 53 1 2024-03-23T11:47:51.729 INFO:tasks.workunit.client.0.smithi063.stdout:6/559: dwrite d1/dc/d3a/f45 [0,4194304] 0 2024-03-23T11:47:51.729 INFO:tasks.workunit.client.0.smithi063.stdout:8/976: truncate d6/d1c/f7b 568084 0 2024-03-23T11:47:51.733 INFO:tasks.workunit.client.1.smithi111.stdout:8/838: mkdir d5/d32/d7d/d112 0 2024-03-23T11:47:51.739 INFO:tasks.workunit.client.0.smithi063.stdout:7/655: mkdir d5/d6/dca 0 2024-03-23T11:47:51.739 INFO:tasks.workunit.client.0.smithi063.stdout:1/644: dwrite d3/f7 [0,4194304] 0 2024-03-23T11:47:51.755 INFO:tasks.workunit.client.1.smithi111.stdout:7/941: write d14/d37/d43/db1/fd7 [63781,62433] 0 2024-03-23T11:47:51.757 INFO:tasks.workunit.client.1.smithi111.stdout:8/839: getdents d5/d26/d2f/d62/db5 0 2024-03-23T11:47:51.758 INFO:tasks.workunit.client.1.smithi111.stdout:7/942: read - d14/d1b/d1e/d2c/d2e/d73/dae/dea/f114 zero size 2024-03-23T11:47:51.759 INFO:tasks.workunit.client.0.smithi063.stdout:7/656: getdents d5/d26/db7/dc6/d6e/da2/dc9/dc8 0 2024-03-23T11:47:51.762 INFO:tasks.workunit.client.1.smithi111.stdout:8/840: creat d5/d26/d2f/d62/d75/d4b/d51/dd1/f113 x:0 0 0 2024-03-23T11:47:51.763 INFO:tasks.workunit.client.0.smithi063.stdout:6/560: link d1/dc/d3a/d82/da2/d41/d42/f9d d1/dc/d21/fa9 0 2024-03-23T11:47:51.763 INFO:tasks.workunit.client.0.smithi063.stdout:2/563: dread d2/de/d22/f78 [0,4194304] 0 2024-03-23T11:47:51.764 INFO:tasks.workunit.client.0.smithi063.stdout:7/657: creat d5/d6/dca/fcb x:0 0 0 2024-03-23T11:47:51.768 INFO:tasks.workunit.client.1.smithi111.stdout:8/841: read d5/d26/d31/d3b/d55/ff1 [515172,50294] 0 2024-03-23T11:47:51.771 INFO:tasks.workunit.client.0.smithi063.stdout:6/561: truncate d1/f3b 1454245 0 2024-03-23T11:47:51.776 INFO:tasks.workunit.client.0.smithi063.stdout:7/658: read d5/d5c/d2f/d73/da/f89 [3410958,53984] 0 2024-03-23T11:47:51.776 INFO:tasks.workunit.client.1.smithi111.stdout:6/960: sync 2024-03-23T11:47:51.779 INFO:tasks.workunit.client.0.smithi063.stdout:8/977: dwrite d6/d1c/d32/d37/d10a/dae/feb [0,4194304] 0 2024-03-23T11:47:51.783 INFO:tasks.workunit.client.0.smithi063.stdout:7/659: write d5/d26/f72 [3009280,11228] 0 2024-03-23T11:47:51.784 INFO:tasks.workunit.client.0.smithi063.stdout:0/853: dread d2/d14/f66 [0,4194304] 0 2024-03-23T11:47:51.791 INFO:tasks.workunit.client.1.smithi111.stdout:7/943: unlink d14/d37/cd6 0 2024-03-23T11:47:51.797 INFO:tasks.workunit.client.0.smithi063.stdout:8/978: symlink d6/d1c/d32/l14a 0 2024-03-23T11:47:51.798 INFO:tasks.workunit.client.0.smithi063.stdout:5/475: write d2/d2f/f37 [105963,4074] 0 2024-03-23T11:47:51.799 INFO:tasks.workunit.client.0.smithi063.stdout:7/660: mkdir d5/d22/dcc 0 2024-03-23T11:47:51.800 INFO:tasks.workunit.client.0.smithi063.stdout:9/692: write d4/d31/d70/d8d/d90/ff1 [1038688,26879] 0 2024-03-23T11:47:51.801 INFO:tasks.workunit.client.0.smithi063.stdout:3/992: dwrite d13/d15/d114/f11b [0,4194304] 0 2024-03-23T11:47:51.805 INFO:tasks.workunit.client.1.smithi111.stdout:8/842: write d5/d26/d31/d3b/f6e [4764536,93637] 0 2024-03-23T11:47:51.807 INFO:tasks.workunit.client.1.smithi111.stdout:8/843: fdatasync d5/d32/ff0 0 2024-03-23T11:47:51.810 INFO:tasks.workunit.client.0.smithi063.stdout:0/854: dread d2/d12/d32/d6d/d124/d28/fdf [0,4194304] 0 2024-03-23T11:47:51.813 INFO:tasks.workunit.client.1.smithi111.stdout:8/844: mkdir d5/d26/d2f/d62/d7e/dae/d114 0 2024-03-23T11:47:51.813 INFO:tasks.workunit.client.0.smithi063.stdout:3/993: chown d13/d22/d2d/d3f/d58/d6e/d8d/l110 21 1 2024-03-23T11:47:51.815 INFO:tasks.workunit.client.0.smithi063.stdout:7/661: fdatasync d5/d26/db7/dc6/d6e/da2/dc9/d64/fb2 0 2024-03-23T11:47:51.816 INFO:tasks.workunit.client.1.smithi111.stdout:0/961: write d5/db/d1f/d19/d45/fad [30782,63959] 0 2024-03-23T11:47:51.819 INFO:tasks.workunit.client.0.smithi063.stdout:4/712: dwrite d5/da/d20/d4b/d16/d50/d8e/f97 [0,4194304] 0 2024-03-23T11:47:51.820 INFO:tasks.workunit.client.1.smithi111.stdout:1/947: dread d0/d2b/d69/fbc [0,4194304] 0 2024-03-23T11:47:51.822 INFO:tasks.workunit.client.1.smithi111.stdout:7/944: mkdir d14/d1b/d1e/d2c/d42/d5e/d88/d5f/d118 0 2024-03-23T11:47:51.824 INFO:tasks.workunit.client.0.smithi063.stdout:0/855: stat d2/d12/d32/d6d/d124/d28/d46/le9 0 2024-03-23T11:47:51.825 INFO:tasks.workunit.client.1.smithi111.stdout:8/845: creat d5/d32/d7d/f115 x:0 0 0 2024-03-23T11:47:51.828 INFO:tasks.workunit.client.1.smithi111.stdout:6/961: link d2/d35/f40 d2/d1f/d45/f136 0 2024-03-23T11:47:51.834 INFO:tasks.workunit.client.0.smithi063.stdout:2/564: dwrite d2/d23/d2e/d87/d4d/f6d [0,4194304] 0 2024-03-23T11:47:51.840 INFO:tasks.workunit.client.1.smithi111.stdout:9/986: dwrite d0/d4/d1c/f45 [0,4194304] 0 2024-03-23T11:47:51.841 INFO:tasks.workunit.client.0.smithi063.stdout:3/994: symlink d13/d22/d2d/d3f/d58/d97/dac/l13e 0 2024-03-23T11:47:51.846 INFO:tasks.workunit.client.0.smithi063.stdout:2/565: write d2/de/d22/f5b [4159996,46287] 0 2024-03-23T11:47:51.849 INFO:tasks.workunit.client.1.smithi111.stdout:6/962: mkdir d2/d1f/d45/d86/d137 0 2024-03-23T11:47:51.852 INFO:tasks.workunit.client.0.smithi063.stdout:7/662: unlink d5/d6/d19/d1f/f6d 0 2024-03-23T11:47:51.855 INFO:tasks.workunit.client.0.smithi063.stdout:9/693: truncate d4/d31/d43/da9/da6/f45 435262 0 2024-03-23T11:47:51.858 INFO:tasks.workunit.client.0.smithi063.stdout:0/856: symlink d2/d6e/l12f 0 2024-03-23T11:47:51.858 INFO:tasks.workunit.client.0.smithi063.stdout:9/694: stat d4/d4c/d44/c4b 0 2024-03-23T11:47:51.862 INFO:tasks.workunit.client.0.smithi063.stdout:2/566: rmdir d2/de/d22 39 2024-03-23T11:47:51.869 INFO:tasks.workunit.client.0.smithi063.stdout:9/695: mkdir d4/d31/d70/d8d/df6 0 2024-03-23T11:47:51.871 INFO:tasks.workunit.client.0.smithi063.stdout:7/663: rename d5/d22/f9d to d5/d26/db7/fcd 0 2024-03-23T11:47:51.873 INFO:tasks.workunit.client.0.smithi063.stdout:6/562: dread d1/dc/d21/d49/f76 [0,4194304] 0 2024-03-23T11:47:51.873 INFO:tasks.workunit.client.0.smithi063.stdout:5/476: write d2/d7/f1a [445629,126797] 0 2024-03-23T11:47:51.877 INFO:tasks.workunit.client.0.smithi063.stdout:8/979: dread d6/dd/d59/d13d/d5d/d7d/db4/f10e [0,4194304] 0 2024-03-23T11:47:51.878 INFO:tasks.workunit.client.0.smithi063.stdout:2/567: symlink d2/de/d7a/d8d/lb3 0 2024-03-23T11:47:51.879 INFO:tasks.workunit.client.1.smithi111.stdout:6/963: mknod d2/d35/d70/c138 0 2024-03-23T11:47:51.881 INFO:tasks.workunit.client.0.smithi063.stdout:9/696: fdatasync d4/d31/d43/da9/da6/dbe/fd2 0 2024-03-23T11:47:51.883 INFO:tasks.workunit.client.1.smithi111.stdout:9/987: mkdir d0/d4/d1c/d2d/d75/d7a/d13f 0 2024-03-23T11:47:51.884 INFO:tasks.workunit.client.1.smithi111.stdout:1/948: link d0/d6/d48/d4a/d6c/d83/d11d/d91/fa1 d0/d6/d48/d4a/d6c/d83/d11d/f142 0 2024-03-23T11:47:51.886 INFO:tasks.workunit.client.0.smithi063.stdout:6/563: truncate d1/dc/d3a/f61 1773446 0 2024-03-23T11:47:51.886 INFO:tasks.workunit.client.0.smithi063.stdout:9/697: read - d4/d31/d9c/d9e/fea zero size 2024-03-23T11:47:51.889 INFO:tasks.workunit.client.1.smithi111.stdout:9/988: dread d0/d31/da0/f125 [0,4194304] 0 2024-03-23T11:47:51.890 INFO:tasks.workunit.client.0.smithi063.stdout:8/980: rmdir d6/dd/d59/d13d/d15 39 2024-03-23T11:47:51.892 INFO:tasks.workunit.client.0.smithi063.stdout:0/857: dwrite d2/d6e/f11e [0,4194304] 0 2024-03-23T11:47:51.897 INFO:tasks.workunit.client.0.smithi063.stdout:0/858: fsync d2/d14/d36/d77/d94/daf/ffb 0 2024-03-23T11:47:51.903 INFO:tasks.workunit.client.1.smithi111.stdout:1/949: fdatasync d0/d2b/fc7 0 2024-03-23T11:47:51.906 INFO:tasks.workunit.client.0.smithi063.stdout:3/995: write d13/d22/d2d/d3f/d67/f84 [3676300,17594] 0 2024-03-23T11:47:51.909 INFO:tasks.workunit.client.0.smithi063.stdout:0/859: fdatasync d2/f56 0 2024-03-23T11:47:51.911 INFO:tasks.workunit.client.1.smithi111.stdout:9/989: creat d0/d31/d41/df4/d103/f140 x:0 0 0 2024-03-23T11:47:51.913 INFO:tasks.workunit.client.0.smithi063.stdout:6/564: creat d1/dc/d3a/d57/d77/d79/faa x:0 0 0 2024-03-23T11:47:51.914 INFO:tasks.workunit.client.0.smithi063.stdout:4/713: dwrite d5/da/d20/d4b/d1f/f21 [8388608,4194304] 0 2024-03-23T11:47:51.917 INFO:tasks.workunit.client.1.smithi111.stdout:8/846: dread d5/d26/d2f/d62/d7e/fa7 [0,4194304] 0 2024-03-23T11:47:51.921 INFO:tasks.workunit.client.1.smithi111.stdout:7/945: mknod d14/d1b/d1e/d2c/d42/d5e/d88/c119 0 2024-03-23T11:47:51.922 INFO:tasks.workunit.client.0.smithi063.stdout:2/568: write d2/d23/d2e/d52/f71 [109428,129264] 0 2024-03-23T11:47:51.923 INFO:tasks.workunit.client.0.smithi063.stdout:8/981: truncate d6/dd/d59/d13d/f49 1877665 0 2024-03-23T11:47:51.923 INFO:tasks.workunit.client.1.smithi111.stdout:0/962: truncate d5/db/d1f/d19/dbd/f94 501750 0 2024-03-23T11:47:51.929 INFO:tasks.workunit.client.1.smithi111.stdout:7/946: mkdir d14/d1b/d1e/d2c/d42/dc1/deb/d103/d11a 0 2024-03-23T11:47:51.934 INFO:tasks.workunit.client.1.smithi111.stdout:0/963: read d5/db/d1f/d47/d6f/ffa [609432,90757] 0 2024-03-23T11:47:51.936 INFO:tasks.workunit.client.0.smithi063.stdout:9/698: dwrite d4/d31/d70/d8d/fbb [0,4194304] 0 2024-03-23T11:47:51.938 INFO:tasks.workunit.client.1.smithi111.stdout:6/964: dwrite d2/d17/f60 [0,4194304] 0 2024-03-23T11:47:51.939 INFO:tasks.workunit.client.0.smithi063.stdout:2/569: rename d2/d23/d2e/d52/d99 to d2/d46/db4 0 2024-03-23T11:47:51.942 INFO:tasks.workunit.client.1.smithi111.stdout:7/947: rename d14/d1b/d1e/d2c/d2e/d33/l64 to d14/d1b/d1e/d2c/d42/d5e/l11b 0 2024-03-23T11:47:51.942 INFO:tasks.workunit.client.0.smithi063.stdout:4/714: getdents d5/da/d20/d4b/d16/d50/d8e/d96/dc0 0 2024-03-23T11:47:51.946 INFO:tasks.workunit.client.0.smithi063.stdout:8/982: fdatasync d6/dd/d59/d13d/f49 0 2024-03-23T11:47:51.946 INFO:tasks.workunit.client.1.smithi111.stdout:8/847: dread d5/d26/d2f/d62/d75/d4b/d51/d5a/f36 [0,4194304] 0 2024-03-23T11:47:51.948 INFO:tasks.workunit.client.0.smithi063.stdout:4/715: readlink d5/da/d20/d4b/d16/d32/l4a 0 2024-03-23T11:47:51.948 INFO:tasks.workunit.client.0.smithi063.stdout:9/699: creat d4/d31/d43/d49/d8e/ff7 x:0 0 0 2024-03-23T11:47:51.948 INFO:tasks.workunit.client.1.smithi111.stdout:0/964: getdents d5/db/d1f/d47/d59 0 2024-03-23T11:47:51.949 INFO:tasks.workunit.client.1.smithi111.stdout:6/965: rename d2/d1f/d45/f136 to d2/d73/f139 0 2024-03-23T11:47:51.953 INFO:tasks.workunit.client.0.smithi063.stdout:9/700: chown d4/d4c/f41 1788494 1 2024-03-23T11:47:51.954 INFO:tasks.workunit.client.1.smithi111.stdout:6/966: readlink d2/d1f/d45/d9a/ld0 0 2024-03-23T11:47:51.955 INFO:tasks.workunit.client.1.smithi111.stdout:6/967: dread - d2/fa2 zero size 2024-03-23T11:47:51.956 INFO:tasks.workunit.client.1.smithi111.stdout:0/965: link d5/db/d1f/de4/f112 d5/f144 0 2024-03-23T11:47:51.957 INFO:tasks.workunit.client.0.smithi063.stdout:6/565: link d1/dc/d3a/d82/da2/f83 d1/dc/fab 0 2024-03-23T11:47:51.957 INFO:tasks.workunit.client.0.smithi063.stdout:0/860: dwrite d2/d12/d32/d6d/d124/d1f/fba [0,4194304] 0 2024-03-23T11:47:51.959 INFO:tasks.workunit.client.1.smithi111.stdout:0/966: readlink d5/db/l33 0 2024-03-23T11:47:51.960 INFO:tasks.workunit.client.1.smithi111.stdout:8/848: mknod d5/d26/d2f/d42/c116 0 2024-03-23T11:47:51.962 INFO:tasks.workunit.client.0.smithi063.stdout:4/716: creat d5/da/dc3/fcb x:0 0 0 2024-03-23T11:47:51.964 INFO:tasks.workunit.client.1.smithi111.stdout:0/967: rmdir d5/db/d4d/da6/d10b/d122 39 2024-03-23T11:47:51.965 INFO:tasks.workunit.client.1.smithi111.stdout:7/948: dwrite d14/d37/d54/fe4 [8388608,4194304] 0 2024-03-23T11:47:51.967 INFO:tasks.workunit.client.1.smithi111.stdout:9/990: dread d0/d4/d10/d29/fa3 [0,4194304] 0 2024-03-23T11:47:51.968 INFO:tasks.workunit.client.1.smithi111.stdout:8/849: rmdir d5/d26/d2f/d62/d75/d4b/d51/dd1 39 2024-03-23T11:47:51.971 INFO:tasks.workunit.client.0.smithi063.stdout:0/861: truncate d2/d14/f79 1254204 0 2024-03-23T11:47:51.972 INFO:tasks.workunit.client.1.smithi111.stdout:7/949: creat d14/d1b/d1e/d2c/d2e/d73/dae/d74/f11c x:0 0 0 2024-03-23T11:47:51.972 INFO:tasks.workunit.client.1.smithi111.stdout:8/850: write d5/d1b/f3a [3360934,58701] 0 2024-03-23T11:47:51.974 INFO:tasks.workunit.client.1.smithi111.stdout:0/968: rmdir d5/db/d1f/d47/de0/d138 0 2024-03-23T11:47:51.974 INFO:tasks.workunit.client.0.smithi063.stdout:6/566: creat d1/dc/d3a/d57/fac x:0 0 0 2024-03-23T11:47:51.975 INFO:tasks.workunit.client.0.smithi063.stdout:2/570: rename d2/d46/l4e to d2/d46/d54/lb5 0 2024-03-23T11:47:51.976 INFO:tasks.workunit.client.1.smithi111.stdout:7/950: creat d14/d1b/d22/f11d x:0 0 0 2024-03-23T11:47:51.980 INFO:tasks.workunit.client.1.smithi111.stdout:0/969: symlink d5/db/d1f/d47/d59/l145 0 2024-03-23T11:47:51.981 INFO:tasks.workunit.client.0.smithi063.stdout:4/717: getdents d5/da/d20/d4b/d16/d32 0 2024-03-23T11:47:51.981 INFO:tasks.workunit.client.0.smithi063.stdout:0/862: truncate d2/d4/f9 3561885 0 2024-03-23T11:47:51.982 INFO:tasks.workunit.client.1.smithi111.stdout:6/968: dwrite d2/d1f/d45/d49/fa1 [4194304,4194304] 0 2024-03-23T11:47:51.983 INFO:tasks.workunit.client.0.smithi063.stdout:9/701: truncate d4/ff 4391800 0 2024-03-23T11:47:51.985 INFO:tasks.workunit.client.0.smithi063.stdout:2/571: truncate d2/d23/d2e/d52/f4c 2984784 0 2024-03-23T11:47:51.986 INFO:tasks.workunit.client.1.smithi111.stdout:7/951: read d14/d1b/d1e/d2c/d42/f45 [1990534,35079] 0 2024-03-23T11:47:51.986 INFO:tasks.workunit.client.0.smithi063.stdout:4/718: chown d5/da/d20/d4b/d16/c5b 78 1 2024-03-23T11:47:51.987 INFO:tasks.workunit.client.0.smithi063.stdout:2/572: chown d2/d1c/f5f 3015 1 2024-03-23T11:47:51.987 INFO:tasks.workunit.client.1.smithi111.stdout:0/970: write d5/db/d1f/f12c [886397,18809] 0 2024-03-23T11:47:51.992 INFO:tasks.workunit.client.0.smithi063.stdout:9/702: symlink d4/d31/d43/d49/d8e/lf8 0 2024-03-23T11:47:51.993 INFO:tasks.workunit.client.1.smithi111.stdout:7/952: rmdir d14/d44/d4b 39 2024-03-23T11:47:51.993 INFO:tasks.workunit.client.1.smithi111.stdout:9/991: dwrite d0/d31/d41/d71/d8f/fa9 [0,4194304] 0 2024-03-23T11:47:51.996 INFO:tasks.workunit.client.0.smithi063.stdout:4/719: fsync d5/da/d20/f48 0 2024-03-23T11:47:51.996 INFO:tasks.workunit.client.0.smithi063.stdout:9/703: stat d4/d4c/d19/l55 0 2024-03-23T11:47:51.998 INFO:tasks.workunit.client.1.smithi111.stdout:6/969: creat d2/d1a/d1c/ddf/d116/f13a x:0 0 0 2024-03-23T11:47:51.999 INFO:tasks.workunit.client.0.smithi063.stdout:8/983: dwrite d6/dd/df/f14 [0,4194304] 0 2024-03-23T11:47:51.999 INFO:tasks.workunit.client.1.smithi111.stdout:7/953: mknod d14/d1b/d1e/d2c/d8b/c11e 0 2024-03-23T11:47:52.001 INFO:tasks.workunit.client.1.smithi111.stdout:6/970: dread - d2/d35/f133 zero size 2024-03-23T11:47:52.003 INFO:tasks.workunit.client.1.smithi111.stdout:7/954: read - d14/d1b/d1e/d2c/d2e/d33/d98/f10c zero size 2024-03-23T11:47:52.009 INFO:tasks.workunit.client.1.smithi111.stdout:6/971: mkdir d2/d17/d13b 0 2024-03-23T11:47:52.009 INFO:tasks.workunit.client.1.smithi111.stdout:0/971: creat d5/db/d4d/da6/d10b/f146 x:0 0 0 2024-03-23T11:47:52.010 INFO:tasks.workunit.client.0.smithi063.stdout:0/863: dwrite d2/d14/d36/ff4 [0,4194304] 0 2024-03-23T11:47:52.015 INFO:tasks.workunit.client.0.smithi063.stdout:9/704: rename d4/d31/d70/d8d/fdc to d4/d4c/d44/ff9 0 2024-03-23T11:47:52.016 INFO:tasks.workunit.client.0.smithi063.stdout:6/567: write d1/f12 [856206,73509] 0 2024-03-23T11:47:52.018 INFO:tasks.workunit.client.1.smithi111.stdout:6/972: mkdir d2/d17/d13c 0 2024-03-23T11:47:52.020 INFO:tasks.workunit.client.1.smithi111.stdout:9/992: dwrite d0/f127 [0,4194304] 0 2024-03-23T11:47:52.020 INFO:tasks.workunit.client.1.smithi111.stdout:1/950: sync 2024-03-23T11:47:52.021 INFO:tasks.workunit.client.0.smithi063.stdout:2/573: unlink d2/de/d22/f5b 0 2024-03-23T11:47:52.025 INFO:tasks.workunit.client.1.smithi111.stdout:9/993: chown d0/d31/d35/l62 7288765 1 2024-03-23T11:47:52.026 INFO:tasks.workunit.client.1.smithi111.stdout:1/951: symlink d0/def/l143 0 2024-03-23T11:47:52.027 INFO:tasks.workunit.client.0.smithi063.stdout:6/568: truncate d1/f9e 522982 0 2024-03-23T11:47:52.027 INFO:tasks.workunit.client.0.smithi063.stdout:0/864: rename d2/d12/d32/d6d/f74 to d2/d14/d36/d77/d94/daf/f130 0 2024-03-23T11:47:52.028 INFO:tasks.workunit.client.1.smithi111.stdout:8/851: dwrite d5/d26/d2f/d62/d75/d4b/d51/d5a/f48 [0,4194304] 0 2024-03-23T11:47:52.029 INFO:tasks.workunit.client.0.smithi063.stdout:6/569: read - d1/dc/d3a/d57/d6d/fa0 zero size 2024-03-23T11:47:52.031 INFO:tasks.workunit.client.0.smithi063.stdout:2/574: dread - d2/d23/d2e/d87/f80 zero size 2024-03-23T11:47:52.032 INFO:tasks.workunit.client.0.smithi063.stdout:0/865: mknod d2/d6e/d8c/c131 0 2024-03-23T11:47:52.034 INFO:tasks.workunit.client.0.smithi063.stdout:9/705: fdatasync d4/ff 0 2024-03-23T11:47:52.037 INFO:tasks.workunit.client.1.smithi111.stdout:8/852: mknod d5/d26/d2f/d62/db5/c117 0 2024-03-23T11:47:52.038 INFO:tasks.workunit.client.0.smithi063.stdout:6/570: unlink d1/dc/d3a/d82/da2/d41/d42/f6f 0 2024-03-23T11:47:52.038 INFO:tasks.workunit.client.0.smithi063.stdout:0/866: rmdir d2/d48/da5 39 2024-03-23T11:47:52.041 INFO:tasks.workunit.client.1.smithi111.stdout:8/853: readlink d5/d26/d31/d4d/l9a 0 2024-03-23T11:47:52.045 INFO:tasks.workunit.client.0.smithi063.stdout:9/706: truncate d4/d4c/d19/f42 7855979 0 2024-03-23T11:47:52.047 INFO:tasks.workunit.client.1.smithi111.stdout:6/973: dread d2/d1f/f92 [0,4194304] 0 2024-03-23T11:47:52.047 INFO:tasks.workunit.client.0.smithi063.stdout:6/571: creat d1/dc/d3a/d57/d77/fad x:0 0 0 2024-03-23T11:47:52.048 INFO:tasks.workunit.client.1.smithi111.stdout:9/994: dread d0/d31/d41/d9d/dd4/ff7 [0,4194304] 0 2024-03-23T11:47:52.052 INFO:tasks.workunit.client.1.smithi111.stdout:8/854: creat d5/d26/d2f/d62/db5/dcc/d105/f118 x:0 0 0 2024-03-23T11:47:52.053 INFO:tasks.workunit.client.1.smithi111.stdout:9/995: dread - d0/d31/d41/d9d/dd4/f10a zero size 2024-03-23T11:47:52.053 INFO:tasks.workunit.client.0.smithi063.stdout:9/707: read - d4/d31/d43/d49/d8e/ff7 zero size 2024-03-23T11:47:52.054 INFO:tasks.workunit.client.0.smithi063.stdout:4/720: dwrite d5/f90 [0,4194304] 0 2024-03-23T11:47:52.054 INFO:tasks.workunit.client.1.smithi111.stdout:6/974: rename d2/d17/l38 to d2/d1f/d45/d86/l13d 0 2024-03-23T11:47:52.055 INFO:tasks.workunit.client.1.smithi111.stdout:8/855: write d5/d26/d31/d3b/d55/ff1 [1679150,39038] 0 2024-03-23T11:47:52.059 INFO:tasks.workunit.client.0.smithi063.stdout:8/984: dwrite d6/d1f/d54/d5c/d66/df1/f107 [0,4194304] 0 2024-03-23T11:47:52.060 INFO:tasks.workunit.client.0.smithi063.stdout:2/575: rename d2/d1c/l39 to d2/d46/db4/lb6 0 2024-03-23T11:47:52.062 INFO:tasks.workunit.client.0.smithi063.stdout:9/708: truncate d4/d4c/d19/f27 1233086 0 2024-03-23T11:47:52.066 INFO:tasks.workunit.client.1.smithi111.stdout:9/996: symlink d0/d4/d6a/d9a/l141 0 2024-03-23T11:47:52.068 INFO:tasks.workunit.client.1.smithi111.stdout:7/955: dread d14/d1b/d1e/d86/fbe [0,4194304] 0 2024-03-23T11:47:52.071 INFO:tasks.workunit.client.1.smithi111.stdout:6/975: unlink d2/d35/d70/c75 0 2024-03-23T11:47:52.073 INFO:tasks.workunit.client.0.smithi063.stdout:9/709: mkdir d4/d31/d43/d49/dfa 0 2024-03-23T11:47:52.074 INFO:tasks.workunit.client.0.smithi063.stdout:9/710: truncate d4/fae 366202 0 2024-03-23T11:47:52.078 INFO:tasks.workunit.client.0.smithi063.stdout:4/721: rename d5/da/d20/d4b/d29/d38/d93/fbc to d5/da/d20/d4b/d16/d50/d8e/dad/fcc 0 2024-03-23T11:47:52.080 INFO:tasks.workunit.client.1.smithi111.stdout:8/856: dwrite d5/d32/d7d/f115 [0,4194304] 0 2024-03-23T11:47:52.081 INFO:tasks.workunit.client.1.smithi111.stdout:0/972: dread d5/db/d1f/f1a [0,4194304] 0 2024-03-23T11:47:52.082 INFO:tasks.workunit.client.1.smithi111.stdout:8/857: fsync d5/f11 0 2024-03-23T11:47:52.083 INFO:tasks.workunit.client.1.smithi111.stdout:9/997: mknod d0/d4/d1c/d2d/d75/d7a/c142 0 2024-03-23T11:47:52.084 INFO:tasks.workunit.client.1.smithi111.stdout:7/956: symlink d14/d1b/d1e/d2c/d42/dc1/deb/d103/l11f 0 2024-03-23T11:47:52.088 INFO:tasks.workunit.client.1.smithi111.stdout:6/976: truncate d2/d1a/f72 1249062 0 2024-03-23T11:47:52.089 INFO:tasks.workunit.client.1.smithi111.stdout:6/977: read d2/f28 [743711,53789] 0 2024-03-23T11:47:52.090 INFO:tasks.workunit.client.0.smithi063.stdout:4/722: creat d5/da/dc3/fcd x:0 0 0 2024-03-23T11:47:52.090 INFO:tasks.workunit.client.0.smithi063.stdout:9/711: truncate d4/d31/d43/d49/d8e/f93 5032223 0 2024-03-23T11:47:52.091 INFO:tasks.workunit.client.1.smithi111.stdout:6/978: dread - d2/d80/d100/f129 zero size 2024-03-23T11:47:52.092 INFO:tasks.workunit.client.1.smithi111.stdout:0/973: symlink d5/db/d1f/d19/dbd/d8b/dd0/l147 0 2024-03-23T11:47:52.093 INFO:tasks.workunit.client.1.smithi111.stdout:6/979: fdatasync d2/d1a/d106/d108/fca 0 2024-03-23T11:47:52.094 INFO:tasks.workunit.client.1.smithi111.stdout:0/974: chown d5/db/d1f/d48/f58 1314367 1 2024-03-23T11:47:52.095 INFO:tasks.workunit.client.1.smithi111.stdout:7/957: fsync d14/d44/d70/d7e/dbc/fc7 0 2024-03-23T11:47:52.103 INFO:tasks.workunit.client.1.smithi111.stdout:1/952: write d0/d6/d48/d4a/d6c/d83/d11d/d1e/d63/ffa [65532,22549] 0 2024-03-23T11:47:52.103 INFO:tasks.workunit.client.1.smithi111.stdout:9/998: creat d0/d4/d1c/d2d/d75/d7a/d13f/f143 x:0 0 0 2024-03-23T11:47:52.105 INFO:tasks.workunit.client.0.smithi063.stdout:4/723: mknod d5/da/d20/d4b/d16/d50/d86/cce 0 2024-03-23T11:47:52.106 INFO:tasks.workunit.client.0.smithi063.stdout:9/712: fsync d4/d4c/f69 0 2024-03-23T11:47:52.106 INFO:tasks.workunit.client.1.smithi111.stdout:7/958: write d14/d1b/d1e/d2c/d2e/d73/dae/d74/f11c [193567,44105] 0 2024-03-23T11:47:52.109 INFO:tasks.workunit.client.1.smithi111.stdout:0/975: read d5/db/d1f/d47/d87/f117 [3893481,93945] 0 2024-03-23T11:47:52.112 INFO:tasks.workunit.client.0.smithi063.stdout:9/713: dread - d4/d31/d9c/d9e/fea zero size 2024-03-23T11:47:52.115 INFO:tasks.workunit.client.1.smithi111.stdout:1/953: dread - d0/f13 zero size 2024-03-23T11:47:52.116 INFO:tasks.workunit.client.1.smithi111.stdout:9/999: write d0/d4/d10/d29/fa3 [1861904,128354] 0 2024-03-23T11:47:52.117 INFO:tasks.workunit.client.1.smithi111.stdout:8/858: getdents d5/d32/d9b 0 2024-03-23T11:47:52.118 INFO:tasks.workunit.client.0.smithi063.stdout:2/576: dwrite d2/f25 [0,4194304] 0 2024-03-23T11:47:52.119 INFO:tasks.workunit.client.1.smithi111.stdout:0/976: unlink d5/db/d1f/d19/dbd/d64/dc7/d100/fd3 0 2024-03-23T11:47:52.121 INFO:tasks.workunit.client.0.smithi063.stdout:8/985: dwrite d6/d1c/d32/d37/fc0 [0,4194304] 0 2024-03-23T11:47:52.121 INFO:tasks.workunit.client.1.smithi111.stdout:8/859: write d5/d26/d31/d3b/f80 [6357604,89631] 0 2024-03-23T11:47:52.122 INFO:tasks.workunit.client.0.smithi063.stdout:9/714: unlink d4/d31/d43/da9/cd3 0 2024-03-23T11:47:52.123 INFO:tasks.workunit.client.0.smithi063.stdout:9/715: stat d4/f5a 0 2024-03-23T11:47:52.126 INFO:tasks.workunit.client.1.smithi111.stdout:1/954: unlink d0/d6/cb 0 2024-03-23T11:47:52.129 INFO:tasks.workunit.client.1.smithi111.stdout:8/860: rename d5/d26/d2f/d62/d75/d8d/la1 to d5/d26/d2f/d62/db5/dcc/l119 0 2024-03-23T11:47:52.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:51 smithi063 ceph-mon[35189]: pgmap v45: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 65 MiB/s rd, 153 MiB/s wr, 363 op/s 2024-03-23T11:47:52.136 INFO:tasks.workunit.client.1.smithi111.stdout:0/977: unlink d5/db/d1f/d19/dbd/d8b/fa8 0 2024-03-23T11:47:52.137 INFO:tasks.workunit.client.1.smithi111.stdout:1/955: creat d0/d2b/d7c/dca/f144 x:0 0 0 2024-03-23T11:47:52.139 INFO:tasks.workunit.client.1.smithi111.stdout:8/861: creat d5/d26/d31/d5e/f11a x:0 0 0 2024-03-23T11:47:52.141 INFO:tasks.workunit.client.1.smithi111.stdout:0/978: mknod d5/db/d1f/d19/dbd/d8b/c148 0 2024-03-23T11:47:52.143 INFO:tasks.workunit.client.0.smithi063.stdout:9/716: truncate d4/d31/d70/d8d/db3/fe0 688485 0 2024-03-23T11:47:52.145 INFO:tasks.workunit.client.1.smithi111.stdout:8/862: mkdir d5/d32/d7d/d112/d11b 0 2024-03-23T11:47:52.146 INFO:tasks.workunit.client.1.smithi111.stdout:1/956: creat d0/dda/f145 x:0 0 0 2024-03-23T11:47:52.148 INFO:tasks.workunit.client.1.smithi111.stdout:7/959: getdents d14/d37 0 2024-03-23T11:47:52.150 INFO:tasks.workunit.client.0.smithi063.stdout:4/724: dwrite d5/d1e/f82 [0,4194304] 0 2024-03-23T11:47:52.153 INFO:tasks.workunit.client.0.smithi063.stdout:9/717: creat d4/d31/d9c/ffb x:0 0 0 2024-03-23T11:47:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:51 smithi111 ceph-mon[44005]: pgmap v45: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 65 MiB/s rd, 153 MiB/s wr, 363 op/s 2024-03-23T11:47:52.154 INFO:tasks.workunit.client.1.smithi111.stdout:8/863: write d5/d26/d31/d4d/f6c [3085164,30289] 0 2024-03-23T11:47:52.154 INFO:tasks.workunit.client.1.smithi111.stdout:7/960: fsync d14/d1b/d1e/d2c/d2e/d33/fcd 0 2024-03-23T11:47:52.165 INFO:tasks.workunit.client.0.smithi063.stdout:8/986: write d6/dd/d59/d13d/d15/d9d/f117 [477256,40273] 0 2024-03-23T11:47:52.167 INFO:tasks.workunit.client.1.smithi111.stdout:8/864: symlink d5/d1b/dbc/l11c 0 2024-03-23T11:47:52.168 INFO:tasks.workunit.client.1.smithi111.stdout:0/979: write d5/db/d1f/f1a [5046619,95131] 0 2024-03-23T11:47:52.171 INFO:tasks.workunit.client.0.smithi063.stdout:8/987: dread - d6/dd/df/f140 zero size 2024-03-23T11:47:52.172 INFO:tasks.workunit.client.0.smithi063.stdout:4/725: write d5/da/d20/f3c [2616142,130167] 0 2024-03-23T11:47:52.173 INFO:tasks.workunit.client.1.smithi111.stdout:8/865: fdatasync d5/d26/d31/f64 0 2024-03-23T11:47:52.173 INFO:tasks.workunit.client.1.smithi111.stdout:7/961: dwrite d14/d37/d43/db1/fd7 [0,4194304] 0 2024-03-23T11:47:52.177 INFO:tasks.workunit.client.0.smithi063.stdout:2/577: dwrite d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:52.179 INFO:tasks.workunit.client.1.smithi111.stdout:8/866: read d5/d26/d2f/d62/d7e/d85/f96 [1176564,95915] 0 2024-03-23T11:47:52.179 INFO:tasks.workunit.client.1.smithi111.stdout:7/962: fdatasync d14/d1b/d1e/d86/fbe 0 2024-03-23T11:47:52.181 INFO:tasks.workunit.client.1.smithi111.stdout:1/957: dwrite d0/d6/fd8 [0,4194304] 0 2024-03-23T11:47:52.184 INFO:tasks.workunit.client.1.smithi111.stdout:8/867: readlink d5/d26/d2f/d62/d75/la3 0 2024-03-23T11:47:52.184 INFO:tasks.workunit.client.0.smithi063.stdout:4/726: creat d5/da/fcf x:0 0 0 2024-03-23T11:47:52.185 INFO:tasks.workunit.client.0.smithi063.stdout:2/578: readlink d2/d23/d2e/d87/d4d/l58 0 2024-03-23T11:47:52.190 INFO:tasks.workunit.client.0.smithi063.stdout:4/727: write d5/da/d20/d4b/d1f/f21 [7055432,66458] 0 2024-03-23T11:47:52.191 INFO:tasks.workunit.client.0.smithi063.stdout:8/988: dread d6/dd/d59/d13d/d15/d9d/f117 [0,4194304] 0 2024-03-23T11:47:52.191 INFO:tasks.workunit.client.0.smithi063.stdout:2/579: fdatasync d2/d23/d2e/d98/faf 0 2024-03-23T11:47:52.191 INFO:tasks.workunit.client.1.smithi111.stdout:7/963: rmdir d14/d1b/d1e/d2c/d42/d9c 39 2024-03-23T11:47:52.194 INFO:tasks.workunit.client.0.smithi063.stdout:9/718: dread d4/d4c/f41 [0,4194304] 0 2024-03-23T11:47:52.194 INFO:tasks.workunit.client.0.smithi063.stdout:2/580: chown d2/d23/d2e/d52 24106850 1 2024-03-23T11:47:52.196 INFO:tasks.workunit.client.0.smithi063.stdout:9/719: read - d4/fc0 zero size 2024-03-23T11:47:52.197 INFO:tasks.workunit.client.1.smithi111.stdout:1/958: creat d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/d8e/f146 x:0 0 0 2024-03-23T11:47:52.206 INFO:tasks.workunit.client.0.smithi063.stdout:4/728: dread d5/d1e/f82 [0,4194304] 0 2024-03-23T11:47:52.210 INFO:tasks.workunit.client.0.smithi063.stdout:2/581: write d2/d23/d2e/f36 [4563316,110202] 0 2024-03-23T11:47:52.211 INFO:tasks.workunit.client.0.smithi063.stdout:8/989: truncate d6/d1c/f6c 1686049 0 2024-03-23T11:47:52.212 INFO:tasks.workunit.client.1.smithi111.stdout:7/964: dwrite d14/d1b/d1e/d2c/d2e/d73/dae/d74/f11c [0,4194304] 0 2024-03-23T11:47:52.214 INFO:tasks.workunit.client.1.smithi111.stdout:8/868: dwrite d5/d32/d7d/faa [0,4194304] 0 2024-03-23T11:47:52.214 INFO:tasks.workunit.client.1.smithi111.stdout:7/965: fsync d14/d1b/d1e/d2c/d2e/d73/dae/f62 0 2024-03-23T11:47:52.217 INFO:tasks.workunit.client.1.smithi111.stdout:8/869: readlink d5/d26/d2f/d62/d75/d4b/d51/d5a/lf7 0 2024-03-23T11:47:52.218 INFO:tasks.workunit.client.0.smithi063.stdout:8/990: mknod d6/d1f/d54/d5c/d76/d85/c14b 0 2024-03-23T11:47:52.223 INFO:tasks.workunit.client.1.smithi111.stdout:8/870: creat d5/d26/d31/f11d x:0 0 0 2024-03-23T11:47:52.224 INFO:tasks.workunit.client.1.smithi111.stdout:8/871: readlink d5/d26/d31/d3b/d55/le8 0 2024-03-23T11:47:52.227 INFO:tasks.workunit.client.1.smithi111.stdout:8/872: read d5/d26/d31/fe6 [137709,82749] 0 2024-03-23T11:47:52.229 INFO:tasks.workunit.client.0.smithi063.stdout:8/991: creat d6/dd/d59/d13d/d5d/d7d/db4/f14c x:0 0 0 2024-03-23T11:47:52.233 INFO:tasks.workunit.client.0.smithi063.stdout:4/729: dwrite d5/d1e/f82 [0,4194304] 0 2024-03-23T11:47:52.236 INFO:tasks.workunit.client.1.smithi111.stdout:7/966: dwrite d14/d1b/d1e/d2c/d2e/d33/f3d [0,4194304] 0 2024-03-23T11:47:52.238 INFO:tasks.workunit.client.0.smithi063.stdout:4/730: fdatasync d5/d1e/d2b/f58 0 2024-03-23T11:47:52.238 INFO:tasks.workunit.client.1.smithi111.stdout:8/873: dread d5/d26/d2f/d8f/fb7 [0,4194304] 0 2024-03-23T11:47:52.245 INFO:tasks.workunit.client.1.smithi111.stdout:7/967: chown d14/d44/d4b/lb3 13282226 1 2024-03-23T11:47:52.246 INFO:tasks.workunit.client.0.smithi063.stdout:2/582: dwrite d2/d46/d54/fa3 [0,4194304] 0 2024-03-23T11:47:52.250 INFO:tasks.workunit.client.0.smithi063.stdout:9/720: dread d4/d4c/f8c [0,4194304] 0 2024-03-23T11:47:52.251 INFO:tasks.workunit.client.1.smithi111.stdout:7/968: mkdir d14/d1b/d1e/d2c/d2e/d73/dae/d74/d120 0 2024-03-23T11:47:52.253 INFO:tasks.workunit.client.1.smithi111.stdout:8/874: rename d5/d26/d2f/d8f/fb7 to d5/d26/d2f/d62/db5/f11e 0 2024-03-23T11:47:52.254 INFO:tasks.workunit.client.1.smithi111.stdout:7/969: write d14/d37/d43/ff4 [760497,129351] 0 2024-03-23T11:47:52.255 INFO:tasks.workunit.client.1.smithi111.stdout:6/980: sync 2024-03-23T11:47:52.257 INFO:tasks.workunit.client.1.smithi111.stdout:6/981: chown d2/d73/f10f 239600 1 2024-03-23T11:47:52.257 INFO:tasks.workunit.client.0.smithi063.stdout:2/583: chown d2/d23/d2e/d52/f71 3763 1 2024-03-23T11:47:52.259 INFO:tasks.workunit.client.0.smithi063.stdout:4/731: creat d5/da/d72/fd0 x:0 0 0 2024-03-23T11:47:52.265 INFO:tasks.workunit.client.0.smithi063.stdout:8/992: dwrite d6/d1c/d32/d37/fcf [0,4194304] 0 2024-03-23T11:47:52.266 INFO:tasks.workunit.client.1.smithi111.stdout:7/970: rename d14/d1b/d1e/d2c/d2e/d33/d98/la6 to d14/d1b/d1e/d2c/d2e/l121 0 2024-03-23T11:47:52.267 INFO:tasks.workunit.client.0.smithi063.stdout:9/721: write d4/d31/d70/d8d/da0/fe2 [754821,79676] 0 2024-03-23T11:47:52.268 INFO:tasks.workunit.client.1.smithi111.stdout:7/971: chown d14/d1b/d1e/d2c/d42/d5e/d88/d5f/cf8 528043723 1 2024-03-23T11:47:52.271 INFO:tasks.workunit.client.1.smithi111.stdout:7/972: chown d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fcc 7208287 1 2024-03-23T11:47:52.271 INFO:tasks.workunit.client.0.smithi063.stdout:2/584: truncate d2/d23/d56/f57 1903744 0 2024-03-23T11:47:52.272 INFO:tasks.workunit.client.1.smithi111.stdout:7/973: write d14/d44/d70/f10e [723767,760] 0 2024-03-23T11:47:52.274 INFO:tasks.workunit.client.0.smithi063.stdout:2/585: rmdir d2/de 39 2024-03-23T11:47:52.280 INFO:tasks.workunit.client.0.smithi063.stdout:2/586: chown d2/de/d22 0 1 2024-03-23T11:47:52.284 INFO:tasks.workunit.client.0.smithi063.stdout:2/587: chown d2/d23/d2e/d52/d86/da0 47311425 1 2024-03-23T11:47:52.286 INFO:tasks.workunit.client.0.smithi063.stdout:2/588: chown d2/de/d22/d91 11136180 1 2024-03-23T11:47:52.290 INFO:tasks.workunit.client.0.smithi063.stdout:2/589: symlink d2/d84/d94/lb7 0 2024-03-23T11:47:52.298 INFO:tasks.workunit.client.0.smithi063.stdout:2/590: write d2/d23/d2e/d87/f82 [2675419,785] 0 2024-03-23T11:47:52.305 INFO:tasks.workunit.client.1.smithi111.stdout:8/875: dread d5/d26/d31/d3b/f60 [4194304,4194304] 0 2024-03-23T11:47:52.305 INFO:tasks.workunit.client.0.smithi063.stdout:2/591: mkdir d2/d46/db4/db8 0 2024-03-23T11:47:52.306 INFO:tasks.workunit.client.0.smithi063.stdout:4/732: dwrite d5/d1e/d2b/daa/fb8 [0,4194304] 0 2024-03-23T11:47:52.307 INFO:tasks.workunit.client.1.smithi111.stdout:8/876: fdatasync d5/f22 0 2024-03-23T11:47:52.310 INFO:tasks.workunit.client.0.smithi063.stdout:9/722: dwrite d4/d31/d43/da9/da6/f7f [0,4194304] 0 2024-03-23T11:47:52.315 INFO:tasks.workunit.client.1.smithi111.stdout:8/877: truncate f1 6065812 0 2024-03-23T11:47:52.315 INFO:tasks.workunit.client.0.smithi063.stdout:8/993: dread d6/d1c/f2d [4194304,4194304] 0 2024-03-23T11:47:52.316 INFO:tasks.workunit.client.1.smithi111.stdout:7/974: dread d14/d1b/d1e/d2c/d42/d5e/d88/d5f/f68 [0,4194304] 0 2024-03-23T11:47:52.316 INFO:tasks.workunit.client.1.smithi111.stdout:8/878: write d5/d32/d79/f104 [1007417,35280] 0 2024-03-23T11:47:52.318 INFO:tasks.workunit.client.0.smithi063.stdout:9/723: dread - d4/d31/d9c/d9e/fd4 zero size 2024-03-23T11:47:52.319 INFO:tasks.workunit.client.1.smithi111.stdout:7/975: chown d14/d1b/d1e/d2c/d2e/d73/dae/dea 929200 1 2024-03-23T11:47:52.319 INFO:tasks.workunit.client.1.smithi111.stdout:8/879: write d5/d32/d79/f104 [404796,129644] 0 2024-03-23T11:47:52.321 INFO:tasks.workunit.client.0.smithi063.stdout:9/724: dread - d4/d31/d43/da9/da6/dbe/fd2 zero size 2024-03-23T11:47:52.323 INFO:tasks.workunit.client.0.smithi063.stdout:8/994: creat d6/d1f/d54/d5c/d66/da4/d113/f14d x:0 0 0 2024-03-23T11:47:52.329 INFO:tasks.workunit.client.1.smithi111.stdout:8/880: rmdir d5/d26/d2f/d62/d75/d4b/d51/da4 39 2024-03-23T11:47:52.331 INFO:tasks.workunit.client.1.smithi111.stdout:8/881: dread - d5/d1b/f95 zero size 2024-03-23T11:47:52.333 INFO:tasks.workunit.client.1.smithi111.stdout:8/882: write d5/d32/d79/f104 [391897,5467] 0 2024-03-23T11:47:52.334 INFO:tasks.workunit.client.1.smithi111.stdout:8/883: chown d5/d26/d2f/f77 341 1 2024-03-23T11:47:52.342 INFO:tasks.workunit.client.0.smithi063.stdout:8/995: dwrite d6/d1c/d32/d37/ff4 [4194304,4194304] 0 2024-03-23T11:47:52.342 INFO:tasks.workunit.client.0.smithi063.stdout:9/725: write d4/f21 [2115339,88894] 0 2024-03-23T11:47:52.348 INFO:tasks.workunit.client.1.smithi111.stdout:7/976: dwrite d14/d1b/d1e/d2c/d42/d5e/d88/d5f/fcc [0,4194304] 0 2024-03-23T11:47:52.351 INFO:tasks.workunit.client.1.smithi111.stdout:8/884: dwrite d5/d26/d31/d4d/fd4 [0,4194304] 0 2024-03-23T11:47:52.353 INFO:tasks.workunit.client.0.smithi063.stdout:9/726: creat d4/d31/d43/da9/da6/ffc x:0 0 0 2024-03-23T11:47:52.356 INFO:tasks.workunit.client.1.smithi111.stdout:7/977: creat d14/d1b/d1e/d2c/d42/dc1/deb/d103/d11a/f122 x:0 0 0 2024-03-23T11:47:52.360 INFO:tasks.workunit.client.1.smithi111.stdout:8/885: fsync d5/d26/d2f/d62/db5/f7c 0 2024-03-23T11:47:52.361 INFO:tasks.workunit.client.1.smithi111.stdout:7/978: mknod d14/d1b/d1e/d2c/d2e/d73/dae/c123 0 2024-03-23T11:47:52.364 INFO:tasks.workunit.client.1.smithi111.stdout:8/886: mkdir d5/d26/d2f/d42/dd3/d11f 0 2024-03-23T11:47:52.364 INFO:tasks.workunit.client.0.smithi063.stdout:8/996: dread d6/d1f/d54/d5c/d66/df1/f107 [0,4194304] 0 2024-03-23T11:47:52.366 INFO:tasks.workunit.client.1.smithi111.stdout:8/887: dread - d5/d32/d7d/fa6 zero size 2024-03-23T11:47:52.368 INFO:tasks.workunit.client.1.smithi111.stdout:7/979: creat d14/d1b/d1e/d2c/d42/d9c/dd3/f124 x:0 0 0 2024-03-23T11:47:52.371 INFO:tasks.workunit.client.1.smithi111.stdout:0/980: sync 2024-03-23T11:47:52.372 INFO:tasks.workunit.client.1.smithi111.stdout:7/980: mkdir d14/d1b/d1e/d2c/d42/dc1/deb/d103/d11a/d125 0 2024-03-23T11:47:52.372 INFO:tasks.workunit.client.1.smithi111.stdout:8/888: getdents d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/db4 0 2024-03-23T11:47:52.375 INFO:tasks.workunit.client.1.smithi111.stdout:8/889: symlink d5/d32/d7d/l120 0 2024-03-23T11:47:52.375 INFO:tasks.workunit.client.0.smithi063.stdout:9/727: dwrite d4/fad [0,4194304] 0 2024-03-23T11:47:52.376 INFO:tasks.workunit.client.1.smithi111.stdout:0/981: chown d5/db/d1f/d47/d59/cb8 506 1 2024-03-23T11:47:52.376 INFO:tasks.workunit.client.1.smithi111.stdout:7/981: truncate d14/d1b/d1e/d86/fed 1012536 0 2024-03-23T11:47:52.379 INFO:tasks.workunit.client.1.smithi111.stdout:8/890: mknod d5/d26/d2f/d62/db5/dce/c121 0 2024-03-23T11:47:52.380 INFO:tasks.workunit.client.0.smithi063.stdout:8/997: dread d6/d1c/d32/d37/d10a/dae/feb [0,4194304] 0 2024-03-23T11:47:52.380 INFO:tasks.workunit.client.0.smithi063.stdout:9/728: truncate d4/d4c/d19/fd6 316163 0 2024-03-23T11:47:52.382 INFO:tasks.workunit.client.1.smithi111.stdout:7/982: link lf d14/d44/d70/daf/l126 0 2024-03-23T11:47:52.382 INFO:tasks.workunit.client.0.smithi063.stdout:9/729: chown d4/f47 7 1 2024-03-23T11:47:52.383 INFO:tasks.workunit.client.0.smithi063.stdout:8/998: stat d6/dd/d59/d13d/d5d/d7d/l122 0 2024-03-23T11:47:52.385 INFO:tasks.workunit.client.1.smithi111.stdout:7/983: mknod d14/d44/d4b/c127 0 2024-03-23T11:47:52.387 INFO:tasks.workunit.client.0.smithi063.stdout:8/999: rmdir d6/d1c/d32/d37/d10a/dc8 39 2024-03-23T11:47:52.387 INFO:tasks.workunit.client.1.smithi111.stdout:0/982: write d5/db/d1f/de4/da7/fc9 [903461,43887] 0 2024-03-23T11:47:52.391 INFO:tasks.workunit.client.1.smithi111.stdout:8/891: write d5/d26/d2f/d42/d63/f69 [2627003,51733] 0 2024-03-23T11:47:52.395 INFO:tasks.workunit.client.1.smithi111.stdout:0/983: rename d5/db/d1f/d47/l55 to d5/db/d1f/d47/d59/d5b/l149 0 2024-03-23T11:47:52.395 INFO:tasks.workunit.client.1.smithi111.stdout:8/892: symlink d5/d26/d31/d3b/d55/dd0/l122 0 2024-03-23T11:47:52.397 INFO:tasks.workunit.client.1.smithi111.stdout:0/984: creat d5/db/d4d/f14a x:0 0 0 2024-03-23T11:47:52.399 INFO:tasks.workunit.client.1.smithi111.stdout:8/893: write d5/d26/d2f/d62/d75/d4b/d51/dd1/f10a [471824,7006] 0 2024-03-23T11:47:52.399 INFO:tasks.workunit.client.1.smithi111.stdout:0/985: readlink d5/db/d4d/da6/dcc/l116 0 2024-03-23T11:47:52.402 INFO:tasks.workunit.client.1.smithi111.stdout:0/986: unlink d5/db/d1f/f18 0 2024-03-23T11:47:52.406 INFO:tasks.workunit.client.1.smithi111.stdout:0/987: unlink d5/db/d1f/d47/de0/ff9 0 2024-03-23T11:47:52.410 INFO:tasks.workunit.client.1.smithi111.stdout:0/988: creat d5/db/d1f/d47/d87/ddf/f14b x:0 0 0 2024-03-23T11:47:52.413 INFO:tasks.workunit.client.1.smithi111.stdout:0/989: creat d5/db/d4d/f14c x:0 0 0 2024-03-23T11:47:52.416 INFO:tasks.workunit.client.1.smithi111.stdout:8/894: dwrite d5/d26/d2f/d62/d75/d4b/fb1 [0,4194304] 0 2024-03-23T11:47:52.418 INFO:tasks.workunit.client.1.smithi111.stdout:8/895: readlink d5/d26/d31/d3b/d55/dd0/l122 0 2024-03-23T11:47:52.422 INFO:tasks.workunit.client.1.smithi111.stdout:0/990: dread d5/db/d1f/d19/dbd/f3f [0,4194304] 0 2024-03-23T11:47:52.424 INFO:tasks.workunit.client.1.smithi111.stdout:8/896: rename d5/d26/d2f/d62/db5/dcc/d105 to d5/d32/d7d/d112/d11b/d123 0 2024-03-23T11:47:52.425 INFO:tasks.workunit.client.1.smithi111.stdout:0/991: creat d5/dc3/f14d x:0 0 0 2024-03-23T11:47:52.426 INFO:tasks.workunit.client.1.smithi111.stdout:0/992: read d5/db/d1f/d47/f5a [5447635,77966] 0 2024-03-23T11:47:52.436 INFO:tasks.workunit.client.1.smithi111.stdout:8/897: dread d5/d26/d2f/d62/d7e/fa7 [0,4194304] 0 2024-03-23T11:47:52.439 INFO:tasks.workunit.client.1.smithi111.stdout:8/898: write d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f81 [1659416,7152] 0 2024-03-23T11:47:52.440 INFO:tasks.workunit.client.1.smithi111.stdout:0/993: write d5/db/d1f/f37 [3712886,119534] 0 2024-03-23T11:47:52.442 INFO:tasks.workunit.client.1.smithi111.stdout:8/899: fsync d5/d26/d2f/d62/d7e/fa7 0 2024-03-23T11:47:52.445 INFO:tasks.workunit.client.1.smithi111.stdout:0/994: symlink d5/db/d1f/d19/dbd/d64/l14e 0 2024-03-23T11:47:52.446 INFO:tasks.workunit.client.1.smithi111.stdout:8/900: symlink d5/d26/d2f/d62/db5/d108/l124 0 2024-03-23T11:47:52.447 INFO:tasks.workunit.client.1.smithi111.stdout:0/995: rmdir d5/db/d1f/d19/dbd/d64/d91 39 2024-03-23T11:47:52.448 INFO:tasks.workunit.client.1.smithi111.stdout:8/901: symlink d5/d26/d2f/d62/d75/df5/l125 0 2024-03-23T11:47:52.451 INFO:tasks.workunit.client.1.smithi111.stdout:8/902: symlink d5/l126 0 2024-03-23T11:47:52.451 INFO:tasks.workunit.client.1.smithi111.stdout:0/996: write d5/db/d1f/d47/f5a [874934,20734] 0 2024-03-23T11:47:52.453 INFO:tasks.workunit.client.1.smithi111.stdout:0/997: readlink d5/db/d4d/da6/lb1 0 2024-03-23T11:47:52.454 INFO:tasks.workunit.client.1.smithi111.stdout:8/903: symlink d5/d26/d2f/d62/db5/dce/l127 0 2024-03-23T11:47:52.457 INFO:tasks.workunit.client.1.smithi111.stdout:8/904: fsync d5/d26/d2f/d62/db5/f87 0 2024-03-23T11:47:52.457 INFO:tasks.workunit.client.1.smithi111.stdout:0/998: creat d5/db/d4d/da6/dcc/f14f x:0 0 0 2024-03-23T11:47:52.459 INFO:tasks.workunit.client.1.smithi111.stdout:8/905: chown d5/d26/d2f/d62/d75/d4b/d51/d5a/dbe/ffb 78 1 2024-03-23T11:47:52.460 INFO:tasks.workunit.client.1.smithi111.stdout:0/999: symlink d5/db/d1f/d47/d87/l150 0 2024-03-23T11:47:52.464 INFO:tasks.workunit.client.1.smithi111.stdout:8/906: creat d5/d26/d2f/d62/d75/d4b/de5/f128 x:0 0 0 2024-03-23T11:47:52.468 INFO:tasks.workunit.client.1.smithi111.stdout:8/907: rename d5/c2a to d5/d26/d2f/d62/d7e/d85/c129 0 2024-03-23T11:47:52.471 INFO:tasks.workunit.client.1.smithi111.stdout:8/908: symlink d5/d26/l12a 0 2024-03-23T11:47:52.472 INFO:tasks.workunit.client.1.smithi111.stdout:8/909: chown d5/d26/d2f/d62/db5/f7c 14876153 1 2024-03-23T11:47:52.474 INFO:tasks.workunit.client.1.smithi111.stdout:8/910: write d5/d26/d2f/d42/dd3/fde [468045,457] 0 2024-03-23T11:47:52.476 INFO:tasks.workunit.client.1.smithi111.stdout:8/911: write d5/d32/d7d/f115 [2506354,101631] 0 2024-03-23T11:47:52.476 INFO:tasks.workunit.client.1.smithi111.stdout:6/982: sync 2024-03-23T11:47:52.476 INFO:tasks.workunit.client.1.smithi111.stdout:1/959: sync 2024-03-23T11:47:52.477 INFO:tasks.workunit.client.1.smithi111.stdout:6/983: chown d2/ff 234020830 1 2024-03-23T11:47:52.480 INFO:tasks.workunit.client.1.smithi111.stdout:1/960: creat d0/d2b/d7c/d13e/f147 x:0 0 0 2024-03-23T11:47:52.480 INFO:tasks.workunit.client.1.smithi111.stdout:6/984: write d2/d1a/d106/d108/fca [2136961,32937] 0 2024-03-23T11:47:52.485 INFO:tasks.workunit.client.1.smithi111.stdout:8/912: getdents d5 0 2024-03-23T11:47:52.485 INFO:tasks.workunit.client.1.smithi111.stdout:6/985: creat d2/d1f/d45/d49/d8d/f13e x:0 0 0 2024-03-23T11:47:52.486 INFO:tasks.workunit.client.1.smithi111.stdout:1/961: link d0/d6/fc d0/d6/d48/d4a/d6c/d83/d11d/f148 0 2024-03-23T11:47:52.490 INFO:tasks.workunit.client.1.smithi111.stdout:8/913: mknod d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/de0/c12b 0 2024-03-23T11:47:52.491 INFO:tasks.workunit.client.1.smithi111.stdout:1/962: fdatasync d0/d6/d48/d4a/d6c/d83/d11d/fb1 0 2024-03-23T11:47:52.494 INFO:tasks.workunit.client.1.smithi111.stdout:7/984: sync 2024-03-23T11:47:52.495 INFO:tasks.workunit.client.1.smithi111.stdout:1/963: write d0/d6/d48/d4a/d6c/d83/d11d/d1e/f40 [869011,83237] 0 2024-03-23T11:47:52.498 INFO:tasks.workunit.client.1.smithi111.stdout:1/964: fsync d0/d6/d48/d4a/d6c/d96/fbf 0 2024-03-23T11:47:52.503 INFO:tasks.workunit.client.1.smithi111.stdout:6/986: dwrite d2/d1f/d45/f56 [0,4194304] 0 2024-03-23T11:47:52.505 INFO:tasks.workunit.client.1.smithi111.stdout:1/965: write d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d87/d8e/f146 [418640,9418] 0 2024-03-23T11:47:52.505 INFO:tasks.workunit.client.1.smithi111.stdout:6/987: readlink d2/d35/dc5/l10a 0 2024-03-23T11:47:52.508 INFO:tasks.workunit.client.1.smithi111.stdout:8/914: dwrite d5/d26/d2f/d62/d7e/fa7 [0,4194304] 0 2024-03-23T11:47:52.514 INFO:tasks.workunit.client.1.smithi111.stdout:1/966: mknod d0/d6/d48/d4a/d6c/d83/d11d/d91/c149 0 2024-03-23T11:47:52.514 INFO:tasks.workunit.client.1.smithi111.stdout:1/967: chown d0/l15 3250 1 2024-03-23T11:47:52.517 INFO:tasks.workunit.client.1.smithi111.stdout:6/988: write d2/d1a/fad [173233,45168] 0 2024-03-23T11:47:52.519 INFO:tasks.workunit.client.1.smithi111.stdout:6/989: fdatasync d2/d17/f20 0 2024-03-23T11:47:52.527 INFO:tasks.workunit.client.1.smithi111.stdout:6/990: rename d2/fa2 to d2/d1a/d1c/d2c/d51/f13f 0 2024-03-23T11:47:52.527 INFO:tasks.workunit.client.1.smithi111.stdout:8/915: link d5/d26/d2f/d62/d75/df5/l125 d5/d26/d2f/d62/db5/d108/l12c 0 2024-03-23T11:47:52.530 INFO:tasks.workunit.client.1.smithi111.stdout:8/916: write d5/d26/d31/fb6 [5112344,39112] 0 2024-03-23T11:47:52.530 INFO:tasks.workunit.client.1.smithi111.stdout:6/991: fsync d2/d80/f99 0 2024-03-23T11:47:52.533 INFO:tasks.workunit.client.1.smithi111.stdout:6/992: creat d2/d80/d100/f140 x:0 0 0 2024-03-23T11:47:52.533 INFO:tasks.workunit.client.1.smithi111.stdout:8/917: write d5/d26/d2f/d62/d75/d4b/d51/dd1/f10a [180306,117149] 0 2024-03-23T11:47:52.537 INFO:tasks.workunit.client.1.smithi111.stdout:8/918: truncate d5/d32/d79/fb3 1237187 0 2024-03-23T11:47:52.539 INFO:tasks.workunit.client.1.smithi111.stdout:8/919: fdatasync d5/d26/d2f/d42/d63/fe7 0 2024-03-23T11:47:52.541 INFO:tasks.workunit.client.1.smithi111.stdout:6/993: link d2/d73/f139 d2/d1a/d106/d108/f141 0 2024-03-23T11:47:52.541 INFO:tasks.workunit.client.1.smithi111.stdout:8/920: stat d5/d32/d7d/da0/c101 0 2024-03-23T11:47:52.543 INFO:tasks.workunit.client.1.smithi111.stdout:1/968: dread d0/d2b/f3e [0,4194304] 0 2024-03-23T11:47:52.545 INFO:tasks.workunit.client.1.smithi111.stdout:6/994: creat d2/d1f/d45/d49/f142 x:0 0 0 2024-03-23T11:47:52.546 INFO:tasks.workunit.client.1.smithi111.stdout:1/969: mkdir d0/d5/d128/d14a 0 2024-03-23T11:47:52.557 INFO:tasks.workunit.client.1.smithi111.stdout:8/921: dwrite d5/d32/d79/f104 [0,4194304] 0 2024-03-23T11:47:52.559 INFO:tasks.workunit.client.1.smithi111.stdout:1/970: dread d0/d6/fc [0,4194304] 0 2024-03-23T11:47:52.560 INFO:tasks.workunit.client.1.smithi111.stdout:8/922: read - d5/d26/d2f/d42/f57 zero size 2024-03-23T11:47:52.563 INFO:tasks.workunit.client.1.smithi111.stdout:6/995: dwrite d2/d17/fc7 [0,4194304] 0 2024-03-23T11:47:52.564 INFO:tasks.workunit.client.1.smithi111.stdout:8/923: read d5/d26/d2f/d62/db5/f87 [2086938,57079] 0 2024-03-23T11:47:52.566 INFO:tasks.workunit.client.1.smithi111.stdout:1/971: mkdir d0/d6/d48/d4a/d6c/d83/d11d/d1e/d63/d14b 0 2024-03-23T11:47:52.568 INFO:tasks.workunit.client.1.smithi111.stdout:8/924: read - d5/d26/d2f/d62/d75/d4b/d51/dd1/f113 zero size 2024-03-23T11:47:52.573 INFO:tasks.workunit.client.1.smithi111.stdout:6/996: write d2/d1a/d1c/d2c/fb9 [2885539,47189] 0 2024-03-23T11:47:52.573 INFO:tasks.workunit.client.1.smithi111.stdout:8/925: symlink d5/d26/d2f/d42/d63/l12d 0 2024-03-23T11:47:52.575 INFO:tasks.workunit.client.1.smithi111.stdout:6/997: read - d2/d35/d70/d98/fa4 zero size 2024-03-23T11:47:52.576 INFO:tasks.workunit.client.1.smithi111.stdout:8/926: write d5/d32/d7d/d112/d11b/d123/f118 [836922,59332] 0 2024-03-23T11:47:52.582 INFO:tasks.workunit.client.1.smithi111.stdout:6/998: link d2/d1f/dce/cfd d2/d1f/d45/d49/c143 0 2024-03-23T11:47:52.586 INFO:tasks.workunit.client.1.smithi111.stdout:6/999: mknod d2/d1a/d1c/d2c/d51/def/d118/c144 0 2024-03-23T11:47:52.594 INFO:tasks.workunit.client.1.smithi111.stdout:7/985: sync 2024-03-23T11:47:52.594 INFO:tasks.workunit.client.1.smithi111.stdout:8/927: dread d5/d26/d31/d4d/fc4 [0,4194304] 0 2024-03-23T11:47:52.604 INFO:tasks.workunit.client.1.smithi111.stdout:1/972: dread d0/d6/d48/fa8 [0,4194304] 0 2024-03-23T11:47:52.605 INFO:tasks.workunit.client.1.smithi111.stdout:7/986: truncate d14/d1b/d1e/d2c/d2e/d73/dae/d74/fa9 921649 0 2024-03-23T11:47:52.611 INFO:tasks.workunit.client.1.smithi111.stdout:7/987: truncate d14/d1b/d1e/d2c/d42/dc1/deb/d103/d11a/f122 323104 0 2024-03-23T11:47:52.613 INFO:tasks.workunit.client.1.smithi111.stdout:7/988: dread - d14/d1b/d1e/d2c/d2e/d33/d98/f10c zero size 2024-03-23T11:47:52.619 INFO:tasks.workunit.client.1.smithi111.stdout:7/989: chown d14/d1b/d1e/d2c/d42/d5e/d88/f57 7495 1 2024-03-23T11:47:52.621 INFO:tasks.workunit.client.1.smithi111.stdout:8/928: dread d5/f2d [0,4194304] 0 2024-03-23T11:47:52.629 INFO:tasks.workunit.client.1.smithi111.stdout:1/973: dwrite d0/d2b/de3/f115 [0,4194304] 0 2024-03-23T11:47:52.632 INFO:tasks.workunit.client.1.smithi111.stdout:8/929: fsync d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/f52 0 2024-03-23T11:47:52.641 INFO:tasks.workunit.client.1.smithi111.stdout:8/930: mkdir d5/d26/d31/d4d/d12e 0 2024-03-23T11:47:52.643 INFO:tasks.workunit.client.1.smithi111.stdout:8/931: fsync d5/f2d 0 2024-03-23T11:47:52.650 INFO:tasks.workunit.client.1.smithi111.stdout:7/990: dwrite d14/d1b/d1e/d2c/d42/d5e/d9e/f9f [0,4194304] 0 2024-03-23T11:47:52.653 INFO:tasks.workunit.client.1.smithi111.stdout:1/974: dwrite d0/d2b/d37/d6a/ff8 [8388608,4194304] 0 2024-03-23T11:47:52.666 INFO:tasks.workunit.client.1.smithi111.stdout:8/932: dread d5/fe [0,4194304] 0 2024-03-23T11:47:52.671 INFO:tasks.workunit.client.1.smithi111.stdout:8/933: symlink d5/d26/d31/d4d/d12e/l12f 0 2024-03-23T11:47:52.671 INFO:tasks.workunit.client.1.smithi111.stdout:7/991: dwrite d14/d1b/d1e/d2c/d2e/d73/dae/f62 [0,4194304] 0 2024-03-23T11:47:52.672 INFO:tasks.workunit.client.1.smithi111.stdout:8/934: chown d5/d26/cbd 131112 1 2024-03-23T11:47:52.673 INFO:tasks.workunit.client.1.smithi111.stdout:1/975: truncate d0/d6/d48/d4a/d6c/d83/d11d/d1e/fa0 954550 0 2024-03-23T11:47:52.673 INFO:tasks.workunit.client.1.smithi111.stdout:7/992: chown d14/d37/fda 50943 1 2024-03-23T11:47:52.677 INFO:tasks.workunit.client.1.smithi111.stdout:8/935: creat d5/d26/d31/d5e/f130 x:0 0 0 2024-03-23T11:47:52.679 INFO:tasks.workunit.client.1.smithi111.stdout:1/976: truncate d0/d6/d48/d4d/dab/dbd/f135 911322 0 2024-03-23T11:47:52.680 INFO:tasks.workunit.client.1.smithi111.stdout:7/993: truncate d14/d1b/d1e/d2c/d42/d9c/dd3/f124 739181 0 2024-03-23T11:47:52.681 INFO:tasks.workunit.client.1.smithi111.stdout:8/936: creat d5/d26/d31/d3b/d55/f131 x:0 0 0 2024-03-23T11:47:52.682 INFO:tasks.workunit.client.1.smithi111.stdout:8/937: stat d5/d26/d2f/d42/c6d 0 2024-03-23T11:47:52.683 INFO:tasks.workunit.client.1.smithi111.stdout:7/994: creat d14/d37/d43/db1/f128 x:0 0 0 2024-03-23T11:47:52.684 INFO:tasks.workunit.client.1.smithi111.stdout:8/938: chown d5/d32/lb0 39088468 1 2024-03-23T11:47:52.685 INFO:tasks.workunit.client.1.smithi111.stdout:7/995: dread - d14/d1b/d1e/d86/f89 zero size 2024-03-23T11:47:52.685 INFO:tasks.workunit.client.0.smithi063.stdout:1/645: sync 2024-03-23T11:47:52.685 INFO:tasks.workunit.client.0.smithi063.stdout:7/664: sync 2024-03-23T11:47:52.687 INFO:tasks.workunit.client.1.smithi111.stdout:7/996: fdatasync d14/d1b/d1e/d2c/d2e/f34 0 2024-03-23T11:47:52.688 INFO:tasks.workunit.client.1.smithi111.stdout:8/939: symlink d5/d32/d7d/d112/l132 0 2024-03-23T11:47:52.691 INFO:tasks.workunit.client.0.smithi063.stdout:7/665: rename d5/d26/db7/dc6/d6e/da2/dc9 to d5/d26/db7/dce 0 2024-03-23T11:47:52.693 INFO:tasks.workunit.client.1.smithi111.stdout:8/940: unlink d5/d26/d31/d4d/faf 0 2024-03-23T11:47:52.696 INFO:tasks.workunit.client.1.smithi111.stdout:7/997: write d14/d1b/d22/f4d [4489243,105496] 0 2024-03-23T11:47:52.697 INFO:tasks.workunit.client.0.smithi063.stdout:1/646: getdents d3/d49/d56/d65/d88/d92 0 2024-03-23T11:47:52.701 INFO:tasks.workunit.client.1.smithi111.stdout:1/977: dwrite d0/d6/f19 [0,4194304] 0 2024-03-23T11:47:52.703 INFO:tasks.workunit.client.1.smithi111.stdout:8/941: getdents d5/d32/d7d/d112/d11b 0 2024-03-23T11:47:52.706 INFO:tasks.workunit.client.1.smithi111.stdout:7/998: rename d14/l18 to d14/d1b/d1e/d2c/d2e/d73/dae/d74/d117/l129 0 2024-03-23T11:47:52.707 INFO:tasks.workunit.client.1.smithi111.stdout:8/942: creat d5/d26/d2f/d62/d75/d4b/de5/f133 x:0 0 0 2024-03-23T11:47:52.709 INFO:tasks.workunit.client.1.smithi111.stdout:7/999: chown d14/d1b/d1e/d2c/d42/d5e/d88/f94 2 1 2024-03-23T11:47:52.713 INFO:tasks.workunit.client.1.smithi111.stdout:1/978: write d0/d6/d48/d4a/d6c/d83/d11d/fb1 [1578888,40078] 0 2024-03-23T11:47:52.714 INFO:tasks.workunit.client.1.smithi111.stdout:8/943: rmdir d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71 39 2024-03-23T11:47:52.715 INFO:tasks.workunit.client.1.smithi111.stdout:8/944: fsync d5/d1b/f3a 0 2024-03-23T11:47:52.719 INFO:tasks.workunit.client.1.smithi111.stdout:8/945: chown d5/d26/d2f/d62/d75/d4b/d51/cd2 5 1 2024-03-23T11:47:52.721 INFO:tasks.workunit.client.0.smithi063.stdout:7/666: dwrite d5/d26/db7/fcd [0,4194304] 0 2024-03-23T11:47:52.722 INFO:tasks.workunit.client.0.smithi063.stdout:7/667: read - d5/d6/dca/fcb zero size 2024-03-23T11:47:52.724 INFO:tasks.workunit.client.0.smithi063.stdout:7/668: chown d5/d6/f27 6088604 1 2024-03-23T11:47:52.726 INFO:tasks.workunit.client.0.smithi063.stdout:1/647: dread d3/df/d31/f5f [0,4194304] 0 2024-03-23T11:47:52.729 INFO:tasks.workunit.client.1.smithi111.stdout:8/946: write d5/d26/d31/d5e/f3e [127523,28237] 0 2024-03-23T11:47:52.731 INFO:tasks.workunit.client.0.smithi063.stdout:1/648: unlink d3/df/d2f/ca4 0 2024-03-23T11:47:52.735 INFO:tasks.workunit.client.0.smithi063.stdout:1/649: symlink d3/d49/d56/d65/d88/lc6 0 2024-03-23T11:47:52.740 INFO:tasks.workunit.client.1.smithi111.stdout:1/979: dwrite d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/dbb/fc1 [0,4194304] 0 2024-03-23T11:47:52.746 INFO:tasks.workunit.client.1.smithi111.stdout:1/980: mkdir d0/d6/d48/d4a/d6c/d83/d11d/d1e/d70/d14c 0 2024-03-23T11:47:52.747 INFO:tasks.workunit.client.1.smithi111.stdout:8/947: dread d5/d26/d2f/d62/d75/d4b/fb1 [0,4194304] 0 2024-03-23T11:47:52.750 INFO:tasks.workunit.client.0.smithi063.stdout:1/650: dread d3/d3f/d4d/f5b [0,4194304] 0 2024-03-23T11:47:52.754 INFO:tasks.workunit.client.0.smithi063.stdout:1/651: read d3/d49/d56/f7f [190884,25199] 0 2024-03-23T11:47:52.755 INFO:tasks.workunit.client.0.smithi063.stdout:1/652: stat d3/df/f12 0 2024-03-23T11:47:52.756 INFO:tasks.workunit.client.0.smithi063.stdout:1/653: chown d3/d35/d40/l4f 317 1 2024-03-23T11:47:52.758 INFO:tasks.workunit.client.1.smithi111.stdout:1/981: write d0/d2b/d37/d6a/fe4 [2183836,63509] 0 2024-03-23T11:47:52.758 INFO:tasks.workunit.client.1.smithi111.stdout:8/948: rename d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9/dad/f109 to d5/d26/d2f/d62/d75/df5/f134 0 2024-03-23T11:47:52.759 INFO:tasks.workunit.client.0.smithi063.stdout:1/654: chown d3/d35/d40/f53 8668014 1 2024-03-23T11:47:52.765 INFO:tasks.workunit.client.1.smithi111.stdout:8/949: rename d5/d32/l5d to d5/d26/d2f/d62/d75/d4b/d51/dd1/l135 0 2024-03-23T11:47:52.765 INFO:tasks.workunit.client.1.smithi111.stdout:1/982: link d0/d6/d48/d4d/lc8 d0/d6/d48/d4a/de5/d125/l14d 0 2024-03-23T11:47:52.767 INFO:tasks.workunit.client.0.smithi063.stdout:7/669: dread d5/d5c/d2f/d73/da/f89 [0,4194304] 0 2024-03-23T11:47:52.770 INFO:tasks.workunit.client.1.smithi111.stdout:8/950: unlink d5/c45 0 2024-03-23T11:47:52.772 INFO:tasks.workunit.client.0.smithi063.stdout:7/670: dread d5/d26/db7/dc6/d6e/f81 [0,4194304] 0 2024-03-23T11:47:52.773 INFO:tasks.workunit.client.1.smithi111.stdout:8/951: mkdir d5/d26/d2f/d62/d136 0 2024-03-23T11:47:52.775 INFO:tasks.workunit.client.1.smithi111.stdout:8/952: chown d5/d26/d2f/d8f/cac 1145 1 2024-03-23T11:47:52.775 INFO:tasks.workunit.client.0.smithi063.stdout:7/671: dread - d5/d6/d19/d65/fba zero size 2024-03-23T11:47:52.776 INFO:tasks.workunit.client.1.smithi111.stdout:1/983: write d0/d2b/f2e [4354633,104045] 0 2024-03-23T11:47:52.778 INFO:tasks.workunit.client.0.smithi063.stdout:7/672: mkdir d5/d6/d19/dcf 0 2024-03-23T11:47:52.779 INFO:tasks.workunit.client.1.smithi111.stdout:1/984: stat d0/d6/d48/d4a/d6c/d83/d11d/d1e 0 2024-03-23T11:47:52.781 INFO:tasks.workunit.client.1.smithi111.stdout:1/985: fdatasync d0/d5/f23 0 2024-03-23T11:47:52.788 INFO:tasks.workunit.client.1.smithi111.stdout:1/986: rename d0/d6/d48/d4d/dab/lc3 to d0/d6/d48/d4a/d6c/d83/d11d/d1e/d54/d137/l14e 0 2024-03-23T11:47:52.789 INFO:tasks.workunit.client.1.smithi111.stdout:8/953: dwrite d5/d26/d31/d5e/f65 [0,4194304] 0 2024-03-23T11:47:52.790 INFO:tasks.workunit.client.1.smithi111.stdout:1/987: fdatasync d0/d2b/d7c/dca/f119 0 2024-03-23T11:47:52.792 INFO:tasks.workunit.client.0.smithi063.stdout:1/655: dread d3/d1c/d4b/f8d [0,4194304] 0 2024-03-23T11:47:52.795 INFO:tasks.workunit.client.1.smithi111.stdout:1/988: write d0/d6/d48/d4a/d6c/d83/d11d/fb1 [2622179,94129] 0 2024-03-23T11:47:52.796 INFO:tasks.workunit.client.0.smithi063.stdout:1/656: mknod d3/df/d7b/cc7 0 2024-03-23T11:47:52.799 INFO:tasks.workunit.client.1.smithi111.stdout:8/954: rename d5/d32/d7d/da0/c101 to d5/d26/dd8/c137 0 2024-03-23T11:47:52.800 INFO:tasks.workunit.client.0.smithi063.stdout:1/657: fdatasync d3/d35/d40/f53 0 2024-03-23T11:47:52.805 INFO:tasks.workunit.client.0.smithi063.stdout:1/658: symlink d3/d49/d56/d65/d84/lc8 0 2024-03-23T11:47:52.808 INFO:tasks.workunit.client.0.smithi063.stdout:1/659: symlink d3/d1c/lc9 0 2024-03-23T11:47:52.810 INFO:tasks.workunit.client.0.smithi063.stdout:1/660: write d3/d49/d56/d65/f7c [8407609,738] 0 2024-03-23T11:47:52.812 INFO:tasks.workunit.client.0.smithi063.stdout:7/673: dread d5/d22/f24 [0,4194304] 0 2024-03-23T11:47:52.813 INFO:tasks.workunit.client.1.smithi111.stdout:1/989: write d0/d6/d48/d4a/d6c/d83/d11d/d1e/fa0 [876130,32] 0 2024-03-23T11:47:52.814 INFO:tasks.workunit.client.0.smithi063.stdout:7/674: fsync d5/d26/f74 0 2024-03-23T11:47:52.817 INFO:tasks.workunit.client.0.smithi063.stdout:7/675: creat d5/d5c/d2f/d8b/fd0 x:0 0 0 2024-03-23T11:47:52.818 INFO:tasks.workunit.client.0.smithi063.stdout:7/676: chown d5/d5c/d2f/d8c 53602236 1 2024-03-23T11:47:52.820 INFO:tasks.workunit.client.1.smithi111.stdout:1/990: truncate d0/f1 3889779 0 2024-03-23T11:47:52.820 INFO:tasks.workunit.client.0.smithi063.stdout:7/677: dread - d5/d6/d19/f7d zero size 2024-03-23T11:47:52.822 INFO:tasks.workunit.client.1.smithi111.stdout:1/991: mkdir d0/d6/d48/d4a/de5/d125/d14f 0 2024-03-23T11:47:52.823 INFO:tasks.workunit.client.0.smithi063.stdout:3/996: sync 2024-03-23T11:47:52.823 INFO:tasks.workunit.client.0.smithi063.stdout:5/477: sync 2024-03-23T11:47:52.823 INFO:tasks.workunit.client.0.smithi063.stdout:0/867: sync 2024-03-23T11:47:52.823 INFO:tasks.workunit.client.0.smithi063.stdout:7/678: symlink d5/d26/db7/dc6/d6e/ld1 0 2024-03-23T11:47:52.826 INFO:tasks.workunit.client.0.smithi063.stdout:3/997: readlink d13/d22/d2d/d3f/d58/d97/l11e 0 2024-03-23T11:47:52.830 INFO:tasks.workunit.client.0.smithi063.stdout:3/998: stat d13/d22/l40 0 2024-03-23T11:47:52.839 INFO:tasks.workunit.client.1.smithi111.stdout:1/992: dwrite d0/d2b/d7c/f9f [0,4194304] 0 2024-03-23T11:47:52.842 INFO:tasks.workunit.client.1.smithi111.stdout:1/993: creat d0/d6/d48/d4a/de5/f150 x:0 0 0 2024-03-23T11:47:52.845 INFO:tasks.workunit.client.1.smithi111.stdout:1/994: mkdir d0/dda/d151 0 2024-03-23T11:47:52.845 INFO:tasks.workunit.client.0.smithi063.stdout:3/999: rename d13/d22/d2d/d3f/d67/fd1 to d13/d15/d7e/f13f 0 2024-03-23T11:47:52.845 INFO:tasks.workunit.client.0.smithi063.stdout:1/661: dread d3/df/d2f/f97 [0,4194304] 0 2024-03-23T11:47:52.847 INFO:tasks.workunit.client.1.smithi111.stdout:1/995: mknod d0/d5/dd5/c152 0 2024-03-23T11:47:52.852 INFO:tasks.workunit.client.1.smithi111.stdout:1/996: dread d0/d6/d48/d4d/dab/dbd/f135 [0,4194304] 0 2024-03-23T11:47:52.852 INFO:tasks.workunit.client.0.smithi063.stdout:1/662: write d3/f9 [1316532,102595] 0 2024-03-23T11:47:52.854 INFO:tasks.workunit.client.1.smithi111.stdout:1/997: chown d0/d2b/d7c/d13e 127091 1 2024-03-23T11:47:52.855 INFO:tasks.workunit.client.0.smithi063.stdout:0/868: dwrite d2/d12/d32/d6d/d124/d1f/f2d [0,4194304] 0 2024-03-23T11:47:52.855 INFO:tasks.workunit.client.0.smithi063.stdout:5/478: dread d2/d7/fd [0,4194304] 0 2024-03-23T11:47:52.856 INFO:tasks.workunit.client.0.smithi063.stdout:1/663: fsync d3/df/db8/fc4 0 2024-03-23T11:47:52.861 INFO:tasks.workunit.client.0.smithi063.stdout:5/479: write d2/d28/d3a/d46/d71/d9f/fa1 [730306,73774] 0 2024-03-23T11:47:52.862 INFO:tasks.workunit.client.0.smithi063.stdout:1/664: mkdir d3/d1c/d4b/db5/dca 0 2024-03-23T11:47:52.863 INFO:tasks.workunit.client.0.smithi063.stdout:0/869: creat d2/d14/d36/d77/f132 x:0 0 0 2024-03-23T11:47:52.865 INFO:tasks.workunit.client.0.smithi063.stdout:7/679: dread d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:52.867 INFO:tasks.workunit.client.0.smithi063.stdout:5/480: mkdir d2/d28/d3a/d46/da4 0 2024-03-23T11:47:52.867 INFO:tasks.workunit.client.1.smithi111.stdout:1/998: dwrite d0/d5/f35 [0,4194304] 0 2024-03-23T11:47:52.872 INFO:tasks.workunit.client.1.smithi111.stdout:1/999: dread d0/d6/fc [0,4194304] 0 2024-03-23T11:47:52.873 INFO:tasks.workunit.client.0.smithi063.stdout:0/870: creat d2/d14/d36/d77/f133 x:0 0 0 2024-03-23T11:47:52.878 INFO:tasks.workunit.client.0.smithi063.stdout:5/481: getdents d2/d28 0 2024-03-23T11:47:52.889 INFO:tasks.workunit.client.0.smithi063.stdout:7/680: write d5/d5c/d2f/d73/da/f17 [2788628,96261] 0 2024-03-23T11:47:52.892 INFO:tasks.workunit.client.0.smithi063.stdout:7/681: truncate d5/fc7 370557 0 2024-03-23T11:47:52.896 INFO:tasks.workunit.client.0.smithi063.stdout:7/682: rmdir d5/d6/d19 39 2024-03-23T11:47:52.899 INFO:tasks.workunit.client.0.smithi063.stdout:1/665: dwrite d3/d49/d56/d65/d84/f8e [0,4194304] 0 2024-03-23T11:47:52.901 INFO:tasks.workunit.client.0.smithi063.stdout:0/871: dwrite d2/d12/d32/d6d/d124/d1f/fb5 [0,4194304] 0 2024-03-23T11:47:52.902 INFO:tasks.workunit.client.0.smithi063.stdout:7/683: chown d5/d5c/d2f/d73/d20/c38 10 1 2024-03-23T11:47:52.907 INFO:tasks.workunit.client.0.smithi063.stdout:1/666: mkdir d3/df/db8/dcb 0 2024-03-23T11:47:52.911 INFO:tasks.workunit.client.0.smithi063.stdout:7/684: mkdir d5/d5c/d2f/db5/dd2 0 2024-03-23T11:47:52.912 INFO:tasks.workunit.client.0.smithi063.stdout:1/667: creat d3/df/d2f/fcc x:0 0 0 2024-03-23T11:47:52.913 INFO:tasks.workunit.client.1.smithi111.stdout:8/955: sync 2024-03-23T11:47:52.914 INFO:tasks.workunit.client.0.smithi063.stdout:7/685: fsync f1 0 2024-03-23T11:47:52.914 INFO:tasks.workunit.client.0.smithi063.stdout:5/482: dread d2/d7/d8/f13 [0,4194304] 0 2024-03-23T11:47:52.918 INFO:tasks.workunit.client.0.smithi063.stdout:1/668: getdents d3/df/d31 0 2024-03-23T11:47:52.922 INFO:tasks.workunit.client.0.smithi063.stdout:5/483: rename d2/f84 to d2/d28/d3a/fa5 0 2024-03-23T11:47:52.923 INFO:tasks.workunit.client.0.smithi063.stdout:7/686: mkdir d5/d26/db7/dc6/d4f/d5b/dd3 0 2024-03-23T11:47:52.925 INFO:tasks.workunit.client.0.smithi063.stdout:6/572: sync 2024-03-23T11:47:52.925 INFO:tasks.workunit.client.0.smithi063.stdout:2/592: sync 2024-03-23T11:47:52.925 INFO:tasks.workunit.client.0.smithi063.stdout:9/730: sync 2024-03-23T11:47:52.925 INFO:tasks.workunit.client.0.smithi063.stdout:4/733: sync 2024-03-23T11:47:52.928 INFO:tasks.workunit.client.1.smithi111.stdout:8/956: getdents d5/d26/d2f/d62/d75/d4b/d51/d5a 0 2024-03-23T11:47:52.929 INFO:tasks.workunit.client.0.smithi063.stdout:9/731: chown d4/d31/c89 2152 1 2024-03-23T11:47:52.929 INFO:tasks.workunit.client.0.smithi063.stdout:7/687: write d5/d5c/d2f/d8c/fb0 [536353,52119] 0 2024-03-23T11:47:52.933 INFO:tasks.workunit.client.0.smithi063.stdout:2/593: symlink d2/lb9 0 2024-03-23T11:47:52.936 INFO:tasks.workunit.client.0.smithi063.stdout:4/734: dread - d5/da/d20/d4b/d16/d32/d56/f9e zero size 2024-03-23T11:47:52.936 INFO:tasks.workunit.client.0.smithi063.stdout:9/732: write d4/f14 [1705599,58951] 0 2024-03-23T11:47:52.937 INFO:tasks.workunit.client.0.smithi063.stdout:6/573: dread d1/dc/d21/f29 [0,4194304] 0 2024-03-23T11:47:52.938 INFO:tasks.workunit.client.1.smithi111.stdout:8/957: rename d5/d32/d7d/l120 to d5/d26/d2f/d42/dd3/l138 0 2024-03-23T11:47:52.940 INFO:tasks.workunit.client.0.smithi063.stdout:5/484: mkdir d2/d7/d8/de/d35/da6 0 2024-03-23T11:47:52.942 INFO:tasks.workunit.client.0.smithi063.stdout:0/872: dwrite d2/d12/d32/d6d/d124/d28/fa3 [0,4194304] 0 2024-03-23T11:47:52.951 INFO:tasks.workunit.client.0.smithi063.stdout:4/735: mknod d5/da/d20/d4b/d16/d50/d5d/cd1 0 2024-03-23T11:47:52.952 INFO:tasks.workunit.client.0.smithi063.stdout:1/669: dwrite d3/f11 [0,4194304] 0 2024-03-23T11:47:52.953 INFO:tasks.workunit.client.0.smithi063.stdout:2/594: rename d2/d23/d2e/f36 to d2/d23/d2e/d98/fba 0 2024-03-23T11:47:52.956 INFO:tasks.workunit.client.0.smithi063.stdout:1/670: readlink d3/df/d31/l3d 0 2024-03-23T11:47:52.956 INFO:tasks.workunit.client.0.smithi063.stdout:5/485: fdatasync d2/d7/d8/de/d49/f63 0 2024-03-23T11:47:52.958 INFO:tasks.workunit.client.0.smithi063.stdout:9/733: unlink d4/d31/d43/da9/da6/c38 0 2024-03-23T11:47:52.959 INFO:tasks.workunit.client.0.smithi063.stdout:6/574: write d1/d15/d4b/f6b [680038,128635] 0 2024-03-23T11:47:52.960 INFO:tasks.workunit.client.0.smithi063.stdout:7/688: dwrite d5/d5c/d2f/f79 [0,4194304] 0 2024-03-23T11:47:52.964 INFO:tasks.workunit.client.1.smithi111.stdout:8/958: dwrite d5/d1b/fc9 [0,4194304] 0 2024-03-23T11:47:52.965 INFO:tasks.workunit.client.0.smithi063.stdout:7/689: chown d5/d26/db7/dc6/d6e/da2 81094652 1 2024-03-23T11:47:52.965 INFO:tasks.workunit.client.0.smithi063.stdout:0/873: write d2/d14/f34 [5020181,113695] 0 2024-03-23T11:47:52.968 INFO:tasks.workunit.client.1.smithi111.stdout:8/959: dread - d5/d32/d7d/fa6 zero size 2024-03-23T11:47:52.969 INFO:tasks.workunit.client.0.smithi063.stdout:7/690: fsync d5/d5c/d2f/d8c/fb0 0 2024-03-23T11:47:52.971 INFO:tasks.workunit.client.1.smithi111.stdout:8/960: fdatasync d5/d26/d31/d4d/f6c 0 2024-03-23T11:47:52.971 INFO:tasks.workunit.client.0.smithi063.stdout:5/486: mkdir d2/d7/d2b/d8b/da7 0 2024-03-23T11:47:52.972 INFO:tasks.workunit.client.0.smithi063.stdout:2/595: symlink d2/lbb 0 2024-03-23T11:47:52.973 INFO:tasks.workunit.client.0.smithi063.stdout:2/596: readlink d2/lbb 0 2024-03-23T11:47:52.974 INFO:tasks.workunit.client.1.smithi111.stdout:8/961: stat d5/f11 0 2024-03-23T11:47:52.975 INFO:tasks.workunit.client.0.smithi063.stdout:4/736: mkdir d5/da/d20/d4b/d16/d50/d8e/d96/dc0/dd2 0 2024-03-23T11:47:52.979 INFO:tasks.workunit.client.1.smithi111.stdout:8/962: truncate d5/d26/d2f/d62/d75/d4b/de5/f133 144401 0 2024-03-23T11:47:52.981 INFO:tasks.workunit.client.0.smithi063.stdout:1/671: mknod d3/d3f/ccd 0 2024-03-23T11:47:52.982 INFO:tasks.workunit.client.0.smithi063.stdout:7/691: mknod d5/d26/d76/cd4 0 2024-03-23T11:47:52.984 INFO:tasks.workunit.client.0.smithi063.stdout:7/692: fdatasync d5/d5c/d2f/d73/f8d 0 2024-03-23T11:47:52.985 INFO:tasks.workunit.client.0.smithi063.stdout:2/597: readlink d2/d46/db4/lb6 0 2024-03-23T11:47:52.995 INFO:tasks.workunit.client.0.smithi063.stdout:9/734: write d4/d4c/fde [60879,78005] 0 2024-03-23T11:47:52.996 INFO:tasks.workunit.client.1.smithi111.stdout:8/963: creat d5/d26/d31/d3b/f139 x:0 0 0 2024-03-23T11:47:52.997 INFO:tasks.workunit.client.0.smithi063.stdout:4/737: rename d5/da/d20/d4b/d16/d32/d6f to d5/da/d20/d4b/d16/d50/d8e/dd3 0 2024-03-23T11:47:53.000 INFO:tasks.workunit.client.0.smithi063.stdout:7/693: rmdir d5/d26/db7/dc6/d6e/da2 39 2024-03-23T11:47:53.000 INFO:tasks.workunit.client.0.smithi063.stdout:2/598: chown d2/d23/d2e/l42 571350132 1 2024-03-23T11:47:53.001 INFO:tasks.workunit.client.0.smithi063.stdout:0/874: creat d2/d48/f134 x:0 0 0 2024-03-23T11:47:53.004 INFO:tasks.workunit.client.1.smithi111.stdout:8/964: creat d5/d1b/dbc/dc3/f13a x:0 0 0 2024-03-23T11:47:53.011 INFO:tasks.workunit.client.1.smithi111.stdout:8/965: rename d5/d26/d2f/d62/db5/f7c to d5/d26/d2f/d62/d75/f13b 0 2024-03-23T11:47:53.017 INFO:tasks.workunit.client.1.smithi111.stdout:8/966: unlink d5/d26/d31/d4d/f107 0 2024-03-23T11:47:53.017 INFO:tasks.workunit.client.0.smithi063.stdout:4/738: write d5/da/d72/fa8 [683419,85699] 0 2024-03-23T11:47:53.017 INFO:tasks.workunit.client.0.smithi063.stdout:1/672: getdents d3/d1c/d4b/db5 0 2024-03-23T11:47:53.021 INFO:tasks.workunit.client.0.smithi063.stdout:2/599: truncate d2/d23/d2e/d52/d6c/f7c 601932 0 2024-03-23T11:47:53.024 INFO:tasks.workunit.client.0.smithi063.stdout:1/673: mkdir d3/d1c/dce 0 2024-03-23T11:47:53.025 INFO:tasks.workunit.client.0.smithi063.stdout:6/575: dread d1/dc/d3a/d82/da2/f62 [0,4194304] 0 2024-03-23T11:47:53.025 INFO:tasks.workunit.client.0.smithi063.stdout:5/487: dwrite d2/d28/d3a/d46/d79/f7e [0,4194304] 0 2024-03-23T11:47:53.036 INFO:tasks.workunit.client.1.smithi111.stdout:8/967: truncate d5/d26/d31/d3b/d55/dd0/fe9 1913469 0 2024-03-23T11:47:53.037 INFO:tasks.workunit.client.1.smithi111.stdout:8/968: stat d5/d26/d31/d3b/d55/ff1 0 2024-03-23T11:47:53.037 INFO:tasks.workunit.client.0.smithi063.stdout:4/739: truncate d5/da/d20/d4b/f5e 436336 0 2024-03-23T11:47:53.039 INFO:tasks.workunit.client.0.smithi063.stdout:5/488: fdatasync d2/d17/f51 0 2024-03-23T11:47:53.041 INFO:tasks.workunit.client.0.smithi063.stdout:2/600: mkdir d2/de/d22/dbc 0 2024-03-23T11:47:53.047 INFO:tasks.workunit.client.0.smithi063.stdout:6/576: mknod d1/dc/d21/cae 0 2024-03-23T11:47:53.047 INFO:tasks.workunit.client.0.smithi063.stdout:9/735: dread d4/f7b [0,4194304] 0 2024-03-23T11:47:53.047 INFO:tasks.workunit.client.0.smithi063.stdout:5/489: stat d2/d17/d4d/d6e 0 2024-03-23T11:47:53.048 INFO:tasks.workunit.client.0.smithi063.stdout:0/875: dwrite d2/d12/d32/fc0 [0,4194304] 0 2024-03-23T11:47:53.053 INFO:tasks.workunit.client.0.smithi063.stdout:7/694: dwrite d5/d26/db7/dce/d64/fb2 [0,4194304] 0 2024-03-23T11:47:53.056 INFO:tasks.workunit.client.0.smithi063.stdout:2/601: chown d2/d23/d2e/d52/f3e 1116704 1 2024-03-23T11:47:53.067 INFO:tasks.workunit.client.0.smithi063.stdout:9/736: mknod d4/d31/cfd 0 2024-03-23T11:47:53.069 INFO:tasks.workunit.client.0.smithi063.stdout:7/695: dread - d5/d26/db7/dc6/f97 zero size 2024-03-23T11:47:53.070 INFO:tasks.workunit.client.1.smithi111.stdout:8/969: truncate d5/d26/d2f/d62/d75/d4b/fb1 1179386 0 2024-03-23T11:47:53.072 INFO:tasks.workunit.client.0.smithi063.stdout:5/490: symlink d2/d7/d8/de/d2a/la8 0 2024-03-23T11:47:53.082 INFO:tasks.workunit.client.0.smithi063.stdout:4/740: mknod d5/da/d20/d4b/d29/d38/cd4 0 2024-03-23T11:47:53.084 INFO:tasks.workunit.client.0.smithi063.stdout:9/737: readlink d4/l8 0 2024-03-23T11:47:53.085 INFO:tasks.workunit.client.0.smithi063.stdout:7/696: symlink d5/d22/ld5 0 2024-03-23T11:47:53.087 INFO:tasks.workunit.client.0.smithi063.stdout:5/491: fsync d2/d7/d2b/f97 0 2024-03-23T11:47:53.091 INFO:tasks.workunit.client.0.smithi063.stdout:1/674: dread d3/df/f78 [0,4194304] 0 2024-03-23T11:47:53.093 INFO:tasks.workunit.client.1.smithi111.stdout:8/970: dwrite d5/d26/d31/fe6 [4194304,4194304] 0 2024-03-23T11:47:53.094 INFO:tasks.workunit.client.0.smithi063.stdout:6/577: write d1/d13/f16 [256683,121010] 0 2024-03-23T11:47:53.095 INFO:tasks.workunit.client.0.smithi063.stdout:0/876: link d2/d12/d32/d6d/d124/d1f/f9b d2/d48/f135 0 2024-03-23T11:47:53.096 INFO:tasks.workunit.client.1.smithi111.stdout:8/971: truncate d5/d1b/dbc/dc3/f13a 730858 0 2024-03-23T11:47:53.097 INFO:tasks.workunit.client.0.smithi063.stdout:7/697: rename d5/d6/d39 to d5/d26/db7/dd6 0 2024-03-23T11:47:53.102 INFO:tasks.workunit.client.0.smithi063.stdout:1/675: dread - d3/d35/f90 zero size 2024-03-23T11:47:53.106 INFO:tasks.workunit.client.0.smithi063.stdout:0/877: read d2/d6e/d8c/ff1 [369379,107451] 0 2024-03-23T11:47:53.106 INFO:tasks.workunit.client.1.smithi111.stdout:8/972: symlink d5/d26/d2f/d62/d136/l13c 0 2024-03-23T11:47:53.114 INFO:tasks.workunit.client.0.smithi063.stdout:2/602: dwrite d2/d23/d56/f75 [0,4194304] 0 2024-03-23T11:47:53.117 INFO:tasks.workunit.client.0.smithi063.stdout:6/578: unlink d1/l2b 0 2024-03-23T11:47:53.124 INFO:tasks.workunit.client.1.smithi111.stdout:8/973: readlink d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/df9/lfd 0 2024-03-23T11:47:53.124 INFO:tasks.workunit.client.0.smithi063.stdout:9/738: rename d4/d31/d43/da9/da6/db5/ld8 to d4/d31/d43/d49/dfa/lfe 0 2024-03-23T11:47:53.126 INFO:tasks.workunit.client.0.smithi063.stdout:5/492: symlink d2/d28/d3a/d46/da4/la9 0 2024-03-23T11:47:53.139 INFO:tasks.workunit.client.0.smithi063.stdout:4/741: dread d5/da/d20/d4b/d1f/f21 [0,4194304] 0 2024-03-23T11:47:53.149 INFO:tasks.workunit.client.0.smithi063.stdout:1/676: symlink d3/d49/d56/d65/d88/d92/d9e/lcf 0 2024-03-23T11:47:53.152 INFO:tasks.workunit.client.0.smithi063.stdout:5/493: symlink d2/d7/d8/de/d2a/d3b/d85/laa 0 2024-03-23T11:47:53.152 INFO:tasks.workunit.client.0.smithi063.stdout:7/698: dread d5/d22/f94 [0,4194304] 0 2024-03-23T11:47:53.153 INFO:tasks.workunit.client.0.smithi063.stdout:9/739: dwrite d4/d4c/fde [4194304,4194304] 0 2024-03-23T11:47:53.157 INFO:tasks.workunit.client.0.smithi063.stdout:0/878: link d2/d4/f8a d2/d14/f136 0 2024-03-23T11:47:53.160 INFO:tasks.workunit.client.1.smithi111.stdout:8/974: write d5/d26/d31/d3b/f89 [218829,124265] 0 2024-03-23T11:47:53.161 INFO:tasks.workunit.client.0.smithi063.stdout:1/677: rmdir d3/d49 39 2024-03-23T11:47:53.163 INFO:tasks.workunit.client.0.smithi063.stdout:2/603: write d2/de/f2b [532356,59045] 0 2024-03-23T11:47:53.174 INFO:tasks.workunit.client.0.smithi063.stdout:5/494: rename d2/d7/d8/de/f41 to d2/d7/d8/de/d47/fab 0 2024-03-23T11:47:53.177 INFO:tasks.workunit.client.0.smithi063.stdout:2/604: truncate d2/d84/f89 172192 0 2024-03-23T11:47:53.177 INFO:tasks.workunit.client.1.smithi111.stdout:8/975: dwrite d5/d26/d2f/d62/d7e/d85/f96 [0,4194304] 0 2024-03-23T11:47:53.178 INFO:tasks.workunit.client.0.smithi063.stdout:0/879: link d2/d14/d36/d77/d94/d129/f10f d2/d14/d36/d77/d94/daf/f137 0 2024-03-23T11:47:53.180 INFO:tasks.workunit.client.0.smithi063.stdout:6/579: dread d1/dc/d3a/d82/da2/f4a [0,4194304] 0 2024-03-23T11:47:53.180 INFO:tasks.workunit.client.0.smithi063.stdout:2/605: chown d2/f25 402 1 2024-03-23T11:47:53.182 INFO:tasks.workunit.client.0.smithi063.stdout:9/740: truncate d4/d31/d9c/d9e/fd4 985471 0 2024-03-23T11:47:53.183 INFO:tasks.workunit.client.0.smithi063.stdout:0/880: mknod d2/d14/d36/d77/deb/c138 0 2024-03-23T11:47:53.185 INFO:tasks.workunit.client.0.smithi063.stdout:6/580: chown d1/dc/d21/da5 128391008 1 2024-03-23T11:47:53.186 INFO:tasks.workunit.client.1.smithi111.stdout:8/976: link d5/d26/d2f/d62/d75/d8d/ccd d5/d32/d7d/d112/c13d 0 2024-03-23T11:47:53.189 INFO:tasks.workunit.client.0.smithi063.stdout:0/881: truncate d2/d14/d36/d77/f132 663659 0 2024-03-23T11:47:53.193 INFO:tasks.workunit.client.0.smithi063.stdout:1/678: link d3/df/d2f/f97 d3/fd0 0 2024-03-23T11:47:53.193 INFO:tasks.workunit.client.0.smithi063.stdout:4/742: dwrite d5/d1e/d2b/f35 [0,4194304] 0 2024-03-23T11:47:53.197 INFO:tasks.workunit.client.0.smithi063.stdout:1/679: readlink d3/df/d2f/l45 0 2024-03-23T11:47:53.202 INFO:tasks.workunit.client.0.smithi063.stdout:9/741: truncate d4/d31/d43/d49/f78 250155 0 2024-03-23T11:47:53.205 INFO:tasks.workunit.client.0.smithi063.stdout:2/606: creat d2/d1c/fbd x:0 0 0 2024-03-23T11:47:53.208 INFO:tasks.workunit.client.0.smithi063.stdout:7/699: dwrite d5/d26/db7/dc6/d4f/f93 [0,4194304] 0 2024-03-23T11:47:53.208 INFO:tasks.workunit.client.0.smithi063.stdout:0/882: creat d2/d6e/d8c/d127/f139 x:0 0 0 2024-03-23T11:47:53.213 INFO:tasks.workunit.client.0.smithi063.stdout:2/607: rmdir d2/de/d7a/d77 39 2024-03-23T11:47:53.219 INFO:tasks.workunit.client.0.smithi063.stdout:2/608: symlink d2/d84/lbe 0 2024-03-23T11:47:53.225 INFO:tasks.workunit.client.1.smithi111.stdout:8/977: dwrite d5/d26/d2f/d62/d75/d4b/d51/da4/f88 [0,4194304] 0 2024-03-23T11:47:53.227 INFO:tasks.workunit.client.0.smithi063.stdout:7/700: rename d5/d6/d19/l43 to d5/d5c/d2f/d73/d20/ld7 0 2024-03-23T11:47:53.234 INFO:tasks.workunit.client.0.smithi063.stdout:9/742: dwrite d4/d31/d9c/d9e/fc5 [0,4194304] 0 2024-03-23T11:47:53.234 INFO:tasks.workunit.client.0.smithi063.stdout:2/609: write d2/f29 [2084836,96997] 0 2024-03-23T11:47:53.234 INFO:tasks.workunit.client.0.smithi063.stdout:7/701: rename d5/d22 to d5/d22/da3/dd8 22 2024-03-23T11:47:53.238 INFO:tasks.workunit.client.0.smithi063.stdout:0/883: dread d2/d14/d36/d77/fbd [0,4194304] 0 2024-03-23T11:47:53.247 INFO:tasks.workunit.client.1.smithi111.stdout:8/978: write d5/d26/d31/fc7 [253787,56999] 0 2024-03-23T11:47:53.250 INFO:tasks.workunit.client.0.smithi063.stdout:2/610: unlink d2/d23/d2e/l42 0 2024-03-23T11:47:53.250 INFO:tasks.workunit.client.0.smithi063.stdout:9/743: truncate d4/fe 2846071 0 2024-03-23T11:47:53.252 INFO:tasks.workunit.client.1.smithi111.stdout:8/979: symlink d5/d26/d2f/d42/l13e 0 2024-03-23T11:47:53.256 INFO:tasks.workunit.client.0.smithi063.stdout:0/884: rmdir d2/d12/d32/d6d/d124/d25/d93/dd3 39 2024-03-23T11:47:53.256 INFO:tasks.workunit.client.0.smithi063.stdout:9/744: write d4/d4c/fde [9367166,118766] 0 2024-03-23T11:47:53.257 INFO:tasks.workunit.client.0.smithi063.stdout:6/581: dread d1/f8 [0,4194304] 0 2024-03-23T11:47:53.264 INFO:tasks.workunit.client.0.smithi063.stdout:4/743: dread d5/da/d20/d4b/d16/d32/d56/f85 [0,4194304] 0 2024-03-23T11:47:53.264 INFO:tasks.workunit.client.0.smithi063.stdout:1/680: dread d3/d35/d40/f4c [4194304,4194304] 0 2024-03-23T11:47:53.268 INFO:tasks.workunit.client.0.smithi063.stdout:6/582: mknod d1/dc/d21/caf 0 2024-03-23T11:47:53.268 INFO:tasks.workunit.client.0.smithi063.stdout:1/681: read d3/f11 [77605,81205] 0 2024-03-23T11:47:53.274 INFO:tasks.workunit.client.1.smithi111.stdout:8/980: dread d5/d26/d31/fb6 [0,4194304] 0 2024-03-23T11:47:53.276 INFO:tasks.workunit.client.0.smithi063.stdout:7/702: creat d5/d6/d19/d1f/d6c/fd9 x:0 0 0 2024-03-23T11:47:53.278 INFO:tasks.workunit.client.0.smithi063.stdout:4/744: write d5/da/d20/d4b/d16/f5c [716935,83712] 0 2024-03-23T11:47:53.281 INFO:tasks.workunit.client.1.smithi111.stdout:8/981: creat d5/d26/d2f/d62/d75/d4b/d51/da4/f13f x:0 0 0 2024-03-23T11:47:53.285 INFO:tasks.workunit.client.0.smithi063.stdout:4/745: dread - d5/da/d72/fd0 zero size 2024-03-23T11:47:53.286 INFO:tasks.workunit.client.0.smithi063.stdout:2/611: dwrite d2/d23/d2e/d87/f82 [4194304,4194304] 0 2024-03-23T11:47:53.286 INFO:tasks.workunit.client.0.smithi063.stdout:9/745: rename d4/d4c/d44/c8a to d4/cff 0 2024-03-23T11:47:53.288 INFO:tasks.workunit.client.0.smithi063.stdout:6/583: creat d1/dc/d3a/d82/da2/d41/d42/d99/fb0 x:0 0 0 2024-03-23T11:47:53.290 INFO:tasks.workunit.client.0.smithi063.stdout:2/612: chown d2/d84/d94 436718080 1 2024-03-23T11:47:53.291 INFO:tasks.workunit.client.0.smithi063.stdout:0/885: dwrite d2/d14/d36/d77/fe5 [0,4194304] 0 2024-03-23T11:47:53.292 INFO:tasks.workunit.client.1.smithi111.stdout:8/982: rename d5/d26/d2f/d42/d63/fab to d5/d26/d2f/d62/d7e/dae/d114/f140 0 2024-03-23T11:47:53.296 INFO:tasks.workunit.client.0.smithi063.stdout:0/886: chown d2/d6e/d8c/d127/f139 0 1 2024-03-23T11:47:53.300 INFO:tasks.workunit.client.0.smithi063.stdout:1/682: truncate d3/d49/d56/f7f 1130610 0 2024-03-23T11:47:53.301 INFO:tasks.workunit.client.1.smithi111.stdout:8/983: rmdir d5/d26/d2f/d42/dd3 39 2024-03-23T11:47:53.304 INFO:tasks.workunit.client.0.smithi063.stdout:4/746: stat d5/d1e/d2b/c87 0 2024-03-23T11:47:53.304 INFO:tasks.workunit.client.0.smithi063.stdout:1/683: chown d3/d49/d56/fb7 2 1 2024-03-23T11:47:53.306 INFO:tasks.workunit.client.1.smithi111.stdout:8/984: truncate d5/d26/d2f/d62/d75/d4b/d51/d5a/fdf 63918 0 2024-03-23T11:47:53.307 INFO:tasks.workunit.client.0.smithi063.stdout:2/613: mkdir d2/d23/d2e/d52/d86/dbf 0 2024-03-23T11:47:53.310 INFO:tasks.workunit.client.0.smithi063.stdout:6/584: mknod d1/dc/d21/cb1 0 2024-03-23T11:47:53.314 INFO:tasks.workunit.client.0.smithi063.stdout:2/614: creat d2/d46/db4/fc0 x:0 0 0 2024-03-23T11:47:53.315 INFO:tasks.workunit.client.0.smithi063.stdout:4/747: mkdir d5/da/d20/d4b/d29/d38/d7a/dd5 0 2024-03-23T11:47:53.316 INFO:tasks.workunit.client.0.smithi063.stdout:0/887: mkdir d2/d12/d13a 0 2024-03-23T11:47:53.318 INFO:tasks.workunit.client.0.smithi063.stdout:7/703: dwrite d5/d26/db7/dc6/d4f/d5b/f66 [0,4194304] 0 2024-03-23T11:47:53.320 INFO:tasks.workunit.client.1.smithi111.stdout:8/985: truncate d5/d32/d79/f104 2682269 0 2024-03-23T11:47:53.322 INFO:tasks.workunit.client.1.smithi111.stdout:8/986: write d5/d26/d31/d3b/f70 [3552790,65639] 0 2024-03-23T11:47:53.323 INFO:tasks.workunit.client.0.smithi063.stdout:9/746: dwrite d4/d31/d43/fdd [4194304,4194304] 0 2024-03-23T11:47:53.326 INFO:tasks.workunit.client.0.smithi063.stdout:6/585: symlink d1/dc/d3a/d57/d77/d79/lb2 0 2024-03-23T11:47:53.329 INFO:tasks.workunit.client.0.smithi063.stdout:2/615: unlink d2/d23/d2e/d52/l38 0 2024-03-23T11:47:53.329 INFO:tasks.workunit.client.0.smithi063.stdout:7/704: creat d5/d5c/d2f/fda x:0 0 0 2024-03-23T11:47:53.330 INFO:tasks.workunit.client.0.smithi063.stdout:0/888: creat d2/d12/d32/d6d/d124/d1f/d101/f13b x:0 0 0 2024-03-23T11:47:53.333 INFO:tasks.workunit.client.0.smithi063.stdout:6/586: symlink d1/dc/d21/lb3 0 2024-03-23T11:47:53.334 INFO:tasks.workunit.client.1.smithi111.stdout:8/987: creat d5/d26/f141 x:0 0 0 2024-03-23T11:47:53.335 INFO:tasks.workunit.client.0.smithi063.stdout:2/616: write d2/d23/d2e/d87/d4d/f79 [1209242,52375] 0 2024-03-23T11:47:53.339 INFO:tasks.workunit.client.1.smithi111.stdout:8/988: stat d5/d26/d2f/d42/dd3 0 2024-03-23T11:47:53.339 INFO:tasks.workunit.client.0.smithi063.stdout:6/587: read d1/d15/fa4 [2313,17779] 0 2024-03-23T11:47:53.343 INFO:tasks.workunit.client.0.smithi063.stdout:9/747: mkdir d4/d31/d43/da9/da6/d100 0 2024-03-23T11:47:53.346 INFO:tasks.workunit.client.0.smithi063.stdout:7/705: fdatasync d5/d22/f94 0 2024-03-23T11:47:53.346 INFO:tasks.workunit.client.1.smithi111.stdout:8/989: mknod d5/d26/d31/d4d/c142 0 2024-03-23T11:47:53.348 INFO:tasks.workunit.client.0.smithi063.stdout:9/748: write d4/d4c/fde [3645054,114454] 0 2024-03-23T11:47:53.349 INFO:tasks.workunit.client.0.smithi063.stdout:7/706: dread - d5/d5c/d2f/d8b/fd0 zero size 2024-03-23T11:47:53.351 INFO:tasks.workunit.client.0.smithi063.stdout:4/748: truncate d5/da/d20/fb7 321076 0 2024-03-23T11:47:53.353 INFO:tasks.workunit.client.1.smithi111.stdout:8/990: symlink d5/d26/d31/d3b/d55/dd0/l143 0 2024-03-23T11:47:53.354 INFO:tasks.workunit.client.0.smithi063.stdout:2/617: creat d2/d84/d92/fc1 x:0 0 0 2024-03-23T11:47:53.357 INFO:tasks.workunit.client.0.smithi063.stdout:6/588: rmdir d1/dc/d3a/d57/d77/d79 39 2024-03-23T11:47:53.362 INFO:tasks.workunit.client.0.smithi063.stdout:1/684: dread d3/d35/f93 [0,4194304] 0 2024-03-23T11:47:53.364 INFO:tasks.workunit.client.0.smithi063.stdout:7/707: rmdir d5 39 2024-03-23T11:47:53.366 INFO:tasks.workunit.client.0.smithi063.stdout:6/589: symlink d1/dc/d3a/d82/da2/d41/lb4 0 2024-03-23T11:47:53.368 INFO:tasks.workunit.client.1.smithi111.stdout:8/991: dwrite d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/f7f [0,4194304] 0 2024-03-23T11:47:53.369 INFO:tasks.workunit.client.0.smithi063.stdout:0/889: dread d2/d12/d32/d6d/d124/d1f/fba [0,4194304] 0 2024-03-23T11:47:53.370 INFO:tasks.workunit.client.0.smithi063.stdout:2/618: rmdir d2/d46/db4/db8 0 2024-03-23T11:47:53.371 INFO:tasks.workunit.client.0.smithi063.stdout:6/590: mkdir d1/dc/d8c/db5 0 2024-03-23T11:47:53.373 INFO:tasks.workunit.client.0.smithi063.stdout:2/619: chown d2/d46/db4/fc0 436746719 1 2024-03-23T11:47:53.373 INFO:tasks.workunit.client.0.smithi063.stdout:7/708: rmdir d5/d26/db7/dc6/d4f 39 2024-03-23T11:47:53.379 INFO:tasks.workunit.client.1.smithi111.stdout:8/992: link d5/d26/d31/d3b/f80 d5/d26/d2f/d62/d75/df5/f144 0 2024-03-23T11:47:53.380 INFO:tasks.workunit.client.0.smithi063.stdout:2/620: mkdir d2/d46/d54/dc2 0 2024-03-23T11:47:53.389 INFO:tasks.workunit.client.0.smithi063.stdout:9/749: dwrite d4/d31/f63 [0,4194304] 0 2024-03-23T11:47:53.390 INFO:tasks.workunit.client.1.smithi111.stdout:8/993: truncate d5/d26/d2f/d62/d75/d4b/fb1 357426 0 2024-03-23T11:47:53.393 INFO:tasks.workunit.client.0.smithi063.stdout:0/890: rename d2/d48/f10c to d2/d12/d32/d6d/d124/d25/f13c 0 2024-03-23T11:47:53.395 INFO:tasks.workunit.client.0.smithi063.stdout:2/621: symlink d2/d23/d2e/d52/lc3 0 2024-03-23T11:47:53.397 INFO:tasks.workunit.client.1.smithi111.stdout:8/994: rename d5/d26/d2f/d62/d75/d4b/d51/d5a/dbe to d5/d26/d2f/d62/d75/d4b/d51/d5a/d145 0 2024-03-23T11:47:53.397 INFO:tasks.workunit.client.0.smithi063.stdout:6/591: dwrite d1/f81 [0,4194304] 0 2024-03-23T11:47:53.406 INFO:tasks.workunit.client.1.smithi111.stdout:8/995: rmdir d5/d26/dd8 39 2024-03-23T11:47:53.409 INFO:tasks.workunit.client.1.smithi111.stdout:8/996: chown d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/d71/dd9 3514 1 2024-03-23T11:47:53.409 INFO:tasks.workunit.client.0.smithi063.stdout:2/622: mkdir d2/d23/d2e/d98/dc4 0 2024-03-23T11:47:53.411 INFO:tasks.workunit.client.0.smithi063.stdout:6/592: symlink d1/d15/d8b/lb6 0 2024-03-23T11:47:53.413 INFO:tasks.workunit.client.0.smithi063.stdout:7/709: rename d5/d26/db7/dc6/d4f/d5b/c5e to d5/d26/cdb 0 2024-03-23T11:47:53.413 INFO:tasks.workunit.client.0.smithi063.stdout:6/593: chown d1/dc/f91 19 1 2024-03-23T11:47:53.415 INFO:tasks.workunit.client.1.smithi111.stdout:8/997: mknod d5/d26/d2f/d62/db5/d108/c146 0 2024-03-23T11:47:53.416 INFO:tasks.workunit.client.0.smithi063.stdout:9/750: getdents d4/d31/d70/d8d/da0 0 2024-03-23T11:47:53.417 INFO:tasks.workunit.client.0.smithi063.stdout:2/623: mknod d2/d23/d2e/cc5 0 2024-03-23T11:47:53.418 INFO:tasks.workunit.client.0.smithi063.stdout:7/710: write d5/d26/db7/dc6/d4f/f93 [1106839,74208] 0 2024-03-23T11:47:53.420 INFO:tasks.workunit.client.0.smithi063.stdout:0/891: dread d2/d14/d36/d77/f80 [0,4194304] 0 2024-03-23T11:47:53.421 INFO:tasks.workunit.client.0.smithi063.stdout:6/594: symlink d1/d15/lb7 0 2024-03-23T11:47:53.421 INFO:tasks.workunit.client.0.smithi063.stdout:2/624: chown d2/de/d22/l48 1 1 2024-03-23T11:47:53.421 INFO:tasks.workunit.client.0.smithi063.stdout:9/751: mknod d4/d4c/d44/c101 0 2024-03-23T11:47:53.422 INFO:tasks.workunit.client.0.smithi063.stdout:7/711: chown d5/d5c/d2f/d73/da/d60/fa6 101004584 1 2024-03-23T11:47:53.424 INFO:tasks.workunit.client.0.smithi063.stdout:6/595: chown d1/l9 194730524 1 2024-03-23T11:47:53.429 INFO:tasks.workunit.client.0.smithi063.stdout:6/596: creat d1/d15/fb8 x:0 0 0 2024-03-23T11:47:53.434 INFO:tasks.workunit.client.0.smithi063.stdout:6/597: truncate d1/dc/d3a/f4f 446772 0 2024-03-23T11:47:53.438 INFO:tasks.workunit.client.0.smithi063.stdout:7/712: dread d5/d26/db7/dd6/f6a [0,4194304] 0 2024-03-23T11:47:53.438 INFO:tasks.workunit.client.0.smithi063.stdout:0/892: dread d2/d12/f7e [0,4194304] 0 2024-03-23T11:47:53.440 INFO:tasks.workunit.client.0.smithi063.stdout:0/893: fsync d2/d14/d36/ff4 0 2024-03-23T11:47:53.442 INFO:tasks.workunit.client.0.smithi063.stdout:7/713: unlink d5/c4a 0 2024-03-23T11:47:53.447 INFO:tasks.workunit.client.0.smithi063.stdout:7/714: dread d5/fc7 [0,4194304] 0 2024-03-23T11:47:53.450 INFO:tasks.workunit.client.0.smithi063.stdout:7/715: mknod d5/d6/d19/cdc 0 2024-03-23T11:47:53.452 INFO:tasks.workunit.client.0.smithi063.stdout:9/752: write d4/d4c/d19/f42 [3126805,98034] 0 2024-03-23T11:47:53.452 INFO:tasks.workunit.client.0.smithi063.stdout:2/625: write d2/d23/d2e/f35 [821460,32622] 0 2024-03-23T11:47:53.454 INFO:tasks.workunit.client.0.smithi063.stdout:7/716: fdatasync d5/d5c/d2f/d73/da/f17 0 2024-03-23T11:47:53.456 INFO:tasks.workunit.client.1.smithi111.stdout:8/998: dwrite d5/d32/d9b/fb2 [0,4194304] 0 2024-03-23T11:47:53.460 INFO:tasks.workunit.client.1.smithi111.stdout:8/999: stat d5/d26/d2f/d62/d75/d4b/d51/d5a/d6a/df9/cfe 0 2024-03-23T11:47:53.460 INFO:tasks.workunit.client.0.smithi063.stdout:2/626: mknod d2/d46/db4/cc6 0 2024-03-23T11:47:53.463 INFO:tasks.workunit.client.0.smithi063.stdout:9/753: write d4/d31/d43/d49/d8e/ff7 [138072,34818] 0 2024-03-23T11:47:53.464 INFO:tasks.workunit.client.0.smithi063.stdout:2/627: readlink d2/d23/d56/laa 0 2024-03-23T11:47:53.468 INFO:tasks.workunit.client.0.smithi063.stdout:6/598: dwrite d1/dc/d21/fa9 [0,4194304] 0 2024-03-23T11:47:53.469 INFO:tasks.workunit.client.0.smithi063.stdout:0/894: dwrite d2/d12/d32/d6d/d124/d28/d46/d51/f82 [0,4194304] 0 2024-03-23T11:47:53.470 INFO:tasks.workunit.client.0.smithi063.stdout:2/628: stat d2/d46/f70 0 2024-03-23T11:47:53.470 INFO:tasks.workunit.client.0.smithi063.stdout:6/599: stat d1 0 2024-03-23T11:47:53.473 INFO:tasks.workunit.client.0.smithi063.stdout:7/717: creat d5/d26/db7/dc6/d4f/d5b/dd3/fdd x:0 0 0 2024-03-23T11:47:53.474 INFO:tasks.workunit.client.1.smithi111.stderr:+ rm -rf -- ./tmp.v3ZauUBSr9 2024-03-23T11:47:53.475 INFO:tasks.workunit.client.0.smithi063.stdout:2/629: truncate d2/d1c/fbd 559377 0 2024-03-23T11:47:53.475 INFO:tasks.workunit.client.0.smithi063.stdout:9/754: symlink d4/d31/d43/da9/da6/dbe/l102 0 2024-03-23T11:47:53.477 INFO:tasks.workunit.client.0.smithi063.stdout:6/600: stat d1/d15/c18 0 2024-03-23T11:47:53.480 INFO:tasks.workunit.client.0.smithi063.stdout:2/630: write d2/d46/d54/fa3 [829285,66240] 0 2024-03-23T11:47:53.487 INFO:tasks.workunit.client.0.smithi063.stdout:0/895: rename d2/d14/d36/d77/d94/d129/c3d to d2/d12/d32/d6d/d124/d28/d46/d51/c13d 0 2024-03-23T11:47:53.496 INFO:tasks.workunit.client.0.smithi063.stdout:0/896: fdatasync d2/d14/de8/f8e 0 2024-03-23T11:47:53.499 INFO:tasks.workunit.client.0.smithi063.stdout:6/601: link d1/l36 d1/dc/d21/da5/lb9 0 2024-03-23T11:47:53.502 INFO:tasks.workunit.client.0.smithi063.stdout:2/631: dread d2/d23/d2e/d52/f71 [0,4194304] 0 2024-03-23T11:47:53.503 INFO:tasks.workunit.client.0.smithi063.stdout:0/897: creat d2/d12/d32/d6d/d124/d1f/d4f/f13e x:0 0 0 2024-03-23T11:47:53.504 INFO:tasks.workunit.client.0.smithi063.stdout:7/718: dwrite d5/d5c/d2f/fda [0,4194304] 0 2024-03-23T11:47:53.517 INFO:tasks.workunit.client.0.smithi063.stdout:6/602: creat d1/dc/d3a/d82/fba x:0 0 0 2024-03-23T11:47:53.525 INFO:tasks.workunit.client.0.smithi063.stdout:6/603: dread - d1/dc/d3a/d82/da2/f93 zero size 2024-03-23T11:47:53.527 INFO:tasks.workunit.client.0.smithi063.stdout:9/755: dwrite d4/d4c/d44/ff9 [0,4194304] 0 2024-03-23T11:47:53.527 INFO:tasks.workunit.client.0.smithi063.stdout:7/719: mkdir d5/d26/dde 0 2024-03-23T11:47:53.528 INFO:tasks.workunit.client.0.smithi063.stdout:0/898: link d2/d14/d36/d77/f132 d2/d12/d32/d6d/d124/d25/d93/dc9/ddd/f13f 0 2024-03-23T11:47:53.532 INFO:tasks.workunit.client.0.smithi063.stdout:6/604: fdatasync d1/d15/f26 0 2024-03-23T11:47:53.541 INFO:tasks.workunit.client.0.smithi063.stdout:2/632: dwrite d2/d1c/f20 [0,4194304] 0 2024-03-23T11:47:53.544 INFO:tasks.workunit.client.0.smithi063.stdout:7/720: truncate d5/d26/db7/dc6/d6e/f81 936113 0 2024-03-23T11:47:53.550 INFO:tasks.workunit.client.0.smithi063.stdout:2/633: creat d2/d23/d56/fc7 x:0 0 0 2024-03-23T11:47:53.553 INFO:tasks.workunit.client.0.smithi063.stdout:7/721: mknod d5/d26/db7/dce/d64/cdf 0 2024-03-23T11:47:53.556 INFO:tasks.workunit.client.0.smithi063.stdout:6/605: rename d1/d15/d8b to d1/dc/d3a/d57/d77/d79/dbb 0 2024-03-23T11:47:53.557 INFO:tasks.workunit.client.0.smithi063.stdout:2/634: mkdir d2/d23/d2e/d98/dc8 0 2024-03-23T11:47:53.558 INFO:tasks.workunit.client.0.smithi063.stdout:6/606: fsync d1/d15/fb8 0 2024-03-23T11:47:53.561 INFO:tasks.workunit.client.0.smithi063.stdout:6/607: chown d1/dc/d3a/d82/da2/l7e 284292344 1 2024-03-23T11:47:53.562 INFO:tasks.workunit.client.0.smithi063.stdout:7/722: creat d5/d22/dcc/fe0 x:0 0 0 2024-03-23T11:47:53.565 INFO:tasks.workunit.client.0.smithi063.stdout:0/899: truncate d2/d4/fd8 687551 0 2024-03-23T11:47:53.566 INFO:tasks.workunit.client.0.smithi063.stdout:7/723: readlink d5/d26/db7/dc6/d4f/l83 0 2024-03-23T11:47:53.567 INFO:tasks.workunit.client.0.smithi063.stdout:2/635: mknod d2/d23/d2e/d87/cc9 0 2024-03-23T11:47:53.570 INFO:tasks.workunit.client.0.smithi063.stdout:2/636: chown d2/d46/db4/lb6 3 1 2024-03-23T11:47:53.570 INFO:tasks.workunit.client.0.smithi063.stdout:9/756: dread d4/ff [0,4194304] 0 2024-03-23T11:47:53.571 INFO:tasks.workunit.client.0.smithi063.stdout:0/900: dread - d2/d12/d32/d6d/d124/d1f/d4f/f13e zero size 2024-03-23T11:47:53.576 INFO:tasks.workunit.client.0.smithi063.stdout:2/637: fsync d2/d23/d2e/d98/fba 0 2024-03-23T11:47:53.583 INFO:tasks.workunit.client.0.smithi063.stdout:6/608: dwrite d1/dc/d3a/d57/d6d/fa0 [0,4194304] 0 2024-03-23T11:47:53.583 INFO:tasks.workunit.client.0.smithi063.stdout:2/638: symlink d2/de/d7a/da2/lca 0 2024-03-23T11:47:53.586 INFO:tasks.workunit.client.0.smithi063.stdout:6/609: stat d1/dc/d3a/d57/d77/fad 0 2024-03-23T11:47:53.598 INFO:tasks.workunit.client.0.smithi063.stdout:7/724: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:53.600 INFO:tasks.workunit.client.0.smithi063.stdout:7/725: chown d5/d26/db7/dce/d64 0 1 2024-03-23T11:47:53.601 INFO:tasks.workunit.client.0.smithi063.stdout:9/757: write d4/d4c/d44/f7e [2975633,73570] 0 2024-03-23T11:47:53.602 INFO:tasks.workunit.client.0.smithi063.stdout:0/901: write d2/d12/f44 [5248443,58743] 0 2024-03-23T11:47:53.608 INFO:tasks.workunit.client.0.smithi063.stdout:9/758: fsync d4/d4c/f30 0 2024-03-23T11:47:53.617 INFO:tasks.workunit.client.0.smithi063.stdout:0/902: rename d2/d12/d32/d6d/d124/d1f/c40 to d2/d12/d32/d6d/d124/d25/c140 0 2024-03-23T11:47:53.620 INFO:tasks.workunit.client.0.smithi063.stdout:2/639: dwrite d2/d84/f89 [0,4194304] 0 2024-03-23T11:47:53.620 INFO:tasks.workunit.client.0.smithi063.stdout:6/610: dread d1/dc/d3a/f40 [0,4194304] 0 2024-03-23T11:47:53.622 INFO:tasks.workunit.client.0.smithi063.stdout:7/726: truncate d5/d6/f25 7071857 0 2024-03-23T11:47:53.629 INFO:tasks.workunit.client.0.smithi063.stdout:0/903: creat d2/d12/d32/d6d/d124/d28/d46/d51/f141 x:0 0 0 2024-03-23T11:47:53.633 INFO:tasks.workunit.client.0.smithi063.stdout:9/759: getdents d4/d31/d9c/d9e 0 2024-03-23T11:47:53.634 INFO:tasks.workunit.client.0.smithi063.stdout:2/640: rmdir d2/d23/d2e/d87 39 2024-03-23T11:47:53.635 INFO:tasks.workunit.client.0.smithi063.stdout:7/727: mkdir d5/d5c/d2f/d8b/daa/de1 0 2024-03-23T11:47:53.641 INFO:tasks.workunit.client.0.smithi063.stdout:6/611: dread d1/d13/f53 [0,4194304] 0 2024-03-23T11:47:53.643 INFO:tasks.workunit.client.0.smithi063.stdout:9/760: rename d4/d4c/c73 to d4/d4c/df4/c103 0 2024-03-23T11:47:53.644 INFO:tasks.workunit.client.0.smithi063.stdout:6/612: chown d1/dc/fab 2083535 1 2024-03-23T11:47:53.647 INFO:tasks.workunit.client.0.smithi063.stdout:2/641: fdatasync d2/d23/d2e/d87/f50 0 2024-03-23T11:47:53.648 INFO:tasks.workunit.client.0.smithi063.stdout:6/613: mknod d1/dc/d3a/d57/d6d/cbc 0 2024-03-23T11:47:53.654 INFO:tasks.workunit.client.0.smithi063.stdout:0/904: dwrite d2/d48/f7f [0,4194304] 0 2024-03-23T11:47:53.656 INFO:tasks.workunit.client.0.smithi063.stdout:9/761: link d4/fc4 d4/d31/d43/d49/f104 0 2024-03-23T11:47:53.657 INFO:tasks.workunit.client.0.smithi063.stdout:6/614: symlink d1/dc/d3a/d82/d85/lbd 0 2024-03-23T11:47:53.659 INFO:tasks.workunit.client.0.smithi063.stdout:9/762: write d4/d31/d9c/ffb [408258,97267] 0 2024-03-23T11:47:53.659 INFO:tasks.workunit.client.0.smithi063.stdout:0/905: fdatasync d2/d14/d36/d77/f132 0 2024-03-23T11:47:53.662 INFO:tasks.workunit.client.0.smithi063.stdout:7/728: dread d5/d26/d76/f78 [0,4194304] 0 2024-03-23T11:47:53.664 INFO:tasks.workunit.client.0.smithi063.stdout:6/615: write d1/dc/d3a/d57/d6d/fa0 [755237,52800] 0 2024-03-23T11:47:53.672 INFO:tasks.workunit.client.0.smithi063.stdout:7/729: creat d5/d5c/d2f/d8b/fe2 x:0 0 0 2024-03-23T11:47:53.674 INFO:tasks.workunit.client.0.smithi063.stdout:2/642: dwrite d2/d84/d92/fb0 [0,4194304] 0 2024-03-23T11:47:53.675 INFO:tasks.workunit.client.0.smithi063.stdout:7/730: write f1 [5596511,46981] 0 2024-03-23T11:47:53.684 INFO:tasks.workunit.client.0.smithi063.stdout:2/643: symlink d2/d46/db4/lcb 0 2024-03-23T11:47:53.684 INFO:tasks.workunit.client.0.smithi063.stdout:0/906: dwrite d2/d6e/f11e [0,4194304] 0 2024-03-23T11:47:53.687 INFO:tasks.workunit.client.0.smithi063.stdout:9/763: write d4/d31/d43/da9/da6/dbe/fd2 [129239,129433] 0 2024-03-23T11:47:53.688 INFO:tasks.workunit.client.0.smithi063.stdout:0/907: symlink d2/d14/de8/ddb/l142 0 2024-03-23T11:47:53.690 INFO:tasks.workunit.client.0.smithi063.stdout:2/644: mkdir d2/d23/d2e/d98/dc4/dcc 0 2024-03-23T11:47:53.692 INFO:tasks.workunit.client.0.smithi063.stdout:6/616: write d1/dc/d3a/d82/da2/d41/d42/f48 [1155888,120081] 0 2024-03-23T11:47:53.695 INFO:tasks.workunit.client.0.smithi063.stdout:7/731: write d5/d6/d19/f7d [401803,80102] 0 2024-03-23T11:47:53.696 INFO:tasks.workunit.client.0.smithi063.stdout:2/645: creat d2/d23/d2e/d87/d4d/fcd x:0 0 0 2024-03-23T11:47:53.698 INFO:tasks.workunit.client.0.smithi063.stdout:6/617: write d1/dc/d8c/f96 [15940,111613] 0 2024-03-23T11:47:53.703 INFO:tasks.workunit.client.0.smithi063.stdout:9/764: write d4/d4c/d44/f84 [2946422,104688] 0 2024-03-23T11:47:53.706 INFO:tasks.workunit.client.0.smithi063.stdout:7/732: symlink d5/d5c/d2f/db5/dd2/le3 0 2024-03-23T11:47:53.707 INFO:tasks.workunit.client.0.smithi063.stdout:9/765: chown d4/d4c/d44/f6b 10232 1 2024-03-23T11:47:53.707 INFO:tasks.workunit.client.0.smithi063.stdout:6/618: symlink d1/dc/d3a/d82/d85/lbe 0 2024-03-23T11:47:53.708 INFO:tasks.workunit.client.0.smithi063.stdout:2/646: truncate d2/d46/d54/f68 590912 0 2024-03-23T11:47:53.715 INFO:tasks.workunit.client.0.smithi063.stdout:9/766: write d4/d4c/f25 [2660999,13989] 0 2024-03-23T11:47:53.717 INFO:tasks.workunit.client.0.smithi063.stdout:2/647: creat d2/d23/fce x:0 0 0 2024-03-23T11:47:53.719 INFO:tasks.workunit.client.0.smithi063.stdout:9/767: truncate d4/ff 5302192 0 2024-03-23T11:47:53.720 INFO:tasks.workunit.client.0.smithi063.stdout:0/908: dwrite d2/d14/f1c [0,4194304] 0 2024-03-23T11:47:53.724 INFO:tasks.workunit.client.0.smithi063.stdout:9/768: read - d4/d31/d43/da9/da6/ffc zero size 2024-03-23T11:47:53.733 INFO:tasks.workunit.client.0.smithi063.stdout:0/909: mknod d2/d12/d32/d6d/d124/d25/c143 0 2024-03-23T11:47:53.736 INFO:tasks.workunit.client.0.smithi063.stdout:2/648: fsync d2/de/d7a/d77/f93 0 2024-03-23T11:47:53.737 INFO:tasks.workunit.client.0.smithi063.stdout:6/619: dwrite d1/dc/d3a/d57/d6d/fa8 [0,4194304] 0 2024-03-23T11:47:53.741 INFO:tasks.workunit.client.0.smithi063.stdout:0/910: stat d2/d12/d32/d6d/d124/d25/d93/dc9/ddd/f13f 0 2024-03-23T11:47:53.742 INFO:tasks.workunit.client.0.smithi063.stdout:7/733: link d5/d5c/d2f/d73/da/cf d5/d26/db7/dc6/d6e/da2/ce4 0 2024-03-23T11:47:53.748 INFO:tasks.workunit.client.0.smithi063.stdout:9/769: creat d4/dcd/f105 x:0 0 0 2024-03-23T11:47:53.749 INFO:tasks.workunit.client.0.smithi063.stdout:2/649: fsync d2/de/d22/f78 0 2024-03-23T11:47:53.751 INFO:tasks.workunit.client.0.smithi063.stdout:6/620: symlink d1/d15/d4b/lbf 0 2024-03-23T11:47:53.753 INFO:tasks.workunit.client.0.smithi063.stdout:0/911: fsync d2/d14/d36/fda 0 2024-03-23T11:47:53.754 INFO:tasks.workunit.client.0.smithi063.stdout:2/650: write d2/d23/d56/fc7 [352836,14736] 0 2024-03-23T11:47:53.756 INFO:tasks.workunit.client.0.smithi063.stdout:0/912: stat d2/d14/de8/ddb/l142 0 2024-03-23T11:47:53.757 INFO:tasks.workunit.client.0.smithi063.stdout:2/651: chown d2/cd 43131 1 2024-03-23T11:47:53.759 INFO:tasks.workunit.client.0.smithi063.stdout:9/770: rename d4/d4c/d44/c4b to d4/d31/d70/d8d/c106 0 2024-03-23T11:47:53.764 INFO:tasks.workunit.client.0.smithi063.stdout:9/771: dread d4/d31/d9c/ffb [0,4194304] 0 2024-03-23T11:47:53.766 INFO:tasks.workunit.client.0.smithi063.stdout:0/913: rename d2/d14/d36/ff4 to d2/d4/f144 0 2024-03-23T11:47:53.768 INFO:tasks.workunit.client.0.smithi063.stdout:9/772: creat d4/d31/d43/d49/d8e/f107 x:0 0 0 2024-03-23T11:47:53.772 INFO:tasks.workunit.client.0.smithi063.stdout:2/652: rename d2/l8 to d2/d84/d94/lcf 0 2024-03-23T11:47:53.772 INFO:tasks.workunit.client.0.smithi063.stdout:6/621: dwrite d1/dc/d3a/d57/d6d/fa0 [0,4194304] 0 2024-03-23T11:47:53.784 INFO:tasks.workunit.client.0.smithi063.stdout:0/914: write d2/d4/f30 [4464864,37966] 0 2024-03-23T11:47:53.787 INFO:tasks.workunit.client.0.smithi063.stdout:0/915: write d2/d6e/f11e [3159785,115709] 0 2024-03-23T11:47:53.789 INFO:tasks.workunit.client.0.smithi063.stdout:6/622: creat d1/dc/d3a/d57/d6d/fc0 x:0 0 0 2024-03-23T11:47:53.796 INFO:tasks.workunit.client.0.smithi063.stdout:2/653: rename d2/d23/f41 to d2/d23/d2e/d52/fd0 0 2024-03-23T11:47:53.838 INFO:tasks.workunit.client.0.smithi063.stdout:0/916: creat d2/d12/d32/d6d/d124/d28/d46/d51/f145 x:0 0 0 2024-03-23T11:47:53.850 INFO:tasks.workunit.client.0.smithi063.stdout:9/773: dwrite d4/d31/d70/d8d/db3/fe0 [0,4194304] 0 2024-03-23T11:47:53.855 INFO:tasks.workunit.client.0.smithi063.stdout:6/623: mkdir d1/dc/d3a/d57/d77/dc1 0 2024-03-23T11:47:53.861 INFO:tasks.workunit.client.0.smithi063.stdout:2/654: mknod d2/d23/d2e/d52/d86/dbf/cd1 0 2024-03-23T11:47:53.867 INFO:tasks.workunit.client.0.smithi063.stdout:2/655: read - d2/f6b zero size 2024-03-23T11:47:53.868 INFO:tasks.workunit.client.0.smithi063.stdout:2/656: chown d2/d1c 0 1 2024-03-23T11:47:53.870 INFO:tasks.workunit.client.0.smithi063.stdout:9/774: mkdir d4/d31/d70/d8d/d90/d108 0 2024-03-23T11:47:53.874 INFO:tasks.workunit.client.0.smithi063.stdout:2/657: truncate d2/d23/d56/fa4 877801 0 2024-03-23T11:47:53.877 INFO:tasks.workunit.client.0.smithi063.stdout:0/917: symlink d2/d12/d32/d6d/d124/d28/l146 0 2024-03-23T11:47:53.877 INFO:tasks.workunit.client.0.smithi063.stdout:2/658: truncate d2/f85 686074 0 2024-03-23T11:47:53.880 INFO:tasks.workunit.client.0.smithi063.stdout:2/659: mkdir d2/d23/d2e/d98/dd2 0 2024-03-23T11:47:53.882 INFO:tasks.workunit.client.0.smithi063.stdout:0/918: creat d2/d14/d36/d77/d94/daf/f147 x:0 0 0 2024-03-23T11:47:53.883 INFO:tasks.workunit.client.0.smithi063.stdout:2/660: unlink d2/d46/f70 0 2024-03-23T11:47:53.889 INFO:tasks.workunit.client.0.smithi063.stdout:2/661: symlink d2/d23/d2e/ld3 0 2024-03-23T11:47:53.907 INFO:tasks.workunit.client.0.smithi063.stdout:6/624: write d1/d15/f88 [984215,3224] 0 2024-03-23T11:47:53.908 INFO:tasks.workunit.client.0.smithi063.stdout:9/775: rmdir d4/d31/d70/d8d 39 2024-03-23T11:47:53.913 INFO:tasks.workunit.client.0.smithi063.stdout:9/776: mkdir d4/d31/d43/da9/da6/db5/d109 0 2024-03-23T11:47:53.917 INFO:tasks.workunit.client.0.smithi063.stdout:2/662: dread d2/f18 [0,4194304] 0 2024-03-23T11:47:53.921 INFO:tasks.workunit.client.0.smithi063.stdout:0/919: truncate d2/d14/de8/fb6 3448863 0 2024-03-23T11:47:53.926 INFO:tasks.workunit.client.0.smithi063.stdout:6/625: write d1/dc/d3a/d82/da2/f83 [321266,34136] 0 2024-03-23T11:47:53.928 INFO:tasks.workunit.client.0.smithi063.stdout:9/777: write d4/f7c [520638,76201] 0 2024-03-23T11:47:53.928 INFO:tasks.workunit.client.0.smithi063.stdout:6/626: chown d1/dc/d21/f3c 6 1 2024-03-23T11:47:53.941 INFO:tasks.workunit.client.0.smithi063.stdout:2/663: dwrite d2/f2f [0,4194304] 0 2024-03-23T11:47:53.942 INFO:tasks.workunit.client.0.smithi063.stdout:0/920: dwrite d2/d12/d32/f43 [4194304,4194304] 0 2024-03-23T11:47:53.946 INFO:tasks.workunit.client.0.smithi063.stdout:9/778: link d4/d31/d43/fdd d4/d4c/df4/f10a 0 2024-03-23T11:47:53.955 INFO:tasks.workunit.client.0.smithi063.stdout:2/664: mkdir d2/d23/d56/dad/dd4 0 2024-03-23T11:47:53.961 INFO:tasks.workunit.client.0.smithi063.stdout:2/665: unlink d2/lbb 0 2024-03-23T11:47:53.967 INFO:tasks.workunit.client.0.smithi063.stdout:0/921: rename d2/f56 to d2/d48/da5/f148 0 2024-03-23T11:47:53.969 INFO:tasks.workunit.client.0.smithi063.stdout:0/922: creat d2/d14/de8/ddb/f149 x:0 0 0 2024-03-23T11:47:53.975 INFO:tasks.workunit.client.0.smithi063.stdout:0/923: truncate d2/d14/de8/f8e 2421264 0 2024-03-23T11:47:53.976 INFO:tasks.workunit.client.0.smithi063.stdout:9/779: dwrite d4/d31/d43/da9/f80 [8388608,4194304] 0 2024-03-23T11:47:53.977 INFO:tasks.workunit.client.0.smithi063.stdout:0/924: fdatasync d2/d4/f30 0 2024-03-23T11:47:53.982 INFO:tasks.workunit.client.0.smithi063.stdout:2/666: dwrite d2/d23/d2e/d87/d4d/f79 [0,4194304] 0 2024-03-23T11:47:53.986 INFO:tasks.workunit.client.0.smithi063.stdout:0/925: truncate d2/d14/d36/d77/d94/daf/fd2 3308245 0 2024-03-23T11:47:53.986 INFO:tasks.workunit.client.0.smithi063.stdout:2/667: mknod d2/de/d7a/d77/cd5 0 2024-03-23T11:47:53.991 INFO:tasks.workunit.client.0.smithi063.stdout:0/926: creat d2/d12/d32/d6d/d124/d1f/d101/f14a x:0 0 0 2024-03-23T11:47:53.992 INFO:tasks.workunit.client.0.smithi063.stdout:2/668: mkdir d2/d23/d56/dad/dd4/dd6 0 2024-03-23T11:47:53.994 INFO:tasks.workunit.client.0.smithi063.stdout:9/780: write d4/d4c/f30 [2608537,129790] 0 2024-03-23T11:47:53.996 INFO:tasks.workunit.client.0.smithi063.stdout:2/669: chown d2/d23/d2e/d87/d4d/l95 30403663 1 2024-03-23T11:47:54.001 INFO:tasks.workunit.client.0.smithi063.stdout:2/670: rename d2/d23/c72 to d2/de/d7a/d77/cd7 0 2024-03-23T11:47:54.007 INFO:tasks.workunit.client.0.smithi063.stdout:2/671: rename d2/d23/d2e/d87/d4d/f6d to d2/d23/d56/dad/fd8 0 2024-03-23T11:47:54.017 INFO:tasks.workunit.client.0.smithi063.stdout:9/781: write d4/d31/d43/fdd [3457125,1302] 0 2024-03-23T11:47:54.026 INFO:tasks.workunit.client.0.smithi063.stdout:0/927: dwrite d2/d12/d32/d6d/d124/d1f/ffa [0,4194304] 0 2024-03-23T11:47:54.036 INFO:tasks.workunit.client.0.smithi063.stdout:0/928: creat d2/d12/d32/d6d/d124/d1f/d4f/d126/f14b x:0 0 0 2024-03-23T11:47:54.039 INFO:tasks.workunit.client.0.smithi063.stdout:2/672: dwrite d2/d23/d2e/d52/f7d [0,4194304] 0 2024-03-23T11:47:54.044 INFO:tasks.workunit.client.0.smithi063.stdout:0/929: mkdir d2/d12/d32/d6d/d124/d1f/d101/d14c 0 2024-03-23T11:47:54.044 INFO:tasks.workunit.client.0.smithi063.stdout:9/782: dwrite d4/d31/d43/d49/d8e/fbc [0,4194304] 0 2024-03-23T11:47:54.046 INFO:tasks.workunit.client.0.smithi063.stdout:2/673: rmdir d2/de/d7a/da2 39 2024-03-23T11:47:54.049 INFO:tasks.workunit.client.0.smithi063.stdout:9/783: mkdir d4/d31/d10b 0 2024-03-23T11:47:54.051 INFO:tasks.workunit.client.0.smithi063.stdout:0/930: creat d2/d12/f14d x:0 0 0 2024-03-23T11:47:54.058 INFO:tasks.workunit.client.0.smithi063.stdout:9/784: dread d4/f5 [4194304,4194304] 0 2024-03-23T11:47:54.058 INFO:tasks.workunit.client.0.smithi063.stdout:0/931: creat d2/d48/de2/f14e x:0 0 0 2024-03-23T11:47:54.064 INFO:tasks.workunit.client.0.smithi063.stdout:2/674: rename d2/d23/d2e/d52/f6f to d2/d23/d56/dad/dd4/fd9 0 2024-03-23T11:47:54.066 INFO:tasks.workunit.client.0.smithi063.stdout:0/932: mknod d2/d6e/c14f 0 2024-03-23T11:47:54.069 INFO:tasks.workunit.client.0.smithi063.stdout:9/785: symlink d4/d31/d43/da9/da6/d100/l10c 0 2024-03-23T11:47:54.076 INFO:tasks.workunit.client.0.smithi063.stdout:2/675: dread d2/d1c/f20 [0,4194304] 0 2024-03-23T11:47:54.076 INFO:tasks.workunit.client.0.smithi063.stdout:9/786: creat d4/d31/d43/f10d x:0 0 0 2024-03-23T11:47:54.078 INFO:tasks.workunit.client.0.smithi063.stdout:2/676: chown d2/de/l26 1 1 2024-03-23T11:47:54.084 INFO:tasks.workunit.client.0.smithi063.stdout:0/933: dwrite d2/d12/f44 [0,4194304] 0 2024-03-23T11:47:54.099 INFO:tasks.workunit.client.0.smithi063.stdout:2/677: write d2/d23/d2e/d87/f8a [199950,16703] 0 2024-03-23T11:47:54.108 INFO:tasks.workunit.client.0.smithi063.stdout:9/787: dwrite d4/d31/d43/d49/f78 [0,4194304] 0 2024-03-23T11:47:54.110 INFO:tasks.workunit.client.0.smithi063.stdout:2/678: rename d2/d1c/f66 to d2/d23/d2e/d52/fda 0 2024-03-23T11:47:54.111 INFO:tasks.workunit.client.0.smithi063.stdout:9/788: fdatasync d4/d31/d43/f7d 0 2024-03-23T11:47:54.119 INFO:tasks.workunit.client.0.smithi063.stdout:9/789: creat d4/d31/d70/f10e x:0 0 0 2024-03-23T11:47:54.121 INFO:tasks.workunit.client.0.smithi063.stdout:2/679: dread d2/de/d7a/d8d/fa9 [0,4194304] 0 2024-03-23T11:47:54.125 INFO:tasks.workunit.client.0.smithi063.stdout:9/790: read d4/d4c/d44/f6b [3479759,24777] 0 2024-03-23T11:47:54.126 INFO:tasks.workunit.client.0.smithi063.stdout:0/934: dread d2/d14/d36/d77/d94/d129/f81 [0,4194304] 0 2024-03-23T11:47:54.128 INFO:tasks.workunit.client.0.smithi063.stdout:2/680: unlink d2/de/f1b 0 2024-03-23T11:47:54.129 INFO:tasks.workunit.client.0.smithi063.stdout:0/935: write d2/d6e/d8c/d127/f139 [1046505,48012] 0 2024-03-23T11:47:54.132 INFO:tasks.workunit.client.0.smithi063.stdout:2/681: chown d2/d23/d2e/c60 3969980 1 2024-03-23T11:47:54.133 INFO:tasks.workunit.client.0.smithi063.stdout:0/936: chown d2/d12/d32/d6d/d124/d25/c105 781499354 1 2024-03-23T11:47:54.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:53 smithi063 ceph-mon[35189]: pgmap v46: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 87 MiB/s rd, 200 MiB/s wr, 477 op/s 2024-03-23T11:47:54.134 INFO:tasks.workunit.client.0.smithi063.stdout:9/791: fdatasync d4/d31/d70/d8d/fc6 0 2024-03-23T11:47:54.137 INFO:tasks.workunit.client.0.smithi063.stdout:9/792: truncate d4/f5a 1279770 0 2024-03-23T11:47:54.139 INFO:tasks.workunit.client.0.smithi063.stdout:2/682: rename d2/d46/db4/fc0 to d2/de/d22/dbc/fdb 0 2024-03-23T11:47:54.140 INFO:tasks.workunit.client.0.smithi063.stdout:9/793: read d4/d4c/f25 [2960518,62873] 0 2024-03-23T11:47:54.141 INFO:tasks.workunit.client.0.smithi063.stdout:2/683: readlink d2/d23/l43 0 2024-03-23T11:47:54.142 INFO:tasks.workunit.client.0.smithi063.stdout:2/684: fdatasync d2/f29 0 2024-03-23T11:47:54.145 INFO:tasks.workunit.client.0.smithi063.stdout:0/937: rename d2/d14/d36/d77/f120 to d2/d48/da5/da7/f150 0 2024-03-23T11:47:54.146 INFO:tasks.workunit.client.0.smithi063.stdout:9/794: mknod d4/d31/c10f 0 2024-03-23T11:47:54.149 INFO:tasks.workunit.client.0.smithi063.stdout:9/795: stat d4/d31/l4e 0 2024-03-23T11:47:54.149 INFO:tasks.workunit.client.0.smithi063.stdout:9/796: readlink d4/d31/d43/da9/laa 0 2024-03-23T11:47:54.151 INFO:tasks.workunit.client.0.smithi063.stdout:0/938: mkdir d2/d6e/d115/d151 0 2024-03-23T11:47:54.151 INFO:tasks.workunit.client.0.smithi063.stdout:9/797: chown d4/fba 0 1 2024-03-23T11:47:54.152 INFO:tasks.workunit.client.0.smithi063.stdout:0/939: readlink d2/d48/de2/l10a 0 2024-03-23T11:47:54.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:53 smithi111 ceph-mon[44005]: pgmap v46: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 87 MiB/s rd, 200 MiB/s wr, 477 op/s 2024-03-23T11:47:54.154 INFO:tasks.workunit.client.0.smithi063.stdout:2/685: creat d2/fdc x:0 0 0 2024-03-23T11:47:54.155 INFO:tasks.workunit.client.0.smithi063.stdout:0/940: chown d2/d48/d110/l12a 1705874882 1 2024-03-23T11:47:54.155 INFO:tasks.workunit.client.0.smithi063.stdout:9/798: chown d4/d31/d43/da9/da6/dbe/fe4 61910065 1 2024-03-23T11:47:54.156 INFO:tasks.workunit.client.0.smithi063.stdout:2/686: fsync d2/d23/d2e/d98/faf 0 2024-03-23T11:47:54.161 INFO:tasks.workunit.client.0.smithi063.stdout:0/941: dread d2/d12/d32/d6d/d124/d25/f122 [0,4194304] 0 2024-03-23T11:47:54.167 INFO:tasks.workunit.client.0.smithi063.stdout:1/685: sync 2024-03-23T11:47:54.167 INFO:tasks.workunit.client.0.smithi063.stdout:5/495: sync 2024-03-23T11:47:54.167 INFO:tasks.workunit.client.0.smithi063.stdout:4/749: sync 2024-03-23T11:47:54.177 INFO:tasks.workunit.client.0.smithi063.stdout:2/687: dread d2/d1c/f20 [0,4194304] 0 2024-03-23T11:47:54.181 INFO:tasks.workunit.client.0.smithi063.stdout:1/686: truncate d3/df/d2f/f97 1409852 0 2024-03-23T11:47:54.183 INFO:tasks.workunit.client.0.smithi063.stdout:9/799: dwrite d4/d31/f63 [0,4194304] 0 2024-03-23T11:47:54.194 INFO:tasks.workunit.client.0.smithi063.stdout:9/800: creat d4/d31/d43/d49/d8e/f110 x:0 0 0 2024-03-23T11:47:54.197 INFO:tasks.workunit.client.0.smithi063.stdout:0/942: dread d2/d14/d36/d77/f7c [0,4194304] 0 2024-03-23T11:47:54.199 INFO:tasks.workunit.client.0.smithi063.stdout:5/496: dwrite d2/d7/d8/de/d2a/d3b/f99 [0,4194304] 0 2024-03-23T11:47:54.203 INFO:tasks.workunit.client.0.smithi063.stdout:4/750: getdents d5/da/d20/d4b/d16/d50/d5d 0 2024-03-23T11:47:54.212 INFO:tasks.workunit.client.0.smithi063.stdout:4/751: mknod d5/d1e/d2b/daa/cd6 0 2024-03-23T11:47:54.217 INFO:tasks.workunit.client.0.smithi063.stdout:0/943: write d2/d12/d32/d6d/d124/d25/d93/fbf [4175832,60468] 0 2024-03-23T11:47:54.220 INFO:tasks.workunit.client.0.smithi063.stdout:5/497: rmdir d2/d17/da3 0 2024-03-23T11:47:54.221 INFO:tasks.workunit.client.0.smithi063.stdout:9/801: write d4/d4c/d19/f27 [670,23719] 0 2024-03-23T11:47:54.222 INFO:tasks.workunit.client.0.smithi063.stdout:1/687: dwrite d3/f4 [4194304,4194304] 0 2024-03-23T11:47:54.227 INFO:tasks.workunit.client.0.smithi063.stdout:5/498: rmdir d2/d17/d4d 39 2024-03-23T11:47:54.229 INFO:tasks.workunit.client.0.smithi063.stdout:9/802: write d4/d31/d43/da9/da6/dbe/fd2 [159872,22920] 0 2024-03-23T11:47:54.230 INFO:tasks.workunit.client.0.smithi063.stdout:4/752: rename d5/da/d20/d4b/d16/d50/d8e/d96/dc0/dd2 to d5/da/d20/d4b/d16/d50/d86/da2/dd7 0 2024-03-23T11:47:54.231 INFO:tasks.workunit.client.0.smithi063.stdout:9/803: fdatasync d4/f7c 0 2024-03-23T11:47:54.235 INFO:tasks.workunit.client.0.smithi063.stdout:0/944: link d2/d14/d36/d77/d94/d129/ld7 d2/d6e/d117/l152 0 2024-03-23T11:47:54.238 INFO:tasks.workunit.client.0.smithi063.stdout:1/688: symlink d3/d49/d56/d65/d84/da7/ld1 0 2024-03-23T11:47:54.240 INFO:tasks.workunit.client.0.smithi063.stdout:5/499: link d2/d7/d8/de/d2a/d3b/d43/f45 d2/d28/fac 0 2024-03-23T11:47:54.242 INFO:tasks.workunit.client.0.smithi063.stdout:1/689: symlink d3/d49/d56/d65/d88/d92/d9e/ld2 0 2024-03-23T11:47:54.244 INFO:tasks.workunit.client.0.smithi063.stdout:1/690: read d3/d1c/da8/faf [65842,58610] 0 2024-03-23T11:47:54.248 INFO:tasks.workunit.client.0.smithi063.stdout:5/500: creat d2/fad x:0 0 0 2024-03-23T11:47:54.250 INFO:tasks.workunit.client.0.smithi063.stdout:1/691: mkdir d3/d1c/d4b/dd3 0 2024-03-23T11:47:54.251 INFO:tasks.workunit.client.0.smithi063.stdout:4/753: dwrite d5/da/d20/d4b/d1f/f21 [0,4194304] 0 2024-03-23T11:47:54.254 INFO:tasks.workunit.client.0.smithi063.stdout:5/501: rename c0 to d2/d7/d8/de/d8f/cae 0 2024-03-23T11:47:54.256 INFO:tasks.workunit.client.0.smithi063.stdout:1/692: creat d3/d1c/d4b/db5/fd4 x:0 0 0 2024-03-23T11:47:54.258 INFO:tasks.workunit.client.0.smithi063.stdout:1/693: fsync d3/df/d2f/fcc 0 2024-03-23T11:47:54.259 INFO:tasks.workunit.client.0.smithi063.stdout:4/754: fsync d5/da/d20/d4b/d16/d37/f57 0 2024-03-23T11:47:54.260 INFO:tasks.workunit.client.0.smithi063.stdout:5/502: mkdir d2/d7/d2b/d8b/daf 0 2024-03-23T11:47:54.261 INFO:tasks.workunit.client.0.smithi063.stdout:1/694: stat d3/d1c/faa 0 2024-03-23T11:47:54.263 INFO:tasks.workunit.client.0.smithi063.stdout:4/755: read d5/da/d20/d4b/f98 [2431812,8129] 0 2024-03-23T11:47:54.269 INFO:tasks.workunit.client.0.smithi063.stdout:9/804: dread d4/f74 [0,4194304] 0 2024-03-23T11:47:54.269 INFO:tasks.workunit.client.0.smithi063.stdout:1/695: rename d3/d3f/d4d/f8c to d3/d62/dad/fd5 0 2024-03-23T11:47:54.277 INFO:tasks.workunit.client.0.smithi063.stdout:1/696: mkdir d3/df/d2f/d9a/dd6 0 2024-03-23T11:47:54.281 INFO:tasks.workunit.client.0.smithi063.stdout:4/756: rename d5/da/d20/d4b/d29/d38/d93/cb2 to d5/da/d7e/cd8 0 2024-03-23T11:47:54.282 INFO:tasks.workunit.client.0.smithi063.stdout:5/503: getdents d2/d17/d4d 0 2024-03-23T11:47:54.284 INFO:tasks.workunit.client.0.smithi063.stdout:4/757: chown d5/da/d20/d4b/f68 203 1 2024-03-23T11:47:54.284 INFO:tasks.workunit.client.0.smithi063.stdout:1/697: fdatasync d3/d1c/f66 0 2024-03-23T11:47:54.285 INFO:tasks.workunit.client.0.smithi063.stdout:9/805: dread d4/d31/d43/d49/d8e/fbc [0,4194304] 0 2024-03-23T11:47:54.289 INFO:tasks.workunit.client.0.smithi063.stdout:4/758: creat d5/da/d7e/dbf/fd9 x:0 0 0 2024-03-23T11:47:54.291 INFO:tasks.workunit.client.0.smithi063.stdout:4/759: stat d5/da/d20/d4b/d16/d37/c6e 0 2024-03-23T11:47:54.292 INFO:tasks.workunit.client.0.smithi063.stdout:1/698: mknod d3/d49/d56/d65/d88/d92/cd7 0 2024-03-23T11:47:54.295 INFO:tasks.workunit.client.0.smithi063.stdout:1/699: chown d3/d49/d56/d65/f7c 2236 1 2024-03-23T11:47:54.296 INFO:tasks.workunit.client.0.smithi063.stdout:1/700: fdatasync d3/df/f36 0 2024-03-23T11:47:54.297 INFO:tasks.workunit.client.0.smithi063.stdout:4/760: symlink d5/da/d20/d4b/d16/d50/d8e/dad/lda 0 2024-03-23T11:47:54.298 INFO:tasks.workunit.client.0.smithi063.stdout:6/627: sync 2024-03-23T11:47:54.298 INFO:tasks.workunit.client.0.smithi063.stdout:7/734: sync 2024-03-23T11:47:54.300 INFO:tasks.workunit.client.0.smithi063.stdout:1/701: mknod d3/df/d7b/cd8 0 2024-03-23T11:47:54.300 INFO:tasks.workunit.client.0.smithi063.stdout:7/735: chown d5/d26/d76/cd4 1 1 2024-03-23T11:47:54.303 INFO:tasks.workunit.client.0.smithi063.stdout:6/628: rmdir d1/dc/d8c 39 2024-03-23T11:47:54.307 INFO:tasks.workunit.client.0.smithi063.stdout:7/736: symlink d5/d26/db7/dce/le5 0 2024-03-23T11:47:54.309 INFO:tasks.workunit.client.0.smithi063.stdout:6/629: creat d1/d15/d4b/fc2 x:0 0 0 2024-03-23T11:47:54.312 INFO:tasks.workunit.client.0.smithi063.stdout:1/702: rename d3/df/d31 to d3/d1c/d4b/da5/dd9 0 2024-03-23T11:47:54.316 INFO:tasks.workunit.client.0.smithi063.stdout:5/504: write d2/d7/f14 [77726,106142] 0 2024-03-23T11:47:54.318 INFO:tasks.workunit.client.0.smithi063.stdout:5/505: fsync d2/d7/d64/fa2 0 2024-03-23T11:47:54.319 INFO:tasks.workunit.client.0.smithi063.stdout:9/806: write d4/d31/d43/f86 [539028,1961] 0 2024-03-23T11:47:54.321 INFO:tasks.workunit.client.0.smithi063.stdout:1/703: creat d3/d49/d56/d65/d88/d92/fda x:0 0 0 2024-03-23T11:47:54.325 INFO:tasks.workunit.client.0.smithi063.stdout:5/506: truncate d2/f7f 3502461 0 2024-03-23T11:47:54.329 INFO:tasks.workunit.client.0.smithi063.stdout:1/704: mkdir d3/d49/d56/d65/db6/ddb 0 2024-03-23T11:47:54.331 INFO:tasks.workunit.client.0.smithi063.stdout:6/630: getdents d1/dc/d3a/d82/d85 0 2024-03-23T11:47:54.332 INFO:tasks.workunit.client.0.smithi063.stdout:5/507: truncate d2/d7/d64/f87 454051 0 2024-03-23T11:47:54.335 INFO:tasks.workunit.client.0.smithi063.stdout:1/705: truncate d3/f39 1571528 0 2024-03-23T11:47:54.336 INFO:tasks.workunit.client.0.smithi063.stdout:6/631: stat d1/dc/d3a/d57/f5c 0 2024-03-23T11:47:54.338 INFO:tasks.workunit.client.0.smithi063.stdout:4/761: dwrite d5/da/d72/f88 [0,4194304] 0 2024-03-23T11:47:54.338 INFO:tasks.workunit.client.0.smithi063.stdout:5/508: mkdir d2/d7/d8/de/d49/db0 0 2024-03-23T11:47:54.345 INFO:tasks.workunit.client.0.smithi063.stdout:6/632: truncate d1/f9e 1091203 0 2024-03-23T11:47:54.349 INFO:tasks.workunit.client.0.smithi063.stdout:9/807: dread d4/f21 [0,4194304] 0 2024-03-23T11:47:54.350 INFO:tasks.workunit.client.0.smithi063.stdout:4/762: rmdir d5/da/d20/d4b/d16/d50 39 2024-03-23T11:47:54.352 INFO:tasks.workunit.client.0.smithi063.stdout:1/706: truncate d3/d35/f90 460189 0 2024-03-23T11:47:54.357 INFO:tasks.workunit.client.0.smithi063.stdout:1/707: mkdir d3/d3f/d4d/ddc 0 2024-03-23T11:47:54.359 INFO:tasks.workunit.client.0.smithi063.stdout:5/509: rename d2/d28/f2d to d2/d2f/fb1 0 2024-03-23T11:47:54.360 INFO:tasks.workunit.client.0.smithi063.stdout:4/763: creat d5/d1e/d2b/dbd/fdb x:0 0 0 2024-03-23T11:47:54.362 INFO:tasks.workunit.client.0.smithi063.stdout:1/708: mkdir d3/d3f/d4d/d52/ddd 0 2024-03-23T11:47:54.365 INFO:tasks.workunit.client.0.smithi063.stdout:6/633: rename d1/dc/d3a/d57/d77/d79 to d1/dc/d3a/d82/da2/d41/d42/d99/dc3 0 2024-03-23T11:47:54.365 INFO:tasks.workunit.client.0.smithi063.stdout:9/808: rename d4 to d4/d31/d43/d49/d8e/d111 22 2024-03-23T11:47:54.368 INFO:tasks.workunit.client.0.smithi063.stdout:5/510: unlink d2/d7/d8/d1d/l96 0 2024-03-23T11:47:54.371 INFO:tasks.workunit.client.0.smithi063.stdout:9/809: rename d4/d31/d43/d49/f78 to d4/d31/f112 0 2024-03-23T11:47:54.372 INFO:tasks.workunit.client.0.smithi063.stdout:4/764: mknod d5/da/cdc 0 2024-03-23T11:47:54.376 INFO:tasks.workunit.client.0.smithi063.stdout:5/511: dread d2/d7/d8/f20 [0,4194304] 0 2024-03-23T11:47:54.378 INFO:tasks.workunit.client.0.smithi063.stdout:9/810: link d4/fad d4/d4c/d44/f113 0 2024-03-23T11:47:54.383 INFO:tasks.workunit.client.0.smithi063.stdout:6/634: dwrite d1/dc/fab [0,4194304] 0 2024-03-23T11:47:54.386 INFO:tasks.workunit.client.0.smithi063.stdout:4/765: mkdir d5/da/d20/d4b/d16/d50/d8e/dad/ddd 0 2024-03-23T11:47:54.387 INFO:tasks.workunit.client.0.smithi063.stdout:9/811: truncate d4/d4c/d19/f1b 3902462 0 2024-03-23T11:47:54.391 INFO:tasks.workunit.client.0.smithi063.stdout:4/766: write d5/da/d72/fa8 [764937,112767] 0 2024-03-23T11:47:54.391 INFO:tasks.workunit.client.0.smithi063.stdout:6/635: symlink d1/dc/d3a/d82/da2/d41/d42/d99/lc4 0 2024-03-23T11:47:54.397 INFO:tasks.workunit.client.0.smithi063.stdout:4/767: chown d5/da/d20/d4b/d16/d50/d8e/dad/ddd 231 1 2024-03-23T11:47:54.397 INFO:tasks.workunit.client.0.smithi063.stdout:9/812: mknod d4/d31/c114 0 2024-03-23T11:47:54.398 INFO:tasks.workunit.client.0.smithi063.stdout:5/512: dwrite d2/d17/f51 [4194304,4194304] 0 2024-03-23T11:47:54.401 INFO:tasks.workunit.client.0.smithi063.stdout:6/636: creat d1/dc/d3a/d57/d6d/fc5 x:0 0 0 2024-03-23T11:47:54.407 INFO:tasks.workunit.client.0.smithi063.stdout:9/813: truncate d4/d31/d70/d8d/fac 4134030 0 2024-03-23T11:47:54.409 INFO:tasks.workunit.client.0.smithi063.stdout:6/637: symlink d1/dc/d8c/lc6 0 2024-03-23T11:47:54.417 INFO:tasks.workunit.client.0.smithi063.stdout:6/638: rmdir d1/dc/d3a/d82/da2/d41/d42 39 2024-03-23T11:47:54.417 INFO:tasks.workunit.client.0.smithi063.stdout:9/814: symlink d4/d4c/dab/l115 0 2024-03-23T11:47:54.418 INFO:tasks.workunit.client.0.smithi063.stdout:6/639: chown d1/dc/f91 13072 1 2024-03-23T11:47:54.427 INFO:tasks.workunit.client.0.smithi063.stdout:6/640: mknod d1/dc/d3a/d57/d77/cc7 0 2024-03-23T11:47:54.431 INFO:tasks.workunit.client.0.smithi063.stdout:6/641: dread - d1/dc/d3a/d57/fac zero size 2024-03-23T11:47:54.431 INFO:tasks.workunit.client.0.smithi063.stdout:5/513: dwrite d2/f5 [0,4194304] 0 2024-03-23T11:47:54.444 INFO:tasks.workunit.client.0.smithi063.stdout:6/642: creat d1/dc/d3a/d82/da2/d41/d42/fc8 x:0 0 0 2024-03-23T11:47:54.445 INFO:tasks.workunit.client.0.smithi063.stdout:9/815: dwrite d4/d31/d9c/d9e/fe7 [0,4194304] 0 2024-03-23T11:47:54.446 INFO:tasks.workunit.client.0.smithi063.stdout:5/514: rename d2/d7/d27/c7b to d2/d7/d8/de/d2a/d3b/d85/cb2 0 2024-03-23T11:47:54.450 INFO:tasks.workunit.client.0.smithi063.stdout:6/643: rmdir d1/dc/d21 39 2024-03-23T11:47:54.455 INFO:tasks.workunit.client.0.smithi063.stdout:5/515: truncate d2/d7/d8/de/d2a/d3b/f73 1015377 0 2024-03-23T11:47:54.458 INFO:tasks.workunit.client.0.smithi063.stdout:9/816: truncate d4/d31/d43/d49/fcf 612085 0 2024-03-23T11:47:54.461 INFO:tasks.workunit.client.0.smithi063.stdout:6/644: creat d1/dc/d3a/d57/d77/fc9 x:0 0 0 2024-03-23T11:47:54.463 INFO:tasks.workunit.client.0.smithi063.stdout:6/645: write d1/f81 [791204,6633] 0 2024-03-23T11:47:54.465 INFO:tasks.workunit.client.0.smithi063.stdout:5/516: rename d2/d7/d2b/l95 to d2/d28/d3a/d46/d79/lb3 0 2024-03-23T11:47:54.468 INFO:tasks.workunit.client.0.smithi063.stdout:6/646: rmdir d1/d15/d4b 39 2024-03-23T11:47:54.472 INFO:tasks.workunit.client.0.smithi063.stdout:6/647: fsync d1/dc/d3a/d82/da2/d41/d42/f9d 0 2024-03-23T11:47:54.477 INFO:tasks.workunit.client.0.smithi063.stdout:9/817: rename d4/d31/d43/da9/da6/ldf to d4/d31/d43/d49/d8e/l116 0 2024-03-23T11:47:54.480 INFO:tasks.workunit.client.0.smithi063.stdout:6/648: truncate d1/d15/f26 1621375 0 2024-03-23T11:47:54.486 INFO:tasks.workunit.client.0.smithi063.stdout:6/649: truncate d1/dc/d3a/d57/f5c 46881 0 2024-03-23T11:47:54.489 INFO:tasks.workunit.client.0.smithi063.stdout:6/650: mknod d1/dc/d3a/cca 0 2024-03-23T11:47:54.498 INFO:tasks.workunit.client.0.smithi063.stdout:5/517: dwrite d2/d7/d8/d1d/f57 [0,4194304] 0 2024-03-23T11:47:54.499 INFO:tasks.workunit.client.0.smithi063.stdout:9/818: dwrite d4/d31/d9c/ffb [0,4194304] 0 2024-03-23T11:47:54.504 INFO:tasks.workunit.client.0.smithi063.stdout:6/651: symlink d1/dc/d3a/d82/da2/d41/d42/d99/dc3/lcb 0 2024-03-23T11:47:54.512 INFO:tasks.workunit.client.0.smithi063.stdout:9/819: write d4/d4c/f25 [276954,123870] 0 2024-03-23T11:47:54.515 INFO:tasks.workunit.client.0.smithi063.stdout:9/820: chown d4/c65 1863587 1 2024-03-23T11:47:54.516 INFO:tasks.workunit.client.0.smithi063.stdout:9/821: readlink d4/d31/d43/l6c 0 2024-03-23T11:47:54.545 INFO:tasks.workunit.client.0.smithi063.stdout:5/518: dwrite d2/d28/f3d [0,4194304] 0 2024-03-23T11:47:54.545 INFO:tasks.workunit.client.0.smithi063.stdout:6/652: dwrite d1/d15/fa4 [0,4194304] 0 2024-03-23T11:47:54.548 INFO:tasks.workunit.client.0.smithi063.stdout:9/822: dwrite d4/d31/d70/ff2 [0,4194304] 0 2024-03-23T11:47:54.553 INFO:tasks.workunit.client.0.smithi063.stdout:5/519: mknod d2/d7/d2b/d8b/da7/cb4 0 2024-03-23T11:47:54.562 INFO:tasks.workunit.client.0.smithi063.stdout:5/520: symlink d2/d7/d8/de/d49/lb5 0 2024-03-23T11:47:54.570 INFO:tasks.workunit.client.0.smithi063.stdout:5/521: mknod d2/d7/d8/de/d2a/d3b/d85/cb6 0 2024-03-23T11:47:54.577 INFO:tasks.workunit.client.0.smithi063.stdout:9/823: dwrite d4/d31/d43/da9/da6/f87 [0,4194304] 0 2024-03-23T11:47:54.577 INFO:tasks.workunit.client.0.smithi063.stdout:6/653: dread d1/d15/f26 [0,4194304] 0 2024-03-23T11:47:54.579 INFO:tasks.workunit.client.0.smithi063.stdout:9/824: chown d4/d31/d43/l6c 216 1 2024-03-23T11:47:54.583 INFO:tasks.workunit.client.0.smithi063.stdout:5/522: getdents d2/d7/d8/de/d2a 0 2024-03-23T11:47:54.586 INFO:tasks.workunit.client.0.smithi063.stdout:6/654: symlink d1/dc/d21/d49/lcc 0 2024-03-23T11:47:54.589 INFO:tasks.workunit.client.0.smithi063.stdout:5/523: unlink d2/d7/c9 0 2024-03-23T11:47:54.600 INFO:tasks.workunit.client.0.smithi063.stdout:6/655: rmdir d1/dc/d3a/d57/d77/dc1 0 2024-03-23T11:47:54.603 INFO:tasks.workunit.client.0.smithi063.stdout:6/656: readlink d1/dc/d3a/d82/d85/lbd 0 2024-03-23T11:47:54.605 INFO:tasks.workunit.client.0.smithi063.stdout:5/524: write d2/d28/d3a/d46/d71/f78 [537138,70361] 0 2024-03-23T11:47:54.609 INFO:tasks.workunit.client.0.smithi063.stdout:5/525: truncate f1 2075660 0 2024-03-23T11:47:54.610 INFO:tasks.workunit.client.0.smithi063.stdout:9/825: dwrite d4/d31/fb9 [0,4194304] 0 2024-03-23T11:47:54.611 INFO:tasks.workunit.client.0.smithi063.stdout:6/657: dread d1/dc/d3a/d82/da2/d41/f5e [0,4194304] 0 2024-03-23T11:47:54.624 INFO:tasks.workunit.client.0.smithi063.stdout:6/658: rename d1/dc/d3a/d57/d6d/cbc to d1/dc/d3a/d82/da2/d41/d42/d99/dc3/ccd 0 2024-03-23T11:47:54.625 INFO:tasks.workunit.client.0.smithi063.stdout:5/526: write d2/f7f [4021925,45849] 0 2024-03-23T11:47:54.634 INFO:tasks.workunit.client.0.smithi063.stdout:9/826: dwrite d4/d4c/f69 [0,4194304] 0 2024-03-23T11:47:54.638 INFO:tasks.workunit.client.0.smithi063.stdout:6/659: dread d1/d15/d4b/f6b [0,4194304] 0 2024-03-23T11:47:54.640 INFO:tasks.workunit.client.0.smithi063.stdout:9/827: read - d4/d31/d43/da9/da6/db5/fc1 zero size 2024-03-23T11:47:54.643 INFO:tasks.workunit.client.0.smithi063.stdout:6/660: getdents d1/d15/d4b 0 2024-03-23T11:47:54.644 INFO:tasks.workunit.client.0.smithi063.stdout:9/828: rmdir d4/d31/d43/d49 39 2024-03-23T11:47:54.645 INFO:tasks.workunit.client.0.smithi063.stdout:9/829: chown d4/d31/fb9 0 1 2024-03-23T11:47:54.652 INFO:tasks.workunit.client.0.smithi063.stdout:5/527: dread d2/d7/d8/d1d/f1f [0,4194304] 0 2024-03-23T11:47:54.654 INFO:tasks.workunit.client.0.smithi063.stdout:9/830: creat d4/d31/d10b/f117 x:0 0 0 2024-03-23T11:47:54.655 INFO:tasks.workunit.client.0.smithi063.stdout:5/528: rmdir d2/d7/d8/de/d2a/d3b/d43/d5c 39 2024-03-23T11:47:54.657 INFO:tasks.workunit.client.0.smithi063.stdout:7/737: sync 2024-03-23T11:47:54.658 INFO:tasks.workunit.client.0.smithi063.stdout:2/688: sync 2024-03-23T11:47:54.658 INFO:tasks.workunit.client.0.smithi063.stdout:0/945: sync 2024-03-23T11:47:54.662 INFO:tasks.workunit.client.0.smithi063.stdout:9/831: write d4/fae [228303,130843] 0 2024-03-23T11:47:54.664 INFO:tasks.workunit.client.0.smithi063.stdout:5/529: symlink d2/d28/d3a/d46/d71/d9f/lb7 0 2024-03-23T11:47:54.671 INFO:tasks.workunit.client.0.smithi063.stdout:6/661: dwrite d1/dc/d3a/f61 [0,4194304] 0 2024-03-23T11:47:54.674 INFO:tasks.workunit.client.0.smithi063.stdout:5/530: creat d2/d17/fb8 x:0 0 0 2024-03-23T11:47:54.676 INFO:tasks.workunit.client.0.smithi063.stdout:2/689: write d2/d84/d94/fa6 [793672,52309] 0 2024-03-23T11:47:54.684 INFO:tasks.workunit.client.0.smithi063.stdout:6/662: mknod d1/d15/cce 0 2024-03-23T11:47:54.688 INFO:tasks.workunit.client.0.smithi063.stdout:5/531: mknod d2/d7/d8/de/cb9 0 2024-03-23T11:47:54.689 INFO:tasks.workunit.client.0.smithi063.stdout:0/946: dwrite d2/d48/f134 [0,4194304] 0 2024-03-23T11:47:54.689 INFO:tasks.workunit.client.0.smithi063.stdout:9/832: creat d4/d31/d43/da9/da6/dbe/f118 x:0 0 0 2024-03-23T11:47:54.689 INFO:tasks.workunit.client.0.smithi063.stdout:7/738: dread d5/d5c/d2f/fda [0,4194304] 0 2024-03-23T11:47:54.694 INFO:tasks.workunit.client.0.smithi063.stdout:9/833: write d4/d4c/fde [3100233,36429] 0 2024-03-23T11:47:54.696 INFO:tasks.workunit.client.0.smithi063.stdout:6/663: readlink d1/dc/d3a/d57/d6d/l9c 0 2024-03-23T11:47:54.699 INFO:tasks.workunit.client.0.smithi063.stdout:0/947: symlink d2/d12/d32/d6d/d124/d25/d93/dd3/l153 0 2024-03-23T11:47:54.710 INFO:tasks.workunit.client.0.smithi063.stdout:2/690: write d2/f3a [423485,69232] 0 2024-03-23T11:47:54.721 INFO:tasks.workunit.client.0.smithi063.stdout:0/948: dread d2/d14/f79 [0,4194304] 0 2024-03-23T11:47:54.734 INFO:tasks.workunit.client.0.smithi063.stdout:7/739: dwrite d5/d6/f7a [4194304,4194304] 0 2024-03-23T11:47:54.744 INFO:tasks.workunit.client.0.smithi063.stdout:6/664: dwrite d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:54.749 INFO:tasks.workunit.client.0.smithi063.stdout:5/532: dwrite d2/d7/d8/de/d2a/d3b/f73 [0,4194304] 0 2024-03-23T11:47:54.755 INFO:tasks.workunit.client.0.smithi063.stdout:2/691: dwrite d2/d23/d2e/d52/f7d [0,4194304] 0 2024-03-23T11:47:54.756 INFO:tasks.workunit.client.0.smithi063.stdout:5/533: chown d2/d2f 1048881202 1 2024-03-23T11:47:54.760 INFO:tasks.workunit.client.0.smithi063.stdout:4/768: sync 2024-03-23T11:47:54.761 INFO:tasks.workunit.client.0.smithi063.stdout:1/709: sync 2024-03-23T11:47:54.762 INFO:tasks.workunit.client.0.smithi063.stdout:9/834: dwrite d4/d31/d70/d8d/da0/fe1 [0,4194304] 0 2024-03-23T11:47:54.768 INFO:tasks.workunit.client.0.smithi063.stdout:2/692: read d2/d1c/f20 [3327873,80058] 0 2024-03-23T11:47:54.770 INFO:tasks.workunit.client.0.smithi063.stdout:1/710: stat d3/df/f78 0 2024-03-23T11:47:54.781 INFO:tasks.workunit.client.0.smithi063.stdout:7/740: dwrite d5/d5c/d2f/d73/d20/f31 [0,4194304] 0 2024-03-23T11:47:54.790 INFO:tasks.workunit.client.0.smithi063.stdout:6/665: rename d1/dc/d3a/c92 to d1/d15/ccf 0 2024-03-23T11:47:54.792 INFO:tasks.workunit.client.0.smithi063.stdout:7/741: dread - d5/d6/dca/fcb zero size 2024-03-23T11:47:54.795 INFO:tasks.workunit.client.0.smithi063.stdout:0/949: dwrite d2/d6e/d8c/fe4 [0,4194304] 0 2024-03-23T11:47:54.798 INFO:tasks.workunit.client.0.smithi063.stdout:5/534: dread d2/d7/d8/f68 [0,4194304] 0 2024-03-23T11:47:54.799 INFO:tasks.workunit.client.0.smithi063.stdout:7/742: readlink d5/d5c/d2f/d8b/daa/lac 0 2024-03-23T11:47:54.802 INFO:tasks.workunit.client.0.smithi063.stdout:9/835: rename d4/fc to d4/d31/d43/da9/da6/dbe/f119 0 2024-03-23T11:47:54.805 INFO:tasks.workunit.client.0.smithi063.stdout:2/693: dread d2/de/f2b [0,4194304] 0 2024-03-23T11:47:54.806 INFO:tasks.workunit.client.0.smithi063.stdout:7/743: read f1 [6849822,87427] 0 2024-03-23T11:47:54.807 INFO:tasks.workunit.client.0.smithi063.stdout:0/950: read d2/d14/d36/d77/fe5 [16747,74513] 0 2024-03-23T11:47:54.810 INFO:tasks.workunit.client.0.smithi063.stdout:6/666: creat d1/d15/fd0 x:0 0 0 2024-03-23T11:47:54.811 INFO:tasks.workunit.client.0.smithi063.stdout:4/769: write d5/da/d20/d4b/d16/d50/d8e/fa7 [5100859,80498] 0 2024-03-23T11:47:54.813 INFO:tasks.workunit.client.0.smithi063.stdout:6/667: readlink d1/dc/d21/lb3 0 2024-03-23T11:47:54.817 INFO:tasks.workunit.client.0.smithi063.stdout:4/770: chown d5/da/d72/fd0 848 1 2024-03-23T11:47:54.818 INFO:tasks.workunit.client.0.smithi063.stdout:6/668: chown d1/dc/d3a/d82/d85/lbe 0 1 2024-03-23T11:47:54.819 INFO:tasks.workunit.client.0.smithi063.stdout:1/711: rename d3/df/f6e to d3/d49/d56/d65/d88/d92/fde 0 2024-03-23T11:47:54.822 INFO:tasks.workunit.client.0.smithi063.stdout:2/694: creat d2/d46/d54/fdd x:0 0 0 2024-03-23T11:47:54.829 INFO:tasks.workunit.client.0.smithi063.stdout:5/535: rename d2/d7/d8/de/d8f to d2/d17/d4d/d6e/dba 0 2024-03-23T11:47:54.833 INFO:tasks.workunit.client.0.smithi063.stdout:1/712: creat d3/d49/d56/d65/db6/dc5/fdf x:0 0 0 2024-03-23T11:47:54.835 INFO:tasks.workunit.client.0.smithi063.stdout:0/951: dread d2/d12/f7e [0,4194304] 0 2024-03-23T11:47:54.835 INFO:tasks.workunit.client.0.smithi063.stdout:1/713: dread - d3/df/db8/fc4 zero size 2024-03-23T11:47:54.837 INFO:tasks.workunit.client.0.smithi063.stdout:0/952: chown d2/d48/c113 479 1 2024-03-23T11:47:54.839 INFO:tasks.workunit.client.0.smithi063.stdout:9/836: link d4/d31/d43/d49/d8e/fbc d4/d31/d43/da9/da6/dbe/f11a 0 2024-03-23T11:47:54.842 INFO:tasks.workunit.client.0.smithi063.stdout:7/744: write d5/fc7 [486249,37345] 0 2024-03-23T11:47:54.848 INFO:tasks.workunit.client.0.smithi063.stdout:5/536: write d2/d7/d8/de/d49/f63 [628341,26940] 0 2024-03-23T11:47:54.851 INFO:tasks.workunit.client.0.smithi063.stdout:6/669: link d1/d15/d4b/f4c d1/d15/fd1 0 2024-03-23T11:47:54.856 INFO:tasks.workunit.client.0.smithi063.stdout:2/695: dwrite d2/d23/d2e/d52/fd0 [0,4194304] 0 2024-03-23T11:47:54.856 INFO:tasks.workunit.client.0.smithi063.stdout:1/714: write d3/df/f78 [3853575,129284] 0 2024-03-23T11:47:54.856 INFO:tasks.workunit.client.0.smithi063.stdout:5/537: fsync d2/d28/d3a/d46/d79/f7e 0 2024-03-23T11:47:54.861 INFO:tasks.workunit.client.0.smithi063.stdout:4/771: dwrite f2 [0,4194304] 0 2024-03-23T11:47:54.864 INFO:tasks.workunit.client.0.smithi063.stdout:5/538: chown d2/d28/d3a/d46/f86 0 1 2024-03-23T11:47:54.864 INFO:tasks.workunit.client.0.smithi063.stdout:9/837: dread d4/d31/d43/da9/da6/f87 [0,4194304] 0 2024-03-23T11:47:54.866 INFO:tasks.workunit.client.0.smithi063.stdout:6/670: mkdir d1/dc/d3a/d82/da2/dd2 0 2024-03-23T11:47:54.867 INFO:tasks.workunit.client.0.smithi063.stdout:2/696: mknod d2/d23/d2e/d87/cde 0 2024-03-23T11:47:54.870 INFO:tasks.workunit.client.0.smithi063.stdout:6/671: chown d1/dc/d21/d49/f76 243 1 2024-03-23T11:47:54.873 INFO:tasks.workunit.client.0.smithi063.stdout:0/953: link d2/d14/de8/f8e d2/d14/d36/d77/deb/f154 0 2024-03-23T11:47:54.879 INFO:tasks.workunit.client.0.smithi063.stdout:7/745: dread d5/d26/db7/dc6/d4f/f93 [0,4194304] 0 2024-03-23T11:47:54.879 INFO:tasks.workunit.client.0.smithi063.stdout:9/838: rename d4/d31/d70 to d4/d31/d9c/d11b 0 2024-03-23T11:47:54.880 INFO:tasks.workunit.client.0.smithi063.stdout:2/697: rmdir d2/d23/d2e/d87 39 2024-03-23T11:47:54.884 INFO:tasks.workunit.client.0.smithi063.stdout:5/539: fdatasync d2/d28/f4b 0 2024-03-23T11:47:54.887 INFO:tasks.workunit.client.0.smithi063.stdout:1/715: write d3/d3f/d4d/f95 [577558,4550] 0 2024-03-23T11:47:54.887 INFO:tasks.workunit.client.0.smithi063.stdout:6/672: creat d1/dc/d21/d49/fd3 x:0 0 0 2024-03-23T11:47:54.889 INFO:tasks.workunit.client.0.smithi063.stdout:0/954: fsync d2/d14/d36/f11b 0 2024-03-23T11:47:54.891 INFO:tasks.workunit.client.0.smithi063.stdout:4/772: unlink d5/da/d20/d4b/f5e 0 2024-03-23T11:47:54.897 INFO:tasks.workunit.client.0.smithi063.stdout:9/839: dwrite d4/d31/d9c/d11b/d8d/d90/ff1 [0,4194304] 0 2024-03-23T11:47:54.902 INFO:tasks.workunit.client.0.smithi063.stdout:7/746: truncate d5/d5c/d2f/d73/da/f4e 5120797 0 2024-03-23T11:47:54.902 INFO:tasks.workunit.client.0.smithi063.stdout:6/673: mkdir d1/dd4 0 2024-03-23T11:47:54.907 INFO:tasks.workunit.client.0.smithi063.stdout:9/840: chown d4/d31/d43/da9/laa 1866107464 1 2024-03-23T11:47:54.910 INFO:tasks.workunit.client.0.smithi063.stdout:0/955: creat d2/d48/d110/f155 x:0 0 0 2024-03-23T11:47:54.915 INFO:tasks.workunit.client.0.smithi063.stdout:2/698: rename d2/d23/d2e/f4b to d2/d23/d56/dad/dd4/dd6/fdf 0 2024-03-23T11:47:54.916 INFO:tasks.workunit.client.0.smithi063.stdout:7/747: symlink d5/d5c/d2f/d8b/le6 0 2024-03-23T11:47:54.920 INFO:tasks.workunit.client.0.smithi063.stdout:5/540: dwrite d2/d7/d8/f20 [0,4194304] 0 2024-03-23T11:47:54.922 INFO:tasks.workunit.client.0.smithi063.stdout:9/841: stat d4/d31/le6 0 2024-03-23T11:47:54.922 INFO:tasks.workunit.client.0.smithi063.stdout:7/748: chown d5/d5c/d2f/cb1 25971496 1 2024-03-23T11:47:54.928 INFO:tasks.workunit.client.0.smithi063.stdout:6/674: dwrite d1/dc/d3a/d57/d6d/fa8 [0,4194304] 0 2024-03-23T11:47:54.932 INFO:tasks.workunit.client.0.smithi063.stdout:1/716: rename d3/d1c to d3/d3f/d4d/dbc/de0 0 2024-03-23T11:47:54.933 INFO:tasks.workunit.client.0.smithi063.stdout:6/675: dread - d1/dc/d3a/d57/d6d/f9a zero size 2024-03-23T11:47:54.934 INFO:tasks.workunit.client.0.smithi063.stdout:4/773: dwrite d5/da/d20/f48 [0,4194304] 0 2024-03-23T11:47:54.939 INFO:tasks.workunit.client.0.smithi063.stdout:5/541: symlink d2/d17/d4d/lbb 0 2024-03-23T11:47:54.941 INFO:tasks.workunit.client.0.smithi063.stdout:9/842: mkdir d4/d31/d9c/d11b/d8d/d11c 0 2024-03-23T11:47:54.942 INFO:tasks.workunit.client.0.smithi063.stdout:7/749: rename d5/d6/lc0 to d5/d5c/d2f/d8b/le7 0 2024-03-23T11:47:54.945 INFO:tasks.workunit.client.0.smithi063.stdout:7/750: write d5/d5c/d2f/f79 [1032117,69467] 0 2024-03-23T11:47:54.945 INFO:tasks.workunit.client.0.smithi063.stdout:6/676: mkdir d1/dc/d3a/d82/da2/dd5 0 2024-03-23T11:47:54.946 INFO:tasks.workunit.client.0.smithi063.stdout:7/751: fdatasync d5/fc7 0 2024-03-23T11:47:54.950 INFO:tasks.workunit.client.0.smithi063.stdout:2/699: mknod d2/d23/d2e/d87/d4d/ce0 0 2024-03-23T11:47:54.954 INFO:tasks.workunit.client.0.smithi063.stdout:5/542: mknod d2/d2f/cbc 0 2024-03-23T11:47:54.959 INFO:tasks.workunit.client.0.smithi063.stdout:9/843: fdatasync d4/d31/d43/da9/da6/dbe/f11a 0 2024-03-23T11:47:54.961 INFO:tasks.workunit.client.0.smithi063.stdout:9/844: readlink d4/d4c/d19/l2b 0 2024-03-23T11:47:54.961 INFO:tasks.workunit.client.0.smithi063.stdout:1/717: write d3/d3f/d4d/dbc/de0/faa [553881,119585] 0 2024-03-23T11:47:54.965 INFO:tasks.workunit.client.0.smithi063.stdout:6/677: unlink d1/d15/f88 0 2024-03-23T11:47:54.967 INFO:tasks.workunit.client.0.smithi063.stdout:0/956: link d2/d12/d32/d6d/d124/d28/d46/d51/c123 d2/d14/d36/d77/deb/c156 0 2024-03-23T11:47:54.968 INFO:tasks.workunit.client.0.smithi063.stdout:6/678: write d1/dc/d3a/d57/d77/fad [226672,84700] 0 2024-03-23T11:47:54.971 INFO:tasks.workunit.client.0.smithi063.stdout:4/774: dread d5/f90 [0,4194304] 0 2024-03-23T11:47:54.980 INFO:tasks.workunit.client.0.smithi063.stdout:2/700: mknod d2/d23/d2e/d98/dd2/ce1 0 2024-03-23T11:47:54.997 INFO:tasks.workunit.client.0.smithi063.stdout:9/845: mknod d4/d31/d43/da9/da6/c11d 0 2024-03-23T11:47:55.000 INFO:tasks.workunit.client.0.smithi063.stdout:7/752: link d5/d26/db7/dc6/d4f/d5b/dd3/fdd d5/d6/d19/fe8 0 2024-03-23T11:47:55.001 INFO:tasks.workunit.client.0.smithi063.stdout:0/957: write d2/d14/d36/fda [712819,91596] 0 2024-03-23T11:47:55.003 INFO:tasks.workunit.client.0.smithi063.stdout:0/958: stat d2/d6e/d115 0 2024-03-23T11:47:55.008 INFO:tasks.workunit.client.0.smithi063.stdout:5/543: creat d2/d7/fbd x:0 0 0 2024-03-23T11:47:55.009 INFO:tasks.workunit.client.0.smithi063.stdout:9/846: mkdir d4/d31/d43/da9/da6/d100/d11e 0 2024-03-23T11:47:55.012 INFO:tasks.workunit.client.0.smithi063.stdout:7/753: unlink d5/d26/db7/dd6/c6b 0 2024-03-23T11:47:55.015 INFO:tasks.workunit.client.0.smithi063.stdout:2/701: symlink d2/d23/d2e/d98/dc4/dcc/le2 0 2024-03-23T11:47:55.022 INFO:tasks.workunit.client.0.smithi063.stdout:2/702: creat d2/d23/d2e/d87/d4d/fe3 x:0 0 0 2024-03-23T11:47:55.023 INFO:tasks.workunit.client.0.smithi063.stdout:1/718: dwrite d3/f2a [0,4194304] 0 2024-03-23T11:47:55.024 INFO:tasks.workunit.client.0.smithi063.stdout:5/544: mkdir d2/d17/d4d/d6e/dba/dbe 0 2024-03-23T11:47:55.025 INFO:tasks.workunit.client.0.smithi063.stdout:0/959: dread d2/d12/d32/d6d/d124/d25/f13c [0,4194304] 0 2024-03-23T11:47:55.029 INFO:tasks.workunit.client.0.smithi063.stdout:7/754: getdents d5/d26/db7/dc6/d6e/d98 0 2024-03-23T11:47:55.032 INFO:tasks.workunit.client.0.smithi063.stdout:6/679: dwrite d1/d15/d4b/f4c [0,4194304] 0 2024-03-23T11:47:55.046 INFO:tasks.workunit.client.0.smithi063.stdout:9/847: dread d4/d31/d43/f7d [0,4194304] 0 2024-03-23T11:47:55.048 INFO:tasks.workunit.client.0.smithi063.stdout:5/545: creat d2/d7/d8/de/d2a/d3b/d43/d5c/fbf x:0 0 0 2024-03-23T11:47:55.048 INFO:tasks.workunit.client.0.smithi063.stdout:0/960: read d2/d14/de8/fb6 [2533104,128347] 0 2024-03-23T11:47:55.050 INFO:tasks.workunit.client.0.smithi063.stdout:6/680: unlink d1/d15/f70 0 2024-03-23T11:47:55.052 INFO:tasks.workunit.client.0.smithi063.stdout:5/546: readlink d2/d7/d8/l11 0 2024-03-23T11:47:55.053 INFO:tasks.workunit.client.0.smithi063.stdout:9/848: unlink d4/l8 0 2024-03-23T11:47:55.055 INFO:tasks.workunit.client.0.smithi063.stdout:2/703: dwrite d2/d23/d2e/d87/d4d/fb1 [0,4194304] 0 2024-03-23T11:47:55.056 INFO:tasks.workunit.client.0.smithi063.stdout:0/961: write d2/d12/d32/d6d/d124/d1f/f2f [2282234,37888] 0 2024-03-23T11:47:55.058 INFO:tasks.workunit.client.0.smithi063.stdout:7/755: dwrite d5/d5c/d2f/d73/d20/f31 [0,4194304] 0 2024-03-23T11:47:55.060 INFO:tasks.workunit.client.0.smithi063.stdout:1/719: dread d3/d49/d56/d65/d84/f98 [4194304,4194304] 0 2024-03-23T11:47:55.063 INFO:tasks.workunit.client.0.smithi063.stdout:1/720: chown d3/d3f/d4d/dbc/de0/d4b/fbb 39 1 2024-03-23T11:47:55.065 INFO:tasks.workunit.client.0.smithi063.stdout:5/547: mkdir d2/d28/d3a/d46/d79/dc0 0 2024-03-23T11:47:55.067 INFO:tasks.workunit.client.0.smithi063.stdout:2/704: rename d2/de/d22/c2a to d2/de/d22/ce4 0 2024-03-23T11:47:55.075 INFO:tasks.workunit.client.0.smithi063.stdout:0/962: getdents d2/d12/d32 0 2024-03-23T11:47:55.075 INFO:tasks.workunit.client.0.smithi063.stdout:1/721: dread d3/df/f7d [0,4194304] 0 2024-03-23T11:47:55.076 INFO:tasks.workunit.client.0.smithi063.stdout:7/756: symlink d5/d6/d19/d1f/d6c/le9 0 2024-03-23T11:47:55.080 INFO:tasks.workunit.client.0.smithi063.stdout:5/548: link d2/d7/d8/c54 d2/d7/d8/d1d/cc1 0 2024-03-23T11:47:55.080 INFO:tasks.workunit.client.0.smithi063.stdout:1/722: fsync d3/d49/d56/f67 0 2024-03-23T11:47:55.081 INFO:tasks.workunit.client.0.smithi063.stdout:0/963: mkdir d2/d6e/d117/d157 0 2024-03-23T11:47:55.082 INFO:tasks.workunit.client.0.smithi063.stdout:5/549: read - d2/d7/fbd zero size 2024-03-23T11:47:55.089 INFO:tasks.workunit.client.0.smithi063.stdout:7/757: truncate d5/d26/db7/dd6/f6a 1878746 0 2024-03-23T11:47:55.091 INFO:tasks.workunit.client.0.smithi063.stdout:9/849: truncate d4/d4c/f30 98503 0 2024-03-23T11:47:55.096 INFO:tasks.workunit.client.0.smithi063.stdout:0/964: link d2/d14/f1c d2/d12/f158 0 2024-03-23T11:47:55.096 INFO:tasks.workunit.client.0.smithi063.stdout:5/550: dread d2/f5 [0,4194304] 0 2024-03-23T11:47:55.098 INFO:tasks.workunit.client.0.smithi063.stdout:2/705: truncate d2/f2f 4032240 0 2024-03-23T11:47:55.103 INFO:tasks.workunit.client.0.smithi063.stdout:9/850: creat d4/d31/d9c/d11b/d8d/d11c/f11f x:0 0 0 2024-03-23T11:47:55.104 INFO:tasks.workunit.client.0.smithi063.stdout:0/965: mkdir d2/d14/d36/d77/d94/daf/d159 0 2024-03-23T11:47:55.106 INFO:tasks.workunit.client.0.smithi063.stdout:9/851: readlink d4/d31/d9c/d11b/l99 0 2024-03-23T11:47:55.108 INFO:tasks.workunit.client.0.smithi063.stdout:9/852: readlink d4/d4c/d44/la5 0 2024-03-23T11:47:55.110 INFO:tasks.workunit.client.0.smithi063.stdout:1/723: dread d3/f7 [8388608,4194304] 0 2024-03-23T11:47:55.115 INFO:tasks.workunit.client.0.smithi063.stdout:0/966: mkdir d2/d14/d36/d77/d15a 0 2024-03-23T11:47:55.121 INFO:tasks.workunit.client.0.smithi063.stdout:9/853: mknod d4/d31/d43/d49/d8e/c120 0 2024-03-23T11:47:55.121 INFO:tasks.workunit.client.0.smithi063.stdout:0/967: truncate d2/d12/d32/d6d/d124/d1f/f9b 1048073 0 2024-03-23T11:47:55.125 INFO:tasks.workunit.client.0.smithi063.stdout:1/724: rename d3/df/f7d to d3/df/db8/dcb/fe1 0 2024-03-23T11:47:55.126 INFO:tasks.workunit.client.0.smithi063.stdout:0/968: chown d2/d12/d32/d6d/d124/d25/c78 1262021521 1 2024-03-23T11:47:55.126 INFO:tasks.workunit.client.0.smithi063.stdout:2/706: dwrite d2/d23/d2e/d87/d4d/fb1 [0,4194304] 0 2024-03-23T11:47:55.133 INFO:tasks.workunit.client.0.smithi063.stdout:5/551: write f1 [2175712,86953] 0 2024-03-23T11:47:55.139 INFO:tasks.workunit.client.0.smithi063.stdout:1/725: mkdir d3/df/d2f/d9a/de2 0 2024-03-23T11:47:55.160 INFO:tasks.workunit.client.0.smithi063.stdout:5/552: fsync d2/d7/d64/f87 0 2024-03-23T11:47:55.166 INFO:tasks.workunit.client.0.smithi063.stdout:0/969: write d2/d12/d32/d6d/d124/d25/f122 [1032879,70646] 0 2024-03-23T11:47:55.167 INFO:tasks.workunit.client.0.smithi063.stdout:1/726: dwrite d3/df/fbf [0,4194304] 0 2024-03-23T11:47:55.168 INFO:tasks.workunit.client.0.smithi063.stdout:9/854: dread d4/d4c/d44/f113 [0,4194304] 0 2024-03-23T11:47:55.170 INFO:tasks.workunit.client.0.smithi063.stdout:5/553: mkdir d2/d7/d8/de/d2a/d3b/d43/d5c/dc2 0 2024-03-23T11:47:55.179 INFO:tasks.workunit.client.0.smithi063.stdout:1/727: dread - d3/d3f/d4d/dbc/de0/d4b/f8b zero size 2024-03-23T11:47:55.179 INFO:tasks.workunit.client.0.smithi063.stdout:2/707: dread d2/d23/d56/dad/dd4/dd6/fdf [0,4194304] 0 2024-03-23T11:47:55.181 INFO:tasks.workunit.client.0.smithi063.stdout:2/708: read - d2/f6b zero size 2024-03-23T11:47:55.181 INFO:tasks.workunit.client.0.smithi063.stdout:9/855: rename d4/d31/d43/f50 to d4/d4c/dab/f121 0 2024-03-23T11:47:55.182 INFO:tasks.workunit.client.0.smithi063.stdout:0/970: mkdir d2/d14/d36/d77/d15a/d15b 0 2024-03-23T11:47:55.184 INFO:tasks.workunit.client.0.smithi063.stdout:2/709: fsync d2/d23/fce 0 2024-03-23T11:47:55.186 INFO:tasks.workunit.client.0.smithi063.stdout:0/971: read d2/d14/f34 [4890885,41716] 0 2024-03-23T11:47:55.194 INFO:tasks.workunit.client.0.smithi063.stdout:2/710: dread d2/d84/d94/fa6 [0,4194304] 0 2024-03-23T11:47:55.199 INFO:tasks.workunit.client.0.smithi063.stdout:9/856: dread d4/d31/d43/f7d [0,4194304] 0 2024-03-23T11:47:55.210 INFO:tasks.workunit.client.0.smithi063.stdout:2/711: link d2/de/l1a d2/d23/d2e/d98/dc8/le5 0 2024-03-23T11:47:55.215 INFO:tasks.workunit.client.0.smithi063.stdout:1/728: dread d3/d3f/d4d/dbc/de0/d4b/da5/dd9/f5f [0,4194304] 0 2024-03-23T11:47:55.220 INFO:tasks.workunit.client.0.smithi063.stdout:9/857: dwrite d4/d4c/fca [0,4194304] 0 2024-03-23T11:47:55.221 INFO:tasks.workunit.client.0.smithi063.stdout:0/972: dwrite d2/d14/de8/ddb/f108 [0,4194304] 0 2024-03-23T11:47:55.229 INFO:tasks.workunit.client.0.smithi063.stdout:1/729: rename d3/d3f/d4d/dbc/de0/d4b/db5 to d3/d49/d56/d65/d84/da7/de3 0 2024-03-23T11:47:55.234 INFO:tasks.workunit.client.0.smithi063.stdout:2/712: getdents d2/d23/d2e/d98/dc8 0 2024-03-23T11:47:55.236 INFO:tasks.workunit.client.0.smithi063.stdout:1/730: fdatasync d3/d62/dad/fd5 0 2024-03-23T11:47:55.243 INFO:tasks.workunit.client.0.smithi063.stdout:2/713: chown d2/d1c/l3d 69648 1 2024-03-23T11:47:55.243 INFO:tasks.workunit.client.0.smithi063.stdout:0/973: dwrite d2/d12/d32/d6d/d124/d28/d46/d51/f82 [4194304,4194304] 0 2024-03-23T11:47:55.249 INFO:tasks.workunit.client.0.smithi063.stdout:2/714: symlink d2/de/d7a/d8d/le6 0 2024-03-23T11:47:55.252 INFO:tasks.workunit.client.0.smithi063.stdout:0/974: creat d2/d14/d36/f15c x:0 0 0 2024-03-23T11:47:55.255 INFO:tasks.workunit.client.0.smithi063.stdout:9/858: dwrite d4/d31/d43/d49/f9a [0,4194304] 0 2024-03-23T11:47:55.258 INFO:tasks.workunit.client.0.smithi063.stdout:2/715: truncate d2/f6b 846977 0 2024-03-23T11:47:55.258 INFO:tasks.workunit.client.0.smithi063.stdout:1/731: dwrite d3/d35/fae [0,4194304] 0 2024-03-23T11:47:55.262 INFO:tasks.workunit.client.0.smithi063.stdout:1/732: chown d3/l2e 2861 1 2024-03-23T11:47:55.266 INFO:tasks.workunit.client.0.smithi063.stdout:1/733: readlink d3/d35/d40/l4f 0 2024-03-23T11:47:55.267 INFO:tasks.workunit.client.0.smithi063.stdout:2/716: dread d2/f37 [0,4194304] 0 2024-03-23T11:47:55.268 INFO:tasks.workunit.client.0.smithi063.stdout:1/734: chown d3/df 1224778185 1 2024-03-23T11:47:55.270 INFO:tasks.workunit.client.0.smithi063.stdout:9/859: rename d4/d31/d43/da9/cf3 to d4/d31/d43/da9/da6/dbe/c122 0 2024-03-23T11:47:55.272 INFO:tasks.workunit.client.0.smithi063.stdout:1/735: fsync d3/d3f/d4d/dbc/de0/faa 0 2024-03-23T11:47:55.273 INFO:tasks.workunit.client.0.smithi063.stdout:2/717: truncate d2/d23/fce 1005283 0 2024-03-23T11:47:55.279 INFO:tasks.workunit.client.0.smithi063.stdout:1/736: fdatasync d3/df/d2f/fcc 0 2024-03-23T11:47:55.280 INFO:tasks.workunit.client.0.smithi063.stdout:9/860: unlink d4/d4c/d19/f27 0 2024-03-23T11:47:55.281 INFO:tasks.workunit.client.0.smithi063.stdout:0/975: dwrite d2/d12/d32/d6d/d124/d1f/f112 [0,4194304] 0 2024-03-23T11:47:55.282 INFO:tasks.workunit.client.0.smithi063.stdout:1/737: chown d3/df/l13 94381 1 2024-03-23T11:47:55.291 INFO:tasks.workunit.client.0.smithi063.stdout:9/861: symlink d4/d31/d9c/d11b/d8d/db3/l123 0 2024-03-23T11:47:55.293 INFO:tasks.workunit.client.0.smithi063.stdout:2/718: dread d2/d23/d56/f57 [0,4194304] 0 2024-03-23T11:47:55.294 INFO:tasks.workunit.client.0.smithi063.stdout:9/862: chown d4/d31/d43/da9/laa 2 1 2024-03-23T11:47:55.299 INFO:tasks.workunit.client.0.smithi063.stdout:1/738: getdents d3/d3f 0 2024-03-23T11:47:55.299 INFO:tasks.workunit.client.0.smithi063.stdout:0/976: link d2/d14/de8/ddb/l142 d2/d12/d32/d6d/d124/d1f/d4f/l15d 0 2024-03-23T11:47:55.301 INFO:tasks.workunit.client.0.smithi063.stdout:2/719: unlink d2/de/d7a/d8d/lb3 0 2024-03-23T11:47:55.303 INFO:tasks.workunit.client.0.smithi063.stdout:1/739: mkdir d3/df/d2f/de4 0 2024-03-23T11:47:55.304 INFO:tasks.workunit.client.0.smithi063.stdout:9/863: truncate d4/d31/d43/d49/fcf 155103 0 2024-03-23T11:47:55.307 INFO:tasks.workunit.client.0.smithi063.stdout:2/720: symlink d2/d23/d2e/d52/d86/le7 0 2024-03-23T11:47:55.310 INFO:tasks.workunit.client.0.smithi063.stdout:0/977: mknod d2/d4/c15e 0 2024-03-23T11:47:55.311 INFO:tasks.workunit.client.0.smithi063.stdout:1/740: creat d3/d49/d56/d65/db6/ddb/fe5 x:0 0 0 2024-03-23T11:47:55.312 INFO:tasks.workunit.client.0.smithi063.stdout:2/721: fsync d2/d23/d56/dad/fd8 0 2024-03-23T11:47:55.317 INFO:tasks.workunit.client.0.smithi063.stdout:2/722: symlink d2/d23/d2e/d98/le8 0 2024-03-23T11:47:55.324 INFO:tasks.workunit.client.0.smithi063.stdout:9/864: dwrite d4/d31/d9c/d11b/d8d/d11c/f11f [0,4194304] 0 2024-03-23T11:47:55.326 INFO:tasks.workunit.client.0.smithi063.stdout:9/865: write d4/fae [124396,89197] 0 2024-03-23T11:47:55.327 INFO:tasks.workunit.client.0.smithi063.stdout:1/741: link d3/d3f/d4d/dbc/de0/d4b/f8b d3/d49/d56/d65/d88/fe6 0 2024-03-23T11:47:55.335 INFO:tasks.workunit.client.0.smithi063.stdout:0/978: write d2/d4/f9 [2932803,109666] 0 2024-03-23T11:47:55.341 INFO:tasks.workunit.client.0.smithi063.stdout:1/742: mknod d3/d49/d56/d65/db6/dc5/ce7 0 2024-03-23T11:47:55.345 INFO:tasks.workunit.client.0.smithi063.stdout:0/979: write d2/d48/fcb [1977943,20778] 0 2024-03-23T11:47:55.345 INFO:tasks.workunit.client.0.smithi063.stdout:9/866: unlink d4/d4c/d19/f33 0 2024-03-23T11:47:55.349 INFO:tasks.workunit.client.0.smithi063.stdout:9/867: fsync d4/d4c/f5c 0 2024-03-23T11:47:55.351 INFO:tasks.workunit.client.0.smithi063.stdout:0/980: symlink d2/d4/l15f 0 2024-03-23T11:47:55.354 INFO:tasks.workunit.client.0.smithi063.stdout:2/723: dread d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:55.355 INFO:tasks.workunit.client.0.smithi063.stdout:1/743: rename d3/d49/d56/d65/d84/c87 to d3/df/ce8 0 2024-03-23T11:47:55.357 INFO:tasks.workunit.client.0.smithi063.stdout:1/744: read d3/d3f/fa3 [2219787,19356] 0 2024-03-23T11:47:55.360 INFO:tasks.workunit.client.0.smithi063.stdout:1/745: stat d3/d3f/d4d/dbc/de0/dce 0 2024-03-23T11:47:55.360 INFO:tasks.workunit.client.0.smithi063.stdout:0/981: unlink d2/d48/de2/c125 0 2024-03-23T11:47:55.363 INFO:tasks.workunit.client.0.smithi063.stdout:2/724: fdatasync d2/de/d7a/da2/fab 0 2024-03-23T11:47:55.369 INFO:tasks.workunit.client.0.smithi063.stdout:9/868: dread d4/d31/d43/d49/d8e/fbc [0,4194304] 0 2024-03-23T11:47:55.370 INFO:tasks.workunit.client.0.smithi063.stdout:9/869: stat d4/d31/d43/da9/f80 0 2024-03-23T11:47:55.378 INFO:tasks.workunit.client.0.smithi063.stdout:1/746: link d3/d3f/d4d/dbc/de0/f6d d3/df/fe9 0 2024-03-23T11:47:55.380 INFO:tasks.workunit.client.0.smithi063.stdout:9/870: mkdir d4/d31/d9c/d11b/d8d/d11c/d124 0 2024-03-23T11:47:55.380 INFO:tasks.workunit.client.0.smithi063.stdout:1/747: chown d3/d3f/d4d/d52/lc2 80492 1 2024-03-23T11:47:55.382 INFO:tasks.workunit.client.0.smithi063.stdout:9/871: write d4/d31/d43/d49/d8e/ff7 [468659,11385] 0 2024-03-23T11:47:55.385 INFO:tasks.workunit.client.0.smithi063.stdout:2/725: write d2/d23/d56/dad/dd4/fd9 [1103414,44870] 0 2024-03-23T11:47:55.387 INFO:tasks.workunit.client.0.smithi063.stdout:1/748: rename d3/d62/c6b to d3/d62/dad/cea 0 2024-03-23T11:47:55.390 INFO:tasks.workunit.client.0.smithi063.stdout:9/872: stat d4/d31/d9c/d11b/d8d/c106 0 2024-03-23T11:47:55.391 INFO:tasks.workunit.client.0.smithi063.stdout:0/982: truncate d2/d12/d32/d6d/d124/d25/f122 878721 0 2024-03-23T11:47:55.392 INFO:tasks.workunit.client.0.smithi063.stdout:2/726: mkdir d2/d23/d2e/d98/dc8/de9 0 2024-03-23T11:47:55.394 INFO:tasks.workunit.client.0.smithi063.stdout:9/873: creat d4/d31/d43/da9/da6/dbe/f125 x:0 0 0 2024-03-23T11:47:55.398 INFO:tasks.workunit.client.0.smithi063.stdout:1/749: creat d3/feb x:0 0 0 2024-03-23T11:47:55.402 INFO:tasks.workunit.client.0.smithi063.stdout:9/874: creat d4/d31/d9c/d11b/d8d/d11c/d124/f126 x:0 0 0 2024-03-23T11:47:55.406 INFO:tasks.workunit.client.0.smithi063.stdout:1/750: rename d3/df/d2f/c70 to d3/d49/d56/d65/db6/dc5/cec 0 2024-03-23T11:47:55.407 INFO:tasks.workunit.client.0.smithi063.stdout:2/727: link d2/de/d22/l48 d2/d23/d2e/d52/lea 0 2024-03-23T11:47:55.410 INFO:tasks.workunit.client.0.smithi063.stdout:9/875: symlink d4/d31/d9c/d11b/d8d/d90/d108/l127 0 2024-03-23T11:47:55.413 INFO:tasks.workunit.client.0.smithi063.stdout:1/751: mkdir d3/d3f/d4d/dbc/de0/da8/ded 0 2024-03-23T11:47:55.413 INFO:tasks.workunit.client.0.smithi063.stdout:0/983: dread d2/d14/d36/f61 [0,4194304] 0 2024-03-23T11:47:55.415 INFO:tasks.workunit.client.0.smithi063.stdout:2/728: getdents d2/d23/d56/dad/dd4 0 2024-03-23T11:47:55.416 INFO:tasks.workunit.client.0.smithi063.stdout:1/752: mkdir d3/d35/d40/dee 0 2024-03-23T11:47:55.419 INFO:tasks.workunit.client.0.smithi063.stdout:9/876: creat d4/d4c/f128 x:0 0 0 2024-03-23T11:47:55.420 INFO:tasks.workunit.client.0.smithi063.stdout:2/729: write d2/d23/d56/fc7 [580509,101305] 0 2024-03-23T11:47:55.424 INFO:tasks.workunit.client.0.smithi063.stdout:9/877: write d4/d31/d9c/d11b/d8d/d11c/d124/f126 [934853,84156] 0 2024-03-23T11:47:55.425 INFO:tasks.workunit.client.0.smithi063.stdout:1/753: rename d3/d3f/d4d/dbc/de0/d4b/f8d to d3/d62/fef 0 2024-03-23T11:47:55.442 INFO:tasks.workunit.client.0.smithi063.stdout:0/984: dread d2/d6e/f11e [0,4194304] 0 2024-03-23T11:47:55.446 INFO:tasks.workunit.client.0.smithi063.stdout:0/985: chown d2/d12/d32/d6d/d124/d1f/d4f/cf5 2 1 2024-03-23T11:47:55.447 INFO:tasks.workunit.client.0.smithi063.stdout:1/754: dwrite d3/df/db8/fc4 [0,4194304] 0 2024-03-23T11:47:55.458 INFO:tasks.workunit.client.0.smithi063.stdout:9/878: dread d4/d31/f59 [4194304,4194304] 0 2024-03-23T11:47:55.462 INFO:tasks.workunit.client.0.smithi063.stdout:2/730: write d2/d23/d2e/d52/f4c [3264478,83291] 0 2024-03-23T11:47:55.470 INFO:tasks.workunit.client.0.smithi063.stdout:1/755: dwrite d3/d49/d56/d65/d88/d92/fda [0,4194304] 0 2024-03-23T11:47:55.474 INFO:tasks.workunit.client.0.smithi063.stdout:2/731: truncate d2/d1c/f96 1002509 0 2024-03-23T11:47:55.480 INFO:tasks.workunit.client.0.smithi063.stdout:2/732: creat d2/d23/d2e/d98/dd2/feb x:0 0 0 2024-03-23T11:47:55.482 INFO:tasks.workunit.client.0.smithi063.stdout:9/879: dwrite d4/d31/d43/da9/da6/f7f [4194304,4194304] 0 2024-03-23T11:47:55.484 INFO:tasks.workunit.client.0.smithi063.stdout:0/986: dwrite d2/d12/d32/d6d/d124/d1f/d4f/fed [0,4194304] 0 2024-03-23T11:47:55.487 INFO:tasks.workunit.client.0.smithi063.stdout:2/733: mkdir d2/d23/d2e/d87/dec 0 2024-03-23T11:47:55.489 INFO:tasks.workunit.client.0.smithi063.stdout:9/880: mkdir d4/d31/d43/da9/da6/db5/d129 0 2024-03-23T11:47:55.497 INFO:tasks.workunit.client.0.smithi063.stdout:9/881: chown d4/d4c/d19/f22 651 1 2024-03-23T11:47:55.498 INFO:tasks.workunit.client.0.smithi063.stdout:2/734: write d2/d1c/f20 [5232845,102312] 0 2024-03-23T11:47:55.501 INFO:tasks.workunit.client.0.smithi063.stdout:9/882: chown d4/d31/d43/da9/da6/f7f 72159 1 2024-03-23T11:47:55.505 INFO:tasks.workunit.client.0.smithi063.stdout:1/756: dwrite d3/df/db8/dcb/fe1 [0,4194304] 0 2024-03-23T11:47:55.506 INFO:tasks.workunit.client.0.smithi063.stdout:2/735: unlink d2/d23/d2e/d52/d6c/c76 0 2024-03-23T11:47:55.514 INFO:tasks.workunit.client.0.smithi063.stdout:2/736: readlink d2/de/l1a 0 2024-03-23T11:47:55.519 INFO:tasks.workunit.client.0.smithi063.stdout:2/737: mknod d2/d23/d56/dad/ced 0 2024-03-23T11:47:55.528 INFO:tasks.workunit.client.0.smithi063.stdout:9/883: write d4/f2a [2752517,58184] 0 2024-03-23T11:47:55.535 INFO:tasks.workunit.client.0.smithi063.stdout:0/987: dwrite d2/d14/d36/d77/d94/daf/f137 [0,4194304] 0 2024-03-23T11:47:55.539 INFO:tasks.workunit.client.0.smithi063.stdout:1/757: dwrite d3/d3f/d4d/dbc/de0/d4b/f71 [0,4194304] 0 2024-03-23T11:47:55.550 INFO:tasks.workunit.client.0.smithi063.stdout:9/884: mkdir d4/dcd/d12a 0 2024-03-23T11:47:55.551 INFO:tasks.workunit.client.0.smithi063.stdout:0/988: rmdir d2 39 2024-03-23T11:47:55.554 INFO:tasks.workunit.client.0.smithi063.stdout:1/758: creat d3/df/d2f/ff0 x:0 0 0 2024-03-23T11:47:55.555 INFO:tasks.workunit.client.0.smithi063.stdout:2/738: symlink d2/d23/d2e/d98/dc8/de9/lee 0 2024-03-23T11:47:55.557 INFO:tasks.workunit.client.0.smithi063.stdout:2/739: dread - d2/de/d22/dbc/fdb zero size 2024-03-23T11:47:55.560 INFO:tasks.workunit.client.0.smithi063.stdout:1/759: rename d3/d35/d40/f42 to d3/df/d2f/d9a/ff1 0 2024-03-23T11:47:55.565 INFO:tasks.workunit.client.0.smithi063.stdout:0/989: unlink d2/cb8 0 2024-03-23T11:47:55.569 INFO:tasks.workunit.client.0.smithi063.stdout:0/990: dread - d2/d14/d36/d77/d94/daf/ffd zero size 2024-03-23T11:47:55.571 INFO:tasks.workunit.client.0.smithi063.stdout:1/760: symlink d3/df/d2f/d9a/dd6/lf2 0 2024-03-23T11:47:55.573 INFO:tasks.workunit.client.0.smithi063.stdout:2/740: link d2/d84/d92/fb0 d2/d23/d2e/d87/fef 0 2024-03-23T11:47:55.576 INFO:tasks.workunit.client.0.smithi063.stdout:2/741: read d2/f29 [3416374,117239] 0 2024-03-23T11:47:55.578 INFO:tasks.workunit.client.0.smithi063.stdout:2/742: write d2/d1c/f5f [3488805,72305] 0 2024-03-23T11:47:55.578 INFO:tasks.workunit.client.0.smithi063.stdout:1/761: fsync d3/d3f/d4d/dbc/de0/d4b/f8b 0 2024-03-23T11:47:55.581 INFO:tasks.workunit.client.0.smithi063.stdout:2/743: chown d2/d23/d2e/ld3 1213 1 2024-03-23T11:47:55.586 INFO:tasks.workunit.client.0.smithi063.stdout:1/762: rename d3/df/c15 to d3/d3f/d4d/cf3 0 2024-03-23T11:47:55.588 INFO:tasks.workunit.client.0.smithi063.stdout:1/763: chown d3/d3f/d4d/dbc/de0/da8/faf 1707755 1 2024-03-23T11:47:55.590 INFO:tasks.workunit.client.0.smithi063.stdout:2/744: symlink d2/d23/d56/lf0 0 2024-03-23T11:47:55.590 INFO:tasks.workunit.client.0.smithi063.stdout:1/764: chown d3/d3f/d4d/dbc/de0/f1e 295013812 1 2024-03-23T11:47:55.595 INFO:tasks.workunit.client.0.smithi063.stdout:2/745: rename d2/d23/d2e/d52/d86/dbf/cd1 to d2/de/d7a/d77/cf1 0 2024-03-23T11:47:55.596 INFO:tasks.workunit.client.0.smithi063.stdout:2/746: chown d2/d1c/f3c 13476756 1 2024-03-23T11:47:55.601 INFO:tasks.workunit.client.0.smithi063.stdout:1/765: rename d3/d49/d56/d65/db6/ddb to d3/d49/df4 0 2024-03-23T11:47:55.603 INFO:tasks.workunit.client.0.smithi063.stdout:1/766: write d3/df/db8/dcb/fe1 [953007,44022] 0 2024-03-23T11:47:55.605 INFO:tasks.workunit.client.0.smithi063.stdout:0/991: dwrite d2/d6e/fc6 [0,4194304] 0 2024-03-23T11:47:55.609 INFO:tasks.workunit.client.0.smithi063.stdout:2/747: symlink d2/d46/lf2 0 2024-03-23T11:47:55.610 INFO:tasks.workunit.client.0.smithi063.stdout:2/748: readlink d2/d1c/l3d 0 2024-03-23T11:47:55.616 INFO:tasks.workunit.client.0.smithi063.stdout:2/749: symlink d2/d84/lf3 0 2024-03-23T11:47:55.625 INFO:tasks.workunit.client.0.smithi063.stdout:1/767: write d3/d3f/d4d/dbc/de0/d4b/f8a [800109,87251] 0 2024-03-23T11:47:55.631 INFO:tasks.workunit.client.0.smithi063.stdout:0/992: dread d2/d48/fcb [0,4194304] 0 2024-03-23T11:47:55.633 INFO:tasks.workunit.client.0.smithi063.stdout:0/993: creat d2/d6e/f160 x:0 0 0 2024-03-23T11:47:55.634 INFO:tasks.workunit.client.0.smithi063.stdout:1/768: write d3/df/d2f/f64 [2046091,61185] 0 2024-03-23T11:47:55.639 INFO:tasks.workunit.client.0.smithi063.stdout:0/994: symlink d2/d14/d36/d77/d94/d129/dcf/l161 0 2024-03-23T11:47:55.640 INFO:tasks.workunit.client.0.smithi063.stdout:1/769: truncate d3/df/fe9 4266632 0 2024-03-23T11:47:55.642 INFO:tasks.workunit.client.0.smithi063.stdout:0/995: truncate d2/d12/f21 4473626 0 2024-03-23T11:47:55.645 INFO:tasks.workunit.client.0.smithi063.stdout:1/770: rename d3/d49/d56/d65/d88/d92/d9e to d3/df/d2f/de4/df5 0 2024-03-23T11:47:55.647 INFO:tasks.workunit.client.0.smithi063.stdout:1/771: fsync d3/d3f/d4d/d52/fab 0 2024-03-23T11:47:55.664 INFO:tasks.workunit.client.0.smithi063.stdout:1/772: dwrite d3/df/d2f/fcc [0,4194304] 0 2024-03-23T11:47:55.670 INFO:tasks.workunit.client.0.smithi063.stdout:0/996: dread d2/d14/f23 [0,4194304] 0 2024-03-23T11:47:55.673 INFO:tasks.workunit.client.0.smithi063.stdout:0/997: chown d2/d12/d32/d6d/d124/d25/d93/ff7 2 1 2024-03-23T11:47:55.676 INFO:tasks.workunit.client.0.smithi063.stdout:0/998: mkdir d2/d12/d32/d6d/d124/d1f/d4f/d126/d162 0 2024-03-23T11:47:55.680 INFO:tasks.workunit.client.0.smithi063.stdout:0/999: truncate d2/d14/d36/d77/f132 311150 0 2024-03-23T11:47:55.813 INFO:tasks.workunit.client.0.smithi063.stdout:4/775: sync 2024-03-23T11:47:55.813 INFO:tasks.workunit.client.0.smithi063.stdout:6/681: sync 2024-03-23T11:47:55.813 INFO:tasks.workunit.client.0.smithi063.stdout:9/885: sync 2024-03-23T11:47:55.832 INFO:tasks.workunit.client.0.smithi063.stdout:9/886: dwrite d4/d31/d43/da9/f95 [0,4194304] 0 2024-03-23T11:47:55.833 INFO:tasks.workunit.client.0.smithi063.stdout:4/776: dwrite d5/da/d20/d4b/fa5 [0,4194304] 0 2024-03-23T11:47:55.841 INFO:tasks.workunit.client.0.smithi063.stdout:4/777: unlink d5/l6 0 2024-03-23T11:47:55.874 INFO:tasks.workunit.client.0.smithi063.stdout:9/887: write d4/d4c/f40 [309309,55780] 0 2024-03-23T11:47:55.883 INFO:tasks.workunit.client.0.smithi063.stdout:9/888: creat d4/d31/d43/f12b x:0 0 0 2024-03-23T11:47:55.885 INFO:tasks.workunit.client.0.smithi063.stdout:4/778: dwrite d5/d1e/f51 [0,4194304] 0 2024-03-23T11:47:55.887 INFO:tasks.workunit.client.0.smithi063.stdout:9/889: read d4/d31/d43/fdd [5264236,13437] 0 2024-03-23T11:47:55.894 INFO:tasks.workunit.client.0.smithi063.stdout:9/890: unlink d4/d4c/c82 0 2024-03-23T11:47:55.896 INFO:tasks.workunit.client.0.smithi063.stdout:4/779: write d5/ff [364852,118570] 0 2024-03-23T11:47:55.899 INFO:tasks.workunit.client.0.smithi063.stdout:9/891: creat d4/d31/d9c/d11b/d8d/d11c/d124/f12c x:0 0 0 2024-03-23T11:47:55.901 INFO:tasks.workunit.client.0.smithi063.stdout:4/780: write d5/da/d20/d4b/d29/d38/f55 [1589490,13589] 0 2024-03-23T11:47:55.905 INFO:tasks.workunit.client.0.smithi063.stdout:9/892: dread d4/d4c/f81 [0,4194304] 0 2024-03-23T11:47:55.907 INFO:tasks.workunit.client.0.smithi063.stdout:4/781: creat d5/da/d20/d4b/d16/d50/d8e/dd3/fde x:0 0 0 2024-03-23T11:47:55.913 INFO:tasks.workunit.client.0.smithi063.stdout:4/782: rename d5/da/d72 to d5/da/d20/d4b/d16/d50/d8e/d96/ddf 0 2024-03-23T11:47:55.923 INFO:tasks.workunit.client.0.smithi063.stdout:4/783: write d5/da/d20/d4b/d16/d50/f5f [4416371,110521] 0 2024-03-23T11:47:55.928 INFO:tasks.workunit.client.0.smithi063.stdout:4/784: creat d5/da/d20/d4b/d29/fe0 x:0 0 0 2024-03-23T11:47:55.931 INFO:tasks.workunit.client.0.smithi063.stdout:9/893: dread d4/d31/d43/d49/f9a [4194304,4194304] 0 2024-03-23T11:47:55.934 INFO:tasks.workunit.client.0.smithi063.stdout:9/894: fdatasync d4/d31/d9c/d11b/d8d/d11c/d124/f12c 0 2024-03-23T11:47:55.934 INFO:tasks.workunit.client.0.smithi063.stdout:4/785: rename d5/da/d20/d4b/d16/c61 to d5/da/d20/d4b/d16/d32/d56/ce1 0 2024-03-23T11:47:55.938 INFO:tasks.workunit.client.0.smithi063.stdout:4/786: mknod d5/d1e/d2b/daa/ce2 0 2024-03-23T11:47:55.945 INFO:tasks.workunit.client.0.smithi063.stdout:4/787: mkdir d5/da/d20/d4b/d29/d38/d7a/dd5/de3 0 2024-03-23T11:47:55.949 INFO:tasks.workunit.client.0.smithi063.stdout:9/895: dwrite d4/d31/d9c/d11b/d8d/db3/fe0 [0,4194304] 0 2024-03-23T11:47:55.955 INFO:tasks.workunit.client.0.smithi063.stdout:9/896: mkdir d4/d31/d9c/d11b/d8d/da0/d12d 0 2024-03-23T11:47:55.955 INFO:tasks.workunit.client.0.smithi063.stdout:4/788: write d5/da/d20/d4b/fa9 [960730,41233] 0 2024-03-23T11:47:55.958 INFO:tasks.workunit.client.0.smithi063.stdout:4/789: dread - d5/da/d20/d4b/d16/d50/d8e/d96/ddf/f92 zero size 2024-03-23T11:47:55.958 INFO:tasks.workunit.client.0.smithi063.stdout:9/897: mknod d4/d31/d43/da9/da6/dbe/c12e 0 2024-03-23T11:47:55.974 INFO:tasks.workunit.client.0.smithi063.stdout:2/750: sync 2024-03-23T11:47:55.974 INFO:tasks.workunit.client.0.smithi063.stdout:7/758: sync 2024-03-23T11:47:55.974 INFO:tasks.workunit.client.0.smithi063.stdout:5/554: sync 2024-03-23T11:47:55.974 INFO:tasks.workunit.client.0.smithi063.stdout:1/773: sync 2024-03-23T11:47:55.977 INFO:tasks.workunit.client.0.smithi063.stdout:9/898: dwrite d4/d4c/fca [0,4194304] 0 2024-03-23T11:47:55.978 INFO:tasks.workunit.client.0.smithi063.stdout:1/774: write d3/d3f/d4d/f95 [156489,91140] 0 2024-03-23T11:47:55.979 INFO:tasks.workunit.client.0.smithi063.stdout:7/759: stat d5/d5c/d2f/d8b/daa/lac 0 2024-03-23T11:47:55.981 INFO:tasks.workunit.client.0.smithi063.stdout:2/751: mknod d2/d23/d56/dad/dd4/dd6/cf4 0 2024-03-23T11:47:55.984 INFO:tasks.workunit.client.0.smithi063.stdout:5/555: symlink d2/d28/lc3 0 2024-03-23T11:47:55.987 INFO:tasks.workunit.client.0.smithi063.stdout:5/556: stat d2/d7/d64/c9c 0 2024-03-23T11:47:55.988 INFO:tasks.workunit.client.0.smithi063.stdout:7/760: write f1 [3917978,44033] 0 2024-03-23T11:47:55.990 INFO:tasks.workunit.client.0.smithi063.stdout:2/752: mkdir d2/d23/d2e/d98/dc4/dcc/df5 0 2024-03-23T11:47:55.994 INFO:tasks.workunit.client.0.smithi063.stdout:7/761: read - d5/d5c/d2f/fc3 zero size 2024-03-23T11:47:55.998 INFO:tasks.workunit.client.0.smithi063.stdout:2/753: mkdir d2/d23/d2e/d98/dc8/df6 0 2024-03-23T11:47:56.002 INFO:tasks.workunit.client.0.smithi063.stdout:1/775: rename d3/c5 to d3/d49/cf6 0 2024-03-23T11:47:56.004 INFO:tasks.workunit.client.0.smithi063.stdout:5/557: truncate d2/d7/d8/de/d47/fab 1249878 0 2024-03-23T11:47:56.005 INFO:tasks.workunit.client.0.smithi063.stdout:4/790: dread d5/da/d20/d4b/d16/d50/d8e/d96/ddf/fb1 [0,4194304] 0 2024-03-23T11:47:56.006 INFO:tasks.workunit.client.0.smithi063.stdout:7/762: unlink d5/d5c/d2f/d8b/daa/cc1 0 2024-03-23T11:47:56.007 INFO:tasks.workunit.client.0.smithi063.stdout:5/558: chown d2/d7/d8/de/d2a/d3b/d43/d5c/dc2 14555612 1 2024-03-23T11:47:56.009 INFO:tasks.workunit.client.0.smithi063.stdout:2/754: creat d2/de/d22/d91/ff7 x:0 0 0 2024-03-23T11:47:56.011 INFO:tasks.workunit.client.0.smithi063.stdout:2/755: truncate d2/fdc 521783 0 2024-03-23T11:47:56.014 INFO:tasks.workunit.client.0.smithi063.stdout:5/559: mknod d2/d7/d8/d1d/cc4 0 2024-03-23T11:47:56.014 INFO:tasks.workunit.client.0.smithi063.stdout:4/791: getdents d5/da/d20/d4b/d16/d50/d8e/dad/ddd 0 2024-03-23T11:47:56.017 INFO:tasks.workunit.client.0.smithi063.stdout:9/899: dread d4/d31/d43/da9/f80 [4194304,4194304] 0 2024-03-23T11:47:56.024 INFO:tasks.workunit.client.0.smithi063.stdout:4/792: unlink d5/da/d20/d4b/d16/d50/d8e/dd3/c73 0 2024-03-23T11:47:56.025 INFO:tasks.workunit.client.0.smithi063.stdout:5/560: mknod d2/d28/d3a/d46/d79/dc0/cc5 0 2024-03-23T11:47:56.026 INFO:tasks.workunit.client.0.smithi063.stdout:7/763: dwrite d5/d5c/d2f/f79 [0,4194304] 0 2024-03-23T11:47:56.035 INFO:tasks.workunit.client.0.smithi063.stdout:5/561: mkdir d2/d7/d2b/d8b/da7/dc6 0 2024-03-23T11:47:56.041 INFO:tasks.workunit.client.0.smithi063.stdout:7/764: mknod d5/d26/db7/dd6/d88/dc2/cea 0 2024-03-23T11:47:56.042 INFO:tasks.workunit.client.0.smithi063.stdout:9/900: write d4/d31/f3d [1240496,96036] 0 2024-03-23T11:47:56.045 INFO:tasks.workunit.client.0.smithi063.stdout:7/765: rmdir d5/d26/db7/dc6/d6e/da2 39 2024-03-23T11:47:56.046 INFO:tasks.workunit.client.0.smithi063.stdout:4/793: getdents d5/da/d20/d4b/d16/dc4 0 2024-03-23T11:47:56.047 INFO:tasks.workunit.client.0.smithi063.stdout:7/766: readlink d5/d26/db7/dc6/d6e/ld1 0 2024-03-23T11:47:56.049 INFO:tasks.workunit.client.0.smithi063.stdout:4/794: truncate d5/d1e/fb9 5421154 0 2024-03-23T11:47:56.051 INFO:tasks.workunit.client.0.smithi063.stdout:9/901: truncate d4/d31/d43/d49/f67 9140467 0 2024-03-23T11:47:56.054 INFO:tasks.workunit.client.0.smithi063.stdout:7/767: mknod d5/d26/db7/dce/dc8/ceb 0 2024-03-23T11:47:56.056 INFO:tasks.workunit.client.0.smithi063.stdout:9/902: rename d4/d4c/d44 to d4/d31/d9c/d9e/d12f 0 2024-03-23T11:47:56.056 INFO:tasks.workunit.client.0.smithi063.stdout:1/776: dread d3/d49/d56/d65/d84/f8e [0,4194304] 0 2024-03-23T11:47:56.058 INFO:tasks.workunit.client.0.smithi063.stdout:9/903: write d4/d4c/d19/f23 [1320480,125353] 0 2024-03-23T11:47:56.059 INFO:tasks.workunit.client.0.smithi063.stdout:1/777: read d3/df/f36 [1304373,54783] 0 2024-03-23T11:47:56.060 INFO:tasks.workunit.client.0.smithi063.stdout:7/768: fdatasync d5/d6/d19/f92 0 2024-03-23T11:47:56.062 INFO:tasks.workunit.client.0.smithi063.stdout:4/795: creat d5/da/d20/d4b/d29/d38/fe4 x:0 0 0 2024-03-23T11:47:56.062 INFO:tasks.workunit.client.0.smithi063.stdout:9/904: read d4/f21 [4746583,74517] 0 2024-03-23T11:47:56.063 INFO:tasks.workunit.client.0.smithi063.stdout:7/769: chown d5/d6/d19/d1f 3 1 2024-03-23T11:47:56.063 INFO:tasks.workunit.client.0.smithi063.stdout:1/778: chown d3/df/db8/dcb/fe1 524671 1 2024-03-23T11:47:56.073 INFO:tasks.workunit.client.0.smithi063.stdout:9/905: mkdir d4/d31/d9c/d11b/d8d/da0/d130 0 2024-03-23T11:47:56.078 INFO:tasks.workunit.client.0.smithi063.stdout:1/779: creat d3/df/db8/dcb/ff7 x:0 0 0 2024-03-23T11:47:56.081 INFO:tasks.workunit.client.0.smithi063.stdout:4/796: truncate d5/d1e/d2b/f54 398979 0 2024-03-23T11:47:56.083 INFO:tasks.workunit.client.0.smithi063.stdout:7/770: creat d5/d22/da3/fec x:0 0 0 2024-03-23T11:47:56.084 INFO:tasks.workunit.client.0.smithi063.stdout:1/780: rmdir d3/d49/d56/d65/d88 39 2024-03-23T11:47:56.087 INFO:tasks.workunit.client.0.smithi063.stdout:9/906: dread d4/d31/d9c/d11b/d8d/fc6 [0,4194304] 0 2024-03-23T11:47:56.098 INFO:tasks.workunit.client.0.smithi063.stdout:4/797: creat d5/da/d20/d4b/d16/d50/fe5 x:0 0 0 2024-03-23T11:47:56.100 INFO:tasks.workunit.client.0.smithi063.stdout:1/781: fsync d3/d49/d56/d65/d88/d92/fde 0 2024-03-23T11:47:56.100 INFO:tasks.workunit.client.0.smithi063.stdout:4/798: chown d5/da/d20/l80 59276402 1 2024-03-23T11:47:56.105 INFO:tasks.workunit.client.0.smithi063.stdout:9/907: rename f2 to d4/d31/d43/f131 0 2024-03-23T11:47:56.107 INFO:tasks.workunit.client.0.smithi063.stdout:7/771: link d5/l14 d5/d5c/d2f/led 0 2024-03-23T11:47:56.107 INFO:tasks.workunit.client.0.smithi063.stdout:9/908: read d4/d31/d9c/d11b/d8d/d11c/d124/f126 [978000,102902] 0 2024-03-23T11:47:56.109 INFO:tasks.workunit.client.0.smithi063.stdout:4/799: rmdir d5/da/d20/d4b/d16/d50/d8e/dd3 39 2024-03-23T11:47:56.115 INFO:tasks.workunit.client.0.smithi063.stdout:1/782: rename d3/d49/d56/d65/f79 to d3/d3f/d4d/d52/ff8 0 2024-03-23T11:47:56.118 INFO:tasks.workunit.client.0.smithi063.stdout:4/800: creat d5/da/d20/d4b/d1f/fe6 x:0 0 0 2024-03-23T11:47:56.125 INFO:tasks.workunit.client.0.smithi063.stdout:4/801: unlink d5/d1e/f51 0 2024-03-23T11:47:56.130 INFO:tasks.workunit.client.0.smithi063.stdout:9/909: getdents d4/d31/d10b 0 2024-03-23T11:47:56.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:55 smithi063 ceph-mon[35189]: pgmap v47: 65 pgs: 65 active+clean; 3.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 78 MiB/s rd, 186 MiB/s wr, 463 op/s 2024-03-23T11:47:56.137 INFO:tasks.workunit.client.0.smithi063.stdout:4/802: link d5/da/dc3/fcb d5/da/d7e/dbf/fe7 0 2024-03-23T11:47:56.138 INFO:tasks.workunit.client.0.smithi063.stdout:1/783: dwrite d3/d49/d56/d65/f7c [4194304,4194304] 0 2024-03-23T11:47:56.142 INFO:tasks.workunit.client.0.smithi063.stdout:7/772: dread d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:56.147 INFO:tasks.workunit.client.0.smithi063.stdout:1/784: read d3/df/d2f/f64 [2615568,52926] 0 2024-03-23T11:47:56.147 INFO:tasks.workunit.client.0.smithi063.stdout:9/910: truncate d4/d31/d9c/d11b/d8d/d11c/d124/f126 1312624 0 2024-03-23T11:47:56.150 INFO:tasks.workunit.client.0.smithi063.stdout:7/773: read - d5/d6/dca/fcb zero size 2024-03-23T11:47:56.152 INFO:tasks.workunit.client.0.smithi063.stdout:4/803: rename d5/da/d20/f3c to d5/da/d20/d4b/d16/d50/d86/da2/fe8 0 2024-03-23T11:47:56.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:55 smithi111 ceph-mon[44005]: pgmap v47: 65 pgs: 65 active+clean; 3.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 78 MiB/s rd, 186 MiB/s wr, 463 op/s 2024-03-23T11:47:56.169 INFO:tasks.workunit.client.0.smithi063.stdout:1/785: dwrite d3/d3f/d4d/f95 [0,4194304] 0 2024-03-23T11:47:56.175 INFO:tasks.workunit.client.0.smithi063.stdout:9/911: truncate d4/d4c/f25 900388 0 2024-03-23T11:47:56.179 INFO:tasks.workunit.client.0.smithi063.stdout:4/804: fdatasync d5/da/d20/d4b/d16/d50/f84 0 2024-03-23T11:47:56.184 INFO:tasks.workunit.client.0.smithi063.stdout:1/786: dread d3/df/fbf [0,4194304] 0 2024-03-23T11:47:56.187 INFO:tasks.workunit.client.0.smithi063.stdout:9/912: getdents d4/d4c/d19 0 2024-03-23T11:47:56.197 INFO:tasks.workunit.client.0.smithi063.stdout:4/805: dread d5/da/d20/d4b/d16/d32/d56/fbb [0,4194304] 0 2024-03-23T11:47:56.198 INFO:tasks.workunit.client.0.smithi063.stdout:1/787: rename d3/df/c2c to d3/d49/d56/d65/db6/dc5/cf9 0 2024-03-23T11:47:56.205 INFO:tasks.workunit.client.0.smithi063.stdout:7/774: dread d5/d5c/d2f/d73/da/f8a [0,4194304] 0 2024-03-23T11:47:56.206 INFO:tasks.workunit.client.0.smithi063.stdout:4/806: creat d5/da/d20/d4b/d16/d50/d86/da2/dd7/fe9 x:0 0 0 2024-03-23T11:47:56.207 INFO:tasks.workunit.client.0.smithi063.stdout:1/788: mkdir d3/d35/d40/dee/dfa 0 2024-03-23T11:47:56.208 INFO:tasks.workunit.client.0.smithi063.stdout:9/913: dwrite d4/d4c/f81 [0,4194304] 0 2024-03-23T11:47:56.211 INFO:tasks.workunit.client.0.smithi063.stdout:9/914: fdatasync d4/d4c/fde 0 2024-03-23T11:47:56.211 INFO:tasks.workunit.client.0.smithi063.stdout:7/775: truncate d5/d5c/d2f/d8b/fe2 432579 0 2024-03-23T11:47:56.212 INFO:tasks.workunit.client.0.smithi063.stdout:4/807: write d5/da/dc3/fcd [288803,98172] 0 2024-03-23T11:47:56.212 INFO:tasks.workunit.client.0.smithi063.stdout:1/789: write d3/d49/d56/d65/db6/dc5/fdf [610868,27645] 0 2024-03-23T11:47:56.217 INFO:tasks.workunit.client.0.smithi063.stdout:1/790: chown d3/df/d2f/d9a/dd6 24 1 2024-03-23T11:47:56.219 INFO:tasks.workunit.client.0.smithi063.stdout:9/915: write d4/d4c/f8c [2364058,66511] 0 2024-03-23T11:47:56.220 INFO:tasks.workunit.client.0.smithi063.stdout:7/776: write d5/f12 [1958249,93577] 0 2024-03-23T11:47:56.222 INFO:tasks.workunit.client.0.smithi063.stdout:4/808: symlink d5/da/d20/d4b/d16/d32/d56/dc7/lea 0 2024-03-23T11:47:56.226 INFO:tasks.workunit.client.0.smithi063.stdout:9/916: rmdir d4/d31/d9c/d11b/d8d/db3 39 2024-03-23T11:47:56.226 INFO:tasks.workunit.client.0.smithi063.stdout:1/791: rename d3/d3f/d4d/dbc/de0/f1e to d3/d62/dad/ffb 0 2024-03-23T11:47:56.230 INFO:tasks.workunit.client.0.smithi063.stdout:7/777: rename d5/d5c/d2f/d73/d20 to d5/d26/db7/dce/dc8/dee 0 2024-03-23T11:47:56.234 INFO:tasks.workunit.client.0.smithi063.stdout:1/792: dread d3/d3f/d4d/dbc/de0/da8/faf [0,4194304] 0 2024-03-23T11:47:56.236 INFO:tasks.workunit.client.0.smithi063.stdout:7/778: creat d5/d26/db7/dc6/d4f/d5b/fef x:0 0 0 2024-03-23T11:47:56.242 INFO:tasks.workunit.client.0.smithi063.stdout:7/779: mkdir d5/d26/db7/dce/d64/df0 0 2024-03-23T11:47:56.243 INFO:tasks.workunit.client.0.smithi063.stdout:9/917: write d4/d31/d9c/d9e/fc5 [2266285,62867] 0 2024-03-23T11:47:56.246 INFO:tasks.workunit.client.0.smithi063.stdout:4/809: dwrite d5/da/d20/d4b/fa9 [0,4194304] 0 2024-03-23T11:47:56.250 INFO:tasks.workunit.client.0.smithi063.stdout:7/780: creat d5/d5c/d2f/ff1 x:0 0 0 2024-03-23T11:47:56.257 INFO:tasks.workunit.client.0.smithi063.stdout:9/918: chown d4/d31/d43/d49/d8e/l116 46399610 1 2024-03-23T11:47:56.266 INFO:tasks.workunit.client.0.smithi063.stdout:1/793: dread d3/d35/d40/f53 [0,4194304] 0 2024-03-23T11:47:56.272 INFO:tasks.workunit.client.0.smithi063.stdout:1/794: mkdir d3/df/d2f/d9a/dfc 0 2024-03-23T11:47:56.284 INFO:tasks.workunit.client.0.smithi063.stdout:9/919: dwrite d4/d31/d43/fdd [4194304,4194304] 0 2024-03-23T11:47:56.286 INFO:tasks.workunit.client.0.smithi063.stdout:1/795: rename d3/f39 to d3/d3f/d4d/dbc/de0/dce/ffd 0 2024-03-23T11:47:56.291 INFO:tasks.workunit.client.0.smithi063.stdout:7/781: dread d5/d26/db7/dd6/d88/fa9 [0,4194304] 0 2024-03-23T11:47:56.291 INFO:tasks.workunit.client.0.smithi063.stdout:1/796: stat d3/d49/d56/d65/d84 0 2024-03-23T11:47:56.291 INFO:tasks.workunit.client.0.smithi063.stdout:9/920: dread - d4/fb1 zero size 2024-03-23T11:47:56.295 INFO:tasks.workunit.client.0.smithi063.stdout:7/782: mknod d5/d26/db7/dd6/d88/cf2 0 2024-03-23T11:47:56.296 INFO:tasks.workunit.client.0.smithi063.stdout:4/810: dread d5/da/d20/d4b/d16/d50/f64 [0,4194304] 0 2024-03-23T11:47:56.299 INFO:tasks.workunit.client.0.smithi063.stdout:9/921: unlink d4/d31/d43/f10d 0 2024-03-23T11:47:56.301 INFO:tasks.workunit.client.0.smithi063.stdout:6/682: sync 2024-03-23T11:47:56.304 INFO:tasks.workunit.client.0.smithi063.stdout:1/797: write d3/d3f/d4d/dbc/de0/d4b/da5/dd9/f5f [2183661,65331] 0 2024-03-23T11:47:56.305 INFO:tasks.workunit.client.0.smithi063.stdout:6/683: read - d1/d15/d4b/f66 zero size 2024-03-23T11:47:56.307 INFO:tasks.workunit.client.0.smithi063.stdout:6/684: dread - d1/dc/d3a/d57/d77/fc9 zero size 2024-03-23T11:47:56.309 INFO:tasks.workunit.client.0.smithi063.stdout:7/783: symlink d5/d22/lf3 0 2024-03-23T11:47:56.311 INFO:tasks.workunit.client.0.smithi063.stdout:7/784: chown l3 1025346 1 2024-03-23T11:47:56.311 INFO:tasks.workunit.client.0.smithi063.stdout:9/922: write d4/d31/f112 [3395834,22401] 0 2024-03-23T11:47:56.313 INFO:tasks.workunit.client.0.smithi063.stdout:1/798: rmdir d3/d3f/d4d 39 2024-03-23T11:47:56.313 INFO:tasks.workunit.client.0.smithi063.stdout:4/811: mkdir d5/da/deb 0 2024-03-23T11:47:56.315 INFO:tasks.workunit.client.0.smithi063.stdout:9/923: unlink d4/f35 0 2024-03-23T11:47:56.334 INFO:tasks.workunit.client.0.smithi063.stdout:6/685: link d1/dc/d3a/f45 d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb/fd6 0 2024-03-23T11:47:56.334 INFO:tasks.workunit.client.0.smithi063.stdout:9/924: getdents d4/d31/d43/da9/da6/d100/d11e 0 2024-03-23T11:47:56.335 INFO:tasks.workunit.client.0.smithi063.stdout:4/812: dwrite d5/da/d20/d4b/d16/d50/d8e/fa7 [0,4194304] 0 2024-03-23T11:47:56.341 INFO:tasks.workunit.client.0.smithi063.stdout:1/799: creat d3/d3f/d4d/dbc/de0/ffe x:0 0 0 2024-03-23T11:47:56.348 INFO:tasks.workunit.client.0.smithi063.stdout:6/686: symlink d1/dc/d3a/d57/ld7 0 2024-03-23T11:47:56.350 INFO:tasks.workunit.client.0.smithi063.stdout:9/925: mkdir d4/d31/d9c/d11b/d8d/da0/d12d/d132 0 2024-03-23T11:47:56.353 INFO:tasks.workunit.client.0.smithi063.stdout:6/687: chown d1/dc/d3a/d57/d77/cc7 1 1 2024-03-23T11:47:56.353 INFO:tasks.workunit.client.0.smithi063.stdout:7/785: dread d5/d6/d19/d1f/d6c/f80 [0,4194304] 0 2024-03-23T11:47:56.355 INFO:tasks.workunit.client.0.smithi063.stdout:1/800: dread d3/d49/d56/f67 [0,4194304] 0 2024-03-23T11:47:56.362 INFO:tasks.workunit.client.0.smithi063.stdout:4/813: dwrite d5/da/d20/d4b/d16/d50/d8e/d96/ddf/fd0 [0,4194304] 0 2024-03-23T11:47:56.362 INFO:tasks.workunit.client.0.smithi063.stdout:6/688: mkdir d1/dc/d3a/d82/da2/dd8 0 2024-03-23T11:47:56.367 INFO:tasks.workunit.client.0.smithi063.stdout:7/786: mkdir d5/d26/db7/dce/d64/df4 0 2024-03-23T11:47:56.367 INFO:tasks.workunit.client.0.smithi063.stdout:9/926: symlink d4/d31/d9c/d11b/d8d/da0/d12d/d132/l133 0 2024-03-23T11:47:56.373 INFO:tasks.workunit.client.0.smithi063.stdout:4/814: symlink d5/da/d20/d4b/d16/dc4/lec 0 2024-03-23T11:47:56.375 INFO:tasks.workunit.client.0.smithi063.stdout:6/689: creat d1/dc/d21/d49/fd9 x:0 0 0 2024-03-23T11:47:56.380 INFO:tasks.workunit.client.0.smithi063.stdout:6/690: chown d1/dc/d3a/d82/da2/d41/d42/fc8 157 1 2024-03-23T11:47:56.381 INFO:tasks.workunit.client.0.smithi063.stdout:1/801: dwrite d3/d3f/d4d/dbc/de0/d4b/da5/dd9/f5f [0,4194304] 0 2024-03-23T11:47:56.383 INFO:tasks.workunit.client.0.smithi063.stdout:2/756: sync 2024-03-23T11:47:56.383 INFO:tasks.workunit.client.0.smithi063.stdout:5/562: sync 2024-03-23T11:47:56.389 INFO:tasks.workunit.client.0.smithi063.stdout:5/563: chown d2/d7/d8/f68 19303 1 2024-03-23T11:47:56.390 INFO:tasks.workunit.client.0.smithi063.stdout:2/757: stat d2/d23/d56/dad/fd8 0 2024-03-23T11:47:56.392 INFO:tasks.workunit.client.0.smithi063.stdout:9/927: dwrite d4/d31/d9c/d9e/fea [0,4194304] 0 2024-03-23T11:47:56.393 INFO:tasks.workunit.client.0.smithi063.stdout:4/815: link d5/da/d20/d4b/d16/d50/f84 d5/da/d7e/fed 0 2024-03-23T11:47:56.399 INFO:tasks.workunit.client.0.smithi063.stdout:2/758: readlink d2/d84/d94/lcf 0 2024-03-23T11:47:56.404 INFO:tasks.workunit.client.0.smithi063.stdout:1/802: dwrite d3/d49/d56/fb7 [0,4194304] 0 2024-03-23T11:47:56.405 INFO:tasks.workunit.client.0.smithi063.stdout:9/928: mkdir d4/d31/d9c/d9e/d134 0 2024-03-23T11:47:56.406 INFO:tasks.workunit.client.0.smithi063.stdout:7/787: truncate f1 194550 0 2024-03-23T11:47:56.414 INFO:tasks.workunit.client.0.smithi063.stdout:6/691: write d1/dc/fd [716299,557] 0 2024-03-23T11:47:56.414 INFO:tasks.workunit.client.0.smithi063.stdout:7/788: chown d5/d6/d19/cdc 38970 1 2024-03-23T11:47:56.417 INFO:tasks.workunit.client.0.smithi063.stdout:6/692: write d1/f12 [118435,56855] 0 2024-03-23T11:47:56.423 INFO:tasks.workunit.client.0.smithi063.stdout:2/759: creat d2/d1c/ff8 x:0 0 0 2024-03-23T11:47:56.427 INFO:tasks.workunit.client.0.smithi063.stdout:2/760: write d2/d23/d56/f75 [930638,87787] 0 2024-03-23T11:47:56.427 INFO:tasks.workunit.client.0.smithi063.stdout:5/564: write d2/f2c [255014,51776] 0 2024-03-23T11:47:56.428 INFO:tasks.workunit.client.0.smithi063.stdout:9/929: rename d4/d31/d43/d49/d8e/f93 to d4/f135 0 2024-03-23T11:47:56.433 INFO:tasks.workunit.client.0.smithi063.stdout:1/803: mknod d3/d3f/d4d/dbc/de0/d4b/dd3/cff 0 2024-03-23T11:47:56.434 INFO:tasks.workunit.client.0.smithi063.stdout:4/816: write d5/da/d20/d4b/d16/d32/d56/f85 [3720171,87379] 0 2024-03-23T11:47:56.441 INFO:tasks.workunit.client.0.smithi063.stdout:2/761: creat d2/d23/d56/ff9 x:0 0 0 2024-03-23T11:47:56.443 INFO:tasks.workunit.client.0.smithi063.stdout:5/565: truncate d2/d2f/f3f 994811 0 2024-03-23T11:47:56.445 INFO:tasks.workunit.client.0.smithi063.stdout:9/930: creat d4/d31/d43/da9/da6/dbe/f136 x:0 0 0 2024-03-23T11:47:56.452 INFO:tasks.workunit.client.0.smithi063.stdout:4/817: creat d5/da/d20/d4b/d16/d50/d86/fee x:0 0 0 2024-03-23T11:47:56.453 INFO:tasks.workunit.client.0.smithi063.stdout:2/762: fsync d2/d23/d2e/d52/d6c/f88 0 2024-03-23T11:47:56.457 INFO:tasks.workunit.client.0.smithi063.stdout:5/566: creat d2/d2f/fc7 x:0 0 0 2024-03-23T11:47:56.458 INFO:tasks.workunit.client.0.smithi063.stdout:6/693: truncate d1/f12 994983 0 2024-03-23T11:47:56.460 INFO:tasks.workunit.client.0.smithi063.stdout:6/694: stat d1/d15/d4b/lbf 0 2024-03-23T11:47:56.461 INFO:tasks.workunit.client.0.smithi063.stdout:9/931: mkdir d4/d31/d43/da9/da6/d100/d137 0 2024-03-23T11:47:56.463 INFO:tasks.workunit.client.0.smithi063.stdout:7/789: dwrite d5/d26/db7/dd6/f6a [0,4194304] 0 2024-03-23T11:47:56.466 INFO:tasks.workunit.client.0.smithi063.stdout:1/804: truncate d3/d35/f90 254915 0 2024-03-23T11:47:56.473 INFO:tasks.workunit.client.0.smithi063.stdout:5/567: creat d2/d7/d27/fc8 x:0 0 0 2024-03-23T11:47:56.477 INFO:tasks.workunit.client.0.smithi063.stdout:5/568: chown d2/d17/d4d/d6e/dba/dbe 26 1 2024-03-23T11:47:56.478 INFO:tasks.workunit.client.0.smithi063.stdout:6/695: creat d1/dc/d3a/d82/da2/dd2/fda x:0 0 0 2024-03-23T11:47:56.483 INFO:tasks.workunit.client.0.smithi063.stdout:4/818: dread - d5/d1e/d2b/faf zero size 2024-03-23T11:47:56.487 INFO:tasks.workunit.client.0.smithi063.stdout:6/696: chown d1/dc/d21/d49/fd9 1634415 1 2024-03-23T11:47:56.491 INFO:tasks.workunit.client.0.smithi063.stdout:9/932: dwrite d4/d31/d43/d49/d8e/ff7 [0,4194304] 0 2024-03-23T11:47:56.493 INFO:tasks.workunit.client.0.smithi063.stdout:2/763: write d2/d23/d2e/d52/f6a [213382,31886] 0 2024-03-23T11:47:56.494 INFO:tasks.workunit.client.0.smithi063.stdout:5/569: unlink d2/d7/d8/de/d2a/c82 0 2024-03-23T11:47:56.494 INFO:tasks.workunit.client.0.smithi063.stdout:4/819: chown d5/da/d20/d4b/d16/d37/fbe 4935495 1 2024-03-23T11:47:56.496 INFO:tasks.workunit.client.0.smithi063.stdout:7/790: dwrite d5/d5c/d2f/d73/da/f17 [0,4194304] 0 2024-03-23T11:47:56.503 INFO:tasks.workunit.client.0.smithi063.stdout:9/933: write d4/d31/d43/da9/da6/dbe/f136 [436421,33086] 0 2024-03-23T11:47:56.504 INFO:tasks.workunit.client.0.smithi063.stdout:2/764: creat d2/d23/d2e/d98/dd2/ffa x:0 0 0 2024-03-23T11:47:56.506 INFO:tasks.workunit.client.0.smithi063.stdout:2/765: chown d2/d23/d56/lf0 13 1 2024-03-23T11:47:56.511 INFO:tasks.workunit.client.0.smithi063.stdout:7/791: mknod d5/d26/db7/dce/dc8/cf5 0 2024-03-23T11:47:56.513 INFO:tasks.workunit.client.0.smithi063.stdout:2/766: truncate d2/d23/d2e/d87/d4d/fb1 4997492 0 2024-03-23T11:47:56.516 INFO:tasks.workunit.client.0.smithi063.stdout:7/792: chown d5/d26/db7/dce/dc8/cf5 22890165 1 2024-03-23T11:47:56.520 INFO:tasks.workunit.client.0.smithi063.stdout:2/767: symlink d2/d23/d2e/d52/d86/dbf/lfb 0 2024-03-23T11:47:56.523 INFO:tasks.workunit.client.0.smithi063.stdout:2/768: creat d2/d23/d56/ffc x:0 0 0 2024-03-23T11:47:56.550 INFO:tasks.workunit.client.0.smithi063.stdout:5/570: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:56.550 INFO:tasks.workunit.client.0.smithi063.stdout:9/934: dread d4/d31/d9c/d11b/d8d/fbb [0,4194304] 0 2024-03-23T11:47:56.551 INFO:tasks.workunit.client.0.smithi063.stdout:4/820: dread d5/da/d20/d4b/d16/d50/d86/da2/fe8 [0,4194304] 0 2024-03-23T11:47:56.554 INFO:tasks.workunit.client.0.smithi063.stdout:5/571: chown d2/d28/d3a/d46/d79/f7e 2736526 1 2024-03-23T11:47:56.554 INFO:tasks.workunit.client.0.smithi063.stdout:2/769: dwrite d2/d23/d2e/d52/f71 [0,4194304] 0 2024-03-23T11:47:56.556 INFO:tasks.workunit.client.0.smithi063.stdout:4/821: stat d5/da/d20/d4b/d16/d50/d8e/dad 0 2024-03-23T11:47:56.557 INFO:tasks.workunit.client.0.smithi063.stdout:7/793: dread d5/f12 [0,4194304] 0 2024-03-23T11:47:56.570 INFO:tasks.workunit.client.0.smithi063.stdout:2/770: mkdir d2/de/d22/dfd 0 2024-03-23T11:47:56.571 INFO:tasks.workunit.client.0.smithi063.stdout:5/572: chown d2/d7/d8/de/c24 166 1 2024-03-23T11:47:56.575 INFO:tasks.workunit.client.0.smithi063.stdout:4/822: read d5/d1e/f75 [3032423,94139] 0 2024-03-23T11:47:56.577 INFO:tasks.workunit.client.0.smithi063.stdout:7/794: link d5/d26/db7/dc6/fb3 d5/d26/db7/dd6/d88/ff6 0 2024-03-23T11:47:56.580 INFO:tasks.workunit.client.0.smithi063.stdout:4/823: write d5/da/d20/d4b/d16/d32/d56/f85 [2737273,21324] 0 2024-03-23T11:47:56.594 INFO:tasks.workunit.client.0.smithi063.stdout:5/573: dwrite d2/f2c [0,4194304] 0 2024-03-23T11:47:56.595 INFO:tasks.workunit.client.0.smithi063.stdout:9/935: dread d4/f7b [0,4194304] 0 2024-03-23T11:47:56.595 INFO:tasks.workunit.client.0.smithi063.stdout:2/771: dwrite d2/d23/d2e/d52/fd0 [0,4194304] 0 2024-03-23T11:47:56.598 INFO:tasks.workunit.client.0.smithi063.stdout:9/936: chown d4/c10 13226381 1 2024-03-23T11:47:56.603 INFO:tasks.workunit.client.0.smithi063.stdout:9/937: chown d4/d31/d9c/d11b/d8d/da0/fe2 308 1 2024-03-23T11:47:56.606 INFO:tasks.workunit.client.0.smithi063.stdout:5/574: rename d2/d17/f42 to d2/d28/fc9 0 2024-03-23T11:47:56.611 INFO:tasks.workunit.client.0.smithi063.stdout:9/938: readlink d4/d31/d43/da9/da6/l8f 0 2024-03-23T11:47:56.618 INFO:tasks.workunit.client.0.smithi063.stdout:9/939: creat d4/d31/d9c/d11b/d8d/d90/d108/f138 x:0 0 0 2024-03-23T11:47:56.622 INFO:tasks.workunit.client.0.smithi063.stdout:5/575: getdents d2 0 2024-03-23T11:47:56.623 INFO:tasks.workunit.client.0.smithi063.stdout:2/772: dread d2/de/d7a/d8d/f9e [0,4194304] 0 2024-03-23T11:47:56.625 INFO:tasks.workunit.client.0.smithi063.stdout:4/824: write d5/da/d20/d4b/d16/d37/f95 [380714,63866] 0 2024-03-23T11:47:56.626 INFO:tasks.workunit.client.0.smithi063.stdout:5/576: chown d2/d7/d8/de/l8a 122 1 2024-03-23T11:47:56.634 INFO:tasks.workunit.client.0.smithi063.stdout:2/773: creat d2/d84/d94/ffe x:0 0 0 2024-03-23T11:47:56.638 INFO:tasks.workunit.client.0.smithi063.stdout:7/795: dwrite f1 [0,4194304] 0 2024-03-23T11:47:56.643 INFO:tasks.workunit.client.0.smithi063.stdout:9/940: truncate d4/d4c/fca 1342409 0 2024-03-23T11:47:56.643 INFO:tasks.workunit.client.0.smithi063.stdout:2/774: truncate d2/d23/f69 2631143 0 2024-03-23T11:47:56.650 INFO:tasks.workunit.client.0.smithi063.stdout:5/577: dread d2/f5 [0,4194304] 0 2024-03-23T11:47:56.657 INFO:tasks.workunit.client.0.smithi063.stdout:4/825: truncate d5/da/d20/d4b/d29/d38/d7a/f8a 48790 0 2024-03-23T11:47:56.663 INFO:tasks.workunit.client.0.smithi063.stdout:9/941: creat d4/d31/d9c/d11b/d8d/d11c/f139 x:0 0 0 2024-03-23T11:47:56.665 INFO:tasks.workunit.client.0.smithi063.stdout:9/942: fdatasync d4/d31/d9c/d11b/d8d/da0/fe2 0 2024-03-23T11:47:56.670 INFO:tasks.workunit.client.0.smithi063.stdout:7/796: mknod d5/d26/db7/dce/d64/df4/cf7 0 2024-03-23T11:47:56.671 INFO:tasks.workunit.client.0.smithi063.stdout:2/775: write d2/d23/d2e/d87/f50 [1048490,42063] 0 2024-03-23T11:47:56.674 INFO:tasks.workunit.client.0.smithi063.stdout:7/797: chown d5/d6/d19/d1f/d6c/le9 182 1 2024-03-23T11:47:56.674 INFO:tasks.workunit.client.0.smithi063.stdout:2/776: write d2/d23/d56/f75 [410732,85285] 0 2024-03-23T11:47:56.675 INFO:tasks.workunit.client.0.smithi063.stdout:4/826: creat d5/d1e/d2b/dbd/fef x:0 0 0 2024-03-23T11:47:56.677 INFO:tasks.workunit.client.0.smithi063.stdout:7/798: write d5/d22/da3/fec [759866,14492] 0 2024-03-23T11:47:56.681 INFO:tasks.workunit.client.0.smithi063.stdout:7/799: write d5/d6/dca/fcb [942340,59022] 0 2024-03-23T11:47:56.685 INFO:tasks.workunit.client.0.smithi063.stdout:4/827: symlink d5/da/d20/d4b/d16/d37/lf0 0 2024-03-23T11:47:56.687 INFO:tasks.workunit.client.0.smithi063.stdout:2/777: rename d2/d23/d2e/d87/d4d/fb1 to d2/d46/d54/fff 0 2024-03-23T11:47:56.688 INFO:tasks.workunit.client.0.smithi063.stdout:5/578: truncate d2/d28/d3a/d46/d71/f78 209168 0 2024-03-23T11:47:56.690 INFO:tasks.workunit.client.0.smithi063.stdout:7/800: creat d5/d6/d19/d65/ff8 x:0 0 0 2024-03-23T11:47:56.693 INFO:tasks.workunit.client.0.smithi063.stdout:7/801: stat d5/d26/db7/dc6/d6e/ld1 0 2024-03-23T11:47:56.693 INFO:tasks.workunit.client.0.smithi063.stdout:2/778: unlink d2/f18 0 2024-03-23T11:47:56.694 INFO:tasks.workunit.client.0.smithi063.stdout:9/943: link d4/d31/d9c/d11b/l99 d4/d31/d43/da9/da6/d100/d137/l13a 0 2024-03-23T11:47:56.705 INFO:tasks.workunit.client.0.smithi063.stdout:7/802: rename d5/d5c/d2f/d73/da/d90/f9b to d5/d6/d19/d1f/ff9 0 2024-03-23T11:47:56.707 INFO:tasks.workunit.client.0.smithi063.stdout:4/828: rmdir d5/da/deb 0 2024-03-23T11:47:56.709 INFO:tasks.workunit.client.0.smithi063.stdout:2/779: mknod d2/de/d22/dfd/c100 0 2024-03-23T11:47:56.710 INFO:tasks.workunit.client.0.smithi063.stdout:4/829: readlink d5/da/d20/d4b/d16/d32/l6a 0 2024-03-23T11:47:56.713 INFO:tasks.workunit.client.0.smithi063.stdout:4/830: write d5/da/d20/d4b/d29/d38/f55 [3313225,54708] 0 2024-03-23T11:47:56.717 INFO:tasks.workunit.client.0.smithi063.stdout:9/944: rename d4/d31/d43/d49/d8e to d4/d31/d13b 0 2024-03-23T11:47:56.725 INFO:tasks.workunit.client.0.smithi063.stdout:7/803: creat d5/d26/db7/dd6/ffa x:0 0 0 2024-03-23T11:47:56.729 INFO:tasks.workunit.client.0.smithi063.stdout:5/579: dwrite d2/d7/d2b/f39 [0,4194304] 0 2024-03-23T11:47:56.739 INFO:tasks.workunit.client.0.smithi063.stdout:9/945: unlink d4/d31/d43/da9/da6/l8f 0 2024-03-23T11:47:56.748 INFO:tasks.workunit.client.0.smithi063.stdout:2/780: fsync d2/d23/f83 0 2024-03-23T11:47:56.755 INFO:tasks.workunit.client.0.smithi063.stdout:7/804: dread d5/d5c/d2f/f79 [0,4194304] 0 2024-03-23T11:47:56.758 INFO:tasks.workunit.client.0.smithi063.stdout:4/831: dwrite d5/da/d20/d4b/f68 [0,4194304] 0 2024-03-23T11:47:56.765 INFO:tasks.workunit.client.0.smithi063.stdout:5/580: mkdir d2/d7/d8/de/d49/db0/dca 0 2024-03-23T11:47:56.768 INFO:tasks.workunit.client.0.smithi063.stdout:7/805: mkdir d5/d6/d19/d1f/dfb 0 2024-03-23T11:47:56.768 INFO:tasks.workunit.client.0.smithi063.stdout:2/781: link d2/d23/f83 d2/de/d22/d91/f101 0 2024-03-23T11:47:56.774 INFO:tasks.workunit.client.0.smithi063.stdout:9/946: dwrite d4/d31/d43/da9/da6/ffc [0,4194304] 0 2024-03-23T11:47:56.780 INFO:tasks.workunit.client.0.smithi063.stdout:4/832: symlink d5/da/d20/d4b/d16/d50/d8e/dad/ddd/lf1 0 2024-03-23T11:47:56.785 INFO:tasks.workunit.client.0.smithi063.stdout:9/947: fdatasync d4/d4c/d19/fbf 0 2024-03-23T11:47:56.793 INFO:tasks.workunit.client.0.smithi063.stdout:4/833: rmdir d5/da/d7e 39 2024-03-23T11:47:56.804 INFO:tasks.workunit.client.0.smithi063.stdout:9/948: mkdir d4/d31/d43/da9/da6/d100/d11e/d13c 0 2024-03-23T11:47:56.809 INFO:tasks.workunit.client.0.smithi063.stdout:9/949: mkdir d4/d31/d9c/d9e/d13d 0 2024-03-23T11:47:56.814 INFO:tasks.workunit.client.0.smithi063.stdout:9/950: creat d4/d4c/dab/f13e x:0 0 0 2024-03-23T11:47:56.816 INFO:tasks.workunit.client.0.smithi063.stdout:4/834: write d5/da/d20/d4b/d16/d37/f53 [3274047,7168] 0 2024-03-23T11:47:56.825 INFO:tasks.workunit.client.0.smithi063.stdout:7/806: dwrite d5/d26/d32/faf [0,4194304] 0 2024-03-23T11:47:56.826 INFO:tasks.workunit.client.0.smithi063.stdout:4/835: chown d5/da/d20/d4b/d16/d50/d86/fee 112386 1 2024-03-23T11:47:56.829 INFO:tasks.workunit.client.0.smithi063.stdout:9/951: creat d4/d31/d43/da9/da6/db5/d129/f13f x:0 0 0 2024-03-23T11:47:56.835 INFO:tasks.workunit.client.0.smithi063.stdout:9/952: chown d4/d4c/d19 1925 1 2024-03-23T11:47:56.836 INFO:tasks.workunit.client.0.smithi063.stdout:4/836: write d5/da/d20/d4b/d16/d50/fe5 [111710,102885] 0 2024-03-23T11:47:56.839 INFO:tasks.workunit.client.0.smithi063.stdout:4/837: stat d5/da/d20/d4b/d1f/f21 0 2024-03-23T11:47:56.841 INFO:tasks.workunit.client.0.smithi063.stdout:9/953: creat d4/d31/d9c/d11b/d8d/d90/f140 x:0 0 0 2024-03-23T11:47:56.846 INFO:tasks.workunit.client.0.smithi063.stdout:9/954: dread - d4/d31/d43/da9/da6/dbe/f125 zero size 2024-03-23T11:47:56.849 INFO:tasks.workunit.client.0.smithi063.stdout:7/807: write d5/d26/db7/dc6/f97 [598798,74871] 0 2024-03-23T11:47:56.858 INFO:tasks.workunit.client.0.smithi063.stdout:9/955: read d4/d31/d9c/d11b/d8d/d11c/f11f [2620884,34567] 0 2024-03-23T11:47:56.866 INFO:tasks.workunit.client.0.smithi063.stdout:9/956: mkdir d4/d31/d9c/d11b/d8d/da0/d141 0 2024-03-23T11:47:56.870 INFO:tasks.workunit.client.0.smithi063.stdout:9/957: rmdir d4/d31 39 2024-03-23T11:47:56.875 INFO:tasks.workunit.client.0.smithi063.stdout:7/808: write d5/d26/db7/dce/dc8/dee/f82 [1789585,113778] 0 2024-03-23T11:47:56.875 INFO:tasks.workunit.client.0.smithi063.stdout:9/958: stat d4/d31/d10b/f117 0 2024-03-23T11:47:56.876 INFO:tasks.workunit.client.0.smithi063.stdout:7/809: dread - d5/d5c/d2f/fc3 zero size 2024-03-23T11:47:56.880 INFO:tasks.workunit.client.0.smithi063.stdout:9/959: mknod d4/d4c/d19/c142 0 2024-03-23T11:47:56.881 INFO:tasks.workunit.client.0.smithi063.stdout:9/960: read - d4/fc0 zero size 2024-03-23T11:47:56.884 INFO:tasks.workunit.client.0.smithi063.stdout:9/961: fsync d4/d31/d43/da9/f95 0 2024-03-23T11:47:56.889 INFO:tasks.workunit.client.0.smithi063.stdout:4/838: dread d5/da/d20/d4b/d1f/f43 [0,4194304] 0 2024-03-23T11:47:56.894 INFO:tasks.workunit.client.0.smithi063.stdout:9/962: symlink d4/d4c/l143 0 2024-03-23T11:47:56.901 INFO:tasks.workunit.client.0.smithi063.stdout:4/839: write d5/da/d20/f77 [745352,72793] 0 2024-03-23T11:47:56.905 INFO:tasks.workunit.client.0.smithi063.stdout:4/840: chown d5/d1e/d2b/f35 68859956 1 2024-03-23T11:47:56.909 INFO:tasks.workunit.client.0.smithi063.stdout:7/810: dwrite d5/d26/db7/dc6/d4f/d5b/f7b [0,4194304] 0 2024-03-23T11:47:56.911 INFO:tasks.workunit.client.0.smithi063.stdout:9/963: rename d4/d31/d43/da9/laa to d4/d31/d43/da9/da6/db5/d109/l144 0 2024-03-23T11:47:56.916 INFO:tasks.workunit.client.0.smithi063.stdout:4/841: symlink d5/da/d20/d4b/d16/d50/d86/da2/lf2 0 2024-03-23T11:47:56.917 INFO:tasks.workunit.client.0.smithi063.stdout:7/811: mknod d5/d26/db7/dd6/cfc 0 2024-03-23T11:47:56.921 INFO:tasks.workunit.client.0.smithi063.stdout:9/964: rmdir d4/d31/d43/da9/da6/db5/d109 39 2024-03-23T11:47:56.922 INFO:tasks.workunit.client.0.smithi063.stdout:9/965: fdatasync d4/d31/d9c/d9e/d12f/f84 0 2024-03-23T11:47:56.923 INFO:tasks.workunit.client.0.smithi063.stdout:7/812: write d5/d26/db7/dd6/f6a [1885861,6907] 0 2024-03-23T11:47:56.925 INFO:tasks.workunit.client.0.smithi063.stdout:1/805: sync 2024-03-23T11:47:56.935 INFO:tasks.workunit.client.0.smithi063.stdout:9/966: rmdir d4/d31/d9c 39 2024-03-23T11:47:56.940 INFO:tasks.workunit.client.0.smithi063.stdout:1/806: symlink d3/d3f/d4d/dbc/de0/l100 0 2024-03-23T11:47:56.942 INFO:tasks.workunit.client.0.smithi063.stdout:4/842: dwrite d5/da/d20/d4b/d16/d50/d86/fee [0,4194304] 0 2024-03-23T11:47:56.948 INFO:tasks.workunit.client.0.smithi063.stdout:7/813: creat d5/d6/d19/dcf/ffd x:0 0 0 2024-03-23T11:47:56.949 INFO:tasks.workunit.client.0.smithi063.stdout:1/807: creat d3/d3f/d4d/dbc/de0/d4b/da5/dd9/f101 x:0 0 0 2024-03-23T11:47:56.953 INFO:tasks.workunit.client.0.smithi063.stdout:1/808: write d3/d49/d56/f67 [215874,114677] 0 2024-03-23T11:47:56.953 INFO:tasks.workunit.client.0.smithi063.stdout:9/967: symlink d4/d31/d9c/d9e/d13d/l145 0 2024-03-23T11:47:56.954 INFO:tasks.workunit.client.0.smithi063.stdout:7/814: symlink d5/d5c/d2f/d73/da/lfe 0 2024-03-23T11:47:56.957 INFO:tasks.workunit.client.0.smithi063.stdout:9/968: chown d4/d31/d9c/d9e/d12f/c85 81148 1 2024-03-23T11:47:56.960 INFO:tasks.workunit.client.0.smithi063.stdout:4/843: truncate d5/da/d20/d4b/d16/d50/d86/f89 1789392 0 2024-03-23T11:47:56.961 INFO:tasks.workunit.client.0.smithi063.stdout:9/969: stat d4/d31/d43/da9/da6/d100/d11e 0 2024-03-23T11:47:56.964 INFO:tasks.workunit.client.0.smithi063.stdout:9/970: write d4/d4c/fde [5222675,56241] 0 2024-03-23T11:47:56.967 INFO:tasks.workunit.client.0.smithi063.stdout:4/844: fsync d5/da/d20/d4b/d29/fe0 0 2024-03-23T11:47:56.972 INFO:tasks.workunit.client.0.smithi063.stdout:1/809: mkdir d3/d49/d56/d65/d102 0 2024-03-23T11:47:56.976 INFO:tasks.workunit.client.0.smithi063.stdout:4/845: mkdir d5/da/d20/d4b/d1f/df3 0 2024-03-23T11:47:56.982 INFO:tasks.workunit.client.0.smithi063.stdout:9/971: creat d4/d31/d43/f146 x:0 0 0 2024-03-23T11:47:56.989 INFO:tasks.workunit.client.0.smithi063.stdout:4/846: rename d5/da/d20/d4b/d29/d38/l62 to d5/d1e/d2b/daa/lf4 0 2024-03-23T11:47:56.990 INFO:tasks.workunit.client.0.smithi063.stdout:7/815: dwrite d5/d5c/d2f/d73/da/d60/fa6 [0,4194304] 0 2024-03-23T11:47:56.994 INFO:tasks.workunit.client.0.smithi063.stdout:9/972: symlink d4/d31/d43/l147 0 2024-03-23T11:47:56.998 INFO:tasks.workunit.client.0.smithi063.stdout:1/810: dwrite d3/d49/d56/f67 [0,4194304] 0 2024-03-23T11:47:57.000 INFO:tasks.workunit.client.0.smithi063.stdout:9/973: mknod d4/d31/d9c/c148 0 2024-03-23T11:47:57.008 INFO:tasks.workunit.client.0.smithi063.stdout:7/816: rename d5/d26/db7/dc6/d4f/d5b to d5/d26/dff 0 2024-03-23T11:47:57.014 INFO:tasks.workunit.client.0.smithi063.stdout:7/817: rename d5/d26/db7/dc6/d6e/d98 to d5/d26/dde/d100 0 2024-03-23T11:47:57.015 INFO:tasks.workunit.client.0.smithi063.stdout:2/782: sync 2024-03-23T11:47:57.015 INFO:tasks.workunit.client.0.smithi063.stdout:6/697: sync 2024-03-23T11:47:57.017 INFO:tasks.workunit.client.0.smithi063.stdout:5/581: sync 2024-03-23T11:47:57.021 INFO:tasks.workunit.client.0.smithi063.stdout:2/783: read d2/d46/d54/fa3 [2846397,63716] 0 2024-03-23T11:47:57.025 INFO:tasks.workunit.client.0.smithi063.stdout:2/784: chown d2/de/d7a/d77/cd5 175190150 1 2024-03-23T11:47:57.030 INFO:tasks.workunit.client.0.smithi063.stdout:5/582: rename d2/d28/fac to d2/d7/d8/d1d/fcb 0 2024-03-23T11:47:57.034 INFO:tasks.workunit.client.0.smithi063.stdout:5/583: write d2/d7/fd [4552259,35660] 0 2024-03-23T11:47:57.037 INFO:tasks.workunit.client.0.smithi063.stdout:4/847: dread d5/da/d20/d4b/d16/d32/d56/f85 [0,4194304] 0 2024-03-23T11:47:57.040 INFO:tasks.workunit.client.0.smithi063.stdout:7/818: link d5/d6/d19/d1f/d6c/fd9 d5/d26/db7/dce/d64/df0/f101 0 2024-03-23T11:47:57.045 INFO:tasks.workunit.client.0.smithi063.stdout:6/698: mkdir d1/dc/d3a/d57/d6d/ddb 0 2024-03-23T11:47:57.046 INFO:tasks.workunit.client.0.smithi063.stdout:6/699: stat d1/dc/d3a/d82/da2/f4a 0 2024-03-23T11:47:57.056 INFO:tasks.workunit.client.0.smithi063.stdout:1/811: dwrite d3/f11 [0,4194304] 0 2024-03-23T11:47:57.059 INFO:tasks.workunit.client.0.smithi063.stdout:5/584: creat d2/d7/d8/de/d49/db0/fcc x:0 0 0 2024-03-23T11:47:57.062 INFO:tasks.workunit.client.0.smithi063.stdout:5/585: dread - d2/d7/d27/f94 zero size 2024-03-23T11:47:57.072 INFO:tasks.workunit.client.0.smithi063.stdout:7/819: symlink d5/d26/db7/dc6/d6e/da2/l102 0 2024-03-23T11:47:57.075 INFO:tasks.workunit.client.0.smithi063.stdout:5/586: creat d2/d7/d8/de/d35/fcd x:0 0 0 2024-03-23T11:47:57.077 INFO:tasks.workunit.client.0.smithi063.stdout:5/587: write d2/d7/d27/fc8 [1037217,28012] 0 2024-03-23T11:47:57.079 INFO:tasks.workunit.client.0.smithi063.stdout:5/588: read d2/d7/d2b/f39 [4127932,23619] 0 2024-03-23T11:47:57.081 INFO:tasks.workunit.client.0.smithi063.stdout:2/785: write d2/d23/d2e/d52/d6c/f88 [5485546,65664] 0 2024-03-23T11:47:57.082 INFO:tasks.workunit.client.0.smithi063.stdout:7/820: mknod d5/d26/db7/dd6/d88/dc2/c103 0 2024-03-23T11:47:57.083 INFO:tasks.workunit.client.0.smithi063.stdout:6/700: rmdir d1/dd4 0 2024-03-23T11:47:57.085 INFO:tasks.workunit.client.0.smithi063.stdout:2/786: write d2/d23/d56/f75 [1336704,62621] 0 2024-03-23T11:47:57.087 INFO:tasks.workunit.client.0.smithi063.stdout:6/701: chown d1/dc/d3a/d57/d6d/ddb 900376513 1 2024-03-23T11:47:57.087 INFO:tasks.workunit.client.0.smithi063.stdout:1/812: write d3/d49/d56/f7f [879760,94635] 0 2024-03-23T11:47:57.091 INFO:tasks.workunit.client.0.smithi063.stdout:5/589: fsync d2/d28/d3a/f8d 0 2024-03-23T11:47:57.098 INFO:tasks.workunit.client.0.smithi063.stdout:4/848: dwrite d5/da/d20/d4b/d29/d38/d7a/f8a [0,4194304] 0 2024-03-23T11:47:57.104 INFO:tasks.workunit.client.0.smithi063.stdout:7/821: rename d5/d5c/d2f/db5/dd2/le3 to d5/d26/d76/l104 0 2024-03-23T11:47:57.107 INFO:tasks.workunit.client.0.smithi063.stdout:2/787: dwrite d2/d23/d2e/d87/f50 [0,4194304] 0 2024-03-23T11:47:57.113 INFO:tasks.workunit.client.0.smithi063.stdout:1/813: mkdir d3/d49/d56/d65/d88/d92/d103 0 2024-03-23T11:47:57.113 INFO:tasks.workunit.client.0.smithi063.stdout:5/590: fsync d2/d7/d27/f83 0 2024-03-23T11:47:57.119 INFO:tasks.workunit.client.0.smithi063.stdout:4/849: symlink d5/da/d20/d4b/d1f/df3/lf5 0 2024-03-23T11:47:57.121 INFO:tasks.workunit.client.0.smithi063.stdout:1/814: fsync d3/d49/d56/d65/db6/dc5/fdf 0 2024-03-23T11:47:57.121 INFO:tasks.workunit.client.0.smithi063.stdout:6/702: dread d1/dc/d21/d49/f76 [0,4194304] 0 2024-03-23T11:47:57.122 INFO:tasks.workunit.client.0.smithi063.stdout:2/788: symlink d2/d23/d2e/d87/l102 0 2024-03-23T11:47:57.124 INFO:tasks.workunit.client.0.smithi063.stdout:7/822: creat d5/d5c/d2f/d73/f105 x:0 0 0 2024-03-23T11:47:57.128 INFO:tasks.workunit.client.0.smithi063.stdout:1/815: read d3/df/f78 [3197961,120018] 0 2024-03-23T11:47:57.133 INFO:tasks.workunit.client.0.smithi063.stdout:5/591: dread d2/d7/d8/d1d/fcb [0,4194304] 0 2024-03-23T11:47:57.136 INFO:tasks.workunit.client.0.smithi063.stdout:4/850: creat d5/da/d20/d4b/d16/dc4/ff6 x:0 0 0 2024-03-23T11:47:57.144 INFO:tasks.workunit.client.0.smithi063.stdout:2/789: creat d2/de/d22/dbc/f103 x:0 0 0 2024-03-23T11:47:57.144 INFO:tasks.workunit.client.0.smithi063.stdout:6/703: dwrite d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:47:57.146 INFO:tasks.workunit.client.0.smithi063.stdout:7/823: creat d5/d26/dde/f106 x:0 0 0 2024-03-23T11:47:57.152 INFO:tasks.workunit.client.0.smithi063.stdout:5/592: symlink d2/d7/d8/d1d/lce 0 2024-03-23T11:47:57.155 INFO:tasks.workunit.client.0.smithi063.stdout:4/851: creat d5/da/d20/d4b/d16/dc4/ff7 x:0 0 0 2024-03-23T11:47:57.158 INFO:tasks.workunit.client.0.smithi063.stdout:1/816: mknod d3/df/d2f/d9a/dfc/c104 0 2024-03-23T11:47:57.159 INFO:tasks.workunit.client.0.smithi063.stdout:7/824: rmdir d5/d5c 39 2024-03-23T11:47:57.164 INFO:tasks.workunit.client.0.smithi063.stdout:4/852: write d5/da/d20/d4b/d16/d50/d86/da2/dd7/fe9 [905180,50669] 0 2024-03-23T11:47:57.168 INFO:tasks.workunit.client.0.smithi063.stdout:1/817: creat d3/d49/d56/d65/d84/da7/de3/f105 x:0 0 0 2024-03-23T11:47:57.169 INFO:tasks.workunit.client.0.smithi063.stdout:2/790: link d2/d84/lbe d2/d23/d2e/d98/dc4/dcc/df5/l104 0 2024-03-23T11:47:57.169 INFO:tasks.workunit.client.0.smithi063.stdout:5/593: dread d2/f2c [0,4194304] 0 2024-03-23T11:47:57.172 INFO:tasks.workunit.client.0.smithi063.stdout:5/594: fsync d2/d7/fbd 0 2024-03-23T11:47:57.176 INFO:tasks.workunit.client.0.smithi063.stdout:2/791: creat d2/d84/d92/f105 x:0 0 0 2024-03-23T11:47:57.179 INFO:tasks.workunit.client.0.smithi063.stdout:5/595: fdatasync d2/d28/d3a/fa5 0 2024-03-23T11:47:57.182 INFO:tasks.workunit.client.0.smithi063.stdout:6/704: write d1/d15/f26 [1193769,30559] 0 2024-03-23T11:47:57.197 INFO:tasks.workunit.client.0.smithi063.stdout:4/853: dwrite d5/da/fcf [0,4194304] 0 2024-03-23T11:47:57.197 INFO:tasks.workunit.client.0.smithi063.stdout:2/792: mkdir d2/d1c/d106 0 2024-03-23T11:47:57.202 INFO:tasks.workunit.client.0.smithi063.stdout:7/825: dwrite d5/d5c/d2f/d73/da/d60/fa6 [4194304,4194304] 0 2024-03-23T11:47:57.204 INFO:tasks.workunit.client.0.smithi063.stdout:2/793: truncate d2/d84/d94/fa6 1864602 0 2024-03-23T11:47:57.207 INFO:tasks.workunit.client.0.smithi063.stdout:5/596: dread d2/d28/d3a/d46/d71/d9f/fa1 [0,4194304] 0 2024-03-23T11:47:57.212 INFO:tasks.workunit.client.0.smithi063.stdout:5/597: dread - d2/d7/fbd zero size 2024-03-23T11:47:57.215 INFO:tasks.workunit.client.0.smithi063.stdout:7/826: symlink d5/d26/db7/dd6/d88/l107 0 2024-03-23T11:47:57.216 INFO:tasks.workunit.client.0.smithi063.stdout:1/818: dread d3/d49/d56/d65/d88/d92/fb0 [0,4194304] 0 2024-03-23T11:47:57.221 INFO:tasks.workunit.client.0.smithi063.stdout:4/854: getdents d5/da/d20/d4b/d16/d32/d56/dc7 0 2024-03-23T11:47:57.224 INFO:tasks.workunit.client.0.smithi063.stdout:5/598: mkdir d2/d7/d8/de/dcf 0 2024-03-23T11:47:57.227 INFO:tasks.workunit.client.0.smithi063.stdout:2/794: dread d2/d23/d2e/f35 [0,4194304] 0 2024-03-23T11:47:57.228 INFO:tasks.workunit.client.0.smithi063.stdout:6/705: dwrite d1/dc/d3a/d57/f5c [0,4194304] 0 2024-03-23T11:47:57.231 INFO:tasks.workunit.client.0.smithi063.stdout:7/827: creat d5/d22/da3/f108 x:0 0 0 2024-03-23T11:47:57.234 INFO:tasks.workunit.client.0.smithi063.stdout:5/599: write d2/d7/d8/d1d/f88 [1881310,127458] 0 2024-03-23T11:47:57.235 INFO:tasks.workunit.client.0.smithi063.stdout:6/706: read - d1/f67 zero size 2024-03-23T11:47:57.238 INFO:tasks.workunit.client.0.smithi063.stdout:2/795: chown d2/de/d7a/da2/lca 184884 1 2024-03-23T11:47:57.239 INFO:tasks.workunit.client.0.smithi063.stdout:7/828: read d5/d5c/d2f/d73/da/d60/fa6 [8026302,75141] 0 2024-03-23T11:47:57.243 INFO:tasks.workunit.client.0.smithi063.stdout:4/855: write d5/da/d20/d4b/d16/d50/d8e/d96/ddf/f92 [840068,9237] 0 2024-03-23T11:47:57.245 INFO:tasks.workunit.client.0.smithi063.stdout:5/600: creat d2/d7/d8/d1d/fd0 x:0 0 0 2024-03-23T11:47:57.246 INFO:tasks.workunit.client.0.smithi063.stdout:2/796: creat d2/d46/db4/f107 x:0 0 0 2024-03-23T11:47:57.247 INFO:tasks.workunit.client.0.smithi063.stdout:1/819: dread d3/d49/d56/d65/d88/d92/fb0 [0,4194304] 0 2024-03-23T11:47:57.249 INFO:tasks.workunit.client.0.smithi063.stdout:6/707: mkdir d1/dc/d3a/d82/ddc 0 2024-03-23T11:47:57.253 INFO:tasks.workunit.client.0.smithi063.stdout:7/829: write d5/d22/f24 [2229770,56754] 0 2024-03-23T11:47:57.256 INFO:tasks.workunit.client.0.smithi063.stdout:1/820: symlink d3/df/d2f/l106 0 2024-03-23T11:47:57.257 INFO:tasks.workunit.client.0.smithi063.stdout:4/856: symlink d5/da/d20/d4b/d29/d38/lf8 0 2024-03-23T11:47:57.260 INFO:tasks.workunit.client.0.smithi063.stdout:1/821: stat d3/c44 0 2024-03-23T11:47:57.261 INFO:tasks.workunit.client.0.smithi063.stdout:5/601: rename d2/d17/l26 to d2/d17/d4d/ld1 0 2024-03-23T11:47:57.265 INFO:tasks.workunit.client.0.smithi063.stdout:1/822: creat d3/df/db8/dcb/f107 x:0 0 0 2024-03-23T11:47:57.267 INFO:tasks.workunit.client.0.smithi063.stdout:2/797: rename d2/d23/d2e/d98/dc8/le5 to d2/d23/d2e/d98/dd2/l108 0 2024-03-23T11:47:57.269 INFO:tasks.workunit.client.0.smithi063.stdout:6/708: dwrite d1/dc/d3a/d82/da2/dd2/fda [0,4194304] 0 2024-03-23T11:47:57.270 INFO:tasks.workunit.client.0.smithi063.stdout:6/709: fdatasync d1/d15/d4b/fc2 0 2024-03-23T11:47:57.287 INFO:tasks.workunit.client.0.smithi063.stdout:1/823: creat d3/d3f/d4d/dbc/de0/f108 x:0 0 0 2024-03-23T11:47:57.288 INFO:tasks.workunit.client.0.smithi063.stdout:2/798: creat d2/de/d7a/d8d/f109 x:0 0 0 2024-03-23T11:47:57.293 INFO:tasks.workunit.client.0.smithi063.stdout:7/830: write d5/d5c/d2f/d73/f3c [2937701,109194] 0 2024-03-23T11:47:57.293 INFO:tasks.workunit.client.0.smithi063.stdout:5/602: getdents d2/d7/d8/de/d49/db0/dca 0 2024-03-23T11:47:57.294 INFO:tasks.workunit.client.0.smithi063.stdout:5/603: readlink d2/d28/lc3 0 2024-03-23T11:47:57.298 INFO:tasks.workunit.client.0.smithi063.stdout:6/710: mknod d1/dc/d3a/d57/cdd 0 2024-03-23T11:47:57.299 INFO:tasks.workunit.client.0.smithi063.stdout:1/824: creat d3/d3f/d4d/d52/ddd/f109 x:0 0 0 2024-03-23T11:47:57.306 INFO:tasks.workunit.client.0.smithi063.stdout:1/825: mknod d3/df/d2f/d9a/c10a 0 2024-03-23T11:47:57.309 INFO:tasks.workunit.client.0.smithi063.stdout:6/711: truncate d1/dc/d21/d49/f6a 470480 0 2024-03-23T11:47:57.309 INFO:tasks.workunit.client.0.smithi063.stdout:2/799: dwrite d2/d23/d2e/d98/dd2/ffa [0,4194304] 0 2024-03-23T11:47:57.312 INFO:tasks.workunit.client.0.smithi063.stdout:5/604: dwrite d2/d7/d8/d1d/f88 [0,4194304] 0 2024-03-23T11:47:57.314 INFO:tasks.workunit.client.0.smithi063.stdout:1/826: unlink d3/d49/d56/f86 0 2024-03-23T11:47:57.314 INFO:tasks.workunit.client.0.smithi063.stdout:5/605: fsync d2/d2f/fc7 0 2024-03-23T11:47:57.317 INFO:tasks.workunit.client.0.smithi063.stdout:1/827: creat d3/df/d2f/f10b x:0 0 0 2024-03-23T11:47:57.320 INFO:tasks.workunit.client.0.smithi063.stdout:7/831: write d5/d6/d19/d1f/ff9 [4684353,6633] 0 2024-03-23T11:47:57.322 INFO:tasks.workunit.client.0.smithi063.stdout:7/832: fdatasync d5/d5c/d2f/ff1 0 2024-03-23T11:47:57.324 INFO:tasks.workunit.client.0.smithi063.stdout:6/712: symlink d1/dc/d3a/d82/da2/dd5/lde 0 2024-03-23T11:47:57.324 INFO:tasks.workunit.client.0.smithi063.stdout:2/800: symlink d2/d23/l10a 0 2024-03-23T11:47:57.329 INFO:tasks.workunit.client.0.smithi063.stdout:2/801: truncate d2/de/d7a/d8d/fa9 792404 0 2024-03-23T11:47:57.332 INFO:tasks.workunit.client.0.smithi063.stdout:6/713: creat d1/dc/d21/d49/fdf x:0 0 0 2024-03-23T11:47:57.335 INFO:tasks.workunit.client.0.smithi063.stdout:2/802: symlink d2/d23/d2e/d52/d86/l10b 0 2024-03-23T11:47:57.336 INFO:tasks.workunit.client.0.smithi063.stdout:1/828: rmdir d3/d3f/d4d/ddc 0 2024-03-23T11:47:57.337 INFO:tasks.workunit.client.0.smithi063.stdout:2/803: chown d2/de/f2b 7407050 1 2024-03-23T11:47:57.347 INFO:tasks.workunit.client.0.smithi063.stdout:2/804: fsync d2/d23/d2e/d87/f80 0 2024-03-23T11:47:57.349 INFO:tasks.workunit.client.0.smithi063.stdout:1/829: symlink d3/d49/d56/d65/d84/da7/de3/l10c 0 2024-03-23T11:47:57.349 INFO:tasks.workunit.client.0.smithi063.stdout:2/805: readlink d2/de/d7a/da2/lca 0 2024-03-23T11:47:57.351 INFO:tasks.workunit.client.0.smithi063.stdout:7/833: write d5/d6/d19/fe8 [1003181,69476] 0 2024-03-23T11:47:57.353 INFO:tasks.workunit.client.0.smithi063.stdout:2/806: mkdir d2/de/d22/dbc/d10c 0 2024-03-23T11:47:57.358 INFO:tasks.workunit.client.0.smithi063.stdout:7/834: fsync d5/d26/db7/dc6/d4f/f93 0 2024-03-23T11:47:57.360 INFO:tasks.workunit.client.0.smithi063.stdout:6/714: write d1/dc/d21/d49/f95 [3279649,80618] 0 2024-03-23T11:47:57.361 INFO:tasks.workunit.client.0.smithi063.stdout:2/807: fdatasync d2/d23/f69 0 2024-03-23T11:47:57.365 INFO:tasks.workunit.client.0.smithi063.stdout:7/835: creat d5/d22/da3/f109 x:0 0 0 2024-03-23T11:47:57.369 INFO:tasks.workunit.client.0.smithi063.stdout:1/830: truncate d3/d35/d40/f4c 4585332 0 2024-03-23T11:47:57.370 INFO:tasks.workunit.client.0.smithi063.stdout:7/836: rmdir d5/d6/d19/d1f 39 2024-03-23T11:47:57.370 INFO:tasks.workunit.client.0.smithi063.stdout:1/831: fsync d3/df/d2f/fcc 0 2024-03-23T11:47:57.374 INFO:tasks.workunit.client.0.smithi063.stdout:2/808: write d2/d1c/ff8 [542279,70159] 0 2024-03-23T11:47:57.377 INFO:tasks.workunit.client.0.smithi063.stdout:7/837: rmdir d5/d26/db7 39 2024-03-23T11:47:57.378 INFO:tasks.workunit.client.0.smithi063.stdout:1/832: symlink d3/d3f/d4d/dbc/de0/dce/l10d 0 2024-03-23T11:47:57.378 INFO:tasks.workunit.client.0.smithi063.stdout:2/809: read d2/d23/d2e/d98/dd2/ffa [3046272,98617] 0 2024-03-23T11:47:57.381 INFO:tasks.workunit.client.0.smithi063.stdout:2/810: chown d2/d46/db4/cc6 6983218 1 2024-03-23T11:47:57.384 INFO:tasks.workunit.client.0.smithi063.stdout:1/833: mknod d3/df/d2f/d9a/dfc/c10e 0 2024-03-23T11:47:57.387 INFO:tasks.workunit.client.0.smithi063.stdout:1/834: fdatasync d3/d3f/d4d/d52/fab 0 2024-03-23T11:47:57.387 INFO:tasks.workunit.client.0.smithi063.stdout:6/715: dread d1/dc/d3a/f45 [0,4194304] 0 2024-03-23T11:47:57.389 INFO:tasks.workunit.client.0.smithi063.stdout:1/835: chown d3/d3f/d4d/dbc/de0/lb3 0 1 2024-03-23T11:47:57.391 INFO:tasks.workunit.client.0.smithi063.stdout:7/838: mknod d5/d26/db7/dce/c10a 0 2024-03-23T11:47:57.394 INFO:tasks.workunit.client.0.smithi063.stdout:6/716: symlink d1/dc/d3a/d57/d77/le0 0 2024-03-23T11:47:57.396 INFO:tasks.workunit.client.0.smithi063.stdout:6/717: chown d1/dc/d3a/d82/da2/c7f 209 1 2024-03-23T11:47:57.397 INFO:tasks.workunit.client.0.smithi063.stdout:7/839: unlink d5/d26/db7/dce/c10a 0 2024-03-23T11:47:57.398 INFO:tasks.workunit.client.0.smithi063.stdout:1/836: unlink d3/cc 0 2024-03-23T11:47:57.400 INFO:tasks.workunit.client.0.smithi063.stdout:6/718: write d1/dc/d21/d49/fd9 [243531,115469] 0 2024-03-23T11:47:57.405 INFO:tasks.workunit.client.0.smithi063.stdout:2/811: dwrite d2/de/d22/dbc/fdb [0,4194304] 0 2024-03-23T11:47:57.406 INFO:tasks.workunit.client.0.smithi063.stdout:6/719: read - d1/dc/d3a/d57/f97 zero size 2024-03-23T11:47:57.413 INFO:tasks.workunit.client.0.smithi063.stdout:2/812: chown d2/de/d7a/d77/cf1 5961336 1 2024-03-23T11:47:57.414 INFO:tasks.workunit.client.0.smithi063.stdout:6/720: symlink d1/d15/le1 0 2024-03-23T11:47:57.416 INFO:tasks.workunit.client.0.smithi063.stdout:7/840: write d5/d6/d19/d65/fba [596275,121525] 0 2024-03-23T11:47:57.421 INFO:tasks.workunit.client.0.smithi063.stdout:6/721: creat d1/dc/d21/fe2 x:0 0 0 2024-03-23T11:47:57.423 INFO:tasks.workunit.client.0.smithi063.stdout:2/813: chown d2/d84/lbe 28125754 1 2024-03-23T11:47:57.425 INFO:tasks.workunit.client.0.smithi063.stdout:7/841: mknod d5/d26/db7/dc6/c10b 0 2024-03-23T11:47:57.427 INFO:tasks.workunit.client.0.smithi063.stdout:1/837: dwrite d3/d35/f90 [0,4194304] 0 2024-03-23T11:47:57.428 INFO:tasks.workunit.client.0.smithi063.stdout:6/722: mknod d1/dc/ce3 0 2024-03-23T11:47:57.428 INFO:tasks.workunit.client.0.smithi063.stdout:7/842: write d5/d26/f74 [1306371,3941] 0 2024-03-23T11:47:57.430 INFO:tasks.workunit.client.0.smithi063.stdout:7/843: stat d5/d26/db7/dd6/cfc 0 2024-03-23T11:47:57.433 INFO:tasks.workunit.client.0.smithi063.stdout:7/844: write d5/d6/d19/d65/fba [72403,5141] 0 2024-03-23T11:47:57.436 INFO:tasks.workunit.client.0.smithi063.stdout:7/845: dread - d5/d5c/d2f/ff1 zero size 2024-03-23T11:47:57.438 INFO:tasks.workunit.client.0.smithi063.stdout:6/723: mknod d1/dc/d3a/d82/ce4 0 2024-03-23T11:47:57.438 INFO:tasks.workunit.client.0.smithi063.stdout:1/838: dread d3/d49/d56/d65/d84/fac [0,4194304] 0 2024-03-23T11:47:57.441 INFO:tasks.workunit.client.0.smithi063.stdout:1/839: chown d3/d49/d56/d65/d102 425684840 1 2024-03-23T11:47:57.446 INFO:tasks.workunit.client.0.smithi063.stdout:2/814: dwrite d2/d84/d94/ffe [0,4194304] 0 2024-03-23T11:47:57.446 INFO:tasks.workunit.client.0.smithi063.stdout:7/846: fdatasync d5/d6/f25 0 2024-03-23T11:47:57.451 INFO:tasks.workunit.client.0.smithi063.stdout:1/840: symlink d3/df/d2f/d9a/dd6/l10f 0 2024-03-23T11:47:57.458 INFO:tasks.workunit.client.0.smithi063.stdout:7/847: creat d5/d26/db7/dce/d64/f10c x:0 0 0 2024-03-23T11:47:57.460 INFO:tasks.workunit.client.0.smithi063.stdout:2/815: mkdir d2/d46/d54/dc2/d10d 0 2024-03-23T11:47:57.460 INFO:tasks.workunit.client.0.smithi063.stdout:1/841: mkdir d3/d35/d40/dee/d110 0 2024-03-23T11:47:57.461 INFO:tasks.workunit.client.0.smithi063.stdout:6/724: link d1/dc/d3a/f40 d1/dc/d3a/d82/da2/dd2/fe5 0 2024-03-23T11:47:57.462 INFO:tasks.workunit.client.0.smithi063.stdout:7/848: creat d5/d5c/d2f/d8b/f10d x:0 0 0 2024-03-23T11:47:57.464 INFO:tasks.workunit.client.0.smithi063.stdout:6/725: stat d1/dc/d3a/d82/da2/d41/lb4 0 2024-03-23T11:47:57.465 INFO:tasks.workunit.client.0.smithi063.stdout:2/816: creat d2/d23/d2e/d52/d86/dbf/f10e x:0 0 0 2024-03-23T11:47:57.467 INFO:tasks.workunit.client.0.smithi063.stdout:6/726: mkdir d1/de6 0 2024-03-23T11:47:57.483 INFO:tasks.workunit.client.0.smithi063.stdout:1/842: dwrite d3/df/d2f/ff0 [0,4194304] 0 2024-03-23T11:47:57.485 INFO:tasks.workunit.client.0.smithi063.stdout:2/817: dwrite d2/f29 [0,4194304] 0 2024-03-23T11:47:57.490 INFO:tasks.workunit.client.0.smithi063.stdout:2/818: dread - d2/d23/d2e/d98/dd2/feb zero size 2024-03-23T11:47:57.496 INFO:tasks.workunit.client.0.smithi063.stdout:2/819: creat d2/d23/d2e/d98/f10f x:0 0 0 2024-03-23T11:47:57.497 INFO:tasks.workunit.client.0.smithi063.stdout:1/843: getdents d3/df/d2f/d9a 0 2024-03-23T11:47:57.499 INFO:tasks.workunit.client.0.smithi063.stdout:6/727: dwrite d1/dc/d3a/d82/da2/f62 [0,4194304] 0 2024-03-23T11:47:57.500 INFO:tasks.workunit.client.0.smithi063.stdout:7/849: dread d5/d5c/d2f/fda [0,4194304] 0 2024-03-23T11:47:57.502 INFO:tasks.workunit.client.0.smithi063.stdout:7/850: readlink l3 0 2024-03-23T11:47:57.502 INFO:tasks.workunit.client.0.smithi063.stdout:9/974: sync 2024-03-23T11:47:57.503 INFO:tasks.workunit.client.0.smithi063.stdout:1/844: write d3/d49/d56/d65/d88/d92/fda [3040881,15360] 0 2024-03-23T11:47:57.508 INFO:tasks.workunit.client.0.smithi063.stdout:2/820: creat d2/d23/d2e/d98/dc8/f110 x:0 0 0 2024-03-23T11:47:57.513 INFO:tasks.workunit.client.0.smithi063.stdout:7/851: symlink d5/d22/dcc/l10e 0 2024-03-23T11:47:57.513 INFO:tasks.workunit.client.0.smithi063.stdout:1/845: rmdir d3/d49 39 2024-03-23T11:47:57.515 INFO:tasks.workunit.client.0.smithi063.stdout:1/846: chown d3/d3f/d4d/dbc 17 1 2024-03-23T11:47:57.516 INFO:tasks.workunit.client.0.smithi063.stdout:6/728: mkdir d1/dc/d3a/d82/da2/d41/d42/d99/dc3/de7 0 2024-03-23T11:47:57.523 INFO:tasks.workunit.client.0.smithi063.stdout:6/729: symlink d1/dc/d3a/d82/da2/dd5/le8 0 2024-03-23T11:47:57.530 INFO:tasks.workunit.client.0.smithi063.stdout:9/975: dwrite d4/d31/d9c/d11b/d8d/d90/f140 [0,4194304] 0 2024-03-23T11:47:57.530 INFO:tasks.workunit.client.0.smithi063.stdout:6/730: mkdir d1/dc/d3a/d82/da2/d41/d42/d99/de9 0 2024-03-23T11:47:57.538 INFO:tasks.workunit.client.0.smithi063.stdout:6/731: mkdir d1/dc/dea 0 2024-03-23T11:47:57.538 INFO:tasks.workunit.client.0.smithi063.stdout:1/847: dread d3/d35/f90 [0,4194304] 0 2024-03-23T11:47:57.554 INFO:tasks.workunit.client.0.smithi063.stdout:2/821: dwrite d2/de/d22/f74 [0,4194304] 0 2024-03-23T11:47:57.557 INFO:tasks.workunit.client.0.smithi063.stdout:2/822: chown d2/d46/d54/fdd 629 1 2024-03-23T11:47:57.558 INFO:tasks.workunit.client.0.smithi063.stdout:7/852: dread d5/d26/f74 [0,4194304] 0 2024-03-23T11:47:57.561 INFO:tasks.workunit.client.0.smithi063.stdout:6/732: dwrite d1/dc/d21/d49/f95 [0,4194304] 0 2024-03-23T11:47:57.564 INFO:tasks.workunit.client.0.smithi063.stdout:6/733: stat d1/dc/d3a/d57/d6d/l9c 0 2024-03-23T11:47:57.567 INFO:tasks.workunit.client.0.smithi063.stdout:6/734: stat d1/dc/d3a/d82/da2/d41/d42 0 2024-03-23T11:47:57.570 INFO:tasks.workunit.client.0.smithi063.stdout:6/735: chown d1/dc/d3a/d82/da2/d41/d42/d99/de9 6245 1 2024-03-23T11:47:57.582 INFO:tasks.workunit.client.0.smithi063.stdout:6/736: dwrite d1/dc/d3a/d57/d6d/fa0 [0,4194304] 0 2024-03-23T11:47:57.599 INFO:tasks.workunit.client.0.smithi063.stdout:9/976: creat d4/d31/d43/da9/da6/f149 x:0 0 0 2024-03-23T11:47:57.603 INFO:tasks.workunit.client.0.smithi063.stdout:2/823: rename d2/f3a to d2/d23/d2e/d98/dc8/f111 0 2024-03-23T11:47:57.610 INFO:tasks.workunit.client.0.smithi063.stdout:2/824: symlink d2/d23/d56/dad/dd4/dd6/l112 0 2024-03-23T11:47:57.613 INFO:tasks.workunit.client.0.smithi063.stdout:9/977: creat d4/f14a x:0 0 0 2024-03-23T11:47:57.614 INFO:tasks.workunit.client.0.smithi063.stdout:6/737: dread d1/f6e [0,4194304] 0 2024-03-23T11:47:57.615 INFO:tasks.workunit.client.0.smithi063.stdout:2/825: creat d2/de/d22/dfd/f113 x:0 0 0 2024-03-23T11:47:57.615 INFO:tasks.workunit.client.0.smithi063.stdout:9/978: fsync d4/d31/d9c/ffb 0 2024-03-23T11:47:57.617 INFO:tasks.workunit.client.0.smithi063.stdout:6/738: chown d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb 1042728 1 2024-03-23T11:47:57.619 INFO:tasks.workunit.client.0.smithi063.stdout:9/979: readlink d4/d31/d9c/d11b/l99 0 2024-03-23T11:47:57.626 INFO:tasks.workunit.client.0.smithi063.stdout:2/826: link d2/d84/d94/fa1 d2/de/d7a/f114 0 2024-03-23T11:47:57.630 INFO:tasks.workunit.client.0.smithi063.stdout:2/827: mknod d2/d23/d2e/d98/dc4/dcc/c115 0 2024-03-23T11:47:57.632 INFO:tasks.workunit.client.0.smithi063.stdout:2/828: fsync d2/d23/d56/f75 0 2024-03-23T11:47:57.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:57 smithi063 ceph-mon[35189]: pgmap v48: 65 pgs: 65 active+clean; 3.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 61 MiB/s rd, 147 MiB/s wr, 347 op/s 2024-03-23T11:47:57.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:57 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:57.635 INFO:tasks.workunit.client.0.smithi063.stdout:9/980: dwrite d4/d31/d43/da9/da6/dbe/fd2 [0,4194304] 0 2024-03-23T11:47:57.638 INFO:tasks.workunit.client.0.smithi063.stdout:2/829: mknod d2/de/d7a/d8d/c116 0 2024-03-23T11:47:57.638 INFO:tasks.workunit.client.0.smithi063.stdout:9/981: chown d4/d4c/d19/l55 1 1 2024-03-23T11:47:57.640 INFO:tasks.workunit.client.0.smithi063.stdout:7/853: write d5/d5c/d2f/d73/da/d60/fa6 [9198196,58722] 0 2024-03-23T11:47:57.642 INFO:tasks.workunit.client.0.smithi063.stdout:2/830: creat d2/d23/d56/dad/dd4/f117 x:0 0 0 2024-03-23T11:47:57.645 INFO:tasks.workunit.client.0.smithi063.stdout:9/982: creat d4/d31/d9c/d11b/d8d/da0/f14b x:0 0 0 2024-03-23T11:47:57.647 INFO:tasks.workunit.client.0.smithi063.stdout:7/854: creat d5/d5c/d2f/f10f x:0 0 0 2024-03-23T11:47:57.649 INFO:tasks.workunit.client.0.smithi063.stdout:2/831: chown d2/de/d7a/d8d/c116 1 1 2024-03-23T11:47:57.649 INFO:tasks.workunit.client.0.smithi063.stdout:6/739: write d1/f6e [4190318,4532] 0 2024-03-23T11:47:57.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:57 smithi111 ceph-mon[44005]: pgmap v48: 65 pgs: 65 active+clean; 3.6 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 61 MiB/s rd, 147 MiB/s wr, 347 op/s 2024-03-23T11:47:57.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:57 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:47:57.654 INFO:tasks.workunit.client.0.smithi063.stdout:1/848: dwrite d3/f10 [0,4194304] 0 2024-03-23T11:47:57.654 INFO:tasks.workunit.client.0.smithi063.stdout:2/832: stat d2/d23/d2e/d52 0 2024-03-23T11:47:57.660 INFO:tasks.workunit.client.0.smithi063.stdout:7/855: mkdir d5/d5c/d2f/d73/da/d60/d110 0 2024-03-23T11:47:57.666 INFO:tasks.workunit.client.0.smithi063.stdout:7/856: chown d5/d5c/d2f/d73/da/d60/l58 7877944 1 2024-03-23T11:47:57.667 INFO:tasks.workunit.client.0.smithi063.stdout:6/740: creat d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb/feb x:0 0 0 2024-03-23T11:47:57.672 INFO:tasks.workunit.client.0.smithi063.stdout:2/833: dread d2/f85 [0,4194304] 0 2024-03-23T11:47:57.672 INFO:tasks.workunit.client.0.smithi063.stdout:7/857: mknod d5/d5c/d2f/db5/dd2/c111 0 2024-03-23T11:47:57.678 INFO:tasks.workunit.client.0.smithi063.stdout:9/983: dwrite d4/d31/d9c/d11b/d8d/d90/ff1 [0,4194304] 0 2024-03-23T11:47:57.679 INFO:tasks.workunit.client.0.smithi063.stdout:2/834: write d2/d23/d2e/d52/d6c/f88 [5834173,121351] 0 2024-03-23T11:47:57.681 INFO:tasks.workunit.client.0.smithi063.stdout:2/835: readlink d2/d84/lbe 0 2024-03-23T11:47:57.692 INFO:tasks.workunit.client.0.smithi063.stdout:2/836: mknod d2/d84/d94/c118 0 2024-03-23T11:47:57.694 INFO:tasks.workunit.client.0.smithi063.stdout:2/837: stat d2/d46/db4 0 2024-03-23T11:47:57.696 INFO:tasks.workunit.client.0.smithi063.stdout:9/984: mkdir d4/d31/d43/da9/da6/db5/d109/d14c 0 2024-03-23T11:47:57.697 INFO:tasks.workunit.client.0.smithi063.stdout:2/838: write d2/d23/d2e/d98/f10f [104545,55481] 0 2024-03-23T11:47:57.698 INFO:tasks.workunit.client.0.smithi063.stdout:7/858: rename d5/d6/d19/fa8 to d5/d26/db7/dc6/f112 0 2024-03-23T11:47:57.701 INFO:tasks.workunit.client.0.smithi063.stdout:1/849: dread d3/d3f/d4d/dbc/de0/f6d [0,4194304] 0 2024-03-23T11:47:57.701 INFO:tasks.workunit.client.0.smithi063.stdout:6/741: dread d1/dc/d3a/d82/da2/f4a [0,4194304] 0 2024-03-23T11:47:57.718 INFO:tasks.workunit.client.0.smithi063.stdout:6/742: rename d1/dc/d3a/d82/da2/dd2/fe5 to d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb/fec 0 2024-03-23T11:47:57.724 INFO:tasks.workunit.client.0.smithi063.stdout:2/839: dwrite d2/d23/d2e/d87/d4d/f79 [0,4194304] 0 2024-03-23T11:47:57.728 INFO:tasks.workunit.client.0.smithi063.stdout:4/857: sync 2024-03-23T11:47:57.728 INFO:tasks.workunit.client.0.smithi063.stdout:5/606: sync 2024-03-23T11:47:57.732 INFO:tasks.workunit.client.0.smithi063.stdout:5/607: stat d2/f33 0 2024-03-23T11:47:57.736 INFO:tasks.workunit.client.0.smithi063.stdout:6/743: write d1/dc/f91 [659139,82689] 0 2024-03-23T11:47:57.737 INFO:tasks.workunit.client.0.smithi063.stdout:9/985: dwrite d4/d4c/fca [0,4194304] 0 2024-03-23T11:47:57.739 INFO:tasks.workunit.client.0.smithi063.stdout:1/850: dwrite d3/d3f/d4d/f5b [0,4194304] 0 2024-03-23T11:47:57.740 INFO:tasks.workunit.client.0.smithi063.stdout:7/859: dread d5/d26/db7/dce/dc8/dee/f31 [4194304,4194304] 0 2024-03-23T11:47:57.742 INFO:tasks.workunit.client.0.smithi063.stdout:2/840: dread d2/f6b [0,4194304] 0 2024-03-23T11:47:57.747 INFO:tasks.workunit.client.0.smithi063.stdout:2/841: truncate d2/d46/db4/f107 412920 0 2024-03-23T11:47:57.756 INFO:tasks.workunit.client.0.smithi063.stdout:5/608: unlink d2/f33 0 2024-03-23T11:47:57.759 INFO:tasks.workunit.client.0.smithi063.stdout:1/851: rename d3/df/fe9 to d3/df/d2f/de4/f111 0 2024-03-23T11:47:57.766 INFO:tasks.workunit.client.0.smithi063.stdout:4/858: mknod d5/da/d20/d4b/d29/d38/cf9 0 2024-03-23T11:47:57.769 INFO:tasks.workunit.client.0.smithi063.stdout:7/860: creat d5/d6/d19/d65/f113 x:0 0 0 2024-03-23T11:47:57.771 INFO:tasks.workunit.client.0.smithi063.stdout:2/842: creat d2/d23/d2e/d52/d86/f119 x:0 0 0 2024-03-23T11:47:57.775 INFO:tasks.workunit.client.0.smithi063.stdout:1/852: creat d3/d3f/d4d/f112 x:0 0 0 2024-03-23T11:47:57.778 INFO:tasks.workunit.client.0.smithi063.stdout:6/744: dwrite d1/d15/f94 [0,4194304] 0 2024-03-23T11:47:57.779 INFO:tasks.workunit.client.0.smithi063.stdout:9/986: dwrite d4/d31/d43/da9/da6/dbe/f11a [0,4194304] 0 2024-03-23T11:47:57.787 INFO:tasks.workunit.client.0.smithi063.stdout:7/861: mkdir d5/d26/dde/d100/da1/d114 0 2024-03-23T11:47:57.792 INFO:tasks.workunit.client.0.smithi063.stdout:2/843: fdatasync d2/de/d22/f78 0 2024-03-23T11:47:57.800 INFO:tasks.workunit.client.0.smithi063.stdout:1/853: truncate d3/d49/df4/fe5 513640 0 2024-03-23T11:47:57.803 INFO:tasks.workunit.client.0.smithi063.stdout:9/987: dwrite d4/d31/d43/da9/da6/dbe/f136 [0,4194304] 0 2024-03-23T11:47:57.811 INFO:tasks.workunit.client.0.smithi063.stdout:6/745: truncate d1/dc/d3a/d57/f98 524328 0 2024-03-23T11:47:57.815 INFO:tasks.workunit.client.0.smithi063.stdout:6/746: chown d1/dc/l9f 107175815 1 2024-03-23T11:47:57.815 INFO:tasks.workunit.client.0.smithi063.stdout:9/988: rmdir d4/d4c/d19 39 2024-03-23T11:47:57.820 INFO:tasks.workunit.client.0.smithi063.stdout:5/609: dwrite d2/d28/d3a/fa5 [0,4194304] 0 2024-03-23T11:47:57.821 INFO:tasks.workunit.client.0.smithi063.stdout:2/844: dwrite d2/de/d22/dbc/fdb [0,4194304] 0 2024-03-23T11:47:57.821 INFO:tasks.workunit.client.0.smithi063.stdout:6/747: chown d1/dc/d3a/d82/da2/d41/d42/f48 107374 1 2024-03-23T11:47:57.825 INFO:tasks.workunit.client.0.smithi063.stdout:4/859: link d5/da/d7e/dbf/fe7 d5/da/d20/d4b/d16/d32/d56/ffa 0 2024-03-23T11:47:57.825 INFO:tasks.workunit.client.0.smithi063.stdout:6/748: write d1/dc/d3a/d82/da2/d41/d42/fc8 [752030,44696] 0 2024-03-23T11:47:57.826 INFO:tasks.workunit.client.0.smithi063.stdout:7/862: rename d5/d5c/d2f/d73/c15 to d5/d26/c115 0 2024-03-23T11:47:57.832 INFO:tasks.workunit.client.0.smithi063.stdout:2/845: rmdir d2/d46/d54 39 2024-03-23T11:47:57.832 INFO:tasks.workunit.client.0.smithi063.stdout:1/854: getdents d3/d3f/d4d/dbc 0 2024-03-23T11:47:57.838 INFO:tasks.workunit.client.0.smithi063.stdout:4/860: mknod d5/da/d7e/cfb 0 2024-03-23T11:47:57.842 INFO:tasks.workunit.client.0.smithi063.stdout:5/610: mkdir d2/d7/d8/de/d49/db0/dd2 0 2024-03-23T11:47:57.843 INFO:tasks.workunit.client.0.smithi063.stdout:2/846: creat d2/d84/f11a x:0 0 0 2024-03-23T11:47:57.845 INFO:tasks.workunit.client.0.smithi063.stdout:4/861: truncate d5/d1e/d2b/faf 936257 0 2024-03-23T11:47:57.846 INFO:tasks.workunit.client.0.smithi063.stdout:9/989: fdatasync d4/d31/d43/da9/da6/dbe/f136 0 2024-03-23T11:47:57.853 INFO:tasks.workunit.client.0.smithi063.stdout:2/847: mknod d2/d23/c11b 0 2024-03-23T11:47:57.855 INFO:tasks.workunit.client.0.smithi063.stdout:1/855: getdents d3/df/d2f/de4 0 2024-03-23T11:47:57.857 INFO:tasks.workunit.client.0.smithi063.stdout:9/990: truncate d4/d31/f59 6136132 0 2024-03-23T11:47:57.862 INFO:tasks.workunit.client.0.smithi063.stdout:9/991: write d4/d31/d9c/d11b/d8d/d90/f140 [3720709,29215] 0 2024-03-23T11:47:57.869 INFO:tasks.workunit.client.0.smithi063.stdout:4/862: dread d5/da/d20/f77 [0,4194304] 0 2024-03-23T11:47:57.869 INFO:tasks.workunit.client.0.smithi063.stdout:7/863: write d5/d5c/d2f/f79 [3284556,6476] 0 2024-03-23T11:47:57.871 INFO:tasks.workunit.client.0.smithi063.stdout:9/992: creat d4/d31/d9c/d11b/d8d/da0/d12d/f14d x:0 0 0 2024-03-23T11:47:57.876 INFO:tasks.workunit.client.0.smithi063.stdout:9/993: fdatasync d4/d31/d43/d49/f9a 0 2024-03-23T11:47:57.879 INFO:tasks.workunit.client.0.smithi063.stdout:6/749: dread d1/dc/d3a/f40 [0,4194304] 0 2024-03-23T11:47:57.939 INFO:tasks.workunit.client.0.smithi063.stdout:9/994: rename d4/d31/d43/fdd to d4/d31/d9c/d9e/f14e 0 2024-03-23T11:47:57.940 INFO:tasks.workunit.client.0.smithi063.stdout:5/611: write d2/d7/d8/de/d2a/d3b/d85/f9e [32205,96481] 0 2024-03-23T11:47:57.944 INFO:tasks.workunit.client.0.smithi063.stdout:9/995: chown d4/d31/d43/da9/da6/dbe/c12e 1277540 1 2024-03-23T11:47:57.946 INFO:tasks.workunit.client.0.smithi063.stdout:1/856: truncate d3/f4 3199708 0 2024-03-23T11:47:57.949 INFO:tasks.workunit.client.0.smithi063.stdout:1/857: readlink d3/df/d2f/l45 0 2024-03-23T11:47:57.950 INFO:tasks.workunit.client.0.smithi063.stdout:5/612: mknod d2/d7/d8/de/d47/cd3 0 2024-03-23T11:47:57.955 INFO:tasks.workunit.client.0.smithi063.stdout:4/863: write d5/da/d20/d4b/d1f/f41 [2179226,9657] 0 2024-03-23T11:47:57.957 INFO:tasks.workunit.client.0.smithi063.stdout:1/858: dread - d3/d49/d56/d65/d84/da7/de3/fd4 zero size 2024-03-23T11:47:57.965 INFO:tasks.workunit.client.0.smithi063.stdout:4/864: dread - d5/da/d7e/dbf/fe7 zero size 2024-03-23T11:47:57.967 INFO:tasks.workunit.client.0.smithi063.stdout:5/613: truncate d2/d7/d8/de/d2a/d3b/d43/d5c/fbf 181849 0 2024-03-23T11:47:57.968 INFO:tasks.workunit.client.0.smithi063.stdout:1/859: mkdir d3/d3f/d4d/dbc/d113 0 2024-03-23T11:47:57.971 INFO:tasks.workunit.client.0.smithi063.stdout:6/750: dwrite d1/d15/d4b/f6b [0,4194304] 0 2024-03-23T11:47:57.971 INFO:tasks.workunit.client.0.smithi063.stdout:7/864: dread d5/d26/db7/dc6/f112 [0,4194304] 0 2024-03-23T11:47:57.976 INFO:tasks.workunit.client.0.smithi063.stdout:6/751: stat d1/dc/d21/d49/f95 0 2024-03-23T11:47:57.977 INFO:tasks.workunit.client.0.smithi063.stdout:4/865: chown d5/da/d20/d4b/d16/d50/d8e/d96/dc0 3 1 2024-03-23T11:47:57.977 INFO:tasks.workunit.client.0.smithi063.stdout:7/865: stat d5/d26/db7/dc6/lbd 0 2024-03-23T11:47:57.983 INFO:tasks.workunit.client.0.smithi063.stdout:1/860: truncate d3/df/f1a 7807746 0 2024-03-23T11:47:57.987 INFO:tasks.workunit.client.0.smithi063.stdout:6/752: symlink d1/dc/d3a/d82/d85/led 0 2024-03-23T11:47:57.992 INFO:tasks.workunit.client.0.smithi063.stdout:1/861: symlink d3/df/d2f/de4/l114 0 2024-03-23T11:47:57.993 INFO:tasks.workunit.client.0.smithi063.stdout:6/753: chown d1/dc/d3a/d82/da2/d41/d42/d99/dc3 61762 1 2024-03-23T11:47:57.999 INFO:tasks.workunit.client.0.smithi063.stdout:6/754: creat d1/dc/d3a/fee x:0 0 0 2024-03-23T11:47:58.001 INFO:tasks.workunit.client.0.smithi063.stdout:4/866: getdents d5/da/d7e 0 2024-03-23T11:47:58.005 INFO:tasks.workunit.client.0.smithi063.stdout:9/996: dread d4/f7b [0,4194304] 0 2024-03-23T11:47:58.007 INFO:tasks.workunit.client.0.smithi063.stdout:6/755: write d1/d15/fb8 [702521,43435] 0 2024-03-23T11:47:58.013 INFO:tasks.workunit.client.0.smithi063.stdout:9/997: write d4/d31/d43/da9/da6/ffc [3467376,119598] 0 2024-03-23T11:47:58.013 INFO:tasks.workunit.client.0.smithi063.stdout:7/866: write d5/d6/d19/d1f/d6c/f80 [598292,32015] 0 2024-03-23T11:47:58.020 INFO:tasks.workunit.client.0.smithi063.stdout:7/867: truncate d5/d5c/d2f/d8b/f10d 794591 0 2024-03-23T11:47:58.025 INFO:tasks.workunit.client.0.smithi063.stdout:6/756: rename d1/dc/d3a/d82/da2/d41/d42/f48 to d1/dc/dea/fef 0 2024-03-23T11:47:58.044 INFO:tasks.workunit.client.0.smithi063.stdout:1/862: dwrite d3/f5d [0,4194304] 0 2024-03-23T11:47:58.047 INFO:tasks.workunit.client.0.smithi063.stdout:9/998: dwrite d4/f14a [0,4194304] 0 2024-03-23T11:47:58.053 INFO:tasks.workunit.client.0.smithi063.stdout:5/614: dread d2/f6 [0,4194304] 0 2024-03-23T11:47:58.056 INFO:tasks.workunit.client.0.smithi063.stdout:4/867: dwrite d5/da/d20/d4b/d1f/f26 [0,4194304] 0 2024-03-23T11:47:58.064 INFO:tasks.workunit.client.0.smithi063.stdout:6/757: write d1/dc/d3a/d82/da2/d41/d42/d99/dc3/faa [356876,125634] 0 2024-03-23T11:47:58.081 INFO:tasks.workunit.client.0.smithi063.stdout:4/868: dread d5/d1e/f82 [0,4194304] 0 2024-03-23T11:47:58.083 INFO:tasks.workunit.client.0.smithi063.stdout:7/868: dread d5/d5c/d2f/d73/da/f89 [0,4194304] 0 2024-03-23T11:47:58.086 INFO:tasks.workunit.client.0.smithi063.stdout:7/869: fsync d5/d6/d19/dcf/ffd 0 2024-03-23T11:47:58.086 INFO:tasks.workunit.client.0.smithi063.stdout:6/758: dwrite d1/dc/d8c/f96 [0,4194304] 0 2024-03-23T11:47:58.089 INFO:tasks.workunit.client.0.smithi063.stdout:5/615: getdents d2/d7/d2b/d8b 0 2024-03-23T11:47:58.090 INFO:tasks.workunit.client.0.smithi063.stdout:6/759: write d1/d15/fd1 [3034551,63961] 0 2024-03-23T11:47:58.092 INFO:tasks.workunit.client.0.smithi063.stdout:1/863: rmdir d3/d3f/d4d/dbc/d113 0 2024-03-23T11:47:58.092 INFO:tasks.workunit.client.0.smithi063.stdout:5/616: write d2/d7/d8/de/d49/db0/fcc [342637,56206] 0 2024-03-23T11:47:58.094 INFO:tasks.workunit.client.0.smithi063.stdout:4/869: creat d5/da/d20/d4b/d16/d37/ffc x:0 0 0 2024-03-23T11:47:58.110 INFO:tasks.workunit.client.0.smithi063.stdout:9/999: dread d4/f47 [0,4194304] 0 2024-03-23T11:47:58.111 INFO:tasks.workunit.client.0.smithi063.stdout:7/870: mkdir d5/d26/dde/d100/da1/db9/d116 0 2024-03-23T11:47:58.113 INFO:tasks.workunit.client.0.smithi063.stdout:4/870: truncate d5/da/d20/d4b/f2d 2838191 0 2024-03-23T11:47:58.115 INFO:tasks.workunit.client.0.smithi063.stdout:5/617: write d2/d17/fb8 [235584,24339] 0 2024-03-23T11:47:58.119 INFO:tasks.workunit.client.0.smithi063.stdout:1/864: dwrite d3/f10 [8388608,4194304] 0 2024-03-23T11:47:58.125 INFO:tasks.workunit.client.0.smithi063.stdout:4/871: symlink d5/da/d20/d4b/d16/d50/lfd 0 2024-03-23T11:47:58.130 INFO:tasks.workunit.client.0.smithi063.stdout:1/865: rename d3/df/d2f/de4/df5/ld2 to d3/d3f/d4d/dbc/de0/dce/l115 0 2024-03-23T11:47:58.135 INFO:tasks.workunit.client.0.smithi063.stdout:7/871: link d5/d26/db7/dd6/d88/la7 d5/d5c/d2f/d73/da/d60/d110/l117 0 2024-03-23T11:47:58.137 INFO:tasks.workunit.client.0.smithi063.stdout:4/872: rmdir d5/da/d20/d4b/d1f/df3 39 2024-03-23T11:47:58.141 INFO:tasks.workunit.client.0.smithi063.stdout:1/866: creat d3/d3f/d4d/dbc/de0/d4b/da5/f116 x:0 0 0 2024-03-23T11:47:58.142 INFO:tasks.workunit.client.0.smithi063.stdout:4/873: creat d5/da/dc3/ffe x:0 0 0 2024-03-23T11:47:58.151 INFO:tasks.workunit.client.0.smithi063.stdout:7/872: symlink d5/d26/db7/dce/dc8/dee/l118 0 2024-03-23T11:47:58.151 INFO:tasks.workunit.client.0.smithi063.stdout:1/867: fdatasync d3/d3f/d4d/dbc/de0/f66 0 2024-03-23T11:47:58.154 INFO:tasks.workunit.client.0.smithi063.stdout:6/760: dwrite d1/dc/d3a/f3f [0,4194304] 0 2024-03-23T11:47:58.156 INFO:tasks.workunit.client.0.smithi063.stdout:7/873: chown d5/d5c/d2f/d73/da/d60/d110 22506437 1 2024-03-23T11:47:58.163 INFO:tasks.workunit.client.0.smithi063.stdout:5/618: dread d2/d17/f19 [0,4194304] 0 2024-03-23T11:47:58.164 INFO:tasks.workunit.client.0.smithi063.stdout:4/874: symlink d5/da/d20/d4b/d16/d50/d8e/dd3/d94/lff 0 2024-03-23T11:47:58.166 INFO:tasks.workunit.client.0.smithi063.stdout:6/761: creat d1/dc/d3a/d82/da2/dd2/ff0 x:0 0 0 2024-03-23T11:47:58.169 INFO:tasks.workunit.client.0.smithi063.stdout:7/874: chown d5/d22/c30 1073 1 2024-03-23T11:47:58.170 INFO:tasks.workunit.client.0.smithi063.stdout:2/848: sync 2024-03-23T11:47:58.170 INFO:tasks.workunit.client.0.smithi063.stdout:1/868: mkdir d3/d3f/d117 0 2024-03-23T11:47:58.177 INFO:tasks.workunit.client.0.smithi063.stdout:4/875: rename d5/d1e/f75 to d5/da/d20/d4b/d16/d50/d8e/d96/dc0/f100 0 2024-03-23T11:47:58.177 INFO:tasks.workunit.client.0.smithi063.stdout:6/762: mkdir d1/dc/dea/df1 0 2024-03-23T11:47:58.177 INFO:tasks.workunit.client.0.smithi063.stdout:5/619: read d2/d28/f4f [2328957,92154] 0 2024-03-23T11:47:58.180 INFO:tasks.workunit.client.0.smithi063.stdout:7/875: mknod d5/d26/d76/c119 0 2024-03-23T11:47:58.180 INFO:tasks.workunit.client.0.smithi063.stdout:5/620: chown d2/d17/d4d/d6e/dba/dbe 2167 1 2024-03-23T11:47:58.182 INFO:tasks.workunit.client.0.smithi063.stdout:6/763: chown d1/dc/d8c/f96 1325 1 2024-03-23T11:47:58.183 INFO:tasks.workunit.client.0.smithi063.stdout:1/869: write d3/d62/fef [1160362,104667] 0 2024-03-23T11:47:58.185 INFO:tasks.workunit.client.0.smithi063.stdout:5/621: dread - d2/d28/fc9 zero size 2024-03-23T11:47:58.193 INFO:tasks.workunit.client.0.smithi063.stdout:6/764: read d1/dc/d21/fa9 [3043270,123104] 0 2024-03-23T11:47:58.193 INFO:tasks.workunit.client.0.smithi063.stdout:7/876: write d5/d5c/d2f/fda [533663,59967] 0 2024-03-23T11:47:58.198 INFO:tasks.workunit.client.0.smithi063.stdout:7/877: readlink d5/d26/db7/dd6/l99 0 2024-03-23T11:47:58.205 INFO:tasks.workunit.client.0.smithi063.stdout:2/849: dwrite d2/de/d7a/d8d/f109 [0,4194304] 0 2024-03-23T11:47:58.209 INFO:tasks.workunit.client.0.smithi063.stdout:5/622: write d2/d7/d8/de/d2a/d3b/d43/d5c/f62 [2760201,1056] 0 2024-03-23T11:47:58.211 INFO:tasks.workunit.client.0.smithi063.stdout:7/878: write d5/d26/f72 [3975028,59448] 0 2024-03-23T11:47:58.211 INFO:tasks.workunit.client.0.smithi063.stdout:6/765: dread d1/dc/f1f [0,4194304] 0 2024-03-23T11:47:58.215 INFO:tasks.workunit.client.0.smithi063.stdout:6/766: readlink d1/dc/d8c/lc6 0 2024-03-23T11:47:58.217 INFO:tasks.workunit.client.0.smithi063.stdout:1/870: dwrite d3/d3f/d4d/dbc/de0/d4b/da5/dd9/f5f [0,4194304] 0 2024-03-23T11:47:58.222 INFO:tasks.workunit.client.0.smithi063.stdout:1/871: stat d3/df/d2f/ff0 0 2024-03-23T11:47:58.230 INFO:tasks.workunit.client.0.smithi063.stdout:5/623: creat d2/d7/d2b/d8b/da7/dc6/fd4 x:0 0 0 2024-03-23T11:47:58.231 INFO:tasks.workunit.client.0.smithi063.stdout:6/767: creat d1/dc/d21/da5/ff2 x:0 0 0 2024-03-23T11:47:58.241 INFO:tasks.workunit.client.0.smithi063.stdout:6/768: readlink d1/dc/d3a/d82/da2/l7e 0 2024-03-23T11:47:58.244 INFO:tasks.workunit.client.0.smithi063.stdout:5/624: creat d2/d7/d2b/d8b/da7/fd5 x:0 0 0 2024-03-23T11:47:58.244 INFO:tasks.workunit.client.0.smithi063.stdout:7/879: dread d5/d5c/d2f/d8c/fb0 [0,4194304] 0 2024-03-23T11:47:58.246 INFO:tasks.workunit.client.0.smithi063.stdout:5/625: readlink d2/d7/d8/d1d/l21 0 2024-03-23T11:47:58.248 INFO:tasks.workunit.client.0.smithi063.stdout:1/872: dread d3/f5d [0,4194304] 0 2024-03-23T11:47:58.248 INFO:tasks.workunit.client.0.smithi063.stdout:6/769: creat d1/dc/d3a/d82/da2/dd8/ff3 x:0 0 0 2024-03-23T11:47:58.248 INFO:tasks.workunit.client.0.smithi063.stdout:2/850: dwrite d2/d23/d56/dad/dd4/dd6/fdf [0,4194304] 0 2024-03-23T11:47:58.254 INFO:tasks.workunit.client.0.smithi063.stdout:1/873: fsync d3/d49/d56/d65/d84/da7/de3/f105 0 2024-03-23T11:47:58.257 INFO:tasks.workunit.client.0.smithi063.stdout:7/880: symlink d5/d26/db7/dce/d64/l11a 0 2024-03-23T11:47:58.261 INFO:tasks.workunit.client.0.smithi063.stdout:1/874: mkdir d3/d3f/d4d/dbc/d118 0 2024-03-23T11:47:58.264 INFO:tasks.workunit.client.0.smithi063.stdout:2/851: read d2/d23/d2e/d52/d6c/f7c [287120,33297] 0 2024-03-23T11:47:58.273 INFO:tasks.workunit.client.0.smithi063.stdout:1/875: dread d3/d49/d56/d65/db6/dc5/fdf [0,4194304] 0 2024-03-23T11:47:58.274 INFO:tasks.workunit.client.0.smithi063.stdout:6/770: rename d1/dc/d21/da5 to d1/dc/d3a/d82/d85/df4 0 2024-03-23T11:47:58.279 INFO:tasks.workunit.client.0.smithi063.stdout:6/771: write d1/dc/d3a/d57/d6d/fc5 [81923,8205] 0 2024-03-23T11:47:58.280 INFO:tasks.workunit.client.0.smithi063.stdout:2/852: link d2/de/d22/f78 d2/f11c 0 2024-03-23T11:47:58.284 INFO:tasks.workunit.client.0.smithi063.stdout:1/876: rename d3/d49/d56/d65/cbe to d3/d3f/d4d/dbc/d118/c119 0 2024-03-23T11:47:58.285 INFO:tasks.workunit.client.0.smithi063.stdout:6/772: fsync d1/f5 0 2024-03-23T11:47:58.287 INFO:tasks.workunit.client.0.smithi063.stdout:2/853: mkdir d2/d23/d56/dad/d11d 0 2024-03-23T11:47:58.291 INFO:tasks.workunit.client.0.smithi063.stdout:6/773: chown d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb 55250031 1 2024-03-23T11:47:58.293 INFO:tasks.workunit.client.0.smithi063.stdout:5/626: dread d2/d28/d3a/d46/f86 [0,4194304] 0 2024-03-23T11:47:58.294 INFO:tasks.workunit.client.0.smithi063.stdout:7/881: dread d5/d26/db7/dce/dc8/dee/f82 [0,4194304] 0 2024-03-23T11:47:58.295 INFO:tasks.workunit.client.0.smithi063.stdout:1/877: creat d3/d35/d40/dee/dfa/f11a x:0 0 0 2024-03-23T11:47:58.295 INFO:tasks.workunit.client.0.smithi063.stdout:2/854: getdents d2/d23/d2e/d87/dec 0 2024-03-23T11:47:58.297 INFO:tasks.workunit.client.0.smithi063.stdout:7/882: rmdir d5 39 2024-03-23T11:47:58.298 INFO:tasks.workunit.client.0.smithi063.stdout:5/627: write d2/d28/f3d [753720,1755] 0 2024-03-23T11:47:58.303 INFO:tasks.workunit.client.0.smithi063.stdout:6/774: link d1/dc/d3a/f4f d1/ff5 0 2024-03-23T11:47:58.307 INFO:tasks.workunit.client.0.smithi063.stdout:6/775: fdatasync d1/dc/d3a/d82/da2/dd2/fda 0 2024-03-23T11:47:58.307 INFO:tasks.workunit.client.0.smithi063.stdout:7/883: creat d5/d6/d19/f11b x:0 0 0 2024-03-23T11:47:58.311 INFO:tasks.workunit.client.0.smithi063.stdout:5/628: truncate d2/d7/d8/de/d35/f60 1539027 0 2024-03-23T11:47:58.312 INFO:tasks.workunit.client.0.smithi063.stdout:7/884: truncate d5/d22/da3/fec 1660593 0 2024-03-23T11:47:58.315 INFO:tasks.workunit.client.0.smithi063.stdout:7/885: fsync d5/d6/d19/d65/fba 0 2024-03-23T11:47:58.318 INFO:tasks.workunit.client.0.smithi063.stdout:6/776: mknod d1/dc/dea/df1/cf6 0 2024-03-23T11:47:58.320 INFO:tasks.workunit.client.0.smithi063.stdout:7/886: mkdir d5/d5c/d2f/d73/da/d11c 0 2024-03-23T11:47:58.324 INFO:tasks.workunit.client.0.smithi063.stdout:5/629: write d2/d7/d8/d1d/fcb [58687,50961] 0 2024-03-23T11:47:58.324 INFO:tasks.workunit.client.0.smithi063.stdout:6/777: dread - d1/dc/d3a/d82/d85/df4/ff2 zero size 2024-03-23T11:47:58.330 INFO:tasks.workunit.client.0.smithi063.stdout:1/878: dwrite d3/d3f/d4d/dbc/de0/d4b/fbb [0,4194304] 0 2024-03-23T11:47:58.331 INFO:tasks.workunit.client.0.smithi063.stdout:7/887: write d5/d6/f25 [360923,36912] 0 2024-03-23T11:47:58.334 INFO:tasks.workunit.client.0.smithi063.stdout:7/888: fsync d5/d26/d32/faf 0 2024-03-23T11:47:58.335 INFO:tasks.workunit.client.0.smithi063.stdout:5/630: read d2/d7/d8/de/d2a/d3b/d43/d5c/f62 [1651775,113586] 0 2024-03-23T11:47:58.335 INFO:tasks.workunit.client.0.smithi063.stdout:2/855: dread d2/d84/d92/fb0 [0,4194304] 0 2024-03-23T11:47:58.336 INFO:tasks.workunit.client.0.smithi063.stdout:6/778: mkdir d1/dc/d3a/d57/df7 0 2024-03-23T11:47:58.337 INFO:tasks.workunit.client.0.smithi063.stdout:5/631: fdatasync f1 0 2024-03-23T11:47:58.342 INFO:tasks.workunit.client.0.smithi063.stdout:6/779: write d1/dc/d3a/d82/da2/f62 [281594,59143] 0 2024-03-23T11:47:58.350 INFO:tasks.workunit.client.0.smithi063.stdout:7/889: unlink d5/d22/da3/f108 0 2024-03-23T11:47:58.354 INFO:tasks.workunit.client.0.smithi063.stdout:2/856: truncate d2/d23/d56/dad/fd8 3458185 0 2024-03-23T11:47:58.357 INFO:tasks.workunit.client.0.smithi063.stdout:7/890: mknod d5/d26/dde/d100/c11d 0 2024-03-23T11:47:58.362 INFO:tasks.workunit.client.0.smithi063.stdout:7/891: mknod d5/d5c/d2f/d73/da/c11e 0 2024-03-23T11:47:58.363 INFO:tasks.workunit.client.0.smithi063.stdout:2/857: truncate d2/d1c/f96 1505987 0 2024-03-23T11:47:58.373 INFO:tasks.workunit.client.0.smithi063.stdout:2/858: getdents d2/d23/d56/dad/dd4 0 2024-03-23T11:47:58.376 INFO:tasks.workunit.client.0.smithi063.stdout:6/780: dread d1/dc/d3a/d82/da2/f83 [0,4194304] 0 2024-03-23T11:47:58.377 INFO:tasks.workunit.client.0.smithi063.stdout:2/859: write d2/f6b [809537,100208] 0 2024-03-23T11:47:58.383 INFO:tasks.workunit.client.0.smithi063.stdout:2/860: creat d2/d23/d56/dad/d11d/f11e x:0 0 0 2024-03-23T11:47:58.384 INFO:tasks.workunit.client.0.smithi063.stdout:6/781: creat d1/dc/d3a/d57/df7/ff8 x:0 0 0 2024-03-23T11:47:58.384 INFO:tasks.workunit.client.0.smithi063.stdout:7/892: dwrite d5/d5c/d2f/d73/da/f89 [4194304,4194304] 0 2024-03-23T11:47:58.399 INFO:tasks.workunit.client.0.smithi063.stdout:6/782: dread d1/dc/f46 [0,4194304] 0 2024-03-23T11:47:58.407 INFO:tasks.workunit.client.0.smithi063.stdout:7/893: dwrite d5/d26/db7/dce/d64/f10c [0,4194304] 0 2024-03-23T11:47:58.409 INFO:tasks.workunit.client.0.smithi063.stdout:6/783: link d1/dc/d3a/d82/d85/led d1/d13/lf9 0 2024-03-23T11:47:58.415 INFO:tasks.workunit.client.0.smithi063.stdout:2/861: dread d2/d23/d2e/d98/fba [0,4194304] 0 2024-03-23T11:47:58.422 INFO:tasks.workunit.client.0.smithi063.stdout:6/784: rename d1/dc/d3a/d82/da2/l47 to d1/dc/d3a/d82/da2/d41/d42/d99/de9/lfa 0 2024-03-23T11:47:58.424 INFO:tasks.workunit.client.0.smithi063.stdout:2/862: creat d2/d23/d2e/d87/dec/f11f x:0 0 0 2024-03-23T11:47:58.426 INFO:tasks.workunit.client.0.smithi063.stdout:6/785: symlink d1/dc/d3a/d82/da2/dd8/lfb 0 2024-03-23T11:47:58.429 INFO:tasks.workunit.client.0.smithi063.stdout:6/786: readlink d1/dc/d3a/d82/da2/d41/d42/l5f 0 2024-03-23T11:47:58.433 INFO:tasks.workunit.client.0.smithi063.stdout:7/894: dwrite d5/d26/db7/dd6/ffa [0,4194304] 0 2024-03-23T11:47:58.435 INFO:tasks.workunit.client.0.smithi063.stdout:7/895: write d5/d26/dff/f7b [2053042,29459] 0 2024-03-23T11:47:58.437 INFO:tasks.workunit.client.0.smithi063.stdout:6/787: creat d1/dc/d3a/d82/da2/d41/d42/ffc x:0 0 0 2024-03-23T11:47:58.443 INFO:tasks.workunit.client.0.smithi063.stdout:6/788: mkdir d1/dc/d3a/d57/d6d/ddb/dfd 0 2024-03-23T11:47:58.444 INFO:tasks.workunit.client.0.smithi063.stdout:2/863: dwrite d2/d23/d56/f75 [0,4194304] 0 2024-03-23T11:47:58.446 INFO:tasks.workunit.client.0.smithi063.stdout:7/896: link d5/d26/db7/dce/d64/l9e d5/d26/db7/dd6/d88/l11f 0 2024-03-23T11:47:58.450 INFO:tasks.workunit.client.0.smithi063.stdout:7/897: readlink d5/d26/db7/dd6/d88/l107 0 2024-03-23T11:47:58.452 INFO:tasks.workunit.client.0.smithi063.stdout:7/898: dread - d5/d6/d19/f11b zero size 2024-03-23T11:47:58.459 INFO:tasks.workunit.client.0.smithi063.stdout:7/899: creat d5/d6/f120 x:0 0 0 2024-03-23T11:47:58.460 INFO:tasks.workunit.client.0.smithi063.stdout:2/864: write d2/de/d7a/d8d/f9e [4355685,63526] 0 2024-03-23T11:47:58.461 INFO:tasks.workunit.client.0.smithi063.stdout:7/900: chown d5/d6/f25 5662757 1 2024-03-23T11:47:58.469 INFO:tasks.workunit.client.0.smithi063.stdout:2/865: unlink d2/d46/d54/fdd 0 2024-03-23T11:47:58.469 INFO:tasks.workunit.client.0.smithi063.stdout:6/789: dwrite d1/f3b [0,4194304] 0 2024-03-23T11:47:58.470 INFO:tasks.workunit.client.0.smithi063.stdout:2/866: readlink d2/d84/lf3 0 2024-03-23T11:47:58.473 INFO:tasks.workunit.client.0.smithi063.stdout:2/867: dread - d2/d23/d2e/d87/f80 zero size 2024-03-23T11:47:58.475 INFO:tasks.workunit.client.0.smithi063.stdout:6/790: mknod d1/dc/d21/cfe 0 2024-03-23T11:47:58.479 INFO:tasks.workunit.client.0.smithi063.stdout:6/791: mknod d1/dc/d21/d49/cff 0 2024-03-23T11:47:58.482 INFO:tasks.workunit.client.0.smithi063.stdout:7/901: truncate d5/d26/dff/f7b 579007 0 2024-03-23T11:47:58.485 INFO:tasks.workunit.client.0.smithi063.stdout:7/902: read d5/d26/db7/dc6/f112 [260554,5948] 0 2024-03-23T11:47:58.485 INFO:tasks.workunit.client.0.smithi063.stdout:6/792: creat d1/dc/dea/df1/f100 x:0 0 0 2024-03-23T11:47:58.488 INFO:tasks.workunit.client.0.smithi063.stdout:6/793: chown d1/dc/d3a/cca 10 1 2024-03-23T11:47:58.490 INFO:tasks.workunit.client.0.smithi063.stdout:6/794: stat d1/dc/d3a/d57/d6d/fa0 0 2024-03-23T11:47:58.493 INFO:tasks.workunit.client.0.smithi063.stdout:6/795: write d1/f6e [3385342,96915] 0 2024-03-23T11:47:58.494 INFO:tasks.workunit.client.0.smithi063.stdout:6/796: dread - d1/f67 zero size 2024-03-23T11:47:58.500 INFO:tasks.workunit.client.0.smithi063.stdout:2/868: dread d2/d23/d2e/d52/f4c [0,4194304] 0 2024-03-23T11:47:58.500 INFO:tasks.workunit.client.0.smithi063.stdout:6/797: fsync d1/dc/d3a/d57/d77/f87 0 2024-03-23T11:47:58.506 INFO:tasks.workunit.client.0.smithi063.stdout:7/903: dread d5/d5c/d2f/d73/da/f89 [0,4194304] 0 2024-03-23T11:47:58.509 INFO:tasks.workunit.client.0.smithi063.stdout:7/904: rmdir d5/d26/dde/d100/da1/db9 39 2024-03-23T11:47:58.510 INFO:tasks.workunit.client.0.smithi063.stdout:2/869: dread d2/de/d7a/d8d/fa9 [0,4194304] 0 2024-03-23T11:47:58.512 INFO:tasks.workunit.client.0.smithi063.stdout:7/905: mkdir d5/d26/dff/d121 0 2024-03-23T11:47:58.515 INFO:tasks.workunit.client.0.smithi063.stdout:7/906: mknod d5/d26/db7/dce/d64/c122 0 2024-03-23T11:47:58.517 INFO:tasks.workunit.client.0.smithi063.stdout:7/907: truncate f1 4469114 0 2024-03-23T11:47:58.518 INFO:tasks.workunit.client.0.smithi063.stdout:2/870: link d2/d84/d94/fa1 d2/d23/d2e/d98/dc4/f120 0 2024-03-23T11:47:58.524 INFO:tasks.workunit.client.0.smithi063.stdout:2/871: mknod d2/d23/d2e/d98/dc8/df6/c121 0 2024-03-23T11:47:58.524 INFO:tasks.workunit.client.0.smithi063.stdout:6/798: dwrite d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb/feb [0,4194304] 0 2024-03-23T11:47:58.526 INFO:tasks.workunit.client.0.smithi063.stdout:6/799: fdatasync d1/d15/fb8 0 2024-03-23T11:47:58.526 INFO:tasks.workunit.client.0.smithi063.stdout:2/872: dread - d2/d23/d2e/d87/f80 zero size 2024-03-23T11:47:58.530 INFO:tasks.workunit.client.0.smithi063.stdout:2/873: mkdir d2/d84/d92/d122 0 2024-03-23T11:47:58.535 INFO:tasks.workunit.client.0.smithi063.stdout:2/874: mkdir d2/d23/d2e/d98/dd2/d123 0 2024-03-23T11:47:58.540 INFO:tasks.workunit.client.0.smithi063.stdout:7/908: dwrite d5/d6/d19/d1f/d6c/fd9 [0,4194304] 0 2024-03-23T11:47:58.544 INFO:tasks.workunit.client.0.smithi063.stdout:6/800: truncate d1/d13/f16 2728618 0 2024-03-23T11:47:58.549 INFO:tasks.workunit.client.0.smithi063.stdout:2/875: rmdir d2 39 2024-03-23T11:47:58.549 INFO:tasks.workunit.client.0.smithi063.stdout:6/801: write d1/dc/fab [1087562,27791] 0 2024-03-23T11:47:58.567 INFO:tasks.workunit.client.0.smithi063.stdout:7/909: dwrite d5/d5c/d2f/d73/da/f4e [0,4194304] 0 2024-03-23T11:47:58.572 INFO:tasks.workunit.client.0.smithi063.stdout:6/802: dread d1/dc/fab [0,4194304] 0 2024-03-23T11:47:58.578 INFO:tasks.workunit.client.0.smithi063.stdout:2/876: mknod d2/d23/d2e/d87/c124 0 2024-03-23T11:47:58.579 INFO:tasks.workunit.client.0.smithi063.stdout:6/803: creat d1/d15/d4b/f101 x:0 0 0 2024-03-23T11:47:58.582 INFO:tasks.workunit.client.0.smithi063.stdout:2/877: fdatasync d2/d23/d56/fa4 0 2024-03-23T11:47:58.586 INFO:tasks.workunit.client.0.smithi063.stdout:2/878: readlink d2/d23/d2e/d98/le8 0 2024-03-23T11:47:58.589 INFO:tasks.workunit.client.0.smithi063.stdout:7/910: dwrite d5/d26/db7/dd6/f6a [0,4194304] 0 2024-03-23T11:47:58.589 INFO:tasks.workunit.client.0.smithi063.stdout:2/879: rmdir d2/de/d7a/da2 39 2024-03-23T11:47:58.595 INFO:tasks.workunit.client.0.smithi063.stdout:7/911: chown d5/d5c/d2f/d73/da/l11 11645403 1 2024-03-23T11:47:58.596 INFO:tasks.workunit.client.0.smithi063.stdout:2/880: mkdir d2/de/d22/dbc/d10c/d125 0 2024-03-23T11:47:58.603 INFO:tasks.workunit.client.0.smithi063.stdout:2/881: creat d2/d84/d92/d122/f126 x:0 0 0 2024-03-23T11:47:58.605 INFO:tasks.workunit.client.0.smithi063.stdout:2/882: chown d2/d23/d2e/d52/d86/f119 1289265 1 2024-03-23T11:47:58.608 INFO:tasks.workunit.client.0.smithi063.stdout:2/883: mknod d2/d23/d2e/d98/c127 0 2024-03-23T11:47:58.611 INFO:tasks.workunit.client.0.smithi063.stdout:2/884: fsync d2/d23/d2e/d87/f80 0 2024-03-23T11:47:58.614 INFO:tasks.workunit.client.0.smithi063.stdout:2/885: chown d2/d23/d2e/d87/f82 0 1 2024-03-23T11:47:58.616 INFO:tasks.workunit.client.0.smithi063.stdout:2/886: chown d2/d23/d56/dad 0 1 2024-03-23T11:47:58.629 INFO:tasks.workunit.client.0.smithi063.stdout:2/887: dread d2/d1c/f96 [0,4194304] 0 2024-03-23T11:47:58.631 INFO:tasks.workunit.client.0.smithi063.stdout:2/888: dread - d2/d23/d2e/d98/faf zero size 2024-03-23T11:47:58.632 INFO:tasks.workunit.client.0.smithi063.stdout:7/912: dread d5/d26/db7/dc6/d4f/f93 [0,4194304] 0 2024-03-23T11:47:58.634 INFO:tasks.workunit.client.0.smithi063.stdout:7/913: write d5/d6/f120 [627191,45668] 0 2024-03-23T11:47:58.635 INFO:tasks.workunit.client.0.smithi063.stdout:2/889: truncate d2/d23/d56/ffc 409202 0 2024-03-23T11:47:58.638 INFO:tasks.workunit.client.0.smithi063.stdout:2/890: creat d2/d84/d92/f128 x:0 0 0 2024-03-23T11:47:58.639 INFO:tasks.workunit.client.0.smithi063.stdout:7/914: truncate d5/d5c/d2f/d8c/fae 4491701 0 2024-03-23T11:47:58.641 INFO:tasks.workunit.client.0.smithi063.stdout:7/915: symlink d5/d5c/d2f/d8b/l123 0 2024-03-23T11:47:58.644 INFO:tasks.workunit.client.0.smithi063.stdout:7/916: creat d5/d22/da3/f124 x:0 0 0 2024-03-23T11:47:58.647 INFO:tasks.workunit.client.0.smithi063.stdout:7/917: mknod d5/d26/db7/dce/d64/c125 0 2024-03-23T11:47:58.654 INFO:tasks.workunit.client.0.smithi063.stdout:2/891: dwrite d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:58.657 INFO:tasks.workunit.client.0.smithi063.stdout:2/892: write d2/f29 [367582,76127] 0 2024-03-23T11:47:58.662 INFO:tasks.workunit.client.0.smithi063.stdout:2/893: write d2/d23/d56/dad/dd4/fd9 [2744782,80990] 0 2024-03-23T11:47:58.667 INFO:tasks.workunit.client.0.smithi063.stdout:7/918: dwrite d5/d5c/d2f/f10f [0,4194304] 0 2024-03-23T11:47:58.670 INFO:tasks.workunit.client.0.smithi063.stdout:7/919: chown d5/d5c/d2f/d8b/daa 6658787 1 2024-03-23T11:47:58.675 INFO:tasks.workunit.client.0.smithi063.stdout:2/894: write d2/d23/d56/f57 [89347,102574] 0 2024-03-23T11:47:58.686 INFO:tasks.workunit.client.0.smithi063.stdout:2/895: write d2/de/d22/f4a [419865,79766] 0 2024-03-23T11:47:58.689 INFO:tasks.workunit.client.0.smithi063.stdout:2/896: write d2/d23/d2e/d98/f10f [783237,22191] 0 2024-03-23T11:47:58.693 INFO:tasks.workunit.client.0.smithi063.stdout:2/897: dread d2/d46/db4/f107 [0,4194304] 0 2024-03-23T11:47:58.695 INFO:tasks.workunit.client.0.smithi063.stdout:2/898: read d2/de/d7a/d8d/f9e [653224,64130] 0 2024-03-23T11:47:58.699 INFO:tasks.workunit.client.0.smithi063.stdout:2/899: mkdir d2/de/d22/dbc/d10c/d125/d129 0 2024-03-23T11:47:58.701 INFO:tasks.workunit.client.0.smithi063.stdout:2/900: dread - d2/de/d22/d91/ff7 zero size 2024-03-23T11:47:58.704 INFO:tasks.workunit.client.0.smithi063.stdout:2/901: mknod d2/d23/d2e/d52/d6c/c12a 0 2024-03-23T11:47:58.705 INFO:tasks.workunit.client.0.smithi063.stdout:2/902: fdatasync d2/de/d7a/d8d/f109 0 2024-03-23T11:47:58.709 INFO:tasks.workunit.client.0.smithi063.stdout:2/903: mknod d2/d23/d2e/d98/dc8/de9/c12b 0 2024-03-23T11:47:58.730 INFO:tasks.workunit.client.0.smithi063.stdout:2/904: dread d2/d84/f89 [0,4194304] 0 2024-03-23T11:47:58.739 INFO:tasks.workunit.client.0.smithi063.stdout:2/905: dread d2/d84/d94/fa6 [0,4194304] 0 2024-03-23T11:47:58.752 INFO:tasks.workunit.client.0.smithi063.stdout:2/906: dwrite d2/d23/d56/dad/dd4/f117 [0,4194304] 0 2024-03-23T11:47:58.770 INFO:tasks.workunit.client.0.smithi063.stdout:2/907: dwrite d2/d23/d2e/d52/d86/f119 [0,4194304] 0 2024-03-23T11:47:58.774 INFO:tasks.workunit.client.0.smithi063.stdout:2/908: dread - d2/d84/d92/d122/f126 zero size 2024-03-23T11:47:58.861 INFO:tasks.workunit.client.0.smithi063.stdout:2/909: dwrite d2/de/d7a/f114 [0,4194304] 0 2024-03-23T11:47:58.868 INFO:tasks.workunit.client.0.smithi063.stdout:2/910: mkdir d2/d23/d2e/d52/d86/dbf/d12c 0 2024-03-23T11:47:58.870 INFO:tasks.workunit.client.0.smithi063.stdout:2/911: fdatasync d2/d23/d56/ffc 0 2024-03-23T11:47:58.876 INFO:tasks.workunit.client.0.smithi063.stdout:2/912: creat d2/de/d22/d91/f12d x:0 0 0 2024-03-23T11:47:58.881 INFO:tasks.workunit.client.0.smithi063.stdout:2/913: creat d2/d46/d54/f12e x:0 0 0 2024-03-23T11:47:58.886 INFO:tasks.workunit.client.0.smithi063.stdout:2/914: stat d2/d23/d2e/d98/dc4/dcc/df5/l104 0 2024-03-23T11:47:58.943 INFO:tasks.workunit.client.0.smithi063.stdout:2/915: rename d2/d23/d2e/d52/fd0 to d2/d23/d2e/d98/f12f 0 2024-03-23T11:47:58.945 INFO:tasks.workunit.client.0.smithi063.stdout:2/916: write d2/d1c/ff8 [1454026,24377] 0 2024-03-23T11:47:58.954 INFO:tasks.workunit.client.0.smithi063.stdout:2/917: getdents d2/de/d22/d91 0 2024-03-23T11:47:58.956 INFO:tasks.workunit.client.0.smithi063.stdout:2/918: write d2/d23/d56/f57 [1239196,98169] 0 2024-03-23T11:47:58.958 INFO:tasks.workunit.client.0.smithi063.stdout:2/919: stat d2/d23/c11b 0 2024-03-23T11:47:58.972 INFO:tasks.workunit.client.0.smithi063.stdout:2/920: dread d2/d23/d56/f75 [0,4194304] 0 2024-03-23T11:47:58.980 INFO:tasks.workunit.client.0.smithi063.stdout:2/921: getdents d2/de 0 2024-03-23T11:47:58.985 INFO:tasks.workunit.client.0.smithi063.stdout:2/922: mknod d2/d23/d56/dad/dd4/dd6/c130 0 2024-03-23T11:47:58.993 INFO:tasks.workunit.client.0.smithi063.stdout:2/923: dread d2/d23/d56/fa4 [0,4194304] 0 2024-03-23T11:47:59.004 INFO:tasks.workunit.client.0.smithi063.stdout:4/876: sync 2024-03-23T11:47:59.007 INFO:tasks.workunit.client.0.smithi063.stdout:4/877: readlink d5/da/d20/l80 0 2024-03-23T11:47:59.009 INFO:tasks.workunit.client.0.smithi063.stdout:2/924: dwrite d2/d23/d56/dad/dd4/dd6/fdf [0,4194304] 0 2024-03-23T11:47:59.011 INFO:tasks.workunit.client.0.smithi063.stdout:4/878: creat d5/da/d20/d4b/d29/d38/d7a/dd5/f101 x:0 0 0 2024-03-23T11:47:59.013 INFO:tasks.workunit.client.0.smithi063.stdout:2/925: write d2/d23/d2e/d87/d4d/fcd [448313,86919] 0 2024-03-23T11:47:59.017 INFO:tasks.workunit.client.0.smithi063.stdout:4/879: read - d5/d1e/d2b/dbd/fef zero size 2024-03-23T11:47:59.018 INFO:tasks.workunit.client.0.smithi063.stdout:2/926: unlink d2/d23/d2e/d87/c5d 0 2024-03-23T11:47:59.022 INFO:tasks.workunit.client.0.smithi063.stdout:2/927: fdatasync d2/de/d22/f78 0 2024-03-23T11:47:59.026 INFO:tasks.workunit.client.0.smithi063.stdout:4/880: creat d5/da/d20/d4b/d16/d50/d8e/f102 x:0 0 0 2024-03-23T11:47:59.030 INFO:tasks.workunit.client.0.smithi063.stdout:4/881: chown d5/da/d20/d4b/d29/d38/f55 0 1 2024-03-23T11:47:59.030 INFO:tasks.workunit.client.0.smithi063.stdout:2/928: dread d2/fdc [0,4194304] 0 2024-03-23T11:47:59.036 INFO:tasks.workunit.client.0.smithi063.stdout:2/929: rmdir d2/d46/db4 39 2024-03-23T11:47:59.041 INFO:tasks.workunit.client.0.smithi063.stdout:2/930: getdents d2/de/d22 0 2024-03-23T11:47:59.046 INFO:tasks.workunit.client.0.smithi063.stdout:2/931: rename d2/d23/c11b to d2/d46/d54/dc2/d10d/c131 0 2024-03-23T11:47:59.048 INFO:tasks.workunit.client.0.smithi063.stdout:5/632: sync 2024-03-23T11:47:59.048 INFO:tasks.workunit.client.0.smithi063.stdout:7/920: sync 2024-03-23T11:47:59.048 INFO:tasks.workunit.client.0.smithi063.stdout:1/879: sync 2024-03-23T11:47:59.049 INFO:tasks.workunit.client.0.smithi063.stdout:6/804: sync 2024-03-23T11:47:59.050 INFO:tasks.workunit.client.0.smithi063.stdout:2/932: unlink d2/d23/d56/dad/dd4/dd6/l112 0 2024-03-23T11:47:59.056 INFO:tasks.workunit.client.0.smithi063.stdout:2/933: creat d2/d23/f132 x:0 0 0 2024-03-23T11:47:59.060 INFO:tasks.workunit.client.0.smithi063.stdout:5/633: mkdir d2/d7/d8/de/d35/da6/dd6 0 2024-03-23T11:47:59.066 INFO:tasks.workunit.client.0.smithi063.stdout:2/934: mkdir d2/d23/d2e/d52/d86/dbf/d12c/d133 0 2024-03-23T11:47:59.068 INFO:tasks.workunit.client.0.smithi063.stdout:2/935: stat d2/d84/d92/d122 0 2024-03-23T11:47:59.068 INFO:tasks.workunit.client.0.smithi063.stdout:1/880: write d3/d3f/d4d/dbc/de0/da8/faf [79706,121804] 0 2024-03-23T11:47:59.069 INFO:tasks.workunit.client.0.smithi063.stdout:6/805: truncate d1/d15/d4b/f6b 440745 0 2024-03-23T11:47:59.075 INFO:tasks.workunit.client.0.smithi063.stdout:6/806: creat d1/dc/d3a/d82/da2/f102 x:0 0 0 2024-03-23T11:47:59.076 INFO:tasks.workunit.client.0.smithi063.stdout:1/881: fsync d3/d49/d56/d65/d84/f8e 0 2024-03-23T11:47:59.082 INFO:tasks.workunit.client.0.smithi063.stdout:7/921: dread d5/d6/d19/d1f/ff9 [0,4194304] 0 2024-03-23T11:47:59.083 INFO:tasks.workunit.client.0.smithi063.stdout:5/634: dwrite d2/f6 [0,4194304] 0 2024-03-23T11:47:59.083 INFO:tasks.workunit.client.0.smithi063.stdout:2/936: getdents d2/d23/d2e/d98/dc4/dcc 0 2024-03-23T11:47:59.084 INFO:tasks.workunit.client.0.smithi063.stdout:1/882: mkdir d3/df/d2f/de4/df5/d11b 0 2024-03-23T11:47:59.086 INFO:tasks.workunit.client.0.smithi063.stdout:6/807: truncate d1/dc/d3a/f4f 1405733 0 2024-03-23T11:47:59.089 INFO:tasks.workunit.client.0.smithi063.stdout:7/922: mkdir d5/d6/d126 0 2024-03-23T11:47:59.089 INFO:tasks.workunit.client.0.smithi063.stdout:1/883: dread - d3/df/db8/dcb/ff7 zero size 2024-03-23T11:47:59.092 INFO:tasks.workunit.client.0.smithi063.stdout:1/884: write d3/f10 [11025657,25123] 0 2024-03-23T11:47:59.094 INFO:tasks.workunit.client.0.smithi063.stdout:2/937: creat d2/de/d22/dbc/d10c/f134 x:0 0 0 2024-03-23T11:47:59.094 INFO:tasks.workunit.client.0.smithi063.stdout:7/923: truncate d5/d6/d19/d65/fba 1381683 0 2024-03-23T11:47:59.096 INFO:tasks.workunit.client.0.smithi063.stdout:5/635: creat d2/d7/d8/de/d2a/d3b/fd7 x:0 0 0 2024-03-23T11:47:59.100 INFO:tasks.workunit.client.0.smithi063.stdout:1/885: read d3/d62/dad/fd5 [3389766,77136] 0 2024-03-23T11:47:59.101 INFO:tasks.workunit.client.0.smithi063.stdout:5/636: fsync d2/d17/f34 0 2024-03-23T11:47:59.103 INFO:tasks.workunit.client.0.smithi063.stdout:2/938: fdatasync d2/d23/d2e/d52/fda 0 2024-03-23T11:47:59.109 INFO:tasks.workunit.client.0.smithi063.stdout:7/924: rename d5/d26/db7/dd6/cfc to d5/d26/db7/dd6/c127 0 2024-03-23T11:47:59.113 INFO:tasks.workunit.client.0.smithi063.stdout:7/925: readlink d5/d26/db7/dd6/d88/l107 0 2024-03-23T11:47:59.115 INFO:tasks.workunit.client.0.smithi063.stdout:2/939: creat d2/d23/d2e/d52/d6c/f135 x:0 0 0 2024-03-23T11:47:59.117 INFO:tasks.workunit.client.0.smithi063.stdout:1/886: symlink d3/d49/d56/d65/d88/d92/l11c 0 2024-03-23T11:47:59.119 INFO:tasks.workunit.client.0.smithi063.stdout:5/637: symlink d2/d17/ld8 0 2024-03-23T11:47:59.121 INFO:tasks.workunit.client.0.smithi063.stdout:7/926: creat d5/d6/d19/d65/f128 x:0 0 0 2024-03-23T11:47:59.121 INFO:tasks.workunit.client.0.smithi063.stdout:2/940: rmdir d2/de/d22 39 2024-03-23T11:47:59.123 INFO:tasks.workunit.client.0.smithi063.stdout:6/808: dread d1/dc/d3a/d82/da2/f4a [0,4194304] 0 2024-03-23T11:47:59.125 INFO:tasks.workunit.client.0.smithi063.stdout:1/887: creat d3/d35/d40/dee/d110/f11d x:0 0 0 2024-03-23T11:47:59.129 INFO:tasks.workunit.client.0.smithi063.stdout:6/809: mknod d1/c103 0 2024-03-23T11:47:59.129 INFO:tasks.workunit.client.0.smithi063.stdout:7/927: mknod d5/d26/dde/d100/c129 0 2024-03-23T11:47:59.130 INFO:tasks.workunit.client.0.smithi063.stdout:1/888: creat d3/d49/df4/f11e x:0 0 0 2024-03-23T11:47:59.134 INFO:tasks.workunit.client.0.smithi063.stdout:2/941: dread d2/d23/d2e/d98/f10f [0,4194304] 0 2024-03-23T11:47:59.139 INFO:tasks.workunit.client.0.smithi063.stdout:1/889: rename d3/d35/d40/dee/dfa to d3/d3f/d4d/d11f 0 2024-03-23T11:47:59.145 INFO:tasks.workunit.client.0.smithi063.stdout:7/928: link d5/d5c/d2f/d73/da/f17 d5/d5c/d2f/d8b/daa/f12a 0 2024-03-23T11:47:59.147 INFO:tasks.workunit.client.0.smithi063.stdout:6/810: link d1/dc/c5a d1/dc/d3a/d82/da2/dd8/c104 0 2024-03-23T11:47:59.147 INFO:tasks.workunit.client.0.smithi063.stdout:7/929: fsync d5/fc7 0 2024-03-23T11:47:59.150 INFO:tasks.workunit.client.0.smithi063.stdout:1/890: mknod d3/d49/d56/d65/d88/d92/d103/c120 0 2024-03-23T11:47:59.151 INFO:tasks.workunit.client.0.smithi063.stdout:2/942: getdents d2/d84/d94 0 2024-03-23T11:47:59.152 INFO:tasks.workunit.client.0.smithi063.stdout:5/638: dwrite d2/d7/d8/de/d35/f60 [0,4194304] 0 2024-03-23T11:47:59.157 INFO:tasks.workunit.client.0.smithi063.stdout:6/811: mknod d1/dc/d3a/d57/d6d/c105 0 2024-03-23T11:47:59.160 INFO:tasks.workunit.client.0.smithi063.stdout:7/930: mknod d5/d6/c12b 0 2024-03-23T11:47:59.163 INFO:tasks.workunit.client.0.smithi063.stdout:2/943: write d2/d23/d2e/d98/dd2/ffa [127613,93938] 0 2024-03-23T11:47:59.163 INFO:tasks.workunit.client.0.smithi063.stdout:1/891: symlink d3/d49/d56/d65/l121 0 2024-03-23T11:47:59.166 INFO:tasks.workunit.client.0.smithi063.stdout:5/639: symlink d2/d7/d8/de/d2a/d3b/d43/d5c/dc2/ld9 0 2024-03-23T11:47:59.168 INFO:tasks.workunit.client.0.smithi063.stdout:5/640: chown d2/d7/d27 78899792 1 2024-03-23T11:47:59.170 INFO:tasks.workunit.client.0.smithi063.stdout:7/931: creat d5/d26/dff/dd3/f12c x:0 0 0 2024-03-23T11:47:59.172 INFO:tasks.workunit.client.0.smithi063.stdout:5/641: write d2/d17/fb8 [649187,126708] 0 2024-03-23T11:47:59.176 INFO:tasks.workunit.client.0.smithi063.stdout:7/932: readlink d5/d5c/d2f/d73/da/l11 0 2024-03-23T11:47:59.178 INFO:tasks.workunit.client.0.smithi063.stdout:2/944: dread d2/de/d7a/d77/f93 [0,4194304] 0 2024-03-23T11:47:59.183 INFO:tasks.workunit.client.0.smithi063.stdout:2/945: chown d2/d1c/c40 4718 1 2024-03-23T11:47:59.183 INFO:tasks.workunit.client.0.smithi063.stdout:6/812: dwrite d1/dc/f1f [0,4194304] 0 2024-03-23T11:47:59.185 INFO:tasks.workunit.client.0.smithi063.stdout:5/642: rename d2/d7/d2b/f97 to d2/d7/d2b/d8b/da7/dc6/fda 0 2024-03-23T11:47:59.186 INFO:tasks.workunit.client.0.smithi063.stdout:7/933: creat d5/d26/db7/dd6/d88/f12d x:0 0 0 2024-03-23T11:47:59.188 INFO:tasks.workunit.client.0.smithi063.stdout:1/892: write d3/d3f/d4d/dbc/de0/dce/ffd [1949188,87914] 0 2024-03-23T11:47:59.190 INFO:tasks.workunit.client.0.smithi063.stdout:5/643: chown d2/d7/d8/de/d35/da6/dd6 882 1 2024-03-23T11:47:59.192 INFO:tasks.workunit.client.0.smithi063.stdout:2/946: read d2/d46/d54/fff [303322,116942] 0 2024-03-23T11:47:59.199 INFO:tasks.workunit.client.0.smithi063.stdout:7/934: write d5/d5c/d2f/d8b/fd0 [364221,87612] 0 2024-03-23T11:47:59.207 INFO:tasks.workunit.client.0.smithi063.stdout:2/947: creat d2/de/d22/dbc/d10c/f136 x:0 0 0 2024-03-23T11:47:59.208 INFO:tasks.workunit.client.0.smithi063.stdout:5/644: truncate d2/d28/d3a/d46/d71/f78 171433 0 2024-03-23T11:47:59.211 INFO:tasks.workunit.client.0.smithi063.stdout:6/813: dread d1/f3b [0,4194304] 0 2024-03-23T11:47:59.214 INFO:tasks.workunit.client.0.smithi063.stdout:5/645: chown d2/d7/d8/d1d/f57 2003891 1 2024-03-23T11:47:59.215 INFO:tasks.workunit.client.0.smithi063.stdout:1/893: dwrite d3/d3f/d4d/d52/fab [0,4194304] 0 2024-03-23T11:47:59.216 INFO:tasks.workunit.client.0.smithi063.stdout:2/948: chown d2/d23/d2e/d52/d86/l9f 66 1 2024-03-23T11:47:59.218 INFO:tasks.workunit.client.0.smithi063.stdout:1/894: chown d3/d49/d56/d65/d88 491 1 2024-03-23T11:47:59.218 INFO:tasks.workunit.client.0.smithi063.stdout:6/814: dread - d1/dc/d3a/d82/d85/df4/ff2 zero size 2024-03-23T11:47:59.221 INFO:tasks.workunit.client.0.smithi063.stdout:2/949: mkdir d2/d23/d2e/d52/d86/d137 0 2024-03-23T11:47:59.224 INFO:tasks.workunit.client.0.smithi063.stdout:5/646: creat d2/d17/d4d/d6e/fdb x:0 0 0 2024-03-23T11:47:59.224 INFO:tasks.workunit.client.0.smithi063.stdout:6/815: creat d1/dc/d3a/d57/df7/f106 x:0 0 0 2024-03-23T11:47:59.230 INFO:tasks.workunit.client.0.smithi063.stdout:5/647: fsync d2/d28/d3a/fa5 0 2024-03-23T11:47:59.244 INFO:tasks.workunit.client.0.smithi063.stdout:2/950: dwrite d2/d1c/f30 [0,4194304] 0 2024-03-23T11:47:59.246 INFO:tasks.workunit.client.0.smithi063.stdout:7/935: dread d5/d26/db7/fcd [0,4194304] 0 2024-03-23T11:47:59.247 INFO:tasks.workunit.client.0.smithi063.stdout:6/816: write d1/dc/d3a/d82/fba [128306,30594] 0 2024-03-23T11:47:59.251 INFO:tasks.workunit.client.0.smithi063.stdout:6/817: readlink d1/dc/d21/l6c 0 2024-03-23T11:47:59.255 INFO:tasks.workunit.client.0.smithi063.stdout:6/818: write d1/d15/fa4 [3571275,67617] 0 2024-03-23T11:47:59.255 INFO:tasks.workunit.client.0.smithi063.stdout:1/895: dread d3/f7 [0,4194304] 0 2024-03-23T11:47:59.265 INFO:tasks.workunit.client.0.smithi063.stdout:5/648: dwrite d2/d28/d3a/d46/d71/d9f/fa1 [0,4194304] 0 2024-03-23T11:47:59.267 INFO:tasks.workunit.client.0.smithi063.stdout:1/896: write d3/d49/d56/d65/d84/f98 [2509576,97937] 0 2024-03-23T11:47:59.268 INFO:tasks.workunit.client.0.smithi063.stdout:2/951: dread d2/de/d7a/d8d/f109 [0,4194304] 0 2024-03-23T11:47:59.280 INFO:tasks.workunit.client.0.smithi063.stdout:7/936: dwrite d5/d26/dff/dd3/fdd [0,4194304] 0 2024-03-23T11:47:59.283 INFO:tasks.workunit.client.0.smithi063.stdout:7/937: fdatasync d5/d5c/d2f/f79 0 2024-03-23T11:47:59.289 INFO:tasks.workunit.client.0.smithi063.stdout:5/649: getdents d2/d28/d3a 0 2024-03-23T11:47:59.293 INFO:tasks.workunit.client.0.smithi063.stdout:5/650: chown d2/d7/d8/f13 13185 1 2024-03-23T11:47:59.293 INFO:tasks.workunit.client.0.smithi063.stdout:1/897: dread d3/d3f/d4d/dbc/de0/d4b/f8a [0,4194304] 0 2024-03-23T11:47:59.294 INFO:tasks.workunit.client.0.smithi063.stdout:6/819: dread d1/dc/d3a/d57/d6d/fa0 [0,4194304] 0 2024-03-23T11:47:59.295 INFO:tasks.workunit.client.0.smithi063.stdout:7/938: write d5/d26/db7/fcd [3261906,40050] 0 2024-03-23T11:47:59.296 INFO:tasks.workunit.client.0.smithi063.stdout:5/651: rmdir d2/d17 39 2024-03-23T11:47:59.300 INFO:tasks.workunit.client.0.smithi063.stdout:1/898: symlink d3/df/d2f/d9a/l122 0 2024-03-23T11:47:59.300 INFO:tasks.workunit.client.0.smithi063.stdout:6/820: creat d1/f107 x:0 0 0 2024-03-23T11:47:59.303 INFO:tasks.workunit.client.0.smithi063.stdout:7/939: getdents d5/d5c/d2f/d73/da/d11c 0 2024-03-23T11:47:59.303 INFO:tasks.workunit.client.0.smithi063.stdout:1/899: readlink d3/d3f/d4d/dbc/de0/d4b/l96 0 2024-03-23T11:47:59.307 INFO:tasks.workunit.client.0.smithi063.stdout:1/900: write d3/d49/d56/d65/f7c [1738920,55154] 0 2024-03-23T11:47:59.311 INFO:tasks.workunit.client.0.smithi063.stdout:6/821: link d1/dc/d3a/d57/f5c d1/dc/d3a/d82/da2/f108 0 2024-03-23T11:47:59.312 INFO:tasks.workunit.client.0.smithi063.stdout:6/822: stat d1/d15/f26 0 2024-03-23T11:47:59.315 INFO:tasks.workunit.client.0.smithi063.stdout:1/901: mkdir d3/d49/d56/d65/d88/d92/d103/d123 0 2024-03-23T11:47:59.319 INFO:tasks.workunit.client.0.smithi063.stdout:7/940: truncate d5/d26/db7/dce/dc8/dee/f82 1513140 0 2024-03-23T11:47:59.321 INFO:tasks.workunit.client.0.smithi063.stdout:1/902: unlink d3/d3f/d4d/dbc/de0/d4b/l80 0 2024-03-23T11:47:59.322 INFO:tasks.workunit.client.0.smithi063.stdout:5/652: dwrite d2/d7/d27/f94 [0,4194304] 0 2024-03-23T11:47:59.322 INFO:tasks.workunit.client.0.smithi063.stdout:7/941: rmdir d5/d26/d76 39 2024-03-23T11:47:59.324 INFO:tasks.workunit.client.0.smithi063.stdout:5/653: readlink d2/l36 0 2024-03-23T11:47:59.326 INFO:tasks.workunit.client.0.smithi063.stdout:1/903: write d3/d49/d56/d65/d84/f98 [1877429,95854] 0 2024-03-23T11:47:59.330 INFO:tasks.workunit.client.0.smithi063.stdout:7/942: truncate d5/d22/f94 2382227 0 2024-03-23T11:47:59.331 INFO:tasks.workunit.client.0.smithi063.stdout:5/654: mknod d2/d28/d3a/cdc 0 2024-03-23T11:47:59.332 INFO:tasks.workunit.client.0.smithi063.stdout:1/904: rmdir d3/d3f/d4d/d52/ddd 39 2024-03-23T11:47:59.334 INFO:tasks.workunit.client.0.smithi063.stdout:5/655: chown d2/d7/d8/de/c24 0 1 2024-03-23T11:47:59.340 INFO:tasks.workunit.client.0.smithi063.stdout:1/905: mkdir d3/d49/d56/d65/d102/d124 0 2024-03-23T11:47:59.344 INFO:tasks.workunit.client.0.smithi063.stdout:1/906: read d3/d3f/d4d/dbc/de0/d4b/fbb [3812053,44449] 0 2024-03-23T11:47:59.345 INFO:tasks.workunit.client.0.smithi063.stdout:5/656: symlink d2/d7/d64/ldd 0 2024-03-23T11:47:59.347 INFO:tasks.workunit.client.0.smithi063.stdout:7/943: truncate d5/d5c/d2f/d8c/fb0 462321 0 2024-03-23T11:47:59.348 INFO:tasks.workunit.client.0.smithi063.stdout:1/907: readlink d3/d3f/d4d/dbc/de0/lc9 0 2024-03-23T11:47:59.349 INFO:tasks.workunit.client.0.smithi063.stdout:7/944: chown d5/d5c/d2f/f10f 0 1 2024-03-23T11:47:59.352 INFO:tasks.workunit.client.0.smithi063.stdout:5/657: creat d2/d7/d8/de/d49/fde x:0 0 0 2024-03-23T11:47:59.355 INFO:tasks.workunit.client.0.smithi063.stdout:1/908: mknod d3/d3f/d4d/dbc/de0/d4b/dd3/c125 0 2024-03-23T11:47:59.356 INFO:tasks.workunit.client.0.smithi063.stdout:5/658: read d2/d7/d8/de/d49/db0/fcc [158931,91342] 0 2024-03-23T11:47:59.356 INFO:tasks.workunit.client.0.smithi063.stdout:7/945: symlink d5/d5c/d2f/d73/da/l12e 0 2024-03-23T11:47:59.357 INFO:tasks.workunit.client.0.smithi063.stdout:5/659: readlink d2/l36 0 2024-03-23T11:47:59.359 INFO:tasks.workunit.client.0.smithi063.stdout:1/909: unlink d3/df/d2f/l106 0 2024-03-23T11:47:59.360 INFO:tasks.workunit.client.0.smithi063.stdout:1/910: stat d3/d49/d56/d65/db6/dc5 0 2024-03-23T11:47:59.362 INFO:tasks.workunit.client.0.smithi063.stdout:5/660: write d2/fc [4733211,128168] 0 2024-03-23T11:47:59.364 INFO:tasks.workunit.client.0.smithi063.stdout:5/661: dread - d2/d28/d3a/f9b zero size 2024-03-23T11:47:59.366 INFO:tasks.workunit.client.0.smithi063.stdout:1/911: dread d3/d3f/d4d/dbc/de0/faa [0,4194304] 0 2024-03-23T11:47:59.369 INFO:tasks.workunit.client.0.smithi063.stdout:5/662: dread d2/d7/d8/de/d2a/d3b/d85/f9e [0,4194304] 0 2024-03-23T11:47:59.370 INFO:tasks.workunit.client.0.smithi063.stdout:1/912: creat d3/d3f/f126 x:0 0 0 2024-03-23T11:47:59.372 INFO:tasks.workunit.client.0.smithi063.stdout:1/913: readlink d3/d49/d56/d65/d84/da7/de3/l10c 0 2024-03-23T11:47:59.384 INFO:tasks.workunit.client.0.smithi063.stdout:1/914: link d3/fd0 d3/d3f/d4d/d52/ddd/f127 0 2024-03-23T11:47:59.388 INFO:tasks.workunit.client.0.smithi063.stdout:1/915: read d3/d3f/d4d/f95 [2772023,62980] 0 2024-03-23T11:47:59.393 INFO:tasks.workunit.client.0.smithi063.stdout:1/916: mkdir d3/df/d2f/d9a/dd6/d128 0 2024-03-23T11:47:59.395 INFO:tasks.workunit.client.0.smithi063.stdout:5/663: dwrite d2/d7/d2b/f39 [0,4194304] 0 2024-03-23T11:47:59.400 INFO:tasks.workunit.client.0.smithi063.stdout:5/664: chown d2/d7/d8/de/d2a/d3b/d43/d5c/f7d 4519280 1 2024-03-23T11:47:59.403 INFO:tasks.workunit.client.0.smithi063.stdout:5/665: mkdir d2/d28/ddf 0 2024-03-23T11:47:59.408 INFO:tasks.workunit.client.0.smithi063.stdout:5/666: readlink d2/d28/d3a/l56 0 2024-03-23T11:47:59.415 INFO:tasks.workunit.client.0.smithi063.stdout:5/667: write d2/d28/d3a/f8d [875238,3678] 0 2024-03-23T11:47:59.418 INFO:tasks.workunit.client.0.smithi063.stdout:5/668: rmdir d2/d28/d3a/d46/da4 39 2024-03-23T11:47:59.419 INFO:tasks.workunit.client.0.smithi063.stdout:1/917: dread d3/d3f/d4d/dbc/de0/d4b/f71 [0,4194304] 0 2024-03-23T11:47:59.423 INFO:tasks.workunit.client.0.smithi063.stdout:1/918: read d3/d49/d56/d65/d84/f98 [2981024,52699] 0 2024-03-23T11:47:59.425 INFO:tasks.workunit.client.0.smithi063.stdout:5/669: link d2/d7/d2b/d8b/da7/fd5 d2/d7/d2b/d8b/da7/fe0 0 2024-03-23T11:47:59.429 INFO:tasks.workunit.client.0.smithi063.stdout:1/919: getdents d3/d3f/d4d/d52 0 2024-03-23T11:47:59.433 INFO:tasks.workunit.client.0.smithi063.stdout:5/670: rename d2/d7/d8/de/d2a/d3b/d85/cb2 to d2/d7/d8/de/d2a/d3b/d43/d5c/ce1 0 2024-03-23T11:47:59.435 INFO:tasks.workunit.client.0.smithi063.stdout:1/920: creat d3/d49/d56/d65/f129 x:0 0 0 2024-03-23T11:47:59.437 INFO:tasks.workunit.client.0.smithi063.stdout:1/921: write d3/f10 [2013477,92664] 0 2024-03-23T11:47:59.438 INFO:tasks.workunit.client.0.smithi063.stdout:5/671: read d2/f2c [1422763,104024] 0 2024-03-23T11:47:59.440 INFO:tasks.workunit.client.0.smithi063.stdout:1/922: symlink d3/df/d7b/l12a 0 2024-03-23T11:47:59.444 INFO:tasks.workunit.client.0.smithi063.stdout:1/923: creat d3/df/d2f/d9a/de2/f12b x:0 0 0 2024-03-23T11:47:59.451 INFO:tasks.workunit.client.0.smithi063.stdout:1/924: rename d3/d3f/d4d/d11f to d3/d49/d56/d65/d88/d92/d103/d12c 0 2024-03-23T11:47:59.456 INFO:tasks.workunit.client.0.smithi063.stdout:1/925: rename d3/d35/d40/l72 to d3/d3f/d117/l12d 0 2024-03-23T11:47:59.458 INFO:tasks.workunit.client.0.smithi063.stdout:1/926: write d3/d49/df4/fe5 [1195308,44273] 0 2024-03-23T11:47:59.462 INFO:tasks.workunit.client.0.smithi063.stdout:1/927: mkdir d3/df/d2f/de4/df5/d11b/d12e 0 2024-03-23T11:47:59.465 INFO:tasks.workunit.client.0.smithi063.stdout:1/928: chown d3/d3f/d4d/dbc/de0/lc9 1598527 1 2024-03-23T11:47:59.465 INFO:tasks.workunit.client.0.smithi063.stdout:5/672: dread d2/d7/d8/d1d/f57 [0,4194304] 0 2024-03-23T11:47:59.469 INFO:tasks.workunit.client.0.smithi063.stdout:1/929: fdatasync d3/d3f/d4d/dbc/de0/d4b/da5/f116 0 2024-03-23T11:47:59.479 INFO:tasks.workunit.client.0.smithi063.stdout:1/930: truncate d3/d49/d56/f7f 213300 0 2024-03-23T11:47:59.483 INFO:tasks.workunit.client.0.smithi063.stdout:1/931: write d3/df/d2f/d9a/ff1 [1583155,49008] 0 2024-03-23T11:47:59.484 INFO:tasks.workunit.client.0.smithi063.stdout:5/673: dwrite d2/d7/f1a [0,4194304] 0 2024-03-23T11:47:59.489 INFO:tasks.workunit.client.0.smithi063.stdout:5/674: write d2/d28/d3a/fa5 [1307390,54767] 0 2024-03-23T11:47:59.490 INFO:tasks.workunit.client.0.smithi063.stdout:1/932: mknod d3/d3f/d4d/dbc/de0/d4b/c12f 0 2024-03-23T11:47:59.495 INFO:tasks.workunit.client.0.smithi063.stdout:5/675: mkdir d2/d17/d4d/d6e/de2 0 2024-03-23T11:47:59.507 INFO:tasks.workunit.client.0.smithi063.stdout:1/933: truncate d3/f10 4469112 0 2024-03-23T11:47:59.510 INFO:tasks.workunit.client.0.smithi063.stdout:1/934: mknod d3/d49/c130 0 2024-03-23T11:47:59.515 INFO:tasks.workunit.client.0.smithi063.stdout:1/935: rename d3/d3f/d4d/d52/fab to d3/d49/d56/d65/db6/f131 0 2024-03-23T11:47:59.521 INFO:tasks.workunit.client.0.smithi063.stdout:1/936: truncate d3/d49/d56/d65/d88/f9c 1372250 0 2024-03-23T11:47:59.531 INFO:tasks.workunit.client.0.smithi063.stdout:5/676: dread d2/d17/f51 [0,4194304] 0 2024-03-23T11:47:59.533 INFO:tasks.workunit.client.0.smithi063.stdout:5/677: chown d2/d7/d8/de/d49/lb5 506509981 1 2024-03-23T11:47:59.535 INFO:tasks.workunit.client.0.smithi063.stdout:5/678: rename d2/d7 to d2/d7/d8/d1d/de3 22 2024-03-23T11:47:59.540 INFO:tasks.workunit.client.0.smithi063.stdout:5/679: creat d2/d7/d8/de/d35/da6/dd6/fe4 x:0 0 0 2024-03-23T11:47:59.543 INFO:tasks.workunit.client.0.smithi063.stdout:5/680: write d2/d7/d8/de/d2a/d3b/fd7 [710684,107344] 0 2024-03-23T11:47:59.550 INFO:tasks.workunit.client.0.smithi063.stdout:5/681: link d2/d7/d8/d1d/c2e d2/d7/d8/de/d2a/d3b/d43/d5c/ce5 0 2024-03-23T11:47:59.553 INFO:tasks.workunit.client.0.smithi063.stdout:5/682: truncate d2/d28/d3a/d46/f86 5205784 0 2024-03-23T11:47:59.581 INFO:tasks.workunit.client.0.smithi063.stdout:5/683: dread d2/d7/d8/de/d2a/d3b/d43/d5c/f62 [0,4194304] 0 2024-03-23T11:47:59.584 INFO:tasks.workunit.client.0.smithi063.stdout:5/684: mkdir d2/d28/de6 0 2024-03-23T11:47:59.611 INFO:tasks.workunit.client.0.smithi063.stdout:4/882: sync 2024-03-23T11:47:59.614 INFO:tasks.workunit.client.0.smithi063.stdout:4/883: truncate d5/da/d20/d4b/d29/fe0 178382 0 2024-03-23T11:47:59.616 INFO:tasks.workunit.client.0.smithi063.stdout:4/884: chown d5/da/d20/d4b/d16/d50/d8e/d96/ddf/c9b 0 1 2024-03-23T11:47:59.618 INFO:tasks.workunit.client.0.smithi063.stdout:4/885: dread - d5/da/d20/d4b/d1f/fe6 zero size 2024-03-23T11:47:59.622 INFO:tasks.workunit.client.0.smithi063.stdout:4/886: write d5/da/d20/d4b/d1f/f21 [2350658,78223] 0 2024-03-23T11:47:59.625 INFO:tasks.workunit.client.0.smithi063.stdout:4/887: chown d5/da/l22 145 1 2024-03-23T11:47:59.631 INFO:tasks.workunit.client.0.smithi063.stdout:4/888: creat d5/da/d20/d4b/d16/d32/f103 x:0 0 0 2024-03-23T11:47:59.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:59 smithi063 ceph-mon[35189]: pgmap v49: 65 pgs: 65 active+clean; 3.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 90 MiB/s rd, 205 MiB/s wr, 511 op/s 2024-03-23T11:47:59.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:47:59 smithi063 ceph-mon[35189]: mgrmap e26: smithi111.nnblsk(active, since 92s), standbys: smithi063.gehkcr 2024-03-23T11:47:59.649 INFO:tasks.workunit.client.0.smithi063.stdout:4/889: dwrite d5/da/d20/d4b/d16/d50/f64 [0,4194304] 0 2024-03-23T11:47:59.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:59 smithi111 ceph-mon[44005]: pgmap v49: 65 pgs: 65 active+clean; 3.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 90 MiB/s rd, 205 MiB/s wr, 511 op/s 2024-03-23T11:47:59.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:47:59 smithi111 ceph-mon[44005]: mgrmap e26: smithi111.nnblsk(active, since 92s), standbys: smithi063.gehkcr 2024-03-23T11:47:59.872 INFO:tasks.workunit.client.0.smithi063.stdout:2/952: sync 2024-03-23T11:48:00.151 INFO:tasks.workunit.client.0.smithi063.stdout:6/823: sync 2024-03-23T11:48:00.151 INFO:tasks.workunit.client.0.smithi063.stdout:1/937: sync 2024-03-23T11:48:00.151 INFO:tasks.workunit.client.0.smithi063.stdout:7/946: sync 2024-03-23T11:48:00.157 INFO:tasks.workunit.client.0.smithi063.stdout:6/824: symlink d1/dc/d3a/d82/d85/df4/l109 0 2024-03-23T11:48:00.162 INFO:tasks.workunit.client.0.smithi063.stdout:6/825: unlink d1/dc/d3a/d82/da2/f83 0 2024-03-23T11:48:00.164 INFO:tasks.workunit.client.0.smithi063.stdout:6/826: stat d1/ff5 0 2024-03-23T11:48:00.171 INFO:tasks.workunit.client.0.smithi063.stdout:7/947: dwrite d5/d26/db7/dce/dc8/dee/f35 [0,4194304] 0 2024-03-23T11:48:00.173 INFO:tasks.workunit.client.0.smithi063.stdout:1/938: dwrite d3/d3f/d4d/dbc/de0/ffe [0,4194304] 0 2024-03-23T11:48:00.176 INFO:tasks.workunit.client.0.smithi063.stdout:1/939: write d3/df/d2f/fcc [1486749,111795] 0 2024-03-23T11:48:00.180 INFO:tasks.workunit.client.0.smithi063.stdout:7/948: mkdir d5/d6/d19/d12f 0 2024-03-23T11:48:00.187 INFO:tasks.workunit.client.0.smithi063.stdout:1/940: link d3/l2e d3/d35/d40/dee/d110/l132 0 2024-03-23T11:48:00.189 INFO:tasks.workunit.client.0.smithi063.stdout:6/827: dwrite d1/dc/d3a/d57/d77/fc9 [0,4194304] 0 2024-03-23T11:48:00.191 INFO:tasks.workunit.client.0.smithi063.stdout:1/941: chown d3/d3f/d4d/dbc/de0/d4b/f8b 505644548 1 2024-03-23T11:48:00.191 INFO:tasks.workunit.client.0.smithi063.stdout:7/949: rename d5/d5c/d2f/d73/da/lfe to d5/d26/db7/dce/l130 0 2024-03-23T11:48:00.193 INFO:tasks.workunit.client.0.smithi063.stdout:6/828: chown d1/f12 7372229 1 2024-03-23T11:48:00.198 INFO:tasks.workunit.client.0.smithi063.stdout:7/950: mknod d5/d5c/d2f/d73/da/c131 0 2024-03-23T11:48:00.201 INFO:tasks.workunit.client.0.smithi063.stdout:1/942: mknod d3/d49/d56/d65/d102/d124/c133 0 2024-03-23T11:48:00.210 INFO:tasks.workunit.client.0.smithi063.stdout:7/951: symlink d5/d26/dff/l132 0 2024-03-23T11:48:00.211 INFO:tasks.workunit.client.0.smithi063.stdout:6/829: mknod d1/dc/d3a/d57/d6d/ddb/c10a 0 2024-03-23T11:48:00.213 INFO:tasks.workunit.client.0.smithi063.stdout:6/830: readlink d1/l36 0 2024-03-23T11:48:00.215 INFO:tasks.workunit.client.0.smithi063.stdout:7/952: creat d5/d26/db7/dc6/f133 x:0 0 0 2024-03-23T11:48:00.216 INFO:tasks.workunit.client.0.smithi063.stdout:7/953: readlink d5/d26/db7/dc6/lbd 0 2024-03-23T11:48:00.227 INFO:tasks.workunit.client.0.smithi063.stdout:7/954: write d5/d5c/d2f/d73/f8d [437086,111149] 0 2024-03-23T11:48:00.231 INFO:tasks.workunit.client.0.smithi063.stdout:6/831: dwrite d1/f6e [0,4194304] 0 2024-03-23T11:48:00.231 INFO:tasks.workunit.client.0.smithi063.stdout:7/955: read d5/d5c/d2f/d8b/f10d [383580,69362] 0 2024-03-23T11:48:00.233 INFO:tasks.workunit.client.0.smithi063.stdout:6/832: chown d1/dc/d3a/f30 1974 1 2024-03-23T11:48:00.234 INFO:tasks.workunit.client.0.smithi063.stdout:6/833: chown d1/dc/d3a/d57/df7/f106 68273045 1 2024-03-23T11:48:00.238 INFO:tasks.workunit.client.0.smithi063.stdout:6/834: creat d1/dc/d3a/d82/da2/f10b x:0 0 0 2024-03-23T11:48:00.244 INFO:tasks.workunit.client.0.smithi063.stdout:6/835: symlink d1/dc/d3a/d82/d85/l10c 0 2024-03-23T11:48:00.247 INFO:tasks.workunit.client.0.smithi063.stdout:6/836: truncate d1/dc/f91 1045902 0 2024-03-23T11:48:00.250 INFO:tasks.workunit.client.0.smithi063.stdout:7/956: dwrite d5/d5c/d2f/d8b/fd0 [0,4194304] 0 2024-03-23T11:48:00.254 INFO:tasks.workunit.client.0.smithi063.stdout:7/957: symlink d5/d26/dde/d100/l134 0 2024-03-23T11:48:00.255 INFO:tasks.workunit.client.0.smithi063.stdout:6/837: link d1/d13/c22 d1/dc/d3a/d57/df7/c10d 0 2024-03-23T11:48:00.260 INFO:tasks.workunit.client.0.smithi063.stdout:6/838: write d1/dc/d3a/d82/da2/d41/d42/f9d [374856,92735] 0 2024-03-23T11:48:00.264 INFO:tasks.workunit.client.0.smithi063.stdout:6/839: mkdir d1/dc/d3a/d57/d6d/ddb/d10e 0 2024-03-23T11:48:00.268 INFO:tasks.workunit.client.0.smithi063.stdout:6/840: mknod d1/dc/d3a/d82/da2/d41/c10f 0 2024-03-23T11:48:00.270 INFO:tasks.workunit.client.0.smithi063.stdout:2/953: sync 2024-03-23T11:48:00.270 INFO:tasks.workunit.client.0.smithi063.stdout:4/890: sync 2024-03-23T11:48:00.271 INFO:tasks.workunit.client.0.smithi063.stdout:5/685: sync 2024-03-23T11:48:00.272 INFO:tasks.workunit.client.0.smithi063.stdout:2/954: chown d2/d84/d94/lb7 11 1 2024-03-23T11:48:00.273 INFO:tasks.workunit.client.0.smithi063.stdout:6/841: unlink d1/dc/d3a/d82/d85/lbe 0 2024-03-23T11:48:00.274 INFO:tasks.workunit.client.0.smithi063.stdout:4/891: chown d5/da/d20/d4b/d29/d38/cf9 2225314 1 2024-03-23T11:48:00.276 INFO:tasks.workunit.client.0.smithi063.stdout:2/955: mknod d2/d84/c138 0 2024-03-23T11:48:00.278 INFO:tasks.workunit.client.0.smithi063.stdout:2/956: chown d2/f2f 16061 1 2024-03-23T11:48:00.281 INFO:tasks.workunit.client.0.smithi063.stdout:4/892: mknod d5/da/d20/d4b/d1f/df3/c104 0 2024-03-23T11:48:00.284 INFO:tasks.workunit.client.0.smithi063.stdout:2/957: chown d2/d1c/f5f 2877278 1 2024-03-23T11:48:00.290 INFO:tasks.workunit.client.0.smithi063.stdout:6/842: getdents d1/dc/d3a 0 2024-03-23T11:48:00.293 INFO:tasks.workunit.client.0.smithi063.stdout:4/893: mkdir d5/d1e/d105 0 2024-03-23T11:48:00.294 INFO:tasks.workunit.client.0.smithi063.stdout:2/958: unlink d2/d23/d2e/d52/c3f 0 2024-03-23T11:48:00.297 INFO:tasks.workunit.client.0.smithi063.stdout:6/843: mkdir d1/dc/d3a/d82/d110 0 2024-03-23T11:48:00.300 INFO:tasks.workunit.client.0.smithi063.stdout:4/894: mknod d5/da/d20/d4b/d1f/c106 0 2024-03-23T11:48:00.300 INFO:tasks.workunit.client.0.smithi063.stdout:6/844: write d1/f9e [3614573,92514] 0 2024-03-23T11:48:00.305 INFO:tasks.workunit.client.0.smithi063.stdout:2/959: link d2/d23/d2e/f35 d2/d23/d2e/d52/f139 0 2024-03-23T11:48:00.307 INFO:tasks.workunit.client.0.smithi063.stdout:5/686: dwrite d2/d2f/f37 [0,4194304] 0 2024-03-23T11:48:00.309 INFO:tasks.workunit.client.0.smithi063.stdout:2/960: chown d2/d23/d2e/d87/d4d/fcd 387878670 1 2024-03-23T11:48:00.309 INFO:tasks.workunit.client.0.smithi063.stdout:6/845: getdents d1/dc/d3a/d82/d110 0 2024-03-23T11:48:00.310 INFO:tasks.workunit.client.0.smithi063.stdout:4/895: creat d5/da/d20/d4b/d16/d50/d8e/d96/ddf/f107 x:0 0 0 2024-03-23T11:48:00.311 INFO:tasks.workunit.client.0.smithi063.stdout:5/687: read d2/d7/d8/de/d35/f60 [3952213,28017] 0 2024-03-23T11:48:00.314 INFO:tasks.workunit.client.0.smithi063.stdout:4/896: chown d5/da/d20/d4b/d29/fe0 176 1 2024-03-23T11:48:00.316 INFO:tasks.workunit.client.0.smithi063.stdout:6/846: creat d1/dc/d21/f111 x:0 0 0 2024-03-23T11:48:00.317 INFO:tasks.workunit.client.0.smithi063.stdout:2/961: symlink d2/d23/d2e/d98/dc8/de9/l13a 0 2024-03-23T11:48:00.320 INFO:tasks.workunit.client.0.smithi063.stdout:6/847: chown d1/dc/d3a/d82/da2/l7e 10 1 2024-03-23T11:48:00.322 INFO:tasks.workunit.client.0.smithi063.stdout:6/848: stat d1/dc/d3a/f8f 0 2024-03-23T11:48:00.323 INFO:tasks.workunit.client.0.smithi063.stdout:2/962: mknod d2/d23/d2e/d98/dc8/de9/c13b 0 2024-03-23T11:48:00.324 INFO:tasks.workunit.client.0.smithi063.stdout:4/897: symlink d5/da/d20/d4b/d29/d38/d7a/dd5/de3/l108 0 2024-03-23T11:48:00.327 INFO:tasks.workunit.client.0.smithi063.stdout:6/849: creat d1/d15/f112 x:0 0 0 2024-03-23T11:48:00.327 INFO:tasks.workunit.client.0.smithi063.stdout:5/688: dread d2/d7/d8/de/d2a/d3b/d43/f5a [0,4194304] 0 2024-03-23T11:48:00.329 INFO:tasks.workunit.client.0.smithi063.stdout:6/850: readlink d1/dc/d3a/d57/d6d/l9c 0 2024-03-23T11:48:00.330 INFO:tasks.workunit.client.0.smithi063.stdout:4/898: symlink d5/da/d20/d4b/d16/d50/d8e/l109 0 2024-03-23T11:48:00.333 INFO:tasks.workunit.client.0.smithi063.stdout:6/851: unlink d1/f6e 0 2024-03-23T11:48:00.334 INFO:tasks.workunit.client.0.smithi063.stdout:6/852: chown d1/dc/d3a/f40 1842295 1 2024-03-23T11:48:00.335 INFO:tasks.workunit.client.0.smithi063.stdout:4/899: unlink d5/d1e/d2b/c87 0 2024-03-23T11:48:00.337 INFO:tasks.workunit.client.0.smithi063.stdout:5/689: getdents d2/d17/d4d/d6e 0 2024-03-23T11:48:00.340 INFO:tasks.workunit.client.0.smithi063.stdout:4/900: unlink l3 0 2024-03-23T11:48:00.342 INFO:tasks.workunit.client.0.smithi063.stdout:6/853: creat d1/dc/d3a/d57/d6d/ddb/dfd/f113 x:0 0 0 2024-03-23T11:48:00.343 INFO:tasks.workunit.client.0.smithi063.stdout:5/690: creat d2/d7/d8/de/d47/fe7 x:0 0 0 2024-03-23T11:48:00.343 INFO:tasks.workunit.client.0.smithi063.stdout:6/854: readlink d1/d15/d4b/lbf 0 2024-03-23T11:48:00.344 INFO:tasks.workunit.client.0.smithi063.stdout:6/855: stat d1/dc/d3a/f45 0 2024-03-23T11:48:00.346 INFO:tasks.workunit.client.0.smithi063.stdout:4/901: creat d5/da/d20/d4b/d16/d50/d8e/d96/f10a x:0 0 0 2024-03-23T11:48:00.359 INFO:tasks.workunit.client.0.smithi063.stdout:5/691: write d2/d7/d8/f68 [1143970,112188] 0 2024-03-23T11:48:00.360 INFO:tasks.workunit.client.0.smithi063.stdout:4/902: write d5/da/d20/d4b/d16/d50/d8e/dd3/f7f [496152,9289] 0 2024-03-23T11:48:00.361 INFO:tasks.workunit.client.0.smithi063.stdout:6/856: truncate d1/dc/d21/fa9 1671008 0 2024-03-23T11:48:00.364 INFO:tasks.workunit.client.0.smithi063.stdout:4/903: write d5/da/d20/f48 [3630963,109014] 0 2024-03-23T11:48:00.367 INFO:tasks.workunit.client.0.smithi063.stdout:4/904: chown d5/da/d20/d4b/d29/d38/d7a/dd5/f101 25 1 2024-03-23T11:48:00.380 INFO:tasks.workunit.client.0.smithi063.stdout:4/905: dwrite d5/ff [4194304,4194304] 0 2024-03-23T11:48:00.383 INFO:tasks.workunit.client.0.smithi063.stdout:4/906: creat d5/da/d20/d4b/d16/d32/f10b x:0 0 0 2024-03-23T11:48:00.386 INFO:tasks.workunit.client.0.smithi063.stdout:4/907: mknod d5/da/d7e/dbf/c10c 0 2024-03-23T11:48:00.389 INFO:tasks.workunit.client.0.smithi063.stdout:4/908: creat d5/da/d20/d4b/d16/d50/d8e/dad/f10d x:0 0 0 2024-03-23T11:48:00.405 INFO:tasks.workunit.client.0.smithi063.stdout:4/909: dread d5/da/d20/d4b/d16/f5c [0,4194304] 0 2024-03-23T11:48:00.407 INFO:tasks.workunit.client.0.smithi063.stdout:4/910: fdatasync d5/da/d20/d4b/d16/d50/d8e/dad/f10d 0 2024-03-23T11:48:00.410 INFO:tasks.workunit.client.0.smithi063.stdout:4/911: write d5/da/d20/d4b/d29/d38/f55 [4140071,31628] 0 2024-03-23T11:48:00.414 INFO:tasks.workunit.client.0.smithi063.stdout:4/912: chown d5/da/d20/d4b/d16/d50/d8e/d96/ddf 767 1 2024-03-23T11:48:00.419 INFO:tasks.workunit.client.0.smithi063.stdout:4/913: symlink d5/da/d20/d4b/d16/d50/d8e/d96/dca/l10e 0 2024-03-23T11:48:00.421 INFO:tasks.workunit.client.0.smithi063.stdout:4/914: fsync d5/da/d20/d4b/d1f/f41 0 2024-03-23T11:48:00.429 INFO:tasks.workunit.client.0.smithi063.stdout:4/915: write d5/fc [568867,104715] 0 2024-03-23T11:48:00.446 INFO:tasks.workunit.client.0.smithi063.stdout:4/916: dwrite d5/da/d20/d4b/f42 [0,4194304] 0 2024-03-23T11:48:00.464 INFO:tasks.workunit.client.0.smithi063.stdout:4/917: dwrite d5/da/d20/d4b/d1f/f21 [4194304,4194304] 0 2024-03-23T11:48:00.482 INFO:tasks.workunit.client.0.smithi063.stdout:4/918: dwrite d5/da/d20/d4b/d16/dc4/ff6 [0,4194304] 0 2024-03-23T11:48:00.500 INFO:tasks.workunit.client.0.smithi063.stdout:4/919: dwrite d5/da/dc3/ffe [0,4194304] 0 2024-03-23T11:48:00.502 INFO:tasks.workunit.client.0.smithi063.stdout:4/920: write d5/fc [1803212,116847] 0 2024-03-23T11:48:00.507 INFO:tasks.workunit.client.0.smithi063.stdout:4/921: chown d5/da/d20/d4b/d16/d32/d56/ffa 8 1 2024-03-23T11:48:00.511 INFO:tasks.workunit.client.0.smithi063.stdout:4/922: symlink d5/da/d7e/dbf/l10f 0 2024-03-23T11:48:00.517 INFO:tasks.workunit.client.0.smithi063.stdout:4/923: rename d5/da/d20/d4b/d16/d32/d56/l71 to d5/da/d20/d4b/d16/d32/d56/l110 0 2024-03-23T11:48:00.523 INFO:tasks.workunit.client.0.smithi063.stdout:4/924: rename d5/da/d20/d4b/d16/d32/d56/dc7 to d5/da/d20/d4b/d16/d50/d8e/d96/dc0/d111 0 2024-03-23T11:48:00.525 INFO:tasks.workunit.client.0.smithi063.stdout:4/925: mknod d5/d1e/d105/c112 0 2024-03-23T11:48:00.532 INFO:tasks.workunit.client.0.smithi063.stdout:4/926: write d5/da/d20/d4b/f5a [637033,85687] 0 2024-03-23T11:48:00.534 INFO:tasks.workunit.client.0.smithi063.stdout:4/927: write d5/da/d20/d4b/d1f/f41 [2961697,80319] 0 2024-03-23T11:48:00.538 INFO:tasks.workunit.client.0.smithi063.stdout:4/928: rename d5/da/d20/d4b/c15 to d5/da/d20/d4b/d16/dc4/c113 0 2024-03-23T11:48:00.541 INFO:tasks.workunit.client.0.smithi063.stdout:4/929: symlink d5/da/d20/d4b/d16/d50/d86/da2/l114 0 2024-03-23T11:48:00.544 INFO:tasks.workunit.client.0.smithi063.stdout:4/930: symlink d5/da/d20/d4b/d16/d50/d8e/dad/ddd/l115 0 2024-03-23T11:48:00.547 INFO:tasks.workunit.client.0.smithi063.stdout:4/931: read - d5/da/d20/d4b/d16/d50/d8e/dd3/fc9 zero size 2024-03-23T11:48:00.549 INFO:tasks.workunit.client.0.smithi063.stdout:4/932: creat d5/d1e/d2b/dbd/f116 x:0 0 0 2024-03-23T11:48:00.554 INFO:tasks.workunit.client.0.smithi063.stdout:4/933: mknod d5/da/d20/d4b/d16/d50/d8e/d96/dc0/d111/c117 0 2024-03-23T11:48:00.559 INFO:tasks.workunit.client.0.smithi063.stdout:4/934: write d5/da/d20/d4b/d16/d50/d8e/d96/ddf/fb1 [5616714,130767] 0 2024-03-23T11:48:00.571 INFO:tasks.workunit.client.0.smithi063.stdout:4/935: dread d5/da/d20/d4b/f8b [0,4194304] 0 2024-03-23T11:48:00.619 INFO:tasks.workunit.client.0.smithi063.stdout:4/936: dread d5/da/d20/d4b/d16/d50/d8e/d96/ddf/f88 [0,4194304] 0 2024-03-23T11:48:00.621 INFO:tasks.workunit.client.0.smithi063.stdout:4/937: rename d5 to d5/da/d20/d4b/d16/d50/d86/d118 22 2024-03-23T11:48:00.623 INFO:tasks.workunit.client.0.smithi063.stdout:1/943: sync 2024-03-23T11:48:00.625 INFO:tasks.workunit.client.0.smithi063.stdout:4/938: creat d5/da/d20/d4b/d16/dc4/f119 x:0 0 0 2024-03-23T11:48:00.628 INFO:tasks.workunit.client.0.smithi063.stdout:4/939: mkdir d5/da/d11a 0 2024-03-23T11:48:00.660 INFO:tasks.workunit.client.0.smithi063.stdout:4/940: write d5/da/d20/d4b/d16/d32/d56/fbb [259800,59922] 0 2024-03-23T11:48:00.674 INFO:tasks.workunit.client.0.smithi063.stdout:4/941: dwrite d5/da/d20/d4b/d16/d37/ffc [0,4194304] 0 2024-03-23T11:48:00.680 INFO:tasks.workunit.client.0.smithi063.stdout:4/942: rename d5/da/d20/d4b/d16/fa3 to d5/f11b 0 2024-03-23T11:48:00.684 INFO:tasks.workunit.client.0.smithi063.stdout:4/943: creat d5/da/d20/d4b/d16/d37/f11c x:0 0 0 2024-03-23T11:48:00.688 INFO:tasks.workunit.client.0.smithi063.stdout:4/944: write d5/da/d20/d4b/d16/d50/d8e/d96/ddf/f92 [1383741,119222] 0 2024-03-23T11:48:00.701 INFO:tasks.workunit.client.0.smithi063.stdout:4/945: dread d5/da/d20/d4b/f42 [0,4194304] 0 2024-03-23T11:48:00.711 INFO:tasks.workunit.client.0.smithi063.stdout:4/946: truncate d5/da/d20/d4b/fa9 2572699 0 2024-03-23T11:48:00.714 INFO:tasks.workunit.client.0.smithi063.stdout:4/947: readlink d5/da/d20/d4b/d16/d32/d56/l9f 0 2024-03-23T11:48:00.716 INFO:tasks.workunit.client.0.smithi063.stdout:4/948: readlink d5/da/d20/d4b/d29/d38/lf8 0 2024-03-23T11:48:00.722 INFO:tasks.workunit.client.0.smithi063.stdout:4/949: rename d5/da/d20/d4b/d29/d38/d7a/d8f/lac to d5/da/d20/d4b/d16/d50/d86/da2/l11d 0 2024-03-23T11:48:00.724 INFO:tasks.workunit.client.0.smithi063.stdout:4/950: stat d5/da/d20/d4b/d16/d50/d8e/l109 0 2024-03-23T11:48:00.741 INFO:tasks.workunit.client.0.smithi063.stdout:4/951: dwrite d5/da/d20/d4b/f8b [0,4194304] 0 2024-03-23T11:48:00.744 INFO:tasks.workunit.client.0.smithi063.stdout:4/952: creat d5/da/dc3/f11e x:0 0 0 2024-03-23T11:48:00.750 INFO:tasks.workunit.client.0.smithi063.stdout:4/953: symlink d5/d1e/d2b/dbd/l11f 0 2024-03-23T11:48:00.755 INFO:tasks.workunit.client.0.smithi063.stdout:4/954: creat d5/da/d20/d4b/d16/d50/d8e/dd3/d94/f120 x:0 0 0 2024-03-23T11:48:00.762 INFO:tasks.workunit.client.0.smithi063.stdout:4/955: write d5/d1e/d2b/f54 [339287,93966] 0 2024-03-23T11:48:00.764 INFO:tasks.workunit.client.0.smithi063.stdout:4/956: chown d5/da/d7e/cfb 857796 1 2024-03-23T11:48:00.766 INFO:tasks.workunit.client.0.smithi063.stdout:4/957: write d5/d1e/d2b/f54 [1046115,47464] 0 2024-03-23T11:48:00.773 INFO:tasks.workunit.client.0.smithi063.stdout:4/958: truncate d5/fc 4108744 0 2024-03-23T11:48:00.782 INFO:tasks.workunit.client.0.smithi063.stdout:4/959: rename d5/da/d20/d4b/d16/d50/d8e/d96/dc0/d111/c117 to d5/da/d20/d4b/d29/c121 0 2024-03-23T11:48:00.786 INFO:tasks.workunit.client.0.smithi063.stdout:4/960: write d5/da/d20/d4b/d16/d50/d86/da2/fe8 [3955410,40150] 0 2024-03-23T11:48:00.789 INFO:tasks.workunit.client.0.smithi063.stdout:4/961: truncate d5/da/d20/d4b/d16/d50/fe5 839618 0 2024-03-23T11:48:00.792 INFO:tasks.workunit.client.0.smithi063.stdout:4/962: rmdir d5/da/d20/d4b/d16/d50/d86/da2 39 2024-03-23T11:48:00.795 INFO:tasks.workunit.client.0.smithi063.stdout:4/963: creat d5/da/d20/d4b/d16/d32/d56/f122 x:0 0 0 2024-03-23T11:48:00.813 INFO:tasks.workunit.client.0.smithi063.stdout:4/964: dread d5/da/d20/d4b/d16/d50/d86/da2/fe8 [0,4194304] 0 2024-03-23T11:48:00.818 INFO:tasks.workunit.client.0.smithi063.stdout:4/965: rename d5/d1e/d2b/dbd/fef to d5/d1e/d2b/dbd/f123 0 2024-03-23T11:48:00.822 INFO:tasks.workunit.client.0.smithi063.stdout:4/966: truncate d5/da/d20/d4b/d16/d32/f10b 378359 0 2024-03-23T11:48:00.827 INFO:tasks.workunit.client.0.smithi063.stdout:7/958: sync 2024-03-23T11:48:00.830 INFO:tasks.workunit.client.0.smithi063.stdout:7/959: mkdir d5/d6/d19/d65/d135 0 2024-03-23T11:48:00.836 INFO:tasks.workunit.client.0.smithi063.stdout:7/960: rename d5/d26/db7/dd6/d88/dc2 to d5/d5c/d2f/d136 0 2024-03-23T11:48:00.837 INFO:tasks.workunit.client.0.smithi063.stdout:4/967: dwrite d5/da/d20/d4b/f5a [4194304,4194304] 0 2024-03-23T11:48:00.837 INFO:tasks.workunit.client.0.smithi063.stdout:7/961: chown d5/d6/d19/f7d 12278 1 2024-03-23T11:48:00.838 INFO:tasks.workunit.client.0.smithi063.stdout:4/968: fsync d5/da/dc3/ffe 0 2024-03-23T11:48:00.843 INFO:tasks.workunit.client.0.smithi063.stdout:4/969: symlink d5/da/l124 0 2024-03-23T11:48:00.846 INFO:tasks.workunit.client.0.smithi063.stdout:4/970: readlink d5/da/d20/d4b/d16/d37/l7b 0 2024-03-23T11:48:00.848 INFO:tasks.workunit.client.0.smithi063.stdout:4/971: chown d5/d1e/d2b/daa 20 1 2024-03-23T11:48:00.849 INFO:tasks.workunit.client.0.smithi063.stdout:2/963: sync 2024-03-23T11:48:00.849 INFO:tasks.workunit.client.0.smithi063.stdout:1/944: sync 2024-03-23T11:48:00.849 INFO:tasks.workunit.client.0.smithi063.stdout:5/692: sync 2024-03-23T11:48:00.849 INFO:tasks.workunit.client.0.smithi063.stdout:6/857: sync 2024-03-23T11:48:00.860 INFO:tasks.workunit.client.0.smithi063.stdout:1/945: chown d3/d3f/d4d/dbc/de0/d4b/da5/f116 841 1 2024-03-23T11:48:00.866 INFO:tasks.workunit.client.0.smithi063.stdout:1/946: mkdir d3/d35/d40/d134 0 2024-03-23T11:48:00.867 INFO:tasks.workunit.client.0.smithi063.stdout:5/693: unlink d2/d17/fb8 0 2024-03-23T11:48:00.868 INFO:tasks.workunit.client.0.smithi063.stdout:5/694: fdatasync d2/d7/d8/de/d2a/d3b/fd7 0 2024-03-23T11:48:00.869 INFO:tasks.workunit.client.0.smithi063.stdout:2/964: mkdir d2/d23/d2e/d98/dc4/dcc/df5/d13c 0 2024-03-23T11:48:00.872 INFO:tasks.workunit.client.0.smithi063.stdout:6/858: rename d1/dc/d3a/d82/da2/d41/l59 to d1/dc/d3a/d57/d6d/ddb/dfd/l114 0 2024-03-23T11:48:00.876 INFO:tasks.workunit.client.0.smithi063.stdout:2/965: chown d2/f2f 29738788 1 2024-03-23T11:48:00.876 INFO:tasks.workunit.client.0.smithi063.stdout:1/947: read d3/d3f/f77 [2370956,43412] 0 2024-03-23T11:48:00.880 INFO:tasks.workunit.client.0.smithi063.stdout:2/966: chown d2/de/l10 76607 1 2024-03-23T11:48:00.882 INFO:tasks.workunit.client.0.smithi063.stdout:1/948: readlink d3/d49/d56/d65/d88/d92/lb1 0 2024-03-23T11:48:00.882 INFO:tasks.workunit.client.0.smithi063.stdout:4/972: dwrite d5/da/dc3/fcb [0,4194304] 0 2024-03-23T11:48:00.884 INFO:tasks.workunit.client.0.smithi063.stdout:5/695: dread d2/d7/d8/d1d/fcb [0,4194304] 0 2024-03-23T11:48:00.886 INFO:tasks.workunit.client.0.smithi063.stdout:2/967: mkdir d2/d46/d13d 0 2024-03-23T11:48:00.888 INFO:tasks.workunit.client.0.smithi063.stdout:1/949: readlink d3/d49/d56/d65/d84/lc8 0 2024-03-23T11:48:00.892 INFO:tasks.workunit.client.0.smithi063.stdout:5/696: fsync d2/fad 0 2024-03-23T11:48:00.894 INFO:tasks.workunit.client.0.smithi063.stdout:4/973: fdatasync d5/da/d20/d4b/d16/d32/fba 0 2024-03-23T11:48:00.902 INFO:tasks.workunit.client.0.smithi063.stdout:5/697: getdents d2/d7/d64 0 2024-03-23T11:48:00.902 INFO:tasks.workunit.client.0.smithi063.stdout:2/968: dread d2/d23/d2e/f35 [0,4194304] 0 2024-03-23T11:48:00.903 INFO:tasks.workunit.client.0.smithi063.stdout:5/698: write d2/d7/f1a [733534,52132] 0 2024-03-23T11:48:00.913 INFO:tasks.workunit.client.0.smithi063.stdout:2/969: creat d2/d23/d56/dad/dd4/f13e x:0 0 0 2024-03-23T11:48:00.918 INFO:tasks.workunit.client.0.smithi063.stdout:5/699: symlink d2/d17/d4d/d5f/le8 0 2024-03-23T11:48:00.920 INFO:tasks.workunit.client.0.smithi063.stdout:5/700: fsync d2/d7/d8/de/d2a/d3b/f73 0 2024-03-23T11:48:00.924 INFO:tasks.workunit.client.0.smithi063.stdout:2/970: unlink d2/de/d7a/f114 0 2024-03-23T11:48:00.925 INFO:tasks.workunit.client.0.smithi063.stdout:2/971: stat d2/d84/d94/c118 0 2024-03-23T11:48:00.930 INFO:tasks.workunit.client.0.smithi063.stdout:1/950: dread d3/d35/f90 [0,4194304] 0 2024-03-23T11:48:00.934 INFO:tasks.workunit.client.0.smithi063.stdout:5/701: rename d2/d7/d8/de/d2a/d3b to d2/d7/de9 0 2024-03-23T11:48:00.937 INFO:tasks.workunit.client.0.smithi063.stdout:2/972: creat d2/d46/d54/f13f x:0 0 0 2024-03-23T11:48:00.944 INFO:tasks.workunit.client.0.smithi063.stdout:6/859: dwrite d1/dc/d21/d49/f6a [0,4194304] 0 2024-03-23T11:48:00.948 INFO:tasks.workunit.client.0.smithi063.stdout:1/951: mknod d3/d49/d56/d65/d88/d92/d103/d12c/c135 0 2024-03-23T11:48:00.955 INFO:tasks.workunit.client.0.smithi063.stdout:6/860: creat d1/dc/d3a/f115 x:0 0 0 2024-03-23T11:48:00.955 INFO:tasks.workunit.client.0.smithi063.stdout:4/974: dwrite d5/f11b [0,4194304] 0 2024-03-23T11:48:00.959 INFO:tasks.workunit.client.0.smithi063.stdout:5/702: dread f1 [0,4194304] 0 2024-03-23T11:48:00.962 INFO:tasks.workunit.client.0.smithi063.stdout:2/973: link d2/de/d22/dfd/f113 d2/d46/d54/f140 0 2024-03-23T11:48:00.962 INFO:tasks.workunit.client.0.smithi063.stdout:6/861: write d1/d15/fa4 [674263,37765] 0 2024-03-23T11:48:00.968 INFO:tasks.workunit.client.0.smithi063.stdout:4/975: creat d5/da/f125 x:0 0 0 2024-03-23T11:48:00.969 INFO:tasks.workunit.client.0.smithi063.stdout:2/974: symlink d2/d23/d56/dad/l141 0 2024-03-23T11:48:00.970 INFO:tasks.workunit.client.0.smithi063.stdout:5/703: chown d2/d17/d4d/d6e/f76 516631834 1 2024-03-23T11:48:00.976 INFO:tasks.workunit.client.0.smithi063.stdout:6/862: read d1/f81 [1678204,57619] 0 2024-03-23T11:48:00.977 INFO:tasks.workunit.client.0.smithi063.stdout:4/976: mkdir d5/da/d20/d4b/d16/d50/d8e/d96/dca/d126 0 2024-03-23T11:48:00.983 INFO:tasks.workunit.client.0.smithi063.stdout:4/977: readlink d5/da/l4d 0 2024-03-23T11:48:00.983 INFO:tasks.workunit.client.0.smithi063.stdout:2/975: dread d2/d23/d2e/d98/dc8/f111 [0,4194304] 0 2024-03-23T11:48:00.986 INFO:tasks.workunit.client.0.smithi063.stdout:5/704: dread d2/d7/d8/f13 [0,4194304] 0 2024-03-23T11:48:00.988 INFO:tasks.workunit.client.0.smithi063.stdout:2/976: fdatasync d2/de/f15 0 2024-03-23T11:48:00.991 INFO:tasks.workunit.client.0.smithi063.stdout:4/978: write d5/da/d20/d4b/d16/d50/d8e/dd3/f7f [1360550,29929] 0 2024-03-23T11:48:00.993 INFO:tasks.workunit.client.0.smithi063.stdout:5/705: dread - d2/d7/fa0 zero size 2024-03-23T11:48:00.995 INFO:tasks.workunit.client.0.smithi063.stdout:4/979: readlink d5/da/d20/d4b/d29/d38/l8c 0 2024-03-23T11:48:01.001 INFO:tasks.workunit.client.0.smithi063.stdout:6/863: dwrite d1/dc/d3a/d82/da2/dd2/ff0 [0,4194304] 0 2024-03-23T11:48:01.009 INFO:tasks.workunit.client.0.smithi063.stdout:2/977: mknod d2/d23/d2e/d52/c142 0 2024-03-23T11:48:01.016 INFO:tasks.workunit.client.0.smithi063.stdout:2/978: creat d2/d23/d2e/d98/f143 x:0 0 0 2024-03-23T11:48:01.019 INFO:tasks.workunit.client.0.smithi063.stdout:2/979: truncate d2/d23/f83 506136 0 2024-03-23T11:48:01.025 INFO:tasks.workunit.client.0.smithi063.stdout:6/864: write d1/dc/d3a/d57/d6d/fa8 [1494856,86609] 0 2024-03-23T11:48:01.029 INFO:tasks.workunit.client.0.smithi063.stdout:6/865: symlink d1/dc/d3a/d82/da2/dd5/l116 0 2024-03-23T11:48:01.032 INFO:tasks.workunit.client.0.smithi063.stdout:2/980: write d2/d23/d2e/f55 [679348,121717] 0 2024-03-23T11:48:01.035 INFO:tasks.workunit.client.0.smithi063.stdout:2/981: mknod d2/d23/d56/c144 0 2024-03-23T11:48:01.037 INFO:tasks.workunit.client.0.smithi063.stdout:6/866: creat d1/dc/d3a/d82/ddc/f117 x:0 0 0 2024-03-23T11:48:01.039 INFO:tasks.workunit.client.0.smithi063.stdout:2/982: mkdir d2/d84/d145 0 2024-03-23T11:48:01.046 INFO:tasks.workunit.client.0.smithi063.stdout:2/983: symlink d2/d84/l146 0 2024-03-23T11:48:01.049 INFO:tasks.workunit.client.0.smithi063.stdout:4/980: dread d5/da/d7e/fed [0,4194304] 0 2024-03-23T11:48:01.049 INFO:tasks.workunit.client.0.smithi063.stdout:6/867: getdents d1/dc/d3a/d82/da2/d41 0 2024-03-23T11:48:01.051 INFO:tasks.workunit.client.0.smithi063.stdout:2/984: mknod d2/d23/d2e/d98/c147 0 2024-03-23T11:48:01.054 INFO:tasks.workunit.client.0.smithi063.stdout:6/868: rmdir d1/dc/d3a/d82/da2/d41/d42 39 2024-03-23T11:48:01.059 INFO:tasks.workunit.client.0.smithi063.stdout:4/981: mkdir d5/da/d20/d4b/d1f/d127 0 2024-03-23T11:48:01.073 INFO:tasks.workunit.client.0.smithi063.stdout:2/985: dwrite d2/d84/d92/fc1 [0,4194304] 0 2024-03-23T11:48:01.077 INFO:tasks.workunit.client.0.smithi063.stdout:2/986: readlink d2/l32 0 2024-03-23T11:48:01.078 INFO:tasks.workunit.client.0.smithi063.stdout:6/869: dwrite d1/d15/d4b/fc2 [0,4194304] 0 2024-03-23T11:48:01.082 INFO:tasks.workunit.client.0.smithi063.stdout:2/987: readlink d2/d1c/l3d 0 2024-03-23T11:48:01.083 INFO:tasks.workunit.client.0.smithi063.stdout:4/982: truncate d5/da/fcf 667878 0 2024-03-23T11:48:01.087 INFO:tasks.workunit.client.0.smithi063.stdout:6/870: mknod d1/dc/d3a/c118 0 2024-03-23T11:48:01.092 INFO:tasks.workunit.client.0.smithi063.stdout:4/983: rmdir d5/da/d20/d4b/d16/d50/d8e/d96 39 2024-03-23T11:48:01.093 INFO:tasks.workunit.client.0.smithi063.stdout:2/988: symlink d2/d23/d2e/d52/d86/d137/l148 0 2024-03-23T11:48:01.096 INFO:tasks.workunit.client.0.smithi063.stdout:6/871: symlink d1/dc/d3a/d57/df7/l119 0 2024-03-23T11:48:01.098 INFO:tasks.workunit.client.0.smithi063.stdout:2/989: mkdir d2/d46/d54/d149 0 2024-03-23T11:48:01.102 INFO:tasks.workunit.client.0.smithi063.stdout:4/984: creat d5/da/d20/d4b/d16/d50/d8e/dad/ddd/f128 x:0 0 0 2024-03-23T11:48:01.106 INFO:tasks.workunit.client.0.smithi063.stdout:4/985: symlink d5/da/d20/d4b/l129 0 2024-03-23T11:48:01.108 INFO:tasks.workunit.client.0.smithi063.stdout:6/872: dread d1/d13/f53 [0,4194304] 0 2024-03-23T11:48:01.112 INFO:tasks.workunit.client.0.smithi063.stdout:4/986: unlink d5/da/d20/d4b/d1f/cc5 0 2024-03-23T11:48:01.121 INFO:tasks.workunit.client.0.smithi063.stdout:6/873: dread d1/dc/d3a/d82/da2/d41/f5e [0,4194304] 0 2024-03-23T11:48:01.122 INFO:tasks.workunit.client.0.smithi063.stdout:4/987: truncate d5/da/d20/d4b/d16/d32/d56/f9e 415566 0 2024-03-23T11:48:01.123 INFO:tasks.workunit.client.0.smithi063.stdout:6/874: chown d1/dc/d21/d49/cff 3291512 1 2024-03-23T11:48:01.125 INFO:tasks.workunit.client.0.smithi063.stdout:4/988: chown d5/da/d20/d4b/d16/d32/d56/l9f 1 1 2024-03-23T11:48:01.127 INFO:tasks.workunit.client.0.smithi063.stdout:6/875: read d1/d15/d4b/f6b [235473,85388] 0 2024-03-23T11:48:01.129 INFO:tasks.workunit.client.0.smithi063.stdout:4/989: mkdir d5/da/d20/d4b/d16/d50/d5d/d12a 0 2024-03-23T11:48:01.131 INFO:tasks.workunit.client.0.smithi063.stdout:6/876: unlink d1/dc/d3a/d57/cdd 0 2024-03-23T11:48:01.134 INFO:tasks.workunit.client.0.smithi063.stdout:6/877: mkdir d1/dc/d3a/d82/ddc/d11a 0 2024-03-23T11:48:01.135 INFO:tasks.workunit.client.0.smithi063.stdout:4/990: creat d5/da/d20/d4b/d16/d50/d8e/d96/dca/f12b x:0 0 0 2024-03-23T11:48:01.138 INFO:tasks.workunit.client.0.smithi063.stdout:6/878: mkdir d1/dc/d3a/d82/ddc/d11a/d11b 0 2024-03-23T11:48:01.140 INFO:tasks.workunit.client.0.smithi063.stdout:4/991: mknod d5/da/d20/d4b/d16/d50/d8e/dad/c12c 0 2024-03-23T11:48:01.143 INFO:tasks.workunit.client.0.smithi063.stdout:6/879: rmdir d1/dc/d3a/d82/da2/d41/d42/d99 39 2024-03-23T11:48:01.149 INFO:tasks.workunit.client.0.smithi063.stdout:6/880: dread d1/d15/d4b/f6b [0,4194304] 0 2024-03-23T11:48:01.160 INFO:tasks.workunit.client.0.smithi063.stdout:4/992: dwrite d5/da/d20/d4b/d16/d50/d8e/d96/dca/f12b [0,4194304] 0 2024-03-23T11:48:01.166 INFO:tasks.workunit.client.0.smithi063.stdout:4/993: stat d5/da/d20/d4b/d16/fb4 0 2024-03-23T11:48:01.167 INFO:tasks.workunit.client.0.smithi063.stdout:4/994: mkdir d5/da/d20/d4b/d16/d37/d12d 0 2024-03-23T11:48:01.175 INFO:tasks.workunit.client.0.smithi063.stdout:6/881: dread d1/f81 [0,4194304] 0 2024-03-23T11:48:01.178 INFO:tasks.workunit.client.0.smithi063.stdout:4/995: write d5/da/d20/d4b/d16/d37/f57 [1416510,7189] 0 2024-03-23T11:48:01.189 INFO:tasks.workunit.client.0.smithi063.stdout:4/996: write d5/da/d20/d4b/d16/d32/d56/f9e [759629,63999] 0 2024-03-23T11:48:01.193 INFO:tasks.workunit.client.0.smithi063.stdout:4/997: mknod d5/d1e/d105/c12e 0 2024-03-23T11:48:01.193 INFO:tasks.workunit.client.0.smithi063.stdout:4/998: chown d5/da 948109 1 2024-03-23T11:48:01.195 INFO:tasks.workunit.client.0.smithi063.stdout:6/882: dwrite d1/ff5 [0,4194304] 0 2024-03-23T11:48:01.197 INFO:tasks.workunit.client.0.smithi063.stdout:6/883: chown d1/f80 8152 1 2024-03-23T11:48:01.202 INFO:tasks.workunit.client.0.smithi063.stdout:4/999: rename d5/da/d20/d4b/d16/d50/d8e/d96/ddf/c9b to d5/da/d7e/dbf/c12f 0 2024-03-23T11:48:01.241 INFO:tasks.workunit.client.0.smithi063.stdout:7/962: sync 2024-03-23T11:48:01.244 INFO:tasks.workunit.client.0.smithi063.stdout:7/963: write d5/d26/db7/dce/dc8/dee/f35 [4305046,113540] 0 2024-03-23T11:48:01.247 INFO:tasks.workunit.client.0.smithi063.stdout:7/964: write d5/d26/dff/fef [555002,107398] 0 2024-03-23T11:48:01.250 INFO:tasks.workunit.client.0.smithi063.stdout:7/965: creat d5/d5c/d2f/f137 x:0 0 0 2024-03-23T11:48:01.253 INFO:tasks.workunit.client.0.smithi063.stdout:7/966: write d5/d26/db7/dce/d64/f10c [3415135,78913] 0 2024-03-23T11:48:01.254 INFO:tasks.workunit.client.0.smithi063.stdout:7/967: chown d5/d6/d19/dcf 5 1 2024-03-23T11:48:01.260 INFO:tasks.workunit.client.0.smithi063.stdout:7/968: truncate d5/d26/d76/f78 1867274 0 2024-03-23T11:48:01.261 INFO:tasks.workunit.client.0.smithi063.stdout:7/969: dread - d5/d5c/d2f/fc3 zero size 2024-03-23T11:48:01.263 INFO:tasks.workunit.client.0.smithi063.stdout:7/970: chown d5/d5c/d2f/d73/da/d60 184883 1 2024-03-23T11:48:01.275 INFO:tasks.workunit.client.0.smithi063.stdout:7/971: dread d5/d5c/d2f/d8c/fae [0,4194304] 0 2024-03-23T11:48:01.287 INFO:tasks.workunit.client.0.smithi063.stdout:7/972: dwrite d5/d5c/d2f/fda [0,4194304] 0 2024-03-23T11:48:01.292 INFO:tasks.workunit.client.0.smithi063.stdout:7/973: getdents d5/d26/d76 0 2024-03-23T11:48:01.294 INFO:tasks.workunit.client.0.smithi063.stdout:7/974: rmdir d5/d26/db7/dce 39 2024-03-23T11:48:01.297 INFO:tasks.workunit.client.0.smithi063.stdout:7/975: mkdir d5/d5c/d2f/d73/da/d138 0 2024-03-23T11:48:01.302 INFO:tasks.workunit.client.0.smithi063.stdout:7/976: symlink d5/d26/dde/d100/da1/d114/l139 0 2024-03-23T11:48:01.326 INFO:tasks.workunit.client.0.smithi063.stdout:7/977: dread d5/d5c/d2f/d73/f8d [0,4194304] 0 2024-03-23T11:48:01.329 INFO:tasks.workunit.client.0.smithi063.stdout:7/978: unlink d5/d5c/d2f/d136/cea 0 2024-03-23T11:48:01.331 INFO:tasks.workunit.client.0.smithi063.stdout:7/979: dread - d5/d5c/d2f/ff1 zero size 2024-03-23T11:48:01.333 INFO:tasks.workunit.client.0.smithi063.stdout:7/980: truncate d5/d5c/d2f/f137 340640 0 2024-03-23T11:48:01.336 INFO:tasks.workunit.client.0.smithi063.stdout:7/981: creat d5/d5c/d2f/d136/f13a x:0 0 0 2024-03-23T11:48:01.342 INFO:tasks.workunit.client.0.smithi063.stdout:7/982: dread d5/d6/d19/f7d [0,4194304] 0 2024-03-23T11:48:01.343 INFO:tasks.workunit.client.0.smithi063.stdout:7/983: read - d5/d5c/d2f/d73/f105 zero size 2024-03-23T11:48:01.345 INFO:tasks.workunit.client.0.smithi063.stdout:7/984: read d5/d5c/d2f/f137 [233975,68414] 0 2024-03-23T11:48:01.347 INFO:tasks.workunit.client.0.smithi063.stdout:7/985: fdatasync d5/d5c/d2f/fda 0 2024-03-23T11:48:01.364 INFO:tasks.workunit.client.0.smithi063.stdout:7/986: dwrite d5/d26/db7/dd6/d88/fa9 [0,4194304] 0 2024-03-23T11:48:01.368 INFO:tasks.workunit.client.0.smithi063.stdout:7/987: truncate d5/d5c/d2f/d8b/f10d 1412119 0 2024-03-23T11:48:01.369 INFO:tasks.workunit.client.0.smithi063.stdout:7/988: stat d5/d6/d19/d1f 0 2024-03-23T11:48:01.382 INFO:tasks.workunit.client.0.smithi063.stdout:7/989: dread d5/d6/d19/d1f/d6c/f80 [0,4194304] 0 2024-03-23T11:48:01.386 INFO:tasks.workunit.client.0.smithi063.stdout:7/990: getdents d5/d5c/d2f/d8b/daa/de1 0 2024-03-23T11:48:01.389 INFO:tasks.workunit.client.0.smithi063.stdout:7/991: symlink d5/d26/dde/l13b 0 2024-03-23T11:48:01.389 INFO:tasks.workunit.client.0.smithi063.stdout:7/992: readlink d5/d22/ld5 0 2024-03-23T11:48:01.392 INFO:tasks.workunit.client.0.smithi063.stdout:7/993: truncate d5/d26/db7/dd6/d88/fa9 4957247 0 2024-03-23T11:48:01.407 INFO:tasks.workunit.client.0.smithi063.stdout:7/994: dwrite d5/d5c/d2f/d73/da/f17 [4194304,4194304] 0 2024-03-23T11:48:01.408 INFO:tasks.workunit.client.0.smithi063.stdout:7/995: fsync d5/d6/dca/fcb 0 2024-03-23T11:48:01.412 INFO:tasks.workunit.client.0.smithi063.stdout:7/996: chown d5/d5c/d2f/d73/da/d60/fa6 7889677 1 2024-03-23T11:48:01.416 INFO:tasks.workunit.client.0.smithi063.stdout:7/997: fsync d5/d6/d19/d65/fba 0 2024-03-23T11:48:01.432 INFO:tasks.workunit.client.0.smithi063.stdout:7/998: dwrite d5/d22/dcc/fe0 [0,4194304] 0 2024-03-23T11:48:01.437 INFO:tasks.workunit.client.0.smithi063.stdout:7/999: mknod d5/d26/db7/dd6/d88/c13c 0 2024-03-23T11:48:01.622 INFO:tasks.workunit.client.0.smithi063.stdout:1/952: sync 2024-03-23T11:48:01.626 INFO:tasks.workunit.client.0.smithi063.stdout:1/953: getdents d3/df/d7b 0 2024-03-23T11:48:01.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:01 smithi063 ceph-mon[35189]: pgmap v50: 65 pgs: 65 active+clean; 3.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 66 MiB/s rd, 146 MiB/s wr, 363 op/s 2024-03-23T11:48:01.650 INFO:tasks.workunit.client.0.smithi063.stdout:1/954: dwrite d3/d49/d56/d65/d88/fe6 [0,4194304] 0 2024-03-23T11:48:01.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:01 smithi111 ceph-mon[44005]: pgmap v50: 65 pgs: 65 active+clean; 3.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 66 MiB/s rd, 146 MiB/s wr, 363 op/s 2024-03-23T11:48:01.677 INFO:tasks.workunit.client.0.smithi063.stdout:1/955: dread d3/d35/d40/f4c [0,4194304] 0 2024-03-23T11:48:01.681 INFO:tasks.workunit.client.0.smithi063.stdout:1/956: symlink d3/d3f/d4d/d52/l136 0 2024-03-23T11:48:01.686 INFO:tasks.workunit.client.0.smithi063.stdout:1/957: rename d3/d3f/d4d/dbc/de0/c59 to d3/d62/dad/c137 0 2024-03-23T11:48:01.687 INFO:tasks.workunit.client.0.smithi063.stdout:1/958: chown d3/d3f/d4d/dbc/d118 2830392 1 2024-03-23T11:48:01.703 INFO:tasks.workunit.client.0.smithi063.stdout:1/959: dwrite d3/f4 [0,4194304] 0 2024-03-23T11:48:01.713 INFO:tasks.workunit.client.0.smithi063.stdout:1/960: dread d3/d3f/d4d/dbc/de0/da8/faf [0,4194304] 0 2024-03-23T11:48:01.720 INFO:tasks.workunit.client.0.smithi063.stdout:5/706: sync 2024-03-23T11:48:01.721 INFO:tasks.workunit.client.0.smithi063.stdout:6/884: sync 2024-03-23T11:48:01.721 INFO:tasks.workunit.client.0.smithi063.stdout:2/990: sync 2024-03-23T11:48:01.723 INFO:tasks.workunit.client.0.smithi063.stdout:1/961: rename d3/d3f/d4d/l55 to d3/d3f/d4d/d52/l138 0 2024-03-23T11:48:01.730 INFO:tasks.workunit.client.0.smithi063.stdout:1/962: truncate d3/df/d2f/fcc 4979339 0 2024-03-23T11:48:01.733 INFO:tasks.workunit.client.0.smithi063.stdout:6/885: truncate d1/dc/d21/d49/f76 716817 0 2024-03-23T11:48:01.736 INFO:tasks.workunit.client.0.smithi063.stdout:2/991: rename d2/d46/db4 to d2/d23/d2e/d14a 0 2024-03-23T11:48:01.741 INFO:tasks.workunit.client.0.smithi063.stdout:5/707: dread d2/d17/f51 [0,4194304] 0 2024-03-23T11:48:01.742 INFO:tasks.workunit.client.0.smithi063.stdout:6/886: mkdir d1/dc/d3a/d82/da2/d41/d11c 0 2024-03-23T11:48:01.745 INFO:tasks.workunit.client.0.smithi063.stdout:6/887: read d1/d13/f1c [18686,82347] 0 2024-03-23T11:48:01.747 INFO:tasks.workunit.client.0.smithi063.stdout:6/888: readlink d1/d15/le1 0 2024-03-23T11:48:01.757 INFO:tasks.workunit.client.0.smithi063.stdout:5/708: rename d2/d17/d4d/d6e/dba/cae to d2/d7/d64/cea 0 2024-03-23T11:48:01.758 INFO:tasks.workunit.client.0.smithi063.stdout:1/963: dwrite d3/d35/f93 [0,4194304] 0 2024-03-23T11:48:01.761 INFO:tasks.workunit.client.0.smithi063.stdout:6/889: rmdir d1/dc/d3a/d82/da2/d41/d42/d99/dc3/de7 0 2024-03-23T11:48:01.767 INFO:tasks.workunit.client.0.smithi063.stdout:5/709: symlink d2/d17/d4d/d6e/dba/dbe/leb 0 2024-03-23T11:48:01.768 INFO:tasks.workunit.client.0.smithi063.stdout:2/992: dwrite d2/d23/d2e/d52/f6a [0,4194304] 0 2024-03-23T11:48:01.768 INFO:tasks.workunit.client.0.smithi063.stdout:6/890: chown d1/d15/fb8 854169 1 2024-03-23T11:48:01.777 INFO:tasks.workunit.client.0.smithi063.stdout:5/710: unlink d2/d28/d3a/d46/d79/c7a 0 2024-03-23T11:48:01.784 INFO:tasks.workunit.client.0.smithi063.stdout:2/993: creat d2/d23/d2e/d87/f14b x:0 0 0 2024-03-23T11:48:01.784 INFO:tasks.workunit.client.0.smithi063.stdout:6/891: link d1/d13/f53 d1/dc/d3a/d82/da2/dd8/f11d 0 2024-03-23T11:48:01.791 INFO:tasks.workunit.client.0.smithi063.stdout:6/892: dread d1/dc/d3a/d57/d6d/fc5 [0,4194304] 0 2024-03-23T11:48:01.792 INFO:tasks.workunit.client.0.smithi063.stdout:2/994: mknod d2/d23/d2e/d87/d4d/c14c 0 2024-03-23T11:48:01.793 INFO:tasks.workunit.client.0.smithi063.stdout:1/964: dwrite d3/d49/d56/d65/f7c [0,4194304] 0 2024-03-23T11:48:01.795 INFO:tasks.workunit.client.0.smithi063.stdout:1/965: stat d3/df/l13 0 2024-03-23T11:48:01.798 INFO:tasks.workunit.client.0.smithi063.stdout:6/893: write d1/dc/d3a/d82/da2/d41/d42/fc8 [904879,112639] 0 2024-03-23T11:48:01.799 INFO:tasks.workunit.client.0.smithi063.stdout:1/966: dread - d3/d35/f94 zero size 2024-03-23T11:48:01.801 INFO:tasks.workunit.client.0.smithi063.stdout:2/995: mkdir d2/d23/d2e/d52/d86/d137/d14d 0 2024-03-23T11:48:01.804 INFO:tasks.workunit.client.0.smithi063.stdout:1/967: readlink d3/d35/d40/l74 0 2024-03-23T11:48:01.807 INFO:tasks.workunit.client.0.smithi063.stdout:2/996: chown d2/l16 127458999 1 2024-03-23T11:48:01.809 INFO:tasks.workunit.client.0.smithi063.stdout:1/968: readlink d3/d49/d56/d65/l121 0 2024-03-23T11:48:01.812 INFO:tasks.workunit.client.0.smithi063.stdout:5/711: dwrite d2/d17/d4d/d6e/fdb [0,4194304] 0 2024-03-23T11:48:01.815 INFO:tasks.workunit.client.0.smithi063.stdout:2/997: mknod d2/d23/d2e/d52/d86/c14e 0 2024-03-23T11:48:01.816 INFO:tasks.workunit.client.0.smithi063.stdout:1/969: creat d3/d3f/d4d/f139 x:0 0 0 2024-03-23T11:48:01.819 INFO:tasks.workunit.client.0.smithi063.stdout:1/970: fsync d3/df/d2f/f10b 0 2024-03-23T11:48:01.823 INFO:tasks.workunit.client.0.smithi063.stdout:5/712: read d2/f6 [4029773,22209] 0 2024-03-23T11:48:01.824 INFO:tasks.workunit.client.0.smithi063.stdout:1/971: readlink d3/d3f/d4d/dbc/de0/d4b/da5/la6 0 2024-03-23T11:48:01.831 INFO:tasks.workunit.client.0.smithi063.stdout:1/972: symlink d3/d49/d56/d65/d102/d124/l13a 0 2024-03-23T11:48:01.834 INFO:tasks.workunit.client.0.smithi063.stdout:6/894: write d1/dc/d3a/d57/f97 [188685,16803] 0 2024-03-23T11:48:01.837 INFO:tasks.workunit.client.0.smithi063.stdout:5/713: rename d2/d17/d4d/d6e/l93 to d2/d7/lec 0 2024-03-23T11:48:01.840 INFO:tasks.workunit.client.0.smithi063.stdout:5/714: chown d2/d7/d8/de/cf 0 1 2024-03-23T11:48:01.840 INFO:tasks.workunit.client.0.smithi063.stdout:6/895: creat d1/dc/d8c/f11e x:0 0 0 2024-03-23T11:48:01.842 INFO:tasks.workunit.client.0.smithi063.stdout:2/998: truncate d2/d23/d2e/d87/d4d/f79 2836104 0 2024-03-23T11:48:01.846 INFO:tasks.workunit.client.0.smithi063.stdout:2/999: mknod d2/d23/d56/c14f 0 2024-03-23T11:48:01.848 INFO:tasks.workunit.client.0.smithi063.stdout:6/896: mkdir d1/dc/d3a/d82/da2/d41/d42/d99/de9/d11f 0 2024-03-23T11:48:01.851 INFO:tasks.workunit.client.0.smithi063.stdout:5/715: link d2/d2f/f37 d2/d2f/fed 0 2024-03-23T11:48:01.851 INFO:tasks.workunit.client.0.smithi063.stdout:1/973: dwrite d3/df/d2f/d9a/de2/f12b [0,4194304] 0 2024-03-23T11:48:01.854 INFO:tasks.workunit.client.0.smithi063.stdout:6/897: fdatasync d1/dc/d3a/d82/da2/f4a 0 2024-03-23T11:48:01.854 INFO:tasks.workunit.client.0.smithi063.stdout:5/716: chown d2/d7/d8/d1d/f88 33323 1 2024-03-23T11:48:01.857 INFO:tasks.workunit.client.0.smithi063.stdout:5/717: write d2/d17/d4d/d6e/fdb [2203540,45838] 0 2024-03-23T11:48:01.860 INFO:tasks.workunit.client.0.smithi063.stdout:6/898: chown d1/dc/d3a/d82/da2/d41/lb4 13 1 2024-03-23T11:48:01.861 INFO:tasks.workunit.client.0.smithi063.stdout:5/718: creat d2/d28/d3a/d46/d79/fee x:0 0 0 2024-03-23T11:48:01.861 INFO:tasks.workunit.client.0.smithi063.stdout:6/899: readlink d1/dc/d3a/l78 0 2024-03-23T11:48:01.866 INFO:tasks.workunit.client.0.smithi063.stdout:5/719: creat d2/d28/d3a/d46/d79/dc0/fef x:0 0 0 2024-03-23T11:48:01.867 INFO:tasks.workunit.client.0.smithi063.stdout:6/900: read d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb/fec [1231305,9652] 0 2024-03-23T11:48:01.873 INFO:tasks.workunit.client.0.smithi063.stdout:5/720: creat d2/d7/ff0 x:0 0 0 2024-03-23T11:48:01.882 INFO:tasks.workunit.client.0.smithi063.stdout:5/721: rename d2/d7/de9/d43/f45 to d2/d28/d3a/d46/d71/d9d/ff1 0 2024-03-23T11:48:01.886 INFO:tasks.workunit.client.0.smithi063.stdout:6/901: dwrite d1/f80 [0,4194304] 0 2024-03-23T11:48:01.887 INFO:tasks.workunit.client.0.smithi063.stdout:5/722: creat d2/d28/de6/ff2 x:0 0 0 2024-03-23T11:48:01.906 INFO:tasks.workunit.client.0.smithi063.stdout:6/902: dwrite d1/dc/d21/fe2 [0,4194304] 0 2024-03-23T11:48:01.909 INFO:tasks.workunit.client.0.smithi063.stdout:6/903: fsync d1/dc/d3a/f30 0 2024-03-23T11:48:01.910 INFO:tasks.workunit.client.0.smithi063.stdout:5/723: dwrite d2/d7/d2b/d8b/da7/dc6/fd4 [0,4194304] 0 2024-03-23T11:48:01.913 INFO:tasks.workunit.client.0.smithi063.stdout:5/724: write d2/d7/d8/f68 [889247,64057] 0 2024-03-23T11:48:01.914 INFO:tasks.workunit.client.0.smithi063.stdout:6/904: mknod d1/dc/d3a/d82/da2/c120 0 2024-03-23T11:48:01.917 INFO:tasks.workunit.client.0.smithi063.stdout:5/725: fdatasync d2/d7/de9/f73 0 2024-03-23T11:48:01.924 INFO:tasks.workunit.client.0.smithi063.stdout:6/905: rename d1/dc/d3a/d82/da2/d41/d42/la6 to d1/dc/l121 0 2024-03-23T11:48:01.926 INFO:tasks.workunit.client.0.smithi063.stdout:6/906: write d1/dc/d21/f111 [64407,60022] 0 2024-03-23T11:48:01.936 INFO:tasks.workunit.client.0.smithi063.stdout:6/907: dread d1/dc/d3a/d82/da2/d41/d42/fc8 [0,4194304] 0 2024-03-23T11:48:01.940 INFO:tasks.workunit.client.0.smithi063.stdout:6/908: unlink d1/dc/d3a/d57/d6d/ddb/c10a 0 2024-03-23T11:48:01.943 INFO:tasks.workunit.client.0.smithi063.stdout:6/909: mknod d1/dc/d3a/d82/d110/c122 0 2024-03-23T11:48:01.945 INFO:tasks.workunit.client.0.smithi063.stdout:6/910: mknod d1/d15/d4b/c123 0 2024-03-23T11:48:01.954 INFO:tasks.workunit.client.0.smithi063.stdout:6/911: write d1/dc/d3a/d82/da2/d41/f5e [583287,8091] 0 2024-03-23T11:48:01.962 INFO:tasks.workunit.client.0.smithi063.stdout:6/912: rename d1/f12 to d1/dc/d3a/d57/f124 0 2024-03-23T11:48:01.967 INFO:tasks.workunit.client.0.smithi063.stdout:6/913: creat d1/dc/d3a/d82/f125 x:0 0 0 2024-03-23T11:48:01.972 INFO:tasks.workunit.client.0.smithi063.stdout:6/914: truncate d1/dc/d3a/f8f 1029408 0 2024-03-23T11:48:01.975 INFO:tasks.workunit.client.0.smithi063.stdout:6/915: chown d1/d15/cce 21 1 2024-03-23T11:48:01.981 INFO:tasks.workunit.client.0.smithi063.stdout:6/916: creat d1/dc/d3a/d82/d85/df4/f126 x:0 0 0 2024-03-23T11:48:01.987 INFO:tasks.workunit.client.0.smithi063.stdout:6/917: creat d1/dc/d3a/d57/d6d/f127 x:0 0 0 2024-03-23T11:48:01.991 INFO:tasks.workunit.client.0.smithi063.stdout:6/918: mknod d1/dc/d3a/d82/da2/c128 0 2024-03-23T11:48:01.998 INFO:tasks.workunit.client.0.smithi063.stdout:6/919: truncate d1/dc/d3a/d57/d77/fad 902942 0 2024-03-23T11:48:02.416 INFO:tasks.workunit.client.0.smithi063.stdout:1/974: sync 2024-03-23T11:48:02.416 INFO:tasks.workunit.client.0.smithi063.stdout:6/920: sync 2024-03-23T11:48:02.416 INFO:tasks.workunit.client.0.smithi063.stdout:5/726: sync 2024-03-23T11:48:02.422 INFO:tasks.workunit.client.0.smithi063.stdout:6/921: fdatasync d1/dc/f1d 0 2024-03-23T11:48:02.425 INFO:tasks.workunit.client.0.smithi063.stdout:6/922: write d1/dc/d3a/d82/da2/dd2/ff0 [611247,75624] 0 2024-03-23T11:48:02.428 INFO:tasks.workunit.client.0.smithi063.stdout:5/727: write d2/f6 [3884345,8619] 0 2024-03-23T11:48:02.431 INFO:tasks.workunit.client.0.smithi063.stdout:6/923: chown d1/dc/d3a/d57/f98 50202 1 2024-03-23T11:48:02.434 INFO:tasks.workunit.client.0.smithi063.stdout:5/728: mkdir d2/d7/d8/de/d49/db0/dca/df3 0 2024-03-23T11:48:02.435 INFO:tasks.workunit.client.0.smithi063.stdout:6/924: chown d1/dc/d3a/d57/d77/fad 0 1 2024-03-23T11:48:02.436 INFO:tasks.workunit.client.0.smithi063.stdout:5/729: rmdir d2/d28/d3a/d46 39 2024-03-23T11:48:02.437 INFO:tasks.workunit.client.0.smithi063.stdout:6/925: stat d1/dc/d3a/d82/da2/dd5/le8 0 2024-03-23T11:48:02.443 INFO:tasks.workunit.client.0.smithi063.stdout:1/975: dread d3/d49/d56/d65/db6/f131 [0,4194304] 0 2024-03-23T11:48:02.445 INFO:tasks.workunit.client.0.smithi063.stdout:1/976: fdatasync d3/d49/d56/d65/f129 0 2024-03-23T11:48:02.448 INFO:tasks.workunit.client.0.smithi063.stdout:1/977: chown d3/d3f/d4d/f95 4136 1 2024-03-23T11:48:02.451 INFO:tasks.workunit.client.0.smithi063.stdout:5/730: dwrite d2/d7/f1a [0,4194304] 0 2024-03-23T11:48:02.452 INFO:tasks.workunit.client.0.smithi063.stdout:1/978: mknod d3/d35/c13b 0 2024-03-23T11:48:02.461 INFO:tasks.workunit.client.0.smithi063.stdout:6/926: dread d1/dc/d21/d49/f95 [0,4194304] 0 2024-03-23T11:48:02.466 INFO:tasks.workunit.client.0.smithi063.stdout:1/979: rename d3/d3f/d4d/dbc/de0/dce/l115 to d3/df/d2f/de4/df5/d11b/l13c 0 2024-03-23T11:48:02.466 INFO:tasks.workunit.client.0.smithi063.stdout:6/927: mkdir d1/dc/d3a/d57/d129 0 2024-03-23T11:48:02.467 INFO:tasks.workunit.client.0.smithi063.stdout:1/980: creat d3/d49/d56/d65/d102/f13d x:0 0 0 2024-03-23T11:48:02.471 INFO:tasks.workunit.client.0.smithi063.stdout:6/928: rmdir d1/dc/d3a/d82 39 2024-03-23T11:48:02.474 INFO:tasks.workunit.client.0.smithi063.stdout:1/981: read d3/d49/d56/d65/d88/f9c [485513,15501] 0 2024-03-23T11:48:02.476 INFO:tasks.workunit.client.0.smithi063.stdout:6/929: creat d1/dc/d3a/d82/f12a x:0 0 0 2024-03-23T11:48:02.478 INFO:tasks.workunit.client.0.smithi063.stdout:6/930: stat d1/dc/d3a/d57/d129 0 2024-03-23T11:48:02.487 INFO:tasks.workunit.client.0.smithi063.stdout:5/731: dwrite d2/d7/d8/de/d47/fab [0,4194304] 0 2024-03-23T11:48:02.497 INFO:tasks.workunit.client.0.smithi063.stdout:5/732: rename d2/d7/d8/de/d49/f8e to d2/d7/d8/de/d35/da6/dd6/ff4 0 2024-03-23T11:48:02.499 INFO:tasks.workunit.client.0.smithi063.stdout:1/982: dwrite d3/d3f/d4d/dbc/de0/d4b/da5/dd9/f41 [0,4194304] 0 2024-03-23T11:48:02.506 INFO:tasks.workunit.client.0.smithi063.stdout:1/983: getdents d3/df/d2f/de4/df5 0 2024-03-23T11:48:02.507 INFO:tasks.workunit.client.0.smithi063.stdout:1/984: stat d3/d49/c130 0 2024-03-23T11:48:02.510 INFO:tasks.workunit.client.0.smithi063.stdout:1/985: write d3/df/db8/dcb/ff7 [521274,58059] 0 2024-03-23T11:48:02.515 INFO:tasks.workunit.client.0.smithi063.stdout:1/986: creat d3/d35/d40/d134/f13e x:0 0 0 2024-03-23T11:48:02.583 INFO:tasks.workunit.client.0.smithi063.stdout:1/987: write d3/d35/d40/d134/f13e [156902,114519] 0 2024-03-23T11:48:02.583 INFO:tasks.workunit.client.0.smithi063.stdout:5/733: dwrite d2/d7/d8/d1d/fcb [0,4194304] 0 2024-03-23T11:48:02.631 INFO:tasks.workunit.client.0.smithi063.stdout:5/734: symlink d2/d7/d8/de/d2a/lf5 0 2024-03-23T11:48:02.644 INFO:tasks.workunit.client.0.smithi063.stdout:1/988: dwrite d3/d49/d56/d65/d84/f98 [4194304,4194304] 0 2024-03-23T11:48:02.652 INFO:tasks.workunit.client.0.smithi063.stdout:1/989: symlink d3/d49/d56/d65/d84/l13f 0 2024-03-23T11:48:02.667 INFO:tasks.workunit.client.0.smithi063.stdout:1/990: truncate d3/d3f/d4d/dbc/de0/d4b/da5/dd9/f41 1888636 0 2024-03-23T11:48:02.669 INFO:tasks.workunit.client.0.smithi063.stdout:1/991: chown d3/d49/d56/d65/d88/d92/fde 3119030 1 2024-03-23T11:48:02.673 INFO:tasks.workunit.client.0.smithi063.stdout:5/735: dwrite d2/d28/d3a/d46/f86 [0,4194304] 0 2024-03-23T11:48:02.676 INFO:tasks.workunit.client.0.smithi063.stdout:1/992: write d3/d3f/f77 [2861350,38897] 0 2024-03-23T11:48:02.682 INFO:tasks.workunit.client.0.smithi063.stdout:5/736: rename d2/d7/de9/d43/d5c/f62 to d2/d28/d3a/d46/d71/d9d/ff6 0 2024-03-23T11:48:02.686 INFO:tasks.workunit.client.0.smithi063.stdout:1/993: rename d3/d49/d56/d65/d88/d92/l11c to d3/d49/d56/d65/d88/d92/l140 0 2024-03-23T11:48:02.686 INFO:tasks.workunit.client.0.smithi063.stdout:6/931: sync 2024-03-23T11:48:02.686 INFO:tasks.workunit.client.0.smithi063.stdout:5/737: creat d2/d7/d8/de/d35/ff7 x:0 0 0 2024-03-23T11:48:02.689 INFO:tasks.workunit.client.0.smithi063.stdout:5/738: symlink d2/d2f/lf8 0 2024-03-23T11:48:02.691 INFO:tasks.workunit.client.0.smithi063.stdout:6/932: creat d1/dc/d3a/d82/ddc/d11a/d11b/f12b x:0 0 0 2024-03-23T11:48:02.692 INFO:tasks.workunit.client.0.smithi063.stdout:5/739: dread - d2/d7/d8/de/d35/da6/dd6/fe4 zero size 2024-03-23T11:48:02.696 INFO:tasks.workunit.client.0.smithi063.stdout:6/933: symlink d1/dc/d3a/d82/da2/d41/d42/d99/de9/l12c 0 2024-03-23T11:48:02.696 INFO:tasks.workunit.client.0.smithi063.stdout:5/740: symlink d2/d28/ddf/lf9 0 2024-03-23T11:48:02.702 INFO:tasks.workunit.client.0.smithi063.stdout:5/741: getdents d2/d28/d3a/d46/d79/dc0 0 2024-03-23T11:48:02.710 INFO:tasks.workunit.client.0.smithi063.stdout:5/742: rename d2/d28/d3a/d46/d71/d9d/ff1 to d2/d28/d3a/d46/da4/ffa 0 2024-03-23T11:48:02.712 INFO:tasks.workunit.client.0.smithi063.stdout:5/743: write d2/d7/ff0 [65814,81656] 0 2024-03-23T11:48:02.716 INFO:tasks.workunit.client.0.smithi063.stdout:6/934: dwrite d1/d15/f26 [0,4194304] 0 2024-03-23T11:48:02.731 INFO:tasks.workunit.client.0.smithi063.stdout:6/935: dwrite d1/dc/d3a/f4f [0,4194304] 0 2024-03-23T11:48:02.732 INFO:tasks.workunit.client.0.smithi063.stdout:6/936: chown d1/f9e 26475460 1 2024-03-23T11:48:02.734 INFO:tasks.workunit.client.0.smithi063.stdout:6/937: chown d1/dc/d3a/d82/da2/f108 378952 1 2024-03-23T11:48:02.737 INFO:tasks.workunit.client.0.smithi063.stdout:6/938: truncate d1/dc/d3a/d57/df7/ff8 371015 0 2024-03-23T11:48:02.741 INFO:tasks.workunit.client.0.smithi063.stdout:6/939: mkdir d1/dc/d3a/d57/df7/d12d 0 2024-03-23T11:48:02.748 INFO:tasks.workunit.client.0.smithi063.stdout:6/940: getdents d1/dc/dea 0 2024-03-23T11:48:02.751 INFO:tasks.workunit.client.0.smithi063.stdout:6/941: write d1/dc/d3a/fee [1001110,127379] 0 2024-03-23T11:48:02.756 INFO:tasks.workunit.client.0.smithi063.stdout:6/942: creat d1/dc/d3a/d57/d6d/ddb/f12e x:0 0 0 2024-03-23T11:48:02.762 INFO:tasks.workunit.client.0.smithi063.stdout:6/943: rename d1/dc/d21/d49 to d1/dc/d3a/d57/d6d/d12f 0 2024-03-23T11:48:02.765 INFO:tasks.workunit.client.0.smithi063.stdout:6/944: rmdir d1/dc/d8c 39 2024-03-23T11:48:02.766 INFO:tasks.workunit.client.0.smithi063.stdout:6/945: chown d1/dc/d3a/d57/d6d/ddb 35976583 1 2024-03-23T11:48:02.770 INFO:tasks.workunit.client.0.smithi063.stdout:6/946: truncate d1/dc/fd 3767307 0 2024-03-23T11:48:02.772 INFO:tasks.workunit.client.0.smithi063.stdout:6/947: write d1/ff5 [584075,91684] 0 2024-03-23T11:48:02.774 INFO:tasks.workunit.client.0.smithi063.stdout:6/948: chown d1/dc/d3a/d57/d77/fad 2044 1 2024-03-23T11:48:02.775 INFO:tasks.workunit.client.0.smithi063.stdout:6/949: readlink d1/dc/d3a/l78 0 2024-03-23T11:48:02.777 INFO:tasks.workunit.client.0.smithi063.stdout:6/950: readlink d1/d15/l7a 0 2024-03-23T11:48:02.779 INFO:tasks.workunit.client.0.smithi063.stdout:6/951: stat d1/dc/c5a 0 2024-03-23T11:48:02.795 INFO:tasks.workunit.client.0.smithi063.stdout:6/952: dwrite d1/dc/d3a/d82/da2/dd8/ff3 [0,4194304] 0 2024-03-23T11:48:02.802 INFO:tasks.workunit.client.0.smithi063.stdout:6/953: mkdir d1/dc/d3a/d57/d6d/ddb/d10e/d130 0 2024-03-23T11:48:02.816 INFO:tasks.workunit.client.0.smithi063.stdout:6/954: dwrite d1/f107 [0,4194304] 0 2024-03-23T11:48:02.820 INFO:tasks.workunit.client.0.smithi063.stdout:6/955: mknod d1/d15/d4b/c131 0 2024-03-23T11:48:02.824 INFO:tasks.workunit.client.0.smithi063.stdout:6/956: mkdir d1/dc/d3a/d82/da2/d41/d132 0 2024-03-23T11:48:02.832 INFO:tasks.workunit.client.0.smithi063.stdout:6/957: dread d1/d15/fb8 [0,4194304] 0 2024-03-23T11:48:03.126 INFO:tasks.workunit.client.0.smithi063.stdout:1/994: sync 2024-03-23T11:48:03.127 INFO:tasks.workunit.client.0.smithi063.stdout:5/744: sync 2024-03-23T11:48:03.127 INFO:tasks.workunit.client.0.smithi063.stdout:6/958: sync 2024-03-23T11:48:03.130 INFO:tasks.workunit.client.0.smithi063.stdout:5/745: mkdir d2/d17/d4d/dfb 0 2024-03-23T11:48:03.134 INFO:tasks.workunit.client.0.smithi063.stdout:1/995: rename d3/d3f/d4d/dbc/de0/d4b to d3/d141 0 2024-03-23T11:48:03.137 INFO:tasks.workunit.client.0.smithi063.stdout:6/959: getdents d1/dc/d3a/d57/d77 0 2024-03-23T11:48:03.142 INFO:tasks.workunit.client.0.smithi063.stdout:1/996: write d3/feb [146450,55353] 0 2024-03-23T11:48:03.144 INFO:tasks.workunit.client.0.smithi063.stdout:6/960: link d1/dc/d3a/d82/da2/c120 d1/d15/d4b/c133 0 2024-03-23T11:48:03.147 INFO:tasks.workunit.client.0.smithi063.stdout:1/997: mknod d3/d49/c142 0 2024-03-23T11:48:03.148 INFO:tasks.workunit.client.0.smithi063.stdout:6/961: symlink d1/dc/dea/l134 0 2024-03-23T11:48:03.149 INFO:tasks.workunit.client.0.smithi063.stdout:1/998: fdatasync d3/feb 0 2024-03-23T11:48:03.154 INFO:tasks.workunit.client.0.smithi063.stdout:5/746: dwrite d2/fad [0,4194304] 0 2024-03-23T11:48:03.156 INFO:tasks.workunit.client.0.smithi063.stdout:6/962: creat d1/dc/d3a/d82/d85/df4/f135 x:0 0 0 2024-03-23T11:48:03.161 INFO:tasks.workunit.client.0.smithi063.stdout:5/747: getdents d2/d17/d4d/d6e/de2 0 2024-03-23T11:48:03.166 INFO:tasks.workunit.client.0.smithi063.stdout:5/748: dread d2/d7/d64/f87 [0,4194304] 0 2024-03-23T11:48:03.168 INFO:tasks.workunit.client.0.smithi063.stdout:5/749: read f1 [1865709,116112] 0 2024-03-23T11:48:03.173 INFO:tasks.workunit.client.0.smithi063.stdout:1/999: dwrite d3/d49/d56/d65/f129 [0,4194304] 0 2024-03-23T11:48:03.175 INFO:tasks.workunit.client.0.smithi063.stdout:5/750: creat d2/d28/ffc x:0 0 0 2024-03-23T11:48:03.181 INFO:tasks.workunit.client.0.smithi063.stdout:5/751: link d2/d7/f23 d2/d7/d8/de/d49/db0/ffd 0 2024-03-23T11:48:03.185 INFO:tasks.workunit.client.0.smithi063.stdout:6/963: dread d1/dc/d3a/f30 [0,4194304] 0 2024-03-23T11:48:03.187 INFO:tasks.workunit.client.0.smithi063.stdout:5/752: creat d2/d7/de9/d43/ffe x:0 0 0 2024-03-23T11:48:03.189 INFO:tasks.workunit.client.0.smithi063.stdout:6/964: rmdir d1/d15 39 2024-03-23T11:48:03.190 INFO:tasks.workunit.client.0.smithi063.stdout:5/753: fsync d2/d28/f4b 0 2024-03-23T11:48:03.198 INFO:tasks.workunit.client.0.smithi063.stdout:6/965: write d1/dc/d3a/d57/d6d/fc0 [38592,117226] 0 2024-03-23T11:48:03.207 INFO:tasks.workunit.client.0.smithi063.stdout:6/966: rename d1/dc/d3a/d82/da2/d41/d42/d99/dc3/ccd to d1/dc/d3a/d82/d85/df4/c136 0 2024-03-23T11:48:03.213 INFO:tasks.workunit.client.0.smithi063.stdout:6/967: rename d1/l1a to d1/dc/d3a/d82/da2/d41/d42/d99/dc3/dbb/l137 0 2024-03-23T11:48:03.215 INFO:tasks.workunit.client.0.smithi063.stdout:5/754: dread d2/d7/d8/d1d/f57 [0,4194304] 0 2024-03-23T11:48:03.216 INFO:tasks.workunit.client.0.smithi063.stdout:6/968: symlink d1/dc/d3a/d82/d85/l138 0 2024-03-23T11:48:03.218 INFO:tasks.workunit.client.0.smithi063.stdout:6/969: write d1/dc/d3a/d82/da2/d41/d42/ffc [54713,96099] 0 2024-03-23T11:48:03.228 INFO:tasks.workunit.client.0.smithi063.stdout:6/970: dread d1/dc/d3a/d57/d6d/d12f/f76 [0,4194304] 0 2024-03-23T11:48:03.230 INFO:tasks.workunit.client.0.smithi063.stdout:6/971: chown d1/dc/d3a/d82/da2/d41/d42/ffc 0 1 2024-03-23T11:48:03.233 INFO:tasks.workunit.client.0.smithi063.stdout:6/972: truncate d1/dc/d3a/d82/f125 128989 0 2024-03-23T11:48:03.236 INFO:tasks.workunit.client.0.smithi063.stdout:6/973: symlink d1/dc/d3a/d57/d6d/ddb/d10e/d130/l139 0 2024-03-23T11:48:03.250 INFO:tasks.workunit.client.0.smithi063.stdout:6/974: dwrite d1/dc/d3a/f4f [0,4194304] 0 2024-03-23T11:48:03.256 INFO:tasks.workunit.client.0.smithi063.stdout:6/975: rename d1/dc/d3a/d82/da2/f93 to d1/dc/d3a/d57/d77/f13a 0 2024-03-23T11:48:03.261 INFO:tasks.workunit.client.0.smithi063.stdout:6/976: mknod d1/dc/d3a/d82/da2/d41/d42/d99/de9/c13b 0 2024-03-23T11:48:03.277 INFO:tasks.workunit.client.0.smithi063.stdout:6/977: dwrite d1/f5 [0,4194304] 0 2024-03-23T11:48:03.280 INFO:tasks.workunit.client.0.smithi063.stdout:6/978: mkdir d1/dc/d3a/d57/d129/d13c 0 2024-03-23T11:48:03.286 INFO:tasks.workunit.client.0.smithi063.stdout:6/979: rename d1/dc/d3a/d57/d6d/d12f/lcc to d1/dc/d3a/d82/da2/d41/l13d 0 2024-03-23T11:48:03.288 INFO:tasks.workunit.client.0.smithi063.stdout:6/980: stat d1/dc/d3a/d82/d85/l138 0 2024-03-23T11:48:03.294 INFO:tasks.workunit.client.0.smithi063.stdout:6/981: link d1/dc/d21/l6c d1/dc/d3a/d82/ddc/d11a/l13e 0 2024-03-23T11:48:03.297 INFO:tasks.workunit.client.0.smithi063.stdout:6/982: unlink d1/lb 0 2024-03-23T11:48:03.300 INFO:tasks.workunit.client.0.smithi063.stdout:6/983: fsync d1/dc/d3a/d57/d6d/f9a 0 2024-03-23T11:48:03.303 INFO:tasks.workunit.client.0.smithi063.stdout:6/984: unlink d1/dc/d3a/d57/df7/l119 0 2024-03-23T11:48:03.306 INFO:tasks.workunit.client.0.smithi063.stdout:6/985: readlink d1/dc/d3a/d82/da2/d41/d42/d99/de9/l12c 0 2024-03-23T11:48:03.311 INFO:tasks.workunit.client.0.smithi063.stdout:6/986: rmdir d1 39 2024-03-23T11:48:03.327 INFO:tasks.workunit.client.0.smithi063.stdout:6/987: dwrite d1/dc/d3a/d82/ddc/f117 [0,4194304] 0 2024-03-23T11:48:03.330 INFO:tasks.workunit.client.0.smithi063.stdout:6/988: chown d1/d15 14186 1 2024-03-23T11:48:03.332 INFO:tasks.workunit.client.0.smithi063.stdout:6/989: stat d1/dc/d3a/d82/d85/df4/f135 0 2024-03-23T11:48:03.334 INFO:tasks.workunit.client.0.smithi063.stdout:6/990: truncate d1/f81 1873804 0 2024-03-23T11:48:03.339 INFO:tasks.workunit.client.0.smithi063.stdout:6/991: dread - d1/dc/d3a/d82/da2/d41/d42/d99/fb0 zero size 2024-03-23T11:48:03.345 INFO:tasks.workunit.client.0.smithi063.stdout:6/992: symlink d1/dc/d3a/d57/d6d/ddb/d10e/l13f 0 2024-03-23T11:48:03.346 INFO:tasks.workunit.client.0.smithi063.stdout:6/993: stat d1/d13/c22 0 2024-03-23T11:48:03.353 INFO:tasks.workunit.client.0.smithi063.stdout:6/994: write d1/dc/fab [1175751,44512] 0 2024-03-23T11:48:03.360 INFO:tasks.workunit.client.0.smithi063.stdout:6/995: dread d1/d13/f1c [0,4194304] 0 2024-03-23T11:48:03.362 INFO:tasks.workunit.client.0.smithi063.stdout:6/996: symlink d1/d15/d4b/l140 0 2024-03-23T11:48:03.364 INFO:tasks.workunit.client.0.smithi063.stdout:6/997: creat d1/de6/f141 x:0 0 0 2024-03-23T11:48:03.365 INFO:tasks.workunit.client.0.smithi063.stdout:6/998: fsync d1/dc/fab 0 2024-03-23T11:48:03.368 INFO:tasks.workunit.client.0.smithi063.stdout:6/999: symlink d1/dc/d3a/l142 0 2024-03-23T11:48:03.504 INFO:tasks.workunit.client.0.smithi063.stdout:5/755: sync 2024-03-23T11:48:03.506 INFO:tasks.workunit.client.0.smithi063.stdout:5/756: write d2/d7/d8/f68 [107628,34445] 0 2024-03-23T11:48:03.512 INFO:tasks.workunit.client.0.smithi063.stdout:5/757: getdents d2/d7/d27 0 2024-03-23T11:48:03.515 INFO:tasks.workunit.client.0.smithi063.stdout:5/758: rmdir d2/d28/d3a/d46 39 2024-03-23T11:48:03.522 INFO:tasks.workunit.client.0.smithi063.stdout:5/759: truncate d2/f6 2051839 0 2024-03-23T11:48:03.524 INFO:tasks.workunit.client.0.smithi063.stdout:5/760: fsync d2/f2c 0 2024-03-23T11:48:03.528 INFO:tasks.workunit.client.0.smithi063.stdout:5/761: unlink d2/f3e 0 2024-03-23T11:48:03.540 INFO:tasks.workunit.client.0.smithi063.stdout:5/762: dread d2/d7/d8/de/d47/fab [0,4194304] 0 2024-03-23T11:48:03.545 INFO:tasks.workunit.client.0.smithi063.stdout:5/763: rename d2/d7/d8/de/d35/c40 to d2/d17/d4d/d6e/de2/cff 0 2024-03-23T11:48:03.548 INFO:tasks.workunit.client.0.smithi063.stdout:5/764: mknod d2/d7/d64/c100 0 2024-03-23T11:48:03.549 INFO:tasks.workunit.client.0.smithi063.stdout:5/765: stat d2/d28/d3a/l48 0 2024-03-23T11:48:03.551 INFO:tasks.workunit.client.0.smithi063.stdout:5/766: rmdir d2/d7/de9/d85 39 2024-03-23T11:48:03.553 INFO:tasks.workunit.client.0.smithi063.stdout:5/767: chown d2/d7/d8/d1d/lce 31 1 2024-03-23T11:48:03.554 INFO:tasks.workunit.client.0.smithi063.stdout:5/768: stat d2/d7/de9/d43/d5c 0 2024-03-23T11:48:03.557 INFO:tasks.workunit.client.0.smithi063.stdout:5/769: creat d2/d17/d4d/d6e/de2/f101 x:0 0 0 2024-03-23T11:48:03.573 INFO:tasks.workunit.client.0.smithi063.stdout:5/770: dwrite d2/d28/d3a/f9b [0,4194304] 0 2024-03-23T11:48:03.580 INFO:tasks.workunit.client.0.smithi063.stdout:5/771: unlink d2/d7/d8/de/d49/f63 0 2024-03-23T11:48:03.583 INFO:tasks.workunit.client.0.smithi063.stdout:5/772: write d2/d28/d3a/f8d [1897766,117397] 0 2024-03-23T11:48:03.586 INFO:tasks.workunit.client.0.smithi063.stdout:5/773: symlink d2/d7/de9/l102 0 2024-03-23T11:48:03.603 INFO:tasks.workunit.client.0.smithi063.stdout:5/774: dread d2/d2f/f3f [0,4194304] 0 2024-03-23T11:48:03.616 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:03 smithi063 ceph-mon[35189]: pgmap v51: 65 pgs: 65 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 75 MiB/s rd, 162 MiB/s wr, 435 op/s 2024-03-23T11:48:03.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:03 smithi111 ceph-mon[44005]: pgmap v51: 65 pgs: 65 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 75 MiB/s rd, 162 MiB/s wr, 435 op/s 2024-03-23T11:48:03.656 INFO:tasks.workunit.client.0.smithi063.stdout:5/775: write d2/d28/d3a/d46/da4/ffa [156055,86027] 0 2024-03-23T11:48:03.662 INFO:tasks.workunit.client.0.smithi063.stdout:5/776: rename d2/d17/d4d/ld1 to d2/d7/d2b/d8b/da7/dc6/l103 0 2024-03-23T11:48:03.664 INFO:tasks.workunit.client.0.smithi063.stdout:5/777: chown d2/d28/d3a/d46/l66 34 1 2024-03-23T11:48:03.667 INFO:tasks.workunit.client.0.smithi063.stdout:5/778: mkdir d2/d7/d8/de/d47/d104 0 2024-03-23T11:48:03.679 INFO:tasks.workunit.client.0.smithi063.stdout:5/779: dwrite f1 [0,4194304] 0 2024-03-23T11:48:03.682 INFO:tasks.workunit.client.0.smithi063.stdout:5/780: write d2/d7/d8/f68 [2059893,113618] 0 2024-03-23T11:48:03.689 INFO:tasks.workunit.client.0.smithi063.stdout:5/781: rename d2/d17/d4d/lbb to d2/d7/d8/de/d2a/l105 0 2024-03-23T11:48:03.690 INFO:tasks.workunit.client.0.smithi063.stdout:5/782: fdatasync d2/d17/d4d/d6e/de2/f101 0 2024-03-23T11:48:03.696 INFO:tasks.workunit.client.0.smithi063.stdout:5/783: mknod d2/d7/d8/de/d49/db0/c106 0 2024-03-23T11:48:03.701 INFO:tasks.workunit.client.0.smithi063.stdout:5/784: creat d2/d28/d3a/f107 x:0 0 0 2024-03-23T11:48:03.714 INFO:tasks.workunit.client.0.smithi063.stdout:5/785: dwrite d2/d7/de9/f73 [0,4194304] 0 2024-03-23T11:48:03.718 INFO:tasks.workunit.client.0.smithi063.stdout:5/786: write d2/d28/d3a/d46/d79/fee [762067,33770] 0 2024-03-23T11:48:03.722 INFO:tasks.workunit.client.0.smithi063.stdout:5/787: chown d2/d28/d3a/d46/d71/d9d/ff6 12197882 1 2024-03-23T11:48:03.743 INFO:tasks.workunit.client.0.smithi063.stdout:5/788: dread d2/d7/f10 [0,4194304] 0 2024-03-23T11:48:03.745 INFO:tasks.workunit.client.0.smithi063.stdout:5/789: rename d2 to d2/d7/d8/de/d35/d108 22 2024-03-23T11:48:03.763 INFO:tasks.workunit.client.0.smithi063.stdout:5/790: dwrite d2/d7/fbd [0,4194304] 0 2024-03-23T11:48:03.783 INFO:tasks.workunit.client.0.smithi063.stdout:5/791: dwrite d2/d17/d4d/d6e/f76 [0,4194304] 0 2024-03-23T11:48:03.788 INFO:tasks.workunit.client.0.smithi063.stdout:5/792: rename d2/d28/de6/ff2 to d2/d28/ddf/f109 0 2024-03-23T11:48:03.790 INFO:tasks.workunit.client.0.smithi063.stdout:5/793: mkdir d2/d7/d2b/d8b/d10a 0 2024-03-23T11:48:03.793 INFO:tasks.workunit.client.0.smithi063.stdout:5/794: mknod d2/d7/d8/de/d49/c10b 0 2024-03-23T11:48:03.865 INFO:tasks.workunit.client.0.smithi063.stdout:5/795: creat d2/d7/d64/f10c x:0 0 0 2024-03-23T11:48:03.868 INFO:tasks.workunit.client.0.smithi063.stdout:5/796: chown d2/d7/d8/f13 235241 1 2024-03-23T11:48:03.925 INFO:tasks.workunit.client.0.smithi063.stdout:5/797: write d2/d7/d8/de/d35/f60 [1533263,27403] 0 2024-03-23T11:48:03.932 INFO:tasks.workunit.client.0.smithi063.stdout:5/798: mkdir d2/d7/de9/d10d 0 2024-03-23T11:48:03.935 INFO:tasks.workunit.client.0.smithi063.stdout:5/799: creat d2/d7/d8/de/d35/f10e x:0 0 0 2024-03-23T11:48:03.937 INFO:tasks.workunit.client.0.smithi063.stdout:5/800: readlink d2/d7/d8/de/l8a 0 2024-03-23T11:48:03.940 INFO:tasks.workunit.client.0.smithi063.stdout:5/801: chown d2/d28/d3a/d46/d79/fee 83971026 1 2024-03-23T11:48:03.942 INFO:tasks.workunit.client.0.smithi063.stdout:5/802: chown d2/d7/d8/de/d49/db0/ffd 5902380 1 2024-03-23T11:48:03.951 INFO:tasks.workunit.client.0.smithi063.stdout:5/803: link d2/d28/d3a/d46/da4/ffa d2/d7/d8/f10f 0 2024-03-23T11:48:03.959 INFO:tasks.workunit.client.0.smithi063.stdout:5/804: truncate d2/d7/d8/de/d47/fab 1549923 0 2024-03-23T11:48:03.962 INFO:tasks.workunit.client.0.smithi063.stdout:5/805: truncate d2/d7/d8/de/d35/ff7 738091 0 2024-03-23T11:48:03.965 INFO:tasks.workunit.client.0.smithi063.stdout:5/806: mkdir d2/d7/d8/de/d2a/d110 0 2024-03-23T11:48:03.969 INFO:tasks.workunit.client.0.smithi063.stdout:5/807: creat d2/d28/d3a/d46/d79/f111 x:0 0 0 2024-03-23T11:48:03.995 INFO:tasks.workunit.client.0.smithi063.stdout:5/808: dwrite d2/d7/d2b/d8b/da7/dc6/fda [0,4194304] 0 2024-03-23T11:48:04.003 INFO:tasks.workunit.client.0.smithi063.stdout:5/809: creat d2/d7/d8/de/d35/f112 x:0 0 0 2024-03-23T11:48:04.023 INFO:tasks.workunit.client.0.smithi063.stdout:5/810: dwrite d2/d7/d2b/f39 [0,4194304] 0 2024-03-23T11:48:04.029 INFO:tasks.workunit.client.0.smithi063.stdout:5/811: mkdir d2/d7/d8/de/d2a/d110/d113 0 2024-03-23T11:48:04.033 INFO:tasks.workunit.client.0.smithi063.stdout:5/812: rmdir d2/d7/d8/de/d35/da6 39 2024-03-23T11:48:04.038 INFO:tasks.workunit.client.0.smithi063.stdout:5/813: getdents d2/d7/de9/d43 0 2024-03-23T11:48:04.041 INFO:tasks.workunit.client.0.smithi063.stdout:5/814: mkdir d2/d7/d2b/d8b/d10a/d114 0 2024-03-23T11:48:04.057 INFO:tasks.workunit.client.0.smithi063.stdout:5/815: dread d2/d7/d8/f10f [0,4194304] 0 2024-03-23T11:48:04.066 INFO:tasks.workunit.client.0.smithi063.stdout:5/816: link d2/d7/d64/f10c d2/d7/d2b/d8b/d10a/d114/f115 0 2024-03-23T11:48:04.068 INFO:tasks.workunit.client.0.smithi063.stdout:5/817: chown d2/d28/d3a/f9b 244741053 1 2024-03-23T11:48:04.070 INFO:tasks.workunit.client.0.smithi063.stdout:5/818: dread - d2/d7/fa0 zero size 2024-03-23T11:48:04.074 INFO:tasks.workunit.client.0.smithi063.stdout:5/819: creat d2/d7/de9/d43/d5c/dc2/f116 x:0 0 0 2024-03-23T11:48:04.079 INFO:tasks.workunit.client.0.smithi063.stdout:5/820: truncate d2/d28/fc9 198475 0 2024-03-23T11:48:04.092 INFO:tasks.workunit.client.0.smithi063.stdout:5/821: dread d2/d17/f51 [0,4194304] 0 2024-03-23T11:48:04.094 INFO:tasks.workunit.client.0.smithi063.stdout:5/822: write d2/fc [4816956,17562] 0 2024-03-23T11:48:04.111 INFO:tasks.workunit.client.0.smithi063.stdout:5/823: dwrite d2/f52 [0,4194304] 0 2024-03-23T11:48:04.114 INFO:tasks.workunit.client.0.smithi063.stdout:5/824: chown d2/d7/f1a 2015910479 1 2024-03-23T11:48:04.129 INFO:tasks.workunit.client.0.smithi063.stdout:5/825: dwrite d2/f52 [0,4194304] 0 2024-03-23T11:48:04.137 INFO:tasks.workunit.client.0.smithi063.stdout:5/826: write d2/f92 [245212,94679] 0 2024-03-23T11:48:04.139 INFO:tasks.workunit.client.0.smithi063.stdout:5/827: write d2/d7/ff0 [845505,38570] 0 2024-03-23T11:48:04.158 INFO:tasks.workunit.client.0.smithi063.stdout:5/828: dwrite d2/d7/d27/f83 [0,4194304] 0 2024-03-23T11:48:04.163 INFO:tasks.workunit.client.0.smithi063.stdout:5/829: creat d2/d7/d8/de/d35/da6/f117 x:0 0 0 2024-03-23T11:48:04.168 INFO:tasks.workunit.client.0.smithi063.stdout:5/830: creat d2/d7/de9/f118 x:0 0 0 2024-03-23T11:48:04.170 INFO:tasks.workunit.client.0.smithi063.stdout:5/831: fdatasync d2/d7/d8/de/d47/fe7 0 2024-03-23T11:48:04.175 INFO:tasks.workunit.client.0.smithi063.stdout:5/832: write d2/d28/f3d [3840230,42552] 0 2024-03-23T11:48:04.182 INFO:tasks.workunit.client.0.smithi063.stdout:5/833: rename d2/d28/d3a/d46/d79 to d2/d17/d4d/d119 0 2024-03-23T11:48:04.759 INFO:tasks.workunit.client.0.smithi063.stdout:5/834: sync 2024-03-23T11:48:04.761 INFO:tasks.workunit.client.0.smithi063.stdout:5/835: dread - d2/d7/d8/de/d35/fcd zero size 2024-03-23T11:48:04.798 INFO:tasks.workunit.client.0.smithi063.stdout:5/836: dwrite d2/d7/fa0 [0,4194304] 0 2024-03-23T11:48:04.799 INFO:tasks.workunit.client.0.smithi063.stdout:5/837: read d2/d7/d64/f87 [132497,62401] 0 2024-03-23T11:48:04.816 INFO:tasks.workunit.client.0.smithi063.stdout:5/838: dwrite d2/d7/d8/de/d35/f60 [0,4194304] 0 2024-03-23T11:48:04.822 INFO:tasks.workunit.client.0.smithi063.stdout:5/839: dread d2/d17/f34 [0,4194304] 0 2024-03-23T11:48:04.836 INFO:tasks.workunit.client.0.smithi063.stdout:5/840: dread d2/f6 [0,4194304] 0 2024-03-23T11:48:04.839 INFO:tasks.workunit.client.0.smithi063.stdout:5/841: dread - d2/d7/d8/de/d49/fde zero size 2024-03-23T11:48:04.843 INFO:tasks.workunit.client.0.smithi063.stdout:5/842: getdents d2/d17/d4d/d6e 0 2024-03-23T11:48:04.845 INFO:tasks.workunit.client.0.smithi063.stdout:5/843: chown d2/d7/d64/f87 21 1 2024-03-23T11:48:04.849 INFO:tasks.workunit.client.0.smithi063.stdout:5/844: creat d2/d28/d3a/d46/da4/f11a x:0 0 0 2024-03-23T11:48:04.853 INFO:tasks.workunit.client.0.smithi063.stdout:5/845: write d2/d17/f34 [1273310,89609] 0 2024-03-23T11:48:04.857 INFO:tasks.workunit.client.0.smithi063.stdout:5/846: mknod d2/d7/d2b/d8b/da7/c11b 0 2024-03-23T11:48:04.858 INFO:tasks.workunit.client.0.smithi063.stdout:5/847: chown d2/d17/d4d/d119/fee 986289803 1 2024-03-23T11:48:04.860 INFO:tasks.workunit.client.0.smithi063.stdout:5/848: write d2/d7/d8/f68 [3100946,38667] 0 2024-03-23T11:48:04.880 INFO:tasks.workunit.client.0.smithi063.stdout:5/849: dread d2/f5 [0,4194304] 0 2024-03-23T11:48:04.886 INFO:tasks.workunit.client.0.smithi063.stdout:5/850: getdents d2/d7/d8/de/d49/db0 0 2024-03-23T11:48:05.187 INFO:tasks.workunit.client.0.smithi063.stdout:5/851: sync 2024-03-23T11:48:05.190 INFO:tasks.workunit.client.0.smithi063.stdout:5/852: write d2/d7/d8/de/d35/f60 [273849,78140] 0 2024-03-23T11:48:05.196 INFO:tasks.workunit.client.0.smithi063.stdout:5/853: getdents d2 0 2024-03-23T11:48:05.218 INFO:tasks.workunit.client.0.smithi063.stdout:5/854: dread d2/d28/d3a/fa5 [0,4194304] 0 2024-03-23T11:48:05.220 INFO:tasks.workunit.client.0.smithi063.stdout:5/855: fsync d2/d7/f23 0 2024-03-23T11:48:05.236 INFO:tasks.workunit.client.0.smithi063.stdout:5/856: dwrite d2/d7/de9/d43/d5c/dc2/f116 [0,4194304] 0 2024-03-23T11:48:05.239 INFO:tasks.workunit.client.0.smithi063.stdout:5/857: fdatasync d2/d7/fbd 0 2024-03-23T11:48:05.241 INFO:tasks.workunit.client.0.smithi063.stdout:5/858: chown d2/d7/de9/f118 345115 1 2024-03-23T11:48:05.245 INFO:tasks.workunit.client.0.smithi063.stdout:5/859: mkdir d2/d7/d8/de/d47/d11c 0 2024-03-23T11:48:05.252 INFO:tasks.workunit.client.0.smithi063.stdout:5/860: getdents d2/d7/d2b/d8b/da7/dc6 0 2024-03-23T11:48:05.255 INFO:tasks.workunit.client.0.smithi063.stdout:5/861: rmdir d2/d7/de9/d43/d5c/dc2 39 2024-03-23T11:48:05.257 INFO:tasks.workunit.client.0.smithi063.stdout:5/862: chown d2/d28/d3a/c4a 204399 1 2024-03-23T11:48:05.260 INFO:tasks.workunit.client.0.smithi063.stdout:5/863: creat d2/d7/d2b/d8b/daf/f11d x:0 0 0 2024-03-23T11:48:05.268 INFO:tasks.workunit.client.0.smithi063.stdout:5/864: truncate d2/d28/d3a/f9b 2023404 0 2024-03-23T11:48:05.282 INFO:tasks.workunit.client.0.smithi063.stdout:5/865: dwrite d2/d7/fa0 [0,4194304] 0 2024-03-23T11:48:05.284 INFO:tasks.workunit.client.0.smithi063.stdout:5/866: dread - d2/d28/d3a/f107 zero size 2024-03-23T11:48:05.290 INFO:tasks.workunit.client.0.smithi063.stdout:5/867: symlink d2/d7/de9/d85/l11e 0 2024-03-23T11:48:05.293 INFO:tasks.workunit.client.0.smithi063.stdout:5/868: mkdir d2/d7/d27/d11f 0 2024-03-23T11:48:05.295 INFO:tasks.workunit.client.0.smithi063.stdout:5/869: fsync d2/d7/d8/de/d35/fcd 0 2024-03-23T11:48:05.297 INFO:tasks.workunit.client.0.smithi063.stdout:5/870: read - d2/d17/d4d/d6e/de2/f101 zero size 2024-03-23T11:48:05.303 INFO:tasks.workunit.client.0.smithi063.stdout:5/871: rename d2/f44 to d2/d17/d4d/d6e/f120 0 2024-03-23T11:48:05.318 INFO:tasks.workunit.client.0.smithi063.stdout:5/872: dwrite d2/d7/de9/d85/f9e [0,4194304] 0 2024-03-23T11:48:05.333 INFO:tasks.workunit.client.0.smithi063.stdout:5/873: dwrite d2/d7/de9/d85/f9e [4194304,4194304] 0 2024-03-23T11:48:05.365 INFO:tasks.workunit.client.0.smithi063.stdout:5/874: dread d2/d7/f14 [0,4194304] 0 2024-03-23T11:48:05.376 INFO:tasks.workunit.client.0.smithi063.stdout:5/875: dread d2/d17/f51 [0,4194304] 0 2024-03-23T11:48:05.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:05 smithi063 ceph-mon[35189]: pgmap v52: 65 pgs: 65 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 61 MiB/s rd, 134 MiB/s wr, 371 op/s 2024-03-23T11:48:05.387 INFO:tasks.workunit.client.0.smithi063.stdout:5/876: dread d2/d28/d3a/d46/f86 [0,4194304] 0 2024-03-23T11:48:05.392 INFO:tasks.workunit.client.0.smithi063.stdout:5/877: link d2/d28/d3a/cdc d2/d28/de6/c121 0 2024-03-23T11:48:05.396 INFO:tasks.workunit.client.0.smithi063.stdout:5/878: creat d2/d7/f122 x:0 0 0 2024-03-23T11:48:05.400 INFO:tasks.workunit.client.0.smithi063.stdout:5/879: write d2/d7/de9/d43/d5c/dc2/f116 [2688950,107541] 0 2024-03-23T11:48:05.404 INFO:tasks.workunit.client.0.smithi063.stdout:5/880: write d2/d17/f51 [7055840,99953] 0 2024-03-23T11:48:05.418 INFO:tasks.workunit.client.0.smithi063.stdout:5/881: dwrite d2/f92 [0,4194304] 0 2024-03-23T11:48:05.421 INFO:tasks.workunit.client.0.smithi063.stdout:5/882: mkdir d2/d17/d4d/dfb/d123 0 2024-03-23T11:48:05.440 INFO:tasks.workunit.client.0.smithi063.stdout:5/883: dwrite d2/d7/d8/de/d35/f60 [0,4194304] 0 2024-03-23T11:48:05.444 INFO:tasks.workunit.client.0.smithi063.stdout:5/884: symlink d2/d17/d4d/dfb/d123/l124 0 2024-03-23T11:48:05.450 INFO:tasks.workunit.client.0.smithi063.stdout:5/885: unlink d2/d7/d8/c54 0 2024-03-23T11:48:05.452 INFO:tasks.workunit.client.0.smithi063.stdout:5/886: chown d2/d28/d3a/f107 3867 1 2024-03-23T11:48:05.462 INFO:tasks.workunit.client.0.smithi063.stdout:5/887: write d2/d2f/fed [2259988,82613] 0 2024-03-23T11:48:05.466 INFO:tasks.workunit.client.0.smithi063.stdout:5/888: dread d2/d7/de9/d43/f5a [0,4194304] 0 2024-03-23T11:48:05.471 INFO:tasks.workunit.client.0.smithi063.stdout:5/889: read d2/d7/d8/d1d/f1f [1481810,43014] 0 2024-03-23T11:48:05.474 INFO:tasks.workunit.client.0.smithi063.stdout:5/890: symlink d2/d17/d4d/l125 0 2024-03-23T11:48:05.476 INFO:tasks.workunit.client.0.smithi063.stdout:5/891: read d2/d7/d2b/d8b/da7/dc6/fda [3521076,123709] 0 2024-03-23T11:48:05.491 INFO:tasks.workunit.client.0.smithi063.stdout:5/892: dwrite d2/d7/d8/de/d49/db0/fcc [0,4194304] 0 2024-03-23T11:48:05.493 INFO:tasks.workunit.client.0.smithi063.stdout:5/893: chown d2/d28/lc3 201816703 1 2024-03-23T11:48:05.496 INFO:tasks.workunit.client.0.smithi063.stdout:5/894: mknod d2/d28/d3a/d46/da4/c126 0 2024-03-23T11:48:05.498 INFO:tasks.workunit.client.0.smithi063.stdout:5/895: symlink d2/d28/l127 0 2024-03-23T11:48:05.506 INFO:tasks.workunit.client.0.smithi063.stdout:5/896: write d2/d7/d64/fa2 [437116,128440] 0 2024-03-23T11:48:05.510 INFO:tasks.workunit.client.0.smithi063.stdout:5/897: mknod d2/d17/d4d/d6e/de2/c128 0 2024-03-23T11:48:05.511 INFO:tasks.workunit.client.0.smithi063.stdout:5/898: chown d2/d28/d3a/d46/d71 585 1 2024-03-23T11:48:05.516 INFO:tasks.workunit.client.0.smithi063.stdout:5/899: creat d2/d7/d2b/d8b/d10a/f129 x:0 0 0 2024-03-23T11:48:05.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:05 smithi111 ceph-mon[44005]: pgmap v52: 65 pgs: 65 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 61 MiB/s rd, 134 MiB/s wr, 371 op/s 2024-03-23T11:48:06.023 INFO:tasks.workunit.client.0.smithi063.stdout:5/900: sync 2024-03-23T11:48:06.026 INFO:tasks.workunit.client.0.smithi063.stdout:5/901: read d2/d7/d8/de/d49/db0/fcc [1102945,2509] 0 2024-03-23T11:48:06.068 INFO:tasks.workunit.client.0.smithi063.stdout:5/902: dread d2/d7/d8/de/d35/f6c [0,4194304] 0 2024-03-23T11:48:06.072 INFO:tasks.workunit.client.0.smithi063.stdout:5/903: mkdir d2/d17/d4d/d6e/dba/dbe/d12a 0 2024-03-23T11:48:06.084 INFO:tasks.workunit.client.0.smithi063.stdout:5/904: write d2/d28/f4b [404374,121863] 0 2024-03-23T11:48:06.088 INFO:tasks.workunit.client.0.smithi063.stdout:5/905: creat d2/d28/f12b x:0 0 0 2024-03-23T11:48:06.091 INFO:tasks.workunit.client.0.smithi063.stdout:5/906: fsync d2/d28/fc9 0 2024-03-23T11:48:06.110 INFO:tasks.workunit.client.0.smithi063.stdout:5/907: dwrite d2/d7/d8/d1d/f1f [4194304,4194304] 0 2024-03-23T11:48:06.112 INFO:tasks.workunit.client.0.smithi063.stdout:5/908: readlink d2/d7/de9/l102 0 2024-03-23T11:48:06.114 INFO:tasks.workunit.client.0.smithi063.stdout:5/909: mkdir d2/d7/de9/d43/d12c 0 2024-03-23T11:48:06.125 INFO:tasks.workunit.client.0.smithi063.stdout:5/910: dread d2/d7/f1a [0,4194304] 0 2024-03-23T11:48:06.129 INFO:tasks.workunit.client.0.smithi063.stdout:5/911: creat d2/d7/d2b/d8b/da7/f12d x:0 0 0 2024-03-23T11:48:06.131 INFO:tasks.workunit.client.0.smithi063.stdout:5/912: creat d2/d17/d4d/dfb/f12e x:0 0 0 2024-03-23T11:48:06.137 INFO:tasks.workunit.client.0.smithi063.stdout:5/913: mkdir d2/d7/de9/d43/d12f 0 2024-03-23T11:48:06.141 INFO:tasks.workunit.client.0.smithi063.stdout:5/914: creat d2/d28/d3a/d46/da4/f130 x:0 0 0 2024-03-23T11:48:06.143 INFO:tasks.workunit.client.0.smithi063.stdout:5/915: dread - d2/d7/de9/d43/ffe zero size 2024-03-23T11:48:06.148 INFO:tasks.workunit.client.0.smithi063.stdout:5/916: write d2/f5 [2429195,98909] 0 2024-03-23T11:48:06.152 INFO:tasks.workunit.client.0.smithi063.stdout:5/917: symlink d2/d7/d27/d11f/l131 0 2024-03-23T11:48:06.166 INFO:tasks.workunit.client.0.smithi063.stdout:5/918: dwrite d2/d2f/fc7 [0,4194304] 0 2024-03-23T11:48:06.168 INFO:tasks.workunit.client.0.smithi063.stdout:5/919: fdatasync d2/d7/de9/f118 0 2024-03-23T11:48:06.171 INFO:tasks.workunit.client.0.smithi063.stdout:5/920: creat d2/d7/d8/de/d47/f132 x:0 0 0 2024-03-23T11:48:06.175 INFO:tasks.workunit.client.0.smithi063.stdout:5/921: mknod d2/d28/d3a/d46/d71/d9f/c133 0 2024-03-23T11:48:06.178 INFO:tasks.workunit.client.0.smithi063.stdout:5/922: rmdir d2/d7/d8/de/d2a/d110 39 2024-03-23T11:48:06.193 INFO:tasks.workunit.client.0.smithi063.stdout:5/923: dwrite d2/d7/d2b/d8b/da7/f12d [0,4194304] 0 2024-03-23T11:48:06.206 INFO:tasks.workunit.client.0.smithi063.stdout:5/924: dread d2/d7/d8/d1d/f57 [0,4194304] 0 2024-03-23T11:48:06.210 INFO:tasks.workunit.client.0.smithi063.stdout:5/925: mknod d2/d7/d8/de/d47/d104/c134 0 2024-03-23T11:48:06.216 INFO:tasks.workunit.client.0.smithi063.stdout:5/926: mknod d2/d7/d2b/d8b/d10a/d114/c135 0 2024-03-23T11:48:06.223 INFO:tasks.workunit.client.0.smithi063.stdout:5/927: getdents d2/d7 0 2024-03-23T11:48:06.226 INFO:tasks.workunit.client.0.smithi063.stdout:5/928: chown d2/d7/d2b/d8b/da7/f12d 25 1 2024-03-23T11:48:06.236 INFO:tasks.workunit.client.0.smithi063.stdout:5/929: dread d2/d7/f10 [0,4194304] 0 2024-03-23T11:48:06.240 INFO:tasks.workunit.client.0.smithi063.stdout:5/930: chown d2/d7/d8/de/cb9 164259801 1 2024-03-23T11:48:06.257 INFO:tasks.workunit.client.0.smithi063.stdout:5/931: dwrite d2/d7/d2b/d8b/da7/fd5 [0,4194304] 0 2024-03-23T11:48:06.264 INFO:tasks.workunit.client.0.smithi063.stdout:5/932: creat d2/d7/d2b/d8b/daf/f136 x:0 0 0 2024-03-23T11:48:06.273 INFO:tasks.workunit.client.0.smithi063.stdout:5/933: dread d2/d7/f10 [0,4194304] 0 2024-03-23T11:48:06.278 INFO:tasks.workunit.client.0.smithi063.stdout:5/934: unlink d2/d7/de9/d43/d5c/f7d 0 2024-03-23T11:48:06.281 INFO:tasks.workunit.client.0.smithi063.stdout:5/935: symlink d2/d7/l137 0 2024-03-23T11:48:06.292 INFO:tasks.workunit.client.0.smithi063.stdout:5/936: creat d2/d7/de9/d43/f138 x:0 0 0 2024-03-23T11:48:06.639 INFO:tasks.workunit.client.0.smithi063.stdout:5/937: sync 2024-03-23T11:48:06.653 INFO:tasks.workunit.client.0.smithi063.stdout:5/938: dwrite d2/d17/f51 [4194304,4194304] 0 2024-03-23T11:48:06.660 INFO:tasks.workunit.client.0.smithi063.stdout:5/939: creat d2/d7/d8/de/d2a/f139 x:0 0 0 2024-03-23T11:48:06.676 INFO:tasks.workunit.client.0.smithi063.stdout:5/940: dwrite d2/fc [0,4194304] 0 2024-03-23T11:48:06.680 INFO:tasks.workunit.client.0.smithi063.stdout:5/941: rmdir d2/d7/d27 39 2024-03-23T11:48:06.685 INFO:tasks.workunit.client.0.smithi063.stdout:5/942: write d2/d7/de9/f73 [4833082,130137] 0 2024-03-23T11:48:06.688 INFO:tasks.workunit.client.0.smithi063.stdout:5/943: stat d2/d7/de9/f73 0 2024-03-23T11:48:06.695 INFO:tasks.workunit.client.0.smithi063.stdout:5/944: rename d2/d7/d8/f20 to d2/d17/d4d/d119/dc0/f13a 0 2024-03-23T11:48:06.698 INFO:tasks.workunit.client.0.smithi063.stdout:5/945: fsync d2/d7/d27/f83 0 2024-03-23T11:48:06.702 INFO:tasks.workunit.client.0.smithi063.stdout:5/946: mknod d2/d7/d2b/d8b/c13b 0 2024-03-23T11:48:06.707 INFO:tasks.workunit.client.0.smithi063.stdout:5/947: getdents d2/d7/de9/d85 0 2024-03-23T11:48:06.713 INFO:tasks.workunit.client.0.smithi063.stdout:5/948: getdents d2/d7/d8/de/d49 0 2024-03-23T11:48:06.719 INFO:tasks.workunit.client.0.smithi063.stdout:5/949: creat d2/d7/d8/de/d2a/d110/f13c x:0 0 0 2024-03-23T11:48:06.725 INFO:tasks.workunit.client.0.smithi063.stdout:5/950: symlink d2/d7/d27/d11f/l13d 0 2024-03-23T11:48:07.033 INFO:tasks.workunit.client.0.smithi063.stdout:5/951: sync 2024-03-23T11:48:07.034 INFO:tasks.workunit.client.0.smithi063.stdout:5/952: truncate f1 4635077 0 2024-03-23T11:48:07.044 INFO:tasks.workunit.client.0.smithi063.stdout:5/953: getdents d2/d7/d8/de/d47/d104 0 2024-03-23T11:48:07.053 INFO:tasks.workunit.client.0.smithi063.stdout:5/954: getdents d2/d17/d4d/d119/dc0 0 2024-03-23T11:48:07.057 INFO:tasks.workunit.client.0.smithi063.stdout:5/955: read d2/d7/d8/de/d35/f60 [3724140,47845] 0 2024-03-23T11:48:07.073 INFO:tasks.workunit.client.0.smithi063.stdout:5/956: dread d2/d7/de9/f73 [0,4194304] 0 2024-03-23T11:48:07.078 INFO:tasks.workunit.client.0.smithi063.stdout:5/957: write d2/d7/d8/de/d2a/d110/f13c [834826,86886] 0 2024-03-23T11:48:07.085 INFO:tasks.workunit.client.0.smithi063.stdout:5/958: mknod d2/d28/d3a/d46/c13e 0 2024-03-23T11:48:07.090 INFO:tasks.workunit.client.0.smithi063.stdout:5/959: mknod d2/d7/d8/de/c13f 0 2024-03-23T11:48:07.095 INFO:tasks.workunit.client.0.smithi063.stdout:5/960: chown d2/d7/d8/de/d2a/d110/d113 1003365 1 2024-03-23T11:48:07.096 INFO:tasks.workunit.client.0.smithi063.stdout:5/961: read - d2/d17/d4d/dfb/f12e zero size 2024-03-23T11:48:07.110 INFO:tasks.workunit.client.0.smithi063.stdout:5/962: write d2/f7f [816459,70973] 0 2024-03-23T11:48:07.112 INFO:tasks.workunit.client.0.smithi063.stdout:5/963: chown d2/d7/de9/l102 95665 1 2024-03-23T11:48:07.121 INFO:tasks.workunit.client.0.smithi063.stdout:5/964: link d2/d28/d3a/l6d d2/d7/l140 0 2024-03-23T11:48:07.123 INFO:tasks.workunit.client.0.smithi063.stdout:5/965: truncate d2/d2f/f70 4840177 0 2024-03-23T11:48:07.126 INFO:tasks.workunit.client.0.smithi063.stdout:5/966: unlink d2/d28/d3a/f107 0 2024-03-23T11:48:07.128 INFO:tasks.workunit.client.0.smithi063.stdout:5/967: rmdir d2/d7/d2b/d8b/d10a/d114 39 2024-03-23T11:48:07.131 INFO:tasks.workunit.client.0.smithi063.stdout:5/968: mkdir d2/d17/d4d/d6e/d141 0 2024-03-23T11:48:07.134 INFO:tasks.workunit.client.0.smithi063.stdout:5/969: creat d2/d28/d3a/d46/d71/f142 x:0 0 0 2024-03-23T11:48:07.158 INFO:tasks.workunit.client.0.smithi063.stdout:5/970: dread d2/d28/d3a/d46/d71/d9f/fa1 [0,4194304] 0 2024-03-23T11:48:07.169 INFO:tasks.workunit.client.0.smithi063.stdout:5/971: truncate d2/d7/f1a 1374429 0 2024-03-23T11:48:07.177 INFO:tasks.workunit.client.0.smithi063.stdout:5/972: rename d2/d28/d3a/fa5 to d2/d7/d8/de/d35/f143 0 2024-03-23T11:48:07.179 INFO:tasks.workunit.client.0.smithi063.stdout:5/973: fdatasync d2/d7/d8/d1d/f57 0 2024-03-23T11:48:07.182 INFO:tasks.workunit.client.0.smithi063.stdout:5/974: write d2/d7/fbd [5223521,16863] 0 2024-03-23T11:48:07.201 INFO:tasks.workunit.client.0.smithi063.stdout:5/975: dwrite d2/d7/d8/d1d/fd0 [0,4194304] 0 2024-03-23T11:48:07.206 INFO:tasks.workunit.client.0.smithi063.stdout:5/976: mknod d2/d7/d8/d1d/c144 0 2024-03-23T11:48:07.210 INFO:tasks.workunit.client.0.smithi063.stdout:5/977: fdatasync d2/d28/ddf/f109 0 2024-03-23T11:48:07.225 INFO:tasks.workunit.client.0.smithi063.stdout:5/978: dwrite d2/d17/f51 [0,4194304] 0 2024-03-23T11:48:07.226 INFO:tasks.workunit.client.0.smithi063.stdout:5/979: stat d2/d7/d8/de/d2a/d110 0 2024-03-23T11:48:07.245 INFO:tasks.workunit.client.0.smithi063.stdout:5/980: dwrite d2/d7/d8/de/d47/fe7 [0,4194304] 0 2024-03-23T11:48:07.257 INFO:tasks.workunit.client.0.smithi063.stdout:5/981: write d2/d7/f14 [897851,120599] 0 2024-03-23T11:48:07.264 INFO:tasks.workunit.client.0.smithi063.stdout:5/982: link d2/d7/d8/d1d/cc4 d2/d7/de9/d43/d12c/c145 0 2024-03-23T11:48:07.265 INFO:tasks.workunit.client.0.smithi063.stdout:5/983: stat d2/d7/l137 0 2024-03-23T11:48:07.282 INFO:tasks.workunit.client.0.smithi063.stdout:5/984: dwrite d2/d7/f122 [0,4194304] 0 2024-03-23T11:48:07.283 INFO:tasks.workunit.client.0.smithi063.stdout:5/985: stat d2/d7/d27/d11f/l13d 0 2024-03-23T11:48:07.594 INFO:tasks.workunit.client.0.smithi063.stdout:5/986: sync 2024-03-23T11:48:07.605 INFO:tasks.workunit.client.0.smithi063.stdout:5/987: dread d2/d7/d8/de/d35/f143 [0,4194304] 0 2024-03-23T11:48:07.625 INFO:tasks.workunit.client.0.smithi063.stdout:5/988: creat d2/d7/d8/f146 x:0 0 0 2024-03-23T11:48:07.629 INFO:tasks.workunit.client.0.smithi063.stdout:5/989: rename d2/d17/d4d/d6e/fdb to d2/d17/d4d/dfb/f147 0 2024-03-23T11:48:07.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:07 smithi063 ceph-mon[35189]: pgmap v53: 65 pgs: 65 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 45 MiB/s rd, 92 MiB/s wr, 286 op/s 2024-03-23T11:48:07.648 INFO:tasks.workunit.client.0.smithi063.stdout:5/990: dread d2/d7/d27/f94 [0,4194304] 0 2024-03-23T11:48:07.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:07 smithi111 ceph-mon[44005]: pgmap v53: 65 pgs: 65 active+clean; 3.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 45 MiB/s rd, 92 MiB/s wr, 286 op/s 2024-03-23T11:48:07.662 INFO:tasks.workunit.client.0.smithi063.stdout:5/991: dwrite d2/d28/f12b [0,4194304] 0 2024-03-23T11:48:07.667 INFO:tasks.workunit.client.0.smithi063.stdout:5/992: creat d2/d7/d8/de/d49/db0/dca/f148 x:0 0 0 2024-03-23T11:48:07.674 INFO:tasks.workunit.client.0.smithi063.stdout:5/993: fsync d2/d7/d2b/d8b/d10a/d114/f115 0 2024-03-23T11:48:07.675 INFO:tasks.workunit.client.0.smithi063.stdout:5/994: chown d2/d28/d3a 12 1 2024-03-23T11:48:07.693 INFO:tasks.workunit.client.0.smithi063.stdout:5/995: dwrite d2/d17/f34 [0,4194304] 0 2024-03-23T11:48:07.699 INFO:tasks.workunit.client.0.smithi063.stdout:5/996: truncate d2/f2c 1780164 0 2024-03-23T11:48:07.707 INFO:tasks.workunit.client.0.smithi063.stdout:5/997: rename d2/d28/d3a/f9b to d2/d7/de9/d43/d5c/dc2/f149 0 2024-03-23T11:48:07.710 INFO:tasks.workunit.client.0.smithi063.stdout:5/998: symlink d2/d7/d8/de/d47/l14a 0 2024-03-23T11:48:07.711 INFO:tasks.workunit.client.0.smithi063.stdout:5/999: chown d2/d28/f12b 1325 1 2024-03-23T11:48:07.720 INFO:tasks.workunit.client.0.smithi063.stderr:+ rm -rf -- ./tmp.3LUi8G1z5V 2024-03-23T11:48:09.573 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:09 smithi063 ceph-mon[35189]: pgmap v54: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 49 MiB/s rd, 106 MiB/s wr, 369 op/s 2024-03-23T11:48:09.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:09 smithi111 ceph-mon[44005]: pgmap v54: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 49 MiB/s rd, 106 MiB/s wr, 369 op/s 2024-03-23T11:48:11.031 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:10 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:11.031 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:10 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:11.031 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:10 smithi111 ceph-mon[44005]: Reconfiguring node-exporter.smithi111 (dependencies changed)... 2024-03-23T11:48:11.031 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:10 smithi111 ceph-mon[44005]: Reconfiguring daemon node-exporter.smithi111 on smithi111 2024-03-23T11:48:11.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:10 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:11.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:10 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:11.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:10 smithi063 ceph-mon[35189]: Reconfiguring node-exporter.smithi111 (dependencies changed)... 2024-03-23T11:48:11.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:10 smithi063 ceph-mon[35189]: Reconfiguring daemon node-exporter.smithi111 on smithi111 2024-03-23T11:48:11.782 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:11 smithi111 ceph-mon[44005]: pgmap v55: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 20 MiB/s rd, 49 MiB/s wr, 205 op/s 2024-03-23T11:48:11.782 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:11 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:12.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:11 smithi063 ceph-mon[35189]: pgmap v55: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 20 MiB/s rd, 49 MiB/s wr, 205 op/s 2024-03-23T11:48:12.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:11 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: pgmap v56: 65 pgs: 65 active+clean; 2.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 21 MiB/s rd, 51 MiB/s wr, 248 op/s 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: Upgrade: Need to upgrade myself (mgr.smithi111.nnblsk) 2024-03-23T11:48:13.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:13 smithi063 ceph-mon[35189]: Upgrade: Need to upgrade myself (mgr.smithi111.nnblsk) 2024-03-23T11:48:13.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.984+0000 7fded9a00700 1 -- 172.21.15.63:0/3297516435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded4102300 msgr2=0x7fded41046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:13.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.984+0000 7fded9a00700 1 --2- 172.21.15.63:0/3297516435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded4102300 0x7fded41046e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdec4004880 tx=0x7fdec4009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:13.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.985+0000 7fded9a00700 1 -- 172.21.15.63:0/3297516435 shutdown_connections 2024-03-23T11:48:13.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.985+0000 7fded9a00700 1 --2- 172.21.15.63:0/3297516435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded4102300 0x7fded41046e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:13.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.985+0000 7fded9a00700 1 --2- 172.21.15.63:0/3297516435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded40662e0 0x7fded4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:13.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.985+0000 7fded9a00700 1 -- 172.21.15.63:0/3297516435 >> 172.21.15.63:0/3297516435 conn(0x7fded40f7430 msgr2=0x7fded40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:13.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.987+0000 7fded9a00700 1 -- 172.21.15.63:0/3297516435 shutdown_connections 2024-03-23T11:48:13.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.987+0000 7fded9a00700 1 -- 172.21.15.63:0/3297516435 wait complete. 2024-03-23T11:48:13.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.987+0000 7fded9a00700 1 Processor -- start 2024-03-23T11:48:13.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.988+0000 7fded9a00700 1 -- start start 2024-03-23T11:48:13.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.989+0000 7fded9a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded40662e0 0x7fded410df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:13.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.989+0000 7fded9a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded4102300 0x7fded410e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:13.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.989+0000 7fded9a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded410ca50 con 0x7fded40662e0 2024-03-23T11:48:13.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.989+0000 7fded9a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded410cb90 con 0x7fded4102300 2024-03-23T11:48:13.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.990+0000 7fded2000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded4102300 0x7fded410e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:13.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.990+0000 7fded2000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded4102300 0x7fded410e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:60128/0 (socket says 172.21.15.63:60128) 2024-03-23T11:48:13.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.990+0000 7fded2000700 1 -- 172.21.15.63:0/48007184 learned_addr learned my addr 172.21.15.63:0/48007184 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:13.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.991+0000 7fded2000700 1 -- 172.21.15.63:0/48007184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded40662e0 msgr2=0x7fded410df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:13.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.991+0000 7fded2000700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded40662e0 0x7fded410df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:13.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.991+0000 7fded2000700 1 -- 172.21.15.63:0/48007184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc009940 con 0x7fded4102300 2024-03-23T11:48:13.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.991+0000 7fded2000700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded4102300 0x7fded410e460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdec4003000 tx=0x7fdec4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:13.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.992+0000 7fdecaa00700 1 -- 172.21.15.63:0/48007184 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdec400e070 con 0x7fded4102300 2024-03-23T11:48:13.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.992+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec4005040 con 0x7fded4102300 2024-03-23T11:48:13.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.992+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded410d1e0 con 0x7fded4102300 2024-03-23T11:48:13.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.993+0000 7fdecaa00700 1 -- 172.21.15.63:0/48007184 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdec401c400 con 0x7fded4102300 2024-03-23T11:48:13.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.993+0000 7fdecaa00700 1 -- 172.21.15.63:0/48007184 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdec4012600 con 0x7fded4102300 2024-03-23T11:48:13.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.995+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb00052e0 con 0x7fded4102300 2024-03-23T11:48:13.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.998+0000 7fdecaa00700 1 -- 172.21.15.63:0/48007184 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 88505+0+0 (secure 0 0 0) 0x7fdec401c9c0 con 0x7fded4102300 2024-03-23T11:48:13.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.999+0000 7fdecaa00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fdeb806ac90 0x7fdeb806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:13.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:13.999+0000 7fdecaa00700 1 -- 172.21.15.63:0/48007184 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fdec408ace0 con 0x7fded4102300 2024-03-23T11:48:13.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.000+0000 7fded2a00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fdeb806ac90 0x7fdeb806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:13.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.001+0000 7fded2a00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fdeb806ac90 0x7fdeb806d140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fded410dcb0 tx=0x7fdecc004d10 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:13.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.003+0000 7fdecaa00700 1 -- 172.21.15.63:0/48007184 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdec4055590 con 0x7fded4102300 2024-03-23T11:48:14.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: pgmap v56: 65 pgs: 65 active+clean; 2.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 21 MiB/s rd, 51 MiB/s wr, 248 op/s 2024-03-23T11:48:14.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:14.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:14.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:48:14.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:48:14.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:48:14.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:48:14.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:48:14.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:48:14.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:14.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: Upgrade: Need to upgrade myself (mgr.smithi111.nnblsk) 2024-03-23T11:48:14.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:13 smithi111 ceph-mon[44005]: Upgrade: Need to upgrade myself (mgr.smithi111.nnblsk) 2024-03-23T11:48:14.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.370+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdeb0000bf0 con 0x7fdeb806ac90 2024-03-23T11:48:14.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.371+0000 7fdecaa00700 1 -- 172.21.15.63:0/48007184 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fdeb0000bf0 con 0x7fdeb806ac90 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.374+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fdeb806ac90 msgr2=0x7fdeb806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.374+0000 7fded9a00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fdeb806ac90 0x7fdeb806d140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fded410dcb0 tx=0x7fdecc004d10 comp rx=0 tx=0).stop 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.374+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded4102300 msgr2=0x7fded410e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.374+0000 7fded9a00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded4102300 0x7fded410e460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdec4003000 tx=0x7fdec4009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.375+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 shutdown_connections 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.375+0000 7fded9a00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded40662e0 0x7fded410df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.375+0000 7fded9a00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fdeb806ac90 0x7fdeb806d140 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.375+0000 7fded9a00700 1 --2- 172.21.15.63:0/48007184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fded4102300 0x7fded410e460 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.375+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 >> 172.21.15.63:0/48007184 conn(0x7fded40f7430 msgr2=0x7fded40fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.375+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 shutdown_connections 2024-03-23T11:48:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.375+0000 7fded9a00700 1 -- 172.21.15.63:0/48007184 wait complete. 2024-03-23T11:48:14.379 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:48:14.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 -- 172.21.15.63:0/1973505918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 msgr2=0x7f656c070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:14.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 --2- 172.21.15.63:0/1973505918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c070d00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6558004940 tx=0x7f6558009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:14.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 -- 172.21.15.63:0/1973505918 shutdown_connections 2024-03-23T11:48:14.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 --2- 172.21.15.63:0/1973505918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c070d00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 --2- 172.21.15.63:0/1973505918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f656c071e40 0x7f656c070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 -- 172.21.15.63:0/1973505918 >> 172.21.15.63:0/1973505918 conn(0x7f656c0f8540 msgr2=0x7f656c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:14.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 -- 172.21.15.63:0/1973505918 shutdown_connections 2024-03-23T11:48:14.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.565+0000 7f6572800700 1 -- 172.21.15.63:0/1973505918 wait complete. 2024-03-23T11:48:14.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.566+0000 7f6572800700 1 Processor -- start 2024-03-23T11:48:14.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.567+0000 7f6572800700 1 -- start start 2024-03-23T11:48:14.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.567+0000 7f6572800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c107820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:14.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.567+0000 7f6572800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f656c071e40 0x7f656c107d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:14.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.567+0000 7f6572800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c1082a0 con 0x7f656c071e40 2024-03-23T11:48:14.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.567+0000 7f6572800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c1083e0 con 0x7f656c070890 2024-03-23T11:48:14.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.568+0000 7f656b400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c107820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:14.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.568+0000 7f656b400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c107820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:60156/0 (socket says 172.21.15.63:60156) 2024-03-23T11:48:14.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.568+0000 7f656b400700 1 -- 172.21.15.63:0/3249108794 learned_addr learned my addr 172.21.15.63:0/3249108794 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:14.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.568+0000 7f656aa00700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f656c071e40 0x7f656c107d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.568+0000 7f656b400700 1 -- 172.21.15.63:0/3249108794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f656c071e40 msgr2=0x7f656c107d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.568+0000 7f656b400700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f656c071e40 0x7f656c107d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.568+0000 7f656b400700 1 -- 172.21.15.63:0/3249108794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6558005040 con 0x7f656c070890 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.569+0000 7f656b400700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c107820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f655c00eb70 tx=0x7f655c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.569+0000 7f6563400700 1 -- 172.21.15.63:0/3249108794 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f655c007cc0 con 0x7f656c070890 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.569+0000 7f6563400700 1 -- 172.21.15.63:0/3249108794 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f655c00f040 con 0x7f656c070890 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.569+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c108660 con 0x7f656c070890 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.570+0000 7f6563400700 1 -- 172.21.15.63:0/3249108794 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f655c00c9d0 con 0x7f656c070890 2024-03-23T11:48:14.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.570+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c1224e0 con 0x7f656c070890 2024-03-23T11:48:14.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.571+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c05f5f0 con 0x7f656c070890 2024-03-23T11:48:14.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.572+0000 7f6563400700 1 -- 172.21.15.63:0/3249108794 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 88505+0+0 (secure 0 0 0) 0x7f655c00cc40 con 0x7f656c070890 2024-03-23T11:48:14.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.574+0000 7f6563400700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f655006ab80 0x7f655006d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:14.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.574+0000 7f6563400700 1 -- 172.21.15.63:0/3249108794 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f655c088eb0 con 0x7f656c070890 2024-03-23T11:48:14.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.574+0000 7f656aa00700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f655006ab80 0x7f655006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:14.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.575+0000 7f656aa00700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f655006ab80 0x7f655006d030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6558009fd0 tx=0x7f6558009e30 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:14.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.578+0000 7f6563400700 1 -- 172.21.15.63:0/3249108794 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f655c04f560 con 0x7f656c070890 2024-03-23T11:48:14.784 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-23T11:48:14.784 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-23T11:48:14.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.988+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f656c0ff8d0 con 0x7f655006ab80 2024-03-23T11:48:14.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.989+0000 7f6563400700 1 -- 172.21.15.63:0/3249108794 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f656c0ff8d0 con 0x7f655006ab80 2024-03-23T11:48:14.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f655006ab80 msgr2=0x7f655006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f655006ab80 0x7f655006d030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6558009fd0 tx=0x7f6558009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 msgr2=0x7f656c107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c107820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f655c00eb70 tx=0x7f655c00b040 comp rx=0 tx=0).stop 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 shutdown_connections 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f655006ab80 0x7f655006d030 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f656c070890 0x7f656c107820 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.993+0000 7f6572800700 1 --2- 172.21.15.63:0/3249108794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f656c071e40 0x7f656c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.994+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 >> 172.21.15.63:0/3249108794 conn(0x7f656c0f8540 msgr2=0x7f656c0fa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.994+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 shutdown_connections 2024-03-23T11:48:14.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:14.994+0000 7f6572800700 1 -- 172.21.15.63:0/3249108794 wait complete. 2024-03-23T11:48:15.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.200+0000 7fc269200700 1 -- 172.21.15.63:0/1443259826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2640feff0 msgr2=0x7fc2640ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:15.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.200+0000 7fc269200700 1 --2- 172.21.15.63:0/1443259826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2640feff0 0x7fc2640ff400 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc254004880 tx=0x7fc254009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:15.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.200+0000 7fc269200700 1 -- 172.21.15.63:0/1443259826 shutdown_connections 2024-03-23T11:48:15.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.200+0000 7fc269200700 1 --2- 172.21.15.63:0/1443259826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2641002a0 0x7fc264100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.200+0000 7fc269200700 1 --2- 172.21.15.63:0/1443259826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2640feff0 0x7fc2640ff400 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.200+0000 7fc269200700 1 -- 172.21.15.63:0/1443259826 >> 172.21.15.63:0/1443259826 conn(0x7fc2640fa950 msgr2=0x7fc2640fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:15.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.201+0000 7fc269200700 1 -- 172.21.15.63:0/1443259826 shutdown_connections 2024-03-23T11:48:15.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.201+0000 7fc269200700 1 -- 172.21.15.63:0/1443259826 wait complete. 2024-03-23T11:48:15.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.202+0000 7fc269200700 1 Processor -- start 2024-03-23T11:48:15.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.203+0000 7fc269200700 1 -- start start 2024-03-23T11:48:15.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.204+0000 7fc269200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640feff0 0x7fc26406e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:15.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.204+0000 7fc269200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2641002a0 0x7fc26406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:15.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.204+0000 7fc269200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26406ef00 con 0x7fc2641002a0 2024-03-23T11:48:15.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc269200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26406f040 con 0x7fc2640feff0 2024-03-23T11:48:15.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640feff0 0x7fc26406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:15.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640feff0 0x7fc26406e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:60168/0 (socket says 172.21.15.63:60168) 2024-03-23T11:48:15.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261e00700 1 -- 172.21.15.63:0/2844700621 learned_addr learned my addr 172.21.15.63:0/2844700621 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:15.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261400700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2641002a0 0x7fc26406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:15.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261e00700 1 -- 172.21.15.63:0/2844700621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2641002a0 msgr2=0x7fc26406e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:15.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261e00700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2641002a0 0x7fc26406e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261e00700 1 -- 172.21.15.63:0/2844700621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc254005040 con 0x7fc2640feff0 2024-03-23T11:48:15.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.205+0000 7fc261400700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2641002a0 0x7fc26406e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:48:15.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.206+0000 7fc261e00700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640feff0 0x7fc26406e480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc254013420 tx=0x7fc254009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:15.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.206+0000 7fc25a000700 1 -- 172.21.15.63:0/2844700621 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25400e030 con 0x7fc2640feff0 2024-03-23T11:48:15.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.206+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26406f2c0 con 0x7fc2640feff0 2024-03-23T11:48:15.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.206+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26411e020 con 0x7fc2640feff0 2024-03-23T11:48:15.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.206+0000 7fc25a000700 1 -- 172.21.15.63:0/2844700621 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc254013c30 con 0x7fc2640feff0 2024-03-23T11:48:15.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.206+0000 7fc25a000700 1 -- 172.21.15.63:0/2844700621 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25401ce90 con 0x7fc2640feff0 2024-03-23T11:48:15.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.209+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2440052e0 con 0x7fc2640feff0 2024-03-23T11:48:15.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.210+0000 7fc25a000700 1 -- 172.21.15.63:0/2844700621 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 88505+0+0 (secure 0 0 0) 0x7fc25401c400 con 0x7fc2640feff0 2024-03-23T11:48:15.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.211+0000 7fc25a000700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fc24806ac50 0x7fc24806d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:15.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.211+0000 7fc25a000700 1 -- 172.21.15.63:0/2844700621 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc25408ac50 con 0x7fc2640feff0 2024-03-23T11:48:15.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.211+0000 7fc261400700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fc24806ac50 0x7fc24806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:15.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.212+0000 7fc261400700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fc24806ac50 0x7fc24806d100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc250007fd0 tx=0x7fc25000b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:15.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.215+0000 7fc25a000700 1 -- 172.21.15.63:0/2844700621 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc254055620 con 0x7fc2640feff0 2024-03-23T11:48:15.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.574+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc244000bf0 con 0x7fc24806ac50 2024-03-23T11:48:15.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.584+0000 7fc25a000700 1 -- 172.21.15.63:0/2844700621 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fc244000bf0 con 0x7fc24806ac50 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 starting - - - - 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (10m) 99s ago 10m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (9m) 95s ago 9m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 starting - - - - 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (4m) 99s ago 4m 70.5M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (4m) 99s ago 4m 42.0M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (4m) 95s ago 4m 41.1M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (4m) 95s ago 4m 31.6M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:9283 running (12m) 99s ago 12m 156M - 17.2.7-726-g9c4e0c34 837124da804f 63833d132315 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (2m) 95s ago 9m 478M - 18.2.2-820-gb630d6cd 881b71eac2fe 278e3d19ec5d 2024-03-23T11:48:15.570 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (12m) 99s ago 12m 48.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (9m) 95s ago 9m 37.0M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 starting - - - - 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 starting - - - - 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (8m) 99s ago 8m 73.2M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (8m) 99s ago 8m 78.9M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (7m) 99s ago 7m 67.7M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (7m) 95s ago 7m 78.8M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (6m) 95s ago 6m 74.8M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (6m) 95s ago 6m 71.7M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:48:15.571 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 starting - - - - 2024-03-23T11:48:15.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fc24806ac50 msgr2=0x7fc24806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:15.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fc24806ac50 0x7fc24806d100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc250007fd0 tx=0x7fc25000b040 comp rx=0 tx=0).stop 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640feff0 msgr2=0x7fc26406e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640feff0 0x7fc26406e480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc254013420 tx=0x7fc254009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 shutdown_connections 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fc24806ac50 0x7fc24806d100 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640feff0 0x7fc26406e480 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 --2- 172.21.15.63:0/2844700621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2641002a0 0x7fc26406e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 >> 172.21.15.63:0/2844700621 conn(0x7fc2640fa950 msgr2=0x7fc2640fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.588+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 shutdown_connections 2024-03-23T11:48:15.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.589+0000 7fc269200700 1 -- 172.21.15.63:0/2844700621 wait complete. 2024-03-23T11:48:15.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.801+0000 7f572a600700 1 -- 172.21.15.63:0/410301353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5724100320 msgr2=0x7f5724100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:15.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.801+0000 7f572a600700 1 --2- 172.21.15.63:0/410301353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5724100320 0x7f5724100790 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5714004940 tx=0x7f5714009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:15.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.801+0000 7f572a600700 1 -- 172.21.15.63:0/410301353 shutdown_connections 2024-03-23T11:48:15.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.801+0000 7f572a600700 1 --2- 172.21.15.63:0/410301353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5724100320 0x7f5724100790 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.801+0000 7f572a600700 1 --2- 172.21.15.63:0/410301353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57240ff070 0x7f57240ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.801+0000 7f572a600700 1 -- 172.21.15.63:0/410301353 >> 172.21.15.63:0/410301353 conn(0x7f57240fa950 msgr2=0x7f57240fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:15.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.801+0000 7f572a600700 1 -- 172.21.15.63:0/410301353 shutdown_connections 2024-03-23T11:48:15.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.802+0000 7f572a600700 1 -- 172.21.15.63:0/410301353 wait complete. 2024-03-23T11:48:15.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.803+0000 7f572a600700 1 Processor -- start 2024-03-23T11:48:15.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.803+0000 7f572a600700 1 -- start start 2024-03-23T11:48:15.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f572a600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57240ff070 0x7f5724113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:15.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f572a600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5724100320 0x7f57241144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:15.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f572a600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724114aa0 con 0x7f57240ff070 2024-03-23T11:48:15.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f572a600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57241152f0 con 0x7f5724100320 2024-03-23T11:48:15.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f5722a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5724100320 0x7f57241144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:15.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f5722a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5724100320 0x7f57241144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:60180/0 (socket says 172.21.15.63:60180) 2024-03-23T11:48:15.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f5722a00700 1 -- 172.21.15.63:0/598612736 learned_addr learned my addr 172.21.15.63:0/598612736 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:15.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.805+0000 7f5722a00700 1 -- 172.21.15.63:0/598612736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57240ff070 msgr2=0x7f5724113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:15.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.804+0000 7f5723400700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57240ff070 0x7f5724113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:15.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.805+0000 7f5722a00700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57240ff070 0x7f5724113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:15.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.805+0000 7f5722a00700 1 -- 172.21.15.63:0/598612736 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714005040 con 0x7f5724100320 2024-03-23T11:48:15.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.805+0000 7f5723400700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57240ff070 0x7f5724113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:48:15.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.806+0000 7f5722a00700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5724100320 0x7f57241144a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f571400b3b0 tx=0x7f5714009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:15.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.806+0000 7f571b400700 1 -- 172.21.15.63:0/598612736 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571400e040 con 0x7f5724100320 2024-03-23T11:48:15.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.806+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5724115570 con 0x7f5724100320 2024-03-23T11:48:15.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.806+0000 7f571b400700 1 -- 172.21.15.63:0/598612736 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5714004290 con 0x7f5724100320 2024-03-23T11:48:15.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.806+0000 7f571b400700 1 -- 172.21.15.63:0/598612736 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571401c3f0 con 0x7f5724100320 2024-03-23T11:48:15.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.806+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5724115ab0 con 0x7f5724100320 2024-03-23T11:48:15.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.808+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572405f5f0 con 0x7f5724100320 2024-03-23T11:48:15.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.809+0000 7f571b400700 1 -- 172.21.15.63:0/598612736 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 88505+0+0 (secure 0 0 0) 0x7f5714013420 con 0x7f5724100320 2024-03-23T11:48:15.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.810+0000 7f571b400700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f570806ab80 0x7f570806d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:15.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.810+0000 7f571b400700 1 -- 172.21.15.63:0/598612736 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f571408a5e0 con 0x7f5724100320 2024-03-23T11:48:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.811+0000 7f5723400700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f570806ab80 0x7f570806d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.812+0000 7f5723400700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f570806ab80 0x7f570806d030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5710007fd0 tx=0x7f5710009040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:15.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:15.816+0000 7f571b400700 1 -- 172.21.15.63:0/598612736 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f57140589e0 con 0x7f5724100320 2024-03-23T11:48:15.928 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:15 smithi063 ceph-mon[35189]: pgmap v57: 65 pgs: 65 active+clean; 2.5 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 13 MiB/s rd, 36 MiB/s wr, 200 op/s 2024-03-23T11:48:15.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:15 smithi063 ceph-mon[35189]: from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:15.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:15 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:15.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:15 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:48:15.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:15 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:48:15.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:15 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:48:15.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:15 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:48:16.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:15 smithi111 ceph-mon[44005]: pgmap v57: 65 pgs: 65 active+clean; 2.5 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 13 MiB/s rd, 36 MiB/s wr, 200 op/s 2024-03-23T11:48:16.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:15 smithi111 ceph-mon[44005]: from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:16.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:15 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:16.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:15 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:48:16.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:15 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:48:16.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:15 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:48:16.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:15 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:48:16.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.271+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5724115dc0 con 0x7f5724100320 2024-03-23T11:48:16.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.272+0000 7f571b400700 1 -- 172.21.15.63:0/598612736 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+848 (secure 0 0 0) 0x7f5714058570 con 0x7f5724100320 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 13, 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:48:16.258 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:48:16.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.276+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f570806ab80 msgr2=0x7f570806d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:16.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.276+0000 7f572a600700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f570806ab80 0x7f570806d030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5710007fd0 tx=0x7f5710009040 comp rx=0 tx=0).stop 2024-03-23T11:48:16.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.276+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5724100320 msgr2=0x7f57241144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.276+0000 7f572a600700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5724100320 0x7f57241144a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f571400b3b0 tx=0x7f5714009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.277+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 shutdown_connections 2024-03-23T11:48:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.277+0000 7f572a600700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57240ff070 0x7f5724113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.277+0000 7f572a600700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f570806ab80 0x7f570806d030 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.277+0000 7f572a600700 1 --2- 172.21.15.63:0/598612736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5724100320 0x7f57241144a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.277+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 >> 172.21.15.63:0/598612736 conn(0x7f57240fa950 msgr2=0x7f57240fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:16.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.277+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 shutdown_connections 2024-03-23T11:48:16.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.278+0000 7f572a600700 1 -- 172.21.15.63:0/598612736 wait complete. 2024-03-23T11:48:16.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 -- 172.21.15.63:0/3432756825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 msgr2=0x7f93f00ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 --2- 172.21.15.63:0/3432756825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f00ff410 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f93e4004940 tx=0x7f93e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 -- 172.21.15.63:0/3432756825 shutdown_connections 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 --2- 172.21.15.63:0/3432756825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93f01002b0 0x7f93f0100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 --2- 172.21.15.63:0/3432756825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f00ff410 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 -- 172.21.15.63:0/3432756825 >> 172.21.15.63:0/3432756825 conn(0x7f93f00fa950 msgr2=0x7f93f00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 -- 172.21.15.63:0/3432756825 shutdown_connections 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.470+0000 7f93f7400700 1 -- 172.21.15.63:0/3432756825 wait complete. 2024-03-23T11:48:16.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.472+0000 7f93f7400700 1 Processor -- start 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.472+0000 7f93f7400700 1 -- start start 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.472+0000 7f93f7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f0075c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.472+0000 7f93f7400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93f01002b0 0x7f93f00761c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.472+0000 7f93f7400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0072720 con 0x7f93f00ff000 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.472+0000 7f93f7400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0072890 con 0x7f93f01002b0 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.473+0000 7f93f4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f0075c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.473+0000 7f93f4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f0075c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40726/0 (socket says 172.21.15.63:40726) 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.473+0000 7f93f4a00700 1 -- 172.21.15.63:0/2657924115 learned_addr learned my addr 172.21.15.63:0/2657924115 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:16.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.473+0000 7f93efe00700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93f01002b0 0x7f93f00761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:16.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.473+0000 7f93f4a00700 1 -- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93f01002b0 msgr2=0x7f93f00761c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:16.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.473+0000 7f93f4a00700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93f01002b0 0x7f93f00761c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.474+0000 7f93f4a00700 1 -- 172.21.15.63:0/2657924115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e4005040 con 0x7f93f00ff000 2024-03-23T11:48:16.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.474+0000 7f93f4a00700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f0075c80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f93e4007d40 tx=0x7f93e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:16.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.475+0000 7f93ecc00700 1 -- 172.21.15.63:0/2657924115 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e400e030 con 0x7f93f00ff000 2024-03-23T11:48:16.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.475+0000 7f93ecc00700 1 -- 172.21.15.63:0/2657924115 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93e401c400 con 0x7f93f00ff000 2024-03-23T11:48:16.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.475+0000 7f93ecc00700 1 -- 172.21.15.63:0/2657924115 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e400b650 con 0x7f93f00ff000 2024-03-23T11:48:16.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.475+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f0072ac0 con 0x7f93f00ff000 2024-03-23T11:48:16.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.475+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f0073080 con 0x7f93f00ff000 2024-03-23T11:48:16.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.476+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f004c710 con 0x7f93f00ff000 2024-03-23T11:48:16.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.478+0000 7f93ecc00700 1 -- 172.21.15.63:0/2657924115 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) v1 ==== 88505+0+0 (secure 0 0 0) 0x7f93e401c570 con 0x7f93f00ff000 2024-03-23T11:48:16.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.480+0000 7f93ecc00700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f93d806ab70 0x7f93d806d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:16.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.480+0000 7f93ecc00700 1 -- 172.21.15.63:0/2657924115 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f93e408a990 con 0x7f93f00ff000 2024-03-23T11:48:16.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.480+0000 7f93efe00700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f93d806ab70 0x7f93d806d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:16.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.481+0000 7f93efe00700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f93d806ab70 0x7f93d806d020 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009b80 tx=0x7f93dc00b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:16.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.487+0000 7f93ecc00700 1 -- 172.21.15.63:0/2657924115 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f93e4058d90 con 0x7f93f00ff000 2024-03-23T11:48:16.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.850+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f93f01045e0 con 0x7f93d806ab70 2024-03-23T11:48:16.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.852+0000 7f93ecc00700 1 -- 172.21.15.63:0/2657924115 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f93f01045e0 con 0x7f93d806ab70 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [], 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "1/2 daemons upgraded", 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mgr daemons", 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:48:16.837 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:48:16.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f93d806ab70 msgr2=0x7f93d806d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:16.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f93d806ab70 0x7f93d806d020 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009b80 tx=0x7f93dc00b040 comp rx=0 tx=0).stop 2024-03-23T11:48:16.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 msgr2=0x7f93f0075c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:16.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f0075c80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f93e4007d40 tx=0x7f93e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:16.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 shutdown_connections 2024-03-23T11:48:16.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00ff000 0x7f93f0075c80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f93d806ab70 0x7f93d806d020 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 --2- 172.21.15.63:0/2657924115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93f01002b0 0x7f93f00761c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:16.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 >> 172.21.15.63:0/2657924115 conn(0x7f93f00fa950 msgr2=0x7f93f00fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:16.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.855+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 shutdown_connections 2024-03-23T11:48:16.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:16.856+0000 7f93f7400700 1 -- 172.21.15.63:0/2657924115 wait complete. 2024-03-23T11:48:17.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:16 smithi063 ceph-mon[35189]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:17.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:16 smithi063 ceph-mon[35189]: Upgrade: Updating mgr.smithi063.gehkcr 2024-03-23T11:48:17.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:16 smithi063 ceph-mon[35189]: Deploying daemon mgr.smithi063.gehkcr on smithi063 2024-03-23T11:48:17.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:16 smithi063 ceph-mon[35189]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:17.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:16 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/598612736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:48:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:16 smithi111 ceph-mon[44005]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:16 smithi111 ceph-mon[44005]: Upgrade: Updating mgr.smithi063.gehkcr 2024-03-23T11:48:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:16 smithi111 ceph-mon[44005]: Deploying daemon mgr.smithi063.gehkcr on smithi063 2024-03-23T11:48:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:16 smithi111 ceph-mon[44005]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:16 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/598612736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:48:17.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:17 smithi063 ceph-mon[35189]: pgmap v58: 65 pgs: 65 active+clean; 2.5 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 5.5 MiB/s rd, 17 MiB/s wr, 150 op/s 2024-03-23T11:48:17.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:17 smithi063 ceph-mon[35189]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:18.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:17 smithi111 ceph-mon[44005]: pgmap v58: 65 pgs: 65 active+clean; 2.5 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 5.5 MiB/s rd, 17 MiB/s wr, 150 op/s 2024-03-23T11:48:18.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:17 smithi111 ceph-mon[44005]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:19.934 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:19 smithi111 ceph-mon[44005]: pgmap v59: 65 pgs: 65 active+clean; 1.6 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 5.9 MiB/s rd, 18 MiB/s wr, 219 op/s 2024-03-23T11:48:20.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:19 smithi063 ceph-mon[35189]: pgmap v59: 65 pgs: 65 active+clean; 1.6 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 5.9 MiB/s rd, 18 MiB/s wr, 219 op/s 2024-03-23T11:48:22.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: pgmap v60: 65 pgs: 65 active+clean; 1.6 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 135 op/s 2024-03-23T11:48:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:21 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: pgmap v60: 65 pgs: 65 active+clean; 1.6 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 135 op/s 2024-03-23T11:48:22.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:22.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:22.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:21 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:24.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:23 smithi063 ceph-mon[35189]: pgmap v61: 65 pgs: 65 active+clean; 1.3 GiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 194 op/s 2024-03-23T11:48:24.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:23 smithi111 ceph-mon[44005]: pgmap v61: 65 pgs: 65 active+clean; 1.3 GiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 194 op/s 2024-03-23T11:48:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:25 smithi063 ceph-mon[35189]: pgmap v62: 65 pgs: 65 active+clean; 1000 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 210 op/s 2024-03-23T11:48:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:25 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:25 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:25 smithi111 ceph-mon[44005]: pgmap v62: 65 pgs: 65 active+clean; 1000 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 210 op/s 2024-03-23T11:48:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:25 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:25 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:27.075 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:26 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:27.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:26 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:27.501 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-23T11:48:27.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-23T11:48:28.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:27 smithi063 ceph-mon[35189]: pgmap v63: 65 pgs: 65 active+clean; 1000 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 185 op/s 2024-03-23T11:48:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:27 smithi111 ceph-mon[44005]: pgmap v63: 65 pgs: 65 active+clean; 1000 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 185 op/s 2024-03-23T11:48:30.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:29 smithi063 ceph-mon[35189]: pgmap v64: 65 pgs: 65 active+clean; 465 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 299 op/s 2024-03-23T11:48:30.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:29 smithi111 ceph-mon[44005]: pgmap v64: 65 pgs: 65 active+clean; 465 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 299 op/s 2024-03-23T11:48:32.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:31 smithi063 ceph-mon[35189]: pgmap v65: 65 pgs: 65 active+clean; 465 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 231 op/s 2024-03-23T11:48:32.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:31 smithi111 ceph-mon[44005]: pgmap v65: 65 pgs: 65 active+clean; 465 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 231 op/s 2024-03-23T11:48:33.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:33 smithi063 ceph-mon[35189]: pgmap v66: 65 pgs: 65 active+clean; 359 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 297 op/s 2024-03-23T11:48:33.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:33 smithi063 ceph-mon[35189]: Standby manager daemon smithi063.gehkcr restarted 2024-03-23T11:48:33.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:33 smithi063 ceph-mon[35189]: Standby manager daemon smithi063.gehkcr started 2024-03-23T11:48:33.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:33 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/crt"}]: dispatch 2024-03-23T11:48:33.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:33 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:48:33.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:33 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/key"}]: dispatch 2024-03-23T11:48:33.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:33 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:48:34.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:33 smithi111 ceph-mon[44005]: pgmap v66: 65 pgs: 65 active+clean; 359 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 297 op/s 2024-03-23T11:48:34.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:33 smithi111 ceph-mon[44005]: Standby manager daemon smithi063.gehkcr restarted 2024-03-23T11:48:34.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:33 smithi111 ceph-mon[44005]: Standby manager daemon smithi063.gehkcr started 2024-03-23T11:48:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:33 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/crt"}]: dispatch 2024-03-23T11:48:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:33 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:48:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:33 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi063.gehkcr/key"}]: dispatch 2024-03-23T11:48:34.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:33 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:48:35.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:34 smithi063 ceph-mon[35189]: mgrmap e27: smithi111.nnblsk(active, since 2m), standbys: smithi063.gehkcr 2024-03-23T11:48:35.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:34 smithi111 ceph-mon[44005]: mgrmap e27: smithi111.nnblsk(active, since 2m), standbys: smithi063.gehkcr 2024-03-23T11:48:36.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:35 smithi063 ceph-mon[35189]: pgmap v67: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 284 op/s 2024-03-23T11:48:36.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:35 smithi111 ceph-mon[44005]: pgmap v67: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 284 op/s 2024-03-23T11:48:38.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:37 smithi063 ceph-mon[35189]: pgmap v68: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 226 op/s 2024-03-23T11:48:38.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:37 smithi111 ceph-mon[44005]: pgmap v68: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 226 op/s 2024-03-23T11:48:40.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:39 smithi063 ceph-mon[35189]: pgmap v69: 65 pgs: 65 active+clean; 302 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 395 op/s 2024-03-23T11:48:40.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:39 smithi111 ceph-mon[44005]: pgmap v69: 65 pgs: 65 active+clean; 302 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 395 op/s 2024-03-23T11:48:42.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:41 smithi063 ceph-mon[35189]: pgmap v70: 65 pgs: 65 active+clean; 302 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 281 op/s 2024-03-23T11:48:42.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:41 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:42.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:41 smithi111 ceph-mon[44005]: pgmap v70: 65 pgs: 65 active+clean; 302 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 281 op/s 2024-03-23T11:48:42.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:41 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:44.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:43 smithi063 ceph-mon[35189]: pgmap v71: 65 pgs: 65 active+clean; 300 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 377 op/s 2024-03-23T11:48:44.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:43 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:44.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:43 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:44.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:43 smithi111 ceph-mon[44005]: pgmap v71: 65 pgs: 65 active+clean; 300 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 377 op/s 2024-03-23T11:48:44.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:43 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:44.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:43 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:45.993 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:45 smithi063 ceph-mon[35189]: pgmap v72: 65 pgs: 65 active+clean; 298 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 387 op/s 2024-03-23T11:48:45.993 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:45 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:45.994 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:45 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:46.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:45 smithi111 ceph-mon[44005]: pgmap v72: 65 pgs: 65 active+clean; 298 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 387 op/s 2024-03-23T11:48:46.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:45 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:46.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:45 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.059+0000 7f5bb1600700 1 -- 172.21.15.63:0/1450340313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 msgr2=0x7f5bac0fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.059+0000 7f5bb1600700 1 --2- 172.21.15.63:0/1450340313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac0fd230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5b98004940 tx=0x7f5b98009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.059+0000 7f5bb1600700 1 -- 172.21.15.63:0/1450340313 shutdown_connections 2024-03-23T11:48:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.059+0000 7f5bb1600700 1 --2- 172.21.15.63:0/1450340313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac0fd230 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.059+0000 7f5bb1600700 1 --2- 172.21.15.63:0/1450340313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bac0fbb10 0x7f5bac0fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.059+0000 7f5bb1600700 1 -- 172.21.15.63:0/1450340313 >> 172.21.15.63:0/1450340313 conn(0x7f5bac0f7270 msgr2=0x7f5bac0f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.060+0000 7f5bb1600700 1 -- 172.21.15.63:0/1450340313 shutdown_connections 2024-03-23T11:48:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.060+0000 7f5bb1600700 1 -- 172.21.15.63:0/1450340313 wait complete. 2024-03-23T11:48:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.061+0000 7f5bb1600700 1 Processor -- start 2024-03-23T11:48:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.062+0000 7f5bb1600700 1 -- start start 2024-03-23T11:48:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.062+0000 7f5bb1600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bac0fbb10 0x7f5bac10f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.062+0000 7f5bb1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac10fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.062+0000 7f5bb1600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac110450 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.062+0000 7f5bb1600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac110ca0 con 0x7f5bac0fbb10 2024-03-23T11:48:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.063+0000 7f5bab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac10fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.063+0000 7f5bab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac10fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35090/0 (socket says 172.21.15.63:35090) 2024-03-23T11:48:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.063+0000 7f5bab400700 1 -- 172.21.15.63:0/445759803 learned_addr learned my addr 172.21.15.63:0/445759803 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:47.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.063+0000 7f5babe00700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bac0fbb10 0x7f5bac10f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:47.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.063+0000 7f5bab400700 1 -- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bac0fbb10 msgr2=0x7f5bac10f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:47.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.063+0000 7f5bab400700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bac0fbb10 0x7f5bac10f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.063+0000 7f5bab400700 1 -- 172.21.15.63:0/445759803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b98005040 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.064+0000 7f5babe00700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bac0fbb10 0x7f5bac10f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:48:47.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.064+0000 7f5bab400700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac10fe50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800b0d0 tx=0x7f5b98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:47.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.064+0000 7f5ba3e00700 1 -- 172.21.15.63:0/445759803 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9800e070 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.064+0000 7f5bb1600700 1 -- 172.21.15.63:0/445759803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bac110ea0 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.065+0000 7f5bb1600700 1 -- 172.21.15.63:0/445759803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bac111460 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.066+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c0052e0 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.066+0000 7f5ba3e00700 1 -- 172.21.15.63:0/445759803 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b980123c0 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.066+0000 7f5ba3e00700 1 -- 172.21.15.63:0/445759803 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b980265f0 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.069+0000 7f5ba3e00700 1 -- 172.21.15.63:0/445759803 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 27) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f5b98026810 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.070+0000 7f5ba3e00700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f5b9006c520 0x7f5b9006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:47.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.070+0000 7f5ba3e00700 1 -- 172.21.15.63:0/445759803 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f5b9808caf0 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.071+0000 7f5babe00700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f5b9006c520 0x7f5b9006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:47.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.072+0000 7f5babe00700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f5b9006c520 0x7f5b9006e9d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c004600 tx=0x7f5b9c00b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.077+0000 7f5ba3e00700 1 -- 172.21.15.63:0/445759803 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f5b9805b4c0 con 0x7f5bac0fcdc0 2024-03-23T11:48:47.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.431+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b8c000bf0 con 0x7f5b9006c520 2024-03-23T11:48:47.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.433+0000 7f5ba3e00700 1 -- 172.21.15.63:0/445759803 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f5b8c000bf0 con 0x7f5b9006c520 2024-03-23T11:48:47.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.437+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f5b9006c520 msgr2=0x7f5b9006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:47.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.437+0000 7f5ba1600700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f5b9006c520 0x7f5b9006e9d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c004600 tx=0x7f5b9c00b040 comp rx=0 tx=0).stop 2024-03-23T11:48:47.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.437+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 msgr2=0x7f5bac10fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.437+0000 7f5ba1600700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac10fe50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800b0d0 tx=0x7f5b98009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.438+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 shutdown_connections 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.438+0000 7f5ba1600700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f5b9006c520 0x7f5b9006e9d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.438+0000 7f5ba1600700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bac0fbb10 0x7f5bac10f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.438+0000 7f5ba1600700 1 --2- 172.21.15.63:0/445759803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5bac0fcdc0 0x7f5bac10fe50 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.438+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 >> 172.21.15.63:0/445759803 conn(0x7f5bac0f7270 msgr2=0x7f5bac065cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.438+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 shutdown_connections 2024-03-23T11:48:47.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.438+0000 7f5ba1600700 1 -- 172.21.15.63:0/445759803 wait complete. 2024-03-23T11:48:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:48:47.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.671+0000 7fbdd6c00700 1 -- 172.21.15.63:0/2678578956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006f9d0 msgr2=0x7fbdd006e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:47.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.671+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/2678578956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006f9d0 0x7fbdd006e460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0004940 tx=0x7fbdc0009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.672+0000 7fbdd6c00700 1 -- 172.21.15.63:0/2678578956 shutdown_connections 2024-03-23T11:48:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.672+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/2678578956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdd006e9a0 0x7fbdd006ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.672+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/2678578956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006f9d0 0x7fbdd006e460 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.672+0000 7fbdd6c00700 1 -- 172.21.15.63:0/2678578956 >> 172.21.15.63:0/2678578956 conn(0x7fbdd0069de0 msgr2=0x7fbdd006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.672+0000 7fbdd6c00700 1 -- 172.21.15.63:0/2678578956 shutdown_connections 2024-03-23T11:48:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.672+0000 7fbdd6c00700 1 -- 172.21.15.63:0/2678578956 wait complete. 2024-03-23T11:48:47.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.673+0000 7fbdd6c00700 1 Processor -- start 2024-03-23T11:48:47.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.674+0000 7fbdd6c00700 1 -- start start 2024-03-23T11:48:47.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.676+0000 7fbdd6c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006e9a0 0x7fbdd0125240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:47.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.676+0000 7fbdd6c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdd006f9d0 0x7fbdd0125780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:47.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.676+0000 7fbdd6c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0125e10 con 0x7fbdd006f9d0 2024-03-23T11:48:47.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.676+0000 7fbdcfe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006e9a0 0x7fbdd0125240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:47.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.676+0000 7fbdcfe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006e9a0 0x7fbdd0125240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33708/0 (socket says 172.21.15.63:33708) 2024-03-23T11:48:47.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.676+0000 7fbdcfe00700 1 -- 172.21.15.63:0/3734765910 learned_addr learned my addr 172.21.15.63:0/3734765910 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:47.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.676+0000 7fbdd6c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd012a370 con 0x7fbdd006e9a0 2024-03-23T11:48:47.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.677+0000 7fbdc7e00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdd006f9d0 0x7fbdd0125780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:47.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.677+0000 7fbdcfe00700 1 -- 172.21.15.63:0/3734765910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdd006f9d0 msgr2=0x7fbdd0125780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:47.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.677+0000 7fbdcfe00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdd006f9d0 0x7fbdd0125780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:47.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.677+0000 7fbdcfe00700 1 -- 172.21.15.63:0/3734765910 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc0005040 con 0x7fbdd006e9a0 2024-03-23T11:48:47.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.678+0000 7fbdcfe00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006e9a0 0x7fbdd0125240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0008f80 tx=0x7fbdc0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:47.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.678+0000 7fbdccc00700 1 -- 172.21.15.63:0/3734765910 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc000e070 con 0x7fbdd006e9a0 2024-03-23T11:48:47.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.678+0000 7fbdccc00700 1 -- 172.21.15.63:0/3734765910 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdc0004290 con 0x7fbdd006e9a0 2024-03-23T11:48:47.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.678+0000 7fbdccc00700 1 -- 172.21.15.63:0/3734765910 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc001c400 con 0x7fbdd006e9a0 2024-03-23T11:48:47.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.679+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd012a570 con 0x7fbdd006e9a0 2024-03-23T11:48:47.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.679+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd012aae0 con 0x7fbdd006e9a0 2024-03-23T11:48:47.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.681+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd004c710 con 0x7fbdd006e9a0 2024-03-23T11:48:47.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.683+0000 7fbdccc00700 1 -- 172.21.15.63:0/3734765910 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 27) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fbdc0013420 con 0x7fbdd006e9a0 2024-03-23T11:48:47.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.684+0000 7fbdccc00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fbdb406c550 0x7fbdb406ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:47.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.684+0000 7fbdccc00700 1 -- 172.21.15.63:0/3734765910 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fbdc008cfa0 con 0x7fbdd006e9a0 2024-03-23T11:48:47.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.684+0000 7fbdc7e00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fbdb406c550 0x7fbdb406ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:47.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.685+0000 7fbdc7e00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fbdb406c550 0x7fbdb406ea00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009bf0 tx=0x7fbdb80091a0 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:47.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:47.689+0000 7fbdccc00700 1 -- 172.21.15.63:0/3734765910 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fbdc005b2e0 con 0x7fbdd006e9a0 2024-03-23T11:48:48.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.045+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdd01081b0 con 0x7fbdb406c550 2024-03-23T11:48:48.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.047+0000 7fbdccc00700 1 -- 172.21.15.63:0/3734765910 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fbdd01081b0 con 0x7fbdb406c550 2024-03-23T11:48:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.050+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fbdb406c550 msgr2=0x7fbdb406ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.050+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fbdb406c550 0x7fbdb406ea00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009bf0 tx=0x7fbdb80091a0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.050+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006e9a0 msgr2=0x7fbdd0125240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.050+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006e9a0 0x7fbdd0125240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0008f80 tx=0x7fbdc0009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.051+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 shutdown_connections 2024-03-23T11:48:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.051+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7fbdb406c550 0x7fbdb406ea00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.051+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd006e9a0 0x7fbdd0125240 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.051+0000 7fbdd6c00700 1 --2- 172.21.15.63:0/3734765910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbdd006f9d0 0x7fbdd0125780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.051+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 >> 172.21.15.63:0/3734765910 conn(0x7fbdd0069de0 msgr2=0x7fbdd006b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.051+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 shutdown_connections 2024-03-23T11:48:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.051+0000 7fbdd6c00700 1 -- 172.21.15.63:0/3734765910 wait complete. 2024-03-23T11:48:48.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:47 smithi063 ceph-mon[35189]: pgmap v73: 65 pgs: 65 active+clean; 298 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 341 op/s 2024-03-23T11:48:48.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:47 smithi111 ceph-mon[44005]: pgmap v73: 65 pgs: 65 active+clean; 298 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 341 op/s 2024-03-23T11:48:48.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.248+0000 7f8f2ac00700 1 -- 172.21.15.63:0/1517714181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 msgr2=0x7f8f240fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.248+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/1517714181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 0x7f8f240fe4e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009ce0 tx=0x7f8f1c004d10 comp rx=0 tx=0).stop 2024-03-23T11:48:48.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.249+0000 7f8f2ac00700 1 -- 172.21.15.63:0/1517714181 shutdown_connections 2024-03-23T11:48:48.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.249+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/1517714181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 0x7f8f240fe4e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.249+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/1517714181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 0x7f8f240fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.249+0000 7f8f2ac00700 1 -- 172.21.15.63:0/1517714181 >> 172.21.15.63:0/1517714181 conn(0x7f8f240f85a0 msgr2=0x7f8f240fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:48.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.249+0000 7f8f2ac00700 1 -- 172.21.15.63:0/1517714181 shutdown_connections 2024-03-23T11:48:48.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.249+0000 7f8f2ac00700 1 -- 172.21.15.63:0/1517714181 wait complete. 2024-03-23T11:48:48.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.250+0000 7f8f2ac00700 1 Processor -- start 2024-03-23T11:48:48.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.250+0000 7f8f2ac00700 1 -- start start 2024-03-23T11:48:48.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f2ac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 0x7f8f24114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:48.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f2ac00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 0x7f8f24114570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:48.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f2ac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f24114b70 con 0x7f8f240fcf00 2024-03-23T11:48:48.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f2ac00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f241153c0 con 0x7f8f240fe070 2024-03-23T11:48:48.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f23e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 0x7f8f24114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:48.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f23e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 0x7f8f24114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35124/0 (socket says 172.21.15.63:35124) 2024-03-23T11:48:48.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f23e00700 1 -- 172.21.15.63:0/3055477173 learned_addr learned my addr 172.21.15.63:0/3055477173 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:48.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.251+0000 7f8f23400700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 0x7f8f24114570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:48.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.252+0000 7f8f23e00700 1 -- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 msgr2=0x7f8f24114570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.252+0000 7f8f23e00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 0x7f8f24114570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.252+0000 7f8f23e00700 1 -- 172.21.15.63:0/3055477173 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c009940 con 0x7f8f240fcf00 2024-03-23T11:48:48.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.252+0000 7f8f23e00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 0x7f8f24114030 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000eb70 tx=0x7f8f1000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:48.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.253+0000 7f8f1be00700 1 -- 172.21.15.63:0/3055477173 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f10007cc0 con 0x7f8f240fcf00 2024-03-23T11:48:48.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.253+0000 7f8f1be00700 1 -- 172.21.15.63:0/3055477173 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f1000f040 con 0x7f8f240fcf00 2024-03-23T11:48:48.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.253+0000 7f8f1be00700 1 -- 172.21.15.63:0/3055477173 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f1000c990 con 0x7f8f240fcf00 2024-03-23T11:48:48.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.253+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f24115640 con 0x7f8f240fcf00 2024-03-23T11:48:48.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.253+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f24115b80 con 0x7f8f240fcf00 2024-03-23T11:48:48.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.257+0000 7f8f1be00700 1 -- 172.21.15.63:0/3055477173 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 27) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f8f1000caf0 con 0x7f8f240fcf00 2024-03-23T11:48:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.257+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2405f5f0 con 0x7f8f240fcf00 2024-03-23T11:48:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.258+0000 7f8f1be00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f8f0806c520 0x7f8f0806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.258+0000 7f8f1be00700 1 -- 172.21.15.63:0/3055477173 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f8f1008afe0 con 0x7f8f240fcf00 2024-03-23T11:48:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.259+0000 7f8f23400700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f8f0806c520 0x7f8f0806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:48.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.260+0000 7f8f23400700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f8f0806c520 0x7f8f0806e9d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0094f0 tx=0x7f8f1c004d10 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:48.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.264+0000 7f8f1be00700 1 -- 172.21.15.63:0/3055477173 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f8f100592a0 con 0x7f8f240fcf00 2024-03-23T11:48:48.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.669+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f24115eb0 con 0x7f8f0806c520 2024-03-23T11:48:48.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.679+0000 7f8f1be00700 1 -- 172.21.15.63:0/3055477173 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8f24115eb0 con 0x7f8f0806c520 2024-03-23T11:48:48.665 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:48:48.665 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (92s) 5s ago 11m 14.7M - 0.25.0 c8568f914cd2 106bde4987fa 2024-03-23T11:48:48.665 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (11m) 5s ago 11m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:48:48.665 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (10m) 22s ago 10m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:48:48.665 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (62s) 5s ago 10m 76.8M - 9.4.7 954c08fa6188 d4ed1ee72841 2024-03-23T11:48:48.665 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (4m) 5s ago 4m 155M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:48:48.666 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (4m) 5s ago 4m 145M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:48:48.666 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (4m) 22s ago 4m 92.8M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:48:48.666 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (4m) 22s ago 4m 84.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:48:48.666 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (28s) 5s ago 12m 77.5M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:48:48.666 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (2m) 22s ago 10m 495M - 18.2.2-820-gb630d6cd 881b71eac2fe 278e3d19ec5d 2024-03-23T11:48:48.666 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (12m) 5s ago 12m 53.3M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:48:48.667 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (10m) 22s ago 10m 39.5M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:48:48.667 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (115s) 5s ago 11m 7495k - 1.5.0 0da6a335fe13 9387979cc7ad 2024-03-23T11:48:48.667 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (36s) 22s ago 10m 2973k - 1.5.0 0da6a335fe13 fd96d8076254 2024-03-23T11:48:48.667 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (9m) 5s ago 9m 198M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:48:48.667 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (8m) 5s ago 8m 212M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:48:48.667 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (8m) 5s ago 8m 168M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:48:48.667 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (7m) 22s ago 7m 236M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:48:48.668 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (7m) 22s ago 7m 206M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:48:48.668 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (6m) 22s ago 6m 217M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:48:48.668 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (39s) 5s ago 10m 38.2M - 2.43.0 a07b618ecd1d f74e0fbb00ad 2024-03-23T11:48:48.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f8f0806c520 msgr2=0x7f8f0806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f8f0806c520 0x7f8f0806e9d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0094f0 tx=0x7f8f1c004d10 comp rx=0 tx=0).stop 2024-03-23T11:48:48.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 msgr2=0x7f8f24114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 0x7f8f24114030 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000eb70 tx=0x7f8f1000b040 comp rx=0 tx=0).stop 2024-03-23T11:48:48.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 shutdown_connections 2024-03-23T11:48:48.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f240fcf00 0x7f8f24114030 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f8f0806c520 0x7f8f0806e9d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 --2- 172.21.15.63:0/3055477173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f240fe070 0x7f8f24114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.682+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 >> 172.21.15.63:0/3055477173 conn(0x7f8f240f85a0 msgr2=0x7f8f240f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:48.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.683+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 shutdown_connections 2024-03-23T11:48:48.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.683+0000 7f8f2ac00700 1 -- 172.21.15.63:0/3055477173 wait complete. 2024-03-23T11:48:48.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 -- 172.21.15.63:0/2838905945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc066280 msgr2=0x7f69fc0ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 --2- 172.21.15.63:0/2838905945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc066280 0x7f69fc0ffa20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f69f0004940 tx=0x7f69f0009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:48.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 -- 172.21.15.63:0/2838905945 shutdown_connections 2024-03-23T11:48:48.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 --2- 172.21.15.63:0/2838905945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc0fff60 0x7f69fc102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 --2- 172.21.15.63:0/2838905945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc066280 0x7f69fc0ffa20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 -- 172.21.15.63:0/2838905945 >> 172.21.15.63:0/2838905945 conn(0x7f69fc0f7430 msgr2=0x7f69fc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:48.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 -- 172.21.15.63:0/2838905945 shutdown_connections 2024-03-23T11:48:48.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.872+0000 7f6a04400700 1 -- 172.21.15.63:0/2838905945 wait complete. 2024-03-23T11:48:48.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.873+0000 7f6a04400700 1 Processor -- start 2024-03-23T11:48:48.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.874+0000 7f6a04400700 1 -- start start 2024-03-23T11:48:48.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a04400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc066280 0x7f69fc10b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:48.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a04400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc0fff60 0x7f69fc10bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:48.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a04400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc10c270 con 0x7f69fc066280 2024-03-23T11:48:48.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a04400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc10cac0 con 0x7f69fc0fff60 2024-03-23T11:48:48.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a01a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc066280 0x7f69fc10b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:48.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a01000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc0fff60 0x7f69fc10bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:48.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a01000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc0fff60 0x7f69fc10bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33728/0 (socket says 172.21.15.63:33728) 2024-03-23T11:48:48.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a01000700 1 -- 172.21.15.63:0/495422321 learned_addr learned my addr 172.21.15.63:0/495422321 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:48.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.875+0000 7f6a01a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc066280 0x7f69fc10b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35140/0 (socket says 172.21.15.63:35140) 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.876+0000 7f6a01000700 1 -- 172.21.15.63:0/495422321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc066280 msgr2=0x7f69fc10b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.876+0000 7f6a01000700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc066280 0x7f69fc10b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.876+0000 7f6a01000700 1 -- 172.21.15.63:0/495422321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0005040 con 0x7f69fc0fff60 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.876+0000 7f6a01a00700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc066280 0x7f69fc10b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.876+0000 7f6a01000700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc0fff60 0x7f69fc10bc20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69e8002a80 tx=0x7f69e80046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.877+0000 7f69f9600700 1 -- 172.21.15.63:0/495422321 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69e8007cc0 con 0x7f69fc0fff60 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.877+0000 7f69f9600700 1 -- 172.21.15.63:0/495422321 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69e800f040 con 0x7f69fc0fff60 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.877+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc10cda0 con 0x7f69fc0fff60 2024-03-23T11:48:48.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.877+0000 7f69f9600700 1 -- 172.21.15.63:0/495422321 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69e8009640 con 0x7f69fc0fff60 2024-03-23T11:48:48.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.877+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc10d360 con 0x7f69fc0fff60 2024-03-23T11:48:48.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.879+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69fc05f5f0 con 0x7f69fc0fff60 2024-03-23T11:48:48.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.880+0000 7f69f9600700 1 -- 172.21.15.63:0/495422321 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 27) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f69e80097a0 con 0x7f69fc0fff60 2024-03-23T11:48:48.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.881+0000 7f69f9600700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f69e406c450 0x7f69e406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:48.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.881+0000 7f69f9600700 1 -- 172.21.15.63:0/495422321 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f69e808ab70 con 0x7f69fc0fff60 2024-03-23T11:48:48.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.882+0000 7f6a01a00700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f69e406c450 0x7f69e406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:48.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.882+0000 7f6a01a00700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f69e406c450 0x7f69e406e900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f69f0000c00 tx=0x7f69f0009e30 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:48.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:48.886+0000 7f69f9600700 1 -- 172.21.15.63:0/495422321 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f69e80553a0 con 0x7f69fc0fff60 2024-03-23T11:48:49.070 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:48 smithi063 ceph-mon[35189]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:48 smithi111 ceph-mon[44005]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.333+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f69fc04bed0 con 0x7f69fc0fff60 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.333+0000 7f69f9600700 1 -- 172.21.15.63:0/495422321 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f69e80589c0 con 0x7f69fc0fff60 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:49.319 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:48:49.320 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f69e406c450 msgr2=0x7f69e406e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f69e406c450 0x7f69e406e900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f69f0000c00 tx=0x7f69f0009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc0fff60 msgr2=0x7f69fc10bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc0fff60 0x7f69fc10bc20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69e8002a80 tx=0x7f69e80046b0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 shutdown_connections 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69fc066280 0x7f69fc10b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f69e406c450 0x7f69e406e900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 --2- 172.21.15.63:0/495422321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69fc0fff60 0x7f69fc10bc20 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69e8002a80 tx=0x7f69e80046b0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 >> 172.21.15.63:0/495422321 conn(0x7f69fc0f7430 msgr2=0x7f69fc0fbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 shutdown_connections 2024-03-23T11:48:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.337+0000 7f6a04400700 1 -- 172.21.15.63:0/495422321 wait complete. 2024-03-23T11:48:49.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.562+0000 7f23bde00700 1 -- 172.21.15.63:0/1379790621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 msgr2=0x7f23b80fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:49.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.562+0000 7f23bde00700 1 --2- 172.21.15.63:0/1379790621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b80fd310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f23a8004940 tx=0x7f23a8009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:49.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.563+0000 7f23bde00700 1 -- 172.21.15.63:0/1379790621 shutdown_connections 2024-03-23T11:48:49.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.563+0000 7f23bde00700 1 --2- 172.21.15.63:0/1379790621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23b80fe070 0x7f23b80fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.563+0000 7f23bde00700 1 --2- 172.21.15.63:0/1379790621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b80fd310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.563+0000 7f23bde00700 1 -- 172.21.15.63:0/1379790621 >> 172.21.15.63:0/1379790621 conn(0x7f23b80f85a0 msgr2=0x7f23b80fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:49.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.563+0000 7f23bde00700 1 -- 172.21.15.63:0/1379790621 shutdown_connections 2024-03-23T11:48:49.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.563+0000 7f23bde00700 1 -- 172.21.15.63:0/1379790621 wait complete. 2024-03-23T11:48:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.564+0000 7f23bde00700 1 Processor -- start 2024-03-23T11:48:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.564+0000 7f23bde00700 1 -- start start 2024-03-23T11:48:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.564+0000 7f23bde00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b8113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.564+0000 7f23bde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23b80fe070 0x7f23b8114400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.564+0000 7f23bde00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8114a00 con 0x7f23b80fe070 2024-03-23T11:48:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.564+0000 7f23bde00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8115250 con 0x7f23b80fcf00 2024-03-23T11:48:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.565+0000 7f23b6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23b80fe070 0x7f23b8114400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.565+0000 7f23b6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23b80fe070 0x7f23b8114400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35164/0 (socket says 172.21.15.63:35164) 2024-03-23T11:48:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.565+0000 7f23b6a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b8113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.565+0000 7f23b6a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b8113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33736/0 (socket says 172.21.15.63:33736) 2024-03-23T11:48:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.565+0000 7f23b6a00700 1 -- 172.21.15.63:0/352005802 learned_addr learned my addr 172.21.15.63:0/352005802 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:48:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.566+0000 7f23b6a00700 1 -- 172.21.15.63:0/352005802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23b80fe070 msgr2=0x7f23b8114400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.566+0000 7f23b6a00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23b80fe070 0x7f23b8114400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.566+0000 7f23b6a00700 1 -- 172.21.15.63:0/352005802 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a8005040 con 0x7f23b80fcf00 2024-03-23T11:48:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.567+0000 7f23b6a00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b8113ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f23a800b0d0 tx=0x7f23a8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.567+0000 7f23aea00700 1 -- 172.21.15.63:0/352005802 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a800e070 con 0x7f23b80fcf00 2024-03-23T11:48:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.568+0000 7f23aea00700 1 -- 172.21.15.63:0/352005802 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23a800b410 con 0x7f23b80fcf00 2024-03-23T11:48:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.568+0000 7f23aea00700 1 -- 172.21.15.63:0/352005802 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a8012600 con 0x7f23b80fcf00 2024-03-23T11:48:49.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.568+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b8115450 con 0x7f23b80fcf00 2024-03-23T11:48:49.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.568+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b8115a10 con 0x7f23b80fcf00 2024-03-23T11:48:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.572+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b805f5f0 con 0x7f23b80fcf00 2024-03-23T11:48:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.572+0000 7f23aea00700 1 -- 172.21.15.63:0/352005802 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 27) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f23a800b9f0 con 0x7f23b80fcf00 2024-03-23T11:48:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.574+0000 7f23aea00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f239c06c500 0x7f239c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:48:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.574+0000 7f23aea00700 1 -- 172.21.15.63:0/352005802 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f23a805e310 con 0x7f23b80fcf00 2024-03-23T11:48:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.575+0000 7f23b6000700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f239c06c500 0x7f239c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:48:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.576+0000 7f23b6000700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f239c06c500 0x7f239c06e9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009b80 tx=0x7f23a400b040 comp rx=0 tx=0).ready entity=mgr.24395 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:48:49.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.582+0000 7f23aea00700 1 -- 172.21.15.63:0/352005802 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f23a8057060 con 0x7f23b80fcf00 2024-03-23T11:48:49.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.947+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 --> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23b8115d70 con 0x7f239c06c500 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.948+0000 7f23aea00700 1 -- 172.21.15.63:0/352005802 <== mgr.24395 v2:172.21.15.111:6828/3542552256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f23b8115d70 con 0x7f239c06c500 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "2/2 daemons upgraded", 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mgr daemons", 2024-03-23T11:48:49.933 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:48:49.934 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:48:49.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.951+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f239c06c500 msgr2=0x7f239c06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:49.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.951+0000 7f23bde00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f239c06c500 0x7f239c06e9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009b80 tx=0x7f23a400b040 comp rx=0 tx=0).stop 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.951+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 msgr2=0x7f23b8113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.951+0000 7f23bde00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b8113ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f23a800b0d0 tx=0x7f23a8009e30 comp rx=0 tx=0).stop 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.952+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 shutdown_connections 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.952+0000 7f23bde00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:6828/3542552256,v1:172.21.15.111:6829/3542552256] conn(0x7f239c06c500 0x7f239c06e9b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.952+0000 7f23bde00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b80fcf00 0x7f23b8113ec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.952+0000 7f23bde00700 1 --2- 172.21.15.63:0/352005802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23b80fe070 0x7f23b8114400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.952+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 >> 172.21.15.63:0/352005802 conn(0x7f23b80f85a0 msgr2=0x7f23b80f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.952+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 shutdown_connections 2024-03-23T11:48:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:48:49.952+0000 7f23bde00700 1 -- 172.21.15.63:0/352005802 wait complete. 2024-03-23T11:48:50.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:49 smithi063 ceph-mon[35189]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:50.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:49 smithi063 ceph-mon[35189]: pgmap v74: 65 pgs: 65 active+clean; 297 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 515 op/s 2024-03-23T11:48:50.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:49 smithi063 ceph-mon[35189]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:50.134 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:49 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/495422321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:48:50.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:49 smithi111 ceph-mon[44005]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:50.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:49 smithi111 ceph-mon[44005]: pgmap v74: 65 pgs: 65 active+clean; 297 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 515 op/s 2024-03-23T11:48:50.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:49 smithi111 ceph-mon[44005]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:50.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:49 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/495422321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:48:52.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:51 smithi063 ceph-mon[35189]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:52.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:51 smithi063 ceph-mon[35189]: pgmap v75: 65 pgs: 65 active+clean; 297 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 346 op/s 2024-03-23T11:48:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:51 smithi111 ceph-mon[44005]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:48:52.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:51 smithi111 ceph-mon[44005]: pgmap v75: 65 pgs: 65 active+clean; 297 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 346 op/s 2024-03-23T11:48:53.602 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: pgmap v76: 65 pgs: 65 active+clean; 297 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 444 op/s 2024-03-23T11:48:53.603 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:53.603 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:53.603 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:48:53.603 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:48:53.603 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:53.603 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:53.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr fail", "who": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr fail", "who": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: osdmap e48: 6 total, 6 up, 6 in 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd='[{"prefix": "mgr fail", "who": "smithi111.nnblsk"}]': finished 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: mgrmap e28: smithi063.gehkcr(active, starting, since 0.00911735s) 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:48:53.884 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:53.885 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:53.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:53.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:48:53.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:48:53.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:48:53.886 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:53 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:48:53.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: pgmap v76: 65 pgs: 65 active+clean; 297 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 444 op/s 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 172.21.15.111:0/367972852' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr fail", "who": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "mgr fail", "who": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: osdmap e48: 6 total, 6 up, 6 in 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.24395 ' entity='mgr.smithi111.nnblsk' cmd='[{"prefix": "mgr fail", "who": "smithi111.nnblsk"}]': finished 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: mgrmap e28: smithi063.gehkcr(active, starting, since 0.00911735s) 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:48:53.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:48:53.905 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:48:53.906 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:53 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:48:55.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:54 smithi063 ceph-mon[35189]: mgrmap e29: smithi063.gehkcr(active, since 1.01737s) 2024-03-23T11:48:55.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:54 smithi063 ceph-mon[35189]: pgmap v3: 65 pgs: 65 active+clean; 293 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:48:55.250 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:54 smithi111 ceph-mon[44005]: mgrmap e29: smithi063.gehkcr(active, since 1.01737s) 2024-03-23T11:48:55.250 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:54 smithi111 ceph-mon[44005]: pgmap v3: 65 pgs: 65 active+clean; 293 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:48:56.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:55 smithi063 ceph-mon[35189]: [23/Mar/2024:11:48:54] ENGINE Bus STARTING 2024-03-23T11:48:56.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:55 smithi063 ceph-mon[35189]: [23/Mar/2024:11:48:54] ENGINE Serving on http://172.21.15.63:8765 2024-03-23T11:48:56.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:55 smithi063 ceph-mon[35189]: [23/Mar/2024:11:48:54] ENGINE Serving on https://172.21.15.63:7150 2024-03-23T11:48:56.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:55 smithi063 ceph-mon[35189]: [23/Mar/2024:11:48:54] ENGINE Bus STARTED 2024-03-23T11:48:56.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:55 smithi063 ceph-mon[35189]: pgmap v4: 65 pgs: 65 active+clean; 293 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:48:56.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:55 smithi111 ceph-mon[44005]: [23/Mar/2024:11:48:54] ENGINE Bus STARTING 2024-03-23T11:48:56.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:55 smithi111 ceph-mon[44005]: [23/Mar/2024:11:48:54] ENGINE Serving on http://172.21.15.63:8765 2024-03-23T11:48:56.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:55 smithi111 ceph-mon[44005]: [23/Mar/2024:11:48:54] ENGINE Serving on https://172.21.15.63:7150 2024-03-23T11:48:56.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:55 smithi111 ceph-mon[44005]: [23/Mar/2024:11:48:54] ENGINE Bus STARTED 2024-03-23T11:48:56.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:55 smithi111 ceph-mon[44005]: pgmap v4: 65 pgs: 65 active+clean; 293 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:48:57.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:56 smithi063 ceph-mon[35189]: mgrmap e30: smithi063.gehkcr(active, since 3s) 2024-03-23T11:48:57.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:56 smithi111 ceph-mon[44005]: mgrmap e30: smithi063.gehkcr(active, since 3s) 2024-03-23T11:48:58.254 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:57 smithi063 ceph-mon[35189]: pgmap v5: 65 pgs: 65 active+clean; 293 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:48:58.340 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:57 smithi111 ceph-mon[44005]: pgmap v5: 65 pgs: 65 active+clean; 293 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:49:00.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:48:59 smithi111 ceph-mon[44005]: pgmap v6: 65 pgs: 65 active+clean; 292 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 25 KiB/s wr, 44 op/s 2024-03-23T11:49:00.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:48:59 smithi063 ceph-mon[35189]: pgmap v6: 65 pgs: 65 active+clean; 292 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 25 KiB/s rd, 25 KiB/s wr, 44 op/s 2024-03-23T11:49:02.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:02 smithi063 ceph-mon[35189]: pgmap v7: 65 pgs: 65 active+clean; 288 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 287 op/s 2024-03-23T11:49:02.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:02 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:02.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:02 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:02.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:02 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:02.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:02 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:02.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:02 smithi111 ceph-mon[44005]: pgmap v7: 65 pgs: 65 active+clean; 288 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 287 op/s 2024-03-23T11:49:02.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:02 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:02.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:02 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:02.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:02 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:02.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:02 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: pgmap v8: 65 pgs: 65 active+clean; 288 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 804 KiB/s rd, 842 KiB/s wr, 223 op/s 2024-03-23T11:49:04.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:49:04.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:49:04.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:49:04.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:49:04.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:03 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: pgmap v8: 65 pgs: 65 active+clean; 288 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 804 KiB/s rd, 842 KiB/s wr, 223 op/s 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:49:04.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:49:04.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:49:04.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:03 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/1553258919' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:49:06.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:05 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:49:06.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:05 smithi063 ceph-mon[35189]: mgrmap e31: smithi063.gehkcr(active, since 11s), standbys: smithi111.nnblsk 2024-03-23T11:49:06.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:05 smithi063 ceph-mon[35189]: pgmap v9: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 387 op/s 2024-03-23T11:49:06.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:05 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:49:06.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:05 smithi111 ceph-mon[44005]: mgrmap e31: smithi063.gehkcr(active, since 11s), standbys: smithi111.nnblsk 2024-03-23T11:49:06.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:05 smithi111 ceph-mon[44005]: pgmap v9: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 387 op/s 2024-03-23T11:49:08.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:08 smithi063 ceph-mon[35189]: pgmap v10: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-23T11:49:08.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:08 smithi111 ceph-mon[44005]: pgmap v10: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-23T11:49:09.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:09 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:09.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:09 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:10.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:10 smithi063 ceph-mon[35189]: pgmap v11: 65 pgs: 65 active+clean; 289 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 383 op/s 2024-03-23T11:49:10.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:10 smithi111 ceph-mon[44005]: pgmap v11: 65 pgs: 65 active+clean; 289 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 383 op/s 2024-03-23T11:49:12.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:12 smithi063 ceph-mon[35189]: pgmap v12: 65 pgs: 65 active+clean; 287 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 491 op/s 2024-03-23T11:49:12.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:12 smithi111 ceph-mon[44005]: pgmap v12: 65 pgs: 65 active+clean; 287 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 491 op/s 2024-03-23T11:49:13.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:13 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:13.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:13 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:13.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:49:13.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:49:13.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:13 smithi063 ceph-mon[35189]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:49:13.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:13 smithi063 ceph-mon[35189]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:49:13.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:13 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:13.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:13 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:13.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:49:13.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:49:13.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:13 smithi111 ceph-mon[44005]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:49:13.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:13 smithi111 ceph-mon[44005]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:49:14.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:14 smithi063 ceph-mon[35189]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:49:14.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:14 smithi063 ceph-mon[35189]: pgmap v13: 65 pgs: 65 active+clean; 287 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 343 op/s 2024-03-23T11:49:14.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:14 smithi063 ceph-mon[35189]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:49:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:14 smithi063 ceph-mon[35189]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:49:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:14 smithi111 ceph-mon[44005]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:49:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:14 smithi111 ceph-mon[44005]: pgmap v13: 65 pgs: 65 active+clean; 287 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 343 op/s 2024-03-23T11:49:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:14 smithi111 ceph-mon[44005]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:49:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:14 smithi111 ceph-mon[44005]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:49:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:15 smithi063 ceph-mon[35189]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:49:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:15 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:15 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:15 smithi111 ceph-mon[44005]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:49:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:15 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:15 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:16.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:16 smithi063 ceph-mon[35189]: pgmap v14: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 485 op/s 2024-03-23T11:49:16.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:16 smithi111 ceph-mon[44005]: pgmap v14: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 485 op/s 2024-03-23T11:49:18.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:18 smithi063 ceph-mon[35189]: pgmap v15: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 297 op/s 2024-03-23T11:49:18.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:18 smithi063 ceph-mon[35189]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:49:18.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:18 smithi111 ceph-mon[44005]: pgmap v15: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 297 op/s 2024-03-23T11:49:18.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:18 smithi111 ceph-mon[44005]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:49:19.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:19 smithi063 ceph-mon[35189]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:49:19.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:19 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:19.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:19 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:19.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:19 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:19.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:19 smithi111 ceph-mon[44005]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:49:19.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:19 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:19.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:19 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:19.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:19 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:20.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 -- 172.21.15.63:0/1451638854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 msgr2=0x7fd3340ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:20.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 --2- 172.21.15.63:0/1451638854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 0x7fd3340ff9c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd328004940 tx=0x7fd328009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:20.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 -- 172.21.15.63:0/1451638854 shutdown_connections 2024-03-23T11:49:20.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 --2- 172.21.15.63:0/1451638854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 0x7fd334102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 --2- 172.21.15.63:0/1451638854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 0x7fd3340ff9c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 -- 172.21.15.63:0/1451638854 >> 172.21.15.63:0/1451638854 conn(0x7fd3340f7430 msgr2=0x7fd3340f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:20.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 -- 172.21.15.63:0/1451638854 shutdown_connections 2024-03-23T11:49:20.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.195+0000 7fd33ba00700 1 -- 172.21.15.63:0/1451638854 wait complete. 2024-03-23T11:49:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.196+0000 7fd33ba00700 1 Processor -- start 2024-03-23T11:49:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.197+0000 7fd33ba00700 1 -- start start 2024-03-23T11:49:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.198+0000 7fd33ba00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 0x7fd334113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.198+0000 7fd33ba00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 0x7fd334114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.198+0000 7fd33ba00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd334114b20 con 0x7fd3340662e0 2024-03-23T11:49:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.198+0000 7fd33ba00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd334115310 con 0x7fd3340fff00 2024-03-23T11:49:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.198+0000 7fd333e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 0x7fd334114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.198+0000 7fd333e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 0x7fd334114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:35484/0 (socket says 172.21.15.63:35484) 2024-03-23T11:49:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.199+0000 7fd333e00700 1 -- 172.21.15.63:0/353539409 learned_addr learned my addr 172.21.15.63:0/353539409 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.199+0000 7fd339000700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 0x7fd334113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.199+0000 7fd333e00700 1 -- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 msgr2=0x7fd334113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.199+0000 7fd333e00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 0x7fd334113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.199+0000 7fd333e00700 1 -- 172.21.15.63:0/353539409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd328005040 con 0x7fd3340fff00 2024-03-23T11:49:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.199+0000 7fd339000700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 0x7fd334113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:49:20.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.200+0000 7fd333e00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 0x7fd334114490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd32400b790 tx=0x7fd324009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.201+0000 7fd330c00700 1 -- 172.21.15.63:0/353539409 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32400f770 con 0x7fd3340fff00 2024-03-23T11:49:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.201+0000 7fd330c00700 1 -- 172.21.15.63:0/353539409 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd32400fd80 con 0x7fd3340fff00 2024-03-23T11:49:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.201+0000 7fd330c00700 1 -- 172.21.15.63:0/353539409 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32400d310 con 0x7fd3340fff00 2024-03-23T11:49:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.201+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3341155f0 con 0x7fd3340fff00 2024-03-23T11:49:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.201+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd334115bb0 con 0x7fd3340fff00 2024-03-23T11:49:20.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.205+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33405f5f0 con 0x7fd3340fff00 2024-03-23T11:49:20.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.205+0000 7fd330c00700 1 -- 172.21.15.63:0/353539409 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90487+0+0 (secure 0 0 0) 0x7fd324010420 con 0x7fd3340fff00 2024-03-23T11:49:20.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.206+0000 7fd330c00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd31c06c620 0x7fd31c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:20.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.206+0000 7fd330c00700 1 -- 172.21.15.63:0/353539409 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd32408b630 con 0x7fd3340fff00 2024-03-23T11:49:20.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.207+0000 7fd339000700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd31c06c620 0x7fd31c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:20.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.208+0000 7fd339000700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd31c06c620 0x7fd31c06ead0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd32800b3b0 tx=0x7fd328009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:20.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.211+0000 7fd330c00700 1 -- 172.21.15.63:0/353539409 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fd324059730 con 0x7fd3340fff00 2024-03-23T11:49:20.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.612+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd334115f00 con 0x7fd31c06c620 2024-03-23T11:49:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.615+0000 7fd330c00700 1 -- 172.21.15.63:0/353539409 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fd334115f00 con 0x7fd31c06c620 2024-03-23T11:49:20.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd31c06c620 msgr2=0x7fd31c06ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd31c06c620 0x7fd31c06ead0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd32800b3b0 tx=0x7fd328009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 msgr2=0x7fd334114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 0x7fd334114490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd32400b790 tx=0x7fd324009250 comp rx=0 tx=0).stop 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 shutdown_connections 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd31c06c620 0x7fd31c06ead0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3340662e0 0x7fd334113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 --2- 172.21.15.63:0/353539409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3340fff00 0x7fd334114490 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 >> 172.21.15.63:0/353539409 conn(0x7fd3340f7430 msgr2=0x7fd3340fbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 shutdown_connections 2024-03-23T11:49:20.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.618+0000 7fd33ba00700 1 -- 172.21.15.63:0/353539409 wait complete. 2024-03-23T11:49:20.620 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:49:20.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:20 smithi063 ceph-mon[35189]: pgmap v16: 65 pgs: 65 active+clean; 287 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 333 op/s 2024-03-23T11:49:20.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:20 smithi063 ceph-mon[35189]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-03-23T11:49:20.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:20 smithi111 ceph-mon[44005]: pgmap v16: 65 pgs: 65 active+clean; 287 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 333 op/s 2024-03-23T11:49:20.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:20 smithi111 ceph-mon[44005]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-03-23T11:49:20.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 -- 172.21.15.63:0/2634830857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102360 msgr2=0x7fad68104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:20.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 --2- 172.21.15.63:0/2634830857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102360 0x7fad68104740 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fad54004880 tx=0x7fad54009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:20.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 -- 172.21.15.63:0/2634830857 shutdown_connections 2024-03-23T11:49:20.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 --2- 172.21.15.63:0/2634830857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68102360 0x7fad68104740 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 --2- 172.21.15.63:0/2634830857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad68066280 0x7fad68101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 -- 172.21.15.63:0/2634830857 >> 172.21.15.63:0/2634830857 conn(0x7fad680f7430 msgr2=0x7fad680f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:20.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 -- 172.21.15.63:0/2634830857 shutdown_connections 2024-03-23T11:49:20.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.825+0000 7fad6ea00700 1 -- 172.21.15.63:0/2634830857 wait complete. 2024-03-23T11:49:20.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.827+0000 7fad6ea00700 1 Processor -- start 2024-03-23T11:49:20.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.827+0000 7fad6ea00700 1 -- start start 2024-03-23T11:49:20.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.827+0000 7fad6ea00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68066280 0x7fad6810fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:20.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.827+0000 7fad6ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad68102360 0x7fad68110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:20.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.827+0000 7fad6ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad68110600 con 0x7fad68102360 2024-03-23T11:49:20.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.827+0000 7fad6ea00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad68110e50 con 0x7fad68066280 2024-03-23T11:49:20.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.828+0000 7fad67e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68066280 0x7fad6810fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:20.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.828+0000 7fad67e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68066280 0x7fad6810fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:35512/0 (socket says 172.21.15.63:35512) 2024-03-23T11:49:20.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.828+0000 7fad67e00700 1 -- 172.21.15.63:0/165028316 learned_addr learned my addr 172.21.15.63:0/165028316 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:20.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.828+0000 7fad67400700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad68102360 0x7fad68110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:20.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.829+0000 7fad67e00700 1 -- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad68102360 msgr2=0x7fad68110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:20.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.829+0000 7fad67e00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad68102360 0x7fad68110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:20.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.829+0000 7fad67e00700 1 -- 172.21.15.63:0/165028316 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad54005040 con 0x7fad68066280 2024-03-23T11:49:20.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.829+0000 7fad67400700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad68102360 0x7fad68110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:49:20.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.829+0000 7fad67e00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68066280 0x7fad6810fac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad5800eb40 tx=0x7fad5800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:20.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.830+0000 7fad5fe00700 1 -- 172.21.15.63:0/165028316 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad58007cc0 con 0x7fad68066280 2024-03-23T11:49:20.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.830+0000 7fad5fe00700 1 -- 172.21.15.63:0/165028316 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad5800f040 con 0x7fad68066280 2024-03-23T11:49:20.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.830+0000 7fad5fe00700 1 -- 172.21.15.63:0/165028316 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad58007cc0 con 0x7fad68066280 2024-03-23T11:49:20.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.830+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad68111130 con 0x7fad68066280 2024-03-23T11:49:20.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.830+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad681115b0 con 0x7fad68066280 2024-03-23T11:49:20.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.832+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6805f5f0 con 0x7fad68066280 2024-03-23T11:49:20.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.833+0000 7fad5fe00700 1 -- 172.21.15.63:0/165028316 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90487+0+0 (secure 0 0 0) 0x7fad5800c870 con 0x7fad68066280 2024-03-23T11:49:20.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.834+0000 7fad5fe00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fad4c06c5d0 0x7fad4c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:20.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.834+0000 7fad5fe00700 1 -- 172.21.15.63:0/165028316 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fad58007e20 con 0x7fad68066280 2024-03-23T11:49:20.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.834+0000 7fad67400700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fad4c06c5d0 0x7fad4c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:20.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.835+0000 7fad67400700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fad4c06c5d0 0x7fad4c06ea80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fad54009fd0 tx=0x7fad54009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:20.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:20.842+0000 7fad5fe00700 1 -- 172.21.15.63:0/165028316 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fad58055f80 con 0x7fad68066280 2024-03-23T11:49:21.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.262+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad681119c0 con 0x7fad4c06c5d0 2024-03-23T11:49:21.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.263+0000 7fad5fe00700 1 -- 172.21.15.63:0/165028316 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fad681119c0 con 0x7fad4c06c5d0 2024-03-23T11:49:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fad4c06c5d0 msgr2=0x7fad4c06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fad4c06c5d0 0x7fad4c06ea80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fad54009fd0 tx=0x7fad54009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68066280 msgr2=0x7fad6810fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68066280 0x7fad6810fac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad5800eb40 tx=0x7fad5800b040 comp rx=0 tx=0).stop 2024-03-23T11:49:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 shutdown_connections 2024-03-23T11:49:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fad4c06c5d0 0x7fad4c06ea80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad68066280 0x7fad6810fac0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 --2- 172.21.15.63:0/165028316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad68102360 0x7fad68110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.266+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 >> 172.21.15.63:0/165028316 conn(0x7fad680f7430 msgr2=0x7fad680f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.267+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 shutdown_connections 2024-03-23T11:49:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.267+0000 7fad6ea00700 1 -- 172.21.15.63:0/165028316 wait complete. 2024-03-23T11:49:21.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.462+0000 7f5ca4600700 1 -- 172.21.15.63:0/816114779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 msgr2=0x7f5c9c100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:21.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.462+0000 7f5ca4600700 1 --2- 172.21.15.63:0/816114779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 0x7f5c9c100790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c88004940 tx=0x7f5c88009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:21.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.462+0000 7f5ca4600700 1 -- 172.21.15.63:0/816114779 shutdown_connections 2024-03-23T11:49:21.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.462+0000 7f5ca4600700 1 --2- 172.21.15.63:0/816114779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 0x7f5c9c100790 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.463+0000 7f5ca4600700 1 --2- 172.21.15.63:0/816114779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 0x7f5c9c0ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.463+0000 7f5ca4600700 1 -- 172.21.15.63:0/816114779 >> 172.21.15.63:0/816114779 conn(0x7f5c9c0fa950 msgr2=0x7f5c9c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:21.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.463+0000 7f5ca4600700 1 -- 172.21.15.63:0/816114779 shutdown_connections 2024-03-23T11:49:21.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.463+0000 7f5ca4600700 1 -- 172.21.15.63:0/816114779 wait complete. 2024-03-23T11:49:21.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.464+0000 7f5ca4600700 1 Processor -- start 2024-03-23T11:49:21.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.465+0000 7f5ca4600700 1 -- start start 2024-03-23T11:49:21.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.465+0000 7f5ca4600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 0x7f5c9c10fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:21.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.465+0000 7f5ca4600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 0x7f5c9c10fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:21.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.465+0000 7f5ca4600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c1105f0 con 0x7f5c9c0ff070 2024-03-23T11:49:21.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.465+0000 7f5ca4600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c110e40 con 0x7f5c9c100320 2024-03-23T11:49:21.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.466+0000 7f5ca1c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 0x7f5c9c10fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:21.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.466+0000 7f5ca1c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 0x7f5c9c10fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52886/0 (socket says 172.21.15.63:52886) 2024-03-23T11:49:21.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.466+0000 7f5ca1200700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 0x7f5c9c10fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:21.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.466+0000 7f5ca1c00700 1 -- 172.21.15.63:0/1699528 learned_addr learned my addr 172.21.15.63:0/1699528 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:21.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.467+0000 7f5ca1200700 1 -- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 msgr2=0x7f5c9c10fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:21.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.467+0000 7f5ca1200700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 0x7f5c9c10fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.467+0000 7f5ca1200700 1 -- 172.21.15.63:0/1699528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88005040 con 0x7f5c9c100320 2024-03-23T11:49:21.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.467+0000 7f5ca1c00700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 0x7f5c9c10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:49:21.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.467+0000 7f5ca1200700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 0x7f5c9c10fff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c88000c00 tx=0x7f5c88009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:21.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.468+0000 7f5c99600700 1 -- 172.21.15.63:0/1699528 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8800e030 con 0x7f5c9c100320 2024-03-23T11:49:21.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.468+0000 7f5c99600700 1 -- 172.21.15.63:0/1699528 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c880093b0 con 0x7f5c9c100320 2024-03-23T11:49:21.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.468+0000 7f5c99600700 1 -- 172.21.15.63:0/1699528 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8801cdb0 con 0x7f5c9c100320 2024-03-23T11:49:21.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.469+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9c1110c0 con 0x7f5c9c100320 2024-03-23T11:49:21.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.469+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9c111600 con 0x7f5c9c100320 2024-03-23T11:49:21.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.471+0000 7f5c99600700 1 -- 172.21.15.63:0/1699528 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90487+0+0 (secure 0 0 0) 0x7f5c8801c600 con 0x7f5c9c100320 2024-03-23T11:49:21.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.471+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9c05f5f0 con 0x7f5c9c100320 2024-03-23T11:49:21.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.472+0000 7f5c99600700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f5c8406c620 0x7f5c8406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:21.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.472+0000 7f5c99600700 1 -- 172.21.15.63:0/1699528 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5c8808cfb0 con 0x7f5c9c100320 2024-03-23T11:49:21.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.473+0000 7f5ca1c00700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f5c8406c620 0x7f5c8406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:21.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.474+0000 7f5ca1c00700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f5c8406c620 0x7f5c8406ead0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5c900041d0 tx=0x7f5c9000b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:21.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.477+0000 7f5c99600700 1 -- 172.21.15.63:0/1699528 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f5c8805afa0 con 0x7f5c9c100320 2024-03-23T11:49:21.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:21 smithi063 ceph-mon[35189]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-03-23T11:49:21.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:21 smithi111 ceph-mon[44005]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-03-23T11:49:21.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.843+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c9c111980 con 0x7f5c8406c620 2024-03-23T11:49:21.858 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:49:21.858 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (2m) 19s ago 11m 14.7M - 0.25.0 c8568f914cd2 106bde4987fa 2024-03-23T11:49:21.858 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (12m) 19s ago 12m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (10m) 20s ago 10m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (95s) 19s ago 11m 77.2M - 9.4.7 954c08fa6188 d4ed1ee72841 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (5m) 19s ago 5m 147M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (5m) 19s ago 5m 142M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (5m) 20s ago 5m 84.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (5m) 20s ago 5m 76.4M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (61s) 19s ago 13m 471M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283 running (3m) 20s ago 10m 67.0M - 18.2.2-820-gb630d6cd 881b71eac2fe 278e3d19ec5d 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (13m) 19s ago 13m 53.4M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (10m) 20s ago 10m 42.4M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (2m) 19s ago 11m 7512k - 1.5.0 0da6a335fe13 9387979cc7ad 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (69s) 20s ago 10m 7666k - 1.5.0 0da6a335fe13 fd96d8076254 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (9m) 19s ago 9m 203M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (9m) 19s ago 9m 214M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:49:21.859 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (8m) 19s ago 8m 177M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:49:21.860 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (8m) 20s ago 8m 242M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:49:21.860 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (7m) 20s ago 7m 214M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:49:21.860 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (7m) 20s ago 7m 214M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:49:21.860 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (72s) 19s ago 11m 41.1M - 2.43.0 a07b618ecd1d f74e0fbb00ad 2024-03-23T11:49:21.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.873+0000 7f5c99600700 1 -- 172.21.15.63:0/1699528 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f5c9c111980 con 0x7f5c8406c620 2024-03-23T11:49:21.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f5c8406c620 msgr2=0x7f5c8406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:21.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f5c8406c620 0x7f5c8406ead0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5c900041d0 tx=0x7f5c9000b040 comp rx=0 tx=0).stop 2024-03-23T11:49:21.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 msgr2=0x7f5c9c10fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:21.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 0x7f5c9c10fff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c88000c00 tx=0x7f5c88009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:21.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 shutdown_connections 2024-03-23T11:49:21.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f5c8406c620 0x7f5c8406ead0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c9c0ff070 0x7f5c9c10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 --2- 172.21.15.63:0/1699528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c9c100320 0x7f5c9c10fff0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.878+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 >> 172.21.15.63:0/1699528 conn(0x7f5c9c0fa950 msgr2=0x7f5c9c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.879+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 shutdown_connections 2024-03-23T11:49:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:21.879+0000 7f5ca4600700 1 -- 172.21.15.63:0/1699528 wait complete. 2024-03-23T11:49:22.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.118+0000 7fee39200700 1 -- 172.21.15.63:0/3846150968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 msgr2=0x7fee34070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:22.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.118+0000 7fee39200700 1 --2- 172.21.15.63:0/3846150968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 0x7fee34070460 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fee20004940 tx=0x7fee20009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:22.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.118+0000 7fee39200700 1 -- 172.21.15.63:0/3846150968 shutdown_connections 2024-03-23T11:49:22.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.118+0000 7fee39200700 1 --2- 172.21.15.63:0/3846150968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 0x7fee34070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.118+0000 7fee39200700 1 --2- 172.21.15.63:0/3846150968 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 0x7fee34070460 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.118+0000 7fee39200700 1 -- 172.21.15.63:0/3846150968 >> 172.21.15.63:0/3846150968 conn(0x7fee340f8560 msgr2=0x7fee340fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:22.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.119+0000 7fee39200700 1 -- 172.21.15.63:0/3846150968 shutdown_connections 2024-03-23T11:49:22.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.119+0000 7fee39200700 1 -- 172.21.15.63:0/3846150968 wait complete. 2024-03-23T11:49:22.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.120+0000 7fee39200700 1 Processor -- start 2024-03-23T11:49:22.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.120+0000 7fee39200700 1 -- start start 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.120+0000 7fee39200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 0x7fee3411cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.120+0000 7fee39200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 0x7fee3411d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.120+0000 7fee39200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3411d7d0 con 0x7fee34071f50 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.120+0000 7fee39200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3411db10 con 0x7fee340709a0 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.121+0000 7fee31e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 0x7fee3411cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.121+0000 7fee31e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 0x7fee3411cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:39990/0 (socket says 172.21.15.63:39990) 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.121+0000 7fee31400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 0x7fee3411d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:22.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.121+0000 7fee31e00700 1 -- 172.21.15.63:0/2929076737 learned_addr learned my addr 172.21.15.63:0/2929076737 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.121+0000 7fee31e00700 1 -- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 msgr2=0x7fee3411d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.121+0000 7fee31e00700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 0x7fee3411d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.121+0000 7fee31e00700 1 -- 172.21.15.63:0/2929076737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20005040 con 0x7fee340709a0 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.122+0000 7fee31e00700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 0x7fee3411cbb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fee2000b900 tx=0x7fee20009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.122+0000 7fee2a000700 1 -- 172.21.15.63:0/2929076737 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee2000e040 con 0x7fee340709a0 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.122+0000 7fee2a000700 1 -- 172.21.15.63:0/2929076737 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee20004290 con 0x7fee340709a0 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.122+0000 7fee2a000700 1 -- 172.21.15.63:0/2929076737 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee2001c3f0 con 0x7fee340709a0 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.122+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3411dd90 con 0x7fee340709a0 2024-03-23T11:49:22.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.122+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3411e350 con 0x7fee340709a0 2024-03-23T11:49:22.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.124+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3405f5f0 con 0x7fee340709a0 2024-03-23T11:49:22.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.125+0000 7fee2a000700 1 -- 172.21.15.63:0/2929076737 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90487+0+0 (secure 0 0 0) 0x7fee2000bb80 con 0x7fee340709a0 2024-03-23T11:49:22.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.126+0000 7fee2a000700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fee1806c560 0x7fee1806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:22.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.126+0000 7fee2a000700 1 -- 172.21.15.63:0/2929076737 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fee2008c9d0 con 0x7fee340709a0 2024-03-23T11:49:22.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.127+0000 7fee31400700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fee1806c560 0x7fee1806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.127+0000 7fee31400700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fee1806c560 0x7fee1806ea10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fee24009bf0 tx=0x7fee240091a0 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:22.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.131+0000 7fee2a000700 1 -- 172.21.15.63:0/2929076737 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fee20056f30 con 0x7fee340709a0 2024-03-23T11:49:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.586+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fee34067f90 con 0x7fee340709a0 2024-03-23T11:49:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.586+0000 7fee2a000700 1 -- 172.21.15.63:0/2929076737 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7fee2005a550 con 0x7fee340709a0 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:49:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-23T11:49:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:49:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:49:22.573 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fee1806c560 msgr2=0x7fee1806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fee1806c560 0x7fee1806ea10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fee24009bf0 tx=0x7fee240091a0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 msgr2=0x7fee3411cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 0x7fee3411cbb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fee2000b900 tx=0x7fee20009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 shutdown_connections 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fee1806c560 0x7fee1806ea10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee340709a0 0x7fee3411cbb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 --2- 172.21.15.63:0/2929076737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee34071f50 0x7fee3411d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 >> 172.21.15.63:0/2929076737 conn(0x7fee340f8560 msgr2=0x7fee340f9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 shutdown_connections 2024-03-23T11:49:22.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.590+0000 7fee39200700 1 -- 172.21.15.63:0/2929076737 wait complete. 2024-03-23T11:49:22.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:22 smithi063 ceph-mon[35189]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:22.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:22 smithi063 ceph-mon[35189]: pgmap v17: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 424 op/s 2024-03-23T11:49:22.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:22 smithi063 ceph-mon[35189]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:22.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:22 smithi111 ceph-mon[44005]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:22.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:22 smithi111 ceph-mon[44005]: pgmap v17: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 424 op/s 2024-03-23T11:49:22.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:22 smithi111 ceph-mon[44005]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:22.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.786+0000 7f0b07e00700 1 -- 172.21.15.63:0/2431193721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 msgr2=0x7f0b000ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:22.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.786+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2431193721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b000ff430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0af8004940 tx=0x7f0af8009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:22.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.787+0000 7f0b07e00700 1 -- 172.21.15.63:0/2431193721 shutdown_connections 2024-03-23T11:49:22.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.787+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2431193721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b001002d0 0x7f0b00100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.787+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2431193721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b000ff430 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.787+0000 7f0b07e00700 1 -- 172.21.15.63:0/2431193721 >> 172.21.15.63:0/2431193721 conn(0x7f0b000fa950 msgr2=0x7f0b000fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.788+0000 7f0b07e00700 1 -- 172.21.15.63:0/2431193721 shutdown_connections 2024-03-23T11:49:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.788+0000 7f0b07e00700 1 -- 172.21.15.63:0/2431193721 wait complete. 2024-03-23T11:49:22.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.789+0000 7f0b07e00700 1 Processor -- start 2024-03-23T11:49:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.789+0000 7f0b07e00700 1 -- start start 2024-03-23T11:49:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.790+0000 7f0b07e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b0010b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.790+0000 7f0b07e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b001002d0 0x7f0b0010bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.790+0000 7f0b07e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0010c140 con 0x7f0b001002d0 2024-03-23T11:49:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.790+0000 7f0b07e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0010c2b0 con 0x7f0b000ff020 2024-03-23T11:49:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.791+0000 7f0b05400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b0010b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.791+0000 7f0b05400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b0010b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:40006/0 (socket says 172.21.15.63:40006) 2024-03-23T11:49:22.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.791+0000 7f0b05400700 1 -- 172.21.15.63:0/2880022086 learned_addr learned my addr 172.21.15.63:0/2880022086 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:22.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.792+0000 7f0b05400700 1 -- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b001002d0 msgr2=0x7f0b0010bb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:49:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.792+0000 7f0b04a00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b001002d0 0x7f0b0010bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.792+0000 7f0b05400700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b001002d0 0x7f0b0010bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.792+0000 7f0b05400700 1 -- 172.21.15.63:0/2880022086 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af8005040 con 0x7f0b000ff020 2024-03-23T11:49:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.792+0000 7f0b04a00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b001002d0 0x7f0b0010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:49:22.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.793+0000 7f0b05400700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b0010b600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0af800b3b0 tx=0x7f0af8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:22.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.794+0000 7f0afd600700 1 -- 172.21.15.63:0/2880022086 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0af800e030 con 0x7f0b000ff020 2024-03-23T11:49:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.794+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0010cc40 con 0x7f0b000ff020 2024-03-23T11:49:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.794+0000 7f0afd600700 1 -- 172.21.15.63:0/2880022086 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0af8013420 con 0x7f0b000ff020 2024-03-23T11:49:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.794+0000 7f0afd600700 1 -- 172.21.15.63:0/2880022086 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0af801c640 con 0x7f0b000ff020 2024-03-23T11:49:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.795+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0010d140 con 0x7f0b000ff020 2024-03-23T11:49:22.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.797+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae00052e0 con 0x7f0b000ff020 2024-03-23T11:49:22.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.799+0000 7f0afd600700 1 -- 172.21.15.63:0/2880022086 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90487+0+0 (secure 0 0 0) 0x7f0af801c7a0 con 0x7f0b000ff020 2024-03-23T11:49:22.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.801+0000 7f0afd600700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0ae406c5a0 0x7f0ae406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:22.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.801+0000 7f0afd600700 1 -- 172.21.15.63:0/2880022086 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0af808cd00 con 0x7f0b000ff020 2024-03-23T11:49:22.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.801+0000 7f0b04a00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0ae406c5a0 0x7f0ae406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:22.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.805+0000 7f0b04a00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0ae406c5a0 0x7f0ae406ea50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0aec004e80 tx=0x7f0aec009040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:22.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:22.808+0000 7f0afd600700 1 -- 172.21.15.63:0/2880022086 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0af8090030 con 0x7f0b000ff020 2024-03-23T11:49:23.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.219+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ae0000bf0 con 0x7f0ae406c5a0 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.220+0000 7f0afd600700 1 -- 172.21.15.63:0/2880022086 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f0ae0000bf0 con 0x7f0ae406c5a0 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "2/2 daemons upgraded", 2024-03-23T11:49:23.205 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "", 2024-03-23T11:49:23.206 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:49:23.206 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:49:23.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0ae406c5a0 msgr2=0x7f0ae406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0ae406c5a0 0x7f0ae406ea50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0aec004e80 tx=0x7f0aec009040 comp rx=0 tx=0).stop 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 msgr2=0x7f0b0010b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b0010b600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0af800b3b0 tx=0x7f0af8009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 shutdown_connections 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0ae406c5a0 0x7f0ae406ea50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b000ff020 0x7f0b0010b600 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 --2- 172.21.15.63:0/2880022086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b001002d0 0x7f0b0010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 >> 172.21.15.63:0/2880022086 conn(0x7f0b000fa950 msgr2=0x7f0b000fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 shutdown_connections 2024-03-23T11:49:23.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:23.223+0000 7f0b07e00700 1 -- 172.21.15.63:0/2880022086 wait complete. 2024-03-23T11:49:23.459 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-23T11:49:23.460 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-03-23T11:49:23.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:23 smithi063 ceph-mon[35189]: from='client.24543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:23.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:23 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2929076737' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:23.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:23 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:23.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:23 smithi111 ceph-mon[44005]: from='client.24543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:23.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:23 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2929076737' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:23.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:23 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:24.487 DEBUG:teuthology.parallel:result is None 2024-03-23T11:49:24.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:24 smithi063 ceph-mon[35189]: pgmap v18: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 297 op/s 2024-03-23T11:49:24.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:24 smithi063 ceph-mon[35189]: from='client.24549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:24.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:24 smithi111 ceph-mon[44005]: pgmap v18: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 297 op/s 2024-03-23T11:49:24.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:24 smithi111 ceph-mon[44005]: from='client.24549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:26.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:26 smithi063 ceph-mon[35189]: pgmap v19: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 434 op/s 2024-03-23T11:49:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:26 smithi111 ceph-mon[44005]: pgmap v19: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 434 op/s 2024-03-23T11:49:28.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:28 smithi063 ceph-mon[35189]: pgmap v20: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1024 KiB/s rd, 1.0 MiB/s wr, 292 op/s 2024-03-23T11:49:28.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:28 smithi111 ceph-mon[44005]: pgmap v20: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1024 KiB/s rd, 1.0 MiB/s wr, 292 op/s 2024-03-23T11:49:30.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:30 smithi063 ceph-mon[35189]: pgmap v21: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1024 KiB/s rd, 1.0 MiB/s wr, 292 op/s 2024-03-23T11:49:30.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:30 smithi111 ceph-mon[44005]: pgmap v21: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1024 KiB/s rd, 1.0 MiB/s wr, 292 op/s 2024-03-23T11:49:32.318 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-23T11:49:32.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-23T11:49:32.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:32 smithi063 ceph-mon[35189]: pgmap v22: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 385 op/s 2024-03-23T11:49:32.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:32 smithi111 ceph-mon[44005]: pgmap v22: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 999 KiB/s rd, 1.1 MiB/s wr, 385 op/s 2024-03-23T11:49:33.459 DEBUG:teuthology.parallel:result is None 2024-03-23T11:49:33.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-23T11:49:33.544 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-23T11:49:33.544 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-03-23T11:49:33.627 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-03-23T11:49:33.628 DEBUG:teuthology.parallel:result is None 2024-03-23T11:49:34.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:34 smithi063 ceph-mon[35189]: pgmap v23: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 797 KiB/s rd, 878 KiB/s wr, 265 op/s 2024-03-23T11:49:34.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:34 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:34.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:34 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:34.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:49:34.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:34 smithi063 ceph-mon[35189]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:49:34.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:49:34.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:34 smithi111 ceph-mon[44005]: pgmap v23: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 797 KiB/s rd, 878 KiB/s wr, 265 op/s 2024-03-23T11:49:34.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:34 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:34.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:34 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:34.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:49:34.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:34 smithi111 ceph-mon[44005]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:49:34.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:49:35.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:35 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:49:35.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:35 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:49:35.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:35 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:49:35.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:35 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:49:35.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:35 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:49:35.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:35 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:49:35.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:35 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:49:35.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:35 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:49:36.379 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:36 smithi111 ceph-mon[44005]: Upgrade: Updating mgr.smithi111.nnblsk 2024-03-23T11:49:36.379 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:36 smithi111 ceph-mon[44005]: Deploying daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:49:36.379 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:36 smithi111 ceph-mon[44005]: pgmap v24: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 362 op/s 2024-03-23T11:49:36.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:36 smithi063 ceph-mon[35189]: Upgrade: Updating mgr.smithi111.nnblsk 2024-03-23T11:49:36.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:36 smithi063 ceph-mon[35189]: Deploying daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:49:36.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:36 smithi063 ceph-mon[35189]: pgmap v24: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 362 op/s 2024-03-23T11:49:38.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:38 smithi063 ceph-mon[35189]: pgmap v25: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 KiB/s rd, 649 KiB/s wr, 225 op/s 2024-03-23T11:49:38.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:38 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:38.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:38 smithi111 ceph-mon[44005]: pgmap v25: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 KiB/s rd, 649 KiB/s wr, 225 op/s 2024-03-23T11:49:38.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:38 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:40.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:39 smithi063 ceph-mon[35189]: pgmap v26: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 KiB/s rd, 649 KiB/s wr, 226 op/s 2024-03-23T11:49:40.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:39 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:40.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:39 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:40.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:39 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:49:40.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:39 smithi111 ceph-mon[44005]: pgmap v26: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 KiB/s rd, 649 KiB/s wr, 226 op/s 2024-03-23T11:49:40.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:39 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:40.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:39 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:40.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:39 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:49:42.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:41 smithi111 ceph-mon[44005]: pgmap v27: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 644 KiB/s rd, 650 KiB/s wr, 239 op/s 2024-03-23T11:49:42.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:41 smithi063 ceph-mon[35189]: pgmap v27: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 644 KiB/s rd, 650 KiB/s wr, 239 op/s 2024-03-23T11:49:43.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:43 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:43.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:43 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:43.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:43 smithi063 ceph-mon[35189]: pgmap v28: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 347 KiB/s rd, 292 KiB/s wr, 111 op/s 2024-03-23T11:49:43.972 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:43 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:43.972 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:43 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:43.972 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:43 smithi111 ceph-mon[44005]: pgmap v28: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 347 KiB/s rd, 292 KiB/s wr, 111 op/s 2024-03-23T11:49:45.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:44 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:45.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:44 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:45.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:44 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:45.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:44 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:46.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:45 smithi063 ceph-mon[35189]: pgmap v29: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 348 KiB/s rd, 292 KiB/s wr, 112 op/s 2024-03-23T11:49:46.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:45 smithi111 ceph-mon[44005]: pgmap v29: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 348 KiB/s rd, 292 KiB/s wr, 112 op/s 2024-03-23T11:49:48.197 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:47 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:48.197 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:47 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:48.198 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:47 smithi111 ceph-mon[44005]: pgmap v30: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.1 KiB/s wr, 15 op/s 2024-03-23T11:49:48.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:47 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:48.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:47 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:48.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:47 smithi063 ceph-mon[35189]: pgmap v30: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.1 KiB/s wr, 15 op/s 2024-03-23T11:49:49.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: Standby manager daemon smithi111.nnblsk restarted 2024-03-23T11:49:49.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:49:49.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:49:49.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:49:49.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:49:49.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:49:49.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi063.gehkcr"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi063.gehkcr"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi063.gehkcr"}]': finished 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.nnblsk"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.nnblsk"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.nnblsk"}]': finished 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.384 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk restarted 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/3832230129' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:49:49.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi063.gehkcr"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi063.gehkcr"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi063.gehkcr"}]': finished 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.nnblsk"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.nnblsk"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.nnblsk"}]': finished 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:49.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all mgr 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all rgw 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all rbd-mirror 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all ceph-exporter 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all iscsi 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all nfs 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all nvmeof 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: mgrmap e32: smithi063.gehkcr(active, since 56s), standbys: smithi111.nnblsk 2024-03-23T11:49:50.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:49 smithi063 ceph-mon[35189]: pgmap v31: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.1 KiB/s wr, 15 op/s 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all mgr 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all rgw 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all rbd-mirror 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all ceph-exporter 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all iscsi 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all nfs 2024-03-23T11:49:50.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all nvmeof 2024-03-23T11:49:50.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: mgrmap e32: smithi063.gehkcr(active, since 56s), standbys: smithi111.nnblsk 2024-03-23T11:49:50.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:49 smithi111 ceph-mon[44005]: pgmap v31: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 1.1 KiB/s wr, 15 op/s 2024-03-23T11:49:51.234 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:50 smithi063 ceph-mon[35189]: Upgrade: Updating node-exporter.smithi063 (1/2) 2024-03-23T11:49:51.235 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:50 smithi063 ceph-mon[35189]: Deploying daemon node-exporter.smithi063 on smithi063 2024-03-23T11:49:51.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:50 smithi111 ceph-mon[44005]: Upgrade: Updating node-exporter.smithi063 (1/2) 2024-03-23T11:49:51.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:50 smithi111 ceph-mon[44005]: Deploying daemon node-exporter.smithi063 on smithi063 2024-03-23T11:49:52.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:51 smithi063 ceph-mon[35189]: pgmap v32: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 853 B/s wr, 15 op/s 2024-03-23T11:49:52.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:51 smithi111 ceph-mon[44005]: pgmap v32: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 853 B/s wr, 15 op/s 2024-03-23T11:49:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 -- 172.21.15.63:0/1638435126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 msgr2=0x7f924c0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 --2- 172.21.15.63:0/1638435126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c0fd310 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9240004940 tx=0x7f9240009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 -- 172.21.15.63:0/1638435126 shutdown_connections 2024-03-23T11:49:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 --2- 172.21.15.63:0/1638435126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c0fe0d0 0x7f924c0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 --2- 172.21.15.63:0/1638435126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c0fd310 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 -- 172.21.15.63:0/1638435126 >> 172.21.15.63:0/1638435126 conn(0x7f924c0f8560 msgr2=0x7f924c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:53.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 -- 172.21.15.63:0/1638435126 shutdown_connections 2024-03-23T11:49:53.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.379+0000 7f9254400700 1 -- 172.21.15.63:0/1638435126 wait complete. 2024-03-23T11:49:53.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.380+0000 7f9254400700 1 Processor -- start 2024-03-23T11:49:53.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9254400700 1 -- start start 2024-03-23T11:49:53.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9254400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:53.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9254400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c0fe0d0 0x7f924c10df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:53.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9254400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c10e510 con 0x7f924c0fcee0 2024-03-23T11:49:53.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9254400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c10ecf0 con 0x7f924c0fe0d0 2024-03-23T11:49:53.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9251a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9251a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c10d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40476/0 (socket says 172.21.15.63:40476) 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9251a00700 1 -- 172.21.15.63:0/2024669123 learned_addr learned my addr 172.21.15.63:0/2024669123 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.381+0000 7f9251000700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c0fe0d0 0x7f924c10df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9251a00700 1 -- 172.21.15.63:0/2024669123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c0fe0d0 msgr2=0x7f924c10df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9251a00700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c0fe0d0 0x7f924c10df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9251a00700 1 -- 172.21.15.63:0/2024669123 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240005040 con 0x7f924c0fcee0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9251a00700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c10d9d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9240009380 tx=0x7f9240009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9249600700 1 -- 172.21.15.63:0/2024669123 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924000e040 con 0x7f924c0fcee0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9249600700 1 -- 172.21.15.63:0/2024669123 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9240004290 con 0x7f924c0fcee0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924c10ef70 con 0x7f924c0fcee0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.382+0000 7f9249600700 1 -- 172.21.15.63:0/2024669123 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924001c3f0 con 0x7f924c0fcee0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.383+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c10f4b0 con 0x7f924c0fcee0 2024-03-23T11:49:53.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.383+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924c05f5f0 con 0x7f924c0fcee0 2024-03-23T11:49:53.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.384+0000 7f9249600700 1 -- 172.21.15.63:0/2024669123 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 32) v1 ==== 90487+0+0 (secure 0 0 0) 0x7f9240013690 con 0x7f924c0fcee0 2024-03-23T11:49:53.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.385+0000 7f9249600700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f923406c560 0x7f923406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:53.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.385+0000 7f9249600700 1 -- 172.21.15.63:0/2024669123 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f924008cae0 con 0x7f924c0fcee0 2024-03-23T11:49:53.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.386+0000 7f9251000700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f923406c560 0x7f923406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:53.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.386+0000 7f9251000700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f923406c560 0x7f923406ea10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9238004f40 tx=0x7f923800b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:53.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.392+0000 7f9249600700 1 -- 172.21.15.63:0/2024669123 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f924005ab80 con 0x7f924c0fcee0 2024-03-23T11:49:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.657+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f924c10f880 con 0x7f923406c560 2024-03-23T11:49:53.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.659+0000 7f9249600700 1 -- 172.21.15.63:0/2024669123 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f924c10f880 con 0x7f923406c560 2024-03-23T11:49:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f923406c560 msgr2=0x7f923406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f923406c560 0x7f923406ea10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9238004f40 tx=0x7f923800b040 comp rx=0 tx=0).stop 2024-03-23T11:49:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 msgr2=0x7f924c10d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c10d9d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9240009380 tx=0x7f9240009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 shutdown_connections 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f923406c560 0x7f923406ea10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f924c0fcee0 0x7f924c10d9d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 --2- 172.21.15.63:0/2024669123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c0fe0d0 0x7f924c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.661+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 >> 172.21.15.63:0/2024669123 conn(0x7f924c0f8560 msgr2=0x7f924c0f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.662+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 shutdown_connections 2024-03-23T11:49:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.662+0000 7f9254400700 1 -- 172.21.15.63:0/2024669123 wait complete. 2024-03-23T11:49:53.661 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 -- 172.21.15.63:0/459838904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 msgr2=0x7fb4400ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 --2- 172.21.15.63:0/459838904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400ff8c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb430004940 tx=0x7fb430009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 -- 172.21.15.63:0/459838904 shutdown_connections 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 --2- 172.21.15.63:0/459838904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4400ffe00 0x7fb4401021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 --2- 172.21.15.63:0/459838904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400ff8c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 -- 172.21.15.63:0/459838904 >> 172.21.15.63:0/459838904 conn(0x7fb4400f7270 msgr2=0x7fb4400f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 -- 172.21.15.63:0/459838904 shutdown_connections 2024-03-23T11:49:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.800+0000 7fb43f200700 1 -- 172.21.15.63:0/459838904 wait complete. 2024-03-23T11:49:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.801+0000 7fb43f200700 1 Processor -- start 2024-03-23T11:49:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43f200700 1 -- start start 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400fd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4400ffe00 0x7fb4400fb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400fbd50 con 0x7fb440066220 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400fbe90 con 0x7fb4400ffe00 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400fd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400fd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40496/0 (socket says 172.21.15.63:40496) 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.802+0000 7fb43de00700 1 -- 172.21.15.63:0/3513833964 learned_addr learned my addr 172.21.15.63:0/3513833964 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb43de00700 1 -- 172.21.15.63:0/3513833964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4400ffe00 msgr2=0x7fb4400fb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb43de00700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4400ffe00 0x7fb4400fb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb43de00700 1 -- 172.21.15.63:0/3513833964 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb430005040 con 0x7fb440066220 2024-03-23T11:49:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb43de00700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400fd2a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb43000b3b0 tx=0x7fb430009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb436000700 1 -- 172.21.15.63:0/3513833964 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb43000e040 con 0x7fb440066220 2024-03-23T11:49:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb436000700 1 -- 172.21.15.63:0/3513833964 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb430004290 con 0x7fb440066220 2024-03-23T11:49:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4400fc110 con 0x7fb440066220 2024-03-23T11:49:53.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb436000700 1 -- 172.21.15.63:0/3513833964 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb43001c3f0 con 0x7fb440066220 2024-03-23T11:49:53.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.803+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb440115890 con 0x7fb440066220 2024-03-23T11:49:53.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.804+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44005f5f0 con 0x7fb440066220 2024-03-23T11:49:53.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.805+0000 7fb436000700 1 -- 172.21.15.63:0/3513833964 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 32) v1 ==== 90487+0+0 (secure 0 0 0) 0x7fb43000bc50 con 0x7fb440066220 2024-03-23T11:49:53.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.806+0000 7fb436000700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb42406c560 0x7fb42406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:53.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.806+0000 7fb436000700 1 -- 172.21.15.63:0/3513833964 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb43008cc00 con 0x7fb440066220 2024-03-23T11:49:53.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.807+0000 7fb43d400700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb42406c560 0x7fb42406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:53.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.807+0000 7fb43d400700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb42406c560 0x7fb42406ea10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4400fd880 tx=0x7fb42c009ce0 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:53.811+0000 7fb436000700 1 -- 172.21.15.63:0/3513833964 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb4300570e0 con 0x7fb440066220 2024-03-23T11:49:54.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.042+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4400f8db0 con 0x7fb42406c560 2024-03-23T11:49:54.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.043+0000 7fb436000700 1 -- 172.21.15.63:0/3513833964 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fb4400f8db0 con 0x7fb42406c560 2024-03-23T11:49:54.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.045+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb42406c560 msgr2=0x7fb42406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.045+0000 7fb43f200700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb42406c560 0x7fb42406ea10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4400fd880 tx=0x7fb42c009ce0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 msgr2=0x7fb4400fd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400fd2a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb43000b3b0 tx=0x7fb430009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 shutdown_connections 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb42406c560 0x7fb42406ea10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb440066220 0x7fb4400fd2a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 --2- 172.21.15.63:0/3513833964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4400ffe00 0x7fb4400fb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 >> 172.21.15.63:0/3513833964 conn(0x7fb4400f7270 msgr2=0x7fb4400f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 shutdown_connections 2024-03-23T11:49:54.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.046+0000 7fb43f200700 1 -- 172.21.15.63:0/3513833964 wait complete. 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 -- 172.21.15.63:0/3677323746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 msgr2=0x7fa138102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3677323746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138102380 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa124004940 tx=0x7fa124009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 -- 172.21.15.63:0/3677323746 shutdown_connections 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3677323746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138102380 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3677323746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa138066240 0x7fa1380ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 -- 172.21.15.63:0/3677323746 >> 172.21.15.63:0/3677323746 conn(0x7fa1380f7430 msgr2=0x7fa1380f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 -- 172.21.15.63:0/3677323746 shutdown_connections 2024-03-23T11:49:54.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.186+0000 7fa13ec00700 1 -- 172.21.15.63:0/3677323746 wait complete. 2024-03-23T11:49:54.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.187+0000 7fa13ec00700 1 Processor -- start 2024-03-23T11:49:54.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.187+0000 7fa13ec00700 1 -- start start 2024-03-23T11:49:54.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa13ec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa138066240 0x7fa13810fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:54.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa13ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:54.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa13ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa138110680 con 0x7fa1380fffa0 2024-03-23T11:49:54.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa13ec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa138110ed0 con 0x7fa138066240 2024-03-23T11:49:54.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa137400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:54.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa137400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40504/0 (socket says 172.21.15.63:40504) 2024-03-23T11:49:54.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa137400700 1 -- 172.21.15.63:0/3436227223 learned_addr learned my addr 172.21.15.63:0/3436227223 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:54.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa137e00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa138066240 0x7fa13810fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:54.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa137400700 1 -- 172.21.15.63:0/3436227223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa138066240 msgr2=0x7fa13810fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa137400700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa138066240 0x7fa13810fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.188+0000 7fa137400700 1 -- 172.21.15.63:0/3436227223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa124005040 con 0x7fa1380fffa0 2024-03-23T11:49:54.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.189+0000 7fa137400700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138110080 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa124009fd0 tx=0x7fa124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:54.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.189+0000 7fa12fe00700 1 -- 172.21.15.63:0/3436227223 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa12400e040 con 0x7fa1380fffa0 2024-03-23T11:49:54.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.189+0000 7fa12fe00700 1 -- 172.21.15.63:0/3436227223 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa124013420 con 0x7fa1380fffa0 2024-03-23T11:49:54.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.189+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa138111150 con 0x7fa1380fffa0 2024-03-23T11:49:54.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.189+0000 7fa12fe00700 1 -- 172.21.15.63:0/3436227223 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa12401c630 con 0x7fa1380fffa0 2024-03-23T11:49:54.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.189+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa138111690 con 0x7fa1380fffa0 2024-03-23T11:49:54.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.190+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1380f8fb0 con 0x7fa1380fffa0 2024-03-23T11:49:54.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.191+0000 7fa12fe00700 1 -- 172.21.15.63:0/3436227223 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 32) v1 ==== 90487+0+0 (secure 0 0 0) 0x7fa1240139e0 con 0x7fa1380fffa0 2024-03-23T11:49:54.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.192+0000 7fa12fe00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fa11c06c560 0x7fa11c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:54.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.192+0000 7fa12fe00700 1 -- 172.21.15.63:0/3436227223 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa12408ca60 con 0x7fa1380fffa0 2024-03-23T11:49:54.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.192+0000 7fa137e00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fa11c06c560 0x7fa11c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:54.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.193+0000 7fa137e00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fa11c06c560 0x7fa11c06ea10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa138065f80 tx=0x7fa128009040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.197+0000 7fa12fe00700 1 -- 172.21.15.63:0/3436227223 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa12405aa50 con 0x7fa1380fffa0 2024-03-23T11:49:54.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:54 smithi063 ceph-mon[35189]: pgmap v33: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-23T11:49:54.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:54.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:54 smithi111 ceph-mon[44005]: pgmap v33: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-23T11:49:54.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:49:54.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.423+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1381058e0 con 0x7fa11c06c560 2024-03-23T11:49:54.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.433+0000 7fa12fe00700 1 -- 172.21.15.63:0/3436227223 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa1381058e0 con 0x7fa11c06c560 2024-03-23T11:49:54.418 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:49:54.418 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (2m) 7s ago 12m 14.8M - 0.25.0 c8568f914cd2 106bde4987fa 2024-03-23T11:49:54.418 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (12m) 7s ago 12m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:49:54.418 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (11m) 11s ago 11m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (2m) 7s ago 12m 77.4M - 9.4.7 954c08fa6188 d4ed1ee72841 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (6m) 7s ago 6m 119M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (5m) 7s ago 5m 119M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (5m) 11s ago 5m 65.1M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (5m) 11s ago 5m 59.6M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (94s) 7s ago 13m 493M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (16s) 11s ago 11m 55.2M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (13m) 7s ago 13m 53.5M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (11m) 11s ago 11m 43.5M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (3m) 7s ago 12m 7222k - 1.5.0 0da6a335fe13 9387979cc7ad 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (102s) 11s ago 11m 7578k - 1.5.0 0da6a335fe13 fd96d8076254 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (10m) 7s ago 10m 218M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:49:54.419 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (9m) 7s ago 9m 231M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:49:54.420 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (9m) 7s ago 9m 194M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:49:54.420 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (8m) 11s ago 8m 256M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:49:54.420 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (8m) 11s ago 8m 229M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:49:54.420 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (7m) 11s ago 7m 225M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:49:54.420 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (21s) 7s ago 11m 38.6M - 2.43.0 a07b618ecd1d 5553c5535d78 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fa11c06c560 msgr2=0x7fa11c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fa11c06c560 0x7fa11c06ea10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa138065f80 tx=0x7fa128009040 comp rx=0 tx=0).stop 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 msgr2=0x7fa138110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138110080 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa124009fd0 tx=0x7fa124009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 shutdown_connections 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fa11c06c560 0x7fa11c06ea10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa138066240 0x7fa13810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 --2- 172.21.15.63:0/3436227223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1380fffa0 0x7fa138110080 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 >> 172.21.15.63:0/3436227223 conn(0x7fa1380f7430 msgr2=0x7fa1380fbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:54.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 shutdown_connections 2024-03-23T11:49:54.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.436+0000 7fa13ec00700 1 -- 172.21.15.63:0/3436227223 wait complete. 2024-03-23T11:49:54.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.572+0000 7f3a96a00700 1 -- 172.21.15.63:0/3762860846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a900709a0 msgr2=0x7f3a90070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.572+0000 7f3a96a00700 1 --2- 172.21.15.63:0/3762860846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a900709a0 0x7f3a90070e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c004940 tx=0x7f3a7c009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:54.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.572+0000 7f3a96a00700 1 -- 172.21.15.63:0/3762860846 shutdown_connections 2024-03-23T11:49:54.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.572+0000 7f3a96a00700 1 --2- 172.21.15.63:0/3762860846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a900709a0 0x7f3a90070e10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.572+0000 7f3a96a00700 1 --2- 172.21.15.63:0/3762860846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a90071f50 0x7f3a90070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.572+0000 7f3a96a00700 1 -- 172.21.15.63:0/3762860846 >> 172.21.15.63:0/3762860846 conn(0x7f3a900f8560 msgr2=0x7f3a900fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:54.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.573+0000 7f3a96a00700 1 -- 172.21.15.63:0/3762860846 shutdown_connections 2024-03-23T11:49:54.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.573+0000 7f3a96a00700 1 -- 172.21.15.63:0/3762860846 wait complete. 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.573+0000 7f3a96a00700 1 Processor -- start 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.573+0000 7f3a96a00700 1 -- start start 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a96a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a900709a0 0x7f3a9010faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a96a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a90071f50 0x7f3a90110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a96a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90110630 con 0x7f3a90071f50 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a96a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90110e80 con 0x7f3a900709a0 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a8f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a90071f50 0x7f3a90110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a8f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a90071f50 0x7f3a90110030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40532/0 (socket says 172.21.15.63:40532) 2024-03-23T11:49:54.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a8f400700 1 -- 172.21.15.63:0/4068373462 learned_addr learned my addr 172.21.15.63:0/4068373462 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.574+0000 7f3a8fe00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a900709a0 0x7f3a9010faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a8f400700 1 -- 172.21.15.63:0/4068373462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a900709a0 msgr2=0x7f3a9010faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a8f400700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a900709a0 0x7f3a9010faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a8f400700 1 -- 172.21.15.63:0/4068373462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c005040 con 0x7f3a90071f50 2024-03-23T11:49:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a8fe00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a900709a0 0x7f3a9010faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:49:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a8f400700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a90071f50 0x7f3a90110030 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00b0d0 tx=0x7f3a7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a87e00700 1 -- 172.21.15.63:0/4068373462 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a7c00e070 con 0x7f3a90071f50 2024-03-23T11:49:54.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a87e00700 1 -- 172.21.15.63:0/4068373462 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a7c00b410 con 0x7f3a90071f50 2024-03-23T11:49:54.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a87e00700 1 -- 172.21.15.63:0/4068373462 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a7c012600 con 0x7f3a90071f50 2024-03-23T11:49:54.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a90111100 con 0x7f3a90071f50 2024-03-23T11:49:54.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.575+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a90111640 con 0x7f3a90071f50 2024-03-23T11:49:54.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.576+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9005f5f0 con 0x7f3a90071f50 2024-03-23T11:49:54.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.578+0000 7f3a87e00700 1 -- 172.21.15.63:0/4068373462 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 32) v1 ==== 90487+0+0 (secure 0 0 0) 0x7f3a7c00b9f0 con 0x7f3a90071f50 2024-03-23T11:49:54.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.579+0000 7f3a87e00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f3a7406c560 0x7f3a7406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:54.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.579+0000 7f3a87e00700 1 -- 172.21.15.63:0/4068373462 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f3a7c08c3f0 con 0x7f3a90071f50 2024-03-23T11:49:54.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.579+0000 7f3a8fe00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f3a7406c560 0x7f3a7406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.580+0000 7f3a8fe00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f3a7406c560 0x7f3a7406ea10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a80004600 tx=0x7f3a8000b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:54.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.583+0000 7f3a87e00700 1 -- 172.21.15.63:0/4068373462 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3a7c05a490 con 0x7f3a90071f50 2024-03-23T11:49:54.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.884+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3a90107a60 con 0x7f3a90071f50 2024-03-23T11:49:54.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.885+0000 7f3a87e00700 1 -- 172.21.15.63:0/4068373462 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f3a7c022300 con 0x7f3a90071f50 2024-03-23T11:49:54.869 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:49:54.869 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:49:54.869 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:49:54.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:49:54.871 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:49:54.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f3a7406c560 msgr2=0x7f3a7406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f3a7406c560 0x7f3a7406ea10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a80004600 tx=0x7f3a8000b040 comp rx=0 tx=0).stop 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a90071f50 msgr2=0x7f3a90110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a90071f50 0x7f3a90110030 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00b0d0 tx=0x7f3a7c009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 shutdown_connections 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f3a7406c560 0x7f3a7406ea10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a900709a0 0x7f3a9010faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 --2- 172.21.15.63:0/4068373462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a90071f50 0x7f3a90110030 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 >> 172.21.15.63:0/4068373462 conn(0x7f3a900f8560 msgr2=0x7f3a900f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 shutdown_connections 2024-03-23T11:49:54.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:54.887+0000 7f3a96a00700 1 -- 172.21.15.63:0/4068373462 wait complete. 2024-03-23T11:49:55.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.025+0000 7f69e2a00700 1 -- 172.21.15.63:0/1576926982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fcc20 msgr2=0x7f69dc0fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:55.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.025+0000 7f69e2a00700 1 --2- 172.21.15.63:0/1576926982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fcc20 0x7f69dc0fd030 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f69d0004940 tx=0x7f69d0009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.025+0000 7f69e2a00700 1 -- 172.21.15.63:0/1576926982 shutdown_connections 2024-03-23T11:49:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.025+0000 7f69e2a00700 1 --2- 172.21.15.63:0/1576926982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69dc0fded0 0x7f69dc0fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.025+0000 7f69e2a00700 1 --2- 172.21.15.63:0/1576926982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fcc20 0x7f69dc0fd030 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.026+0000 7f69e2a00700 1 -- 172.21.15.63:0/1576926982 >> 172.21.15.63:0/1576926982 conn(0x7f69dc0f83a0 msgr2=0x7f69dc0fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.026+0000 7f69e2a00700 1 -- 172.21.15.63:0/1576926982 shutdown_connections 2024-03-23T11:49:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.026+0000 7f69e2a00700 1 -- 172.21.15.63:0/1576926982 wait complete. 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.026+0000 7f69e2a00700 1 Processor -- start 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e2a00700 1 -- start start 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e2a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69dc0fcc20 0x7f69dc10f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fded0 0x7f69dc10fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e2a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc1104a0 con 0x7f69dc0fded0 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e2a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc110cf0 con 0x7f69dc0fcc20 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fded0 0x7f69dc10fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:55.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fded0 0x7f69dc10fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40538/0 (socket says 172.21.15.63:40538) 2024-03-23T11:49:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e0c00700 1 -- 172.21.15.63:0/3867490748 learned_addr learned my addr 172.21.15.63:0/3867490748 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:49:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.027+0000 7f69e1600700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69dc0fcc20 0x7f69dc10f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69e0c00700 1 -- 172.21.15.63:0/3867490748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69dc0fcc20 msgr2=0x7f69dc10f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69e0c00700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69dc0fcc20 0x7f69dc10f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69e0c00700 1 -- 172.21.15.63:0/3867490748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0005040 con 0x7f69dc0fded0 2024-03-23T11:49:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69e0c00700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fded0 0x7f69dc10fea0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f69c8002a50 tx=0x7f69c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69d9600700 1 -- 172.21.15.63:0/3867490748 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c8007cc0 con 0x7f69dc0fded0 2024-03-23T11:49:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69d9600700 1 -- 172.21.15.63:0/3867490748 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69c800f040 con 0x7f69dc0fded0 2024-03-23T11:49:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc110fd0 con 0x7f69dc0fded0 2024-03-23T11:49:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69d9600700 1 -- 172.21.15.63:0/3867490748 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c800d9c0 con 0x7f69dc0fded0 2024-03-23T11:49:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.028+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc111510 con 0x7f69dc0fded0 2024-03-23T11:49:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.030+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc05f5f0 con 0x7f69dc0fded0 2024-03-23T11:49:55.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.031+0000 7f69d9600700 1 -- 172.21.15.63:0/3867490748 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 32) v1 ==== 90487+0+0 (secure 0 0 0) 0x7f69c800db20 con 0x7f69dc0fded0 2024-03-23T11:49:55.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.032+0000 7f69d9600700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f69c406c560 0x7f69c406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:49:55.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.032+0000 7f69d9600700 1 -- 172.21.15.63:0/3867490748 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f69c808b3c0 con 0x7f69dc0fded0 2024-03-23T11:49:55.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.032+0000 7f69e1600700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f69c406c560 0x7f69c406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:49:55.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.032+0000 7f69e1600700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f69c406c560 0x7f69c406ea10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69d0004f90 tx=0x7f69d0009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:49:55.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.036+0000 7f69d9600700 1 -- 172.21.15.63:0/3867490748 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f69c80593d0 con 0x7f69dc0fded0 2024-03-23T11:49:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.297+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f69dc111890 con 0x7f69c406c560 2024-03-23T11:49:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.298+0000 7f69d9600700 1 -- 172.21.15.63:0/3867490748 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f69dc111890 con 0x7f69c406c560 2024-03-23T11:49:55.283 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:49:55.283 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "2/2 daemons upgraded", 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:49:55.284 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f69c406c560 msgr2=0x7f69c406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f69c406c560 0x7f69c406ea10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69d0004f90 tx=0x7f69d0009e30 comp rx=0 tx=0).stop 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fded0 msgr2=0x7f69dc10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fded0 0x7f69dc10fea0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f69c8002a50 tx=0x7f69c80046b0 comp rx=0 tx=0).stop 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 shutdown_connections 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f69c406c560 0x7f69c406ea10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69dc0fcc20 0x7f69dc10f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 --2- 172.21.15.63:0/3867490748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69dc0fded0 0x7f69dc10fea0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 >> 172.21.15.63:0/3867490748 conn(0x7f69dc0f83a0 msgr2=0x7f69dc0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 shutdown_connections 2024-03-23T11:49:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:49:55.301+0000 7f69e2a00700 1 -- 172.21.15.63:0/3867490748 wait complete. 2024-03-23T11:49:55.369 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:55 smithi063 ceph-mon[35189]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:55.370 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:55 smithi063 ceph-mon[35189]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:55.370 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:55 smithi063 ceph-mon[35189]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:55.370 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:55 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/4068373462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:55 smithi111 ceph-mon[44005]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:55 smithi111 ceph-mon[44005]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:55 smithi111 ceph-mon[44005]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:55 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/4068373462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:49:56.848 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:56 smithi111 ceph-mon[44005]: pgmap v34: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-23T11:49:56.849 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:56 smithi111 ceph-mon[44005]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:56.849 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:56 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:56.849 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:56 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:56.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:56 smithi063 ceph-mon[35189]: pgmap v34: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-23T11:49:56.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:56 smithi063 ceph-mon[35189]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:49:56.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:56 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:56.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:56 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:49:58.828 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:58 smithi111 ceph-mon[44005]: pgmap v35: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:49:58.828 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:58 smithi111 ceph-mon[44005]: Upgrade: Updating node-exporter.smithi111 (2/2) 2024-03-23T11:49:58.828 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:49:58 smithi111 ceph-mon[44005]: Deploying daemon node-exporter.smithi111 on smithi111 2024-03-23T11:49:58.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:58 smithi063 ceph-mon[35189]: pgmap v35: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:49:58.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:58 smithi063 ceph-mon[35189]: Upgrade: Updating node-exporter.smithi111 (2/2) 2024-03-23T11:49:58.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:49:58 smithi063 ceph-mon[35189]: Deploying daemon node-exporter.smithi111 on smithi111 2024-03-23T11:50:00.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:00 smithi063 ceph-mon[35189]: pgmap v36: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:00.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:00 smithi063 ceph-mon[35189]: overall HEALTH_OK 2024-03-23T11:50:00.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:00 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:00.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:00 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:00.856 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:00 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:00.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:00 smithi111 ceph-mon[44005]: pgmap v36: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:00.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:00 smithi111 ceph-mon[44005]: overall HEALTH_OK 2024-03-23T11:50:00.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:00 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:00.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:00 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:00.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:00 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:02.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:02 smithi063 ceph-mon[35189]: pgmap v37: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:02.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:02 smithi111 ceph-mon[44005]: pgmap v37: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:04.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:04 smithi063 ceph-mon[35189]: pgmap v38: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:04.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:04 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:04.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:04 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:04.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:04 smithi111 ceph-mon[44005]: pgmap v38: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:04.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:04 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:04.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:04 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:06.556 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:06 smithi111 ceph-mon[44005]: pgmap v39: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:06.556 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:06 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:06.556 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:06 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:06.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:06 smithi063 ceph-mon[35189]: pgmap v39: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:06.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:06 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:06.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:06 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:07.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:07 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:07.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:07 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:07.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:07 smithi111 ceph-mon[44005]: pgmap v40: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:08.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:07 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:08.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:07 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:08.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:07 smithi063 ceph-mon[35189]: pgmap v40: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:08.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:08 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:08.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:08 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:08.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:08 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:09.105 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:08 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:09.105 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:08 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:09.105 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:08 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:10.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:10 smithi063 ceph-mon[35189]: pgmap v41: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:10.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:10 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:10.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:10 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:10.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:10 smithi111 ceph-mon[44005]: pgmap v41: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:10.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:10 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:10.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:10 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:12.760 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:12 smithi063 ceph-mon[35189]: pgmap v42: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:12.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:12 smithi111 ceph-mon[44005]: pgmap v42: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:13.817 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:13.817 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:13.817 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.818 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:13 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:13.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:13.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:50:13.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:50:13.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:13.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:13.904 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:13 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:14.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:14 smithi063 ceph-mon[35189]: pgmap v43: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:14.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:14 smithi111 ceph-mon[44005]: pgmap v43: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:16.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:16 smithi063 ceph-mon[35189]: Upgrade: Updating prometheus.smithi063 2024-03-23T11:50:16.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:16 smithi063 ceph-mon[35189]: Deploying daemon prometheus.smithi063 on smithi063 2024-03-23T11:50:16.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:16 smithi063 ceph-mon[35189]: pgmap v44: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:16.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:16 smithi111 ceph-mon[44005]: Upgrade: Updating prometheus.smithi063 2024-03-23T11:50:16.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:16 smithi111 ceph-mon[44005]: Deploying daemon prometheus.smithi063 on smithi063 2024-03-23T11:50:16.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:16 smithi111 ceph-mon[44005]: pgmap v44: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:18.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:18 smithi063 ceph-mon[35189]: pgmap v45: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:18.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:18 smithi111 ceph-mon[44005]: pgmap v45: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:20.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:20 smithi063 ceph-mon[35189]: pgmap v46: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:20.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:20 smithi111 ceph-mon[44005]: pgmap v46: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:22.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:22 smithi063 ceph-mon[35189]: pgmap v47: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:22.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:22 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:22.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:22 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:22.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:22 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:22.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:22 smithi111 ceph-mon[44005]: pgmap v47: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:22.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:22 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:22.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:22 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:22.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:22 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:23.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:23 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:23.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:23 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:24.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:24 smithi063 ceph-mon[35189]: pgmap v48: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:24.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:24 smithi111 ceph-mon[44005]: pgmap v48: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 -- 172.21.15.63:0/1082474061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fcee0 msgr2=0x7fe2100fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 --2- 172.21.15.63:0/1082474061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fcee0 0x7fe2100fd310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe204004940 tx=0x7fe204009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 -- 172.21.15.63:0/1082474061 shutdown_connections 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 --2- 172.21.15.63:0/1082474061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2100fe0d0 0x7fe2100fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 --2- 172.21.15.63:0/1082474061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fcee0 0x7fe2100fd310 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 -- 172.21.15.63:0/1082474061 >> 172.21.15.63:0/1082474061 conn(0x7fe2100f8560 msgr2=0x7fe2100fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 -- 172.21.15.63:0/1082474061 shutdown_connections 2024-03-23T11:50:25.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.449+0000 7fe217800700 1 -- 172.21.15.63:0/1082474061 wait complete. 2024-03-23T11:50:25.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.450+0000 7fe217800700 1 Processor -- start 2024-03-23T11:50:25.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.450+0000 7fe217800700 1 -- start start 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.450+0000 7fe217800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2100fcee0 0x7fe21010fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.450+0000 7fe217800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fe0d0 0x7fe2101100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe217800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2101106a0 con 0x7fe2100fe0d0 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe217800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe210110ef0 con 0x7fe2100fcee0 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe20fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fe0d0 0x7fe2101100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe20fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fe0d0 0x7fe2101100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35242/0 (socket says 172.21.15.63:35242) 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe20fe00700 1 -- 172.21.15.63:0/4260108214 learned_addr learned my addr 172.21.15.63:0/4260108214 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe214e00700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2100fcee0 0x7fe21010fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe20fe00700 1 -- 172.21.15.63:0/4260108214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2100fcee0 msgr2=0x7fe21010fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe20fe00700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2100fcee0 0x7fe21010fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe20fe00700 1 -- 172.21.15.63:0/4260108214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204005040 con 0x7fe2100fe0d0 2024-03-23T11:50:25.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.451+0000 7fe20fe00700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fe0d0 0x7fe2101100a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc002a50 tx=0x7fe1fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:25.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.452+0000 7fe20cc00700 1 -- 172.21.15.63:0/4260108214 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1fc007cc0 con 0x7fe2100fe0d0 2024-03-23T11:50:25.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.452+0000 7fe20cc00700 1 -- 172.21.15.63:0/4260108214 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1fc00f040 con 0x7fe2100fe0d0 2024-03-23T11:50:25.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.452+0000 7fe20cc00700 1 -- 172.21.15.63:0/4260108214 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1fc009540 con 0x7fe2100fe0d0 2024-03-23T11:50:25.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.452+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2101111d0 con 0x7fe2100fe0d0 2024-03-23T11:50:25.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.452+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe210111710 con 0x7fe2100fe0d0 2024-03-23T11:50:25.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.453+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21005f5f0 con 0x7fe2100fe0d0 2024-03-23T11:50:25.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.454+0000 7fe20cc00700 1 -- 172.21.15.63:0/4260108214 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fe1fc0096a0 con 0x7fe2100fe0d0 2024-03-23T11:50:25.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.455+0000 7fe20cc00700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe1f806c450 0x7fe1f806e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:25.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.455+0000 7fe20cc00700 1 -- 172.21.15.63:0/4260108214 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe1fc08b260 con 0x7fe2100fe0d0 2024-03-23T11:50:25.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.455+0000 7fe214e00700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe1f806c450 0x7fe1f806e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:25.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.456+0000 7fe214e00700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe1f806c450 0x7fe1f806e900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe20400b3b0 tx=0x7fe204009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:25.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.460+0000 7fe20cc00700 1 -- 172.21.15.63:0/4260108214 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fe1fc059430 con 0x7fe2100fe0d0 2024-03-23T11:50:25.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:25 smithi063 ceph-mon[35189]: mgrmap e33: smithi063.gehkcr(active, since 91s), standbys: smithi111.nnblsk 2024-03-23T11:50:25.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:25 smithi111 ceph-mon[44005]: mgrmap e33: smithi063.gehkcr(active, since 91s), standbys: smithi111.nnblsk 2024-03-23T11:50:25.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.699+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe210111a90 con 0x7fe1f806c450 2024-03-23T11:50:25.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.700+0000 7fe20cc00700 1 -- 172.21.15.63:0/4260108214 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fe210111a90 con 0x7fe1f806c450 2024-03-23T11:50:25.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe1f806c450 msgr2=0x7fe1f806e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:25.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe1f806c450 0x7fe1f806e900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe20400b3b0 tx=0x7fe204009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:25.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fe0d0 msgr2=0x7fe2101100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:25.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fe0d0 0x7fe2101100a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc002a50 tx=0x7fe1fc0046b0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 shutdown_connections 2024-03-23T11:50:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe1f806c450 0x7fe1f806e900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2100fcee0 0x7fe21010fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 --2- 172.21.15.63:0/4260108214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2100fe0d0 0x7fe2101100a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.702+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 >> 172.21.15.63:0/4260108214 conn(0x7fe2100f8560 msgr2=0x7fe2100f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.703+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 shutdown_connections 2024-03-23T11:50:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.703+0000 7fe217800700 1 -- 172.21.15.63:0/4260108214 wait complete. 2024-03-23T11:50:25.702 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:50:25.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 -- 172.21.15.63:0/2870290984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fce40 msgr2=0x7feee00fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:25.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 --2- 172.21.15.63:0/2870290984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fce40 0x7feee00fd250 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feed0004940 tx=0x7feed0009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:25.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 -- 172.21.15.63:0/2870290984 shutdown_connections 2024-03-23T11:50:25.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 --2- 172.21.15.63:0/2870290984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee00fe040 0x7feee00fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 --2- 172.21.15.63:0/2870290984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fce40 0x7feee00fd250 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 -- 172.21.15.63:0/2870290984 >> 172.21.15.63:0/2870290984 conn(0x7feee00f8550 msgr2=0x7feee00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:25.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 -- 172.21.15.63:0/2870290984 shutdown_connections 2024-03-23T11:50:25.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.848+0000 7feee5200700 1 -- 172.21.15.63:0/2870290984 wait complete. 2024-03-23T11:50:25.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.849+0000 7feee5200700 1 Processor -- start 2024-03-23T11:50:25.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.849+0000 7feee5200700 1 -- start start 2024-03-23T11:50:25.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feee5200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee00fce40 0x7feee010fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:25.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feee5200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fe040 0x7feee0110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feee5200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0110610 con 0x7feee00fe040 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feee5200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0110e60 con 0x7feee00fce40 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fe040 0x7feee0110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fe040 0x7feee0110010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35260/0 (socket says 172.21.15.63:35260) 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedd400700 1 -- 172.21.15.63:0/3059603991 learned_addr learned my addr 172.21.15.63:0/3059603991 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedde00700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee00fce40 0x7feee010fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedd400700 1 -- 172.21.15.63:0/3059603991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee00fce40 msgr2=0x7feee010fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedd400700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee00fce40 0x7feee010fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedd400700 1 -- 172.21.15.63:0/3059603991 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0005040 con 0x7feee00fe040 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.850+0000 7feedd400700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fe040 0x7feee0110010 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feecc002a80 tx=0x7feecc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.851+0000 7feed6000700 1 -- 172.21.15.63:0/3059603991 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feecc007cc0 con 0x7feee00fe040 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.851+0000 7feed6000700 1 -- 172.21.15.63:0/3059603991 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feecc00f040 con 0x7feee00fe040 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.851+0000 7feed6000700 1 -- 172.21.15.63:0/3059603991 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feecc00da60 con 0x7feee00fe040 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.851+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee0111140 con 0x7feee00fe040 2024-03-23T11:50:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.851+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee0111680 con 0x7feee00fe040 2024-03-23T11:50:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.852+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee01019d0 con 0x7feee00fe040 2024-03-23T11:50:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.853+0000 7feed6000700 1 -- 172.21.15.63:0/3059603991 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7feecc00dbc0 con 0x7feee00fe040 2024-03-23T11:50:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.854+0000 7feed6000700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7feec406c450 0x7feec406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.854+0000 7feed6000700 1 -- 172.21.15.63:0/3059603991 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7feecc08b130 con 0x7feee00fe040 2024-03-23T11:50:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.854+0000 7feedde00700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7feec406c450 0x7feec406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.855+0000 7feedde00700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7feec406c450 0x7feec406e900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feed000b3b0 tx=0x7feed0009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:25.859+0000 7feed6000700 1 -- 172.21.15.63:0/3059603991 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7feecc0592c0 con 0x7feee00fe040 2024-03-23T11:50:26.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.090+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feee01118f0 con 0x7feec406c450 2024-03-23T11:50:26.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.092+0000 7feed6000700 1 -- 172.21.15.63:0/3059603991 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7feee01118f0 con 0x7feec406c450 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7feec406c450 msgr2=0x7feec406e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7feec406c450 0x7feec406e900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feed000b3b0 tx=0x7feed0009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fe040 msgr2=0x7feee0110010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fe040 0x7feee0110010 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feecc002a80 tx=0x7feecc0046b0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 shutdown_connections 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7feec406c450 0x7feec406e900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee00fce40 0x7feee010fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 --2- 172.21.15.63:0/3059603991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feee00fe040 0x7feee0110010 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.094+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 >> 172.21.15.63:0/3059603991 conn(0x7feee00f8550 msgr2=0x7feee00f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:26.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.095+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 shutdown_connections 2024-03-23T11:50:26.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.095+0000 7feee5200700 1 -- 172.21.15.63:0/3059603991 wait complete. 2024-03-23T11:50:26.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 -- 172.21.15.63:0/3422599182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 msgr2=0x7f97b0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 --2- 172.21.15.63:0/3422599182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b0070460 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f97a0004940 tx=0x7f97a0009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:26.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 -- 172.21.15.63:0/3422599182 shutdown_connections 2024-03-23T11:50:26.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 --2- 172.21.15.63:0/3422599182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b00709a0 0x7f97b0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 --2- 172.21.15.63:0/3422599182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b0070460 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 -- 172.21.15.63:0/3422599182 >> 172.21.15.63:0/3422599182 conn(0x7f97b00f8560 msgr2=0x7f97b00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:26.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 -- 172.21.15.63:0/3422599182 shutdown_connections 2024-03-23T11:50:26.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.253+0000 7f97b4c00700 1 -- 172.21.15.63:0/3422599182 wait complete. 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.254+0000 7f97b4c00700 1 Processor -- start 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.254+0000 7f97b4c00700 1 -- start start 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.254+0000 7f97b4c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b00709a0 0x7f97b0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.254+0000 7f97b4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b01144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.254+0000 7f97b4c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b0114aa0 con 0x7f97b0071f50 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.254+0000 7f97b4c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b01152f0 con 0x7f97b00709a0 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ad400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b01144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:26.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ad400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b01144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35280/0 (socket says 172.21.15.63:35280) 2024-03-23T11:50:26.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ad400700 1 -- 172.21.15.63:0/1320111015 learned_addr learned my addr 172.21.15.63:0/1320111015 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:26.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ade00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b00709a0 0x7f97b0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:26.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ad400700 1 -- 172.21.15.63:0/1320111015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b00709a0 msgr2=0x7f97b0113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ad400700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b00709a0 0x7f97b0113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ad400700 1 -- 172.21.15.63:0/1320111015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a0005040 con 0x7f97b0071f50 2024-03-23T11:50:26.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.255+0000 7f97ad400700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b01144a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f979c002a50 tx=0x7f979c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:26.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.256+0000 7f97a6000700 1 -- 172.21.15.63:0/1320111015 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f979c007cc0 con 0x7f97b0071f50 2024-03-23T11:50:26.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.256+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b01155d0 con 0x7f97b0071f50 2024-03-23T11:50:26.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.256+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b0115b10 con 0x7f97b0071f50 2024-03-23T11:50:26.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.256+0000 7f97a6000700 1 -- 172.21.15.63:0/1320111015 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f979c00f040 con 0x7f97b0071f50 2024-03-23T11:50:26.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.256+0000 7f97a6000700 1 -- 172.21.15.63:0/1320111015 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f979c00d9c0 con 0x7f97b0071f50 2024-03-23T11:50:26.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.257+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97900052e0 con 0x7f97b0071f50 2024-03-23T11:50:26.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.258+0000 7f97a6000700 1 -- 172.21.15.63:0/1320111015 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f979c00db40 con 0x7f97b0071f50 2024-03-23T11:50:26.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.259+0000 7f97a6000700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f979406c450 0x7f979406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:26.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.259+0000 7f97a6000700 1 -- 172.21.15.63:0/1320111015 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f979c08b140 con 0x7f97b0071f50 2024-03-23T11:50:26.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.260+0000 7f97ade00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f979406c450 0x7f979406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:26.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.260+0000 7f97ade00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f979406c450 0x7f979406e900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f97a000b3b0 tx=0x7f97a0009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:26.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.265+0000 7f97a6000700 1 -- 172.21.15.63:0/1320111015 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f979c0592d0 con 0x7f97b0071f50 2024-03-23T11:50:26.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.493+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9790000bf0 con 0x7f979406c450 2024-03-23T11:50:26.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.503+0000 7f97a6000700 1 -- 172.21.15.63:0/1320111015 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f9790000bf0 con 0x7f979406c450 2024-03-23T11:50:26.487 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (3m) 19s ago 12m 14.9M - 0.25.0 c8568f914cd2 106bde4987fa 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (13m) 19s ago 13m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (11m) 22s ago 11m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (2m) 19s ago 12m 73.4M - 9.4.7 954c08fa6188 d4ed1ee72841 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (6m) 19s ago 6m 119M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (6m) 19s ago 6m 119M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (6m) 22s ago 6m 65.0M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (6m) 22s ago 6m 59.8M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (2m) 19s ago 14m 494M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (48s) 22s ago 11m 446M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (14m) 19s ago 14m 55.0M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (11m) 22s ago 11m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (33s) 19s ago 13m 2805k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (26s) 22s ago 11m 2558k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (10m) 19s ago 10m 219M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:50:26.488 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (10m) 19s ago 10m 231M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:50:26.489 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (9m) 19s ago 9m 194M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:50:26.489 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (9m) 22s ago 9m 256M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:50:26.489 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (8m) 22s ago 8m 230M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:50:26.489 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (8m) 22s ago 8m 225M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:50:26.489 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 starting - - - - 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f979406c450 msgr2=0x7f979406e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f979406c450 0x7f979406e900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f97a000b3b0 tx=0x7f97a0009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 msgr2=0x7f97b01144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b01144a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f979c002a50 tx=0x7f979c0046b0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 shutdown_connections 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f979406c450 0x7f979406e900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97b00709a0 0x7f97b0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 --2- 172.21.15.63:0/1320111015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97b0071f50 0x7f97b01144a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 >> 172.21.15.63:0/1320111015 conn(0x7f97b00f8560 msgr2=0x7f97b00f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 shutdown_connections 2024-03-23T11:50:26.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.505+0000 7f97b4c00700 1 -- 172.21.15.63:0/1320111015 wait complete. 2024-03-23T11:50:26.564 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:26 smithi063 ceph-mon[35189]: pgmap v49: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T11:50:26.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 -- 172.21.15.63:0/714200392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 msgr2=0x7f06900ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 --2- 172.21.15.63:0/714200392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900ff9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0680004940 tx=0x7f0680009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:26.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 -- 172.21.15.63:0/714200392 shutdown_connections 2024-03-23T11:50:26.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 --2- 172.21.15.63:0/714200392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06900fff00 0x7f0690102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 --2- 172.21.15.63:0/714200392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900ff9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 -- 172.21.15.63:0/714200392 >> 172.21.15.63:0/714200392 conn(0x7f06900f7450 msgr2=0x7f06900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:26.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 -- 172.21.15.63:0/714200392 shutdown_connections 2024-03-23T11:50:26.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.658+0000 7f0695600700 1 -- 172.21.15.63:0/714200392 wait complete. 2024-03-23T11:50:26.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f0695600700 1 Processor -- start 2024-03-23T11:50:26.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f0695600700 1 -- start start 2024-03-23T11:50:26.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f0695600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900fd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:26.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f0695600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06900fff00 0x7f06900fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:26.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f0695600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06900fd810 con 0x7f0690066f50 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f0695600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06900fbe00 con 0x7f06900fff00 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f068ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900fd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f068ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900fd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35292/0 (socket says 172.21.15.63:35292) 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.659+0000 7f068ea00700 1 -- 172.21.15.63:0/1140211649 learned_addr learned my addr 172.21.15.63:0/1140211649 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f068ea00700 1 -- 172.21.15.63:0/1140211649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06900fff00 msgr2=0x7f06900fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f068ea00700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06900fff00 0x7f06900fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f068ea00700 1 -- 172.21.15.63:0/1140211649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680005040 con 0x7f0690066f50 2024-03-23T11:50:26.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f068ea00700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900fd210 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f068000b900 tx=0x7f0680009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:26.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f0686a00700 1 -- 172.21.15.63:0/1140211649 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f068000e040 con 0x7f0690066f50 2024-03-23T11:50:26.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f0686a00700 1 -- 172.21.15.63:0/1140211649 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0680004290 con 0x7f0690066f50 2024-03-23T11:50:26.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f0686a00700 1 -- 172.21.15.63:0/1140211649 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f068001c3f0 con 0x7f0690066f50 2024-03-23T11:50:26.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.660+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06900fc050 con 0x7f0690066f50 2024-03-23T11:50:26.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.661+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06900fc5c0 con 0x7f0690066f50 2024-03-23T11:50:26.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.662+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069004c710 con 0x7f0690066f50 2024-03-23T11:50:26.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.663+0000 7f0686a00700 1 -- 172.21.15.63:0/1140211649 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f0680013420 con 0x7f0690066f50 2024-03-23T11:50:26.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.663+0000 7f0686a00700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f067406c440 0x7f067406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:26.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.664+0000 7f0686a00700 1 -- 172.21.15.63:0/1140211649 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f068008c920 con 0x7f0690066f50 2024-03-23T11:50:26.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.664+0000 7f068e000700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f067406c440 0x7f067406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:26.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.664+0000 7f068e000700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f067406c440 0x7f067406e8f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f067c009b80 tx=0x7f067c009d60 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:26.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.668+0000 7f0686a00700 1 -- 172.21.15.63:0/1140211649 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f068005aab0 con 0x7f0690066f50 2024-03-23T11:50:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:26 smithi111 ceph-mon[44005]: pgmap v49: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T11:50:26.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.971+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0690115900 con 0x7f0690066f50 2024-03-23T11:50:26.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.972+0000 7f0686a00700 1 -- 172.21.15.63:0/1140211649 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f068005a640 con 0x7f0690066f50 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:50:26.957 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:50:26.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:26.958 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:50:26.958 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-23T11:50:26.958 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:50:26.958 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:50:26.958 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:50:26.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f067406c440 msgr2=0x7f067406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f067406c440 0x7f067406e8f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f067c009b80 tx=0x7f067c009d60 comp rx=0 tx=0).stop 2024-03-23T11:50:26.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 msgr2=0x7f06900fd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:26.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900fd210 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f068000b900 tx=0x7f0680009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:26.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 shutdown_connections 2024-03-23T11:50:26.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f067406c440 0x7f067406e8f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0690066f50 0x7f06900fd210 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 --2- 172.21.15.63:0/1140211649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06900fff00 0x7f06900fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:26.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 >> 172.21.15.63:0/1140211649 conn(0x7f06900f7450 msgr2=0x7f06900f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:26.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 shutdown_connections 2024-03-23T11:50:26.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:26.975+0000 7f0695600700 1 -- 172.21.15.63:0/1140211649 wait complete. 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 -- 172.21.15.63:0/2501419558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 msgr2=0x7fabf8070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 --2- 172.21.15.63:0/2501419558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8070d60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fabe4004940 tx=0x7fabe4009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 -- 172.21.15.63:0/2501419558 shutdown_connections 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 --2- 172.21.15.63:0/2501419558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8070d60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 --2- 172.21.15.63:0/2501419558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf8071ea0 0x7fabf80703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 -- 172.21.15.63:0/2501419558 >> 172.21.15.63:0/2501419558 conn(0x7fabf80f83d0 msgr2=0x7fabf80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 -- 172.21.15.63:0/2501419558 shutdown_connections 2024-03-23T11:50:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.122+0000 7fabfce00700 1 -- 172.21.15.63:0/2501419558 wait complete. 2024-03-23T11:50:27.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.123+0000 7fabfce00700 1 Processor -- start 2024-03-23T11:50:27.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.123+0000 7fabfce00700 1 -- start start 2024-03-23T11:50:27.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabfce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:27.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabfce00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf8071ea0 0x7fabf81142d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:27.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabfce00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf81148d0 con 0x7fabf80708f0 2024-03-23T11:50:27.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabfce00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8115010 con 0x7fabf8071ea0 2024-03-23T11:50:27.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabf7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:27.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabf7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8113d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35310/0 (socket says 172.21.15.63:35310) 2024-03-23T11:50:27.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabf7400700 1 -- 172.21.15.63:0/3197760134 learned_addr learned my addr 172.21.15.63:0/3197760134 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:27.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabf6a00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf8071ea0 0x7fabf81142d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabf7400700 1 -- 172.21.15.63:0/3197760134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf8071ea0 msgr2=0x7fabf81142d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabf7400700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf8071ea0 0x7fabf81142d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.124+0000 7fabf7400700 1 -- 172.21.15.63:0/3197760134 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe4005040 con 0x7fabf80708f0 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.125+0000 7fabf7400700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8113d90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fabe8002a80 tx=0x7fabe80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.125+0000 7fabef400700 1 -- 172.21.15.63:0/3197760134 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe8007cc0 con 0x7fabf80708f0 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.125+0000 7fabef400700 1 -- 172.21.15.63:0/3197760134 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabe800f040 con 0x7fabf80708f0 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.125+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf81152f0 con 0x7fabf80708f0 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.125+0000 7fabef400700 1 -- 172.21.15.63:0/3197760134 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe800d9e0 con 0x7fabf80708f0 2024-03-23T11:50:27.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.125+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf81158b0 con 0x7fabf80708f0 2024-03-23T11:50:27.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.127+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf805f5f0 con 0x7fabf80708f0 2024-03-23T11:50:27.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.128+0000 7fabef400700 1 -- 172.21.15.63:0/3197760134 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fabe800db40 con 0x7fabf80708f0 2024-03-23T11:50:27.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.129+0000 7fabef400700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fabdc06c450 0x7fabdc06e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:27.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.129+0000 7fabef400700 1 -- 172.21.15.63:0/3197760134 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fabe808ac10 con 0x7fabf80708f0 2024-03-23T11:50:27.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.129+0000 7fabf6a00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fabdc06c450 0x7fabdc06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:27.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.130+0000 7fabf6a00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fabdc06c450 0x7fabdc06e900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009fd0 tx=0x7fabe4009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.134+0000 7fabef400700 1 -- 172.21.15.63:0/3197760134 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fabe8058e40 con 0x7fabf80708f0 2024-03-23T11:50:27.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.389+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fabf8115c90 con 0x7fabdc06c450 2024-03-23T11:50:27.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.390+0000 7fabef400700 1 -- 172.21.15.63:0/3197760134 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fabf8115c90 con 0x7fabdc06c450 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "2/2 daemons upgraded", 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:50:27.375 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.392+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fabdc06c450 msgr2=0x7fabdc06e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.392+0000 7fabfce00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fabdc06c450 0x7fabdc06e900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009fd0 tx=0x7fabe4009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.392+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 msgr2=0x7fabf8113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.392+0000 7fabfce00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8113d90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fabe8002a80 tx=0x7fabe80046b0 comp rx=0 tx=0).stop 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.393+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 shutdown_connections 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.393+0000 7fabfce00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fabdc06c450 0x7fabdc06e900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.393+0000 7fabfce00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf80708f0 0x7fabf8113d90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:27.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.393+0000 7fabfce00700 1 --2- 172.21.15.63:0/3197760134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf8071ea0 0x7fabf81142d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:27.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.393+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 >> 172.21.15.63:0/3197760134 conn(0x7fabf80f83d0 msgr2=0x7fabf80f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:27.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.393+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 shutdown_connections 2024-03-23T11:50:27.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:27.393+0000 7fabfce00700 1 -- 172.21.15.63:0/3197760134 wait complete. 2024-03-23T11:50:27.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:27 smithi063 ceph-mon[35189]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:27.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:27 smithi063 ceph-mon[35189]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:27.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:27 smithi063 ceph-mon[35189]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:27.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:27 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1140211649' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:27.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:27 smithi111 ceph-mon[44005]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:27.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:27 smithi111 ceph-mon[44005]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:27.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:27 smithi111 ceph-mon[44005]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:27.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:27 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1140211649' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:28.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:28 smithi063 ceph-mon[35189]: pgmap v50: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-23T11:50:28.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:28 smithi063 ceph-mon[35189]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:28.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:28 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:28.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:28 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:28.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:28 smithi111 ceph-mon[44005]: pgmap v50: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-23T11:50:28.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:28 smithi111 ceph-mon[44005]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:28.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:28 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:28.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:28 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:29.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:29 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:29.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:29 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:29.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:29 smithi063 ceph-mon[35189]: pgmap v51: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:50:29.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:29 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:29.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:29 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:29.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:29 smithi111 ceph-mon[44005]: pgmap v51: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-23T11:50:32.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:32 smithi063 ceph-mon[35189]: pgmap v52: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T11:50:32.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:32 smithi111 ceph-mon[44005]: pgmap v52: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T11:50:34.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: pgmap v53: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-23T11:50:34.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:34.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:34.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:50:34.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:34 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: pgmap v53: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:34.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:34.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:34 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:35.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:35 smithi063 ceph-mon[35189]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:50:35.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:35 smithi111 ceph-mon[44005]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-23T11:50:36.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:36 smithi063 ceph-mon[35189]: pgmap v54: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T11:50:36.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:36 smithi063 ceph-mon[35189]: Upgrade: Updating alertmanager.smithi063 2024-03-23T11:50:36.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:36 smithi063 ceph-mon[35189]: Deploying daemon alertmanager.smithi063 on smithi063 2024-03-23T11:50:36.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:36 smithi111 ceph-mon[44005]: pgmap v54: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-23T11:50:36.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:36 smithi111 ceph-mon[44005]: Upgrade: Updating alertmanager.smithi063 2024-03-23T11:50:36.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:36 smithi111 ceph-mon[44005]: Deploying daemon alertmanager.smithi063 on smithi063 2024-03-23T11:50:38.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:38 smithi063 ceph-mon[35189]: pgmap v55: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:38.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:38 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:38.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:38 smithi111 ceph-mon[44005]: pgmap v55: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:38.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:38 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:40.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:40 smithi063 ceph-mon[35189]: pgmap v56: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:50:40.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:40 smithi111 ceph-mon[44005]: pgmap v56: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:50:42.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:42 smithi063 ceph-mon[35189]: pgmap v57: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:50:42.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:42 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:42.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:42 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:42.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:42 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:42.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:42 smithi111 ceph-mon[44005]: pgmap v57: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:50:42.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:42 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:42.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:42 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:42.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:42 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:44.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:44 smithi063 ceph-mon[35189]: pgmap v58: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:44.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:44 smithi111 ceph-mon[44005]: pgmap v58: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:46.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:46 smithi063 ceph-mon[35189]: pgmap v59: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:46.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:46 smithi111 ceph-mon[44005]: pgmap v59: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:48.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:48 smithi063 ceph-mon[35189]: pgmap v60: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:48.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:48.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:48 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:48.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:48 smithi111 ceph-mon[44005]: pgmap v60: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:48.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:48.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:48 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:50.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:50 smithi063 ceph-mon[35189]: pgmap v61: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:50:50.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:50 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:50.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:50 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:50 smithi111 ceph-mon[44005]: pgmap v61: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:50:50.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:50 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:50.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:50 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:52.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:52 smithi063 ceph-mon[35189]: pgmap v62: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:50:52.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:52 smithi111 ceph-mon[44005]: pgmap v62: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:50:53.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:53.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: pgmap v63: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:54 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: pgmap v63: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:54.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:54 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:55.474 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:55 smithi063 ceph-mon[35189]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:50:55.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:55 smithi111 ceph-mon[44005]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-23T11:50:56.622 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:56 smithi063 ceph-mon[35189]: pgmap v64: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:56.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:56 smithi111 ceph-mon[44005]: pgmap v64: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:50:57.475 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:57 smithi063 ceph-mon[35189]: Upgrade: Updating grafana.smithi063 2024-03-23T11:50:57.475 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:57 smithi063 ceph-mon[35189]: Deploying daemon grafana.smithi063 on smithi063 2024-03-23T11:50:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.539+0000 7f4701800700 1 -- 172.21.15.63:0/4273059337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0fff00 msgr2=0x7f46fc102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.539+0000 7f4701800700 1 --2- 172.21.15.63:0/4273059337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0fff00 0x7f46fc102320 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f46e8004940 tx=0x7f46e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.540+0000 7f4701800700 1 -- 172.21.15.63:0/4273059337 shutdown_connections 2024-03-23T11:50:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.540+0000 7f4701800700 1 --2- 172.21.15.63:0/4273059337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0fff00 0x7f46fc102320 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.540+0000 7f4701800700 1 --2- 172.21.15.63:0/4273059337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46fc0662e0 0x7f46fc0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.540+0000 7f4701800700 1 -- 172.21.15.63:0/4273059337 >> 172.21.15.63:0/4273059337 conn(0x7f46fc0f7430 msgr2=0x7f46fc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.540+0000 7f4701800700 1 -- 172.21.15.63:0/4273059337 shutdown_connections 2024-03-23T11:50:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.540+0000 7f4701800700 1 -- 172.21.15.63:0/4273059337 wait complete. 2024-03-23T11:50:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.541+0000 7f4701800700 1 Processor -- start 2024-03-23T11:50:57.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.541+0000 7f4701800700 1 -- start start 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.541+0000 7f4701800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0662e0 0x7f46fc0fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.541+0000 7f4701800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46fc0fff00 0x7f46fc0fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.541+0000 7f4701800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc1115f0 con 0x7f46fc0662e0 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.541+0000 7f4701800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc111760 con 0x7f46fc0fff00 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.542+0000 7f46faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0662e0 0x7f46fc0fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.542+0000 7f46faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0662e0 0x7f46fc0fd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41102/0 (socket says 172.21.15.63:41102) 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.542+0000 7f46faa00700 1 -- 172.21.15.63:0/302143339 learned_addr learned my addr 172.21.15.63:0/302143339 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.542+0000 7f46faa00700 1 -- 172.21.15.63:0/302143339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46fc0fff00 msgr2=0x7f46fc0fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.542+0000 7f46faa00700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46fc0fff00 0x7f46fc0fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.542+0000 7f46faa00700 1 -- 172.21.15.63:0/302143339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e8005040 con 0x7f46fc0662e0 2024-03-23T11:50:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.543+0000 7f46faa00700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0662e0 0x7f46fc0fd3f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f46ec002a50 tx=0x7f46ec004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.543+0000 7f46f2a00700 1 -- 172.21.15.63:0/302143339 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46ec007cc0 con 0x7f46fc0662e0 2024-03-23T11:50:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.543+0000 7f46f2a00700 1 -- 172.21.15.63:0/302143339 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46ec00f040 con 0x7f46fc0662e0 2024-03-23T11:50:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.543+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc111a40 con 0x7f46fc0662e0 2024-03-23T11:50:57.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.543+0000 7f46f2a00700 1 -- 172.21.15.63:0/302143339 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46ec00d990 con 0x7f46fc0662e0 2024-03-23T11:50:57.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.543+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc112000 con 0x7f46fc0662e0 2024-03-23T11:50:57.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.545+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc05f5f0 con 0x7f46fc0662e0 2024-03-23T11:50:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.545+0000 7f46f2a00700 1 -- 172.21.15.63:0/302143339 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f46ec009710 con 0x7f46fc0662e0 2024-03-23T11:50:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.546+0000 7f46f2a00700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f46e006c490 0x7f46e006e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.547+0000 7f46f2a00700 1 -- 172.21.15.63:0/302143339 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f46ec08b390 con 0x7f46fc0662e0 2024-03-23T11:50:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.547+0000 7f46fa000700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f46e006c490 0x7f46e006e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.547+0000 7f46fa000700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f46e006c490 0x7f46e006e940 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46e800b900 tx=0x7f46e8009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.552+0000 7f46f2a00700 1 -- 172.21.15.63:0/302143339 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f46ec0594a0 con 0x7f46fc0662e0 2024-03-23T11:50:57.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:57 smithi111 ceph-mon[44005]: Upgrade: Updating grafana.smithi063 2024-03-23T11:50:57.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:57 smithi111 ceph-mon[44005]: Deploying daemon grafana.smithi063 on smithi063 2024-03-23T11:50:57.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.783+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f46fc101b70 con 0x7f46e006c490 2024-03-23T11:50:57.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.785+0000 7f46f2a00700 1 -- 172.21.15.63:0/302143339 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f46fc101b70 con 0x7f46e006c490 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f46e006c490 msgr2=0x7f46e006e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f46e006c490 0x7f46e006e940 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46e800b900 tx=0x7f46e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0662e0 msgr2=0x7f46fc0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0662e0 0x7f46fc0fd3f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f46ec002a50 tx=0x7f46ec004600 comp rx=0 tx=0).stop 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 shutdown_connections 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f46e006c490 0x7f46e006e940 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc0662e0 0x7f46fc0fd3f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 --2- 172.21.15.63:0/302143339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46fc0fff00 0x7f46fc0fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.787+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 >> 172.21.15.63:0/302143339 conn(0x7f46fc0f7430 msgr2=0x7f46fc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.788+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 shutdown_connections 2024-03-23T11:50:57.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.788+0000 7f4701800700 1 -- 172.21.15.63:0/302143339 wait complete. 2024-03-23T11:50:57.788 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:50:57.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.931+0000 7fb203400700 1 -- 172.21.15.63:0/983641975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fe0d0 msgr2=0x7fb1fc0fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:57.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.931+0000 7fb203400700 1 --2- 172.21.15.63:0/983641975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fe0d0 0x7fb1fc0fe540 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8004940 tx=0x7fb1e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:57.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.932+0000 7fb203400700 1 -- 172.21.15.63:0/983641975 shutdown_connections 2024-03-23T11:50:57.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.932+0000 7fb203400700 1 --2- 172.21.15.63:0/983641975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fe0d0 0x7fb1fc0fe540 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.932+0000 7fb203400700 1 --2- 172.21.15.63:0/983641975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1fc0fcee0 0x7fb1fc0fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.932+0000 7fb203400700 1 -- 172.21.15.63:0/983641975 >> 172.21.15.63:0/983641975 conn(0x7fb1fc0f8560 msgr2=0x7fb1fc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:57.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.932+0000 7fb203400700 1 -- 172.21.15.63:0/983641975 shutdown_connections 2024-03-23T11:50:57.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.932+0000 7fb203400700 1 -- 172.21.15.63:0/983641975 wait complete. 2024-03-23T11:50:57.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.933+0000 7fb203400700 1 Processor -- start 2024-03-23T11:50:57.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.933+0000 7fb203400700 1 -- start start 2024-03-23T11:50:57.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb203400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fcee0 0x7fb1fc10d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:57.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb203400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1fc0fe0d0 0x7fb1fc10deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:57.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb203400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc10e4b0 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb203400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc10ed00 con 0x7fb1fc0fe0d0 2024-03-23T11:50:57.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb200a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fcee0 0x7fb1fc10d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:57.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb200a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fcee0 0x7fb1fc10d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41112/0 (socket says 172.21.15.63:41112) 2024-03-23T11:50:57.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb200a00700 1 -- 172.21.15.63:0/3626731594 learned_addr learned my addr 172.21.15.63:0/3626731594 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:57.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb200a00700 1 -- 172.21.15.63:0/3626731594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1fc0fe0d0 msgr2=0x7fb1fc10deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:57.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb200a00700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1fc0fe0d0 0x7fb1fc10deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:57.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb200a00700 1 -- 172.21.15.63:0/3626731594 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8005040 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.934+0000 7fb200a00700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fcee0 0x7fb1fc10d970 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000c8e0 tx=0x7fb1f00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:57.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.935+0000 7fb1f8c00700 1 -- 172.21.15.63:0/3626731594 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f000a940 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.935+0000 7fb1f8c00700 1 -- 172.21.15.63:0/3626731594 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1f000ce80 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.935+0000 7fb1f8c00700 1 -- 172.21.15.63:0/3626731594 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f00113a0 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.935+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc10efe0 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.935+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc10f520 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.936+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc05f5f0 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.937+0000 7fb1f8c00700 1 -- 172.21.15.63:0/3626731594 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fb1f0012400 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.938+0000 7fb1f8c00700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb1e406c450 0x7fb1e406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:57.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.938+0000 7fb1f8c00700 1 -- 172.21.15.63:0/3626731594 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb1f008b5b0 con 0x7fb1fc0fcee0 2024-03-23T11:50:57.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.938+0000 7fb1fbe00700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb1e406c450 0x7fb1e406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:57.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.939+0000 7fb1fbe00700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb1e406c450 0x7fb1e406e900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800b900 tx=0x7fb1e8009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:57.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:57.942+0000 7fb1f8c00700 1 -- 172.21.15.63:0/3626731594 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb1f00597c0 con 0x7fb1fc0fcee0 2024-03-23T11:50:58.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.185+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1fc10f960 con 0x7fb1e406c450 2024-03-23T11:50:58.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.186+0000 7fb1f8c00700 1 -- 172.21.15.63:0/3626731594 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fb1fc10f960 con 0x7fb1e406c450 2024-03-23T11:50:58.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.188+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb1e406c450 msgr2=0x7fb1e406e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.188+0000 7fb203400700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb1e406c450 0x7fb1e406e900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800b900 tx=0x7fb1e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:58.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.188+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fcee0 msgr2=0x7fb1fc10d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fcee0 0x7fb1fc10d970 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000c8e0 tx=0x7fb1f00044c0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 shutdown_connections 2024-03-23T11:50:58.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fb1e406c450 0x7fb1e406e900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1fc0fcee0 0x7fb1fc10d970 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 --2- 172.21.15.63:0/3626731594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1fc0fe0d0 0x7fb1fc10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 >> 172.21.15.63:0/3626731594 conn(0x7fb1fc0f8560 msgr2=0x7fb1fc0f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:58.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 shutdown_connections 2024-03-23T11:50:58.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.189+0000 7fb203400700 1 -- 172.21.15.63:0/3626731594 wait complete. 2024-03-23T11:50:58.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 -- 172.21.15.63:0/1264918008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc1002a0 msgr2=0x7fcefc100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 --2- 172.21.15.63:0/1264918008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc1002a0 0x7fcefc100710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcee8004940 tx=0x7fcee8009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:58.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 -- 172.21.15.63:0/1264918008 shutdown_connections 2024-03-23T11:50:58.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 --2- 172.21.15.63:0/1264918008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc1002a0 0x7fcefc100710 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 --2- 172.21.15.63:0/1264918008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcefc0feff0 0x7fcefc0ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 -- 172.21.15.63:0/1264918008 >> 172.21.15.63:0/1264918008 conn(0x7fcefc0fa950 msgr2=0x7fcefc0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:58.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 -- 172.21.15.63:0/1264918008 shutdown_connections 2024-03-23T11:50:58.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.325+0000 7fcf02800700 1 -- 172.21.15.63:0/1264918008 wait complete. 2024-03-23T11:50:58.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.326+0000 7fcf02800700 1 Processor -- start 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.326+0000 7fcf02800700 1 -- start start 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.326+0000 7fcf02800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc0feff0 0x7fcefc10fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.326+0000 7fcf02800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcefc1002a0 0x7fcefc110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.326+0000 7fcf02800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc110610 con 0x7fcefc0feff0 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.326+0000 7fcf02800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc110e60 con 0x7fcefc1002a0 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc0feff0 0x7fcefc10fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc0feff0 0x7fcefc10fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41132/0 (socket says 172.21.15.63:41132) 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefb400700 1 -- 172.21.15.63:0/2592255451 learned_addr learned my addr 172.21.15.63:0/2592255451 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefaa00700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcefc1002a0 0x7fcefc110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefb400700 1 -- 172.21.15.63:0/2592255451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcefc1002a0 msgr2=0x7fcefc110010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefb400700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcefc1002a0 0x7fcefc110010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefb400700 1 -- 172.21.15.63:0/2592255451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8005040 con 0x7fcefc0feff0 2024-03-23T11:50:58.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.327+0000 7fcefb400700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc0feff0 0x7fcefc10fad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fceec00eb40 tx=0x7fceec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.328+0000 7fcef3400700 1 -- 172.21.15.63:0/2592255451 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceec007cc0 con 0x7fcefc0feff0 2024-03-23T11:50:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.328+0000 7fcef3400700 1 -- 172.21.15.63:0/2592255451 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fceec00f040 con 0x7fcefc0feff0 2024-03-23T11:50:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.328+0000 7fcef3400700 1 -- 172.21.15.63:0/2592255451 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceec009570 con 0x7fcefc0feff0 2024-03-23T11:50:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.328+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc1110e0 con 0x7fcefc0feff0 2024-03-23T11:50:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.328+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc111620 con 0x7fcefc0feff0 2024-03-23T11:50:58.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.329+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc05f5f0 con 0x7fcefc0feff0 2024-03-23T11:50:58.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.330+0000 7fcef3400700 1 -- 172.21.15.63:0/2592255451 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fceec0096d0 con 0x7fcefc0feff0 2024-03-23T11:50:58.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.331+0000 7fcef3400700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fcee006c450 0x7fcee006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:58.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.331+0000 7fcef3400700 1 -- 172.21.15.63:0/2592255451 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fceec08b170 con 0x7fcefc0feff0 2024-03-23T11:50:58.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.331+0000 7fcefaa00700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fcee006c450 0x7fcee006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:58.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.332+0000 7fcefaa00700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fcee006c450 0x7fcee006e900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcee800b0d0 tx=0x7fcee8009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:58.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.335+0000 7fcef3400700 1 -- 172.21.15.63:0/2592255451 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fceec0593b0 con 0x7fcefc0feff0 2024-03-23T11:50:58.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.561+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcefc111a10 con 0x7fcee006c450 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.571+0000 7fcef3400700 1 -- 172.21.15.63:0/2592255451 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fcefc111a10 con 0x7fcee006c450 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (17s) 10s ago 13m 14.0M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (13m) 10s ago 13m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (12m) 54s ago 12m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (3m) 10s ago 13m 74.1M - 9.4.7 954c08fa6188 d4ed1ee72841 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (7m) 10s ago 7m 119M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (6m) 10s ago 6m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (6m) 54s ago 6m 65.0M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (7m) 54s ago 7m 59.8M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:50:58.556 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (2m) 10s ago 14m 497M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (80s) 54s ago 12m 446M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (14m) 10s ago 15m 55.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (12m) 54s ago 12m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (65s) 10s ago 13m 7029k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (58s) 54s ago 12m 2558k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (11m) 10s ago 11m 220M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (10m) 10s ago 10m 232M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (10m) 10s ago 10m 195M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (9m) 54s ago 9m 256M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (9m) 54s ago 9m 230M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (8m) 54s ago 8m 225M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:50:58.557 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (37s) 10s ago 12m 40.7M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:50:58.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fcee006c450 msgr2=0x7fcee006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fcee006c450 0x7fcee006e900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcee800b0d0 tx=0x7fcee8009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc0feff0 msgr2=0x7fcefc10fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc0feff0 0x7fcefc10fad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fceec00eb40 tx=0x7fceec00b040 comp rx=0 tx=0).stop 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 shutdown_connections 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fcee006c450 0x7fcee006e900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcefc0feff0 0x7fcefc10fad0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 --2- 172.21.15.63:0/2592255451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcefc1002a0 0x7fcefc110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 >> 172.21.15.63:0/2592255451 conn(0x7fcefc0fa950 msgr2=0x7fcefc0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 shutdown_connections 2024-03-23T11:50:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.574+0000 7fcf02800700 1 -- 172.21.15.63:0/2592255451 wait complete. 2024-03-23T11:50:58.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:58 smithi063 ceph-mon[35189]: pgmap v65: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:58.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:58 smithi111 ceph-mon[44005]: pgmap v65: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:50:58.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.712+0000 7f0b66800700 1 -- 172.21.15.63:0/54138326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 msgr2=0x7f0b600ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.712+0000 7f0b66800700 1 --2- 172.21.15.63:0/54138326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b600ff9c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0b50004940 tx=0x7f0b50009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:58.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.713+0000 7f0b66800700 1 -- 172.21.15.63:0/54138326 shutdown_connections 2024-03-23T11:50:58.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.713+0000 7f0b66800700 1 --2- 172.21.15.63:0/54138326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b600fff00 0x7f0b60102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.713+0000 7f0b66800700 1 --2- 172.21.15.63:0/54138326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b600ff9c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.713+0000 7f0b66800700 1 -- 172.21.15.63:0/54138326 >> 172.21.15.63:0/54138326 conn(0x7f0b600f7430 msgr2=0x7f0b600f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:58.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.713+0000 7f0b66800700 1 -- 172.21.15.63:0/54138326 shutdown_connections 2024-03-23T11:50:58.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.713+0000 7f0b66800700 1 -- 172.21.15.63:0/54138326 wait complete. 2024-03-23T11:50:58.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b66800700 1 Processor -- start 2024-03-23T11:50:58.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b66800700 1 -- start start 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b66800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b6010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b66800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b600fff00 0x7f0b601100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b66800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b601106b0 con 0x7f0b600662e0 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b66800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b60110f00 con 0x7f0b600fff00 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b5f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b6010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b5f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b6010fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41146/0 (socket says 172.21.15.63:41146) 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.714+0000 7f0b5f400700 1 -- 172.21.15.63:0/1540396951 learned_addr learned my addr 172.21.15.63:0/1540396951 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:58.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b5ea00700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b600fff00 0x7f0b601100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b5f400700 1 -- 172.21.15.63:0/1540396951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b600fff00 msgr2=0x7f0b601100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b5f400700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b600fff00 0x7f0b601100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b5f400700 1 -- 172.21.15.63:0/1540396951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b50005040 con 0x7f0b600662e0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b5f400700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b6010fb70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000b900 tx=0x7f0b50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b57400700 1 -- 172.21.15.63:0/1540396951 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b5000e040 con 0x7f0b600662e0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b57400700 1 -- 172.21.15.63:0/1540396951 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b50004290 con 0x7f0b600662e0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b60111180 con 0x7f0b600662e0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.715+0000 7f0b57400700 1 -- 172.21.15.63:0/1540396951 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b5001c3f0 con 0x7f0b600662e0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.716+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b601116c0 con 0x7f0b600662e0 2024-03-23T11:50:58.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.717+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6005f5f0 con 0x7f0b600662e0 2024-03-23T11:50:58.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.718+0000 7f0b57400700 1 -- 172.21.15.63:0/1540396951 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f0b50013420 con 0x7f0b600662e0 2024-03-23T11:50:58.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.718+0000 7f0b57400700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0b4406c450 0x7f0b4406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:58.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.719+0000 7f0b57400700 1 -- 172.21.15.63:0/1540396951 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0b5008c800 con 0x7f0b600662e0 2024-03-23T11:50:58.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.719+0000 7f0b5ea00700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0b4406c450 0x7f0b4406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:58.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.719+0000 7f0b5ea00700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0b4406c450 0x7f0b4406e900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c004990 tx=0x7f0b4c009ee0 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:58.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:58.723+0000 7f0b57400700 1 -- 172.21.15.63:0/1540396951 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0b5005aa10 con 0x7f0b600662e0 2024-03-23T11:50:59.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.054+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0b60111a90 con 0x7f0b600662e0 2024-03-23T11:50:59.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.055+0000 7f0b57400700 1 -- 172.21.15.63:0/1540396951 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f0b5005a5a0 con 0x7f0b600662e0 2024-03-23T11:50:59.039 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:50:59.039 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:50:59.040 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:50:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:50:59.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0b4406c450 msgr2=0x7f0b4406e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0b4406c450 0x7f0b4406e900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c004990 tx=0x7f0b4c009ee0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 msgr2=0x7f0b6010fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b6010fb70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000b900 tx=0x7f0b50009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 shutdown_connections 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f0b4406c450 0x7f0b4406e900 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b600662e0 0x7f0b6010fb70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 --2- 172.21.15.63:0/1540396951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b600fff00 0x7f0b601100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 >> 172.21.15.63:0/1540396951 conn(0x7f0b600f7430 msgr2=0x7f0b600fbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 shutdown_connections 2024-03-23T11:50:59.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.057+0000 7f0b66800700 1 -- 172.21.15.63:0/1540396951 wait complete. 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 -- 172.21.15.63:0/1411205709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 msgr2=0x7fddd80fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1411205709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd80fe540 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fddc4004940 tx=0x7fddc4009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 -- 172.21.15.63:0/1411205709 shutdown_connections 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1411205709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd80fe540 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1411205709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd80fcee0 0x7fddd80fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 -- 172.21.15.63:0/1411205709 >> 172.21.15.63:0/1411205709 conn(0x7fddd80f8560 msgr2=0x7fddd80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 -- 172.21.15.63:0/1411205709 shutdown_connections 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.198+0000 7fdddfa00700 1 -- 172.21.15.63:0/1411205709 wait complete. 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.199+0000 7fdddfa00700 1 Processor -- start 2024-03-23T11:50:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fdddfa00700 1 -- start start 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fdddfa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd80fcee0 0x7fddd810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fdddfa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd81100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fdddfa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81106c0 con 0x7fddd80fe0d0 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fdddfa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8110f10 con 0x7fddd80fcee0 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fddd7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd81100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fddd7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd81100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41162/0 (socket says 172.21.15.63:41162) 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fddd7e00700 1 -- 172.21.15.63:0/1800585468 learned_addr learned my addr 172.21.15.63:0/1800585468 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.200+0000 7fdddd000700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd80fcee0 0x7fddd810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fddd7e00700 1 -- 172.21.15.63:0/1800585468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd80fcee0 msgr2=0x7fddd810fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fddd7e00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd80fcee0 0x7fddd810fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fddd7e00700 1 -- 172.21.15.63:0/1800585468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc4005040 con 0x7fddd80fe0d0 2024-03-23T11:50:59.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fddd7e00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd81100c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009380 tx=0x7fddc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:59.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fddd4c00700 1 -- 172.21.15.63:0/1800585468 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc400e040 con 0x7fddd80fe0d0 2024-03-23T11:50:59.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fddd4c00700 1 -- 172.21.15.63:0/1800585468 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddc4004290 con 0x7fddd80fe0d0 2024-03-23T11:50:59.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fddd4c00700 1 -- 172.21.15.63:0/1800585468 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddc401c3f0 con 0x7fddd80fe0d0 2024-03-23T11:50:59.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.201+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd8111190 con 0x7fddd80fe0d0 2024-03-23T11:50:59.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.202+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd81116d0 con 0x7fddd80fe0d0 2024-03-23T11:50:59.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.203+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd805f5f0 con 0x7fddd80fe0d0 2024-03-23T11:50:59.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.204+0000 7fddd4c00700 1 -- 172.21.15.63:0/1800585468 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fddc4013420 con 0x7fddd80fe0d0 2024-03-23T11:50:59.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.205+0000 7fddd4c00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fddc006c450 0x7fddc006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:50:59.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.205+0000 7fddd4c00700 1 -- 172.21.15.63:0/1800585468 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fddc408c820 con 0x7fddd80fe0d0 2024-03-23T11:50:59.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.205+0000 7fdddd000700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fddc006c450 0x7fddc006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:50:59.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.206+0000 7fdddd000700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fddc006c450 0x7fddc006e900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fddd80fcc20 tx=0x7fddcc00b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:50:59.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.210+0000 7fddd4c00700 1 -- 172.21.15.63:0/1800585468 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fddc405aaf0 con 0x7fddd80fe0d0 2024-03-23T11:50:59.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.461+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fddd8111a00 con 0x7fddc006c450 2024-03-23T11:50:59.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.462+0000 7fddd4c00700 1 -- 172.21.15.63:0/1800585468 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fddd8111a00 con 0x7fddc006c450 2024-03-23T11:50:59.446 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:50:59.446 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "2/2 daemons upgraded", 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading grafana daemons", 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:50:59.447 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fddc006c450 msgr2=0x7fddc006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fddc006c450 0x7fddc006e900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fddd80fcc20 tx=0x7fddcc00b040 comp rx=0 tx=0).stop 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 msgr2=0x7fddd81100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd81100c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009380 tx=0x7fddc4009e30 comp rx=0 tx=0).stop 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 shutdown_connections 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fddc006c450 0x7fddc006e900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd80fcee0 0x7fddd810fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 --2- 172.21.15.63:0/1800585468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddd80fe0d0 0x7fddd81100c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 >> 172.21.15.63:0/1800585468 conn(0x7fddd80f8560 msgr2=0x7fddd80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.464+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 shutdown_connections 2024-03-23T11:50:59.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:50:59.465+0000 7fdddfa00700 1 -- 172.21.15.63:0/1800585468 wait complete. 2024-03-23T11:50:59.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:59 smithi063 ceph-mon[35189]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:59.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:59 smithi063 ceph-mon[35189]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:59.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:59 smithi063 ceph-mon[35189]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:59.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:50:59 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1540396951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:50:59.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:59 smithi111 ceph-mon[44005]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:59.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:59 smithi111 ceph-mon[44005]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:59.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:59 smithi111 ceph-mon[44005]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:50:59.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:50:59 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1540396951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:00.602 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:00 smithi063 ceph-mon[35189]: pgmap v66: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:00.602 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:00 smithi063 ceph-mon[35189]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:00.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:00 smithi111 ceph-mon[44005]: pgmap v66: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:00.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:00 smithi111 ceph-mon[44005]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:02.699 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:02 smithi063 ceph-mon[35189]: pgmap v67: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:02.699 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:02 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:02.699 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:02 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:02.699 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:02 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:02.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:02 smithi111 ceph-mon[44005]: pgmap v67: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:02.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:02 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:02.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:02 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:02.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:02 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:04.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:04 smithi063 ceph-mon[35189]: pgmap v68: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:04.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:04 smithi111 ceph-mon[44005]: pgmap v68: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:06.873 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:06 smithi063 ceph-mon[35189]: pgmap v69: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:06.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:06 smithi111 ceph-mon[44005]: pgmap v69: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:08.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:08 smithi063 ceph-mon[35189]: pgmap v70: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:08.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:08 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:08.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:08 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:08.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:08 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:08.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:08 smithi111 ceph-mon[44005]: pgmap v70: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:08.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:08 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:08.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:08 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:08.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:08 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:10.239 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:09 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:10.239 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:09 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:10.239 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:09 smithi063 ceph-mon[35189]: pgmap v71: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:10.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:09 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:10.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:09 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:10.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:09 smithi111 ceph-mon[44005]: pgmap v71: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:12.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:12 smithi063 ceph-mon[35189]: pgmap v72: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:12 smithi111 ceph-mon[44005]: pgmap v72: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:14.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: pgmap v73: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: Upgrade: Finalizing container_image settings 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-23T11:51:14.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-23T11:51:14.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-23T11:51:14.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-23T11:51:14.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-23T11:51:14.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-23T11:51:14.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-23T11:51:14.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: Upgrade: Complete! 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-23T11:51:14.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:14 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: pgmap v73: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-23T11:51:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: Upgrade: Finalizing container_image settings 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-23T11:51:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-23T11:51:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: Upgrade: Complete! 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-23T11:51:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:14 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:15.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:15 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:15.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:15 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:15 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:15 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:15 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:15 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:15 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:15 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:15 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:15 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:15 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:15 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:15 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:15.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:15 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:16.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:16 smithi063 ceph-mon[35189]: pgmap v74: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:16.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:16 smithi111 ceph-mon[44005]: pgmap v74: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:18.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:18 smithi063 ceph-mon[35189]: pgmap v75: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:18.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:18 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:18.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:18 smithi111 ceph-mon[44005]: pgmap v75: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:18.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:18 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:20.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:20 smithi063 ceph-mon[35189]: pgmap v76: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:20.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:20 smithi111 ceph-mon[44005]: pgmap v76: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:22.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:22 smithi063 ceph-mon[35189]: pgmap v77: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:22.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:22 smithi111 ceph-mon[44005]: pgmap v77: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:51:23.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:23 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:23.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:23 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:24.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:24 smithi063 ceph-mon[35189]: pgmap v78: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:24.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:24 smithi111 ceph-mon[44005]: pgmap v78: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:26.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:26 smithi063 ceph-mon[35189]: pgmap v79: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:26.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:26 smithi111 ceph-mon[44005]: pgmap v79: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:28.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:28 smithi063 ceph-mon[35189]: pgmap v80: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:28.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:28 smithi111 ceph-mon[44005]: pgmap v80: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 -- 172.21.15.63:0/2416654392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 msgr2=0x7f0a080fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2416654392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a080fd310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f09fc004940 tx=0x7f09fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 -- 172.21.15.63:0/2416654392 shutdown_connections 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2416654392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a080fe0d0 0x7f0a080fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2416654392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a080fd310 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 -- 172.21.15.63:0/2416654392 >> 172.21.15.63:0/2416654392 conn(0x7f0a080f8560 msgr2=0x7f0a080fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 -- 172.21.15.63:0/2416654392 shutdown_connections 2024-03-23T11:51:29.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.621+0000 7f0a0f400700 1 -- 172.21.15.63:0/2416654392 wait complete. 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.622+0000 7f0a0f400700 1 Processor -- start 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.622+0000 7f0a0f400700 1 -- start start 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.622+0000 7f0a0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a08071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.622+0000 7f0a0f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a080fe0d0 0x7f0a08070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.622+0000 7f0a0f400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a080708a0 con 0x7f0a080fcee0 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.622+0000 7f0a0f400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a080709e0 con 0x7f0a080fe0d0 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a0ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a08071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a0ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a08071dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38568/0 (socket says 172.21.15.63:38568) 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a0ca00700 1 -- 172.21.15.63:0/2650824100 learned_addr learned my addr 172.21.15.63:0/2650824100 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a07e00700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a080fe0d0 0x7f0a08070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a0ca00700 1 -- 172.21.15.63:0/2650824100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a080fe0d0 msgr2=0x7f0a08070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a0ca00700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a080fe0d0 0x7f0a08070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a0ca00700 1 -- 172.21.15.63:0/2650824100 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc005040 con 0x7f0a080fcee0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.623+0000 7f0a0ca00700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a08071dd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009380 tx=0x7f09fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.624+0000 7f0a04c00700 1 -- 172.21.15.63:0/2650824100 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09fc00e040 con 0x7f0a080fcee0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.624+0000 7f0a04c00700 1 -- 172.21.15.63:0/2650824100 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09fc004290 con 0x7f0a080fcee0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.624+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a08070c30 con 0x7f0a080fcee0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.624+0000 7f0a04c00700 1 -- 172.21.15.63:0/2650824100 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09fc01c3f0 con 0x7f0a080fcee0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.624+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a08115a00 con 0x7f0a080fcee0 2024-03-23T11:51:29.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.625+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0805f5f0 con 0x7f0a080fcee0 2024-03-23T11:51:29.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.626+0000 7f0a04c00700 1 -- 172.21.15.63:0/2650824100 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f09fc013690 con 0x7f0a080fcee0 2024-03-23T11:51:29.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.627+0000 7f0a04c00700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f09f006c450 0x7f09f006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:29.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.627+0000 7f0a04c00700 1 -- 172.21.15.63:0/2650824100 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f09fc08c860 con 0x7f0a080fcee0 2024-03-23T11:51:29.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.627+0000 7f0a07e00700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f09f006c450 0x7f09f006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:29.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.628+0000 7f0a07e00700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f09f006c450 0x7f09f006e900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09f4004f40 tx=0x7f09f400b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:29.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.632+0000 7f0a04c00700 1 -- 172.21.15.63:0/2650824100 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f09fc056f60 con 0x7f0a080fcee0 2024-03-23T11:51:29.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.863+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a08102580 con 0x7f09f006c450 2024-03-23T11:51:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.864+0000 7f0a04c00700 1 -- 172.21.15.63:0/2650824100 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0a08102580 con 0x7f09f006c450 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f09f006c450 msgr2=0x7f09f006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f09f006c450 0x7f09f006e900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09f4004f40 tx=0x7f09f400b040 comp rx=0 tx=0).stop 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 msgr2=0x7f0a08071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a08071dd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009380 tx=0x7f09fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 shutdown_connections 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f09f006c450 0x7f09f006e900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a080fcee0 0x7f0a08071dd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 --2- 172.21.15.63:0/2650824100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a080fe0d0 0x7f0a08070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 >> 172.21.15.63:0/2650824100 conn(0x7f0a080f8560 msgr2=0x7f0a080f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 shutdown_connections 2024-03-23T11:51:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:29.866+0000 7f0a0f400700 1 -- 172.21.15.63:0/2650824100 wait complete. 2024-03-23T11:51:30.551 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-23T11:51:30.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:30 smithi063 ceph-mon[35189]: pgmap v81: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:30.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:30 smithi111 ceph-mon[44005]: pgmap v81: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:30.990 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:31.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:31 smithi063 ceph-mon[35189]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:31.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:31 smithi111 ceph-mon[44005]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:32.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:32 smithi063 ceph-mon[35189]: pgmap v82: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:51:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:32 smithi111 ceph-mon[44005]: pgmap v82: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 -- 172.21.15.63:0/3415954396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 msgr2=0x7f395c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 --2- 172.21.15.63:0/3415954396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c1031d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f394c004940 tx=0x7f394c009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 -- 172.21.15.63:0/3415954396 shutdown_connections 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 --2- 172.21.15.63:0/3415954396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f395c1037a0 0x7f395c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 --2- 172.21.15.63:0/3415954396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c1031d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 -- 172.21.15.63:0/3415954396 >> 172.21.15.63:0/3415954396 conn(0x7f395c0f89b0 msgr2=0x7f395c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 -- 172.21.15.63:0/3415954396 shutdown_connections 2024-03-23T11:51:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.988+0000 7f3962c00700 1 -- 172.21.15.63:0/3415954396 wait complete. 2024-03-23T11:51:32.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.989+0000 7f3962c00700 1 Processor -- start 2024-03-23T11:51:32.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.989+0000 7f3962c00700 1 -- start start 2024-03-23T11:51:32.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.989+0000 7f3962c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:32.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f3962c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f395c1037a0 0x7f395c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:32.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f3962c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c1168a0 con 0x7f395c102e00 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f3962c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c10fc70 con 0x7f395c1037a0 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f395be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f395be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41800/0 (socket says 172.21.15.63:41800) 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f395be00700 1 -- 172.21.15.63:0/1847560756 learned_addr learned my addr 172.21.15.63:0/1847560756 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f395be00700 1 -- 172.21.15.63:0/1847560756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f395c1037a0 msgr2=0x7f395c116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f3953e00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f395c1037a0 0x7f395c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f395be00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f395c1037a0 0x7f395c116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f395be00700 1 -- 172.21.15.63:0/1847560756 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c005040 con 0x7f395c102e00 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f3953e00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f395c1037a0 0x7f395c116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.990+0000 7f395be00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c115d10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f394c009390 tx=0x7f394c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.991+0000 7f3958c00700 1 -- 172.21.15.63:0/1847560756 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f394c00e040 con 0x7f395c102e00 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.991+0000 7f3958c00700 1 -- 172.21.15.63:0/1847560756 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f394c013420 con 0x7f395c102e00 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.991+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c10fef0 con 0x7f395c102e00 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.991+0000 7f3958c00700 1 -- 172.21.15.63:0/1847560756 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f394c01c630 con 0x7f395c102e00 2024-03-23T11:51:32.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.991+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c1104b0 con 0x7f395c102e00 2024-03-23T11:51:32.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.992+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c0fe550 con 0x7f395c102e00 2024-03-23T11:51:32.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.993+0000 7f3958c00700 1 -- 172.21.15.63:0/1847560756 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f394c0139e0 con 0x7f395c102e00 2024-03-23T11:51:32.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.994+0000 7f3958c00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f394006c450 0x7f394006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:32.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.994+0000 7f3958c00700 1 -- 172.21.15.63:0/1847560756 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f394c08c700 con 0x7f395c102e00 2024-03-23T11:51:32.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.994+0000 7f3953e00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f394006c450 0x7f394006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:32.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.995+0000 7f3953e00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f394006c450 0x7f394006e900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3944007fd0 tx=0x7f3944009040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:32.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:32.999+0000 7f3958c00700 1 -- 172.21.15.63:0/1847560756 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f394c05a9c0 con 0x7f395c102e00 2024-03-23T11:51:33.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.302+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f395c109e20 con 0x7f395c102e00 2024-03-23T11:51:33.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.302+0000 7f3958c00700 1 -- 172.21.15.63:0/1847560756 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f394c05a550 con 0x7f395c102e00 2024-03-23T11:51:33.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f394006c450 msgr2=0x7f394006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:33.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f394006c450 0x7f394006e900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3944007fd0 tx=0x7f3944009040 comp rx=0 tx=0).stop 2024-03-23T11:51:33.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 msgr2=0x7f395c115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:33.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c115d10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f394c009390 tx=0x7f394c009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:33.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 shutdown_connections 2024-03-23T11:51:33.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f394006c450 0x7f394006e900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:33.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f395c102e00 0x7f395c115d10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:33.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 --2- 172.21.15.63:0/1847560756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f395c1037a0 0x7f395c116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:33.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 >> 172.21.15.63:0/1847560756 conn(0x7f395c0f89b0 msgr2=0x7f395c0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:33.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 shutdown_connections 2024-03-23T11:51:33.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:33.305+0000 7f3962c00700 1 -- 172.21.15.63:0/1847560756 wait complete. 2024-03-23T11:51:33.302 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:51:33.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:33 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/1847560756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:33.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:33 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/1847560756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:34.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-23T11:51:34.472 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:34.816 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:34 smithi063 ceph-mon[35189]: pgmap v83: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:34.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:34 smithi111 ceph-mon[44005]: pgmap v83: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:35.715 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:35 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:51:35.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:35 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:51:36.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 -- 172.21.15.63:0/3839471507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 msgr2=0x7f30c00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:36.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 --2- 172.21.15.63:0/3839471507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c00fe260 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f30b4004940 tx=0x7f30b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:36.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 -- 172.21.15.63:0/3839471507 shutdown_connections 2024-03-23T11:51:36.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 --2- 172.21.15.63:0/3839471507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30c00fe830 0x7f30c0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:36.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 --2- 172.21.15.63:0/3839471507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c00fe260 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:36.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 -- 172.21.15.63:0/3839471507 >> 172.21.15.63:0/3839471507 conn(0x7f30c0072bc0 msgr2=0x7f30c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:36.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 -- 172.21.15.63:0/3839471507 shutdown_connections 2024-03-23T11:51:36.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.458+0000 7f30c7600700 1 -- 172.21.15.63:0/3839471507 wait complete. 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.459+0000 7f30c7600700 1 Processor -- start 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.459+0000 7f30c7600700 1 -- start start 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.459+0000 7f30c7600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c7600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30c00fe830 0x7f30c0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c7600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0114680 con 0x7f30c00fde90 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c7600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c010da70 con 0x7f30c00fe830 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c0113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41814/0 (socket says 172.21.15.63:41814) 2024-03-23T11:51:36.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c4c00700 1 -- 172.21.15.63:0/3110387008 learned_addr learned my addr 172.21.15.63:0/3110387008 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:36.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30bfe00700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30c00fe830 0x7f30c0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:36.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c4c00700 1 -- 172.21.15.63:0/3110387008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30c00fe830 msgr2=0x7f30c0114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:36.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c4c00700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30c00fe830 0x7f30c0114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c4c00700 1 -- 172.21.15.63:0/3110387008 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4005040 con 0x7f30c00fde90 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30bfe00700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30c00fe830 0x7f30c0114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.460+0000 7f30c4c00700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c0113af0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009360 tx=0x7f30b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.461+0000 7f30bcc00700 1 -- 172.21.15.63:0/3110387008 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b400e040 con 0x7f30c00fde90 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.461+0000 7f30bcc00700 1 -- 172.21.15.63:0/3110387008 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30b4013420 con 0x7f30c00fde90 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.461+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c010dcf0 con 0x7f30c00fde90 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.461+0000 7f30bcc00700 1 -- 172.21.15.63:0/3110387008 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b401c630 con 0x7f30c00fde90 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.461+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c010e230 con 0x7f30c00fde90 2024-03-23T11:51:36.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.462+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c01019b0 con 0x7f30c00fde90 2024-03-23T11:51:36.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.463+0000 7f30bcc00700 1 -- 172.21.15.63:0/3110387008 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f30b40139e0 con 0x7f30c00fde90 2024-03-23T11:51:36.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.464+0000 7f30bcc00700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f30a806c440 0x7f30a806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:36.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.464+0000 7f30bcc00700 1 -- 172.21.15.63:0/3110387008 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f30b408c700 con 0x7f30c00fde90 2024-03-23T11:51:36.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.464+0000 7f30bfe00700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f30a806c440 0x7f30a806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:36.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.465+0000 7f30bfe00700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f30a806c440 0x7f30a806e8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009fd0 tx=0x7f30ac00b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:36.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.468+0000 7f30bcc00700 1 -- 172.21.15.63:0/3110387008 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f30b405a910 con 0x7f30c00fde90 2024-03-23T11:51:36.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.767+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f30c010f140 con 0x7f30c00fde90 2024-03-23T11:51:36.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.767+0000 7f30bcc00700 1 -- 172.21.15.63:0/3110387008 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f30b405a4a0 con 0x7f30c00fde90 2024-03-23T11:51:36.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f30a806c440 msgr2=0x7f30a806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:36.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f30a806c440 0x7f30a806e8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009fd0 tx=0x7f30ac00b040 comp rx=0 tx=0).stop 2024-03-23T11:51:36.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 msgr2=0x7f30c0113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:36.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c0113af0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009360 tx=0x7f30b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:36.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 shutdown_connections 2024-03-23T11:51:36.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f30a806c440 0x7f30a806e8f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:36.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30c00fde90 0x7f30c0113af0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:36.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 --2- 172.21.15.63:0/3110387008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30c00fe830 0x7f30c0114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:36.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 >> 172.21.15.63:0/3110387008 conn(0x7f30c0072bc0 msgr2=0x7f30c00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:36.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 shutdown_connections 2024-03-23T11:51:36.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:36.770+0000 7f30c7600700 1 -- 172.21.15.63:0/3110387008 wait complete. 2024-03-23T11:51:36.769 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)" 2024-03-23T11:51:36.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:36 smithi063 ceph-mon[35189]: pgmap v84: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:36.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:36 smithi111 ceph-mon[44005]: pgmap v84: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:37.690 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-23T11:51:37.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:37 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/3110387008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:37.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:37 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/3110387008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:38.129 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:38.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:38 smithi063 ceph-mon[35189]: pgmap v85: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:38.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:38 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:38.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:38 smithi111 ceph-mon[44005]: pgmap v85: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:38.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:38 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:40.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 -- 172.21.15.63:0/3878881455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d4102e00 msgr2=0x7f74d41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:40.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 --2- 172.21.15.63:0/3878881455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d4102e00 0x7f74d41031d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f74c8004940 tx=0x7f74c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:40.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 -- 172.21.15.63:0/3878881455 shutdown_connections 2024-03-23T11:51:40.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 --2- 172.21.15.63:0/3878881455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d41037a0 0x7f74d40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:40.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 --2- 172.21.15.63:0/3878881455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d4102e00 0x7f74d41031d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:40.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 -- 172.21.15.63:0/3878881455 >> 172.21.15.63:0/3878881455 conn(0x7f74d40f89b0 msgr2=0x7f74d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:40.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 -- 172.21.15.63:0/3878881455 shutdown_connections 2024-03-23T11:51:40.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.080+0000 7f74dba00700 1 -- 172.21.15.63:0/3878881455 wait complete. 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.081+0000 7f74dba00700 1 Processor -- start 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.081+0000 7f74dba00700 1 -- start start 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.081+0000 7f74dba00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d4102e00 0x7f74d4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.081+0000 7f74dba00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d41037a0 0x7f74d4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.081+0000 7f74dba00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d4116860 con 0x7f74d41037a0 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.081+0000 7f74dba00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d410fc30 con 0x7f74d4102e00 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d41037a0 0x7f74d4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d41037a0 0x7f74d4116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41820/0 (socket says 172.21.15.63:41820) 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d3e00700 1 -- 172.21.15.63:0/488626501 learned_addr learned my addr 172.21.15.63:0/488626501 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d9000700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d4102e00 0x7f74d4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d3e00700 1 -- 172.21.15.63:0/488626501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d4102e00 msgr2=0x7f74d4115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:40.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d3e00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d4102e00 0x7f74d4115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d3e00700 1 -- 172.21.15.63:0/488626501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c8005040 con 0x7f74d41037a0 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d3e00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d41037a0 0x7f74d4116210 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f74c000c890 tx=0x7f74c00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.082+0000 7f74d0c00700 1 -- 172.21.15.63:0/488626501 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c000a700 con 0x7f74d41037a0 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.083+0000 7f74d0c00700 1 -- 172.21.15.63:0/488626501 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74c000ad10 con 0x7f74d41037a0 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.083+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d410ff10 con 0x7f74d41037a0 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.083+0000 7f74d0c00700 1 -- 172.21.15.63:0/488626501 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c0012e70 con 0x7f74d41037a0 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.083+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d41104d0 con 0x7f74d41037a0 2024-03-23T11:51:40.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.084+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d40fe550 con 0x7f74d41037a0 2024-03-23T11:51:40.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.085+0000 7f74d0c00700 1 -- 172.21.15.63:0/488626501 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f74c000a860 con 0x7f74d41037a0 2024-03-23T11:51:40.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.086+0000 7f74d0c00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f74bc06c450 0x7f74bc06e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:40.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.086+0000 7f74d0c00700 1 -- 172.21.15.63:0/488626501 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f74c008b5e0 con 0x7f74d41037a0 2024-03-23T11:51:40.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.086+0000 7f74d9000700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f74bc06c450 0x7f74bc06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:40.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.087+0000 7f74d9000700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f74bc06c450 0x7f74bc06e900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f74c800b840 tx=0x7f74c8009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:40.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.090+0000 7f74d0c00700 1 -- 172.21.15.63:0/488626501 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f74c0059770 con 0x7f74d41037a0 2024-03-23T11:51:40.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.387+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f74d405f5f0 con 0x7f74d41037a0 2024-03-23T11:51:40.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.388+0000 7f74d0c00700 1 -- 172.21.15.63:0/488626501 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f74c0059300 con 0x7f74d41037a0 2024-03-23T11:51:40.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f74bc06c450 msgr2=0x7f74bc06e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f74bc06c450 0x7f74bc06e900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f74c800b840 tx=0x7f74c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d41037a0 msgr2=0x7f74d4116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d41037a0 0x7f74d4116210 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f74c000c890 tx=0x7f74c00044c0 comp rx=0 tx=0).stop 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 shutdown_connections 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f74bc06c450 0x7f74bc06e900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d4102e00 0x7f74d4115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 --2- 172.21.15.63:0/488626501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74d41037a0 0x7f74d4116210 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 >> 172.21.15.63:0/488626501 conn(0x7f74d40f89b0 msgr2=0x7f74d40f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 shutdown_connections 2024-03-23T11:51:40.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:40.390+0000 7f74dba00700 1 -- 172.21.15.63:0/488626501 wait complete. 2024-03-23T11:51:40.389 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:51:40.768 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:40 smithi063 ceph-mon[35189]: pgmap v86: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:40.768 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:40 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/488626501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:40.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:40 smithi111 ceph-mon[44005]: pgmap v86: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:40.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:40 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/488626501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:51:41.113 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-23T11:51:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:42.752 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:42 smithi063 ceph-mon[35189]: pgmap v87: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:42.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:42 smithi111 ceph-mon[44005]: pgmap v87: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 -- 172.21.15.63:0/2839282049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c4070bf0 msgr2=0x7fc3c4109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/2839282049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c4070bf0 0x7fc3c4109190 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac004940 tx=0x7fc3ac009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 -- 172.21.15.63:0/2839282049 shutdown_connections 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/2839282049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c4070bf0 0x7fc3c4109190 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/2839282049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3c40702e0 0x7fc3c40706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 -- 172.21.15.63:0/2839282049 >> 172.21.15.63:0/2839282049 conn(0x7fc3c40f89b0 msgr2=0x7fc3c40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 -- 172.21.15.63:0/2839282049 shutdown_connections 2024-03-23T11:51:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.526+0000 7fc3c9e00700 1 -- 172.21.15.63:0/2839282049 wait complete. 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.527+0000 7fc3c9e00700 1 Processor -- start 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.527+0000 7fc3c9e00700 1 -- start start 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.527+0000 7fc3c9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c40702e0 0x7fc3c4115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.527+0000 7fc3c9e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3c4070bf0 0x7fc3c4116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.527+0000 7fc3c9e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c4116650 con 0x7fc3c40702e0 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c9e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c410fc30 con 0x7fc3c4070bf0 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c40702e0 0x7fc3c4115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c40702e0 0x7fc3c4115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40008/0 (socket says 172.21.15.63:40008) 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c2a00700 1 -- 172.21.15.63:0/221489932 learned_addr learned my addr 172.21.15.63:0/221489932 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c2a00700 1 -- 172.21.15.63:0/221489932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3c4070bf0 msgr2=0x7fc3c4116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:43.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c2a00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3c4070bf0 0x7fc3c4116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c2a00700 1 -- 172.21.15.63:0/221489932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac005040 con 0x7fc3c40702e0 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.528+0000 7fc3c2a00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c40702e0 0x7fc3c4115ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400eb40 tx=0x7fc3b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.529+0000 7fc3baa00700 1 -- 172.21.15.63:0/221489932 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3b4007cc0 con 0x7fc3c40702e0 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.529+0000 7fc3baa00700 1 -- 172.21.15.63:0/221489932 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3b400f040 con 0x7fc3c40702e0 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.529+0000 7fc3baa00700 1 -- 172.21.15.63:0/221489932 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3b4009570 con 0x7fc3c40702e0 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.529+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c410feb0 con 0x7fc3c40702e0 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.529+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c41103f0 con 0x7fc3c40702e0 2024-03-23T11:51:43.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.530+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c4071a20 con 0x7fc3c40702e0 2024-03-23T11:51:43.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.531+0000 7fc3baa00700 1 -- 172.21.15.63:0/221489932 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fc3b40096d0 con 0x7fc3c40702e0 2024-03-23T11:51:43.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.532+0000 7fc3baa00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc3a806c440 0x7fc3a806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:43.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.532+0000 7fc3baa00700 1 -- 172.21.15.63:0/221489932 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc3b408b260 con 0x7fc3c40702e0 2024-03-23T11:51:43.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.532+0000 7fc3bbe00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc3a806c440 0x7fc3a806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:43.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.533+0000 7fc3bbe00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc3a806c440 0x7fc3a806e8f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00b0d0 tx=0x7fc3ac009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:43.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.536+0000 7fc3baa00700 1 -- 172.21.15.63:0/221489932 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc3b4059420 con 0x7fc3c40702e0 2024-03-23T11:51:43.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:43.772+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3c4110610 con 0x7fc3a806c440 2024-03-23T11:51:44.781 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:44 smithi063 ceph-mon[35189]: pgmap v88: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:44.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:44 smithi111 ceph-mon[44005]: pgmap v88: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:45.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.492+0000 7fc3baa00700 1 -- 172.21.15.63:0/221489932 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5093 (secure 0 0 0) 0x7fc3c4110610 con 0x7fc3a806c440 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.494+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc3a806c440 msgr2=0x7fc3a806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.494+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc3a806c440 0x7fc3a806e8f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00b0d0 tx=0x7fc3ac009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.494+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c40702e0 msgr2=0x7fc3c4115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.494+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c40702e0 0x7fc3c4115ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400eb40 tx=0x7fc3b400b040 comp rx=0 tx=0).stop 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.495+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 shutdown_connections 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.495+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc3a806c440 0x7fc3a806e8f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.495+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c40702e0 0x7fc3c4115ac0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.495+0000 7fc3c9e00700 1 --2- 172.21.15.63:0/221489932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3c4070bf0 0x7fc3c4116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:45.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.495+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 >> 172.21.15.63:0/221489932 conn(0x7fc3c40f89b0 msgr2=0x7fc3c4105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:45.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.495+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 shutdown_connections 2024-03-23T11:51:45.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:45.495+0000 7fc3c9e00700 1 -- 172.21.15.63:0/221489932 wait complete. 2024-03-23T11:51:45.493 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:51:45.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:45 smithi063 ceph-mon[35189]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:45.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:45 smithi111 ceph-mon[44005]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:46.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph orch ps' 2024-03-23T11:51:46.691 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:46.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:46 smithi063 ceph-mon[35189]: pgmap v89: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:46.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:46 smithi111 ceph-mon[44005]: pgmap v89: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:47.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:47 smithi063 ceph-mon[35189]: pgmap v90: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:47.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:47 smithi111 ceph-mon[44005]: pgmap v90: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:48.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 -- 172.21.15.63:0/237678101 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 msgr2=0x7fdbcc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 --2- 172.21.15.63:0/237678101 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc0fce10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8004940 tx=0x7fdbb8009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 -- 172.21.15.63:0/237678101 shutdown_connections 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 --2- 172.21.15.63:0/237678101 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc0fce10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 --2- 172.21.15.63:0/237678101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbcc102e00 0x7fdbcc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 -- 172.21.15.63:0/237678101 >> 172.21.15.63:0/237678101 conn(0x7fdbcc0f89b0 msgr2=0x7fdbcc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 -- 172.21.15.63:0/237678101 shutdown_connections 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.608+0000 7fdbd1400700 1 -- 172.21.15.63:0/237678101 wait complete. 2024-03-23T11:51:48.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.609+0000 7fdbd1400700 1 Processor -- start 2024-03-23T11:51:48.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.609+0000 7fdbd1400700 1 -- start start 2024-03-23T11:51:48.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbd1400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbcc102e00 0x7fdbcc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:48.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbd1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:48.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbd1400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0722a0 con 0x7fdbcc1037a0 2024-03-23T11:51:48.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbd1400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc070d40 con 0x7fdbcc102e00 2024-03-23T11:51:48.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbc9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbc9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40022/0 (socket says 172.21.15.63:40022) 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbc9e00700 1 -- 172.21.15.63:0/2854442411 learned_addr learned my addr 172.21.15.63:0/2854442411 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.610+0000 7fdbca800700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbcc102e00 0x7fdbcc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbc9e00700 1 -- 172.21.15.63:0/2854442411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbcc102e00 msgr2=0x7fdbcc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbc9e00700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbcc102e00 0x7fdbcc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbc9e00700 1 -- 172.21.15.63:0/2854442411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb8005040 con 0x7fdbcc1037a0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbc9e00700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc070800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009390 tx=0x7fdbb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbc2a00700 1 -- 172.21.15.63:0/2854442411 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbb800e040 con 0x7fdbcc1037a0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbc2a00700 1 -- 172.21.15.63:0/2854442411 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbb8013420 con 0x7fdbcc1037a0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbcc070fc0 con 0x7fdbcc1037a0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbc2a00700 1 -- 172.21.15.63:0/2854442411 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbb801c630 con 0x7fdbcc1037a0 2024-03-23T11:51:48.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.611+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbcc11a0c0 con 0x7fdbcc1037a0 2024-03-23T11:51:48.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.612+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc0fe550 con 0x7fdbcc1037a0 2024-03-23T11:51:48.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.613+0000 7fdbc2a00700 1 -- 172.21.15.63:0/2854442411 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fdbb8013a00 con 0x7fdbcc1037a0 2024-03-23T11:51:48.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.614+0000 7fdbc2a00700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdbb006c440 0x7fdbb006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:48.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.614+0000 7fdbc2a00700 1 -- 172.21.15.63:0/2854442411 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdbb808c3a0 con 0x7fdbcc1037a0 2024-03-23T11:51:48.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.614+0000 7fdbca800700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdbb006c440 0x7fdbb006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:48.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.615+0000 7fdbca800700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdbb006c440 0x7fdbb006e8f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009bf0 tx=0x7fdbbc0091a0 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:48.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.619+0000 7fdbc2a00700 1 -- 172.21.15.63:0/2854442411 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdbb8056b20 con 0x7fdbcc1037a0 2024-03-23T11:51:48.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.842+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbcc0fa0e0 con 0x7fdbb006c440 2024-03-23T11:51:48.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.852+0000 7fdbc2a00700 1 -- 172.21.15.63:0/2854442411 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fdbcc0fa0e0 con 0x7fdbb006c440 2024-03-23T11:51:48.836 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:51:48.836 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (67s) 41s ago 14m 14.1M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (14m) 41s ago 14m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (13m) 104s ago 13m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (47s) 41s ago 13m 44.1M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (7m) 41s ago 7m 119M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (7m) 41s ago 7m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (7m) 104s ago 7m 65.0M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (7m) 104s ago 7m 59.8M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (3m) 41s ago 15m 498M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (2m) 104s ago 13m 446M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (15m) 41s ago 15m 55.1M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (13m) 104s ago 13m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (115s) 41s ago 14m 7390k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (109s) 104s ago 13m 2558k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (12m) 41s ago 12m 220M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (11m) 41s ago 11m 232M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:51:48.837 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (11m) 41s ago 11m 195M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:51:48.838 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (10m) 104s ago 10m 256M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:51:48.838 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (10m) 104s ago 10m 230M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:51:48.838 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (9m) 104s ago 9m 225M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:51:48.838 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (88s) 41s ago 13m 41.4M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.854+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdbb006c440 msgr2=0x7fdbb006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.854+0000 7fdbd1400700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdbb006c440 0x7fdbb006e8f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009bf0 tx=0x7fdbbc0091a0 comp rx=0 tx=0).stop 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.854+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 msgr2=0x7fdbcc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.854+0000 7fdbd1400700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc070800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009390 tx=0x7fdbb8009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.855+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 shutdown_connections 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.855+0000 7fdbd1400700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdbb006c440 0x7fdbb006e8f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.855+0000 7fdbd1400700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbcc102e00 0x7fdbcc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.855+0000 7fdbd1400700 1 --2- 172.21.15.63:0/2854442411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbcc1037a0 0x7fdbcc070800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.855+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 >> 172.21.15.63:0/2854442411 conn(0x7fdbcc0f89b0 msgr2=0x7fdbcc0f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.855+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 shutdown_connections 2024-03-23T11:51:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:48.855+0000 7fdbd1400700 1 -- 172.21.15.63:0/2854442411 wait complete. 2024-03-23T11:51:48.860 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:48 smithi063 ceph-mon[35189]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:51:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:48 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:51:49.586 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-23T11:51:49.587 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:51:49.587 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-03-23T11:51:49.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:49 smithi063 ceph-mon[35189]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:49.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:49 smithi063 ceph-mon[35189]: pgmap v91: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:49.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:49 smithi111 ceph-mon[44005]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:51:49.903 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:49 smithi111 ceph-mon[44005]: pgmap v91: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:50.027 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:51.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 -- 172.21.15.63:0/3855148301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e4102e20 msgr2=0x7fe3e41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 --2- 172.21.15.63:0/3855148301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e4102e20 0x7fe3e41031f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8004940 tx=0x7fe3d8009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 -- 172.21.15.63:0/3855148301 shutdown_connections 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 --2- 172.21.15.63:0/3855148301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3e41037c0 0x7fe3e40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 --2- 172.21.15.63:0/3855148301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e4102e20 0x7fe3e41031f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 -- 172.21.15.63:0/3855148301 >> 172.21.15.63:0/3855148301 conn(0x7fe3e40f89d0 msgr2=0x7fe3e40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 -- 172.21.15.63:0/3855148301 shutdown_connections 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.875+0000 7fe3ec400700 1 -- 172.21.15.63:0/3855148301 wait complete. 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.876+0000 7fe3ec400700 1 Processor -- start 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.876+0000 7fe3ec400700 1 -- start start 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3ec400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3e4102e20 0x7fe3e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3ec400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e41037c0 0x7fe3e4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3ec400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e4070d90 con 0x7fe3e41037c0 2024-03-23T11:51:51.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3ec400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e4070ed0 con 0x7fe3e4102e20 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e41037c0 0x7fe3e4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e41037c0 0x7fe3e4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34294/0 (socket says 172.21.15.63:34294) 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9000700 1 -- 172.21.15.63:0/138123245 learned_addr learned my addr 172.21.15.63:0/138123245 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9a00700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3e4102e20 0x7fe3e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9000700 1 -- 172.21.15.63:0/138123245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3e4102e20 msgr2=0x7fe3e40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9000700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3e4102e20 0x7fe3e40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9000700 1 -- 172.21.15.63:0/138123245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8005040 con 0x7fe3e41037c0 2024-03-23T11:51:51.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.877+0000 7fe3e9000700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e41037c0 0x7fe3e4070800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0002a50 tx=0x7fe3d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:51.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.878+0000 7fe3e1600700 1 -- 172.21.15.63:0/138123245 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3d0007cc0 con 0x7fe3e41037c0 2024-03-23T11:51:51.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.878+0000 7fe3e1600700 1 -- 172.21.15.63:0/138123245 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3d000f040 con 0x7fe3e41037c0 2024-03-23T11:51:51.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.878+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e4119e00 con 0x7fe3e41037c0 2024-03-23T11:51:51.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.878+0000 7fe3e1600700 1 -- 172.21.15.63:0/138123245 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3d000d9a0 con 0x7fe3e41037c0 2024-03-23T11:51:51.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.878+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e411a280 con 0x7fe3e41037c0 2024-03-23T11:51:51.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.879+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e40fe570 con 0x7fe3e41037c0 2024-03-23T11:51:51.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.880+0000 7fe3e1600700 1 -- 172.21.15.63:0/138123245 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fe3d000dc10 con 0x7fe3e41037c0 2024-03-23T11:51:51.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.881+0000 7fe3e1600700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe3cc06c450 0x7fe3cc06e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:51.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.881+0000 7fe3e1600700 1 -- 172.21.15.63:0/138123245 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe3d008aff0 con 0x7fe3e41037c0 2024-03-23T11:51:51.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.881+0000 7fe3e9a00700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe3cc06c450 0x7fe3cc06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:51.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.882+0000 7fe3e9a00700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe3cc06c450 0x7fe3cc06e900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800b3b0 tx=0x7fe3d8009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:51.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:51.886+0000 7fe3e1600700 1 -- 172.21.15.63:0/138123245 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fe3d0055790 con 0x7fe3e41037c0 2024-03-23T11:51:52.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.102+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fe3e405f5f0 con 0x7fe3e41037c0 2024-03-23T11:51:52.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.109+0000 7fe3e1600700 1 -- 172.21.15.63:0/138123245 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v47)=0 v47) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe3e405f5f0 con 0x7fe3e41037c0 2024-03-23T11:51:52.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe3cc06c450 msgr2=0x7fe3cc06e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:52.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe3cc06c450 0x7fe3cc06e900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800b3b0 tx=0x7fe3d8009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:52.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e41037c0 msgr2=0x7fe3e4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:52.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e41037c0 0x7fe3e4070800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0002a50 tx=0x7fe3d0004600 comp rx=0 tx=0).stop 2024-03-23T11:51:52.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 shutdown_connections 2024-03-23T11:51:52.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fe3cc06c450 0x7fe3cc06e900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:52.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3e4102e20 0x7fe3e40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:52.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 --2- 172.21.15.63:0/138123245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3e41037c0 0x7fe3e4070800 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:52.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 >> 172.21.15.63:0/138123245 conn(0x7fe3e40f89d0 msgr2=0x7fe3e40f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:52.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 shutdown_connections 2024-03-23T11:51:52.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:52.112+0000 7fe3ec400700 1 -- 172.21.15.63:0/138123245 wait complete. 2024-03-23T11:51:52.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:52 smithi063 ceph-mon[35189]: pgmap v92: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:52.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:52 smithi111 ceph-mon[44005]: pgmap v92: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:52.965 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-23T11:51:52.966 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:51:52.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-23T11:51:53.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:53 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/138123245' entity='client.admin' 2024-03-23T11:51:53.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:53.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:53.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:53.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:53 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:53.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:53 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:53.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:53 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/138123245' entity='client.admin' 2024-03-23T11:51:53.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:51:53.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:51:53.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:51:53.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:53 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:51:53.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:53 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:51:53.410 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:54.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:54 smithi111 ceph-mon[44005]: pgmap v93: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:54.466 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:54 smithi063 ceph-mon[35189]: pgmap v93: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:55.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 -- 172.21.15.63:0/1870354637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c102e00 msgr2=0x7f885c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 --2- 172.21.15.63:0/1870354637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c102e00 0x7f885c1031d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f884c004940 tx=0x7f884c009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 -- 172.21.15.63:0/1870354637 shutdown_connections 2024-03-23T11:51:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 --2- 172.21.15.63:0/1870354637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c1037a0 0x7f885c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 --2- 172.21.15.63:0/1870354637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c102e00 0x7f885c1031d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 -- 172.21.15.63:0/1870354637 >> 172.21.15.63:0/1870354637 conn(0x7f885c0f89b0 msgr2=0x7f885c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 -- 172.21.15.63:0/1870354637 shutdown_connections 2024-03-23T11:51:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.525+0000 7f8861200700 1 -- 172.21.15.63:0/1870354637 wait complete. 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.526+0000 7f8861200700 1 Processor -- start 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.526+0000 7f8861200700 1 -- start start 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.526+0000 7f8861200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c102e00 0x7f885c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.526+0000 7f8861200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c1037a0 0x7f885c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.526+0000 7f8861200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c070e20 con 0x7f885c1037a0 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.526+0000 7f8861200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c070f60 con 0x7f885c102e00 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8853e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c1037a0 0x7f885c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8853e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c1037a0 0x7f885c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34320/0 (socket says 172.21.15.63:34320) 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8853e00700 1 -- 172.21.15.63:0/1023897140 learned_addr learned my addr 172.21.15.63:0/1023897140 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8859e00700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c102e00 0x7f885c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8853e00700 1 -- 172.21.15.63:0/1023897140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c102e00 msgr2=0x7f885c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8853e00700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c102e00 0x7f885c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8853e00700 1 -- 172.21.15.63:0/1023897140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c005040 con 0x7f885c1037a0 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.527+0000 7f8853e00700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c1037a0 0x7f885c070800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f884400b790 tx=0x7f8844009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.528+0000 7f8852000700 1 -- 172.21.15.63:0/1023897140 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f884400f6f0 con 0x7f885c1037a0 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.528+0000 7f8852000700 1 -- 172.21.15.63:0/1023897140 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f884400fd00 con 0x7f885c1037a0 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.528+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c119de0 con 0x7f885c1037a0 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.528+0000 7f8852000700 1 -- 172.21.15.63:0/1023897140 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88440103a0 con 0x7f885c1037a0 2024-03-23T11:51:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.528+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c11a290 con 0x7f885c1037a0 2024-03-23T11:51:55.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.529+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885c0fe550 con 0x7f885c1037a0 2024-03-23T11:51:55.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.530+0000 7f8852000700 1 -- 172.21.15.63:0/1023897140 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f884400f850 con 0x7f885c1037a0 2024-03-23T11:51:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.531+0000 7f8852000700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f884006c450 0x7f884006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.531+0000 7f8852000700 1 -- 172.21.15.63:0/1023897140 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f884408b350 con 0x7f885c1037a0 2024-03-23T11:51:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.531+0000 7f8859e00700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f884006c450 0x7f884006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:55.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.532+0000 7f8859e00700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f884006c450 0x7f884006e900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f884c00b020 tx=0x7f884c009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:55.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.535+0000 7f8852000700 1 -- 172.21.15.63:0/1023897140 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f8844055a50 con 0x7f885c1037a0 2024-03-23T11:51:55.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.749+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f885c05f5f0 con 0x7f885c1037a0 2024-03-23T11:51:55.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.750+0000 7f8852000700 1 -- 172.21.15.63:0/1023897140 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v47)=0 v47) v1 ==== 155+0+0 (secure 0 0 0) 0x7f8844059070 con 0x7f885c1037a0 2024-03-23T11:51:55.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f884006c450 msgr2=0x7f884006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:55.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f884006c450 0x7f884006e900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f884c00b020 tx=0x7f884c009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:55.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c1037a0 msgr2=0x7f885c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:55.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c1037a0 0x7f885c070800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f884400b790 tx=0x7f8844009220 comp rx=0 tx=0).stop 2024-03-23T11:51:55.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 shutdown_connections 2024-03-23T11:51:55.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f884006c450 0x7f884006e900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:55.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c102e00 0x7f885c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:55.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 --2- 172.21.15.63:0/1023897140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f885c1037a0 0x7f885c070800 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:55.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.752+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 >> 172.21.15.63:0/1023897140 conn(0x7f885c0f89b0 msgr2=0x7f885c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:55.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.753+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 shutdown_connections 2024-03-23T11:51:55.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:55.753+0000 7f8861200700 1 -- 172.21.15.63:0/1023897140 wait complete. 2024-03-23T11:51:56.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:56 smithi063 ceph-mon[35189]: pgmap v94: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:56.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:56 smithi111 ceph-mon[44005]: pgmap v94: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:51:56.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-23T11:51:56.892 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:51:58.573 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:51:58 smithi063 ceph-mon[35189]: pgmap v95: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:58.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:51:58 smithi111 ceph-mon[44005]: pgmap v95: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:51:58.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 -- 172.21.15.63:0/1845486745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 msgr2=0x7fc8f81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:58.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 --2- 172.21.15.63:0/1845486745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f81031d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec004940 tx=0x7fc8ec009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:58.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 -- 172.21.15.63:0/1845486745 shutdown_connections 2024-03-23T11:51:58.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 --2- 172.21.15.63:0/1845486745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f81037a0 0x7fc8f80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:58.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 --2- 172.21.15.63:0/1845486745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f81031d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:58.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 -- 172.21.15.63:0/1845486745 >> 172.21.15.63:0/1845486745 conn(0x7fc8f80f89b0 msgr2=0x7fc8f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:58.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 -- 172.21.15.63:0/1845486745 shutdown_connections 2024-03-23T11:51:58.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.888+0000 7fc900800700 1 -- 172.21.15.63:0/1845486745 wait complete. 2024-03-23T11:51:58.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.889+0000 7fc900800700 1 Processor -- start 2024-03-23T11:51:58.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.889+0000 7fc900800700 1 -- start start 2024-03-23T11:51:58.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.890+0000 7fc900800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:58.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.890+0000 7fc900800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f81037a0 0x7fc8f8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:58.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.890+0000 7fc900800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8070d90 con 0x7fc8f8102e00 2024-03-23T11:51:58.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.890+0000 7fc900800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8070ed0 con 0x7fc8f81037a0 2024-03-23T11:51:58.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.890+0000 7fc8fde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:58.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.890+0000 7fc8fde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34328/0 (socket says 172.21.15.63:34328) 2024-03-23T11:51:58.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.890+0000 7fc8fde00700 1 -- 172.21.15.63:0/490288334 learned_addr learned my addr 172.21.15.63:0/490288334 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8fd400700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f81037a0 0x7fc8f8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8fde00700 1 -- 172.21.15.63:0/490288334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f81037a0 msgr2=0x7fc8f8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8fde00700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f81037a0 0x7fc8f8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8fde00700 1 -- 172.21.15.63:0/490288334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec005040 con 0x7fc8f8102e00 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8fde00700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f80702c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009390 tx=0x7fc8ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8f2000700 1 -- 172.21.15.63:0/490288334 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8ec00e040 con 0x7fc8f8102e00 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8f2000700 1 -- 172.21.15.63:0/490288334 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8ec013420 con 0x7fc8f8102e00 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f8071120 con 0x7fc8f8102e00 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc8f2000700 1 -- 172.21.15.63:0/490288334 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8ec01c630 con 0x7fc8f8102e00 2024-03-23T11:51:58.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.891+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f811a220 con 0x7fc8f8102e00 2024-03-23T11:51:58.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.892+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f80fe550 con 0x7fc8f8102e00 2024-03-23T11:51:58.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.894+0000 7fc8f2000700 1 -- 172.21.15.63:0/490288334 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fc8ec0139e0 con 0x7fc8f8102e00 2024-03-23T11:51:58.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.894+0000 7fc8f2000700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc8e006c450 0x7fc8e006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:51:58.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.894+0000 7fc8f2000700 1 -- 172.21.15.63:0/490288334 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc8ec08c7d0 con 0x7fc8f8102e00 2024-03-23T11:51:58.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.895+0000 7fc8fd400700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc8e006c450 0x7fc8e006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:51:58.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.895+0000 7fc8fd400700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc8e006c450 0x7fc8e006e900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8071c00 tx=0x7fc8e8009ee0 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:51:58.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:58.900+0000 7fc8f2000700 1 -- 172.21.15.63:0/490288334 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc8ec056ed0 con 0x7fc8f8102e00 2024-03-23T11:51:59.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.115+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fc8f804bed0 con 0x7fc8f8102e00 2024-03-23T11:51:59.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.115+0000 7fc8f2000700 1 -- 172.21.15.63:0/490288334 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v47)=0 v47) v1 ==== 163+0+0 (secure 0 0 0) 0x7fc8ec05a4f0 con 0x7fc8f8102e00 2024-03-23T11:51:59.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc8e006c450 msgr2=0x7fc8e006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:59.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc8e006c450 0x7fc8e006e900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8071c00 tx=0x7fc8e8009ee0 comp rx=0 tx=0).stop 2024-03-23T11:51:59.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 msgr2=0x7fc8f80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:51:59.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f80702c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009390 tx=0x7fc8ec009e30 comp rx=0 tx=0).stop 2024-03-23T11:51:59.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 shutdown_connections 2024-03-23T11:51:59.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fc8e006c450 0x7fc8e006e900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:59.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f8102e00 0x7fc8f80702c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:59.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 --2- 172.21.15.63:0/490288334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f81037a0 0x7fc8f8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:51:59.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 >> 172.21.15.63:0/490288334 conn(0x7fc8f80f89b0 msgr2=0x7fc8f80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:51:59.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 shutdown_connections 2024-03-23T11:51:59.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:51:59.118+0000 7fc900800700 1 -- 172.21.15.63:0/490288334 wait complete. 2024-03-23T11:51:59.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-23T11:52:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:52:00.517 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:00 smithi063 ceph-mon[35189]: pgmap v96: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:00.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:00 smithi111 ceph-mon[44005]: pgmap v96: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 -- 172.21.15.63:0/3670094552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc105990 msgr2=0x7f08cc0ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 --2- 172.21.15.63:0/3670094552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc105990 0x7f08cc0ff000 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f08b8004940 tx=0x7f08b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 -- 172.21.15.63:0/3670094552 shutdown_connections 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 --2- 172.21.15.63:0/3670094552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc105990 0x7f08cc0ff000 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 --2- 172.21.15.63:0/3670094552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08cc104ff0 0x7f08cc1053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 -- 172.21.15.63:0/3670094552 >> 172.21.15.63:0/3670094552 conn(0x7f08cc0faba0 msgr2=0x7f08cc0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 -- 172.21.15.63:0/3670094552 shutdown_connections 2024-03-23T11:52:02.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.202+0000 7f08d4400700 1 -- 172.21.15.63:0/3670094552 wait complete. 2024-03-23T11:52:02.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.203+0000 7f08d4400700 1 Processor -- start 2024-03-23T11:52:02.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.203+0000 7f08d4400700 1 -- start start 2024-03-23T11:52:02.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d4400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc104ff0 0x7f08cc115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:02.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d4400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08cc105990 0x7f08cc115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d4400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc116620 con 0x7f08cc104ff0 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d4400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc10fc00 con 0x7f08cc105990 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d1a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc104ff0 0x7f08cc115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d1a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc104ff0 0x7f08cc115a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34650/0 (socket says 172.21.15.63:34650) 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d1a00700 1 -- 172.21.15.63:0/4217759151 learned_addr learned my addr 172.21.15.63:0/4217759151 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d1a00700 1 -- 172.21.15.63:0/4217759151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08cc105990 msgr2=0x7f08cc115fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d1a00700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08cc105990 0x7f08cc115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:02.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.204+0000 7f08d1a00700 1 -- 172.21.15.63:0/4217759151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b8005040 con 0x7f08cc104ff0 2024-03-23T11:52:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.205+0000 7f08d1a00700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc104ff0 0x7f08cc115a90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f08c000c8c0 tx=0x7f08c00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.205+0000 7f08c9600700 1 -- 172.21.15.63:0/4217759151 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08c000a980 con 0x7f08cc104ff0 2024-03-23T11:52:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.205+0000 7f08c9600700 1 -- 172.21.15.63:0/4217759151 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08c000ce60 con 0x7f08cc104ff0 2024-03-23T11:52:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.205+0000 7f08c9600700 1 -- 172.21.15.63:0/4217759151 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08c00113a0 con 0x7f08cc104ff0 2024-03-23T11:52:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.205+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc10fee0 con 0x7f08cc104ff0 2024-03-23T11:52:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.205+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc110450 con 0x7f08cc104ff0 2024-03-23T11:52:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.206+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc102300 con 0x7f08cc104ff0 2024-03-23T11:52:02.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.207+0000 7f08c9600700 1 -- 172.21.15.63:0/4217759151 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f08c0012400 con 0x7f08cc104ff0 2024-03-23T11:52:02.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.208+0000 7f08c9600700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f08b406c440 0x7f08b406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:02.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.208+0000 7f08c9600700 1 -- 172.21.15.63:0/4217759151 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f08c008b5f0 con 0x7f08cc104ff0 2024-03-23T11:52:02.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.208+0000 7f08d1000700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f08b406c440 0x7f08b406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:02.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.209+0000 7f08d1000700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f08b406c440 0x7f08b406e8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f08b8004f90 tx=0x7f08b8009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:02.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.213+0000 7f08c9600700 1 -- 172.21.15.63:0/4217759151 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f08c0059820 con 0x7f08cc104ff0 2024-03-23T11:52:02.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.427+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f08cc100740 con 0x7f08cc104ff0 2024-03-23T11:52:02.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.428+0000 7f08c9600700 1 -- 172.21.15.63:0/4217759151 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v47)=0 v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7f08c00593b0 con 0x7f08cc104ff0 2024-03-23T11:52:02.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f08b406c440 msgr2=0x7f08b406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f08b406c440 0x7f08b406e8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f08b8004f90 tx=0x7f08b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc104ff0 msgr2=0x7f08cc115a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc104ff0 0x7f08cc115a90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f08c000c8c0 tx=0x7f08c00044c0 comp rx=0 tx=0).stop 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 shutdown_connections 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f08b406c440 0x7f08b406e8f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08cc104ff0 0x7f08cc115a90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 --2- 172.21.15.63:0/4217759151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08cc105990 0x7f08cc115fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 >> 172.21.15.63:0/4217759151 conn(0x7f08cc0faba0 msgr2=0x7f08cc0fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 shutdown_connections 2024-03-23T11:52:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:02.430+0000 7f08d4400700 1 -- 172.21.15.63:0/4217759151 wait complete. 2024-03-23T11:52:02.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:02 smithi063 ceph-mon[35189]: pgmap v97: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:02.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:02 smithi111 ceph-mon[44005]: pgmap v97: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:03.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-23T11:52:03.601 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:52:04.532 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:04 smithi063 ceph-mon[35189]: pgmap v98: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:04.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:04 smithi111 ceph-mon[44005]: pgmap v98: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:05.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 -- 172.21.15.63:0/1703215671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 msgr2=0x7f8ebc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:05.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/1703215671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc1031d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8eac004940 tx=0x7f8eac009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:05.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 -- 172.21.15.63:0/1703215671 shutdown_connections 2024-03-23T11:52:05.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/1703215671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc1037a0 0x7f8ebc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:05.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/1703215671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc1031d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:05.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 -- 172.21.15.63:0/1703215671 >> 172.21.15.63:0/1703215671 conn(0x7f8ebc0f89b0 msgr2=0x7f8ebc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:05.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 -- 172.21.15.63:0/1703215671 shutdown_connections 2024-03-23T11:52:05.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.588+0000 7f8ec0a00700 1 -- 172.21.15.63:0/1703215671 wait complete. 2024-03-23T11:52:05.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.589+0000 7f8ec0a00700 1 Processor -- start 2024-03-23T11:52:05.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.590+0000 7f8ec0a00700 1 -- start start 2024-03-23T11:52:05.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.590+0000 7f8ec0a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:05.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.590+0000 7f8ec0a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc1037a0 0x7f8ebc116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:05.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.590+0000 7f8ec0a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc1168a0 con 0x7f8ebc102e00 2024-03-23T11:52:05.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.590+0000 7f8ec0a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc10fc70 con 0x7f8ebc1037a0 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34674/0 (socket says 172.21.15.63:34674) 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9e00700 1 -- 172.21.15.63:0/2882726370 learned_addr learned my addr 172.21.15.63:0/2882726370 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9400700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc1037a0 0x7f8ebc116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9e00700 1 -- 172.21.15.63:0/2882726370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc1037a0 msgr2=0x7f8ebc116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9e00700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc1037a0 0x7f8ebc116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9e00700 1 -- 172.21.15.63:0/2882726370 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac005040 con 0x7f8ebc102e00 2024-03-23T11:52:05.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb9e00700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc115d10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009390 tx=0x7f8eac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:05.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.591+0000 7f8eb2000700 1 -- 172.21.15.63:0/2882726370 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eac00e040 con 0x7f8ebc102e00 2024-03-23T11:52:05.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.592+0000 7f8eb2000700 1 -- 172.21.15.63:0/2882726370 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8eac013420 con 0x7f8ebc102e00 2024-03-23T11:52:05.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.592+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc10fef0 con 0x7f8ebc102e00 2024-03-23T11:52:05.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.592+0000 7f8eb2000700 1 -- 172.21.15.63:0/2882726370 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eac01c630 con 0x7f8ebc102e00 2024-03-23T11:52:05.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.592+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc1104b0 con 0x7f8ebc102e00 2024-03-23T11:52:05.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.593+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc0fe550 con 0x7f8ebc102e00 2024-03-23T11:52:05.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.594+0000 7f8eb2000700 1 -- 172.21.15.63:0/2882726370 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f8eac0139e0 con 0x7f8ebc102e00 2024-03-23T11:52:05.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.594+0000 7f8eb2000700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f8ea006c450 0x7f8ea006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:05.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.595+0000 7f8eb2000700 1 -- 172.21.15.63:0/2882726370 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f8eac08c7e0 con 0x7f8ebc102e00 2024-03-23T11:52:05.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.595+0000 7f8eb9400700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f8ea006c450 0x7f8ea006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:05.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.595+0000 7f8eb9400700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f8ea006c450 0x7f8ea006e900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc1115b0 tx=0x7f8ea8009ee0 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:05.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.599+0000 7f8eb2000700 1 -- 172.21.15.63:0/2882726370 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f8eac05aa20 con 0x7f8ebc102e00 2024-03-23T11:52:05.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.832+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ebc04bf40 con 0x7f8ea006c450 2024-03-23T11:52:05.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.840+0000 7f8eb2000700 1 -- 172.21.15.63:0/2882726370 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f8ebc04bf40 con 0x7f8ea006c450 2024-03-23T11:52:05.824 INFO:teuthology.orchestra.run.smithi063.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f8ea006c450 msgr2=0x7f8ea006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f8ea006c450 0x7f8ea006e900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc1115b0 tx=0x7f8ea8009ee0 comp rx=0 tx=0).stop 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 msgr2=0x7f8ebc115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc115d10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009390 tx=0x7f8eac009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 shutdown_connections 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f8ea006c450 0x7f8ea006e900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ebc102e00 0x7f8ebc115d10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 --2- 172.21.15.63:0/2882726370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc1037a0 0x7f8ebc116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 >> 172.21.15.63:0/2882726370 conn(0x7f8ebc0f89b0 msgr2=0x7f8ebc0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 shutdown_connections 2024-03-23T11:52:05.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:05.842+0000 7f8ec0a00700 1 -- 172.21.15.63:0/2882726370 wait complete. 2024-03-23T11:52:06.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:06 smithi063 ceph-mon[35189]: pgmap v99: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:06.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:06 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:06.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:06 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:06.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:06 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:06.559 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:06 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:52:06.560 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:06 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:06.582 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-23T11:52:06.583 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T11:52:06.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-23T11:52:06.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:06 smithi111 ceph-mon[44005]: pgmap v99: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:06.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:06 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:06.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:06 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:06.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:06 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:06.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:06 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:52:06.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:06 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:07.597 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:07 smithi063 ceph-mon[35189]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:07.597 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:07 smithi063 ceph-mon[35189]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:52:07.597 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:07 smithi063 ceph-mon[35189]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:52:07.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:07 smithi111 ceph-mon[44005]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:07.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:07 smithi111 ceph-mon[44005]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:52:07.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:07 smithi111 ceph-mon[44005]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:52:07.898 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T11:52:08.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: pgmap v100: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:08.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:08.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:08.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:08.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:52:08.505 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:08.506 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-23T11:52:08.506 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:08 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:08.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: pgmap v100: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:08.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:08.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:08.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:08.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:52:08.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:08.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-23T11:52:08.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:08 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:09.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:09 smithi063 ceph-mon[35189]: Upgrade: Target is version 18.2.2-820-gb630d6cd (reef) 2024-03-23T11:52:09.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:09 smithi063 ceph-mon[35189]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313'] 2024-03-23T11:52:09.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:09 smithi063 ceph-mon[35189]: Upgrade: Setting container_image for all mgr 2024-03-23T11:52:09.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:09 smithi111 ceph-mon[44005]: Upgrade: Target is version 18.2.2-820-gb630d6cd (reef) 2024-03-23T11:52:09.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:09 smithi111 ceph-mon[44005]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313'] 2024-03-23T11:52:09.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:09 smithi111 ceph-mon[44005]: Upgrade: Setting container_image for all mgr 2024-03-23T11:52:10.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:10 smithi063 ceph-mon[35189]: pgmap v101: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:10.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:10 smithi111 ceph-mon[44005]: pgmap v101: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:10.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 -- 172.21.15.63:0/826447656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 msgr2=0x7f14b0109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:10.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 --2- 172.21.15.63:0/826447656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b0109110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f149c004970 tx=0x7f149c009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:10.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 -- 172.21.15.63:0/826447656 shutdown_connections 2024-03-23T11:52:10.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 --2- 172.21.15.63:0/826447656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b0109110 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:10.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 --2- 172.21.15.63:0/826447656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b0070360 0x7f14b0070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:10.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 -- 172.21.15.63:0/826447656 >> 172.21.15.63:0/826447656 conn(0x7f14b00f89b0 msgr2=0x7f14b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:10.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 -- 172.21.15.63:0/826447656 shutdown_connections 2024-03-23T11:52:10.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.691+0000 7f14b5000700 1 -- 172.21.15.63:0/826447656 wait complete. 2024-03-23T11:52:10.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.692+0000 7f14b5000700 1 Processor -- start 2024-03-23T11:52:10.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.692+0000 7f14b5000700 1 -- start start 2024-03-23T11:52:10.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.692+0000 7f14b5000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b0070360 0x7f14b0115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:10.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.692+0000 7f14b5000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b01161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:10.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.692+0000 7f14b5000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b0116830 con 0x7f14b0070c70 2024-03-23T11:52:10.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.692+0000 7f14b5000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b010fc00 con 0x7f14b0070360 2024-03-23T11:52:10.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.692+0000 7f14ad400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b01161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ad400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b01161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34694/0 (socket says 172.21.15.63:34694) 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ad400700 1 -- 172.21.15.63:0/1885489229 learned_addr learned my addr 172.21.15.63:0/1885489229 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ade00700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b0070360 0x7f14b0115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ad400700 1 -- 172.21.15.63:0/1885489229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b0070360 msgr2=0x7f14b0115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ad400700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b0070360 0x7f14b0115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ad400700 1 -- 172.21.15.63:0/1885489229 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c005040 con 0x7f14b0070c70 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ade00700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b0070360 0x7f14b0115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:52:10.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14ad400700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b01161e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f149c009380 tx=0x7f149c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:10.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14a6000700 1 -- 172.21.15.63:0/1885489229 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f149c00e040 con 0x7f14b0070c70 2024-03-23T11:52:10.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14a6000700 1 -- 172.21.15.63:0/1885489229 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f149c013420 con 0x7f14b0070c70 2024-03-23T11:52:10.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14a6000700 1 -- 172.21.15.63:0/1885489229 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f149c01c630 con 0x7f14b0070c70 2024-03-23T11:52:10.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.693+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b010fe80 con 0x7f14b0070c70 2024-03-23T11:52:10.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.694+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b01103c0 con 0x7f14b0070c70 2024-03-23T11:52:10.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.695+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b0103bd0 con 0x7f14b0070c70 2024-03-23T11:52:10.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.696+0000 7f14a6000700 1 -- 172.21.15.63:0/1885489229 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f149c0139e0 con 0x7f14b0070c70 2024-03-23T11:52:10.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.697+0000 7f14a6000700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f149406c440 0x7f149406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:10.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.697+0000 7f14a6000700 1 -- 172.21.15.63:0/1885489229 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f149c08c660 con 0x7f14b0070c70 2024-03-23T11:52:10.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.697+0000 7f14ade00700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f149406c440 0x7f149406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:10.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.697+0000 7f14ade00700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f149406c440 0x7f149406e8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f14a0009fd0 tx=0x7f14a000b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:10.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.702+0000 7f14a6000700 1 -- 172.21.15.63:0/1885489229 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f149c05a930 con 0x7f14b0070c70 2024-03-23T11:52:10.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.934+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14b01112f0 con 0x7f149406c440 2024-03-23T11:52:10.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.936+0000 7f14a6000700 1 -- 172.21.15.63:0/1885489229 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f14b01112f0 con 0x7f149406c440 2024-03-23T11:52:10.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f149406c440 msgr2=0x7f149406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:10.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f149406c440 0x7f149406e8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f14a0009fd0 tx=0x7f14a000b040 comp rx=0 tx=0).stop 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 msgr2=0x7f14b01161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b01161e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f149c009380 tx=0x7f149c009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 shutdown_connections 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f149406c440 0x7f149406e8f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b0070360 0x7f14b0115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 --2- 172.21.15.63:0/1885489229 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14b0070c70 0x7f14b01161e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 >> 172.21.15.63:0/1885489229 conn(0x7f14b00f89b0 msgr2=0x7f14b01015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 shutdown_connections 2024-03-23T11:52:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:10.938+0000 7f14b5000700 1 -- 172.21.15.63:0/1885489229 wait complete. 2024-03-23T11:52:10.937 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.093+0000 7fdb2c000700 1 -- 172.21.15.63:0/1306232174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 msgr2=0x7fdb24070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.093+0000 7fdb2c000700 1 --2- 172.21.15.63:0/1306232174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24070730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdb18004970 tx=0x7fdb18009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.093+0000 7fdb2c000700 1 -- 172.21.15.63:0/1306232174 shutdown_connections 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.093+0000 7fdb2c000700 1 --2- 172.21.15.63:0/1306232174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb24070c70 0x7fdb24109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.093+0000 7fdb2c000700 1 --2- 172.21.15.63:0/1306232174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24070730 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.093+0000 7fdb2c000700 1 -- 172.21.15.63:0/1306232174 >> 172.21.15.63:0/1306232174 conn(0x7fdb240f89b0 msgr2=0x7fdb240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.093+0000 7fdb2c000700 1 -- 172.21.15.63:0/1306232174 shutdown_connections 2024-03-23T11:52:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.094+0000 7fdb2c000700 1 -- 172.21.15.63:0/1306232174 wait complete. 2024-03-23T11:52:11.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.094+0000 7fdb2c000700 1 Processor -- start 2024-03-23T11:52:11.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb2c000700 1 -- start start 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb2c000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb2c000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb24070c70 0x7fdb241161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb2c000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24116810 con 0x7fdb24070360 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb2c000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2410fbe0 con 0x7fdb24070c70 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb29600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb29600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34710/0 (socket says 172.21.15.63:34710) 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.095+0000 7fdb29600700 1 -- 172.21.15.63:0/350376511 learned_addr learned my addr 172.21.15.63:0/350376511 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:11.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.096+0000 7fdb28c00700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb24070c70 0x7fdb241161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.096+0000 7fdb29600700 1 -- 172.21.15.63:0/350376511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb24070c70 msgr2=0x7fdb241161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.096+0000 7fdb29600700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb24070c70 0x7fdb241161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.096+0000 7fdb29600700 1 -- 172.21.15.63:0/350376511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb18005040 con 0x7fdb24070360 2024-03-23T11:52:11.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.096+0000 7fdb29600700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24115c80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdb18004f50 tx=0x7fdb18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:11.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.096+0000 7fdb21600700 1 -- 172.21.15.63:0/350376511 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb1800e040 con 0x7fdb24070360 2024-03-23T11:52:11.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.097+0000 7fdb21600700 1 -- 172.21.15.63:0/350376511 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb18013420 con 0x7fdb24070360 2024-03-23T11:52:11.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.097+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2410fe60 con 0x7fdb24070360 2024-03-23T11:52:11.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.097+0000 7fdb21600700 1 -- 172.21.15.63:0/350376511 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb1801c610 con 0x7fdb24070360 2024-03-23T11:52:11.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.097+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb241103a0 con 0x7fdb24070360 2024-03-23T11:52:11.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.098+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb24103b90 con 0x7fdb24070360 2024-03-23T11:52:11.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.099+0000 7fdb21600700 1 -- 172.21.15.63:0/350376511 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fdb18013590 con 0x7fdb24070360 2024-03-23T11:52:11.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.100+0000 7fdb21600700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdb0c06c450 0x7fdb0c06e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.100+0000 7fdb21600700 1 -- 172.21.15.63:0/350376511 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdb1808c790 con 0x7fdb24070360 2024-03-23T11:52:11.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.100+0000 7fdb28c00700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdb0c06c450 0x7fdb0c06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.101+0000 7fdb28c00700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdb0c06c450 0x7fdb0c06e900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdb24111520 tx=0x7fdb10009ee0 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:11.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.105+0000 7fdb21600700 1 -- 172.21.15.63:0/350376511 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdb1805a8e0 con 0x7fdb24070360 2024-03-23T11:52:11.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.344+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb2410bfb0 con 0x7fdb0c06c450 2024-03-23T11:52:11.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.346+0000 7fdb21600700 1 -- 172.21.15.63:0/350376511 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fdb2410bfb0 con 0x7fdb0c06c450 2024-03-23T11:52:11.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdb0c06c450 msgr2=0x7fdb0c06e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdb0c06c450 0x7fdb0c06e900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdb24111520 tx=0x7fdb10009ee0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 msgr2=0x7fdb24115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24115c80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdb18004f50 tx=0x7fdb18009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:11.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 shutdown_connections 2024-03-23T11:52:11.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fdb0c06c450 0x7fdb0c06e900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb24070360 0x7fdb24115c80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 --2- 172.21.15.63:0/350376511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb24070c70 0x7fdb241161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 >> 172.21.15.63:0/350376511 conn(0x7fdb240f89b0 msgr2=0x7fdb241015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:11.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 shutdown_connections 2024-03-23T11:52:11.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.349+0000 7fdb2c000700 1 -- 172.21.15.63:0/350376511 wait complete. 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.491+0000 7ff3caa00700 1 -- 172.21.15.63:0/1813329811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 msgr2=0x7ff3c4070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.491+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1813329811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c4070360 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4004970 tx=0x7ff3b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.491+0000 7ff3caa00700 1 -- 172.21.15.63:0/1813329811 shutdown_connections 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.491+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1813329811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c40708a0 0x7ff3c4070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.491+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1813329811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c4070360 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.491+0000 7ff3caa00700 1 -- 172.21.15.63:0/1813329811 >> 172.21.15.63:0/1813329811 conn(0x7ff3c40f89b0 msgr2=0x7ff3c40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.492+0000 7ff3caa00700 1 -- 172.21.15.63:0/1813329811 shutdown_connections 2024-03-23T11:52:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.492+0000 7ff3caa00700 1 -- 172.21.15.63:0/1813329811 wait complete. 2024-03-23T11:52:11.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.493+0000 7ff3caa00700 1 Processor -- start 2024-03-23T11:52:11.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.493+0000 7ff3caa00700 1 -- start start 2024-03-23T11:52:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.493+0000 7ff3caa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c40708a0 0x7ff3c411a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3caa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c411a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3caa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c411ac50 con 0x7ff3c4071f90 2024-03-23T11:52:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3caa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4114020 con 0x7ff3c40708a0 2024-03-23T11:52:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3c3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c411a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3c3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c411a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47752/0 (socket says 172.21.15.63:47752) 2024-03-23T11:52:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3c3400700 1 -- 172.21.15.63:0/1663988801 learned_addr learned my addr 172.21.15.63:0/1663988801 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:11.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3c3e00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c40708a0 0x7ff3c411a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3c3400700 1 -- 172.21.15.63:0/1663988801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c40708a0 msgr2=0x7ff3c411a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3c3400700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c40708a0 0x7ff3c411a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.494+0000 7ff3c3400700 1 -- 172.21.15.63:0/1663988801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4005040 con 0x7ff3c4071f90 2024-03-23T11:52:11.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.495+0000 7ff3c3400700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c411a600 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0002a50 tx=0x7ff3b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:11.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.495+0000 7ff3bbe00700 1 -- 172.21.15.63:0/1663988801 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b0007cc0 con 0x7ff3c4071f90 2024-03-23T11:52:11.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.495+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c4114300 con 0x7ff3c4071f90 2024-03-23T11:52:11.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.495+0000 7ff3bbe00700 1 -- 172.21.15.63:0/1663988801 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3b000f040 con 0x7ff3c4071f90 2024-03-23T11:52:11.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.495+0000 7ff3bbe00700 1 -- 172.21.15.63:0/1663988801 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b000d9c0 con 0x7ff3c4071f90 2024-03-23T11:52:11.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.495+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c4114840 con 0x7ff3c4071f90 2024-03-23T11:52:11.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.496+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4071aa0 con 0x7ff3c4071f90 2024-03-23T11:52:11.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.497+0000 7ff3bbe00700 1 -- 172.21.15.63:0/1663988801 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7ff3b000db20 con 0x7ff3c4071f90 2024-03-23T11:52:11.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.498+0000 7ff3bbe00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff3a806c440 0x7ff3a806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.498+0000 7ff3bbe00700 1 -- 172.21.15.63:0/1663988801 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff3b008b200 con 0x7ff3c4071f90 2024-03-23T11:52:11.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.498+0000 7ff3c3e00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff3a806c440 0x7ff3a806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.499+0000 7ff3c3e00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff3a806c440 0x7ff3a806e8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4004f50 tx=0x7ff3b4009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:11.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.503+0000 7ff3bbe00700 1 -- 172.21.15.63:0/1663988801 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff3b00593b0 con 0x7ff3c4071f90 2024-03-23T11:52:11.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.734+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3c4110450 con 0x7ff3a806c440 2024-03-23T11:52:11.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.745+0000 7ff3bbe00700 1 -- 172.21.15.63:0/1663988801 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff3c4110450 con 0x7ff3a806c440 2024-03-23T11:52:11.729 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:52:11.729 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (90s) 64s ago 14m 14.1M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:52:11.729 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (14m) 64s ago 14m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:52:11.729 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (13m) 2m ago 13m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (70s) 64s ago 14m 44.1M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (8m) 64s ago 8m 119M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (8m) 64s ago 8m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (8m) 2m ago 8m 65.0M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (8m) 2m ago 8m 59.8M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (3m) 64s ago 16m 498M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (2m) 2m ago 13m 446M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (16m) 64s ago 16m 55.1M 2048M 17.2.7-726-g9c4e0c34 837124da804f 16d320c90a94 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (13m) 2m ago 13m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (2m) 64s ago 14m 7390k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (2m) 2m ago 13m 2558k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (12m) 64s ago 12m 220M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (11m) 64s ago 11m 232M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:52:11.730 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (11m) 64s ago 11m 195M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:52:11.731 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (10m) 2m ago 10m 256M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:52:11.731 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (10m) 2m ago 10m 230M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:52:11.731 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (9m) 2m ago 9m 225M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:52:11.731 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (111s) 64s ago 14m 41.4M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.747+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff3a806c440 msgr2=0x7ff3a806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.747+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff3a806c440 0x7ff3a806e8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4004f50 tx=0x7ff3b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.747+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 msgr2=0x7ff3c411a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.747+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c411a600 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0002a50 tx=0x7ff3b00046b0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.748+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 shutdown_connections 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.748+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff3a806c440 0x7ff3a806e8f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.748+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c40708a0 0x7ff3c411a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.748+0000 7ff3caa00700 1 --2- 172.21.15.63:0/1663988801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3c4071f90 0x7ff3c411a600 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.748+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 >> 172.21.15.63:0/1663988801 conn(0x7ff3c40f89b0 msgr2=0x7ff3c40f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:11.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.748+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 shutdown_connections 2024-03-23T11:52:11.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.748+0000 7ff3caa00700 1 -- 172.21.15.63:0/1663988801 wait complete. 2024-03-23T11:52:11.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 -- 172.21.15.63:0/603071025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c105010 msgr2=0x7fd02c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 --2- 172.21.15.63:0/603071025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c105010 0x7fd02c1053e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd01c004940 tx=0x7fd01c009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:11.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 -- 172.21.15.63:0/603071025 shutdown_connections 2024-03-23T11:52:11.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 --2- 172.21.15.63:0/603071025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd02c1059b0 0x7fd02c0ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 --2- 172.21.15.63:0/603071025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c105010 0x7fd02c1053e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 -- 172.21.15.63:0/603071025 >> 172.21.15.63:0/603071025 conn(0x7fd02c0faba0 msgr2=0x7fd02c0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:11.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 -- 172.21.15.63:0/603071025 shutdown_connections 2024-03-23T11:52:11.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.889+0000 7fd033200700 1 -- 172.21.15.63:0/603071025 wait complete. 2024-03-23T11:52:11.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.890+0000 7fd033200700 1 Processor -- start 2024-03-23T11:52:11.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.890+0000 7fd033200700 1 -- start start 2024-03-23T11:52:11.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd033200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd02c105010 0x7fd02c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd033200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c1059b0 0x7fd02c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd033200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c116830 con 0x7fd02c1059b0 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd033200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c10fc00 con 0x7fd02c105010 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd02b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c1059b0 0x7fd02c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd02b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c1059b0 0x7fd02c1161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47780/0 (socket says 172.21.15.63:47780) 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd02b400700 1 -- 172.21.15.63:0/2997905678 learned_addr learned my addr 172.21.15.63:0/2997905678 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd02be00700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd02c105010 0x7fd02c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd02b400700 1 -- 172.21.15.63:0/2997905678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd02c105010 msgr2=0x7fd02c115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd02b400700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd02c105010 0x7fd02c115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.891+0000 7fd02b400700 1 -- 172.21.15.63:0/2997905678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c005040 con 0x7fd02c1059b0 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.892+0000 7fd02b400700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c1059b0 0x7fd02c1161e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd018002a50 tx=0x7fd018004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.892+0000 7fd023e00700 1 -- 172.21.15.63:0/2997905678 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd018007cc0 con 0x7fd02c1059b0 2024-03-23T11:52:11.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.892+0000 7fd023e00700 1 -- 172.21.15.63:0/2997905678 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd01800f040 con 0x7fd02c1059b0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.892+0000 7fd023e00700 1 -- 172.21.15.63:0/2997905678 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd01800d9a0 con 0x7fd02c1059b0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.892+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c10fee0 con 0x7fd02c1059b0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.892+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c1104a0 con 0x7fd02c1059b0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.893+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c100760 con 0x7fd02c1059b0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.894+0000 7fd023e00700 1 -- 172.21.15.63:0/2997905678 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7fd01800dc10 con 0x7fd02c1059b0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.895+0000 7fd023e00700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd01006c450 0x7fd01006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.895+0000 7fd023e00700 1 -- 172.21.15.63:0/2997905678 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd01808b2f0 con 0x7fd02c1059b0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.895+0000 7fd02be00700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd01006c450 0x7fd01006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:11.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.896+0000 7fd02be00700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd01006c450 0x7fd01006e900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00b3b0 tx=0x7fd01c009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:11.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:11.900+0000 7fd023e00700 1 -- 172.21.15.63:0/2997905678 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fd018059480 con 0x7fd02c1059b0 2024-03-23T11:52:12.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.206+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd02c04bed0 con 0x7fd02c1059b0 2024-03-23T11:52:12.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.207+0000 7fd023e00700 1 -- 172.21.15.63:0/2997905678 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7fd018059010 con 0x7fd02c1059b0 2024-03-23T11:52:12.191 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:52:12.191 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:52:12.191 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-23T11:52:12.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:52:12.192 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.209+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd01006c450 msgr2=0x7fd01006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.209+0000 7fd033200700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd01006c450 0x7fd01006e900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00b3b0 tx=0x7fd01c009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.209+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c1059b0 msgr2=0x7fd02c1161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.209+0000 7fd033200700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c1059b0 0x7fd02c1161e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd018002a50 tx=0x7fd018004600 comp rx=0 tx=0).stop 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.209+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 shutdown_connections 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.210+0000 7fd033200700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7fd01006c450 0x7fd01006e900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.210+0000 7fd033200700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd02c105010 0x7fd02c115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.210+0000 7fd033200700 1 --2- 172.21.15.63:0/2997905678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd02c1059b0 0x7fd02c1161e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.210+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 >> 172.21.15.63:0/2997905678 conn(0x7fd02c0faba0 msgr2=0x7fd02c0fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.210+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 shutdown_connections 2024-03-23T11:52:12.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.210+0000 7fd033200700 1 -- 172.21.15.63:0/2997905678 wait complete. 2024-03-23T11:52:12.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 -- 172.21.15.63:0/1616899355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 msgr2=0x7ff508109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 --2- 172.21.15.63:0/1616899355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff508109110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0004970 tx=0x7ff4f0009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:12.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 -- 172.21.15.63:0/1616899355 shutdown_connections 2024-03-23T11:52:12.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 --2- 172.21.15.63:0/1616899355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff508109110 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 --2- 172.21.15.63:0/1616899355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508070360 0x7ff508070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 -- 172.21.15.63:0/1616899355 >> 172.21.15.63:0/1616899355 conn(0x7ff5080f89b0 msgr2=0x7ff5080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:12.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 -- 172.21.15.63:0/1616899355 shutdown_connections 2024-03-23T11:52:12.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.349+0000 7ff50f000700 1 -- 172.21.15.63:0/1616899355 wait complete. 2024-03-23T11:52:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.350+0000 7ff50f000700 1 Processor -- start 2024-03-23T11:52:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.350+0000 7ff50f000700 1 -- start start 2024-03-23T11:52:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.350+0000 7ff50f000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508070360 0x7ff508115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.350+0000 7ff50f000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff5081161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.350+0000 7ff50f000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff508116830 con 0x7ff508070c70 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.350+0000 7ff50f000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50810fc00 con 0x7ff508070360 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff4ffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff5081161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff4ffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff5081161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47796/0 (socket says 172.21.15.63:47796) 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff4ffe00700 1 -- 172.21.15.63:0/673643714 learned_addr learned my addr 172.21.15.63:0/673643714 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff507e00700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508070360 0x7ff508115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff4ffe00700 1 -- 172.21.15.63:0/673643714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508070360 msgr2=0x7ff508115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff4ffe00700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508070360 0x7ff508115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff4ffe00700 1 -- 172.21.15.63:0/673643714 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0005040 con 0x7ff508070c70 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff507e00700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508070360 0x7ff508115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.351+0000 7ff4ffe00700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff5081161e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0004f50 tx=0x7ff4f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.352+0000 7ff504c00700 1 -- 172.21.15.63:0/673643714 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4f000e040 con 0x7ff508070c70 2024-03-23T11:52:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.352+0000 7ff504c00700 1 -- 172.21.15.63:0/673643714 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4f0013420 con 0x7ff508070c70 2024-03-23T11:52:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.352+0000 7ff504c00700 1 -- 172.21.15.63:0/673643714 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4f001c610 con 0x7ff508070c70 2024-03-23T11:52:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.352+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50810fe80 con 0x7ff508070c70 2024-03-23T11:52:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.352+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5081103c0 con 0x7ff508070c70 2024-03-23T11:52:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.353+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff508103bd0 con 0x7ff508070c70 2024-03-23T11:52:12.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.354+0000 7ff504c00700 1 -- 172.21.15.63:0/673643714 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7ff4f0013590 con 0x7ff508070c70 2024-03-23T11:52:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.355+0000 7ff504c00700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff4ec06c450 0x7ff4ec06e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.355+0000 7ff504c00700 1 -- 172.21.15.63:0/673643714 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff4f008c720 con 0x7ff508070c70 2024-03-23T11:52:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.355+0000 7ff507e00700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff4ec06c450 0x7ff4ec06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.356+0000 7ff507e00700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff4ec06c450 0x7ff4ec06e900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009fd0 tx=0x7ff4f800b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:12.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.359+0000 7ff504c00700 1 -- 172.21.15.63:0/673643714 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff4f005a8b0 con 0x7ff508070c70 2024-03-23T11:52:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: Upgrade: Updating mon.smithi063 2024-03-23T11:52:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: pgmap v102: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:52:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:52:12.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:12.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: Deploying daemon mon.smithi063 on smithi063 2024-03-23T11:52:12.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:12 smithi063 ceph-mon[35189]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: Upgrade: Updating mon.smithi063 2024-03-23T11:52:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: pgmap v102: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:12.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:52:12.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:52:12.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: from='mgr.14684 172.21.15.63:0/486235938' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:12.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: Deploying daemon mon.smithi063 on smithi063 2024-03-23T11:52:12.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:12 smithi111 ceph-mon[44005]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:12.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.620+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff5081112f0 con 0x7ff508070c70 2024-03-23T11:52:12.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.621+0000 7ff504c00700 1 -- 172.21.15.63:0/673643714 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1832 (secure 0 0 0) 0x7ff4f005a440 con 0x7ff508070c70 2024-03-23T11:52:12.605 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:52:12.605 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:52:12.605 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:12.605 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:52:12.605 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:52:12.605 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:52:12.606 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:52:12.607 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:12.608 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:12.608 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:12.608 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:12.608 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:52:12.608 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff4ec06c450 msgr2=0x7ff4ec06e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff4ec06c450 0x7ff4ec06e900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009fd0 tx=0x7ff4f800b040 comp rx=0 tx=0).stop 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 msgr2=0x7ff5081161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff5081161e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0004f50 tx=0x7ff4f0009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 shutdown_connections 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7ff4ec06c450 0x7ff4ec06e900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508070360 0x7ff508115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 --2- 172.21.15.63:0/673643714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff508070c70 0x7ff5081161e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 >> 172.21.15.63:0/673643714 conn(0x7ff5080f89b0 msgr2=0x7ff5081015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.623+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 shutdown_connections 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.624+0000 7ff50f000700 1 -- 172.21.15.63:0/673643714 wait complete. 2024-03-23T11:52:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:52:12.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.786+0000 7f28e3e00700 1 -- 172.21.15.63:0/773842659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28dc1098a0 msgr2=0x7f28dc06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.786+0000 7f28e3e00700 1 --2- 172.21.15.63:0/773842659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28dc1098a0 0x7f28dc06e650 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f28d0004940 tx=0x7f28d0009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:12.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.787+0000 7f28e3e00700 1 -- 172.21.15.63:0/773842659 shutdown_connections 2024-03-23T11:52:12.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.787+0000 7f28e3e00700 1 --2- 172.21.15.63:0/773842659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 0x7f28dc06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.787+0000 7f28e3e00700 1 --2- 172.21.15.63:0/773842659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28dc1098a0 0x7f28dc06e650 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.787+0000 7f28e3e00700 1 -- 172.21.15.63:0/773842659 >> 172.21.15.63:0/773842659 conn(0x7f28dc0690e0 msgr2=0x7f28dc0694e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:12.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.787+0000 7f28e3e00700 1 -- 172.21.15.63:0/773842659 shutdown_connections 2024-03-23T11:52:12.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.787+0000 7f28e3e00700 1 -- 172.21.15.63:0/773842659 wait complete. 2024-03-23T11:52:12.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.788+0000 7f28e3e00700 1 Processor -- start 2024-03-23T11:52:12.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.791+0000 7f28e3e00700 1 -- start start 2024-03-23T11:52:12.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.791+0000 7f28e3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28dc1098a0 0x7f28dc11c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:12.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.792+0000 7f28e3e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 0x7f28dc11c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:12.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.792+0000 7f28e3e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc11cbe0 con 0x7f28dc1098a0 2024-03-23T11:52:12.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.792+0000 7f28e3e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc1161c0 con 0x7f28dc109d00 2024-03-23T11:52:12.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.793+0000 7f28e0a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 0x7f28dc11c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:12.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.793+0000 7f28e0a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 0x7f28dc11c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33636/0 (socket says 172.21.15.63:33636) 2024-03-23T11:52:12.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.793+0000 7f28e0a00700 1 -- 172.21.15.63:0/3992359033 learned_addr learned my addr 172.21.15.63:0/3992359033 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:12.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.793+0000 7f28e0a00700 1 -- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28dc1098a0 msgr2=0x7f28dc11c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:12.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.793+0000 7f28e0a00700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28dc1098a0 0x7f28dc11c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:12.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.793+0000 7f28e0a00700 1 -- 172.21.15.63:0/3992359033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0005040 con 0x7f28dc109d00 2024-03-23T11:52:12.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.794+0000 7f28e0a00700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 0x7f28dc11c590 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f28c8004ee0 tx=0x7f28c80046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:12.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.794+0000 7f28d9600700 1 -- 172.21.15.63:0/3992359033 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28c8007ea0 con 0x7f28dc109d00 2024-03-23T11:52:12.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.794+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc116440 con 0x7f28dc109d00 2024-03-23T11:52:12.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.794+0000 7f28d9600700 1 -- 172.21.15.63:0/3992359033 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28c8009580 con 0x7f28dc109d00 2024-03-23T11:52:12.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.794+0000 7f28d9600700 1 -- 172.21.15.63:0/3992359033 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28c800dd90 con 0x7f28dc109d00 2024-03-23T11:52:12.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.795+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc116a00 con 0x7f28dc109d00 2024-03-23T11:52:12.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.797+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc06fd60 con 0x7f28dc109d00 2024-03-23T11:52:12.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.798+0000 7f28d9600700 1 -- 172.21.15.63:0/3992359033 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f28c8018070 con 0x7f28dc109d00 2024-03-23T11:52:12.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.799+0000 7f28d9600700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f28c406c440 0x7f28c406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:12.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.799+0000 7f28d9600700 1 -- 172.21.15.63:0/3992359033 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f28c808f510 con 0x7f28dc109d00 2024-03-23T11:52:12.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.799+0000 7f28e1400700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f28c406c440 0x7f28c406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:12.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.800+0000 7f28e1400700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f28c406c440 0x7f28c406e8f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28d0004f90 tx=0x7f28d0009e30 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:12.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:12.804+0000 7f28d9600700 1 -- 172.21.15.63:0/3992359033 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f28c805d6a0 con 0x7f28dc109d00 2024-03-23T11:52:13.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.041+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 --> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28dc10f790 con 0x7f28c406c440 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.043+0000 7f28d9600700 1 -- 172.21.15.63:0/3992359033 <== mgr.14684 v2:172.21.15.63:6800/335624352 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f28dc10f790 con 0x7f28c406c440 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:52:13.027 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "2/21 daemons upgraded", 2024-03-23T11:52:13.028 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mon daemons", 2024-03-23T11:52:13.028 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:52:13.028 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:52:13.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f28c406c440 msgr2=0x7f28c406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:13.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f28c406c440 0x7f28c406e8f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28d0004f90 tx=0x7f28d0009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:13.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 msgr2=0x7f28dc11c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 0x7f28dc11c590 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f28c8004ee0 tx=0x7f28c80046b0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 shutdown_connections 2024-03-23T11:52:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f28c406c440 0x7f28c406e8f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28dc1098a0 0x7f28dc11c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 --2- 172.21.15.63:0/3992359033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28dc109d00 0x7f28dc11c590 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.045+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 >> 172.21.15.63:0/3992359033 conn(0x7f28dc0690e0 msgr2=0x7f28dc10c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:13.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.046+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 shutdown_connections 2024-03-23T11:52:13.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.046+0000 7f28e3e00700 1 -- 172.21.15.63:0/3992359033 wait complete. 2024-03-23T11:52:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 -- 172.21.15.63:0/3868855613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 msgr2=0x7f270c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:13.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 --2- 172.21.15.63:0/3868855613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c0fce10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f26f8004940 tx=0x7f26f8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 -- 172.21.15.63:0/3868855613 shutdown_connections 2024-03-23T11:52:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 --2- 172.21.15.63:0/3868855613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c0fce10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 --2- 172.21.15.63:0/3868855613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f270c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 -- 172.21.15.63:0/3868855613 >> 172.21.15.63:0/3868855613 conn(0x7f270c0f89b0 msgr2=0x7f270c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 -- 172.21.15.63:0/3868855613 shutdown_connections 2024-03-23T11:52:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.191+0000 7f2713800700 1 -- 172.21.15.63:0/3868855613 wait complete. 2024-03-23T11:52:13.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.192+0000 7f2713800700 1 Processor -- start 2024-03-23T11:52:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.192+0000 7f2713800700 1 -- start start 2024-03-23T11:52:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f2713800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f270c10af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f2713800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c10b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:13.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f2713800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c10f0b0 con 0x7f270c1037a0 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f2713800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c10b9a0 con 0x7f270c102e00 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f270be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c10b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f270be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c10b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47836/0 (socket says 172.21.15.63:47836) 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f270be00700 1 -- 172.21.15.63:0/333297678 learned_addr learned my addr 172.21.15.63:0/333297678 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.193+0000 7f2710e00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f270c10af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f270be00700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 msgr2=0x7f270c10af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f270be00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f270c10af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f270be00700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f8005040 con 0x7f270c1037a0 2024-03-23T11:52:13.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f270be00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c10b460 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009390 tx=0x7f26f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:13.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f800e040 con 0x7f270c1037a0 2024-03-23T11:52:13.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26f8013420 con 0x7f270c1037a0 2024-03-23T11:52:13.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c10bc20 con 0x7f270c1037a0 2024-03-23T11:52:13.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f801c630 con 0x7f270c1037a0 2024-03-23T11:52:13.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.194+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c11e660 con 0x7f270c1037a0 2024-03-23T11:52:13.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.196+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c0fe550 con 0x7f270c1037a0 2024-03-23T11:52:13.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.197+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f26f8013a00 con 0x7f270c1037a0 2024-03-23T11:52:13.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.198+0000 7f2708c00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f26f406c450 0x7f26f406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:13.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.198+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f26f808c670 con 0x7f270c1037a0 2024-03-23T11:52:13.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.198+0000 7f2710e00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f26f406c450 0x7f26f406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:13.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.199+0000 7f2710e00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f26f406c450 0x7f26f406e900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2700004f40 tx=0x7f270000b040 comp rx=0 tx=0).ready entity=mgr.14684 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:13.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.203+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f26f8056d70 con 0x7f270c1037a0 2024-03-23T11:52:13.309 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:13 smithi063 systemd[1]: Stopping Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:52:13.309 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:13 smithi063 ceph-mon[35189]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:13.309 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:13 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/2997905678' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:52:13.309 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:13 smithi063 ceph-mon[35189]: from='client.? 172.21.15.63:0/673643714' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f270be00700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 msgr2=0x7f270c10b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f270be00700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 msgr2=0x7f270c10b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f270be00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c10b460 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009390 tx=0x7f26f8009e30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f270be00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c10b460 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009390 tx=0x7f26f8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 msgr2=0x7f270c10b460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f2708c00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f270c1037a0 0x7f270c10b460 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f2708c00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f40740a0 0x7f26f4076480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f2708c00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f26f4078cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4079650 con 0x7f26f40740a0 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.325+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4079790 con 0x7f270c102e00 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.326+0000 7f270be00700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f40740a0 msgr2=0x7f26f4076480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3300/0 2024-03-23T11:52:13.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.326+0000 7f270be00700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f40740a0 0x7f26f4076480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:52:13.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.326+0000 7f2711800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f26f4078cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:13.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.328+0000 7f2711800700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f40740a0 msgr2=0x7f26f4076480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:52:13.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.328+0000 7f2711800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f40740a0 0x7f26f4076480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.328+0000 7f2711800700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7f26f4079930 con 0x7f270c102e00 2024-03-23T11:52:13.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.330+0000 7f2711800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f26f4078cd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2704009f20 tx=0x7f27040095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:13.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.331+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f270400be20 con 0x7f270c102e00 2024-03-23T11:52:13.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.332+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27040148c0 con 0x7f270c102e00 2024-03-23T11:52:13.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.334+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mgrmap(e 33) v1 ==== 90385+0+0 (secure 0 0 0) 0x7f2704014410 con 0x7f270c102e00 2024-03-23T11:52:13.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:13 smithi111 ceph-mon[44005]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:13.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:13 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/2997905678' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:52:13.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:13 smithi111 ceph-mon[44005]: from='client.? 172.21.15.63:0/673643714' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:52:13.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.517+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f270c04bed0 con 0x7f270c102e00 2024-03-23T11:52:13.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.518+0000 7f2708c00700 1 -- 172.21.15.63:0/333297678 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f270405b6c0 con 0x7f270c102e00 2024-03-23T11:52:13.502 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T11:52:13.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f26f406c450 msgr2=0x7f26f406e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:13.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f26f406c450 0x7f26f406e900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2700004f40 tx=0x7f270000b040 comp rx=0 tx=0).stop 2024-03-23T11:52:13.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 msgr2=0x7f26f4078cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:13.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f26f4078cd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2704009f20 tx=0x7f27040095d0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 shutdown_connections 2024-03-23T11:52:13.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f270c102e00 0x7f26f4078cd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f40740a0 0x7f26f4076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 --2- 172.21.15.63:0/333297678 >> [v2:172.21.15.63:6800/335624352,v1:172.21.15.63:6801/335624352] conn(0x7f26f406c450 0x7f26f406e900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:13.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 >> 172.21.15.63:0/333297678 conn(0x7f270c0f89b0 msgr2=0x7f270c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:13.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 shutdown_connections 2024-03-23T11:52:13.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:13.520+0000 7f2713800700 1 -- 172.21.15.63:0/333297678 wait complete. 2024-03-23T11:52:13.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:13 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[35166]: 2024-03-23T11:52:13.324+0000 7f6c5ec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi063 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:52:13.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:13 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[35166]: 2024-03-23T11:52:13.324+0000 7f6c5ec00700 -1 mon.smithi063@0(leader) e2 *** Got Signal Terminated *** 2024-03-23T11:52:14.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:13 smithi063 bash[140913]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063 2024-03-23T11:52:14.570 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:14 smithi063 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063.service: Succeeded. 2024-03-23T11:52:14.571 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:14 smithi063 systemd[1]: Stopped Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:52:14.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:14 smithi063 systemd[1]: Starting Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:52:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: set uid:gid to 167:167 (ceph:ceph) 2024-03-23T11:52:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable), process ceph-mon, pid 7 2024-03-23T11:52:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: pidfile_write: ignore empty --pid-file 2024-03-23T11:52:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: load: jerasure load: lrc 2024-03-23T11:52:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: RocksDB version: 7.9.2 2024-03-23T11:52:15.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Git sha 0 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Compile date 2024-03-23 00:21:31 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: DB SUMMARY 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: DB Session ID: MG209K9XCXXZQZVAETII 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: CURRENT file: CURRENT 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: IDENTITY file: IDENTITY 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1147 Bytes 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi063/store.db dir, Total Num: 1, files: 000033.sst 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi063/store.db: 000031.log size: 3731979 ; 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.error_if_exists: 0 2024-03-23T11:52:15.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.create_if_missing: 0 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.paranoid_checks: 1 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.env: 0x55d702286720 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.fs: PosixFileSystem 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.info_log: 0x55d704fab3c0 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_file_opening_threads: 16 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.statistics: (nil) 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.use_fsync: 0 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_log_file_size: 0 2024-03-23T11:52:15.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.keep_log_file_num: 1000 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.recycle_log_file_num: 0 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.allow_fallocate: 1 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.allow_mmap_reads: 0 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.allow_mmap_writes: 0 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.use_direct_reads: 0 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.create_missing_column_families: 0 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.db_log_dir: 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.wal_dir: 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-23T11:52:15.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.advise_random_on_open: 1 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.db_write_buffer_size: 0 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.write_buffer_manager: 0x55d704224140 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.rate_limiter: (nil) 2024-03-23T11:52:15.637 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.wal_recovery_mode: 2 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.enable_thread_tracking: 0 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.enable_pipelined_write: 0 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.unordered_write: 0 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.row_cache: None 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.wal_filter: None 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-23T11:52:15.638 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.allow_ingest_behind: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.two_write_queues: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.manual_wal_flush: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.wal_compression: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.atomic_flush: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.log_readahead_size: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.best_efforts_recovery: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.allow_data_in_errors: 0 2024-03-23T11:52:15.639 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.db_host_id: __hostname__ 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_background_jobs: 2 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_background_compactions: -1 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_subcompactions: 1 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-23T11:52:15.641 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_total_wal_size: 0 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_open_files: -1 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bytes_per_sync: 0 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_readahead_size: 0 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_background_flushes: -1 2024-03-23T11:52:15.642 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Compression algorithms supported: 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kZSTD supported: 0 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kXpressCompression supported: 0 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kLZ4HCCompression supported: 1 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kZlibCompression supported: 1 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kSnappyCompression supported: 1 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kLZ4Compression supported: 1 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: kBZip2Compression supported: 0 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi063/store.db/MANIFEST-000015 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-23T11:52:15.643 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.merge_operator: 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_filter: None 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_filter_factory: None 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.sst_partitioner_factory: None 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d704fab4e0) 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: cache_index_and_filter_blocks: 1 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: pin_top_level_index_and_filter: 1 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_type: 0 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: data_block_index_type: 0 2024-03-23T11:52:15.644 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_shortening: 1 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: checksum: 4 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: no_block_cache: 0 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache: 0x55d7042c5350 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_name: BinnedLRUCache 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_options: 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: capacity : 536870912 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: num_shard_bits : 4 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: strict_capacity_limit : 0 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: high_pri_pool_ratio: 0.000 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_compressed: (nil) 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: persistent_cache: (nil) 2024-03-23T11:52:15.645 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_size: 4096 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_size_deviation: 10 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_restart_interval: 16 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_block_restart_interval: 1 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: metadata_block_size: 4096 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: partition_filters: 0 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: use_delta_encoding: 1 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: filter_policy: bloomfilter 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: whole_key_filtering: 1 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: verify_compression: 0 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: read_amp_bytes_per_bit: 0 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: format_version: 5 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: enable_index_compression: 1 2024-03-23T11:52:15.646 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_align: 0 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: max_auto_readahead_size: 262144 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: prepopulate_block_cache: 0 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: initial_auto_readahead_size: 8192 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: num_file_reads_for_auto_readahead: 2 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.write_buffer_size: 33554432 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_write_buffer_number: 2 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression: NoCompression 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression: Disabled 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.prefix_extractor: nullptr 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.num_levels: 7 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-23T11:52:15.647 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-23T11:52:15.648 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.level: 32767 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.strategy: 0 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.enabled: false 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.target_file_size_base: 67108864 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-23T11:52:15.649 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-23T11:52:15.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.arena_block_size: 1048576 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.disable_auto_compactions: 0 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-23T11:52:15.651 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.table_properties_collectors: 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.inplace_update_support: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.bloom_locality: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.max_successive_merges: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.paranoid_file_checks: 0 2024-03-23T11:52:15.652 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.force_consistency_checks: 1 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.report_bg_io_stats: 0 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.ttl: 2592000 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.enable_blob_files: false 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.min_blob_size: 0 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.blob_file_size: 268435456 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-23T11:52:15.653 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.blob_file_starting_level: 0 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi063/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11886, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 79d1dda2-776a-437f-a2c3-60ff8d040140 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711194735246897, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711194735262869, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 3300171, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11887, "largest_seqno": 14891, "table_properties": {"data_size": 3287533, "index_size": 7905, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3717, "raw_key_size": 32682, "raw_average_key_size": 22, "raw_value_size": 3258583, "raw_average_value_size": 2227, "num_data_blocks": 358, "num_entries": 1463, "num_filter_entries": 1463, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711194735, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "79d1dda2-776a-437f-a2c3-60ff8d040140", "db_session_id": "MG209K9XCXXZQZVAETII", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711194735263025, "job": 1, "event": "recovery_finished"} 2024-03-23T11:52:15.654 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi063/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d704362000 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: DB pointer 0x55d704350000 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** DB Stats ** 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T11:52:15.655 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: L0 1/0 3.15 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 492.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: L6 1/0 6.23 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Sum 2/0 9.37 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 492.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 492.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T11:52:15.656 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 492.7 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative compaction: 0.00 GB write, 137.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval compaction: 0.00 GB write, 137.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:52:15.657 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Block cache BinnedLRUCache@0x55d7042c5350#7 capacity: 512.00 MB usage: 12.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: starting mon.smithi063 rank 0 at public addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] at bind addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi063 fsid 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???) e2 preinit fsid 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???).mds e24 new map 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???).mds e24 print_map 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: e24 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:15.658 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: legacy client fscid: 1 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Filesystem 'cephfs' (1) 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: fs_name cephfs 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: epoch 24 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: tableserver 0 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: root 0 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: session_timeout 60 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: session_autoclose 300 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: max_file_size 1099511627776 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: max_xattr_size 65536 2024-03-23T11:52:15.659 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: required_client_features {} 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: last_failure 0 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: last_failure_osd_epoch 46 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: max_mds 2 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: in 0,1 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: up {0=14462,1=14478} 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: failed 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: damaged 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: stopped 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: data_pools [3] 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: metadata_pool 2 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: inline_data disabled 2024-03-23T11:52:15.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: balancer 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: bal_rank_mask -1 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: standby_count_wanted 1 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: [mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: [mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: [mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: [mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-23T11:52:15.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 ceph-mon[141124]: mon.smithi063@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-23T11:52:15.662 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 bash[141086]: 48414a51fe0ee87fee9aa80ecd5ef1e2123ed48787ce3e6cd9789113c76e64f6 2024-03-23T11:52:15.662 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:15 smithi063 systemd[1]: Started Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:52:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: pgmap v103: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: from='client.14876 172.21.15.63:0/333297678' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: pgmap v104: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: mon.smithi063 calling monitor election 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: mon.smithi063 is new leader, mons smithi063,smithi111 in quorum (ranks 0,1) 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: monmap e2: 2 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: osdmap e48: 6 total, 6 up, 6 in 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: mgrmap e33: smithi063.gehkcr(active, since 3m), standbys: smithi111.nnblsk 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: mgrmap e34: smithi063.gehkcr(active, since 3m), standbys: smithi111.nnblsk 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: overall HEALTH_OK 2024-03-23T11:52:17.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:16 smithi063 ceph-mon[141124]: from='mgr.14684 ' entity='' 2024-03-23T11:52:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: pgmap v103: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: from='client.14876 172.21.15.63:0/333297678' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: pgmap v104: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: mon.smithi063 calling monitor election 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: mon.smithi063 is new leader, mons smithi063,smithi111 in quorum (ranks 0,1) 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: monmap e2: 2 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: osdmap e48: 6 total, 6 up, 6 in 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: mgrmap e33: smithi063.gehkcr(active, since 3m), standbys: smithi111.nnblsk 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: mgrmap e34: smithi063.gehkcr(active, since 3m), standbys: smithi111.nnblsk 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: overall HEALTH_OK 2024-03-23T11:52:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:16 smithi111 ceph-mon[44005]: from='mgr.14684 ' entity='' 2024-03-23T11:52:26.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: Standby manager daemon smithi111.nnblsk restarted 2024-03-23T11:52:26.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:52:26.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:52:26.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:52:26.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: Active manager daemon smithi063.gehkcr restarted 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: Activating manager daemon smithi063.gehkcr 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: osdmap e49: 6 total, 6 up, 6 in 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: mgrmap e35: smithi063.gehkcr(active, starting, since 0.0092021s), standbys: smithi111.nnblsk 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:52:26.634 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:26.635 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:52:26.636 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk restarted 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: Standby manager daemon smithi111.nnblsk started 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/crt"}]: dispatch 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.nnblsk/key"}]: dispatch 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.? 172.21.15.111:0/391342575' entity='mgr.smithi111.nnblsk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: Active manager daemon smithi063.gehkcr restarted 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: Activating manager daemon smithi063.gehkcr 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: osdmap e49: 6 total, 6 up, 6 in 2024-03-23T11:52:26.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: mgrmap e35: smithi063.gehkcr(active, starting, since 0.0092021s), standbys: smithi111.nnblsk 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi063.gehkcr", "id": "smithi063.gehkcr"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr metadata", "who": "smithi111.nnblsk", "id": "smithi111.nnblsk"}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:52:26.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: Manager daemon smithi063.gehkcr is now available 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/mirror_snapshot_schedule"}]: dispatch 2024-03-23T11:52:26.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:26 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.gehkcr/trash_purge_schedule"}]: dispatch 2024-03-23T11:52:28.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:27 smithi063 ceph-mon[141124]: [23/Mar/2024:11:52:26] ENGINE Bus STARTING 2024-03-23T11:52:28.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:27 smithi063 ceph-mon[141124]: [23/Mar/2024:11:52:26] ENGINE Serving on http://172.21.15.63:8765 2024-03-23T11:52:28.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:27 smithi063 ceph-mon[141124]: [23/Mar/2024:11:52:26] ENGINE Serving on https://172.21.15.63:7150 2024-03-23T11:52:28.026 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:27 smithi063 ceph-mon[141124]: [23/Mar/2024:11:52:26] ENGINE Bus STARTED 2024-03-23T11:52:28.026 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:27 smithi063 ceph-mon[141124]: mgrmap e36: smithi063.gehkcr(active, since 1.01333s), standbys: smithi111.nnblsk 2024-03-23T11:52:28.026 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:27 smithi063 ceph-mon[141124]: pgmap v3: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:52:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:27 smithi111 ceph-mon[44005]: [23/Mar/2024:11:52:26] ENGINE Bus STARTING 2024-03-23T11:52:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:27 smithi111 ceph-mon[44005]: [23/Mar/2024:11:52:26] ENGINE Serving on http://172.21.15.63:8765 2024-03-23T11:52:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:27 smithi111 ceph-mon[44005]: [23/Mar/2024:11:52:26] ENGINE Serving on https://172.21.15.63:7150 2024-03-23T11:52:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:27 smithi111 ceph-mon[44005]: [23/Mar/2024:11:52:26] ENGINE Bus STARTED 2024-03-23T11:52:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:27 smithi111 ceph-mon[44005]: mgrmap e36: smithi063.gehkcr(active, since 1.01333s), standbys: smithi111.nnblsk 2024-03-23T11:52:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:27 smithi111 ceph-mon[44005]: pgmap v3: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:52:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:28 smithi063 ceph-mon[141124]: mgrmap e37: smithi063.gehkcr(active, since 2s), standbys: smithi111.nnblsk 2024-03-23T11:52:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:28 smithi063 ceph-mon[141124]: pgmap v4: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:52:29.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:28 smithi111 ceph-mon[44005]: mgrmap e37: smithi063.gehkcr(active, since 2s), standbys: smithi111.nnblsk 2024-03-23T11:52:29.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:28 smithi111 ceph-mon[44005]: pgmap v4: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:52:30.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:30.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:30.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:30 smithi063 ceph-mon[141124]: pgmap v5: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:52:30.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:30 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:30.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:30 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:30.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:30 smithi111 ceph-mon[44005]: pgmap v5: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:52:31.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:31 smithi063 ceph-mon[141124]: mgrmap e38: smithi063.gehkcr(active, since 4s), standbys: smithi111.nnblsk 2024-03-23T11:52:31.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:31.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:31.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:31 smithi111 ceph-mon[44005]: mgrmap e38: smithi063.gehkcr(active, since 4s), standbys: smithi111.nnblsk 2024-03-23T11:52:31.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:31 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:31.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:31 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:32.801 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:32 smithi111 ceph-mon[44005]: pgmap v6: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-03-23T11:52:32.801 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:32 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:32.801 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:32 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:32 smithi063 ceph-mon[141124]: pgmap v6: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-03-23T11:52:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:34.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:34 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:34.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:34 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:34.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:34 smithi111 ceph-mon[44005]: pgmap v7: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-23T11:52:34.801 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:34 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:34.801 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:34 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:34.801 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:34 smithi063 ceph-mon[141124]: pgmap v7: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-23T11:52:36.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:36 smithi063 ceph-mon[141124]: Detected new or changed devices on smithi111 2024-03-23T11:52:36.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:36 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:36.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:36 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:36.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:36 smithi063 ceph-mon[141124]: pgmap v8: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-23T11:52:36.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:36 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi111 2024-03-23T11:52:36.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:36 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:36.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:36 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:36.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:36 smithi111 ceph-mon[44005]: pgmap v8: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-23T11:52:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:38 smithi063 ceph-mon[141124]: pgmap v9: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-23T11:52:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:39.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:52:39.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:38 smithi111 ceph-mon[44005]: pgmap v9: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-23T11:52:39.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:38 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:39.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:38 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:39.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:38 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:39.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:38 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:52:40.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:39 smithi063 ceph-mon[141124]: Detected new or changed devices on smithi063 2024-03-23T11:52:40.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:39 smithi063 ceph-mon[141124]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:52:40.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:39 smithi063 ceph-mon[141124]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:52:40.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:39 smithi063 ceph-mon[141124]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:52:40.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:39 smithi063 ceph-mon[141124]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:52:40.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:39 smithi111 ceph-mon[44005]: Detected new or changed devices on smithi063 2024-03-23T11:52:40.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:39 smithi111 ceph-mon[44005]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-23T11:52:40.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:39 smithi111 ceph-mon[44005]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-23T11:52:40.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:39 smithi111 ceph-mon[44005]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:52:40.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:39 smithi111 ceph-mon[44005]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-23T11:52:41.101 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:52:41.101 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:52:41.101 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: pgmap v10: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:52:41.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:40 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-23T11:52:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:52:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-23T11:52:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: Updating smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:52:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: Updating smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: pgmap v10: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:52:41.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-23T11:52:42.071 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:41 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:42.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:43.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:42 smithi063 ceph-mon[141124]: pgmap v11: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-23T11:52:43.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:43.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:52:43.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:52:43.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:43.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:42 smithi111 ceph-mon[44005]: pgmap v11: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-23T11:52:43.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:42 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:43.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:42 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:52:43.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:42 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:52:43.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:42 smithi111 ceph-mon[44005]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:43.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 -- 172.21.15.63:0/1001885920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070c70 msgr2=0x7f8cbc109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:43.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1001885920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070c70 0x7f8cbc109110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8004970 tx=0x7f8ca8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:43.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 -- 172.21.15.63:0/1001885920 shutdown_connections 2024-03-23T11:52:43.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1001885920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070c70 0x7f8cbc109110 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:43.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1001885920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cbc070360 0x7f8cbc070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:43.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 -- 172.21.15.63:0/1001885920 >> 172.21.15.63:0/1001885920 conn(0x7f8cbc0f89b0 msgr2=0x7f8cbc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:43.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 -- 172.21.15.63:0/1001885920 shutdown_connections 2024-03-23T11:52:43.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.676+0000 7f8cc3000700 1 -- 172.21.15.63:0/1001885920 wait complete. 2024-03-23T11:52:43.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.677+0000 7f8cc3000700 1 Processor -- start 2024-03-23T11:52:43.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.677+0000 7f8cc3000700 1 -- start start 2024-03-23T11:52:43.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cc3000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070360 0x7f8cbc115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:43.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cc3000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cbc070c70 0x7f8cbc1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:43.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cc3000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc116830 con 0x7f8cbc070360 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cc3000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc10fc00 con 0x7f8cbc070c70 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cbbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070360 0x7f8cbc115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cbbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070360 0x7f8cbc115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48838/0 (socket says 172.21.15.63:48838) 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cbbe00700 1 -- 172.21.15.63:0/1826725520 learned_addr learned my addr 172.21.15.63:0/1826725520 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cbb400700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cbc070c70 0x7f8cbc1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cbbe00700 1 -- 172.21.15.63:0/1826725520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cbc070c70 msgr2=0x7f8cbc1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cbbe00700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cbc070c70 0x7f8cbc1161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.678+0000 7f8cbbe00700 1 -- 172.21.15.63:0/1826725520 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca8005040 con 0x7f8cbc070360 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.679+0000 7f8cbbe00700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070360 0x7f8cbc115ca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cac002a50 tx=0x7f8cac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.679+0000 7f8cb3e00700 1 -- 172.21.15.63:0/1826725520 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cac007cc0 con 0x7f8cbc070360 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.679+0000 7f8cb3e00700 1 -- 172.21.15.63:0/1826725520 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cac00f040 con 0x7f8cbc070360 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.679+0000 7f8cb3e00700 1 -- 172.21.15.63:0/1826725520 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cac009540 con 0x7f8cbc070360 2024-03-23T11:52:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.679+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc10fee0 con 0x7f8cbc070360 2024-03-23T11:52:43.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.679+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc110420 con 0x7f8cbc070360 2024-03-23T11:52:43.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.680+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc103bd0 con 0x7f8cbc070360 2024-03-23T11:52:43.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.681+0000 7f8cb3e00700 1 -- 172.21.15.63:0/1826725520 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f8cac0096a0 con 0x7f8cbc070360 2024-03-23T11:52:43.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.682+0000 7f8cb3e00700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca006c8e0 0x7f8ca006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:43.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.682+0000 7f8cb3e00700 1 -- 172.21.15.63:0/1826725520 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f8cac08ba30 con 0x7f8cbc070360 2024-03-23T11:52:43.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.682+0000 7f8cbb400700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca006c8e0 0x7f8ca006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:43.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.683+0000 7f8cbb400700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca006c8e0 0x7f8ca006ed90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800b800 tx=0x7f8ca8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:43.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.688+0000 7f8cb3e00700 1 -- 172.21.15.63:0/1826725520 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8cac059520 con 0x7f8cbc070360 2024-03-23T11:52:43.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:43 smithi111 ceph-mon[44005]: Upgrade: Updating mon.smithi111 2024-03-23T11:52:43.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:43 smithi111 ceph-mon[44005]: Deploying daemon mon.smithi111 on smithi111 2024-03-23T11:52:43.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.920+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cbc10c020 con 0x7f8ca006c8e0 2024-03-23T11:52:43.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.921+0000 7f8cb3e00700 1 -- 172.21.15.63:0/1826725520 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f8cbc10c020 con 0x7f8ca006c8e0 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca006c8e0 msgr2=0x7f8ca006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca006c8e0 0x7f8ca006ed90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800b800 tx=0x7f8ca8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070360 msgr2=0x7f8cbc115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070360 0x7f8cbc115ca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cac002a50 tx=0x7f8cac0046b0 comp rx=0 tx=0).stop 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 shutdown_connections 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca006c8e0 0x7f8ca006ed90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cbc070360 0x7f8cbc115ca0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 --2- 172.21.15.63:0/1826725520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cbc070c70 0x7f8cbc1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 >> 172.21.15.63:0/1826725520 conn(0x7f8cbc0f89b0 msgr2=0x7f8cbc1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:43.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 shutdown_connections 2024-03-23T11:52:43.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:43.924+0000 7f8cc3000700 1 -- 172.21.15.63:0/1826725520 wait complete. 2024-03-23T11:52:43.922 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.061+0000 7f3e9f800700 1 -- 172.21.15.63:0/1885772527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e981037a0 msgr2=0x7f3e980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.061+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1885772527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e981037a0 0x7f3e980fce10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e84004940 tx=0x7f3e84009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.062+0000 7f3e9f800700 1 -- 172.21.15.63:0/1885772527 shutdown_connections 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.062+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1885772527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e981037a0 0x7f3e980fce10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.062+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1885772527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e98102e00 0x7f3e981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.062+0000 7f3e9f800700 1 -- 172.21.15.63:0/1885772527 >> 172.21.15.63:0/1885772527 conn(0x7f3e980f89b0 msgr2=0x7f3e980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.062+0000 7f3e9f800700 1 -- 172.21.15.63:0/1885772527 shutdown_connections 2024-03-23T11:52:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.062+0000 7f3e9f800700 1 -- 172.21.15.63:0/1885772527 wait complete. 2024-03-23T11:52:44.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9f800700 1 Processor -- start 2024-03-23T11:52:44.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9f800700 1 -- start start 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9f800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e98102e00 0x7f3e98115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9f800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e981037a0 0x7f3e98116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9f800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e98116890 con 0x7f3e98102e00 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9f800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9810fc60 con 0x7f3e981037a0 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e98102e00 0x7f3e98115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e98102e00 0x7f3e98115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48864/0 (socket says 172.21.15.63:48864) 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.063+0000 7f3e9ce00700 1 -- 172.21.15.63:0/1253187768 learned_addr learned my addr 172.21.15.63:0/1253187768 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e97e00700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e981037a0 0x7f3e98116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e9ce00700 1 -- 172.21.15.63:0/1253187768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e981037a0 msgr2=0x7f3e98116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e9ce00700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e981037a0 0x7f3e98116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e9ce00700 1 -- 172.21.15.63:0/1253187768 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84005040 con 0x7f3e98102e00 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e9ce00700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e98102e00 0x7f3e98115d00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c002a50 tx=0x7f3e8c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e94c00700 1 -- 172.21.15.63:0/1253187768 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e8c007cc0 con 0x7f3e98102e00 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e94c00700 1 -- 172.21.15.63:0/1253187768 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e8c00f040 con 0x7f3e98102e00 2024-03-23T11:52:44.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9810ff40 con 0x7f3e98102e00 2024-03-23T11:52:44.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.064+0000 7f3e94c00700 1 -- 172.21.15.63:0/1253187768 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e8c00d9c0 con 0x7f3e98102e00 2024-03-23T11:52:44.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.065+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e98110500 con 0x7f3e98102e00 2024-03-23T11:52:44.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.066+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e980fe550 con 0x7f3e98102e00 2024-03-23T11:52:44.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.066+0000 7f3e94c00700 1 -- 172.21.15.63:0/1253187768 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f3e8c00db20 con 0x7f3e98102e00 2024-03-23T11:52:44.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.067+0000 7f3e94c00700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e8006c8e0 0x7f3e8006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.067+0000 7f3e94c00700 1 -- 172.21.15.63:0/1253187768 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3e8c08b970 con 0x7f3e98102e00 2024-03-23T11:52:44.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.068+0000 7f3e97e00700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e8006c8e0 0x7f3e8006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:44.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.068+0000 7f3e97e00700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e8006c8e0 0x7f3e8006ed90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e981115a0 tx=0x7f3e84009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:44.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.072+0000 7f3e94c00700 1 -- 172.21.15.63:0/1253187768 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3e8c059460 con 0x7f3e98102e00 2024-03-23T11:52:44.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:43 smithi063 ceph-mon[141124]: Upgrade: Updating mon.smithi111 2024-03-23T11:52:44.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:43 smithi063 ceph-mon[141124]: Deploying daemon mon.smithi111 on smithi111 2024-03-23T11:52:44.280 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:43 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:52:44.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.310+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e9804bf40 con 0x7f3e8006c8e0 2024-03-23T11:52:44.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.311+0000 7f3e94c00700 1 -- 172.21.15.63:0/1253187768 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f3e9804bf40 con 0x7f3e8006c8e0 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e8006c8e0 msgr2=0x7f3e8006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e8006c8e0 0x7f3e8006ed90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e981115a0 tx=0x7f3e84009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e98102e00 msgr2=0x7f3e98115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e98102e00 0x7f3e98115d00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c002a50 tx=0x7f3e8c0046b0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 shutdown_connections 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e8006c8e0 0x7f3e8006ed90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e98102e00 0x7f3e98115d00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 --2- 172.21.15.63:0/1253187768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e981037a0 0x7f3e98116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 >> 172.21.15.63:0/1253187768 conn(0x7f3e980f89b0 msgr2=0x7f3e980f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 shutdown_connections 2024-03-23T11:52:44.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.314+0000 7f3e9f800700 1 -- 172.21.15.63:0/1253187768 wait complete. 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.450+0000 7f5a0c400700 1 -- 172.21.15.63:0/289644384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a04102e00 msgr2=0x7f5a041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.450+0000 7f5a0c400700 1 --2- 172.21.15.63:0/289644384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a04102e00 0x7f5a041031d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59f8004940 tx=0x7f59f8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.450+0000 7f5a0c400700 1 -- 172.21.15.63:0/289644384 shutdown_connections 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.450+0000 7f5a0c400700 1 --2- 172.21.15.63:0/289644384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a041037a0 0x7f5a040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.450+0000 7f5a0c400700 1 --2- 172.21.15.63:0/289644384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a04102e00 0x7f5a041031d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.450+0000 7f5a0c400700 1 -- 172.21.15.63:0/289644384 >> 172.21.15.63:0/289644384 conn(0x7f5a040f89b0 msgr2=0x7f5a040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.450+0000 7f5a0c400700 1 -- 172.21.15.63:0/289644384 shutdown_connections 2024-03-23T11:52:44.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.451+0000 7f5a0c400700 1 -- 172.21.15.63:0/289644384 wait complete. 2024-03-23T11:52:44.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.451+0000 7f5a0c400700 1 Processor -- start 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.451+0000 7f5a0c400700 1 -- start start 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a0c400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a04102e00 0x7f5a040702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a0c400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a041037a0 0x7f5a04070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a0c400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04070e20 con 0x7f5a041037a0 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a0c400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04070f60 con 0x7f5a04102e00 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a09a00700 1 -- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a04102e00 msgr2=0x7f5a040702c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:3300/0 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a09a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a04102e00 0x7f5a040702c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a09000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a041037a0 0x7f5a04070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a09000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a041037a0 0x7f5a04070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48944/0 (socket says 172.21.15.63:48944) 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.452+0000 7f5a09000700 1 -- 172.21.15.63:0/727345429 learned_addr learned my addr 172.21.15.63:0/727345429 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a09000700 1 -- 172.21.15.63:0/727345429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a04102e00 msgr2=0x7f5a040702c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a09000700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a04102e00 0x7f5a040702c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a09000700 1 -- 172.21.15.63:0/727345429 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f8005040 con 0x7f5a041037a0 2024-03-23T11:52:44.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a09000700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a041037a0 0x7f5a04070800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b790 tx=0x7f59f0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:44.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a01600700 1 -- 172.21.15.63:0/727345429 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59f0010790 con 0x7f5a041037a0 2024-03-23T11:52:44.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a01600700 1 -- 172.21.15.63:0/727345429 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59f0010da0 con 0x7f5a041037a0 2024-03-23T11:52:44.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a01600700 1 -- 172.21.15.63:0/727345429 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59f000f300 con 0x7f5a041037a0 2024-03-23T11:52:44.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.453+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a04119de0 con 0x7f5a041037a0 2024-03-23T11:52:44.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.454+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0411a2f0 con 0x7f5a041037a0 2024-03-23T11:52:44.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.455+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a040fe550 con 0x7f5a041037a0 2024-03-23T11:52:44.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.456+0000 7f5a01600700 1 -- 172.21.15.63:0/727345429 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f59f0011440 con 0x7f5a041037a0 2024-03-23T11:52:44.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.456+0000 7f5a01600700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f59ec06c880 0x7f59ec06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.457+0000 7f5a01600700 1 -- 172.21.15.63:0/727345429 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f59f008bd80 con 0x7f5a041037a0 2024-03-23T11:52:44.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.457+0000 7f5a09a00700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f59ec06c880 0x7f59ec06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:44.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.457+0000 7f5a09a00700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f59ec06c880 0x7f59ec06ed30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b020 tx=0x7f59f8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:44.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.461+0000 7f5a01600700 1 -- 172.21.15.63:0/727345429 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f59f0055e70 con 0x7f5a041037a0 2024-03-23T11:52:44.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:44 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi111[43958]: 2024-03-23T11:52:44.294+0000 7f4ca5400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:52:44.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:44 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi111[43958]: 2024-03-23T11:52:44.294+0000 7f4ca5400700 -1 mon.smithi111@1(peon) e2 *** Got Signal Terminated *** 2024-03-23T11:52:44.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.688+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a0404bed0 con 0x7f59ec06c880 2024-03-23T11:52:44.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.699+0000 7f5a01600700 1 -- 172.21.15.63:0/727345429 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5a0404bed0 con 0x7f59ec06c880 2024-03-23T11:52:44.683 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:52:44.683 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (2m) 12s ago 15m 14.8M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:52:44.683 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (15m) 12s ago 15m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (14m) 15s ago 14m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (103s) 12s ago 14m 73.2M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (8m) 12s ago 8m 119M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (8m) 12s ago 8m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (8m) 15s ago 8m 65.2M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (8m) 15s ago 8m 59.7M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (4m) 12s ago 16m 469M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (3m) 15s ago 14m 447M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (29s) 12s ago 16m 27.8M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (13m) 15s ago 13m 51.6M 2048M 17.2.7-726-g9c4e0c34 837124da804f 02d10f77a670 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (2m) 12s ago 15m 7398k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (2m) 15s ago 14m 7260k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (13m) 12s ago 13m 222M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:52:44.684 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (12m) 12s ago 12m 233M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:52:44.685 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (11m) 12s ago 11m 196M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:52:44.685 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (11m) 15s ago 11m 259M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:52:44.685 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (10m) 15s ago 10m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:52:44.685 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (10m) 15s ago 10m 227M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:52:44.685 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (2m) 12s ago 14m 45.5M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f59ec06c880 msgr2=0x7f59ec06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f59ec06c880 0x7f59ec06ed30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b020 tx=0x7f59f8009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a041037a0 msgr2=0x7f5a04070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a041037a0 0x7f5a04070800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b790 tx=0x7f59f0009220 comp rx=0 tx=0).stop 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 shutdown_connections 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f59ec06c880 0x7f59ec06ed30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a04102e00 0x7f5a040702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 --2- 172.21.15.63:0/727345429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a041037a0 0x7f5a04070800 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.701+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 >> 172.21.15.63:0/727345429 conn(0x7f5a040f89b0 msgr2=0x7f5a040f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.702+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 shutdown_connections 2024-03-23T11:52:44.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.702+0000 7f5a0c400700 1 -- 172.21.15.63:0/727345429 wait complete. 2024-03-23T11:52:44.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 -- 172.21.15.63:0/900406626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 msgr2=0x7ff0e00706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:44.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 --2- 172.21.15.63:0/900406626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e00706d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4004970 tx=0x7ff0d4009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:44.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 -- 172.21.15.63:0/900406626 shutdown_connections 2024-03-23T11:52:44.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 --2- 172.21.15.63:0/900406626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0070c10 0x7ff0e0109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 --2- 172.21.15.63:0/900406626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e00706d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 -- 172.21.15.63:0/900406626 >> 172.21.15.63:0/900406626 conn(0x7ff0e00f8970 msgr2=0x7ff0e00fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:44.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 -- 172.21.15.63:0/900406626 shutdown_connections 2024-03-23T11:52:44.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.838+0000 7ff0e8400700 1 -- 172.21.15.63:0/900406626 wait complete. 2024-03-23T11:52:44.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.839+0000 7ff0e8400700 1 Processor -- start 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.839+0000 7ff0e8400700 1 -- start start 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.839+0000 7ff0e8400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.839+0000 7ff0e8400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0070c10 0x7ff0e0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.839+0000 7ff0e8400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e0116900 con 0x7ff0e0070300 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.839+0000 7ff0e8400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e010fc40 con 0x7ff0e0070c10 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5000700 1 -- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0070c10 msgr2=0x7ff0e0116220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:3300/0 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0070c10 0x7ff0e0116220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e0115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48958/0 (socket says 172.21.15.63:48958) 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5a00700 1 -- 172.21.15.63:0/178950996 learned_addr learned my addr 172.21.15.63:0/178950996 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5a00700 1 -- 172.21.15.63:0/178950996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0070c10 msgr2=0x7ff0e0116220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5a00700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0070c10 0x7ff0e0116220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:44.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5a00700 1 -- 172.21.15.63:0/178950996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d4005040 con 0x7ff0e0070300 2024-03-23T11:52:44.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0e5a00700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e0115ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4000c00 tx=0x7ff0d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:44.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.840+0000 7ff0dd600700 1 -- 172.21.15.63:0/178950996 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0d400e070 con 0x7ff0e0070300 2024-03-23T11:52:44.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.841+0000 7ff0dd600700 1 -- 172.21.15.63:0/178950996 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0d4013420 con 0x7ff0e0070300 2024-03-23T11:52:44.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.841+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e010fe60 con 0x7ff0e0070300 2024-03-23T11:52:44.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.841+0000 7ff0dd600700 1 -- 172.21.15.63:0/178950996 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0d401c640 con 0x7ff0e0070300 2024-03-23T11:52:44.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.841+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e0110340 con 0x7ff0e0070300 2024-03-23T11:52:44.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.842+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e0071a40 con 0x7ff0e0070300 2024-03-23T11:52:44.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.843+0000 7ff0dd600700 1 -- 172.21.15.63:0/178950996 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7ff0d4013590 con 0x7ff0e0070300 2024-03-23T11:52:44.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.844+0000 7ff0dd600700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff0c806c880 0x7ff0c806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:44.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.844+0000 7ff0dd600700 1 -- 172.21.15.63:0/178950996 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff0d408d350 con 0x7ff0e0070300 2024-03-23T11:52:44.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.844+0000 7ff0e5000700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff0c806c880 0x7ff0c806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:44.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.845+0000 7ff0e5000700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff0c806c880 0x7ff0c806ed30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc006fd0 tx=0x7ff0cc006df0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:44.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:44.849+0000 7ff0dd600700 1 -- 172.21.15.63:0/178950996 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff0d405aec0 con 0x7ff0e0070300 2024-03-23T11:52:45.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.148+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff0e010c0d0 con 0x7ff0e0070300 2024-03-23T11:52:45.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.148+0000 7ff0dd600700 1 -- 172.21.15.63:0/178950996 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7ff0d405aa50 con 0x7ff0e0070300 2024-03-23T11:52:45.132 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:52:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 11, 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T11:52:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:52:45.134 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff0c806c880 msgr2=0x7ff0c806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff0c806c880 0x7ff0c806ed30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc006fd0 tx=0x7ff0cc006df0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 msgr2=0x7ff0e0115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e0115ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4000c00 tx=0x7ff0d4009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 shutdown_connections 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff0c806c880 0x7ff0c806ed30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0e0070300 0x7ff0e0115ce0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 --2- 172.21.15.63:0/178950996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0070c10 0x7ff0e0116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 >> 172.21.15.63:0/178950996 conn(0x7ff0e00f8970 msgr2=0x7ff0e00f9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:45.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 shutdown_connections 2024-03-23T11:52:45.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.151+0000 7ff0e8400700 1 -- 172.21.15.63:0/178950996 wait complete. 2024-03-23T11:52:45.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:44 smithi111 bash[106708]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi111 2024-03-23T11:52:45.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.289+0000 7f4a60000700 1 -- 172.21.15.63:0/2391722906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070c10 msgr2=0x7f4a58109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.289+0000 7f4a60000700 1 --2- 172.21.15.63:0/2391722906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070c10 0x7f4a58109190 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4a44004970 tx=0x7f4a44009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.289+0000 7f4a60000700 1 -- 172.21.15.63:0/2391722906 shutdown_connections 2024-03-23T11:52:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.289+0000 7f4a60000700 1 --2- 172.21.15.63:0/2391722906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070c10 0x7f4a58109190 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.289+0000 7f4a60000700 1 --2- 172.21.15.63:0/2391722906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a58070300 0x7f4a580706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.289+0000 7f4a60000700 1 -- 172.21.15.63:0/2391722906 >> 172.21.15.63:0/2391722906 conn(0x7f4a580f8970 msgr2=0x7f4a580fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.290+0000 7f4a60000700 1 -- 172.21.15.63:0/2391722906 shutdown_connections 2024-03-23T11:52:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.290+0000 7f4a60000700 1 -- 172.21.15.63:0/2391722906 wait complete. 2024-03-23T11:52:45.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.290+0000 7f4a60000700 1 Processor -- start 2024-03-23T11:52:45.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a60000700 1 -- start start 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a60000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070300 0x7f4a58115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a60000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a58070c10 0x7f4a58116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a60000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a58116900 con 0x7f4a58070300 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a60000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5810fc40 con 0x7f4a58070c10 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a5cc00700 1 -- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a58070c10 msgr2=0x7f4a58116220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:3300/0 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a5cc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a58070c10 0x7f4a58116220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a5d600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070300 0x7f4a58115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a5d600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070300 0x7f4a58115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48980/0 (socket says 172.21.15.63:48980) 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.291+0000 7f4a5d600700 1 -- 172.21.15.63:0/2337368156 learned_addr learned my addr 172.21.15.63:0/2337368156 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a5d600700 1 -- 172.21.15.63:0/2337368156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a58070c10 msgr2=0x7f4a58116220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:52:45.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a5d600700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a58070c10 0x7f4a58116220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a5d600700 1 -- 172.21.15.63:0/2337368156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a44005040 con 0x7f4a58070300 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a5d600700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070300 0x7f4a58115ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00b760 tx=0x7f4a4c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a55600700 1 -- 172.21.15.63:0/2337368156 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a4c010790 con 0x7f4a58070300 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a55600700 1 -- 172.21.15.63:0/2337368156 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a4c010da0 con 0x7f4a58070300 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a55600700 1 -- 172.21.15.63:0/2337368156 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a4c00f300 con 0x7f4a58070300 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5810fec0 con 0x7f4a58070300 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.292+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a58110400 con 0x7f4a58070300 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.293+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a58071a40 con 0x7f4a58070300 2024-03-23T11:52:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.294+0000 7f4a55600700 1 -- 172.21.15.63:0/2337368156 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f4a4c0116d0 con 0x7f4a58070300 2024-03-23T11:52:45.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.295+0000 7f4a55600700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a4006c880 0x7f4a4006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:45.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.295+0000 7f4a55600700 1 -- 172.21.15.63:0/2337368156 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4a4c08bcd0 con 0x7f4a58070300 2024-03-23T11:52:45.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.296+0000 7f4a5cc00700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a4006c880 0x7f4a4006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:45.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.296+0000 7f4a5cc00700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a4006c880 0x7f4a4006ed30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a44000c00 tx=0x7f4a44009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:45.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.300+0000 7f4a55600700 1 -- 172.21.15.63:0/2337368156 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a4c059640 con 0x7f4a58070300 2024-03-23T11:52:45.501 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:45 smithi111 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi111.service: Succeeded. 2024-03-23T11:52:45.501 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:45 smithi111 systemd[1]: Stopped Ceph mon.smithi111 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:52:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.555+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4a5805f5f0 con 0x7f4a58070300 2024-03-23T11:52:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.556+0000 7f4a55600700 1 -- 172.21.15.63:0/2337368156 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f4a4c059460 con 0x7f4a58070300 2024-03-23T11:52:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:52:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:52:45.541 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:52:45.542 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:52:45.543 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:52:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a4006c880 msgr2=0x7f4a4006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a4006c880 0x7f4a4006ed30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a44000c00 tx=0x7f4a44009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070300 msgr2=0x7f4a58115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070300 0x7f4a58115ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00b760 tx=0x7f4a4c009220 comp rx=0 tx=0).stop 2024-03-23T11:52:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 shutdown_connections 2024-03-23T11:52:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a4006c880 0x7f4a4006ed30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a58070300 0x7f4a58115ce0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 --2- 172.21.15.63:0/2337368156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a58070c10 0x7f4a58116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.558+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 >> 172.21.15.63:0/2337368156 conn(0x7f4a580f8970 msgr2=0x7f4a580f9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.559+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 shutdown_connections 2024-03-23T11:52:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.559+0000 7f4a60000700 1 -- 172.21.15.63:0/2337368156 wait complete. 2024-03-23T11:52:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 -- 172.21.15.63:0/1765340314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 msgr2=0x7f3fa0103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/1765340314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0103010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f90004940 tx=0x7f3f90009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 -- 172.21.15.63:0/1765340314 shutdown_connections 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/1765340314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa01035e0 0x7f3fa00fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/1765340314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0103010 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 -- 172.21.15.63:0/1765340314 >> 172.21.15.63:0/1765340314 conn(0x7f3fa00f8830 msgr2=0x7f3fa00fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 -- 172.21.15.63:0/1765340314 shutdown_connections 2024-03-23T11:52:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.697+0000 7f3f9fe00700 1 -- 172.21.15.63:0/1765340314 wait complete. 2024-03-23T11:52:45.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.698+0000 7f3f9fe00700 1 Processor -- start 2024-03-23T11:52:45.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.698+0000 7f3f9fe00700 1 -- start start 2024-03-23T11:52:45.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0115b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:45.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa01035e0 0x7f3fa0116060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa0116740 con 0x7f3fa0102c40 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9fe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa010faa0 con 0x7f3fa01035e0 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9e000700 1 -- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa01035e0 msgr2=0x7f3fa0116060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:3300/0 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0115b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9e000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa01035e0 0x7f3fa0116060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0115b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49000/0 (socket says 172.21.15.63:49000) 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9ea00700 1 -- 172.21.15.63:0/976275859 learned_addr learned my addr 172.21.15.63:0/976275859 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9ea00700 1 -- 172.21.15.63:0/976275859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa01035e0 msgr2=0x7f3fa0116060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9ea00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa01035e0 0x7f3fa0116060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.699+0000 7f3f9ea00700 1 -- 172.21.15.63:0/976275859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f90005040 con 0x7f3fa0102c40 2024-03-23T11:52:45.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.700+0000 7f3f9ea00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0115b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3f90004090 tx=0x7f3f90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:45.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.700+0000 7f3f96a00700 1 -- 172.21.15.63:0/976275859 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9000e070 con 0x7f3fa0102c40 2024-03-23T11:52:45.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.700+0000 7f3f96a00700 1 -- 172.21.15.63:0/976275859 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f90004290 con 0x7f3fa0102c40 2024-03-23T11:52:45.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.700+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa010fd20 con 0x7f3fa0102c40 2024-03-23T11:52:45.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.700+0000 7f3f96a00700 1 -- 172.21.15.63:0/976275859 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f90013cb0 con 0x7f3fa0102c40 2024-03-23T11:52:45.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.700+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa0110280 con 0x7f3fa0102c40 2024-03-23T11:52:45.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.701+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa00fe390 con 0x7f3fa0102c40 2024-03-23T11:52:45.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.702+0000 7f3f96a00700 1 -- 172.21.15.63:0/976275859 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f3f9000ba80 con 0x7f3fa0102c40 2024-03-23T11:52:45.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.703+0000 7f3f96a00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f8406c880 0x7f3f8406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:45.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.703+0000 7f3f96a00700 1 -- 172.21.15.63:0/976275859 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3f9008d390 con 0x7f3fa0102c40 2024-03-23T11:52:45.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.703+0000 7f3f9e000700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f8406c880 0x7f3f8406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:45.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.704+0000 7f3f9e000700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f8406c880 0x7f3f8406ed30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c006fd0 tx=0x7f3f8c006df0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:45.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.708+0000 7f3f96a00700 1 -- 172.21.15.63:0/976275859 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f9005af00 con 0x7f3fa0102c40 2024-03-23T11:52:45.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:45 smithi111 systemd[1]: Starting Ceph mon.smithi111 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:52:45.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.938+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fa0109ce0 con 0x7f3f8406c880 2024-03-23T11:52:45.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.939+0000 7f3f96a00700 1 -- 172.21.15.63:0/976275859 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f3fa0109ce0 con 0x7f3f8406c880 2024-03-23T11:52:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:52:45.923 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:52:45.923 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:52:45.923 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:52:45.923 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:52:45.924 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T11:52:45.924 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:52:45.924 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "3/21 daemons upgraded", 2024-03-23T11:52:45.924 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mon daemons", 2024-03-23T11:52:45.924 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:52:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:52:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f8406c880 msgr2=0x7f3f8406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f8406c880 0x7f3f8406ed30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c006fd0 tx=0x7f3f8c006df0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 msgr2=0x7f3fa0115b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0115b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3f90004090 tx=0x7f3f90009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 shutdown_connections 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f8406c880 0x7f3f8406ed30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa0102c40 0x7f3fa0115b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 --2- 172.21.15.63:0/976275859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fa01035e0 0x7f3fa0116060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 >> 172.21.15.63:0/976275859 conn(0x7f3fa00f8830 msgr2=0x7f3fa0070680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.941+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 shutdown_connections 2024-03-23T11:52:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:45.942+0000 7f3f9fe00700 1 -- 172.21.15.63:0/976275859 wait complete. 2024-03-23T11:52:46.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 -- 172.21.15.63:0/2564037091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c48102e00 msgr2=0x7f7c481031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:46.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2564037091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c48102e00 0x7f7c481031d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c38004940 tx=0x7f7c38009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:46.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 -- 172.21.15.63:0/2564037091 shutdown_connections 2024-03-23T11:52:46.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2564037091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c481037a0 0x7f7c480fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:46.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2564037091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c48102e00 0x7f7c481031d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 -- 172.21.15.63:0/2564037091 >> 172.21.15.63:0/2564037091 conn(0x7f7c480f89b0 msgr2=0x7f7c480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 -- 172.21.15.63:0/2564037091 shutdown_connections 2024-03-23T11:52:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.078+0000 7f7c4f200700 1 -- 172.21.15.63:0/2564037091 wait complete. 2024-03-23T11:52:46.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c4f200700 1 Processor -- start 2024-03-23T11:52:46.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c4f200700 1 -- start start 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c4f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c48102e00 0x7f7c48115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c4f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c481037a0 0x7f7c48116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c4f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48116930 con 0x7f7c481037a0 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c4f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4810fc70 con 0x7f7c48102e00 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c47e00700 1 -- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c48102e00 msgr2=0x7f7c48115d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:3300/0 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c3fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c481037a0 0x7f7c48116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c47e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c48102e00 0x7f7c48115d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c3fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c481037a0 0x7f7c48116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49018/0 (socket says 172.21.15.63:49018) 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.079+0000 7f7c3fe00700 1 -- 172.21.15.63:0/2095586441 learned_addr learned my addr 172.21.15.63:0/2095586441 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c3fe00700 1 -- 172.21.15.63:0/2095586441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c48102e00 msgr2=0x7f7c48115d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c3fe00700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c48102e00 0x7f7c48115d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c3fe00700 1 -- 172.21.15.63:0/2095586441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c38005040 con 0x7f7c481037a0 2024-03-23T11:52:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c3fe00700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c481037a0 0x7f7c48116250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b790 tx=0x7f7c30009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c44c00700 1 -- 172.21.15.63:0/2095586441 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c30010790 con 0x7f7c481037a0 2024-03-23T11:52:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4810fef0 con 0x7f7c481037a0 2024-03-23T11:52:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c44c00700 1 -- 172.21.15.63:0/2095586441 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c30010da0 con 0x7f7c481037a0 2024-03-23T11:52:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.080+0000 7f7c44c00700 1 -- 172.21.15.63:0/2095586441 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c3000f300 con 0x7f7c481037a0 2024-03-23T11:52:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.081+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c481104b0 con 0x7f7c481037a0 2024-03-23T11:52:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.082+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c480fe550 con 0x7f7c481037a0 2024-03-23T11:52:46.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.082+0000 7f7c44c00700 1 -- 172.21.15.63:0/2095586441 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f7c30011440 con 0x7f7c481037a0 2024-03-23T11:52:46.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.083+0000 7f7c44c00700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7c2c06c880 0x7f7c2c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:52:46.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.083+0000 7f7c44c00700 1 -- 172.21.15.63:0/2095586441 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7c3008bdc0 con 0x7f7c481037a0 2024-03-23T11:52:46.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.084+0000 7f7c47e00700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7c2c06c880 0x7f7c2c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:52:46.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.084+0000 7f7c47e00700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7c2c06c880 0x7f7c2c06ed30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800b020 tx=0x7f7c38009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:52:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.088+0000 7f7c44c00700 1 -- 172.21.15.63:0/2095586441 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7c3008f030 con 0x7f7c481037a0 2024-03-23T11:52:46.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.395+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7c4805f5f0 con 0x7f7c481037a0 2024-03-23T11:52:46.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.395+0000 7f7c44c00700 1 -- 172.21.15.63:0/2095586441 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f7c30059940 con 0x7f7c481037a0 2024-03-23T11:52:46.379 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.397+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7c2c06c880 msgr2=0x7f7c2c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.397+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7c2c06c880 0x7f7c2c06ed30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800b020 tx=0x7f7c38009e30 comp rx=0 tx=0).stop 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.397+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c481037a0 msgr2=0x7f7c48116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.397+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c481037a0 0x7f7c48116250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b790 tx=0x7f7c30009220 comp rx=0 tx=0).stop 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.398+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 shutdown_connections 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.398+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7c2c06c880 0x7f7c2c06ed30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.398+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c48102e00 0x7f7c48115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.398+0000 7f7c4f200700 1 --2- 172.21.15.63:0/2095586441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c481037a0 0x7f7c48116250 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.398+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 >> 172.21.15.63:0/2095586441 conn(0x7f7c480f89b0 msgr2=0x7f7c480f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:52:46.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.398+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 shutdown_connections 2024-03-23T11:52:46.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:52:46.398+0000 7f7c4f200700 1 -- 172.21.15.63:0/2095586441 wait complete. 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: set uid:gid to 167:167 (ceph:ceph) 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable), process ceph-mon, pid 7 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: pidfile_write: ignore empty --pid-file 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: load: jerasure load: lrc 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: RocksDB version: 7.9.2 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Git sha 0 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Compile date 2024-03-23 00:21:31 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: DB SUMMARY 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: DB Session ID: 6MARC8TR0EV8ARHBEQXL 2024-03-23T11:52:46.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: CURRENT file: CURRENT 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: IDENTITY file: IDENTITY 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1072 Bytes 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi111/store.db dir, Total Num: 1, files: 000027.sst 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi111/store.db: 000025.log size: 8242034 ; 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.error_if_exists: 0 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.create_if_missing: 0 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.paranoid_checks: 1 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-23T11:52:46.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.env: 0x562a7ed1a720 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.fs: PosixFileSystem 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.info_log: 0x562a822333c0 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_file_opening_threads: 16 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.statistics: (nil) 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.use_fsync: 0 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_log_file_size: 0 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.keep_log_file_num: 1000 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.recycle_log_file_num: 0 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.allow_fallocate: 1 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.allow_mmap_reads: 0 2024-03-23T11:52:46.405 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.allow_mmap_writes: 0 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.use_direct_reads: 0 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.create_missing_column_families: 0 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.db_log_dir: 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.wal_dir: 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-23T11:52:46.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.advise_random_on_open: 1 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.db_write_buffer_size: 0 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.write_buffer_manager: 0x562a814ac140 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.rate_limiter: (nil) 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.wal_recovery_mode: 2 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.enable_thread_tracking: 0 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.enable_pipelined_write: 0 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.unordered_write: 0 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-23T11:52:46.407 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.row_cache: None 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.wal_filter: None 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.allow_ingest_behind: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.two_write_queues: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.manual_wal_flush: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.wal_compression: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.atomic_flush: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-23T11:52:46.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-23T11:52:46.409 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.log_readahead_size: 0 2024-03-23T11:52:46.409 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-23T11:52:46.409 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.best_efforts_recovery: 0 2024-03-23T11:52:46.409 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-23T11:52:46.409 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-23T11:52:46.409 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.allow_data_in_errors: 0 2024-03-23T11:52:46.409 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.db_host_id: __hostname__ 2024-03-23T11:52:46.410 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-23T11:52:46.410 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_background_jobs: 2 2024-03-23T11:52:46.410 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_background_compactions: -1 2024-03-23T11:52:46.410 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_subcompactions: 1 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_total_wal_size: 0 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_open_files: -1 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bytes_per_sync: 0 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_readahead_size: 0 2024-03-23T11:52:46.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_background_flushes: -1 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Compression algorithms supported: 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kZSTD supported: 0 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kXpressCompression supported: 0 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kLZ4HCCompression supported: 1 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kZlibCompression supported: 1 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kSnappyCompression supported: 1 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kLZ4Compression supported: 1 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: kBZip2Compression supported: 0 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000009 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.merge_operator: 2024-03-23T11:52:46.412 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_filter: None 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_filter_factory: None 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.sst_partitioner_factory: None 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562a822334e0) 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_type: 0 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_index_type: 0 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_shortening: 1 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: checksum: 4 2024-03-23T11:52:46.413 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: no_block_cache: 0 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache: 0x562a8154d1f0 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_options: 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: capacity : 536870912 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: num_shard_bits : 4 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: strict_capacity_limit : 0 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_compressed: (nil) 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: persistent_cache: (nil) 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size: 4096 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size_deviation: 10 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_restart_interval: 16 2024-03-23T11:52:46.414 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_block_restart_interval: 1 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: metadata_block_size: 4096 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: partition_filters: 0 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: use_delta_encoding: 1 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: filter_policy: bloomfilter 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: whole_key_filtering: 1 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: verify_compression: 0 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: format_version: 5 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_index_compression: 1 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_align: 0 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_auto_readahead_size: 262144 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: prepopulate_block_cache: 0 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: initial_auto_readahead_size: 8192 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: num_file_reads_for_auto_readahead: 2 2024-03-23T11:52:46.415 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.write_buffer_size: 33554432 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_write_buffer_number: 2 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression: NoCompression 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression: Disabled 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.prefix_extractor: nullptr 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.num_levels: 7 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-23T11:52:46.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.level: 32767 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.strategy: 0 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.enabled: false 2024-03-23T11:52:46.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.target_file_size_base: 67108864 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-23T11:52:46.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.arena_block_size: 1048576 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.disable_auto_compactions: 0 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-23T11:52:46.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.table_properties_collectors: 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.inplace_update_support: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.bloom_locality: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.max_successive_merges: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.paranoid_file_checks: 0 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.force_consistency_checks: 1 2024-03-23T11:52:46.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.report_bg_io_stats: 0 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.ttl: 2592000 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.enable_blob_files: false 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.min_blob_size: 0 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.blob_file_size: 268435456 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.blob_file_starting_level: 0 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-23T11:52:46.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12084, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 94f629b3-3df2-4bd8-9d58-a65b1e42bf71 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711194766157852, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 bash[106862]: df03633d2cc1c824d64f4c985e607d7d837c51aa73e91ea1f6b53fc8384b57dd 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 systemd[1]: Started Ceph mon.smithi111 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711194766188563, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4948820, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12089, "largest_seqno": 15985, "table_properties": {"data_size": 4934016, "index_size": 9559, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4229, "raw_key_size": 37854, "raw_average_key_size": 22, "raw_value_size": 4900261, "raw_average_value_size": 2902, "num_data_blocks": 431, "num_entries": 1688, "num_filter_entries": 1688, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711194766, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "94f629b3-3df2-4bd8-9d58-a65b1e42bf71", "db_session_id": "6MARC8TR0EV8ARHBEQXL", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711194766188719, "job": 1, "event": "recovery_finished"} 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi111/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-23T11:52:46.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562a815ea000 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: DB pointer 0x562a815d8000 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** DB Stats ** 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T11:52:46.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L0 1/0 4.72 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 503.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L6 1/0 6.23 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Sum 2/0 10.95 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 503.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 503.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 503.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-23T11:52:46.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative compaction: 0.00 GB write, 129.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval compaction: 0.00 GB write, 129.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Block cache BinnedLRUCache@0x562a8154d1f0#7 capacity: 512.00 MB usage: 14.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,10.08 KB,0.00192225%) Misc(1,0.00 KB,0%) 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: starting mon.smithi111 rank 1 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi111 fsid 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???) e2 preinit fsid 6880ffb6-e909-11ee-95cd-87774f69a715 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???).mds e24 new map 2024-03-23T11:52:46.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???).mds e24 print_map 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: e24 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: legacy client fscid: 1 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Filesystem 'cephfs' (1) 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: fs_name cephfs 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: epoch 24 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:52:46.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: tableserver 0 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: root 0 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: session_timeout 60 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: session_autoclose 300 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_file_size 1099511627776 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_xattr_size 65536 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: required_client_features {} 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: last_failure 0 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: last_failure_osd_epoch 46 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_mds 2 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: in 0,1 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: up {0=14462,1=14478} 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: failed 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: damaged 2024-03-23T11:52:46.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: stopped 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_pools [3] 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: metadata_pool 2 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: inline_data disabled 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: balancer 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: bal_rank_mask -1 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: standby_count_wanted 1 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-23T11:52:46.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-23T11:52:46.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:46 smithi111 ceph-mon[106899]: mon.smithi111@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-03-23T11:52:48.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:47 smithi063 ceph-mon[141124]: pgmap v12: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:48.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:47 smithi111 ceph-mon[106899]: pgmap v12: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: pgmap v14: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: overall HEALTH_OK 2024-03-23T11:52:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:52:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: mon.smithi111 calling monitor election 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: mon.smithi063 calling monitor election 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: mon.smithi063 is new leader, mons smithi063,smithi111 in quorum (ranks 0,1) 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: monmap e3: 2 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: osdmap e49: 6 total, 6 up, 6 in 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: mgrmap e38: smithi063.gehkcr(active, since 22s), standbys: smithi111.nnblsk 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: overall HEALTH_OK 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:49.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: pgmap v14: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: overall HEALTH_OK 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: mon.smithi111 calling monitor election 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: mon.smithi063 calling monitor election 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: mon.smithi063 is new leader, mons smithi063,smithi111 in quorum (ranks 0,1) 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: monmap e3: 2 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: osdmap e49: 6 total, 6 up, 6 in 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: mgrmap e38: smithi063.gehkcr(active, since 22s), standbys: smithi111.nnblsk 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: overall HEALTH_OK 2024-03-23T11:52:49.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:49.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:52:50.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:50 smithi111 ceph-mon[106899]: pgmap v15: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:50 smithi063 ceph-mon[141124]: pgmap v15: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:52:52.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:52.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:52.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:52 smithi063 ceph-mon[141124]: pgmap v16: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:52:52.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:52.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:52.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:52 smithi111 ceph-mon[106899]: pgmap v16: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:52:53.775 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:53 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:53.775 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:53 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:53.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:53 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:53.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:53 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:54.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:54 smithi063 ceph-mon[141124]: pgmap v17: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:52:54.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:54 smithi111 ceph-mon[106899]: pgmap v17: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:52:56.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:56 smithi111 ceph-mon[106899]: pgmap v18: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:52:56.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:56.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:56 smithi063 ceph-mon[141124]: pgmap v18: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:52:56.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:52:58.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:58.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: Reconfiguring mon.smithi063 (monmap changed)... 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: Reconfiguring daemon mon.smithi063 on smithi063 2024-03-23T11:52:58.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:57 smithi063 ceph-mon[141124]: pgmap v19: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:52:58.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: Reconfiguring mon.smithi063 (monmap changed)... 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: Reconfiguring daemon mon.smithi063 on smithi063 2024-03-23T11:52:58.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:57 smithi111 ceph-mon[106899]: pgmap v19: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:53:00.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:00.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:00.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: Reconfiguring mgr.smithi063.gehkcr (monmap changed)... 2024-03-23T11:53:00.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:53:00.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:53:00.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:00.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: Reconfiguring daemon mgr.smithi063.gehkcr on smithi063 2024-03-23T11:53:00.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:52:59 smithi063 ceph-mon[141124]: pgmap v20: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:53:00.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:00.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:00.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: Reconfiguring mgr.smithi063.gehkcr (monmap changed)... 2024-03-23T11:53:00.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.gehkcr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:53:00.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:53:00.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:00.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: Reconfiguring daemon mgr.smithi063.gehkcr on smithi063 2024-03-23T11:53:00.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:52:59 smithi111 ceph-mon[106899]: pgmap v20: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T11:53:01.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:01 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:01.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:01 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:01.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:01 smithi063 ceph-mon[141124]: Reconfiguring crash.smithi063 (monmap changed)... 2024-03-23T11:53:01.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:01 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:01.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:01 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:01.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:01 smithi063 ceph-mon[141124]: Reconfiguring daemon crash.smithi063 on smithi063 2024-03-23T11:53:01.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:01 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:01.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:01 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:01.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:01 smithi111 ceph-mon[106899]: Reconfiguring crash.smithi063 (monmap changed)... 2024-03-23T11:53:01.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:01 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:01.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:01 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:01.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:01 smithi111 ceph-mon[106899]: Reconfiguring daemon crash.smithi063 on smithi063 2024-03-23T11:53:02.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:02 smithi063 ceph-mon[141124]: pgmap v21: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:53:02.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:02.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:02.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T11:53:02.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:02.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:02 smithi111 ceph-mon[106899]: pgmap v21: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:53:02.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:02.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:02.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T11:53:02.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:03.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:03 smithi063 ceph-mon[141124]: Reconfiguring osd.0 (monmap changed)... 2024-03-23T11:53:03.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:03 smithi063 ceph-mon[141124]: Reconfiguring daemon osd.0 on smithi063 2024-03-23T11:53:03.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:03 smithi111 ceph-mon[106899]: Reconfiguring osd.0 (monmap changed)... 2024-03-23T11:53:03.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:03 smithi111 ceph-mon[106899]: Reconfiguring daemon osd.0 on smithi063 2024-03-23T11:53:04.854 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:04 smithi063 ceph-mon[141124]: pgmap v22: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:04.854 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:04 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:04.854 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:04 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:04.854 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:04 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T11:53:04.855 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:04 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:04.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:04 smithi111 ceph-mon[106899]: pgmap v22: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:04.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:04 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:04.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:04 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:04.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:04 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T11:53:04.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:04 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:05.766 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:05 smithi063 ceph-mon[141124]: Reconfiguring osd.1 (monmap changed)... 2024-03-23T11:53:05.766 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:05 smithi063 ceph-mon[141124]: Reconfiguring daemon osd.1 on smithi063 2024-03-23T11:53:05.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:05 smithi111 ceph-mon[106899]: Reconfiguring osd.1 (monmap changed)... 2024-03-23T11:53:05.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:05 smithi111 ceph-mon[106899]: Reconfiguring daemon osd.1 on smithi063 2024-03-23T11:53:06.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:06 smithi063 ceph-mon[141124]: pgmap v23: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:06.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:06.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:06.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T11:53:06.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:06.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:06 smithi111 ceph-mon[106899]: pgmap v23: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:06.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:06.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:06.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T11:53:06.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:07.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:07 smithi063 ceph-mon[141124]: Reconfiguring osd.2 (monmap changed)... 2024-03-23T11:53:07.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:07 smithi063 ceph-mon[141124]: Reconfiguring daemon osd.2 on smithi063 2024-03-23T11:53:07.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:07 smithi111 ceph-mon[106899]: Reconfiguring osd.2 (monmap changed)... 2024-03-23T11:53:07.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:07 smithi111 ceph-mon[106899]: Reconfiguring daemon osd.2 on smithi063 2024-03-23T11:53:09.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:08 smithi063 ceph-mon[141124]: pgmap v24: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:09.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:09.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:09.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:08 smithi063 ceph-mon[141124]: Reconfiguring mds.cephfs.smithi063.brtzgy (monmap changed)... 2024-03-23T11:53:09.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:09.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:09.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:08 smithi063 ceph-mon[141124]: Reconfiguring daemon mds.cephfs.smithi063.brtzgy on smithi063 2024-03-23T11:53:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:08 smithi111 ceph-mon[106899]: pgmap v24: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:08 smithi111 ceph-mon[106899]: Reconfiguring mds.cephfs.smithi063.brtzgy (monmap changed)... 2024-03-23T11:53:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:09.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:09.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:08 smithi111 ceph-mon[106899]: Reconfiguring daemon mds.cephfs.smithi063.brtzgy on smithi063 2024-03-23T11:53:10.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:10.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:10.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:10 smithi063 ceph-mon[141124]: Reconfiguring mds.cephfs.smithi063.mdbdrz (monmap changed)... 2024-03-23T11:53:10.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:10.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:10.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:10 smithi063 ceph-mon[141124]: Reconfiguring daemon mds.cephfs.smithi063.mdbdrz on smithi063 2024-03-23T11:53:10.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:10 smithi063 ceph-mon[141124]: pgmap v25: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:10.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:10.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:10.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:10 smithi111 ceph-mon[106899]: Reconfiguring mds.cephfs.smithi063.mdbdrz (monmap changed)... 2024-03-23T11:53:10.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:10.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:10.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:10 smithi111 ceph-mon[106899]: Reconfiguring daemon mds.cephfs.smithi063.mdbdrz on smithi063 2024-03-23T11:53:10.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:10 smithi111 ceph-mon[106899]: pgmap v25: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:11.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:11.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:11.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:11.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:11.633 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:11.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:11.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:11.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:11.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:11.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:12.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:12 smithi111 ceph-mon[106899]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-23T11:53:12.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:12 smithi111 ceph-mon[106899]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-23T11:53:12.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:12 smithi111 ceph-mon[106899]: pgmap v26: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:12.759 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:12 smithi063 ceph-mon[141124]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-23T11:53:12.759 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:12 smithi063 ceph-mon[141124]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-23T11:53:12.760 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:12 smithi063 ceph-mon[141124]: pgmap v26: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:14.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:13 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:14.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:13 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:14.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:13 smithi063 ceph-mon[141124]: Reconfiguring mgr.smithi111.nnblsk (monmap changed)... 2024-03-23T11:53:14.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:13 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:53:14.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:13 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:53:14.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:13 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:14.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:13 smithi063 ceph-mon[141124]: Reconfiguring daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:53:14.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:13 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:14.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:13 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:14.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:13 smithi111 ceph-mon[106899]: Reconfiguring mgr.smithi111.nnblsk (monmap changed)... 2024-03-23T11:53:14.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:13 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.nnblsk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-23T11:53:14.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:13 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-23T11:53:14.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:13 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:14.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:13 smithi111 ceph-mon[106899]: Reconfiguring daemon mgr.smithi111.nnblsk on smithi111 2024-03-23T11:53:15.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:14 smithi111 ceph-mon[106899]: pgmap v27: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:15.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:15.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:15.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:53:15.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:53:15.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:15.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:14 smithi063 ceph-mon[141124]: pgmap v27: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:15.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:15.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:15.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-23T11:53:15.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-23T11:53:15.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:15.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:15 smithi111 ceph-mon[106899]: Reconfiguring mon.smithi111 (monmap changed)... 2024-03-23T11:53:15.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:15 smithi111 ceph-mon[106899]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-23T11:53:16.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:15 smithi063 ceph-mon[141124]: Reconfiguring mon.smithi111 (monmap changed)... 2024-03-23T11:53:16.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:15 smithi063 ceph-mon[141124]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.553+0000 7f617a200700 1 -- 172.21.15.63:0/2791216204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 msgr2=0x7f61740fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.553+0000 7f617a200700 1 --2- 172.21.15.63:0/2791216204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61740fce10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6160004940 tx=0x7f6160009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.554+0000 7f617a200700 1 -- 172.21.15.63:0/2791216204 shutdown_connections 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.554+0000 7f617a200700 1 --2- 172.21.15.63:0/2791216204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61740fce10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.554+0000 7f617a200700 1 --2- 172.21.15.63:0/2791216204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6174102e00 0x7f61741031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.554+0000 7f617a200700 1 -- 172.21.15.63:0/2791216204 >> 172.21.15.63:0/2791216204 conn(0x7f61740f89b0 msgr2=0x7f61740fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.554+0000 7f617a200700 1 -- 172.21.15.63:0/2791216204 shutdown_connections 2024-03-23T11:53:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.554+0000 7f617a200700 1 -- 172.21.15.63:0/2791216204 wait complete. 2024-03-23T11:53:16.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.554+0000 7f617a200700 1 Processor -- start 2024-03-23T11:53:16.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f617a200700 1 -- start start 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f617a200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6174102e00 0x7f6174115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f617a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61741162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f617a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6174116930 con 0x7f61741037a0 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f617a200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617410fd00 con 0x7f6174102e00 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f6172a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61741162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f6173400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6174102e00 0x7f6174115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f6172a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61741162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42114/0 (socket says 172.21.15.63:42114) 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.555+0000 7f6172a00700 1 -- 172.21.15.63:0/4021381376 learned_addr learned my addr 172.21.15.63:0/4021381376 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f6172a00700 1 -- 172.21.15.63:0/4021381376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6174102e00 msgr2=0x7f6174115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f6172a00700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6174102e00 0x7f6174115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f6172a00700 1 -- 172.21.15.63:0/4021381376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6160005040 con 0x7f61741037a0 2024-03-23T11:53:16.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f6172a00700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61741162e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6160009390 tx=0x7f6160009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:16.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f616b400700 1 -- 172.21.15.63:0/4021381376 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f616000e040 con 0x7f61741037a0 2024-03-23T11:53:16.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f616b400700 1 -- 172.21.15.63:0/4021381376 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6160013420 con 0x7f61741037a0 2024-03-23T11:53:16.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617410ff80 con 0x7f61741037a0 2024-03-23T11:53:16.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.556+0000 7f616b400700 1 -- 172.21.15.63:0/4021381376 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f616001c630 con 0x7f61741037a0 2024-03-23T11:53:16.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.557+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61741104c0 con 0x7f61741037a0 2024-03-23T11:53:16.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.558+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61740fe550 con 0x7f61741037a0 2024-03-23T11:53:16.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.559+0000 7f616b400700 1 -- 172.21.15.63:0/4021381376 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f6160013a00 con 0x7f61741037a0 2024-03-23T11:53:16.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.560+0000 7f616b400700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f615806c8e0 0x7f615806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:16.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.560+0000 7f616b400700 1 -- 172.21.15.63:0/4021381376 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f616008cd60 con 0x7f61741037a0 2024-03-23T11:53:16.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.560+0000 7f6173400700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f615806c8e0 0x7f615806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:16.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.561+0000 7f6173400700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f615806c8e0 0x7f615806ed90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6164004f40 tx=0x7f616400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:16.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.565+0000 7f616b400700 1 -- 172.21.15.63:0/4021381376 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f616005a8f0 con 0x7f61741037a0 2024-03-23T11:53:16.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.799+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6174109e10 con 0x7f615806c8e0 2024-03-23T11:53:16.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.801+0000 7f616b400700 1 -- 172.21.15.63:0/4021381376 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f6174109e10 con 0x7f615806c8e0 2024-03-23T11:53:16.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f615806c8e0 msgr2=0x7f615806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f615806c8e0 0x7f615806ed90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6164004f40 tx=0x7f616400b040 comp rx=0 tx=0).stop 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 msgr2=0x7f61741162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61741162e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6160009390 tx=0x7f6160009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 shutdown_connections 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f615806c8e0 0x7f615806ed90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6174102e00 0x7f6174115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 --2- 172.21.15.63:0/4021381376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61741037a0 0x7f61741162e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.803+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 >> 172.21.15.63:0/4021381376 conn(0x7f61740f89b0 msgr2=0x7f61740f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.804+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 shutdown_connections 2024-03-23T11:53:16.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.804+0000 7f617a200700 1 -- 172.21.15.63:0/4021381376 wait complete. 2024-03-23T11:53:16.800 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:53:16.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 -- 172.21.15.63:0/1679102025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5844102e00 msgr2=0x7f58441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:16.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 --2- 172.21.15.63:0/1679102025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5844102e00 0x7f58441031d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5834004940 tx=0x7f5834009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:16.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 -- 172.21.15.63:0/1679102025 shutdown_connections 2024-03-23T11:53:16.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 --2- 172.21.15.63:0/1679102025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58441037a0 0x7f58440fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 --2- 172.21.15.63:0/1679102025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5844102e00 0x7f58441031d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 -- 172.21.15.63:0/1679102025 >> 172.21.15.63:0/1679102025 conn(0x7f58440f89b0 msgr2=0x7f58440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:16.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 -- 172.21.15.63:0/1679102025 shutdown_connections 2024-03-23T11:53:16.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.941+0000 7f5849000700 1 -- 172.21.15.63:0/1679102025 wait complete. 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.942+0000 7f5849000700 1 Processor -- start 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.942+0000 7f5849000700 1 -- start start 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f5849000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5844102e00 0x7f5844115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f5849000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58441037a0 0x7f58441161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f5849000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5844116800 con 0x7f58441037a0 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f5849000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584410fbd0 con 0x7f5844102e00 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f583be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58441037a0 0x7f58441161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f5841e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5844102e00 0x7f5844115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f583be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58441037a0 0x7f58441161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42120/0 (socket says 172.21.15.63:42120) 2024-03-23T11:53:16.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f583be00700 1 -- 172.21.15.63:0/1575582508 learned_addr learned my addr 172.21.15.63:0/1575582508 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:16.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f583be00700 1 -- 172.21.15.63:0/1575582508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5844102e00 msgr2=0x7f5844115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:16.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f583be00700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5844102e00 0x7f5844115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:16.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.943+0000 7f583be00700 1 -- 172.21.15.63:0/1575582508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5834005040 con 0x7f58441037a0 2024-03-23T11:53:16.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.944+0000 7f583be00700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58441037a0 0x7f58441161b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f582c00eb40 tx=0x7f582c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:16.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.944+0000 7f583a000700 1 -- 172.21.15.63:0/1575582508 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f582c007cc0 con 0x7f58441037a0 2024-03-23T11:53:16.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.944+0000 7f583a000700 1 -- 172.21.15.63:0/1575582508 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f582c00f040 con 0x7f58441037a0 2024-03-23T11:53:16.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.944+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584410fe50 con 0x7f58441037a0 2024-03-23T11:53:16.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.944+0000 7f583a000700 1 -- 172.21.15.63:0/1575582508 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f582c00c8f0 con 0x7f58441037a0 2024-03-23T11:53:16.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.944+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5844110410 con 0x7f58441037a0 2024-03-23T11:53:16.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.945+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58440fe550 con 0x7f58441037a0 2024-03-23T11:53:16.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.946+0000 7f583a000700 1 -- 172.21.15.63:0/1575582508 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f582c0090d0 con 0x7f58441037a0 2024-03-23T11:53:16.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.947+0000 7f583a000700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f582806c8e0 0x7f582806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:16.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.947+0000 7f583a000700 1 -- 172.21.15.63:0/1575582508 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f582c08b6e0 con 0x7f58441037a0 2024-03-23T11:53:16.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.948+0000 7f5841e00700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f582806c8e0 0x7f582806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:16.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.948+0000 7f5841e00700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f582806c8e0 0x7f582806ed90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58340045a0 tx=0x7f5834009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:16.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:16.952+0000 7f583a000700 1 -- 172.21.15.63:0/1575582508 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f582c059280 con 0x7f58441037a0 2024-03-23T11:53:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:16 smithi063 ceph-mon[141124]: pgmap v28: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:16 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:16 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:16 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T11:53:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:16 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:16 smithi111 ceph-mon[106899]: pgmap v28: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:16 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:16 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:16 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T11:53:17.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:16 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:17.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.184+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5844109e30 con 0x7f582806c8e0 2024-03-23T11:53:17.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.185+0000 7f583a000700 1 -- 172.21.15.63:0/1575582508 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f5844109e30 con 0x7f582806c8e0 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f582806c8e0 msgr2=0x7f582806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f582806c8e0 0x7f582806ed90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58340045a0 tx=0x7f5834009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58441037a0 msgr2=0x7f58441161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58441037a0 0x7f58441161b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f582c00eb40 tx=0x7f582c00b040 comp rx=0 tx=0).stop 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 shutdown_connections 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f582806c8e0 0x7f582806ed90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5844102e00 0x7f5844115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 --2- 172.21.15.63:0/1575582508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58441037a0 0x7f58441161b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 >> 172.21.15.63:0/1575582508 conn(0x7f58440f89b0 msgr2=0x7f58440f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 shutdown_connections 2024-03-23T11:53:17.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.188+0000 7f5849000700 1 -- 172.21.15.63:0/1575582508 wait complete. 2024-03-23T11:53:17.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 -- 172.21.15.63:0/2131077128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 msgr2=0x7f8784105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 --2- 172.21.15.63:0/2131077128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784105460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8774004940 tx=0x7f8774009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:17.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 -- 172.21.15.63:0/2131077128 shutdown_connections 2024-03-23T11:53:17.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 --2- 172.21.15.63:0/2131077128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8784105a30 0x7f87840ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 --2- 172.21.15.63:0/2131077128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784105460 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 -- 172.21.15.63:0/2131077128 >> 172.21.15.63:0/2131077128 conn(0x7f87840fac00 msgr2=0x7f87840fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:17.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 -- 172.21.15.63:0/2131077128 shutdown_connections 2024-03-23T11:53:17.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.327+0000 7f878ae00700 1 -- 172.21.15.63:0/2131077128 wait complete. 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.328+0000 7f878ae00700 1 Processor -- start 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.328+0000 7f878ae00700 1 -- start start 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f878ae00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784115b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f878ae00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8784105a30 0x7f8784116040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f878ae00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8784116720 con 0x7f8784105090 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f878ae00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878410fc70 con 0x7f8784105a30 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784115b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784115b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42144/0 (socket says 172.21.15.63:42144) 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783e00700 1 -- 172.21.15.63:0/3035887802 learned_addr learned my addr 172.21.15.63:0/3035887802 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783400700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8784105a30 0x7f8784116040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:17.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783e00700 1 -- 172.21.15.63:0/3035887802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8784105a30 msgr2=0x7f8784116040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783e00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8784105a30 0x7f8784116040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783e00700 1 -- 172.21.15.63:0/3035887802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8774005040 con 0x7f8784105090 2024-03-23T11:53:17.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783400700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8784105a30 0x7f8784116040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:53:17.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.329+0000 7f8783e00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784115b00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8774008d00 tx=0x7f8774009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:17.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.330+0000 7f877be00700 1 -- 172.21.15.63:0/3035887802 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f877400e070 con 0x7f8784105090 2024-03-23T11:53:17.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.330+0000 7f877be00700 1 -- 172.21.15.63:0/3035887802 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8774004290 con 0x7f8784105090 2024-03-23T11:53:17.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.330+0000 7f877be00700 1 -- 172.21.15.63:0/3035887802 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8774013d90 con 0x7f8784105090 2024-03-23T11:53:17.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.330+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878410fe90 con 0x7f8784105090 2024-03-23T11:53:17.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.330+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87841103f0 con 0x7f8784105090 2024-03-23T11:53:17.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.331+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87841023a0 con 0x7f8784105090 2024-03-23T11:53:17.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.332+0000 7f877be00700 1 -- 172.21.15.63:0/3035887802 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f87740135b0 con 0x7f8784105090 2024-03-23T11:53:17.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.333+0000 7f877be00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f876806c8f0 0x7f876806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:17.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.333+0000 7f877be00700 1 -- 172.21.15.63:0/3035887802 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f877408d3a0 con 0x7f8784105090 2024-03-23T11:53:17.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.333+0000 7f8783400700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f876806c8f0 0x7f876806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:17.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.334+0000 7f8783400700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f876806c8f0 0x7f876806eda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87841113a0 tx=0x7f8770008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:17.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.337+0000 7f877be00700 1 -- 172.21.15.63:0/3035887802 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f877405af10 con 0x7f8784105090 2024-03-23T11:53:17.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.561+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f878410c160 con 0x7f876806c8f0 2024-03-23T11:53:17.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.571+0000 7f877be00700 1 -- 172.21.15.63:0/3035887802 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f877000d630 con 0x7f876806c8f0 2024-03-23T11:53:17.554 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:53:17.554 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (2m) 45s ago 15m 14.8M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:53:17.554 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (15m) 45s ago 15m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 2fb70d0660e5 2024-03-23T11:53:17.554 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (14m) 26s ago 14m 6651k - 17.2.7-726-g9c4e0c34 837124da804f 69339b7fe002 2024-03-23T11:53:17.554 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (2m) 45s ago 15m 73.2M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:53:17.554 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (9m) 45s ago 9m 119M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (9m) 45s ago 9m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (9m) 26s ago 9m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (9m) 26s ago 9m 60.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (4m) 45s ago 17m 469M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (3m) 26s ago 14m 449M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (62s) 45s ago 17m 27.8M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (31s) 26s ago 14m 24.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (3m) 45s ago 15m 7398k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (3m) 26s ago 14m 7327k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (13m) 45s ago 13m 222M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (13m) 45s ago 13m 233M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (12m) 45s ago 12m 196M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (12m) 26s ago 12m 259M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:53:17.555 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (11m) 26s ago 11m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:53:17.556 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (11m) 26s ago 11m 228M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:53:17.556 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (2m) 45s ago 15m 45.5M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:53:17.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f876806c8f0 msgr2=0x7f876806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f876806c8f0 0x7f876806eda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87841113a0 tx=0x7f8770008040 comp rx=0 tx=0).stop 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 msgr2=0x7f8784115b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784115b00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8774008d00 tx=0x7f8774009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 shutdown_connections 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f876806c8f0 0x7f876806eda0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8784105090 0x7f8784115b00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 --2- 172.21.15.63:0/3035887802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8784105a30 0x7f8784116040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.573+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 >> 172.21.15.63:0/3035887802 conn(0x7f87840fac00 msgr2=0x7f87840fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.574+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 shutdown_connections 2024-03-23T11:53:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.574+0000 7f878ae00700 1 -- 172.21.15.63:0/3035887802 wait complete. 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.710+0000 7fb38ce00700 1 -- 172.21.15.63:0/1405303470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb388102e00 msgr2=0x7fb3881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.710+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1405303470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb388102e00 0x7fb3881031d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb378004940 tx=0x7fb378009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.710+0000 7fb38ce00700 1 -- 172.21.15.63:0/1405303470 shutdown_connections 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.710+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1405303470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3881037a0 0x7fb3880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.710+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1405303470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb388102e00 0x7fb3881031d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.710+0000 7fb38ce00700 1 -- 172.21.15.63:0/1405303470 >> 172.21.15.63:0/1405303470 conn(0x7fb3880f89b0 msgr2=0x7fb3880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.711+0000 7fb38ce00700 1 -- 172.21.15.63:0/1405303470 shutdown_connections 2024-03-23T11:53:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.711+0000 7fb38ce00700 1 -- 172.21.15.63:0/1405303470 wait complete. 2024-03-23T11:53:17.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.711+0000 7fb38ce00700 1 Processor -- start 2024-03-23T11:53:17.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb38ce00700 1 -- start start 2024-03-23T11:53:17.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb38ce00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb388102e00 0x7fb388113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb38ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3881037a0 0x7fb388114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb38ce00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3881146b0 con 0x7fb3881037a0 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb38ce00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38810daa0 con 0x7fb388102e00 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb37fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3881037a0 0x7fb388114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb385e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb388102e00 0x7fb388113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb37fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3881037a0 0x7fb388114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42164/0 (socket says 172.21.15.63:42164) 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.712+0000 7fb37fe00700 1 -- 172.21.15.63:0/1974589029 learned_addr learned my addr 172.21.15.63:0/1974589029 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb37fe00700 1 -- 172.21.15.63:0/1974589029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb388102e00 msgr2=0x7fb388113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb37fe00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb388102e00 0x7fb388113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb37fe00700 1 -- 172.21.15.63:0/1974589029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378005040 con 0x7fb3881037a0 2024-03-23T11:53:17.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb385e00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb388102e00 0x7fb388113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:53:17.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb37fe00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3881037a0 0x7fb388114060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb370002a50 tx=0x7fb370004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:17.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb37e000700 1 -- 172.21.15.63:0/1974589029 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb370007cc0 con 0x7fb3881037a0 2024-03-23T11:53:17.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb37e000700 1 -- 172.21.15.63:0/1974589029 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb37000f040 con 0x7fb3881037a0 2024-03-23T11:53:17.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38810dd80 con 0x7fb3881037a0 2024-03-23T11:53:17.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb37e000700 1 -- 172.21.15.63:0/1974589029 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb37000d920 con 0x7fb3881037a0 2024-03-23T11:53:17.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.713+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38810e340 con 0x7fb3881037a0 2024-03-23T11:53:17.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.714+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3880fe550 con 0x7fb3881037a0 2024-03-23T11:53:17.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.715+0000 7fb37e000700 1 -- 172.21.15.63:0/1974589029 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7fb370009680 con 0x7fb3881037a0 2024-03-23T11:53:17.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.716+0000 7fb37e000700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb36c06c8e0 0x7fb36c06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:17.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.716+0000 7fb37e000700 1 -- 172.21.15.63:0/1974589029 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb37008b9d0 con 0x7fb3881037a0 2024-03-23T11:53:17.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.717+0000 7fb385e00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb36c06c8e0 0x7fb36c06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:17.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.717+0000 7fb385e00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb36c06c8e0 0x7fb36c06ed90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb37800b3b0 tx=0x7fb378009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:17.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:17.721+0000 7fb37e000700 1 -- 172.21.15.63:0/1974589029 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb3700594c0 con 0x7fb3881037a0 2024-03-23T11:53:17.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:17 smithi111 ceph-mon[106899]: Reconfiguring osd.3 (monmap changed)... 2024-03-23T11:53:17.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:17 smithi111 ceph-mon[106899]: Reconfiguring daemon osd.3 on smithi111 2024-03-23T11:53:17.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:17 smithi111 ceph-mon[106899]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:18.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.030+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb388109dc0 con 0x7fb3881037a0 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.031+0000 7fb37e000700 1 -- 172.21.15.63:0/1974589029 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fb370059050 con 0x7fb3881037a0 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:18.014 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 10, 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:53:18.015 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:53:18.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb36c06c8e0 msgr2=0x7fb36c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb36c06c8e0 0x7fb36c06ed90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb37800b3b0 tx=0x7fb378009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:18.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3881037a0 msgr2=0x7fb388114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3881037a0 0x7fb388114060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb370002a50 tx=0x7fb370004600 comp rx=0 tx=0).stop 2024-03-23T11:53:18.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 shutdown_connections 2024-03-23T11:53:18.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb36c06c8e0 0x7fb36c06ed90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb388102e00 0x7fb388113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 --2- 172.21.15.63:0/1974589029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3881037a0 0x7fb388114060 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.033+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 >> 172.21.15.63:0/1974589029 conn(0x7fb3880f89b0 msgr2=0x7fb3880f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:18.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.034+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 shutdown_connections 2024-03-23T11:53:18.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.034+0000 7fb38ce00700 1 -- 172.21.15.63:0/1974589029 wait complete. 2024-03-23T11:53:18.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:17 smithi063 ceph-mon[141124]: Reconfiguring osd.3 (monmap changed)... 2024-03-23T11:53:18.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:17 smithi063 ceph-mon[141124]: Reconfiguring daemon osd.3 on smithi111 2024-03-23T11:53:18.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:17 smithi063 ceph-mon[141124]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 -- 172.21.15.63:0/376077435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e94102e00 msgr2=0x7f7e941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/376077435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e94102e00 0x7f7e941031d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e88004940 tx=0x7f7e88009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 -- 172.21.15.63:0/376077435 shutdown_connections 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/376077435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e941037a0 0x7f7e940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/376077435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e94102e00 0x7f7e941031d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 -- 172.21.15.63:0/376077435 >> 172.21.15.63:0/376077435 conn(0x7f7e940f89b0 msgr2=0x7f7e940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 -- 172.21.15.63:0/376077435 shutdown_connections 2024-03-23T11:53:18.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.170+0000 7f7e9bc00700 1 -- 172.21.15.63:0/376077435 wait complete. 2024-03-23T11:53:18.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.171+0000 7f7e9bc00700 1 Processor -- start 2024-03-23T11:53:18.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.171+0000 7f7e9bc00700 1 -- start start 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.171+0000 7f7e9bc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94102e00 0x7f7e94115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.171+0000 7f7e9bc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e941037a0 0x7f7e94116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.171+0000 7f7e9bc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94116860 con 0x7f7e941037a0 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.171+0000 7f7e9bc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9410fc30 con 0x7f7e94102e00 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e93e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e941037a0 0x7f7e94116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e93e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e941037a0 0x7f7e94116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42180/0 (socket says 172.21.15.63:42180) 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e93e00700 1 -- 172.21.15.63:0/4149466740 learned_addr learned my addr 172.21.15.63:0/4149466740 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e99200700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94102e00 0x7f7e94115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e93e00700 1 -- 172.21.15.63:0/4149466740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94102e00 msgr2=0x7f7e94115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e93e00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94102e00 0x7f7e94115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e93e00700 1 -- 172.21.15.63:0/4149466740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e88005040 con 0x7f7e941037a0 2024-03-23T11:53:18.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.172+0000 7f7e93e00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e941037a0 0x7f7e94116210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e80002a50 tx=0x7f7e80004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:18.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.173+0000 7f7e90c00700 1 -- 172.21.15.63:0/4149466740 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e80007cc0 con 0x7f7e941037a0 2024-03-23T11:53:18.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.173+0000 7f7e90c00700 1 -- 172.21.15.63:0/4149466740 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e8000f040 con 0x7f7e941037a0 2024-03-23T11:53:18.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.173+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9410ff10 con 0x7f7e941037a0 2024-03-23T11:53:18.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.173+0000 7f7e90c00700 1 -- 172.21.15.63:0/4149466740 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e8000d920 con 0x7f7e941037a0 2024-03-23T11:53:18.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.173+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e941104d0 con 0x7f7e941037a0 2024-03-23T11:53:18.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.174+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e940fe550 con 0x7f7e941037a0 2024-03-23T11:53:18.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.175+0000 7f7e90c00700 1 -- 172.21.15.63:0/4149466740 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f7e80009680 con 0x7f7e941037a0 2024-03-23T11:53:18.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.176+0000 7f7e90c00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e7c06c8e0 0x7f7e7c06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.176+0000 7f7e90c00700 1 -- 172.21.15.63:0/4149466740 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7e8008b920 con 0x7f7e941037a0 2024-03-23T11:53:18.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.176+0000 7f7e99200700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e7c06c8e0 0x7f7e7c06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.177+0000 7f7e99200700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e7c06c8e0 0x7f7e7c06ed90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009fd0 tx=0x7f7e88009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:18.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.180+0000 7f7e90c00700 1 -- 172.21.15.63:0/4149466740 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7e80059450 con 0x7f7e941037a0 2024-03-23T11:53:18.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.441+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7e9404bed0 con 0x7f7e941037a0 2024-03-23T11:53:18.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.442+0000 7f7e90c00700 1 -- 172.21.15.63:0/4149466740 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f7e80058fe0 con 0x7f7e941037a0 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:53:18.425 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:53:18.426 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:53:18.427 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:53:18.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e7c06c8e0 msgr2=0x7f7e7c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e7c06c8e0 0x7f7e7c06ed90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009fd0 tx=0x7f7e88009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:18.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e941037a0 msgr2=0x7f7e94116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e941037a0 0x7f7e94116210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e80002a50 tx=0x7f7e80004600 comp rx=0 tx=0).stop 2024-03-23T11:53:18.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 shutdown_connections 2024-03-23T11:53:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e7c06c8e0 0x7f7e7c06ed90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94102e00 0x7f7e94115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 --2- 172.21.15.63:0/4149466740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e941037a0 0x7f7e94116210 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 >> 172.21.15.63:0/4149466740 conn(0x7f7e940f89b0 msgr2=0x7f7e940f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 shutdown_connections 2024-03-23T11:53:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.444+0000 7f7e9bc00700 1 -- 172.21.15.63:0/4149466740 wait complete. 2024-03-23T11:53:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:53:18.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.580+0000 7fbeef000700 1 -- 172.21.15.63:0/242279616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee8102e00 msgr2=0x7fbee81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.580+0000 7fbeef000700 1 --2- 172.21.15.63:0/242279616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee8102e00 0x7fbee81031d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbed8004940 tx=0x7fbed8009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:18.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.581+0000 7fbeef000700 1 -- 172.21.15.63:0/242279616 shutdown_connections 2024-03-23T11:53:18.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.581+0000 7fbeef000700 1 --2- 172.21.15.63:0/242279616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee81037a0 0x7fbee80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.581+0000 7fbeef000700 1 --2- 172.21.15.63:0/242279616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee8102e00 0x7fbee81031d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.581+0000 7fbeef000700 1 -- 172.21.15.63:0/242279616 >> 172.21.15.63:0/242279616 conn(0x7fbee80f89b0 msgr2=0x7fbee80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:18.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.581+0000 7fbeef000700 1 -- 172.21.15.63:0/242279616 shutdown_connections 2024-03-23T11:53:18.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.581+0000 7fbeef000700 1 -- 172.21.15.63:0/242279616 wait complete. 2024-03-23T11:53:18.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.581+0000 7fbeef000700 1 Processor -- start 2024-03-23T11:53:18.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbeef000700 1 -- start start 2024-03-23T11:53:18.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbeef000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee8102e00 0x7fbee8115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbeef000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee81037a0 0x7fbee81161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbeef000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee8116820 con 0x7fbee81037a0 2024-03-23T11:53:18.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbeef000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee810fbf0 con 0x7fbee8102e00 2024-03-23T11:53:18.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbee7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee8102e00 0x7fbee8115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbee7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee81037a0 0x7fbee81161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbee7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee81037a0 0x7fbee81161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42200/0 (socket says 172.21.15.63:42200) 2024-03-23T11:53:18.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbee7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee8102e00 0x7fbee8115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:47862/0 (socket says 172.21.15.63:47862) 2024-03-23T11:53:18.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.582+0000 7fbee7400700 1 -- 172.21.15.63:0/3239524242 learned_addr learned my addr 172.21.15.63:0/3239524242 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbee7e00700 1 -- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee81037a0 msgr2=0x7fbee81161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbee7e00700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee81037a0 0x7fbee81161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbee7e00700 1 -- 172.21.15.63:0/3239524242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed8005040 con 0x7fbee8102e00 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbee7400700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee81037a0 0x7fbee81161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbee7e00700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee8102e00 0x7fbee8115c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbed800b0d0 tx=0x7fbed8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbedfe00700 1 -- 172.21.15.63:0/3239524242 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbed800e070 con 0x7fbee8102e00 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbedfe00700 1 -- 172.21.15.63:0/3239524242 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbed8013e00 con 0x7fbee8102e00 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbedfe00700 1 -- 172.21.15.63:0/3239524242 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbed800bcf0 con 0x7fbee8102e00 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.583+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee810fe70 con 0x7fbee8102e00 2024-03-23T11:53:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.584+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee8110430 con 0x7fbee8102e00 2024-03-23T11:53:18.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.585+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee80fe550 con 0x7fbee8102e00 2024-03-23T11:53:18.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.585+0000 7fbedfe00700 1 -- 172.21.15.63:0/3239524242 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7fbed8013950 con 0x7fbee8102e00 2024-03-23T11:53:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.586+0000 7fbedfe00700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbecc06c8e0 0x7fbecc06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.586+0000 7fbedfe00700 1 -- 172.21.15.63:0/3239524242 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fbed808c430 con 0x7fbee8102e00 2024-03-23T11:53:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.587+0000 7fbee7400700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbecc06c8e0 0x7fbecc06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.587+0000 7fbee7400700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbecc06c8e0 0x7fbecc06ed90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbee8111530 tx=0x7fbed400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:18.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.591+0000 7fbedfe00700 1 -- 172.21.15.63:0/3239524242 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbed805a060 con 0x7fbee8102e00 2024-03-23T11:53:18.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.826+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbee804bed0 con 0x7fbecc06c8e0 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.827+0000 7fbedfe00700 1 -- 172.21.15.63:0/3239524242 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fbee804bed0 con 0x7fbecc06c8e0 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "4/21 daemons upgraded", 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mon daemons", 2024-03-23T11:53:18.810 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:53:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:53:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbecc06c8e0 msgr2=0x7fbecc06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbecc06c8e0 0x7fbecc06ed90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbee8111530 tx=0x7fbed400b040 comp rx=0 tx=0).stop 2024-03-23T11:53:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee8102e00 msgr2=0x7fbee8115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee8102e00 0x7fbee8115c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbed800b0d0 tx=0x7fbed8009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 shutdown_connections 2024-03-23T11:53:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbecc06c8e0 0x7fbecc06ed90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee8102e00 0x7fbee8115c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 --2- 172.21.15.63:0/3239524242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbee81037a0 0x7fbee81161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 >> 172.21.15.63:0/3239524242 conn(0x7fbee80f89b0 msgr2=0x7fbee80f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.829+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 shutdown_connections 2024-03-23T11:53:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.830+0000 7fbeef000700 1 -- 172.21.15.63:0/3239524242 wait complete. 2024-03-23T11:53:18.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 -- 172.21.15.63:0/1848467749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 msgr2=0x7fdce4070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 --2- 172.21.15.63:0/1848467749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4070730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4004970 tx=0x7fdcd4009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:18.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 -- 172.21.15.63:0/1848467749 shutdown_connections 2024-03-23T11:53:18.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 --2- 172.21.15.63:0/1848467749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdce4070c70 0x7fdce4108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 --2- 172.21.15.63:0/1848467749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4070730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 -- 172.21.15.63:0/1848467749 >> 172.21.15.63:0/1848467749 conn(0x7fdce40f8830 msgr2=0x7fdce40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:18.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 -- 172.21.15.63:0/1848467749 shutdown_connections 2024-03-23T11:53:18.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.965+0000 7fdce8a00700 1 -- 172.21.15.63:0/1848467749 wait complete. 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.966+0000 7fdce8a00700 1 Processor -- start 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.966+0000 7fdce8a00700 1 -- start start 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce8a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdce4070c70 0x7fdce4115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce8a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce4116620 con 0x7fdce4070360 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce8a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce410fa50 con 0x7fdce4070c70 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce2800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce2800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4115a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42226/0 (socket says 172.21.15.63:42226) 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce2800700 1 -- 172.21.15.63:0/3208812141 learned_addr learned my addr 172.21.15.63:0/3208812141 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce1e00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdce4070c70 0x7fdce4115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce2800700 1 -- 172.21.15.63:0/3208812141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdce4070c70 msgr2=0x7fdce4115fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce2800700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdce4070c70 0x7fdce4115fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.967+0000 7fdce2800700 1 -- 172.21.15.63:0/3208812141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4005040 con 0x7fdce4070360 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.968+0000 7fdce2800700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4115a90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4005010 tx=0x7fdcd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.968+0000 7fdcdaa00700 1 -- 172.21.15.63:0/3208812141 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcd400e040 con 0x7fdce4070360 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.968+0000 7fdcdaa00700 1 -- 172.21.15.63:0/3208812141 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcd40098b0 con 0x7fdce4070360 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.968+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce410fcd0 con 0x7fdce4070360 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.968+0000 7fdcdaa00700 1 -- 172.21.15.63:0/3208812141 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcd4013e80 con 0x7fdce4070360 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.968+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce4110210 con 0x7fdce4070360 2024-03-23T11:53:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.969+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce4071aa0 con 0x7fdce4070360 2024-03-23T11:53:18.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.970+0000 7fdcdaa00700 1 -- 172.21.15.63:0/3208812141 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7fdcd40136b0 con 0x7fdce4070360 2024-03-23T11:53:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.971+0000 7fdcdaa00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc806c870 0x7fdcc806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.971+0000 7fdcdaa00700 1 -- 172.21.15.63:0/3208812141 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fdcd408cef0 con 0x7fdce4070360 2024-03-23T11:53:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.971+0000 7fdce1e00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc806c870 0x7fdcc806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.972+0000 7fdce1e00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc806c870 0x7fdcc806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdce4104940 tx=0x7fdcd000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:18.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:18.975+0000 7fdcdaa00700 1 -- 172.21.15.63:0/3208812141 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdcd405a9e0 con 0x7fdce4070360 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: pgmap v29: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1974589029' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:18 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4149466740' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: pgmap v29: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1974589029' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:19.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:18 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4149466740' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:53:19.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.271+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdce404bed0 con 0x7fdce4070360 2024-03-23T11:53:19.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.272+0000 7fdcdaa00700 1 -- 172.21.15.63:0/3208812141 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fdcd405a570 con 0x7fdce4070360 2024-03-23T11:53:19.254 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc806c870 msgr2=0x7fdcc806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc806c870 0x7fdcc806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdce4104940 tx=0x7fdcd000b040 comp rx=0 tx=0).stop 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 msgr2=0x7fdce4115a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4115a90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4005010 tx=0x7fdcd4009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 shutdown_connections 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc806c870 0x7fdcc806ed20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdce4070360 0x7fdce4115a90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 --2- 172.21.15.63:0/3208812141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdce4070c70 0x7fdce4115fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 >> 172.21.15.63:0/3208812141 conn(0x7fdce40f8830 msgr2=0x7fdce40f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 shutdown_connections 2024-03-23T11:53:19.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:19.274+0000 7fdce8a00700 1 -- 172.21.15.63:0/3208812141 wait complete. 2024-03-23T11:53:20.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: Reconfiguring osd.4 (monmap changed)... 2024-03-23T11:53:20.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: Reconfiguring daemon osd.4 on smithi111 2024-03-23T11:53:20.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: from='client.44119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:20.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3208812141' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:53:20.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:20.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:20.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T11:53:20.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:19 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: Reconfiguring osd.4 (monmap changed)... 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: Reconfiguring daemon osd.4 on smithi111 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: from='client.44119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3208812141' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T11:53:20.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:19 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:21.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:20 smithi063 ceph-mon[141124]: Reconfiguring osd.5 (monmap changed)... 2024-03-23T11:53:21.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:20 smithi063 ceph-mon[141124]: Reconfiguring daemon osd.5 on smithi111 2024-03-23T11:53:21.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:20 smithi063 ceph-mon[141124]: pgmap v30: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:21.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:20 smithi111 ceph-mon[106899]: Reconfiguring osd.5 (monmap changed)... 2024-03-23T11:53:21.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:20 smithi111 ceph-mon[106899]: Reconfiguring daemon osd.5 on smithi111 2024-03-23T11:53:21.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:20 smithi111 ceph-mon[106899]: pgmap v30: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:22.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:22.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:22.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:22 smithi111 ceph-mon[106899]: Reconfiguring mds.cephfs.smithi111.ylnmzy (monmap changed)... 2024-03-23T11:53:22.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:22.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:22.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:22 smithi111 ceph-mon[106899]: Reconfiguring daemon mds.cephfs.smithi111.ylnmzy on smithi111 2024-03-23T11:53:22.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:22 smithi111 ceph-mon[106899]: pgmap v31: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:22 smithi063 ceph-mon[141124]: Reconfiguring mds.cephfs.smithi111.ylnmzy (monmap changed)... 2024-03-23T11:53:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:22 smithi063 ceph-mon[141124]: Reconfiguring daemon mds.cephfs.smithi111.ylnmzy on smithi111 2024-03-23T11:53:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:22 smithi063 ceph-mon[141124]: pgmap v31: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:24.284 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:24 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:24.284 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:24 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:24.284 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:24 smithi111 ceph-mon[106899]: Reconfiguring mds.cephfs.smithi111.atkibb (monmap changed)... 2024-03-23T11:53:24.284 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:24 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:24.284 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:24 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:24.284 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:24 smithi111 ceph-mon[106899]: Reconfiguring daemon mds.cephfs.smithi111.atkibb on smithi111 2024-03-23T11:53:24.285 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:24 smithi111 ceph-mon[106899]: pgmap v32: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:24.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:24 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:24.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:24 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:24.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:24 smithi063 ceph-mon[141124]: Reconfiguring mds.cephfs.smithi111.atkibb (monmap changed)... 2024-03-23T11:53:24.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:24 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:53:24.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:24 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:24.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:24 smithi063 ceph-mon[141124]: Reconfiguring daemon mds.cephfs.smithi111.atkibb on smithi111 2024-03-23T11:53:24.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:24 smithi063 ceph-mon[141124]: pgmap v32: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:25.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:25.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all mon 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi063"}]: dispatch 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi063"}]': finished 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]: dispatch 2024-03-23T11:53:25.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]': finished 2024-03-23T11:53:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:53:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all mon 2024-03-23T11:53:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi063"}]: dispatch 2024-03-23T11:53:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi063"}]': finished 2024-03-23T11:53:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]: dispatch 2024-03-23T11:53:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]': finished 2024-03-23T11:53:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:26 smithi063 ceph-mon[141124]: pgmap v33: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:27.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:27.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:26 smithi111 ceph-mon[106899]: pgmap v33: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:28.002 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:27 smithi063 ceph-mon[141124]: Upgrade: Updating crash.smithi063 (1/2) 2024-03-23T11:53:28.002 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:28.002 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:28.003 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:27 smithi063 ceph-mon[141124]: Deploying daemon crash.smithi063 on smithi063 2024-03-23T11:53:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:27 smithi111 ceph-mon[106899]: Upgrade: Updating crash.smithi063 (1/2) 2024-03-23T11:53:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:27 smithi111 ceph-mon[106899]: Deploying daemon crash.smithi063 on smithi063 2024-03-23T11:53:28.972 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:28 smithi063 ceph-mon[141124]: pgmap v34: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:29.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:28 smithi111 ceph-mon[106899]: pgmap v34: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:31.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:30 smithi063 ceph-mon[141124]: pgmap v35: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:30 smithi111 ceph-mon[106899]: pgmap v35: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:32.265 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:32.265 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:32.265 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:31 smithi111 ceph-mon[106899]: pgmap v36: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:32.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:32.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:32.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:31 smithi063 ceph-mon[141124]: pgmap v36: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:33.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:33 smithi111 ceph-mon[106899]: Upgrade: Updating crash.smithi111 (2/2) 2024-03-23T11:53:33.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:33.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:33.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:33.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:33 smithi111 ceph-mon[106899]: Deploying daemon crash.smithi111 on smithi111 2024-03-23T11:53:34.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:33 smithi063 ceph-mon[141124]: Upgrade: Updating crash.smithi111 (2/2) 2024-03-23T11:53:34.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:34.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-23T11:53:34.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:34.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:33 smithi063 ceph-mon[141124]: Deploying daemon crash.smithi111 on smithi111 2024-03-23T11:53:34.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:34 smithi111 ceph-mon[106899]: pgmap v37: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:35.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:34 smithi063 ceph-mon[141124]: pgmap v37: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:36.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:36 smithi063 ceph-mon[141124]: pgmap v38: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:36.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:36 smithi111 ceph-mon[106899]: pgmap v38: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:38.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:37 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:38.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:37 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:38.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:37 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:53:38.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:37 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:38.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:37 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:38.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:37 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:53:38.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:38 smithi111 ceph-mon[106899]: pgmap v39: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:38 smithi063 ceph-mon[141124]: pgmap v39: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:40 smithi063 ceph-mon[141124]: pgmap v40: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:40 smithi111 ceph-mon[106899]: pgmap v40: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:42.758 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:42.759 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:42.759 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:42 smithi063 ceph-mon[141124]: pgmap v41: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:42.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:42 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:42.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:42 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:42.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:42 smithi111 ceph-mon[106899]: pgmap v41: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:44.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:44 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:44.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:44 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:44.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:44 smithi063 ceph-mon[141124]: pgmap v42: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:44.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:44 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:44.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:44 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:44.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:44 smithi111 ceph-mon[106899]: pgmap v42: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:45.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:45 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:45.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:45 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:45.553 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:45 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:45.553 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:45 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:46.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:46 smithi063 ceph-mon[141124]: pgmap v43: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:46.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:46 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:46.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:46 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:46.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:46 smithi111 ceph-mon[106899]: pgmap v43: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:46.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:46 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:46.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:46 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:48 smithi063 ceph-mon[141124]: pgmap v44: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:49.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:48 smithi111 ceph-mon[106899]: pgmap v44: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 -- 172.21.15.63:0/1606160459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105a30 msgr2=0x7ff4080ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 --2- 172.21.15.63:0/1606160459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105a30 0x7ff4080ff0a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4004940 tx=0x7ff3f4009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 -- 172.21.15.63:0/1606160459 shutdown_connections 2024-03-23T11:53:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 --2- 172.21.15.63:0/1606160459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105a30 0x7ff4080ff0a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 --2- 172.21.15.63:0/1606160459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff408105090 0x7ff408105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 -- 172.21.15.63:0/1606160459 >> 172.21.15.63:0/1606160459 conn(0x7ff4080fac00 msgr2=0x7ff4080fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 -- 172.21.15.63:0/1606160459 shutdown_connections 2024-03-23T11:53:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.418+0000 7ff40da00700 1 -- 172.21.15.63:0/1606160459 wait complete. 2024-03-23T11:53:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.419+0000 7ff40da00700 1 Processor -- start 2024-03-23T11:53:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.419+0000 7ff40da00700 1 -- start start 2024-03-23T11:53:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff40da00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105090 0x7ff408115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff40da00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff408105a30 0x7ff408115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff40da00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff408116600 con 0x7ff408105090 2024-03-23T11:53:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff40da00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810fbe0 con 0x7ff408105a30 2024-03-23T11:53:49.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff406a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105090 0x7ff408115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:49.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff406a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105090 0x7ff408115a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43362/0 (socket says 172.21.15.63:43362) 2024-03-23T11:53:49.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff406a00700 1 -- 172.21.15.63:0/1783227911 learned_addr learned my addr 172.21.15.63:0/1783227911 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.420+0000 7ff406000700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff408105a30 0x7ff408115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff406a00700 1 -- 172.21.15.63:0/1783227911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff408105a30 msgr2=0x7ff408115fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff406a00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff408105a30 0x7ff408115fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff406a00700 1 -- 172.21.15.63:0/1783227911 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4005040 con 0x7ff408105090 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff406a00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105090 0x7ff408115a70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8002a50 tx=0x7ff3f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff3fea00700 1 -- 172.21.15.63:0/1783227911 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f8007cc0 con 0x7ff408105090 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff3fea00700 1 -- 172.21.15.63:0/1783227911 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3f800f040 con 0x7ff408105090 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40810fec0 con 0x7ff408105090 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff3fea00700 1 -- 172.21.15.63:0/1783227911 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f800d9c0 con 0x7ff408105090 2024-03-23T11:53:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.421+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff408110480 con 0x7ff408105090 2024-03-23T11:53:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.423+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4081023a0 con 0x7ff408105090 2024-03-23T11:53:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.425+0000 7ff3fea00700 1 -- 172.21.15.63:0/1783227911 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7ff3f800db20 con 0x7ff408105090 2024-03-23T11:53:49.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.426+0000 7ff3fea00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3ec06c8f0 0x7ff3ec06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:49.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.426+0000 7ff3fea00700 1 -- 172.21.15.63:0/1783227911 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff3f808ba30 con 0x7ff408105090 2024-03-23T11:53:49.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.426+0000 7ff406000700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3ec06c8f0 0x7ff3ec06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:49.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.427+0000 7ff406000700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3ec06c8f0 0x7ff3ec06eda0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff408111310 tx=0x7ff3f4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:49.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.431+0000 7ff3fea00700 1 -- 172.21.15.63:0/1783227911 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff3f80594a0 con 0x7ff408105090 2024-03-23T11:53:49.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.674+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff40810c0a0 con 0x7ff3ec06c8f0 2024-03-23T11:53:49.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.675+0000 7ff3fea00700 1 -- 172.21.15.63:0/1783227911 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff40810c0a0 con 0x7ff3ec06c8f0 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3ec06c8f0 msgr2=0x7ff3ec06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3ec06c8f0 0x7ff3ec06eda0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff408111310 tx=0x7ff3f4009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105090 msgr2=0x7ff408115a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105090 0x7ff408115a70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8002a50 tx=0x7ff3f80046b0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 shutdown_connections 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3ec06c8f0 0x7ff3ec06eda0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff408105090 0x7ff408115a70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 --2- 172.21.15.63:0/1783227911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff408105a30 0x7ff408115fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 >> 172.21.15.63:0/1783227911 conn(0x7ff4080fac00 msgr2=0x7ff4080fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 shutdown_connections 2024-03-23T11:53:49.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.678+0000 7ff40da00700 1 -- 172.21.15.63:0/1783227911 wait complete. 2024-03-23T11:53:49.676 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:53:49.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 -- 172.21.15.63:0/1592198429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c102e00 msgr2=0x7fc56c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:49.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 --2- 172.21.15.63:0/1592198429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c102e00 0x7fc56c1031d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc55c004940 tx=0x7fc55c009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:49.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 -- 172.21.15.63:0/1592198429 shutdown_connections 2024-03-23T11:53:49.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 --2- 172.21.15.63:0/1592198429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc56c1037a0 0x7fc56c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 --2- 172.21.15.63:0/1592198429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c102e00 0x7fc56c1031d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 -- 172.21.15.63:0/1592198429 >> 172.21.15.63:0/1592198429 conn(0x7fc56c0f89b0 msgr2=0x7fc56c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:49.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 -- 172.21.15.63:0/1592198429 shutdown_connections 2024-03-23T11:53:49.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.825+0000 7fc572800700 1 -- 172.21.15.63:0/1592198429 wait complete. 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.826+0000 7fc572800700 1 Processor -- start 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.826+0000 7fc572800700 1 -- start start 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.826+0000 7fc572800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc56c102e00 0x7fc56c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc572800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c1037a0 0x7fc56c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc572800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c116810 con 0x7fc56c1037a0 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc572800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c10fbe0 con 0x7fc56c102e00 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc56aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c1037a0 0x7fc56c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:49.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc56aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c1037a0 0x7fc56c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43370/0 (socket says 172.21.15.63:43370) 2024-03-23T11:53:49.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc56aa00700 1 -- 172.21.15.63:0/3074681398 learned_addr learned my addr 172.21.15.63:0/3074681398 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:49.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc56b400700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc56c102e00 0x7fc56c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:49.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc56aa00700 1 -- 172.21.15.63:0/3074681398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc56c102e00 msgr2=0x7fc56c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:49.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc56aa00700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc56c102e00 0x7fc56c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:49.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.827+0000 7fc56aa00700 1 -- 172.21.15.63:0/3074681398 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c005040 con 0x7fc56c1037a0 2024-03-23T11:53:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.828+0000 7fc56aa00700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c1037a0 0x7fc56c1161c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc55800eb40 tx=0x7fc55800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.828+0000 7fc563400700 1 -- 172.21.15.63:0/3074681398 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc558007cc0 con 0x7fc56c1037a0 2024-03-23T11:53:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.828+0000 7fc563400700 1 -- 172.21.15.63:0/3074681398 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc55800f040 con 0x7fc56c1037a0 2024-03-23T11:53:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.828+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56c10fe60 con 0x7fc56c1037a0 2024-03-23T11:53:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.828+0000 7fc563400700 1 -- 172.21.15.63:0/3074681398 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc55800c8f0 con 0x7fc56c1037a0 2024-03-23T11:53:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.828+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56c110420 con 0x7fc56c1037a0 2024-03-23T11:53:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.829+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56c0fe550 con 0x7fc56c1037a0 2024-03-23T11:53:49.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.831+0000 7fc563400700 1 -- 172.21.15.63:0/3074681398 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7fc5580090d0 con 0x7fc56c1037a0 2024-03-23T11:53:49.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.832+0000 7fc563400700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc55006c8e0 0x7fc55006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:49.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.832+0000 7fc563400700 1 -- 172.21.15.63:0/3074681398 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc55808b6a0 con 0x7fc56c1037a0 2024-03-23T11:53:49.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.832+0000 7fc56b400700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc55006c8e0 0x7fc55006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:49.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.833+0000 7fc56b400700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc55006c8e0 0x7fc55006ed90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0045a0 tx=0x7fc55c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:49.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:49.838+0000 7fc563400700 1 -- 172.21.15.63:0/3074681398 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc558059240 con 0x7fc56c1037a0 2024-03-23T11:53:50.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.074+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc56c109e60 con 0x7fc55006c8e0 2024-03-23T11:53:50.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.075+0000 7fc563400700 1 -- 172.21.15.63:0/3074681398 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc55c0096f0 con 0x7fc55006c8e0 2024-03-23T11:53:50.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc55006c8e0 msgr2=0x7fc55006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc55006c8e0 0x7fc55006ed90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0045a0 tx=0x7fc55c009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:50.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c1037a0 msgr2=0x7fc56c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c1037a0 0x7fc56c1161c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc55800eb40 tx=0x7fc55800b040 comp rx=0 tx=0).stop 2024-03-23T11:53:50.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 shutdown_connections 2024-03-23T11:53:50.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc55006c8e0 0x7fc55006ed90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc56c102e00 0x7fc56c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 --2- 172.21.15.63:0/3074681398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc56c1037a0 0x7fc56c1161c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.077+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 >> 172.21.15.63:0/3074681398 conn(0x7fc56c0f89b0 msgr2=0x7fc56c0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:50.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.078+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 shutdown_connections 2024-03-23T11:53:50.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.078+0000 7fc572800700 1 -- 172.21.15.63:0/3074681398 wait complete. 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:50.095 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:50.096 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:50.096 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.096 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi063"}]: dispatch 2024-03-23T11:53:50.096 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi063"}]': finished 2024-03-23T11:53:50.096 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]: dispatch 2024-03-23T11:53:50.096 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]': finished 2024-03-23T11:53:50.096 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-23T11:53:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:53:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi063"}]: dispatch 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi063"}]': finished 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]: dispatch 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]': finished 2024-03-23T11:53:50.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-23T11:53:50.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 -- 172.21.15.63:0/8269045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 msgr2=0x7fc7c80706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 --2- 172.21.15.63:0/8269045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c80706b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8004940 tx=0x7fc7b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:50.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 -- 172.21.15.63:0/8269045 shutdown_connections 2024-03-23T11:53:50.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 --2- 172.21.15.63:0/8269045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c8070bf0 0x7fc7c8109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 --2- 172.21.15.63:0/8269045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c80706b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 -- 172.21.15.63:0/8269045 >> 172.21.15.63:0/8269045 conn(0x7fc7c80f89b0 msgr2=0x7fc7c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:50.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 -- 172.21.15.63:0/8269045 shutdown_connections 2024-03-23T11:53:50.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.219+0000 7fc7ce600700 1 -- 172.21.15.63:0/8269045 wait complete. 2024-03-23T11:53:50.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.221+0000 7fc7ce600700 1 Processor -- start 2024-03-23T11:53:50.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.221+0000 7fc7ce600700 1 -- start start 2024-03-23T11:53:50.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.221+0000 7fc7ce600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c8115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:50.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.221+0000 7fc7ce600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c8070bf0 0x7fc7c8116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:50.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.221+0000 7fc7ce600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8116650 con 0x7fc7c80702e0 2024-03-23T11:53:50.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.221+0000 7fc7ce600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c810fc30 con 0x7fc7c8070bf0 2024-03-23T11:53:50.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c8115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:50.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c8115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43382/0 (socket says 172.21.15.63:43382) 2024-03-23T11:53:50.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c7400700 1 -- 172.21.15.63:0/3936328339 learned_addr learned my addr 172.21.15.63:0/3936328339 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:50.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c6a00700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c8070bf0 0x7fc7c8116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c7400700 1 -- 172.21.15.63:0/3936328339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c8070bf0 msgr2=0x7fc7c8116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c7400700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c8070bf0 0x7fc7c8116000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c7400700 1 -- 172.21.15.63:0/3936328339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8005040 con 0x7fc7c80702e0 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c6a00700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c8070bf0 0x7fc7c8116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.222+0000 7fc7c7400700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c8115ac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009360 tx=0x7fc7b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.223+0000 7fc7bf400700 1 -- 172.21.15.63:0/3936328339 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b800e040 con 0x7fc7c80702e0 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.223+0000 7fc7bf400700 1 -- 172.21.15.63:0/3936328339 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7b8013420 con 0x7fc7c80702e0 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.223+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c810feb0 con 0x7fc7c80702e0 2024-03-23T11:53:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.223+0000 7fc7bf400700 1 -- 172.21.15.63:0/3936328339 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b801c5f0 con 0x7fc7c80702e0 2024-03-23T11:53:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.223+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c81103f0 con 0x7fc7c80702e0 2024-03-23T11:53:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.224+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c8103c10 con 0x7fc7c80702e0 2024-03-23T11:53:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.225+0000 7fc7bf400700 1 -- 172.21.15.63:0/3936328339 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7fc7b8013590 con 0x7fc7c80702e0 2024-03-23T11:53:50.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.226+0000 7fc7bf400700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7ac06c8f0 0x7fc7ac06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:50.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.226+0000 7fc7bf400700 1 -- 172.21.15.63:0/3936328339 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc7b808cf90 con 0x7fc7c80702e0 2024-03-23T11:53:50.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.226+0000 7fc7c6a00700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7ac06c8f0 0x7fc7ac06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:50.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.227+0000 7fc7c6a00700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7ac06c8f0 0x7fc7ac06eda0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009fd0 tx=0x7fc7b4009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:50.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.230+0000 7fc7bf400700 1 -- 172.21.15.63:0/3936328339 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc7b805aa80 con 0x7fc7c80702e0 2024-03-23T11:53:50.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.458+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7c810c010 con 0x7fc7ac06c8f0 2024-03-23T11:53:50.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.468+0000 7fc7bf400700 1 -- 172.21.15.63:0/3936328339 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fc7c810c010 con 0x7fc7ac06c8f0 2024-03-23T11:53:50.452 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:53:50.452 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (3m) 7s ago 16m 14.1M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:53:50.452 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (20s) 7s ago 16m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:53:50.452 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (14s) 10s ago 15m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:53:50.452 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (2m) 7s ago 15m 73.2M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (9m) 7s ago 9m 118M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (9m) 7s ago 9m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (9m) 10s ago 9m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (9m) 10s ago 9m 60.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (5m) 7s ago 17m 492M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (4m) 10s ago 15m 450M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (95s) 7s ago 17m 40.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (64s) 10s ago 15m 36.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (3m) 7s ago 16m 7721k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (3m) 10s ago 15m 7860k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (14m) 7s ago 14m 222M 1618M 17.2.7-726-g9c4e0c34 837124da804f edfdc170d301 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (13m) 7s ago 13m 233M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (13m) 7s ago 13m 197M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:53:50.453 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (12m) 10s ago 12m 259M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:53:50.454 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (12m) 10s ago 12m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:53:50.454 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (11m) 10s ago 11m 228M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:53:50.454 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (3m) 7s ago 15m 48.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:53:50.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7ac06c8f0 msgr2=0x7fc7ac06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7ac06c8f0 0x7fc7ac06eda0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009fd0 tx=0x7fc7b4009ee0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 msgr2=0x7fc7c8115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c8115ac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009360 tx=0x7fc7b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 shutdown_connections 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7ac06c8f0 0x7fc7ac06eda0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c80702e0 0x7fc7c8115ac0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 --2- 172.21.15.63:0/3936328339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c8070bf0 0x7fc7c8116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 >> 172.21.15.63:0/3936328339 conn(0x7fc7c80f89b0 msgr2=0x7fc7c8101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 shutdown_connections 2024-03-23T11:53:50.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.470+0000 7fc7ce600700 1 -- 172.21.15.63:0/3936328339 wait complete. 2024-03-23T11:53:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 -- 172.21.15.63:0/808040197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 msgr2=0x7fb0e40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 --2- 172.21.15.63:0/808040197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e40fce30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc004940 tx=0x7fb0cc009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 -- 172.21.15.63:0/808040197 shutdown_connections 2024-03-23T11:53:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 --2- 172.21.15.63:0/808040197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e40fce30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 --2- 172.21.15.63:0/808040197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e4102e20 0x7fb0e41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 -- 172.21.15.63:0/808040197 >> 172.21.15.63:0/808040197 conn(0x7fb0e40f89d0 msgr2=0x7fb0e40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 -- 172.21.15.63:0/808040197 shutdown_connections 2024-03-23T11:53:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.607+0000 7fb0e9000700 1 -- 172.21.15.63:0/808040197 wait complete. 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.608+0000 7fb0e9000700 1 Processor -- start 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.608+0000 7fb0e9000700 1 -- start start 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0e9000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e4102e20 0x7fb0e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0e9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0e9000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e4070d90 con 0x7fb0e41037c0 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0e9000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e4070ed0 con 0x7fb0e4102e20 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0dbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0dbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43402/0 (socket says 172.21.15.63:43402) 2024-03-23T11:53:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0dbe00700 1 -- 172.21.15.63:0/2338748381 learned_addr learned my addr 172.21.15.63:0/2338748381 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0e1e00700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e4102e20 0x7fb0e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0dbe00700 1 -- 172.21.15.63:0/2338748381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e4102e20 msgr2=0x7fb0e40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0dbe00700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e4102e20 0x7fb0e40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.609+0000 7fb0dbe00700 1 -- 172.21.15.63:0/2338748381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc005040 con 0x7fb0e41037c0 2024-03-23T11:53:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.610+0000 7fb0dbe00700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e4070800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009390 tx=0x7fb0cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.610+0000 7fb0da000700 1 -- 172.21.15.63:0/2338748381 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0cc00e040 con 0x7fb0e41037c0 2024-03-23T11:53:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.610+0000 7fb0da000700 1 -- 172.21.15.63:0/2338748381 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0cc013420 con 0x7fb0e41037c0 2024-03-23T11:53:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.610+0000 7fb0da000700 1 -- 172.21.15.63:0/2338748381 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0cc01c630 con 0x7fb0e41037c0 2024-03-23T11:53:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.610+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e4071120 con 0x7fb0e41037c0 2024-03-23T11:53:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.610+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e411a170 con 0x7fb0e41037c0 2024-03-23T11:53:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.611+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e40fe570 con 0x7fb0e41037c0 2024-03-23T11:53:50.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.612+0000 7fb0da000700 1 -- 172.21.15.63:0/2338748381 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7fb0cc013a00 con 0x7fb0e41037c0 2024-03-23T11:53:50.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.613+0000 7fb0da000700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0c806c8e0 0x7fb0c806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:50.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.613+0000 7fb0da000700 1 -- 172.21.15.63:0/2338748381 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb0cc08cd30 con 0x7fb0e41037c0 2024-03-23T11:53:50.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.613+0000 7fb0e1e00700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0c806c8e0 0x7fb0c806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:50.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.614+0000 7fb0e1e00700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0c806c8e0 0x7fb0c806ed90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009fd0 tx=0x7fb0d400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:50.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.618+0000 7fb0da000700 1 -- 172.21.15.63:0/2338748381 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb0cc056e10 con 0x7fb0e41037c0 2024-03-23T11:53:50.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.934+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb0e404bed0 con 0x7fb0e41037c0 2024-03-23T11:53:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.934+0000 7fb0da000700 1 -- 172.21.15.63:0/2338748381 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fb0cc05a430 con 0x7fb0e41037c0 2024-03-23T11:53:50.918 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:53:50.918 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:53:50.918 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:53:50.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 10, 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:53:50.919 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:53:50.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.936+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0c806c8e0 msgr2=0x7fb0c806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.936+0000 7fb0e9000700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0c806c8e0 0x7fb0c806ed90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009fd0 tx=0x7fb0d400b040 comp rx=0 tx=0).stop 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.936+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 msgr2=0x7fb0e4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.936+0000 7fb0e9000700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e4070800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009390 tx=0x7fb0cc009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.937+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 shutdown_connections 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.937+0000 7fb0e9000700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0c806c8e0 0x7fb0c806ed90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.937+0000 7fb0e9000700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e4102e20 0x7fb0e40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.937+0000 7fb0e9000700 1 --2- 172.21.15.63:0/2338748381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0e41037c0 0x7fb0e4070800 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.937+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 >> 172.21.15.63:0/2338748381 conn(0x7fb0e40f89d0 msgr2=0x7fb0e40f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.937+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 shutdown_connections 2024-03-23T11:53:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:50.937+0000 7fb0e9000700 1 -- 172.21.15.63:0/2338748381 wait complete. 2024-03-23T11:53:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 -- 172.21.15.63:0/160318375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 msgr2=0x7fa234070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 --2- 172.21.15.63:0/160318375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234070730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa228004970 tx=0x7fa228009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 -- 172.21.15.63:0/160318375 shutdown_connections 2024-03-23T11:53:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 --2- 172.21.15.63:0/160318375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa234070c70 0x7fa234109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 --2- 172.21.15.63:0/160318375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234070730 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 -- 172.21.15.63:0/160318375 >> 172.21.15.63:0/160318375 conn(0x7fa2340f89b0 msgr2=0x7fa2340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 -- 172.21.15.63:0/160318375 shutdown_connections 2024-03-23T11:53:51.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.078+0000 7fa23b400700 1 -- 172.21.15.63:0/160318375 wait complete. 2024-03-23T11:53:51.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.079+0000 7fa23b400700 1 Processor -- start 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.079+0000 7fa23b400700 1 -- start start 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.079+0000 7fa23b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.079+0000 7fa23b400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa234070c70 0x7fa2341161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.079+0000 7fa23b400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234116810 con 0x7fa234070360 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.079+0000 7fa23b400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23410fbe0 con 0x7fa234070c70 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.080+0000 7fa238a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.080+0000 7fa238a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43426/0 (socket says 172.21.15.63:43426) 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.080+0000 7fa238a00700 1 -- 172.21.15.63:0/1230057108 learned_addr learned my addr 172.21.15.63:0/1230057108 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:51.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.080+0000 7fa233e00700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa234070c70 0x7fa2341161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.080+0000 7fa238a00700 1 -- 172.21.15.63:0/1230057108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa234070c70 msgr2=0x7fa2341161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.080+0000 7fa238a00700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa234070c70 0x7fa2341161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.080+0000 7fa238a00700 1 -- 172.21.15.63:0/1230057108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228005040 con 0x7fa234070360 2024-03-23T11:53:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.081+0000 7fa238a00700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234115c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa228005010 tx=0x7fa228009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.081+0000 7fa230c00700 1 -- 172.21.15.63:0/1230057108 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa22800e040 con 0x7fa234070360 2024-03-23T11:53:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.081+0000 7fa230c00700 1 -- 172.21.15.63:0/1230057108 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa228004290 con 0x7fa234070360 2024-03-23T11:53:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.081+0000 7fa230c00700 1 -- 172.21.15.63:0/1230057108 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa228013d00 con 0x7fa234070360 2024-03-23T11:53:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.081+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23410fe60 con 0x7fa234070360 2024-03-23T11:53:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.081+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2341103a0 con 0x7fa234070360 2024-03-23T11:53:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.083+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa234103b90 con 0x7fa234070360 2024-03-23T11:53:51.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.084+0000 7fa230c00700 1 -- 172.21.15.63:0/1230057108 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7fa2280134b0 con 0x7fa234070360 2024-03-23T11:53:51.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.085+0000 7fa230c00700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa21c06c8e0 0x7fa21c06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.085+0000 7fa230c00700 1 -- 172.21.15.63:0/1230057108 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa22808cde0 con 0x7fa234070360 2024-03-23T11:53:51.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.086+0000 7fa233e00700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa21c06c8e0 0x7fa21c06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.086+0000 7fa233e00700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa21c06c8e0 0x7fa21c06ed90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa234111520 tx=0x7fa22000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:51.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.090+0000 7fa230c00700 1 -- 172.21.15.63:0/1230057108 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa22805a910 con 0x7fa234070360 2024-03-23T11:53:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:50 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all crash 2024-03-23T11:53:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:50 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-23T11:53:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:50 smithi063 ceph-mon[141124]: Upgrade: osd.0 is safe to restart 2024-03-23T11:53:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:50 smithi063 ceph-mon[141124]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:50 smithi063 ceph-mon[141124]: pgmap v45: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:50 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all crash 2024-03-23T11:53:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:50 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-23T11:53:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:50 smithi111 ceph-mon[106899]: Upgrade: osd.0 is safe to restart 2024-03-23T11:53:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:50 smithi111 ceph-mon[106899]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:50 smithi111 ceph-mon[106899]: pgmap v45: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:51.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.370+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa23410bfb0 con 0x7fa234070360 2024-03-23T11:53:51.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.370+0000 7fa230c00700 1 -- 172.21.15.63:0/1230057108 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7fa22805a4a0 con 0x7fa234070360 2024-03-23T11:53:51.354 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:53:51.355 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:53:51.356 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:53:51.357 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:53:51.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa21c06c8e0 msgr2=0x7fa21c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa21c06c8e0 0x7fa21c06ed90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa234111520 tx=0x7fa22000b040 comp rx=0 tx=0).stop 2024-03-23T11:53:51.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 msgr2=0x7fa234115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234115c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa228005010 tx=0x7fa228009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:51.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 shutdown_connections 2024-03-23T11:53:51.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa21c06c8e0 0x7fa21c06ed90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa234070360 0x7fa234115c80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 --2- 172.21.15.63:0/1230057108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa234070c70 0x7fa2341161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 >> 172.21.15.63:0/1230057108 conn(0x7fa2340f89b0 msgr2=0x7fa2341015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:51.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 shutdown_connections 2024-03-23T11:53:51.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.373+0000 7fa23b400700 1 -- 172.21.15.63:0/1230057108 wait complete. 2024-03-23T11:53:51.359 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.516+0000 7ff996600700 1 -- 172.21.15.63:0/2578304146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 msgr2=0x7ff9900fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.516+0000 7ff996600700 1 --2- 172.21.15.63:0/2578304146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff9900fce30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff97c004940 tx=0x7ff97c009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.516+0000 7ff996600700 1 -- 172.21.15.63:0/2578304146 shutdown_connections 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.516+0000 7ff996600700 1 --2- 172.21.15.63:0/2578304146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff9900fce30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.516+0000 7ff996600700 1 --2- 172.21.15.63:0/2578304146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff990102e20 0x7ff9901031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.516+0000 7ff996600700 1 -- 172.21.15.63:0/2578304146 >> 172.21.15.63:0/2578304146 conn(0x7ff9900f89d0 msgr2=0x7ff9900fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.517+0000 7ff996600700 1 -- 172.21.15.63:0/2578304146 shutdown_connections 2024-03-23T11:53:51.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.517+0000 7ff996600700 1 -- 172.21.15.63:0/2578304146 wait complete. 2024-03-23T11:53:51.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.517+0000 7ff996600700 1 Processor -- start 2024-03-23T11:53:51.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff996600700 1 -- start start 2024-03-23T11:53:51.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff996600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff990102e20 0x7ff990113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff996600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff990113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff996600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9901145d0 con 0x7ff9901037c0 2024-03-23T11:53:51.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff996600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99010d9c0 con 0x7ff990102e20 2024-03-23T11:53:51.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff98ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff990113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff98ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff990113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50666/0 (socket says 172.21.15.63:50666) 2024-03-23T11:53:51.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff98ea00700 1 -- 172.21.15.63:0/3004473013 learned_addr learned my addr 172.21.15.63:0/3004473013 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:51.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.518+0000 7ff98f400700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff990102e20 0x7ff990113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.519+0000 7ff98ea00700 1 -- 172.21.15.63:0/3004473013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff990102e20 msgr2=0x7ff990113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.519+0000 7ff98ea00700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff990102e20 0x7ff990113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.519+0000 7ff98ea00700 1 -- 172.21.15.63:0/3004473013 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c005040 con 0x7ff9901037c0 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.519+0000 7ff98ea00700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff990113f80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00b0d0 tx=0x7ff97c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.519+0000 7ff987400700 1 -- 172.21.15.63:0/3004473013 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff97c021070 con 0x7ff9901037c0 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.519+0000 7ff987400700 1 -- 172.21.15.63:0/3004473013 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff97c013e00 con 0x7ff9901037c0 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.519+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99010dc40 con 0x7ff9901037c0 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.520+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99010e1a0 con 0x7ff9901037c0 2024-03-23T11:53:51.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.520+0000 7ff987400700 1 -- 172.21.15.63:0/3004473013 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff97c00e070 con 0x7ff9901037c0 2024-03-23T11:53:51.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.521+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9900fe570 con 0x7ff9901037c0 2024-03-23T11:53:51.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.522+0000 7ff987400700 1 -- 172.21.15.63:0/3004473013 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7ff97c013950 con 0x7ff9901037c0 2024-03-23T11:53:51.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.523+0000 7ff987400700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff97406c8e0 0x7ff97406ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.523+0000 7ff987400700 1 -- 172.21.15.63:0/3004473013 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff97c08ccd0 con 0x7ff9901037c0 2024-03-23T11:53:51.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.523+0000 7ff98f400700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff97406c8e0 0x7ff97406ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.524+0000 7ff98f400700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff97406c8e0 0x7ff97406ed90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff980009b80 tx=0x7ff98000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:51.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.527+0000 7ff987400700 1 -- 172.21.15.63:0/3004473013 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff97c05a7c0 con 0x7ff9901037c0 2024-03-23T11:53:51.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.768+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff990109dc0 con 0x7ff97406c8e0 2024-03-23T11:53:51.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.770+0000 7ff987400700 1 -- 172.21.15.63:0/3004473013 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff990109dc0 con 0x7ff97406c8e0 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "6/21 daemons upgraded", 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading osd daemons", 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:53:51.755 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:53:51.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.772+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff97406c8e0 msgr2=0x7ff97406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.772+0000 7ff996600700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff97406c8e0 0x7ff97406ed90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff980009b80 tx=0x7ff98000b040 comp rx=0 tx=0).stop 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.772+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 msgr2=0x7ff990113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.772+0000 7ff996600700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff990113f80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00b0d0 tx=0x7ff97c009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.773+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 shutdown_connections 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.773+0000 7ff996600700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff97406c8e0 0x7ff97406ed90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.773+0000 7ff996600700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff990102e20 0x7ff990113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.773+0000 7ff996600700 1 --2- 172.21.15.63:0/3004473013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9901037c0 0x7ff990113f80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.773+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 >> 172.21.15.63:0/3004473013 conn(0x7ff9900f89d0 msgr2=0x7ff9900f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.773+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 shutdown_connections 2024-03-23T11:53:51.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.773+0000 7ff996600700 1 -- 172.21.15.63:0/3004473013 wait complete. 2024-03-23T11:53:51.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 -- 172.21.15.63:0/2355821690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 msgr2=0x7f62500fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 --2- 172.21.15.63:0/2355821690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f62500fce30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f623c004940 tx=0x7f623c009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:51.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 -- 172.21.15.63:0/2355821690 shutdown_connections 2024-03-23T11:53:51.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 --2- 172.21.15.63:0/2355821690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f62500fce30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 --2- 172.21.15.63:0/2355821690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6250102e20 0x7f62501031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 -- 172.21.15.63:0/2355821690 >> 172.21.15.63:0/2355821690 conn(0x7f62500f89d0 msgr2=0x7f62500fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:51.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 -- 172.21.15.63:0/2355821690 shutdown_connections 2024-03-23T11:53:51.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.914+0000 7f6256600700 1 -- 172.21.15.63:0/2355821690 wait complete. 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.915+0000 7f6256600700 1 Processor -- start 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.915+0000 7f6256600700 1 -- start start 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.915+0000 7f6256600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6250102e20 0x7f6250115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.915+0000 7f6256600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f6250116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.915+0000 7f6256600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250116890 con 0x7f62501037c0 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.915+0000 7f6256600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625010fc60 con 0x7f6250102e20 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f6250116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f6250116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50680/0 (socket says 172.21.15.63:50680) 2024-03-23T11:53:51.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624ea00700 1 -- 172.21.15.63:0/3329554137 learned_addr learned my addr 172.21.15.63:0/3329554137 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:53:51.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624f400700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6250102e20 0x7f6250115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624ea00700 1 -- 172.21.15.63:0/3329554137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6250102e20 msgr2=0x7f6250115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:51.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624ea00700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6250102e20 0x7f6250115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:51.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624ea00700 1 -- 172.21.15.63:0/3329554137 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c005040 con 0x7f62501037c0 2024-03-23T11:53:51.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.916+0000 7f624ea00700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f6250116240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f623c009390 tx=0x7f623c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:51.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.917+0000 7f6247400700 1 -- 172.21.15.63:0/3329554137 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623c00e040 con 0x7f62501037c0 2024-03-23T11:53:51.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.917+0000 7f6247400700 1 -- 172.21.15.63:0/3329554137 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f623c013420 con 0x7f62501037c0 2024-03-23T11:53:51.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.917+0000 7f6247400700 1 -- 172.21.15.63:0/3329554137 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623c01c630 con 0x7f62501037c0 2024-03-23T11:53:51.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.917+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625010fee0 con 0x7f62501037c0 2024-03-23T11:53:51.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.917+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6250110450 con 0x7f62501037c0 2024-03-23T11:53:51.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.918+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62500fe570 con 0x7f62501037c0 2024-03-23T11:53:51.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.919+0000 7f6247400700 1 -- 172.21.15.63:0/3329554137 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 38) v1 ==== 90626+0+0 (secure 0 0 0) 0x7f623c013a00 con 0x7f62501037c0 2024-03-23T11:53:51.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.920+0000 7f6247400700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623406c8e0 0x7f623406ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:53:51.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.920+0000 7f6247400700 1 -- 172.21.15.63:0/3329554137 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f623c08ce30 con 0x7f62501037c0 2024-03-23T11:53:51.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.920+0000 7f624f400700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623406c8e0 0x7f623406ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:53:51.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.921+0000 7f624f400700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623406c8e0 0x7f623406ed90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6240004f40 tx=0x7f624000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:53:51.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:51.924+0000 7f6247400700 1 -- 172.21.15.63:0/3329554137 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f623c05a920 con 0x7f62501037c0 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: Upgrade: Updating osd.0 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: Deploying daemon osd.0 on smithi063 2024-03-23T11:53:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2338748381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:52.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:51 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1230057108' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: Upgrade: Updating osd.0 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: Deploying daemon osd.0 on smithi063 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2338748381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:53:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:51 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1230057108' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:53:52.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.230+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6250109dc0 con 0x7f62501037c0 2024-03-23T11:53:52.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.230+0000 7f6247400700 1 -- 172.21.15.63:0/3329554137 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f623c05a4b0 con 0x7f62501037c0 2024-03-23T11:53:52.214 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623406c8e0 msgr2=0x7f623406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623406c8e0 0x7f623406ed90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6240004f40 tx=0x7f624000b040 comp rx=0 tx=0).stop 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 msgr2=0x7f6250116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f6250116240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f623c009390 tx=0x7f623c009e30 comp rx=0 tx=0).stop 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 shutdown_connections 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623406c8e0 0x7f623406ed90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6250102e20 0x7f6250115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 --2- 172.21.15.63:0/3329554137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62501037c0 0x7f6250116240 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 >> 172.21.15.63:0/3329554137 conn(0x7f62500f89d0 msgr2=0x7f62500f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:53:52.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 shutdown_connections 2024-03-23T11:53:52.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:53:52.233+0000 7f6256600700 1 -- 172.21.15.63:0/3329554137 wait complete. 2024-03-23T11:53:53.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:52 smithi063 ceph-mon[141124]: pgmap v46: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:53.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:52 smithi063 ceph-mon[141124]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:53.034 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:52 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3329554137' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:53:53.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:52 smithi111 ceph-mon[106899]: pgmap v46: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:53:53.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:52 smithi111 ceph-mon[106899]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:53:53.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:52 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3329554137' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:53:55.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:54 smithi063 ceph-mon[141124]: pgmap v47: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:55.025 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:54 smithi063 systemd[1]: Stopping Ceph osd.0 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:53:55.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:54 smithi111 ceph-mon[106899]: pgmap v47: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:55.382 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:55 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[58550]: 2024-03-23T11:53:55.039+0000 7f85e3400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:53:55.382 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:55 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[58550]: 2024-03-23T11:53:55.040+0000 7f85e3400700 -1 osd.0 49 *** Got signal Terminated *** 2024-03-23T11:53:55.382 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:55 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[58550]: 2024-03-23T11:53:55.040+0000 7f85e3400700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T11:53:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:55 smithi063 ceph-mon[141124]: osd.0 marked itself down and dead 2024-03-23T11:53:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:55 smithi111 ceph-mon[106899]: osd.0 marked itself down and dead 2024-03-23T11:53:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:53:57.132 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:56 smithi063 bash[154396]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0 2024-03-23T11:53:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:56 smithi063 ceph-mon[141124]: pgmap v48: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:56 smithi063 ceph-mon[141124]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:53:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:56 smithi063 ceph-mon[141124]: osdmap e50: 6 total, 5 up, 6 in 2024-03-23T11:53:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:56 smithi111 ceph-mon[106899]: pgmap v48: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:53:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:56 smithi111 ceph-mon[106899]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:53:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:56 smithi111 ceph-mon[106899]: osdmap e50: 6 total, 5 up, 6 in 2024-03-23T11:53:58.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:57 smithi063 ceph-mon[141124]: osdmap e51: 6 total, 5 up, 6 in 2024-03-23T11:53:58.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:57 smithi111 ceph-mon[106899]: osdmap e51: 6 total, 5 up, 6 in 2024-03-23T11:53:59.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:58 smithi063 ceph-mon[141124]: pgmap v51: 65 pgs: 34 peering, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 639 B/s wr, 2 op/s 2024-03-23T11:53:59.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:58 smithi063 ceph-mon[141124]: Health check failed: Reduced data availability: 33 pgs peering (PG_AVAILABILITY) 2024-03-23T11:53:59.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:58 smithi063 ceph-mon[141124]: mgrmap e39: smithi063.gehkcr(active, since 92s), standbys: smithi111.nnblsk 2024-03-23T11:53:59.132 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:58 smithi063 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.0.service: Succeeded. 2024-03-23T11:53:59.133 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:58 smithi063 systemd[1]: Stopped Ceph osd.0 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:53:59.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:58 smithi111 ceph-mon[106899]: pgmap v51: 65 pgs: 34 peering, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 639 B/s wr, 2 op/s 2024-03-23T11:53:59.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:58 smithi111 ceph-mon[106899]: Health check failed: Reduced data availability: 33 pgs peering (PG_AVAILABILITY) 2024-03-23T11:53:59.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:58 smithi111 ceph-mon[106899]: mgrmap e39: smithi063.gehkcr(active, since 92s), standbys: smithi111.nnblsk 2024-03-23T11:53:59.632 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:53:59 smithi063 systemd[1]: Starting Ceph osd.0 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:54:00.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:53:59 smithi063 ceph-mon[141124]: pgmap v52: 65 pgs: 34 peering, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 639 B/s wr, 0 op/s 2024-03-23T11:54:00.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:53:59 smithi111 ceph-mon[106899]: pgmap v52: 65 pgs: 34 peering, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 639 B/s wr, 0 op/s 2024-03-23T11:54:00.882 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:00 smithi063 bash[154666]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-23T11:54:00.882 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:00 smithi063 bash[154666]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-23T11:54:00.882 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:00 smithi063 bash[154666]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-23T11:54:00.882 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:00 smithi063 bash[154666]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-23T11:54:00.883 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:00 smithi063 bash[154666]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-23T11:54:00.883 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:00 smithi063 bash[154666]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-23T11:54:00.883 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:00 smithi063 bash[154666]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-23T11:54:02.075 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:01 smithi063 ceph-mon[141124]: Health check failed: Degraded data redundancy: 65/330 objects degraded (19.697%), 19 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:02.075 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:01 smithi063 ceph-mon[141124]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 33 pgs peering) 2024-03-23T11:54:02.075 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:01 smithi063 ceph-osd[154959]: -- 172.21.15.63:0/1807778338 <== mon.0 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56267f75f1e0 con 0x56267f72ec00 2024-03-23T11:54:02.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:01 smithi111 ceph-mon[106899]: Health check failed: Degraded data redundancy: 65/330 objects degraded (19.697%), 19 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:02.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:01 smithi111 ceph-mon[106899]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 33 pgs peering) 2024-03-23T11:54:02.382 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:02 smithi063 bash[154921]: c0dca2307f3c3117ffcffb308a70e28295460f479b4c5c74620aa1a3dff208b5 2024-03-23T11:54:02.382 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:02 smithi063 systemd[1]: Started Ceph osd.0 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:54:02.795 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:02 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[154936]: 2024-03-23T11:54:02.523+0000 7f95b65de640 -1 Falling back to public interface 2024-03-23T11:54:03.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:02 smithi063 ceph-mon[141124]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 639 B/s wr, 2 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:03.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:03.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:03.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:03.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:02 smithi111 ceph-mon[106899]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 639 B/s wr, 2 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:03.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:03.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:03.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:05.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:04 smithi063 ceph-mon[141124]: pgmap v54: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 639 B/s wr, 2 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:05.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:04 smithi111 ceph-mon[106899]: pgmap v54: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 639 B/s wr, 2 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:06.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:06 smithi063 ceph-mon[141124]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 519 B/s wr, 1 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:06.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:06 smithi111 ceph-mon[106899]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 519 B/s wr, 1 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:07.632 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:07 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[154936]: 2024-03-23T11:54:07.361+0000 7f95b65de640 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-23T11:54:08.132 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:07 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[154936]: 2024-03-23T11:54:07.871+0000 7f95b65de640 -1 osd.0 49 log_to_monitors true 2024-03-23T11:54:09.132 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 11:54:08 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[154936]: 2024-03-23T11:54:08.781+0000 7f95a0a00700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:54:09.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:08 smithi063 ceph-mon[141124]: pgmap v56: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:09.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:08 smithi063 ceph-mon[141124]: from='osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-23T11:54:09.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:09.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:08 smithi111 ceph-mon[106899]: pgmap v56: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:08 smithi111 ceph-mon[106899]: from='osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-23T11:54:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:10.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:09 smithi063 ceph-mon[141124]: from='osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-23T11:54:10.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:09 smithi063 ceph-mon[141124]: osdmap e52: 6 total, 5 up, 6 in 2024-03-23T11:54:10.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:09 smithi063 ceph-mon[141124]: from='osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:54:10.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:09 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:10.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:09 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:10.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:09 smithi111 ceph-mon[106899]: from='osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-23T11:54:10.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:09 smithi111 ceph-mon[106899]: osdmap e52: 6 total, 5 up, 6 in 2024-03-23T11:54:10.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:09 smithi111 ceph-mon[106899]: from='osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:54:10.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:09 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:10.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:09 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:11.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:10 smithi063 ceph-mon[141124]: pgmap v58: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:11.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:10 smithi063 ceph-mon[141124]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:54:11.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:10 smithi063 ceph-mon[141124]: osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827] boot 2024-03-23T11:54:11.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:10 smithi063 ceph-mon[141124]: osdmap e53: 6 total, 6 up, 6 in 2024-03-23T11:54:11.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:54:11.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:11.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:10 smithi111 ceph-mon[106899]: pgmap v58: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s; 65/330 objects degraded (19.697%) 2024-03-23T11:54:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:10 smithi111 ceph-mon[106899]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:54:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:10 smithi111 ceph-mon[106899]: osd.0 [v2:172.21.15.63:6802/2164791827,v1:172.21.15.63:6803/2164791827] boot 2024-03-23T11:54:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:10 smithi111 ceph-mon[106899]: osdmap e53: 6 total, 6 up, 6 in 2024-03-23T11:54:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-23T11:54:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:12.082 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:11 smithi063 ceph-mon[141124]: osdmap e54: 6 total, 6 up, 6 in 2024-03-23T11:54:12.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:11 smithi111 ceph-mon[106899]: osdmap e54: 6 total, 6 up, 6 in 2024-03-23T11:54:13.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:12 smithi063 ceph-mon[141124]: pgmap v61: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.2 KiB/s rd, 6 op/s; 11/330 objects degraded (3.333%) 2024-03-23T11:54:13.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:12 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 11/330 objects degraded (3.333%), 3 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:13.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:12 smithi111 ceph-mon[106899]: pgmap v61: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.2 KiB/s rd, 6 op/s; 11/330 objects degraded (3.333%) 2024-03-23T11:54:13.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:12 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 11/330 objects degraded (3.333%), 3 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:14.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: pgmap v62: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 11/330 objects degraded (3.333%) 2024-03-23T11:54:14.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:14.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:14 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:15.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: pgmap v62: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 11/330 objects degraded (3.333%) 2024-03-23T11:54:15.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:15.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:15.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:54:15.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:54:15.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:15.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:15.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:15.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:15.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:15.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:14 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:16.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:15 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:16.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:15 smithi063 ceph-mon[141124]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-23T11:54:16.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:15 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:16.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:15 smithi111 ceph-mon[106899]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-23T11:54:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:16 smithi063 ceph-mon[141124]: pgmap v63: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-23T11:54:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:16 smithi063 ceph-mon[141124]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/330 objects degraded (3.333%), 3 pgs degraded) 2024-03-23T11:54:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:16 smithi063 ceph-mon[141124]: Cluster is now healthy 2024-03-23T11:54:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:16 smithi111 ceph-mon[106899]: pgmap v63: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-23T11:54:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:16 smithi111 ceph-mon[106899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/330 objects degraded (3.333%), 3 pgs degraded) 2024-03-23T11:54:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:16 smithi111 ceph-mon[106899]: Cluster is now healthy 2024-03-23T11:54:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:18 smithi063 ceph-mon[141124]: pgmap v64: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:54:19.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:18 smithi111 ceph-mon[106899]: pgmap v64: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-23T11:54:21.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:20 smithi063 ceph-mon[141124]: pgmap v65: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-23T11:54:21.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:21.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:20 smithi111 ceph-mon[106899]: pgmap v65: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-23T11:54:21.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:22.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 -- 172.21.15.63:0/780531109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3200708a0 msgr2=0x7fb320070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:22.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 --2- 172.21.15.63:0/780531109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3200708a0 0x7fb320070d10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb30c004970 tx=0x7fb30c009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:22.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 -- 172.21.15.63:0/780531109 shutdown_connections 2024-03-23T11:54:22.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 --2- 172.21.15.63:0/780531109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3200708a0 0x7fb320070d10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 --2- 172.21.15.63:0/780531109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb320071f90 0x7fb320070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 -- 172.21.15.63:0/780531109 >> 172.21.15.63:0/780531109 conn(0x7fb3200f89b0 msgr2=0x7fb3200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:22.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 -- 172.21.15.63:0/780531109 shutdown_connections 2024-03-23T11:54:22.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.401+0000 7fb325c00700 1 -- 172.21.15.63:0/780531109 wait complete. 2024-03-23T11:54:22.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.402+0000 7fb325c00700 1 Processor -- start 2024-03-23T11:54:22.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.402+0000 7fb325c00700 1 -- start start 2024-03-23T11:54:22.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.402+0000 7fb325c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3200708a0 0x7fb32011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.402+0000 7fb325c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb320071f90 0x7fb32011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.402+0000 7fb325c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32011ac40 con 0x7fb320071f90 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.402+0000 7fb325c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb320114010 con 0x7fb3200708a0 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.403+0000 7fb31e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb320071f90 0x7fb32011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.403+0000 7fb31e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb320071f90 0x7fb32011a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43732/0 (socket says 172.21.15.63:43732) 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.403+0000 7fb31e000700 1 -- 172.21.15.63:0/2789030871 learned_addr learned my addr 172.21.15.63:0/2789030871 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.403+0000 7fb31ea00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3200708a0 0x7fb32011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.403+0000 7fb31e000700 1 -- 172.21.15.63:0/2789030871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3200708a0 msgr2=0x7fb32011a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.403+0000 7fb31e000700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3200708a0 0x7fb32011a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.403+0000 7fb31e000700 1 -- 172.21.15.63:0/2789030871 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c005040 con 0x7fb320071f90 2024-03-23T11:54:22.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.404+0000 7fb31e000700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb320071f90 0x7fb32011a5f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0043d0 tx=0x7fb30c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:22.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.404+0000 7fb316a00700 1 -- 172.21.15.63:0/2789030871 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb30c00e040 con 0x7fb320071f90 2024-03-23T11:54:22.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.404+0000 7fb316a00700 1 -- 172.21.15.63:0/2789030871 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb30c013420 con 0x7fb320071f90 2024-03-23T11:54:22.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.404+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb320114290 con 0x7fb320071f90 2024-03-23T11:54:22.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.404+0000 7fb316a00700 1 -- 172.21.15.63:0/2789030871 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb30c01c630 con 0x7fb320071f90 2024-03-23T11:54:22.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.404+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3201147d0 con 0x7fb320071f90 2024-03-23T11:54:22.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.405+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb320071aa0 con 0x7fb320071f90 2024-03-23T11:54:22.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.406+0000 7fb316a00700 1 -- 172.21.15.63:0/2789030871 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb30c0139e0 con 0x7fb320071f90 2024-03-23T11:54:22.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.407+0000 7fb316a00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb30406c7c0 0x7fb30406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:22.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.407+0000 7fb316a00700 1 -- 172.21.15.63:0/2789030871 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb30c08cbc0 con 0x7fb320071f90 2024-03-23T11:54:22.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.408+0000 7fb31ea00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb30406c7c0 0x7fb30406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:22.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.408+0000 7fb31ea00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb30406c7c0 0x7fb30406ec70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb310007fd0 tx=0x7fb310009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:22.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.413+0000 7fb316a00700 1 -- 172.21.15.63:0/2789030871 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb30c05aa10 con 0x7fb320071f90 2024-03-23T11:54:22.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.646+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3201103e0 con 0x7fb30406c7c0 2024-03-23T11:54:22.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.647+0000 7fb316a00700 1 -- 172.21.15.63:0/2789030871 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb3201103e0 con 0x7fb30406c7c0 2024-03-23T11:54:22.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb30406c7c0 msgr2=0x7fb30406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb30406c7c0 0x7fb30406ec70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb310007fd0 tx=0x7fb310009040 comp rx=0 tx=0).stop 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb320071f90 msgr2=0x7fb32011a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb320071f90 0x7fb32011a5f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0043d0 tx=0x7fb30c009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 shutdown_connections 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb30406c7c0 0x7fb30406ec70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3200708a0 0x7fb32011a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 --2- 172.21.15.63:0/2789030871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb320071f90 0x7fb32011a5f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 >> 172.21.15.63:0/2789030871 conn(0x7fb3200f89b0 msgr2=0x7fb3200f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:22.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 shutdown_connections 2024-03-23T11:54:22.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.650+0000 7fb325c00700 1 -- 172.21.15.63:0/2789030871 wait complete. 2024-03-23T11:54:22.648 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:54:22.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 -- 172.21.15.63:0/913853999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fe830 msgr2=0x7ff838106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 --2- 172.21.15.63:0/913853999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fe830 0x7ff838106f00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff820004940 tx=0x7ff820009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 -- 172.21.15.63:0/913853999 shutdown_connections 2024-03-23T11:54:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 --2- 172.21.15.63:0/913853999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fe830 0x7ff838106f00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 --2- 172.21.15.63:0/913853999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8380fde90 0x7ff8380fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 -- 172.21.15.63:0/913853999 >> 172.21.15.63:0/913853999 conn(0x7ff838072bc0 msgr2=0x7ff838072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 -- 172.21.15.63:0/913853999 shutdown_connections 2024-03-23T11:54:22.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.787+0000 7ff83da00700 1 -- 172.21.15.63:0/913853999 wait complete. 2024-03-23T11:54:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.788+0000 7ff83da00700 1 Processor -- start 2024-03-23T11:54:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.788+0000 7ff83da00700 1 -- start start 2024-03-23T11:54:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff83da00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fde90 0x7ff838115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff83da00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8380fe830 0x7ff8381161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff83da00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff838116820 con 0x7ff8380fde90 2024-03-23T11:54:22.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff83da00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83810fbf0 con 0x7ff8380fe830 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff836a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fde90 0x7ff838115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff836a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fde90 0x7ff838115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43758/0 (socket says 172.21.15.63:43758) 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff836a00700 1 -- 172.21.15.63:0/186398601 learned_addr learned my addr 172.21.15.63:0/186398601 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff82fe00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8380fe830 0x7ff8381161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff836a00700 1 -- 172.21.15.63:0/186398601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8380fe830 msgr2=0x7ff8381161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff836a00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8380fe830 0x7ff8381161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.789+0000 7ff836a00700 1 -- 172.21.15.63:0/186398601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff820005040 con 0x7ff8380fde90 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.790+0000 7ff836a00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fde90 0x7ff838115c90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff82800eb70 tx=0x7ff82800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.790+0000 7ff82ea00700 1 -- 172.21.15.63:0/186398601 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff828007cc0 con 0x7ff8380fde90 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.790+0000 7ff82ea00700 1 -- 172.21.15.63:0/186398601 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff82800f040 con 0x7ff8380fde90 2024-03-23T11:54:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.790+0000 7ff82ea00700 1 -- 172.21.15.63:0/186398601 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8280095f0 con 0x7ff8380fde90 2024-03-23T11:54:22.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.790+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83810fe70 con 0x7ff8380fde90 2024-03-23T11:54:22.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.790+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8381103b0 con 0x7ff8380fde90 2024-03-23T11:54:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.791+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff838106a20 con 0x7ff8380fde90 2024-03-23T11:54:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.792+0000 7ff82ea00700 1 -- 172.21.15.63:0/186398601 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff828009750 con 0x7ff8380fde90 2024-03-23T11:54:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.793+0000 7ff82ea00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff81c06c7d0 0x7ff81c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.793+0000 7ff82ea00700 1 -- 172.21.15.63:0/186398601 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff82808b5c0 con 0x7ff8380fde90 2024-03-23T11:54:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.793+0000 7ff82fe00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff81c06c7d0 0x7ff81c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:22.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.794+0000 7ff82fe00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff81c06c7d0 0x7ff81c06ec80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff82000b0d0 tx=0x7ff820009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:22.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:22.797+0000 7ff82ea00700 1 -- 172.21.15.63:0/186398601 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff828059370 con 0x7ff8380fde90 2024-03-23T11:54:23.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.033+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff838109e60 con 0x7ff81c06c7d0 2024-03-23T11:54:23.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.034+0000 7ff82ea00700 1 -- 172.21.15.63:0/186398601 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff838109e60 con 0x7ff81c06c7d0 2024-03-23T11:54:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff81c06c7d0 msgr2=0x7ff81c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff81c06c7d0 0x7ff81c06ec80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff82000b0d0 tx=0x7ff820009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fde90 msgr2=0x7ff838115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fde90 0x7ff838115c90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff82800eb70 tx=0x7ff82800b040 comp rx=0 tx=0).stop 2024-03-23T11:54:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 shutdown_connections 2024-03-23T11:54:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff81c06c7d0 0x7ff81c06ec80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8380fde90 0x7ff838115c90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 --2- 172.21.15.63:0/186398601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8380fe830 0x7ff8381161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 >> 172.21.15.63:0/186398601 conn(0x7ff838072bc0 msgr2=0x7ff8380fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:23.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 shutdown_connections 2024-03-23T11:54:23.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.037+0000 7ff83da00700 1 -- 172.21.15.63:0/186398601 wait complete. 2024-03-23T11:54:23.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:22 smithi063 ceph-mon[141124]: pgmap v66: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-23T11:54:23.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:22 smithi111 ceph-mon[106899]: pgmap v66: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-23T11:54:23.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 -- 172.21.15.63:0/2773715124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 msgr2=0x7fc6d81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/2773715124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d81031f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8004940 tx=0x7fc6c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:23.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 -- 172.21.15.63:0/2773715124 shutdown_connections 2024-03-23T11:54:23.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/2773715124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d81037c0 0x7fc6d80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/2773715124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d81031f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 -- 172.21.15.63:0/2773715124 >> 172.21.15.63:0/2773715124 conn(0x7fc6d80f89d0 msgr2=0x7fc6d80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:23.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 -- 172.21.15.63:0/2773715124 shutdown_connections 2024-03-23T11:54:23.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.174+0000 7fc6dcc00700 1 -- 172.21.15.63:0/2773715124 wait complete. 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.175+0000 7fc6dcc00700 1 Processor -- start 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.175+0000 7fc6dcc00700 1 -- start start 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6dcc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6dcc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d81037c0 0x7fc6d8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6dcc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d8116880 con 0x7fc6d8102e20 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6dcc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d810fc50 con 0x7fc6d81037c0 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6d5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6d5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43780/0 (socket says 172.21.15.63:43780) 2024-03-23T11:54:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6d5e00700 1 -- 172.21.15.63:0/1461235005 learned_addr learned my addr 172.21.15.63:0/1461235005 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6d5400700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d81037c0 0x7fc6d8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6d5e00700 1 -- 172.21.15.63:0/1461235005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d81037c0 msgr2=0x7fc6d8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.176+0000 7fc6d5e00700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d81037c0 0x7fc6d8116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.177+0000 7fc6d5e00700 1 -- 172.21.15.63:0/1461235005 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8005040 con 0x7fc6d8102e20 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.177+0000 7fc6d5e00700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d8115cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009390 tx=0x7fc6c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.177+0000 7fc6ce000700 1 -- 172.21.15.63:0/1461235005 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c800e040 con 0x7fc6d8102e20 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.177+0000 7fc6ce000700 1 -- 172.21.15.63:0/1461235005 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c8013420 con 0x7fc6d8102e20 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.177+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d810fed0 con 0x7fc6d8102e20 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.177+0000 7fc6ce000700 1 -- 172.21.15.63:0/1461235005 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c801c630 con 0x7fc6d8102e20 2024-03-23T11:54:23.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.177+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d8110490 con 0x7fc6d8102e20 2024-03-23T11:54:23.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.178+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d80fe570 con 0x7fc6d8102e20 2024-03-23T11:54:23.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.179+0000 7fc6ce000700 1 -- 172.21.15.63:0/1461235005 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc6c80139e0 con 0x7fc6d8102e20 2024-03-23T11:54:23.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.180+0000 7fc6ce000700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6bc06c7d0 0x7fc6bc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:23.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.180+0000 7fc6ce000700 1 -- 172.21.15.63:0/1461235005 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc6c808ccf0 con 0x7fc6d8102e20 2024-03-23T11:54:23.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.180+0000 7fc6d5400700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6bc06c7d0 0x7fc6bc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:23.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.181+0000 7fc6d5400700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6bc06c7d0 0x7fc6bc06ec80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8111590 tx=0x7fc6c400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:23.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.184+0000 7fc6ce000700 1 -- 172.21.15.63:0/1461235005 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc6c805aaa0 con 0x7fc6d8102e20 2024-03-23T11:54:23.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.410+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6d80fe570 con 0x7fc6bc06c7d0 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.420+0000 7fc6ce000700 1 -- 172.21.15.63:0/1461235005 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fc6d80fe570 con 0x7fc6bc06c7d0 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (3m) 14s ago 16m 14.1M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (53s) 14s ago 17m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (47s) 43s ago 15m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (3m) 14s ago 16m 73.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (10m) 14s ago 10m 118M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (10m) 14s ago 10m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (10m) 43s ago 10m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:54:23.404 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (10m) 43s ago 10m 60.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (6m) 14s ago 18m 495M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (4m) 43s ago 15m 450M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (2m) 14s ago 18m 43.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (97s) 43s ago 15m 36.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (4m) 14s ago 16m 7906k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (4m) 43s ago 15m 7860k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (21s) 14s ago 14m 13.1M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (14m) 14s ago 14m 232M 1618M 17.2.7-726-g9c4e0c34 837124da804f 852f52cf200f 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (13m) 14s ago 13m 197M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (13m) 43s ago 13m 259M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (12m) 43s ago 12m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (12m) 43s ago 12m 228M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:54:23.405 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (4m) 14s ago 16m 49.1M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:54:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6bc06c7d0 msgr2=0x7fc6bc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6bc06c7d0 0x7fc6bc06ec80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8111590 tx=0x7fc6c400b040 comp rx=0 tx=0).stop 2024-03-23T11:54:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 msgr2=0x7fc6d8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d8115cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009390 tx=0x7fc6c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 shutdown_connections 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6bc06c7d0 0x7fc6bc06ec80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6d8102e20 0x7fc6d8115cf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 --2- 172.21.15.63:0/1461235005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d81037c0 0x7fc6d8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.422+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 >> 172.21.15.63:0/1461235005 conn(0x7fc6d80f89d0 msgr2=0x7fc6d80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.423+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 shutdown_connections 2024-03-23T11:54:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.423+0000 7fc6dcc00700 1 -- 172.21.15.63:0/1461235005 wait complete. 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.559+0000 7f2e44800700 1 -- 172.21.15.63:0/1717271821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 msgr2=0x7f2e3c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.559+0000 7f2e44800700 1 --2- 172.21.15.63:0/1717271821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c1031d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e30004940 tx=0x7f2e30009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.559+0000 7f2e44800700 1 -- 172.21.15.63:0/1717271821 shutdown_connections 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.559+0000 7f2e44800700 1 --2- 172.21.15.63:0/1717271821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e3c1037a0 0x7f2e3c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.559+0000 7f2e44800700 1 --2- 172.21.15.63:0/1717271821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c1031d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.559+0000 7f2e44800700 1 -- 172.21.15.63:0/1717271821 >> 172.21.15.63:0/1717271821 conn(0x7f2e3c0f89b0 msgr2=0x7f2e3c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.560+0000 7f2e44800700 1 -- 172.21.15.63:0/1717271821 shutdown_connections 2024-03-23T11:54:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.560+0000 7f2e44800700 1 -- 172.21.15.63:0/1717271821 wait complete. 2024-03-23T11:54:23.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.560+0000 7f2e44800700 1 Processor -- start 2024-03-23T11:54:23.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e44800700 1 -- start start 2024-03-23T11:54:23.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e44800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:23.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e44800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e3c1037a0 0x7f2e3c116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e44800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c116880 con 0x7f2e3c102e00 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e44800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c10fc50 con 0x7f2e3c1037a0 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43806/0 (socket says 172.21.15.63:43806) 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.561+0000 7f2e41e00700 1 -- 172.21.15.63:0/101153364 learned_addr learned my addr 172.21.15.63:0/101153364 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e41400700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e3c1037a0 0x7f2e3c116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e41e00700 1 -- 172.21.15.63:0/101153364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e3c1037a0 msgr2=0x7f2e3c116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e41e00700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e3c1037a0 0x7f2e3c116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e41e00700 1 -- 172.21.15.63:0/101153364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30005040 con 0x7f2e3c102e00 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e41e00700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c115cf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009180 tx=0x7f2e30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e36000700 1 -- 172.21.15.63:0/101153364 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e3000e040 con 0x7f2e3c102e00 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e36000700 1 -- 172.21.15.63:0/101153364 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e30004290 con 0x7f2e3c102e00 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c10fed0 con 0x7f2e3c102e00 2024-03-23T11:54:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e36000700 1 -- 172.21.15.63:0/101153364 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e3001c3f0 con 0x7f2e3c102e00 2024-03-23T11:54:23.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.562+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c110490 con 0x7f2e3c102e00 2024-03-23T11:54:23.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.564+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c0fe550 con 0x7f2e3c102e00 2024-03-23T11:54:23.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.565+0000 7f2e36000700 1 -- 172.21.15.63:0/101153364 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2e30013790 con 0x7f2e3c102e00 2024-03-23T11:54:23.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.566+0000 7f2e36000700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2e2406c7c0 0x7f2e2406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:23.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.566+0000 7f2e36000700 1 -- 172.21.15.63:0/101153364 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2e3008cc90 con 0x7f2e3c102e00 2024-03-23T11:54:23.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.566+0000 7f2e41400700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2e2406c7c0 0x7f2e2406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:23.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.567+0000 7f2e41400700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2e2406c7c0 0x7f2e2406ec70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c111590 tx=0x7f2e2c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:23.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.570+0000 7f2e36000700 1 -- 172.21.15.63:0/101153364 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2e3005aa40 con 0x7f2e3c102e00 2024-03-23T11:54:23.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.878+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2e3c0fe550 con 0x7f2e3c102e00 2024-03-23T11:54:23.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.878+0000 7f2e36000700 1 -- 172.21.15.63:0/101153364 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7f2e30013420 con 0x7f2e3c102e00 2024-03-23T11:54:23.862 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:54:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:54:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:54:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 5, 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 9, 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 5 2024-03-23T11:54:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:54:23.864 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:54:23.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.880+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2e2406c7c0 msgr2=0x7f2e2406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2e2406c7c0 0x7f2e2406ec70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c111590 tx=0x7f2e2c009ee0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 msgr2=0x7f2e3c115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c115cf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009180 tx=0x7f2e30009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 shutdown_connections 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2e2406c7c0 0x7f2e2406ec70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e3c102e00 0x7f2e3c115cf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 --2- 172.21.15.63:0/101153364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e3c1037a0 0x7f2e3c116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 >> 172.21.15.63:0/101153364 conn(0x7f2e3c0f89b0 msgr2=0x7f2e3c0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 shutdown_connections 2024-03-23T11:54:23.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:23.881+0000 7f2e44800700 1 -- 172.21.15.63:0/101153364 wait complete. 2024-03-23T11:54:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 -- 172.21.15.63:0/1589526893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0102ca0 msgr2=0x7f68c0103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 --2- 172.21.15.63:0/1589526893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0102ca0 0x7f68c0103070 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f68b0004940 tx=0x7f68b0009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 -- 172.21.15.63:0/1589526893 shutdown_connections 2024-03-23T11:54:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 --2- 172.21.15.63:0/1589526893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68c0065630 0x7f68c0065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 --2- 172.21.15.63:0/1589526893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0102ca0 0x7f68c0103070 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 -- 172.21.15.63:0/1589526893 >> 172.21.15.63:0/1589526893 conn(0x7f68c0071b70 msgr2=0x7f68c0071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 -- 172.21.15.63:0/1589526893 shutdown_connections 2024-03-23T11:54:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.017+0000 7f68bf200700 1 -- 172.21.15.63:0/1589526893 wait complete. 2024-03-23T11:54:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.018+0000 7f68bf200700 1 Processor -- start 2024-03-23T11:54:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.019+0000 7f68bf200700 1 -- start start 2024-03-23T11:54:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.019+0000 7f68bf200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0065630 0x7f68c01138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.019+0000 7f68bf200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68c0102ca0 0x7f68c0113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.019+0000 7f68bf200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c0114430 con 0x7f68c0065630 2024-03-23T11:54:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.019+0000 7f68bf200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c010d820 con 0x7f68c0102ca0 2024-03-23T11:54:24.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0065630 0x7f68c01138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0065630 0x7f68c01138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43824/0 (socket says 172.21.15.63:43824) 2024-03-23T11:54:24.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bde00700 1 -- 172.21.15.63:0/1169220238 learned_addr learned my addr 172.21.15.63:0/1169220238 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:24.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bde00700 1 -- 172.21.15.63:0/1169220238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68c0102ca0 msgr2=0x7f68c0113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bde00700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68c0102ca0 0x7f68c0113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bde00700 1 -- 172.21.15.63:0/1169220238 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0005040 con 0x7f68c0065630 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bde00700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0065630 0x7f68c01138a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009390 tx=0x7f68b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68b6000700 1 -- 172.21.15.63:0/1169220238 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b000e040 con 0x7f68c0065630 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68b6000700 1 -- 172.21.15.63:0/1169220238 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68b0013420 con 0x7f68c0065630 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.020+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c010daa0 con 0x7f68c0065630 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.021+0000 7f68b6000700 1 -- 172.21.15.63:0/1169220238 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b001c630 con 0x7f68c0065630 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.021+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c010e060 con 0x7f68c0065630 2024-03-23T11:54:24.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.022+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c00fe380 con 0x7f68c0065630 2024-03-23T11:54:24.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.022+0000 7f68b6000700 1 -- 172.21.15.63:0/1169220238 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f68b00139e0 con 0x7f68c0065630 2024-03-23T11:54:24.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.023+0000 7f68b6000700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68a406c7d0 0x7f68a406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.023+0000 7f68b6000700 1 -- 172.21.15.63:0/1169220238 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f68b008cc40 con 0x7f68c0065630 2024-03-23T11:54:24.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.024+0000 7f68bd400700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68a406c7d0 0x7f68a406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.024+0000 7f68bd400700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68a406c7d0 0x7f68a406ec80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68c010e9e0 tx=0x7f68ac009ce0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:24.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.028+0000 7f68b6000700 1 -- 172.21.15.63:0/1169220238 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f68b005aaa0 con 0x7f68c0065630 2024-03-23T11:54:24.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:23 smithi063 ceph-mon[141124]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:24.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:23 smithi111 ceph-mon[106899]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:24.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.288+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f68c0109ba0 con 0x7f68c0065630 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.289+0000 7f68b6000700 1 -- 172.21.15.63:0/1169220238 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f68b005a630 con 0x7f68c0065630 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:54:24.273 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:54:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68a406c7d0 msgr2=0x7f68a406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68a406c7d0 0x7f68a406ec80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68c010e9e0 tx=0x7f68ac009ce0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0065630 msgr2=0x7f68c01138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0065630 0x7f68c01138a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009390 tx=0x7f68b0009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 shutdown_connections 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68a406c7d0 0x7f68a406ec80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68c0065630 0x7f68c01138a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 --2- 172.21.15.63:0/1169220238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68c0102ca0 0x7f68c0113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 >> 172.21.15.63:0/1169220238 conn(0x7f68c0071b70 msgr2=0x7f68c01059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.291+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 shutdown_connections 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.292+0000 7f68bf200700 1 -- 172.21.15.63:0/1169220238 wait complete. 2024-03-23T11:54:24.278 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:54:24.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 -- 172.21.15.63:0/2127933226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 msgr2=0x7fd38c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 --2- 172.21.15.63:0/2127933226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c1031d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd380004940 tx=0x7fd380009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:24.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 -- 172.21.15.63:0/2127933226 shutdown_connections 2024-03-23T11:54:24.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 --2- 172.21.15.63:0/2127933226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd38c1037a0 0x7fd38c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 --2- 172.21.15.63:0/2127933226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c1031d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 -- 172.21.15.63:0/2127933226 >> 172.21.15.63:0/2127933226 conn(0x7fd38c0f89b0 msgr2=0x7fd38c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:24.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 -- 172.21.15.63:0/2127933226 shutdown_connections 2024-03-23T11:54:24.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.428+0000 7fd393800700 1 -- 172.21.15.63:0/2127933226 wait complete. 2024-03-23T11:54:24.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.429+0000 7fd393800700 1 Processor -- start 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.429+0000 7fd393800700 1 -- start start 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd393800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd393800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd38c1037a0 0x7fd38c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd393800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c070d90 con 0x7fd38c102e00 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd393800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c070ed0 con 0x7fd38c1037a0 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd390e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd390e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43848/0 (socket says 172.21.15.63:43848) 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd390e00700 1 -- 172.21.15.63:0/2759931458 learned_addr learned my addr 172.21.15.63:0/2759931458 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd390e00700 1 -- 172.21.15.63:0/2759931458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd38c1037a0 msgr2=0x7fd38c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd38be00700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd38c1037a0 0x7fd38c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd390e00700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd38c1037a0 0x7fd38c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd390e00700 1 -- 172.21.15.63:0/2759931458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd380005040 con 0x7fd38c102e00 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.430+0000 7fd38be00700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd38c1037a0 0x7fd38c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.431+0000 7fd390e00700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c0702c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd380009390 tx=0x7fd380009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.431+0000 7fd388c00700 1 -- 172.21.15.63:0/2759931458 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd38000e040 con 0x7fd38c102e00 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.431+0000 7fd388c00700 1 -- 172.21.15.63:0/2759931458 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd380013420 con 0x7fd38c102e00 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.431+0000 7fd388c00700 1 -- 172.21.15.63:0/2759931458 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd38001c630 con 0x7fd38c102e00 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.431+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c071120 con 0x7fd38c102e00 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.431+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c11a250 con 0x7fd38c102e00 2024-03-23T11:54:24.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.432+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c0fe550 con 0x7fd38c102e00 2024-03-23T11:54:24.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.433+0000 7fd388c00700 1 -- 172.21.15.63:0/2759931458 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd3800139e0 con 0x7fd38c102e00 2024-03-23T11:54:24.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.434+0000 7fd388c00700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd37406c7d0 0x7fd37406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.434+0000 7fd388c00700 1 -- 172.21.15.63:0/2759931458 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd38008cb60 con 0x7fd38c102e00 2024-03-23T11:54:24.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.434+0000 7fd38be00700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd37406c7d0 0x7fd37406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.435+0000 7fd38be00700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd37406c7d0 0x7fd37406ec80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd378007fd0 tx=0x7fd378009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:24.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.438+0000 7fd388c00700 1 -- 172.21.15.63:0/2759931458 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd380056fd0 con 0x7fd38c102e00 2024-03-23T11:54:24.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.672+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd38c0719b0 con 0x7fd37406c7d0 2024-03-23T11:54:24.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.673+0000 7fd388c00700 1 -- 172.21.15.63:0/2759931458 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd38c0719b0 con 0x7fd37406c7d0 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:54:24.657 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "7/21 daemons upgraded", 2024-03-23T11:54:24.658 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading osd daemons", 2024-03-23T11:54:24.658 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:54:24.658 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:54:24.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.675+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd37406c7d0 msgr2=0x7fd37406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.675+0000 7fd393800700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd37406c7d0 0x7fd37406ec80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd378007fd0 tx=0x7fd378009040 comp rx=0 tx=0).stop 2024-03-23T11:54:24.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.675+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 msgr2=0x7fd38c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.675+0000 7fd393800700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c0702c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd380009390 tx=0x7fd380009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:24.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.676+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 shutdown_connections 2024-03-23T11:54:24.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.676+0000 7fd393800700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd37406c7d0 0x7fd37406ec80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.676+0000 7fd393800700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd38c102e00 0x7fd38c0702c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.676+0000 7fd393800700 1 --2- 172.21.15.63:0/2759931458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd38c1037a0 0x7fd38c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.676+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 >> 172.21.15.63:0/2759931458 conn(0x7fd38c0f89b0 msgr2=0x7fd38c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:24.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.676+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 shutdown_connections 2024-03-23T11:54:24.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.676+0000 7fd393800700 1 -- 172.21.15.63:0/2759931458 wait complete. 2024-03-23T11:54:24.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 -- 172.21.15.63:0/2069730849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 msgr2=0x7f7f981031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 --2- 172.21.15.63:0/2069730849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f981031f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7f88004940 tx=0x7f7f88009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:24.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 -- 172.21.15.63:0/2069730849 shutdown_connections 2024-03-23T11:54:24.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 --2- 172.21.15.63:0/2069730849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f981037c0 0x7f7f980fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 --2- 172.21.15.63:0/2069730849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f981031f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 -- 172.21.15.63:0/2069730849 >> 172.21.15.63:0/2069730849 conn(0x7f7f980f89d0 msgr2=0x7f7f980fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:24.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 -- 172.21.15.63:0/2069730849 shutdown_connections 2024-03-23T11:54:24.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.813+0000 7f7f9d800700 1 -- 172.21.15.63:0/2069730849 wait complete. 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.814+0000 7f7f9d800700 1 Processor -- start 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.814+0000 7f7f9d800700 1 -- start start 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f9d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f98115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f9d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f981037c0 0x7f7f981161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f9d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f98116800 con 0x7f7f98102e20 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f9d800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9810fbd0 con 0x7f7f981037c0 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f96a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f98115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f96a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f98115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43866/0 (socket says 172.21.15.63:43866) 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f96a00700 1 -- 172.21.15.63:0/1608097397 learned_addr learned my addr 172.21.15.63:0/1608097397 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:24.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f8fe00700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f981037c0 0x7f7f981161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f96a00700 1 -- 172.21.15.63:0/1608097397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f981037c0 msgr2=0x7f7f981161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:24.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f96a00700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f981037c0 0x7f7f981161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:24.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f96a00700 1 -- 172.21.15.63:0/1608097397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f88005040 con 0x7f7f98102e20 2024-03-23T11:54:24.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.815+0000 7f7f8fe00700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f981037c0 0x7f7f981161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:54:24.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.816+0000 7f7f96a00700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f98115c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009390 tx=0x7f7f88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:24.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.819+0000 7f7f8ea00700 1 -- 172.21.15.63:0/1608097397 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f8800e040 con 0x7f7f98102e20 2024-03-23T11:54:24.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.819+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9810fe50 con 0x7f7f98102e20 2024-03-23T11:54:24.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.819+0000 7f7f8ea00700 1 -- 172.21.15.63:0/1608097397 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f88013420 con 0x7f7f98102e20 2024-03-23T11:54:24.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.820+0000 7f7f8ea00700 1 -- 172.21.15.63:0/1608097397 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f8801c630 con 0x7f7f98102e20 2024-03-23T11:54:24.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.820+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f98110410 con 0x7f7f98102e20 2024-03-23T11:54:24.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.821+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f980fe570 con 0x7f7f98102e20 2024-03-23T11:54:24.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.822+0000 7f7f8ea00700 1 -- 172.21.15.63:0/1608097397 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7f880139e0 con 0x7f7f98102e20 2024-03-23T11:54:24.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.822+0000 7f7f8ea00700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7f7c06c7d0 0x7f7f7c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:24.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.823+0000 7f7f8ea00700 1 -- 172.21.15.63:0/1608097397 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7f8808cc00 con 0x7f7f98102e20 2024-03-23T11:54:24.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.823+0000 7f7f8fe00700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7f7c06c7d0 0x7f7f7c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:24.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.823+0000 7f7f8fe00700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7f7c06c7d0 0x7f7f7c06ec80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f80007fd0 tx=0x7f7f80009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:24.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:24.827+0000 7f7f8ea00700 1 -- 172.21.15.63:0/1608097397 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f8805aa40 con 0x7f7f98102e20 2024-03-23T11:54:25.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.127+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f9804bed0 con 0x7f7f98102e20 2024-03-23T11:54:25.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.128+0000 7f7f8ea00700 1 -- 172.21.15.63:0/1608097397 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f7f8805a5d0 con 0x7f7f98102e20 2024-03-23T11:54:25.112 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T11:54:25.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.130+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7f7c06c7d0 msgr2=0x7f7f7c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7f7c06c7d0 0x7f7f7c06ec80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f80007fd0 tx=0x7f7f80009f60 comp rx=0 tx=0).stop 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 msgr2=0x7f7f98115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f98115c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009390 tx=0x7f7f88009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 shutdown_connections 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7f7c06c7d0 0x7f7f7c06ec80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f98102e20 0x7f7f98115c70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 --2- 172.21.15.63:0/1608097397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f981037c0 0x7f7f981161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 >> 172.21.15.63:0/1608097397 conn(0x7f7f980f89d0 msgr2=0x7f7f98070620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 shutdown_connections 2024-03-23T11:54:25.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:25.131+0000 7f7f9d800700 1 -- 172.21.15.63:0/1608097397 wait complete. 2024-03-23T11:54:25.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:24 smithi063 ceph-mon[141124]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:25.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:24 smithi063 ceph-mon[141124]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:25.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:24 smithi063 ceph-mon[141124]: pgmap v67: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:25.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:24 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/101153364' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:25.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:24 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1169220238' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:54:25.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:24 smithi111 ceph-mon[106899]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:25.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:24 smithi111 ceph-mon[106899]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:25.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:24 smithi111 ceph-mon[106899]: pgmap v67: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:25.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:24 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/101153364' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:25.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:24 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1169220238' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:54:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:25 smithi063 ceph-mon[141124]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:25 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1608097397' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:54:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:25 smithi111 ceph-mon[106899]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:25 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1608097397' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:54:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:26 smithi063 ceph-mon[141124]: pgmap v68: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:26 smithi111 ceph-mon[106899]: pgmap v68: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:28 smithi063 ceph-mon[141124]: pgmap v69: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:54:29.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:28 smithi111 ceph-mon[106899]: pgmap v69: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:54:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:29 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:29 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:30.981 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:30 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:30.981 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:30 smithi063 ceph-mon[141124]: Upgrade: osd.1 is safe to restart 2024-03-23T11:54:30.981 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:30 smithi063 ceph-mon[141124]: pgmap v70: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:30 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-23T11:54:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:30 smithi111 ceph-mon[106899]: Upgrade: osd.1 is safe to restart 2024-03-23T11:54:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:30 smithi111 ceph-mon[106899]: pgmap v70: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:32 smithi063 ceph-mon[141124]: Upgrade: Updating osd.1 2024-03-23T11:54:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T11:54:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:54:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:32 smithi063 ceph-mon[141124]: Deploying daemon osd.1 on smithi063 2024-03-23T11:54:32.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:32 smithi063 ceph-mon[141124]: pgmap v71: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:54:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:32 smithi111 ceph-mon[106899]: Upgrade: Updating osd.1 2024-03-23T11:54:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-23T11:54:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:54:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:32 smithi111 ceph-mon[106899]: Deploying daemon osd.1 on smithi063 2024-03-23T11:54:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:32 smithi111 ceph-mon[106899]: pgmap v71: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:54:35.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:34 smithi063 ceph-mon[141124]: pgmap v72: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:35.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:34 smithi111 ceph-mon[106899]: pgmap v72: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:35.632 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:35 smithi063 systemd[1]: Stopping Ceph osd.1 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:54:36.132 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:35 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[68510]: 2024-03-23T11:54:35.747+0000 7fde4c200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:54:36.132 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:35 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[68510]: 2024-03-23T11:54:35.747+0000 7fde4c200700 -1 osd.1 54 *** Got signal Terminated *** 2024-03-23T11:54:36.132 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:35 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[68510]: 2024-03-23T11:54:35.747+0000 7fde4c200700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T11:54:36.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:36 smithi063 ceph-mon[141124]: pgmap v73: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:36.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:36 smithi063 ceph-mon[141124]: osd.1 marked itself down and dead 2024-03-23T11:54:36.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:36 smithi111 ceph-mon[106899]: pgmap v73: 65 pgs: 65 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:54:36.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:36 smithi111 ceph-mon[106899]: osd.1 marked itself down and dead 2024-03-23T11:54:37.295 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:37 smithi063 bash[160943]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1 2024-03-23T11:54:37.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:37 smithi063 ceph-mon[141124]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:54:37.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:37 smithi063 ceph-mon[141124]: osdmap e55: 6 total, 5 up, 6 in 2024-03-23T11:54:37.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:37 smithi111 ceph-mon[106899]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:54:37.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:37 smithi111 ceph-mon[106899]: osdmap e55: 6 total, 5 up, 6 in 2024-03-23T11:54:38.592 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:38 smithi063 ceph-mon[141124]: osdmap e56: 6 total, 5 up, 6 in 2024-03-23T11:54:38.592 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:38 smithi063 ceph-mon[141124]: pgmap v76: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:54:38.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:38 smithi111 ceph-mon[106899]: osdmap e56: 6 total, 5 up, 6 in 2024-03-23T11:54:38.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:38 smithi111 ceph-mon[106899]: pgmap v76: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-23T11:54:39.602 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:39 smithi063 ceph-mon[141124]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-23T11:54:39.602 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:39 smithi063 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.1.service: Succeeded. 2024-03-23T11:54:39.602 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:39 smithi063 systemd[1]: Stopped Ceph osd.1 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:54:39.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:39 smithi111 ceph-mon[106899]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-23T11:54:39.882 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:39 smithi063 systemd[1]: Starting Ceph osd.1 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:54:40.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:40 smithi063 ceph-mon[141124]: pgmap v77: 65 pgs: 2 active+undersized+degraded, 29 peering, 3 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6/330 objects degraded (1.818%) 2024-03-23T11:54:40.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:40 smithi111 ceph-mon[106899]: pgmap v77: 65 pgs: 2 active+undersized+degraded, 29 peering, 3 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6/330 objects degraded (1.818%) 2024-03-23T11:54:41.288 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:40 smithi063 bash[161205]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-23T11:54:41.288 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:40 smithi063 bash[161205]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-23T11:54:41.288 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:40 smithi063 bash[161205]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-23T11:54:41.288 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:40 smithi063 bash[161205]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-23T11:54:41.288 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:40 smithi063 bash[161205]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-23T11:54:41.288 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:40 smithi063 bash[161205]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-23T11:54:41.288 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:40 smithi063 bash[161205]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-23T11:54:41.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:41 smithi063 ceph-mon[141124]: Health check failed: Degraded data redundancy: 6/330 objects degraded (1.818%), 2 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:41.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:41.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:41.681 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:41 smithi111 ceph-mon[106899]: Health check failed: Degraded data redundancy: 6/330 objects degraded (1.818%), 2 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:41.681 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:41.681 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:42.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:42 smithi063 ceph-mon[141124]: pgmap v78: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/330 objects degraded (15.758%) 2024-03-23T11:54:42.632 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:42 smithi063 ceph-osd[161498]: -- 172.21.15.63:0/2253767655 <== mon.0 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556c6b9b91e0 con 0x556c6b989000 2024-03-23T11:54:42.632 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:42 smithi063 bash[161460]: c3e531c0c9f75690bd93802dd7a1a2de6249b64aaba283ad19bb372bef963b22 2024-03-23T11:54:42.632 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:42 smithi063 systemd[1]: Started Ceph osd.1 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:54:42.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:42 smithi111 ceph-mon[106899]: pgmap v78: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/330 objects degraded (15.758%) 2024-03-23T11:54:43.132 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:43 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[161475]: 2024-03-23T11:54:43.014+0000 7fa70bdaa640 -1 Falling back to public interface 2024-03-23T11:54:43.608 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:43 smithi063 ceph-mon[141124]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-23T11:54:43.609 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:43 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:43.609 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:43 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:43.609 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:43 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:43.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:43 smithi111 ceph-mon[106899]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs peering) 2024-03-23T11:54:43.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:43 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:43.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:43 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:43.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:43 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:44.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:44 smithi063 ceph-mon[141124]: pgmap v79: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/330 objects degraded (15.758%) 2024-03-23T11:54:44.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:44 smithi111 ceph-mon[106899]: pgmap v79: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/330 objects degraded (15.758%) 2024-03-23T11:54:46.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:46 smithi063 ceph-mon[141124]: pgmap v80: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/330 objects degraded (15.758%) 2024-03-23T11:54:46.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:46 smithi111 ceph-mon[106899]: pgmap v80: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/330 objects degraded (15.758%) 2024-03-23T11:54:48.211 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:47 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[161475]: 2024-03-23T11:54:47.899+0000 7fa70bdaa640 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-23T11:54:48.759 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:48 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[161475]: 2024-03-23T11:54:48.488+0000 7fa70bdaa640 -1 osd.1 54 log_to_monitors true 2024-03-23T11:54:49.132 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 11:54:48 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[161475]: 2024-03-23T11:54:48.795+0000 7fa6f8a00700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:54:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:48 smithi063 ceph-mon[141124]: pgmap v81: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/330 objects degraded (15.758%) 2024-03-23T11:54:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:48 smithi063 ceph-mon[141124]: from='osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T11:54:49.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:48 smithi111 ceph-mon[106899]: pgmap v81: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/330 objects degraded (15.758%) 2024-03-23T11:54:49.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:48 smithi111 ceph-mon[106899]: from='osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-23T11:54:50.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:49 smithi063 ceph-mon[141124]: from='osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-23T11:54:50.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:49 smithi063 ceph-mon[141124]: osdmap e57: 6 total, 5 up, 6 in 2024-03-23T11:54:50.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:49 smithi063 ceph-mon[141124]: from='osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:54:50.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:50.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:49 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:49 smithi111 ceph-mon[106899]: from='osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-23T11:54:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:49 smithi111 ceph-mon[106899]: osdmap e57: 6 total, 5 up, 6 in 2024-03-23T11:54:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:49 smithi111 ceph-mon[106899]: from='osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:54:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:50.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:49 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: pgmap v83: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 52/330 objects degraded (15.758%) 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315] boot 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: osdmap e58: 6 total, 6 up, 6 in 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:50 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 52/330 objects degraded (15.758%), 16 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: pgmap v83: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 52/330 objects degraded (15.758%) 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: osd.1 [v2:172.21.15.63:6810/3286820315,v1:172.21.15.63:6811/3286820315] boot 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: osdmap e58: 6 total, 6 up, 6 in 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:50 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 52/330 objects degraded (15.758%), 16 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:52.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:51 smithi063 ceph-mon[141124]: osdmap e59: 6 total, 6 up, 6 in 2024-03-23T11:54:52.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:51 smithi111 ceph-mon[106899]: osdmap e59: 6 total, 6 up, 6 in 2024-03-23T11:54:52.802 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:52 smithi111 ceph-mon[106899]: pgmap v86: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 4 op/s; 22/330 objects degraded (6.667%) 2024-03-23T11:54:53.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:52 smithi063 ceph-mon[141124]: pgmap v86: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 4 op/s; 22/330 objects degraded (6.667%) 2024-03-23T11:54:55.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:54 smithi063 ceph-mon[141124]: pgmap v87: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 22/330 objects degraded (6.667%) 2024-03-23T11:54:55.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:55.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:55.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:54:55.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:54:55.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:55.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:54 smithi111 ceph-mon[106899]: pgmap v87: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 22/330 objects degraded (6.667%) 2024-03-23T11:54:55.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:55.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:55.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:54:55.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:54:55.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 -- 172.21.15.63:0/367216891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c102e20 msgr2=0x7f7e5c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 --2- 172.21.15.63:0/367216891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c102e20 0x7f7e5c1031f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c004940 tx=0x7f7e4c009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 -- 172.21.15.63:0/367216891 shutdown_connections 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 --2- 172.21.15.63:0/367216891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e5c1037c0 0x7f7e5c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 --2- 172.21.15.63:0/367216891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c102e20 0x7f7e5c1031f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 -- 172.21.15.63:0/367216891 >> 172.21.15.63:0/367216891 conn(0x7f7e5c0f89d0 msgr2=0x7f7e5c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 -- 172.21.15.63:0/367216891 shutdown_connections 2024-03-23T11:54:55.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.275+0000 7f7e63200700 1 -- 172.21.15.63:0/367216891 wait complete. 2024-03-23T11:54:55.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.276+0000 7f7e63200700 1 Processor -- start 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.276+0000 7f7e63200700 1 -- start start 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.276+0000 7f7e63200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e5c102e20 0x7f7e5c113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.276+0000 7f7e63200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c1037c0 0x7f7e5c114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.276+0000 7f7e63200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c114690 con 0x7f7e5c1037c0 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.276+0000 7f7e63200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c10da80 con 0x7f7e5c102e20 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e53e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c1037c0 0x7f7e5c114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e53e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c1037c0 0x7f7e5c114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33756/0 (socket says 172.21.15.63:33756) 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e53e00700 1 -- 172.21.15.63:0/1039184688 learned_addr learned my addr 172.21.15.63:0/1039184688 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e5be00700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e5c102e20 0x7f7e5c113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e53e00700 1 -- 172.21.15.63:0/1039184688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e5c102e20 msgr2=0x7f7e5c113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e53e00700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e5c102e20 0x7f7e5c113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e53e00700 1 -- 172.21.15.63:0/1039184688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c005040 con 0x7f7e5c1037c0 2024-03-23T11:54:55.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e5be00700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e5c102e20 0x7f7e5c113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:54:55.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.277+0000 7f7e53e00700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c1037c0 0x7f7e5c114040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7e44002a50 tx=0x7f7e44004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:55.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.278+0000 7f7e58c00700 1 -- 172.21.15.63:0/1039184688 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e44007cc0 con 0x7f7e5c1037c0 2024-03-23T11:54:55.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.278+0000 7f7e58c00700 1 -- 172.21.15.63:0/1039184688 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e4400f040 con 0x7f7e5c1037c0 2024-03-23T11:54:55.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.278+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5c10dd60 con 0x7f7e5c1037c0 2024-03-23T11:54:55.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.278+0000 7f7e58c00700 1 -- 172.21.15.63:0/1039184688 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e4400d9a0 con 0x7f7e5c1037c0 2024-03-23T11:54:55.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.278+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c10e320 con 0x7f7e5c1037c0 2024-03-23T11:54:55.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.279+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c0fe570 con 0x7f7e5c1037c0 2024-03-23T11:54:55.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.280+0000 7f7e58c00700 1 -- 172.21.15.63:0/1039184688 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7e4400dc60 con 0x7f7e5c1037c0 2024-03-23T11:54:55.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.281+0000 7f7e58c00700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e4006c7d0 0x7f7e4006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:55.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.281+0000 7f7e58c00700 1 -- 172.21.15.63:0/1039184688 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7e4408b810 con 0x7f7e5c1037c0 2024-03-23T11:54:55.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.281+0000 7f7e5be00700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e4006c7d0 0x7f7e4006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:55.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.282+0000 7f7e5be00700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e4006c7d0 0x7f7e4006ec80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c000c00 tx=0x7f7e4c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:55.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.285+0000 7f7e58c00700 1 -- 172.21.15.63:0/1039184688 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7e44059540 con 0x7f7e5c1037c0 2024-03-23T11:54:55.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.517+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e5c109de0 con 0x7f7e4006c7d0 2024-03-23T11:54:55.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.519+0000 7f7e58c00700 1 -- 172.21.15.63:0/1039184688 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7e5c109de0 con 0x7f7e4006c7d0 2024-03-23T11:54:55.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.521+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e4006c7d0 msgr2=0x7f7e4006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.521+0000 7f7e63200700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e4006c7d0 0x7f7e4006ec80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c000c00 tx=0x7f7e4c009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:55.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.521+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c1037c0 msgr2=0x7f7e5c114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.521+0000 7f7e63200700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c1037c0 0x7f7e5c114040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7e44002a50 tx=0x7f7e44004600 comp rx=0 tx=0).stop 2024-03-23T11:54:55.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.522+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 shutdown_connections 2024-03-23T11:54:55.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.522+0000 7f7e63200700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7e4006c7d0 0x7f7e4006ec80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.522+0000 7f7e63200700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e5c102e20 0x7f7e5c113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.522+0000 7f7e63200700 1 --2- 172.21.15.63:0/1039184688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e5c1037c0 0x7f7e5c114040 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.522+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 >> 172.21.15.63:0/1039184688 conn(0x7f7e5c0f89d0 msgr2=0x7f7e5c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:55.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.522+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 shutdown_connections 2024-03-23T11:54:55.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.522+0000 7f7e63200700 1 -- 172.21.15.63:0/1039184688 wait complete. 2024-03-23T11:54:55.518 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:54:55.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 -- 172.21.15.63:0/3758756576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff274102e20 msgr2=0x7ff2741031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 --2- 172.21.15.63:0/3758756576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff274102e20 0x7ff2741031f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff268004940 tx=0x7ff268009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:55.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 -- 172.21.15.63:0/3758756576 shutdown_connections 2024-03-23T11:54:55.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 --2- 172.21.15.63:0/3758756576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2741037c0 0x7ff2740fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 --2- 172.21.15.63:0/3758756576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff274102e20 0x7ff2741031f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 -- 172.21.15.63:0/3758756576 >> 172.21.15.63:0/3758756576 conn(0x7ff2740f8970 msgr2=0x7ff2740fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:55.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 -- 172.21.15.63:0/3758756576 shutdown_connections 2024-03-23T11:54:55.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.660+0000 7ff27be00700 1 -- 172.21.15.63:0/3758756576 wait complete. 2024-03-23T11:54:55.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.661+0000 7ff27be00700 1 Processor -- start 2024-03-23T11:54:55.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.661+0000 7ff27be00700 1 -- start start 2024-03-23T11:54:55.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.661+0000 7ff27be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff274102e20 0x7ff274113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:55.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.661+0000 7ff27be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741037c0 0x7ff274114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:55.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.661+0000 7ff27be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff274114670 con 0x7ff2741037c0 2024-03-23T11:54:55.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.661+0000 7ff27be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27410da60 con 0x7ff274102e20 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.661+0000 7ff278a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741037c0 0x7ff274114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff278a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741037c0 0x7ff274114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33774/0 (socket says 172.21.15.63:33774) 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff278a00700 1 -- 172.21.15.63:0/2662675011 learned_addr learned my addr 172.21.15.63:0/2662675011 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff278a00700 1 -- 172.21.15.63:0/2662675011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff274102e20 msgr2=0x7ff274113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff278a00700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff274102e20 0x7ff274113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff278a00700 1 -- 172.21.15.63:0/2662675011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268005040 con 0x7ff2741037c0 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff278a00700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741037c0 0x7ff274114020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff260002a50 tx=0x7ff260004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff271600700 1 -- 172.21.15.63:0/2662675011 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff260007cc0 con 0x7ff2741037c0 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff271600700 1 -- 172.21.15.63:0/2662675011 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff26000f040 con 0x7ff2741037c0 2024-03-23T11:54:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff271600700 1 -- 172.21.15.63:0/2662675011 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff26000d990 con 0x7ff2741037c0 2024-03-23T11:54:55.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.662+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27410dd40 con 0x7ff2741037c0 2024-03-23T11:54:55.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.663+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27410e300 con 0x7ff2741037c0 2024-03-23T11:54:55.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.664+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2740fe570 con 0x7ff2741037c0 2024-03-23T11:54:55.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.665+0000 7ff271600700 1 -- 172.21.15.63:0/2662675011 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff260009710 con 0x7ff2741037c0 2024-03-23T11:54:55.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.665+0000 7ff271600700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff25c06c7d0 0x7ff25c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:55.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.666+0000 7ff271600700 1 -- 172.21.15.63:0/2662675011 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff26008b7e0 con 0x7ff2741037c0 2024-03-23T11:54:55.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.666+0000 7ff279400700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff25c06c7d0 0x7ff25c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:55.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.666+0000 7ff279400700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff25c06c7d0 0x7ff25c06ec80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff268004f90 tx=0x7ff268009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:55.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.670+0000 7ff271600700 1 -- 172.21.15.63:0/2662675011 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff260059260 con 0x7ff2741037c0 2024-03-23T11:54:55.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.906+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff274109de0 con 0x7ff25c06c7d0 2024-03-23T11:54:55.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.907+0000 7ff271600700 1 -- 172.21.15.63:0/2662675011 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff274109de0 con 0x7ff25c06c7d0 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff25c06c7d0 msgr2=0x7ff25c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff25c06c7d0 0x7ff25c06ec80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff268004f90 tx=0x7ff268009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741037c0 msgr2=0x7ff274114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741037c0 0x7ff274114020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff260002a50 tx=0x7ff260004600 comp rx=0 tx=0).stop 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 shutdown_connections 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff25c06c7d0 0x7ff25c06ec80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff274102e20 0x7ff274113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 --2- 172.21.15.63:0/2662675011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741037c0 0x7ff274114020 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 >> 172.21.15.63:0/2662675011 conn(0x7ff2740f8970 msgr2=0x7ff2740f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 shutdown_connections 2024-03-23T11:54:55.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:55.910+0000 7ff27be00700 1 -- 172.21.15.63:0/2662675011 wait complete. 2024-03-23T11:54:56.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 -- 172.21.15.63:0/3248785172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55041035e0 msgr2=0x7f55040fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 --2- 172.21.15.63:0/3248785172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55041035e0 0x7f55040fcc50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f54f0004940 tx=0x7f54f0009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:56.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 -- 172.21.15.63:0/3248785172 shutdown_connections 2024-03-23T11:54:56.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 --2- 172.21.15.63:0/3248785172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55041035e0 0x7f55040fcc50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 --2- 172.21.15.63:0/3248785172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5504102c40 0x7f5504103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 -- 172.21.15.63:0/3248785172 >> 172.21.15.63:0/3248785172 conn(0x7f55040f8830 msgr2=0x7f55040fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:56.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 -- 172.21.15.63:0/3248785172 shutdown_connections 2024-03-23T11:54:56.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.047+0000 7f5508c00700 1 -- 172.21.15.63:0/3248785172 wait complete. 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.048+0000 7f5508c00700 1 Processor -- start 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.048+0000 7f5508c00700 1 -- start start 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5508c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5504102c40 0x7f5504115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5508c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55041035e0 0x7f5504115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5508c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55041165e0 con 0x7f5504102c40 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5508c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550410f9d0 con 0x7f55041035e0 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5503400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5504102c40 0x7f5504115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5503400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5504102c40 0x7f5504115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33794/0 (socket says 172.21.15.63:33794) 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5503400700 1 -- 172.21.15.63:0/2405816905 learned_addr learned my addr 172.21.15.63:0/2405816905 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:56.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.049+0000 7f5502a00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55041035e0 0x7f5504115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f5503400700 1 -- 172.21.15.63:0/2405816905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55041035e0 msgr2=0x7f5504115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f5503400700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55041035e0 0x7f5504115f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f5503400700 1 -- 172.21.15.63:0/2405816905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f0005040 con 0x7f5504102c40 2024-03-23T11:54:56.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f5503400700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5504102c40 0x7f5504115a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f54f4002a80 tx=0x7f54f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:56.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f54fb400700 1 -- 172.21.15.63:0/2405816905 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54f4007cc0 con 0x7f5504102c40 2024-03-23T11:54:56.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f54fb400700 1 -- 172.21.15.63:0/2405816905 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54f400f040 con 0x7f5504102c40 2024-03-23T11:54:56.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f54fb400700 1 -- 172.21.15.63:0/2405816905 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54f400da60 con 0x7f5504102c40 2024-03-23T11:54:56.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550410fcb0 con 0x7f5504102c40 2024-03-23T11:54:56.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.050+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5504110240 con 0x7f5504102c40 2024-03-23T11:54:56.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.052+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55040fe390 con 0x7f5504102c40 2024-03-23T11:54:56.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.052+0000 7f54fb400700 1 -- 172.21.15.63:0/2405816905 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f54f400dbc0 con 0x7f5504102c40 2024-03-23T11:54:56.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.053+0000 7f54fb400700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54e806c7d0 0x7f54e806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.053+0000 7f54fb400700 1 -- 172.21.15.63:0/2405816905 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f54f408b630 con 0x7f5504102c40 2024-03-23T11:54:56.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.054+0000 7f5502a00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54e806c7d0 0x7f54e806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.054+0000 7f5502a00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54e806c7d0 0x7f54e806ec80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5504110cb0 tx=0x7f54f0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:56.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.058+0000 7f54fb400700 1 -- 172.21.15.63:0/2405816905 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f54f4059340 con 0x7f5504102c40 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 22/330 objects degraded (6.667%), 7 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:56.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 22/330 objects degraded (6.667%), 7 pgs degraded (PG_DEGRADED) 2024-03-23T11:54:56.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:54:56.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:54:56.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.281+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5504109bd0 con 0x7f54e806c7d0 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.291+0000 7f54fb400700 1 -- 172.21.15.63:0/2405816905 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f5504109bd0 con 0x7f54e806c7d0 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (4m) 7s ago 17m 14.1M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (85s) 7s ago 17m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (80s) 75s ago 16m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (3m) 7s ago 17m 73.4M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (11m) 7s ago 11m 118M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (10m) 7s ago 10m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (10m) 75s ago 10m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:54:56.274 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (10m) 75s ago 10m 60.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (6m) 7s ago 18m 497M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (5m) 75s ago 16m 450M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (2m) 7s ago 19m 44.3M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (2m) 75s ago 16m 36.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (5m) 7s ago 17m 7608k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (4m) 75s ago 16m 7860k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (54s) 7s ago 15m 87.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (14s) 7s ago 14m 9.87M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (14m) 7s ago 14m 196M 1618M 17.2.7-726-g9c4e0c34 837124da804f 00110f77ac28 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (13m) 75s ago 13m 259M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (13m) 75s ago 13m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (12m) 75s ago 12m 228M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:54:56.275 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (4m) 7s ago 16m 52.3M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54e806c7d0 msgr2=0x7f54e806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54e806c7d0 0x7f54e806ec80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5504110cb0 tx=0x7f54f0009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5504102c40 msgr2=0x7f5504115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5504102c40 0x7f5504115a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f54f4002a80 tx=0x7f54f40046b0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 shutdown_connections 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54e806c7d0 0x7f54e806ec80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5504102c40 0x7f5504115a50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 --2- 172.21.15.63:0/2405816905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55041035e0 0x7f5504115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 >> 172.21.15.63:0/2405816905 conn(0x7f55040f8830 msgr2=0x7f55040f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 shutdown_connections 2024-03-23T11:54:56.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.294+0000 7f5508c00700 1 -- 172.21.15.63:0/2405816905 wait complete. 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 -- 172.21.15.63:0/918192863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 msgr2=0x7f9c0c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 --2- 172.21.15.63:0/918192863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c106f40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8004940 tx=0x7f9bf8009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 -- 172.21.15.63:0/918192863 shutdown_connections 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 --2- 172.21.15.63:0/918192863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c106f40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 --2- 172.21.15.63:0/918192863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c0c0fded0 0x7f9c0c0fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 -- 172.21.15.63:0/918192863 >> 172.21.15.63:0/918192863 conn(0x7f9c0c072bc0 msgr2=0x7f9c0c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 -- 172.21.15.63:0/918192863 shutdown_connections 2024-03-23T11:54:56.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.429+0000 7f9c11200700 1 -- 172.21.15.63:0/918192863 wait complete. 2024-03-23T11:54:56.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.430+0000 7f9c11200700 1 Processor -- start 2024-03-23T11:54:56.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.430+0000 7f9c11200700 1 -- start start 2024-03-23T11:54:56.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c11200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c0c0fded0 0x7f9c0c115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c11200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c11200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c1167d0 con 0x7f9c0c0fe870 2024-03-23T11:54:56.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c11200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c10fba0 con 0x7f9c0c0fded0 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c09400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c09400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c116180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33802/0 (socket says 172.21.15.63:33802) 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c09400700 1 -- 172.21.15.63:0/1794173080 learned_addr learned my addr 172.21.15.63:0/1794173080 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c09e00700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c0c0fded0 0x7f9c0c115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c09400700 1 -- 172.21.15.63:0/1794173080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c0c0fded0 msgr2=0x7f9c0c115c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c09400700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c0c0fded0 0x7f9c0c115c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.431+0000 7f9c09400700 1 -- 172.21.15.63:0/1794173080 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8005040 con 0x7f9c0c0fe870 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.432+0000 7f9c09400700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c116180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009390 tx=0x7f9bf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.432+0000 7f9c02000700 1 -- 172.21.15.63:0/1794173080 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf800e040 con 0x7f9c0c0fe870 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.432+0000 7f9c02000700 1 -- 172.21.15.63:0/1794173080 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bf8013420 con 0x7f9c0c0fe870 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.432+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c10fe80 con 0x7f9c0c0fe870 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.432+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c1103c0 con 0x7f9c0c0fe870 2024-03-23T11:54:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.432+0000 7f9c02000700 1 -- 172.21.15.63:0/1794173080 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf80255e0 con 0x7f9c0c0fe870 2024-03-23T11:54:56.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.433+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c106a00 con 0x7f9c0c0fe870 2024-03-23T11:54:56.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.434+0000 7f9c02000700 1 -- 172.21.15.63:0/1794173080 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9bf8013a20 con 0x7f9c0c0fe870 2024-03-23T11:54:56.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.435+0000 7f9c02000700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9bf006c7d0 0x7f9bf006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.435+0000 7f9c02000700 1 -- 172.21.15.63:0/1794173080 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f9bf808cc00 con 0x7f9c0c0fe870 2024-03-23T11:54:56.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.435+0000 7f9c09e00700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9bf006c7d0 0x7f9bf006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.436+0000 7f9c09e00700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9bf006c7d0 0x7f9bf006ec80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc004f40 tx=0x7f9bfc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.439+0000 7f9c02000700 1 -- 172.21.15.63:0/1794173080 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9bf805a910 con 0x7f9c0c0fe870 2024-03-23T11:54:56.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.742+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9c0c111290 con 0x7f9c0c0fe870 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.742+0000 7f9c02000700 1 -- 172.21.15.63:0/1794173080 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7f9bf805a4a0 con 0x7f9c0c0fe870 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4, 2024-03-23T11:54:56.725 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 8, 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:54:56.726 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:54:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.744+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9bf006c7d0 msgr2=0x7f9bf006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.744+0000 7f9c11200700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9bf006c7d0 0x7f9bf006ec80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc004f40 tx=0x7f9bfc00b040 comp rx=0 tx=0).stop 2024-03-23T11:54:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.744+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 msgr2=0x7f9c0c116180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.744+0000 7f9c11200700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c116180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009390 tx=0x7f9bf8009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:56.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.745+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 shutdown_connections 2024-03-23T11:54:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.745+0000 7f9c11200700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9bf006c7d0 0x7f9bf006ec80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.745+0000 7f9c11200700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c0c0fded0 0x7f9c0c115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.745+0000 7f9c11200700 1 --2- 172.21.15.63:0/1794173080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c0c0fe870 0x7f9c0c116180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.745+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 >> 172.21.15.63:0/1794173080 conn(0x7f9c0c072bc0 msgr2=0x7f9c0c0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.745+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 shutdown_connections 2024-03-23T11:54:56.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.745+0000 7f9c11200700 1 -- 172.21.15.63:0/1794173080 wait complete. 2024-03-23T11:54:56.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.882+0000 7fb0f9200700 1 -- 172.21.15.63:0/2876721482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f41059a0 msgr2=0x7fb0f40ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.882+0000 7fb0f9200700 1 --2- 172.21.15.63:0/2876721482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f41059a0 0x7fb0f40ff010 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0004940 tx=0x7fb0e0009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:56.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.882+0000 7fb0f9200700 1 -- 172.21.15.63:0/2876721482 shutdown_connections 2024-03-23T11:54:56.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.882+0000 7fb0f9200700 1 --2- 172.21.15.63:0/2876721482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f41059a0 0x7fb0f40ff010 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.882+0000 7fb0f9200700 1 --2- 172.21.15.63:0/2876721482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f4105000 0x7fb0f41053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.882+0000 7fb0f9200700 1 -- 172.21.15.63:0/2876721482 >> 172.21.15.63:0/2876721482 conn(0x7fb0f40fabd0 msgr2=0x7fb0f40fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:56.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.882+0000 7fb0f9200700 1 -- 172.21.15.63:0/2876721482 shutdown_connections 2024-03-23T11:54:56.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.883+0000 7fb0f9200700 1 -- 172.21.15.63:0/2876721482 wait complete. 2024-03-23T11:54:56.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.883+0000 7fb0f9200700 1 Processor -- start 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.883+0000 7fb0f9200700 1 -- start start 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f4105000 0x7fb0f4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f9200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f41059a0 0x7fb0f4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f9200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f4116860 con 0x7fb0f4105000 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f9200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f410fc30 con 0x7fb0f41059a0 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f4105000 0x7fb0f4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f4105000 0x7fb0f4115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33806/0 (socket says 172.21.15.63:33806) 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f1e00700 1 -- 172.21.15.63:0/3636782824 learned_addr learned my addr 172.21.15.63:0/3636782824 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:56.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f1400700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f41059a0 0x7fb0f4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f1e00700 1 -- 172.21.15.63:0/3636782824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f41059a0 msgr2=0x7fb0f4116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f1e00700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f41059a0 0x7fb0f4116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.884+0000 7fb0f1e00700 1 -- 172.21.15.63:0/3636782824 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0005040 con 0x7fb0f4105000 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.885+0000 7fb0f1e00700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f4105000 0x7fb0f4115cd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4002a50 tx=0x7fb0e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.885+0000 7fb0ea000700 1 -- 172.21.15.63:0/3636782824 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0e4007cc0 con 0x7fb0f4105000 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.885+0000 7fb0ea000700 1 -- 172.21.15.63:0/3636782824 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0e400f040 con 0x7fb0f4105000 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.885+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f410ff10 con 0x7fb0f4105000 2024-03-23T11:54:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.885+0000 7fb0ea000700 1 -- 172.21.15.63:0/3636782824 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0e400d9c0 con 0x7fb0f4105000 2024-03-23T11:54:56.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.885+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f41104d0 con 0x7fb0f4105000 2024-03-23T11:54:56.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.886+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f4102310 con 0x7fb0f4105000 2024-03-23T11:54:56.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.887+0000 7fb0ea000700 1 -- 172.21.15.63:0/3636782824 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb0e400db40 con 0x7fb0f4105000 2024-03-23T11:54:56.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.888+0000 7fb0ea000700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:56.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.888+0000 7fb0ea000700 1 -- 172.21.15.63:0/3636782824 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb0e408b700 con 0x7fb0f4105000 2024-03-23T11:54:56.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.889+0000 7fb0f1400700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:56.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.889+0000 7fb0f1400700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4111570 tx=0x7fb0e0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:56.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:56.893+0000 7fb0ea000700 1 -- 172.21.15.63:0/3636782824 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb0e40594e0 con 0x7fb0f4105000 2024-03-23T11:54:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:56 smithi063 ceph-mon[141124]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:56 smithi063 ceph-mon[141124]: pgmap v88: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-03-23T11:54:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:56 smithi063 ceph-mon[141124]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/330 objects degraded (6.667%), 7 pgs degraded) 2024-03-23T11:54:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:56 smithi063 ceph-mon[141124]: Cluster is now healthy 2024-03-23T11:54:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:56 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1794173080' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.151+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb0f410bfa0 con 0x7fb0f4105000 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.151+0000 7fb0ea000700 1 -- 172.21.15.63:0/3636782824 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7fb0e400ddf0 con 0x7fb0f4105000 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:54:57.134 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:54:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:54:57.136 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 msgr2=0x7fb0d806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4111570 tx=0x7fb0e0009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f4105000 msgr2=0x7fb0f4115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f4105000 0x7fb0f4115cd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4002a50 tx=0x7fb0e40046b0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 shutdown_connections 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f4105000 0x7fb0f4115cd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 --2- 172.21.15.63:0/3636782824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f41059a0 0x7fb0f4116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 >> 172.21.15.63:0/3636782824 conn(0x7fb0f40fabd0 msgr2=0x7fb0f40fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 shutdown_connections 2024-03-23T11:54:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.154+0000 7fb0f9200700 1 -- 172.21.15.63:0/3636782824 wait complete. 2024-03-23T11:54:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:54:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:56 smithi111 ceph-mon[106899]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:56 smithi111 ceph-mon[106899]: pgmap v88: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-03-23T11:54:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:56 smithi111 ceph-mon[106899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/330 objects degraded (6.667%), 7 pgs degraded) 2024-03-23T11:54:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:56 smithi111 ceph-mon[106899]: Cluster is now healthy 2024-03-23T11:54:57.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:56 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1794173080' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:54:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 -- 172.21.15.63:0/2313662415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c102c50 msgr2=0x7f364c103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 --2- 172.21.15.63:0/2313662415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c102c50 0x7f364c103020 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f363c004940 tx=0x7f363c009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 -- 172.21.15.63:0/2313662415 shutdown_connections 2024-03-23T11:54:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 --2- 172.21.15.63:0/2313662415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c1035f0 0x7f364c0fcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 --2- 172.21.15.63:0/2313662415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c102c50 0x7f364c103020 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 -- 172.21.15.63:0/2313662415 >> 172.21.15.63:0/2313662415 conn(0x7f364c0f87e0 msgr2=0x7f364c0fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 -- 172.21.15.63:0/2313662415 shutdown_connections 2024-03-23T11:54:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.292+0000 7f3650a00700 1 -- 172.21.15.63:0/2313662415 wait complete. 2024-03-23T11:54:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.293+0000 7f3650a00700 1 Processor -- start 2024-03-23T11:54:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.294+0000 7f3650a00700 1 -- start start 2024-03-23T11:54:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.294+0000 7f3650a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c102c50 0x7f364c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.294+0000 7f3650a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c1035f0 0x7f364c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.294+0000 7f3650a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c070d90 con 0x7f364c1035f0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.294+0000 7f3650a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c070ed0 con 0x7f364c102c50 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3649e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c1035f0 0x7f364c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3649e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c1035f0 0x7f364c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33820/0 (socket says 172.21.15.63:33820) 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3649e00700 1 -- 172.21.15.63:0/2057588194 learned_addr learned my addr 172.21.15.63:0/2057588194 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f364a800700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c102c50 0x7f364c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3649e00700 1 -- 172.21.15.63:0/2057588194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c102c50 msgr2=0x7f364c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3649e00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c102c50 0x7f364c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3649e00700 1 -- 172.21.15.63:0/2057588194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c005040 con 0x7f364c1035f0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3649e00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c1035f0 0x7f364c070800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f363800eb40 tx=0x7f363800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3642a00700 1 -- 172.21.15.63:0/2057588194 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3638007cc0 con 0x7f364c1035f0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3642a00700 1 -- 172.21.15.63:0/2057588194 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f363800f040 con 0x7f364c1035f0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c071120 con 0x7f364c1035f0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.295+0000 7f3642a00700 1 -- 172.21.15.63:0/2057588194 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3638009570 con 0x7f364c1035f0 2024-03-23T11:54:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.296+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c119fa0 con 0x7f364c1035f0 2024-03-23T11:54:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.297+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c0fe3a0 con 0x7f364c1035f0 2024-03-23T11:54:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.298+0000 7f3642a00700 1 -- 172.21.15.63:0/2057588194 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f36380096d0 con 0x7f364c1035f0 2024-03-23T11:54:57.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.298+0000 7f3642a00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f363006c7d0 0x7f363006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:57.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.299+0000 7f3642a00700 1 -- 172.21.15.63:0/2057588194 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f363808b4b0 con 0x7f364c1035f0 2024-03-23T11:54:57.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.299+0000 7f364a800700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f363006c7d0 0x7f363006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:57.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.299+0000 7f364a800700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f363006c7d0 0x7f363006ec80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f363c00b0d0 tx=0x7f363c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:57.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.303+0000 7f3642a00700 1 -- 172.21.15.63:0/2057588194 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f36380557e0 con 0x7f364c1035f0 2024-03-23T11:54:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.537+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f364c109c00 con 0x7f363006c7d0 2024-03-23T11:54:57.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.539+0000 7f3642a00700 1 -- 172.21.15.63:0/2057588194 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f364c109c00 con 0x7f363006c7d0 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "8/21 daemons upgraded", 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading osd daemons", 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:54:57.523 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:54:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f363006c7d0 msgr2=0x7f363006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f363006c7d0 0x7f363006ec80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f363c00b0d0 tx=0x7f363c009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c1035f0 msgr2=0x7f364c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c1035f0 0x7f364c070800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f363800eb40 tx=0x7f363800b040 comp rx=0 tx=0).stop 2024-03-23T11:54:57.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 shutdown_connections 2024-03-23T11:54:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f363006c7d0 0x7f363006ec80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c102c50 0x7f364c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 --2- 172.21.15.63:0/2057588194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f364c1035f0 0x7f364c070800 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 >> 172.21.15.63:0/2057588194 conn(0x7f364c0f87e0 msgr2=0x7f364c0f97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 shutdown_connections 2024-03-23T11:54:57.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.541+0000 7f3650a00700 1 -- 172.21.15.63:0/2057588194 wait complete. 2024-03-23T11:54:57.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 -- 172.21.15.63:0/3685374230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 msgr2=0x7fe348109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 --2- 172.21.15.63:0/3685374230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348109190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe334004970 tx=0x7fe334009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 -- 172.21.15.63:0/3685374230 shutdown_connections 2024-03-23T11:54:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 --2- 172.21.15.63:0/3685374230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348109190 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 --2- 172.21.15.63:0/3685374230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe348070300 0x7fe3480706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 -- 172.21.15.63:0/3685374230 >> 172.21.15.63:0/3685374230 conn(0x7fe3480f8970 msgr2=0x7fe3480fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 -- 172.21.15.63:0/3685374230 shutdown_connections 2024-03-23T11:54:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.678+0000 7fe34f000700 1 -- 172.21.15.63:0/3685374230 wait complete. 2024-03-23T11:54:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.679+0000 7fe34f000700 1 Processor -- start 2024-03-23T11:54:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.679+0000 7fe34f000700 1 -- start start 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.679+0000 7fe34f000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe348070300 0x7fe348115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.679+0000 7fe34f000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.679+0000 7fe34f000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe348116850 con 0x7fe348070c10 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.679+0000 7fe34f000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34810fc20 con 0x7fe348070300 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.679+0000 7fe347400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe347400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33840/0 (socket says 172.21.15.63:33840) 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe347400700 1 -- 172.21.15.63:0/957780291 learned_addr learned my addr 172.21.15.63:0/957780291 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:54:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe347e00700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe348070300 0x7fe348115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:57.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe347400700 1 -- 172.21.15.63:0/957780291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe348070300 msgr2=0x7fe348115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe347400700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe348070300 0x7fe348115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe347400700 1 -- 172.21.15.63:0/957780291 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe334005040 con 0x7fe348070c10 2024-03-23T11:54:57.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe347400700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348116200 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe334004f50 tx=0x7fe334009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:57.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe33fe00700 1 -- 172.21.15.63:0/957780291 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33400e040 con 0x7fe348070c10 2024-03-23T11:54:57.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe33fe00700 1 -- 172.21.15.63:0/957780291 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe334013420 con 0x7fe348070c10 2024-03-23T11:54:57.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe33fe00700 1 -- 172.21.15.63:0/957780291 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33401c630 con 0x7fe348070c10 2024-03-23T11:54:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.680+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34810fea0 con 0x7fe348070c10 2024-03-23T11:54:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.681+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3481103e0 con 0x7fe348070c10 2024-03-23T11:54:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.682+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe348071a40 con 0x7fe348070c10 2024-03-23T11:54:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.683+0000 7fe33fe00700 1 -- 172.21.15.63:0/957780291 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe334013590 con 0x7fe348070c10 2024-03-23T11:54:57.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.684+0000 7fe33fe00700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32c06c7d0 0x7fe32c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:54:57.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.684+0000 7fe33fe00700 1 -- 172.21.15.63:0/957780291 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe33408ca90 con 0x7fe348070c10 2024-03-23T11:54:57.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.684+0000 7fe347e00700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32c06c7d0 0x7fe32c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:54:57.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.684+0000 7fe347e00700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32c06c7d0 0x7fe32c06ec80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe338009fd0 tx=0x7fe33800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:54:57.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.688+0000 7fe33fe00700 1 -- 172.21.15.63:0/957780291 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe33405a8e0 con 0x7fe348070c10 2024-03-23T11:54:57.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.987+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe34804bed0 con 0x7fe348070c10 2024-03-23T11:54:57.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.987+0000 7fe33fe00700 1 -- 172.21.15.63:0/957780291 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fe33405a470 con 0x7fe348070c10 2024-03-23T11:54:57.970 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T11:54:57.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32c06c7d0 msgr2=0x7fe32c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32c06c7d0 0x7fe32c06ec80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe338009fd0 tx=0x7fe33800b040 comp rx=0 tx=0).stop 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 msgr2=0x7fe348116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348116200 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe334004f50 tx=0x7fe334009e30 comp rx=0 tx=0).stop 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 shutdown_connections 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32c06c7d0 0x7fe32c06ec80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe348070300 0x7fe348115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 --2- 172.21.15.63:0/957780291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe348070c10 0x7fe348116200 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 >> 172.21.15.63:0/957780291 conn(0x7fe3480f8970 msgr2=0x7fe3480f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 shutdown_connections 2024-03-23T11:54:57.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:54:57.990+0000 7fe34f000700 1 -- 172.21.15.63:0/957780291 wait complete. 2024-03-23T11:54:58.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:57 smithi063 ceph-mon[141124]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:58.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:57 smithi063 ceph-mon[141124]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:58.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:57 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3636782824' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:54:58.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:57 smithi111 ceph-mon[106899]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:58.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:57 smithi111 ceph-mon[106899]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:58.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:57 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3636782824' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:54:59.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:58 smithi063 ceph-mon[141124]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:59.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:58 smithi063 ceph-mon[141124]: pgmap v89: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-03-23T11:54:59.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:54:58 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/957780291' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:54:59.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:58 smithi111 ceph-mon[106899]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:54:59.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:58 smithi111 ceph-mon[106899]: pgmap v89: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-03-23T11:54:59.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:54:58 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/957780291' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:55:01.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:00 smithi063 ceph-mon[141124]: pgmap v90: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:55:01.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:00 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:01.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:00 smithi111 ceph-mon[106899]: pgmap v90: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:55:01.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:00 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:03.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:02 smithi063 ceph-mon[141124]: pgmap v91: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 3 op/s 2024-03-23T11:55:03.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:02 smithi111 ceph-mon[106899]: pgmap v91: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 3 op/s 2024-03-23T11:55:05.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:04 smithi063 ceph-mon[141124]: pgmap v92: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:55:05.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:04 smithi111 ceph-mon[106899]: pgmap v92: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:55:06.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:06 smithi063 ceph-mon[141124]: pgmap v93: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:55:06.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:06 smithi111 ceph-mon[106899]: pgmap v93: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:55:09.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:08 smithi063 ceph-mon[141124]: pgmap v94: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:09.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:08 smithi111 ceph-mon[106899]: pgmap v94: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:10.876 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:10 smithi063 ceph-mon[141124]: pgmap v95: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:10.876 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:55:10.877 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:10 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:55:10.877 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:10 smithi063 ceph-mon[141124]: Upgrade: osd.2 is safe to restart 2024-03-23T11:55:10.877 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:10.877 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:10 smithi111 ceph-mon[106899]: pgmap v95: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:55:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:10 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-23T11:55:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:10 smithi111 ceph-mon[106899]: Upgrade: osd.2 is safe to restart 2024-03-23T11:55:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:11.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:12.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:12 smithi063 ceph-mon[141124]: Upgrade: Updating osd.2 2024-03-23T11:55:12.912 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:12.913 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T11:55:12.913 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:55:12.913 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:12 smithi063 ceph-mon[141124]: Deploying daemon osd.2 on smithi063 2024-03-23T11:55:12.913 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:12 smithi063 ceph-mon[141124]: pgmap v96: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:12.913 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:12 smithi111 ceph-mon[106899]: Upgrade: Updating osd.2 2024-03-23T11:55:12.914 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:12.914 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-23T11:55:12.914 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:55:12.914 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:12 smithi111 ceph-mon[106899]: Deploying daemon osd.2 on smithi063 2024-03-23T11:55:12.914 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:12 smithi111 ceph-mon[106899]: pgmap v96: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:14.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:14 smithi063 ceph-mon[141124]: pgmap v97: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:15.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:14 smithi111 ceph-mon[106899]: pgmap v97: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:16.090 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:15 smithi063 systemd[1]: Stopping Ceph osd.2 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:55:16.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:16 smithi063 ceph-mon[141124]: pgmap v98: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:16.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:16 smithi063 ceph-mon[141124]: osd.2 marked itself down and dead 2024-03-23T11:55:16.382 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:16 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[77498]: 2024-03-23T11:55:16.106+0000 7ff431200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:55:16.382 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:16 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[77498]: 2024-03-23T11:55:16.106+0000 7ff431200700 -1 osd.2 59 *** Got signal Terminated *** 2024-03-23T11:55:16.382 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:16 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[77498]: 2024-03-23T11:55:16.106+0000 7ff431200700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T11:55:16.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:16 smithi111 ceph-mon[106899]: pgmap v98: 65 pgs: 65 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:16.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:16 smithi111 ceph-mon[106899]: osd.2 marked itself down and dead 2024-03-23T11:55:17.321 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:17 smithi063 bash[167683]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2 2024-03-23T11:55:17.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:17 smithi063 ceph-mon[141124]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:55:17.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:17 smithi063 ceph-mon[141124]: osdmap e60: 6 total, 5 up, 6 in 2024-03-23T11:55:17.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:17 smithi111 ceph-mon[106899]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:55:17.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:17 smithi111 ceph-mon[106899]: osdmap e60: 6 total, 5 up, 6 in 2024-03-23T11:55:18.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:18 smithi063 ceph-mon[141124]: osdmap e61: 6 total, 5 up, 6 in 2024-03-23T11:55:18.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:18 smithi063 ceph-mon[141124]: pgmap v101: 65 pgs: 12 peering, 7 stale+active+clean, 46 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-23T11:55:18.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:18 smithi111 ceph-mon[106899]: osdmap e61: 6 total, 5 up, 6 in 2024-03-23T11:55:18.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:18 smithi111 ceph-mon[106899]: pgmap v101: 65 pgs: 12 peering, 7 stale+active+clean, 46 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-23T11:55:19.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:19 smithi063 ceph-mon[141124]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-23T11:55:19.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:19 smithi111 ceph-mon[106899]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-23T11:55:19.797 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:19 smithi063 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.2.service: Succeeded. 2024-03-23T11:55:19.797 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:19 smithi063 systemd[1]: Stopped Ceph osd.2 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:55:20.131 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:19 smithi063 systemd[1]: Starting Ceph osd.2 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:55:20.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:20 smithi063 ceph-mon[141124]: pgmap v102: 65 pgs: 7 active+undersized+degraded, 12 peering, 8 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 26/330 objects degraded (7.879%) 2024-03-23T11:55:20.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:20 smithi111 ceph-mon[106899]: pgmap v102: 65 pgs: 7 active+undersized+degraded, 12 peering, 8 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 26/330 objects degraded (7.879%) 2024-03-23T11:55:21.555 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:21 smithi063 ceph-mon[141124]: Health check failed: Degraded data redundancy: 26/330 objects degraded (7.879%), 7 pgs degraded (PG_DEGRADED) 2024-03-23T11:55:21.555 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:21 smithi063 bash[167955]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-23T11:55:21.555 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:21 smithi063 bash[167955]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-23T11:55:21.555 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:21 smithi063 bash[167955]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-23T11:55:21.555 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:21 smithi063 bash[167955]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-23T11:55:21.555 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:21 smithi063 bash[167955]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-23T11:55:21.556 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:21 smithi063 bash[167955]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-23T11:55:21.556 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:21 smithi063 bash[167955]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-23T11:55:21.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:21 smithi111 ceph-mon[106899]: Health check failed: Degraded data redundancy: 26/330 objects degraded (7.879%), 7 pgs degraded (PG_DEGRADED) 2024-03-23T11:55:22.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:22 smithi063 ceph-mon[141124]: pgmap v103: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:22.632 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:22 smithi063 ceph-osd[168249]: -- 172.21.15.63:0/2897448316 <== mon.0 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558e95bb51e0 con 0x558e95b85000 2024-03-23T11:55:22.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:22 smithi111 ceph-mon[106899]: pgmap v103: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:23.063 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:22 smithi063 bash[168210]: 03b76453ce489fdd637192d9d3748cea7d6cae28f4554100b6b5287bc8618a0b 2024-03-23T11:55:23.063 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:22 smithi063 systemd[1]: Started Ceph osd.2 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:55:23.343 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:23 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[168226]: 2024-03-23T11:55:23.079+0000 7f92b8e91640 -1 Falling back to public interface 2024-03-23T11:55:23.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:23 smithi063 ceph-mon[141124]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-23T11:55:23.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:23 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:23.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:23 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:23.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:23 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:55:23.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:23 smithi111 ceph-mon[106899]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-23T11:55:23.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:23 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:23.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:23 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:23.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:23 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:55:24.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:24 smithi063 ceph-mon[141124]: pgmap v104: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:24.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:24 smithi111 ceph-mon[106899]: pgmap v104: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:26.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:26 smithi063 ceph-mon[141124]: pgmap v105: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:26.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:26.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:26 smithi111 ceph-mon[106899]: pgmap v105: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:27.881 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:27 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[168226]: 2024-03-23T11:55:27.567+0000 7f92b8e91640 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 -- 172.21.15.63:0/4046659879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 msgr2=0x7f92840fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 --2- 172.21.15.63:0/4046659879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f92840fce30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9274004940 tx=0x7f9274009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 -- 172.21.15.63:0/4046659879 shutdown_connections 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 --2- 172.21.15.63:0/4046659879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f92840fce30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 --2- 172.21.15.63:0/4046659879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9284102e20 0x7f92841031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 -- 172.21.15.63:0/4046659879 >> 172.21.15.63:0/4046659879 conn(0x7f92840f89d0 msgr2=0x7f92840fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 -- 172.21.15.63:0/4046659879 shutdown_connections 2024-03-23T11:55:28.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.180+0000 7f928c800700 1 -- 172.21.15.63:0/4046659879 wait complete. 2024-03-23T11:55:28.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.182+0000 7f928c800700 1 Processor -- start 2024-03-23T11:55:28.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.182+0000 7f928c800700 1 -- start start 2024-03-23T11:55:28.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.182+0000 7f928c800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9284102e20 0x7f92840702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.182+0000 7f928c800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f9284070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.182+0000 7f928c800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284070d90 con 0x7f92841037c0 2024-03-23T11:55:28.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.182+0000 7f928c800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284070ed0 con 0x7f9284102e20 2024-03-23T11:55:28.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f9284070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f9284070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36948/0 (socket says 172.21.15.63:36948) 2024-03-23T11:55:28.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289400700 1 -- 172.21.15.63:0/2689604346 learned_addr learned my addr 172.21.15.63:0/2689604346 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:55:28.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289e00700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9284102e20 0x7f92840702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289400700 1 -- 172.21.15.63:0/2689604346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9284102e20 msgr2=0x7f92840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289400700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9284102e20 0x7f92840702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289400700 1 -- 172.21.15.63:0/2689604346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9274005040 con 0x7f92841037c0 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289e00700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9284102e20 0x7f92840702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.183+0000 7f9289400700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f9284070800 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9274009390 tx=0x7f9274009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.184+0000 7f927e000700 1 -- 172.21.15.63:0/2689604346 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f927400e040 con 0x7f92841037c0 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.184+0000 7f927e000700 1 -- 172.21.15.63:0/2689604346 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9274013420 con 0x7f92841037c0 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.184+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9284071120 con 0x7f92841037c0 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.184+0000 7f927e000700 1 -- 172.21.15.63:0/2689604346 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f927401c630 con 0x7f92841037c0 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.184+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928411a200 con 0x7f92841037c0 2024-03-23T11:55:28.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.185+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92840fe570 con 0x7f92841037c0 2024-03-23T11:55:28.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.186+0000 7f927e000700 1 -- 172.21.15.63:0/2689604346 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f92740139e0 con 0x7f92841037c0 2024-03-23T11:55:28.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.187+0000 7f927e000700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926c06c7d0 0x7f926c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.187+0000 7f927e000700 1 -- 172.21.15.63:0/2689604346 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f927408cca0 con 0x7f92841037c0 2024-03-23T11:55:28.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.187+0000 7f9289e00700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926c06c7d0 0x7f926c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.188+0000 7f9289e00700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926c06c7d0 0x7f926c06ec80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9278009fd0 tx=0x7f9278009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:28.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.192+0000 7f927e000700 1 -- 172.21.15.63:0/2689604346 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9274091070 con 0x7f92841037c0 2024-03-23T11:55:28.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.428+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9284109e60 con 0x7f926c06c7d0 2024-03-23T11:55:28.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.430+0000 7f927e000700 1 -- 172.21.15.63:0/2689604346 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9278009970 con 0x7f926c06c7d0 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926c06c7d0 msgr2=0x7f926c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926c06c7d0 0x7f926c06ec80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9278009fd0 tx=0x7f9278009ee0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 msgr2=0x7f9284070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f9284070800 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9274009390 tx=0x7f9274009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 shutdown_connections 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926c06c7d0 0x7f926c06ec80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9284102e20 0x7f92840702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 --2- 172.21.15.63:0/2689604346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92841037c0 0x7f9284070800 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.432+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 >> 172.21.15.63:0/2689604346 conn(0x7f92840f89d0 msgr2=0x7f92840f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.433+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 shutdown_connections 2024-03-23T11:55:28.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.433+0000 7f928c800700 1 -- 172.21.15.63:0/2689604346 wait complete. 2024-03-23T11:55:28.430 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:55:28.455 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:28 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[168226]: 2024-03-23T11:55:28.158+0000 7f92b8e91640 -1 osd.2 59 log_to_monitors true 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.575+0000 7fd348a00700 1 -- 172.21.15.63:0/2865110391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 msgr2=0x7fd344103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.575+0000 7fd348a00700 1 --2- 172.21.15.63:0/2865110391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344103010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd334004940 tx=0x7fd334009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.576+0000 7fd348a00700 1 -- 172.21.15.63:0/2865110391 shutdown_connections 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.576+0000 7fd348a00700 1 --2- 172.21.15.63:0/2865110391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3441035e0 0x7fd3440fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.576+0000 7fd348a00700 1 --2- 172.21.15.63:0/2865110391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344103010 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.576+0000 7fd348a00700 1 -- 172.21.15.63:0/2865110391 >> 172.21.15.63:0/2865110391 conn(0x7fd3440f8830 msgr2=0x7fd3440fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.576+0000 7fd348a00700 1 -- 172.21.15.63:0/2865110391 shutdown_connections 2024-03-23T11:55:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.576+0000 7fd348a00700 1 -- 172.21.15.63:0/2865110391 wait complete. 2024-03-23T11:55:28.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.576+0000 7fd348a00700 1 Processor -- start 2024-03-23T11:55:28.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd348a00700 1 -- start start 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd348a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd348a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3441035e0 0x7fd344115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd348a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd344116630 con 0x7fd344102c40 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd348a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34410fa20 con 0x7fd3441035e0 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd342800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd342800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344115aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36956/0 (socket says 172.21.15.63:36956) 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.577+0000 7fd342800700 1 -- 172.21.15.63:0/1154096629 learned_addr learned my addr 172.21.15.63:0/1154096629 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:55:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd341e00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3441035e0 0x7fd344115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd342800700 1 -- 172.21.15.63:0/1154096629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3441035e0 msgr2=0x7fd344115fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd342800700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3441035e0 0x7fd344115fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd342800700 1 -- 172.21.15.63:0/1154096629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334005040 con 0x7fd344102c40 2024-03-23T11:55:28.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd342800700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344115aa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd334007f50 tx=0x7fd334009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:28.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd33aa00700 1 -- 172.21.15.63:0/1154096629 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd334021070 con 0x7fd344102c40 2024-03-23T11:55:28.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd33aa00700 1 -- 172.21.15.63:0/1154096629 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3340093b0 con 0x7fd344102c40 2024-03-23T11:55:28.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34410fca0 con 0x7fd344102c40 2024-03-23T11:55:28.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.578+0000 7fd33aa00700 1 -- 172.21.15.63:0/1154096629 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd33400e070 con 0x7fd344102c40 2024-03-23T11:55:28.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.579+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd344110200 con 0x7fd344102c40 2024-03-23T11:55:28.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.580+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3440fe390 con 0x7fd344102c40 2024-03-23T11:55:28.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.580+0000 7fd33aa00700 1 -- 172.21.15.63:0/1154096629 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd33401c400 con 0x7fd344102c40 2024-03-23T11:55:28.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.581+0000 7fd33aa00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd32806c7d0 0x7fd32806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.581+0000 7fd33aa00700 1 -- 172.21.15.63:0/1154096629 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd33408cae0 con 0x7fd344102c40 2024-03-23T11:55:28.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.582+0000 7fd341e00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd32806c7d0 0x7fd32806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.582+0000 7fd341e00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd32806c7d0 0x7fd32806ec80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd344110be0 tx=0x7fd33000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:28.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.586+0000 7fd33aa00700 1 -- 172.21.15.63:0/1154096629 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd33405a870 con 0x7fd344102c40 2024-03-23T11:55:28.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.833+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd344109bf0 con 0x7fd32806c7d0 2024-03-23T11:55:28.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.836+0000 7fd33aa00700 1 -- 172.21.15.63:0/1154096629 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd344109bf0 con 0x7fd32806c7d0 2024-03-23T11:55:28.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd32806c7d0 msgr2=0x7fd32806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd32806c7d0 0x7fd32806ec80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd344110be0 tx=0x7fd33000b040 comp rx=0 tx=0).stop 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 msgr2=0x7fd344115aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344115aa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd334007f50 tx=0x7fd334009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 shutdown_connections 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd32806c7d0 0x7fd32806ec80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd344102c40 0x7fd344115aa0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 --2- 172.21.15.63:0/1154096629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3441035e0 0x7fd344115fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.838+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 >> 172.21.15.63:0/1154096629 conn(0x7fd3440f8830 msgr2=0x7fd3440f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.839+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 shutdown_connections 2024-03-23T11:55:28.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.839+0000 7fd348a00700 1 -- 172.21.15.63:0/1154096629 wait complete. 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 -- 172.21.15.63:0/1607603432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 msgr2=0x7f11f0070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 --2- 172.21.15.63:0/1607603432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f0070d10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f11dc004970 tx=0x7f11dc009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 -- 172.21.15.63:0/1607603432 shutdown_connections 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 --2- 172.21.15.63:0/1607603432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f0070d10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 --2- 172.21.15.63:0/1607603432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f0071f90 0x7f11f0070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 -- 172.21.15.63:0/1607603432 >> 172.21.15.63:0/1607603432 conn(0x7f11f00f89b0 msgr2=0x7f11f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 -- 172.21.15.63:0/1607603432 shutdown_connections 2024-03-23T11:55:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.980+0000 7f11f6400700 1 -- 172.21.15.63:0/1607603432 wait complete. 2024-03-23T11:55:28.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.981+0000 7f11f6400700 1 Processor -- start 2024-03-23T11:55:28.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.981+0000 7f11f6400700 1 -- start start 2024-03-23T11:55:28.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11f6400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11f6400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f0071f90 0x7f11f011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11f6400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f011ac40 con 0x7f11f00708a0 2024-03-23T11:55:28.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11f6400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0114010 con 0x7f11f0071f90 2024-03-23T11:55:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11ef400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11ef400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f011a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36980/0 (socket says 172.21.15.63:36980) 2024-03-23T11:55:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11ef400700 1 -- 172.21.15.63:0/3574635267 learned_addr learned my addr 172.21.15.63:0/3574635267 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:55:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.982+0000 7f11eea00700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f0071f90 0x7f11f011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11ef400700 1 -- 172.21.15.63:0/3574635267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f0071f90 msgr2=0x7f11f011a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11ef400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f0071f90 0x7f11f011a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11ef400700 1 -- 172.21.15.63:0/3574635267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc005040 con 0x7f11f00708a0 2024-03-23T11:55:28.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11ef400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f011a0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11e000eb40 tx=0x7f11e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:28.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11e7400700 1 -- 172.21.15.63:0/3574635267 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e0007cc0 con 0x7f11f00708a0 2024-03-23T11:55:28.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11e7400700 1 -- 172.21.15.63:0/3574635267 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11e000f040 con 0x7f11f00708a0 2024-03-23T11:55:28.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f0114290 con 0x7f11f00708a0 2024-03-23T11:55:28.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11e7400700 1 -- 172.21.15.63:0/3574635267 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e000c990 con 0x7f11f00708a0 2024-03-23T11:55:28.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.983+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f01147d0 con 0x7f11f00708a0 2024-03-23T11:55:28.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.985+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f0071aa0 con 0x7f11f00708a0 2024-03-23T11:55:28.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.986+0000 7f11e7400700 1 -- 172.21.15.63:0/3574635267 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f11e000caf0 con 0x7f11f00708a0 2024-03-23T11:55:28.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.987+0000 7f11e7400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11d406c7d0 0x7f11d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:28.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.987+0000 7f11e7400700 1 -- 172.21.15.63:0/3574635267 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f11e008b5c0 con 0x7f11f00708a0 2024-03-23T11:55:28.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.988+0000 7f11eea00700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11d406c7d0 0x7f11d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:28.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.989+0000 7f11eea00700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11d406c7d0 0x7f11d406ec80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11f0115950 tx=0x7f11dc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:28.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:28.991+0000 7f11e7400700 1 -- 172.21.15.63:0/3574635267 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f11e0059380 con 0x7f11f00708a0 2024-03-23T11:55:29.131 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 11:55:28 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[168226]: 2024-03-23T11:55:28.790+0000 7f92a6e00700 -1 osd.2 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:55:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:28 smithi063 ceph-mon[141124]: pgmap v106: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:28 smithi063 ceph-mon[141124]: from='osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T11:55:29.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:28 smithi111 ceph-mon[106899]: pgmap v106: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:29.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:28 smithi111 ceph-mon[106899]: from='osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-23T11:55:29.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.225+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f11f004bf40 con 0x7f11d406c7d0 2024-03-23T11:55:29.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.239+0000 7f11e7400700 1 -- 172.21.15.63:0/3574635267 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f11f004bf40 con 0x7f11d406c7d0 2024-03-23T11:55:29.222 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:55:29.222 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (4m) 0s ago 17m 14.2M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:55:29.222 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (118s) 0s ago 18m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (113s) 108s ago 16m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (4m) 0s ago 17m 73.4M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (11m) 0s ago 11m 118M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (11m) 0s ago 11m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (11m) 108s ago 11m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (11m) 108s ago 11m 60.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (7m) 0s ago 19m 502M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (5m) 108s ago 16m 450M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (3m) 0s ago 19m 45.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (2m) 108s ago 16m 36.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (5m) 0s ago 18m 7532k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (5m) 108s ago 16m 7860k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (87s) 0s ago 15m 87.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (47s) 0s ago 15m 91.1M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (7s) 0s ago 14m 11.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (14m) 108s ago 14m 259M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:55:29.223 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (13m) 108s ago 13m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:55:29.224 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (13m) 108s ago 13m 228M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:55:29.224 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (5m) 0s ago 17m 53.4M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11d406c7d0 msgr2=0x7f11d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11d406c7d0 0x7f11d406ec80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11f0115950 tx=0x7f11dc009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 msgr2=0x7f11f011a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f011a0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11e000eb40 tx=0x7f11e000b040 comp rx=0 tx=0).stop 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 shutdown_connections 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11d406c7d0 0x7f11d406ec80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11f00708a0 0x7f11f011a0b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 --2- 172.21.15.63:0/3574635267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11f0071f90 0x7f11f011a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 >> 172.21.15.63:0/3574635267 conn(0x7f11f00f89b0 msgr2=0x7f11f00f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.241+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 shutdown_connections 2024-03-23T11:55:29.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.242+0000 7f11f6400700 1 -- 172.21.15.63:0/3574635267 wait complete. 2024-03-23T11:55:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 -- 172.21.15.63:0/392080745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 msgr2=0x7f18340fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 --2- 172.21.15.63:0/392080745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f18340fe1f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1824004940 tx=0x7f1824009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 -- 172.21.15.63:0/392080745 shutdown_connections 2024-03-23T11:55:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 --2- 172.21.15.63:0/392080745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18340fe7c0 0x7f1834106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 --2- 172.21.15.63:0/392080745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f18340fe1f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 -- 172.21.15.63:0/392080745 >> 172.21.15.63:0/392080745 conn(0x7f18340f7870 msgr2=0x7f18340f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 -- 172.21.15.63:0/392080745 shutdown_connections 2024-03-23T11:55:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.379+0000 7f183a600700 1 -- 172.21.15.63:0/392080745 wait complete. 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.380+0000 7f183a600700 1 Processor -- start 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.380+0000 7f183a600700 1 -- start start 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.380+0000 7f183a600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f1834115b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.380+0000 7f183a600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18340fe7c0 0x7f18341160b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.380+0000 7f183a600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1834116700 con 0x7f18340fde20 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.380+0000 7f183a600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183410fce0 con 0x7f18340fe7c0 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.380+0000 7f1833400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f1834115b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f1833400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f1834115b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36996/0 (socket says 172.21.15.63:36996) 2024-03-23T11:55:29.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f1833400700 1 -- 172.21.15.63:0/1927181591 learned_addr learned my addr 172.21.15.63:0/1927181591 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:55:29.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f182be00700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18340fe7c0 0x7f18341160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:29.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f1833400700 1 -- 172.21.15.63:0/1927181591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18340fe7c0 msgr2=0x7f18341160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f1833400700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18340fe7c0 0x7f18341160b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f1833400700 1 -- 172.21.15.63:0/1927181591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824005040 con 0x7f18340fde20 2024-03-23T11:55:29.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f1833400700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f1834115b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f182400b3b0 tx=0x7f1824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:29.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f182b400700 1 -- 172.21.15.63:0/1927181591 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f182400e040 con 0x7f18340fde20 2024-03-23T11:55:29.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f182b400700 1 -- 172.21.15.63:0/1927181591 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1824013420 con 0x7f18340fde20 2024-03-23T11:55:29.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.381+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183410ff60 con 0x7f18340fde20 2024-03-23T11:55:29.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.382+0000 7f182b400700 1 -- 172.21.15.63:0/1927181591 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f182401c630 con 0x7f18340fde20 2024-03-23T11:55:29.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.382+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18341104a0 con 0x7f18340fde20 2024-03-23T11:55:29.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.383+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1834066b50 con 0x7f18340fde20 2024-03-23T11:55:29.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.384+0000 7f182b400700 1 -- 172.21.15.63:0/1927181591 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f1824013590 con 0x7f18340fde20 2024-03-23T11:55:29.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.385+0000 7f182b400700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181806c7c0 0x7f181806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:29.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.386+0000 7f182b400700 1 -- 172.21.15.63:0/1927181591 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f182408c5d0 con 0x7f18340fde20 2024-03-23T11:55:29.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.386+0000 7f182be00700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181806c7c0 0x7f181806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:29.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.387+0000 7f182be00700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181806c7c0 0x7f181806ec70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1834111410 tx=0x7f181c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:29.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.390+0000 7f182b400700 1 -- 172.21.15.63:0/1927181591 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f182405a2e0 con 0x7f18340fde20 2024-03-23T11:55:29.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.744+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f18341111c0 con 0x7f18340fde20 2024-03-23T11:55:29.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.745+0000 7f182b400700 1 -- 172.21.15.63:0/1927181591 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7f1824059e70 con 0x7f18340fde20 2024-03-23T11:55:29.728 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:55:29.728 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:55:29.728 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 3, 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 7, 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:55:29.729 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181806c7c0 msgr2=0x7f181806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181806c7c0 0x7f181806ec70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1834111410 tx=0x7f181c00b040 comp rx=0 tx=0).stop 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 msgr2=0x7f1834115b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f1834115b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f182400b3b0 tx=0x7f1824009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 shutdown_connections 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181806c7c0 0x7f181806ec70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18340fde20 0x7f1834115b70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 --2- 172.21.15.63:0/1927181591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18340fe7c0 0x7f18341160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 >> 172.21.15.63:0/1927181591 conn(0x7f18340f7870 msgr2=0x7f18340f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.747+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 shutdown_connections 2024-03-23T11:55:29.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.748+0000 7f183a600700 1 -- 172.21.15.63:0/1927181591 wait complete. 2024-03-23T11:55:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 -- 172.21.15.63:0/1557107943 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c0708a0 msgr2=0x7f034c070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 --2- 172.21.15.63:0/1557107943 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c0708a0 0x7f034c070d10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0338004970 tx=0x7f0338009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:29.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 -- 172.21.15.63:0/1557107943 shutdown_connections 2024-03-23T11:55:29.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 --2- 172.21.15.63:0/1557107943 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c0708a0 0x7f034c070d10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 --2- 172.21.15.63:0/1557107943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f034c071f90 0x7f034c070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 -- 172.21.15.63:0/1557107943 >> 172.21.15.63:0/1557107943 conn(0x7f034c0f89b0 msgr2=0x7f034c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:29.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 -- 172.21.15.63:0/1557107943 shutdown_connections 2024-03-23T11:55:29.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.910+0000 7f0352400700 1 -- 172.21.15.63:0/1557107943 wait complete. 2024-03-23T11:55:29.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.911+0000 7f0352400700 1 Processor -- start 2024-03-23T11:55:29.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.911+0000 7f0352400700 1 -- start start 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.911+0000 7f0352400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f034c0708a0 0x7f034c119ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f0352400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c071f90 0x7f034c11a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f0352400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c11aa30 con 0x7f034c071f90 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f0352400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c114010 con 0x7f034c0708a0 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c071f90 0x7f034c11a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c071f90 0x7f034c11a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37008/0 (socket says 172.21.15.63:37008) 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034aa00700 1 -- 172.21.15.63:0/825677244 learned_addr learned my addr 172.21.15.63:0/825677244 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034b400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f034c0708a0 0x7f034c119ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034aa00700 1 -- 172.21.15.63:0/825677244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f034c0708a0 msgr2=0x7f034c119ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034aa00700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f034c0708a0 0x7f034c119ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034aa00700 1 -- 172.21.15.63:0/825677244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0338005040 con 0x7f034c071f90 2024-03-23T11:55:29.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.912+0000 7f034aa00700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c071f90 0x7f034c11a3e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0338004f50 tx=0x7f0338009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:29.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.913+0000 7f0343400700 1 -- 172.21.15.63:0/825677244 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033800e040 con 0x7f034c071f90 2024-03-23T11:55:29.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.913+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c114290 con 0x7f034c071f90 2024-03-23T11:55:29.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.913+0000 7f0343400700 1 -- 172.21.15.63:0/825677244 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0338013420 con 0x7f034c071f90 2024-03-23T11:55:29.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.913+0000 7f0343400700 1 -- 172.21.15.63:0/825677244 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033801c610 con 0x7f034c071f90 2024-03-23T11:55:29.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.913+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c1147d0 con 0x7f034c071f90 2024-03-23T11:55:29.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.914+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c071aa0 con 0x7f034c071f90 2024-03-23T11:55:29.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.916+0000 7f0343400700 1 -- 172.21.15.63:0/825677244 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0338013590 con 0x7f034c071f90 2024-03-23T11:55:29.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.917+0000 7f0343400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f033006c7c0 0x7f033006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:29.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.917+0000 7f0343400700 1 -- 172.21.15.63:0/825677244 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f033808cdc0 con 0x7f034c071f90 2024-03-23T11:55:29.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.917+0000 7f034b400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f033006c7c0 0x7f033006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.918+0000 7f034b400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f033006c7c0 0x7f033006ec70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f033c007fd0 tx=0x7f033c009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:29.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:29.922+0000 7f0343400700 1 -- 172.21.15.63:0/825677244 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0338090030 con 0x7f034c071f90 2024-03-23T11:55:30.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: from='osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-23T11:55:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: osdmap e62: 6 total, 5 up, 6 in 2024-03-23T11:55:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: from='osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:55:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:29 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1927181591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: from='osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: osdmap e62: 6 total, 5 up, 6 in 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: from='osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:30.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:29 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1927181591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:30.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.195+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f034c1103e0 con 0x7f034c071f90 2024-03-23T11:55:30.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.196+0000 7f0343400700 1 -- 172.21.15.63:0/825677244 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f033805aa50 con 0x7f034c071f90 2024-03-23T11:55:30.179 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:55:30.179 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:55:30.179 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:55:30.179 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:55:30.179 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:55:30.180 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:55:30.181 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:55:30.182 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:55:30.182 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:55:30.182 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:55:30.182 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:55:30.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f033006c7c0 msgr2=0x7f033006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f033006c7c0 0x7f033006ec70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f033c007fd0 tx=0x7f033c009040 comp rx=0 tx=0).stop 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c071f90 msgr2=0x7f034c11a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c071f90 0x7f034c11a3e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0338004f50 tx=0x7f0338009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 shutdown_connections 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f033006c7c0 0x7f033006ec70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f034c0708a0 0x7f034c119ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 --2- 172.21.15.63:0/825677244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f034c071f90 0x7f034c11a3e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 >> 172.21.15.63:0/825677244 conn(0x7f034c0f89b0 msgr2=0x7f034c0f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 shutdown_connections 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.199+0000 7f0352400700 1 -- 172.21.15.63:0/825677244 wait complete. 2024-03-23T11:55:30.183 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:55:30.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 -- 172.21.15.63:0/2236342190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 msgr2=0x7f8174101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 --2- 172.21.15.63:0/2236342190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174101f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8164004940 tx=0x7f8164009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 -- 172.21.15.63:0/2236342190 shutdown_connections 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 --2- 172.21.15.63:0/2236342190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81740fbb70 0x7f81740fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 --2- 172.21.15.63:0/2236342190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174101f70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 -- 172.21.15.63:0/2236342190 >> 172.21.15.63:0/2236342190 conn(0x7f8174071b70 msgr2=0x7f8174071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 -- 172.21.15.63:0/2236342190 shutdown_connections 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.339+0000 7f8179200700 1 -- 172.21.15.63:0/2236342190 wait complete. 2024-03-23T11:55:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.340+0000 7f8179200700 1 Processor -- start 2024-03-23T11:55:30.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.340+0000 7f8179200700 1 -- start start 2024-03-23T11:55:30.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8179200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81740fbb70 0x7f8174115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8179200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8179200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8174116650 con 0x7f8174101ba0 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8179200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817410fa40 con 0x7f81740fbb70 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8172a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8172a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37022/0 (socket says 172.21.15.63:37022) 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8172a00700 1 -- 172.21.15.63:0/1603054131 learned_addr learned my addr 172.21.15.63:0/1603054131 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8173400700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81740fbb70 0x7f8174115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8172a00700 1 -- 172.21.15.63:0/1603054131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81740fbb70 msgr2=0x7f8174115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8172a00700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81740fbb70 0x7f8174115ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.341+0000 7f8172a00700 1 -- 172.21.15.63:0/1603054131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164005040 con 0x7f8174101ba0 2024-03-23T11:55:30.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.342+0000 7f8172a00700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174116000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8160002a50 tx=0x7f81600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:30.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.342+0000 7f816b400700 1 -- 172.21.15.63:0/1603054131 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8160007cc0 con 0x7f8174101ba0 2024-03-23T11:55:30.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.342+0000 7f816b400700 1 -- 172.21.15.63:0/1603054131 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f816000f040 con 0x7f8174101ba0 2024-03-23T11:55:30.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.342+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817410fd20 con 0x7f8174101ba0 2024-03-23T11:55:30.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.342+0000 7f816b400700 1 -- 172.21.15.63:0/1603054131 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f816000d9c0 con 0x7f8174101ba0 2024-03-23T11:55:30.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.342+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81741102e0 con 0x7f8174101ba0 2024-03-23T11:55:30.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.343+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81740fd200 con 0x7f8174101ba0 2024-03-23T11:55:30.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.344+0000 7f816b400700 1 -- 172.21.15.63:0/1603054131 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f816000db20 con 0x7f8174101ba0 2024-03-23T11:55:30.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.345+0000 7f816b400700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f815806c7d0 0x7f815806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:30.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.345+0000 7f816b400700 1 -- 172.21.15.63:0/1603054131 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f816008b7a0 con 0x7f8174101ba0 2024-03-23T11:55:30.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.345+0000 7f8173400700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f815806c7d0 0x7f815806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:30.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.346+0000 7f8173400700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f815806c7d0 0x7f815806ec80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8164004560 tx=0x7f8164009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:30.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.354+0000 7f816b400700 1 -- 172.21.15.63:0/1603054131 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8160059430 con 0x7f8174101ba0 2024-03-23T11:55:30.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.603+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8174115790 con 0x7f815806c7d0 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.604+0000 7f816b400700 1 -- 172.21.15.63:0/1603054131 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f8174115790 con 0x7f815806c7d0 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "9/21 daemons upgraded", 2024-03-23T11:55:30.588 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading osd daemons", 2024-03-23T11:55:30.589 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:55:30.589 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:55:30.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f815806c7d0 msgr2=0x7f815806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f815806c7d0 0x7f815806ec80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8164004560 tx=0x7f8164009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:30.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 msgr2=0x7f8174116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174116000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8160002a50 tx=0x7f81600046b0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 shutdown_connections 2024-03-23T11:55:30.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f815806c7d0 0x7f815806ec80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81740fbb70 0x7f8174115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 --2- 172.21.15.63:0/1603054131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8174101ba0 0x7f8174116000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 >> 172.21.15.63:0/1603054131 conn(0x7f8174071b70 msgr2=0x7f81740fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:30.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 shutdown_connections 2024-03-23T11:55:30.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.607+0000 7f8179200700 1 -- 172.21.15.63:0/1603054131 wait complete. 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 -- 172.21.15.63:0/1030861679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4102e20 msgr2=0x7f28d41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 --2- 172.21.15.63:0/1030861679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4102e20 0x7f28d41031f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f28c4004940 tx=0x7f28c4009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 -- 172.21.15.63:0/1030861679 shutdown_connections 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 --2- 172.21.15.63:0/1030861679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d41037c0 0x7f28d40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 --2- 172.21.15.63:0/1030861679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4102e20 0x7f28d41031f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 -- 172.21.15.63:0/1030861679 >> 172.21.15.63:0/1030861679 conn(0x7f28d40f8970 msgr2=0x7f28d40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 -- 172.21.15.63:0/1030861679 shutdown_connections 2024-03-23T11:55:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.757+0000 7f28db000700 1 -- 172.21.15.63:0/1030861679 wait complete. 2024-03-23T11:55:30.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.759+0000 7f28db000700 1 Processor -- start 2024-03-23T11:55:30.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.759+0000 7f28db000700 1 -- start start 2024-03-23T11:55:30.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.759+0000 7f28db000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4102e20 0x7f28d4115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:30.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.759+0000 7f28db000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d41037c0 0x7f28d41161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:30.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.759+0000 7f28db000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4116840 con 0x7f28d41037c0 2024-03-23T11:55:30.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.759+0000 7f28db000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d410fc10 con 0x7f28d4102e20 2024-03-23T11:55:30.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d41037c0 0x7f28d41161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:30.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d41037c0 0x7f28d41161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37046/0 (socket says 172.21.15.63:37046) 2024-03-23T11:55:30.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3400700 1 -- 172.21.15.63:0/2741474770 learned_addr learned my addr 172.21.15.63:0/2741474770 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:55:30.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3e00700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4102e20 0x7f28d4115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:30.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3400700 1 -- 172.21.15.63:0/2741474770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4102e20 msgr2=0x7f28d4115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3400700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4102e20 0x7f28d4115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3400700 1 -- 172.21.15.63:0/2741474770 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c4005040 con 0x7f28d41037c0 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.760+0000 7f28d3e00700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4102e20 0x7f28d4115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.761+0000 7f28d3400700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d41037c0 0x7f28d41161f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f28c000eb40 tx=0x7f28c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.761+0000 7f28cbe00700 1 -- 172.21.15.63:0/2741474770 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28c0007cc0 con 0x7f28d41037c0 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.761+0000 7f28cbe00700 1 -- 172.21.15.63:0/2741474770 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28c000f040 con 0x7f28d41037c0 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.761+0000 7f28cbe00700 1 -- 172.21.15.63:0/2741474770 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28c000c8f0 con 0x7f28d41037c0 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.761+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d410fe90 con 0x7f28d41037c0 2024-03-23T11:55:30.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.761+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d4110450 con 0x7f28d41037c0 2024-03-23T11:55:30.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.763+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d40fe570 con 0x7f28d41037c0 2024-03-23T11:55:30.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.764+0000 7f28cbe00700 1 -- 172.21.15.63:0/2741474770 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f28c00090d0 con 0x7f28d41037c0 2024-03-23T11:55:30.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.765+0000 7f28cbe00700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28b806c7d0 0x7f28b806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:55:30.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.766+0000 7f28cbe00700 1 -- 172.21.15.63:0/2741474770 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f28c008b370 con 0x7f28d41037c0 2024-03-23T11:55:30.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.766+0000 7f28d3e00700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28b806c7d0 0x7f28b806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:55:30.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.766+0000 7f28d3e00700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28b806c7d0 0x7f28b806ec80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b0d0 tx=0x7f28c4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:55:30.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:30.770+0000 7f28cbe00700 1 -- 172.21.15.63:0/2741474770 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f28c00591d0 con 0x7f28d41037c0 2024-03-23T11:55:31.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:31.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: pgmap v108: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:31.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:55:31.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428] boot 2024-03-23T11:55:31.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: osdmap e63: 6 total, 6 up, 6 in 2024-03-23T11:55:31.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:55:31.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:31.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:31.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/825677244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:55:31.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 40/330 objects degraded (12.121%), 12 pgs degraded (PG_DEGRADED) 2024-03-23T11:55:31.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:30 smithi063 ceph-mon[141124]: osdmap e64: 6 total, 6 up, 6 in 2024-03-23T11:55:31.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.108+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f28d4109de0 con 0x7f28d41037c0 2024-03-23T11:55:31.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.108+0000 7f28cbe00700 1 -- 172.21.15.63:0/2741474770 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+852 (secure 0 0 0) 0x7f28c0058d60 con 0x7f28d41037c0 2024-03-23T11:55:31.091 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_WARN Degraded data redundancy: 40/330 objects degraded (12.121%), 12 pgs degraded 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 40/330 objects degraded (12.121%), 12 pgs degraded 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.1 is active+undersized+degraded, acting [1,0] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.3 is active+undersized+degraded, acting [5,1] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.7 is active+undersized+degraded, acting [3,4] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.e is active+undersized+degraded, acting [0,3] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.16 is active+undersized+degraded, acting [5,3] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.17 is active+undersized+degraded, acting [5,1] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.19 is active+undersized+degraded, acting [0,4] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.1c is active+undersized+degraded, acting [4,5] 2024-03-23T11:55:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-03-23T11:55:31.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.110+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28b806c7d0 msgr2=0x7f28b806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:31.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.110+0000 7f28db000700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28b806c7d0 0x7f28b806ec80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b0d0 tx=0x7f28c4009e30 comp rx=0 tx=0).stop 2024-03-23T11:55:31.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.110+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d41037c0 msgr2=0x7f28d41161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:55:31.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.110+0000 7f28db000700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d41037c0 0x7f28d41161f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f28c000eb40 tx=0x7f28c000b040 comp rx=0 tx=0).stop 2024-03-23T11:55:31.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.111+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 shutdown_connections 2024-03-23T11:55:31.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.111+0000 7f28db000700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28b806c7d0 0x7f28b806ec80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:31.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.111+0000 7f28db000700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4102e20 0x7f28d4115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:31.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.111+0000 7f28db000700 1 --2- 172.21.15.63:0/2741474770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d41037c0 0x7f28d41161f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:55:31.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.111+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 >> 172.21.15.63:0/2741474770 conn(0x7f28d40f8970 msgr2=0x7f28d40703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:55:31.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.111+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 shutdown_connections 2024-03-23T11:55:31.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:55:31.111+0000 7f28db000700 1 -- 172.21.15.63:0/2741474770 wait complete. 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: pgmap v108: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 40/330 objects degraded (12.121%) 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: osd.2 [v2:172.21.15.63:6818/3910577428,v1:172.21.15.63:6819/3910577428] boot 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: osdmap e63: 6 total, 6 up, 6 in 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/825677244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 40/330 objects degraded (12.121%), 12 pgs degraded (PG_DEGRADED) 2024-03-23T11:55:31.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:30 smithi111 ceph-mon[106899]: osdmap e64: 6 total, 6 up, 6 in 2024-03-23T11:55:32.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:31 smithi063 ceph-mon[141124]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:32.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:31 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2741474770' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:55:32.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:31 smithi111 ceph-mon[106899]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:55:32.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:31 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2741474770' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:55:33.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:32 smithi063 ceph-mon[141124]: pgmap v111: 65 pgs: 7 active+undersized+degraded, 10 active+undersized, 48 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 5 op/s; 26/330 objects degraded (7.879%) 2024-03-23T11:55:33.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:32 smithi111 ceph-mon[106899]: pgmap v111: 65 pgs: 7 active+undersized+degraded, 10 active+undersized, 48 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 5 op/s; 26/330 objects degraded (7.879%) 2024-03-23T11:55:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:34 smithi063 ceph-mon[141124]: pgmap v112: 65 pgs: 7 active+undersized+degraded, 10 active+undersized, 48 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 26/330 objects degraded (7.879%) 2024-03-23T11:55:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:34 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:35.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:34 smithi111 ceph-mon[106899]: pgmap v112: 65 pgs: 7 active+undersized+degraded, 10 active+undersized, 48 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 26/330 objects degraded (7.879%) 2024-03-23T11:55:35.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:34 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:36.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:36.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:55:36.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:55:36.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:36.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:55:36.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:36.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:36.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:36.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:36.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:35 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 26/330 objects degraded (7.879%), 7 pgs degraded (PG_DEGRADED) 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:36.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:55:36.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:36.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:35 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 26/330 objects degraded (7.879%), 7 pgs degraded (PG_DEGRADED) 2024-03-23T11:55:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:36 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:36 smithi063 ceph-mon[141124]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-23T11:55:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:36 smithi063 ceph-mon[141124]: pgmap v113: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-23T11:55:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:36 smithi063 ceph-mon[141124]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/330 objects degraded (7.879%), 7 pgs degraded) 2024-03-23T11:55:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:36 smithi063 ceph-mon[141124]: Cluster is now healthy 2024-03-23T11:55:37.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:36 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:37.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:36 smithi111 ceph-mon[106899]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-23T11:55:37.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:36 smithi111 ceph-mon[106899]: pgmap v113: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-23T11:55:37.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:36 smithi111 ceph-mon[106899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/330 objects degraded (7.879%), 7 pgs degraded) 2024-03-23T11:55:37.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:36 smithi111 ceph-mon[106899]: Cluster is now healthy 2024-03-23T11:55:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:38 smithi063 ceph-mon[141124]: pgmap v114: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:55:39.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:38 smithi111 ceph-mon[106899]: pgmap v114: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:55:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:40 smithi063 ceph-mon[141124]: pgmap v115: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:40 smithi111 ceph-mon[106899]: pgmap v115: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:41.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:43.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:42 smithi063 ceph-mon[141124]: pgmap v116: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:42 smithi111 ceph-mon[106899]: pgmap v116: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:45.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:44 smithi063 ceph-mon[141124]: pgmap v117: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:45.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:44 smithi111 ceph-mon[106899]: pgmap v117: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:46.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:46 smithi063 ceph-mon[141124]: pgmap v118: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:46.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:46 smithi111 ceph-mon[106899]: pgmap v118: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:49.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:48 smithi063 ceph-mon[141124]: pgmap v119: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:49.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:48 smithi111 ceph-mon[106899]: pgmap v119: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:50 smithi063 ceph-mon[141124]: pgmap v120: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:51.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:50 smithi111 ceph-mon[106899]: pgmap v120: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:51.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:51.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:51 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:51.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:51 smithi111 ceph-mon[106899]: Upgrade: osd.3 is safe to restart 2024-03-23T11:55:52.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:51 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-23T11:55:52.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:51 smithi063 ceph-mon[141124]: Upgrade: osd.3 is safe to restart 2024-03-23T11:55:53.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:52 smithi063 ceph-mon[141124]: pgmap v121: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:53.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:53.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T11:55:53.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:55:53.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:52 smithi111 ceph-mon[106899]: pgmap v121: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-23T11:55:53.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:55:53.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-23T11:55:53.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:55:54.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:53 smithi063 ceph-mon[141124]: Upgrade: Updating osd.3 2024-03-23T11:55:54.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:53 smithi063 ceph-mon[141124]: Deploying daemon osd.3 on smithi111 2024-03-23T11:55:54.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:53 smithi111 ceph-mon[106899]: Upgrade: Updating osd.3 2024-03-23T11:55:54.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:53 smithi111 ceph-mon[106899]: Deploying daemon osd.3 on smithi111 2024-03-23T11:55:54.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:54 smithi111 ceph-mon[106899]: pgmap v122: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:55.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:54 smithi063 ceph-mon[141124]: pgmap v122: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:55.902 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:55 smithi111 systemd[1]: Stopping Ceph osd.3 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:55:56.326 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:56 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[50390]: 2024-03-23T11:55:56.057+0000 7fd18ae00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:55:56.326 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:56 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[50390]: 2024-03-23T11:55:56.057+0000 7fd18ae00700 -1 osd.3 64 *** Got signal Terminated *** 2024-03-23T11:55:56.326 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:56 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[50390]: 2024-03-23T11:55:56.057+0000 7fd18ae00700 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T11:55:56.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:56 smithi063 ceph-mon[141124]: pgmap v123: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:56.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:56.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:56 smithi063 ceph-mon[141124]: osd.3 marked itself down and dead 2024-03-23T11:55:56.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:56 smithi111 ceph-mon[106899]: pgmap v123: 65 pgs: 65 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:55:56.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:55:56.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:56 smithi111 ceph-mon[106899]: osd.3 marked itself down and dead 2024-03-23T11:55:57.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:57 smithi111 ceph-mon[106899]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:55:57.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:57 smithi111 ceph-mon[106899]: osdmap e65: 6 total, 5 up, 6 in 2024-03-23T11:55:57.402 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:57 smithi111 bash[119260]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3 2024-03-23T11:55:57.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:57 smithi063 ceph-mon[141124]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:55:57.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:57 smithi063 ceph-mon[141124]: osdmap e65: 6 total, 5 up, 6 in 2024-03-23T11:55:58.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:58 smithi111 ceph-mon[106899]: osdmap e66: 6 total, 5 up, 6 in 2024-03-23T11:55:58.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:58 smithi111 ceph-mon[106899]: pgmap v126: 65 pgs: 18 peering, 6 stale+active+clean, 41 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:55:58.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:58 smithi063 ceph-mon[141124]: osdmap e66: 6 total, 5 up, 6 in 2024-03-23T11:55:58.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:58 smithi063 ceph-mon[141124]: pgmap v126: 65 pgs: 18 peering, 6 stale+active+clean, 41 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:55:59.585 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:55:59 smithi111 ceph-mon[106899]: Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY) 2024-03-23T11:55:59.586 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:59 smithi111 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.3.service: Succeeded. 2024-03-23T11:55:59.586 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:59 smithi111 systemd[1]: Stopped Ceph osd.3 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:55:59.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:55:59 smithi063 ceph-mon[141124]: Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY) 2024-03-23T11:55:59.902 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:55:59 smithi111 systemd[1]: Starting Ceph osd.3 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:56:00.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:00 smithi063 ceph-mon[141124]: pgmap v127: 65 pgs: 10 active+undersized+degraded, 18 peering, 10 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 29/330 objects degraded (8.788%) 2024-03-23T11:56:00.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:00 smithi111 ceph-mon[106899]: pgmap v127: 65 pgs: 10 active+undersized+degraded, 18 peering, 10 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 29/330 objects degraded (8.788%) 2024-03-23T11:56:01.152 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:00 smithi111 bash[119536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-23T11:56:01.152 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:00 smithi111 bash[119536]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-23T11:56:01.152 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:00 smithi111 bash[119536]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-23T11:56:01.152 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:00 smithi111 bash[119536]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-23T11:56:01.152 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:00 smithi111 bash[119536]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-23T11:56:01.152 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:00 smithi111 bash[119536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-23T11:56:01.152 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:00 smithi111 bash[119536]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.267+0000 7fb2a1200700 1 -- 172.21.15.63:0/3762941892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c1037a0 msgr2=0x7fb29c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.267+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3762941892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c1037a0 0x7fb29c0fce10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb288004940 tx=0x7fb288009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.268+0000 7fb2a1200700 1 -- 172.21.15.63:0/3762941892 shutdown_connections 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.268+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3762941892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c1037a0 0x7fb29c0fce10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.268+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3762941892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29c102e00 0x7fb29c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.268+0000 7fb2a1200700 1 -- 172.21.15.63:0/3762941892 >> 172.21.15.63:0/3762941892 conn(0x7fb29c0f89b0 msgr2=0x7fb29c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.268+0000 7fb2a1200700 1 -- 172.21.15.63:0/3762941892 shutdown_connections 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.268+0000 7fb2a1200700 1 -- 172.21.15.63:0/3762941892 wait complete. 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb2a1200700 1 Processor -- start 2024-03-23T11:56:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb2a1200700 1 -- start start 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb2a1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c102e00 0x7fb29c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb2a1200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29c1037a0 0x7fb29c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb2a1200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c114600 con 0x7fb29c102e00 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb2a1200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c10d9f0 con 0x7fb29c1037a0 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb299e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c102e00 0x7fb29c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb299e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c102e00 0x7fb29c113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37514/0 (socket says 172.21.15.63:37514) 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.269+0000 7fb299e00700 1 -- 172.21.15.63:0/3090297600 learned_addr learned my addr 172.21.15.63:0/3090297600 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb299e00700 1 -- 172.21.15.63:0/3090297600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29c1037a0 msgr2=0x7fb29c113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb299400700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29c1037a0 0x7fb29c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb299e00700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29c1037a0 0x7fb29c113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb299e00700 1 -- 172.21.15.63:0/3090297600 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288005040 con 0x7fb29c102e00 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb299400700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29c1037a0 0x7fb29c113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:56:01.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb299e00700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c102e00 0x7fb29c113a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00eb40 tx=0x7fb28c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:01.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb292000700 1 -- 172.21.15.63:0/3090297600 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb28c007cc0 con 0x7fb29c102e00 2024-03-23T11:56:01.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb292000700 1 -- 172.21.15.63:0/3090297600 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb28c00f040 con 0x7fb29c102e00 2024-03-23T11:56:01.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c10dc70 con 0x7fb29c102e00 2024-03-23T11:56:01.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.270+0000 7fb292000700 1 -- 172.21.15.63:0/3090297600 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb28c009570 con 0x7fb29c102e00 2024-03-23T11:56:01.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.271+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c10e230 con 0x7fb29c102e00 2024-03-23T11:56:01.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.272+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c0fe550 con 0x7fb29c102e00 2024-03-23T11:56:01.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.273+0000 7fb292000700 1 -- 172.21.15.63:0/3090297600 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb28c0096d0 con 0x7fb29c102e00 2024-03-23T11:56:01.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.274+0000 7fb292000700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb28006c7d0 0x7fb28006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:01.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.274+0000 7fb292000700 1 -- 172.21.15.63:0/3090297600 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb28c08b670 con 0x7fb29c102e00 2024-03-23T11:56:01.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.274+0000 7fb299400700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb28006c7d0 0x7fb28006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:01.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.275+0000 7fb299400700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb28006c7d0 0x7fb28006ec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb288009fd0 tx=0x7fb288009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:01.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.279+0000 7fb292000700 1 -- 172.21.15.63:0/3090297600 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb28c059430 con 0x7fb29c102e00 2024-03-23T11:56:01.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.512+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb29c109e10 con 0x7fb28006c7d0 2024-03-23T11:56:01.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.513+0000 7fb292000700 1 -- 172.21.15.63:0/3090297600 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb29c109e10 con 0x7fb28006c7d0 2024-03-23T11:56:01.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb28006c7d0 msgr2=0x7fb28006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb28006c7d0 0x7fb28006ec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb288009fd0 tx=0x7fb288009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c102e00 msgr2=0x7fb29c113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c102e00 0x7fb29c113a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00eb40 tx=0x7fb28c00b040 comp rx=0 tx=0).stop 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 shutdown_connections 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb28006c7d0 0x7fb28006ec80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c102e00 0x7fb29c113a70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 --2- 172.21.15.63:0/3090297600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29c1037a0 0x7fb29c113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 >> 172.21.15.63:0/3090297600 conn(0x7fb29c0f89b0 msgr2=0x7fb29c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 shutdown_connections 2024-03-23T11:56:01.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.516+0000 7fb2a1200700 1 -- 172.21.15.63:0/3090297600 wait complete. 2024-03-23T11:56:01.515 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:56:01.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:01 smithi063 ceph-mon[141124]: Health check failed: Degraded data redundancy: 29/330 objects degraded (8.788%), 10 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.655+0000 7f595ac00700 1 -- 172.21.15.63:0/3870202729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 msgr2=0x7f59541031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.655+0000 7f595ac00700 1 --2- 172.21.15.63:0/3870202729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f59541031d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5944004940 tx=0x7f5944009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.655+0000 7f595ac00700 1 -- 172.21.15.63:0/3870202729 shutdown_connections 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.655+0000 7f595ac00700 1 --2- 172.21.15.63:0/3870202729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59541037a0 0x7f59540fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.655+0000 7f595ac00700 1 --2- 172.21.15.63:0/3870202729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f59541031d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.655+0000 7f595ac00700 1 -- 172.21.15.63:0/3870202729 >> 172.21.15.63:0/3870202729 conn(0x7f59540f89b0 msgr2=0x7f59540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.656+0000 7f595ac00700 1 -- 172.21.15.63:0/3870202729 shutdown_connections 2024-03-23T11:56:01.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.656+0000 7f595ac00700 1 -- 172.21.15.63:0/3870202729 wait complete. 2024-03-23T11:56:01.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.656+0000 7f595ac00700 1 Processor -- start 2024-03-23T11:56:01.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.656+0000 7f595ac00700 1 -- start start 2024-03-23T11:56:01.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f595ac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f5954115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:01.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f595ac00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59541037a0 0x7f5954116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:01.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f595ac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59541167d0 con 0x7f5954102e00 2024-03-23T11:56:01.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f595ac00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595410fba0 con 0x7f59541037a0 2024-03-23T11:56:01.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f5953e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f5954115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:01.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f5953e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f5954115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59426/0 (socket says 172.21.15.63:59426) 2024-03-23T11:56:01.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f5953e00700 1 -- 172.21.15.63:0/3687024590 learned_addr learned my addr 172.21.15.63:0/3687024590 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:01.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f5953400700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59541037a0 0x7f5954116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f5953e00700 1 -- 172.21.15.63:0/3687024590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59541037a0 msgr2=0x7f5954116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f5953e00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59541037a0 0x7f5954116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.657+0000 7f5953e00700 1 -- 172.21.15.63:0/3687024590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5944005040 con 0x7f5954102e00 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.658+0000 7f5953e00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f5954115c40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5944009390 tx=0x7f5944009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.658+0000 7f594be00700 1 -- 172.21.15.63:0/3687024590 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f594400e040 con 0x7f5954102e00 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.658+0000 7f594be00700 1 -- 172.21.15.63:0/3687024590 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5944013420 con 0x7f5954102e00 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.658+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595410fe20 con 0x7f5954102e00 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.658+0000 7f594be00700 1 -- 172.21.15.63:0/3687024590 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f594401c630 con 0x7f5954102e00 2024-03-23T11:56:01.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.658+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59541103e0 con 0x7f5954102e00 2024-03-23T11:56:01.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.659+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59540fe550 con 0x7f5954102e00 2024-03-23T11:56:01.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.660+0000 7f594be00700 1 -- 172.21.15.63:0/3687024590 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f59440139e0 con 0x7f5954102e00 2024-03-23T11:56:01.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.661+0000 7f594be00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f593806c7d0 0x7f593806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:01.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.661+0000 7f594be00700 1 -- 172.21.15.63:0/3687024590 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f594408cd00 con 0x7f5954102e00 2024-03-23T11:56:01.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.661+0000 7f5953400700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f593806c7d0 0x7f593806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:01.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.662+0000 7f5953400700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f593806c7d0 0x7f593806ec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59541114e0 tx=0x7f594000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:01.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.666+0000 7f594be00700 1 -- 172.21.15.63:0/3687024590 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f594405aa90 con 0x7f5954102e00 2024-03-23T11:56:01.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:01 smithi111 ceph-mon[106899]: Health check failed: Degraded data redundancy: 29/330 objects degraded (8.788%), 10 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:01.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.899+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f595404bf40 con 0x7f593806c7d0 2024-03-23T11:56:01.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.901+0000 7f594be00700 1 -- 172.21.15.63:0/3687024590 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f595404bf40 con 0x7f593806c7d0 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f593806c7d0 msgr2=0x7f593806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f593806c7d0 0x7f593806ec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59541114e0 tx=0x7f594000b040 comp rx=0 tx=0).stop 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 msgr2=0x7f5954115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f5954115c40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5944009390 tx=0x7f5944009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 shutdown_connections 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f593806c7d0 0x7f593806ec80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5954102e00 0x7f5954115c40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 --2- 172.21.15.63:0/3687024590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59541037a0 0x7f5954116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 >> 172.21.15.63:0/3687024590 conn(0x7f59540f89b0 msgr2=0x7f59540f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 shutdown_connections 2024-03-23T11:56:01.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:01.903+0000 7f595ac00700 1 -- 172.21.15.63:0/3687024590 wait complete. 2024-03-23T11:56:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 -- 172.21.15.63:0/941891960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d01037a0 msgr2=0x7fc1d00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/941891960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d01037a0 0x7fc1d00fce10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8004940 tx=0x7fc1b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 -- 172.21.15.63:0/941891960 shutdown_connections 2024-03-23T11:56:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/941891960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d01037a0 0x7fc1d00fce10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/941891960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1d0102e00 0x7fc1d01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 -- 172.21.15.63:0/941891960 >> 172.21.15.63:0/941891960 conn(0x7fc1d00f89b0 msgr2=0x7fc1d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 -- 172.21.15.63:0/941891960 shutdown_connections 2024-03-23T11:56:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.040+0000 7fc1d6a00700 1 -- 172.21.15.63:0/941891960 wait complete. 2024-03-23T11:56:02.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1d6a00700 1 Processor -- start 2024-03-23T11:56:02.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1d6a00700 1 -- start start 2024-03-23T11:56:02.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1d6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d0102e00 0x7fc1d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1d6a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1d01037a0 0x7fc1d0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1d6a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0070d90 con 0x7fc1d0102e00 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1d6a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0070ed0 con 0x7fc1d01037a0 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1cfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d0102e00 0x7fc1d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.041+0000 7fc1cfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d0102e00 0x7fc1d00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59432/0 (socket says 172.21.15.63:59432) 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1cfe00700 1 -- 172.21.15.63:0/757327566 learned_addr learned my addr 172.21.15.63:0/757327566 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1c7e00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1d01037a0 0x7fc1d0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1cfe00700 1 -- 172.21.15.63:0/757327566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1d01037a0 msgr2=0x7fc1d0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1cfe00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1d01037a0 0x7fc1d0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1cfe00700 1 -- 172.21.15.63:0/757327566 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8005040 con 0x7fc1d0102e00 2024-03-23T11:56:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1cfe00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d0102e00 0x7fc1d00702c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000eb70 tx=0x7fc1c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1ccc00700 1 -- 172.21.15.63:0/757327566 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1c0007cc0 con 0x7fc1d0102e00 2024-03-23T11:56:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1ccc00700 1 -- 172.21.15.63:0/757327566 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1c000f040 con 0x7fc1d0102e00 2024-03-23T11:56:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1ccc00700 1 -- 172.21.15.63:0/757327566 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1c00095f0 con 0x7fc1d0102e00 2024-03-23T11:56:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.042+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d0071120 con 0x7fc1d0102e00 2024-03-23T11:56:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.043+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d011a150 con 0x7fc1d0102e00 2024-03-23T11:56:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.044+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d00fe550 con 0x7fc1d0102e00 2024-03-23T11:56:02.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.045+0000 7fc1ccc00700 1 -- 172.21.15.63:0/757327566 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc1c0009750 con 0x7fc1d0102e00 2024-03-23T11:56:02.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.045+0000 7fc1ccc00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1b406c7d0 0x7fc1b406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.046+0000 7fc1ccc00700 1 -- 172.21.15.63:0/757327566 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc1c008b620 con 0x7fc1d0102e00 2024-03-23T11:56:02.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.046+0000 7fc1c7e00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1b406c7d0 0x7fc1b406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.046+0000 7fc1c7e00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1b406c7d0 0x7fc1b406ec80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009fd0 tx=0x7fc1b8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:02.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.050+0000 7fc1ccc00700 1 -- 172.21.15.63:0/757327566 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc1c00558c0 con 0x7fc1d0102e00 2024-03-23T11:56:02.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.271+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1d004bf40 con 0x7fc1b406c7d0 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.281+0000 7fc1ccc00700 1 -- 172.21.15.63:0/757327566 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fc1d004bf40 con 0x7fc1b406c7d0 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (5m) 33s ago 18m 14.2M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (2m) 33s ago 18m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (2m) 2m ago 17m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (5m) 33s ago 18m 73.4M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (12m) 33s ago 12m 118M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (12m) 33s ago 12m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:56:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (11m) 2m ago 11m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (12m) 2m ago 12m 60.2M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (7m) 33s ago 19m 502M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (6m) 2m ago 17m 450M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (3m) 33s ago 20m 45.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (3m) 2m ago 17m 36.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (6m) 33s ago 18m 7532k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (6m) 2m ago 17m 7860k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (2m) 33s ago 16m 87.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (80s) 33s ago 15m 91.1M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (40s) 33s ago 15m 11.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (14m) 2m ago 14m 259M 2642M 17.2.7-726-g9c4e0c34 837124da804f b44b6673c218 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (14m) 2m ago 14m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (13m) 2m ago 13m 228M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:56:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (5m) 33s ago 17m 53.4M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1b406c7d0 msgr2=0x7fc1b406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1b406c7d0 0x7fc1b406ec80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009fd0 tx=0x7fc1b8009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d0102e00 msgr2=0x7fc1d00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d0102e00 0x7fc1d00702c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000eb70 tx=0x7fc1c000b040 comp rx=0 tx=0).stop 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 shutdown_connections 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1b406c7d0 0x7fc1b406ec80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc1d0102e00 0x7fc1d00702c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 --2- 172.21.15.63:0/757327566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1d01037a0 0x7fc1d0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 >> 172.21.15.63:0/757327566 conn(0x7fc1d00f89b0 msgr2=0x7fc1d00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 shutdown_connections 2024-03-23T11:56:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.284+0000 7fc1d6a00700 1 -- 172.21.15.63:0/757327566 wait complete. 2024-03-23T11:56:02.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 -- 172.21.15.63:0/3329009373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 msgr2=0x7ff2b81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 --2- 172.21.15.63:0/3329009373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 0x7ff2b81031f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac004940 tx=0x7ff2ac009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:02.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 -- 172.21.15.63:0/3329009373 shutdown_connections 2024-03-23T11:56:02.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 --2- 172.21.15.63:0/3329009373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 0x7ff2b80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 --2- 172.21.15.63:0/3329009373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 0x7ff2b81031f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 -- 172.21.15.63:0/3329009373 >> 172.21.15.63:0/3329009373 conn(0x7ff2b80f89d0 msgr2=0x7ff2b80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:02.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 -- 172.21.15.63:0/3329009373 shutdown_connections 2024-03-23T11:56:02.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.424+0000 7ff2bf800700 1 -- 172.21.15.63:0/3329009373 wait complete. 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.425+0000 7ff2bf800700 1 Processor -- start 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.425+0000 7ff2bf800700 1 -- start start 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.425+0000 7ff2bf800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 0x7ff2b80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.425+0000 7ff2bf800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 0x7ff2b8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.425+0000 7ff2bf800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b8070d90 con 0x7ff2b81037c0 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.425+0000 7ff2bf800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b8070ed0 con 0x7ff2b8102e20 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.425+0000 7ff2b7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 0x7ff2b8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 0x7ff2b8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59438/0 (socket says 172.21.15.63:59438) 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b7e00700 1 -- 172.21.15.63:0/638824935 learned_addr learned my addr 172.21.15.63:0/638824935 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:02.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2bce00700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 0x7ff2b80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b7e00700 1 -- 172.21.15.63:0/638824935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 msgr2=0x7ff2b80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b7e00700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 0x7ff2b80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b7e00700 1 -- 172.21.15.63:0/638824935 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a4005040 con 0x7ff2b81037c0 2024-03-23T11:56:02.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b7e00700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 0x7ff2b8070800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009700 tx=0x7ff2a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:02.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b4c00700 1 -- 172.21.15.63:0/638824935 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2a400e7e0 con 0x7ff2b81037c0 2024-03-23T11:56:02.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac005040 con 0x7ff2b81037c0 2024-03-23T11:56:02.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.426+0000 7ff2b4c00700 1 -- 172.21.15.63:0/638824935 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2a400f040 con 0x7ff2b81037c0 2024-03-23T11:56:02.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.427+0000 7ff2b4c00700 1 -- 172.21.15.63:0/638824935 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2a4013c30 con 0x7ff2b81037c0 2024-03-23T11:56:02.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.427+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b8119ff0 con 0x7ff2b81037c0 2024-03-23T11:56:02.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.428+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b80fe570 con 0x7ff2b81037c0 2024-03-23T11:56:02.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.429+0000 7ff2b4c00700 1 -- 172.21.15.63:0/638824935 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff2a400e940 con 0x7ff2b81037c0 2024-03-23T11:56:02.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.429+0000 7ff2b4c00700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff2a006c7d0 0x7ff2a006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.430+0000 7ff2b4c00700 1 -- 172.21.15.63:0/638824935 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff2a408b870 con 0x7ff2b81037c0 2024-03-23T11:56:02.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.430+0000 7ff2bce00700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff2a006c7d0 0x7ff2a006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.430+0000 7ff2bce00700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff2a006c7d0 0x7ff2a006ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0045a0 tx=0x7ff2ac009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:02.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.434+0000 7ff2b4c00700 1 -- 172.21.15.63:0/638824935 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff2a4055a70 con 0x7ff2b81037c0 2024-03-23T11:56:02.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:02 smithi063 ceph-mon[141124]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:02.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:02 smithi063 ceph-mon[141124]: pgmap v128: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:02.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:02.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:02.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:02 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:02.651 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:02 smithi111 ceph-osd[119827]: -- 172.21.15.111:0/2807531518 <== mon.1 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556b4b56d1e0 con 0x556b4b53d000 2024-03-23T11:56:02.651 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:02 smithi111 bash[119790]: 060d49a70bd25d603a7537032a8dfd0bf7be80ffb7a461709c200302cc153ba9 2024-03-23T11:56:02.652 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:02 smithi111 systemd[1]: Started Ceph osd.3 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:56:02.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:02 smithi111 ceph-mon[106899]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:02.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:02 smithi111 ceph-mon[106899]: pgmap v128: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:02.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:02.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:02.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:02 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:02.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.732+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff2b804bed0 con 0x7ff2b81037c0 2024-03-23T11:56:02.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.733+0000 7ff2b4c00700 1 -- 172.21.15.63:0/638824935 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7ff2a4059090 con 0x7ff2b81037c0 2024-03-23T11:56:02.716 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:56:02.716 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:56:02.716 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:56:02.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2, 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6, 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 7 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:56:02.717 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.735+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff2a006c7d0 msgr2=0x7ff2a006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.735+0000 7ff2bf800700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff2a006c7d0 0x7ff2a006ec80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0045a0 tx=0x7ff2ac009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.735+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 msgr2=0x7ff2b8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.735+0000 7ff2bf800700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 0x7ff2b8070800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009700 tx=0x7ff2a4009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.736+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 shutdown_connections 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.736+0000 7ff2bf800700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff2a006c7d0 0x7ff2a006ec80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.736+0000 7ff2bf800700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2b8102e20 0x7ff2b80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.736+0000 7ff2bf800700 1 --2- 172.21.15.63:0/638824935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2b81037c0 0x7ff2b8070800 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.736+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 >> 172.21.15.63:0/638824935 conn(0x7ff2b80f89d0 msgr2=0x7ff2b80f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:02.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.736+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 shutdown_connections 2024-03-23T11:56:02.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.736+0000 7ff2bf800700 1 -- 172.21.15.63:0/638824935 wait complete. 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 -- 172.21.15.63:0/3302606166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b38102e00 msgr2=0x7f7b381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 --2- 172.21.15.63:0/3302606166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b38102e00 0x7f7b381031d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c004940 tx=0x7f7b2c009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 -- 172.21.15.63:0/3302606166 shutdown_connections 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 --2- 172.21.15.63:0/3302606166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b381037a0 0x7f7b380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 --2- 172.21.15.63:0/3302606166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b38102e00 0x7f7b381031d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 -- 172.21.15.63:0/3302606166 >> 172.21.15.63:0/3302606166 conn(0x7f7b380f89b0 msgr2=0x7f7b380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 -- 172.21.15.63:0/3302606166 shutdown_connections 2024-03-23T11:56:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.876+0000 7f7b40400700 1 -- 172.21.15.63:0/3302606166 wait complete. 2024-03-23T11:56:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.877+0000 7f7b40400700 1 Processor -- start 2024-03-23T11:56:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.877+0000 7f7b40400700 1 -- start start 2024-03-23T11:56:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b40400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b38102e00 0x7f7b380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b40400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b381037a0 0x7f7b38070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b40400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b38070d90 con 0x7f7b381037a0 2024-03-23T11:56:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b40400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b38070ed0 con 0x7f7b38102e00 2024-03-23T11:56:02.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b3d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b381037a0 0x7f7b38070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b3d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b381037a0 0x7f7b38070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59460/0 (socket says 172.21.15.63:59460) 2024-03-23T11:56:02.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b3d000700 1 -- 172.21.15.63:0/3982500132 learned_addr learned my addr 172.21.15.63:0/3982500132 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:02.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b3da00700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b38102e00 0x7f7b380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b3d000700 1 -- 172.21.15.63:0/3982500132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b38102e00 msgr2=0x7f7b380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b3d000700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b38102e00 0x7f7b380702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.878+0000 7f7b3d000700 1 -- 172.21.15.63:0/3982500132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c005040 con 0x7f7b381037a0 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.879+0000 7f7b3d000700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b381037a0 0x7f7b38070800 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b24002a50 tx=0x7f7b240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.879+0000 7f7b35600700 1 -- 172.21.15.63:0/3982500132 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b24007cc0 con 0x7f7b381037a0 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.879+0000 7f7b35600700 1 -- 172.21.15.63:0/3982500132 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b2400f040 con 0x7f7b381037a0 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.879+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b38119d40 con 0x7f7b381037a0 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.879+0000 7f7b35600700 1 -- 172.21.15.63:0/3982500132 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b2400da30 con 0x7f7b381037a0 2024-03-23T11:56:02.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.879+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3811a1c0 con 0x7f7b381037a0 2024-03-23T11:56:02.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.880+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b380fe550 con 0x7f7b381037a0 2024-03-23T11:56:02.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.881+0000 7f7b35600700 1 -- 172.21.15.63:0/3982500132 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7b2400db90 con 0x7f7b381037a0 2024-03-23T11:56:02.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.882+0000 7f7b35600700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2006c7d0 0x7f7b2006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:02.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.882+0000 7f7b35600700 1 -- 172.21.15.63:0/3982500132 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7b2408b640 con 0x7f7b381037a0 2024-03-23T11:56:02.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.882+0000 7f7b3da00700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2006c7d0 0x7f7b2006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:02.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.883+0000 7f7b3da00700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2006c7d0 0x7f7b2006ec80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009fd0 tx=0x7f7b2c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:02.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:02.886+0000 7f7b35600700 1 -- 172.21.15.63:0/3982500132 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b240558c0 con 0x7f7b381037a0 2024-03-23T11:56:03.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.145+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7b3804bed0 con 0x7f7b381037a0 2024-03-23T11:56:03.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.145+0000 7f7b35600700 1 -- 172.21.15.63:0/3982500132 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f7b24058ee0 con 0x7f7b381037a0 2024-03-23T11:56:03.128 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:56:03.129 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:56:03.130 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:56:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:56:03.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.147+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2006c7d0 msgr2=0x7f7b2006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.147+0000 7f7b40400700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2006c7d0 0x7f7b2006ec80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009fd0 tx=0x7f7b2c009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:03.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.147+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b381037a0 msgr2=0x7f7b38070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.147+0000 7f7b40400700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b381037a0 0x7f7b38070800 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b24002a50 tx=0x7f7b240046b0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.148+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 shutdown_connections 2024-03-23T11:56:03.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.148+0000 7f7b40400700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2006c7d0 0x7f7b2006ec80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.148+0000 7f7b40400700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b38102e00 0x7f7b380702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.148+0000 7f7b40400700 1 --2- 172.21.15.63:0/3982500132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b381037a0 0x7f7b38070800 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.148+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 >> 172.21.15.63:0/3982500132 conn(0x7f7b380f89b0 msgr2=0x7f7b380f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:03.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.148+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 shutdown_connections 2024-03-23T11:56:03.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.148+0000 7f7b40400700 1 -- 172.21.15.63:0/3982500132 wait complete. 2024-03-23T11:56:03.133 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:56:03.151 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:02 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T11:56:02.717+0000 7f4b7ee9f640 -1 Falling back to public interface 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 -- 172.21.15.63:0/4253221024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070360 msgr2=0x7fb820070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 --2- 172.21.15.63:0/4253221024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070360 0x7fb820070730 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb810004970 tx=0x7fb810009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 -- 172.21.15.63:0/4253221024 shutdown_connections 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 --2- 172.21.15.63:0/4253221024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb820070c70 0x7fb820108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 --2- 172.21.15.63:0/4253221024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070360 0x7fb820070730 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 -- 172.21.15.63:0/4253221024 >> 172.21.15.63:0/4253221024 conn(0x7fb8200f8830 msgr2=0x7fb8200fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 -- 172.21.15.63:0/4253221024 shutdown_connections 2024-03-23T11:56:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.285+0000 7fb825400700 1 -- 172.21.15.63:0/4253221024 wait complete. 2024-03-23T11:56:03.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.286+0000 7fb825400700 1 Processor -- start 2024-03-23T11:56:03.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.286+0000 7fb825400700 1 -- start start 2024-03-23T11:56:03.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb825400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb820070360 0x7fb820115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:03.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb825400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070c70 0x7fb820116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb825400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb820116680 con 0x7fb820070c70 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb825400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82010fa70 con 0x7fb820070360 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb81f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070c70 0x7fb820116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb81f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070c70 0x7fb820116030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59478/0 (socket says 172.21.15.63:59478) 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb81f400700 1 -- 172.21.15.63:0/346532077 learned_addr learned my addr 172.21.15.63:0/346532077 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb81fe00700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb820070360 0x7fb820115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb81f400700 1 -- 172.21.15.63:0/346532077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb820070360 msgr2=0x7fb820115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb81f400700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb820070360 0x7fb820115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.287+0000 7fb81f400700 1 -- 172.21.15.63:0/346532077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb810005040 con 0x7fb820070c70 2024-03-23T11:56:03.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.288+0000 7fb81f400700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070c70 0x7fb820116030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb80c002a80 tx=0x7fb80c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:03.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.288+0000 7fb817e00700 1 -- 172.21.15.63:0/346532077 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80c007cc0 con 0x7fb820070c70 2024-03-23T11:56:03.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.288+0000 7fb817e00700 1 -- 172.21.15.63:0/346532077 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb80c00f040 con 0x7fb820070c70 2024-03-23T11:56:03.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.288+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82010fd50 con 0x7fb820070c70 2024-03-23T11:56:03.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.288+0000 7fb817e00700 1 -- 172.21.15.63:0/346532077 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80c00da60 con 0x7fb820070c70 2024-03-23T11:56:03.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.288+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb820110290 con 0x7fb820070c70 2024-03-23T11:56:03.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.289+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb820071aa0 con 0x7fb820070c70 2024-03-23T11:56:03.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.290+0000 7fb817e00700 1 -- 172.21.15.63:0/346532077 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb80c00dbc0 con 0x7fb820070c70 2024-03-23T11:56:03.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.291+0000 7fb817e00700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb80406c7d0 0x7fb80406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:03.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.291+0000 7fb817e00700 1 -- 172.21.15.63:0/346532077 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb80c08b630 con 0x7fb820070c70 2024-03-23T11:56:03.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.291+0000 7fb81fe00700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb80406c7d0 0x7fb80406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:03.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.292+0000 7fb81fe00700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb80406c7d0 0x7fb80406ec80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb81000b370 tx=0x7fb810009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:03.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.295+0000 7fb817e00700 1 -- 172.21.15.63:0/346532077 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb80c059340 con 0x7fb820070c70 2024-03-23T11:56:03.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.530+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb82010bdf0 con 0x7fb80406c7d0 2024-03-23T11:56:03.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.532+0000 7fb817e00700 1 -- 172.21.15.63:0/346532077 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb82010bdf0 con 0x7fb80406c7d0 2024-03-23T11:56:03.515 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:56:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:56:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:56:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "9/21 daemons upgraded", 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading osd daemons", 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:56:03.516 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb80406c7d0 msgr2=0x7fb80406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb80406c7d0 0x7fb80406ec80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb81000b370 tx=0x7fb810009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070c70 msgr2=0x7fb820116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070c70 0x7fb820116030 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb80c002a80 tx=0x7fb80c0046b0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 shutdown_connections 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb80406c7d0 0x7fb80406ec80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb820070360 0x7fb820115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 --2- 172.21.15.63:0/346532077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb820070c70 0x7fb820116030 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 >> 172.21.15.63:0/346532077 conn(0x7fb8200f8830 msgr2=0x7fb8200f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 shutdown_connections 2024-03-23T11:56:03.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.534+0000 7fb825400700 1 -- 172.21.15.63:0/346532077 wait complete. 2024-03-23T11:56:03.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:03 smithi063 ceph-mon[141124]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:03.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:03 smithi063 ceph-mon[141124]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:03.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:03 smithi063 ceph-mon[141124]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs peering) 2024-03-23T11:56:03.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:03 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/638824935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:03.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:03 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3982500132' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:56:03.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:03 smithi111 ceph-mon[106899]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:03.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:03 smithi111 ceph-mon[106899]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:03.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:03 smithi111 ceph-mon[106899]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs peering) 2024-03-23T11:56:03.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:03 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/638824935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:03.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:03 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3982500132' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 -- 172.21.15.63:0/2352106960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff980102c40 msgr2=0x7ff980103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 --2- 172.21.15.63:0/2352106960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff980102c40 0x7ff980103010 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff970004940 tx=0x7ff970009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 -- 172.21.15.63:0/2352106960 shutdown_connections 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 --2- 172.21.15.63:0/2352106960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9801035e0 0x7ff9800fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 --2- 172.21.15.63:0/2352106960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff980102c40 0x7ff980103010 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 -- 172.21.15.63:0/2352106960 >> 172.21.15.63:0/2352106960 conn(0x7ff9800f8830 msgr2=0x7ff9800fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 -- 172.21.15.63:0/2352106960 shutdown_connections 2024-03-23T11:56:03.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.671+0000 7ff97f200700 1 -- 172.21.15.63:0/2352106960 wait complete. 2024-03-23T11:56:03.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.672+0000 7ff97f200700 1 Processor -- start 2024-03-23T11:56:03.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.672+0000 7ff97f200700 1 -- start start 2024-03-23T11:56:03.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff980102c40 0x7ff980113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:03.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9801035e0 0x7ff980113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:03.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980114400 con 0x7ff9801035e0 2024-03-23T11:56:03.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98010d7f0 con 0x7ff980102c40 2024-03-23T11:56:03.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9801035e0 0x7ff980113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:03.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9801035e0 0x7ff980113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59498/0 (socket says 172.21.15.63:59498) 2024-03-23T11:56:03.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97d400700 1 -- 172.21.15.63:0/3971692070 learned_addr learned my addr 172.21.15.63:0/3971692070 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:03.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97de00700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff980102c40 0x7ff980113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97d400700 1 -- 172.21.15.63:0/3971692070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff980102c40 msgr2=0x7ff980113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97d400700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff980102c40 0x7ff980113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.673+0000 7ff97d400700 1 -- 172.21.15.63:0/3971692070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff970005040 con 0x7ff9801035e0 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.674+0000 7ff97d400700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9801035e0 0x7ff980113db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00eb40 tx=0x7ff96c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.674+0000 7ff976000700 1 -- 172.21.15.63:0/3971692070 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff96c007cc0 con 0x7ff9801035e0 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.674+0000 7ff976000700 1 -- 172.21.15.63:0/3971692070 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff96c00f040 con 0x7ff9801035e0 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.674+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98010da70 con 0x7ff9801035e0 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.674+0000 7ff976000700 1 -- 172.21.15.63:0/3971692070 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff96c00c8f0 con 0x7ff9801035e0 2024-03-23T11:56:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.674+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98010e030 con 0x7ff9801035e0 2024-03-23T11:56:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.675+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9800fe390 con 0x7ff9801035e0 2024-03-23T11:56:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.676+0000 7ff976000700 1 -- 172.21.15.63:0/3971692070 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff96c0090d0 con 0x7ff9801035e0 2024-03-23T11:56:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.677+0000 7ff976000700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff96406c7d0 0x7ff96406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.677+0000 7ff976000700 1 -- 172.21.15.63:0/3971692070 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff96c08b410 con 0x7ff9801035e0 2024-03-23T11:56:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.677+0000 7ff97de00700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff96406c7d0 0x7ff96406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.678+0000 7ff97de00700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff96406c7d0 0x7ff96406ec80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff97000b0d0 tx=0x7ff970009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:03.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.681+0000 7ff976000700 1 -- 172.21.15.63:0/3971692070 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff96c0591d0 con 0x7ff9801035e0 2024-03-23T11:56:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.982+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff980109bf0 con 0x7ff9801035e0 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.982+0000 7ff976000700 1 -- 172.21.15.63:0/3971692070 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1328 (secure 0 0 0) 0x7ff96c058d60 con 0x7ff9801035e0 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 68/330 objects degraded (20.606%), 19 pgs degraded 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout: osd.3 (root=default,host=smithi111) is down 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 68/330 objects degraded (20.606%), 19 pgs degraded 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout: pg 1.0 is active+undersized+degraded, acting [0,1] 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-23T11:56:03.966 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.7 is active+undersized+degraded, acting [4,2] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.8 is active+undersized+degraded, acting [5,0] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.a is active+undersized+degraded, acting [1,4] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.d is active+undersized+degraded, acting [1,2] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.e is active+undersized+degraded, acting [2,0] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.12 is active+undersized+degraded, acting [1,0] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.14 is active+undersized+degraded, acting [4,5] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.1a is active+undersized+degraded, acting [4,5] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.1d is active+undersized+degraded, acting [5,0] 2024-03-23T11:56:03.967 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-03-23T11:56:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff96406c7d0 msgr2=0x7ff96406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff96406c7d0 0x7ff96406ec80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff97000b0d0 tx=0x7ff970009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9801035e0 msgr2=0x7ff980113db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9801035e0 0x7ff980113db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00eb40 tx=0x7ff96c00b040 comp rx=0 tx=0).stop 2024-03-23T11:56:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 shutdown_connections 2024-03-23T11:56:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff96406c7d0 0x7ff96406ec80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff980102c40 0x7ff980113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 --2- 172.21.15.63:0/3971692070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9801035e0 0x7ff980113db0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 >> 172.21.15.63:0/3971692070 conn(0x7ff9800f8830 msgr2=0x7ff9800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 shutdown_connections 2024-03-23T11:56:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:03.985+0000 7ff97f200700 1 -- 172.21.15.63:0/3971692070 wait complete. 2024-03-23T11:56:04.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:04 smithi063 ceph-mon[141124]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:04.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:04 smithi063 ceph-mon[141124]: pgmap v129: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:04.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:04 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3971692070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:56:04.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:04 smithi111 ceph-mon[106899]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:04.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:04 smithi111 ceph-mon[106899]: pgmap v129: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:04.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:04 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3971692070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:56:07.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:06 smithi063 ceph-mon[141124]: pgmap v130: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:07.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:07.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:07.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:06 smithi111 ceph-mon[106899]: pgmap v130: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:07.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:07.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:08.042 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:07 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T11:56:07.874+0000 7f4b7ee9f640 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-23T11:56:08.042 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:08.042 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:08.042 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:08 smithi111 ceph-mon[106899]: pgmap v131: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:08.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:08.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:08.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:08 smithi063 ceph-mon[141124]: pgmap v131: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 68/330 objects degraded (20.606%) 2024-03-23T11:56:08.757 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:08 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T11:56:08.454+0000 7f4b7ee9f640 -1 osd.3 64 log_to_monitors true 2024-03-23T11:56:09.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:09 smithi063 ceph-mon[141124]: from='osd.3 [v2:172.21.15.111:6800/1545952189,v1:172.21.15.111:6801/1545952189]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T11:56:09.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:09 smithi063 ceph-mon[141124]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T11:56:09.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:09 smithi111 ceph-mon[106899]: from='osd.3 [v2:172.21.15.111:6800/1545952189,v1:172.21.15.111:6801/1545952189]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T11:56:09.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:09 smithi111 ceph-mon[106899]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-23T11:56:09.402 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 11:56:09 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T11:56:09.047+0000 7f4b69c00700 -1 osd.3 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:56:10.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:10 smithi063 ceph-mon[141124]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-23T11:56:10.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:10 smithi063 ceph-mon[141124]: osdmap e67: 6 total, 5 up, 6 in 2024-03-23T11:56:10.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:10 smithi063 ceph-mon[141124]: from='osd.3 [v2:172.21.15.111:6800/1545952189,v1:172.21.15.111:6801/1545952189]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:10.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:10 smithi063 ceph-mon[141124]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:10.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:10 smithi063 ceph-mon[141124]: pgmap v133: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 68/330 objects degraded (20.606%) 2024-03-23T11:56:10.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:10 smithi111 ceph-mon[106899]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-23T11:56:10.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:10 smithi111 ceph-mon[106899]: osdmap e67: 6 total, 5 up, 6 in 2024-03-23T11:56:10.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:10 smithi111 ceph-mon[106899]: from='osd.3 [v2:172.21.15.111:6800/1545952189,v1:172.21.15.111:6801/1545952189]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:10.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:10 smithi111 ceph-mon[106899]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:10.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:10 smithi111 ceph-mon[106899]: pgmap v133: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 68/330 objects degraded (20.606%) 2024-03-23T11:56:11.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:11 smithi063 ceph-mon[141124]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:56:11.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:11 smithi063 ceph-mon[141124]: osd.3 [v2:172.21.15.111:6800/1545952189,v1:172.21.15.111:6801/1545952189] boot 2024-03-23T11:56:11.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:11 smithi063 ceph-mon[141124]: osdmap e68: 6 total, 6 up, 6 in 2024-03-23T11:56:11.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:56:11.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:11 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 68/330 objects degraded (20.606%), 19 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:11.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:11.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:11 smithi111 ceph-mon[106899]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:56:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:11 smithi111 ceph-mon[106899]: osd.3 [v2:172.21.15.111:6800/1545952189,v1:172.21.15.111:6801/1545952189] boot 2024-03-23T11:56:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:11 smithi111 ceph-mon[106899]: osdmap e68: 6 total, 6 up, 6 in 2024-03-23T11:56:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-23T11:56:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:11 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 68/330 objects degraded (20.606%), 19 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:12.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: osdmap e69: 6 total, 6 up, 6 in 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-23T11:56:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:12 smithi063 ceph-mon[141124]: pgmap v136: 65 pgs: 16 active+undersized+degraded, 14 active+undersized, 35 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s; 58/330 objects degraded (17.576%) 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: osdmap e69: 6 total, 6 up, 6 in 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:12.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-23T11:56:12.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:12 smithi111 ceph-mon[106899]: pgmap v136: 65 pgs: 16 active+undersized+degraded, 14 active+undersized, 35 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s; 58/330 objects degraded (17.576%) 2024-03-23T11:56:15.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:14 smithi063 ceph-mon[141124]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 14 active+undersized, 35 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 1 op/s; 58/330 objects degraded (17.576%) 2024-03-23T11:56:15.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:14 smithi111 ceph-mon[106899]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 14 active+undersized, 35 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 1 op/s; 58/330 objects degraded (17.576%) 2024-03-23T11:56:16.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:15 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 58/330 objects degraded (17.576%), 16 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:16.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:15 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 58/330 objects degraded (17.576%), 16 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:17.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:16 smithi063 ceph-mon[141124]: pgmap v138: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-23T11:56:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:16 smithi063 ceph-mon[141124]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/330 objects degraded (17.576%), 16 pgs degraded) 2024-03-23T11:56:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:16 smithi063 ceph-mon[141124]: Cluster is now healthy 2024-03-23T11:56:17.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:16 smithi111 ceph-mon[106899]: pgmap v138: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-23T11:56:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:16 smithi111 ceph-mon[106899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/330 objects degraded (17.576%), 16 pgs degraded) 2024-03-23T11:56:17.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:16 smithi111 ceph-mon[106899]: Cluster is now healthy 2024-03-23T11:56:19.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:18 smithi063 ceph-mon[141124]: pgmap v139: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:56:19.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:18 smithi111 ceph-mon[106899]: pgmap v139: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:56:21.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:20 smithi063 ceph-mon[141124]: pgmap v140: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T11:56:21.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:20 smithi111 ceph-mon[106899]: pgmap v140: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T11:56:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:23.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:22 smithi063 ceph-mon[141124]: pgmap v141: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-23T11:56:23.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:22 smithi111 ceph-mon[106899]: pgmap v141: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-23T11:56:25.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:24 smithi063 ceph-mon[141124]: pgmap v142: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:56:25.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:24 smithi111 ceph-mon[106899]: pgmap v142: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:56:27.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:26 smithi063 ceph-mon[141124]: pgmap v143: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:26 smithi111 ceph-mon[106899]: pgmap v143: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:28.053 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:27 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:28.053 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:27 smithi111 ceph-mon[106899]: Upgrade: osd.4 is safe to restart 2024-03-23T11:56:28.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:27 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-23T11:56:28.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:27 smithi063 ceph-mon[141124]: Upgrade: osd.4 is safe to restart 2024-03-23T11:56:29.069 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:28 smithi111 ceph-mon[106899]: pgmap v144: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:29.069 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:28 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:29.069 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:28 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T11:56:29.069 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:28 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:56:29.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:28 smithi063 ceph-mon[141124]: pgmap v144: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:28 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:28 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-23T11:56:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:28 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:56:30.120 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:29 smithi111 ceph-mon[106899]: Upgrade: Updating osd.4 2024-03-23T11:56:30.120 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:29 smithi111 ceph-mon[106899]: Deploying daemon osd.4 on smithi111 2024-03-23T11:56:30.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:29 smithi063 ceph-mon[141124]: Upgrade: Updating osd.4 2024-03-23T11:56:30.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:29 smithi063 ceph-mon[141124]: Deploying daemon osd.4 on smithi111 2024-03-23T11:56:30.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:30 smithi111 ceph-mon[106899]: pgmap v145: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:31.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:30 smithi063 ceph-mon[141124]: pgmap v145: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:32.402 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:32 smithi111 systemd[1]: Stopping Ceph osd.4 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:56:32.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:32 smithi111 ceph-mon[106899]: pgmap v146: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:32.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:32 smithi111 ceph-mon[106899]: osd.4 marked itself down and dead 2024-03-23T11:56:32.902 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:32 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[58272]: 2024-03-23T11:56:32.594+0000 7f5c87a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:56:32.902 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:32 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[58272]: 2024-03-23T11:56:32.594+0000 7f5c87a00700 -1 osd.4 69 *** Got signal Terminated *** 2024-03-23T11:56:32.902 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:32 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[58272]: 2024-03-23T11:56:32.594+0000 7f5c87a00700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T11:56:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:32 smithi063 ceph-mon[141124]: pgmap v146: 65 pgs: 65 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:56:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:32 smithi063 ceph-mon[141124]: osd.4 marked itself down and dead 2024-03-23T11:56:33.765 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:33 smithi111 bash[125915]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4 2024-03-23T11:56:34.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 -- 172.21.15.63:0/3421140572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac68102e20 msgr2=0x7fac681031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 --2- 172.21.15.63:0/3421140572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac68102e20 0x7fac681031f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fac58004940 tx=0x7fac58009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 -- 172.21.15.63:0/3421140572 shutdown_connections 2024-03-23T11:56:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 --2- 172.21.15.63:0/3421140572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac681037c0 0x7fac680fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 --2- 172.21.15.63:0/3421140572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac68102e20 0x7fac681031f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 -- 172.21.15.63:0/3421140572 >> 172.21.15.63:0/3421140572 conn(0x7fac680f89d0 msgr2=0x7fac680fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 -- 172.21.15.63:0/3421140572 shutdown_connections 2024-03-23T11:56:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.136+0000 7fac6e800700 1 -- 172.21.15.63:0/3421140572 wait complete. 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.137+0000 7fac6e800700 1 Processor -- start 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.137+0000 7fac6e800700 1 -- start start 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.137+0000 7fac6e800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac68102e20 0x7fac68113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.137+0000 7fac6e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac681037c0 0x7fac68113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.137+0000 7fac6e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac68114610 con 0x7fac681037c0 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac6e800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6810da00 con 0x7fac68102e20 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac66a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac681037c0 0x7fac68113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac66a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac681037c0 0x7fac68113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38280/0 (socket says 172.21.15.63:38280) 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac66a00700 1 -- 172.21.15.63:0/2180675227 learned_addr learned my addr 172.21.15.63:0/2180675227 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac67400700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac68102e20 0x7fac68113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac66a00700 1 -- 172.21.15.63:0/2180675227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac68102e20 msgr2=0x7fac68113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac66a00700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac68102e20 0x7fac68113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.138+0000 7fac66a00700 1 -- 172.21.15.63:0/2180675227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58005040 con 0x7fac681037c0 2024-03-23T11:56:34.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.139+0000 7fac66a00700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac681037c0 0x7fac68113fc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac54002a50 tx=0x7fac540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:34.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.139+0000 7fac5f400700 1 -- 172.21.15.63:0/2180675227 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac54007cc0 con 0x7fac681037c0 2024-03-23T11:56:34.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.139+0000 7fac5f400700 1 -- 172.21.15.63:0/2180675227 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac5400f040 con 0x7fac681037c0 2024-03-23T11:56:34.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.139+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6810dce0 con 0x7fac681037c0 2024-03-23T11:56:34.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.139+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6810e2a0 con 0x7fac681037c0 2024-03-23T11:56:34.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.139+0000 7fac5f400700 1 -- 172.21.15.63:0/2180675227 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac5400d9f0 con 0x7fac681037c0 2024-03-23T11:56:34.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.140+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac680fe570 con 0x7fac681037c0 2024-03-23T11:56:34.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.141+0000 7fac5f400700 1 -- 172.21.15.63:0/2180675227 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fac5400db50 con 0x7fac681037c0 2024-03-23T11:56:34.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.142+0000 7fac5f400700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac4c06c7d0 0x7fac4c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.142+0000 7fac5f400700 1 -- 172.21.15.63:0/2180675227 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fac5408b7b0 con 0x7fac681037c0 2024-03-23T11:56:34.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.143+0000 7fac67400700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac4c06c7d0 0x7fac4c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.143+0000 7fac67400700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac4c06c7d0 0x7fac4c06ec80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fac58009fd0 tx=0x7fac58009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:34.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.147+0000 7fac5f400700 1 -- 172.21.15.63:0/2180675227 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac540594e0 con 0x7fac681037c0 2024-03-23T11:56:34.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:33 smithi063 ceph-mon[141124]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:56:34.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:33 smithi063 ceph-mon[141124]: osdmap e70: 6 total, 5 up, 6 in 2024-03-23T11:56:34.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:33 smithi111 ceph-mon[106899]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:56:34.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:33 smithi111 ceph-mon[106899]: osdmap e70: 6 total, 5 up, 6 in 2024-03-23T11:56:34.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.380+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac68113750 con 0x7fac4c06c7d0 2024-03-23T11:56:34.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.382+0000 7fac5f400700 1 -- 172.21.15.63:0/2180675227 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fac68113750 con 0x7fac4c06c7d0 2024-03-23T11:56:34.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac4c06c7d0 msgr2=0x7fac4c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac4c06c7d0 0x7fac4c06ec80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fac58009fd0 tx=0x7fac58009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:34.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac681037c0 msgr2=0x7fac68113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac681037c0 0x7fac68113fc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac54002a50 tx=0x7fac540046b0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 shutdown_connections 2024-03-23T11:56:34.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac4c06c7d0 0x7fac4c06ec80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac68102e20 0x7fac68113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 --2- 172.21.15.63:0/2180675227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac681037c0 0x7fac68113fc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 >> 172.21.15.63:0/2180675227 conn(0x7fac680f89d0 msgr2=0x7fac68105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:34.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 shutdown_connections 2024-03-23T11:56:34.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.384+0000 7fac6e800700 1 -- 172.21.15.63:0/2180675227 wait complete. 2024-03-23T11:56:34.382 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:56:34.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 -- 172.21.15.63:0/2231231940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c102e00 msgr2=0x7ff50c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 --2- 172.21.15.63:0/2231231940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c102e00 0x7ff50c1031d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc004940 tx=0x7ff4fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 -- 172.21.15.63:0/2231231940 shutdown_connections 2024-03-23T11:56:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 --2- 172.21.15.63:0/2231231940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c1037a0 0x7ff50c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 --2- 172.21.15.63:0/2231231940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c102e00 0x7ff50c1031d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 -- 172.21.15.63:0/2231231940 >> 172.21.15.63:0/2231231940 conn(0x7ff50c0f89b0 msgr2=0x7ff50c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 -- 172.21.15.63:0/2231231940 shutdown_connections 2024-03-23T11:56:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.523+0000 7ff510e00700 1 -- 172.21.15.63:0/2231231940 wait complete. 2024-03-23T11:56:34.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.524+0000 7ff510e00700 1 Processor -- start 2024-03-23T11:56:34.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.524+0000 7ff510e00700 1 -- start start 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff510e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c102e00 0x7ff50c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff510e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c1037a0 0x7ff50c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff510e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c070d90 con 0x7ff50c1037a0 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff510e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c070ed0 con 0x7ff50c102e00 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff509400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c1037a0 0x7ff50c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff509e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c102e00 0x7ff50c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff509e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c102e00 0x7ff50c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33770/0 (socket says 172.21.15.63:33770) 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff509400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c1037a0 0x7ff50c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38304/0 (socket says 172.21.15.63:38304) 2024-03-23T11:56:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.525+0000 7ff509e00700 1 -- 172.21.15.63:0/868310111 learned_addr learned my addr 172.21.15.63:0/868310111 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff509e00700 1 -- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c1037a0 msgr2=0x7ff50c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff509e00700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c1037a0 0x7ff50c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff509e00700 1 -- 172.21.15.63:0/868310111 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc005040 con 0x7ff50c102e00 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff509e00700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c102e00 0x7ff50c0702c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009390 tx=0x7ff4fc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff502000700 1 -- 172.21.15.63:0/868310111 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4fc00e040 con 0x7ff50c102e00 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c071120 con 0x7ff50c102e00 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff502000700 1 -- 172.21.15.63:0/868310111 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4fc013420 con 0x7ff50c102e00 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.526+0000 7ff502000700 1 -- 172.21.15.63:0/868310111 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4fc01c630 con 0x7ff50c102e00 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.527+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c11a250 con 0x7ff50c102e00 2024-03-23T11:56:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.528+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c0fe550 con 0x7ff50c102e00 2024-03-23T11:56:34.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.529+0000 7ff502000700 1 -- 172.21.15.63:0/868310111 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff4fc0139e0 con 0x7ff50c102e00 2024-03-23T11:56:34.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.530+0000 7ff502000700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff4f006c7d0 0x7ff4f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.530+0000 7ff502000700 1 -- 172.21.15.63:0/868310111 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff4fc08cd70 con 0x7ff50c102e00 2024-03-23T11:56:34.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.530+0000 7ff509400700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff4f006c7d0 0x7ff4f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.531+0000 7ff509400700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff4f006c7d0 0x7ff4f006ec80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff50c071c00 tx=0x7ff4f8009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:34.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.535+0000 7ff502000700 1 -- 172.21.15.63:0/868310111 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff4fc056ff0 con 0x7ff50c102e00 2024-03-23T11:56:34.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.767+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff50c0719b0 con 0x7ff4f006c7d0 2024-03-23T11:56:34.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.769+0000 7ff502000700 1 -- 172.21.15.63:0/868310111 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ff50c0719b0 con 0x7ff4f006c7d0 2024-03-23T11:56:34.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff4f006c7d0 msgr2=0x7ff4f006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff4f006c7d0 0x7ff4f006ec80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff50c071c00 tx=0x7ff4f8009040 comp rx=0 tx=0).stop 2024-03-23T11:56:34.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c102e00 msgr2=0x7ff50c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c102e00 0x7ff50c0702c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009390 tx=0x7ff4fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:34.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 shutdown_connections 2024-03-23T11:56:34.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff4f006c7d0 0x7ff4f006ec80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff50c102e00 0x7ff50c0702c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 --2- 172.21.15.63:0/868310111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff50c1037a0 0x7ff50c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 >> 172.21.15.63:0/868310111 conn(0x7ff50c0f89b0 msgr2=0x7ff50c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:34.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 shutdown_connections 2024-03-23T11:56:34.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.771+0000 7ff510e00700 1 -- 172.21.15.63:0/868310111 wait complete. 2024-03-23T11:56:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.910+0000 7f3357a00700 1 -- 172.21.15.63:0/579955072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33501037a0 msgr2=0x7f33500fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.910+0000 7f3357a00700 1 --2- 172.21.15.63:0/579955072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33501037a0 0x7f33500fce10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f333c004940 tx=0x7f333c009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.910+0000 7f3357a00700 1 -- 172.21.15.63:0/579955072 shutdown_connections 2024-03-23T11:56:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.910+0000 7f3357a00700 1 --2- 172.21.15.63:0/579955072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33501037a0 0x7f33500fce10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.910+0000 7f3357a00700 1 --2- 172.21.15.63:0/579955072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3350102e00 0x7f33501031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.910+0000 7f3357a00700 1 -- 172.21.15.63:0/579955072 >> 172.21.15.63:0/579955072 conn(0x7f33500f89b0 msgr2=0x7f33500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.911+0000 7f3357a00700 1 -- 172.21.15.63:0/579955072 shutdown_connections 2024-03-23T11:56:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.911+0000 7f3357a00700 1 -- 172.21.15.63:0/579955072 wait complete. 2024-03-23T11:56:34.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.911+0000 7f3357a00700 1 Processor -- start 2024-03-23T11:56:34.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3357a00700 1 -- start start 2024-03-23T11:56:34.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3357a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3350102e00 0x7f3350113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3357a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33501037a0 0x7f3350113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3357a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3350114600 con 0x7f3350102e00 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3357a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335010d9f0 con 0x7f33501037a0 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3355000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3350102e00 0x7f3350113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3355000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3350102e00 0x7f3350113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38324/0 (socket says 172.21.15.63:38324) 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f3355000700 1 -- 172.21.15.63:0/1509946994 learned_addr learned my addr 172.21.15.63:0/1509946994 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.912+0000 7f334fe00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33501037a0 0x7f3350113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f3355000700 1 -- 172.21.15.63:0/1509946994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33501037a0 msgr2=0x7f3350113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f3355000700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33501037a0 0x7f3350113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:34.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f3355000700 1 -- 172.21.15.63:0/1509946994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c005040 con 0x7f3350102e00 2024-03-23T11:56:34.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f3355000700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3350102e00 0x7f3350113a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f334400eb40 tx=0x7f334400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:34.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f334cc00700 1 -- 172.21.15.63:0/1509946994 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3344007cc0 con 0x7f3350102e00 2024-03-23T11:56:34.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f334cc00700 1 -- 172.21.15.63:0/1509946994 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f334400f040 con 0x7f3350102e00 2024-03-23T11:56:34.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335010dc70 con 0x7f3350102e00 2024-03-23T11:56:34.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f334cc00700 1 -- 172.21.15.63:0/1509946994 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f334400c990 con 0x7f3350102e00 2024-03-23T11:56:34.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.913+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335010e200 con 0x7f3350102e00 2024-03-23T11:56:34.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.914+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33500fe550 con 0x7f3350102e00 2024-03-23T11:56:34.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.915+0000 7f334cc00700 1 -- 172.21.15.63:0/1509946994 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f334400caf0 con 0x7f3350102e00 2024-03-23T11:56:34.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.916+0000 7f334cc00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f333806c7d0 0x7f333806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:34.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.916+0000 7f334cc00700 1 -- 172.21.15.63:0/1509946994 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f334408b520 con 0x7f3350102e00 2024-03-23T11:56:34.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.917+0000 7f334fe00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f333806c7d0 0x7f333806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:34.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.917+0000 7f334fe00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f333806c7d0 0x7f333806ec80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f335010ecd0 tx=0x7f333c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:34.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:34.921+0000 7f334cc00700 1 -- 172.21.15.63:0/1509946994 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3344059380 con 0x7f3350102e00 2024-03-23T11:56:35.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.146+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3350109da0 con 0x7f333806c7d0 2024-03-23T11:56:35.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:34 smithi063 ceph-mon[141124]: pgmap v148: 65 pgs: 6 stale+active+clean, 59 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T11:56:35.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:34 smithi063 ceph-mon[141124]: osdmap e71: 6 total, 5 up, 6 in 2024-03-23T11:56:35.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.156+0000 7f334cc00700 1 -- 172.21.15.63:0/1509946994 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3350109da0 con 0x7f333806c7d0 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (5m) 66s ago 18m 14.2M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (3m) 66s ago 19m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (2m) 29s ago 18m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (5m) 66s ago 18m 73.4M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (12m) 66s ago 12m 118M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (12m) 66s ago 12m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (12m) 29s ago 12m 65.2M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (12m) 29s ago 12m 60.1M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (8m) 66s ago 20m 502M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:56:35.140 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (6m) 29s ago 17m 451M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (4m) 66s ago 20m 45.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (3m) 29s ago 17m 38.8M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (6m) 66s ago 19m 7532k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (6m) 29s ago 17m 7880k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (2m) 66s ago 16m 87.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (113s) 66s ago 16m 91.1M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (73s) 66s ago 15m 11.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (33s) 29s ago 15m 11.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (14m) 29s ago 14m 232M 2642M 17.2.7-726-g9c4e0c34 837124da804f 0737a85269a3 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (14m) 29s ago 14m 228M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:56:35.141 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (6m) 66s ago 18m 53.4M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:56:35.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.158+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f333806c7d0 msgr2=0x7f333806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.158+0000 7f3357a00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f333806c7d0 0x7f333806ec80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f335010ecd0 tx=0x7f333c009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:35.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.158+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3350102e00 msgr2=0x7f3350113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.158+0000 7f3357a00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3350102e00 0x7f3350113a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f334400eb40 tx=0x7f334400b040 comp rx=0 tx=0).stop 2024-03-23T11:56:35.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.159+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 shutdown_connections 2024-03-23T11:56:35.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.159+0000 7f3357a00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f333806c7d0 0x7f333806ec80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.159+0000 7f3357a00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3350102e00 0x7f3350113a70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.159+0000 7f3357a00700 1 --2- 172.21.15.63:0/1509946994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33501037a0 0x7f3350113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.159+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 >> 172.21.15.63:0/1509946994 conn(0x7f33500f89b0 msgr2=0x7f33500f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:35.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.159+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 shutdown_connections 2024-03-23T11:56:35.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.159+0000 7f3357a00700 1 -- 172.21.15.63:0/1509946994 wait complete. 2024-03-23T11:56:35.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:34 smithi111 ceph-mon[106899]: pgmap v148: 65 pgs: 6 stale+active+clean, 59 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T11:56:35.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:34 smithi111 ceph-mon[106899]: osdmap e71: 6 total, 5 up, 6 in 2024-03-23T11:56:35.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.294+0000 7eff16e00700 1 -- 172.21.15.63:0/3536147057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070360 msgr2=0x7eff10070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.294+0000 7eff16e00700 1 --2- 172.21.15.63:0/3536147057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070360 0x7eff10070730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff00004970 tx=0x7eff00009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:35.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.295+0000 7eff16e00700 1 -- 172.21.15.63:0/3536147057 shutdown_connections 2024-03-23T11:56:35.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.295+0000 7eff16e00700 1 --2- 172.21.15.63:0/3536147057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070c70 0x7eff10109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.295+0000 7eff16e00700 1 --2- 172.21.15.63:0/3536147057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070360 0x7eff10070730 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.295+0000 7eff16e00700 1 -- 172.21.15.63:0/3536147057 >> 172.21.15.63:0/3536147057 conn(0x7eff100f89b0 msgr2=0x7eff100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:35.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.295+0000 7eff16e00700 1 -- 172.21.15.63:0/3536147057 shutdown_connections 2024-03-23T11:56:35.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.295+0000 7eff16e00700 1 -- 172.21.15.63:0/3536147057 wait complete. 2024-03-23T11:56:35.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff16e00700 1 Processor -- start 2024-03-23T11:56:35.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff16e00700 1 -- start start 2024-03-23T11:56:35.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff16e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070360 0x7eff10115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:35.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff16e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070c70 0x7eff101161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:35.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff16e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff10116810 con 0x7eff10070c70 2024-03-23T11:56:35.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff16e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1010fbe0 con 0x7eff10070360 2024-03-23T11:56:35.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070c70 0x7eff101161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:35.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff0fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070360 0x7eff10115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:35.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070c70 0x7eff101161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38348/0 (socket says 172.21.15.63:38348) 2024-03-23T11:56:35.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff0f400700 1 -- 172.21.15.63:0/95061107 learned_addr learned my addr 172.21.15.63:0/95061107 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:35.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.296+0000 7eff0fe00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070360 0x7eff10115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:33806/0 (socket says 172.21.15.63:33806) 2024-03-23T11:56:35.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff0fe00700 1 -- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070c70 msgr2=0x7eff101161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff0fe00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070c70 0x7eff101161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff0fe00700 1 -- 172.21.15.63:0/95061107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff00005040 con 0x7eff10070360 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff0fe00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070360 0x7eff10115c80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7eff00005010 tx=0x7eff00009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff07e00700 1 -- 172.21.15.63:0/95061107 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0000e040 con 0x7eff10070360 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff07e00700 1 -- 172.21.15.63:0/95061107 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff00004290 con 0x7eff10070360 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff07e00700 1 -- 172.21.15.63:0/95061107 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff00013c10 con 0x7eff10070360 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.297+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1010fe60 con 0x7eff10070360 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.298+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff101103a0 con 0x7eff10070360 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.299+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff10103b90 con 0x7eff10070360 2024-03-23T11:56:35.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.300+0000 7eff07e00700 1 -- 172.21.15.63:0/95061107 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7eff00013420 con 0x7eff10070360 2024-03-23T11:56:35.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.300+0000 7eff07e00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efef406c7d0 0x7efef406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:35.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.300+0000 7eff07e00700 1 -- 172.21.15.63:0/95061107 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7eff0008ca40 con 0x7eff10070360 2024-03-23T11:56:35.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.301+0000 7eff0f400700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efef406c7d0 0x7efef406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:35.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.301+0000 7eff0f400700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efef406c7d0 0x7efef406ec80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efefc008fd0 tx=0x7efefc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:35.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.305+0000 7eff07e00700 1 -- 172.21.15.63:0/95061107 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff0005a750 con 0x7eff10070360 2024-03-23T11:56:35.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.607+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7eff1004bed0 con 0x7eff10070360 2024-03-23T11:56:35.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.608+0000 7eff07e00700 1 -- 172.21.15.63:0/95061107 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7eff0005a2e0 con 0x7eff10070360 2024-03-23T11:56:35.591 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:56:35.592 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 5, 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 8 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:56:35.593 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:56:35.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.610+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efef406c7d0 msgr2=0x7efef406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.610+0000 7eff16e00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efef406c7d0 0x7efef406ec80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efefc008fd0 tx=0x7efefc00b040 comp rx=0 tx=0).stop 2024-03-23T11:56:35.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.610+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070360 msgr2=0x7eff10115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.610+0000 7eff16e00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070360 0x7eff10115c80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7eff00005010 tx=0x7eff00009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:35.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.611+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 shutdown_connections 2024-03-23T11:56:35.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.611+0000 7eff16e00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efef406c7d0 0x7efef406ec80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.611+0000 7eff16e00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff10070360 0x7eff10115c80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.611+0000 7eff16e00700 1 --2- 172.21.15.63:0/95061107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff10070c70 0x7eff101161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.611+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 >> 172.21.15.63:0/95061107 conn(0x7eff100f89b0 msgr2=0x7eff101015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:35.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.611+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 shutdown_connections 2024-03-23T11:56:35.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.611+0000 7eff16e00700 1 -- 172.21.15.63:0/95061107 wait complete. 2024-03-23T11:56:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 -- 172.21.15.63:0/4001061709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fe830 msgr2=0x7f8e00106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 --2- 172.21.15.63:0/4001061709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fe830 0x7f8e00106f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004940 tx=0x7f8dec009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 -- 172.21.15.63:0/4001061709 shutdown_connections 2024-03-23T11:56:35.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 --2- 172.21.15.63:0/4001061709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fe830 0x7f8e00106f00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 --2- 172.21.15.63:0/4001061709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e000fde90 0x7f8e000fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 -- 172.21.15.63:0/4001061709 >> 172.21.15.63:0/4001061709 conn(0x7f8e00072bc0 msgr2=0x7f8e00072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:35.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 -- 172.21.15.63:0/4001061709 shutdown_connections 2024-03-23T11:56:35.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.747+0000 7f8e04c00700 1 -- 172.21.15.63:0/4001061709 wait complete. 2024-03-23T11:56:35.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.748+0000 7f8e04c00700 1 Processor -- start 2024-03-23T11:56:35.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.748+0000 7f8e04c00700 1 -- start start 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8e04c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fde90 0x7f8e000fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8e04c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e000fe830 0x7f8e000fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8e04c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000fc820 con 0x7f8e000fde90 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8e04c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000fc960 con 0x7f8e000fe830 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8dfde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fde90 0x7f8e000fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8dfde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fde90 0x7f8e000fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38366/0 (socket says 172.21.15.63:38366) 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8dfde00700 1 -- 172.21.15.63:0/2716023508 learned_addr learned my addr 172.21.15.63:0/2716023508 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.749+0000 7f8dfd400700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e000fe830 0x7f8e000fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8dfde00700 1 -- 172.21.15.63:0/2716023508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e000fe830 msgr2=0x7f8e000fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8dfde00700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e000fe830 0x7f8e000fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8dfde00700 1 -- 172.21.15.63:0/2716023508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec005040 con 0x7f8e000fde90 2024-03-23T11:56:35.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8dfde00700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fde90 0x7f8e000fbd50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8df000eb40 tx=0x7f8df000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:35.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8df6000700 1 -- 172.21.15.63:0/2716023508 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df0007cc0 con 0x7f8e000fde90 2024-03-23T11:56:35.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8df6000700 1 -- 172.21.15.63:0/2716023508 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8df000f040 con 0x7f8e000fde90 2024-03-23T11:56:35.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e000fcbb0 con 0x7f8e000fde90 2024-03-23T11:56:35.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8df6000700 1 -- 172.21.15.63:0/2716023508 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df000c990 con 0x7f8e000fde90 2024-03-23T11:56:35.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.750+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0011a130 con 0x7f8e000fde90 2024-03-23T11:56:35.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.751+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e00065e30 con 0x7f8e000fde90 2024-03-23T11:56:35.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.752+0000 7f8df6000700 1 -- 172.21.15.63:0/2716023508 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8df000cc00 con 0x7f8e000fde90 2024-03-23T11:56:35.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.753+0000 7f8df6000700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7d0 0x7f8de406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.753+0000 7f8df6000700 1 -- 172.21.15.63:0/2716023508 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f8df008b670 con 0x7f8e000fde90 2024-03-23T11:56:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.754+0000 7f8dfd400700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7d0 0x7f8de406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:35.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.754+0000 7f8dfd400700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7d0 0x7f8de406ec80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e000fd690 tx=0x7f8dec009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:35.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:35.758+0000 7f8df6000700 1 -- 172.21.15.63:0/2716023508 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8df0055920 con 0x7f8e000fde90 2024-03-23T11:56:35.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:35 smithi111 ceph-mon[106899]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:35.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:35 smithi111 ceph-mon[106899]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:35.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:35 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/95061107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:35.902 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:35 smithi111 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.4.service: Succeeded. 2024-03-23T11:56:35.902 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:35 smithi111 systemd[1]: Stopped Ceph osd.4 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:56:36.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.017+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8e000fd440 con 0x7f8e000fde90 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.018+0000 7f8df6000700 1 -- 172.21.15.63:0/2716023508 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f8df0058f40 con 0x7f8e000fde90 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:56:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:56:36.003 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:36.004 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:36.005 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:56:36.005 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:56:36.005 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7d0 msgr2=0x7f8de406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7d0 0x7f8de406ec80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e000fd690 tx=0x7f8dec009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fde90 msgr2=0x7f8e000fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fde90 0x7f8e000fbd50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8df000eb40 tx=0x7f8df000b040 comp rx=0 tx=0).stop 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 shutdown_connections 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7d0 0x7f8de406ec80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e000fde90 0x7f8e000fbd50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 --2- 172.21.15.63:0/2716023508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e000fe830 0x7f8e000fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 >> 172.21.15.63:0/2716023508 conn(0x7f8e00072bc0 msgr2=0x7f8e000ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:36.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 shutdown_connections 2024-03-23T11:56:36.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.020+0000 7f8e04c00700 1 -- 172.21.15.63:0/2716023508 wait complete. 2024-03-23T11:56:36.007 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:56:36.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:35 smithi063 ceph-mon[141124]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:36.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:35 smithi063 ceph-mon[141124]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:36.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:35 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/95061107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.157+0000 7f925a200700 1 -- 172.21.15.63:0/499857340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 msgr2=0x7f92540fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.157+0000 7f925a200700 1 --2- 172.21.15.63:0/499857340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f92540fce10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9240004940 tx=0x7f9240009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.157+0000 7f925a200700 1 -- 172.21.15.63:0/499857340 shutdown_connections 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.157+0000 7f925a200700 1 --2- 172.21.15.63:0/499857340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f92540fce10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.157+0000 7f925a200700 1 --2- 172.21.15.63:0/499857340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9254102e00 0x7f92541031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.157+0000 7f925a200700 1 -- 172.21.15.63:0/499857340 >> 172.21.15.63:0/499857340 conn(0x7f92540f89b0 msgr2=0x7f92540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.158+0000 7f925a200700 1 -- 172.21.15.63:0/499857340 shutdown_connections 2024-03-23T11:56:36.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.158+0000 7f925a200700 1 -- 172.21.15.63:0/499857340 wait complete. 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.158+0000 7f925a200700 1 Processor -- start 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f925a200700 1 -- start start 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f925a200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9254102e00 0x7f92540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f925a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f9254070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f925a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9254070d90 con 0x7f92541037a0 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f925a200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9254070ed0 con 0x7f9254102e00 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f9252a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f9254070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f9252a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f9254070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38388/0 (socket says 172.21.15.63:38388) 2024-03-23T11:56:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f9252a00700 1 -- 172.21.15.63:0/4050581617 learned_addr learned my addr 172.21.15.63:0/4050581617 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:36.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.159+0000 7f9253400700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9254102e00 0x7f92540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:36.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f9252a00700 1 -- 172.21.15.63:0/4050581617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9254102e00 msgr2=0x7f92540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f9252a00700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9254102e00 0x7f92540702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f9252a00700 1 -- 172.21.15.63:0/4050581617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240005040 con 0x7f92541037a0 2024-03-23T11:56:36.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f9252a00700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f9254070800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9240008d50 tx=0x7f9240009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:36.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f924b400700 1 -- 172.21.15.63:0/4050581617 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924000e030 con 0x7f92541037a0 2024-03-23T11:56:36.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f924b400700 1 -- 172.21.15.63:0/4050581617 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92400093b0 con 0x7f92541037a0 2024-03-23T11:56:36.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9254071120 con 0x7f92541037a0 2024-03-23T11:56:36.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.160+0000 7f924b400700 1 -- 172.21.15.63:0/4050581617 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924001cbb0 con 0x7f92541037a0 2024-03-23T11:56:36.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.161+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925411a1a0 con 0x7f92541037a0 2024-03-23T11:56:36.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.162+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92540fe550 con 0x7f92541037a0 2024-03-23T11:56:36.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.163+0000 7f924b400700 1 -- 172.21.15.63:0/4050581617 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9240013930 con 0x7f92541037a0 2024-03-23T11:56:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.164+0000 7f924b400700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f923806c7d0 0x7f923806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.164+0000 7f924b400700 1 -- 172.21.15.63:0/4050581617 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f924008d1a0 con 0x7f92541037a0 2024-03-23T11:56:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.164+0000 7f9253400700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f923806c7d0 0x7f923806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.164+0000 7f9253400700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f923806c7d0 0x7f923806ec80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9244009b80 tx=0x7f924400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:36.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.168+0000 7f924b400700 1 -- 172.21.15.63:0/4050581617 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9240057440 con 0x7f92541037a0 2024-03-23T11:56:36.380 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:35 smithi111 systemd[1]: Starting Ceph osd.4 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:56:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.401+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9254109e40 con 0x7f923806c7d0 2024-03-23T11:56:36.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.402+0000 7f924b400700 1 -- 172.21.15.63:0/4050581617 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f924400f040 con 0x7f923806c7d0 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "10/21 daemons upgraded", 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading osd daemons", 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:56:36.386 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:56:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f923806c7d0 msgr2=0x7f923806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f923806c7d0 0x7f923806ec80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9244009b80 tx=0x7f924400b040 comp rx=0 tx=0).stop 2024-03-23T11:56:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 msgr2=0x7f9254070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f9254070800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9240008d50 tx=0x7f9240009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 shutdown_connections 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f923806c7d0 0x7f923806ec80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9254102e00 0x7f92540702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 --2- 172.21.15.63:0/4050581617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92541037a0 0x7f9254070800 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 >> 172.21.15.63:0/4050581617 conn(0x7f92540f89b0 msgr2=0x7f92540f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 shutdown_connections 2024-03-23T11:56:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.405+0000 7f925a200700 1 -- 172.21.15.63:0/4050581617 wait complete. 2024-03-23T11:56:36.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 -- 172.21.15.63:0/3095641641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 msgr2=0x7f730c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 --2- 172.21.15.63:0/3095641641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c1031f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f72fc004940 tx=0x7f72fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:36.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 -- 172.21.15.63:0/3095641641 shutdown_connections 2024-03-23T11:56:36.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 --2- 172.21.15.63:0/3095641641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f730c1037c0 0x7f730c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 --2- 172.21.15.63:0/3095641641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c1031f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 -- 172.21.15.63:0/3095641641 >> 172.21.15.63:0/3095641641 conn(0x7f730c0f89d0 msgr2=0x7f730c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:36.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 -- 172.21.15.63:0/3095641641 shutdown_connections 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.542+0000 7f7311200700 1 -- 172.21.15.63:0/3095641641 wait complete. 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.543+0000 7f7311200700 1 Processor -- start 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.543+0000 7f7311200700 1 -- start start 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.543+0000 7f7311200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.543+0000 7f7311200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f730c1037c0 0x7f730c114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.543+0000 7f7311200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c114690 con 0x7f730c102e20 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.543+0000 7f7311200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c10da80 con 0x7f730c1037c0 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38402/0 (socket says 172.21.15.63:38402) 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309e00700 1 -- 172.21.15.63:0/4278030441 learned_addr learned my addr 172.21.15.63:0/4278030441 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309400700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f730c1037c0 0x7f730c114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309e00700 1 -- 172.21.15.63:0/4278030441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f730c1037c0 msgr2=0x7f730c114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309e00700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f730c1037c0 0x7f730c114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309e00700 1 -- 172.21.15.63:0/4278030441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc005040 con 0x7f730c102e20 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7309e00700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c113b00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009390 tx=0x7f72fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.544+0000 7f7302000700 1 -- 172.21.15.63:0/4278030441 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72fc00e040 con 0x7f730c102e20 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.545+0000 7f7302000700 1 -- 172.21.15.63:0/4278030441 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72fc013420 con 0x7f730c102e20 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.545+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c10dd00 con 0x7f730c102e20 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.545+0000 7f7302000700 1 -- 172.21.15.63:0/4278030441 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72fc01c630 con 0x7f730c102e20 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.545+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c10e2c0 con 0x7f730c102e20 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.546+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c0fe570 con 0x7f730c102e20 2024-03-23T11:56:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.547+0000 7f7302000700 1 -- 172.21.15.63:0/4278030441 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f72fc0139e0 con 0x7f730c102e20 2024-03-23T11:56:36.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.548+0000 7f7302000700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72f006c7d0 0x7f72f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:56:36.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.548+0000 7f7302000700 1 -- 172.21.15.63:0/4278030441 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f72fc08cdb0 con 0x7f730c102e20 2024-03-23T11:56:36.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.548+0000 7f7309400700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72f006c7d0 0x7f72f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:56:36.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.549+0000 7f7309400700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72f006c7d0 0x7f72f006ec80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f730c10ec40 tx=0x7f72f800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:56:36.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.552+0000 7f7302000700 1 -- 172.21.15.63:0/4278030441 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f72fc05aac0 con 0x7f730c102e20 2024-03-23T11:56:36.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.850+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f730c0fe570 con 0x7f730c102e20 2024-03-23T11:56:36.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.851+0000 7f7302000700 1 -- 172.21.15.63:0/4278030441 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+942 (secure 0 0 0) 0x7f72fc05a650 con 0x7f730c102e20 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 3 pgs peering; Degraded data redundancy: 36/330 objects degraded (10.909%), 6 pgs degraded 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout: osd.4 (root=default,host=smithi111) is down 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs peering 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.f is stuck peering for 8m, current state peering, last acting [0,5] 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout: pg 3.3 is stuck peering for 7m, current state peering, last acting [0,3] 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout: pg 3.9 is stuck peering for 7m, current state peering, last acting [2,5] 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 36/330 objects degraded (10.909%), 6 pgs degraded 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.5 is active+undersized+degraded, acting [3,0] 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.7 is active+undersized+degraded, acting [3,2] 2024-03-23T11:56:36.835 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.b is active+undersized+degraded, acting [3,5] 2024-03-23T11:56:36.836 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.11 is active+undersized+degraded, acting [3,1] 2024-03-23T11:56:36.836 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-23T11:56:36.836 INFO:teuthology.orchestra.run.smithi063.stdout: pg 2.1a is active+undersized+degraded, acting [3,5] 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72f006c7d0 msgr2=0x7f72f006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72f006c7d0 0x7f72f006ec80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f730c10ec40 tx=0x7f72f800b040 comp rx=0 tx=0).stop 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 msgr2=0x7f730c113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c113b00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009390 tx=0x7f72fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 shutdown_connections 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72f006c7d0 0x7f72f006ec80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c102e20 0x7f730c113b00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 --2- 172.21.15.63:0/4278030441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f730c1037c0 0x7f730c114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:56:36.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 >> 172.21.15.63:0/4278030441 conn(0x7f730c0f89d0 msgr2=0x7f730c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:56:36.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 shutdown_connections 2024-03-23T11:56:36.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:56:36.853+0000 7f7311200700 1 -- 172.21.15.63:0/4278030441 wait complete. 2024-03-23T11:56:37.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:36 smithi063 ceph-mon[141124]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:36 smithi063 ceph-mon[141124]: pgmap v150: 65 pgs: 10 peering, 6 active+undersized+degraded, 8 activating+undersized, 1 stale+active+clean, 5 active+undersized, 35 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 36/330 objects degraded (10.909%) 2024-03-23T11:56:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:36 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2716023508' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:56:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:36 smithi063 ceph-mon[141124]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-23T11:56:37.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:36 smithi063 ceph-mon[141124]: Health check failed: Degraded data redundancy: 36/330 objects degraded (10.909%), 6 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:37.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:36 smithi111 ceph-mon[106899]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:37.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:36 smithi111 ceph-mon[106899]: pgmap v150: 65 pgs: 10 peering, 6 active+undersized+degraded, 8 activating+undersized, 1 stale+active+clean, 5 active+undersized, 35 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 36/330 objects degraded (10.909%) 2024-03-23T11:56:37.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:36 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2716023508' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:56:37.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:36 smithi111 ceph-mon[106899]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-23T11:56:37.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:36 smithi111 ceph-mon[106899]: Health check failed: Degraded data redundancy: 36/330 objects degraded (10.909%), 6 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:37.575 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:37 smithi111 bash[126190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-23T11:56:37.575 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:37 smithi111 bash[126190]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-23T11:56:37.575 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:37 smithi111 bash[126190]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-23T11:56:37.575 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:37 smithi111 bash[126190]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-23T11:56:37.575 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:37 smithi111 bash[126190]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-23T11:56:37.575 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:37 smithi111 bash[126190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-23T11:56:37.576 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:37 smithi111 bash[126190]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-23T11:56:37.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:37 smithi111 ceph-mon[106899]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:37.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:37 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4278030441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:56:38.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:37 smithi063 ceph-mon[141124]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:56:38.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:37 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4278030441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:56:38.790 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:38 smithi111 ceph-osd[126512]: -- 172.21.15.111:0/1821968047 <== mon.1 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d69be71e0 con 0x557d69bb6c00 2024-03-23T11:56:38.790 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:38 smithi111 bash[126474]: 22ff65335e1de2c26d71d542f75243c7ec4ca30d73d7bbf979dbeaee738b2ca8 2024-03-23T11:56:38.790 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:38 smithi111 systemd[1]: Started Ceph osd.4 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:56:39.069 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:38 smithi111 ceph-mon[106899]: pgmap v151: 65 pgs: 10 peering, 8 active+undersized+degraded, 8 activating+undersized, 8 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 41/330 objects degraded (12.424%) 2024-03-23T11:56:39.070 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:38 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:39.070 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:38 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:39.070 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:38 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:39.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:38 smithi063 ceph-mon[141124]: pgmap v151: 65 pgs: 10 peering, 8 active+undersized+degraded, 8 activating+undersized, 8 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 41/330 objects degraded (12.424%) 2024-03-23T11:56:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:39.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:39.402 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:39 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T11:56:39.085+0000 7f249dfcb640 -1 Falling back to public interface 2024-03-23T11:56:40.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:39 smithi063 ceph-mon[141124]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-23T11:56:40.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:39 smithi111 ceph-mon[106899]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-23T11:56:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:40 smithi063 ceph-mon[141124]: pgmap v152: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/330 objects degraded (16.364%) 2024-03-23T11:56:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:41.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:40 smithi111 ceph-mon[106899]: pgmap v152: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/330 objects degraded (16.364%) 2024-03-23T11:56:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:43.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:42 smithi063 ceph-mon[141124]: pgmap v153: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/330 objects degraded (16.364%) 2024-03-23T11:56:43.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:43.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:42 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:42 smithi111 ceph-mon[106899]: pgmap v153: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/330 objects degraded (16.364%) 2024-03-23T11:56:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:42 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:42 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:44.232 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:43 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T11:56:43.954+0000 7f249dfcb640 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-23T11:56:44.533 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:44 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:44.533 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:44 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:44.533 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:44 smithi111 ceph-mon[106899]: pgmap v154: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/330 objects degraded (16.364%) 2024-03-23T11:56:44.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:44 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:44.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:44 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:44.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:44 smithi063 ceph-mon[141124]: pgmap v154: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/330 objects degraded (16.364%) 2024-03-23T11:56:44.888 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:44 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T11:56:44.549+0000 7f249dfcb640 -1 osd.4 69 log_to_monitors true 2024-03-23T11:56:45.453 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:45 smithi111 ceph-mon[106899]: from='osd.4 [v2:172.21.15.111:6808/2121500845,v1:172.21.15.111:6809/2121500845]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:56:45.453 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:45 smithi111 ceph-mon[106899]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:56:45.453 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:45 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 54/330 objects degraded (16.364%), 16 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:45.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:45 smithi063 ceph-mon[141124]: from='osd.4 [v2:172.21.15.111:6808/2121500845,v1:172.21.15.111:6809/2121500845]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:56:45.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:45 smithi063 ceph-mon[141124]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-23T11:56:45.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:45 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 54/330 objects degraded (16.364%), 16 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:45.902 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 11:56:45 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T11:56:45.468+0000 7f2489800700 -1 osd.4 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:56:46.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:46 smithi063 ceph-mon[141124]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-23T11:56:46.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:46 smithi063 ceph-mon[141124]: osdmap e72: 6 total, 5 up, 6 in 2024-03-23T11:56:46.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:46 smithi063 ceph-mon[141124]: from='osd.4 [v2:172.21.15.111:6808/2121500845,v1:172.21.15.111:6809/2121500845]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:46.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:46 smithi063 ceph-mon[141124]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:46.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:46 smithi063 ceph-mon[141124]: pgmap v156: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 54/330 objects degraded (16.364%) 2024-03-23T11:56:46.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:46 smithi111 ceph-mon[106899]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-23T11:56:46.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:46 smithi111 ceph-mon[106899]: osdmap e72: 6 total, 5 up, 6 in 2024-03-23T11:56:46.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:46 smithi111 ceph-mon[106899]: from='osd.4 [v2:172.21.15.111:6808/2121500845,v1:172.21.15.111:6809/2121500845]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:46.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:46 smithi111 ceph-mon[106899]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:56:46.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:46 smithi111 ceph-mon[106899]: pgmap v156: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 54/330 objects degraded (16.364%) 2024-03-23T11:56:47.790 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:47 smithi111 ceph-mon[106899]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:56:47.790 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:47 smithi111 ceph-mon[106899]: osd.4 [v2:172.21.15.111:6808/2121500845,v1:172.21.15.111:6809/2121500845] boot 2024-03-23T11:56:47.790 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:47 smithi111 ceph-mon[106899]: osdmap e73: 6 total, 6 up, 6 in 2024-03-23T11:56:47.790 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:47 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:56:47.790 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:47 smithi111 ceph-mon[106899]: osdmap e74: 6 total, 6 up, 6 in 2024-03-23T11:56:47.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:47 smithi063 ceph-mon[141124]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:56:47.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:47 smithi063 ceph-mon[141124]: osd.4 [v2:172.21.15.111:6808/2121500845,v1:172.21.15.111:6809/2121500845] boot 2024-03-23T11:56:47.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:47 smithi063 ceph-mon[141124]: osdmap e73: 6 total, 6 up, 6 in 2024-03-23T11:56:47.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:47 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-23T11:56:47.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:47 smithi063 ceph-mon[141124]: osdmap e74: 6 total, 6 up, 6 in 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: pgmap v159: 65 pgs: 4 peering, 15 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 51/330 objects degraded (15.455%) 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:48.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:48.883 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:48 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:56:48.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: pgmap v159: 65 pgs: 4 peering, 15 active+undersized+degraded, 15 active+undersized, 31 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 51/330 objects degraded (15.455%) 2024-03-23T11:56:48.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:56:48.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:48 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:56:49.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:49 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:56:49.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:49 smithi063 ceph-mon[141124]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-23T11:56:49.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:49 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:56:49.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:49 smithi111 ceph-mon[106899]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-23T11:56:50.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:50 smithi063 ceph-mon[141124]: pgmap v160: 65 pgs: 4 peering, 3 active+undersized+degraded, 3 active+undersized, 55 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 7/330 objects degraded (2.121%) 2024-03-23T11:56:50.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:50 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 51/330 objects degraded (15.455%), 15 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:50.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:50 smithi111 ceph-mon[106899]: pgmap v160: 65 pgs: 4 peering, 3 active+undersized+degraded, 3 active+undersized, 55 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 7/330 objects degraded (2.121%) 2024-03-23T11:56:50.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:50 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 51/330 objects degraded (15.455%), 15 pgs degraded (PG_DEGRADED) 2024-03-23T11:56:52.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:51 smithi063 ceph-mon[141124]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/330 objects degraded (2.121%), 3 pgs degraded) 2024-03-23T11:56:52.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:51 smithi063 ceph-mon[141124]: Cluster is now healthy 2024-03-23T11:56:52.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:51 smithi111 ceph-mon[106899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/330 objects degraded (2.121%), 3 pgs degraded) 2024-03-23T11:56:52.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:51 smithi111 ceph-mon[106899]: Cluster is now healthy 2024-03-23T11:56:53.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:52 smithi063 ceph-mon[141124]: pgmap v161: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1 op/s 2024-03-23T11:56:53.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:52 smithi111 ceph-mon[106899]: pgmap v161: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1 op/s 2024-03-23T11:56:55.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:54 smithi063 ceph-mon[141124]: pgmap v162: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-23T11:56:55.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:54 smithi111 ceph-mon[106899]: pgmap v162: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-23T11:56:57.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:56 smithi063 ceph-mon[141124]: pgmap v163: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-03-23T11:56:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:57.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:57.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:56 smithi111 ceph-mon[106899]: pgmap v163: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-03-23T11:56:57.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:56:57.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:56:59.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:56:58 smithi063 ceph-mon[141124]: pgmap v164: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:56:59.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:56:58 smithi111 ceph-mon[106899]: pgmap v164: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-23T11:57:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:00 smithi063 ceph-mon[141124]: pgmap v165: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-03-23T11:57:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:00 smithi111 ceph-mon[106899]: pgmap v165: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-03-23T11:57:02.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:01 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:02.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:01 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:03.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:02 smithi063 ceph-mon[141124]: pgmap v166: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:57:03.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:02 smithi111 ceph-mon[106899]: pgmap v166: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:57:04.087 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:03 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:57:04.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:03 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:57:05.089 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:04 smithi111 ceph-mon[106899]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:57:05.089 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:04 smithi111 ceph-mon[106899]: Upgrade: osd.5 is safe to restart 2024-03-23T11:57:05.089 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:04 smithi111 ceph-mon[106899]: pgmap v167: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:57:05.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:04 smithi063 ceph-mon[141124]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-23T11:57:05.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:04 smithi063 ceph-mon[141124]: Upgrade: osd.5 is safe to restart 2024-03-23T11:57:05.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:04 smithi063 ceph-mon[141124]: pgmap v167: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T11:57:06.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:06 smithi111 ceph-mon[106899]: Upgrade: Updating osd.5 2024-03-23T11:57:06.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:06.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T11:57:06.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:06 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:06.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:06 smithi111 ceph-mon[106899]: Deploying daemon osd.5 on smithi111 2024-03-23T11:57:06.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:06 smithi111 ceph-mon[106899]: pgmap v168: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:57:06.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:06 smithi063 ceph-mon[141124]: Upgrade: Updating osd.5 2024-03-23T11:57:06.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:06.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-23T11:57:06.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:06 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:06.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:06 smithi063 ceph-mon[141124]: Deploying daemon osd.5 on smithi111 2024-03-23T11:57:06.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:06 smithi063 ceph-mon[141124]: pgmap v168: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:57:06.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.007+0000 7ffb3fc00700 1 -- 172.21.15.63:0/1054992265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 msgr2=0x7ffb380fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:06.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.007+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/1054992265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb380fce10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffb24004940 tx=0x7ffb24009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:06.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.007+0000 7ffb3fc00700 1 -- 172.21.15.63:0/1054992265 shutdown_connections 2024-03-23T11:57:06.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.007+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/1054992265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb380fce10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:06.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.007+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/1054992265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb38102e00 0x7ffb381031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:06.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.007+0000 7ffb3fc00700 1 -- 172.21.15.63:0/1054992265 >> 172.21.15.63:0/1054992265 conn(0x7ffb380f89b0 msgr2=0x7ffb380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:06.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.008+0000 7ffb3fc00700 1 -- 172.21.15.63:0/1054992265 shutdown_connections 2024-03-23T11:57:06.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.008+0000 7ffb3fc00700 1 -- 172.21.15.63:0/1054992265 wait complete. 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.008+0000 7ffb3fc00700 1 Processor -- start 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb3fc00700 1 -- start start 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb3fc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb38102e00 0x7ffb38113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb3fc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb38113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb3fc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38114600 con 0x7ffb381037a0 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb3fc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3810d9f0 con 0x7ffb38102e00 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb38113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb3d200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb38102e00 0x7ffb38113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:06.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb38113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37234/0 (socket says 172.21.15.63:37234) 2024-03-23T11:57:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.009+0000 7ffb37e00700 1 -- 172.21.15.63:0/2189049541 learned_addr learned my addr 172.21.15.63:0/2189049541 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb3d200700 1 -- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 msgr2=0x7ffb38113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb3d200700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb38113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb3d200700 1 -- 172.21.15.63:0/2189049541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24005040 con 0x7ffb38102e00 2024-03-23T11:57:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb37e00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb38113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:57:06.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb3d200700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb38102e00 0x7ffb38113a70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009fd0 tx=0x7ffb2c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb34c00700 1 -- 172.21.15.63:0/2189049541 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb2c004e90 con 0x7ffb38102e00 2024-03-23T11:57:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb34c00700 1 -- 172.21.15.63:0/2189049541 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb2c00f040 con 0x7ffb38102e00 2024-03-23T11:57:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.010+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3810dc70 con 0x7ffb38102e00 2024-03-23T11:57:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.011+0000 7ffb34c00700 1 -- 172.21.15.63:0/2189049541 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb2c00c9b0 con 0x7ffb38102e00 2024-03-23T11:57:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.011+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3810e230 con 0x7ffb38102e00 2024-03-23T11:57:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.012+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb380fe550 con 0x7ffb38102e00 2024-03-23T11:57:06.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.013+0000 7ffb34c00700 1 -- 172.21.15.63:0/2189049541 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ffb2c007cc0 con 0x7ffb38102e00 2024-03-23T11:57:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.014+0000 7ffb34c00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb2006c7d0 0x7ffb2006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.014+0000 7ffb34c00700 1 -- 172.21.15.63:0/2189049541 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ffb2c08b420 con 0x7ffb38102e00 2024-03-23T11:57:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.014+0000 7ffb37e00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb2006c7d0 0x7ffb2006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:06.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.014+0000 7ffb37e00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb2006c7d0 0x7ffb2006ec80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb3810ebb0 tx=0x7ffb24009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:07.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.020+0000 7ffb34c00700 1 -- 172.21.15.63:0/2189049541 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb2c0591e0 con 0x7ffb38102e00 2024-03-23T11:57:07.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.254+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb38109da0 con 0x7ffb2006c7d0 2024-03-23T11:57:07.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.255+0000 7ffb34c00700 1 -- 172.21.15.63:0/2189049541 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ffb38109da0 con 0x7ffb2006c7d0 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.257+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb2006c7d0 msgr2=0x7ffb2006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.257+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb2006c7d0 0x7ffb2006ec80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb3810ebb0 tx=0x7ffb24009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.257+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb38102e00 msgr2=0x7ffb38113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.257+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb38102e00 0x7ffb38113a70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009fd0 tx=0x7ffb2c00b040 comp rx=0 tx=0).stop 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.258+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 shutdown_connections 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.258+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb2006c7d0 0x7ffb2006ec80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.258+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb38102e00 0x7ffb38113a70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.258+0000 7ffb3fc00700 1 --2- 172.21.15.63:0/2189049541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb381037a0 0x7ffb38113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.258+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 >> 172.21.15.63:0/2189049541 conn(0x7ffb380f89b0 msgr2=0x7ffb380f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.258+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 shutdown_connections 2024-03-23T11:57:07.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.258+0000 7ffb3fc00700 1 -- 172.21.15.63:0/2189049541 wait complete. 2024-03-23T11:57:07.255 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 -- 172.21.15.63:0/4122078094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2301037a0 msgr2=0x7fc2300fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 --2- 172.21.15.63:0/4122078094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2301037a0 0x7fc2300fce10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc21c004940 tx=0x7fc21c009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 -- 172.21.15.63:0/4122078094 shutdown_connections 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 --2- 172.21.15.63:0/4122078094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2301037a0 0x7fc2300fce10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 --2- 172.21.15.63:0/4122078094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc230102e00 0x7fc2301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 -- 172.21.15.63:0/4122078094 >> 172.21.15.63:0/4122078094 conn(0x7fc2300f89b0 msgr2=0x7fc2300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 -- 172.21.15.63:0/4122078094 shutdown_connections 2024-03-23T11:57:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.395+0000 7fc235c00700 1 -- 172.21.15.63:0/4122078094 wait complete. 2024-03-23T11:57:07.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.396+0000 7fc235c00700 1 Processor -- start 2024-03-23T11:57:07.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.396+0000 7fc235c00700 1 -- start start 2024-03-23T11:57:07.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.396+0000 7fc235c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc230102e00 0x7fc230115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:07.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.396+0000 7fc235c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2301037a0 0x7fc230116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:07.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.396+0000 7fc235c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc230116890 con 0x7fc230102e00 2024-03-23T11:57:07.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.396+0000 7fc235c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23010fc60 con 0x7fc2301037a0 2024-03-23T11:57:07.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc230102e00 0x7fc230115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc230102e00 0x7fc230115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37266/0 (socket says 172.21.15.63:37266) 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22ea00700 1 -- 172.21.15.63:0/873018781 learned_addr learned my addr 172.21.15.63:0/873018781 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22e000700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2301037a0 0x7fc230116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22ea00700 1 -- 172.21.15.63:0/873018781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2301037a0 msgr2=0x7fc230116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22ea00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2301037a0 0x7fc230116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22ea00700 1 -- 172.21.15.63:0/873018781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c005040 con 0x7fc230102e00 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc22ea00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc230102e00 0x7fc230115d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc220002a50 tx=0x7fc2200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.397+0000 7fc226a00700 1 -- 172.21.15.63:0/873018781 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc220007cc0 con 0x7fc230102e00 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.398+0000 7fc226a00700 1 -- 172.21.15.63:0/873018781 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc22000f040 con 0x7fc230102e00 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.398+0000 7fc226a00700 1 -- 172.21.15.63:0/873018781 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc220009540 con 0x7fc230102e00 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.398+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23010ff40 con 0x7fc230102e00 2024-03-23T11:57:07.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.398+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc230110500 con 0x7fc230102e00 2024-03-23T11:57:07.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.399+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2300fe550 con 0x7fc230102e00 2024-03-23T11:57:07.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.400+0000 7fc226a00700 1 -- 172.21.15.63:0/873018781 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc2200096a0 con 0x7fc230102e00 2024-03-23T11:57:07.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.401+0000 7fc226a00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21406c7d0 0x7fc21406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:07.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.401+0000 7fc226a00700 1 -- 172.21.15.63:0/873018781 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc22008b7a0 con 0x7fc230102e00 2024-03-23T11:57:07.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.401+0000 7fc22e000700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21406c7d0 0x7fc21406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:07.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.402+0000 7fc22e000700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21406c7d0 0x7fc21406ec80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc2301115a0 tx=0x7fc21c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:07.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.406+0000 7fc226a00700 1 -- 172.21.15.63:0/873018781 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc2200594b0 con 0x7fc230102e00 2024-03-23T11:57:07.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.638+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc23004bf40 con 0x7fc21406c7d0 2024-03-23T11:57:07.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.639+0000 7fc226a00700 1 -- 172.21.15.63:0/873018781 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc23004bf40 con 0x7fc21406c7d0 2024-03-23T11:57:07.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21406c7d0 msgr2=0x7fc21406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21406c7d0 0x7fc21406ec80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc2301115a0 tx=0x7fc21c009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc230102e00 msgr2=0x7fc230115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc230102e00 0x7fc230115d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc220002a50 tx=0x7fc2200046b0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 shutdown_connections 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21406c7d0 0x7fc21406ec80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc230102e00 0x7fc230115d00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 --2- 172.21.15.63:0/873018781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2301037a0 0x7fc230116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 >> 172.21.15.63:0/873018781 conn(0x7fc2300f89b0 msgr2=0x7fc2300f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 shutdown_connections 2024-03-23T11:57:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.642+0000 7fc235c00700 1 -- 172.21.15.63:0/873018781 wait complete. 2024-03-23T11:57:07.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 -- 172.21.15.63:0/1499611511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 msgr2=0x7fc234106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 --2- 172.21.15.63:0/1499611511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc234106f20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc220004970 tx=0x7fc220009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 -- 172.21.15.63:0/1499611511 shutdown_connections 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 --2- 172.21.15.63:0/1499611511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc234106f20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 --2- 172.21.15.63:0/1499611511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340fdeb0 0x7fc2340fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 -- 172.21.15.63:0/1499611511 >> 172.21.15.63:0/1499611511 conn(0x7fc234072bc0 msgr2=0x7fc234072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 -- 172.21.15.63:0/1499611511 shutdown_connections 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.779+0000 7fc239200700 1 -- 172.21.15.63:0/1499611511 wait complete. 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.780+0000 7fc239200700 1 Processor -- start 2024-03-23T11:57:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.780+0000 7fc239200700 1 -- start start 2024-03-23T11:57:07.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.780+0000 7fc239200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340fdeb0 0x7fc234115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:07.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.780+0000 7fc239200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc2341161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:07.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.780+0000 7fc239200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2341167f0 con 0x7fc2340fe850 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.780+0000 7fc239200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23410fc00 con 0x7fc2340fdeb0 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.780+0000 7fc231400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc2341161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc231e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340fdeb0 0x7fc234115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc231400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc2341161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37280/0 (socket says 172.21.15.63:37280) 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc231400700 1 -- 172.21.15.63:0/2487879144 learned_addr learned my addr 172.21.15.63:0/2487879144 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc231400700 1 -- 172.21.15.63:0/2487879144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340fdeb0 msgr2=0x7fc234115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc231400700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340fdeb0 0x7fc234115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc231400700 1 -- 172.21.15.63:0/2487879144 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc220005040 con 0x7fc2340fe850 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc231400700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc2341161a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc220004f50 tx=0x7fc220009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc22a000700 1 -- 172.21.15.63:0/2487879144 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22000e040 con 0x7fc2340fe850 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc22a000700 1 -- 172.21.15.63:0/2487879144 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc220013420 con 0x7fc2340fe850 2024-03-23T11:57:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc22a000700 1 -- 172.21.15.63:0/2487879144 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22001c630 con 0x7fc2340fe850 2024-03-23T11:57:07.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.781+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23410fe80 con 0x7fc2340fe850 2024-03-23T11:57:07.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.782+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2341103c0 con 0x7fc2340fe850 2024-03-23T11:57:07.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.783+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2341069f0 con 0x7fc2340fe850 2024-03-23T11:57:07.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.784+0000 7fc22a000700 1 -- 172.21.15.63:0/2487879144 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc220013590 con 0x7fc2340fe850 2024-03-23T11:57:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.785+0000 7fc22a000700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21806c7d0 0x7fc21806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.785+0000 7fc22a000700 1 -- 172.21.15.63:0/2487879144 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc22008cb80 con 0x7fc2340fe850 2024-03-23T11:57:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.785+0000 7fc231e00700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21806c7d0 0x7fc21806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.786+0000 7fc231e00700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21806c7d0 0x7fc21806ec80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc224007fd0 tx=0x7fc224009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:07.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:07.789+0000 7fc22a000700 1 -- 172.21.15.63:0/2487879144 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc220056e80 con 0x7fc2340fe850 2024-03-23T11:57:07.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.013+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2341112f0 con 0x7fc21806c7d0 2024-03-23T11:57:08.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.023+0000 7fc22a000700 1 -- 172.21.15.63:0/2487879144 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc2341112f0 con 0x7fc21806c7d0 2024-03-23T11:57:08.006 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (6m) 99s ago 19m 14.2M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (3m) 99s ago 19m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (3m) 25s ago 18m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (6m) 99s ago 19m 73.4M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (13m) 99s ago 13m 118M - 17.2.7-726-g9c4e0c34 837124da804f dcb10a562a96 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (13m) 99s ago 13m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (13m) 25s ago 13m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (13m) 25s ago 13m 60.1M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (8m) 99s ago 21m 502M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (7m) 25s ago 18m 451M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (4m) 99s ago 21m 45.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (4m) 25s ago 18m 39.8M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (7m) 99s ago 19m 7532k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:57:08.007 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 25s ago 18m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:57:08.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (3m) 99s ago 17m 87.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:57:08.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (2m) 99s ago 16m 91.1M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:57:08.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (105s) 99s ago 16m 11.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:57:08.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (66s) 25s ago 15m 95.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T11:57:08.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (29s) 25s ago 15m 11.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T11:57:08.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (14m) 25s ago 14m 227M 2642M 17.2.7-726-g9c4e0c34 837124da804f 7e31ed95c234 2024-03-23T11:57:08.008 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (6m) 99s ago 19m 53.4M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.025+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21806c7d0 msgr2=0x7fc21806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.025+0000 7fc239200700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21806c7d0 0x7fc21806ec80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc224007fd0 tx=0x7fc224009040 comp rx=0 tx=0).stop 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.025+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 msgr2=0x7fc2341161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.025+0000 7fc239200700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc2341161a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc220004f50 tx=0x7fc220009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.026+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 shutdown_connections 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.026+0000 7fc239200700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc21806c7d0 0x7fc21806ec80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.026+0000 7fc239200700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340fdeb0 0x7fc234115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.026+0000 7fc239200700 1 --2- 172.21.15.63:0/2487879144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2340fe850 0x7fc2341161a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.026+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 >> 172.21.15.63:0/2487879144 conn(0x7fc234072bc0 msgr2=0x7fc2340fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.026+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 shutdown_connections 2024-03-23T11:57:08.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.026+0000 7fc239200700 1 -- 172.21.15.63:0/2487879144 wait complete. 2024-03-23T11:57:08.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 -- 172.21.15.63:0/3796902913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e381037a0 msgr2=0x7f1e380fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/3796902913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e381037a0 0x7f1e380fce10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e20004940 tx=0x7f1e20009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:08.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 -- 172.21.15.63:0/3796902913 shutdown_connections 2024-03-23T11:57:08.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/3796902913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e381037a0 0x7f1e380fce10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/3796902913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e38102e00 0x7f1e381031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 -- 172.21.15.63:0/3796902913 >> 172.21.15.63:0/3796902913 conn(0x7f1e380f89b0 msgr2=0x7f1e380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:08.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 -- 172.21.15.63:0/3796902913 shutdown_connections 2024-03-23T11:57:08.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.162+0000 7f1e3ec00700 1 -- 172.21.15.63:0/3796902913 wait complete. 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.163+0000 7f1e3ec00700 1 Processor -- start 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.163+0000 7f1e3ec00700 1 -- start start 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e3ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38102e00 0x7f1e38115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e3ec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037a0 0x7f1e381161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e3ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e38116840 con 0x7f1e38102e00 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e3ec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3810fc10 con 0x7f1e381037a0 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38102e00 0x7f1e38115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38102e00 0x7f1e38115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37310/0 (socket says 172.21.15.63:37310) 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e37e00700 1 -- 172.21.15.63:0/1928878249 learned_addr learned my addr 172.21.15.63:0/1928878249 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:08.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e2fe00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037a0 0x7f1e381161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e37e00700 1 -- 172.21.15.63:0/1928878249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037a0 msgr2=0x7f1e381161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e37e00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037a0 0x7f1e381161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e37e00700 1 -- 172.21.15.63:0/1928878249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e20005040 con 0x7f1e38102e00 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.164+0000 7f1e37e00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38102e00 0x7f1e38115cb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800eb40 tx=0x7f1e2800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.165+0000 7f1e34c00700 1 -- 172.21.15.63:0/1928878249 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e28007cc0 con 0x7f1e38102e00 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.165+0000 7f1e34c00700 1 -- 172.21.15.63:0/1928878249 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e2800f040 con 0x7f1e38102e00 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.165+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3810fe90 con 0x7f1e38102e00 2024-03-23T11:57:08.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.165+0000 7f1e34c00700 1 -- 172.21.15.63:0/1928878249 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e28009570 con 0x7f1e38102e00 2024-03-23T11:57:08.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.165+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e38110450 con 0x7f1e38102e00 2024-03-23T11:57:08.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.166+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e380fe550 con 0x7f1e38102e00 2024-03-23T11:57:08.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.167+0000 7f1e34c00700 1 -- 172.21.15.63:0/1928878249 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f1e280096d0 con 0x7f1e38102e00 2024-03-23T11:57:08.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.168+0000 7f1e34c00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f1e1c06c7d0 0x7f1e1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:08.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.168+0000 7f1e34c00700 1 -- 172.21.15.63:0/1928878249 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f1e2808b670 con 0x7f1e38102e00 2024-03-23T11:57:08.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.168+0000 7f1e2fe00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f1e1c06c7d0 0x7f1e1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:08.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.169+0000 7f1e2fe00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f1e1c06c7d0 0x7f1e1c06ec80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e20009fd0 tx=0x7f1e20009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:08.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.172+0000 7f1e34c00700 1 -- 172.21.15.63:0/1928878249 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1e28059430 con 0x7f1e38102e00 2024-03-23T11:57:08.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.473+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1e3804bed0 con 0x7f1e38102e00 2024-03-23T11:57:08.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.474+0000 7f1e34c00700 1 -- 172.21.15.63:0/1928878249 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7f1e28058fc0 con 0x7f1e38102e00 2024-03-23T11:57:08.457 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:57:08.457 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:57:08.457 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 5 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 5, 2024-03-23T11:57:08.458 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 9 2024-03-23T11:57:08.459 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:57:08.459 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f1e1c06c7d0 msgr2=0x7f1e1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f1e1c06c7d0 0x7f1e1c06ec80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e20009fd0 tx=0x7f1e20009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38102e00 msgr2=0x7f1e38115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38102e00 0x7f1e38115cb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800eb40 tx=0x7f1e2800b040 comp rx=0 tx=0).stop 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 shutdown_connections 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f1e1c06c7d0 0x7f1e1c06ec80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38102e00 0x7f1e38115cb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 --2- 172.21.15.63:0/1928878249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037a0 0x7f1e381161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.476+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 >> 172.21.15.63:0/1928878249 conn(0x7f1e380f89b0 msgr2=0x7f1e38070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.477+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 shutdown_connections 2024-03-23T11:57:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.477+0000 7f1e3ec00700 1 -- 172.21.15.63:0/1928878249 wait complete. 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.612+0000 7f2b0fa00700 1 -- 172.21.15.63:0/1198583444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 msgr2=0x7f2b080fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.612+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/1198583444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fe260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2afc004940 tx=0x7f2afc009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.613+0000 7f2b0fa00700 1 -- 172.21.15.63:0/1198583444 shutdown_connections 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.613+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/1198583444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b080fe830 0x7f2b08106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.613+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/1198583444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fe260 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.613+0000 7f2b0fa00700 1 -- 172.21.15.63:0/1198583444 >> 172.21.15.63:0/1198583444 conn(0x7f2b08072bc0 msgr2=0x7f2b08072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.613+0000 7f2b0fa00700 1 -- 172.21.15.63:0/1198583444 shutdown_connections 2024-03-23T11:57:08.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.613+0000 7f2b0fa00700 1 -- 172.21.15.63:0/1198583444 wait complete. 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.613+0000 7f2b0fa00700 1 Processor -- start 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0fa00700 1 -- start start 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0fa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0fa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b080fe830 0x7f2b080fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0fa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b080fc8b0 con 0x7f2b080fde90 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0fa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b080fc9f0 con 0x7f2b080fe830 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37328/0 (socket says 172.21.15.63:37328) 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b0d000700 1 -- 172.21.15.63:0/2345085030 learned_addr learned my addr 172.21.15.63:0/2345085030 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.614+0000 7f2b07e00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b080fe830 0x7f2b080fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b0d000700 1 -- 172.21.15.63:0/2345085030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b080fe830 msgr2=0x7f2b080fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b0d000700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b080fe830 0x7f2b080fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b0d000700 1 -- 172.21.15.63:0/2345085030 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc005040 con 0x7f2b080fde90 2024-03-23T11:57:08.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b0d000700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fbd50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2afc000c00 tx=0x7f2afc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:08.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b04c00700 1 -- 172.21.15.63:0/2345085030 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2afc00e040 con 0x7f2b080fde90 2024-03-23T11:57:08.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b04c00700 1 -- 172.21.15.63:0/2345085030 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2afc013420 con 0x7f2b080fde90 2024-03-23T11:57:08.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b08119df0 con 0x7f2b080fde90 2024-03-23T11:57:08.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b04c00700 1 -- 172.21.15.63:0/2345085030 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2afc01c630 con 0x7f2b080fde90 2024-03-23T11:57:08.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.615+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0811a250 con 0x7f2b080fde90 2024-03-23T11:57:08.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.617+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08065e30 con 0x7f2b080fde90 2024-03-23T11:57:08.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.618+0000 7f2b04c00700 1 -- 172.21.15.63:0/2345085030 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2afc013970 con 0x7f2b080fde90 2024-03-23T11:57:08.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.618+0000 7f2b04c00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2af006c7d0 0x7f2af006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:08.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.618+0000 7f2b04c00700 1 -- 172.21.15.63:0/2345085030 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2afc08cb30 con 0x7f2b080fde90 2024-03-23T11:57:08.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.619+0000 7f2b07e00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2af006c7d0 0x7f2af006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:08.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.619+0000 7f2b07e00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2af006c7d0 0x7f2af006ec80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2af4006fd0 tx=0x7f2af4008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:08.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.623+0000 7f2b04c00700 1 -- 172.21.15.63:0/2345085030 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2afc056ef0 con 0x7f2b080fde90 2024-03-23T11:57:08.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.885+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2b0804bf00 con 0x7f2b080fde90 2024-03-23T11:57:08.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.885+0000 7f2b04c00700 1 -- 172.21.15.63:0/2345085030 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1870 (secure 0 0 0) 0x7f2afc05a510 con 0x7f2b080fde90 2024-03-23T11:57:08.868 INFO:teuthology.orchestra.run.smithi063.stdout:e24 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 24 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:51:47.612215+0000 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:57:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 46 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=14462,1=14478} 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:57:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.63:6826/3230334326,v1:172.21.15.63:6827/3230334326] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{0:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6826/3582848147,v1:172.21.15.111:6827/3582848147] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:14478} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.63:6828/2516647170,v1:172.21.15.63:6829/2516647170] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{1:24299} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/315773837,v1:172.21.15.111:6825/315773837] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:57:08.871 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2af006c7d0 msgr2=0x7f2af006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2af006c7d0 0x7f2af006ec80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2af4006fd0 tx=0x7f2af4008040 comp rx=0 tx=0).stop 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 msgr2=0x7f2b080fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fbd50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2afc000c00 tx=0x7f2afc009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 shutdown_connections 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2af006c7d0 0x7f2af006ec80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b080fde90 0x7f2b080fbd50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 --2- 172.21.15.63:0/2345085030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b080fe830 0x7f2b080fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 >> 172.21.15.63:0/2345085030 conn(0x7f2b08072bc0 msgr2=0x7f2b081037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 shutdown_connections 2024-03-23T11:57:08.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:08.888+0000 7f2b0fa00700 1 -- 172.21.15.63:0/2345085030 wait complete. 2024-03-23T11:57:08.873 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T11:57:08.907 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:08 smithi111 ceph-mon[106899]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:08.907 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:08 smithi111 ceph-mon[106899]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:08.907 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:08 smithi111 ceph-mon[106899]: pgmap v169: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:57:08.907 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:08 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1928878249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:09.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 -- 172.21.15.63:0/558370581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070300 msgr2=0x7f43700706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 --2- 172.21.15.63:0/558370581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070300 0x7f43700706d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4364004970 tx=0x7f4364009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:09.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 -- 172.21.15.63:0/558370581 shutdown_connections 2024-03-23T11:57:09.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 --2- 172.21.15.63:0/558370581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370070c10 0x7f4370109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 --2- 172.21.15.63:0/558370581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070300 0x7f43700706d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 -- 172.21.15.63:0/558370581 >> 172.21.15.63:0/558370581 conn(0x7f43700f8970 msgr2=0x7f43700fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:09.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 -- 172.21.15.63:0/558370581 shutdown_connections 2024-03-23T11:57:09.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.025+0000 7f4378200700 1 -- 172.21.15.63:0/558370581 wait complete. 2024-03-23T11:57:09.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.026+0000 7f4378200700 1 Processor -- start 2024-03-23T11:57:09.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.026+0000 7f4378200700 1 -- start start 2024-03-23T11:57:09.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.026+0000 7f4378200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370070300 0x7f4370115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:09.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.026+0000 7f4378200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070c10 0x7f43701161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:09.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.026+0000 7f4378200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4370116820 con 0x7f4370070c10 2024-03-23T11:57:09.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.026+0000 7f4378200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437010fc30 con 0x7f4370070300 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4374e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070c10 0x7f43701161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4374e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070c10 0x7f43701161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37336/0 (socket says 172.21.15.63:37336) 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4374e00700 1 -- 172.21.15.63:0/1492489200 learned_addr learned my addr 172.21.15.63:0/1492489200 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4375800700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370070300 0x7f4370115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4374e00700 1 -- 172.21.15.63:0/1492489200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370070300 msgr2=0x7f4370115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4374e00700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370070300 0x7f4370115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4374e00700 1 -- 172.21.15.63:0/1492489200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364005040 con 0x7f4370070c10 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.027+0000 7f4374e00700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070c10 0x7f43701161d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f435c00eb40 tx=0x7f435c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.028+0000 7f436d600700 1 -- 172.21.15.63:0/1492489200 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c007cc0 con 0x7f4370070c10 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.028+0000 7f436d600700 1 -- 172.21.15.63:0/1492489200 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f435c00f040 con 0x7f4370070c10 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.028+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437010feb0 con 0x7f4370070c10 2024-03-23T11:57:09.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.028+0000 7f436d600700 1 -- 172.21.15.63:0/1492489200 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c00c990 con 0x7f4370070c10 2024-03-23T11:57:09.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.028+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43701103f0 con 0x7f4370070c10 2024-03-23T11:57:09.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.029+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4370071a40 con 0x7f4370070c10 2024-03-23T11:57:09.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.030+0000 7f436d600700 1 -- 172.21.15.63:0/1492489200 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f435c00cc00 con 0x7f4370070c10 2024-03-23T11:57:09.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.031+0000 7f436d600700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f435806c7d0 0x7f435806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:09.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.031+0000 7f436d600700 1 -- 172.21.15.63:0/1492489200 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f435c08b690 con 0x7f4370070c10 2024-03-23T11:57:09.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.031+0000 7f4375800700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f435806c7d0 0x7f435806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:09.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.032+0000 7f4375800700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f435806c7d0 0x7f435806ec80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4364009fd0 tx=0x7f4364009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:09.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.035+0000 7f436d600700 1 -- 172.21.15.63:0/1492489200 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f435c055940 con 0x7f4370070c10 2024-03-23T11:57:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:08 smithi063 ceph-mon[141124]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:08 smithi063 ceph-mon[141124]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:08 smithi063 ceph-mon[141124]: pgmap v169: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T11:57:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:08 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1928878249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:09.213 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:08 smithi111 systemd[1]: Stopping Ceph osd.5 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:57:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.270+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f437010c040 con 0x7f435806c7d0 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.272+0000 7f436d600700 1 -- 172.21.15.63:0/1492489200 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f437010c040 con 0x7f435806c7d0 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:57:09.255 INFO:teuthology.orchestra.run.smithi063.stdout: "mon" 2024-03-23T11:57:09.256 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:57:09.256 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "11/21 daemons upgraded", 2024-03-23T11:57:09.256 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading osd daemons", 2024-03-23T11:57:09.256 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:57:09.256 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:57:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f435806c7d0 msgr2=0x7f435806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f435806c7d0 0x7f435806ec80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4364009fd0 tx=0x7f4364009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070c10 msgr2=0x7f43701161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070c10 0x7f43701161d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f435c00eb40 tx=0x7f435c00b040 comp rx=0 tx=0).stop 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 shutdown_connections 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f435806c7d0 0x7f435806ec80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370070300 0x7f4370115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 --2- 172.21.15.63:0/1492489200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4370070c10 0x7f43701161d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 >> 172.21.15.63:0/1492489200 conn(0x7f43700f8970 msgr2=0x7f43700f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 shutdown_connections 2024-03-23T11:57:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.274+0000 7f4378200700 1 -- 172.21.15.63:0/1492489200 wait complete. 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 -- 172.21.15.63:0/2253005117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105090 msgr2=0x7f9e08105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 --2- 172.21.15.63:0/2253005117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105090 0x7f9e08105460 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9df8004940 tx=0x7f9df8009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 -- 172.21.15.63:0/2253005117 shutdown_connections 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 --2- 172.21.15.63:0/2253005117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e08105a30 0x7f9e080ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 --2- 172.21.15.63:0/2253005117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105090 0x7f9e08105460 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 -- 172.21.15.63:0/2253005117 >> 172.21.15.63:0/2253005117 conn(0x7f9e080fac00 msgr2=0x7f9e080fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 -- 172.21.15.63:0/2253005117 shutdown_connections 2024-03-23T11:57:09.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.410+0000 7f9e0d600700 1 -- 172.21.15.63:0/2253005117 wait complete. 2024-03-23T11:57:09.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.411+0000 7f9e0d600700 1 Processor -- start 2024-03-23T11:57:09.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.411+0000 7f9e0d600700 1 -- start start 2024-03-23T11:57:09.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e0d600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e08105090 0x7f9e08115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e0d600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105a30 0x7f9e08115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e0d600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08116620 con 0x7f9e08105a30 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e0d600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0810fc00 con 0x7f9e08105090 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e06000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105a30 0x7f9e08115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e06000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105a30 0x7f9e08115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37348/0 (socket says 172.21.15.63:37348) 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e06000700 1 -- 172.21.15.63:0/3874514917 learned_addr learned my addr 172.21.15.63:0/3874514917 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e06a00700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e08105090 0x7f9e08115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e06000700 1 -- 172.21.15.63:0/3874514917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e08105090 msgr2=0x7f9e08115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e06000700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e08105090 0x7f9e08115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.412+0000 7f9e06000700 1 -- 172.21.15.63:0/3874514917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df8005040 con 0x7f9e08105a30 2024-03-23T11:57:09.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.413+0000 7f9e06000700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105a30 0x7f9e08115fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9df4002a50 tx=0x7f9df4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:09.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.413+0000 7f9dfea00700 1 -- 172.21.15.63:0/3874514917 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df4007cc0 con 0x7f9e08105a30 2024-03-23T11:57:09.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.413+0000 7f9dfea00700 1 -- 172.21.15.63:0/3874514917 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9df400f040 con 0x7f9e08105a30 2024-03-23T11:57:09.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.413+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0810fee0 con 0x7f9e08105a30 2024-03-23T11:57:09.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.413+0000 7f9dfea00700 1 -- 172.21.15.63:0/3874514917 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df400d9a0 con 0x7f9e08105a30 2024-03-23T11:57:09.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.413+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e081104a0 con 0x7f9e08105a30 2024-03-23T11:57:09.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.414+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e081023a0 con 0x7f9e08105a30 2024-03-23T11:57:09.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.415+0000 7f9dfea00700 1 -- 172.21.15.63:0/3874514917 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9df400dc10 con 0x7f9e08105a30 2024-03-23T11:57:09.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.416+0000 7f9dfea00700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9dec06c7c0 0x7f9dec06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:09.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.416+0000 7f9dfea00700 1 -- 172.21.15.63:0/3874514917 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f9df408b750 con 0x7f9e08105a30 2024-03-23T11:57:09.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.417+0000 7f9e06a00700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9dec06c7c0 0x7f9dec06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:09.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.417+0000 7f9e06a00700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9dec06c7c0 0x7f9dec06ec70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9df800b020 tx=0x7f9df8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:09.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.421+0000 7f9dfea00700 1 -- 172.21.15.63:0/3874514917 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9df4059500 con 0x7f9e08105a30 2024-03-23T11:57:09.651 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:09 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[66502]: 2024-03-23T11:57:09.228+0000 7f1c3a200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T11:57:09.652 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:09 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[66502]: 2024-03-23T11:57:09.228+0000 7f1c3a200700 -1 osd.5 74 *** Got signal Terminated *** 2024-03-23T11:57:09.652 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:09 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[66502]: 2024-03-23T11:57:09.228+0000 7f1c3a200700 -1 osd.5 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T11:57:09.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.718+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9e081007e0 con 0x7f9e08105a30 2024-03-23T11:57:09.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.719+0000 7f9dfea00700 1 -- 172.21.15.63:0/3874514917 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f9df4059090 con 0x7f9e08105a30 2024-03-23T11:57:09.702 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.721+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9dec06c7c0 msgr2=0x7f9dec06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.721+0000 7f9e0d600700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9dec06c7c0 0x7f9dec06ec70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9df800b020 tx=0x7f9df8009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.721+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105a30 msgr2=0x7f9e08115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.721+0000 7f9e0d600700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105a30 0x7f9e08115fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9df4002a50 tx=0x7f9df4004600 comp rx=0 tx=0).stop 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.722+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 shutdown_connections 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.722+0000 7f9e0d600700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9dec06c7c0 0x7f9dec06ec70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.722+0000 7f9e0d600700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e08105090 0x7f9e08115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.722+0000 7f9e0d600700 1 --2- 172.21.15.63:0/3874514917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e08105a30 0x7f9e08115fd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.722+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 >> 172.21.15.63:0/3874514917 conn(0x7f9e080fac00 msgr2=0x7f9e080fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:09.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.722+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 shutdown_connections 2024-03-23T11:57:09.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:09.722+0000 7f9e0d600700 1 -- 172.21.15.63:0/3874514917 wait complete. 2024-03-23T11:57:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:09 smithi063 ceph-mon[141124]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:09 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2345085030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:57:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:09 smithi063 ceph-mon[141124]: osd.5 marked itself down and dead 2024-03-23T11:57:10.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:09 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3874514917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:57:10.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:09 smithi111 ceph-mon[106899]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:10.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:09 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2345085030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:57:10.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:09 smithi111 ceph-mon[106899]: osd.5 marked itself down and dead 2024-03-23T11:57:10.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:09 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3874514917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:57:10.796 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:10 smithi111 bash[132709]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5 2024-03-23T11:57:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:10 smithi063 ceph-mon[141124]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:10 smithi063 ceph-mon[141124]: pgmap v170: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:57:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:10 smithi063 ceph-mon[141124]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:57:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:10 smithi063 ceph-mon[141124]: osdmap e75: 6 total, 5 up, 6 in 2024-03-23T11:57:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:11.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:10 smithi111 ceph-mon[106899]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:10 smithi111 ceph-mon[106899]: pgmap v170: 65 pgs: 65 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T11:57:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:10 smithi111 ceph-mon[106899]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-23T11:57:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:10 smithi111 ceph-mon[106899]: osdmap e75: 6 total, 5 up, 6 in 2024-03-23T11:57:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:12.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:11 smithi063 ceph-mon[141124]: osdmap e76: 6 total, 5 up, 6 in 2024-03-23T11:57:12.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:11 smithi111 ceph-mon[106899]: osdmap e76: 6 total, 5 up, 6 in 2024-03-23T11:57:12.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:12 smithi111 ceph-mon[106899]: pgmap v173: 65 pgs: 7 peering, 9 stale+active+clean, 49 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s 2024-03-23T11:57:12.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:12 smithi111 ceph-mon[106899]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-23T11:57:12.887 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:12 smithi111 systemd[1]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.5.service: Succeeded. 2024-03-23T11:57:12.887 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:12 smithi111 systemd[1]: Stopped Ceph osd.5 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:57:13.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:12 smithi063 ceph-mon[141124]: pgmap v173: 65 pgs: 7 peering, 9 stale+active+clean, 49 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s 2024-03-23T11:57:13.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:12 smithi063 ceph-mon[141124]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-23T11:57:13.151 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:12 smithi111 systemd[1]: Starting Ceph osd.5 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T11:57:14.637 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:14 smithi111 bash[132980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-23T11:57:14.637 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:14 smithi111 bash[132980]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-23T11:57:14.637 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:14 smithi111 bash[132980]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-23T11:57:14.637 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:14 smithi111 bash[132980]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-23T11:57:14.637 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:14 smithi111 bash[132980]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-23T11:57:14.637 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:14 smithi111 bash[132980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-23T11:57:14.637 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:14 smithi111 bash[132980]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-23T11:57:14.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:14 smithi111 ceph-mon[106899]: pgmap v174: 65 pgs: 7 peering, 9 stale+active+clean, 49 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-23T11:57:15.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:14 smithi063 ceph-mon[141124]: pgmap v174: 65 pgs: 7 peering, 9 stale+active+clean, 49 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-23T11:57:15.901 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:15 smithi111 ceph-osd[133266]: -- 172.21.15.111:0/3614575320 <== mon.1 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5649c33331e0 con 0x5649c3302c00 2024-03-23T11:57:15.901 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:15 smithi111 bash[133229]: b1ef416c095d04de319ed877fdca5cea43e26d24fd58978eeb8e243916c8d55c 2024-03-23T11:57:15.901 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:15 smithi111 systemd[1]: Started Ceph osd.5 for 6880ffb6-e909-11ee-95cd-87774f69a715. 2024-03-23T11:57:16.401 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:16 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T11:57:16.122+0000 7f3bfa8ee640 -1 Falling back to public interface 2024-03-23T11:57:17.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:16 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:17.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:16 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:16 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:17.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:16 smithi063 ceph-mon[141124]: pgmap v175: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:17.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:16 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:17.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:16 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:17.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:16 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:17.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:16 smithi111 ceph-mon[106899]: pgmap v175: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:17.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:17 smithi111 ceph-mon[106899]: Health check failed: Degraded data redundancy: 51/330 objects degraded (15.455%), 14 pgs degraded (PG_DEGRADED) 2024-03-23T11:57:17.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:17 smithi111 ceph-mon[106899]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-03-23T11:57:18.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:17 smithi063 ceph-mon[141124]: Health check failed: Degraded data redundancy: 51/330 objects degraded (15.455%), 14 pgs degraded (PG_DEGRADED) 2024-03-23T11:57:18.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:17 smithi063 ceph-mon[141124]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-03-23T11:57:19.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:18 smithi063 ceph-mon[141124]: pgmap v176: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:19.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:18 smithi111 ceph-mon[106899]: pgmap v176: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:20.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:20.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:20.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:20 smithi063 ceph-mon[141124]: pgmap v177: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:20.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:20.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:20.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:20 smithi111 ceph-mon[106899]: pgmap v177: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:21.151 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:20 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T11:57:20.675+0000 7f3bfa8ee640 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-23T11:57:21.435 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T11:57:21.190+0000 7f3bfa8ee640 -1 osd.5 74 log_to_monitors true 2024-03-23T11:57:21.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:21 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:21.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:21 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:21.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:21 smithi063 ceph-mon[141124]: from='osd.5 [v2:172.21.15.111:6816/983556337,v1:172.21.15.111:6817/983556337]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:57:21.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:21 smithi063 ceph-mon[141124]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:57:21.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:21 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:21.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:21 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:21.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:21 smithi111 ceph-mon[106899]: from='osd.5 [v2:172.21.15.111:6816/983556337,v1:172.21.15.111:6817/983556337]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:57:21.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:21 smithi111 ceph-mon[106899]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-23T11:57:21.902 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 11:57:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T11:57:21.507+0000 7f3be4c00700 -1 osd.5 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:57:22.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:22 smithi111 ceph-mon[106899]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-23T11:57:22.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:22 smithi111 ceph-mon[106899]: osdmap e77: 6 total, 5 up, 6 in 2024-03-23T11:57:22.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:22 smithi111 ceph-mon[106899]: from='osd.5 [v2:172.21.15.111:6816/983556337,v1:172.21.15.111:6817/983556337]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:57:22.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:22 smithi111 ceph-mon[106899]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:57:22.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:22 smithi111 ceph-mon[106899]: pgmap v179: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:22.756 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:22 smithi063 ceph-mon[141124]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-23T11:57:22.756 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:22 smithi063 ceph-mon[141124]: osdmap e77: 6 total, 5 up, 6 in 2024-03-23T11:57:22.756 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:22 smithi063 ceph-mon[141124]: from='osd.5 [v2:172.21.15.111:6816/983556337,v1:172.21.15.111:6817/983556337]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:57:22.756 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:22 smithi063 ceph-mon[141124]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-23T11:57:22.757 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:22 smithi063 ceph-mon[141124]: pgmap v179: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 51/330 objects degraded (15.455%) 2024-03-23T11:57:23.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:23 smithi063 ceph-mon[141124]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:57:23.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:23 smithi063 ceph-mon[141124]: osd.5 [v2:172.21.15.111:6816/983556337,v1:172.21.15.111:6817/983556337] boot 2024-03-23T11:57:23.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:23 smithi063 ceph-mon[141124]: osdmap e78: 6 total, 6 up, 6 in 2024-03-23T11:57:23.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:23 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:57:23.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:23 smithi063 ceph-mon[141124]: osdmap e79: 6 total, 6 up, 6 in 2024-03-23T11:57:23.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:23 smithi111 ceph-mon[106899]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-23T11:57:23.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:23 smithi111 ceph-mon[106899]: osd.5 [v2:172.21.15.111:6816/983556337,v1:172.21.15.111:6817/983556337] boot 2024-03-23T11:57:23.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:23 smithi111 ceph-mon[106899]: osdmap e78: 6 total, 6 up, 6 in 2024-03-23T11:57:23.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:23 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-23T11:57:23.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:23 smithi111 ceph-mon[106899]: osdmap e79: 6 total, 6 up, 6 in 2024-03-23T11:57:24.801 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:24 smithi111 ceph-mon[106899]: pgmap v182: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 51/330 objects degraded (15.455%) 2024-03-23T11:57:24.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:24 smithi063 ceph-mon[141124]: pgmap v182: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 51/330 objects degraded (15.455%) 2024-03-23T11:57:26.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-23T11:57:26.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.133 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:26.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:26.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-23T11:57:26.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-23T11:57:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-23T11:57:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-23T11:57:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-23T11:57:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-23T11:57:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-03-23T11:57:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:26.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:27.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all osd 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: Upgrade: Setting require_osd_release to 18 reef 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: pgmap v183: 65 pgs: 5 active+undersized+degraded, 8 active+undersized, 52 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 2 op/s; 15/330 objects degraded (4.545%) 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: osdmap e80: 6 total, 6 up, 6 in 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: osdmap e81: 6 total, 6 up, 6 in 2024-03-23T11:57:27.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:26 smithi063 ceph-mon[141124]: Health check update: Degraded data redundancy: 15/330 objects degraded (4.545%), 5 pgs degraded (PG_DEGRADED) 2024-03-23T11:57:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all osd 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: Upgrade: Setting require_osd_release to 18 reef 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: pgmap v183: 65 pgs: 5 active+undersized+degraded, 8 active+undersized, 52 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 2 op/s; 15/330 objects degraded (4.545%) 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: osdmap e80: 6 total, 6 up, 6 in 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: osdmap e81: 6 total, 6 up, 6 in 2024-03-23T11:57:27.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:26 smithi111 ceph-mon[106899]: Health check update: Degraded data redundancy: 15/330 objects degraded (4.545%), 5 pgs degraded (PG_DEGRADED) 2024-03-23T11:57:27.860 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T11:57:27.440+0000 7fc325a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.121 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: osdmap e82: 6 total, 6 up, 6 in 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.111:6824/4216607936,v1:172.21.15.111:6826/4216607936] up:boot 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] up:boot 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:57:28.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:27 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:57:28.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-03-23T11:57:28.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-23T11:57:28.152 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: osdmap e82: 6 total, 6 up, 6 in 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.111:6824/4216607936,v1:172.21.15.111:6826/4216607936] up:boot 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] up:boot 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:57:28.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:27 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: pgmap v187: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/330 objects degraded (4.545%), 5 pgs degraded) 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6826/856600159,v1:172.21.15.63:6827/856600159] up:boot 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6828/754248720,v1:172.21.15.63:6829/754248720] up:boot 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:57:29.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:28 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: pgmap v187: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/330 objects degraded (4.545%), 5 pgs degraded) 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6826/856600159,v1:172.21.15.63:6827/856600159] up:boot 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6828/754248720,v1:172.21.15.63:6829/754248720] up:boot 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:57:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:28 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:57:30.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:30 smithi063 ceph-mon[141124]: Upgrade: Updating mds.cephfs.smithi063.brtzgy 2024-03-23T11:57:30.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:30.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:57:30.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:30.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:30 smithi063 ceph-mon[141124]: Deploying daemon mds.cephfs.smithi063.brtzgy on smithi063 2024-03-23T11:57:30.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:30 smithi063 ceph-mon[141124]: pgmap v188: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 170 B/s rd, 2 op/s 2024-03-23T11:57:30.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:30 smithi111 ceph-mon[106899]: Upgrade: Updating mds.cephfs.smithi063.brtzgy 2024-03-23T11:57:30.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:30.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.brtzgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:57:30.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:30.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:30 smithi111 ceph-mon[106899]: Deploying daemon mds.cephfs.smithi063.brtzgy on smithi063 2024-03-23T11:57:30.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:30 smithi111 ceph-mon[106899]: pgmap v188: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 170 B/s rd, 2 op/s 2024-03-23T11:57:32.757 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:32 smithi063 ceph-mon[141124]: osdmap e83: 6 total, 6 up, 6 in 2024-03-23T11:57:32.757 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:32 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T11:57:32.757 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:32 smithi063 ceph-mon[141124]: pgmap v190: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-23T11:57:32.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:32 smithi111 ceph-mon[106899]: osdmap e83: 6 total, 6 up, 6 in 2024-03-23T11:57:32.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:32 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T11:57:32.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:32 smithi111 ceph-mon[106899]: pgmap v190: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-23T11:57:33.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:57:33.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:57:34.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:34 smithi063 ceph-mon[141124]: pgmap v191: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-23T11:57:34.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:34 smithi111 ceph-mon[106899]: pgmap v191: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-23T11:57:37.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:36 smithi063 ceph-mon[141124]: pgmap v192: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 492 B/s rd, 1 op/s 2024-03-23T11:57:37.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:36 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:37.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:36 smithi111 ceph-mon[106899]: pgmap v192: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 492 B/s rd, 1 op/s 2024-03-23T11:57:37.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:36 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:38.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:38.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:38.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:38.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:38 smithi063 ceph-mon[141124]: pgmap v193: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2024-03-23T11:57:38.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:38 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:boot 2024-03-23T11:57:38.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:38 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:57:38.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:38 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:57:38.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:38 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:38.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:38 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:38.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:38 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:38.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:38 smithi111 ceph-mon[106899]: pgmap v193: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2024-03-23T11:57:38.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:38 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:boot 2024-03-23T11:57:38.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:38 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:57:38.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:38 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T11:57:39.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 -- 172.21.15.63:0/1977649850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 msgr2=0x7f0410070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 --2- 172.21.15.63:0/1977649850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f0410070360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0400004970 tx=0x7f0400009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 -- 172.21.15.63:0/1977649850 shutdown_connections 2024-03-23T11:57:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 --2- 172.21.15.63:0/1977649850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04100708a0 0x7f0410070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 --2- 172.21.15.63:0/1977649850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f0410070360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 -- 172.21.15.63:0/1977649850 >> 172.21.15.63:0/1977649850 conn(0x7f04100f89d0 msgr2=0x7f04100fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 -- 172.21.15.63:0/1977649850 shutdown_connections 2024-03-23T11:57:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.875+0000 7f0416a00700 1 -- 172.21.15.63:0/1977649850 wait complete. 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.876+0000 7f0416a00700 1 Processor -- start 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.876+0000 7f0416a00700 1 -- start start 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f0416a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04100708a0 0x7f041011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f0416a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f041011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f0416a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041011ac40 con 0x7f0410071f90 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f0416a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0410114010 con 0x7f04100708a0 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f040f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f041011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f040f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f041011a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59602/0 (socket says 172.21.15.63:59602) 2024-03-23T11:57:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f040fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04100708a0 0x7f041011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.877+0000 7f040f400700 1 -- 172.21.15.63:0/1116121615 learned_addr learned my addr 172.21.15.63:0/1116121615 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f040f400700 1 -- 172.21.15.63:0/1116121615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04100708a0 msgr2=0x7f041011a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f040f400700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04100708a0 0x7f041011a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f040f400700 1 -- 172.21.15.63:0/1116121615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400005040 con 0x7f0410071f90 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f040fe00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04100708a0 0x7f041011a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f040f400700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f041011a5f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f03fc002a80 tx=0x7f03fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f0407e00700 1 -- 172.21.15.63:0/1116121615 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03fc007cc0 con 0x7f0410071f90 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f0407e00700 1 -- 172.21.15.63:0/1116121615 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03fc00f040 con 0x7f0410071f90 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f0407e00700 1 -- 172.21.15.63:0/1116121615 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03fc00da60 con 0x7f0410071f90 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04101142f0 con 0x7f0410071f90 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.878+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0410114830 con 0x7f0410071f90 2024-03-23T11:57:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.879+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0410071aa0 con 0x7f0410071f90 2024-03-23T11:57:39.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.881+0000 7f0407e00700 1 -- 172.21.15.63:0/1116121615 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f03fc00dbc0 con 0x7f0410071f90 2024-03-23T11:57:39.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.882+0000 7f0407e00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f03f406c7d0 0x7f03f406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:39.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.882+0000 7f0407e00700 1 -- 172.21.15.63:0/1116121615 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f03fc08b630 con 0x7f0410071f90 2024-03-23T11:57:39.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.882+0000 7f040fe00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f03f406c7d0 0x7f03f406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:39.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.883+0000 7f040fe00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f03f406c7d0 0x7f03f406ec80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f040000b370 tx=0x7f0400009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:39.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:39.888+0000 7f0407e00700 1 -- 172.21.15.63:0/1116121615 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f03fc059340 con 0x7f0410071f90 2024-03-23T11:57:40.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.128+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0410110470 con 0x7f03f406c7d0 2024-03-23T11:57:40.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.130+0000 7f0407e00700 1 -- 172.21.15.63:0/1116121615 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0410110470 con 0x7f03f406c7d0 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f03f406c7d0 msgr2=0x7f03f406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f03f406c7d0 0x7f03f406ec80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f040000b370 tx=0x7f0400009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 msgr2=0x7f041011a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f041011a5f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f03fc002a80 tx=0x7f03fc0046b0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 shutdown_connections 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f03f406c7d0 0x7f03f406ec80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04100708a0 0x7f041011a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 --2- 172.21.15.63:0/1116121615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0410071f90 0x7f041011a5f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 >> 172.21.15.63:0/1116121615 conn(0x7f04100f89d0 msgr2=0x7f04100f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:40.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 shutdown_connections 2024-03-23T11:57:40.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.133+0000 7f0416a00700 1 -- 172.21.15.63:0/1116121615 wait complete. 2024-03-23T11:57:40.131 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:57:40.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.277+0000 7f203da00700 1 -- 172.21.15.63:0/1192169953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2038102e00 msgr2=0x7f20381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.277+0000 7f203da00700 1 --2- 172.21.15.63:0/1192169953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2038102e00 0x7f20381031d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2028004940 tx=0x7f2028009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:40.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.277+0000 7f203da00700 1 -- 172.21.15.63:0/1192169953 shutdown_connections 2024-03-23T11:57:40.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.277+0000 7f203da00700 1 --2- 172.21.15.63:0/1192169953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20381037a0 0x7f20380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.277+0000 7f203da00700 1 --2- 172.21.15.63:0/1192169953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2038102e00 0x7f20381031d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.277+0000 7f203da00700 1 -- 172.21.15.63:0/1192169953 >> 172.21.15.63:0/1192169953 conn(0x7f20380f89b0 msgr2=0x7f20380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:40.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.277+0000 7f203da00700 1 -- 172.21.15.63:0/1192169953 shutdown_connections 2024-03-23T11:57:40.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.278+0000 7f203da00700 1 -- 172.21.15.63:0/1192169953 wait complete. 2024-03-23T11:57:40.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.278+0000 7f203da00700 1 Processor -- start 2024-03-23T11:57:40.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.278+0000 7f203da00700 1 -- start start 2024-03-23T11:57:40.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f203da00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2038102e00 0x7f2038115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f203da00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20381037a0 0x7f2038116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f203da00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2038116880 con 0x7f20381037a0 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f203da00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203810fc50 con 0x7f2038102e00 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f2036000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20381037a0 0x7f2038116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f2036000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20381037a0 0x7f2038116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59622/0 (socket says 172.21.15.63:59622) 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f2036000700 1 -- 172.21.15.63:0/4014351556 learned_addr learned my addr 172.21.15.63:0/4014351556 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.279+0000 7f2036a00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2038102e00 0x7f2038115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f2036000700 1 -- 172.21.15.63:0/4014351556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2038102e00 msgr2=0x7f2038115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f2036000700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2038102e00 0x7f2038115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f2036000700 1 -- 172.21.15.63:0/4014351556 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2028005040 con 0x7f20381037a0 2024-03-23T11:57:40.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f2036000700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20381037a0 0x7f2038116230 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2024002a50 tx=0x7f2024004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:40.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f202ea00700 1 -- 172.21.15.63:0/4014351556 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2024007cc0 con 0x7f20381037a0 2024-03-23T11:57:40.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f202ea00700 1 -- 172.21.15.63:0/4014351556 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f202400f040 con 0x7f20381037a0 2024-03-23T11:57:40.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203810ff30 con 0x7f20381037a0 2024-03-23T11:57:40.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f202ea00700 1 -- 172.21.15.63:0/4014351556 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f202400d920 con 0x7f20381037a0 2024-03-23T11:57:40.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.280+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20381104f0 con 0x7f20381037a0 2024-03-23T11:57:40.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.281+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20380fe550 con 0x7f20381037a0 2024-03-23T11:57:40.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.282+0000 7f202ea00700 1 -- 172.21.15.63:0/4014351556 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2024009680 con 0x7f20381037a0 2024-03-23T11:57:40.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.283+0000 7f202ea00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f201c06c7d0 0x7f201c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:40.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.283+0000 7f202ea00700 1 -- 172.21.15.63:0/4014351556 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f202408b750 con 0x7f20381037a0 2024-03-23T11:57:40.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.283+0000 7f2036a00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f201c06c7d0 0x7f201c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:40.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.284+0000 7f2036a00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f201c06c7d0 0x7f201c06ec80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2028009fd0 tx=0x7f2028009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:40.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.288+0000 7f202ea00700 1 -- 172.21.15.63:0/4014351556 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f20240593e0 con 0x7f20381037a0 2024-03-23T11:57:40.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.523+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2038109e30 con 0x7f201c06c7d0 2024-03-23T11:57:40.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.525+0000 7f202ea00700 1 -- 172.21.15.63:0/4014351556 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2038109e30 con 0x7f201c06c7d0 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f201c06c7d0 msgr2=0x7f201c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f201c06c7d0 0x7f201c06ec80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2028009fd0 tx=0x7f2028009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20381037a0 msgr2=0x7f2038116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20381037a0 0x7f2038116230 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2024002a50 tx=0x7f2024004600 comp rx=0 tx=0).stop 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 shutdown_connections 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f201c06c7d0 0x7f201c06ec80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2038102e00 0x7f2038115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 --2- 172.21.15.63:0/4014351556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20381037a0 0x7f2038116230 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.527+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 >> 172.21.15.63:0/4014351556 conn(0x7f20380f89b0 msgr2=0x7f20380706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.528+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 shutdown_connections 2024-03-23T11:57:40.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.528+0000 7f203da00700 1 -- 172.21.15.63:0/4014351556 wait complete. 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.666+0000 7fad3be00700 1 -- 172.21.15.63:0/3346595331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 msgr2=0x7fad340fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.666+0000 7fad3be00700 1 --2- 172.21.15.63:0/3346595331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad340fce10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fad20004940 tx=0x7fad20009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.667+0000 7fad3be00700 1 -- 172.21.15.63:0/3346595331 shutdown_connections 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.667+0000 7fad3be00700 1 --2- 172.21.15.63:0/3346595331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad340fce10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.667+0000 7fad3be00700 1 --2- 172.21.15.63:0/3346595331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad34102e00 0x7fad341031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.667+0000 7fad3be00700 1 -- 172.21.15.63:0/3346595331 >> 172.21.15.63:0/3346595331 conn(0x7fad340f89b0 msgr2=0x7fad340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.667+0000 7fad3be00700 1 -- 172.21.15.63:0/3346595331 shutdown_connections 2024-03-23T11:57:40.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.667+0000 7fad3be00700 1 -- 172.21.15.63:0/3346595331 wait complete. 2024-03-23T11:57:40.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.668+0000 7fad3be00700 1 Processor -- start 2024-03-23T11:57:40.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.668+0000 7fad3be00700 1 -- start start 2024-03-23T11:57:40.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.668+0000 7fad3be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad34102e00 0x7fad34113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.668+0000 7fad3be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad34114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.668+0000 7fad3be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad34114690 con 0x7fad341037a0 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.668+0000 7fad3be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3410da80 con 0x7fad34102e00 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.668+0000 7fad38a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad34114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad38a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad34114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59644/0 (socket says 172.21.15.63:59644) 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad38a00700 1 -- 172.21.15.63:0/4233111476 learned_addr learned my addr 172.21.15.63:0/4233111476 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad39400700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad34102e00 0x7fad34113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad38a00700 1 -- 172.21.15.63:0/4233111476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad34102e00 msgr2=0x7fad34113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad38a00700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad34102e00 0x7fad34113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad38a00700 1 -- 172.21.15.63:0/4233111476 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad20005040 con 0x7fad341037a0 2024-03-23T11:57:40.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad38a00700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad34114040 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fad20009390 tx=0x7fad20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:40.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.669+0000 7fad31600700 1 -- 172.21.15.63:0/4233111476 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2000e040 con 0x7fad341037a0 2024-03-23T11:57:40.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.670+0000 7fad31600700 1 -- 172.21.15.63:0/4233111476 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad20013420 con 0x7fad341037a0 2024-03-23T11:57:40.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.670+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3410dd00 con 0x7fad341037a0 2024-03-23T11:57:40.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.670+0000 7fad31600700 1 -- 172.21.15.63:0/4233111476 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2001c630 con 0x7fad341037a0 2024-03-23T11:57:40.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.670+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3410e2c0 con 0x7fad341037a0 2024-03-23T11:57:40.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.671+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad340fe550 con 0x7fad341037a0 2024-03-23T11:57:40.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.672+0000 7fad31600700 1 -- 172.21.15.63:0/4233111476 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fad20013a00 con 0x7fad341037a0 2024-03-23T11:57:40.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.673+0000 7fad31600700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:40.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.673+0000 7fad31600700 1 -- 172.21.15.63:0/4233111476 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fad2008cb20 con 0x7fad341037a0 2024-03-23T11:57:40.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.674+0000 7fad39400700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:40.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.674+0000 7fad39400700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad28004f40 tx=0x7fad2800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:40.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.679+0000 7fad31600700 1 -- 172.21.15.63:0/4233111476 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fad2005a830 con 0x7fad341037a0 2024-03-23T11:57:40.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.916+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fad34109da0 con 0x7fad1c06c7d0 2024-03-23T11:57:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.926+0000 7fad31600700 1 -- 172.21.15.63:0/4233111476 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fad34109da0 con 0x7fad1c06c7d0 2024-03-23T11:57:40.909 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:57:40.909 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (6m) 2m ago 20m 14.2M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:57:40.909 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (4m) 2m ago 20m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:57:40.909 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (4m) 21s ago 19m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:57:40.909 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (6m) 2m ago 19m 73.4M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 starting - - - - 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (13m) 2m ago 13m 118M - 17.2.7-726-g9c4e0c34 837124da804f cac567648a8d 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (13m) 21s ago 13m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (13m) 21s ago 13m 60.1M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (9m) 2m ago 21m 502M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (8m) 21s ago 19m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (5m) 2m ago 21m 45.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (4m) 21s ago 18m 40.5M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (7m) 2m ago 20m 7532k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 21s ago 19m 7654k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (3m) 2m ago 18m 87.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (2m) 2m ago 17m 91.1M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (2m) 2m ago 16m 11.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (99s) 21s ago 16m 96.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T11:57:40.910 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (62s) 21s ago 15m 90.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T11:57:40.911 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (25s) 21s ago 15m 10.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T11:57:40.911 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (7m) 2m ago 19m 53.4M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.928+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 msgr2=0x7fad1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.928+0000 7fad3be00700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad28004f40 tx=0x7fad2800b040 comp rx=0 tx=0).stop 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.928+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 msgr2=0x7fad34114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.928+0000 7fad3be00700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad34114040 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fad20009390 tx=0x7fad20009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.929+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 shutdown_connections 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.929+0000 7fad3be00700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.929+0000 7fad3be00700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad34102e00 0x7fad34113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.929+0000 7fad3be00700 1 --2- 172.21.15.63:0/4233111476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad341037a0 0x7fad34114040 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.929+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 >> 172.21.15.63:0/4233111476 conn(0x7fad340f89b0 msgr2=0x7fad340f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.929+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 shutdown_connections 2024-03-23T11:57:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:40.929+0000 7fad3be00700 1 -- 172.21.15.63:0/4233111476 wait complete. 2024-03-23T11:57:41.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 -- 172.21.15.63:0/1284490561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 msgr2=0x7f71d80fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 --2- 172.21.15.63:0/1284490561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d80fe1f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f71c8004940 tx=0x7f71c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:41.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 -- 172.21.15.63:0/1284490561 shutdown_connections 2024-03-23T11:57:41.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 --2- 172.21.15.63:0/1284490561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71d80fe7c0 0x7f71d8106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 --2- 172.21.15.63:0/1284490561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d80fe1f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 -- 172.21.15.63:0/1284490561 >> 172.21.15.63:0/1284490561 conn(0x7f71d80f7870 msgr2=0x7f71d80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 -- 172.21.15.63:0/1284490561 shutdown_connections 2024-03-23T11:57:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.067+0000 7f71de600700 1 -- 172.21.15.63:0/1284490561 wait complete. 2024-03-23T11:57:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.068+0000 7f71de600700 1 Processor -- start 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.068+0000 7f71de600700 1 -- start start 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.068+0000 7f71de600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d81159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.068+0000 7f71de600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71d80fe7c0 0x7f71d8115f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.068+0000 7f71de600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d8116580 con 0x7f71d80fde20 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.068+0000 7f71de600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d810fba0 con 0x7f71d80fe7c0 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d81159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d81159f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59674/0 (socket says 172.21.15.63:59674) 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d7400700 1 -- 172.21.15.63:0/2005895798 learned_addr learned my addr 172.21.15.63:0/2005895798 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d6a00700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71d80fe7c0 0x7f71d8115f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d7400700 1 -- 172.21.15.63:0/2005895798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71d80fe7c0 msgr2=0x7f71d8115f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d7400700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71d80fe7c0 0x7f71d8115f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d7400700 1 -- 172.21.15.63:0/2005895798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8005040 con 0x7f71d80fde20 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.069+0000 7f71d7400700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d81159f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.070+0000 7f71cf400700 1 -- 172.21.15.63:0/2005895798 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71c8021070 con 0x7f71d80fde20 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.070+0000 7f71cf400700 1 -- 172.21.15.63:0/2005895798 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71c8013e00 con 0x7f71d80fde20 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.070+0000 7f71cf400700 1 -- 172.21.15.63:0/2005895798 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71c800e070 con 0x7f71d80fde20 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.070+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d810fe20 con 0x7f71d80fde20 2024-03-23T11:57:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.070+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d8110360 con 0x7f71d80fde20 2024-03-23T11:57:41.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.071+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d8066b50 con 0x7f71d80fde20 2024-03-23T11:57:41.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.072+0000 7f71cf400700 1 -- 172.21.15.63:0/2005895798 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f71c8013950 con 0x7f71d80fde20 2024-03-23T11:57:41.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.073+0000 7f71cf400700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f71bc06c7c0 0x7f71bc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.073+0000 7f71cf400700 1 -- 172.21.15.63:0/2005895798 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f71c808c9a0 con 0x7f71d80fde20 2024-03-23T11:57:41.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.073+0000 7f71d6a00700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f71bc06c7c0 0x7f71bc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.074+0000 7f71d6a00700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f71bc06c7c0 0x7f71bc06ec70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f71d81112d0 tx=0x7f71c400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:41.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.078+0000 7f71cf400700 1 -- 172.21.15.63:0/2005895798 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f71c8056ce0 con 0x7f71d80fde20 2024-03-23T11:57:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:40 smithi063 ceph-mon[141124]: pgmap v194: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-23T11:57:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:40 smithi111 ceph-mon[106899]: pgmap v194: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-23T11:57:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:41.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.410+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f71d8111080 con 0x7f71d80fde20 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.411+0000 7f71cf400700 1 -- 172.21.15.63:0/2005895798 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f71c805a300 con 0x7f71d80fde20 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:41.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 3, 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 1 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 3, 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 11 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:57:41.395 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:57:41.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f71bc06c7c0 msgr2=0x7f71bc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f71bc06c7c0 0x7f71bc06ec70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f71d81112d0 tx=0x7f71c400b040 comp rx=0 tx=0).stop 2024-03-23T11:57:41.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 msgr2=0x7f71d81159f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d81159f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c8009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:41.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 shutdown_connections 2024-03-23T11:57:41.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f71bc06c7c0 0x7f71bc06ec70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71d80fde20 0x7f71d81159f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 --2- 172.21.15.63:0/2005895798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71d80fe7c0 0x7f71d8115f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 >> 172.21.15.63:0/2005895798 conn(0x7f71d80f7870 msgr2=0x7f71d80fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:41.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 shutdown_connections 2024-03-23T11:57:41.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.413+0000 7f71de600700 1 -- 172.21.15.63:0/2005895798 wait complete. 2024-03-23T11:57:41.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 -- 172.21.15.63:0/2742743697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 msgr2=0x7fb294070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 --2- 172.21.15.63:0/2742743697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb294070d10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb280004970 tx=0x7fb280009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:41.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 -- 172.21.15.63:0/2742743697 shutdown_connections 2024-03-23T11:57:41.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 --2- 172.21.15.63:0/2742743697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb294070d10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 --2- 172.21.15.63:0/2742743697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb294071f90 0x7fb294070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 -- 172.21.15.63:0/2742743697 >> 172.21.15.63:0/2742743697 conn(0x7fb2940f89b0 msgr2=0x7fb2940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:41.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 -- 172.21.15.63:0/2742743697 shutdown_connections 2024-03-23T11:57:41.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.550+0000 7fb29a600700 1 -- 172.21.15.63:0/2742743697 wait complete. 2024-03-23T11:57:41.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.551+0000 7fb29a600700 1 Processor -- start 2024-03-23T11:57:41.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.551+0000 7fb29a600700 1 -- start start 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.551+0000 7fb29a600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb29411a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb29a600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb294071f90 0x7fb29411a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb29a600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29411ac40 con 0x7fb2940708a0 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb29a600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294114010 con 0x7fb294071f90 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb293400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb29411a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb293400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb29411a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37086/0 (socket says 172.21.15.63:37086) 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb293400700 1 -- 172.21.15.63:0/1151055015 learned_addr learned my addr 172.21.15.63:0/1151055015 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb292a00700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb294071f90 0x7fb29411a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb293400700 1 -- 172.21.15.63:0/1151055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb294071f90 msgr2=0x7fb29411a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb293400700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb294071f90 0x7fb29411a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.552+0000 7fb293400700 1 -- 172.21.15.63:0/1151055015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb280005040 con 0x7fb2940708a0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.553+0000 7fb293400700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb29411a0b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb284002a50 tx=0x7fb2840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.553+0000 7fb28b400700 1 -- 172.21.15.63:0/1151055015 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb284007cc0 con 0x7fb2940708a0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.553+0000 7fb28b400700 1 -- 172.21.15.63:0/1151055015 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb28400f040 con 0x7fb2940708a0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.553+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2941142f0 con 0x7fb2940708a0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.553+0000 7fb28b400700 1 -- 172.21.15.63:0/1151055015 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb284009540 con 0x7fb2940708a0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.553+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb294114830 con 0x7fb2940708a0 2024-03-23T11:57:41.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.554+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb294071aa0 con 0x7fb2940708a0 2024-03-23T11:57:41.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.556+0000 7fb28b400700 1 -- 172.21.15.63:0/1151055015 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb2840096a0 con 0x7fb2940708a0 2024-03-23T11:57:41.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.557+0000 7fb28b400700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb27806c7d0 0x7fb27806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.557+0000 7fb28b400700 1 -- 172.21.15.63:0/1151055015 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb28408b920 con 0x7fb2940708a0 2024-03-23T11:57:41.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.557+0000 7fb292a00700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb27806c7d0 0x7fb27806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.558+0000 7fb292a00700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb27806c7d0 0x7fb27806ec80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb280009fd0 tx=0x7fb280009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:41.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.562+0000 7fb28b400700 1 -- 172.21.15.63:0/1151055015 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb284059530 con 0x7fb2940708a0 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.838+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb29404bed0 con 0x7fb2940708a0 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.839+0000 7fb28b400700 1 -- 172.21.15.63:0/1151055015 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fb2840590c0 con 0x7fb2940708a0 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stdout:e29 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:57:41.822 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:57:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:57:41.824 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:34408} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/4216607936,v1:172.21.15.111:6826/4216607936] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:41.825 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34414} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/856600159,v1:172.21.15.63:6827/856600159] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:41.825 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:41.825 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:57:41.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb27806c7d0 msgr2=0x7fb27806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb27806c7d0 0x7fb27806ec80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb280009fd0 tx=0x7fb280009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:41.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 msgr2=0x7fb29411a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb29411a0b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb284002a50 tx=0x7fb2840046b0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 shutdown_connections 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb27806c7d0 0x7fb27806ec80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940708a0 0x7fb29411a0b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 --2- 172.21.15.63:0/1151055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb294071f90 0x7fb29411a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.841+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 >> 172.21.15.63:0/1151055015 conn(0x7fb2940f89b0 msgr2=0x7fb2940f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.842+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 shutdown_connections 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.842+0000 7fb29a600700 1 -- 172.21.15.63:0/1151055015 wait complete. 2024-03-23T11:57:41.826 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 29 2024-03-23T11:57:41.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 -- 172.21.15.63:0/2591760128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f81037a0 msgr2=0x7f95f80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 --2- 172.21.15.63:0/2591760128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f81037a0 0x7f95f80fce10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f95e4004940 tx=0x7f95e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:41.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 -- 172.21.15.63:0/2591760128 shutdown_connections 2024-03-23T11:57:41.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 --2- 172.21.15.63:0/2591760128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f81037a0 0x7f95f80fce10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 --2- 172.21.15.63:0/2591760128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f8102e00 0x7f95f81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 -- 172.21.15.63:0/2591760128 >> 172.21.15.63:0/2591760128 conn(0x7f95f80f89b0 msgr2=0x7f95f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:41.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 -- 172.21.15.63:0/2591760128 shutdown_connections 2024-03-23T11:57:41.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.982+0000 7f9600400700 1 -- 172.21.15.63:0/2591760128 wait complete. 2024-03-23T11:57:41.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.983+0000 7f9600400700 1 Processor -- start 2024-03-23T11:57:41.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f9600400700 1 -- start start 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f9600400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f8102e00 0x7f95f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f9600400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f81037a0 0x7f95f8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f9600400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8070d90 con 0x7f95f8102e00 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f9600400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8070ed0 con 0x7f95f81037a0 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f95fda00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f8102e00 0x7f95f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f95fda00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f8102e00 0x7f95f80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37092/0 (socket says 172.21.15.63:37092) 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.984+0000 7f95fda00700 1 -- 172.21.15.63:0/125563192 learned_addr learned my addr 172.21.15.63:0/125563192 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95fd000700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f81037a0 0x7f95f8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95fda00700 1 -- 172.21.15.63:0/125563192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f81037a0 msgr2=0x7f95f8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95fda00700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f81037a0 0x7f95f8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95fda00700 1 -- 172.21.15.63:0/125563192 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4005040 con 0x7f95f8102e00 2024-03-23T11:57:41.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95fda00700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f8102e00 0x7f95f80702c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f95ec002a50 tx=0x7f95ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:41.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95f5600700 1 -- 172.21.15.63:0/125563192 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95ec007cc0 con 0x7f95f8102e00 2024-03-23T11:57:41.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f8119e00 con 0x7f95f8102e00 2024-03-23T11:57:41.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95f5600700 1 -- 172.21.15.63:0/125563192 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95ec00f040 con 0x7f95f8102e00 2024-03-23T11:57:41.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.985+0000 7f95f5600700 1 -- 172.21.15.63:0/125563192 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95ec009500 con 0x7f95f8102e00 2024-03-23T11:57:41.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.986+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f811a280 con 0x7f95f8102e00 2024-03-23T11:57:41.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.987+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f80fe550 con 0x7f95f8102e00 2024-03-23T11:57:41.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.988+0000 7f95f5600700 1 -- 172.21.15.63:0/125563192 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f95ec0096a0 con 0x7f95f8102e00 2024-03-23T11:57:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.989+0000 7f95f5600700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f95e006c7d0 0x7f95e006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.989+0000 7f95f5600700 1 -- 172.21.15.63:0/125563192 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f95ec08b8c0 con 0x7f95f8102e00 2024-03-23T11:57:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.989+0000 7f95fd000700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f95e006c7d0 0x7f95e006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.989+0000 7f95fd000700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f95e006c7d0 0x7f95e006ec80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95f8071c00 tx=0x7f95e4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:41.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:41.997+0000 7f95f5600700 1 -- 172.21.15.63:0/125563192 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f95ec055ac0 con 0x7f95f8102e00 2024-03-23T11:57:42.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:41 smithi063 ceph-mon[141124]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:42.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:41 smithi063 ceph-mon[141124]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:42.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:41 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2005895798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:42.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:41 smithi111 ceph-mon[106899]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:42.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:41 smithi111 ceph-mon[106899]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:42.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:41 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2005895798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:42.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.248+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f95f80719b0 con 0x7f95e006c7d0 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.249+0000 7f95f5600700 1 -- 172.21.15.63:0/125563192 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f95f80719b0 con 0x7f95e006c7d0 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:57:42.232 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:57:42.233 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T11:57:42.233 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T11:57:42.233 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:57:42.233 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "12/21 daemons upgraded", 2024-03-23T11:57:42.233 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T11:57:42.233 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:57:42.233 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.251+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f95e006c7d0 msgr2=0x7f95e006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.251+0000 7f9600400700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f95e006c7d0 0x7f95e006ec80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95f8071c00 tx=0x7f95e4009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.251+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f8102e00 msgr2=0x7f95f80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.251+0000 7f9600400700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f8102e00 0x7f95f80702c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f95ec002a50 tx=0x7f95ec0046b0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.252+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 shutdown_connections 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.252+0000 7f9600400700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f95e006c7d0 0x7f95e006ec80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.252+0000 7f9600400700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f8102e00 0x7f95f80702c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.252+0000 7f9600400700 1 --2- 172.21.15.63:0/125563192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f81037a0 0x7f95f8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.252+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 >> 172.21.15.63:0/125563192 conn(0x7f95f80f89b0 msgr2=0x7f95f80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.252+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 shutdown_connections 2024-03-23T11:57:42.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.252+0000 7f9600400700 1 -- 172.21.15.63:0/125563192 wait complete. 2024-03-23T11:57:42.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 -- 172.21.15.63:0/3028595965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 msgr2=0x7f2f400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:42.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 --2- 172.21.15.63:0/3028595965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f400fce10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f28004940 tx=0x7f2f28009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:42.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 -- 172.21.15.63:0/3028595965 shutdown_connections 2024-03-23T11:57:42.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 --2- 172.21.15.63:0/3028595965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f400fce10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 --2- 172.21.15.63:0/3028595965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 -- 172.21.15.63:0/3028595965 >> 172.21.15.63:0/3028595965 conn(0x7f2f400f89b0 msgr2=0x7f2f400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 -- 172.21.15.63:0/3028595965 shutdown_connections 2024-03-23T11:57:42.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.393+0000 7f2f46400700 1 -- 172.21.15.63:0/3028595965 wait complete. 2024-03-23T11:57:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.394+0000 7f2f46400700 1 Processor -- start 2024-03-23T11:57:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.394+0000 7f2f46400700 1 -- start start 2024-03-23T11:57:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.394+0000 7f2f46400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.394+0000 7f2f46400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.394+0000 7f2f46400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40116930 con 0x7f2f401037a0 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.394+0000 7f2f46400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4010fd00 con 0x7f2f40102e00 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37102/0 (socket says 172.21.15.63:37102) 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37e00700 1 -- 172.21.15.63:0/3051986440 learned_addr learned my addr 172.21.15.63:0/3051986440 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37e00700 1 -- 172.21.15.63:0/3051986440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 msgr2=0x7f2f40115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37e00700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37e00700 1 -- 172.21.15.63:0/3051986440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f28005040 con 0x7f2f401037a0 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:57:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37e00700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401162e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009390 tx=0x7f2f28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.395+0000 7f2f37400700 1 -- 172.21.15.63:0/3051986440 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2800e040 con 0x7f2f401037a0 2024-03-23T11:57:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.396+0000 7f2f37400700 1 -- 172.21.15.63:0/3051986440 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f28013420 con 0x7f2f401037a0 2024-03-23T11:57:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.396+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4010ff80 con 0x7f2f401037a0 2024-03-23T11:57:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.396+0000 7f2f37400700 1 -- 172.21.15.63:0/3051986440 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2801c630 con 0x7f2f401037a0 2024-03-23T11:57:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.396+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f401104c0 con 0x7f2f401037a0 2024-03-23T11:57:42.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.397+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f400fe550 con 0x7f2f401037a0 2024-03-23T11:57:42.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.398+0000 7f2f37400700 1 -- 172.21.15.63:0/3051986440 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2f28013a00 con 0x7f2f401037a0 2024-03-23T11:57:42.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.399+0000 7f2f37400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:57:42.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.399+0000 7f2f37400700 1 -- 172.21.15.63:0/3051986440 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f2808cd00 con 0x7f2f401037a0 2024-03-23T11:57:42.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.399+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:57:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.400+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009fd0 tx=0x7f2f30009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:57:42.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.403+0000 7f2f37400700 1 -- 172.21.15.63:0/3051986440 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f2805aa10 con 0x7f2f401037a0 2024-03-23T11:57:42.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.757+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2f4005f5f0 con 0x7f2f401037a0 2024-03-23T11:57:42.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.757+0000 7f2f37400700 1 -- 172.21.15.63:0/3051986440 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2f2805a5a0 con 0x7f2f401037a0 2024-03-23T11:57:42.741 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T11:57:42.741 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T11:57:42.741 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T11:57:42.741 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T11:57:42.741 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 msgr2=0x7f2f2406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009fd0 tx=0x7f2f30009ee0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 msgr2=0x7f2f401162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401162e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009390 tx=0x7f2f28009e30 comp rx=0 tx=0).stop 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 shutdown_connections 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 --2- 172.21.15.63:0/3051986440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401162e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 >> 172.21.15.63:0/3051986440 conn(0x7f2f400f89b0 msgr2=0x7f2f400f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:57:42.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 shutdown_connections 2024-03-23T11:57:42.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:57:42.760+0000 7f2f46400700 1 -- 172.21.15.63:0/3051986440 wait complete. 2024-03-23T11:57:43.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:42 smithi063 ceph-mon[141124]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:43.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:42 smithi063 ceph-mon[141124]: pgmap v195: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 99 B/s rd, 0 op/s 2024-03-23T11:57:43.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:42 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1151055015' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:57:43.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:42 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3051986440' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:57:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:42 smithi111 ceph-mon[106899]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:42 smithi111 ceph-mon[106899]: pgmap v195: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 99 B/s rd, 0 op/s 2024-03-23T11:57:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:42 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1151055015' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:57:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:42 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3051986440' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:57:44.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:43 smithi063 ceph-mon[141124]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:44.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:43 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:44.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:43 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:44.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:43 smithi111 ceph-mon[106899]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:57:44.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:43 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:44.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:43 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:45.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:44 smithi063 ceph-mon[141124]: pgmap v196: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:45.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:44 smithi111 ceph-mon[106899]: pgmap v196: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:46.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:45 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:46.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:45 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:46.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:45 smithi063 ceph-mon[141124]: pgmap v197: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:46.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:45 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:46.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:45 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:46.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:45 smithi111 ceph-mon[106899]: pgmap v197: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:49.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:48 smithi063 ceph-mon[141124]: pgmap v198: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:49.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:48 smithi111 ceph-mon[106899]: pgmap v198: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:50.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:50.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:50.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:50.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:57:50.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:50.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:50.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:50 smithi063 ceph-mon[141124]: pgmap v199: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:50.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:57:50.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:50 smithi111 ceph-mon[106899]: pgmap v199: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:52.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:52 smithi063 ceph-mon[141124]: Upgrade: Updating mds.cephfs.smithi063.mdbdrz 2024-03-23T11:57:52.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:52.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:57:52.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:52.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:52 smithi063 ceph-mon[141124]: Deploying daemon mds.cephfs.smithi063.mdbdrz on smithi063 2024-03-23T11:57:52.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:52 smithi063 ceph-mon[141124]: pgmap v200: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:52.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:52 smithi111 ceph-mon[106899]: Upgrade: Updating mds.cephfs.smithi063.mdbdrz 2024-03-23T11:57:52.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:52.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi063.mdbdrz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:57:52.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:57:52.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:52 smithi111 ceph-mon[106899]: Deploying daemon mds.cephfs.smithi063.mdbdrz on smithi063 2024-03-23T11:57:52.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:52 smithi111 ceph-mon[106899]: pgmap v200: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:54.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:54 smithi063 ceph-mon[141124]: osdmap e84: 6 total, 6 up, 6 in 2024-03-23T11:57:54.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:54 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T11:57:54.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:57:54.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:54 smithi063 ceph-mon[141124]: pgmap v202: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:54.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:54 smithi111 ceph-mon[106899]: osdmap e84: 6 total, 6 up, 6 in 2024-03-23T11:57:54.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:54 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T11:57:54.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:57:54.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:54 smithi111 ceph-mon[106899]: pgmap v202: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:56.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:56.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:56 smithi063 ceph-mon[141124]: pgmap v203: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:56.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:57:56.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:56 smithi111 ceph-mon[106899]: pgmap v203: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:58.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:58 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:boot 2024-03-23T11:57:58.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:58 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:57:58.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:58 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:57:58.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:58 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:58.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:58 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:58.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:58 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:58.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:57:58 smithi063 ceph-mon[141124]: pgmap v204: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:57:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:58 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:boot 2024-03-23T11:57:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:58 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:57:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:58 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T11:57:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:58 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:58 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:57:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:58 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:57:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:57:58 smithi111 ceph-mon[106899]: pgmap v204: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:00 smithi063 ceph-mon[141124]: pgmap v205: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:00 smithi111 ceph-mon[106899]: pgmap v205: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:03.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:02 smithi063 ceph-mon[141124]: pgmap v206: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:03.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:02 smithi111 ceph-mon[106899]: pgmap v206: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:04.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:03 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:04.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:03 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:04.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:03 smithi111 ceph-mon[106899]: pgmap v207: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:04.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:03 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:04.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:03 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:04.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:03 smithi063 ceph-mon[141124]: pgmap v207: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:05.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:05 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:05.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:05 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:05.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:05 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:05.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:05 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:06.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:06 smithi063 ceph-mon[141124]: pgmap v208: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:06.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:06 smithi111 ceph-mon[106899]: pgmap v208: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:08 smithi063 ceph-mon[141124]: pgmap v209: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:58:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:58:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:08 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:08 smithi111 ceph-mon[106899]: pgmap v209: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:58:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T11:58:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:08 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:09.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:09 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:58:09.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:09 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:09.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:09 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:09.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:09 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:09.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:09 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:09 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:58:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:09 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:09 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:09 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:10.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:09 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:10 smithi063 ceph-mon[141124]: pgmap v210: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:10 smithi111 ceph-mon[106899]: pgmap v210: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:12.211 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:11 smithi111 ceph-mon[106899]: Upgrade: Updating mds.cephfs.smithi111.ylnmzy 2024-03-23T11:58:12.212 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:12.212 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:58:12.212 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:58:12.212 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:11 smithi111 ceph-mon[106899]: Deploying daemon mds.cephfs.smithi111.ylnmzy on smithi111 2024-03-23T11:58:12.212 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:11 smithi111 ceph-mon[106899]: pgmap v211: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:12.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:11 smithi063 ceph-mon[141124]: Upgrade: Updating mds.cephfs.smithi111.ylnmzy 2024-03-23T11:58:12.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:12.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.ylnmzy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T11:58:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T11:58:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:11 smithi063 ceph-mon[141124]: Deploying daemon mds.cephfs.smithi111.ylnmzy on smithi111 2024-03-23T11:58:12.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:11 smithi063 ceph-mon[141124]: pgmap v211: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 -- 172.21.15.63:0/3515095083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0068102e00 msgr2=0x7f00681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 --2- 172.21.15.63:0/3515095083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0068102e00 0x7f00681031d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f005c004940 tx=0x7f005c009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 -- 172.21.15.63:0/3515095083 shutdown_connections 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 --2- 172.21.15.63:0/3515095083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00681037a0 0x7f00680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 --2- 172.21.15.63:0/3515095083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0068102e00 0x7f00681031d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 -- 172.21.15.63:0/3515095083 >> 172.21.15.63:0/3515095083 conn(0x7f00680f89b0 msgr2=0x7f00680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 -- 172.21.15.63:0/3515095083 shutdown_connections 2024-03-23T11:58:12.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.914+0000 7f006f400700 1 -- 172.21.15.63:0/3515095083 wait complete. 2024-03-23T11:58:12.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.915+0000 7f006f400700 1 Processor -- start 2024-03-23T11:58:12.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.915+0000 7f006f400700 1 -- start start 2024-03-23T11:58:12.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f006f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068102e00 0x7f00680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:12.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f006f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f00681037a0 0x7f0068070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:12.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f006f400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0068070d90 con 0x7f00681037a0 2024-03-23T11:58:12.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f006f400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0068070ed0 con 0x7f0068102e00 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f0067e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f00681037a0 0x7f0068070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f0067e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f00681037a0 0x7f0068070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50336/0 (socket says 172.21.15.63:50336) 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f0067e00700 1 -- 172.21.15.63:0/1388792525 learned_addr learned my addr 172.21.15.63:0/1388792525 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.916+0000 7f006ca00700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068102e00 0x7f00680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f0067e00700 1 -- 172.21.15.63:0/1388792525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068102e00 msgr2=0x7f00680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f0067e00700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068102e00 0x7f00680702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f0067e00700 1 -- 172.21.15.63:0/1388792525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c005040 con 0x7f00681037a0 2024-03-23T11:58:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f0067e00700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f00681037a0 0x7f0068070800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f005400eb40 tx=0x7f005400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:12.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f0064c00700 1 -- 172.21.15.63:0/1388792525 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0054007cc0 con 0x7f00681037a0 2024-03-23T11:58:12.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f0064c00700 1 -- 172.21.15.63:0/1388792525 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f005400f040 con 0x7f00681037a0 2024-03-23T11:58:12.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0068071120 con 0x7f00681037a0 2024-03-23T11:58:12.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f0064c00700 1 -- 172.21.15.63:0/1388792525 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f005400c8f0 con 0x7f00681037a0 2024-03-23T11:58:12.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.917+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006811a170 con 0x7f00681037a0 2024-03-23T11:58:12.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.918+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00680fe550 con 0x7f00681037a0 2024-03-23T11:58:12.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.919+0000 7f0064c00700 1 -- 172.21.15.63:0/1388792525 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f00540090d0 con 0x7f00681037a0 2024-03-23T11:58:12.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.920+0000 7f0064c00700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f005006c7d0 0x7f005006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:12.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.920+0000 7f0064c00700 1 -- 172.21.15.63:0/1388792525 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f005408b470 con 0x7f00681037a0 2024-03-23T11:58:12.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.920+0000 7f006ca00700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f005006c7d0 0x7f005006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:12.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.921+0000 7f006ca00700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f005006c7d0 0x7f005006ec80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f005c00b0d0 tx=0x7f005c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:12.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:12.925+0000 7f0064c00700 1 -- 172.21.15.63:0/1388792525 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0054055720 con 0x7f00681037a0 2024-03-23T11:58:13.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.158+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f006804bed0 con 0x7f005006c7d0 2024-03-23T11:58:13.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.160+0000 7f0064c00700 1 -- 172.21.15.63:0/1388792525 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f006804bed0 con 0x7f005006c7d0 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f005006c7d0 msgr2=0x7f005006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f005006c7d0 0x7f005006ec80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f005c00b0d0 tx=0x7f005c009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f00681037a0 msgr2=0x7f0068070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f00681037a0 0x7f0068070800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f005400eb40 tx=0x7f005400b040 comp rx=0 tx=0).stop 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 shutdown_connections 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f005006c7d0 0x7f005006ec80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068102e00 0x7f00680702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 --2- 172.21.15.63:0/1388792525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f00681037a0 0x7f0068070800 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 >> 172.21.15.63:0/1388792525 conn(0x7f00680f89b0 msgr2=0x7f00680f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:13.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 shutdown_connections 2024-03-23T11:58:13.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.162+0000 7f006f400700 1 -- 172.21.15.63:0/1388792525 wait complete. 2024-03-23T11:58:13.159 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 -- 172.21.15.63:0/3125682616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 msgr2=0x7f3370103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 --2- 172.21.15.63:0/3125682616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f3370103070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3360004940 tx=0x7f3360009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 -- 172.21.15.63:0/3125682616 shutdown_connections 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 --2- 172.21.15.63:0/3125682616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3370065630 0x7f3370065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 --2- 172.21.15.63:0/3125682616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f3370103070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 -- 172.21.15.63:0/3125682616 >> 172.21.15.63:0/3125682616 conn(0x7f3370071b70 msgr2=0x7f3370071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 -- 172.21.15.63:0/3125682616 shutdown_connections 2024-03-23T11:58:13.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.302+0000 7f336f200700 1 -- 172.21.15.63:0/3125682616 wait complete. 2024-03-23T11:58:13.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.303+0000 7f336f200700 1 Processor -- start 2024-03-23T11:58:13.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.303+0000 7f336f200700 1 -- start start 2024-03-23T11:58:13.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3370065630 0x7f33700fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:13.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f33700fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:13.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700fc6f0 con 0x7f3370102ca0 2024-03-23T11:58:13.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700fc830 con 0x7f3370065630 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f33700fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336de00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3370065630 0x7f33700fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f33700fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50348/0 (socket says 172.21.15.63:50348) 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336de00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3370065630 0x7f33700fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:41280/0 (socket says 172.21.15.63:41280) 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.304+0000 7f336d400700 1 -- 172.21.15.63:0/3654274324 learned_addr learned my addr 172.21.15.63:0/3654274324 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f336d400700 1 -- 172.21.15.63:0/3654274324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3370065630 msgr2=0x7f33700fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f336d400700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3370065630 0x7f33700fbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f336d400700 1 -- 172.21.15.63:0/3654274324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c005040 con 0x7f3370102ca0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f336d400700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f33700fc160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f335c0096d0 tx=0x7f335c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f3366000700 1 -- 172.21.15.63:0/3654274324 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f335c00e840 con 0x7f3370102ca0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f3366000700 1 -- 172.21.15.63:0/3654274324 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f335c00f040 con 0x7f3370102ca0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3360005040 con 0x7f3370102ca0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f3366000700 1 -- 172.21.15.63:0/3654274324 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f335c013d30 con 0x7f3370102ca0 2024-03-23T11:58:13.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.305+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3370119db0 con 0x7f3370102ca0 2024-03-23T11:58:13.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.306+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33700fe380 con 0x7f3370102ca0 2024-03-23T11:58:13.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.307+0000 7f3366000700 1 -- 172.21.15.63:0/3654274324 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f335c00e9a0 con 0x7f3370102ca0 2024-03-23T11:58:13.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.309+0000 7f3366000700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f335406c7d0 0x7f335406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:13.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.309+0000 7f3366000700 1 -- 172.21.15.63:0/3654274324 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f335c08b810 con 0x7f3370102ca0 2024-03-23T11:58:13.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.309+0000 7f336de00700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f335406c7d0 0x7f335406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:13.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.310+0000 7f336de00700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f335406c7d0 0x7f335406ec80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f33600045a0 tx=0x7f3360009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:13.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.313+0000 7f3366000700 1 -- 172.21.15.63:0/3654274324 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f335c055a10 con 0x7f3370102ca0 2024-03-23T11:58:13.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.545+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f337004bed0 con 0x7f335406c7d0 2024-03-23T11:58:13.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.546+0000 7f3366000700 1 -- 172.21.15.63:0/3654274324 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f337004bed0 con 0x7f335406c7d0 2024-03-23T11:58:13.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f335406c7d0 msgr2=0x7f335406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f335406c7d0 0x7f335406ec80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f33600045a0 tx=0x7f3360009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 msgr2=0x7f33700fc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f33700fc160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f335c0096d0 tx=0x7f335c009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 shutdown_connections 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f335406c7d0 0x7f335406ec80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3370065630 0x7f33700fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 --2- 172.21.15.63:0/3654274324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3370102ca0 0x7f33700fc160 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 >> 172.21.15.63:0/3654274324 conn(0x7f3370071b70 msgr2=0x7f3370105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.548+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 shutdown_connections 2024-03-23T11:58:13.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.549+0000 7f336f200700 1 -- 172.21.15.63:0/3654274324 wait complete. 2024-03-23T11:58:13.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.685+0000 7f0e00200700 1 -- 172.21.15.63:0/1418203453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 msgr2=0x7f0df81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.685+0000 7f0e00200700 1 --2- 172.21.15.63:0/1418203453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df81031d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0dec004940 tx=0x7f0dec009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:13.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.685+0000 7f0e00200700 1 -- 172.21.15.63:0/1418203453 shutdown_connections 2024-03-23T11:58:13.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.685+0000 7f0e00200700 1 --2- 172.21.15.63:0/1418203453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df81037a0 0x7f0df80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.685+0000 7f0e00200700 1 --2- 172.21.15.63:0/1418203453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df81031d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.685+0000 7f0e00200700 1 -- 172.21.15.63:0/1418203453 >> 172.21.15.63:0/1418203453 conn(0x7f0df80f89b0 msgr2=0x7f0df80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:13.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.686+0000 7f0e00200700 1 -- 172.21.15.63:0/1418203453 shutdown_connections 2024-03-23T11:58:13.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.686+0000 7f0e00200700 1 -- 172.21.15.63:0/1418203453 wait complete. 2024-03-23T11:58:13.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.686+0000 7f0e00200700 1 Processor -- start 2024-03-23T11:58:13.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0e00200700 1 -- start start 2024-03-23T11:58:13.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0e00200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df8115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:13.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0e00200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df81037a0 0x7f0df81161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:13.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0e00200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8116800 con 0x7f0df8102e00 2024-03-23T11:58:13.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0e00200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df810fbd0 con 0x7f0df81037a0 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0dfd800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df8115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0dfd800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df8115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50374/0 (socket says 172.21.15.63:50374) 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0dfd800700 1 -- 172.21.15.63:0/2036467421 learned_addr learned my addr 172.21.15.63:0/2036467421 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.687+0000 7f0dfce00700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df81037a0 0x7f0df81161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0dfd800700 1 -- 172.21.15.63:0/2036467421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df81037a0 msgr2=0x7f0df81161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0dfd800700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df81037a0 0x7f0df81161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0dfd800700 1 -- 172.21.15.63:0/2036467421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec005040 con 0x7f0df8102e00 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0dfd800700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df8115c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0dec007f50 tx=0x7f0dec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0df5600700 1 -- 172.21.15.63:0/2036467421 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dec021070 con 0x7f0df8102e00 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0df5600700 1 -- 172.21.15.63:0/2036467421 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dec0093b0 con 0x7f0df8102e00 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0df5600700 1 -- 172.21.15.63:0/2036467421 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dec00e070 con 0x7f0df8102e00 2024-03-23T11:58:13.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df810fe50 con 0x7f0df8102e00 2024-03-23T11:58:13.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.688+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df81103b0 con 0x7f0df8102e00 2024-03-23T11:58:13.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.689+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df80fe550 con 0x7f0df8102e00 2024-03-23T11:58:13.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.690+0000 7f0df5600700 1 -- 172.21.15.63:0/2036467421 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0dec01c400 con 0x7f0df8102e00 2024-03-23T11:58:13.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.691+0000 7f0df5600700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0de006c7d0 0x7f0de006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:13.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.691+0000 7f0df5600700 1 -- 172.21.15.63:0/2036467421 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0dec08cb60 con 0x7f0df8102e00 2024-03-23T11:58:13.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.691+0000 7f0dfce00700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0de006c7d0 0x7f0de006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:13.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.692+0000 7f0dfce00700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0de006c7d0 0x7f0de006ec80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0df8111510 tx=0x7f0de400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:13.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.696+0000 7f0df5600700 1 -- 172.21.15.63:0/2036467421 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0dec05a870 con 0x7f0df8102e00 2024-03-23T11:58:13.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.924+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0df804bed0 con 0x7f0de006c7d0 2024-03-23T11:58:13.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.934+0000 7f0df5600700 1 -- 172.21.15.63:0/2036467421 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f0df804bed0 con 0x7f0de006c7d0 2024-03-23T11:58:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:58:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (7m) 11s ago 20m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:58:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (4m) 11s ago 20m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:58:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (4m) 54s ago 19m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:58:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (7m) 11s ago 20m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:58:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (37s) 11s ago 14m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (16s) 11s ago 14m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (14m) 54s ago 14m 65.3M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (14m) 54s ago 14m 60.1M - 17.2.7-726-g9c4e0c34 837124da804f b6ddb96ba4e9 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (9m) 11s ago 22m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (8m) 54s ago 19m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (5m) 11s ago 22m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (5m) 54s ago 19m 40.5M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (8m) 11s ago 20m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (8m) 54s ago 19m 7654k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (4m) 11s ago 18m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (3m) 11s ago 18m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:58:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (2m) 11s ago 17m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:58:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (2m) 54s ago 16m 96.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T11:58:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (95s) 54s ago 16m 90.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T11:58:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (58s) 54s ago 15m 10.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T11:58:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (7m) 11s ago 20m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:58:13.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.936+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0de006c7d0 msgr2=0x7f0de006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.936+0000 7f0e00200700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0de006c7d0 0x7f0de006ec80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0df8111510 tx=0x7f0de400b040 comp rx=0 tx=0).stop 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.936+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 msgr2=0x7f0df8115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.936+0000 7f0e00200700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df8115c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0dec007f50 tx=0x7f0dec009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.937+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 shutdown_connections 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.937+0000 7f0e00200700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0de006c7d0 0x7f0de006ec80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.937+0000 7f0e00200700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8102e00 0x7f0df8115c70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.937+0000 7f0e00200700 1 --2- 172.21.15.63:0/2036467421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df81037a0 0x7f0df81161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.937+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 >> 172.21.15.63:0/2036467421 conn(0x7f0df80f89b0 msgr2=0x7f0df80f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.937+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 shutdown_connections 2024-03-23T11:58:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:13.937+0000 7f0e00200700 1 -- 172.21.15.63:0/2036467421 wait complete. 2024-03-23T11:58:14.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 -- 172.21.15.63:0/2477252474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 msgr2=0x7fd64c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 --2- 172.21.15.63:0/2477252474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c0fce10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd638004940 tx=0x7fd638009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:14.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 -- 172.21.15.63:0/2477252474 shutdown_connections 2024-03-23T11:58:14.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 --2- 172.21.15.63:0/2477252474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c0fce10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 --2- 172.21.15.63:0/2477252474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd64c102e00 0x7fd64c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 -- 172.21.15.63:0/2477252474 >> 172.21.15.63:0/2477252474 conn(0x7fd64c0f89b0 msgr2=0x7fd64c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:14.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 -- 172.21.15.63:0/2477252474 shutdown_connections 2024-03-23T11:58:14.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.073+0000 7fd653800700 1 -- 172.21.15.63:0/2477252474 wait complete. 2024-03-23T11:58:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.074+0000 7fd653800700 1 Processor -- start 2024-03-23T11:58:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.074+0000 7fd653800700 1 -- start start 2024-03-23T11:58:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.074+0000 7fd653800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd64c102e00 0x7fd64c10e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd653800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c10b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd653800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c10ee80 con 0x7fd64c1037a0 2024-03-23T11:58:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd653800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c10efc0 con 0x7fd64c102e00 2024-03-23T11:58:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd64be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c10b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd64be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c10b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50390/0 (socket says 172.21.15.63:50390) 2024-03-23T11:58:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd64be00700 1 -- 172.21.15.63:0/3094810282 learned_addr learned my addr 172.21.15.63:0/3094810282 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd650e00700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd64c102e00 0x7fd64c10e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd64be00700 1 -- 172.21.15.63:0/3094810282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd64c102e00 msgr2=0x7fd64c10e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd64be00700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd64c102e00 0x7fd64c10e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.075+0000 7fd64be00700 1 -- 172.21.15.63:0/3094810282 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd638005040 con 0x7fd64c1037a0 2024-03-23T11:58:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.076+0000 7fd64be00700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c10b7c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd638009fd0 tx=0x7fd638009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.076+0000 7fd648c00700 1 -- 172.21.15.63:0/3094810282 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd63800e030 con 0x7fd64c1037a0 2024-03-23T11:58:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.076+0000 7fd648c00700 1 -- 172.21.15.63:0/3094810282 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6380093b0 con 0x7fd64c1037a0 2024-03-23T11:58:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.076+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c10bd00 con 0x7fd64c1037a0 2024-03-23T11:58:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.076+0000 7fd648c00700 1 -- 172.21.15.63:0/3094810282 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd63801cc80 con 0x7fd64c1037a0 2024-03-23T11:58:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.076+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c10c2c0 con 0x7fd64c1037a0 2024-03-23T11:58:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.077+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c0fe550 con 0x7fd64c1037a0 2024-03-23T11:58:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.078+0000 7fd648c00700 1 -- 172.21.15.63:0/3094810282 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd63801c400 con 0x7fd64c1037a0 2024-03-23T11:58:14.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.079+0000 7fd648c00700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd63406c7d0 0x7fd63406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.079+0000 7fd648c00700 1 -- 172.21.15.63:0/3094810282 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd63808d270 con 0x7fd64c1037a0 2024-03-23T11:58:14.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.079+0000 7fd650e00700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd63406c7d0 0x7fd63406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.080+0000 7fd650e00700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd63406c7d0 0x7fd63406ec80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd640009b80 tx=0x7fd64000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:14.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.083+0000 7fd648c00700 1 -- 172.21.15.63:0/3094810282 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd6380574f0 con 0x7fd64c1037a0 2024-03-23T11:58:14.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.386+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd64c04bed0 con 0x7fd64c1037a0 2024-03-23T11:58:14.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.387+0000 7fd648c00700 1 -- 172.21.15.63:0/3094810282 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fd638004490 con 0x7fd64c1037a0 2024-03-23T11:58:14.370 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:58:14.370 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:58:14.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:58:14.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:14.370 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:58:14.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 12 2024-03-23T11:58:14.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:58:14.372 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd63406c7d0 msgr2=0x7fd63406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd63406c7d0 0x7fd63406ec80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd640009b80 tx=0x7fd64000b040 comp rx=0 tx=0).stop 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 msgr2=0x7fd64c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c10b7c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd638009fd0 tx=0x7fd638009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 shutdown_connections 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd63406c7d0 0x7fd63406ec80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd64c102e00 0x7fd64c10e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 --2- 172.21.15.63:0/3094810282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd64c1037a0 0x7fd64c10b7c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 >> 172.21.15.63:0/3094810282 conn(0x7fd64c0f89b0 msgr2=0x7fd64c0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 shutdown_connections 2024-03-23T11:58:14.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.390+0000 7fd653800700 1 -- 172.21.15.63:0/3094810282 wait complete. 2024-03-23T11:58:14.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 -- 172.21.15.63:0/1620322116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fe830 msgr2=0x7f9e2c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 --2- 172.21.15.63:0/1620322116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fe830 0x7f9e2c106f00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9e18004940 tx=0x7f9e18009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:14.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 -- 172.21.15.63:0/1620322116 shutdown_connections 2024-03-23T11:58:14.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 --2- 172.21.15.63:0/1620322116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fe830 0x7f9e2c106f00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 --2- 172.21.15.63:0/1620322116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c0fde90 0x7f9e2c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 -- 172.21.15.63:0/1620322116 >> 172.21.15.63:0/1620322116 conn(0x7f9e2c072bc0 msgr2=0x7f9e2c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:14.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 -- 172.21.15.63:0/1620322116 shutdown_connections 2024-03-23T11:58:14.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.527+0000 7f9e33400700 1 -- 172.21.15.63:0/1620322116 wait complete. 2024-03-23T11:58:14.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.528+0000 7f9e33400700 1 Processor -- start 2024-03-23T11:58:14.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.528+0000 7f9e33400700 1 -- start start 2024-03-23T11:58:14.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e33400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fde90 0x7f9e2c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e33400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c0fe830 0x7f9e2c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e33400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c1168a0 con 0x7f9e2c0fde90 2024-03-23T11:58:14.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e33400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c10fc70 con 0x7f9e2c0fe830 2024-03-23T11:58:14.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e30a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fde90 0x7f9e2c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e30a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fde90 0x7f9e2c115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50410/0 (socket says 172.21.15.63:50410) 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e30a00700 1 -- 172.21.15.63:0/2166182569 learned_addr learned my addr 172.21.15.63:0/2166182569 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e2be00700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c0fe830 0x7f9e2c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e30a00700 1 -- 172.21.15.63:0/2166182569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c0fe830 msgr2=0x7f9e2c116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e30a00700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c0fe830 0x7f9e2c116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.529+0000 7f9e30a00700 1 -- 172.21.15.63:0/2166182569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18005040 con 0x7f9e2c0fde90 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.530+0000 7f9e30a00700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fde90 0x7f9e2c115d10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9e20002a50 tx=0x7f9e200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.530+0000 7f9e28c00700 1 -- 172.21.15.63:0/2166182569 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e20007cc0 con 0x7f9e2c0fde90 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.530+0000 7f9e28c00700 1 -- 172.21.15.63:0/2166182569 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e2000f040 con 0x7f9e2c0fde90 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.530+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c10ff50 con 0x7f9e2c0fde90 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.530+0000 7f9e28c00700 1 -- 172.21.15.63:0/2166182569 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e2000d9c0 con 0x7f9e2c0fde90 2024-03-23T11:58:14.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.530+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c110490 con 0x7f9e2c0fde90 2024-03-23T11:58:14.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.531+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c106a20 con 0x7f9e2c0fde90 2024-03-23T11:58:14.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.532+0000 7f9e28c00700 1 -- 172.21.15.63:0/2166182569 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9e2000db20 con 0x7f9e2c0fde90 2024-03-23T11:58:14.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.533+0000 7f9e28c00700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1406c7d0 0x7f9e1406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.533+0000 7f9e28c00700 1 -- 172.21.15.63:0/2166182569 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9e2008b820 con 0x7f9e2c0fde90 2024-03-23T11:58:14.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.533+0000 7f9e2be00700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1406c7d0 0x7f9e1406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.534+0000 7f9e2be00700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1406c7d0 0x7f9e1406ec80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c1115b0 tx=0x7f9e18009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:14.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.538+0000 7f9e28c00700 1 -- 172.21.15.63:0/2166182569 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e20059430 con 0x7f9e2c0fde90 2024-03-23T11:58:14.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:14 smithi063 ceph-mon[141124]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:14.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:14 smithi063 ceph-mon[141124]: osdmap e85: 6 total, 6 up, 6 in 2024-03-23T11:58:14.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:14 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T11:58:14.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:14 smithi063 ceph-mon[141124]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:14.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:14 smithi063 ceph-mon[141124]: pgmap v213: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:14 smithi111 ceph-mon[106899]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:14 smithi111 ceph-mon[106899]: osdmap e85: 6 total, 6 up, 6 in 2024-03-23T11:58:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:14 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T11:58:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:14 smithi111 ceph-mon[106899]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:14 smithi111 ceph-mon[106899]: pgmap v213: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:14.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.798+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9e2c04bed0 con 0x7f9e2c0fde90 2024-03-23T11:58:14.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.799+0000 7f9e28c00700 1 -- 172.21.15.63:0/2166182569 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1661 (secure 0 0 0) 0x7f9e20058fc0 con 0x7f9e2c0fde90 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:e32 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:58:14.782 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:58:14.783 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T11:58:14.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:14.785 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:58:14.785 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:58:14.785 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1406c7d0 msgr2=0x7f9e1406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1406c7d0 0x7f9e1406ec80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c1115b0 tx=0x7f9e18009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fde90 msgr2=0x7f9e2c115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fde90 0x7f9e2c115d10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9e20002a50 tx=0x7f9e200046b0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 shutdown_connections 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1406c7d0 0x7f9e1406ec80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e2c0fde90 0x7f9e2c115d10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 --2- 172.21.15.63:0/2166182569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c0fe830 0x7f9e2c116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 >> 172.21.15.63:0/2166182569 conn(0x7f9e2c072bc0 msgr2=0x7f9e2c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:14.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 shutdown_connections 2024-03-23T11:58:14.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.801+0000 7f9e33400700 1 -- 172.21.15.63:0/2166182569 wait complete. 2024-03-23T11:58:14.787 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 32 2024-03-23T11:58:14.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 -- 172.21.15.63:0/2657007814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b601037a0 msgr2=0x7f4b600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 --2- 172.21.15.63:0/2657007814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b601037a0 0x7f4b600fce10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c004940 tx=0x7f4b4c009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:14.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 -- 172.21.15.63:0/2657007814 shutdown_connections 2024-03-23T11:58:14.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 --2- 172.21.15.63:0/2657007814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b601037a0 0x7f4b600fce10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 --2- 172.21.15.63:0/2657007814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b60102e00 0x7f4b601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 -- 172.21.15.63:0/2657007814 >> 172.21.15.63:0/2657007814 conn(0x7f4b600f89b0 msgr2=0x7f4b600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:14.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 -- 172.21.15.63:0/2657007814 shutdown_connections 2024-03-23T11:58:14.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.940+0000 7f4b67000700 1 -- 172.21.15.63:0/2657007814 wait complete. 2024-03-23T11:58:14.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.941+0000 7f4b67000700 1 Processor -- start 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.941+0000 7f4b67000700 1 -- start start 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.941+0000 7f4b67000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b60102e00 0x7f4b60115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.941+0000 7f4b67000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b601037a0 0x7f4b601162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.941+0000 7f4b67000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b60116930 con 0x7f4b60102e00 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.941+0000 7f4b67000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6010fd00 con 0x7f4b601037a0 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b60102e00 0x7f4b60115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b60102e00 0x7f4b60115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50420/0 (socket says 172.21.15.63:50420) 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5fe00700 1 -- 172.21.15.63:0/1704493016 learned_addr learned my addr 172.21.15.63:0/1704493016 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5f400700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b601037a0 0x7f4b601162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5fe00700 1 -- 172.21.15.63:0/1704493016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b601037a0 msgr2=0x7f4b601162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:14.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5fe00700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b601037a0 0x7f4b601162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5fe00700 1 -- 172.21.15.63:0/1704493016 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c005040 con 0x7f4b60102e00 2024-03-23T11:58:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.942+0000 7f4b5fe00700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b60102e00 0x7f4b60115da0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000c8c0 tx=0x7f4b500044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.943+0000 7f4b57e00700 1 -- 172.21.15.63:0/1704493016 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5000a760 con 0x7f4b60102e00 2024-03-23T11:58:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.943+0000 7f4b57e00700 1 -- 172.21.15.63:0/1704493016 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b5000ad70 con 0x7f4b60102e00 2024-03-23T11:58:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.943+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6010ffe0 con 0x7f4b60102e00 2024-03-23T11:58:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.943+0000 7f4b57e00700 1 -- 172.21.15.63:0/1704493016 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b500113a0 con 0x7f4b60102e00 2024-03-23T11:58:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.943+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b60110520 con 0x7f4b60102e00 2024-03-23T11:58:14.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.944+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b600fe550 con 0x7f4b60102e00 2024-03-23T11:58:14.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.945+0000 7f4b57e00700 1 -- 172.21.15.63:0/1704493016 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4b50012400 con 0x7f4b60102e00 2024-03-23T11:58:14.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.946+0000 7f4b57e00700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4b4406c7d0 0x7f4b4406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:14.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.946+0000 7f4b57e00700 1 -- 172.21.15.63:0/1704493016 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4b5008ba60 con 0x7f4b60102e00 2024-03-23T11:58:14.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.946+0000 7f4b5f400700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4b4406c7d0 0x7f4b4406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:14.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.947+0000 7f4b5f400700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4b4406c7d0 0x7f4b4406ec80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4b60111640 tx=0x7f4b4c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:14.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:14.950+0000 7f4b57e00700 1 -- 172.21.15.63:0/1704493016 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4b50059770 con 0x7f4b60102e00 2024-03-23T11:58:15.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.181+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b6004bf40 con 0x7f4b4406c7d0 2024-03-23T11:58:15.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.182+0000 7f4b57e00700 1 -- 172.21.15.63:0/1704493016 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4b6004bf40 con 0x7f4b4406c7d0 2024-03-23T11:58:15.165 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:58:15.165 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "14/21 daemons upgraded", 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:58:15.166 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4b4406c7d0 msgr2=0x7f4b4406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4b4406c7d0 0x7f4b4406ec80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4b60111640 tx=0x7f4b4c009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b60102e00 msgr2=0x7f4b60115da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b60102e00 0x7f4b60115da0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000c8c0 tx=0x7f4b500044c0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 shutdown_connections 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4b4406c7d0 0x7f4b4406ec80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b60102e00 0x7f4b60115da0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 --2- 172.21.15.63:0/1704493016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b601037a0 0x7f4b601162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 >> 172.21.15.63:0/1704493016 conn(0x7f4b600f89b0 msgr2=0x7f4b600f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 shutdown_connections 2024-03-23T11:58:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.185+0000 7f4b67000700 1 -- 172.21.15.63:0/1704493016 wait complete. 2024-03-23T11:58:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 -- 172.21.15.63:0/2615180476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4640102e00 msgr2=0x7f46401031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 --2- 172.21.15.63:0/2615180476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4640102e00 0x7f46401031d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4630004940 tx=0x7f4630009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 -- 172.21.15.63:0/2615180476 shutdown_connections 2024-03-23T11:58:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 --2- 172.21.15.63:0/2615180476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46401037a0 0x7f46400fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 --2- 172.21.15.63:0/2615180476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4640102e00 0x7f46401031d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 -- 172.21.15.63:0/2615180476 >> 172.21.15.63:0/2615180476 conn(0x7f46400f89b0 msgr2=0x7f46400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:15.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 -- 172.21.15.63:0/2615180476 shutdown_connections 2024-03-23T11:58:15.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.323+0000 7f4644c00700 1 -- 172.21.15.63:0/2615180476 wait complete. 2024-03-23T11:58:15.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.324+0000 7f4644c00700 1 Processor -- start 2024-03-23T11:58:15.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.324+0000 7f4644c00700 1 -- start start 2024-03-23T11:58:15.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.324+0000 7f4644c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640102e00 0x7f4640115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.324+0000 7f4644c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46401037a0 0x7f4640116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.324+0000 7f4644c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4640116860 con 0x7f46401037a0 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.324+0000 7f4644c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464010fc30 con 0x7f4640102e00 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4637e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46401037a0 0x7f4640116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4637e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46401037a0 0x7f4640116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50428/0 (socket says 172.21.15.63:50428) 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4637e00700 1 -- 172.21.15.63:0/3505785154 learned_addr learned my addr 172.21.15.63:0/3505785154 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f463de00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640102e00 0x7f4640115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4637e00700 1 -- 172.21.15.63:0/3505785154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640102e00 msgr2=0x7f4640115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4637e00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640102e00 0x7f4640115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4637e00700 1 -- 172.21.15.63:0/3505785154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630005040 con 0x7f46401037a0 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4637e00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46401037a0 0x7f4640116210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4628002a50 tx=0x7f4628004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.325+0000 7f4636000700 1 -- 172.21.15.63:0/3505785154 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4628007cc0 con 0x7f46401037a0 2024-03-23T11:58:15.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.326+0000 7f4636000700 1 -- 172.21.15.63:0/3505785154 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f462800f040 con 0x7f46401037a0 2024-03-23T11:58:15.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.326+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464010ff10 con 0x7f46401037a0 2024-03-23T11:58:15.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.326+0000 7f4636000700 1 -- 172.21.15.63:0/3505785154 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f462800d920 con 0x7f46401037a0 2024-03-23T11:58:15.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.326+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46401104d0 con 0x7f46401037a0 2024-03-23T11:58:15.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.327+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46400fe550 con 0x7f46401037a0 2024-03-23T11:58:15.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.328+0000 7f4636000700 1 -- 172.21.15.63:0/3505785154 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4628009680 con 0x7f46401037a0 2024-03-23T11:58:15.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.329+0000 7f4636000700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f462406c7d0 0x7f462406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:15.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.329+0000 7f4636000700 1 -- 172.21.15.63:0/3505785154 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f462808b780 con 0x7f46401037a0 2024-03-23T11:58:15.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.329+0000 7f463de00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f462406c7d0 0x7f462406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:15.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.329+0000 7f463de00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f462406c7d0 0x7f462406ec80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4630009fd0 tx=0x7f4630009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:15.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.333+0000 7f4636000700 1 -- 172.21.15.63:0/3505785154 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4628059450 con 0x7f46401037a0 2024-03-23T11:58:15.585 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:15 smithi111 ceph-mon[106899]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:15.585 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:15 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3094810282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:15.585 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:15 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2166182569' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:58:15.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.636+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f464004bed0 con 0x7f46401037a0 2024-03-23T11:58:15.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.636+0000 7f4636000700 1 -- 172.21.15.63:0/3505785154 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4628058fe0 con 0x7f46401037a0 2024-03-23T11:58:15.619 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T11:58:15.619 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T11:58:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T11:58:15.620 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T11:58:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.638+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f462406c7d0 msgr2=0x7f462406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.638+0000 7f4644c00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f462406c7d0 0x7f462406ec80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4630009fd0 tx=0x7f4630009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.638+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46401037a0 msgr2=0x7f4640116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.638+0000 7f4644c00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46401037a0 0x7f4640116210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4628002a50 tx=0x7f4628004600 comp rx=0 tx=0).stop 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.639+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 shutdown_connections 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.639+0000 7f4644c00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f462406c7d0 0x7f462406ec80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.639+0000 7f4644c00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4640102e00 0x7f4640115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.639+0000 7f4644c00700 1 --2- 172.21.15.63:0/3505785154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46401037a0 0x7f4640116210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.639+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 >> 172.21.15.63:0/3505785154 conn(0x7f46400f89b0 msgr2=0x7f46400f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:15.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.639+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 shutdown_connections 2024-03-23T11:58:15.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:15.639+0000 7f4644c00700 1 -- 172.21.15.63:0/3505785154 wait complete. 2024-03-23T11:58:15.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:15 smithi063 ceph-mon[141124]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:15.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:15 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3094810282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:15.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:15 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2166182569' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:58:16.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:16 smithi111 ceph-mon[106899]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:16.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:16 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3505785154' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:58:16.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:16 smithi111 ceph-mon[106899]: pgmap v214: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:16.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:16 smithi063 ceph-mon[141124]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:16.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:16 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3505785154' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:58:16.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:16 smithi063 ceph-mon[141124]: pgmap v214: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:18.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:18.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:18.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:58:18.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:18 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] up:boot 2024-03-23T11:58:18.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:18 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:58:18.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:18 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:58:18.632 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:18 smithi063 ceph-mon[141124]: pgmap v215: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:18.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:18.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:18.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T11:58:18.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:18 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] up:boot 2024-03-23T11:58:18.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:18 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T11:58:18.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:18 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T11:58:18.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:18 smithi111 ceph-mon[106899]: pgmap v215: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:20 smithi063 ceph-mon[141124]: pgmap v216: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:20 smithi111 ceph-mon[106899]: pgmap v216: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:22.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:21 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:22.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:21 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:22.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:21 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:22.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:21 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:23.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:22 smithi063 ceph-mon[141124]: pgmap v217: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:23.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:23.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:23.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:22 smithi111 ceph-mon[106899]: pgmap v217: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:23.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:23.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T11:58:25.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:24 smithi063 ceph-mon[141124]: pgmap v218: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:25.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:24 smithi111 ceph-mon[106899]: pgmap v218: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:26.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:26.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:26 smithi063 ceph-mon[141124]: pgmap v219: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:26.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:26.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:26 smithi111 ceph-mon[106899]: pgmap v219: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:29.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:28 smithi063 ceph-mon[141124]: pgmap v220: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:28 smithi111 ceph-mon[106899]: pgmap v220: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:31.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:30 smithi063 ceph-mon[141124]: pgmap v221: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:31.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:30 smithi111 ceph-mon[106899]: pgmap v221: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:32 smithi063 ceph-mon[141124]: pgmap v222: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:33.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:32 smithi111 ceph-mon[106899]: pgmap v222: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:35.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:34 smithi063 ceph-mon[141124]: pgmap v223: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:35.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:34 smithi111 ceph-mon[106899]: pgmap v223: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:36.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:36 smithi063 ceph-mon[141124]: pgmap v224: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:36.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:36 smithi111 ceph-mon[106899]: pgmap v224: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:39.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:38 smithi063 ceph-mon[141124]: pgmap v225: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:39.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:38 smithi111 ceph-mon[106899]: pgmap v225: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:40 smithi063 ceph-mon[141124]: pgmap v226: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:40 smithi111 ceph-mon[106899]: pgmap v226: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:43.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:42 smithi063 ceph-mon[141124]: pgmap v227: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:42 smithi111 ceph-mon[106899]: pgmap v227: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:45.132 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:44 smithi063 ceph-mon[141124]: pgmap v228: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:45.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:44 smithi111 ceph-mon[106899]: pgmap v228: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 -- 172.21.15.63:0/2765072997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82080708a0 msgr2=0x7f8208070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 --2- 172.21.15.63:0/2765072997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82080708a0 0x7f8208070d10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f81f4004970 tx=0x7f81f4009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 -- 172.21.15.63:0/2765072997 shutdown_connections 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 --2- 172.21.15.63:0/2765072997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82080708a0 0x7f8208070d10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 --2- 172.21.15.63:0/2765072997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8208071f90 0x7f8208070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 -- 172.21.15.63:0/2765072997 >> 172.21.15.63:0/2765072997 conn(0x7f82080f89b0 msgr2=0x7f82080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 -- 172.21.15.63:0/2765072997 shutdown_connections 2024-03-23T11:58:45.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.794+0000 7f820e600700 1 -- 172.21.15.63:0/2765072997 wait complete. 2024-03-23T11:58:45.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.795+0000 7f820e600700 1 Processor -- start 2024-03-23T11:58:45.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.795+0000 7f820e600700 1 -- start start 2024-03-23T11:58:45.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f820e600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82080708a0 0x7f820811a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:45.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f820e600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8208071f90 0x7f820811a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:45.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f820e600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820811ac40 con 0x7f8208071f90 2024-03-23T11:58:45.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f820e600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8208114010 con 0x7f82080708a0 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f8206a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8208071f90 0x7f820811a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f8206a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8208071f90 0x7f820811a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35360/0 (socket says 172.21.15.63:35360) 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f8206a00700 1 -- 172.21.15.63:0/552631749 learned_addr learned my addr 172.21.15.63:0/552631749 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f8207400700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82080708a0 0x7f820811a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f8206a00700 1 -- 172.21.15.63:0/552631749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82080708a0 msgr2=0x7f820811a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f8206a00700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82080708a0 0x7f820811a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.796+0000 7f8206a00700 1 -- 172.21.15.63:0/552631749 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4005040 con 0x7f8208071f90 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.797+0000 7f8206a00700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8208071f90 0x7f820811a5f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f81f4004f50 tx=0x7f81f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.797+0000 7f81ff400700 1 -- 172.21.15.63:0/552631749 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81f400e040 con 0x7f8208071f90 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.797+0000 7f81ff400700 1 -- 172.21.15.63:0/552631749 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81f4013420 con 0x7f8208071f90 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.797+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8208114290 con 0x7f8208071f90 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.797+0000 7f81ff400700 1 -- 172.21.15.63:0/552631749 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81f401c630 con 0x7f8208071f90 2024-03-23T11:58:45.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.797+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82081147d0 con 0x7f8208071f90 2024-03-23T11:58:45.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.798+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8208071aa0 con 0x7f8208071f90 2024-03-23T11:58:45.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.799+0000 7f81ff400700 1 -- 172.21.15.63:0/552631749 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f81f4013590 con 0x7f8208071f90 2024-03-23T11:58:45.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.801+0000 7f81ff400700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f81ec06c7d0 0x7f81ec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:45.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.801+0000 7f81ff400700 1 -- 172.21.15.63:0/552631749 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f81f408cc20 con 0x7f8208071f90 2024-03-23T11:58:45.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.801+0000 7f8207400700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f81ec06c7d0 0x7f81ec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:45.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.802+0000 7f8207400700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f81ec06c7d0 0x7f81ec06ec80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009fd0 tx=0x7f81f800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:45.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:45.806+0000 7f81ff400700 1 -- 172.21.15.63:0/552631749 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f81f405a930 con 0x7f8208071f90 2024-03-23T11:58:46.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.041+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8208110480 con 0x7f81ec06c7d0 2024-03-23T11:58:46.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.043+0000 7f81ff400700 1 -- 172.21.15.63:0/552631749 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f81f800f040 con 0x7f81ec06c7d0 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f81ec06c7d0 msgr2=0x7f81ec06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f81ec06c7d0 0x7f81ec06ec80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009fd0 tx=0x7f81f800b040 comp rx=0 tx=0).stop 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8208071f90 msgr2=0x7f820811a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8208071f90 0x7f820811a5f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f81f4004f50 tx=0x7f81f4009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 shutdown_connections 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f81ec06c7d0 0x7f81ec06ec80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82080708a0 0x7f820811a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 --2- 172.21.15.63:0/552631749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8208071f90 0x7f820811a5f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 >> 172.21.15.63:0/552631749 conn(0x7f82080f89b0 msgr2=0x7f82080f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:46.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 shutdown_connections 2024-03-23T11:58:46.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.045+0000 7f820e600700 1 -- 172.21.15.63:0/552631749 wait complete. 2024-03-23T11:58:46.043 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:58:46.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 -- 172.21.15.63:0/787553503 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 msgr2=0x7fba0c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 --2- 172.21.15.63:0/787553503 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c070730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc004970 tx=0x7fb9fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:46.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 -- 172.21.15.63:0/787553503 shutdown_connections 2024-03-23T11:58:46.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 --2- 172.21.15.63:0/787553503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c070c70 0x7fba0c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 --2- 172.21.15.63:0/787553503 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c070730 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 -- 172.21.15.63:0/787553503 >> 172.21.15.63:0/787553503 conn(0x7fba0c0f89b0 msgr2=0x7fba0c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:46.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 -- 172.21.15.63:0/787553503 shutdown_connections 2024-03-23T11:58:46.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.184+0000 7fba11e00700 1 -- 172.21.15.63:0/787553503 wait complete. 2024-03-23T11:58:46.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.185+0000 7fba11e00700 1 Processor -- start 2024-03-23T11:58:46.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.185+0000 7fba11e00700 1 -- start start 2024-03-23T11:58:46.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.185+0000 7fba11e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.185+0000 7fba11e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c070c70 0x7fba0c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.185+0000 7fba11e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c116810 con 0x7fba0c070360 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.185+0000 7fba11e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c10fbe0 con 0x7fba0c070c70 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba0aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba0aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35366/0 (socket says 172.21.15.63:35366) 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba0aa00700 1 -- 172.21.15.63:0/3917200972 learned_addr learned my addr 172.21.15.63:0/3917200972 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba0aa00700 1 -- 172.21.15.63:0/3917200972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c070c70 msgr2=0x7fba0c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba0aa00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c070c70 0x7fba0c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba0aa00700 1 -- 172.21.15.63:0/3917200972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc005040 con 0x7fba0c070360 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba0aa00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c115c80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009360 tx=0x7fb9fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.186+0000 7fba02a00700 1 -- 172.21.15.63:0/3917200972 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9fc00e040 con 0x7fba0c070360 2024-03-23T11:58:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.187+0000 7fba02a00700 1 -- 172.21.15.63:0/3917200972 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9fc004290 con 0x7fba0c070360 2024-03-23T11:58:46.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.187+0000 7fba02a00700 1 -- 172.21.15.63:0/3917200972 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9fc01c3f0 con 0x7fba0c070360 2024-03-23T11:58:46.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.187+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c10fe60 con 0x7fba0c070360 2024-03-23T11:58:46.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.187+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c1103a0 con 0x7fba0c070360 2024-03-23T11:58:46.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.188+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c103b90 con 0x7fba0c070360 2024-03-23T11:58:46.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.189+0000 7fba02a00700 1 -- 172.21.15.63:0/3917200972 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb9fc013420 con 0x7fba0c070360 2024-03-23T11:58:46.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.190+0000 7fba02a00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f006c7d0 0x7fb9f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.190+0000 7fba02a00700 1 -- 172.21.15.63:0/3917200972 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb9fc08ce60 con 0x7fba0c070360 2024-03-23T11:58:46.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.190+0000 7fba0a000700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f006c7d0 0x7fb9f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:46.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.191+0000 7fba0a000700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f006c7d0 0x7fb9f006ec80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fba0c111520 tx=0x7fb9f8007cc0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:46.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.194+0000 7fba02a00700 1 -- 172.21.15.63:0/3917200972 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb9fc05ab70 con 0x7fba0c070360 2024-03-23T11:58:46.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.426+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba0c10bfb0 con 0x7fb9f006c7d0 2024-03-23T11:58:46.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.427+0000 7fba02a00700 1 -- 172.21.15.63:0/3917200972 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fba0c10bfb0 con 0x7fb9f006c7d0 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f006c7d0 msgr2=0x7fb9f006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f006c7d0 0x7fb9f006ec80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fba0c111520 tx=0x7fb9f8007cc0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 msgr2=0x7fba0c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c115c80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009360 tx=0x7fb9fc009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 shutdown_connections 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f006c7d0 0x7fb9f006ec80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c070360 0x7fba0c115c80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 --2- 172.21.15.63:0/3917200972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c070c70 0x7fba0c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 >> 172.21.15.63:0/3917200972 conn(0x7fba0c0f89b0 msgr2=0x7fba0c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:46.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 shutdown_connections 2024-03-23T11:58:46.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.430+0000 7fba11e00700 1 -- 172.21.15.63:0/3917200972 wait complete. 2024-03-23T11:58:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 -- 172.21.15.63:0/1192911316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 msgr2=0x7f15301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 --2- 172.21.15.63:0/1192911316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15301031d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1524004940 tx=0x7f1524009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 -- 172.21.15.63:0/1192911316 shutdown_connections 2024-03-23T11:58:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 --2- 172.21.15.63:0/1192911316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15301037a0 0x7f15300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 --2- 172.21.15.63:0/1192911316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15301031d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 -- 172.21.15.63:0/1192911316 >> 172.21.15.63:0/1192911316 conn(0x7f15300f89b0 msgr2=0x7f15300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 -- 172.21.15.63:0/1192911316 shutdown_connections 2024-03-23T11:58:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.567+0000 7f1538200700 1 -- 172.21.15.63:0/1192911316 wait complete. 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.568+0000 7f1538200700 1 Processor -- start 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.568+0000 7f1538200700 1 -- start start 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1538200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1538200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15301037a0 0x7f1530070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1538200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1530070d90 con 0x7f1530102e00 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1538200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1530070ed0 con 0x7f15301037a0 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1535800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1535800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15300702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35394/0 (socket says 172.21.15.63:35394) 2024-03-23T11:58:46.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1535800700 1 -- 172.21.15.63:0/3694161232 learned_addr learned my addr 172.21.15.63:0/3694161232 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1535800700 1 -- 172.21.15.63:0/3694161232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15301037a0 msgr2=0x7f1530070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1535800700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15301037a0 0x7f1530070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.569+0000 7f1535800700 1 -- 172.21.15.63:0/3694161232 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1524005040 con 0x7f1530102e00 2024-03-23T11:58:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.570+0000 7f1535800700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15300702c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1524009390 tx=0x7f1524009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.570+0000 7f152d600700 1 -- 172.21.15.63:0/3694161232 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152400e040 con 0x7f1530102e00 2024-03-23T11:58:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.570+0000 7f152d600700 1 -- 172.21.15.63:0/3694161232 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1524013420 con 0x7f1530102e00 2024-03-23T11:58:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.570+0000 7f152d600700 1 -- 172.21.15.63:0/3694161232 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152401c630 con 0x7f1530102e00 2024-03-23T11:58:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.570+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1530071120 con 0x7f1530102e00 2024-03-23T11:58:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.570+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153011a250 con 0x7f1530102e00 2024-03-23T11:58:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.571+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15300fe550 con 0x7f1530102e00 2024-03-23T11:58:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.572+0000 7f152d600700 1 -- 172.21.15.63:0/3694161232 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f15240139e0 con 0x7f1530102e00 2024-03-23T11:58:46.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.573+0000 7f152d600700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f151806c7d0 0x7f151806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.573+0000 7f152d600700 1 -- 172.21.15.63:0/3694161232 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f152408cbf0 con 0x7f1530102e00 2024-03-23T11:58:46.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.573+0000 7f1534e00700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f151806c7d0 0x7f151806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:46.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.574+0000 7f1534e00700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f151806c7d0 0x7f151806ec80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1530071c00 tx=0x7f151c009ce0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:46.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.577+0000 7f152d600700 1 -- 172.21.15.63:0/3694161232 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1524056fe0 con 0x7f1530102e00 2024-03-23T11:58:46.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:46 smithi063 ceph-mon[141124]: pgmap v229: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:46.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:46 smithi111 ceph-mon[106899]: pgmap v229: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:46.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.800+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f15300fe550 con 0x7f151806c7d0 2024-03-23T11:58:46.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.811+0000 7f152d600700 1 -- 172.21.15.63:0/3694161232 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f15300fe550 con 0x7f151806c7d0 2024-03-23T11:58:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (8m) 43s ago 21m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (5m) 43s ago 21m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (5m) 25s ago 20m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (7m) 43s ago 20m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (69s) 43s ago 14m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (49s) 43s ago 14m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (14m) 25s ago 14m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (30s) 25s ago 14m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (10m) 43s ago 22m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (9m) 25s ago 20m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (6m) 43s ago 22m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (6m) 25s ago 20m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:58:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (8m) 43s ago 21m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (8m) 25s ago 20m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (4m) 43s ago 19m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (4m) 43s ago 18m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (3m) 43s ago 18m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (2m) 25s ago 17m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (2m) 25s ago 17m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (91s) 25s ago 16m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T11:58:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (8m) 43s ago 20m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f151806c7d0 msgr2=0x7f151806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f151806c7d0 0x7f151806ec80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1530071c00 tx=0x7f151c009ce0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 msgr2=0x7f15300702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15300702c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1524009390 tx=0x7f1524009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 shutdown_connections 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f151806c7d0 0x7f151806ec80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1530102e00 0x7f15300702c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 --2- 172.21.15.63:0/3694161232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15301037a0 0x7f1530070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 >> 172.21.15.63:0/3694161232 conn(0x7f15300f89b0 msgr2=0x7f15300f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:46.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 shutdown_connections 2024-03-23T11:58:46.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.813+0000 7f1538200700 1 -- 172.21.15.63:0/3694161232 wait complete. 2024-03-23T11:58:46.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.948+0000 7f556a200700 1 -- 172.21.15.63:0/2234035105 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 msgr2=0x7f55641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.948+0000 7f556a200700 1 --2- 172.21.15.63:0/2234035105 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f55641031d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5554004940 tx=0x7f5554009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.948+0000 7f556a200700 1 -- 172.21.15.63:0/2234035105 shutdown_connections 2024-03-23T11:58:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.948+0000 7f556a200700 1 --2- 172.21.15.63:0/2234035105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55641037a0 0x7f55640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.948+0000 7f556a200700 1 --2- 172.21.15.63:0/2234035105 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f55641031d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.948+0000 7f556a200700 1 -- 172.21.15.63:0/2234035105 >> 172.21.15.63:0/2234035105 conn(0x7f55640f89b0 msgr2=0x7f55640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.948+0000 7f556a200700 1 -- 172.21.15.63:0/2234035105 shutdown_connections 2024-03-23T11:58:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.949+0000 7f556a200700 1 -- 172.21.15.63:0/2234035105 wait complete. 2024-03-23T11:58:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.949+0000 7f556a200700 1 Processor -- start 2024-03-23T11:58:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f556a200700 1 -- start start 2024-03-23T11:58:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f556a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f5564115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f556a200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55641037a0 0x7f5564116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f556a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55641167d0 con 0x7f5564102e00 2024-03-23T11:58:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f556a200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556410fba0 con 0x7f55641037a0 2024-03-23T11:58:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f5563400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f5564115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f5563400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f5564115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35418/0 (socket says 172.21.15.63:35418) 2024-03-23T11:58:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f5563400700 1 -- 172.21.15.63:0/2092637044 learned_addr learned my addr 172.21.15.63:0/2092637044 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f5563400700 1 -- 172.21.15.63:0/2092637044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55641037a0 msgr2=0x7f5564116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.950+0000 7f5563400700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55641037a0 0x7f5564116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.951+0000 7f5563400700 1 -- 172.21.15.63:0/2092637044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5554005040 con 0x7f5564102e00 2024-03-23T11:58:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.951+0000 7f5563400700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f5564115c40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5554008d80 tx=0x7f5554009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.951+0000 7f555b400700 1 -- 172.21.15.63:0/2092637044 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f555400e030 con 0x7f5564102e00 2024-03-23T11:58:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.951+0000 7f555b400700 1 -- 172.21.15.63:0/2092637044 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5554013de0 con 0x7f5564102e00 2024-03-23T11:58:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.951+0000 7f555b400700 1 -- 172.21.15.63:0/2092637044 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f555401cd60 con 0x7f5564102e00 2024-03-23T11:58:46.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.951+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556410fe20 con 0x7f5564102e00 2024-03-23T11:58:46.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.951+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55641103e0 con 0x7f5564102e00 2024-03-23T11:58:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.952+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55640fe550 con 0x7f5564102e00 2024-03-23T11:58:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.954+0000 7f555b400700 1 -- 172.21.15.63:0/2092637044 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f555401c590 con 0x7f5564102e00 2024-03-23T11:58:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.954+0000 7f555b400700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f554806c7d0 0x7f554806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.954+0000 7f555b400700 1 -- 172.21.15.63:0/2092637044 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f555408d230 con 0x7f5564102e00 2024-03-23T11:58:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.955+0000 7f5562a00700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f554806c7d0 0x7f554806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.955+0000 7f5562a00700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f554806c7d0 0x7f554806ec80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f55641114e0 tx=0x7f555000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:46.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:46.959+0000 7f555b400700 1 -- 172.21.15.63:0/2092637044 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f555405b000 con 0x7f5564102e00 2024-03-23T11:58:47.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.257+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5564109e10 con 0x7f5564102e00 2024-03-23T11:58:47.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.257+0000 7f555b400700 1 -- 172.21.15.63:0/2092637044 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f555405ab90 con 0x7f5564102e00 2024-03-23T11:58:47.240 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:58:47.240 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:58:47.241 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:58:47.242 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T11:58:47.242 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:58:47.242 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:58:47.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.259+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f554806c7d0 msgr2=0x7f554806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.259+0000 7f556a200700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f554806c7d0 0x7f554806ec80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f55641114e0 tx=0x7f555000b040 comp rx=0 tx=0).stop 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 msgr2=0x7f5564115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f5564115c40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5554008d80 tx=0x7f5554009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 shutdown_connections 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f554806c7d0 0x7f554806ec80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5564102e00 0x7f5564115c40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 --2- 172.21.15.63:0/2092637044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55641037a0 0x7f5564116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 >> 172.21.15.63:0/2092637044 conn(0x7f55640f89b0 msgr2=0x7f55640f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 shutdown_connections 2024-03-23T11:58:47.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.260+0000 7f556a200700 1 -- 172.21.15.63:0/2092637044 wait complete. 2024-03-23T11:58:47.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 -- 172.21.15.63:0/3569865756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4318102e00 msgr2=0x7f43181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 --2- 172.21.15.63:0/3569865756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4318102e00 0x7f43181031d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f430c004940 tx=0x7f430c009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:47.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 -- 172.21.15.63:0/3569865756 shutdown_connections 2024-03-23T11:58:47.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 --2- 172.21.15.63:0/3569865756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43181037a0 0x7f43180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 --2- 172.21.15.63:0/3569865756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4318102e00 0x7f43181031d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 -- 172.21.15.63:0/3569865756 >> 172.21.15.63:0/3569865756 conn(0x7f43180f89b0 msgr2=0x7f43180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:47.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 -- 172.21.15.63:0/3569865756 shutdown_connections 2024-03-23T11:58:47.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.398+0000 7f431fe00700 1 -- 172.21.15.63:0/3569865756 wait complete. 2024-03-23T11:58:47.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.399+0000 7f431fe00700 1 Processor -- start 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.399+0000 7f431fe00700 1 -- start start 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.399+0000 7f431fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4318102e00 0x7f4318115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.399+0000 7f431fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43181037a0 0x7f4318116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.399+0000 7f431fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4318116880 con 0x7f43181037a0 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.399+0000 7f431fe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431810fc50 con 0x7f4318102e00 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43181037a0 0x7f4318116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43181037a0 0x7f4318116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35424/0 (socket says 172.21.15.63:35424) 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431ca00700 1 -- 172.21.15.63:0/3052252960 learned_addr learned my addr 172.21.15.63:0/3052252960 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431d400700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4318102e00 0x7f4318115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431ca00700 1 -- 172.21.15.63:0/3052252960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4318102e00 msgr2=0x7f4318115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431ca00700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4318102e00 0x7f4318115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431ca00700 1 -- 172.21.15.63:0/3052252960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c005040 con 0x7f43181037a0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.400+0000 7f431ca00700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43181037a0 0x7f4318116230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f430400c800 tx=0x7f4304004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.401+0000 7f4315600700 1 -- 172.21.15.63:0/3052252960 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43040106f0 con 0x7f43181037a0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.401+0000 7f4315600700 1 -- 172.21.15.63:0/3052252960 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4304010d00 con 0x7f43181037a0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.401+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431810ff30 con 0x7f43181037a0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.401+0000 7f4315600700 1 -- 172.21.15.63:0/3052252960 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43040113a0 con 0x7f43181037a0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.401+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43181104f0 con 0x7f43181037a0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.402+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43180fe550 con 0x7f43181037a0 2024-03-23T11:58:47.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.403+0000 7f4315600700 1 -- 172.21.15.63:0/3052252960 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4304012400 con 0x7f43181037a0 2024-03-23T11:58:47.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.404+0000 7f4315600700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f430006c7d0 0x7f430006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:47.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.404+0000 7f4315600700 1 -- 172.21.15.63:0/3052252960 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f430408bad0 con 0x7f43181037a0 2024-03-23T11:58:47.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.404+0000 7f431d400700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f430006c7d0 0x7f430006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:47.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.405+0000 7f431d400700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f430006c7d0 0x7f430006ec80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f430c009fd0 tx=0x7f430c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:47.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.409+0000 7f4315600700 1 -- 172.21.15.63:0/3052252960 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4304059760 con 0x7f43181037a0 2024-03-23T11:58:47.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:47 smithi063 ceph-mon[141124]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:47.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:47 smithi063 ceph-mon[141124]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:47.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:47 smithi063 ceph-mon[141124]: from='client.34498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:47.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:47 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2092637044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:47.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:47 smithi111 ceph-mon[106899]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:47.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:47 smithi111 ceph-mon[106899]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:47.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:47 smithi111 ceph-mon[106899]: from='client.34498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:47.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:47 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2092637044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:58:47.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.668+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4318109e20 con 0x7f43181037a0 2024-03-23T11:58:47.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.669+0000 7f4315600700 1 -- 172.21.15.63:0/3052252960 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f43040592f0 con 0x7f43181037a0 2024-03-23T11:58:47.652 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T11:58:47.652 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:58:47.652 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:58:47.652 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:58:47.653 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:58:47.654 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:58:47.655 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:58:47.656 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:58:47.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f430006c7d0 msgr2=0x7f430006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f430006c7d0 0x7f430006ec80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f430c009fd0 tx=0x7f430c009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:47.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43181037a0 msgr2=0x7f4318116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43181037a0 0x7f4318116230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f430400c800 tx=0x7f4304004470 comp rx=0 tx=0).stop 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 shutdown_connections 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f430006c7d0 0x7f430006ec80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4318102e00 0x7f4318115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 --2- 172.21.15.63:0/3052252960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43181037a0 0x7f4318116230 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 >> 172.21.15.63:0/3052252960 conn(0x7f43180f89b0 msgr2=0x7f43180706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.671+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 shutdown_connections 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.672+0000 7f431fe00700 1 -- 172.21.15.63:0/3052252960 wait complete. 2024-03-23T11:58:47.657 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.815+0000 7f3724800700 1 -- 172.21.15.63:0/3811142886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 msgr2=0x7f371c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.815+0000 7f3724800700 1 --2- 172.21.15.63:0/3811142886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c1031d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3710004940 tx=0x7f3710009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.816+0000 7f3724800700 1 -- 172.21.15.63:0/3811142886 shutdown_connections 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.816+0000 7f3724800700 1 --2- 172.21.15.63:0/3811142886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f371c1037a0 0x7f371c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.816+0000 7f3724800700 1 --2- 172.21.15.63:0/3811142886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c1031d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.816+0000 7f3724800700 1 -- 172.21.15.63:0/3811142886 >> 172.21.15.63:0/3811142886 conn(0x7f371c0f89b0 msgr2=0x7f371c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.816+0000 7f3724800700 1 -- 172.21.15.63:0/3811142886 shutdown_connections 2024-03-23T11:58:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.816+0000 7f3724800700 1 -- 172.21.15.63:0/3811142886 wait complete. 2024-03-23T11:58:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.816+0000 7f3724800700 1 Processor -- start 2024-03-23T11:58:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3724800700 1 -- start start 2024-03-23T11:58:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3724800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3724800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f371c1037a0 0x7f371c113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3724800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c1145d0 con 0x7f371c102e00 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3724800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c10d9c0 con 0x7f371c1037a0 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3721e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3721e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35452/0 (socket says 172.21.15.63:35452) 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.817+0000 7f3721e00700 1 -- 172.21.15.63:0/2663040628 learned_addr learned my addr 172.21.15.63:0/2663040628 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3721400700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f371c1037a0 0x7f371c113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3721e00700 1 -- 172.21.15.63:0/2663040628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f371c1037a0 msgr2=0x7f371c113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3721e00700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f371c1037a0 0x7f371c113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3721e00700 1 -- 172.21.15.63:0/2663040628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710005040 con 0x7f371c102e00 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3721400700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f371c1037a0 0x7f371c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3721e00700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c113a40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3710008d80 tx=0x7f3710009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3716000700 1 -- 172.21.15.63:0/2663040628 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371000e030 con 0x7f371c102e00 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3716000700 1 -- 172.21.15.63:0/2663040628 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3710013de0 con 0x7f371c102e00 2024-03-23T11:58:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3716000700 1 -- 172.21.15.63:0/2663040628 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371001cd60 con 0x7f371c102e00 2024-03-23T11:58:47.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.818+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371c10dc40 con 0x7f371c102e00 2024-03-23T11:58:47.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.819+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371c10e200 con 0x7f371c102e00 2024-03-23T11:58:47.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.820+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371c0fe550 con 0x7f371c102e00 2024-03-23T11:58:47.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.820+0000 7f3716000700 1 -- 172.21.15.63:0/2663040628 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f371001c590 con 0x7f371c102e00 2024-03-23T11:58:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.822+0000 7f3716000700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370406c7d0 0x7f370406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:47.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.822+0000 7f3716000700 1 -- 172.21.15.63:0/2663040628 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f371008d2f0 con 0x7f371c102e00 2024-03-23T11:58:47.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.822+0000 7f3721400700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370406c7d0 0x7f370406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:47.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.823+0000 7f3721400700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370406c7d0 0x7f370406ec80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f370c004600 tx=0x7f370c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:47.827+0000 7f3716000700 1 -- 172.21.15.63:0/2663040628 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f371005b000 con 0x7f371c102e00 2024-03-23T11:58:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.059+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f371c109e30 con 0x7f370406c7d0 2024-03-23T11:58:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.061+0000 7f3716000700 1 -- 172.21.15.63:0/2663040628 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f371c109e30 con 0x7f370406c7d0 2024-03-23T11:58:48.043 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:58:48.043 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:58:48.043 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:58:48.044 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370406c7d0 msgr2=0x7f370406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370406c7d0 0x7f370406ec80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f370c004600 tx=0x7f370c00b040 comp rx=0 tx=0).stop 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 msgr2=0x7f371c113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c113a40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3710008d80 tx=0x7f3710009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 shutdown_connections 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370406c7d0 0x7f370406ec80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f371c102e00 0x7f371c113a40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 --2- 172.21.15.63:0/2663040628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f371c1037a0 0x7f371c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 >> 172.21.15.63:0/2663040628 conn(0x7f371c0f89b0 msgr2=0x7f371c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 shutdown_connections 2024-03-23T11:58:48.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.063+0000 7f3724800700 1 -- 172.21.15.63:0/2663040628 wait complete. 2024-03-23T11:58:48.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 -- 172.21.15.63:0/1550897380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 msgr2=0x7fbaf8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:48.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 --2- 172.21.15.63:0/1550897380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf8106f00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbae0004940 tx=0x7fbae0009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:48.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 -- 172.21.15.63:0/1550897380 shutdown_connections 2024-03-23T11:58:48.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 --2- 172.21.15.63:0/1550897380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf8106f00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 --2- 172.21.15.63:0/1550897380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaf80fde90 0x7fbaf80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 -- 172.21.15.63:0/1550897380 >> 172.21.15.63:0/1550897380 conn(0x7fbaf8072bc0 msgr2=0x7fbaf8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:48.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 -- 172.21.15.63:0/1550897380 shutdown_connections 2024-03-23T11:58:48.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.200+0000 7fbafca00700 1 -- 172.21.15.63:0/1550897380 wait complete. 2024-03-23T11:58:48.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.201+0000 7fbafca00700 1 Processor -- start 2024-03-23T11:58:48.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.201+0000 7fbafca00700 1 -- start start 2024-03-23T11:58:48.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbafca00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaf80fde90 0x7fbaf8115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:48.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbafca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf81161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:48.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbafca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf8116800 con 0x7fbaf80fe830 2024-03-23T11:58:48.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbafca00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf810fbd0 con 0x7fbaf80fde90 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaefe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf81161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaefe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf81161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35484/0 (socket says 172.21.15.63:35484) 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaefe00700 1 -- 172.21.15.63:0/4140044327 learned_addr learned my addr 172.21.15.63:0/4140044327 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaf5e00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaf80fde90 0x7fbaf8115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaefe00700 1 -- 172.21.15.63:0/4140044327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaf80fde90 msgr2=0x7fbaf8115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaefe00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaf80fde90 0x7fbaf8115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaefe00700 1 -- 172.21.15.63:0/4140044327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0005040 con 0x7fbaf80fe830 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.202+0000 7fbaf5e00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaf80fde90 0x7fbaf8115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.203+0000 7fbaefe00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf81161b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000c00 tx=0x7fbae0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:48.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.203+0000 7fbaee000700 1 -- 172.21.15.63:0/4140044327 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbae0021070 con 0x7fbaf80fe830 2024-03-23T11:58:48.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.203+0000 7fbaee000700 1 -- 172.21.15.63:0/4140044327 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbae0013e00 con 0x7fbaf80fe830 2024-03-23T11:58:48.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.203+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf810fe50 con 0x7fbaf80fe830 2024-03-23T11:58:48.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.203+0000 7fbaee000700 1 -- 172.21.15.63:0/4140044327 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbae000e070 con 0x7fbaf80fe830 2024-03-23T11:58:48.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.203+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf8110390 con 0x7fbaf80fe830 2024-03-23T11:58:48.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.204+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf8106a20 con 0x7fbaf80fe830 2024-03-23T11:58:48.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.205+0000 7fbaee000700 1 -- 172.21.15.63:0/4140044327 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbae0013950 con 0x7fbaf80fe830 2024-03-23T11:58:48.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.206+0000 7fbaee000700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbadc06c7d0 0x7fbadc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:58:48.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.206+0000 7fbaee000700 1 -- 172.21.15.63:0/4140044327 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbae008cb60 con 0x7fbaf80fe830 2024-03-23T11:58:48.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.206+0000 7fbaf5e00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbadc06c7d0 0x7fbadc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:58:48.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.207+0000 7fbaf5e00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbadc06c7d0 0x7fbadc06ec80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbae8004600 tx=0x7fbae800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:58:48.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.210+0000 7fbaee000700 1 -- 172.21.15.63:0/4140044327 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbae005a7f0 con 0x7fbaf80fe830 2024-03-23T11:58:48.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.507+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbaf81112c0 con 0x7fbaf80fe830 2024-03-23T11:58:48.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.507+0000 7fbaee000700 1 -- 172.21.15.63:0/4140044327 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fbae005a380 con 0x7fbaf80fe830 2024-03-23T11:58:48.491 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T11:58:48.491 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T11:58:48.491 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T11:58:48.491 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T11:58:48.491 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbadc06c7d0 msgr2=0x7fbadc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbadc06c7d0 0x7fbadc06ec80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbae8004600 tx=0x7fbae800b040 comp rx=0 tx=0).stop 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 msgr2=0x7fbaf81161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf81161b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000c00 tx=0x7fbae0009e30 comp rx=0 tx=0).stop 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 shutdown_connections 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbadc06c7d0 0x7fbadc06ec80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaf80fde90 0x7fbaf8115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 --2- 172.21.15.63:0/4140044327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbaf80fe830 0x7fbaf81161b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 >> 172.21.15.63:0/4140044327 conn(0x7fbaf8072bc0 msgr2=0x7fbaf80fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:58:48.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 shutdown_connections 2024-03-23T11:58:48.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:58:48.510+0000 7fbafca00700 1 -- 172.21.15.63:0/4140044327 wait complete. 2024-03-23T11:58:48.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:48 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3052252960' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:58:48.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:48 smithi063 ceph-mon[141124]: pgmap v230: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:48.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:48 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3052252960' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:58:48.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:48 smithi111 ceph-mon[106899]: pgmap v230: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:49.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:49 smithi063 ceph-mon[141124]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:49.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:49 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4140044327' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:58:49.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:49 smithi111 ceph-mon[106899]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:58:49.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:49 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4140044327' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:58:50.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:50 smithi063 ceph-mon[141124]: pgmap v231: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:50.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:50 smithi111 ceph-mon[106899]: pgmap v231: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:53.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:52 smithi063 ceph-mon[141124]: pgmap v232: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:53.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:52 smithi111 ceph-mon[106899]: pgmap v232: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:55.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:54 smithi063 ceph-mon[141124]: pgmap v233: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:55.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:54 smithi111 ceph-mon[106899]: pgmap v233: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:56.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:56.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:56 smithi063 ceph-mon[141124]: pgmap v234: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:56.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:58:56.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:56 smithi111 ceph-mon[106899]: pgmap v234: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:59.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:58:58 smithi063 ceph-mon[141124]: pgmap v235: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:58:59.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:58:58 smithi111 ceph-mon[106899]: pgmap v235: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:00 smithi063 ceph-mon[141124]: pgmap v236: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:00 smithi111 ceph-mon[106899]: pgmap v236: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:03.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:02 smithi063 ceph-mon[141124]: pgmap v237: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:03.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:02 smithi111 ceph-mon[106899]: pgmap v237: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:05.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:04 smithi063 ceph-mon[141124]: pgmap v238: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:05.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:04 smithi111 ceph-mon[106899]: pgmap v238: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:06.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:06 smithi063 ceph-mon[141124]: pgmap v239: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:06.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:06 smithi111 ceph-mon[106899]: pgmap v239: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:08 smithi063 ceph-mon[141124]: pgmap v240: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:08 smithi111 ceph-mon[106899]: pgmap v240: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:10 smithi063 ceph-mon[141124]: pgmap v241: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:10 smithi111 ceph-mon[106899]: pgmap v241: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:13.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:12 smithi063 ceph-mon[141124]: pgmap v242: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:13.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:12 smithi111 ceph-mon[106899]: pgmap v242: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:15.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:14 smithi063 ceph-mon[141124]: pgmap v243: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:15.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:14 smithi111 ceph-mon[106899]: pgmap v243: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:16.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:16 smithi063 ceph-mon[141124]: pgmap v244: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:16.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:16 smithi111 ceph-mon[106899]: pgmap v244: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:18.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 -- 172.21.15.63:0/566532723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 msgr2=0x7fe5a8070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:18.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 --2- 172.21.15.63:0/566532723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8070730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe598004970 tx=0x7fe598009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:18.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 -- 172.21.15.63:0/566532723 shutdown_connections 2024-03-23T11:59:18.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 --2- 172.21.15.63:0/566532723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a8070c70 0x7fe5a8109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:18.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 --2- 172.21.15.63:0/566532723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8070730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:18.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 -- 172.21.15.63:0/566532723 >> 172.21.15.63:0/566532723 conn(0x7fe5a80f89b0 msgr2=0x7fe5a80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:18.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 -- 172.21.15.63:0/566532723 shutdown_connections 2024-03-23T11:59:18.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.666+0000 7fe5acc00700 1 -- 172.21.15.63:0/566532723 wait complete. 2024-03-23T11:59:18.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.667+0000 7fe5acc00700 1 Processor -- start 2024-03-23T11:59:18.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.667+0000 7fe5acc00700 1 -- start start 2024-03-23T11:59:18.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5acc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:18.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5acc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a8070c70 0x7fe5a81161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:18.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5acc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8116810 con 0x7fe5a8070360 2024-03-23T11:59:18.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5acc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a810fbe0 con 0x7fe5a8070c70 2024-03-23T11:59:18.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44828/0 (socket says 172.21.15.63:44828) 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5a5e00700 1 -- 172.21.15.63:0/4167920423 learned_addr learned my addr 172.21.15.63:0/4167920423 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe59fe00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a8070c70 0x7fe5a81161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5a5e00700 1 -- 172.21.15.63:0/4167920423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a8070c70 msgr2=0x7fe5a81161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5a5e00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a8070c70 0x7fe5a81161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe5a5e00700 1 -- 172.21.15.63:0/4167920423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe598005040 con 0x7fe5a8070360 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.668+0000 7fe59fe00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a8070c70 0x7fe5a81161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:59:18.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.669+0000 7fe5a5e00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8115c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe598009360 tx=0x7fe598009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:18.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.669+0000 7fe59e000700 1 -- 172.21.15.63:0/4167920423 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe59800e040 con 0x7fe5a8070360 2024-03-23T11:59:18.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.669+0000 7fe59e000700 1 -- 172.21.15.63:0/4167920423 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe598004290 con 0x7fe5a8070360 2024-03-23T11:59:18.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.669+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a810fe60 con 0x7fe5a8070360 2024-03-23T11:59:18.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.669+0000 7fe59e000700 1 -- 172.21.15.63:0/4167920423 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe59801c3f0 con 0x7fe5a8070360 2024-03-23T11:59:18.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.669+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a81103a0 con 0x7fe5a8070360 2024-03-23T11:59:18.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.670+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a8103b90 con 0x7fe5a8070360 2024-03-23T11:59:18.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.671+0000 7fe59e000700 1 -- 172.21.15.63:0/4167920423 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe598013420 con 0x7fe5a8070360 2024-03-23T11:59:18.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.672+0000 7fe59e000700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe58c06c7d0 0x7fe58c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:18.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.672+0000 7fe59e000700 1 -- 172.21.15.63:0/4167920423 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe59808cda0 con 0x7fe5a8070360 2024-03-23T11:59:18.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.672+0000 7fe59fe00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe58c06c7d0 0x7fe58c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:18.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.673+0000 7fe59fe00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe58c06c7d0 0x7fe58c06ec80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe590009fd0 tx=0x7fe59000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:18.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.677+0000 7fe59e000700 1 -- 172.21.15.63:0/4167920423 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe59805ab70 con 0x7fe5a8070360 2024-03-23T11:59:18.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.910+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5a810bfb0 con 0x7fe58c06c7d0 2024-03-23T11:59:18.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.911+0000 7fe59e000700 1 -- 172.21.15.63:0/4167920423 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe5a810bfb0 con 0x7fe58c06c7d0 2024-03-23T11:59:18.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.913+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe58c06c7d0 msgr2=0x7fe58c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.913+0000 7fe5acc00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe58c06c7d0 0x7fe58c06ec80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe590009fd0 tx=0x7fe59000b040 comp rx=0 tx=0).stop 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.913+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 msgr2=0x7fe5a8115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.913+0000 7fe5acc00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8115c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe598009360 tx=0x7fe598009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.914+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 shutdown_connections 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.914+0000 7fe5acc00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe58c06c7d0 0x7fe58c06ec80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.914+0000 7fe5acc00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5a8070360 0x7fe5a8115c80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.914+0000 7fe5acc00700 1 --2- 172.21.15.63:0/4167920423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a8070c70 0x7fe5a81161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.914+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 >> 172.21.15.63:0/4167920423 conn(0x7fe5a80f89b0 msgr2=0x7fe5a81015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.914+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 shutdown_connections 2024-03-23T11:59:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:18.914+0000 7fe5acc00700 1 -- 172.21.15.63:0/4167920423 wait complete. 2024-03-23T11:59:18.911 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.051+0000 7fb800800700 1 -- 172.21.15.63:0/2073905987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 msgr2=0x7fb7f80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.051+0000 7fb800800700 1 --2- 172.21.15.63:0/2073905987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f80fce10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8004940 tx=0x7fb7e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.051+0000 7fb800800700 1 -- 172.21.15.63:0/2073905987 shutdown_connections 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.052+0000 7fb800800700 1 --2- 172.21.15.63:0/2073905987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f80fce10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.052+0000 7fb800800700 1 --2- 172.21.15.63:0/2073905987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f8102e00 0x7fb7f81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.052+0000 7fb800800700 1 -- 172.21.15.63:0/2073905987 >> 172.21.15.63:0/2073905987 conn(0x7fb7f80f89b0 msgr2=0x7fb7f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.052+0000 7fb800800700 1 -- 172.21.15.63:0/2073905987 shutdown_connections 2024-03-23T11:59:19.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.052+0000 7fb800800700 1 -- 172.21.15.63:0/2073905987 wait complete. 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.052+0000 7fb800800700 1 Processor -- start 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.053+0000 7fb800800700 1 -- start start 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.053+0000 7fb800800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f8102e00 0x7fb7f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.053+0000 7fb800800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.053+0000 7fb800800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8070d90 con 0x7fb7f81037a0 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.053+0000 7fb800800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8070ed0 con 0x7fb7f8102e00 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44842/0 (socket says 172.21.15.63:44842) 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fd400700 1 -- 172.21.15.63:0/1325378537 learned_addr learned my addr 172.21.15.63:0/1325378537 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:19.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fde00700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f8102e00 0x7fb7f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fd400700 1 -- 172.21.15.63:0/1325378537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f8102e00 msgr2=0x7fb7f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fd400700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f8102e00 0x7fb7f80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fd400700 1 -- 172.21.15.63:0/1325378537 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e8005040 con 0x7fb7f81037a0 2024-03-23T11:59:19.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.054+0000 7fb7fd400700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f8070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009390 tx=0x7fb7e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:19.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.055+0000 7fb7f2000700 1 -- 172.21.15.63:0/1325378537 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7e800e040 con 0x7fb7f81037a0 2024-03-23T11:59:19.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.055+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f8071120 con 0x7fb7f81037a0 2024-03-23T11:59:19.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.055+0000 7fb7f2000700 1 -- 172.21.15.63:0/1325378537 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7e8013420 con 0x7fb7f81037a0 2024-03-23T11:59:19.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.055+0000 7fb7f2000700 1 -- 172.21.15.63:0/1325378537 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7e801c630 con 0x7fb7f81037a0 2024-03-23T11:59:19.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.055+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f811a230 con 0x7fb7f81037a0 2024-03-23T11:59:19.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.056+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f80fe550 con 0x7fb7f81037a0 2024-03-23T11:59:19.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.057+0000 7fb7f2000700 1 -- 172.21.15.63:0/1325378537 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb7e8013a00 con 0x7fb7f81037a0 2024-03-23T11:59:19.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.058+0000 7fb7f2000700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7e006c7d0 0x7fb7e006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.058+0000 7fb7f2000700 1 -- 172.21.15.63:0/1325378537 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb7e808ca60 con 0x7fb7f81037a0 2024-03-23T11:59:19.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.058+0000 7fb7fde00700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7e006c7d0 0x7fb7e006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.059+0000 7fb7fde00700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7e006c7d0 0x7fb7e006ec80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0041d0 tx=0x7fb7ec0043c0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:19.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.062+0000 7fb7f2000700 1 -- 172.21.15.63:0/1325378537 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7e8056da0 con 0x7fb7f81037a0 2024-03-23T11:59:19.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:18 smithi063 ceph-mon[141124]: pgmap v245: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:19.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:18 smithi111 ceph-mon[106899]: pgmap v245: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:19.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.295+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7f804bed0 con 0x7fb7e006c7d0 2024-03-23T11:59:19.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.296+0000 7fb7f2000700 1 -- 172.21.15.63:0/1325378537 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb7f804bed0 con 0x7fb7e006c7d0 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7e006c7d0 msgr2=0x7fb7e006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7e006c7d0 0x7fb7e006ec80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0041d0 tx=0x7fb7ec0043c0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 msgr2=0x7fb7f8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f8070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009390 tx=0x7fb7e8009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 shutdown_connections 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7e006c7d0 0x7fb7e006ec80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f8102e00 0x7fb7f80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 --2- 172.21.15.63:0/1325378537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7f81037a0 0x7fb7f8070800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 >> 172.21.15.63:0/1325378537 conn(0x7fb7f80f89b0 msgr2=0x7fb7f80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:19.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 shutdown_connections 2024-03-23T11:59:19.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.299+0000 7fb800800700 1 -- 172.21.15.63:0/1325378537 wait complete. 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 -- 172.21.15.63:0/3122435577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 msgr2=0x7f49601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 --2- 172.21.15.63:0/3122435577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f49601031d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4950004940 tx=0x7f4950009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 -- 172.21.15.63:0/3122435577 shutdown_connections 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 --2- 172.21.15.63:0/3122435577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49601037a0 0x7f49600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 --2- 172.21.15.63:0/3122435577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f49601031d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 -- 172.21.15.63:0/3122435577 >> 172.21.15.63:0/3122435577 conn(0x7f49600f89b0 msgr2=0x7f49600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 -- 172.21.15.63:0/3122435577 shutdown_connections 2024-03-23T11:59:19.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.436+0000 7f4966a00700 1 -- 172.21.15.63:0/3122435577 wait complete. 2024-03-23T11:59:19.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.437+0000 7f4966a00700 1 Processor -- start 2024-03-23T11:59:19.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.437+0000 7f4966a00700 1 -- start start 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.437+0000 7f4966a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f4960115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.437+0000 7f4966a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49601037a0 0x7f4960116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.437+0000 7f4966a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4960116860 con 0x7f4960102e00 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.437+0000 7f4966a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496010fc30 con 0x7f49601037a0 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.438+0000 7f495fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f4960115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.438+0000 7f495fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f4960115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44860/0 (socket says 172.21.15.63:44860) 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.438+0000 7f495fe00700 1 -- 172.21.15.63:0/1987503677 learned_addr learned my addr 172.21.15.63:0/1987503677 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:19.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.438+0000 7f495f400700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49601037a0 0x7f4960116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.438+0000 7f495fe00700 1 -- 172.21.15.63:0/1987503677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49601037a0 msgr2=0x7f4960116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.438+0000 7f495fe00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49601037a0 0x7f4960116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.438+0000 7f495fe00700 1 -- 172.21.15.63:0/1987503677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4950005040 con 0x7f4960102e00 2024-03-23T11:59:19.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.439+0000 7f495fe00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f4960115cd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4950009390 tx=0x7f4950009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:19.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.439+0000 7f4957e00700 1 -- 172.21.15.63:0/1987503677 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f495000e040 con 0x7f4960102e00 2024-03-23T11:59:19.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.439+0000 7f4957e00700 1 -- 172.21.15.63:0/1987503677 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4950013420 con 0x7f4960102e00 2024-03-23T11:59:19.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.439+0000 7f4957e00700 1 -- 172.21.15.63:0/1987503677 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f495001c630 con 0x7f4960102e00 2024-03-23T11:59:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.439+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496010feb0 con 0x7f4960102e00 2024-03-23T11:59:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.439+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4960110470 con 0x7f4960102e00 2024-03-23T11:59:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.440+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49600fe550 con 0x7f4960102e00 2024-03-23T11:59:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.441+0000 7f4957e00700 1 -- 172.21.15.63:0/1987503677 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f49500139e0 con 0x7f4960102e00 2024-03-23T11:59:19.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.442+0000 7f4957e00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f494406c7d0 0x7f494406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.442+0000 7f4957e00700 1 -- 172.21.15.63:0/1987503677 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f495008ce10 con 0x7f4960102e00 2024-03-23T11:59:19.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.442+0000 7f495f400700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f494406c7d0 0x7f494406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.443+0000 7f495f400700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f494406c7d0 0x7f494406ec80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4960111570 tx=0x7f494c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:19.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.447+0000 7f4957e00700 1 -- 172.21.15.63:0/1987503677 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f495005aaa0 con 0x7f4960102e00 2024-03-23T11:59:19.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.670+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f496004bf70 con 0x7f494406c7d0 2024-03-23T11:59:19.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.680+0000 7f4957e00700 1 -- 172.21.15.63:0/1987503677 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f494c00f040 con 0x7f494406c7d0 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (8m) 76s ago 21m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (5m) 76s ago 21m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (5m) 58s ago 20m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (8m) 76s ago 21m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (102s) 76s ago 15m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (82s) 76s ago 15m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (15m) 58s ago 15m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (62s) 58s ago 15m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T11:59:19.664 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (10m) 76s ago 23m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (9m) 58s ago 20m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (7m) 76s ago 23m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (6m) 58s ago 20m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (9m) 76s ago 21m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (9m) 58s ago 20m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (5m) 76s ago 19m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (4m) 76s ago 19m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (3m) 76s ago 18m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (3m) 58s ago 18m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (2m) 58s ago 17m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (2m) 58s ago 17m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T11:59:19.665 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (8m) 76s ago 21m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.682+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f494406c7d0 msgr2=0x7f494406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.682+0000 7f4966a00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f494406c7d0 0x7f494406ec80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4960111570 tx=0x7f494c00b040 comp rx=0 tx=0).stop 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.682+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 msgr2=0x7f4960115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.682+0000 7f4966a00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f4960115cd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4950009390 tx=0x7f4950009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.683+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 shutdown_connections 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.683+0000 7f4966a00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f494406c7d0 0x7f494406ec80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.683+0000 7f4966a00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4960102e00 0x7f4960115cd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.683+0000 7f4966a00700 1 --2- 172.21.15.63:0/1987503677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49601037a0 0x7f4960116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.683+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 >> 172.21.15.63:0/1987503677 conn(0x7f49600f89b0 msgr2=0x7f49600f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:19.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.683+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 shutdown_connections 2024-03-23T11:59:19.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.683+0000 7f4966a00700 1 -- 172.21.15.63:0/1987503677 wait complete. 2024-03-23T11:59:19.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.819+0000 7f86a3200700 1 -- 172.21.15.63:0/3500371845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 msgr2=0x7f86a40706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.819+0000 7f86a3200700 1 --2- 172.21.15.63:0/3500371845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a40706b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8694004940 tx=0x7f8694009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:19.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.820+0000 7f86a3200700 1 -- 172.21.15.63:0/3500371845 shutdown_connections 2024-03-23T11:59:19.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.820+0000 7f86a3200700 1 --2- 172.21.15.63:0/3500371845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86a4070bf0 0x7f86a4108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.820+0000 7f86a3200700 1 --2- 172.21.15.63:0/3500371845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a40706b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.820+0000 7f86a3200700 1 -- 172.21.15.63:0/3500371845 >> 172.21.15.63:0/3500371845 conn(0x7f86a40f87d0 msgr2=0x7f86a40fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:19.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.820+0000 7f86a3200700 1 -- 172.21.15.63:0/3500371845 shutdown_connections 2024-03-23T11:59:19.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.820+0000 7f86a3200700 1 -- 172.21.15.63:0/3500371845 wait complete. 2024-03-23T11:59:19.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.820+0000 7f86a3200700 1 Processor -- start 2024-03-23T11:59:19.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a3200700 1 -- start start 2024-03-23T11:59:19.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a3200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a4115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a3200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86a4070bf0 0x7f86a41160a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a3200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a41166f0 con 0x7f86a40702e0 2024-03-23T11:59:19.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a3200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a410fae0 con 0x7f86a4070bf0 2024-03-23T11:59:19.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a4115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a4115b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44876/0 (socket says 172.21.15.63:44876) 2024-03-23T11:59:19.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a1e00700 1 -- 172.21.15.63:0/1452523677 learned_addr learned my addr 172.21.15.63:0/1452523677 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:19.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.821+0000 7f86a1400700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86a4070bf0 0x7f86a41160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f86a1e00700 1 -- 172.21.15.63:0/1452523677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86a4070bf0 msgr2=0x7f86a41160a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:19.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f86a1e00700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86a4070bf0 0x7f86a41160a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:19.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f86a1e00700 1 -- 172.21.15.63:0/1452523677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694005040 con 0x7f86a40702e0 2024-03-23T11:59:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f86a1e00700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a4115b60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8694009390 tx=0x7f8694009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f869a000700 1 -- 172.21.15.63:0/1452523677 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869400e040 con 0x7f86a40702e0 2024-03-23T11:59:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f869a000700 1 -- 172.21.15.63:0/1452523677 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8694013420 con 0x7f86a40702e0 2024-03-23T11:59:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f869a000700 1 -- 172.21.15.63:0/1452523677 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869401c630 con 0x7f86a40702e0 2024-03-23T11:59:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a410fd60 con 0x7f86a40702e0 2024-03-23T11:59:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.822+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a41102a0 con 0x7f86a40702e0 2024-03-23T11:59:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.823+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a4071a20 con 0x7f86a40702e0 2024-03-23T11:59:19.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.824+0000 7f869a000700 1 -- 172.21.15.63:0/1452523677 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f86940139e0 con 0x7f86a40702e0 2024-03-23T11:59:19.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.825+0000 7f869a000700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f868806c7d0 0x7f868806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:19.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.825+0000 7f869a000700 1 -- 172.21.15.63:0/1452523677 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f869408ccc0 con 0x7f86a40702e0 2024-03-23T11:59:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.825+0000 7f86a1400700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f868806c7d0 0x7f868806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.826+0000 7f86a1400700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f868806c7d0 0x7f868806ec80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f86a4110ca0 tx=0x7f869000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:19.830+0000 7f869a000700 1 -- 172.21.15.63:0/1452523677 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f869405aa90 con 0x7f86a40702e0 2024-03-23T11:59:20.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.132+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f86a410be50 con 0x7f86a40702e0 2024-03-23T11:59:20.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.133+0000 7f869a000700 1 -- 172.21.15.63:0/1452523677 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f869405a620 con 0x7f86a40702e0 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:20.116 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:59:20.117 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:59:20.117 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T11:59:20.117 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:59:20.117 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f868806c7d0 msgr2=0x7f868806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f868806c7d0 0x7f868806ec80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f86a4110ca0 tx=0x7f869000b040 comp rx=0 tx=0).stop 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 msgr2=0x7f86a4115b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a4115b60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8694009390 tx=0x7f8694009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 shutdown_connections 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f868806c7d0 0x7f868806ec80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86a40702e0 0x7f86a4115b60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 --2- 172.21.15.63:0/1452523677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86a4070bf0 0x7f86a41160a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 >> 172.21.15.63:0/1452523677 conn(0x7f86a40f87d0 msgr2=0x7f86a40f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 shutdown_connections 2024-03-23T11:59:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.135+0000 7f86a3200700 1 -- 172.21.15.63:0/1452523677 wait complete. 2024-03-23T11:59:20.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.272+0000 7f57dbe00700 1 -- 172.21.15.63:0/827629458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 msgr2=0x7f57d4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.272+0000 7f57dbe00700 1 --2- 172.21.15.63:0/827629458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d4106f20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f57c0004970 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:20.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.272+0000 7f57dbe00700 1 -- 172.21.15.63:0/827629458 shutdown_connections 2024-03-23T11:59:20.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.272+0000 7f57dbe00700 1 --2- 172.21.15.63:0/827629458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d4106f20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.272+0000 7f57dbe00700 1 --2- 172.21.15.63:0/827629458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40fdeb0 0x7f57d40fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.272+0000 7f57dbe00700 1 -- 172.21.15.63:0/827629458 >> 172.21.15.63:0/827629458 conn(0x7f57d4072bc0 msgr2=0x7f57d4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:20.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.273+0000 7f57dbe00700 1 -- 172.21.15.63:0/827629458 shutdown_connections 2024-03-23T11:59:20.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.273+0000 7f57dbe00700 1 -- 172.21.15.63:0/827629458 wait complete. 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.273+0000 7f57dbe00700 1 Processor -- start 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57dbe00700 1 -- start start 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57dbe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40fdeb0 0x7f57d40fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57dbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d40fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57dbe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d40fc840 con 0x7f57d40fe850 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57dbe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d40fc980 con 0x7f57d40fdeb0 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57d8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d40fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57d8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d40fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44902/0 (socket says 172.21.15.63:44902) 2024-03-23T11:59:20.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57d8a00700 1 -- 172.21.15.63:0/960615677 learned_addr learned my addr 172.21.15.63:0/960615677 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.274+0000 7f57d9400700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40fdeb0 0x7f57d40fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57d8a00700 1 -- 172.21.15.63:0/960615677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40fdeb0 msgr2=0x7f57d40fbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57d8a00700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40fdeb0 0x7f57d40fbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57d8a00700 1 -- 172.21.15.63:0/960615677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0005040 con 0x7f57d40fe850 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57d8a00700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d40fc2b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009360 tx=0x7f57c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57d1600700 1 -- 172.21.15.63:0/960615677 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c000e040 con 0x7f57d40fe850 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57d1600700 1 -- 172.21.15.63:0/960615677 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57c0004290 con 0x7f57d40fe850 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57d1600700 1 -- 172.21.15.63:0/960615677 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c0013d90 con 0x7f57d40fe850 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d40fcbd0 con 0x7f57d40fe850 2024-03-23T11:59:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.275+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d411a0c0 con 0x7f57d40fe850 2024-03-23T11:59:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.276+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4065e10 con 0x7f57d40fe850 2024-03-23T11:59:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.277+0000 7f57d1600700 1 -- 172.21.15.63:0/960615677 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f57c000ba40 con 0x7f57d40fe850 2024-03-23T11:59:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.279+0000 7f57d1600700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f57bc06c7d0 0x7f57bc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.279+0000 7f57d1600700 1 -- 172.21.15.63:0/960615677 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f57c008cd40 con 0x7f57d40fe850 2024-03-23T11:59:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.279+0000 7f57d9400700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f57bc06c7d0 0x7f57bc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.280+0000 7f57d9400700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f57bc06c7d0 0x7f57bc06ec80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f57c8004f40 tx=0x7f57c800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:20.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.283+0000 7f57d1600700 1 -- 172.21.15.63:0/960615677 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f57c0056f40 con 0x7f57d40fe850 2024-03-23T11:59:20.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.542+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f57d404bed0 con 0x7f57d40fe850 2024-03-23T11:59:20.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.543+0000 7f57d1600700 1 -- 172.21.15.63:0/960615677 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f57c000bcf0 con 0x7f57d40fe850 2024-03-23T11:59:20.525 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:59:20.526 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:59:20.527 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:20.528 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:20.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f57bc06c7d0 msgr2=0x7f57bc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f57bc06c7d0 0x7f57bc06ec80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f57c8004f40 tx=0x7f57c800b040 comp rx=0 tx=0).stop 2024-03-23T11:59:20.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 msgr2=0x7f57d40fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d40fc2b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009360 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:20.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 shutdown_connections 2024-03-23T11:59:20.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f57bc06c7d0 0x7f57bc06ec80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40fdeb0 0x7f57d40fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 --2- 172.21.15.63:0/960615677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57d40fe850 0x7f57d40fc2b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 >> 172.21.15.63:0/960615677 conn(0x7f57d4072bc0 msgr2=0x7f57d4103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:20.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 shutdown_connections 2024-03-23T11:59:20.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.545+0000 7f57dbe00700 1 -- 172.21.15.63:0/960615677 wait complete. 2024-03-23T11:59:20.530 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.683+0000 7ff432200700 1 -- 172.21.15.63:0/2234403510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c102e00 msgr2=0x7ff42c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.683+0000 7ff432200700 1 --2- 172.21.15.63:0/2234403510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c102e00 0x7ff42c1031d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff41c004940 tx=0x7ff41c009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.683+0000 7ff432200700 1 -- 172.21.15.63:0/2234403510 shutdown_connections 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.683+0000 7ff432200700 1 --2- 172.21.15.63:0/2234403510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff42c1037a0 0x7ff42c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.683+0000 7ff432200700 1 --2- 172.21.15.63:0/2234403510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c102e00 0x7ff42c1031d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.683+0000 7ff432200700 1 -- 172.21.15.63:0/2234403510 >> 172.21.15.63:0/2234403510 conn(0x7ff42c0f89b0 msgr2=0x7ff42c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.684+0000 7ff432200700 1 -- 172.21.15.63:0/2234403510 shutdown_connections 2024-03-23T11:59:20.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.684+0000 7ff432200700 1 -- 172.21.15.63:0/2234403510 wait complete. 2024-03-23T11:59:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.684+0000 7ff432200700 1 Processor -- start 2024-03-23T11:59:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff432200700 1 -- start start 2024-03-23T11:59:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff432200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff42c102e00 0x7ff42c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff432200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c1037a0 0x7ff42c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff432200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c1168a0 con 0x7ff42c1037a0 2024-03-23T11:59:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff432200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c10fc70 con 0x7ff42c102e00 2024-03-23T11:59:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff42aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c1037a0 0x7ff42c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff42aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c1037a0 0x7ff42c116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44934/0 (socket says 172.21.15.63:44934) 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff42aa00700 1 -- 172.21.15.63:0/3240098940 learned_addr learned my addr 172.21.15.63:0/3240098940 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.685+0000 7ff42b400700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff42c102e00 0x7ff42c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff42aa00700 1 -- 172.21.15.63:0/3240098940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff42c102e00 msgr2=0x7ff42c115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff42aa00700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff42c102e00 0x7ff42c115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff42aa00700 1 -- 172.21.15.63:0/3240098940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c005040 con 0x7ff42c1037a0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff42aa00700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c1037a0 0x7ff42c116250 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff418002a80 tx=0x7ff418004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff423400700 1 -- 172.21.15.63:0/3240098940 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff418004e80 con 0x7ff42c1037a0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff423400700 1 -- 172.21.15.63:0/3240098940 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff41800f040 con 0x7ff42c1037a0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c10ff50 con 0x7ff42c1037a0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff423400700 1 -- 172.21.15.63:0/3240098940 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41800da00 con 0x7ff42c1037a0 2024-03-23T11:59:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.686+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c110510 con 0x7ff42c1037a0 2024-03-23T11:59:20.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.687+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c0fe550 con 0x7ff42c1037a0 2024-03-23T11:59:20.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.688+0000 7ff423400700 1 -- 172.21.15.63:0/3240098940 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff418007cc0 con 0x7ff42c1037a0 2024-03-23T11:59:20.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.689+0000 7ff423400700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff41006c7d0 0x7ff41006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:20.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.689+0000 7ff423400700 1 -- 172.21.15.63:0/3240098940 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff41808b0a0 con 0x7ff42c1037a0 2024-03-23T11:59:20.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.689+0000 7ff42b400700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff41006c7d0 0x7ff41006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:20.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.690+0000 7ff42b400700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff41006c7d0 0x7ff41006ec80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009fd0 tx=0x7ff41c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:20.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.694+0000 7ff423400700 1 -- 172.21.15.63:0/3240098940 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff418058d30 con 0x7ff42c1037a0 2024-03-23T11:59:20.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.930+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff42c109de0 con 0x7ff41006c7d0 2024-03-23T11:59:20.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.931+0000 7ff423400700 1 -- 172.21.15.63:0/3240098940 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff42c109de0 con 0x7ff41006c7d0 2024-03-23T11:59:20.914 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:59:20.914 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:59:20.914 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:59:20.914 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:59:20.915 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:59:20.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff41006c7d0 msgr2=0x7ff41006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff41006c7d0 0x7ff41006ec80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009fd0 tx=0x7ff41c009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c1037a0 msgr2=0x7ff42c116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c1037a0 0x7ff42c116250 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff418002a80 tx=0x7ff418004630 comp rx=0 tx=0).stop 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 shutdown_connections 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff41006c7d0 0x7ff41006ec80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff42c102e00 0x7ff42c115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 --2- 172.21.15.63:0/3240098940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff42c1037a0 0x7ff42c116250 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 >> 172.21.15.63:0/3240098940 conn(0x7ff42c0f89b0 msgr2=0x7ff42c0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 shutdown_connections 2024-03-23T11:59:20.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:20.934+0000 7ff432200700 1 -- 172.21.15.63:0/3240098940 wait complete. 2024-03-23T11:59:21.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 -- 172.21.15.63:0/3769489364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb344102e00 msgr2=0x7fb3441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:21.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 --2- 172.21.15.63:0/3769489364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb344102e00 0x7fb3441031d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb338004940 tx=0x7fb338009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:21.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 -- 172.21.15.63:0/3769489364 shutdown_connections 2024-03-23T11:59:21.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 --2- 172.21.15.63:0/3769489364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3441037a0 0x7fb3440fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:21.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 --2- 172.21.15.63:0/3769489364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb344102e00 0x7fb3441031d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:21.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 -- 172.21.15.63:0/3769489364 >> 172.21.15.63:0/3769489364 conn(0x7fb3440f89b0 msgr2=0x7fb3440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:21.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 -- 172.21.15.63:0/3769489364 shutdown_connections 2024-03-23T11:59:21.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.071+0000 7fb34c000700 1 -- 172.21.15.63:0/3769489364 wait complete. 2024-03-23T11:59:21.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.072+0000 7fb34c000700 1 Processor -- start 2024-03-23T11:59:21.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.072+0000 7fb34c000700 1 -- start start 2024-03-23T11:59:21.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.072+0000 7fb34c000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb344102e00 0x7fb344115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:21.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.072+0000 7fb34c000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3441037a0 0x7fb344116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.072+0000 7fb34c000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3441167e0 con 0x7fb3441037a0 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.072+0000 7fb34c000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34410fbb0 con 0x7fb344102e00 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb348c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3441037a0 0x7fb344116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb348c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3441037a0 0x7fb344116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44944/0 (socket says 172.21.15.63:44944) 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb348c00700 1 -- 172.21.15.63:0/229208735 learned_addr learned my addr 172.21.15.63:0/229208735 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb349600700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb344102e00 0x7fb344115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb348c00700 1 -- 172.21.15.63:0/229208735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb344102e00 msgr2=0x7fb344115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb348c00700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb344102e00 0x7fb344115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb348c00700 1 -- 172.21.15.63:0/229208735 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338005040 con 0x7fb3441037a0 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb349600700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb344102e00 0x7fb344115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.073+0000 7fb348c00700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3441037a0 0x7fb344116190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb33000eb40 tx=0x7fb33000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.074+0000 7fb341600700 1 -- 172.21.15.63:0/229208735 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb330007cc0 con 0x7fb3441037a0 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.074+0000 7fb341600700 1 -- 172.21.15.63:0/229208735 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb33000f040 con 0x7fb3441037a0 2024-03-23T11:59:21.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.074+0000 7fb341600700 1 -- 172.21.15.63:0/229208735 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb330009570 con 0x7fb3441037a0 2024-03-23T11:59:21.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.074+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34410fe30 con 0x7fb3441037a0 2024-03-23T11:59:21.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.074+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3441103f0 con 0x7fb3441037a0 2024-03-23T11:59:21.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.075+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3440fe550 con 0x7fb3441037a0 2024-03-23T11:59:21.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.076+0000 7fb341600700 1 -- 172.21.15.63:0/229208735 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb3300096d0 con 0x7fb3441037a0 2024-03-23T11:59:21.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.077+0000 7fb341600700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb32c06c7d0 0x7fb32c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:21.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.077+0000 7fb341600700 1 -- 172.21.15.63:0/229208735 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb33008b4c0 con 0x7fb3441037a0 2024-03-23T11:59:21.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.077+0000 7fb349600700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb32c06c7d0 0x7fb32c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:21.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.078+0000 7fb349600700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb32c06c7d0 0x7fb32c06ec80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb3380136f0 tx=0x7fb338009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:21.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.081+0000 7fb341600700 1 -- 172.21.15.63:0/229208735 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb330059280 con 0x7fb3441037a0 2024-03-23T11:59:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:20 smithi063 ceph-mon[141124]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:20 smithi063 ceph-mon[141124]: from='client.34522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:20 smithi063 ceph-mon[141124]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:20 smithi063 ceph-mon[141124]: pgmap v246: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:20 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1452523677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:59:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:20 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/960615677' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:59:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:20 smithi111 ceph-mon[106899]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:20 smithi111 ceph-mon[106899]: from='client.34522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:20 smithi111 ceph-mon[106899]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:20 smithi111 ceph-mon[106899]: pgmap v246: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:20 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1452523677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:59:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:20 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/960615677' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:59:21.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.383+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb34404bed0 con 0x7fb3441037a0 2024-03-23T11:59:21.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.384+0000 7fb341600700 1 -- 172.21.15.63:0/229208735 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fb330058e10 con 0x7fb3441037a0 2024-03-23T11:59:21.367 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T11:59:21.367 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T11:59:21.367 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T11:59:21.367 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T11:59:21.367 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb32c06c7d0 msgr2=0x7fb32c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb32c06c7d0 0x7fb32c06ec80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb3380136f0 tx=0x7fb338009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3441037a0 msgr2=0x7fb344116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3441037a0 0x7fb344116190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb33000eb40 tx=0x7fb33000b040 comp rx=0 tx=0).stop 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 shutdown_connections 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb32c06c7d0 0x7fb32c06ec80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb344102e00 0x7fb344115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 --2- 172.21.15.63:0/229208735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3441037a0 0x7fb344116190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:21.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 >> 172.21.15.63:0/229208735 conn(0x7fb3440f89b0 msgr2=0x7fb3440f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:21.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 shutdown_connections 2024-03-23T11:59:21.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:21.386+0000 7fb34c000700 1 -- 172.21.15.63:0/229208735 wait complete. 2024-03-23T11:59:22.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:21 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/229208735' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:59:22.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:21 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/229208735' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:59:23.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:22 smithi063 ceph-mon[141124]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:23.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:22 smithi063 ceph-mon[141124]: pgmap v247: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:23.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:22 smithi111 ceph-mon[106899]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:23.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:22 smithi111 ceph-mon[106899]: pgmap v247: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:25.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:24 smithi063 ceph-mon[141124]: pgmap v248: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:25.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:24 smithi111 ceph-mon[106899]: pgmap v248: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:26.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:26.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:26 smithi063 ceph-mon[141124]: pgmap v249: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:26.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:26.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:26 smithi111 ceph-mon[106899]: pgmap v249: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:29.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:28 smithi063 ceph-mon[141124]: pgmap v250: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:28 smithi111 ceph-mon[106899]: pgmap v250: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:31.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:30 smithi063 ceph-mon[141124]: pgmap v251: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:31.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:30 smithi111 ceph-mon[106899]: pgmap v251: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:32 smithi063 ceph-mon[141124]: pgmap v252: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:33.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:32 smithi111 ceph-mon[106899]: pgmap v252: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:35.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:34 smithi063 ceph-mon[141124]: pgmap v253: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:35.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:34 smithi111 ceph-mon[106899]: pgmap v253: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:36.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:36 smithi063 ceph-mon[141124]: pgmap v254: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:36.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:36 smithi111 ceph-mon[106899]: pgmap v254: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:39.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:38 smithi063 ceph-mon[141124]: pgmap v255: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:39.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:38 smithi111 ceph-mon[106899]: pgmap v255: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:40 smithi063 ceph-mon[141124]: pgmap v256: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:40 smithi111 ceph-mon[106899]: pgmap v256: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:43.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:42 smithi063 ceph-mon[141124]: pgmap v257: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:43.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:42 smithi111 ceph-mon[106899]: pgmap v257: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:45.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:44 smithi063 ceph-mon[141124]: pgmap v258: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:45.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:44 smithi111 ceph-mon[106899]: pgmap v258: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:46.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:46 smithi063 ceph-mon[141124]: pgmap v259: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:46.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:46 smithi111 ceph-mon[106899]: pgmap v259: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:49.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:48 smithi063 ceph-mon[141124]: pgmap v260: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:49.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:48 smithi111 ceph-mon[106899]: pgmap v260: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:51.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:50 smithi063 ceph-mon[141124]: pgmap v261: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:51.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:50 smithi111 ceph-mon[106899]: pgmap v261: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:51.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 -- 172.21.15.63:0/2461823208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070c70 msgr2=0x7f5e1c109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:51.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 --2- 172.21.15.63:0/2461823208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070c70 0x7f5e1c109110 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5e08004970 tx=0x7f5e08009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:51.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 -- 172.21.15.63:0/2461823208 shutdown_connections 2024-03-23T11:59:51.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 --2- 172.21.15.63:0/2461823208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070c70 0x7f5e1c109110 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 --2- 172.21.15.63:0/2461823208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e1c070360 0x7f5e1c070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 -- 172.21.15.63:0/2461823208 >> 172.21.15.63:0/2461823208 conn(0x7f5e1c0f89b0 msgr2=0x7f5e1c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:51.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 -- 172.21.15.63:0/2461823208 shutdown_connections 2024-03-23T11:59:51.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.541+0000 7f5e23800700 1 -- 172.21.15.63:0/2461823208 wait complete. 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.542+0000 7f5e23800700 1 Processor -- start 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.542+0000 7f5e23800700 1 -- start start 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e23800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070360 0x7f5e1c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e23800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e1c070c70 0x7f5e1c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e23800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c116830 con 0x7f5e1c070360 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e23800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c10fc00 con 0x7f5e1c070c70 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e20e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070360 0x7f5e1c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e20e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070360 0x7f5e1c115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52044/0 (socket says 172.21.15.63:52044) 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e20e00700 1 -- 172.21.15.63:0/2650715 learned_addr learned my addr 172.21.15.63:0/2650715 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e1be00700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e1c070c70 0x7f5e1c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e20e00700 1 -- 172.21.15.63:0/2650715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e1c070c70 msgr2=0x7f5e1c1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e20e00700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e1c070c70 0x7f5e1c1161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.543+0000 7f5e20e00700 1 -- 172.21.15.63:0/2650715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08005040 con 0x7f5e1c070360 2024-03-23T11:59:51.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.544+0000 7f5e20e00700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070360 0x7f5e1c115ca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000eb40 tx=0x7f5e1000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:51.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.544+0000 7f5e18c00700 1 -- 172.21.15.63:0/2650715 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e10007cc0 con 0x7f5e1c070360 2024-03-23T11:59:51.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.544+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c10fe80 con 0x7f5e1c070360 2024-03-23T11:59:51.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.544+0000 7f5e18c00700 1 -- 172.21.15.63:0/2650715 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e1000f040 con 0x7f5e1c070360 2024-03-23T11:59:51.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.544+0000 7f5e18c00700 1 -- 172.21.15.63:0/2650715 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e10009570 con 0x7f5e1c070360 2024-03-23T11:59:51.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.544+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c1103c0 con 0x7f5e1c070360 2024-03-23T11:59:51.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.545+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c103bd0 con 0x7f5e1c070360 2024-03-23T11:59:51.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.546+0000 7f5e18c00700 1 -- 172.21.15.63:0/2650715 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5e100096d0 con 0x7f5e1c070360 2024-03-23T11:59:51.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.547+0000 7f5e18c00700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e0406c7d0 0x7f5e0406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:51.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.548+0000 7f5e18c00700 1 -- 172.21.15.63:0/2650715 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e1008b7e0 con 0x7f5e1c070360 2024-03-23T11:59:51.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.548+0000 7f5e1be00700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e0406c7d0 0x7f5e0406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:51.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.548+0000 7f5e1be00700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e0406c7d0 0x7f5e0406ec80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c111540 tx=0x7f5e08009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:51.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.553+0000 7f5e18c00700 1 -- 172.21.15.63:0/2650715 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5e100594a0 con 0x7f5e1c070360 2024-03-23T11:59:51.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.781+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e1c10c020 con 0x7f5e0406c7d0 2024-03-23T11:59:51.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.783+0000 7f5e18c00700 1 -- 172.21.15.63:0/2650715 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5e1c10c020 con 0x7f5e0406c7d0 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e0406c7d0 msgr2=0x7f5e0406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e0406c7d0 0x7f5e0406ec80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c111540 tx=0x7f5e08009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070360 msgr2=0x7f5e1c115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070360 0x7f5e1c115ca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000eb40 tx=0x7f5e1000b040 comp rx=0 tx=0).stop 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 shutdown_connections 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e0406c7d0 0x7f5e0406ec80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e1c070360 0x7f5e1c115ca0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 --2- 172.21.15.63:0/2650715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e1c070c70 0x7f5e1c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 >> 172.21.15.63:0/2650715 conn(0x7f5e1c0f89b0 msgr2=0x7f5e1c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 shutdown_connections 2024-03-23T11:59:51.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.785+0000 7f5e23800700 1 -- 172.21.15.63:0/2650715 wait complete. 2024-03-23T11:59:51.782 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.923+0000 7fd232c00700 1 -- 172.21.15.63:0/520018191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c1037a0 msgr2=0x7fd22c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.923+0000 7fd232c00700 1 --2- 172.21.15.63:0/520018191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c1037a0 0x7fd22c0fce10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd218004940 tx=0x7fd218009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.924+0000 7fd232c00700 1 -- 172.21.15.63:0/520018191 shutdown_connections 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.924+0000 7fd232c00700 1 --2- 172.21.15.63:0/520018191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c1037a0 0x7fd22c0fce10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.924+0000 7fd232c00700 1 --2- 172.21.15.63:0/520018191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd22c102e00 0x7fd22c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.924+0000 7fd232c00700 1 -- 172.21.15.63:0/520018191 >> 172.21.15.63:0/520018191 conn(0x7fd22c0f89b0 msgr2=0x7fd22c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.924+0000 7fd232c00700 1 -- 172.21.15.63:0/520018191 shutdown_connections 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.924+0000 7fd232c00700 1 -- 172.21.15.63:0/520018191 wait complete. 2024-03-23T11:59:51.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.924+0000 7fd232c00700 1 Processor -- start 2024-03-23T11:59:51.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd232c00700 1 -- start start 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd232c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c102e00 0x7fd22c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd232c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd22c1037a0 0x7fd22c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd232c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c116890 con 0x7fd22c102e00 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd232c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c10fc60 con 0x7fd22c1037a0 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd22be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c102e00 0x7fd22c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd22be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c102e00 0x7fd22c115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52062/0 (socket says 172.21.15.63:52062) 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.925+0000 7fd22be00700 1 -- 172.21.15.63:0/2628952483 learned_addr learned my addr 172.21.15.63:0/2628952483 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd22b400700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd22c1037a0 0x7fd22c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd22be00700 1 -- 172.21.15.63:0/2628952483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd22c1037a0 msgr2=0x7fd22c116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd22be00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd22c1037a0 0x7fd22c116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd22be00700 1 -- 172.21.15.63:0/2628952483 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd218005040 con 0x7fd22c102e00 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd22be00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c102e00 0x7fd22c115d00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd21c002a50 tx=0x7fd21c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd223e00700 1 -- 172.21.15.63:0/2628952483 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21c007cc0 con 0x7fd22c102e00 2024-03-23T11:59:51.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c10ff40 con 0x7fd22c102e00 2024-03-23T11:59:51.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd223e00700 1 -- 172.21.15.63:0/2628952483 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd21c00f040 con 0x7fd22c102e00 2024-03-23T11:59:51.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd223e00700 1 -- 172.21.15.63:0/2628952483 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21c009500 con 0x7fd22c102e00 2024-03-23T11:59:51.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.926+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c110500 con 0x7fd22c102e00 2024-03-23T11:59:51.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.927+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c0fe550 con 0x7fd22c102e00 2024-03-23T11:59:51.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.928+0000 7fd223e00700 1 -- 172.21.15.63:0/2628952483 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd21c0096a0 con 0x7fd22c102e00 2024-03-23T11:59:51.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.929+0000 7fd223e00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd21006c7d0 0x7fd21006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:51.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.929+0000 7fd223e00700 1 -- 172.21.15.63:0/2628952483 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd21c08b8c0 con 0x7fd22c102e00 2024-03-23T11:59:51.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.930+0000 7fd22b400700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd21006c7d0 0x7fd21006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:51.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.930+0000 7fd22b400700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd21006c7d0 0x7fd21006ec80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd21800b020 tx=0x7fd218009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:51.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:51.934+0000 7fd223e00700 1 -- 172.21.15.63:0/2628952483 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd21c059550 con 0x7fd22c102e00 2024-03-23T11:59:52.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.167+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd22c04bf40 con 0x7fd21006c7d0 2024-03-23T11:59:52.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.168+0000 7fd223e00700 1 -- 172.21.15.63:0/2628952483 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd22c04bf40 con 0x7fd21006c7d0 2024-03-23T11:59:52.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd21006c7d0 msgr2=0x7fd21006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd21006c7d0 0x7fd21006ec80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd21800b020 tx=0x7fd218009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c102e00 msgr2=0x7fd22c115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c102e00 0x7fd22c115d00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd21c002a50 tx=0x7fd21c0046b0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 shutdown_connections 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd21006c7d0 0x7fd21006ec80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd22c102e00 0x7fd22c115d00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 --2- 172.21.15.63:0/2628952483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd22c1037a0 0x7fd22c116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 >> 172.21.15.63:0/2628952483 conn(0x7fd22c0f89b0 msgr2=0x7fd22c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 shutdown_connections 2024-03-23T11:59:52.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.171+0000 7fd232c00700 1 -- 172.21.15.63:0/2628952483 wait complete. 2024-03-23T11:59:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 -- 172.21.15.63:0/3718435403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 msgr2=0x7feb8c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 --2- 172.21.15.63:0/3718435403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c0fce10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feb78004940 tx=0x7feb78009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 -- 172.21.15.63:0/3718435403 shutdown_connections 2024-03-23T11:59:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 --2- 172.21.15.63:0/3718435403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c0fce10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 --2- 172.21.15.63:0/3718435403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb8c102e00 0x7feb8c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 -- 172.21.15.63:0/3718435403 >> 172.21.15.63:0/3718435403 conn(0x7feb8c0f89b0 msgr2=0x7feb8c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 -- 172.21.15.63:0/3718435403 shutdown_connections 2024-03-23T11:59:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.322+0000 7feb92a00700 1 -- 172.21.15.63:0/3718435403 wait complete. 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.323+0000 7feb92a00700 1 Processor -- start 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.323+0000 7feb92a00700 1 -- start start 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.323+0000 7feb92a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb8c102e00 0x7feb8c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb92a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb92a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c116840 con 0x7feb8c1037a0 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb92a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c10fc10 con 0x7feb8c102e00 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb8b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb8b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c1161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52084/0 (socket says 172.21.15.63:52084) 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb8b400700 1 -- 172.21.15.63:0/3174876893 learned_addr learned my addr 172.21.15.63:0/3174876893 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb8be00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb8c102e00 0x7feb8c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb8b400700 1 -- 172.21.15.63:0/3174876893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb8c102e00 msgr2=0x7feb8c115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb8b400700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb8c102e00 0x7feb8c115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.324+0000 7feb8b400700 1 -- 172.21.15.63:0/3174876893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78005040 con 0x7feb8c1037a0 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.325+0000 7feb8b400700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c1161f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feb78008d50 tx=0x7feb78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.325+0000 7feb83e00700 1 -- 172.21.15.63:0/3174876893 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7800e030 con 0x7feb8c1037a0 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.325+0000 7feb83e00700 1 -- 172.21.15.63:0/3174876893 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb780093b0 con 0x7feb8c1037a0 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.325+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c10fe90 con 0x7feb8c1037a0 2024-03-23T11:59:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.325+0000 7feb83e00700 1 -- 172.21.15.63:0/3174876893 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7801cbb0 con 0x7feb8c1037a0 2024-03-23T11:59:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.325+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c110450 con 0x7feb8c1037a0 2024-03-23T11:59:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.326+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8c0fe550 con 0x7feb8c1037a0 2024-03-23T11:59:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.327+0000 7feb83e00700 1 -- 172.21.15.63:0/3174876893 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7feb78013930 con 0x7feb8c1037a0 2024-03-23T11:59:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.328+0000 7feb83e00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb7006c7d0 0x7feb7006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.328+0000 7feb83e00700 1 -- 172.21.15.63:0/3174876893 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb7808d320 con 0x7feb8c1037a0 2024-03-23T11:59:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.328+0000 7feb8be00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb7006c7d0 0x7feb7006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.329+0000 7feb8be00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb7006c7d0 0x7feb7006ec80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009b80 tx=0x7feb7c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:52.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.333+0000 7feb83e00700 1 -- 172.21.15.63:0/3174876893 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb7805af30 con 0x7feb8c1037a0 2024-03-23T11:59:52.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.553+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feb8c109e10 con 0x7feb7006c7d0 2024-03-23T11:59:52.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.564+0000 7feb83e00700 1 -- 172.21.15.63:0/3174876893 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7feb8c109e10 con 0x7feb7006c7d0 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (9m) 109s ago 22m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (6m) 109s ago 22m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (6m) 91s ago 21m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (8m) 109s ago 21m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (2m) 109s ago 15m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (115s) 109s ago 15m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (15m) 91s ago 15m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (95s) 91s ago 15m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (11m) 109s ago 23m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (10m) 91s ago 21m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (7m) 109s ago 23m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (7m) 91s ago 21m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T11:59:52.547 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (9m) 109s ago 22m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (9m) 91s ago 21m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (5m) 109s ago 20m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (5m) 109s ago 19m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (4m) 109s ago 19m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (3m) 91s ago 18m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (3m) 91s ago 18m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (2m) 91s ago 17m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T11:59:52.548 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (9m) 109s ago 21m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb7006c7d0 msgr2=0x7feb7006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb7006c7d0 0x7feb7006ec80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009b80 tx=0x7feb7c00b040 comp rx=0 tx=0).stop 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 msgr2=0x7feb8c1161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c1161f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feb78008d50 tx=0x7feb78009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 shutdown_connections 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb7006c7d0 0x7feb7006ec80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb8c102e00 0x7feb8c115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 --2- 172.21.15.63:0/3174876893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb8c1037a0 0x7feb8c1161f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 >> 172.21.15.63:0/3174876893 conn(0x7feb8c0f89b0 msgr2=0x7feb8c070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 shutdown_connections 2024-03-23T11:59:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.566+0000 7feb92a00700 1 -- 172.21.15.63:0/3174876893 wait complete. 2024-03-23T11:59:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 -- 172.21.15.63:0/3879958753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa180071f90 msgr2=0x7fa180070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 --2- 172.21.15.63:0/3879958753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa180071f90 0x7fa180070360 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa174004970 tx=0x7fa174009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 -- 172.21.15.63:0/3879958753 shutdown_connections 2024-03-23T11:59:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 --2- 172.21.15.63:0/3879958753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1800708a0 0x7fa180070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 --2- 172.21.15.63:0/3879958753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa180071f90 0x7fa180070360 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 -- 172.21.15.63:0/3879958753 >> 172.21.15.63:0/3879958753 conn(0x7fa1800f89b0 msgr2=0x7fa1800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 -- 172.21.15.63:0/3879958753 shutdown_connections 2024-03-23T11:59:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.702+0000 7fa188400700 1 -- 172.21.15.63:0/3879958753 wait complete. 2024-03-23T11:59:52.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.703+0000 7fa188400700 1 Processor -- start 2024-03-23T11:59:52.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.703+0000 7fa188400700 1 -- start start 2024-03-23T11:59:52.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.703+0000 7fa188400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1800708a0 0x7fa18011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.703+0000 7fa188400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa180071f90 0x7fa18011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.703+0000 7fa188400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18011ac50 con 0x7fa1800708a0 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.703+0000 7fa188400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa180114020 con 0x7fa180071f90 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.703+0000 7fa185a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1800708a0 0x7fa18011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa185a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1800708a0 0x7fa18011a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52096/0 (socket says 172.21.15.63:52096) 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa185a00700 1 -- 172.21.15.63:0/185895303 learned_addr learned my addr 172.21.15.63:0/185895303 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa185000700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa180071f90 0x7fa18011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa185a00700 1 -- 172.21.15.63:0/185895303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa180071f90 msgr2=0x7fa18011a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa185a00700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa180071f90 0x7fa18011a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa185a00700 1 -- 172.21.15.63:0/185895303 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa174005040 con 0x7fa1800708a0 2024-03-23T11:59:52.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa185a00700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1800708a0 0x7fa18011a0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa174004f50 tx=0x7fa174009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:52.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa17d600700 1 -- 172.21.15.63:0/185895303 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17400e040 con 0x7fa1800708a0 2024-03-23T11:59:52.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.704+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1801142a0 con 0x7fa1800708a0 2024-03-23T11:59:52.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.705+0000 7fa17d600700 1 -- 172.21.15.63:0/185895303 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa174013420 con 0x7fa1800708a0 2024-03-23T11:59:52.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.705+0000 7fa17d600700 1 -- 172.21.15.63:0/185895303 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17401c610 con 0x7fa1800708a0 2024-03-23T11:59:52.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.705+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1801147e0 con 0x7fa1800708a0 2024-03-23T11:59:52.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.706+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa180071aa0 con 0x7fa1800708a0 2024-03-23T11:59:52.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.707+0000 7fa17d600700 1 -- 172.21.15.63:0/185895303 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa174013590 con 0x7fa1800708a0 2024-03-23T11:59:52.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.708+0000 7fa17d600700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa16806c7d0 0x7fa16806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:52.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.708+0000 7fa17d600700 1 -- 172.21.15.63:0/185895303 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa17408ccd0 con 0x7fa1800708a0 2024-03-23T11:59:52.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.708+0000 7fa185000700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa16806c7d0 0x7fa16806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:52.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.709+0000 7fa185000700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa16806c7d0 0x7fa16806ec80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa180115960 tx=0x7fa16c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:52.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:52.712+0000 7fa17d600700 1 -- 172.21.15.63:0/185895303 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa17405aa90 con 0x7fa1800708a0 2024-03-23T11:59:52.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.011+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa180071aa0 con 0x7fa1800708a0 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.012+0000 7fa17d600700 1 -- 172.21.15.63:0/185895303 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fa17405a620 con 0x7fa1800708a0 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T11:59:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T11:59:52.996 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:59:52.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa16806c7d0 msgr2=0x7fa16806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa16806c7d0 0x7fa16806ec80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa180115960 tx=0x7fa16c00b040 comp rx=0 tx=0).stop 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1800708a0 msgr2=0x7fa18011a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1800708a0 0x7fa18011a0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa174004f50 tx=0x7fa174009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 shutdown_connections 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa16806c7d0 0x7fa16806ec80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1800708a0 0x7fa18011a0c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 --2- 172.21.15.63:0/185895303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa180071f90 0x7fa18011a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 >> 172.21.15.63:0/185895303 conn(0x7fa1800f89b0 msgr2=0x7fa1800f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 shutdown_connections 2024-03-23T11:59:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.015+0000 7fa188400700 1 -- 172.21.15.63:0/185895303 wait complete. 2024-03-23T11:59:53.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:52 smithi063 ceph-mon[141124]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:53.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:52 smithi063 ceph-mon[141124]: pgmap v262: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:53.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 -- 172.21.15.63:0/356358294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fe830 msgr2=0x7febe4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 --2- 172.21.15.63:0/356358294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fe830 0x7febe4106f00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7febd0004940 tx=0x7febd0009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:53.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 -- 172.21.15.63:0/356358294 shutdown_connections 2024-03-23T11:59:53.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 --2- 172.21.15.63:0/356358294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fe830 0x7febe4106f00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 --2- 172.21.15.63:0/356358294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febe40fde90 0x7febe40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 -- 172.21.15.63:0/356358294 >> 172.21.15.63:0/356358294 conn(0x7febe4072bc0 msgr2=0x7febe4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:53.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 -- 172.21.15.63:0/356358294 shutdown_connections 2024-03-23T11:59:53.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.151+0000 7febeb800700 1 -- 172.21.15.63:0/356358294 wait complete. 2024-03-23T11:59:53.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.152+0000 7febeb800700 1 Processor -- start 2024-03-23T11:59:53.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febeb800700 1 -- start start 2024-03-23T11:59:53.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febeb800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fde90 0x7febe40fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febeb800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febe40fe830 0x7febe40fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febeb800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40fc820 con 0x7febe40fde90 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febeb800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40fc960 con 0x7febe40fe830 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febe8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fde90 0x7febe40fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febe8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fde90 0x7febe40fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52112/0 (socket says 172.21.15.63:52112) 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.153+0000 7febe8e00700 1 -- 172.21.15.63:0/1716133766 learned_addr learned my addr 172.21.15.63:0/1716133766 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe3e00700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febe40fe830 0x7febe40fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe8e00700 1 -- 172.21.15.63:0/1716133766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febe40fe830 msgr2=0x7febe40fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe8e00700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febe40fe830 0x7febe40fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe8e00700 1 -- 172.21.15.63:0/1716133766 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd0005040 con 0x7febe40fde90 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe3e00700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febe40fe830 0x7febe40fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:59:53.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe8e00700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fde90 0x7febe40fbd50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7febd8002a50 tx=0x7febd80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:53.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe0c00700 1 -- 172.21.15.63:0/1716133766 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd8007cc0 con 0x7febe40fde90 2024-03-23T11:59:53.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe0c00700 1 -- 172.21.15.63:0/1716133766 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febd800f040 con 0x7febe40fde90 2024-03-23T11:59:53.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febe0c00700 1 -- 172.21.15.63:0/1716133766 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd800da70 con 0x7febe40fde90 2024-03-23T11:59:53.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.154+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe4119e00 con 0x7febe40fde90 2024-03-23T11:59:53.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.155+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe411a200 con 0x7febe40fde90 2024-03-23T11:59:53.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.156+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe4065e30 con 0x7febe40fde90 2024-03-23T11:59:53.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.156+0000 7febe0c00700 1 -- 172.21.15.63:0/1716133766 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7febd800dc10 con 0x7febe40fde90 2024-03-23T11:59:53.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.158+0000 7febe0c00700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7febcc06c7d0 0x7febcc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.158+0000 7febe0c00700 1 -- 172.21.15.63:0/1716133766 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7febd808b290 con 0x7febe40fde90 2024-03-23T11:59:53.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.158+0000 7febe3e00700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7febcc06c7d0 0x7febcc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.159+0000 7febe3e00700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7febcc06c7d0 0x7febcc06ec80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7febe40fd690 tx=0x7febd0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:53.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.162+0000 7febe0c00700 1 -- 172.21.15.63:0/1716133766 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7febd8055410 con 0x7febe40fde90 2024-03-23T11:59:53.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:52 smithi111 ceph-mon[106899]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:53.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:52 smithi111 ceph-mon[106899]: pgmap v262: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:53.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.421+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7febe405f5f0 con 0x7febe40fde90 2024-03-23T11:59:53.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.421+0000 7febe0c00700 1 -- 172.21.15.63:0/1716133766 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7febd8058a30 con 0x7febe40fde90 2024-03-23T11:59:53.404 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T11:59:53.404 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T11:59:53.404 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:59:53.404 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T11:59:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:53.406 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:53.407 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T11:59:53.407 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T11:59:53.407 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:53.407 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:53.407 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:53.407 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7febcc06c7d0 msgr2=0x7febcc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7febcc06c7d0 0x7febcc06ec80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7febe40fd690 tx=0x7febd0009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fde90 msgr2=0x7febe40fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fde90 0x7febe40fbd50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7febd8002a50 tx=0x7febd80046b0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 shutdown_connections 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7febcc06c7d0 0x7febcc06ec80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe40fde90 0x7febe40fbd50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 --2- 172.21.15.63:0/1716133766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febe40fe830 0x7febe40fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 >> 172.21.15.63:0/1716133766 conn(0x7febe4072bc0 msgr2=0x7febe4103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 shutdown_connections 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.424+0000 7febeb800700 1 -- 172.21.15.63:0/1716133766 wait complete. 2024-03-23T11:59:53.408 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T11:59:53.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.563+0000 7f10cd200700 1 -- 172.21.15.63:0/911007062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 msgr2=0x7f10c8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.563+0000 7f10cd200700 1 --2- 172.21.15.63:0/911007062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c8106f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f10b4004940 tx=0x7f10b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:53.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.563+0000 7f10cd200700 1 -- 172.21.15.63:0/911007062 shutdown_connections 2024-03-23T11:59:53.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.564+0000 7f10cd200700 1 --2- 172.21.15.63:0/911007062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c8106f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.564+0000 7f10cd200700 1 --2- 172.21.15.63:0/911007062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10c80fde90 0x7f10c80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.564+0000 7f10cd200700 1 -- 172.21.15.63:0/911007062 >> 172.21.15.63:0/911007062 conn(0x7f10c8072bc0 msgr2=0x7f10c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:53.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.564+0000 7f10cd200700 1 -- 172.21.15.63:0/911007062 shutdown_connections 2024-03-23T11:59:53.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.564+0000 7f10cd200700 1 -- 172.21.15.63:0/911007062 wait complete. 2024-03-23T11:59:53.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.564+0000 7f10cd200700 1 Processor -- start 2024-03-23T11:59:53.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10cd200700 1 -- start start 2024-03-23T11:59:53.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10cd200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10c80fde90 0x7f10c80fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10cd200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c80fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10cd200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80fc820 con 0x7f10c80fe830 2024-03-23T11:59:53.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10cd200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80fc960 con 0x7f10c80fde90 2024-03-23T11:59:53.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10c5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c80fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10c5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c80fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52116/0 (socket says 172.21.15.63:52116) 2024-03-23T11:59:53.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10c5400700 1 -- 172.21.15.63:0/2203288375 learned_addr learned my addr 172.21.15.63:0/2203288375 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:53.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.565+0000 7f10c5e00700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10c80fde90 0x7f10c80fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10c5400700 1 -- 172.21.15.63:0/2203288375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10c80fde90 msgr2=0x7f10c80fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10c5400700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10c80fde90 0x7f10c80fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10c5400700 1 -- 172.21.15.63:0/2203288375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4005040 con 0x7f10c80fe830 2024-03-23T11:59:53.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10c5400700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c80fc290 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f10b4000c00 tx=0x7f10b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:53.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10be000700 1 -- 172.21.15.63:0/2203288375 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10b400e040 con 0x7f10c80fe830 2024-03-23T11:59:53.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10be000700 1 -- 172.21.15.63:0/2203288375 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10b400bed0 con 0x7f10c80fe830 2024-03-23T11:59:53.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c80fcbb0 con 0x7f10c80fe830 2024-03-23T11:59:53.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10be000700 1 -- 172.21.15.63:0/2203288375 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10b401c3f0 con 0x7f10c80fe830 2024-03-23T11:59:53.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.566+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c811a190 con 0x7f10c80fe830 2024-03-23T11:59:53.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.567+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c8065e30 con 0x7f10c80fe830 2024-03-23T11:59:53.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.568+0000 7f10be000700 1 -- 172.21.15.63:0/2203288375 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f10b4013420 con 0x7f10c80fe830 2024-03-23T11:59:53.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.570+0000 7f10be000700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10ac06c7d0 0x7f10ac06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.570+0000 7f10be000700 1 -- 172.21.15.63:0/2203288375 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f10b408ce70 con 0x7f10c80fe830 2024-03-23T11:59:53.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.570+0000 7f10c5e00700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10ac06c7d0 0x7f10ac06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.571+0000 7f10c5e00700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10ac06c7d0 0x7f10ac06ec80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f10b8004f40 tx=0x7f10b800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:53.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.574+0000 7f10be000700 1 -- 172.21.15.63:0/2203288375 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f10b40570f0 con 0x7f10c80fe830 2024-03-23T11:59:53.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.809+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10c804bed0 con 0x7f10ac06c7d0 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.810+0000 7f10be000700 1 -- 172.21.15.63:0/2203288375 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f10c804bed0 con 0x7f10ac06c7d0 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T11:59:53.793 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T11:59:53.794 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T11:59:53.794 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T11:59:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.812+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10ac06c7d0 msgr2=0x7f10ac06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.812+0000 7f10cd200700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10ac06c7d0 0x7f10ac06ec80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f10b8004f40 tx=0x7f10b800b040 comp rx=0 tx=0).stop 2024-03-23T11:59:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.812+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 msgr2=0x7f10c80fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.812+0000 7f10cd200700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c80fc290 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f10b4000c00 tx=0x7f10b4009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.813+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 shutdown_connections 2024-03-23T11:59:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.813+0000 7f10cd200700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10ac06c7d0 0x7f10ac06ec80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.813+0000 7f10cd200700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10c80fde90 0x7f10c80fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.813+0000 7f10cd200700 1 --2- 172.21.15.63:0/2203288375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10c80fe830 0x7f10c80fc290 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.813+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 >> 172.21.15.63:0/2203288375 conn(0x7f10c8072bc0 msgr2=0x7f10c8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.813+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 shutdown_connections 2024-03-23T11:59:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.813+0000 7f10cd200700 1 -- 172.21.15.63:0/2203288375 wait complete. 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 -- 172.21.15.63:0/2535697872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 msgr2=0x7fb3a81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 --2- 172.21.15.63:0/2535697872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a81031d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb39c004940 tx=0x7fb39c009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 -- 172.21.15.63:0/2535697872 shutdown_connections 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 --2- 172.21.15.63:0/2535697872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a81037a0 0x7fb3a80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 --2- 172.21.15.63:0/2535697872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a81031d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 -- 172.21.15.63:0/2535697872 >> 172.21.15.63:0/2535697872 conn(0x7fb3a80f89b0 msgr2=0x7fb3a80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 -- 172.21.15.63:0/2535697872 shutdown_connections 2024-03-23T11:59:53.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.951+0000 7fb3afe00700 1 -- 172.21.15.63:0/2535697872 wait complete. 2024-03-23T11:59:53.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3afe00700 1 Processor -- start 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3afe00700 1 -- start start 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3afe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3afe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a81037a0 0x7fb3a8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3afe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8116860 con 0x7fb3a8102e00 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3afe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a810fc30 con 0x7fb3a81037a0 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3ad400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3ad400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a8115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52130/0 (socket says 172.21.15.63:52130) 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.952+0000 7fb3ad400700 1 -- 172.21.15.63:0/4259202281 learned_addr learned my addr 172.21.15.63:0/4259202281 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3aca00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a81037a0 0x7fb3a8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3ad400700 1 -- 172.21.15.63:0/4259202281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a81037a0 msgr2=0x7fb3a8116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3ad400700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a81037a0 0x7fb3a8116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3ad400700 1 -- 172.21.15.63:0/4259202281 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c005040 con 0x7fb3a8102e00 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3aca00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a81037a0 0x7fb3a8116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:59:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3ad400700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a8115cd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009360 tx=0x7fb39c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3a5600700 1 -- 172.21.15.63:0/4259202281 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb39c00e040 con 0x7fb3a8102e00 2024-03-23T11:59:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a810feb0 con 0x7fb3a8102e00 2024-03-23T11:59:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.953+0000 7fb3a5600700 1 -- 172.21.15.63:0/4259202281 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb39c013420 con 0x7fb3a8102e00 2024-03-23T11:59:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.954+0000 7fb3a5600700 1 -- 172.21.15.63:0/4259202281 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb39c0255e0 con 0x7fb3a8102e00 2024-03-23T11:59:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.954+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a8110470 con 0x7fb3a8102e00 2024-03-23T11:59:53.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.955+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a80fe550 con 0x7fb3a8102e00 2024-03-23T11:59:53.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.956+0000 7fb3a5600700 1 -- 172.21.15.63:0/4259202281 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb39c013590 con 0x7fb3a8102e00 2024-03-23T11:59:53.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.957+0000 7fb3a5600700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb39006c7d0 0x7fb39006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:59:53.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.957+0000 7fb3a5600700 1 -- 172.21.15.63:0/4259202281 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb39c08cd10 con 0x7fb3a8102e00 2024-03-23T11:59:53.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.957+0000 7fb3aca00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb39006c7d0 0x7fb39006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:59:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.957+0000 7fb3aca00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb39006c7d0 0x7fb39006ec80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb394007fd0 tx=0x7fb394009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:59:53.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:53.961+0000 7fb3a5600700 1 -- 172.21.15.63:0/4259202281 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb39c05a9a0 con 0x7fb3a8102e00 2024-03-23T11:59:54.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:53 smithi063 ceph-mon[141124]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:54.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:53 smithi063 ceph-mon[141124]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:54.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:53 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/185895303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:59:54.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:53 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1716133766' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:59:54.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:53 smithi111 ceph-mon[106899]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:54.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:53 smithi111 ceph-mon[106899]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:54.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:53 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/185895303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T11:59:54.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:53 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1716133766' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T11:59:54.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.258+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb3a804bed0 con 0x7fb3a8102e00 2024-03-23T11:59:54.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.259+0000 7fb3a5600700 1 -- 172.21.15.63:0/4259202281 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fb39c05a530 con 0x7fb3a8102e00 2024-03-23T11:59:54.241 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T11:59:54.242 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T11:59:54.242 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T11:59:54.242 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T11:59:54.242 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T11:59:54.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb39006c7d0 msgr2=0x7fb39006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:54.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb39006c7d0 0x7fb39006ec80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb394007fd0 tx=0x7fb394009040 comp rx=0 tx=0).stop 2024-03-23T11:59:54.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 msgr2=0x7fb3a8115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:59:54.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a8115cd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009360 tx=0x7fb39c009e30 comp rx=0 tx=0).stop 2024-03-23T11:59:54.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 shutdown_connections 2024-03-23T11:59:54.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb39006c7d0 0x7fb39006ec80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:54.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a8102e00 0x7fb3a8115cd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:54.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 --2- 172.21.15.63:0/4259202281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a81037a0 0x7fb3a8116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:59:54.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.261+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 >> 172.21.15.63:0/4259202281 conn(0x7fb3a80f89b0 msgr2=0x7fb3a80f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:59:54.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.262+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 shutdown_connections 2024-03-23T11:59:54.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T11:59:54.262+0000 7fb3afe00700 1 -- 172.21.15.63:0/4259202281 wait complete. 2024-03-23T11:59:55.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:54 smithi063 ceph-mon[141124]: pgmap v263: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:55.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:54 smithi063 ceph-mon[141124]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:55.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:54 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4259202281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:59:55.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:54 smithi111 ceph-mon[106899]: pgmap v263: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:55.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:54 smithi111 ceph-mon[106899]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T11:59:55.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:54 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4259202281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T11:59:56.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:56.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:56 smithi063 ceph-mon[141124]: pgmap v264: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:56.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T11:59:56.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:56 smithi111 ceph-mon[106899]: pgmap v264: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:59.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 11:59:58 smithi063 ceph-mon[141124]: pgmap v265: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T11:59:59.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 11:59:58 smithi111 ceph-mon[106899]: pgmap v265: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T11:59:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:00:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T11:59:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:00:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T11:59:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-03-23T12:00:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T11:59:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:00:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T11:59:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-03-23T12:00:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-mon[141124]: pgmap v266: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-mon[141124]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:00:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-mon[141124]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:00:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-mon[141124]: fs cephfs is degraded 2024-03-23T12:00:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-mon[141124]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:00:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:00 smithi063 ceph-mon[141124]: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:00:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:00 smithi111 ceph-mon[106899]: pgmap v266: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:00 smithi111 ceph-mon[106899]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:00:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:00 smithi111 ceph-mon[106899]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:00:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:00 smithi111 ceph-mon[106899]: fs cephfs is degraded 2024-03-23T12:00:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:00 smithi111 ceph-mon[106899]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:00:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:00 smithi111 ceph-mon[106899]: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:00:03.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:02 smithi063 ceph-mon[141124]: pgmap v267: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:03.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:02 smithi111 ceph-mon[106899]: pgmap v267: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:05.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:04 smithi063 ceph-mon[141124]: pgmap v268: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:05.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:04 smithi111 ceph-mon[106899]: pgmap v268: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:06.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:06 smithi111 ceph-mon[106899]: pgmap v269: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:06.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:06 smithi063 ceph-mon[141124]: pgmap v269: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:08 smithi063 ceph-mon[141124]: pgmap v270: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:09.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:08 smithi111 ceph-mon[106899]: pgmap v270: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:10 smithi063 ceph-mon[141124]: pgmap v271: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:10 smithi111 ceph-mon[106899]: pgmap v271: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:13.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:12 smithi063 ceph-mon[141124]: pgmap v272: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:13.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:12 smithi111 ceph-mon[106899]: pgmap v272: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:15.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:14 smithi063 ceph-mon[141124]: pgmap v273: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:15.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:14 smithi111 ceph-mon[106899]: pgmap v273: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:16.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:16 smithi111 ceph-mon[106899]: pgmap v274: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:16.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:16 smithi063 ceph-mon[141124]: pgmap v274: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:19.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:18 smithi063 ceph-mon[141124]: pgmap v275: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:19.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:18 smithi111 ceph-mon[106899]: pgmap v275: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:21.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:20 smithi063 ceph-mon[141124]: pgmap v276: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:21.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:20 smithi111 ceph-mon[106899]: pgmap v276: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:23.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:22 smithi063 ceph-mon[141124]: pgmap v277: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:23.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:22 smithi111 ceph-mon[106899]: pgmap v277: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:24.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 -- 172.21.15.63:0/3788917588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c102e00 msgr2=0x7fc13c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 --2- 172.21.15.63:0/3788917588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c102e00 0x7fc13c1031d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc12c004940 tx=0x7fc12c009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 -- 172.21.15.63:0/3788917588 shutdown_connections 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 --2- 172.21.15.63:0/3788917588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13c1037a0 0x7fc13c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 --2- 172.21.15.63:0/3788917588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c102e00 0x7fc13c1031d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 -- 172.21.15.63:0/3788917588 >> 172.21.15.63:0/3788917588 conn(0x7fc13c0f89b0 msgr2=0x7fc13c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 -- 172.21.15.63:0/3788917588 shutdown_connections 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.417+0000 7fc140e00700 1 -- 172.21.15.63:0/3788917588 wait complete. 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.418+0000 7fc140e00700 1 Processor -- start 2024-03-23T12:00:24.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.418+0000 7fc140e00700 1 -- start start 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.418+0000 7fc140e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13c102e00 0x7fc13c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.418+0000 7fc140e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c1037a0 0x7fc13c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.418+0000 7fc140e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c116860 con 0x7fc13c1037a0 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.418+0000 7fc140e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10fc30 con 0x7fc13c102e00 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.419+0000 7fc139400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c1037a0 0x7fc13c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.419+0000 7fc139400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c1037a0 0x7fc13c116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52916/0 (socket says 172.21.15.63:52916) 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.419+0000 7fc139400700 1 -- 172.21.15.63:0/1158588094 learned_addr learned my addr 172.21.15.63:0/1158588094 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:24.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.419+0000 7fc139e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13c102e00 0x7fc13c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.419+0000 7fc139400700 1 -- 172.21.15.63:0/1158588094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13c102e00 msgr2=0x7fc13c115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.419+0000 7fc139400700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13c102e00 0x7fc13c115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.419+0000 7fc139400700 1 -- 172.21.15.63:0/1158588094 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c005040 con 0x7fc13c1037a0 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.420+0000 7fc139400700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c1037a0 0x7fc13c116210 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c890 tx=0x7fc1280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.420+0000 7fc132000700 1 -- 172.21.15.63:0/1158588094 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc12800a700 con 0x7fc13c1037a0 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.420+0000 7fc132000700 1 -- 172.21.15.63:0/1158588094 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc12800ad10 con 0x7fc13c1037a0 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.420+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c10ff10 con 0x7fc13c1037a0 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.420+0000 7fc132000700 1 -- 172.21.15.63:0/1158588094 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc128012e70 con 0x7fc13c1037a0 2024-03-23T12:00:24.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.420+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c1104d0 con 0x7fc13c1037a0 2024-03-23T12:00:24.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.421+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c0fe550 con 0x7fc13c1037a0 2024-03-23T12:00:24.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.422+0000 7fc132000700 1 -- 172.21.15.63:0/1158588094 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc12800a860 con 0x7fc13c1037a0 2024-03-23T12:00:24.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.423+0000 7fc132000700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc12006c7d0 0x7fc12006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:24.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.423+0000 7fc132000700 1 -- 172.21.15.63:0/1158588094 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc12808bae0 con 0x7fc13c1037a0 2024-03-23T12:00:24.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.423+0000 7fc139e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc12006c7d0 0x7fc12006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:24.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.424+0000 7fc139e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc12006c7d0 0x7fc12006ec80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b840 tx=0x7fc12c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:24.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.428+0000 7fc132000700 1 -- 172.21.15.63:0/1158588094 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc1280597f0 con 0x7fc13c1037a0 2024-03-23T12:00:24.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.664+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc13c109e30 con 0x7fc12006c7d0 2024-03-23T12:00:24.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.665+0000 7fc132000700 1 -- 172.21.15.63:0/1158588094 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc13c109e30 con 0x7fc12006c7d0 2024-03-23T12:00:24.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc12006c7d0 msgr2=0x7fc12006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:24.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc12006c7d0 0x7fc12006ec80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b840 tx=0x7fc12c009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:24.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c1037a0 msgr2=0x7fc13c116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c1037a0 0x7fc13c116210 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c890 tx=0x7fc1280044c0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 shutdown_connections 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc12006c7d0 0x7fc12006ec80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc13c102e00 0x7fc13c115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 --2- 172.21.15.63:0/1158588094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc13c1037a0 0x7fc13c116210 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 >> 172.21.15.63:0/1158588094 conn(0x7fc13c0f89b0 msgr2=0x7fc13c0f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 shutdown_connections 2024-03-23T12:00:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.668+0000 7fc140e00700 1 -- 172.21.15.63:0/1158588094 wait complete. 2024-03-23T12:00:24.665 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:00:24.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 -- 172.21.15.63:0/2387224885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 msgr2=0x7fae2c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:24.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 --2- 172.21.15.63:0/2387224885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c106f00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fae18004940 tx=0x7fae18009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:24.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 -- 172.21.15.63:0/2387224885 shutdown_connections 2024-03-23T12:00:24.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 --2- 172.21.15.63:0/2387224885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c106f00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 --2- 172.21.15.63:0/2387224885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae2c0fde90 0x7fae2c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 -- 172.21.15.63:0/2387224885 >> 172.21.15.63:0/2387224885 conn(0x7fae2c072bc0 msgr2=0x7fae2c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:24.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 -- 172.21.15.63:0/2387224885 shutdown_connections 2024-03-23T12:00:24.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.806+0000 7fae33a00700 1 -- 172.21.15.63:0/2387224885 wait complete. 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.807+0000 7fae33a00700 1 Processor -- start 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.807+0000 7fae33a00700 1 -- start start 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae33a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae2c0fde90 0x7fae2c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae33a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae33a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c1168a0 con 0x7fae2c0fe830 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae33a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c10fc70 con 0x7fae2c0fde90 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae2be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae2be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52938/0 (socket says 172.21.15.63:52938) 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae2be00700 1 -- 172.21.15.63:0/1822771693 learned_addr learned my addr 172.21.15.63:0/1822771693 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:24.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae31000700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae2c0fde90 0x7fae2c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:24.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae2be00700 1 -- 172.21.15.63:0/1822771693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae2c0fde90 msgr2=0x7fae2c115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:24.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae2be00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae2c0fde90 0x7fae2c115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:24.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.808+0000 7fae2be00700 1 -- 172.21.15.63:0/1822771693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae18005040 con 0x7fae2c0fe830 2024-03-23T12:00:24.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.809+0000 7fae2be00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c116250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae18000c00 tx=0x7fae18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:24.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.809+0000 7fae28c00700 1 -- 172.21.15.63:0/1822771693 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1800e040 con 0x7fae2c0fe830 2024-03-23T12:00:24.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.809+0000 7fae28c00700 1 -- 172.21.15.63:0/1822771693 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae1800bed0 con 0x7fae2c0fe830 2024-03-23T12:00:24.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.809+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c10fef0 con 0x7fae2c0fe830 2024-03-23T12:00:24.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.809+0000 7fae28c00700 1 -- 172.21.15.63:0/1822771693 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1801c3f0 con 0x7fae2c0fe830 2024-03-23T12:00:24.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.809+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c110430 con 0x7fae2c0fe830 2024-03-23T12:00:24.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.810+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c106a20 con 0x7fae2c0fe830 2024-03-23T12:00:24.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.811+0000 7fae28c00700 1 -- 172.21.15.63:0/1822771693 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fae18013420 con 0x7fae2c0fe830 2024-03-23T12:00:24.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.812+0000 7fae28c00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae1406c7d0 0x7fae1406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:24.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.812+0000 7fae28c00700 1 -- 172.21.15.63:0/1822771693 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fae1808cf20 con 0x7fae2c0fe830 2024-03-23T12:00:24.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.812+0000 7fae31000700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae1406c7d0 0x7fae1406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:24.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.813+0000 7fae31000700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae1406c7d0 0x7fae1406ec80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fae20004f40 tx=0x7fae2000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:24.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:24.817+0000 7fae28c00700 1 -- 172.21.15.63:0/1822771693 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fae1805abb0 con 0x7fae2c0fe830 2024-03-23T12:00:25.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.047+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae2c1113d0 con 0x7fae1406c7d0 2024-03-23T12:00:25.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.048+0000 7fae28c00700 1 -- 172.21.15.63:0/1822771693 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fae2c1113d0 con 0x7fae1406c7d0 2024-03-23T12:00:25.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.050+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae1406c7d0 msgr2=0x7fae1406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.050+0000 7fae33a00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae1406c7d0 0x7fae1406ec80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fae20004f40 tx=0x7fae2000b040 comp rx=0 tx=0).stop 2024-03-23T12:00:25.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.050+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 msgr2=0x7fae2c116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.050+0000 7fae33a00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c116250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae18000c00 tx=0x7fae18009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:25.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.051+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 shutdown_connections 2024-03-23T12:00:25.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.051+0000 7fae33a00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae1406c7d0 0x7fae1406ec80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.051+0000 7fae33a00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae2c0fde90 0x7fae2c115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.051+0000 7fae33a00700 1 --2- 172.21.15.63:0/1822771693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae2c0fe830 0x7fae2c116250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.051+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 >> 172.21.15.63:0/1822771693 conn(0x7fae2c072bc0 msgr2=0x7fae2c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:25.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.051+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 shutdown_connections 2024-03-23T12:00:25.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.051+0000 7fae33a00700 1 -- 172.21.15.63:0/1822771693 wait complete. 2024-03-23T12:00:25.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:24 smithi063 ceph-mon[141124]: pgmap v278: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:25.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:24 smithi111 ceph-mon[106899]: pgmap v278: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.186+0000 7f4e10800700 1 -- 172.21.15.63:0/4047267305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e08102e00 msgr2=0x7f4e081031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.186+0000 7f4e10800700 1 --2- 172.21.15.63:0/4047267305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e08102e00 0x7f4e081031d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc004940 tx=0x7f4dfc009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.187+0000 7f4e10800700 1 -- 172.21.15.63:0/4047267305 shutdown_connections 2024-03-23T12:00:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.187+0000 7f4e10800700 1 --2- 172.21.15.63:0/4047267305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e081037a0 0x7f4e080fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.187+0000 7f4e10800700 1 --2- 172.21.15.63:0/4047267305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e08102e00 0x7f4e081031d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.187+0000 7f4e10800700 1 -- 172.21.15.63:0/4047267305 >> 172.21.15.63:0/4047267305 conn(0x7f4e080f89b0 msgr2=0x7f4e080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.187+0000 7f4e10800700 1 -- 172.21.15.63:0/4047267305 shutdown_connections 2024-03-23T12:00:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.187+0000 7f4e10800700 1 -- 172.21.15.63:0/4047267305 wait complete. 2024-03-23T12:00:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e10800700 1 Processor -- start 2024-03-23T12:00:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e10800700 1 -- start start 2024-03-23T12:00:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e10800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e08102e00 0x7f4e08115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e10800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e081037a0 0x7f4e08116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e10800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e081167d0 con 0x7f4e081037a0 2024-03-23T12:00:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e10800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0810fba0 con 0x7f4e08102e00 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e0d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e081037a0 0x7f4e08116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e0d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e081037a0 0x7f4e08116180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52956/0 (socket says 172.21.15.63:52956) 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.188+0000 7f4e0d400700 1 -- 172.21.15.63:0/1351131208 learned_addr learned my addr 172.21.15.63:0/1351131208 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e0de00700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e08102e00 0x7f4e08115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e0d400700 1 -- 172.21.15.63:0/1351131208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e08102e00 msgr2=0x7f4e08115c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e0d400700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e08102e00 0x7f4e08115c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e0d400700 1 -- 172.21.15.63:0/1351131208 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc005040 con 0x7f4e081037a0 2024-03-23T12:00:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e0d400700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e081037a0 0x7f4e08116180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4df800eb40 tx=0x7f4df800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:25.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e02000700 1 -- 172.21.15.63:0/1351131208 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df8007cc0 con 0x7f4e081037a0 2024-03-23T12:00:25.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e02000700 1 -- 172.21.15.63:0/1351131208 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4df800f040 con 0x7f4e081037a0 2024-03-23T12:00:25.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0810fe20 con 0x7f4e081037a0 2024-03-23T12:00:25.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.189+0000 7f4e02000700 1 -- 172.21.15.63:0/1351131208 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df8009570 con 0x7f4e081037a0 2024-03-23T12:00:25.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.190+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e081103e0 con 0x7f4e081037a0 2024-03-23T12:00:25.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.191+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e080fe550 con 0x7f4e081037a0 2024-03-23T12:00:25.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.192+0000 7f4e02000700 1 -- 172.21.15.63:0/1351131208 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4df80096d0 con 0x7f4e081037a0 2024-03-23T12:00:25.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.192+0000 7f4e02000700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4df006c7d0 0x7f4df006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:25.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.192+0000 7f4e02000700 1 -- 172.21.15.63:0/1351131208 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4df808b650 con 0x7f4e081037a0 2024-03-23T12:00:25.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.193+0000 7f4e0de00700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4df006c7d0 0x7f4df006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:25.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.193+0000 7f4e0de00700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4df006c7d0 0x7f4df006ec80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00b0d0 tx=0x7f4dfc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:25.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.197+0000 7f4e02000700 1 -- 172.21.15.63:0/1351131208 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4df8059310 con 0x7f4e081037a0 2024-03-23T12:00:25.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.419+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e08109e30 con 0x7f4df006c7d0 2024-03-23T12:00:25.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.429+0000 7f4e02000700 1 -- 172.21.15.63:0/1351131208 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f4e08109e30 con 0x7f4df006c7d0 2024-03-23T12:00:25.410 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:00:25.410 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (9m) 2m ago 22m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:00:25.410 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (6m) 2m ago 23m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:00:25.410 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (6m) 2m ago 21m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:00:25.410 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (9m) 2m ago 22m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (2m) 2m ago 16m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (2m) 2m ago 16m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (16m) 2m ago 16m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (2m) 2m ago 16m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (12m) 2m ago 24m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (10m) 2m ago 21m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (8m) 2m ago 24m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (7m) 2m ago 21m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (10m) 2m ago 22m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (10m) 2m ago 21m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (6m) 2m ago 20m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (5m) 2m ago 20m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (5m) 2m ago 19m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:00:25.411 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (4m) 2m ago 19m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:00:25.412 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (3m) 2m ago 18m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:00:25.412 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (3m) 2m ago 18m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:00:25.412 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (10m) 2m ago 22m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:00:25.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4df006c7d0 msgr2=0x7f4df006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4df006c7d0 0x7f4df006ec80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00b0d0 tx=0x7f4dfc009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:25.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e081037a0 msgr2=0x7f4e08116180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e081037a0 0x7f4e08116180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4df800eb40 tx=0x7f4df800b040 comp rx=0 tx=0).stop 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 shutdown_connections 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4df006c7d0 0x7f4df006ec80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e08102e00 0x7f4e08115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 --2- 172.21.15.63:0/1351131208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e081037a0 0x7f4e08116180 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 >> 172.21.15.63:0/1351131208 conn(0x7f4e080f89b0 msgr2=0x7f4e080f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 shutdown_connections 2024-03-23T12:00:25.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.431+0000 7f4e10800700 1 -- 172.21.15.63:0/1351131208 wait complete. 2024-03-23T12:00:25.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2776243433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa4102e00 msgr2=0x7f3aa41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2776243433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa4102e00 0x7f3aa41031d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3a94004940 tx=0x7f3a94009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2776243433 shutdown_connections 2024-03-23T12:00:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2776243433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa41037a0 0x7f3aa40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2776243433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa4102e00 0x7f3aa41031d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2776243433 >> 172.21.15.63:0/2776243433 conn(0x7f3aa40f89b0 msgr2=0x7f3aa40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2776243433 shutdown_connections 2024-03-23T12:00:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.568+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2776243433 wait complete. 2024-03-23T12:00:25.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.569+0000 7f3aa9a00700 1 Processor -- start 2024-03-23T12:00:25.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.569+0000 7f3aa9a00700 1 -- start start 2024-03-23T12:00:25.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.569+0000 7f3aa9a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa4102e00 0x7f3aa4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.569+0000 7f3aa9a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa41037a0 0x7f3aa41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.569+0000 7f3aa9a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa41167f0 con 0x7f3aa41037a0 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.569+0000 7f3aa9a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa410fbc0 con 0x7f3aa4102e00 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.570+0000 7f3aa2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa41037a0 0x7f3aa41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.570+0000 7f3aa2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa41037a0 0x7f3aa41161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52984/0 (socket says 172.21.15.63:52984) 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.570+0000 7f3aa2000700 1 -- 172.21.15.63:0/2946673121 learned_addr learned my addr 172.21.15.63:0/2946673121 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.570+0000 7f3aa2a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa4102e00 0x7f3aa4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.570+0000 7f3aa2000700 1 -- 172.21.15.63:0/2946673121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa4102e00 msgr2=0x7f3aa4115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.570+0000 7f3aa2000700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa4102e00 0x7f3aa4115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.570+0000 7f3aa2000700 1 -- 172.21.15.63:0/2946673121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a94005040 con 0x7f3aa41037a0 2024-03-23T12:00:25.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.571+0000 7f3aa2000700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa41037a0 0x7f3aa41161a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000eb40 tx=0x7f3a9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:25.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.571+0000 7f3a9aa00700 1 -- 172.21.15.63:0/2946673121 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a90007cc0 con 0x7f3aa41037a0 2024-03-23T12:00:25.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.571+0000 7f3a9aa00700 1 -- 172.21.15.63:0/2946673121 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a9000f040 con 0x7f3aa41037a0 2024-03-23T12:00:25.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.571+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa410fe40 con 0x7f3aa41037a0 2024-03-23T12:00:25.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.571+0000 7f3a9aa00700 1 -- 172.21.15.63:0/2946673121 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a9000ca30 con 0x7f3aa41037a0 2024-03-23T12:00:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.571+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa4110400 con 0x7f3aa41037a0 2024-03-23T12:00:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.572+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa40fe550 con 0x7f3aa41037a0 2024-03-23T12:00:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.573+0000 7f3a9aa00700 1 -- 172.21.15.63:0/2946673121 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3a9000cb90 con 0x7f3aa41037a0 2024-03-23T12:00:25.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.574+0000 7f3a9aa00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3a8806c7d0 0x7f3a8806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:25.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.574+0000 7f3a9aa00700 1 -- 172.21.15.63:0/2946673121 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3a9008b5e0 con 0x7f3aa41037a0 2024-03-23T12:00:25.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.574+0000 7f3aa2a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3a8806c7d0 0x7f3a8806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.575+0000 7f3aa2a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3a8806c7d0 0x7f3a8806ec80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3a940045a0 tx=0x7f3a94009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:25.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.579+0000 7f3a9aa00700 1 -- 172.21.15.63:0/2946673121 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3a90059270 con 0x7f3aa41037a0 2024-03-23T12:00:25.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.877+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3aa4109de0 con 0x7f3aa41037a0 2024-03-23T12:00:25.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.878+0000 7f3a9aa00700 1 -- 172.21.15.63:0/2946673121 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f3a90058e00 con 0x7f3aa41037a0 2024-03-23T12:00:25.859 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:00:25.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:25.861 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:00:25.861 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:00:25.861 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:00:25.861 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:00:25.861 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3a8806c7d0 msgr2=0x7f3a8806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3a8806c7d0 0x7f3a8806ec80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3a940045a0 tx=0x7f3a94009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa41037a0 msgr2=0x7f3aa41161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa41037a0 0x7f3aa41161a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000eb40 tx=0x7f3a9000b040 comp rx=0 tx=0).stop 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 shutdown_connections 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3a8806c7d0 0x7f3a8806ec80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa4102e00 0x7f3aa4115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 --2- 172.21.15.63:0/2946673121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3aa41037a0 0x7f3aa41161a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.880+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 >> 172.21.15.63:0/2946673121 conn(0x7f3aa40f89b0 msgr2=0x7f3aa40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:25.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.881+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 shutdown_connections 2024-03-23T12:00:25.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:25.881+0000 7f3aa9a00700 1 -- 172.21.15.63:0/2946673121 wait complete. 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 -- 172.21.15.63:0/2512511168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 msgr2=0x7fdab00fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 --2- 172.21.15.63:0/2512511168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fe2a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0004940 tx=0x7fdaa0009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 -- 172.21.15.63:0/2512511168 shutdown_connections 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 --2- 172.21.15.63:0/2512511168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdab00fe870 0x7fdab0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 --2- 172.21.15.63:0/2512511168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fe2a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 -- 172.21.15.63:0/2512511168 >> 172.21.15.63:0/2512511168 conn(0x7fdab0072bc0 msgr2=0x7fdab0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 -- 172.21.15.63:0/2512511168 shutdown_connections 2024-03-23T12:00:26.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.018+0000 7fdab6800700 1 -- 172.21.15.63:0/2512511168 wait complete. 2024-03-23T12:00:26.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.019+0000 7fdab6800700 1 Processor -- start 2024-03-23T12:00:26.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.019+0000 7fdab6800700 1 -- start start 2024-03-23T12:00:26.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdab6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdab6800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdab00fe870 0x7fdab00ff490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdab6800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab01030e0 con 0x7fdab00fded0 2024-03-23T12:00:26.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdab6800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00ff9d0 con 0x7fdab00fe870 2024-03-23T12:00:26.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52992/0 (socket says 172.21.15.63:52992) 2024-03-23T12:00:26.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaf400700 1 -- 172.21.15.63:0/4189899574 learned_addr learned my addr 172.21.15.63:0/4189899574 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaf400700 1 -- 172.21.15.63:0/4189899574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdab00fe870 msgr2=0x7fdab00ff490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaea00700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdab00fe870 0x7fdab00ff490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaf400700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdab00fe870 0x7fdab00ff490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaf400700 1 -- 172.21.15.63:0/4189899574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa0005040 con 0x7fdab00fded0 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.020+0000 7fdaaea00700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdab00fe870 0x7fdab00ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.021+0000 7fdaaf400700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fef50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009390 tx=0x7fdaa0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.021+0000 7fdaa7400700 1 -- 172.21.15.63:0/4189899574 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa000e040 con 0x7fdab00fded0 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.021+0000 7fdaa7400700 1 -- 172.21.15.63:0/4189899574 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaa0013420 con 0x7fdab00fded0 2024-03-23T12:00:26.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.021+0000 7fdaa7400700 1 -- 172.21.15.63:0/4189899574 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa001c630 con 0x7fdab00fded0 2024-03-23T12:00:26.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.021+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab00ffcb0 con 0x7fdab00fded0 2024-03-23T12:00:26.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.021+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab011e560 con 0x7fdab00fded0 2024-03-23T12:00:26.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.022+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab0106a00 con 0x7fdab00fded0 2024-03-23T12:00:26.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.023+0000 7fdaa7400700 1 -- 172.21.15.63:0/4189899574 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdaa0013a20 con 0x7fdab00fded0 2024-03-23T12:00:26.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.024+0000 7fdaa7400700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda9406c7d0 0x7fda9406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.024+0000 7fdaa7400700 1 -- 172.21.15.63:0/4189899574 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdaa008ccb0 con 0x7fdab00fded0 2024-03-23T12:00:26.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.024+0000 7fdaaea00700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda9406c7d0 0x7fda9406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.025+0000 7fdaaea00700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda9406c7d0 0x7fda9406ec80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009fd0 tx=0x7fda9c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:26.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.029+0000 7fdaa7400700 1 -- 172.21.15.63:0/4189899574 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdaa0056f30 con 0x7fdab00fded0 2024-03-23T12:00:26.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:25 smithi063 ceph-mon[141124]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:26.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:26.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:25 smithi111 ceph-mon[106899]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:26.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:26.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.286+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdab0106a00 con 0x7fdab00fded0 2024-03-23T12:00:26.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.287+0000 7fdaa7400700 1 -- 172.21.15.63:0/4189899574 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fdaa005a550 con 0x7fdab00fded0 2024-03-23T12:00:26.268 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:00:26.268 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:00:26.268 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:00:26.269 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:00:26.270 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda9406c7d0 msgr2=0x7fda9406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda9406c7d0 0x7fda9406ec80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009fd0 tx=0x7fda9c00b040 comp rx=0 tx=0).stop 2024-03-23T12:00:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 msgr2=0x7fdab00fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fef50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009390 tx=0x7fdaa0009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 shutdown_connections 2024-03-23T12:00:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda9406c7d0 0x7fda9406ec80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdab00fded0 0x7fdab00fef50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 --2- 172.21.15.63:0/4189899574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdab00fe870 0x7fdab00ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 >> 172.21.15.63:0/4189899574 conn(0x7fdab0072bc0 msgr2=0x7fdab00fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:26.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 shutdown_connections 2024-03-23T12:00:26.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.289+0000 7fdab6800700 1 -- 172.21.15.63:0/4189899574 wait complete. 2024-03-23T12:00:26.273 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:00:26.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.426+0000 7f2ac7600700 1 -- 172.21.15.63:0/3871800842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 msgr2=0x7f2ac00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.426+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3871800842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac00fce10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2aac004940 tx=0x7f2aac009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:26.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.426+0000 7f2ac7600700 1 -- 172.21.15.63:0/3871800842 shutdown_connections 2024-03-23T12:00:26.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.426+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3871800842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac00fce10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.426+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3871800842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ac0102e00 0x7f2ac01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.426+0000 7f2ac7600700 1 -- 172.21.15.63:0/3871800842 >> 172.21.15.63:0/3871800842 conn(0x7f2ac00f89b0 msgr2=0x7f2ac00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:26.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.427+0000 7f2ac7600700 1 -- 172.21.15.63:0/3871800842 shutdown_connections 2024-03-23T12:00:26.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.427+0000 7f2ac7600700 1 -- 172.21.15.63:0/3871800842 wait complete. 2024-03-23T12:00:26.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.427+0000 7f2ac7600700 1 Processor -- start 2024-03-23T12:00:26.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2ac7600700 1 -- start start 2024-03-23T12:00:26.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2ac7600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ac0102e00 0x7f2ac0115db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2ac7600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac01162f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2ac7600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac0116940 con 0x7f2ac01037a0 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2ac7600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac010fd10 con 0x7f2ac0102e00 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2abfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac01162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2abfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac01162f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53020/0 (socket says 172.21.15.63:53020) 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.428+0000 7f2abfe00700 1 -- 172.21.15.63:0/3590570439 learned_addr learned my addr 172.21.15.63:0/3590570439 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2ac4c00700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ac0102e00 0x7f2ac0115db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2abfe00700 1 -- 172.21.15.63:0/3590570439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ac0102e00 msgr2=0x7f2ac0115db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2abfe00700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ac0102e00 0x7f2ac0115db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2abfe00700 1 -- 172.21.15.63:0/3590570439 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac005040 con 0x7f2ac01037a0 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2abfe00700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac01162f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009390 tx=0x7f2aac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2abcc00700 1 -- 172.21.15.63:0/3590570439 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aac00e040 con 0x7f2ac01037a0 2024-03-23T12:00:26.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2abcc00700 1 -- 172.21.15.63:0/3590570439 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aac013420 con 0x7f2ac01037a0 2024-03-23T12:00:26.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac010ff90 con 0x7f2ac01037a0 2024-03-23T12:00:26.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2abcc00700 1 -- 172.21.15.63:0/3590570439 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aac01c630 con 0x7f2ac01037a0 2024-03-23T12:00:26.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.429+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac01104d0 con 0x7f2ac01037a0 2024-03-23T12:00:26.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.431+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac00fe550 con 0x7f2ac01037a0 2024-03-23T12:00:26.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.431+0000 7f2abcc00700 1 -- 172.21.15.63:0/3590570439 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2aac013a00 con 0x7f2ac01037a0 2024-03-23T12:00:26.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.432+0000 7f2abcc00700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2aa806c7d0 0x7f2aa806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.432+0000 7f2abcc00700 1 -- 172.21.15.63:0/3590570439 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2aac08cd10 con 0x7f2ac01037a0 2024-03-23T12:00:26.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.433+0000 7f2ac4c00700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2aa806c7d0 0x7f2aa806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.433+0000 7f2ac4c00700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2aa806c7d0 0x7f2aa806ec80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4004d60 tx=0x7f2ab400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:26.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.437+0000 7f2abcc00700 1 -- 172.21.15.63:0/3590570439 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2aac05aa20 con 0x7f2ac01037a0 2024-03-23T12:00:26.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.670+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ac0109e30 con 0x7f2aa806c7d0 2024-03-23T12:00:26.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.672+0000 7f2abcc00700 1 -- 172.21.15.63:0/3590570439 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2ac0109e30 con 0x7f2aa806c7d0 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:00:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:00:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:00:26.657 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:00:26.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2aa806c7d0 msgr2=0x7f2aa806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2aa806c7d0 0x7f2aa806ec80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4004d60 tx=0x7f2ab400b040 comp rx=0 tx=0).stop 2024-03-23T12:00:26.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 msgr2=0x7f2ac01162f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac01162f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009390 tx=0x7f2aac009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 shutdown_connections 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2aa806c7d0 0x7f2aa806ec80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ac0102e00 0x7f2ac0115db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 --2- 172.21.15.63:0/3590570439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01037a0 0x7f2ac01162f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 >> 172.21.15.63:0/3590570439 conn(0x7f2ac00f89b0 msgr2=0x7f2ac00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 shutdown_connections 2024-03-23T12:00:26.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.674+0000 7f2ac7600700 1 -- 172.21.15.63:0/3590570439 wait complete. 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 -- 172.21.15.63:0/930366168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 msgr2=0x7f8c441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/930366168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c441031d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8c38004940 tx=0x7f8c38009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 -- 172.21.15.63:0/930366168 shutdown_connections 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/930366168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c441037a0 0x7f8c440fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/930366168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c441031d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 -- 172.21.15.63:0/930366168 >> 172.21.15.63:0/930366168 conn(0x7f8c440f89b0 msgr2=0x7f8c440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 -- 172.21.15.63:0/930366168 shutdown_connections 2024-03-23T12:00:26.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.815+0000 7f8c4bc00700 1 -- 172.21.15.63:0/930366168 wait complete. 2024-03-23T12:00:26.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.816+0000 7f8c4bc00700 1 Processor -- start 2024-03-23T12:00:26.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.816+0000 7f8c4bc00700 1 -- start start 2024-03-23T12:00:26.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c4bc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c44115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c4bc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c441037a0 0x7f8c441161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c4bc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44116800 con 0x7f8c44102e00 2024-03-23T12:00:26.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c4bc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4410fbd0 con 0x7f8c441037a0 2024-03-23T12:00:26.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c49200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c44115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c49200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c44115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53036/0 (socket says 172.21.15.63:53036) 2024-03-23T12:00:26.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c49200700 1 -- 172.21.15.63:0/3099398573 learned_addr learned my addr 172.21.15.63:0/3099398573 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:26.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c49200700 1 -- 172.21.15.63:0/3099398573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c441037a0 msgr2=0x7f8c441161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:26.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c49200700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c441037a0 0x7f8c441161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c49200700 1 -- 172.21.15.63:0/3099398573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c38005040 con 0x7f8c44102e00 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.817+0000 7f8c49200700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c44115c70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8c38008d80 tx=0x7f8c38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.818+0000 7f8c40c00700 1 -- 172.21.15.63:0/3099398573 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3800e030 con 0x7f8c44102e00 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.818+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4410fe50 con 0x7f8c44102e00 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.818+0000 7f8c40c00700 1 -- 172.21.15.63:0/3099398573 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c38013de0 con 0x7f8c44102e00 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.818+0000 7f8c40c00700 1 -- 172.21.15.63:0/3099398573 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3801cd60 con 0x7f8c44102e00 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.818+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c44110410 con 0x7f8c44102e00 2024-03-23T12:00:26.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.819+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c440fe550 con 0x7f8c44102e00 2024-03-23T12:00:26.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.820+0000 7f8c40c00700 1 -- 172.21.15.63:0/3099398573 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8c3801c590 con 0x7f8c44102e00 2024-03-23T12:00:26.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.821+0000 7f8c40c00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8c2c06c7d0 0x7f8c2c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:26.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.821+0000 7f8c40c00700 1 -- 172.21.15.63:0/3099398573 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c3808d360 con 0x7f8c44102e00 2024-03-23T12:00:26.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.821+0000 7f8c43e00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8c2c06c7d0 0x7f8c2c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:26.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.822+0000 7f8c43e00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8c2c06c7d0 0x7f8c2c06ec80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8c44111510 tx=0x7f8c3000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:26.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:26.825+0000 7f8c40c00700 1 -- 172.21.15.63:0/3099398573 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8c3805b070 con 0x7f8c44102e00 2024-03-23T12:00:27.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.132+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8c4404bed0 con 0x7f8c44102e00 2024-03-23T12:00:27.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.133+0000 7f8c40c00700 1 -- 172.21.15.63:0/3099398573 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f8c3805ac00 con 0x7f8c44102e00 2024-03-23T12:00:27.115 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:00:27.115 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:00:27.115 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:00:27.116 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:00:27.116 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8c2c06c7d0 msgr2=0x7f8c2c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8c2c06c7d0 0x7f8c2c06ec80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8c44111510 tx=0x7f8c3000b040 comp rx=0 tx=0).stop 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 msgr2=0x7f8c44115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c44115c70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8c38008d80 tx=0x7f8c38009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 shutdown_connections 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8c2c06c7d0 0x7f8c2c06ec80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c44102e00 0x7f8c44115c70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 --2- 172.21.15.63:0/3099398573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c441037a0 0x7f8c441161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 >> 172.21.15.63:0/3099398573 conn(0x7f8c440f89b0 msgr2=0x7f8c440f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 shutdown_connections 2024-03-23T12:00:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:27.135+0000 7f8c4bc00700 1 -- 172.21.15.63:0/3099398573 wait complete. 2024-03-23T12:00:27.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:26 smithi063 ceph-mon[141124]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:27.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:26 smithi063 ceph-mon[141124]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:27.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:26 smithi063 ceph-mon[141124]: pgmap v279: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:27.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:26 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2946673121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:00:27.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:26 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4189899574' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:00:27.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:26 smithi111 ceph-mon[106899]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:27.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:26 smithi111 ceph-mon[106899]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:26 smithi111 ceph-mon[106899]: pgmap v279: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:26 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2946673121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:00:27.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:26 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4189899574' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:00:28.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:27 smithi063 ceph-mon[141124]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:28.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:27 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3099398573' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:00:28.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:27 smithi111 ceph-mon[106899]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:28.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:27 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3099398573' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:00:29.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:28 smithi063 ceph-mon[141124]: pgmap v280: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:29.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:28 smithi111 ceph-mon[106899]: pgmap v280: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:31.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:30 smithi063 ceph-mon[141124]: pgmap v281: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:31.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:30 smithi111 ceph-mon[106899]: pgmap v281: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:32 smithi063 ceph-mon[141124]: pgmap v282: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:33.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:32 smithi111 ceph-mon[106899]: pgmap v282: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:35.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:34 smithi063 ceph-mon[141124]: pgmap v283: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:35.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:34 smithi111 ceph-mon[106899]: pgmap v283: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:36.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:36 smithi111 ceph-mon[106899]: pgmap v284: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:36.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:36 smithi063 ceph-mon[141124]: pgmap v284: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:39.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:38 smithi063 ceph-mon[141124]: pgmap v285: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:39.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:38 smithi111 ceph-mon[106899]: pgmap v285: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:40 smithi063 ceph-mon[141124]: pgmap v286: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:41.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:41.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:40 smithi111 ceph-mon[106899]: pgmap v286: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:41.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:43.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:42 smithi063 ceph-mon[141124]: pgmap v287: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:43.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:42 smithi111 ceph-mon[106899]: pgmap v287: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:45.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:44 smithi063 ceph-mon[141124]: pgmap v288: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:45.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:44 smithi111 ceph-mon[106899]: pgmap v288: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:46.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:46 smithi111 ceph-mon[106899]: pgmap v289: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:46.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:46 smithi063 ceph-mon[141124]: pgmap v289: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:49.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:48 smithi063 ceph-mon[141124]: pgmap v290: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:49.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:48 smithi111 ceph-mon[106899]: pgmap v290: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:51.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:50 smithi063 ceph-mon[141124]: pgmap v291: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:51.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:50 smithi111 ceph-mon[106899]: pgmap v291: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:53.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:52 smithi063 ceph-mon[141124]: pgmap v292: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:53.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:52 smithi111 ceph-mon[106899]: pgmap v292: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:55.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:54 smithi063 ceph-mon[141124]: pgmap v293: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:55.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:54 smithi111 ceph-mon[106899]: pgmap v293: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:56.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:56.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:56 smithi111 ceph-mon[106899]: pgmap v294: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:00:56.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:56 smithi063 ceph-mon[141124]: pgmap v294: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.289+0000 7fcafb400700 1 -- 172.21.15.63:0/2248306961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf4102e00 msgr2=0x7fcaf41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.289+0000 7fcafb400700 1 --2- 172.21.15.63:0/2248306961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf4102e00 0x7fcaf41031d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcae8004940 tx=0x7fcae8009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.289+0000 7fcafb400700 1 -- 172.21.15.63:0/2248306961 shutdown_connections 2024-03-23T12:00:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.289+0000 7fcafb400700 1 --2- 172.21.15.63:0/2248306961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaf41037a0 0x7fcaf40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.289+0000 7fcafb400700 1 --2- 172.21.15.63:0/2248306961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf4102e00 0x7fcaf41031d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.289+0000 7fcafb400700 1 -- 172.21.15.63:0/2248306961 >> 172.21.15.63:0/2248306961 conn(0x7fcaf40f89b0 msgr2=0x7fcaf40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.290+0000 7fcafb400700 1 -- 172.21.15.63:0/2248306961 shutdown_connections 2024-03-23T12:00:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.290+0000 7fcafb400700 1 -- 172.21.15.63:0/2248306961 wait complete. 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.290+0000 7fcafb400700 1 Processor -- start 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcafb400700 1 -- start start 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcafb400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaf4102e00 0x7fcaf4115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcafb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf41037a0 0x7fcaf4116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcafb400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf41168a0 con 0x7fcaf41037a0 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcafb400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf410fc70 con 0x7fcaf4102e00 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcaf3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf41037a0 0x7fcaf4116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcaf3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf41037a0 0x7fcaf4116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37604/0 (socket says 172.21.15.63:37604) 2024-03-23T12:00:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.291+0000 7fcaf3e00700 1 -- 172.21.15.63:0/843800095 learned_addr learned my addr 172.21.15.63:0/843800095 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf8a00700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaf4102e00 0x7fcaf4115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf3e00700 1 -- 172.21.15.63:0/843800095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaf4102e00 msgr2=0x7fcaf4115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf3e00700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaf4102e00 0x7fcaf4115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf3e00700 1 -- 172.21.15.63:0/843800095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae8005040 con 0x7fcaf41037a0 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf3e00700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf41037a0 0x7fcaf4116250 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcae0002a50 tx=0x7fcae0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf0c00700 1 -- 172.21.15.63:0/843800095 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcae0007cc0 con 0x7fcaf41037a0 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf0c00700 1 -- 172.21.15.63:0/843800095 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcae000f040 con 0x7fcaf41037a0 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcaf0c00700 1 -- 172.21.15.63:0/843800095 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcae000d920 con 0x7fcaf41037a0 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.292+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf410ff50 con 0x7fcaf41037a0 2024-03-23T12:00:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.293+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf4110510 con 0x7fcaf41037a0 2024-03-23T12:00:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.294+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf40fe550 con 0x7fcaf41037a0 2024-03-23T12:00:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.295+0000 7fcaf0c00700 1 -- 172.21.15.63:0/843800095 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fcae0009680 con 0x7fcaf41037a0 2024-03-23T12:00:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.296+0000 7fcaf0c00700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcadc06c7d0 0x7fcadc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.296+0000 7fcaf0c00700 1 -- 172.21.15.63:0/843800095 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcae008b750 con 0x7fcaf41037a0 2024-03-23T12:00:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.296+0000 7fcaf8a00700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcadc06c7d0 0x7fcadc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:57.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.296+0000 7fcaf8a00700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcadc06c7d0 0x7fcadc06ec80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009fd0 tx=0x7fcae8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.300+0000 7fcaf0c00700 1 -- 172.21.15.63:0/843800095 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcae00593e0 con 0x7fcaf41037a0 2024-03-23T12:00:57.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.533+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcaf4109e30 con 0x7fcadc06c7d0 2024-03-23T12:00:57.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.534+0000 7fcaf0c00700 1 -- 172.21.15.63:0/843800095 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcaf4109e30 con 0x7fcadc06c7d0 2024-03-23T12:00:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcadc06c7d0 msgr2=0x7fcadc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcadc06c7d0 0x7fcadc06ec80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009fd0 tx=0x7fcae8009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf41037a0 msgr2=0x7fcaf4116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf41037a0 0x7fcaf4116250 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcae0002a50 tx=0x7fcae0004600 comp rx=0 tx=0).stop 2024-03-23T12:00:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 shutdown_connections 2024-03-23T12:00:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcadc06c7d0 0x7fcadc06ec80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaf4102e00 0x7fcaf4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 --2- 172.21.15.63:0/843800095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaf41037a0 0x7fcaf4116250 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 >> 172.21.15.63:0/843800095 conn(0x7fcaf40f89b0 msgr2=0x7fcaf40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 shutdown_connections 2024-03-23T12:00:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.537+0000 7fcafb400700 1 -- 172.21.15.63:0/843800095 wait complete. 2024-03-23T12:00:57.534 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 -- 172.21.15.63:0/2863095199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fe830 msgr2=0x7fc7c0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 --2- 172.21.15.63:0/2863095199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fe830 0x7fc7c0106f00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac004940 tx=0x7fc7ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 -- 172.21.15.63:0/2863095199 shutdown_connections 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 --2- 172.21.15.63:0/2863095199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fe830 0x7fc7c0106f00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 --2- 172.21.15.63:0/2863095199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c00fde90 0x7fc7c00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 -- 172.21.15.63:0/2863095199 >> 172.21.15.63:0/2863095199 conn(0x7fc7c0072bc0 msgr2=0x7fc7c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 -- 172.21.15.63:0/2863095199 shutdown_connections 2024-03-23T12:00:57.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.676+0000 7fc7c7800700 1 -- 172.21.15.63:0/2863095199 wait complete. 2024-03-23T12:00:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.677+0000 7fc7c7800700 1 Processor -- start 2024-03-23T12:00:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.677+0000 7fc7c7800700 1 -- start start 2024-03-23T12:00:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.677+0000 7fc7c7800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fde90 0x7fc7c0115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.677+0000 7fc7c7800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c00fe830 0x7fc7c0116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.677+0000 7fc7c7800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01168e0 con 0x7fc7c00fde90 2024-03-23T12:00:57.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.677+0000 7fc7c7800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c010fcb0 con 0x7fc7c00fe830 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7c4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fde90 0x7fc7c0115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7c4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fde90 0x7fc7c0115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37624/0 (socket says 172.21.15.63:37624) 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7c4e00700 1 -- 172.21.15.63:0/1667519836 learned_addr learned my addr 172.21.15.63:0/1667519836 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7bfe00700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c00fe830 0x7fc7c0116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7c4e00700 1 -- 172.21.15.63:0/1667519836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c00fe830 msgr2=0x7fc7c0116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7c4e00700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c00fe830 0x7fc7c0116290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7c4e00700 1 -- 172.21.15.63:0/1667519836 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac005040 con 0x7fc7c00fde90 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.678+0000 7fc7c4e00700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fde90 0x7fc7c0115d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4002a50 tx=0x7fc7b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.679+0000 7fc7bcc00700 1 -- 172.21.15.63:0/1667519836 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b4007cc0 con 0x7fc7c00fde90 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.679+0000 7fc7bcc00700 1 -- 172.21.15.63:0/1667519836 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7b400f040 con 0x7fc7c00fde90 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.679+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c010ff90 con 0x7fc7c00fde90 2024-03-23T12:00:57.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.679+0000 7fc7bcc00700 1 -- 172.21.15.63:0/1667519836 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b400d9c0 con 0x7fc7c00fde90 2024-03-23T12:00:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.679+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c01104d0 con 0x7fc7c00fde90 2024-03-23T12:00:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.680+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c0106a20 con 0x7fc7c00fde90 2024-03-23T12:00:57.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.681+0000 7fc7bcc00700 1 -- 172.21.15.63:0/1667519836 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc7b400db60 con 0x7fc7c00fde90 2024-03-23T12:00:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.682+0000 7fc7bcc00700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7a806c7d0 0x7fc7a806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.682+0000 7fc7bcc00700 1 -- 172.21.15.63:0/1667519836 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc7b408b120 con 0x7fc7c00fde90 2024-03-23T12:00:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.683+0000 7fc7bfe00700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7a806c7d0 0x7fc7a806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:57.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.683+0000 7fc7bfe00700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7a806c7d0 0x7fc7a806ec80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7c01115f0 tx=0x7fc7ac009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:57.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.687+0000 7fc7bcc00700 1 -- 172.21.15.63:0/1667519836 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc7b4058db0 con 0x7fc7c00fde90 2024-03-23T12:00:57.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.923+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7c0109df0 con 0x7fc7a806c7d0 2024-03-23T12:00:57.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.924+0000 7fc7bcc00700 1 -- 172.21.15.63:0/1667519836 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc7c0109df0 con 0x7fc7a806c7d0 2024-03-23T12:00:57.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7a806c7d0 msgr2=0x7fc7a806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7a806c7d0 0x7fc7a806ec80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7c01115f0 tx=0x7fc7ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:57.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fde90 msgr2=0x7fc7c0115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:57.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fde90 0x7fc7c0115d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4002a50 tx=0x7fc7b40046b0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 shutdown_connections 2024-03-23T12:00:57.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc7a806c7d0 0x7fc7a806ec80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7c00fde90 0x7fc7c0115d50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 --2- 172.21.15.63:0/1667519836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c00fe830 0x7fc7c0116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:57.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 >> 172.21.15.63:0/1667519836 conn(0x7fc7c0072bc0 msgr2=0x7fc7c00fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:57.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 shutdown_connections 2024-03-23T12:00:57.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:57.927+0000 7fc7c7800700 1 -- 172.21.15.63:0/1667519836 wait complete. 2024-03-23T12:00:58.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 -- 172.21.15.63:0/3613880670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 msgr2=0x7ff9b81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 --2- 172.21.15.63:0/3613880670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b81031d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8004940 tx=0x7ff9a8009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 -- 172.21.15.63:0/3613880670 shutdown_connections 2024-03-23T12:00:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 --2- 172.21.15.63:0/3613880670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b81037a0 0x7ff9b80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 --2- 172.21.15.63:0/3613880670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b81031d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 -- 172.21.15.63:0/3613880670 >> 172.21.15.63:0/3613880670 conn(0x7ff9b80f89b0 msgr2=0x7ff9b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 -- 172.21.15.63:0/3613880670 shutdown_connections 2024-03-23T12:00:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.064+0000 7ff9be000700 1 -- 172.21.15.63:0/3613880670 wait complete. 2024-03-23T12:00:58.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.065+0000 7ff9be000700 1 Processor -- start 2024-03-23T12:00:58.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.065+0000 7ff9be000700 1 -- start start 2024-03-23T12:00:58.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.065+0000 7ff9be000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b8115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.065+0000 7ff9be000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b81037a0 0x7ff9b81161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.065+0000 7ff9be000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b8116820 con 0x7ff9b8102e00 2024-03-23T12:00:58.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.065+0000 7ff9be000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b810fbf0 con 0x7ff9b81037a0 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9b7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b8115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9b7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b8115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37636/0 (socket says 172.21.15.63:37636) 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9b7400700 1 -- 172.21.15.63:0/932862279 learned_addr learned my addr 172.21.15.63:0/932862279 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9afe00700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b81037a0 0x7ff9b81161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9b7400700 1 -- 172.21.15.63:0/932862279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b81037a0 msgr2=0x7ff9b81161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9b7400700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b81037a0 0x7ff9b81161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9b7400700 1 -- 172.21.15.63:0/932862279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8005040 con 0x7ff9b8102e00 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9afe00700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b81037a0 0x7ff9b81161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.066+0000 7ff9b7400700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b8115c90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8007f50 tx=0x7ff9a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.067+0000 7ff9af400700 1 -- 172.21.15.63:0/932862279 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9a8021070 con 0x7ff9b8102e00 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.067+0000 7ff9af400700 1 -- 172.21.15.63:0/932862279 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9a8013e00 con 0x7ff9b8102e00 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.067+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b810fe70 con 0x7ff9b8102e00 2024-03-23T12:00:58.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.067+0000 7ff9af400700 1 -- 172.21.15.63:0/932862279 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9a800e070 con 0x7ff9b8102e00 2024-03-23T12:00:58.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.067+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b8110430 con 0x7ff9b8102e00 2024-03-23T12:00:58.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.068+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b80fe550 con 0x7ff9b8102e00 2024-03-23T12:00:58.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.069+0000 7ff9af400700 1 -- 172.21.15.63:0/932862279 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff9a8013950 con 0x7ff9b8102e00 2024-03-23T12:00:58.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.070+0000 7ff9af400700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff99c06c7d0 0x7ff99c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.070+0000 7ff9af400700 1 -- 172.21.15.63:0/932862279 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff9a808cbf0 con 0x7ff9b8102e00 2024-03-23T12:00:58.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.070+0000 7ff9afe00700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff99c06c7d0 0x7ff99c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.071+0000 7ff9afe00700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff99c06c7d0 0x7ff99c06ec80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0004600 tx=0x7ff9a000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:58.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.074+0000 7ff9af400700 1 -- 172.21.15.63:0/932862279 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff9a805a9c0 con 0x7ff9b8102e00 2024-03-23T12:00:58.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.296+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9b80fe550 con 0x7ff99c06c7d0 2024-03-23T12:00:58.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.306+0000 7ff9af400700 1 -- 172.21.15.63:0/932862279 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7ff9b80fe550 con 0x7ff99c06c7d0 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (10m) 2m ago 23m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (7m) 2m ago 23m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (7m) 2m ago 22m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (9m) 2m ago 23m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (3m) 2m ago 17m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (3m) 2m ago 16m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (16m) 2m ago 16m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (2m) 2m ago 17m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (12m) 2m ago 24m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (11m) 2m ago 22m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:00:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (8m) 2m ago 25m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (8m) 2m ago 22m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (11m) 2m ago 23m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (10m) 2m ago 22m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (6m) 2m ago 21m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (6m) 2m ago 20m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (5m) 2m ago 20m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (4m) 2m ago 19m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (4m) 2m ago 19m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (3m) 2m ago 18m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:00:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (10m) 2m ago 22m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff99c06c7d0 msgr2=0x7ff99c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff99c06c7d0 0x7ff99c06ec80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0004600 tx=0x7ff9a000b040 comp rx=0 tx=0).stop 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 msgr2=0x7ff9b8115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b8115c90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8007f50 tx=0x7ff9a8009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 shutdown_connections 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff99c06c7d0 0x7ff99c06ec80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9b8102e00 0x7ff9b8115c90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 --2- 172.21.15.63:0/932862279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b81037a0 0x7ff9b81161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 >> 172.21.15.63:0/932862279 conn(0x7ff9b80f89b0 msgr2=0x7ff9b80f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 shutdown_connections 2024-03-23T12:00:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.308+0000 7ff9be000700 1 -- 172.21.15.63:0/932862279 wait complete. 2024-03-23T12:00:58.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 -- 172.21.15.63:0/3913275344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424065630 msgr2=0x7f6424065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 --2- 172.21.15.63:0/3913275344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424065630 0x7f6424065a80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6410004940 tx=0x7f6410009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:58.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 -- 172.21.15.63:0/3913275344 shutdown_connections 2024-03-23T12:00:58.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 --2- 172.21.15.63:0/3913275344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424065630 0x7f6424065a80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 --2- 172.21.15.63:0/3913275344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6424102ca0 0x7f6424103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 -- 172.21.15.63:0/3913275344 >> 172.21.15.63:0/3913275344 conn(0x7f6424071b70 msgr2=0x7f6424071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:58.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 -- 172.21.15.63:0/3913275344 shutdown_connections 2024-03-23T12:00:58.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.445+0000 7f6429e00700 1 -- 172.21.15.63:0/3913275344 wait complete. 2024-03-23T12:00:58.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.446+0000 7f6429e00700 1 Processor -- start 2024-03-23T12:00:58.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.446+0000 7f6429e00700 1 -- start start 2024-03-23T12:00:58.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.446+0000 7f6429e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6424065630 0x7f64240fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.446+0000 7f6429e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424102ca0 0x7f64240fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6429e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64240fc6f0 con 0x7f6424102ca0 2024-03-23T12:00:58.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6429e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64240fc830 con 0x7f6424065630 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6423e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424102ca0 0x7f64240fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6423e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424102ca0 0x7f64240fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37664/0 (socket says 172.21.15.63:37664) 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6423e00700 1 -- 172.21.15.63:0/3928479767 learned_addr learned my addr 172.21.15.63:0/3928479767 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6428a00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6424065630 0x7f64240fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6423e00700 1 -- 172.21.15.63:0/3928479767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6424065630 msgr2=0x7f64240fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6423e00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6424065630 0x7f64240fbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6423e00700 1 -- 172.21.15.63:0/3928479767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410005040 con 0x7f6424102ca0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.447+0000 7f6423e00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424102ca0 0x7f64240fc160 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6410007f50 tx=0x7f6410009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.448+0000 7f6420c00700 1 -- 172.21.15.63:0/3928479767 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6410021070 con 0x7f6424102ca0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.448+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64240fca80 con 0x7f6424102ca0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.448+0000 7f6420c00700 1 -- 172.21.15.63:0/3928479767 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64100093b0 con 0x7f6424102ca0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.448+0000 7f6420c00700 1 -- 172.21.15.63:0/3928479767 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641000e070 con 0x7f6424102ca0 2024-03-23T12:00:58.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.448+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6424119ef0 con 0x7f6424102ca0 2024-03-23T12:00:58.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.449+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64240fe380 con 0x7f6424102ca0 2024-03-23T12:00:58.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.450+0000 7f6420c00700 1 -- 172.21.15.63:0/3928479767 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f641001c400 con 0x7f6424102ca0 2024-03-23T12:00:58.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.451+0000 7f6420c00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f640c06c7d0 0x7f640c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.451+0000 7f6420c00700 1 -- 172.21.15.63:0/3928479767 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f641008cb60 con 0x7f6424102ca0 2024-03-23T12:00:58.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.451+0000 7f6428a00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f640c06c7d0 0x7f640c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.452+0000 7f6428a00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f640c06c7d0 0x7f640c06ec80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6418009b80 tx=0x7f641800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.455+0000 7f6420c00700 1 -- 172.21.15.63:0/3928479767 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6410056de0 con 0x7f6424102ca0 2024-03-23T12:00:58.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.757+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f642405f5f0 con 0x7f6424102ca0 2024-03-23T12:00:58.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.758+0000 7f6420c00700 1 -- 172.21.15.63:0/3928479767 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f641005a400 con 0x7f6424102ca0 2024-03-23T12:00:58.740 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:00:58.740 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:00:58.741 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:00:58.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:00:58.742 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:00:58.742 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:00:58.742 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:00:58.742 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:00:58.742 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:00:58.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f640c06c7d0 msgr2=0x7f640c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f640c06c7d0 0x7f640c06ec80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6418009b80 tx=0x7f641800b040 comp rx=0 tx=0).stop 2024-03-23T12:00:58.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424102ca0 msgr2=0x7f64240fc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424102ca0 0x7f64240fc160 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6410007f50 tx=0x7f6410009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:58.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 shutdown_connections 2024-03-23T12:00:58.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f640c06c7d0 0x7f640c06ec80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6424065630 0x7f64240fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 --2- 172.21.15.63:0/3928479767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6424102ca0 0x7f64240fc160 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 >> 172.21.15.63:0/3928479767 conn(0x7f6424071b70 msgr2=0x7f64241059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:58.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.760+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 shutdown_connections 2024-03-23T12:00:58.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.761+0000 7f6429e00700 1 -- 172.21.15.63:0/3928479767 wait complete. 2024-03-23T12:00:58.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.899+0000 7f4af0c00700 1 -- 172.21.15.63:0/2282627774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 msgr2=0x7f4aec109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.899+0000 7f4af0c00700 1 --2- 172.21.15.63:0/2282627774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec109110 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8004970 tx=0x7f4ad8009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:58.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.900+0000 7f4af0c00700 1 -- 172.21.15.63:0/2282627774 shutdown_connections 2024-03-23T12:00:58.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.900+0000 7f4af0c00700 1 --2- 172.21.15.63:0/2282627774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec109110 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.900+0000 7f4af0c00700 1 --2- 172.21.15.63:0/2282627774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4aec070360 0x7f4aec070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.900+0000 7f4af0c00700 1 -- 172.21.15.63:0/2282627774 >> 172.21.15.63:0/2282627774 conn(0x7f4aec0f89b0 msgr2=0x7f4aec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:58.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.900+0000 7f4af0c00700 1 -- 172.21.15.63:0/2282627774 shutdown_connections 2024-03-23T12:00:58.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.900+0000 7f4af0c00700 1 -- 172.21.15.63:0/2282627774 wait complete. 2024-03-23T12:00:58.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.900+0000 7f4af0c00700 1 Processor -- start 2024-03-23T12:00:58.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4af0c00700 1 -- start start 2024-03-23T12:00:58.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4af0c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4aec070360 0x7f4aec115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4af0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4af0c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec116830 con 0x7f4aec070c70 2024-03-23T12:00:58.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4af0c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec10fc00 con 0x7f4aec070360 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4ae9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4ae9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec1161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37692/0 (socket says 172.21.15.63:37692) 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4ae9400700 1 -- 172.21.15.63:0/3727018062 learned_addr learned my addr 172.21.15.63:0/3727018062 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.901+0000 7f4ae9e00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4aec070360 0x7f4aec115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4ae9400700 1 -- 172.21.15.63:0/3727018062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4aec070360 msgr2=0x7f4aec115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4ae9400700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4aec070360 0x7f4aec115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4ae9400700 1 -- 172.21.15.63:0/3727018062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad8005040 con 0x7f4aec070c70 2024-03-23T12:00:58.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4ae9400700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec1161e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8005010 tx=0x7f4ad8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:58.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4ae2000700 1 -- 172.21.15.63:0/3727018062 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ad800e040 con 0x7f4aec070c70 2024-03-23T12:00:58.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec10fe80 con 0x7f4aec070c70 2024-03-23T12:00:58.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4ae2000700 1 -- 172.21.15.63:0/3727018062 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ad8004290 con 0x7f4aec070c70 2024-03-23T12:00:58.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4ae2000700 1 -- 172.21.15.63:0/3727018062 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ad8013d00 con 0x7f4aec070c70 2024-03-23T12:00:58.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.902+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec1103c0 con 0x7f4aec070c70 2024-03-23T12:00:58.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.903+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aec103bd0 con 0x7f4aec070c70 2024-03-23T12:00:58.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.904+0000 7f4ae2000700 1 -- 172.21.15.63:0/3727018062 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4ad80134b0 con 0x7f4aec070c70 2024-03-23T12:00:58.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.905+0000 7f4ae2000700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4ad006c7d0 0x7f4ad006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:58.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.905+0000 7f4ae2000700 1 -- 172.21.15.63:0/3727018062 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4ad808ccb0 con 0x7f4aec070c70 2024-03-23T12:00:58.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.905+0000 7f4ae9e00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4ad006c7d0 0x7f4ad006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:58.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.906+0000 7f4ae9e00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4ad006c7d0 0x7f4ad006ec80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009b80 tx=0x7f4adc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:58.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:58.910+0000 7f4ae2000700 1 -- 172.21.15.63:0/3727018062 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4ad805a8c0 con 0x7f4aec070c70 2024-03-23T12:00:59.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:58 smithi063 ceph-mon[141124]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:59.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:58 smithi063 ceph-mon[141124]: pgmap v295: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:59.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:58 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3928479767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:00:59.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.167+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4aec103bd0 con 0x7f4aec070c70 2024-03-23T12:00:59.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.168+0000 7f4ae2000700 1 -- 172.21.15.63:0/3727018062 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f4ad8013e60 con 0x7f4aec070c70 2024-03-23T12:00:59.150 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:00:59.150 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:00:59.150 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:00:59.150 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:00:59.150 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:00:59.151 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:59.152 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:00:59.153 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:00:59.153 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:59.153 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:59.153 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:59.153 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:00:59.153 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:58 smithi111 ceph-mon[106899]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:00:59.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:58 smithi111 ceph-mon[106899]: pgmap v295: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:00:59.154 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:58 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3928479767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:00:59.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4ad006c7d0 msgr2=0x7f4ad006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4ad006c7d0 0x7f4ad006ec80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009b80 tx=0x7f4adc00b040 comp rx=0 tx=0).stop 2024-03-23T12:00:59.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 msgr2=0x7f4aec1161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec1161e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8005010 tx=0x7f4ad8009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 shutdown_connections 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4ad006c7d0 0x7f4ad006ec80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4aec070360 0x7f4aec115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 --2- 172.21.15.63:0/3727018062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4aec070c70 0x7f4aec1161e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 >> 172.21.15.63:0/3727018062 conn(0x7f4aec0f89b0 msgr2=0x7f4aec1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.170+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 shutdown_connections 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.171+0000 7f4af0c00700 1 -- 172.21.15.63:0/3727018062 wait complete. 2024-03-23T12:00:59.155 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 -- 172.21.15.63:0/3667531079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d01035f0 msgr2=0x7f17d00fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3667531079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d01035f0 0x7f17d00fcc60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f17bc004940 tx=0x7f17bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 -- 172.21.15.63:0/3667531079 shutdown_connections 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3667531079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d01035f0 0x7f17d00fcc60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3667531079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d0102c50 0x7f17d0103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 -- 172.21.15.63:0/3667531079 >> 172.21.15.63:0/3667531079 conn(0x7f17d00f87e0 msgr2=0x7f17d00fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 -- 172.21.15.63:0/3667531079 shutdown_connections 2024-03-23T12:00:59.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.309+0000 7f17d5a00700 1 -- 172.21.15.63:0/3667531079 wait complete. 2024-03-23T12:00:59.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.310+0000 7f17d5a00700 1 Processor -- start 2024-03-23T12:00:59.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.310+0000 7f17d5a00700 1 -- start start 2024-03-23T12:00:59.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.310+0000 7f17d5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d0102c50 0x7f17d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.310+0000 7f17d5a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d01035f0 0x7f17d0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.310+0000 7f17d5a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d0070e20 con 0x7f17d0102c50 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.310+0000 7f17d5a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d0070f60 con 0x7f17d01035f0 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d0102c50 0x7f17d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d0102c50 0x7f17d00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37700/0 (socket says 172.21.15.63:37700) 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cfe00700 1 -- 172.21.15.63:0/3260755723 learned_addr learned my addr 172.21.15.63:0/3260755723 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cf400700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d01035f0 0x7f17d0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cfe00700 1 -- 172.21.15.63:0/3260755723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d01035f0 msgr2=0x7f17d0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cfe00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d01035f0 0x7f17d0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cfe00700 1 -- 172.21.15.63:0/3260755723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc005040 con 0x7f17d0102c50 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cf400700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d01035f0 0x7f17d0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.311+0000 7f17cfe00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d0102c50 0x7f17d00702c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17c000b790 tx=0x7f17c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.312+0000 7f17c7e00700 1 -- 172.21.15.63:0/3260755723 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17c000f770 con 0x7f17d0102c50 2024-03-23T12:00:59.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.312+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d0119bf0 con 0x7f17d0102c50 2024-03-23T12:00:59.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.312+0000 7f17c7e00700 1 -- 172.21.15.63:0/3260755723 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17c000fd80 con 0x7f17d0102c50 2024-03-23T12:00:59.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.312+0000 7f17c7e00700 1 -- 172.21.15.63:0/3260755723 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17c000d310 con 0x7f17d0102c50 2024-03-23T12:00:59.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.312+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d011a100 con 0x7f17d0102c50 2024-03-23T12:00:59.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.313+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d00fe3a0 con 0x7f17d0102c50 2024-03-23T12:00:59.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.314+0000 7f17c7e00700 1 -- 172.21.15.63:0/3260755723 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f17c0010420 con 0x7f17d0102c50 2024-03-23T12:00:59.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.315+0000 7f17c7e00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17b406c7d0 0x7f17b406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:59.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.315+0000 7f17c7e00700 1 -- 172.21.15.63:0/3260755723 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f17c008bb40 con 0x7f17d0102c50 2024-03-23T12:00:59.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.315+0000 7f17cf400700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17b406c7d0 0x7f17b406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:59.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.316+0000 7f17cf400700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17b406c7d0 0x7f17b406ec80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00b3b0 tx=0x7f17bc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:59.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.319+0000 7f17c7e00700 1 -- 172.21.15.63:0/3260755723 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f17c0055d40 con 0x7f17d0102c50 2024-03-23T12:00:59.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.551+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17d00719b0 con 0x7f17b406c7d0 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.553+0000 7f17c7e00700 1 -- 172.21.15.63:0/3260755723 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f17d00719b0 con 0x7f17b406c7d0 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:00:59.535 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:00:59.536 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:00:59.536 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:00:59.536 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:00:59.536 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:00:59.536 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:00:59.536 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17b406c7d0 msgr2=0x7f17b406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17b406c7d0 0x7f17b406ec80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00b3b0 tx=0x7f17bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d0102c50 msgr2=0x7f17d00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d0102c50 0x7f17d00702c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17c000b790 tx=0x7f17c0009250 comp rx=0 tx=0).stop 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 shutdown_connections 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17b406c7d0 0x7f17b406ec80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17d0102c50 0x7f17d00702c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 --2- 172.21.15.63:0/3260755723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d01035f0 0x7f17d0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 >> 172.21.15.63:0/3260755723 conn(0x7f17d00f87e0 msgr2=0x7f17d00f9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:59.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 shutdown_connections 2024-03-23T12:00:59.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.555+0000 7f17d5a00700 1 -- 172.21.15.63:0/3260755723 wait complete. 2024-03-23T12:00:59.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 -- 172.21.15.63:0/1868178934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21341037a0 msgr2=0x7f21340fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 --2- 172.21.15.63:0/1868178934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21341037a0 0x7f21340fce10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2124004940 tx=0x7f2124009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:59.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 -- 172.21.15.63:0/1868178934 shutdown_connections 2024-03-23T12:00:59.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 --2- 172.21.15.63:0/1868178934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21341037a0 0x7f21340fce10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 --2- 172.21.15.63:0/1868178934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134102e00 0x7f21341031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 -- 172.21.15.63:0/1868178934 >> 172.21.15.63:0/1868178934 conn(0x7f21340f89b0 msgr2=0x7f21340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:59.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 -- 172.21.15.63:0/1868178934 shutdown_connections 2024-03-23T12:00:59.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.693+0000 7f213c800700 1 -- 172.21.15.63:0/1868178934 wait complete. 2024-03-23T12:00:59.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.694+0000 7f213c800700 1 Processor -- start 2024-03-23T12:00:59.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.694+0000 7f213c800700 1 -- start start 2024-03-23T12:00:59.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.694+0000 7f213c800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2134102e00 0x7f2134113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:59.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.694+0000 7f213c800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21341037a0 0x7f2134114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:59.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.694+0000 7f213c800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134114690 con 0x7f2134102e00 2024-03-23T12:00:59.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.694+0000 7f213c800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213410da80 con 0x7f21341037a0 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.694+0000 7f2139e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2134102e00 0x7f2134113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2134102e00 0x7f2134113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37714/0 (socket says 172.21.15.63:37714) 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139e00700 1 -- 172.21.15.63:0/2311181495 learned_addr learned my addr 172.21.15.63:0/2311181495 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139400700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21341037a0 0x7f2134114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139e00700 1 -- 172.21.15.63:0/2311181495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21341037a0 msgr2=0x7f2134114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139e00700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21341037a0 0x7f2134114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139e00700 1 -- 172.21.15.63:0/2311181495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2124005040 con 0x7f2134102e00 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139400700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21341037a0 0x7f2134114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f2139e00700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2134102e00 0x7f2134113b00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2128002a50 tx=0x7f21280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f212e000700 1 -- 172.21.15.63:0/2311181495 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2128007cc0 con 0x7f2134102e00 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f212e000700 1 -- 172.21.15.63:0/2311181495 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f212800f040 con 0x7f2134102e00 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f212e000700 1 -- 172.21.15.63:0/2311181495 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2128009540 con 0x7f2134102e00 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.695+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213410dd60 con 0x7f2134102e00 2024-03-23T12:00:59.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.696+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213410e2f0 con 0x7f2134102e00 2024-03-23T12:00:59.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.697+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21340fe550 con 0x7f2134102e00 2024-03-23T12:00:59.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.698+0000 7f212e000700 1 -- 172.21.15.63:0/2311181495 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f21280096a0 con 0x7f2134102e00 2024-03-23T12:00:59.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.698+0000 7f212e000700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f211c06c7d0 0x7f211c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:00:59.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.699+0000 7f212e000700 1 -- 172.21.15.63:0/2311181495 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f212808b7a0 con 0x7f2134102e00 2024-03-23T12:00:59.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.699+0000 7f2139400700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f211c06c7d0 0x7f211c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:00:59.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.699+0000 7f2139400700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f211c06c7d0 0x7f211c06ec80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f213410ed60 tx=0x7f2124009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:00:59.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:00:59.703+0000 7f212e000700 1 -- 172.21.15.63:0/2311181495 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f21280594b0 con 0x7f2134102e00 2024-03-23T12:00:59.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.004+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2134109df0 con 0x7f2134102e00 2024-03-23T12:00:59.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.004+0000 7f212e000700 1 -- 172.21.15.63:0/2311181495 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2128059040 con 0x7f2134102e00 2024-03-23T12:00:59.987 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:00:59.987 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:00:59.987 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:00:59.987 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:00:59.987 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f211c06c7d0 msgr2=0x7f211c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f211c06c7d0 0x7f211c06ec80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f213410ed60 tx=0x7f2124009e30 comp rx=0 tx=0).stop 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2134102e00 msgr2=0x7f2134113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2134102e00 0x7f2134113b00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2128002a50 tx=0x7f21280046b0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 shutdown_connections 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f211c06c7d0 0x7f211c06ec80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2134102e00 0x7f2134113b00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 --2- 172.21.15.63:0/2311181495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21341037a0 0x7f2134114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:00:59.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 >> 172.21.15.63:0/2311181495 conn(0x7f21340f89b0 msgr2=0x7f21340f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:00:59.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 shutdown_connections 2024-03-23T12:00:59.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:00.006+0000 7f213c800700 1 -- 172.21.15.63:0/2311181495 wait complete. 2024-03-23T12:01:00.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:59 smithi063 ceph-mon[141124]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:00.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:59 smithi063 ceph-mon[141124]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:00.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:00:59 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3727018062' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:01:00.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:59 smithi111 ceph-mon[106899]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:00.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:59 smithi111 ceph-mon[106899]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:00.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:00:59 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3727018062' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:01:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:00 smithi063 ceph-mon[141124]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:00 smithi063 ceph-mon[141124]: pgmap v296: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:00 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2311181495' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:01:01.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:00 smithi111 ceph-mon[106899]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:00 smithi111 ceph-mon[106899]: pgmap v296: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:01.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:00 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2311181495' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:01:03.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:02 smithi063 ceph-mon[141124]: pgmap v297: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:03.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:02 smithi111 ceph-mon[106899]: pgmap v297: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:05.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:04 smithi063 ceph-mon[141124]: pgmap v298: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:05.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:04 smithi111 ceph-mon[106899]: pgmap v298: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:06.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:06 smithi111 ceph-mon[106899]: pgmap v299: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:06.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:06 smithi063 ceph-mon[141124]: pgmap v299: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:09.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:08 smithi063 ceph-mon[141124]: pgmap v300: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:09.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:08 smithi111 ceph-mon[106899]: pgmap v300: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:10 smithi063 ceph-mon[141124]: pgmap v301: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:11.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:11.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:10 smithi111 ceph-mon[106899]: pgmap v301: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:13.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:12 smithi063 ceph-mon[141124]: pgmap v302: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:13.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:12 smithi111 ceph-mon[106899]: pgmap v302: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:15.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:14 smithi063 ceph-mon[141124]: pgmap v303: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:15.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:14 smithi111 ceph-mon[106899]: pgmap v303: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:16.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:16 smithi111 ceph-mon[106899]: pgmap v304: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:16.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:16 smithi063 ceph-mon[141124]: pgmap v304: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:19.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:18 smithi063 ceph-mon[141124]: pgmap v305: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:19.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:18 smithi111 ceph-mon[106899]: pgmap v305: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:21.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:20 smithi063 ceph-mon[141124]: pgmap v306: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:21.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:20 smithi111 ceph-mon[106899]: pgmap v306: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:23.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:22 smithi063 ceph-mon[141124]: pgmap v307: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:23.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:22 smithi111 ceph-mon[106899]: pgmap v307: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:25.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:24 smithi063 ceph-mon[141124]: pgmap v308: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:25.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:24 smithi111 ceph-mon[106899]: pgmap v308: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:26.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:26.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:26 smithi063 ceph-mon[141124]: pgmap v309: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:26.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:26 smithi111 ceph-mon[106899]: pgmap v309: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:29.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:28 smithi063 ceph-mon[141124]: pgmap v310: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:29.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:28 smithi111 ceph-mon[106899]: pgmap v310: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:30.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 -- 172.21.15.63:0/3691778748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6418102e00 msgr2=0x7f64181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 --2- 172.21.15.63:0/3691778748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6418102e00 0x7f64181031d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6408004940 tx=0x7f6408009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:30.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 -- 172.21.15.63:0/3691778748 shutdown_connections 2024-03-23T12:01:30.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 --2- 172.21.15.63:0/3691778748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64181037a0 0x7f64180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 --2- 172.21.15.63:0/3691778748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6418102e00 0x7f64181031d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 -- 172.21.15.63:0/3691778748 >> 172.21.15.63:0/3691778748 conn(0x7f64180f89b0 msgr2=0x7f64180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:30.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 -- 172.21.15.63:0/3691778748 shutdown_connections 2024-03-23T12:01:30.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.161+0000 7f641d000700 1 -- 172.21.15.63:0/3691778748 wait complete. 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.162+0000 7f641d000700 1 Processor -- start 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.162+0000 7f641d000700 1 -- start start 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.162+0000 7f641d000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6418102e00 0x7f6418115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.162+0000 7f641d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64181037a0 0x7f6418116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.162+0000 7f641d000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6418116880 con 0x7f64181037a0 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.162+0000 7f641d000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641810fc50 con 0x7f6418102e00 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f640fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64181037a0 0x7f6418116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f640fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64181037a0 0x7f6418116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55112/0 (socket says 172.21.15.63:55112) 2024-03-23T12:01:30.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f640fe00700 1 -- 172.21.15.63:0/714886772 learned_addr learned my addr 172.21.15.63:0/714886772 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:01:30.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f6415e00700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6418102e00 0x7f6418115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f640fe00700 1 -- 172.21.15.63:0/714886772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6418102e00 msgr2=0x7f6418115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f640fe00700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6418102e00 0x7f6418115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f640fe00700 1 -- 172.21.15.63:0/714886772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6408005040 con 0x7f64181037a0 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f6415e00700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6418102e00 0x7f6418115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.163+0000 7f640fe00700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64181037a0 0x7f6418116230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f640000c890 tx=0x7f64000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.164+0000 7f640e000700 1 -- 172.21.15.63:0/714886772 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f640000a7d0 con 0x7f64181037a0 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.164+0000 7f640e000700 1 -- 172.21.15.63:0/714886772 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f640000ade0 con 0x7f64181037a0 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.164+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641810ff30 con 0x7f64181037a0 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.164+0000 7f640e000700 1 -- 172.21.15.63:0/714886772 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64000113a0 con 0x7f64181037a0 2024-03-23T12:01:30.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.164+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64181104f0 con 0x7f64181037a0 2024-03-23T12:01:30.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.165+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64180fe550 con 0x7f64181037a0 2024-03-23T12:01:30.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.166+0000 7f640e000700 1 -- 172.21.15.63:0/714886772 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6400012400 con 0x7f64181037a0 2024-03-23T12:01:30.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.167+0000 7f640e000700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f63fc06c7d0 0x7f63fc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.167+0000 7f640e000700 1 -- 172.21.15.63:0/714886772 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f640008bb30 con 0x7f64181037a0 2024-03-23T12:01:30.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.167+0000 7f6415e00700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f63fc06c7d0 0x7f63fc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.168+0000 7f6415e00700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f63fc06c7d0 0x7f63fc06ec80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f640800b840 tx=0x7f6408009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:30.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.172+0000 7f640e000700 1 -- 172.21.15.63:0/714886772 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6400059840 con 0x7f64181037a0 2024-03-23T12:01:30.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.403+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6418109dc0 con 0x7f63fc06c7d0 2024-03-23T12:01:30.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.404+0000 7f640e000700 1 -- 172.21.15.63:0/714886772 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6418109dc0 con 0x7f63fc06c7d0 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f63fc06c7d0 msgr2=0x7f63fc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f63fc06c7d0 0x7f63fc06ec80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f640800b840 tx=0x7f6408009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64181037a0 msgr2=0x7f6418116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64181037a0 0x7f6418116230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f640000c890 tx=0x7f64000044c0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 shutdown_connections 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f63fc06c7d0 0x7f63fc06ec80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6418102e00 0x7f6418115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 --2- 172.21.15.63:0/714886772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64181037a0 0x7f6418116230 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 >> 172.21.15.63:0/714886772 conn(0x7f64180f89b0 msgr2=0x7f64180706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 shutdown_connections 2024-03-23T12:01:30.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.406+0000 7f641d000700 1 -- 172.21.15.63:0/714886772 wait complete. 2024-03-23T12:01:30.403 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 -- 172.21.15.63:0/117705586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 msgr2=0x7f845c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 --2- 172.21.15.63:0/117705586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c1031d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8450004940 tx=0x7f8450009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 -- 172.21.15.63:0/117705586 shutdown_connections 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 --2- 172.21.15.63:0/117705586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f845c1037a0 0x7f845c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 --2- 172.21.15.63:0/117705586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c1031d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 -- 172.21.15.63:0/117705586 >> 172.21.15.63:0/117705586 conn(0x7f845c0f89b0 msgr2=0x7f845c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 -- 172.21.15.63:0/117705586 shutdown_connections 2024-03-23T12:01:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.545+0000 7f8463600700 1 -- 172.21.15.63:0/117705586 wait complete. 2024-03-23T12:01:30.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.546+0000 7f8463600700 1 Processor -- start 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.546+0000 7f8463600700 1 -- start start 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.546+0000 7f8463600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.546+0000 7f8463600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f845c1037a0 0x7f845c116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.546+0000 7f8463600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1167e0 con 0x7f845c102e00 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.546+0000 7f8463600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c10fbb0 con 0x7f845c1037a0 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f8460c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f8460c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55138/0 (socket says 172.21.15.63:55138) 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f8460c00700 1 -- 172.21.15.63:0/1455203589 learned_addr learned my addr 172.21.15.63:0/1455203589 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f845be00700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f845c1037a0 0x7f845c116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f8460c00700 1 -- 172.21.15.63:0/1455203589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f845c1037a0 msgr2=0x7f845c116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f8460c00700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f845c1037a0 0x7f845c116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f8460c00700 1 -- 172.21.15.63:0/1455203589 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450005040 con 0x7f845c102e00 2024-03-23T12:01:30.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.547+0000 7f8460c00700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c115c50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8450000c00 tx=0x7f8450009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:30.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.548+0000 7f8458c00700 1 -- 172.21.15.63:0/1455203589 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8450021070 con 0x7f845c102e00 2024-03-23T12:01:30.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.548+0000 7f8458c00700 1 -- 172.21.15.63:0/1455203589 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8450013de0 con 0x7f845c102e00 2024-03-23T12:01:30.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.548+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c10fe30 con 0x7f845c102e00 2024-03-23T12:01:30.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.548+0000 7f8458c00700 1 -- 172.21.15.63:0/1455203589 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845000e070 con 0x7f845c102e00 2024-03-23T12:01:30.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.548+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c110390 con 0x7f845c102e00 2024-03-23T12:01:30.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.549+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c0fe550 con 0x7f845c102e00 2024-03-23T12:01:30.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.550+0000 7f8458c00700 1 -- 172.21.15.63:0/1455203589 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f845001c600 con 0x7f845c102e00 2024-03-23T12:01:30.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.551+0000 7f8458c00700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f844406c7c0 0x7f844406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.551+0000 7f8458c00700 1 -- 172.21.15.63:0/1455203589 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f845008cbe0 con 0x7f845c102e00 2024-03-23T12:01:30.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.551+0000 7f845be00700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f844406c7c0 0x7f844406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.552+0000 7f845be00700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f844406c7c0 0x7f844406ec70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8448004600 tx=0x7f844800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:30.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.556+0000 7f8458c00700 1 -- 172.21.15.63:0/1455203589 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f845005a870 con 0x7f845c102e00 2024-03-23T12:01:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.785+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f845c04bf70 con 0x7f844406c7c0 2024-03-23T12:01:30.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.786+0000 7f8458c00700 1 -- 172.21.15.63:0/1455203589 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f844800f040 con 0x7f844406c7c0 2024-03-23T12:01:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f844406c7c0 msgr2=0x7f844406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f844406c7c0 0x7f844406ec70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8448004600 tx=0x7f844800b040 comp rx=0 tx=0).stop 2024-03-23T12:01:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 msgr2=0x7f845c115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c115c50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8450000c00 tx=0x7f8450009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 shutdown_connections 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f844406c7c0 0x7f844406ec70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f845c102e00 0x7f845c115c50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 --2- 172.21.15.63:0/1455203589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f845c1037a0 0x7f845c116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 >> 172.21.15.63:0/1455203589 conn(0x7f845c0f89b0 msgr2=0x7f845c0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 shutdown_connections 2024-03-23T12:01:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.789+0000 7f8463600700 1 -- 172.21.15.63:0/1455203589 wait complete. 2024-03-23T12:01:30.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.927+0000 7f5f26e00700 1 -- 172.21.15.63:0/340015856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 msgr2=0x7f5f20106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.927+0000 7f5f26e00700 1 --2- 172.21.15.63:0/340015856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20106f00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c004940 tx=0x7f5f0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:30.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.927+0000 7f5f26e00700 1 -- 172.21.15.63:0/340015856 shutdown_connections 2024-03-23T12:01:30.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.927+0000 7f5f26e00700 1 --2- 172.21.15.63:0/340015856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20106f00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.927+0000 7f5f26e00700 1 --2- 172.21.15.63:0/340015856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f200fde90 0x7f5f200fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.927+0000 7f5f26e00700 1 -- 172.21.15.63:0/340015856 >> 172.21.15.63:0/340015856 conn(0x7f5f20072bc0 msgr2=0x7f5f20072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:30.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.928+0000 7f5f26e00700 1 -- 172.21.15.63:0/340015856 shutdown_connections 2024-03-23T12:01:30.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.928+0000 7f5f26e00700 1 -- 172.21.15.63:0/340015856 wait complete. 2024-03-23T12:01:30.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.928+0000 7f5f26e00700 1 Processor -- start 2024-03-23T12:01:30.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.928+0000 7f5f26e00700 1 -- start start 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f26e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f200fde90 0x7f5f20115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f26e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f26e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f201168c0 con 0x7f5f200fe830 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f26e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2010fc90 con 0x7f5f200fde90 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55146/0 (socket says 172.21.15.63:55146) 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1f400700 1 -- 172.21.15.63:0/1339712233 learned_addr learned my addr 172.21.15.63:0/1339712233 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1fe00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f200fde90 0x7f5f20115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1f400700 1 -- 172.21.15.63:0/1339712233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f200fde90 msgr2=0x7f5f20115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1f400700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f200fde90 0x7f5f20115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1f400700 1 -- 172.21.15.63:0/1339712233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c005040 con 0x7f5f200fe830 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.929+0000 7f5f1fe00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f200fde90 0x7f5f20115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:01:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.930+0000 7f5f1f400700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20116270 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c000c00 tx=0x7f5f0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.930+0000 7f5f17e00700 1 -- 172.21.15.63:0/1339712233 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0c00e040 con 0x7f5f200fe830 2024-03-23T12:01:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.930+0000 7f5f17e00700 1 -- 172.21.15.63:0/1339712233 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f0c013420 con 0x7f5f200fe830 2024-03-23T12:01:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.930+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2010ff10 con 0x7f5f200fe830 2024-03-23T12:01:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.930+0000 7f5f17e00700 1 -- 172.21.15.63:0/1339712233 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0c01c630 con 0x7f5f200fe830 2024-03-23T12:01:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.930+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f20110450 con 0x7f5f200fe830 2024-03-23T12:01:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.931+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f20106a20 con 0x7f5f200fe830 2024-03-23T12:01:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.932+0000 7f5f17e00700 1 -- 172.21.15.63:0/1339712233 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5f0c013970 con 0x7f5f200fe830 2024-03-23T12:01:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.934+0000 7f5f17e00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0406c7d0 0x7f5f0406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.934+0000 7f5f17e00700 1 -- 172.21.15.63:0/1339712233 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5f0c08cca0 con 0x7f5f200fe830 2024-03-23T12:01:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.934+0000 7f5f1fe00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0406c7d0 0x7f5f0406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.934+0000 7f5f1fe00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0406c7d0 0x7f5f0406ec80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009fd0 tx=0x7f5f10009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:30.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:30.938+0000 7f5f17e00700 1 -- 172.21.15.63:0/1339712233 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5f0c05a9b0 con 0x7f5f200fe830 2024-03-23T12:01:31.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:30 smithi063 ceph-mon[141124]: pgmap v311: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:31.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.166+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f20111380 con 0x7f5f0406c7d0 2024-03-23T12:01:31.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:30 smithi111 ceph-mon[106899]: pgmap v311: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:31.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.176+0000 7f5f17e00700 1 -- 172.21.15.63:0/1339712233 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f5f20111380 con 0x7f5f0406c7d0 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (10m) 3m ago 23m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (8m) 3m ago 24m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (7m) 3m ago 22m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (10m) 3m ago 23m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (3m) 3m ago 17m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (3m) 3m ago 17m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (17m) 3m ago 17m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (3m) 3m ago 17m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (13m) 3m ago 25m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (11m) 3m ago 22m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (9m) 3m ago 25m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:01:31.159 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (8m) 3m ago 22m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (11m) 3m ago 24m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (11m) 3m ago 22m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (7m) 3m ago 21m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (6m) 3m ago 21m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (6m) 3m ago 20m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (5m) 3m ago 20m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (4m) 3m ago 19m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (4m) 3m ago 19m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:01:31.160 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (11m) 3m ago 23m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:01:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.178+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0406c7d0 msgr2=0x7f5f0406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.178+0000 7f5f26e00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0406c7d0 0x7f5f0406ec80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009fd0 tx=0x7f5f10009ee0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.178+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 msgr2=0x7f5f20116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.178+0000 7f5f26e00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20116270 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c000c00 tx=0x7f5f0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.179+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 shutdown_connections 2024-03-23T12:01:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.179+0000 7f5f26e00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0406c7d0 0x7f5f0406ec80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.179+0000 7f5f26e00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f200fde90 0x7f5f20115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.179+0000 7f5f26e00700 1 --2- 172.21.15.63:0/1339712233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f200fe830 0x7f5f20116270 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.179+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 >> 172.21.15.63:0/1339712233 conn(0x7f5f20072bc0 msgr2=0x7f5f200fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:31.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.179+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 shutdown_connections 2024-03-23T12:01:31.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.179+0000 7f5f26e00700 1 -- 172.21.15.63:0/1339712233 wait complete. 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 -- 172.21.15.63:0/628613852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 msgr2=0x7fb74c103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 --2- 172.21.15.63:0/628613852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c103070 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb73c004940 tx=0x7fb73c009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 -- 172.21.15.63:0/628613852 shutdown_connections 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 --2- 172.21.15.63:0/628613852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb74c065630 0x7fb74c065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 --2- 172.21.15.63:0/628613852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c103070 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 -- 172.21.15.63:0/628613852 >> 172.21.15.63:0/628613852 conn(0x7fb74c071b70 msgr2=0x7fb74c071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 -- 172.21.15.63:0/628613852 shutdown_connections 2024-03-23T12:01:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.315+0000 7fb751200700 1 -- 172.21.15.63:0/628613852 wait complete. 2024-03-23T12:01:31.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.316+0000 7fb751200700 1 Processor -- start 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.316+0000 7fb751200700 1 -- start start 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.316+0000 7fb751200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb74c065630 0x7fb74c1149b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb751200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb751200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1154f0 con 0x7fb74c102ca0 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb751200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c10fac0 con 0x7fb74c065630 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb74aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb74aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c114ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55150/0 (socket says 172.21.15.63:55150) 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb74aa00700 1 -- 172.21.15.63:0/1857707335 learned_addr learned my addr 172.21.15.63:0/1857707335 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb74b400700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb74c065630 0x7fb74c1149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb74aa00700 1 -- 172.21.15.63:0/1857707335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb74c065630 msgr2=0x7fb74c1149b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb74aa00700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb74c065630 0x7fb74c1149b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.317+0000 7fb74aa00700 1 -- 172.21.15.63:0/1857707335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c005040 con 0x7fb74c102ca0 2024-03-23T12:01:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.318+0000 7fb74aa00700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c114ef0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb738002a50 tx=0x7fb7380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.318+0000 7fb743400700 1 -- 172.21.15.63:0/1857707335 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb738007cc0 con 0x7fb74c102ca0 2024-03-23T12:01:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.318+0000 7fb743400700 1 -- 172.21.15.63:0/1857707335 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb73800f040 con 0x7fb74c102ca0 2024-03-23T12:01:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.318+0000 7fb743400700 1 -- 172.21.15.63:0/1857707335 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb73800d9c0 con 0x7fb74c102ca0 2024-03-23T12:01:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.318+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c10fda0 con 0x7fb74c102ca0 2024-03-23T12:01:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.318+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c110360 con 0x7fb74c102ca0 2024-03-23T12:01:31.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.319+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c106840 con 0x7fb74c102ca0 2024-03-23T12:01:31.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.320+0000 7fb743400700 1 -- 172.21.15.63:0/1857707335 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb73800db20 con 0x7fb74c102ca0 2024-03-23T12:01:31.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.321+0000 7fb743400700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb73006c7d0 0x7fb73006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:31.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.321+0000 7fb743400700 1 -- 172.21.15.63:0/1857707335 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb73808b760 con 0x7fb74c102ca0 2024-03-23T12:01:31.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.321+0000 7fb74b400700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb73006c7d0 0x7fb73006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:31.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.322+0000 7fb74b400700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb73006c7d0 0x7fb73006ec80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fd0 tx=0x7fb73c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:31.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.325+0000 7fb743400700 1 -- 172.21.15.63:0/1857707335 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb738055960 con 0x7fb74c102ca0 2024-03-23T12:01:31.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.628+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb74c1110f0 con 0x7fb74c102ca0 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.628+0000 7fb743400700 1 -- 172.21.15.63:0/1857707335 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fb738058f80 con 0x7fb74c102ca0 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:01:31.611 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:01:31.612 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb73006c7d0 msgr2=0x7fb73006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb73006c7d0 0x7fb73006ec80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fd0 tx=0x7fb73c009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 msgr2=0x7fb74c114ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c114ef0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb738002a50 tx=0x7fb7380046b0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 shutdown_connections 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb73006c7d0 0x7fb73006ec80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb74c065630 0x7fb74c1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 --2- 172.21.15.63:0/1857707335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb74c102ca0 0x7fb74c114ef0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 >> 172.21.15.63:0/1857707335 conn(0x7fb74c071b70 msgr2=0x7fb74c1039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:31.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 shutdown_connections 2024-03-23T12:01:31.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.631+0000 7fb751200700 1 -- 172.21.15.63:0/1857707335 wait complete. 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 -- 172.21.15.63:0/4001402444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 msgr2=0x7f7c001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 --2- 172.21.15.63:0/4001402444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c001031d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4004940 tx=0x7f7bf4009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 -- 172.21.15.63:0/4001402444 shutdown_connections 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 --2- 172.21.15.63:0/4001402444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c001037a0 0x7f7c000fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 --2- 172.21.15.63:0/4001402444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c001031d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 -- 172.21.15.63:0/4001402444 >> 172.21.15.63:0/4001402444 conn(0x7f7c000f89b0 msgr2=0x7f7c000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 -- 172.21.15.63:0/4001402444 shutdown_connections 2024-03-23T12:01:31.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.769+0000 7f7c08800700 1 -- 172.21.15.63:0/4001402444 wait complete. 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.770+0000 7f7c08800700 1 Processor -- start 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.770+0000 7f7c08800700 1 -- start start 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.770+0000 7f7c08800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.770+0000 7f7c08800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c001037a0 0x7f7c00070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.770+0000 7f7c08800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00070d90 con 0x7f7c00102e00 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.770+0000 7f7c08800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00070ed0 con 0x7f7c001037a0 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c000702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47710/0 (socket says 172.21.15.63:47710) 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05e00700 1 -- 172.21.15.63:0/3418831711 learned_addr learned my addr 172.21.15.63:0/3418831711 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05400700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c001037a0 0x7f7c00070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05e00700 1 -- 172.21.15.63:0/3418831711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c001037a0 msgr2=0x7f7c00070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05e00700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c001037a0 0x7f7c00070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:31.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05e00700 1 -- 172.21.15.63:0/3418831711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4005040 con 0x7f7c00102e00 2024-03-23T12:01:31.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.771+0000 7f7c05e00700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c000702c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009360 tx=0x7f7bf4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:31.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.772+0000 7f7bfa000700 1 -- 172.21.15.63:0/3418831711 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf400e040 con 0x7f7c00102e00 2024-03-23T12:01:31.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.772+0000 7f7bfa000700 1 -- 172.21.15.63:0/3418831711 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bf4013420 con 0x7f7c00102e00 2024-03-23T12:01:31.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.772+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00071120 con 0x7f7c00102e00 2024-03-23T12:01:31.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.772+0000 7f7bfa000700 1 -- 172.21.15.63:0/3418831711 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf401c5d0 con 0x7f7c00102e00 2024-03-23T12:01:31.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.772+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0011a250 con 0x7f7c00102e00 2024-03-23T12:01:31.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.773+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c000fe550 con 0x7f7c00102e00 2024-03-23T12:01:31.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.774+0000 7f7bfa000700 1 -- 172.21.15.63:0/3418831711 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7bf4013590 con 0x7f7c00102e00 2024-03-23T12:01:31.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.775+0000 7f7bfa000700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7be806c7d0 0x7f7be806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:31.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.775+0000 7f7bfa000700 1 -- 172.21.15.63:0/3418831711 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7bf408cb80 con 0x7f7c00102e00 2024-03-23T12:01:31.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.775+0000 7f7c05400700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7be806c7d0 0x7f7be806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:31.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.776+0000 7f7c05400700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7be806c7d0 0x7f7be806ec80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7c00071c00 tx=0x7f7bf000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:31.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:31.779+0000 7f7bfa000700 1 -- 172.21.15.63:0/3418831711 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7bf4056e00 con 0x7f7c00102e00 2024-03-23T12:01:32.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.038+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7c000719b0 con 0x7f7c00102e00 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.038+0000 7f7bfa000700 1 -- 172.21.15.63:0/3418831711 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f7bf405a420 con 0x7f7c00102e00 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:01:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:01:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:01:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:01:32.024 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:01:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:01:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:01:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:01:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:01:32.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7be806c7d0 msgr2=0x7f7be806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7be806c7d0 0x7f7be806ec80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7c00071c00 tx=0x7f7bf000b040 comp rx=0 tx=0).stop 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 msgr2=0x7f7c000702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c000702c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009360 tx=0x7f7bf4009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 shutdown_connections 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7be806c7d0 0x7f7be806ec80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c00102e00 0x7f7c000702c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 --2- 172.21.15.63:0/3418831711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c001037a0 0x7f7c00070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 >> 172.21.15.63:0/3418831711 conn(0x7f7c000f89b0 msgr2=0x7f7c000f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 shutdown_connections 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.041+0000 7f7c08800700 1 -- 172.21.15.63:0/3418831711 wait complete. 2024-03-23T12:01:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:01:32.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:31 smithi063 ceph-mon[141124]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:32.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:31 smithi063 ceph-mon[141124]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:32.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:31 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1857707335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:01:32.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:31 smithi111 ceph-mon[106899]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:32.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:31 smithi111 ceph-mon[106899]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:32.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:31 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1857707335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:01:32.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 -- 172.21.15.63:0/3558214958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 msgr2=0x7fb6140fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3558214958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb6140fe260 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb604004940 tx=0x7fb604009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:32.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 -- 172.21.15.63:0/3558214958 shutdown_connections 2024-03-23T12:01:32.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3558214958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6140fe830 0x7fb614106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3558214958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb6140fe260 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 -- 172.21.15.63:0/3558214958 >> 172.21.15.63:0/3558214958 conn(0x7fb614072bc0 msgr2=0x7fb614072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:32.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 -- 172.21.15.63:0/3558214958 shutdown_connections 2024-03-23T12:01:32.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.180+0000 7fb61ac00700 1 -- 172.21.15.63:0/3558214958 wait complete. 2024-03-23T12:01:32.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.181+0000 7fb61ac00700 1 Processor -- start 2024-03-23T12:01:32.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.181+0000 7fb61ac00700 1 -- start start 2024-03-23T12:01:32.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.181+0000 7fb61ac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb614113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:32.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.181+0000 7fb61ac00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6140fe830 0x7fb614113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:32.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.181+0000 7fb61ac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614114610 con 0x7fb6140fde90 2024-03-23T12:01:32.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.181+0000 7fb61ac00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61410da00 con 0x7fb6140fe830 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb613e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb614113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb613e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb614113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47734/0 (socket says 172.21.15.63:47734) 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb613e00700 1 -- 172.21.15.63:0/3198761438 learned_addr learned my addr 172.21.15.63:0/3198761438 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb60be00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6140fe830 0x7fb614113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb613e00700 1 -- 172.21.15.63:0/3198761438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6140fe830 msgr2=0x7fb614113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb613e00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6140fe830 0x7fb614113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb613e00700 1 -- 172.21.15.63:0/3198761438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb604005040 con 0x7fb6140fde90 2024-03-23T12:01:32.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.182+0000 7fb613e00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb614113a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb604000c00 tx=0x7fb604009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:32.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.183+0000 7fb610c00700 1 -- 172.21.15.63:0/3198761438 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb604021070 con 0x7fb6140fde90 2024-03-23T12:01:32.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.183+0000 7fb610c00700 1 -- 172.21.15.63:0/3198761438 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb604013e00 con 0x7fb6140fde90 2024-03-23T12:01:32.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.183+0000 7fb610c00700 1 -- 172.21.15.63:0/3198761438 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb60400e070 con 0x7fb6140fde90 2024-03-23T12:01:32.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.183+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61410dc80 con 0x7fb6140fde90 2024-03-23T12:01:32.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.183+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61410e1c0 con 0x7fb6140fde90 2024-03-23T12:01:32.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.184+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6141019c0 con 0x7fb6140fde90 2024-03-23T12:01:32.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.185+0000 7fb610c00700 1 -- 172.21.15.63:0/3198761438 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb604013950 con 0x7fb6140fde90 2024-03-23T12:01:32.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.186+0000 7fb610c00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb5f806c7d0 0x7fb5f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:32.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.186+0000 7fb610c00700 1 -- 172.21.15.63:0/3198761438 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb60408cac0 con 0x7fb6140fde90 2024-03-23T12:01:32.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.186+0000 7fb60be00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb5f806c7d0 0x7fb5f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:32.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.187+0000 7fb60be00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb5f806c7d0 0x7fb5f806ec80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb61410ebc0 tx=0x7fb5fc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:32.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.191+0000 7fb610c00700 1 -- 172.21.15.63:0/3198761438 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb60405a750 con 0x7fb6140fde90 2024-03-23T12:01:32.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.427+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb61410f170 con 0x7fb5f806c7d0 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.428+0000 7fb610c00700 1 -- 172.21.15.63:0/3198761438 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb61410f170 con 0x7fb5f806c7d0 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:01:32.411 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:01:32.412 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:01:32.412 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:01:32.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb5f806c7d0 msgr2=0x7fb5f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb5f806c7d0 0x7fb5f806ec80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb61410ebc0 tx=0x7fb5fc00b040 comp rx=0 tx=0).stop 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 msgr2=0x7fb614113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb614113a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb604000c00 tx=0x7fb604009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 shutdown_connections 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb5f806c7d0 0x7fb5f806ec80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6140fde90 0x7fb614113a80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 --2- 172.21.15.63:0/3198761438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6140fe830 0x7fb614113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 >> 172.21.15.63:0/3198761438 conn(0x7fb614072bc0 msgr2=0x7fb6140ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 shutdown_connections 2024-03-23T12:01:32.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.431+0000 7fb61ac00700 1 -- 172.21.15.63:0/3198761438 wait complete. 2024-03-23T12:01:32.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 -- 172.21.15.63:0/477374617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 msgr2=0x7f91000fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 --2- 172.21.15.63:0/477374617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f91000fcc50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f90ec004940 tx=0x7f90ec009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 -- 172.21.15.63:0/477374617 shutdown_connections 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 --2- 172.21.15.63:0/477374617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f91000fcc50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 --2- 172.21.15.63:0/477374617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9100102c40 0x7f9100103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 -- 172.21.15.63:0/477374617 >> 172.21.15.63:0/477374617 conn(0x7f91000f8830 msgr2=0x7f91000fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 -- 172.21.15.63:0/477374617 shutdown_connections 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.569+0000 7f9106800700 1 -- 172.21.15.63:0/477374617 wait complete. 2024-03-23T12:01:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.570+0000 7f9106800700 1 Processor -- start 2024-03-23T12:01:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9106800700 1 -- start start 2024-03-23T12:01:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9106800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9100102c40 0x7f9100113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9106800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f9100113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9106800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001143e0 con 0x7f91001035e0 2024-03-23T12:01:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9106800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910010d7d0 con 0x7f9100102c40 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9104a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f9100113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9104a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f9100113d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47756/0 (socket says 172.21.15.63:47756) 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.571+0000 7f9104a00700 1 -- 172.21.15.63:0/1567582251 learned_addr learned my addr 172.21.15.63:0/1567582251 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f9105400700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9100102c40 0x7f9100113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f9104a00700 1 -- 172.21.15.63:0/1567582251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9100102c40 msgr2=0x7f9100113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f9104a00700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9100102c40 0x7f9100113850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f9104a00700 1 -- 172.21.15.63:0/1567582251 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec005040 con 0x7f91001035e0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f9104a00700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f9100113d90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009390 tx=0x7f90ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f90fd600700 1 -- 172.21.15.63:0/1567582251 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90ec00e040 con 0x7f91001035e0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f90fd600700 1 -- 172.21.15.63:0/1567582251 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90ec013420 con 0x7f91001035e0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910010da50 con 0x7f91001035e0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.572+0000 7f90fd600700 1 -- 172.21.15.63:0/1567582251 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90ec01c630 con 0x7f91001035e0 2024-03-23T12:01:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.573+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910010e010 con 0x7f91001035e0 2024-03-23T12:01:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.574+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91000fe390 con 0x7f91001035e0 2024-03-23T12:01:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.575+0000 7f90fd600700 1 -- 172.21.15.63:0/1567582251 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f90ec013a00 con 0x7f91001035e0 2024-03-23T12:01:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.576+0000 7f90fd600700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f90e806c7d0 0x7f90e806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:01:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.576+0000 7f90fd600700 1 -- 172.21.15.63:0/1567582251 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f90ec08ca60 con 0x7f91001035e0 2024-03-23T12:01:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.576+0000 7f9105400700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f90e806c7d0 0x7f90e806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:01:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.576+0000 7f9105400700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f90e806c7d0 0x7f90e806ec80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009fd0 tx=0x7f90f400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:01:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.580+0000 7f90fd600700 1 -- 172.21.15.63:0/1567582251 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f90ec05a830 con 0x7f91001035e0 2024-03-23T12:01:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.881+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9100109bd0 con 0x7f91001035e0 2024-03-23T12:01:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.881+0000 7f90fd600700 1 -- 172.21.15.63:0/1567582251 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f90ec05a3c0 con 0x7f91001035e0 2024-03-23T12:01:32.864 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:01:32.864 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:01:32.864 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:01:32.864 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:01:32.864 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:01:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f90e806c7d0 msgr2=0x7f90e806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f90e806c7d0 0x7f90e806ec80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009fd0 tx=0x7f90f400b040 comp rx=0 tx=0).stop 2024-03-23T12:01:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 msgr2=0x7f9100113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f9100113d90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009390 tx=0x7f90ec009e30 comp rx=0 tx=0).stop 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 shutdown_connections 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f90e806c7d0 0x7f90e806ec80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9100102c40 0x7f9100113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 --2- 172.21.15.63:0/1567582251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91001035e0 0x7f9100113d90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 >> 172.21.15.63:0/1567582251 conn(0x7f91000f8830 msgr2=0x7f91000f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 shutdown_connections 2024-03-23T12:01:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:01:32.884+0000 7f9106800700 1 -- 172.21.15.63:0/1567582251 wait complete. 2024-03-23T12:01:33.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:32 smithi063 ceph-mon[141124]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:32 smithi063 ceph-mon[141124]: pgmap v312: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:32 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3418831711' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:01:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:32 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1567582251' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:01:33.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:32 smithi111 ceph-mon[106899]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:33.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:32 smithi111 ceph-mon[106899]: pgmap v312: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:33.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:32 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3418831711' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:01:33.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:32 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1567582251' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:01:34.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:33 smithi063 ceph-mon[141124]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:34.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:33 smithi111 ceph-mon[106899]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:01:35.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:34 smithi063 ceph-mon[141124]: pgmap v313: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:35.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:34 smithi111 ceph-mon[106899]: pgmap v313: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:36.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:36 smithi063 ceph-mon[141124]: pgmap v314: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:36.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:36 smithi111 ceph-mon[106899]: pgmap v314: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:39.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:38 smithi063 ceph-mon[141124]: pgmap v315: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:39.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:38 smithi111 ceph-mon[106899]: pgmap v315: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:41.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:40 smithi063 ceph-mon[141124]: pgmap v316: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:41.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:41.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:40 smithi111 ceph-mon[106899]: pgmap v316: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:41.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:43.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:42 smithi063 ceph-mon[141124]: pgmap v317: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:43.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:42 smithi111 ceph-mon[106899]: pgmap v317: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:45.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:44 smithi063 ceph-mon[141124]: pgmap v318: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:45.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:44 smithi111 ceph-mon[106899]: pgmap v318: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:46.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:46 smithi063 ceph-mon[141124]: pgmap v319: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:46.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:46 smithi111 ceph-mon[106899]: pgmap v319: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:49.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:48 smithi063 ceph-mon[141124]: pgmap v320: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:49.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:48 smithi111 ceph-mon[106899]: pgmap v320: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:51.093 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:50 smithi111 ceph-mon[106899]: pgmap v321: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:51.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:50 smithi063 ceph-mon[141124]: pgmap v321: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:53.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:52 smithi063 ceph-mon[141124]: pgmap v322: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:53.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:52 smithi111 ceph-mon[106899]: pgmap v322: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:55.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:54 smithi063 ceph-mon[141124]: pgmap v323: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:55.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:54 smithi111 ceph-mon[106899]: pgmap v323: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:56 smithi063 ceph-mon[141124]: pgmap v324: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:01:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:56 smithi111 ceph-mon[106899]: pgmap v324: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:59.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:01:58 smithi063 ceph-mon[141124]: pgmap v325: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:01:59.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:01:58 smithi111 ceph-mon[106899]: pgmap v325: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:01.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:00 smithi063 ceph-mon[141124]: pgmap v326: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:01.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:00 smithi111 ceph-mon[106899]: pgmap v326: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:03.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 -- 172.21.15.63:0/2687989758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 msgr2=0x7fbbe0070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/2687989758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe0070d10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc004970 tx=0x7fbbcc009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 -- 172.21.15.63:0/2687989758 shutdown_connections 2024-03-23T12:02:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/2687989758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe0070d10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/2687989758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe0071f90 0x7fbbe0070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 -- 172.21.15.63:0/2687989758 >> 172.21.15.63:0/2687989758 conn(0x7fbbe00f89d0 msgr2=0x7fbbe00fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 -- 172.21.15.63:0/2687989758 shutdown_connections 2024-03-23T12:02:03.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.039+0000 7fbbe7e00700 1 -- 172.21.15.63:0/2687989758 wait complete. 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.040+0000 7fbbe7e00700 1 Processor -- start 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.040+0000 7fbbe7e00700 1 -- start start 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.040+0000 7fbbe7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe011a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.040+0000 7fbbe7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe0071f90 0x7fbbe011a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.040+0000 7fbbe7e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe011ac30 con 0x7fbbe00708a0 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.040+0000 7fbbe7e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe0114000 con 0x7fbbe0071f90 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe011a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe011a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41760/0 (socket says 172.21.15.63:41760) 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe5400700 1 -- 172.21.15.63:0/226982188 learned_addr learned my addr 172.21.15.63:0/226982188 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe5400700 1 -- 172.21.15.63:0/226982188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe0071f90 msgr2=0x7fbbe011a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe4a00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe0071f90 0x7fbbe011a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe5400700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe0071f90 0x7fbbe011a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe5400700 1 -- 172.21.15.63:0/226982188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc005040 con 0x7fbbe00708a0 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe4a00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe0071f90 0x7fbbe011a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.041+0000 7fbbe5400700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe011a0a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400eb40 tx=0x7fbbd400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.042+0000 7fbbdd600700 1 -- 172.21.15.63:0/226982188 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbd4007cc0 con 0x7fbbe00708a0 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.042+0000 7fbbdd600700 1 -- 172.21.15.63:0/226982188 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbd400f040 con 0x7fbbe00708a0 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.042+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe0114280 con 0x7fbbe00708a0 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.042+0000 7fbbdd600700 1 -- 172.21.15.63:0/226982188 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbd4009570 con 0x7fbbe00708a0 2024-03-23T12:02:03.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.042+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe01147c0 con 0x7fbbe00708a0 2024-03-23T12:02:03.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.043+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe0071aa0 con 0x7fbbe00708a0 2024-03-23T12:02:03.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.044+0000 7fbbdd600700 1 -- 172.21.15.63:0/226982188 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbbd40096d0 con 0x7fbbe00708a0 2024-03-23T12:02:03.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.045+0000 7fbbdd600700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbbc806c7c0 0x7fbbc806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.045+0000 7fbbdd600700 1 -- 172.21.15.63:0/226982188 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbbd408b7e0 con 0x7fbbe00708a0 2024-03-23T12:02:03.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.045+0000 7fbbe4a00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbbc806c7c0 0x7fbbc806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.046+0000 7fbbe4a00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbbc806c7c0 0x7fbbc806ec70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc004560 tx=0x7fbbcc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:03.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.050+0000 7fbbdd600700 1 -- 172.21.15.63:0/226982188 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbbd40594a0 con 0x7fbbe00708a0 2024-03-23T12:02:03.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:02 smithi063 ceph-mon[141124]: pgmap v327: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:03.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:02 smithi111 ceph-mon[106899]: pgmap v327: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:03.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.283+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbe004bf40 con 0x7fbbc806c7c0 2024-03-23T12:02:03.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.285+0000 7fbbdd600700 1 -- 172.21.15.63:0/226982188 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbbe004bf40 con 0x7fbbc806c7c0 2024-03-23T12:02:03.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbbc806c7c0 msgr2=0x7fbbc806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbbc806c7c0 0x7fbbc806ec70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc004560 tx=0x7fbbcc009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:03.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 msgr2=0x7fbbe011a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe011a0a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400eb40 tx=0x7fbbd400b040 comp rx=0 tx=0).stop 2024-03-23T12:02:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 shutdown_connections 2024-03-23T12:02:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbbc806c7c0 0x7fbbc806ec70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbe00708a0 0x7fbbe011a0a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 --2- 172.21.15.63:0/226982188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe0071f90 0x7fbbe011a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 >> 172.21.15.63:0/226982188 conn(0x7fbbe00f89d0 msgr2=0x7fbbe00f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 shutdown_connections 2024-03-23T12:02:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.287+0000 7fbbe7e00700 1 -- 172.21.15.63:0/226982188 wait complete. 2024-03-23T12:02:03.282 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:02:03.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 -- 172.21.15.63:0/2218574462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070c10 msgr2=0x7fe53c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 --2- 172.21.15.63:0/2218574462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070c10 0x7fe53c109190 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe528004970 tx=0x7fe528009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:03.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 -- 172.21.15.63:0/2218574462 shutdown_connections 2024-03-23T12:02:03.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 --2- 172.21.15.63:0/2218574462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070c10 0x7fe53c109190 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 --2- 172.21.15.63:0/2218574462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c070300 0x7fe53c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 -- 172.21.15.63:0/2218574462 >> 172.21.15.63:0/2218574462 conn(0x7fe53c0f8970 msgr2=0x7fe53c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:03.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 -- 172.21.15.63:0/2218574462 shutdown_connections 2024-03-23T12:02:03.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.426+0000 7fe541400700 1 -- 172.21.15.63:0/2218574462 wait complete. 2024-03-23T12:02:03.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.427+0000 7fe541400700 1 Processor -- start 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.427+0000 7fe541400700 1 -- start start 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe541400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070300 0x7fe53c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe541400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c070c10 0x7fe53c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe541400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c1168c0 con 0x7fe53c070300 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe541400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c10fc90 con 0x7fe53c070c10 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe53a800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070300 0x7fe53c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe53a800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070300 0x7fe53c115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41766/0 (socket says 172.21.15.63:41766) 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe53a800700 1 -- 172.21.15.63:0/4045741609 learned_addr learned my addr 172.21.15.63:0/4045741609 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe539e00700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c070c10 0x7fe53c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe53a800700 1 -- 172.21.15.63:0/4045741609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c070c10 msgr2=0x7fe53c116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe53a800700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c070c10 0x7fe53c116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.428+0000 7fe53a800700 1 -- 172.21.15.63:0/4045741609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528005040 con 0x7fe53c070300 2024-03-23T12:02:03.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.429+0000 7fe53a800700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070300 0x7fe53c115d30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe52c002a50 tx=0x7fe52c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:03.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.429+0000 7fe532a00700 1 -- 172.21.15.63:0/4045741609 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe52c007cc0 con 0x7fe53c070300 2024-03-23T12:02:03.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.429+0000 7fe532a00700 1 -- 172.21.15.63:0/4045741609 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe52c00f040 con 0x7fe53c070300 2024-03-23T12:02:03.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.429+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c10ff70 con 0x7fe53c070300 2024-03-23T12:02:03.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.429+0000 7fe532a00700 1 -- 172.21.15.63:0/4045741609 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe52c00da70 con 0x7fe53c070300 2024-03-23T12:02:03.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.429+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c1104b0 con 0x7fe53c070300 2024-03-23T12:02:03.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.430+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c071a40 con 0x7fe53c070300 2024-03-23T12:02:03.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.432+0000 7fe532a00700 1 -- 172.21.15.63:0/4045741609 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe52c00dc10 con 0x7fe53c070300 2024-03-23T12:02:03.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.433+0000 7fe532a00700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe52006c7d0 0x7fe52006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.433+0000 7fe532a00700 1 -- 172.21.15.63:0/4045741609 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe52c08b180 con 0x7fe53c070300 2024-03-23T12:02:03.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.433+0000 7fe539e00700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe52006c7d0 0x7fe52006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.434+0000 7fe539e00700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe52006c7d0 0x7fe52006ec80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe53c1115d0 tx=0x7fe528009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:03.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.437+0000 7fe532a00700 1 -- 172.21.15.63:0/4045741609 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe52c058e10 con 0x7fe53c070300 2024-03-23T12:02:03.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.674+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe53c04bf40 con 0x7fe52006c7d0 2024-03-23T12:02:03.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.676+0000 7fe532a00700 1 -- 172.21.15.63:0/4045741609 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe53c04bf40 con 0x7fe52006c7d0 2024-03-23T12:02:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe52006c7d0 msgr2=0x7fe52006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe52006c7d0 0x7fe52006ec80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe53c1115d0 tx=0x7fe528009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070300 msgr2=0x7fe53c115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070300 0x7fe53c115d30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe52c002a50 tx=0x7fe52c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 shutdown_connections 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe52006c7d0 0x7fe52006ec80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe53c070300 0x7fe53c115d30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 --2- 172.21.15.63:0/4045741609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c070c10 0x7fe53c116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 >> 172.21.15.63:0/4045741609 conn(0x7fe53c0f8970 msgr2=0x7fe53c1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 shutdown_connections 2024-03-23T12:02:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.678+0000 7fe541400700 1 -- 172.21.15.63:0/4045741609 wait complete. 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.817+0000 7f2f18600700 1 -- 172.21.15.63:0/3430649539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 msgr2=0x7f2f101031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.817+0000 7f2f18600700 1 --2- 172.21.15.63:0/3430649539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f101031d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2f04004940 tx=0x7f2f04009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.818+0000 7f2f18600700 1 -- 172.21.15.63:0/3430649539 shutdown_connections 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.818+0000 7f2f18600700 1 --2- 172.21.15.63:0/3430649539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f101037a0 0x7f2f100fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.818+0000 7f2f18600700 1 --2- 172.21.15.63:0/3430649539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f101031d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.818+0000 7f2f18600700 1 -- 172.21.15.63:0/3430649539 >> 172.21.15.63:0/3430649539 conn(0x7f2f100f89b0 msgr2=0x7f2f100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.818+0000 7f2f18600700 1 -- 172.21.15.63:0/3430649539 shutdown_connections 2024-03-23T12:02:03.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.818+0000 7f2f18600700 1 -- 172.21.15.63:0/3430649539 wait complete. 2024-03-23T12:02:03.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.818+0000 7f2f18600700 1 Processor -- start 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f18600700 1 -- start start 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f18600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f10115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f18600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f101037a0 0x7f2f10116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f18600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f10116860 con 0x7f2f10102e00 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f18600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1010fc30 con 0x7f2f101037a0 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f15c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f10115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f15c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f10115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41786/0 (socket says 172.21.15.63:41786) 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f15c00700 1 -- 172.21.15.63:0/2501071250 learned_addr learned my addr 172.21.15.63:0/2501071250 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.819+0000 7f2f15200700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f101037a0 0x7f2f10116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f15c00700 1 -- 172.21.15.63:0/2501071250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f101037a0 msgr2=0x7f2f10116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f15c00700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f101037a0 0x7f2f10116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:03.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f15c00700 1 -- 172.21.15.63:0/2501071250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f04005040 con 0x7f2f10102e00 2024-03-23T12:02:03.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f15c00700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f10115cd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009390 tx=0x7f2f04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:03.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f0d600700 1 -- 172.21.15.63:0/2501071250 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f0400e040 con 0x7f2f10102e00 2024-03-23T12:02:03.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f0d600700 1 -- 172.21.15.63:0/2501071250 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f04013420 con 0x7f2f10102e00 2024-03-23T12:02:03.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f0d600700 1 -- 172.21.15.63:0/2501071250 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f0401c630 con 0x7f2f10102e00 2024-03-23T12:02:03.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.820+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1010feb0 con 0x7f2f10102e00 2024-03-23T12:02:03.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.821+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f10110470 con 0x7f2f10102e00 2024-03-23T12:02:03.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.822+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f100fe550 con 0x7f2f10102e00 2024-03-23T12:02:03.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.822+0000 7f2f0d600700 1 -- 172.21.15.63:0/2501071250 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2f040139e0 con 0x7f2f10102e00 2024-03-23T12:02:03.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.823+0000 7f2f0d600700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2ef806c7d0 0x7f2ef806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:03.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.823+0000 7f2f0d600700 1 -- 172.21.15.63:0/2501071250 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f0408cd80 con 0x7f2f10102e00 2024-03-23T12:02:03.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.824+0000 7f2f15200700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2ef806c7d0 0x7f2ef806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:03.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.824+0000 7f2f15200700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2ef806c7d0 0x7f2ef806ec80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2f10111570 tx=0x7f2efc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:03.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:03.828+0000 7f2f0d600700 1 -- 172.21.15.63:0/2501071250 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f0405aa90 con 0x7f2f10102e00 2024-03-23T12:02:04.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.051+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f1004bf40 con 0x7f2ef806c7d0 2024-03-23T12:02:04.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.061+0000 7f2f0d600700 1 -- 172.21.15.63:0/2501071250 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f2f1004bf40 con 0x7f2ef806c7d0 2024-03-23T12:02:04.042 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:02:04.042 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (11m) 4m ago 24m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:02:04.042 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (8m) 4m ago 24m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:02:04.042 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (8m) 3m ago 23m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:02:04.042 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (11m) 4m ago 24m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:02:04.042 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (4m) 4m ago 18m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (4m) 4m ago 18m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (18m) 3m ago 18m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (3m) 3m ago 18m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (13m) 4m ago 26m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (12m) 3m ago 23m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (9m) 4m ago 26m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (9m) 3m ago 23m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (12m) 4m ago 24m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (12m) 3m ago 23m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (8m) 4m ago 22m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (7m) 4m ago 21m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (6m) 4m ago 21m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (6m) 3m ago 20m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (5m) 3m ago 20m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:02:04.043 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (4m) 3m ago 19m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:02:04.044 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (11m) 4m ago 23m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:02:04.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2ef806c7d0 msgr2=0x7f2ef806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2ef806c7d0 0x7f2ef806ec80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2f10111570 tx=0x7f2efc00b040 comp rx=0 tx=0).stop 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 msgr2=0x7f2f10115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f10115cd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009390 tx=0x7f2f04009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 shutdown_connections 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2ef806c7d0 0x7f2ef806ec80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f10102e00 0x7f2f10115cd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 --2- 172.21.15.63:0/2501071250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f101037a0 0x7f2f10116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 >> 172.21.15.63:0/2501071250 conn(0x7f2f100f89b0 msgr2=0x7f2f100f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 shutdown_connections 2024-03-23T12:02:04.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.063+0000 7f2f18600700 1 -- 172.21.15.63:0/2501071250 wait complete. 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 -- 172.21.15.63:0/461813947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 msgr2=0x7fea780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 --2- 172.21.15.63:0/461813947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea780fce10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fea68004940 tx=0x7fea68009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 -- 172.21.15.63:0/461813947 shutdown_connections 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 --2- 172.21.15.63:0/461813947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea780fce10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 --2- 172.21.15.63:0/461813947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea78102e00 0x7fea781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 -- 172.21.15.63:0/461813947 >> 172.21.15.63:0/461813947 conn(0x7fea780f89b0 msgr2=0x7fea780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 -- 172.21.15.63:0/461813947 shutdown_connections 2024-03-23T12:02:04.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.200+0000 7fea80800700 1 -- 172.21.15.63:0/461813947 wait complete. 2024-03-23T12:02:04.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea80800700 1 Processor -- start 2024-03-23T12:02:04.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea80800700 1 -- start start 2024-03-23T12:02:04.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea80800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea78102e00 0x7fea78115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:04.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea80800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea78116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:04.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea80800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea78116880 con 0x7fea781037a0 2024-03-23T12:02:04.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea80800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7810fc50 con 0x7fea78102e00 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea7d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea78116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea7d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea78116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41792/0 (socket says 172.21.15.63:41792) 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.201+0000 7fea7d400700 1 -- 172.21.15.63:0/894742837 learned_addr learned my addr 172.21.15.63:0/894742837 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea7de00700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea78102e00 0x7fea78115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea7d400700 1 -- 172.21.15.63:0/894742837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea78102e00 msgr2=0x7fea78115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea7d400700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea78102e00 0x7fea78115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea7d400700 1 -- 172.21.15.63:0/894742837 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea68005040 con 0x7fea781037a0 2024-03-23T12:02:04.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea7d400700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea78116230 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fea68009390 tx=0x7fea68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:04.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea72000700 1 -- 172.21.15.63:0/894742837 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea6800e040 con 0x7fea781037a0 2024-03-23T12:02:04.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea72000700 1 -- 172.21.15.63:0/894742837 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea68013420 con 0x7fea781037a0 2024-03-23T12:02:04.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea72000700 1 -- 172.21.15.63:0/894742837 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea6801c630 con 0x7fea781037a0 2024-03-23T12:02:04.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.202+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7810fed0 con 0x7fea781037a0 2024-03-23T12:02:04.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.203+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea78110440 con 0x7fea781037a0 2024-03-23T12:02:04.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.204+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea780fe550 con 0x7fea781037a0 2024-03-23T12:02:04.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.205+0000 7fea72000700 1 -- 172.21.15.63:0/894742837 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fea680139e0 con 0x7fea781037a0 2024-03-23T12:02:04.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.206+0000 7fea72000700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea6006c7d0 0x7fea6006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:04.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.206+0000 7fea72000700 1 -- 172.21.15.63:0/894742837 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fea6808cd20 con 0x7fea781037a0 2024-03-23T12:02:04.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.206+0000 7fea7de00700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea6006c7d0 0x7fea6006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:04.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.206+0000 7fea7de00700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea6006c7d0 0x7fea6006ec80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea6c004990 tx=0x7fea6c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:04.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.210+0000 7fea72000700 1 -- 172.21.15.63:0/894742837 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fea6805aa30 con 0x7fea781037a0 2024-03-23T12:02:04.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.506+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fea7804bed0 con 0x7fea781037a0 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.507+0000 7fea72000700 1 -- 172.21.15.63:0/894742837 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fea6805a5c0 con 0x7fea781037a0 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:02:04.489 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea6006c7d0 msgr2=0x7fea6006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea6006c7d0 0x7fea6006ec80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea6c004990 tx=0x7fea6c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 msgr2=0x7fea78116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea78116230 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fea68009390 tx=0x7fea68009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 shutdown_connections 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea6006c7d0 0x7fea6006ec80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea78102e00 0x7fea78115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 --2- 172.21.15.63:0/894742837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea781037a0 0x7fea78116230 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 >> 172.21.15.63:0/894742837 conn(0x7fea780f89b0 msgr2=0x7fea780f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 shutdown_connections 2024-03-23T12:02:04.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.509+0000 7fea80800700 1 -- 172.21.15.63:0/894742837 wait complete. 2024-03-23T12:02:04.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 -- 172.21.15.63:0/159533136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070c10 msgr2=0x7f39bc109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 --2- 172.21.15.63:0/159533136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070c10 0x7f39bc109190 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f39a8004970 tx=0x7f39a8009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:04.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 -- 172.21.15.63:0/159533136 shutdown_connections 2024-03-23T12:02:04.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 --2- 172.21.15.63:0/159533136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070c10 0x7f39bc109190 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 --2- 172.21.15.63:0/159533136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39bc070300 0x7f39bc0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 -- 172.21.15.63:0/159533136 >> 172.21.15.63:0/159533136 conn(0x7f39bc0f8970 msgr2=0x7f39bc0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:04.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 -- 172.21.15.63:0/159533136 shutdown_connections 2024-03-23T12:02:04.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.647+0000 7f39c4000700 1 -- 172.21.15.63:0/159533136 wait complete. 2024-03-23T12:02:04.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.648+0000 7f39c4000700 1 Processor -- start 2024-03-23T12:02:04.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.648+0000 7f39c4000700 1 -- start start 2024-03-23T12:02:04.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.648+0000 7f39c4000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070300 0x7f39bc115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:04.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.648+0000 7f39c4000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39bc070c10 0x7f39bc116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.648+0000 7f39c4000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc116850 con 0x7f39bc070300 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.648+0000 7f39c4000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc10fc20 con 0x7f39bc070c10 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070300 0x7f39bc115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070300 0x7f39bc115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41802/0 (socket says 172.21.15.63:41802) 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c1600700 1 -- 172.21.15.63:0/4021074511 learned_addr learned my addr 172.21.15.63:0/4021074511 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c0c00700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39bc070c10 0x7f39bc116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c1600700 1 -- 172.21.15.63:0/4021074511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39bc070c10 msgr2=0x7f39bc116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c1600700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39bc070c10 0x7f39bc116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c1600700 1 -- 172.21.15.63:0/4021074511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8005040 con 0x7f39bc070300 2024-03-23T12:02:04.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.649+0000 7f39c1600700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070300 0x7f39bc115cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f39b000eb40 tx=0x7f39b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:04.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.650+0000 7f39b9600700 1 -- 172.21.15.63:0/4021074511 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39b0007cc0 con 0x7f39bc070300 2024-03-23T12:02:04.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.650+0000 7f39b9600700 1 -- 172.21.15.63:0/4021074511 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39b000f040 con 0x7f39bc070300 2024-03-23T12:02:04.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.650+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc10fea0 con 0x7f39bc070300 2024-03-23T12:02:04.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.650+0000 7f39b9600700 1 -- 172.21.15.63:0/4021074511 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39b0009570 con 0x7f39bc070300 2024-03-23T12:02:04.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.650+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc1103b0 con 0x7f39bc070300 2024-03-23T12:02:04.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.651+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc071a40 con 0x7f39bc070300 2024-03-23T12:02:04.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.652+0000 7f39b9600700 1 -- 172.21.15.63:0/4021074511 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f39b00096d0 con 0x7f39bc070300 2024-03-23T12:02:04.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.653+0000 7f39b9600700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f39a406c7d0 0x7f39a406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:04.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.653+0000 7f39b9600700 1 -- 172.21.15.63:0/4021074511 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f39b008b760 con 0x7f39bc070300 2024-03-23T12:02:04.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.653+0000 7f39c0c00700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f39a406c7d0 0x7f39a406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:04.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.654+0000 7f39c0c00700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f39a406c7d0 0x7f39a406ec80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009fd0 tx=0x7f39a8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:04.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.657+0000 7f39b9600700 1 -- 172.21.15.63:0/4021074511 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f39b00594a0 con 0x7f39bc070300 2024-03-23T12:02:04.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.915+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f39bc04bed0 con 0x7f39bc070300 2024-03-23T12:02:04.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.915+0000 7f39b9600700 1 -- 172.21.15.63:0/4021074511 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f39b0059030 con 0x7f39bc070300 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:02:04.897 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:02:04.898 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:04.899 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:04.900 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:04.900 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:04.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f39a406c7d0 msgr2=0x7f39a406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f39a406c7d0 0x7f39a406ec80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009fd0 tx=0x7f39a8009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070300 msgr2=0x7f39bc115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070300 0x7f39bc115cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f39b000eb40 tx=0x7f39b000b040 comp rx=0 tx=0).stop 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 shutdown_connections 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f39a406c7d0 0x7f39a406ec80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39bc070300 0x7f39bc115cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 --2- 172.21.15.63:0/4021074511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39bc070c10 0x7f39bc116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 >> 172.21.15.63:0/4021074511 conn(0x7f39bc0f8970 msgr2=0x7f39bc0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 shutdown_connections 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:04.918+0000 7f39c4000700 1 -- 172.21.15.63:0/4021074511 wait complete. 2024-03-23T12:02:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:02:05.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 -- 172.21.15.63:0/2723029212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688104fa0 msgr2=0x7ff688105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 --2- 172.21.15.63:0/2723029212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688104fa0 0x7ff688105370 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff678004940 tx=0x7ff678009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:05.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 -- 172.21.15.63:0/2723029212 shutdown_connections 2024-03-23T12:02:05.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 --2- 172.21.15.63:0/2723029212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff688105940 0x7ff6880fefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 --2- 172.21.15.63:0/2723029212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688104fa0 0x7ff688105370 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 -- 172.21.15.63:0/2723029212 >> 172.21.15.63:0/2723029212 conn(0x7ff6880fab10 msgr2=0x7ff6880fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:05.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 -- 172.21.15.63:0/2723029212 shutdown_connections 2024-03-23T12:02:05.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.056+0000 7ff68ca00700 1 -- 172.21.15.63:0/2723029212 wait complete. 2024-03-23T12:02:05.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.057+0000 7ff68ca00700 1 Processor -- start 2024-03-23T12:02:05.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.057+0000 7ff68ca00700 1 -- start start 2024-03-23T12:02:05.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.057+0000 7ff68ca00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff688104fa0 0x7ff688115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:05.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.057+0000 7ff68ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688105940 0x7ff6881160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:05.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.057+0000 7ff68ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff688116740 con 0x7ff688105940 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.057+0000 7ff68ca00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68810fb50 con 0x7ff688104fa0 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688105940 0x7ff6881160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688105940 0x7ff6881160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41814/0 (socket says 172.21.15.63:41814) 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685400700 1 -- 172.21.15.63:0/4167203647 learned_addr learned my addr 172.21.15.63:0/4167203647 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685e00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff688104fa0 0x7ff688115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685400700 1 -- 172.21.15.63:0/4167203647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff688104fa0 msgr2=0x7ff688115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685400700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff688104fa0 0x7ff688115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685400700 1 -- 172.21.15.63:0/4167203647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff678005040 con 0x7ff688105940 2024-03-23T12:02:05.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.058+0000 7ff685400700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688105940 0x7ff6881160f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff674002a50 tx=0x7ff6740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:05.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.059+0000 7ff67e000700 1 -- 172.21.15.63:0/4167203647 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff674007cc0 con 0x7ff688105940 2024-03-23T12:02:05.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.059+0000 7ff67e000700 1 -- 172.21.15.63:0/4167203647 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff67400f040 con 0x7ff688105940 2024-03-23T12:02:05.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.059+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68810fe30 con 0x7ff688105940 2024-03-23T12:02:05.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.059+0000 7ff67e000700 1 -- 172.21.15.63:0/4167203647 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff67400da30 con 0x7ff688105940 2024-03-23T12:02:05.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.059+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6881103f0 con 0x7ff688105940 2024-03-23T12:02:05.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.060+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6881022b0 con 0x7ff688105940 2024-03-23T12:02:05.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.061+0000 7ff67e000700 1 -- 172.21.15.63:0/4167203647 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff67400db90 con 0x7ff688105940 2024-03-23T12:02:05.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.062+0000 7ff67e000700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff66c06c7c0 0x7ff66c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:05.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.062+0000 7ff67e000700 1 -- 172.21.15.63:0/4167203647 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff67408b660 con 0x7ff688105940 2024-03-23T12:02:05.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.062+0000 7ff685e00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff66c06c7c0 0x7ff66c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:05.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.063+0000 7ff685e00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff66c06c7c0 0x7ff66c06ec70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff678009fd0 tx=0x7ff678009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:05.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.067+0000 7ff67e000700 1 -- 172.21.15.63:0/4167203647 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff6740558e0 con 0x7ff688105940 2024-03-23T12:02:05.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:04 smithi063 ceph-mon[141124]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:05.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:04 smithi063 ceph-mon[141124]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:05.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:04 smithi063 ceph-mon[141124]: pgmap v328: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:05.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:04 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/894742837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:02:05.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:04 smithi111 ceph-mon[106899]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:05.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:04 smithi111 ceph-mon[106899]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:05.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:04 smithi111 ceph-mon[106899]: pgmap v328: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:05.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:04 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/894742837' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:02:05.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.300+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff688111200 con 0x7ff66c06c7c0 2024-03-23T12:02:05.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.302+0000 7ff67e000700 1 -- 172.21.15.63:0/4167203647 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff688111200 con 0x7ff66c06c7c0 2024-03-23T12:02:05.283 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:02:05.283 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:02:05.283 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:02:05.283 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:02:05.283 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:02:05.283 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:02:05.284 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff66c06c7c0 msgr2=0x7ff66c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff66c06c7c0 0x7ff66c06ec70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff678009fd0 tx=0x7ff678009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688105940 msgr2=0x7ff6881160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688105940 0x7ff6881160f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff674002a50 tx=0x7ff6740046b0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 shutdown_connections 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff66c06c7c0 0x7ff66c06ec70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff688104fa0 0x7ff688115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 --2- 172.21.15.63:0/4167203647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff688105940 0x7ff6881160f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 >> 172.21.15.63:0/4167203647 conn(0x7ff6880fab10 msgr2=0x7ff688109590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.304+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 shutdown_connections 2024-03-23T12:02:05.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.305+0000 7ff68ca00700 1 -- 172.21.15.63:0/4167203647 wait complete. 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.441+0000 7f8e04800700 1 -- 172.21.15.63:0/1934367713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc1037a0 msgr2=0x7f8dfc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.441+0000 7f8e04800700 1 --2- 172.21.15.63:0/1934367713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc1037a0 0x7f8dfc0fce10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004940 tx=0x7f8dec009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.441+0000 7f8e04800700 1 -- 172.21.15.63:0/1934367713 shutdown_connections 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.441+0000 7f8e04800700 1 --2- 172.21.15.63:0/1934367713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc1037a0 0x7f8dfc0fce10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.441+0000 7f8e04800700 1 --2- 172.21.15.63:0/1934367713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dfc102e00 0x7f8dfc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.441+0000 7f8e04800700 1 -- 172.21.15.63:0/1934367713 >> 172.21.15.63:0/1934367713 conn(0x7f8dfc0f89b0 msgr2=0x7f8dfc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.441+0000 7f8e04800700 1 -- 172.21.15.63:0/1934367713 shutdown_connections 2024-03-23T12:02:05.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.442+0000 7f8e04800700 1 -- 172.21.15.63:0/1934367713 wait complete. 2024-03-23T12:02:05.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.442+0000 7f8e04800700 1 Processor -- start 2024-03-23T12:02:05.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.442+0000 7f8e04800700 1 -- start start 2024-03-23T12:02:05.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e04800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc102e00 0x7f8dfc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e04800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dfc1037a0 0x7f8dfc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e04800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc070d90 con 0x7f8dfc102e00 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e04800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc070ed0 con 0x7f8dfc1037a0 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e01e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc102e00 0x7f8dfc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e01e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc102e00 0x7f8dfc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41822/0 (socket says 172.21.15.63:41822) 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e01e00700 1 -- 172.21.15.63:0/3250912118 learned_addr learned my addr 172.21.15.63:0/3250912118 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e01400700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dfc1037a0 0x7f8dfc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e01e00700 1 -- 172.21.15.63:0/3250912118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dfc1037a0 msgr2=0x7f8dfc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e01e00700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dfc1037a0 0x7f8dfc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.443+0000 7f8e01e00700 1 -- 172.21.15.63:0/3250912118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec005040 con 0x7f8dfc102e00 2024-03-23T12:02:05.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.444+0000 7f8e01e00700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc102e00 0x7f8dfc0702c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8df0002a50 tx=0x7f8df00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:05.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.444+0000 7f8df6000700 1 -- 172.21.15.63:0/3250912118 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df0007cc0 con 0x7f8dfc102e00 2024-03-23T12:02:05.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.444+0000 7f8df6000700 1 -- 172.21.15.63:0/3250912118 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8df000f040 con 0x7f8dfc102e00 2024-03-23T12:02:05.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.444+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc119e00 con 0x7f8dfc102e00 2024-03-23T12:02:05.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.444+0000 7f8df6000700 1 -- 172.21.15.63:0/3250912118 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df000d9c0 con 0x7f8dfc102e00 2024-03-23T12:02:05.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.444+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc11a280 con 0x7f8dfc102e00 2024-03-23T12:02:05.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.445+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc0fe550 con 0x7f8dfc102e00 2024-03-23T12:02:05.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.446+0000 7f8df6000700 1 -- 172.21.15.63:0/3250912118 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8df000db20 con 0x7f8dfc102e00 2024-03-23T12:02:05.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.447+0000 7f8df6000700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7c0 0x7f8de406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:05.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.447+0000 7f8df6000700 1 -- 172.21.15.63:0/3250912118 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8df008b860 con 0x7f8dfc102e00 2024-03-23T12:02:05.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.447+0000 7f8e01400700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7c0 0x7f8de406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:05.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.448+0000 7f8e01400700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7c0 0x7f8de406ec70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004890 tx=0x7f8dec009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:05.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.452+0000 7f8df6000700 1 -- 172.21.15.63:0/3250912118 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8df00559e0 con 0x7f8dfc102e00 2024-03-23T12:02:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.748+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8dfc0719b0 con 0x7f8dfc102e00 2024-03-23T12:02:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.749+0000 7f8df6000700 1 -- 172.21.15.63:0/3250912118 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f8df0059000 con 0x7f8dfc102e00 2024-03-23T12:02:05.730 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:02:05.730 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:02:05.730 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:02:05.731 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:02:05.731 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.751+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7c0 msgr2=0x7f8de406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.751+0000 7f8e04800700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7c0 0x7f8de406ec70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004890 tx=0x7f8dec009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.751+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc102e00 msgr2=0x7f8dfc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.751+0000 7f8e04800700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc102e00 0x7f8dfc0702c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8df0002a50 tx=0x7f8df00046b0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.752+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 shutdown_connections 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.752+0000 7f8e04800700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8de406c7c0 0x7f8de406ec70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.752+0000 7f8e04800700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dfc102e00 0x7f8dfc0702c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.752+0000 7f8e04800700 1 --2- 172.21.15.63:0/3250912118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dfc1037a0 0x7f8dfc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:05.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.752+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 >> 172.21.15.63:0/3250912118 conn(0x7f8dfc0f89b0 msgr2=0x7f8dfc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:05.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.752+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 shutdown_connections 2024-03-23T12:02:05.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:05.752+0000 7f8e04800700 1 -- 172.21.15.63:0/3250912118 wait complete. 2024-03-23T12:02:06.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:05 smithi063 ceph-mon[141124]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:06.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:05 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4021074511' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:02:06.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:05 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3250912118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:02:06.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:05 smithi111 ceph-mon[106899]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:06.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:05 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4021074511' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:02:06.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:05 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3250912118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:02:07.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:06 smithi111 ceph-mon[106899]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:07.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:06 smithi111 ceph-mon[106899]: pgmap v329: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:07.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:06 smithi063 ceph-mon[141124]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:07.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:06 smithi063 ceph-mon[141124]: pgmap v329: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:09.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:08 smithi111 ceph-mon[106899]: pgmap v330: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:09.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:08 smithi063 ceph-mon[141124]: pgmap v330: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:10 smithi111 ceph-mon[106899]: pgmap v331: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:10 smithi063 ceph-mon[141124]: pgmap v331: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:13.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:12 smithi063 ceph-mon[141124]: pgmap v332: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:13.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:12 smithi111 ceph-mon[106899]: pgmap v332: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:15.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:14 smithi111 ceph-mon[106899]: pgmap v333: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:15.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:14 smithi063 ceph-mon[141124]: pgmap v333: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:16.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:16 smithi063 ceph-mon[141124]: pgmap v334: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:16.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:16 smithi111 ceph-mon[106899]: pgmap v334: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:19.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:18 smithi063 ceph-mon[141124]: pgmap v335: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:19.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:18 smithi111 ceph-mon[106899]: pgmap v335: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:21.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:20 smithi063 ceph-mon[141124]: pgmap v336: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:21.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:20 smithi111 ceph-mon[106899]: pgmap v336: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:23.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:22 smithi063 ceph-mon[141124]: pgmap v337: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:23.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:22 smithi111 ceph-mon[106899]: pgmap v337: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:25.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:24 smithi111 ceph-mon[106899]: pgmap v338: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:25.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:24 smithi063 ceph-mon[141124]: pgmap v338: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:26 smithi063 ceph-mon[141124]: pgmap v339: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:26 smithi111 ceph-mon[106899]: pgmap v339: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:29.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:28 smithi063 ceph-mon[141124]: pgmap v340: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:29.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:28 smithi111 ceph-mon[106899]: pgmap v340: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:31.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:30 smithi063 ceph-mon[141124]: pgmap v341: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:31.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:30 smithi111 ceph-mon[106899]: pgmap v341: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:33.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:32 smithi063 ceph-mon[141124]: pgmap v342: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:33.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:32 smithi111 ceph-mon[106899]: pgmap v342: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:35.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:34 smithi111 ceph-mon[106899]: pgmap v343: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:35.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:34 smithi063 ceph-mon[141124]: pgmap v343: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:35.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.905+0000 7f0735400700 1 -- 172.21.15.63:0/3635306871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 msgr2=0x7f0730103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:35.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.905+0000 7f0735400700 1 --2- 172.21.15.63:0/3635306871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730103010 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0720004940 tx=0x7f0720009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:35.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.905+0000 7f0735400700 1 -- 172.21.15.63:0/3635306871 shutdown_connections 2024-03-23T12:02:35.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.905+0000 7f0735400700 1 --2- 172.21.15.63:0/3635306871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07301035e0 0x7f07300fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:35.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.905+0000 7f0735400700 1 --2- 172.21.15.63:0/3635306871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730103010 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:35.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.905+0000 7f0735400700 1 -- 172.21.15.63:0/3635306871 >> 172.21.15.63:0/3635306871 conn(0x7f07300f8830 msgr2=0x7f07300fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:35.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.906+0000 7f0735400700 1 -- 172.21.15.63:0/3635306871 shutdown_connections 2024-03-23T12:02:35.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.906+0000 7f0735400700 1 -- 172.21.15.63:0/3635306871 wait complete. 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.906+0000 7f0735400700 1 Processor -- start 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f0735400700 1 -- start start 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f0735400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f0735400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07301035e0 0x7f0730113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f0735400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07301144e0 con 0x7f0730102c40 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f0735400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073010d8d0 con 0x7f07301035e0 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f072fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f072fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35066/0 (socket says 172.21.15.63:35066) 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f072fe00700 1 -- 172.21.15.63:0/169170960 learned_addr learned my addr 172.21.15.63:0/169170960 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.907+0000 7f072f400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07301035e0 0x7f0730113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f072fe00700 1 -- 172.21.15.63:0/169170960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07301035e0 msgr2=0x7f0730113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f072fe00700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07301035e0 0x7f0730113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f072fe00700 1 -- 172.21.15.63:0/169170960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0720005040 con 0x7f0730102c40 2024-03-23T12:02:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f072fe00700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730113950 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0720009390 tx=0x7f0720009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:35.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f0727e00700 1 -- 172.21.15.63:0/169170960 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f072000e040 con 0x7f0730102c40 2024-03-23T12:02:35.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f0727e00700 1 -- 172.21.15.63:0/169170960 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0720013420 con 0x7f0730102c40 2024-03-23T12:02:35.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073010db50 con 0x7f0730102c40 2024-03-23T12:02:35.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f0727e00700 1 -- 172.21.15.63:0/169170960 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f072001c630 con 0x7f0730102c40 2024-03-23T12:02:35.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.908+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073010e110 con 0x7f0730102c40 2024-03-23T12:02:35.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.910+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07300fe390 con 0x7f0730102c40 2024-03-23T12:02:35.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.910+0000 7f0727e00700 1 -- 172.21.15.63:0/169170960 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f07200139e0 con 0x7f0730102c40 2024-03-23T12:02:35.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.911+0000 7f0727e00700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f071406c7d0 0x7f071406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:35.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.911+0000 7f0727e00700 1 -- 172.21.15.63:0/169170960 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f072008ce10 con 0x7f0730102c40 2024-03-23T12:02:35.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.912+0000 7f072f400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f071406c7d0 0x7f071406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:35.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.912+0000 7f072f400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f071406c7d0 0x7f071406ec80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f073010ea90 tx=0x7f071c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:35.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:35.916+0000 7f0727e00700 1 -- 172.21.15.63:0/169170960 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f072005aaa0 con 0x7f0730102c40 2024-03-23T12:02:36.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.148+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0730109bf0 con 0x7f071406c7d0 2024-03-23T12:02:36.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.150+0000 7f0727e00700 1 -- 172.21.15.63:0/169170960 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0730109bf0 con 0x7f071406c7d0 2024-03-23T12:02:36.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f071406c7d0 msgr2=0x7f071406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f071406c7d0 0x7f071406ec80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f073010ea90 tx=0x7f071c00b040 comp rx=0 tx=0).stop 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 msgr2=0x7f0730113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730113950 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0720009390 tx=0x7f0720009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 shutdown_connections 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f071406c7d0 0x7f071406ec80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0730102c40 0x7f0730113950 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 --2- 172.21.15.63:0/169170960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07301035e0 0x7f0730113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 >> 172.21.15.63:0/169170960 conn(0x7f07300f8830 msgr2=0x7f07300f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.152+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 shutdown_connections 2024-03-23T12:02:36.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.153+0000 7f0735400700 1 -- 172.21.15.63:0/169170960 wait complete. 2024-03-23T12:02:36.149 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:02:36.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 -- 172.21.15.63:0/2996937082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 msgr2=0x7fba0c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 --2- 172.21.15.63:0/2996937082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c1031d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fba00004940 tx=0x7fba00009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:36.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 -- 172.21.15.63:0/2996937082 shutdown_connections 2024-03-23T12:02:36.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 --2- 172.21.15.63:0/2996937082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c1037a0 0x7fba0c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 --2- 172.21.15.63:0/2996937082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c1031d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 -- 172.21.15.63:0/2996937082 >> 172.21.15.63:0/2996937082 conn(0x7fba0c0f89b0 msgr2=0x7fba0c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:36.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 -- 172.21.15.63:0/2996937082 shutdown_connections 2024-03-23T12:02:36.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.291+0000 7fba13a00700 1 -- 172.21.15.63:0/2996937082 wait complete. 2024-03-23T12:02:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.292+0000 7fba13a00700 1 Processor -- start 2024-03-23T12:02:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.292+0000 7fba13a00700 1 -- start start 2024-03-23T12:02:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.292+0000 7fba13a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.292+0000 7fba13a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c1037a0 0x7fba0c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.292+0000 7fba13a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c070d90 con 0x7fba0c102e00 2024-03-23T12:02:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.292+0000 7fba13a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c070ed0 con 0x7fba0c1037a0 2024-03-23T12:02:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba11000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba11000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35092/0 (socket says 172.21.15.63:35092) 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba11000700 1 -- 172.21.15.63:0/329886688 learned_addr learned my addr 172.21.15.63:0/329886688 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba11000700 1 -- 172.21.15.63:0/329886688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c1037a0 msgr2=0x7fba0c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba0be00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c1037a0 0x7fba0c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba11000700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c1037a0 0x7fba0c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba11000700 1 -- 172.21.15.63:0/329886688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00005040 con 0x7fba0c102e00 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba0be00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c1037a0 0x7fba0c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:02:36.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.293+0000 7fba11000700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c0702c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fba00009390 tx=0x7fba00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.294+0000 7fba08c00700 1 -- 172.21.15.63:0/329886688 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba0000e040 con 0x7fba0c102e00 2024-03-23T12:02:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.294+0000 7fba08c00700 1 -- 172.21.15.63:0/329886688 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba00013420 con 0x7fba0c102e00 2024-03-23T12:02:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.294+0000 7fba08c00700 1 -- 172.21.15.63:0/329886688 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba0001c630 con 0x7fba0c102e00 2024-03-23T12:02:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.294+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c071120 con 0x7fba0c102e00 2024-03-23T12:02:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.294+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c11a270 con 0x7fba0c102e00 2024-03-23T12:02:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.295+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c0fe550 con 0x7fba0c102e00 2024-03-23T12:02:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.296+0000 7fba08c00700 1 -- 172.21.15.63:0/329886688 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fba000139e0 con 0x7fba0c102e00 2024-03-23T12:02:36.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.297+0000 7fba08c00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f406c7d0 0x7fb9f406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:36.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.297+0000 7fba08c00700 1 -- 172.21.15.63:0/329886688 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fba0008cca0 con 0x7fba0c102e00 2024-03-23T12:02:36.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.297+0000 7fba0be00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f406c7d0 0x7fb9f406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:36.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.298+0000 7fba0be00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f406c7d0 0x7fb9f406ec80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8007fd0 tx=0x7fb9f8009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:36.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.301+0000 7fba08c00700 1 -- 172.21.15.63:0/329886688 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fba00056f20 con 0x7fba0c102e00 2024-03-23T12:02:36.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.534+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba0c0719b0 con 0x7fb9f406c7d0 2024-03-23T12:02:36.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.535+0000 7fba08c00700 1 -- 172.21.15.63:0/329886688 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fba0c0719b0 con 0x7fb9f406c7d0 2024-03-23T12:02:36.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f406c7d0 msgr2=0x7fb9f406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f406c7d0 0x7fb9f406ec80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8007fd0 tx=0x7fb9f8009040 comp rx=0 tx=0).stop 2024-03-23T12:02:36.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 msgr2=0x7fba0c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c0702c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fba00009390 tx=0x7fba00009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 shutdown_connections 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb9f406c7d0 0x7fb9f406ec80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba0c102e00 0x7fba0c0702c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 --2- 172.21.15.63:0/329886688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba0c1037a0 0x7fba0c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 >> 172.21.15.63:0/329886688 conn(0x7fba0c0f89b0 msgr2=0x7fba0c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 shutdown_connections 2024-03-23T12:02:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.537+0000 7fba13a00700 1 -- 172.21.15.63:0/329886688 wait complete. 2024-03-23T12:02:36.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.674+0000 7fcea1200700 1 -- 172.21.15.63:0/564357464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 msgr2=0x7fce9c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.674+0000 7fcea1200700 1 --2- 172.21.15.63:0/564357464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c0fce10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fce88004940 tx=0x7fce88009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.674+0000 7fcea1200700 1 -- 172.21.15.63:0/564357464 shutdown_connections 2024-03-23T12:02:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.674+0000 7fcea1200700 1 --2- 172.21.15.63:0/564357464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c0fce10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.674+0000 7fcea1200700 1 --2- 172.21.15.63:0/564357464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce9c102e00 0x7fce9c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.674+0000 7fcea1200700 1 -- 172.21.15.63:0/564357464 >> 172.21.15.63:0/564357464 conn(0x7fce9c0f89b0 msgr2=0x7fce9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.674+0000 7fcea1200700 1 -- 172.21.15.63:0/564357464 shutdown_connections 2024-03-23T12:02:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.675+0000 7fcea1200700 1 -- 172.21.15.63:0/564357464 wait complete. 2024-03-23T12:02:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.675+0000 7fcea1200700 1 Processor -- start 2024-03-23T12:02:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.675+0000 7fcea1200700 1 -- start start 2024-03-23T12:02:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fcea1200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce9c102e00 0x7fce9c115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fcea1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fcea1200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c1168b0 con 0x7fce9c1037a0 2024-03-23T12:02:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fcea1200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c10fc80 con 0x7fce9c102e00 2024-03-23T12:02:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fce99400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fce99400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35122/0 (socket says 172.21.15.63:35122) 2024-03-23T12:02:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fce99400700 1 -- 172.21.15.63:0/2024396910 learned_addr learned my addr 172.21.15.63:0/2024396910 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fce99e00700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce9c102e00 0x7fce9c115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fce99400700 1 -- 172.21.15.63:0/2024396910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce9c102e00 msgr2=0x7fce9c115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.676+0000 7fce99400700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce9c102e00 0x7fce9c115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.677+0000 7fce99400700 1 -- 172.21.15.63:0/2024396910 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce88005040 con 0x7fce9c1037a0 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.677+0000 7fce99400700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c116260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce88009390 tx=0x7fce88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.677+0000 7fce92000700 1 -- 172.21.15.63:0/2024396910 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8800e040 con 0x7fce9c1037a0 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.677+0000 7fce92000700 1 -- 172.21.15.63:0/2024396910 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce88004290 con 0x7fce9c1037a0 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.677+0000 7fce92000700 1 -- 172.21.15.63:0/2024396910 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce88013e70 con 0x7fce9c1037a0 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.677+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9c10ff00 con 0x7fce9c1037a0 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.677+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9c1104c0 con 0x7fce9c1037a0 2024-03-23T12:02:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.678+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c0fe550 con 0x7fce9c1037a0 2024-03-23T12:02:36.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.679+0000 7fce92000700 1 -- 172.21.15.63:0/2024396910 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fce88013690 con 0x7fce9c1037a0 2024-03-23T12:02:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.680+0000 7fce92000700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce8006c7d0 0x7fce8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.680+0000 7fce92000700 1 -- 172.21.15.63:0/2024396910 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fce8808cc20 con 0x7fce9c1037a0 2024-03-23T12:02:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.680+0000 7fce99e00700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce8006c7d0 0x7fce8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:36.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.681+0000 7fce99e00700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce8006c7d0 0x7fce8006ec80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fce8c004f40 tx=0x7fce8c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:36.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.685+0000 7fce92000700 1 -- 172.21.15.63:0/2024396910 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fce8805a9e0 con 0x7fce9c1037a0 2024-03-23T12:02:36.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:36 smithi063 ceph-mon[141124]: pgmap v344: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:36.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.909+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fce9c109e10 con 0x7fce8006c7d0 2024-03-23T12:02:36.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:36 smithi111 ceph-mon[106899]: pgmap v344: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.919+0000 7fce92000700 1 -- 172.21.15.63:0/2024396910 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fce9c109e10 con 0x7fce8006c7d0 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (11m) 4m ago 24m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (9m) 4m ago 25m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (9m) 4m ago 24m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (11m) 4m ago 24m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (5m) 4m ago 18m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (4m) 4m ago 18m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (18m) 4m ago 18m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:02:36.901 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (4m) 4m ago 18m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (14m) 4m ago 26m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (12m) 4m ago 23m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (10m) 4m ago 26m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (9m) 4m ago 23m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (12m) 4m ago 25m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (12m) 4m ago 24m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (8m) 4m ago 22m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (7m) 4m ago 22m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (7m) 4m ago 21m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (6m) 4m ago 21m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (5m) 4m ago 20m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (5m) 4m ago 20m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:02:36.902 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (12m) 4m ago 24m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:02:36.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce8006c7d0 msgr2=0x7fce8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce8006c7d0 0x7fce8006ec80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fce8c004f40 tx=0x7fce8c00b040 comp rx=0 tx=0).stop 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 msgr2=0x7fce9c116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c116260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce88009390 tx=0x7fce88009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 shutdown_connections 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce8006c7d0 0x7fce8006ec80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce9c102e00 0x7fce9c115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 --2- 172.21.15.63:0/2024396910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce9c1037a0 0x7fce9c116260 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 >> 172.21.15.63:0/2024396910 conn(0x7fce9c0f89b0 msgr2=0x7fce9c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.921+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 shutdown_connections 2024-03-23T12:02:36.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:36.922+0000 7fcea1200700 1 -- 172.21.15.63:0/2024396910 wait complete. 2024-03-23T12:02:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 -- 172.21.15.63:0/4227519342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 msgr2=0x7f9c781031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 --2- 172.21.15.63:0/4227519342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c781031d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9c68004940 tx=0x7f9c68009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 -- 172.21.15.63:0/4227519342 shutdown_connections 2024-03-23T12:02:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 --2- 172.21.15.63:0/4227519342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781037a0 0x7f9c780fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 --2- 172.21.15.63:0/4227519342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c781031d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 -- 172.21.15.63:0/4227519342 >> 172.21.15.63:0/4227519342 conn(0x7f9c780f89b0 msgr2=0x7f9c780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 -- 172.21.15.63:0/4227519342 shutdown_connections 2024-03-23T12:02:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.058+0000 7f9c7de00700 1 -- 172.21.15.63:0/4227519342 wait complete. 2024-03-23T12:02:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.059+0000 7f9c7de00700 1 Processor -- start 2024-03-23T12:02:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.060+0000 7f9c7de00700 1 -- start start 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.060+0000 7f9c7de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c78113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.060+0000 7f9c7de00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781037a0 0x7f9c78114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.060+0000 7f9c7de00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c781146b0 con 0x7f9c78102e00 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.060+0000 7f9c7de00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7810daa0 con 0x7f9c781037a0 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.060+0000 7f9c76a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c78113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c76a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c78113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35136/0 (socket says 172.21.15.63:35136) 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c76a00700 1 -- 172.21.15.63:0/2118351359 learned_addr learned my addr 172.21.15.63:0/2118351359 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c76000700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781037a0 0x7f9c78114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c76a00700 1 -- 172.21.15.63:0/2118351359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781037a0 msgr2=0x7f9c78114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c76a00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781037a0 0x7f9c78114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c76a00700 1 -- 172.21.15.63:0/2118351359 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68005040 con 0x7f9c78102e00 2024-03-23T12:02:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c76a00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c78113b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009180 tx=0x7f9c68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c6ea00700 1 -- 172.21.15.63:0/2118351359 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c6800e040 con 0x7f9c78102e00 2024-03-23T12:02:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c6ea00700 1 -- 172.21.15.63:0/2118351359 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c68004290 con 0x7f9c78102e00 2024-03-23T12:02:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7810dd20 con 0x7f9c78102e00 2024-03-23T12:02:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.061+0000 7f9c6ea00700 1 -- 172.21.15.63:0/2118351359 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c6801c3f0 con 0x7f9c78102e00 2024-03-23T12:02:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.062+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7810e2e0 con 0x7f9c78102e00 2024-03-23T12:02:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.063+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c780fe550 con 0x7f9c78102e00 2024-03-23T12:02:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.064+0000 7f9c6ea00700 1 -- 172.21.15.63:0/2118351359 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9c68013790 con 0x7f9c78102e00 2024-03-23T12:02:37.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.064+0000 7f9c6ea00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c5c06c7c0 0x7f9c5c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.065+0000 7f9c6ea00700 1 -- 172.21.15.63:0/2118351359 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c6808cdb0 con 0x7f9c78102e00 2024-03-23T12:02:37.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.065+0000 7f9c76000700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c5c06c7c0 0x7f9c5c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.065+0000 7f9c76000700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c5c06c7c0 0x7f9c5c06ec70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c7810ec60 tx=0x7f9c64009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:37.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.069+0000 7f9c6ea00700 1 -- 172.21.15.63:0/2118351359 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c6805aa40 con 0x7f9c78102e00 2024-03-23T12:02:37.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.369+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9c780fe550 con 0x7f9c78102e00 2024-03-23T12:02:37.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.369+0000 7f9c6ea00700 1 -- 172.21.15.63:0/2118351359 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f9c68013420 con 0x7f9c78102e00 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:02:37.352 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:02:37.353 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:02:37.353 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:02:37.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:02:37.353 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:02:37.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c5c06c7c0 msgr2=0x7f9c5c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c5c06c7c0 0x7f9c5c06ec70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c7810ec60 tx=0x7f9c64009ee0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 msgr2=0x7f9c78113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c78113b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009180 tx=0x7f9c68009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:37.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 shutdown_connections 2024-03-23T12:02:37.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c5c06c7c0 0x7f9c5c06ec70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c78102e00 0x7f9c78113b20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 --2- 172.21.15.63:0/2118351359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781037a0 0x7f9c78114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 >> 172.21.15.63:0/2118351359 conn(0x7f9c780f89b0 msgr2=0x7f9c780f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:37.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 shutdown_connections 2024-03-23T12:02:37.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.372+0000 7f9c7de00700 1 -- 172.21.15.63:0/2118351359 wait complete. 2024-03-23T12:02:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 -- 172.21.15.63:0/2244304598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b81035e0 msgr2=0x7fe8b80fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2244304598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b81035e0 0x7fe8b80fcc50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4004940 tx=0x7fe8a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 -- 172.21.15.63:0/2244304598 shutdown_connections 2024-03-23T12:02:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2244304598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b81035e0 0x7fe8b80fcc50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2244304598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8b8102c40 0x7fe8b8103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 -- 172.21.15.63:0/2244304598 >> 172.21.15.63:0/2244304598 conn(0x7fe8b80f8830 msgr2=0x7fe8b80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 -- 172.21.15.63:0/2244304598 shutdown_connections 2024-03-23T12:02:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.509+0000 7fe8bca00700 1 -- 172.21.15.63:0/2244304598 wait complete. 2024-03-23T12:02:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.510+0000 7fe8bca00700 1 Processor -- start 2024-03-23T12:02:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.510+0000 7fe8bca00700 1 -- start start 2024-03-23T12:02:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8bca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b8102c40 0x7fe8b8115a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8bca00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8b81035e0 0x7fe8b8115fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8bca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b81165f0 con 0x7fe8b8102c40 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8bca00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b810f9e0 con 0x7fe8b81035e0 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b8102c40 0x7fe8b8115a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b8102c40 0x7fe8b8115a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35160/0 (socket says 172.21.15.63:35160) 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b7400700 1 -- 172.21.15.63:0/2881821260 learned_addr learned my addr 172.21.15.63:0/2881821260 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b6a00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8b81035e0 0x7fe8b8115fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b7400700 1 -- 172.21.15.63:0/2881821260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8b81035e0 msgr2=0x7fe8b8115fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b7400700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8b81035e0 0x7fe8b8115fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b7400700 1 -- 172.21.15.63:0/2881821260 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a4005040 con 0x7fe8b8102c40 2024-03-23T12:02:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.511+0000 7fe8b7400700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b8102c40 0x7fe8b8115a60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8002a80 tx=0x7fe8a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.512+0000 7fe8af400700 1 -- 172.21.15.63:0/2881821260 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8a8007cc0 con 0x7fe8b8102c40 2024-03-23T12:02:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.512+0000 7fe8af400700 1 -- 172.21.15.63:0/2881821260 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8a800f040 con 0x7fe8b8102c40 2024-03-23T12:02:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.512+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b810fcc0 con 0x7fe8b8102c40 2024-03-23T12:02:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.512+0000 7fe8af400700 1 -- 172.21.15.63:0/2881821260 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8a8009640 con 0x7fe8b8102c40 2024-03-23T12:02:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.512+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b8110280 con 0x7fe8b8102c40 2024-03-23T12:02:37.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.513+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b80fe390 con 0x7fe8b8102c40 2024-03-23T12:02:37.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.514+0000 7fe8af400700 1 -- 172.21.15.63:0/2881821260 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe8a80097a0 con 0x7fe8b8102c40 2024-03-23T12:02:37.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.515+0000 7fe8af400700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe89c06c7d0 0x7fe89c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.515+0000 7fe8af400700 1 -- 172.21.15.63:0/2881821260 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe8a808b220 con 0x7fe8b8102c40 2024-03-23T12:02:37.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.515+0000 7fe8b6a00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe89c06c7d0 0x7fe89c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.516+0000 7fe8b6a00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe89c06c7d0 0x7fe89c06ec80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8110ba0 tx=0x7fe8a4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:37.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.519+0000 7fe8af400700 1 -- 172.21.15.63:0/2881821260 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe8a8058eb0 con 0x7fe8b8102c40 2024-03-23T12:02:37.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.779+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe8b8109bd0 con 0x7fe8b8102c40 2024-03-23T12:02:37.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.779+0000 7fe8af400700 1 -- 172.21.15.63:0/2881821260 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fe8a8058a40 con 0x7fe8b8102c40 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:02:37.762 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:02:37.763 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:37.764 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:37.765 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:02:37.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.781+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe89c06c7d0 msgr2=0x7fe89c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.781+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe89c06c7d0 0x7fe89c06ec80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8110ba0 tx=0x7fe8a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:37.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.781+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b8102c40 msgr2=0x7fe8b8115a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.781+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b8102c40 0x7fe8b8115a60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8002a80 tx=0x7fe8a80046b0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.782+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 shutdown_connections 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.782+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe89c06c7d0 0x7fe89c06ec80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.782+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8b8102c40 0x7fe8b8115a60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.782+0000 7fe8bca00700 1 --2- 172.21.15.63:0/2881821260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8b81035e0 0x7fe8b8115fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.782+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 >> 172.21.15.63:0/2881821260 conn(0x7fe8b80f8830 msgr2=0x7fe8b80f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.782+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 shutdown_connections 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.782+0000 7fe8bca00700 1 -- 172.21.15.63:0/2881821260 wait complete. 2024-03-23T12:02:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:02:37.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:37 smithi063 ceph-mon[141124]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:37.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:37 smithi063 ceph-mon[141124]: from='client.34690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:37.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:37 smithi063 ceph-mon[141124]: from='client.34694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:37.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:37 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2118351359' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:02:37.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:37 smithi111 ceph-mon[106899]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:37.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:37 smithi111 ceph-mon[106899]: from='client.34690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:37.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:37 smithi111 ceph-mon[106899]: from='client.34694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:37.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:37 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2118351359' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:02:37.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 -- 172.21.15.63:0/1473037175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec102e00 msgr2=0x7f86ec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 --2- 172.21.15.63:0/1473037175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec102e00 0x7f86ec1031d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f86e0004940 tx=0x7f86e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:37.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 -- 172.21.15.63:0/1473037175 shutdown_connections 2024-03-23T12:02:37.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 --2- 172.21.15.63:0/1473037175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86ec1037a0 0x7f86ec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 --2- 172.21.15.63:0/1473037175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec102e00 0x7f86ec1031d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 -- 172.21.15.63:0/1473037175 >> 172.21.15.63:0/1473037175 conn(0x7f86ec0f89b0 msgr2=0x7f86ec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:37.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 -- 172.21.15.63:0/1473037175 shutdown_connections 2024-03-23T12:02:37.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.921+0000 7f86f3600700 1 -- 172.21.15.63:0/1473037175 wait complete. 2024-03-23T12:02:37.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.922+0000 7f86f3600700 1 Processor -- start 2024-03-23T12:02:37.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.922+0000 7f86f3600700 1 -- start start 2024-03-23T12:02:37.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.922+0000 7f86f3600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86ec102e00 0x7f86ec115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.922+0000 7f86f3600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec1037a0 0x7f86ec116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.922+0000 7f86f3600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec116860 con 0x7f86ec1037a0 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.922+0000 7f86f3600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec10fc30 con 0x7f86ec102e00 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec1037a0 0x7f86ec116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec1037a0 0x7f86ec116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35170/0 (socket says 172.21.15.63:35170) 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86ebe00700 1 -- 172.21.15.63:0/1904939790 learned_addr learned my addr 172.21.15.63:0/1904939790 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86f0c00700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86ec102e00 0x7f86ec115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86ebe00700 1 -- 172.21.15.63:0/1904939790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86ec102e00 msgr2=0x7f86ec115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86ebe00700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86ec102e00 0x7f86ec115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86ebe00700 1 -- 172.21.15.63:0/1904939790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0005040 con 0x7f86ec1037a0 2024-03-23T12:02:37.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.923+0000 7f86ebe00700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec1037a0 0x7f86ec116210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f86d8002a50 tx=0x7f86d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:37.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.924+0000 7f86e8c00700 1 -- 172.21.15.63:0/1904939790 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d8007cc0 con 0x7f86ec1037a0 2024-03-23T12:02:37.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.924+0000 7f86e8c00700 1 -- 172.21.15.63:0/1904939790 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d800f040 con 0x7f86ec1037a0 2024-03-23T12:02:37.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.924+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec10ff10 con 0x7f86ec1037a0 2024-03-23T12:02:37.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.924+0000 7f86e8c00700 1 -- 172.21.15.63:0/1904939790 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d800d990 con 0x7f86ec1037a0 2024-03-23T12:02:37.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.924+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec1104d0 con 0x7f86ec1037a0 2024-03-23T12:02:37.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.925+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec0fe550 con 0x7f86ec1037a0 2024-03-23T12:02:37.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.926+0000 7f86e8c00700 1 -- 172.21.15.63:0/1904939790 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f86d8009710 con 0x7f86ec1037a0 2024-03-23T12:02:37.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.927+0000 7f86e8c00700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86d406c7d0 0x7f86d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:37.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.927+0000 7f86e8c00700 1 -- 172.21.15.63:0/1904939790 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f86d808b860 con 0x7f86ec1037a0 2024-03-23T12:02:37.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.927+0000 7f86f0c00700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86d406c7d0 0x7f86d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:37.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.928+0000 7f86f0c00700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86d406c7d0 0x7f86d406ec80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f86e000b020 tx=0x7f86e0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:37.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:37.931+0000 7f86e8c00700 1 -- 172.21.15.63:0/1904939790 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f86d80594f0 con 0x7f86ec1037a0 2024-03-23T12:02:38.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.164+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f86ec109e30 con 0x7f86d406c7d0 2024-03-23T12:02:38.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.165+0000 7f86e8c00700 1 -- 172.21.15.63:0/1904939790 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f86ec109e30 con 0x7f86d406c7d0 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:02:38.148 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:02:38.149 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:02:38.149 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:02:38.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86d406c7d0 msgr2=0x7f86d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:38.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86d406c7d0 0x7f86d406ec80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f86e000b020 tx=0x7f86e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:38.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec1037a0 msgr2=0x7f86ec116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:38.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec1037a0 0x7f86ec116210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f86d8002a50 tx=0x7f86d8004600 comp rx=0 tx=0).stop 2024-03-23T12:02:38.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 shutdown_connections 2024-03-23T12:02:38.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86d406c7d0 0x7f86d406ec80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86ec102e00 0x7f86ec115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 --2- 172.21.15.63:0/1904939790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ec1037a0 0x7f86ec116210 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 >> 172.21.15.63:0/1904939790 conn(0x7f86ec0f89b0 msgr2=0x7f86ec0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:38.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 shutdown_connections 2024-03-23T12:02:38.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.168+0000 7f86f3600700 1 -- 172.21.15.63:0/1904939790 wait complete. 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.305+0000 7f01d0000700 1 -- 172.21.15.63:0/2540062462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fe830 msgr2=0x7f01c8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.305+0000 7f01d0000700 1 --2- 172.21.15.63:0/2540062462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fe830 0x7f01c8106f00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f01b4004940 tx=0x7f01b4009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.305+0000 7f01d0000700 1 -- 172.21.15.63:0/2540062462 shutdown_connections 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.305+0000 7f01d0000700 1 --2- 172.21.15.63:0/2540062462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fe830 0x7f01c8106f00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.305+0000 7f01d0000700 1 --2- 172.21.15.63:0/2540062462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01c80fde90 0x7f01c80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.305+0000 7f01d0000700 1 -- 172.21.15.63:0/2540062462 >> 172.21.15.63:0/2540062462 conn(0x7f01c8072bc0 msgr2=0x7f01c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.306+0000 7f01d0000700 1 -- 172.21.15.63:0/2540062462 shutdown_connections 2024-03-23T12:02:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.306+0000 7f01d0000700 1 -- 172.21.15.63:0/2540062462 wait complete. 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.306+0000 7f01d0000700 1 Processor -- start 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01d0000700 1 -- start start 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01d0000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fde90 0x7f01c8115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01d0000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01c80fe830 0x7f01c8116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01d0000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c81168a0 con 0x7f01c80fde90 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01d0000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c810fc70 con 0x7f01c80fe830 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01cd600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fde90 0x7f01c8115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01cd600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fde90 0x7f01c8115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35186/0 (socket says 172.21.15.63:35186) 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.307+0000 7f01cd600700 1 -- 172.21.15.63:0/942672569 learned_addr learned my addr 172.21.15.63:0/942672569 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01ccc00700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01c80fe830 0x7f01c8116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:38.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01cd600700 1 -- 172.21.15.63:0/942672569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01c80fe830 msgr2=0x7f01c8116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01cd600700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01c80fe830 0x7f01c8116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01cd600700 1 -- 172.21.15.63:0/942672569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4005040 con 0x7f01c80fde90 2024-03-23T12:02:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01cd600700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fde90 0x7f01c8115d10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f01bc002a50 tx=0x7f01bc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01c5600700 1 -- 172.21.15.63:0/942672569 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01bc007cc0 con 0x7f01c80fde90 2024-03-23T12:02:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c810ff50 con 0x7f01c80fde90 2024-03-23T12:02:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.308+0000 7f01c5600700 1 -- 172.21.15.63:0/942672569 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01bc00f040 con 0x7f01c80fde90 2024-03-23T12:02:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.309+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c8110490 con 0x7f01c80fde90 2024-03-23T12:02:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.309+0000 7f01c5600700 1 -- 172.21.15.63:0/942672569 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01bc009500 con 0x7f01c80fde90 2024-03-23T12:02:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.309+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c8106a20 con 0x7f01c80fde90 2024-03-23T12:02:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.311+0000 7f01c5600700 1 -- 172.21.15.63:0/942672569 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f01bc0096a0 con 0x7f01c80fde90 2024-03-23T12:02:38.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.311+0000 7f01c5600700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f01b006c7d0 0x7f01b006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:02:38.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.312+0000 7f01c5600700 1 -- 172.21.15.63:0/942672569 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f01bc08b940 con 0x7f01c80fde90 2024-03-23T12:02:38.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.312+0000 7f01ccc00700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f01b006c7d0 0x7f01b006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:02:38.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.312+0000 7f01ccc00700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f01b006c7d0 0x7f01b006ec80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01c81115b0 tx=0x7f01b4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:02:38.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.316+0000 7f01c5600700 1 -- 172.21.15.63:0/942672569 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f01bc059550 con 0x7f01c80fde90 2024-03-23T12:02:38.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.612+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f01c804bed0 con 0x7f01c80fde90 2024-03-23T12:02:38.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.612+0000 7f01c5600700 1 -- 172.21.15.63:0/942672569 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f01bc0590e0 con 0x7f01c80fde90 2024-03-23T12:02:38.595 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:02:38.595 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:02:38.595 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:02:38.595 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:02:38.595 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:02:38.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.614+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f01b006c7d0 msgr2=0x7f01b006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.614+0000 7f01d0000700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f01b006c7d0 0x7f01b006ec80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01c81115b0 tx=0x7f01b4009e30 comp rx=0 tx=0).stop 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.614+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fde90 msgr2=0x7f01c8115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.614+0000 7f01d0000700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fde90 0x7f01c8115d10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f01bc002a50 tx=0x7f01bc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.615+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 shutdown_connections 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.615+0000 7f01d0000700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f01b006c7d0 0x7f01b006ec80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.615+0000 7f01d0000700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01c80fde90 0x7f01c8115d10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.615+0000 7f01d0000700 1 --2- 172.21.15.63:0/942672569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01c80fe830 0x7f01c8116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.615+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 >> 172.21.15.63:0/942672569 conn(0x7f01c8072bc0 msgr2=0x7f01c80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.615+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 shutdown_connections 2024-03-23T12:02:38.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:02:38.615+0000 7f01d0000700 1 -- 172.21.15.63:0/942672569 wait complete. 2024-03-23T12:02:38.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:38 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2881821260' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:02:38.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:38 smithi063 ceph-mon[141124]: pgmap v345: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:38.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:38 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2881821260' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:02:38.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:38 smithi111 ceph-mon[106899]: pgmap v345: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:39.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:39 smithi063 ceph-mon[141124]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:39.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:39 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/942672569' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:02:39.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:39 smithi111 ceph-mon[106899]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:02:39.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:39 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/942672569' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:02:40.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:40 smithi063 ceph-mon[141124]: pgmap v346: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:40.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:40 smithi111 ceph-mon[106899]: pgmap v346: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:41.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:41.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:42.755 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:42 smithi063 ceph-mon[141124]: pgmap v347: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:42.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:42 smithi111 ceph-mon[106899]: pgmap v347: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:45.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:44 smithi063 ceph-mon[141124]: pgmap v348: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:45.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:44 smithi111 ceph-mon[106899]: pgmap v348: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:46.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:46 smithi063 ceph-mon[141124]: pgmap v349: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:46.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:46 smithi111 ceph-mon[106899]: pgmap v349: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:49.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:48 smithi063 ceph-mon[141124]: pgmap v350: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:49.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:48 smithi111 ceph-mon[106899]: pgmap v350: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:51.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:50 smithi111 ceph-mon[106899]: pgmap v351: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:51.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:50 smithi063 ceph-mon[141124]: pgmap v351: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:53.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:52 smithi063 ceph-mon[141124]: pgmap v352: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:53.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:52 smithi111 ceph-mon[106899]: pgmap v352: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:55.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:54 smithi111 ceph-mon[106899]: pgmap v353: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:55.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:54 smithi063 ceph-mon[141124]: pgmap v353: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:56 smithi063 ceph-mon[141124]: pgmap v354: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:02:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:56 smithi111 ceph-mon[106899]: pgmap v354: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:59.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:02:58 smithi111 ceph-mon[106899]: pgmap v355: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:02:59.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:02:58 smithi063 ceph-mon[141124]: pgmap v355: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:01.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:00 smithi111 ceph-mon[106899]: pgmap v356: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:00 smithi063 ceph-mon[141124]: pgmap v356: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:03.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:02 smithi063 ceph-mon[141124]: pgmap v357: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:03.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:02 smithi111 ceph-mon[106899]: pgmap v357: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:05.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:04 smithi111 ceph-mon[106899]: pgmap v358: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:05.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:04 smithi063 ceph-mon[141124]: pgmap v358: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:06.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:06 smithi063 ceph-mon[141124]: pgmap v359: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:06.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:06 smithi111 ceph-mon[106899]: pgmap v359: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:08.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 -- 172.21.15.63:0/3055252792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d781037a0 msgr2=0x7f7d780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:08.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 --2- 172.21.15.63:0/3055252792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d781037a0 0x7f7d780fce10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7d64004940 tx=0x7f7d64009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:08.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 -- 172.21.15.63:0/3055252792 shutdown_connections 2024-03-23T12:03:08.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 --2- 172.21.15.63:0/3055252792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d781037a0 0x7f7d780fce10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:08.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 --2- 172.21.15.63:0/3055252792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78102e00 0x7f7d781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:08.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 -- 172.21.15.63:0/3055252792 >> 172.21.15.63:0/3055252792 conn(0x7f7d780f89b0 msgr2=0x7f7d780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:08.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 -- 172.21.15.63:0/3055252792 shutdown_connections 2024-03-23T12:03:08.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.770+0000 7f7d80600700 1 -- 172.21.15.63:0/3055252792 wait complete. 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.771+0000 7f7d80600700 1 Processor -- start 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.771+0000 7f7d80600700 1 -- start start 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.771+0000 7f7d80600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d78102e00 0x7f7d78115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.771+0000 7f7d80600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d781037a0 0x7f7d78116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.771+0000 7f7d80600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78116880 con 0x7f7d78102e00 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.771+0000 7f7d80600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7810fc50 con 0x7f7d781037a0 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.771+0000 7f7d7dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d78102e00 0x7f7d78115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.772+0000 7f7d7dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d78102e00 0x7f7d78115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35272/0 (socket says 172.21.15.63:35272) 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.772+0000 7f7d7dc00700 1 -- 172.21.15.63:0/470666087 learned_addr learned my addr 172.21.15.63:0/470666087 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.772+0000 7f7d7dc00700 1 -- 172.21.15.63:0/470666087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d781037a0 msgr2=0x7f7d78116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:08.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.772+0000 7f7d7dc00700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d781037a0 0x7f7d78116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:08.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.772+0000 7f7d7dc00700 1 -- 172.21.15.63:0/470666087 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64005040 con 0x7f7d78102e00 2024-03-23T12:03:08.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.772+0000 7f7d7dc00700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d78102e00 0x7f7d78115cf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c8c0 tx=0x7f7d6c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:08.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.772+0000 7f7d75600700 1 -- 172.21.15.63:0/470666087 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d6c00a980 con 0x7f7d78102e00 2024-03-23T12:03:08.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.773+0000 7f7d75600700 1 -- 172.21.15.63:0/470666087 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d6c00ce60 con 0x7f7d78102e00 2024-03-23T12:03:08.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.773+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7810ff30 con 0x7f7d78102e00 2024-03-23T12:03:08.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.773+0000 7f7d75600700 1 -- 172.21.15.63:0/470666087 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d6c0113a0 con 0x7f7d78102e00 2024-03-23T12:03:08.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.773+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d781104f0 con 0x7f7d78102e00 2024-03-23T12:03:08.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.774+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d780fe550 con 0x7f7d78102e00 2024-03-23T12:03:08.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.775+0000 7f7d75600700 1 -- 172.21.15.63:0/470666087 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7d6c012400 con 0x7f7d78102e00 2024-03-23T12:03:08.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.776+0000 7f7d75600700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d6006c7d0 0x7f7d6006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:08.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.776+0000 7f7d75600700 1 -- 172.21.15.63:0/470666087 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d6c08bc10 con 0x7f7d78102e00 2024-03-23T12:03:08.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.776+0000 7f7d7d200700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d6006c7d0 0x7f7d6006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:08.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.777+0000 7f7d7d200700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d6006c7d0 0x7f7d6006ec80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7d64004f90 tx=0x7f7d64009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:08.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:08.781+0000 7f7d75600700 1 -- 172.21.15.63:0/470666087 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d6c0598a0 con 0x7f7d78102e00 2024-03-23T12:03:08.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.015+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d78109da0 con 0x7f7d6006c7d0 2024-03-23T12:03:08.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.016+0000 7f7d75600700 1 -- 172.21.15.63:0/470666087 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7d78109da0 con 0x7f7d6006c7d0 2024-03-23T12:03:09.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d6006c7d0 msgr2=0x7f7d6006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d6006c7d0 0x7f7d6006ec80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7d64004f90 tx=0x7f7d64009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d78102e00 msgr2=0x7f7d78115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d78102e00 0x7f7d78115cf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c8c0 tx=0x7f7d6c0044c0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 shutdown_connections 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d6006c7d0 0x7f7d6006ec80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d78102e00 0x7f7d78115cf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 --2- 172.21.15.63:0/470666087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d781037a0 0x7f7d78116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.018+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 >> 172.21.15.63:0/470666087 conn(0x7f7d780f89b0 msgr2=0x7f7d780f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.019+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 shutdown_connections 2024-03-23T12:03:09.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.019+0000 7f7d80600700 1 -- 172.21.15.63:0/470666087 wait complete. 2024-03-23T12:03:09.015 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:03:09.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 -- 172.21.15.63:0/3319885867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 msgr2=0x7f89580fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 --2- 172.21.15.63:0/3319885867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f89580fce10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8944004940 tx=0x7f8944009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:09.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 -- 172.21.15.63:0/3319885867 shutdown_connections 2024-03-23T12:03:09.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 --2- 172.21.15.63:0/3319885867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f89580fce10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 --2- 172.21.15.63:0/3319885867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8958102e00 0x7f89581031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 -- 172.21.15.63:0/3319885867 >> 172.21.15.63:0/3319885867 conn(0x7f89580f89b0 msgr2=0x7f89580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:09.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 -- 172.21.15.63:0/3319885867 shutdown_connections 2024-03-23T12:03:09.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.157+0000 7f895d400700 1 -- 172.21.15.63:0/3319885867 wait complete. 2024-03-23T12:03:09.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.158+0000 7f895d400700 1 Processor -- start 2024-03-23T12:03:09.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.158+0000 7f895d400700 1 -- start start 2024-03-23T12:03:09.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.158+0000 7f895d400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8958102e00 0x7f8958115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.158+0000 7f895d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f8958116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.158+0000 7f895d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89581168b0 con 0x7f89581037a0 2024-03-23T12:03:09.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.158+0000 7f895d400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895810fc80 con 0x7f8958102e00 2024-03-23T12:03:09.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8955e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f8958116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8955e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f8958116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35294/0 (socket says 172.21.15.63:35294) 2024-03-23T12:03:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8955e00700 1 -- 172.21.15.63:0/4071704106 learned_addr learned my addr 172.21.15.63:0/4071704106 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8956800700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8958102e00 0x7f8958115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8955e00700 1 -- 172.21.15.63:0/4071704106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8958102e00 msgr2=0x7f8958115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8955e00700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8958102e00 0x7f8958115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8955e00700 1 -- 172.21.15.63:0/4071704106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8944005040 con 0x7f89581037a0 2024-03-23T12:03:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.159+0000 7f8955e00700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f8958116260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8944009390 tx=0x7f8944009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:09.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.160+0000 7f894ea00700 1 -- 172.21.15.63:0/4071704106 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894400e040 con 0x7f89581037a0 2024-03-23T12:03:09.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.160+0000 7f894ea00700 1 -- 172.21.15.63:0/4071704106 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8944013420 con 0x7f89581037a0 2024-03-23T12:03:09.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.160+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895810ff00 con 0x7f89581037a0 2024-03-23T12:03:09.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.160+0000 7f894ea00700 1 -- 172.21.15.63:0/4071704106 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894401c630 con 0x7f89581037a0 2024-03-23T12:03:09.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.160+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89581104c0 con 0x7f89581037a0 2024-03-23T12:03:09.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.161+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89580fe550 con 0x7f89581037a0 2024-03-23T12:03:09.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.162+0000 7f894ea00700 1 -- 172.21.15.63:0/4071704106 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8944013a00 con 0x7f89581037a0 2024-03-23T12:03:09.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.163+0000 7f894ea00700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f893c06c7d0 0x7f893c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.163+0000 7f894ea00700 1 -- 172.21.15.63:0/4071704106 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f894408cd50 con 0x7f89581037a0 2024-03-23T12:03:09.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.163+0000 7f8956800700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f893c06c7d0 0x7f893c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.164+0000 7f8956800700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f893c06c7d0 0x7f893c06ec80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8948004990 tx=0x7f8948009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:09.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.167+0000 7f894ea00700 1 -- 172.21.15.63:0/4071704106 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f894405aa60 con 0x7f89581037a0 2024-03-23T12:03:09.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:08 smithi111 ceph-mon[106899]: pgmap v360: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:09.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:08 smithi063 ceph-mon[141124]: pgmap v360: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:09.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.400+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8958109e10 con 0x7f893c06c7d0 2024-03-23T12:03:09.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.401+0000 7f894ea00700 1 -- 172.21.15.63:0/4071704106 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8958109e10 con 0x7f893c06c7d0 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f893c06c7d0 msgr2=0x7f893c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f893c06c7d0 0x7f893c06ec80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8948004990 tx=0x7f8948009ee0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 msgr2=0x7f8958116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f8958116260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8944009390 tx=0x7f8944009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 shutdown_connections 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f893c06c7d0 0x7f893c06ec80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8958102e00 0x7f8958115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 --2- 172.21.15.63:0/4071704106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89581037a0 0x7f8958116260 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 >> 172.21.15.63:0/4071704106 conn(0x7f89580f89b0 msgr2=0x7f89580f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:09.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 shutdown_connections 2024-03-23T12:03:09.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.404+0000 7f895d400700 1 -- 172.21.15.63:0/4071704106 wait complete. 2024-03-23T12:03:09.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 -- 172.21.15.63:0/1696602003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c102e00 msgr2=0x7faa9c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1696602003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c102e00 0x7faa9c1031d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faa8c004940 tx=0x7faa8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:09.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 -- 172.21.15.63:0/1696602003 shutdown_connections 2024-03-23T12:03:09.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1696602003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c1037a0 0x7faa9c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1696602003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c102e00 0x7faa9c1031d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 -- 172.21.15.63:0/1696602003 >> 172.21.15.63:0/1696602003 conn(0x7faa9c0f89b0 msgr2=0x7faa9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:09.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 -- 172.21.15.63:0/1696602003 shutdown_connections 2024-03-23T12:03:09.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.541+0000 7faaa0a00700 1 -- 172.21.15.63:0/1696602003 wait complete. 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.542+0000 7faaa0a00700 1 Processor -- start 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.542+0000 7faaa0a00700 1 -- start start 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faaa0a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faaa0a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faaa0a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c070d90 con 0x7faa9c1037a0 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faaa0a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c070ed0 con 0x7faa9c102e00 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faa99400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faa99400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35322/0 (socket says 172.21.15.63:35322) 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faa99400700 1 -- 172.21.15.63:0/1149073879 learned_addr learned my addr 172.21.15.63:0/1149073879 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faa99e00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faa99400700 1 -- 172.21.15.63:0/1149073879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 msgr2=0x7faa9c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faa99400700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.543+0000 7faa99400700 1 -- 172.21.15.63:0/1149073879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c005040 con 0x7faa9c1037a0 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.544+0000 7faa99400700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c070800 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faa8800eb40 tx=0x7faa8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.544+0000 7faa92000700 1 -- 172.21.15.63:0/1149073879 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa88007cc0 con 0x7faa9c1037a0 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.544+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c071120 con 0x7faa9c1037a0 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.544+0000 7faa92000700 1 -- 172.21.15.63:0/1149073879 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa8800f040 con 0x7faa9c1037a0 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.544+0000 7faa92000700 1 -- 172.21.15.63:0/1149073879 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8800c8f0 con 0x7faa9c1037a0 2024-03-23T12:03:09.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.544+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c11a170 con 0x7faa9c1037a0 2024-03-23T12:03:09.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.545+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c0fe550 con 0x7faa9c1037a0 2024-03-23T12:03:09.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.546+0000 7faa92000700 1 -- 172.21.15.63:0/1149073879 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7faa8800cb60 con 0x7faa9c1037a0 2024-03-23T12:03:09.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.547+0000 7faa92000700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.547+0000 7faa92000700 1 -- 172.21.15.63:0/1149073879 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa8808b500 con 0x7faa9c1037a0 2024-03-23T12:03:09.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.548+0000 7faa99e00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.548+0000 7faa99e00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0045a0 tx=0x7faa8c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:09.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.552+0000 7faa92000700 1 -- 172.21.15.63:0/1149073879 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faa880557b0 con 0x7faa9c1037a0 2024-03-23T12:03:09.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.773+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faa9c04bed0 con 0x7faa8006c7d0 2024-03-23T12:03:09.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.783+0000 7faa92000700 1 -- 172.21.15.63:0/1149073879 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7faa9c04bed0 con 0x7faa8006c7d0 2024-03-23T12:03:09.765 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:03:09.765 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (12m) 5m ago 25m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (9m) 5m ago 25m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (9m) 4m ago 24m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (12m) 5m ago 25m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (5m) 5m ago 19m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (5m) 5m ago 19m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (19m) 4m ago 19m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (4m) 4m ago 19m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (14m) 5m ago 27m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (13m) 4m ago 24m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (10m) 5m ago 27m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (10m) 4m ago 24m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (13m) 5m ago 25m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (13m) 4m ago 24m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:03:09.766 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (9m) 5m ago 23m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:03:09.767 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (8m) 5m ago 22m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:03:09.767 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (7m) 5m ago 22m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:03:09.767 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (7m) 4m ago 21m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:03:09.767 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (6m) 4m ago 21m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:03:09.767 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (5m) 4m ago 20m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:03:09.767 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (12m) 5m ago 25m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 msgr2=0x7faa8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0045a0 tx=0x7faa8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 msgr2=0x7faa9c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c070800 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faa8800eb40 tx=0x7faa8800b040 comp rx=0 tx=0).stop 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 shutdown_connections 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 --2- 172.21.15.63:0/1149073879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c070800 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 >> 172.21.15.63:0/1149073879 conn(0x7faa9c0f89b0 msgr2=0x7faa9c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:09.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 shutdown_connections 2024-03-23T12:03:09.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.786+0000 7faaa0a00700 1 -- 172.21.15.63:0/1149073879 wait complete. 2024-03-23T12:03:09.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.922+0000 7f96fb800700 1 -- 172.21.15.63:0/2603710981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 msgr2=0x7f96f41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.922+0000 7f96fb800700 1 --2- 172.21.15.63:0/2603710981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f41031d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f96e8004940 tx=0x7f96e8009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:09.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.922+0000 7f96fb800700 1 -- 172.21.15.63:0/2603710981 shutdown_connections 2024-03-23T12:03:09.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.922+0000 7f96fb800700 1 --2- 172.21.15.63:0/2603710981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f41037a0 0x7f96f40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.922+0000 7f96fb800700 1 --2- 172.21.15.63:0/2603710981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f41031d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.922+0000 7f96fb800700 1 -- 172.21.15.63:0/2603710981 >> 172.21.15.63:0/2603710981 conn(0x7f96f40f89b0 msgr2=0x7f96f40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:09.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.922+0000 7f96fb800700 1 -- 172.21.15.63:0/2603710981 shutdown_connections 2024-03-23T12:03:09.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.923+0000 7f96fb800700 1 -- 172.21.15.63:0/2603710981 wait complete. 2024-03-23T12:03:09.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.923+0000 7f96fb800700 1 Processor -- start 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96fb800700 1 -- start start 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96fb800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f4115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96fb800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f41037a0 0x7f96f4116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96fb800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f41167e0 con 0x7f96f4102e00 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96fb800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f410fbb0 con 0x7f96f41037a0 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96f8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f4115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96f8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f4115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35342/0 (socket says 172.21.15.63:35342) 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96f8e00700 1 -- 172.21.15.63:0/1088042803 learned_addr learned my addr 172.21.15.63:0/1088042803 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96f3e00700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f41037a0 0x7f96f4116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.924+0000 7f96f8e00700 1 -- 172.21.15.63:0/1088042803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f41037a0 msgr2=0x7f96f4116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96f8e00700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f41037a0 0x7f96f4116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:09.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96f8e00700 1 -- 172.21.15.63:0/1088042803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8005040 con 0x7f96f4102e00 2024-03-23T12:03:09.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96f3e00700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f41037a0 0x7f96f4116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:03:09.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96f8e00700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f4115c50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009390 tx=0x7f96e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:09.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96f0c00700 1 -- 172.21.15.63:0/1088042803 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e800e040 con 0x7f96f4102e00 2024-03-23T12:03:09.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96f0c00700 1 -- 172.21.15.63:0/1088042803 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96e8013420 con 0x7f96f4102e00 2024-03-23T12:03:09.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f410fe30 con 0x7f96f4102e00 2024-03-23T12:03:09.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96f0c00700 1 -- 172.21.15.63:0/1088042803 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e801c630 con 0x7f96f4102e00 2024-03-23T12:03:09.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.925+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f41103f0 con 0x7f96f4102e00 2024-03-23T12:03:09.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.926+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f40fe550 con 0x7f96f4102e00 2024-03-23T12:03:09.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.927+0000 7f96f0c00700 1 -- 172.21.15.63:0/1088042803 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f96e80139e0 con 0x7f96f4102e00 2024-03-23T12:03:09.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.928+0000 7f96f0c00700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f96dc06c7d0 0x7f96dc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:09.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.928+0000 7f96f0c00700 1 -- 172.21.15.63:0/1088042803 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f96e808cce0 con 0x7f96f4102e00 2024-03-23T12:03:09.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.928+0000 7f96f3e00700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f96dc06c7d0 0x7f96dc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:09.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.929+0000 7f96f3e00700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f96dc06c7d0 0x7f96dc06ec80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009b80 tx=0x7f96e000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:09.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:09.932+0000 7f96f0c00700 1 -- 172.21.15.63:0/1088042803 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f96e8091070 con 0x7f96f4102e00 2024-03-23T12:03:10.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.234+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f96f404bed0 con 0x7f96f4102e00 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.235+0000 7f96f0c00700 1 -- 172.21.15.63:0/1088042803 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f96e805aaa0 con 0x7f96f4102e00 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:03:10.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:03:10.218 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:03:10.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f96dc06c7d0 msgr2=0x7f96dc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f96dc06c7d0 0x7f96dc06ec80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009b80 tx=0x7f96e000b040 comp rx=0 tx=0).stop 2024-03-23T12:03:10.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 msgr2=0x7f96f4115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f4115c50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009390 tx=0x7f96e8009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:10.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 shutdown_connections 2024-03-23T12:03:10.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f96dc06c7d0 0x7f96dc06ec80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96f4102e00 0x7f96f4115c50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 --2- 172.21.15.63:0/1088042803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f41037a0 0x7f96f4116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 >> 172.21.15.63:0/1088042803 conn(0x7f96f40f89b0 msgr2=0x7f96f40f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:10.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 shutdown_connections 2024-03-23T12:03:10.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.237+0000 7f96fb800700 1 -- 172.21.15.63:0/1088042803 wait complete. 2024-03-23T12:03:10.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 -- 172.21.15.63:0/3914087453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 msgr2=0x7fd7901031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 --2- 172.21.15.63:0/3914087453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd7901031d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd780004940 tx=0x7fd780009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:10.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 -- 172.21.15.63:0/3914087453 shutdown_connections 2024-03-23T12:03:10.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 --2- 172.21.15.63:0/3914087453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7901037a0 0x7fd7900fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 --2- 172.21.15.63:0/3914087453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd7901031d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 -- 172.21.15.63:0/3914087453 >> 172.21.15.63:0/3914087453 conn(0x7fd7900f89b0 msgr2=0x7fd7900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:10.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 -- 172.21.15.63:0/3914087453 shutdown_connections 2024-03-23T12:03:10.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.374+0000 7fd797000700 1 -- 172.21.15.63:0/3914087453 wait complete. 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.375+0000 7fd797000700 1 Processor -- start 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.375+0000 7fd797000700 1 -- start start 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd797000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd790115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd797000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7901037a0 0x7fd790115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd797000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7901166b0 con 0x7fd790102e00 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd797000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79010fc00 con 0x7fd7901037a0 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd790115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:10.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd790115a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35366/0 (socket says 172.21.15.63:35366) 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78fe00700 1 -- 172.21.15.63:0/2101869799 learned_addr learned my addr 172.21.15.63:0/2101869799 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78f400700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7901037a0 0x7fd790115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78fe00700 1 -- 172.21.15.63:0/2101869799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7901037a0 msgr2=0x7fd790115fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78fe00700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7901037a0 0x7fd790115fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78fe00700 1 -- 172.21.15.63:0/2101869799 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780005040 con 0x7fd790102e00 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.376+0000 7fd78f400700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7901037a0 0x7fd790115fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.377+0000 7fd78fe00700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd790115a90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd780009360 tx=0x7fd780009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.377+0000 7fd787e00700 1 -- 172.21.15.63:0/2101869799 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd78000e040 con 0x7fd790102e00 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.377+0000 7fd787e00700 1 -- 172.21.15.63:0/2101869799 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd780013420 con 0x7fd790102e00 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.377+0000 7fd787e00700 1 -- 172.21.15.63:0/2101869799 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd78001c5f0 con 0x7fd790102e00 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.377+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79010fe20 con 0x7fd790102e00 2024-03-23T12:03:10.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.377+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7901103e0 con 0x7fd790102e00 2024-03-23T12:03:10.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.378+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7900fe550 con 0x7fd790102e00 2024-03-23T12:03:10.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.379+0000 7fd787e00700 1 -- 172.21.15.63:0/2101869799 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd780013590 con 0x7fd790102e00 2024-03-23T12:03:10.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.380+0000 7fd787e00700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7c0 0x7fd77406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:10.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.380+0000 7fd787e00700 1 -- 172.21.15.63:0/2101869799 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd78008cbc0 con 0x7fd790102e00 2024-03-23T12:03:10.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.380+0000 7fd78f400700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7c0 0x7fd77406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:10.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.381+0000 7fd78f400700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7c0 0x7fd77406ec70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd790111330 tx=0x7fd77c008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:10.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.385+0000 7fd787e00700 1 -- 172.21.15.63:0/2101869799 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd78005a990 con 0x7fd790102e00 2024-03-23T12:03:10.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.650+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd790110650 con 0x7fd790102e00 2024-03-23T12:03:10.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.650+0000 7fd787e00700 1 -- 172.21.15.63:0/2101869799 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fd78005a520 con 0x7fd790102e00 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:03:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:03:10.634 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:10.635 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:10.636 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.652+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7c0 msgr2=0x7fd77406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.652+0000 7fd797000700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7c0 0x7fd77406ec70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd790111330 tx=0x7fd77c008040 comp rx=0 tx=0).stop 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 msgr2=0x7fd790115a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd790115a90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd780009360 tx=0x7fd780009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 shutdown_connections 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7c0 0x7fd77406ec70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd790102e00 0x7fd790115a90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 --2- 172.21.15.63:0/2101869799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7901037a0 0x7fd790115fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 >> 172.21.15.63:0/2101869799 conn(0x7fd7900f89b0 msgr2=0x7fd790071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 shutdown_connections 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.653+0000 7fd797000700 1 -- 172.21.15.63:0/2101869799 wait complete. 2024-03-23T12:03:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:03:10.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 -- 172.21.15.63:0/1268485535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070360 msgr2=0x7f36f0070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 --2- 172.21.15.63:0/1268485535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070360 0x7f36f0070730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009d10 tx=0x7f36e0004d10 comp rx=0 tx=0).stop 2024-03-23T12:03:10.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 -- 172.21.15.63:0/1268485535 shutdown_connections 2024-03-23T12:03:10.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 --2- 172.21.15.63:0/1268485535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36f0070c70 0x7f36f0109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 --2- 172.21.15.63:0/1268485535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070360 0x7f36f0070730 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 -- 172.21.15.63:0/1268485535 >> 172.21.15.63:0/1268485535 conn(0x7f36f00f89b0 msgr2=0x7f36f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:10.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 -- 172.21.15.63:0/1268485535 shutdown_connections 2024-03-23T12:03:10.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.793+0000 7f36f5c00700 1 -- 172.21.15.63:0/1268485535 wait complete. 2024-03-23T12:03:10.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.794+0000 7f36f5c00700 1 Processor -- start 2024-03-23T12:03:10.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.794+0000 7f36f5c00700 1 -- start start 2024-03-23T12:03:10.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.794+0000 7f36f5c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36f0070360 0x7f36f0115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.794+0000 7f36f5c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070c70 0x7f36f01163f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.794+0000 7f36f5c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f0116a40 con 0x7f36f0070c70 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.794+0000 7f36f5c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f010fe10 con 0x7f36f0070360 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36ee000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070c70 0x7f36f01163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36ee000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070c70 0x7f36f01163f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35398/0 (socket says 172.21.15.63:35398) 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36ee000700 1 -- 172.21.15.63:0/84469703 learned_addr learned my addr 172.21.15.63:0/84469703 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36eea00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36f0070360 0x7f36f0115eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36ee000700 1 -- 172.21.15.63:0/84469703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36f0070360 msgr2=0x7f36f0115eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36ee000700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36f0070360 0x7f36f0115eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36ee000700 1 -- 172.21.15.63:0/84469703 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e0009940 con 0x7f36f0070c70 2024-03-23T12:03:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36eea00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36f0070360 0x7f36f0115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:03:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36ee000700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070c70 0x7f36f01163f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f36dc002a50 tx=0x7f36dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.795+0000 7f36e6a00700 1 -- 172.21.15.63:0/84469703 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36dc007cc0 con 0x7f36f0070c70 2024-03-23T12:03:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.796+0000 7f36e6a00700 1 -- 172.21.15.63:0/84469703 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36dc00f040 con 0x7f36f0070c70 2024-03-23T12:03:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.796+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f01100f0 con 0x7f36f0070c70 2024-03-23T12:03:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.796+0000 7f36e6a00700 1 -- 172.21.15.63:0/84469703 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36dc009540 con 0x7f36f0070c70 2024-03-23T12:03:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.796+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f0110630 con 0x7f36f0070c70 2024-03-23T12:03:10.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.797+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f0103bd0 con 0x7f36f0070c70 2024-03-23T12:03:10.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.798+0000 7f36e6a00700 1 -- 172.21.15.63:0/84469703 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f36dc0096a0 con 0x7f36f0070c70 2024-03-23T12:03:10.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.799+0000 7f36e6a00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f36d406c7d0 0x7f36d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:10.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.799+0000 7f36e6a00700 1 -- 172.21.15.63:0/84469703 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f36dc08b920 con 0x7f36f0070c70 2024-03-23T12:03:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.799+0000 7f36eea00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f36d406c7d0 0x7f36d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.800+0000 7f36eea00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f36d406c7d0 0x7f36d406ec80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005010 tx=0x7f36e0004d10 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:10.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:10.803+0000 7f36e6a00700 1 -- 172.21.15.63:0/84469703 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f36dc059530 con 0x7f36f0070c70 2024-03-23T12:03:11.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.039+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36f0111570 con 0x7f36d406c7d0 2024-03-23T12:03:11.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.041+0000 7f36e6a00700 1 -- 172.21.15.63:0/84469703 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f36f0111570 con 0x7f36d406c7d0 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:03:11.023 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:03:11.024 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:03:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f36d406c7d0 msgr2=0x7f36d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f36d406c7d0 0x7f36d406ec80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005010 tx=0x7f36e0004d10 comp rx=0 tx=0).stop 2024-03-23T12:03:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070c70 msgr2=0x7f36f01163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070c70 0x7f36f01163f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f36dc002a50 tx=0x7f36dc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 shutdown_connections 2024-03-23T12:03:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f36d406c7d0 0x7f36d406ec80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36f0070360 0x7f36f0115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 --2- 172.21.15.63:0/84469703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f0070c70 0x7f36f01163f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 >> 172.21.15.63:0/84469703 conn(0x7f36f00f89b0 msgr2=0x7f36f01015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:11.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 shutdown_connections 2024-03-23T12:03:11.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.043+0000 7f36f5c00700 1 -- 172.21.15.63:0/84469703 wait complete. 2024-03-23T12:03:11.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:10 smithi111 ceph-mon[106899]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:11.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:10 smithi111 ceph-mon[106899]: from='client.34718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:11.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:10 smithi111 ceph-mon[106899]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:10 smithi111 ceph-mon[106899]: pgmap v361: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:10 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1088042803' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:03:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:10 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2101869799' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:03:11.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:11.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 -- 172.21.15.63:0/3082485478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070360 msgr2=0x7f5e38070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:11.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3082485478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070360 0x7f5e38070730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5e28004970 tx=0x7f5e28009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:11.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 -- 172.21.15.63:0/3082485478 shutdown_connections 2024-03-23T12:03:11.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3082485478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e38070c70 0x7f5e38109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3082485478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070360 0x7f5e38070730 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 -- 172.21.15.63:0/3082485478 >> 172.21.15.63:0/3082485478 conn(0x7f5e380f89b0 msgr2=0x7f5e380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:11.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 -- 172.21.15.63:0/3082485478 shutdown_connections 2024-03-23T12:03:11.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.180+0000 7f5e3e200700 1 -- 172.21.15.63:0/3082485478 wait complete. 2024-03-23T12:03:11.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.181+0000 7f5e3e200700 1 Processor -- start 2024-03-23T12:03:11.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.181+0000 7f5e3e200700 1 -- start start 2024-03-23T12:03:11.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.182+0000 7f5e3e200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e38070360 0x7f5e38115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:11.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.182+0000 7f5e3e200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070c70 0x7f5e381161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:11.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.182+0000 7f5e3e200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e38116810 con 0x7f5e38070c70 2024-03-23T12:03:11.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.182+0000 7f5e3e200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3810fbe0 con 0x7f5e38070360 2024-03-23T12:03:11.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.182+0000 7f5e36a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070c70 0x7f5e381161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:11.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.182+0000 7f5e36a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070c70 0x7f5e381161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35416/0 (socket says 172.21.15.63:35416) 2024-03-23T12:03:11.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.182+0000 7f5e36a00700 1 -- 172.21.15.63:0/3190476444 learned_addr learned my addr 172.21.15.63:0/3190476444 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e37400700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e38070360 0x7f5e38115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e36a00700 1 -- 172.21.15.63:0/3190476444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e38070360 msgr2=0x7f5e38115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e36a00700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e38070360 0x7f5e38115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e36a00700 1 -- 172.21.15.63:0/3190476444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e28005040 con 0x7f5e38070c70 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e37400700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e38070360 0x7f5e38115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e36a00700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070c70 0x7f5e381161c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400eb40 tx=0x7f5e2400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e2f400700 1 -- 172.21.15.63:0/3190476444 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e24007cc0 con 0x7f5e38070c70 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e2f400700 1 -- 172.21.15.63:0/3190476444 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e2400f040 con 0x7f5e38070c70 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3810fe60 con 0x7f5e38070c70 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e2f400700 1 -- 172.21.15.63:0/3190476444 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e24009570 con 0x7f5e38070c70 2024-03-23T12:03:11.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.183+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e381103a0 con 0x7f5e38070c70 2024-03-23T12:03:11.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.185+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e38103b90 con 0x7f5e38070c70 2024-03-23T12:03:11.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.185+0000 7f5e2f400700 1 -- 172.21.15.63:0/3190476444 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5e240096d0 con 0x7f5e38070c70 2024-03-23T12:03:11.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.186+0000 7f5e2f400700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e1c06c7d0 0x7f5e1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:11.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.186+0000 7f5e2f400700 1 -- 172.21.15.63:0/3190476444 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e2408b670 con 0x7f5e38070c70 2024-03-23T12:03:11.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.187+0000 7f5e37400700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e1c06c7d0 0x7f5e1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:11.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.187+0000 7f5e37400700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e1c06c7d0 0x7f5e1c06ec80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5e28004e30 tx=0x7f5e28009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:11.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.191+0000 7f5e2f400700 1 -- 172.21.15.63:0/3190476444 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5e24059430 con 0x7f5e38070c70 2024-03-23T12:03:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:10 smithi063 ceph-mon[141124]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:10 smithi063 ceph-mon[141124]: from='client.34718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:10 smithi063 ceph-mon[141124]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:10 smithi063 ceph-mon[141124]: pgmap v361: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:10 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1088042803' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:03:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:10 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2101869799' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:03:11.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.488+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5e38071aa0 con 0x7f5e38070c70 2024-03-23T12:03:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.489+0000 7f5e2f400700 1 -- 172.21.15.63:0/3190476444 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5e24058fc0 con 0x7f5e38070c70 2024-03-23T12:03:11.471 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:03:11.471 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:03:11.471 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:03:11.472 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:03:11.472 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:03:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e1c06c7d0 msgr2=0x7f5e1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e1c06c7d0 0x7f5e1c06ec80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5e28004e30 tx=0x7f5e28009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070c70 msgr2=0x7f5e381161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070c70 0x7f5e381161c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400eb40 tx=0x7f5e2400b040 comp rx=0 tx=0).stop 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 shutdown_connections 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e1c06c7d0 0x7f5e1c06ec80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e38070360 0x7f5e38115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 --2- 172.21.15.63:0/3190476444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38070c70 0x7f5e381161c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.491+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 >> 172.21.15.63:0/3190476444 conn(0x7f5e380f89b0 msgr2=0x7f5e381015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.492+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 shutdown_connections 2024-03-23T12:03:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:11.492+0000 7f5e3e200700 1 -- 172.21.15.63:0/3190476444 wait complete. 2024-03-23T12:03:12.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:11 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3190476444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:03:12.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:11 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3190476444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:03:13.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:12 smithi063 ceph-mon[141124]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:13.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:12 smithi063 ceph-mon[141124]: pgmap v362: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:13.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:12 smithi111 ceph-mon[106899]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:13.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:12 smithi111 ceph-mon[106899]: pgmap v362: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:15.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:14 smithi111 ceph-mon[106899]: pgmap v363: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:15.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:14 smithi063 ceph-mon[141124]: pgmap v363: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:16.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:16 smithi063 ceph-mon[141124]: pgmap v364: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:16.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:16 smithi111 ceph-mon[106899]: pgmap v364: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:19.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:18 smithi111 ceph-mon[106899]: pgmap v365: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:19.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:18 smithi063 ceph-mon[141124]: pgmap v365: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:21.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:20 smithi111 ceph-mon[106899]: pgmap v366: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:21.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:20 smithi063 ceph-mon[141124]: pgmap v366: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:23.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:22 smithi063 ceph-mon[141124]: pgmap v367: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:23.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:22 smithi111 ceph-mon[106899]: pgmap v367: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:25.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:24 smithi111 ceph-mon[106899]: pgmap v368: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:25.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:24 smithi063 ceph-mon[141124]: pgmap v368: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:26.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:26.881 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:26 smithi063 ceph-mon[141124]: pgmap v369: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:26 smithi111 ceph-mon[106899]: pgmap v369: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:29.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:28 smithi111 ceph-mon[106899]: pgmap v370: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:29.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:28 smithi063 ceph-mon[141124]: pgmap v370: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:31.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:30 smithi111 ceph-mon[106899]: pgmap v371: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:31.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:30 smithi063 ceph-mon[141124]: pgmap v371: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:33.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:32 smithi063 ceph-mon[141124]: pgmap v372: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:33.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:32 smithi111 ceph-mon[106899]: pgmap v372: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:35.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:34 smithi063 ceph-mon[141124]: pgmap v373: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:35.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:34 smithi111 ceph-mon[106899]: pgmap v373: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:36.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:36 smithi063 ceph-mon[141124]: pgmap v374: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:36.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:36 smithi111 ceph-mon[106899]: pgmap v374: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:39.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:38 smithi111 ceph-mon[106899]: pgmap v375: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:39.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:38 smithi063 ceph-mon[141124]: pgmap v375: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:41.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:40 smithi111 ceph-mon[106899]: pgmap v376: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:41.151 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:40 smithi063 ceph-mon[141124]: pgmap v376: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:41.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 -- 172.21.15.63:0/2516811871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 msgr2=0x7efddc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:41.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 --2- 172.21.15.63:0/2516811871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc0fce10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efdc8004940 tx=0x7efdc8009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:41.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 -- 172.21.15.63:0/2516811871 shutdown_connections 2024-03-23T12:03:41.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 --2- 172.21.15.63:0/2516811871 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc0fce10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:41.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 --2- 172.21.15.63:0/2516811871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efddc102e00 0x7efddc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:41.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 -- 172.21.15.63:0/2516811871 >> 172.21.15.63:0/2516811871 conn(0x7efddc0f89b0 msgr2=0x7efddc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:41.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 -- 172.21.15.63:0/2516811871 shutdown_connections 2024-03-23T12:03:41.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.646+0000 7efde3400700 1 -- 172.21.15.63:0/2516811871 wait complete. 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.647+0000 7efde3400700 1 Processor -- start 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.647+0000 7efde3400700 1 -- start start 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efde3400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efddc102e00 0x7efddc115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efde3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efde3400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efddc116890 con 0x7efddc1037a0 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efde3400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efddc10fc60 con 0x7efddc102e00 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efddbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:41.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efddbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35236/0 (socket says 172.21.15.63:35236) 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efddbe00700 1 -- 172.21.15.63:0/3010776701 learned_addr learned my addr 172.21.15.63:0/3010776701 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.648+0000 7efde0a00700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efddc102e00 0x7efddc115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efddbe00700 1 -- 172.21.15.63:0/3010776701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efddc102e00 msgr2=0x7efddc115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efddbe00700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efddc102e00 0x7efddc115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efddbe00700 1 -- 172.21.15.63:0/3010776701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8005040 con 0x7efddc1037a0 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efddbe00700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc116240 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009390 tx=0x7efdc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efdd8c00700 1 -- 172.21.15.63:0/3010776701 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdc800e040 con 0x7efddc1037a0 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efdd8c00700 1 -- 172.21.15.63:0/3010776701 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdc8013420 con 0x7efddc1037a0 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efdd8c00700 1 -- 172.21.15.63:0/3010776701 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdc801c630 con 0x7efddc1037a0 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efddc10fee0 con 0x7efddc1037a0 2024-03-23T12:03:41.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.649+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efddc1104a0 con 0x7efddc1037a0 2024-03-23T12:03:41.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.650+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efddc0fe550 con 0x7efddc1037a0 2024-03-23T12:03:41.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.651+0000 7efdd8c00700 1 -- 172.21.15.63:0/3010776701 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7efdc8013a00 con 0x7efddc1037a0 2024-03-23T12:03:41.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.652+0000 7efdd8c00700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdc406c7d0 0x7efdc406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:41.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.652+0000 7efdd8c00700 1 -- 172.21.15.63:0/3010776701 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efdc808cd10 con 0x7efddc1037a0 2024-03-23T12:03:41.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.653+0000 7efde0a00700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdc406c7d0 0x7efdc406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:41.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.653+0000 7efde0a00700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdc406c7d0 0x7efdc406ec80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efdd0004990 tx=0x7efdd0009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:41.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.657+0000 7efdd8c00700 1 -- 172.21.15.63:0/3010776701 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efdc805aa20 con 0x7efddc1037a0 2024-03-23T12:03:41.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.892+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efddc109e10 con 0x7efdc406c7d0 2024-03-23T12:03:41.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.893+0000 7efdd8c00700 1 -- 172.21.15.63:0/3010776701 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efddc109e10 con 0x7efdc406c7d0 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdc406c7d0 msgr2=0x7efdc406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdc406c7d0 0x7efdc406ec80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efdd0004990 tx=0x7efdd0009ee0 comp rx=0 tx=0).stop 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 msgr2=0x7efddc116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc116240 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009390 tx=0x7efdc8009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 shutdown_connections 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdc406c7d0 0x7efdc406ec80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efddc102e00 0x7efddc115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 --2- 172.21.15.63:0/3010776701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efddc1037a0 0x7efddc116240 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 >> 172.21.15.63:0/3010776701 conn(0x7efddc0f89b0 msgr2=0x7efddc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 shutdown_connections 2024-03-23T12:03:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:41.896+0000 7efde3400700 1 -- 172.21.15.63:0/3010776701 wait complete. 2024-03-23T12:03:41.891 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 -- 172.21.15.63:0/88572896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30102c40 msgr2=0x7f8d30103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 --2- 172.21.15.63:0/88572896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30102c40 0x7f8d30103010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8d20004940 tx=0x7f8d20009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 -- 172.21.15.63:0/88572896 shutdown_connections 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 --2- 172.21.15.63:0/88572896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d301035e0 0x7f8d300fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 --2- 172.21.15.63:0/88572896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30102c40 0x7f8d30103010 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 -- 172.21.15.63:0/88572896 >> 172.21.15.63:0/88572896 conn(0x7f8d300f8830 msgr2=0x7f8d300fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 -- 172.21.15.63:0/88572896 shutdown_connections 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.034+0000 7f8d2f200700 1 -- 172.21.15.63:0/88572896 wait complete. 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.035+0000 7f8d2f200700 1 Processor -- start 2024-03-23T12:03:42.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.035+0000 7f8d2f200700 1 -- start start 2024-03-23T12:03:42.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.035+0000 7f8d2f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d30102c40 0x7f8d30115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.035+0000 7f8d2f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d301035e0 0x7f8d301160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d30116710 con 0x7f8d301035e0 2024-03-23T12:03:42.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3010fb20 con 0x7f8d30102c40 2024-03-23T12:03:42.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d301035e0 0x7f8d301160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d301035e0 0x7f8d301160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35252/0 (socket says 172.21.15.63:35252) 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2d400700 1 -- 172.21.15.63:0/185102870 learned_addr learned my addr 172.21.15.63:0/185102870 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2de00700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d30102c40 0x7f8d30115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2d400700 1 -- 172.21.15.63:0/185102870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d30102c40 msgr2=0x7f8d30115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2d400700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d30102c40 0x7f8d30115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2d400700 1 -- 172.21.15.63:0/185102870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d20005040 con 0x7f8d301035e0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.036+0000 7f8d2de00700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d30102c40 0x7f8d30115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.037+0000 7f8d2d400700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d301035e0 0x7f8d301160c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c002a50 tx=0x7f8d1c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.037+0000 7f8d26000700 1 -- 172.21.15.63:0/185102870 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1c007cc0 con 0x7f8d301035e0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.037+0000 7f8d26000700 1 -- 172.21.15.63:0/185102870 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d1c00f040 con 0x7f8d301035e0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.037+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3010fe00 con 0x7f8d301035e0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.037+0000 7f8d26000700 1 -- 172.21.15.63:0/185102870 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1c00d9a0 con 0x7f8d301035e0 2024-03-23T12:03:42.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.037+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d30110340 con 0x7f8d301035e0 2024-03-23T12:03:42.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.038+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d300fe390 con 0x7f8d301035e0 2024-03-23T12:03:42.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.039+0000 7f8d26000700 1 -- 172.21.15.63:0/185102870 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8d1c00dc60 con 0x7f8d301035e0 2024-03-23T12:03:42.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.040+0000 7f8d26000700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d1406c7d0 0x7f8d1406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.040+0000 7f8d26000700 1 -- 172.21.15.63:0/185102870 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8d1c08b930 con 0x7f8d301035e0 2024-03-23T12:03:42.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.040+0000 7f8d2de00700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d1406c7d0 0x7f8d1406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.041+0000 7f8d2de00700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d1406c7d0 0x7f8d1406ec80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000b3b0 tx=0x7f8d20009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:42.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.045+0000 7f8d26000700 1 -- 172.21.15.63:0/185102870 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8d1c055ab0 con 0x7f8d301035e0 2024-03-23T12:03:42.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.278+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d30109bf0 con 0x7f8d1406c7d0 2024-03-23T12:03:42.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.280+0000 7f8d26000700 1 -- 172.21.15.63:0/185102870 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8d30109bf0 con 0x7f8d1406c7d0 2024-03-23T12:03:42.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d1406c7d0 msgr2=0x7f8d1406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d1406c7d0 0x7f8d1406ec80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000b3b0 tx=0x7f8d20009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:42.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d301035e0 msgr2=0x7f8d301160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d301035e0 0x7f8d301160c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c002a50 tx=0x7f8d1c004600 comp rx=0 tx=0).stop 2024-03-23T12:03:42.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 shutdown_connections 2024-03-23T12:03:42.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d1406c7d0 0x7f8d1406ec80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d30102c40 0x7f8d30115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 --2- 172.21.15.63:0/185102870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d301035e0 0x7f8d301160c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.282+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 >> 172.21.15.63:0/185102870 conn(0x7f8d300f8830 msgr2=0x7f8d300f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:42.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.283+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 shutdown_connections 2024-03-23T12:03:42.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.283+0000 7f8d2f200700 1 -- 172.21.15.63:0/185102870 wait complete. 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.419+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2783931220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 msgr2=0x7f2b38106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.419+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2783931220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b38106f00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2b24004940 tx=0x7f2b24009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.419+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2783931220 shutdown_connections 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.419+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2783931220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b38106f00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.419+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2783931220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b380fde90 0x7f2b380fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.419+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2783931220 >> 172.21.15.63:0/2783931220 conn(0x7f2b38072bc0 msgr2=0x7f2b38072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.420+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2783931220 shutdown_connections 2024-03-23T12:03:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.420+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2783931220 wait complete. 2024-03-23T12:03:42.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.420+0000 7f2b3ca00700 1 Processor -- start 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.420+0000 7f2b3ca00700 1 -- start start 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b3ca00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b380fde90 0x7f2b380fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b3ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b380fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b3ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b380fc820 con 0x7f2b380fe830 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b3ca00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b380fc960 con 0x7f2b380fde90 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b35400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b380fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b35400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b380fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35278/0 (socket says 172.21.15.63:35278) 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b35400700 1 -- 172.21.15.63:0/2507562276 learned_addr learned my addr 172.21.15.63:0/2507562276 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.421+0000 7f2b35e00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b380fde90 0x7f2b380fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b35400700 1 -- 172.21.15.63:0/2507562276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b380fde90 msgr2=0x7f2b380fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b35400700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b380fde90 0x7f2b380fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b35400700 1 -- 172.21.15.63:0/2507562276 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24005040 con 0x7f2b380fe830 2024-03-23T12:03:42.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b35e00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b380fde90 0x7f2b380fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:03:42.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b35400700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b380fc290 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b240093b0 tx=0x7f2b24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:42.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b2e000700 1 -- 172.21.15.63:0/2507562276 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b2400e040 con 0x7f2b380fe830 2024-03-23T12:03:42.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b2e000700 1 -- 172.21.15.63:0/2507562276 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b24004290 con 0x7f2b380fe830 2024-03-23T12:03:42.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b380fcbb0 con 0x7f2b380fe830 2024-03-23T12:03:42.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b2e000700 1 -- 172.21.15.63:0/2507562276 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b2401c3f0 con 0x7f2b380fe830 2024-03-23T12:03:42.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.422+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3811a190 con 0x7f2b380fe830 2024-03-23T12:03:42.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.423+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b38065e30 con 0x7f2b380fe830 2024-03-23T12:03:42.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.424+0000 7f2b2e000700 1 -- 172.21.15.63:0/2507562276 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2b24013790 con 0x7f2b380fe830 2024-03-23T12:03:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.425+0000 7f2b2e000700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2b1c06c7d0 0x7f2b1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.425+0000 7f2b2e000700 1 -- 172.21.15.63:0/2507562276 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2b2408ccb0 con 0x7f2b380fe830 2024-03-23T12:03:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.425+0000 7f2b35e00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2b1c06c7d0 0x7f2b1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.426+0000 7f2b35e00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2b1c06c7d0 0x7f2b1c06ec80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007fd0 tx=0x7f2b28009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:42.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.430+0000 7f2b2e000700 1 -- 172.21.15.63:0/2507562276 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2b24056f30 con 0x7f2b380fe830 2024-03-23T12:03:42.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.653+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b38065e30 con 0x7f2b1c06c7d0 2024-03-23T12:03:42.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.663+0000 7f2b2e000700 1 -- 172.21.15.63:0/2507562276 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f2b38065e30 con 0x7f2b1c06c7d0 2024-03-23T12:03:42.644 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:03:42.644 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (13m) 5m ago 26m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:03:42.644 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (10m) 5m ago 26m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:03:42.644 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (10m) 5m ago 25m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (12m) 5m ago 25m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (6m) 5m ago 19m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (5m) 5m ago 19m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (19m) 5m ago 19m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (5m) 5m ago 19m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (15m) 5m ago 27m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (14m) 5m ago 25m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (11m) 5m ago 27m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (10m) 5m ago 24m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (13m) 5m ago 26m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (13m) 5m ago 25m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (9m) 5m ago 24m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (9m) 5m ago 23m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (8m) 5m ago 22m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (7m) 5m ago 22m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:03:42.645 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (7m) 5m ago 21m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:03:42.646 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (6m) 5m ago 21m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:03:42.646 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (13m) 5m ago 25m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2b1c06c7d0 msgr2=0x7f2b1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2b1c06c7d0 0x7f2b1c06ec80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007fd0 tx=0x7f2b28009040 comp rx=0 tx=0).stop 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 msgr2=0x7f2b380fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b380fc290 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b240093b0 tx=0x7f2b24009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 shutdown_connections 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2b1c06c7d0 0x7f2b1c06ec80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b380fde90 0x7f2b380fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 --2- 172.21.15.63:0/2507562276 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b380fe830 0x7f2b380fc290 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 >> 172.21.15.63:0/2507562276 conn(0x7f2b38072bc0 msgr2=0x7f2b38103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 shutdown_connections 2024-03-23T12:03:42.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.666+0000 7f2b3ca00700 1 -- 172.21.15.63:0/2507562276 wait complete. 2024-03-23T12:03:42.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 -- 172.21.15.63:0/1946743838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 msgr2=0x7fe6681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 --2- 172.21.15.63:0/1946743838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe6681031d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe65c004940 tx=0x7fe65c009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:42.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 -- 172.21.15.63:0/1946743838 shutdown_connections 2024-03-23T12:03:42.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 --2- 172.21.15.63:0/1946743838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6681037a0 0x7fe6680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 --2- 172.21.15.63:0/1946743838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe6681031d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 -- 172.21.15.63:0/1946743838 >> 172.21.15.63:0/1946743838 conn(0x7fe6680f89b0 msgr2=0x7fe6680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:42.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 -- 172.21.15.63:0/1946743838 shutdown_connections 2024-03-23T12:03:42.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.803+0000 7fe670200700 1 -- 172.21.15.63:0/1946743838 wait complete. 2024-03-23T12:03:42.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.804+0000 7fe670200700 1 Processor -- start 2024-03-23T12:03:42.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.804+0000 7fe670200700 1 -- start start 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe670200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe668115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe670200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6681037a0 0x7fe668116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe670200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6681168a0 con 0x7fe668102e00 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe670200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810fc70 con 0x7fe6681037a0 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe66d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe668115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe66d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe668115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35290/0 (socket says 172.21.15.63:35290) 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe66d800700 1 -- 172.21.15.63:0/2978844042 learned_addr learned my addr 172.21.15.63:0/2978844042 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe66ce00700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6681037a0 0x7fe668116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe66d800700 1 -- 172.21.15.63:0/2978844042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6681037a0 msgr2=0x7fe668116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe66d800700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6681037a0 0x7fe668116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.805+0000 7fe66d800700 1 -- 172.21.15.63:0/2978844042 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c005040 con 0x7fe668102e00 2024-03-23T12:03:42.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.806+0000 7fe66d800700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe668115d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009180 tx=0x7fe65c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:42.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.806+0000 7fe665600700 1 -- 172.21.15.63:0/2978844042 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65c00e040 con 0x7fe668102e00 2024-03-23T12:03:42.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.806+0000 7fe665600700 1 -- 172.21.15.63:0/2978844042 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe65c004290 con 0x7fe668102e00 2024-03-23T12:03:42.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.806+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66810fef0 con 0x7fe668102e00 2024-03-23T12:03:42.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.806+0000 7fe665600700 1 -- 172.21.15.63:0/2978844042 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65c01c3f0 con 0x7fe668102e00 2024-03-23T12:03:42.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.806+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6681104b0 con 0x7fe668102e00 2024-03-23T12:03:42.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.807+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6680fe550 con 0x7fe668102e00 2024-03-23T12:03:42.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.808+0000 7fe665600700 1 -- 172.21.15.63:0/2978844042 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe65c013790 con 0x7fe668102e00 2024-03-23T12:03:42.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.809+0000 7fe665600700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe65006c7c0 0x7fe65006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:42.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.809+0000 7fe665600700 1 -- 172.21.15.63:0/2978844042 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe65c08cc30 con 0x7fe668102e00 2024-03-23T12:03:42.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.809+0000 7fe66ce00700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe65006c7c0 0x7fe65006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:42.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.809+0000 7fe66ce00700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe65006c7c0 0x7fe65006ec70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6681115b0 tx=0x7fe654009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:42.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:42.813+0000 7fe665600700 1 -- 172.21.15.63:0/2978844042 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe65c05aa00 con 0x7fe668102e00 2024-03-23T12:03:43.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.114+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe66804bed0 con 0x7fe668102e00 2024-03-23T12:03:43.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.115+0000 7fe665600700 1 -- 172.21.15.63:0/2978844042 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fe65c013420 con 0x7fe668102e00 2024-03-23T12:03:43.096 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:03:43.096 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:03:43.096 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:03:43.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:43.096 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:03:43.096 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:03:43.097 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.117+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe65006c7c0 msgr2=0x7fe65006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.117+0000 7fe670200700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe65006c7c0 0x7fe65006ec70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6681115b0 tx=0x7fe654009ee0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.117+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 msgr2=0x7fe668115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.117+0000 7fe670200700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe668115d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009180 tx=0x7fe65c009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.118+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 shutdown_connections 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.118+0000 7fe670200700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe65006c7c0 0x7fe65006ec70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.118+0000 7fe670200700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe668102e00 0x7fe668115d10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.118+0000 7fe670200700 1 --2- 172.21.15.63:0/2978844042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6681037a0 0x7fe668116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.118+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 >> 172.21.15.63:0/2978844042 conn(0x7fe6680f89b0 msgr2=0x7fe6680704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.118+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 shutdown_connections 2024-03-23T12:03:43.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.118+0000 7fe670200700 1 -- 172.21.15.63:0/2978844042 wait complete. 2024-03-23T12:03:43.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:42 smithi063 ceph-mon[141124]: pgmap v377: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:43.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:42 smithi063 ceph-mon[141124]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:43.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:42 smithi111 ceph-mon[106899]: pgmap v377: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:43.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:42 smithi111 ceph-mon[106899]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 -- 172.21.15.63:0/24728749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc1037a0 msgr2=0x7f93cc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 --2- 172.21.15.63:0/24728749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc1037a0 0x7f93cc0fce10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f93b8004940 tx=0x7f93b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 -- 172.21.15.63:0/24728749 shutdown_connections 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 --2- 172.21.15.63:0/24728749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc1037a0 0x7f93cc0fce10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 --2- 172.21.15.63:0/24728749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc102e00 0x7f93cc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 -- 172.21.15.63:0/24728749 >> 172.21.15.63:0/24728749 conn(0x7f93cc0f89b0 msgr2=0x7f93cc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 -- 172.21.15.63:0/24728749 shutdown_connections 2024-03-23T12:03:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.255+0000 7f93d0e00700 1 -- 172.21.15.63:0/24728749 wait complete. 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93d0e00700 1 Processor -- start 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93d0e00700 1 -- start start 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93d0e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93d0e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93d0e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc116880 con 0x7f93cc102e00 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93d0e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc10fc50 con 0x7f93cc1037a0 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93c9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93c9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35314/0 (socket says 172.21.15.63:35314) 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.256+0000 7f93c9e00700 1 -- 172.21.15.63:0/3960849909 learned_addr learned my addr 172.21.15.63:0/3960849909 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c9400700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c9e00700 1 -- 172.21.15.63:0/3960849909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 msgr2=0x7f93cc116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c9e00700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c9e00700 1 -- 172.21.15.63:0/3960849909 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b8005040 con 0x7f93cc102e00 2024-03-23T12:03:43.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c9e00700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f93bc002a50 tx=0x7f93bc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:43.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c2000700 1 -- 172.21.15.63:0/3960849909 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93bc007cc0 con 0x7f93cc102e00 2024-03-23T12:03:43.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c2000700 1 -- 172.21.15.63:0/3960849909 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93bc00f040 con 0x7f93cc102e00 2024-03-23T12:03:43.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc10ff30 con 0x7f93cc102e00 2024-03-23T12:03:43.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.257+0000 7f93c2000700 1 -- 172.21.15.63:0/3960849909 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93bc00d9c0 con 0x7f93cc102e00 2024-03-23T12:03:43.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.258+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc1104f0 con 0x7f93cc102e00 2024-03-23T12:03:43.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.259+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc0fe550 con 0x7f93cc102e00 2024-03-23T12:03:43.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.259+0000 7f93c2000700 1 -- 172.21.15.63:0/3960849909 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f93bc00db20 con 0x7f93cc102e00 2024-03-23T12:03:43.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.261+0000 7f93c2000700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:43.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.261+0000 7f93c2000700 1 -- 172.21.15.63:0/3960849909 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f93bc08b6e0 con 0x7f93cc102e00 2024-03-23T12:03:43.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.261+0000 7f93c9400700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:43.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.262+0000 7f93c9400700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93cc111590 tx=0x7f93b8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.265+0000 7f93c2000700 1 -- 172.21.15.63:0/3960849909 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f93bc0593f0 con 0x7f93cc102e00 2024-03-23T12:03:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.525+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f93cc04bed0 con 0x7f93cc102e00 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.526+0000 7f93c2000700 1 -- 172.21.15.63:0/3960849909 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f93bc058f80 con 0x7f93cc102e00 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:03:43.507 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:03:43.508 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:03:43.509 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:43.510 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:43.510 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:43.510 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:03:43.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 msgr2=0x7f93b006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93cc111590 tx=0x7f93b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 msgr2=0x7f93cc115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f93bc002a50 tx=0x7f93bc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 shutdown_connections 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 --2- 172.21.15.63:0/3960849909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 >> 172.21.15.63:0/3960849909 conn(0x7f93cc0f89b0 msgr2=0x7f93cc0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 shutdown_connections 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.528+0000 7f93d0e00700 1 -- 172.21.15.63:0/3960849909 wait complete. 2024-03-23T12:03:43.511 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:03:43.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 -- 172.21.15.63:0/1714161539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 msgr2=0x7faae40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 --2- 172.21.15.63:0/1714161539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae40fce10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faad0004940 tx=0x7faad0009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:43.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 -- 172.21.15.63:0/1714161539 shutdown_connections 2024-03-23T12:03:43.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 --2- 172.21.15.63:0/1714161539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae40fce10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 --2- 172.21.15.63:0/1714161539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae4102e00 0x7faae41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 -- 172.21.15.63:0/1714161539 >> 172.21.15.63:0/1714161539 conn(0x7faae40f89b0 msgr2=0x7faae40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:43.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 -- 172.21.15.63:0/1714161539 shutdown_connections 2024-03-23T12:03:43.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.667+0000 7faaeb400700 1 -- 172.21.15.63:0/1714161539 wait complete. 2024-03-23T12:03:43.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.668+0000 7faaeb400700 1 Processor -- start 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.668+0000 7faaeb400700 1 -- start start 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.668+0000 7faaeb400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae4102e00 0x7faae4115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.668+0000 7faaeb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae4116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.668+0000 7faaeb400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41168b0 con 0x7faae41037a0 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.668+0000 7faaeb400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae410fc80 con 0x7faae4102e00 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae4116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae4116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35326/0 (socket says 172.21.15.63:35326) 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae3e00700 1 -- 172.21.15.63:0/3319339478 learned_addr learned my addr 172.21.15.63:0/3319339478 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae8a00700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae4102e00 0x7faae4115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae3e00700 1 -- 172.21.15.63:0/3319339478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae4102e00 msgr2=0x7faae4115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae3e00700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae4102e00 0x7faae4115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae3e00700 1 -- 172.21.15.63:0/3319339478 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad0005040 con 0x7faae41037a0 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae8a00700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae4102e00 0x7faae4115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:03:43.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.669+0000 7faae3e00700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae4116260 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7faad0009390 tx=0x7faad0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:43.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.670+0000 7faae0c00700 1 -- 172.21.15.63:0/3319339478 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faad000e040 con 0x7faae41037a0 2024-03-23T12:03:43.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.670+0000 7faae0c00700 1 -- 172.21.15.63:0/3319339478 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faad0013420 con 0x7faae41037a0 2024-03-23T12:03:43.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.670+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae410ff00 con 0x7faae41037a0 2024-03-23T12:03:43.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.670+0000 7faae0c00700 1 -- 172.21.15.63:0/3319339478 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faad001c630 con 0x7faae41037a0 2024-03-23T12:03:43.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.670+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae41104c0 con 0x7faae41037a0 2024-03-23T12:03:43.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.671+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae40fe550 con 0x7faae41037a0 2024-03-23T12:03:43.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.672+0000 7faae0c00700 1 -- 172.21.15.63:0/3319339478 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7faad0013a00 con 0x7faae41037a0 2024-03-23T12:03:43.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.673+0000 7faae0c00700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faacc06c7d0 0x7faacc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:43.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.673+0000 7faae0c00700 1 -- 172.21.15.63:0/3319339478 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faad008cd10 con 0x7faae41037a0 2024-03-23T12:03:43.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.673+0000 7faae8a00700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faacc06c7d0 0x7faacc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:43.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.674+0000 7faae8a00700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faacc06c7d0 0x7faacc06ec80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faad8009fd0 tx=0x7faad800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:43.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.677+0000 7faae0c00700 1 -- 172.21.15.63:0/3319339478 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faad005aa20 con 0x7faae41037a0 2024-03-23T12:03:43.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.910+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faae4109e30 con 0x7faacc06c7d0 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.911+0000 7faae0c00700 1 -- 172.21.15.63:0/3319339478 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faae4109e30 con 0x7faacc06c7d0 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:03:43.892 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:03:43.893 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:03:43.893 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:03:43.893 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:03:43.893 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:03:43.893 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:03:43.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faacc06c7d0 msgr2=0x7faacc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faacc06c7d0 0x7faacc06ec80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faad8009fd0 tx=0x7faad800b040 comp rx=0 tx=0).stop 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 msgr2=0x7faae4116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae4116260 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7faad0009390 tx=0x7faad0009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 shutdown_connections 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faacc06c7d0 0x7faacc06ec80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae4102e00 0x7faae4115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 --2- 172.21.15.63:0/3319339478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faae41037a0 0x7faae4116260 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.913+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 >> 172.21.15.63:0/3319339478 conn(0x7faae40f89b0 msgr2=0x7faae4070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.914+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 shutdown_connections 2024-03-23T12:03:43.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:43.914+0000 7faaeb400700 1 -- 172.21.15.63:0/3319339478 wait complete. 2024-03-23T12:03:44.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.052+0000 7f9db9e00700 1 -- 172.21.15.63:0/390645935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db41037a0 msgr2=0x7f9db40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:44.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.052+0000 7f9db9e00700 1 --2- 172.21.15.63:0/390645935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db41037a0 0x7f9db40fce10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9da0004940 tx=0x7f9da0009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:44.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.052+0000 7f9db9e00700 1 -- 172.21.15.63:0/390645935 shutdown_connections 2024-03-23T12:03:44.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.052+0000 7f9db9e00700 1 --2- 172.21.15.63:0/390645935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db41037a0 0x7f9db40fce10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:44.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.052+0000 7f9db9e00700 1 --2- 172.21.15.63:0/390645935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9db4102e00 0x7f9db41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:44.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.052+0000 7f9db9e00700 1 -- 172.21.15.63:0/390645935 >> 172.21.15.63:0/390645935 conn(0x7f9db40f89b0 msgr2=0x7f9db40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:44.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.053+0000 7f9db9e00700 1 -- 172.21.15.63:0/390645935 shutdown_connections 2024-03-23T12:03:44.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.053+0000 7f9db9e00700 1 -- 172.21.15.63:0/390645935 wait complete. 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.053+0000 7f9db9e00700 1 Processor -- start 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db9e00700 1 -- start start 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db4102e00 0x7f9db4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db9e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9db41037a0 0x7f9db41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db9e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db4116810 con 0x7f9db4102e00 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db9e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db410fbe0 con 0x7f9db41037a0 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db4102e00 0x7f9db4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db4102e00 0x7f9db4115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35346/0 (socket says 172.21.15.63:35346) 2024-03-23T12:03:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.054+0000 7f9db2a00700 1 -- 172.21.15.63:0/99263305 learned_addr learned my addr 172.21.15.63:0/99263305 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:03:44.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db2000700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9db41037a0 0x7f9db41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db2a00700 1 -- 172.21.15.63:0/99263305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9db41037a0 msgr2=0x7f9db41161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db2a00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9db41037a0 0x7f9db41161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db2a00700 1 -- 172.21.15.63:0/99263305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da0005040 con 0x7f9db4102e00 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db2000700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9db41037a0 0x7f9db41161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db2a00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db4102e00 0x7f9db4115c80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9da400eb40 tx=0x7f9da400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9daaa00700 1 -- 172.21.15.63:0/99263305 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9da4007cc0 con 0x7f9db4102e00 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9daaa00700 1 -- 172.21.15.63:0/99263305 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9da400f040 con 0x7f9db4102e00 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db410fe60 con 0x7f9db4102e00 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9daaa00700 1 -- 172.21.15.63:0/99263305 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9da4009570 con 0x7f9db4102e00 2024-03-23T12:03:44.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.055+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db4110420 con 0x7f9db4102e00 2024-03-23T12:03:44.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.056+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db40fe550 con 0x7f9db4102e00 2024-03-23T12:03:44.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.058+0000 7f9daaa00700 1 -- 172.21.15.63:0/99263305 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9da40096d0 con 0x7f9db4102e00 2024-03-23T12:03:44.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.059+0000 7f9daaa00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9d9806c7d0 0x7f9d9806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:03:44.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.059+0000 7f9daaa00700 1 -- 172.21.15.63:0/99263305 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9da408b670 con 0x7f9db4102e00 2024-03-23T12:03:44.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.059+0000 7f9db2000700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9d9806c7d0 0x7f9d9806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:03:44.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.059+0000 7f9db2000700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9d9806c7d0 0x7f9d9806ec80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9da0000c00 tx=0x7f9da0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:03:44.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.063+0000 7f9daaa00700 1 -- 172.21.15.63:0/99263305 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9da4059430 con 0x7f9db4102e00 2024-03-23T12:03:44.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:43 smithi111 ceph-mon[106899]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:44.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:43 smithi111 ceph-mon[106899]: from='client.34750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:44.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:43 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2978844042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:03:44.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:43 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3960849909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:03:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.364+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9db404bed0 con 0x7f9db4102e00 2024-03-23T12:03:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.364+0000 7f9daaa00700 1 -- 172.21.15.63:0/99263305 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f9da4058fc0 con 0x7f9db4102e00 2024-03-23T12:03:44.345 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:03:44.345 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:03:44.346 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:03:44.346 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:03:44.346 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:03:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.366+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9d9806c7d0 msgr2=0x7f9d9806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9d9806c7d0 0x7f9d9806ec80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9da0000c00 tx=0x7f9da0009e30 comp rx=0 tx=0).stop 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db4102e00 msgr2=0x7f9db4115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db4102e00 0x7f9db4115c80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9da400eb40 tx=0x7f9da400b040 comp rx=0 tx=0).stop 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 shutdown_connections 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9d9806c7d0 0x7f9d9806ec80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9db4102e00 0x7f9db4115c80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 --2- 172.21.15.63:0/99263305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9db41037a0 0x7f9db41161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 >> 172.21.15.63:0/99263305 conn(0x7f9db40f89b0 msgr2=0x7f9db40f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 shutdown_connections 2024-03-23T12:03:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:03:44.367+0000 7f9db9e00700 1 -- 172.21.15.63:0/99263305 wait complete. 2024-03-23T12:03:44.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:43 smithi063 ceph-mon[141124]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:44.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:43 smithi063 ceph-mon[141124]: from='client.34750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:44.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:43 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2978844042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:03:44.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:43 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3960849909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:03:45.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:44 smithi063 ceph-mon[141124]: pgmap v378: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:45.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:44 smithi063 ceph-mon[141124]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:45.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:44 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/99263305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:03:45.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:44 smithi111 ceph-mon[106899]: pgmap v378: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:45.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:44 smithi111 ceph-mon[106899]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:03:45.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:44 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/99263305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:03:46.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:46 smithi063 ceph-mon[141124]: pgmap v379: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:46.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:46 smithi111 ceph-mon[106899]: pgmap v379: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:49.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:48 smithi111 ceph-mon[106899]: pgmap v380: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:49.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:48 smithi063 ceph-mon[141124]: pgmap v380: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:51.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:50 smithi063 ceph-mon[141124]: pgmap v381: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:51.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:50 smithi111 ceph-mon[106899]: pgmap v381: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:53.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:52 smithi063 ceph-mon[141124]: pgmap v382: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:53.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:52 smithi111 ceph-mon[106899]: pgmap v382: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:55.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:54 smithi063 ceph-mon[141124]: pgmap v383: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:54 smithi111 ceph-mon[106899]: pgmap v383: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:56 smithi063 ceph-mon[141124]: pgmap v384: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:03:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:56 smithi111 ceph-mon[106899]: pgmap v384: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:59.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:03:58 smithi111 ceph-mon[106899]: pgmap v385: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:03:59.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:03:58 smithi063 ceph-mon[141124]: pgmap v385: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:00 smithi063 ceph-mon[141124]: pgmap v386: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:01.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:00 smithi111 ceph-mon[106899]: pgmap v386: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:03.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:02 smithi063 ceph-mon[141124]: pgmap v387: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:03.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:02 smithi111 ceph-mon[106899]: pgmap v387: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:05.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:04 smithi063 ceph-mon[141124]: pgmap v388: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:05.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:04 smithi111 ceph-mon[106899]: pgmap v388: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:06.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:06 smithi063 ceph-mon[141124]: pgmap v389: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:06.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:06 smithi111 ceph-mon[106899]: pgmap v389: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:09.150 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:08 smithi111 ceph-mon[106899]: pgmap v390: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:09.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:08 smithi063 ceph-mon[141124]: pgmap v390: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:10 smithi063 ceph-mon[141124]: pgmap v391: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:11.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:10 smithi111 ceph-mon[106899]: pgmap v391: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:11.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:13.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:12 smithi063 ceph-mon[141124]: pgmap v392: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:13.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:12 smithi111 ceph-mon[106899]: pgmap v392: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:14.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 -- 172.21.15.63:0/1770325066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 msgr2=0x7fd6841031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:14.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 --2- 172.21.15.63:0/1770325066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd6841031d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd678004940 tx=0x7fd678009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:14.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 -- 172.21.15.63:0/1770325066 shutdown_connections 2024-03-23T12:04:14.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 --2- 172.21.15.63:0/1770325066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6841037a0 0x7fd6840fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 --2- 172.21.15.63:0/1770325066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd6841031d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 -- 172.21.15.63:0/1770325066 >> 172.21.15.63:0/1770325066 conn(0x7fd6840f89b0 msgr2=0x7fd6840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:14.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 -- 172.21.15.63:0/1770325066 shutdown_connections 2024-03-23T12:04:14.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.519+0000 7fd68b600700 1 -- 172.21.15.63:0/1770325066 wait complete. 2024-03-23T12:04:14.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.520+0000 7fd68b600700 1 Processor -- start 2024-03-23T12:04:14.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.520+0000 7fd68b600700 1 -- start start 2024-03-23T12:04:14.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.520+0000 7fd68b600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd684115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:14.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.520+0000 7fd68b600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6841037a0 0x7fd684116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:14.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd68b600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684116880 con 0x7fd684102e00 2024-03-23T12:04:14.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd68b600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68410fc50 con 0x7fd6841037a0 2024-03-23T12:04:14.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd688c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd684115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd688c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd684115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39096/0 (socket says 172.21.15.63:39096) 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd688c00700 1 -- 172.21.15.63:0/1182820315 learned_addr learned my addr 172.21.15.63:0/1182820315 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd688c00700 1 -- 172.21.15.63:0/1182820315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6841037a0 msgr2=0x7fd684116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd683e00700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6841037a0 0x7fd684116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd688c00700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6841037a0 0x7fd684116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd688c00700 1 -- 172.21.15.63:0/1182820315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678005040 con 0x7fd684102e00 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.521+0000 7fd683e00700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6841037a0 0x7fd684116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.522+0000 7fd688c00700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd684115cf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd678009390 tx=0x7fd678009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.522+0000 7fd680c00700 1 -- 172.21.15.63:0/1182820315 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd67800e040 con 0x7fd684102e00 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.522+0000 7fd680c00700 1 -- 172.21.15.63:0/1182820315 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd678013420 con 0x7fd684102e00 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.522+0000 7fd680c00700 1 -- 172.21.15.63:0/1182820315 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd67801c630 con 0x7fd684102e00 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.522+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68410fed0 con 0x7fd684102e00 2024-03-23T12:04:14.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.522+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd684110490 con 0x7fd684102e00 2024-03-23T12:04:14.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.523+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6840fe550 con 0x7fd684102e00 2024-03-23T12:04:14.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.524+0000 7fd680c00700 1 -- 172.21.15.63:0/1182820315 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd6780139e0 con 0x7fd684102e00 2024-03-23T12:04:14.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.525+0000 7fd680c00700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd66c06c7d0 0x7fd66c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:14.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.525+0000 7fd680c00700 1 -- 172.21.15.63:0/1182820315 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd67808cd60 con 0x7fd684102e00 2024-03-23T12:04:14.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.525+0000 7fd683e00700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd66c06c7d0 0x7fd66c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:14.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.526+0000 7fd683e00700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd66c06c7d0 0x7fd66c06ec80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd670009fd0 tx=0x7fd67000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:14.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.530+0000 7fd680c00700 1 -- 172.21.15.63:0/1182820315 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd678091070 con 0x7fd684102e00 2024-03-23T12:04:14.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.762+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd684111340 con 0x7fd66c06c7d0 2024-03-23T12:04:14.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.764+0000 7fd680c00700 1 -- 172.21.15.63:0/1182820315 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd684111340 con 0x7fd66c06c7d0 2024-03-23T12:04:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd66c06c7d0 msgr2=0x7fd66c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd66c06c7d0 0x7fd66c06ec80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd670009fd0 tx=0x7fd67000b040 comp rx=0 tx=0).stop 2024-03-23T12:04:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 msgr2=0x7fd684115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd684115cf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd678009390 tx=0x7fd678009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 shutdown_connections 2024-03-23T12:04:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd66c06c7d0 0x7fd66c06ec80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd684102e00 0x7fd684115cf0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 --2- 172.21.15.63:0/1182820315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6841037a0 0x7fd684116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 >> 172.21.15.63:0/1182820315 conn(0x7fd6840f89b0 msgr2=0x7fd6840706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 shutdown_connections 2024-03-23T12:04:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.766+0000 7fd68b600700 1 -- 172.21.15.63:0/1182820315 wait complete. 2024-03-23T12:04:14.763 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 -- 172.21.15.63:0/3163583330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b81037a0 msgr2=0x7f60b80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 --2- 172.21.15.63:0/3163583330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b81037a0 0x7f60b80fce10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f60a4004940 tx=0x7f60a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 -- 172.21.15.63:0/3163583330 shutdown_connections 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 --2- 172.21.15.63:0/3163583330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b81037a0 0x7f60b80fce10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 --2- 172.21.15.63:0/3163583330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b8102e00 0x7f60b81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 -- 172.21.15.63:0/3163583330 >> 172.21.15.63:0/3163583330 conn(0x7f60b80f89b0 msgr2=0x7f60b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 -- 172.21.15.63:0/3163583330 shutdown_connections 2024-03-23T12:04:14.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.905+0000 7f60bca00700 1 -- 172.21.15.63:0/3163583330 wait complete. 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.906+0000 7f60bca00700 1 Processor -- start 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.906+0000 7f60bca00700 1 -- start start 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.906+0000 7f60bca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b8102e00 0x7f60b80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.906+0000 7f60bca00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b81037a0 0x7f60b8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.906+0000 7f60bca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8070e20 con 0x7f60b8102e00 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.906+0000 7f60bca00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8070f60 con 0x7f60b81037a0 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b8102e00 0x7f60b80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b8102e00 0x7f60b80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39118/0 (socket says 172.21.15.63:39118) 2024-03-23T12:04:14.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5e00700 1 -- 172.21.15.63:0/1686103200 learned_addr learned my addr 172.21.15.63:0/1686103200 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:14.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5400700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b81037a0 0x7f60b8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:14.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5e00700 1 -- 172.21.15.63:0/1686103200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b81037a0 msgr2=0x7f60b8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:14.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5e00700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b81037a0 0x7f60b8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:14.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5e00700 1 -- 172.21.15.63:0/1686103200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a4005040 con 0x7f60b8102e00 2024-03-23T12:04:14.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60b5e00700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b8102e00 0x7f60b80702c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f60a800b790 tx=0x7f60a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:14.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.907+0000 7f60ae000700 1 -- 172.21.15.63:0/1686103200 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a800f6f0 con 0x7f60b8102e00 2024-03-23T12:04:14.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.908+0000 7f60ae000700 1 -- 172.21.15.63:0/1686103200 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60a800fd00 con 0x7f60b8102e00 2024-03-23T12:04:14.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.908+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b8119de0 con 0x7f60b8102e00 2024-03-23T12:04:14.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.908+0000 7f60ae000700 1 -- 172.21.15.63:0/1686103200 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a80103a0 con 0x7f60b8102e00 2024-03-23T12:04:14.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.908+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b811a290 con 0x7f60b8102e00 2024-03-23T12:04:14.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.909+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b80fe550 con 0x7f60b8102e00 2024-03-23T12:04:14.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.910+0000 7f60ae000700 1 -- 172.21.15.63:0/1686103200 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f60a800f850 con 0x7f60b8102e00 2024-03-23T12:04:14.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.911+0000 7f60ae000700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f609c06c7d0 0x7f609c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:14.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.911+0000 7f60ae000700 1 -- 172.21.15.63:0/1686103200 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f60a808b8a0 con 0x7f60b8102e00 2024-03-23T12:04:14.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.912+0000 7f60b5400700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f609c06c7d0 0x7f609c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:14.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.912+0000 7f60b5400700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f609c06c7d0 0x7f609c06ec80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60b8071c00 tx=0x7f60a4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:14.916+0000 7f60ae000700 1 -- 172.21.15.63:0/1686103200 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f60a8055b20 con 0x7f60b8102e00 2024-03-23T12:04:15.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.147+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60b80719b0 con 0x7f609c06c7d0 2024-03-23T12:04:15.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.148+0000 7f60ae000700 1 -- 172.21.15.63:0/1686103200 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f60b80719b0 con 0x7f609c06c7d0 2024-03-23T12:04:15.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.150+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f609c06c7d0 msgr2=0x7f609c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.150+0000 7f60bca00700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f609c06c7d0 0x7f609c06ec80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f60b8071c00 tx=0x7f60a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b8102e00 msgr2=0x7f60b80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b8102e00 0x7f60b80702c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f60a800b790 tx=0x7f60a8009250 comp rx=0 tx=0).stop 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 shutdown_connections 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f609c06c7d0 0x7f609c06ec80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60b8102e00 0x7f60b80702c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 --2- 172.21.15.63:0/1686103200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b81037a0 0x7f60b8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 >> 172.21.15.63:0/1686103200 conn(0x7f60b80f89b0 msgr2=0x7f60b80f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 shutdown_connections 2024-03-23T12:04:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.151+0000 7f60bca00700 1 -- 172.21.15.63:0/1686103200 wait complete. 2024-03-23T12:04:15.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.288+0000 7fee18000700 1 -- 172.21.15.63:0/2969860589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070360 msgr2=0x7fee10070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.288+0000 7fee18000700 1 --2- 172.21.15.63:0/2969860589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070360 0x7fee10070730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fee04004970 tx=0x7fee04009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:15.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.288+0000 7fee18000700 1 -- 172.21.15.63:0/2969860589 shutdown_connections 2024-03-23T12:04:15.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.288+0000 7fee18000700 1 --2- 172.21.15.63:0/2969860589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10070c70 0x7fee10109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.288+0000 7fee18000700 1 --2- 172.21.15.63:0/2969860589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070360 0x7fee10070730 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.288+0000 7fee18000700 1 -- 172.21.15.63:0/2969860589 >> 172.21.15.63:0/2969860589 conn(0x7fee100f89b0 msgr2=0x7fee100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:15.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.288+0000 7fee18000700 1 -- 172.21.15.63:0/2969860589 shutdown_connections 2024-03-23T12:04:15.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.289+0000 7fee18000700 1 -- 172.21.15.63:0/2969860589 wait complete. 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.289+0000 7fee18000700 1 Processor -- start 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.289+0000 7fee18000700 1 -- start start 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee18000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10070360 0x7fee10115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee18000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070c70 0x7fee101161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee18000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101167f0 con 0x7fee10070c70 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee18000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1010fc00 con 0x7fee10070360 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee14c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070c70 0x7fee101161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee14c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070c70 0x7fee101161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39130/0 (socket says 172.21.15.63:39130) 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee14c00700 1 -- 172.21.15.63:0/292038457 learned_addr learned my addr 172.21.15.63:0/292038457 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:15.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.290+0000 7fee15600700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10070360 0x7fee10115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee14c00700 1 -- 172.21.15.63:0/292038457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10070360 msgr2=0x7fee10115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee14c00700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10070360 0x7fee10115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee14c00700 1 -- 172.21.15.63:0/292038457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04005040 con 0x7fee10070c70 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee14c00700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070c70 0x7fee101161a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fedfc002a50 tx=0x7fedfc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee0d600700 1 -- 172.21.15.63:0/292038457 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc007cc0 con 0x7fee10070c70 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee0d600700 1 -- 172.21.15.63:0/292038457 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedfc00f040 con 0x7fee10070c70 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee0d600700 1 -- 172.21.15.63:0/292038457 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc00d9c0 con 0x7fee10070c70 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1010fee0 con 0x7fee10070c70 2024-03-23T12:04:15.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.291+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee101103d0 con 0x7fee10070c70 2024-03-23T12:04:15.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.292+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10071aa0 con 0x7fee10070c70 2024-03-23T12:04:15.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.293+0000 7fee0d600700 1 -- 172.21.15.63:0/292038457 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fedfc00db60 con 0x7fee10070c70 2024-03-23T12:04:15.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.294+0000 7fee0d600700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf806c7d0 0x7fedf806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:15.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.294+0000 7fee0d600700 1 -- 172.21.15.63:0/292038457 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fedfc08b1a0 con 0x7fee10070c70 2024-03-23T12:04:15.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.294+0000 7fee15600700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf806c7d0 0x7fedf806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:15.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.295+0000 7fee15600700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf806c7d0 0x7fedf806ec80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee0400b370 tx=0x7fee04009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:15.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.299+0000 7fee0d600700 1 -- 172.21.15.63:0/292038457 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fedfc055320 con 0x7fee10070c70 2024-03-23T12:04:15.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:14 smithi063 ceph-mon[141124]: pgmap v393: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:15.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:14 smithi111 ceph-mon[106899]: pgmap v393: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:15.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.523+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fee1010bfd0 con 0x7fedf806c7d0 2024-03-23T12:04:15.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.533+0000 7fee0d600700 1 -- 172.21.15.63:0/292038457 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fee1010bfd0 con 0x7fedf806c7d0 2024-03-23T12:04:15.515 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:04:15.515 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (13m) 6m ago 26m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:04:15.515 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (10m) 6m ago 26m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:04:15.515 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (10m) 5m ago 25m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (13m) 6m ago 26m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (6m) 6m ago 20m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (6m) 6m ago 20m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (20m) 5m ago 20m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (5m) 5m ago 20m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (15m) 6m ago 28m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (14m) 5m ago 25m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (12m) 6m ago 28m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (11m) 5m ago 25m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (14m) 6m ago 26m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (14m) 5m ago 25m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (10m) 6m ago 24m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (9m) 6m ago 24m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (8m) 6m ago 23m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:04:15.516 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (8m) 5m ago 23m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:04:15.517 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (7m) 5m ago 22m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:04:15.517 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (7m) 5m ago 22m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:04:15.517 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (13m) 6m ago 26m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf806c7d0 msgr2=0x7fedf806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf806c7d0 0x7fedf806ec80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee0400b370 tx=0x7fee04009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070c70 msgr2=0x7fee101161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070c70 0x7fee101161a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fedfc002a50 tx=0x7fedfc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 shutdown_connections 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf806c7d0 0x7fedf806ec80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10070360 0x7fee10115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 --2- 172.21.15.63:0/292038457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee10070c70 0x7fee101161a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 >> 172.21.15.63:0/292038457 conn(0x7fee100f89b0 msgr2=0x7fee100f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:15.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 shutdown_connections 2024-03-23T12:04:15.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.536+0000 7fee18000700 1 -- 172.21.15.63:0/292038457 wait complete. 2024-03-23T12:04:15.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 -- 172.21.15.63:0/4049141923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 msgr2=0x7f45701031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 --2- 172.21.15.63:0/4049141923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f45701031e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4560004940 tx=0x7f4560009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 -- 172.21.15.63:0/4049141923 shutdown_connections 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 --2- 172.21.15.63:0/4049141923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45701037b0 0x7f45700fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 --2- 172.21.15.63:0/4049141923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f45701031e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 -- 172.21.15.63:0/4049141923 >> 172.21.15.63:0/4049141923 conn(0x7f45700f89a0 msgr2=0x7f45700fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 -- 172.21.15.63:0/4049141923 shutdown_connections 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.672+0000 7f4576600700 1 -- 172.21.15.63:0/4049141923 wait complete. 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.673+0000 7f4576600700 1 Processor -- start 2024-03-23T12:04:15.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.673+0000 7f4576600700 1 -- start start 2024-03-23T12:04:15.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.673+0000 7f4576600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f4570115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.673+0000 7f4576600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45701037b0 0x7f4570116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.673+0000 7f4576600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4570116890 con 0x7f4570102e10 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.673+0000 7f4576600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457010fc60 con 0x7f45701037b0 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f4570115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f4570115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39158/0 (socket says 172.21.15.63:39158) 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456f400700 1 -- 172.21.15.63:0/2833411693 learned_addr learned my addr 172.21.15.63:0/2833411693 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456ea00700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45701037b0 0x7f4570116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456f400700 1 -- 172.21.15.63:0/2833411693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45701037b0 msgr2=0x7f4570116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456f400700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45701037b0 0x7f4570116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456f400700 1 -- 172.21.15.63:0/2833411693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560005040 con 0x7f4570102e10 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456ea00700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45701037b0 0x7f4570116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.674+0000 7f456f400700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f4570115d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4560009360 tx=0x7f4560009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.675+0000 7f4567400700 1 -- 172.21.15.63:0/2833411693 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f456000e040 con 0x7f4570102e10 2024-03-23T12:04:15.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.675+0000 7f4567400700 1 -- 172.21.15.63:0/2833411693 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4560013420 con 0x7f4570102e10 2024-03-23T12:04:15.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.675+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457010fee0 con 0x7f4570102e10 2024-03-23T12:04:15.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.675+0000 7f4567400700 1 -- 172.21.15.63:0/2833411693 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f456001c5f0 con 0x7f4570102e10 2024-03-23T12:04:15.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.675+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45701104a0 con 0x7f4570102e10 2024-03-23T12:04:15.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.676+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45700fe560 con 0x7f4570102e10 2024-03-23T12:04:15.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.677+0000 7f4567400700 1 -- 172.21.15.63:0/2833411693 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4560013590 con 0x7f4570102e10 2024-03-23T12:04:15.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.678+0000 7f4567400700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f455406c7d0 0x7f455406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:15.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.678+0000 7f4567400700 1 -- 172.21.15.63:0/2833411693 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f456008cce0 con 0x7f4570102e10 2024-03-23T12:04:15.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.678+0000 7f456ea00700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f455406c7d0 0x7f455406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:15.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.679+0000 7f456ea00700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f455406c7d0 0x7f455406ec80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f455c009fd0 tx=0x7f455c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:15.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.683+0000 7f4567400700 1 -- 172.21.15.63:0/2833411693 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4560091070 con 0x7f4570102e10 2024-03-23T12:04:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.979+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f457005f5f0 con 0x7f4570102e10 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.980+0000 7f4567400700 1 -- 172.21.15.63:0/2833411693 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f456005aaa0 con 0x7f4570102e10 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:04:15.963 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:04:15.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f455406c7d0 msgr2=0x7f455406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f455406c7d0 0x7f455406ec80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f455c009fd0 tx=0x7f455c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 msgr2=0x7f4570115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:15.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f4570115d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4560009360 tx=0x7f4560009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:15.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 shutdown_connections 2024-03-23T12:04:15.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f455406c7d0 0x7f455406ec80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4570102e10 0x7f4570115d00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 --2- 172.21.15.63:0/2833411693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45701037b0 0x7f4570116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.982+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 >> 172.21.15.63:0/2833411693 conn(0x7f45700f89a0 msgr2=0x7f45700f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.983+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 shutdown_connections 2024-03-23T12:04:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:15.983+0000 7f4576600700 1 -- 172.21.15.63:0/2833411693 wait complete. 2024-03-23T12:04:16.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.119+0000 7fb86e200700 1 -- 172.21.15.63:0/2323773748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 msgr2=0x7fb8681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.119+0000 7fb86e200700 1 --2- 172.21.15.63:0/2323773748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8681031d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb858004940 tx=0x7fb858009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.119+0000 7fb86e200700 1 -- 172.21.15.63:0/2323773748 shutdown_connections 2024-03-23T12:04:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.119+0000 7fb86e200700 1 --2- 172.21.15.63:0/2323773748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8681037a0 0x7fb8680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.119+0000 7fb86e200700 1 --2- 172.21.15.63:0/2323773748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8681031d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.119+0000 7fb86e200700 1 -- 172.21.15.63:0/2323773748 >> 172.21.15.63:0/2323773748 conn(0x7fb8680f89b0 msgr2=0x7fb8680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.120+0000 7fb86e200700 1 -- 172.21.15.63:0/2323773748 shutdown_connections 2024-03-23T12:04:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.120+0000 7fb86e200700 1 -- 172.21.15.63:0/2323773748 wait complete. 2024-03-23T12:04:16.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.120+0000 7fb86e200700 1 Processor -- start 2024-03-23T12:04:16.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb86e200700 1 -- start start 2024-03-23T12:04:16.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb86e200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb86e200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8681037a0 0x7fb868070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb86e200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8680720b0 con 0x7fb868102e00 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb86e200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb868072220 con 0x7fb8681037a0 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb867400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb867400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39180/0 (socket says 172.21.15.63:39180) 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb867400700 1 -- 172.21.15.63:0/497829915 learned_addr learned my addr 172.21.15.63:0/497829915 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.121+0000 7fb866a00700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8681037a0 0x7fb868070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb867400700 1 -- 172.21.15.63:0/497829915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8681037a0 msgr2=0x7fb868070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb867400700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8681037a0 0x7fb868070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb867400700 1 -- 172.21.15.63:0/497829915 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb858005040 con 0x7fb868102e00 2024-03-23T12:04:16.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb867400700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8680702c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb858009390 tx=0x7fb858009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:16.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb85f400700 1 -- 172.21.15.63:0/497829915 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb85800e040 con 0x7fb868102e00 2024-03-23T12:04:16.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb85f400700 1 -- 172.21.15.63:0/497829915 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb858013420 con 0x7fb868102e00 2024-03-23T12:04:16.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb868070e00 con 0x7fb868102e00 2024-03-23T12:04:16.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb85f400700 1 -- 172.21.15.63:0/497829915 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb85801c630 con 0x7fb868102e00 2024-03-23T12:04:16.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.122+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86811e6d0 con 0x7fb868102e00 2024-03-23T12:04:16.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.123+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8680fe550 con 0x7fb868102e00 2024-03-23T12:04:16.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.124+0000 7fb85f400700 1 -- 172.21.15.63:0/497829915 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb8580139e0 con 0x7fb868102e00 2024-03-23T12:04:16.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.125+0000 7fb85f400700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb84c06c7d0 0x7fb84c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.125+0000 7fb85f400700 1 -- 172.21.15.63:0/497829915 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb85808cd60 con 0x7fb868102e00 2024-03-23T12:04:16.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.125+0000 7fb866a00700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb84c06c7d0 0x7fb84c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.126+0000 7fb866a00700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb84c06c7d0 0x7fb84c06ec80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb854009b80 tx=0x7fb85400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:16.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.130+0000 7fb85f400700 1 -- 172.21.15.63:0/497829915 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb858091070 con 0x7fb868102e00 2024-03-23T12:04:16.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.387+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb86804bf00 con 0x7fb868102e00 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.388+0000 7fb85f400700 1 -- 172.21.15.63:0/497829915 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fb85805aaa0 con 0x7fb868102e00 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:04:16.370 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:04:16.371 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:16.372 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:16.373 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:16.373 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:16.373 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:16.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb84c06c7d0 msgr2=0x7fb84c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb84c06c7d0 0x7fb84c06ec80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb854009b80 tx=0x7fb85400b040 comp rx=0 tx=0).stop 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 msgr2=0x7fb8680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8680702c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb858009390 tx=0x7fb858009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 shutdown_connections 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb84c06c7d0 0x7fb84c06ec80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb868102e00 0x7fb8680702c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 --2- 172.21.15.63:0/497829915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8681037a0 0x7fb868070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.390+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 >> 172.21.15.63:0/497829915 conn(0x7fb8680f89b0 msgr2=0x7fb8680f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.391+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 shutdown_connections 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.391+0000 7fb86e200700 1 -- 172.21.15.63:0/497829915 wait complete. 2024-03-23T12:04:16.374 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:04:16.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:15 smithi063 ceph-mon[141124]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:16.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:15 smithi111 ceph-mon[106899]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:16.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 -- 172.21.15.63:0/819653085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f0102e00 msgr2=0x7f70f01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 --2- 172.21.15.63:0/819653085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f0102e00 0x7f70f01031d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f70e0004940 tx=0x7f70e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:16.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 -- 172.21.15.63:0/819653085 shutdown_connections 2024-03-23T12:04:16.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 --2- 172.21.15.63:0/819653085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70f01037a0 0x7f70f00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 --2- 172.21.15.63:0/819653085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f0102e00 0x7f70f01031d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 -- 172.21.15.63:0/819653085 >> 172.21.15.63:0/819653085 conn(0x7f70f00f89b0 msgr2=0x7f70f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:16.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 -- 172.21.15.63:0/819653085 shutdown_connections 2024-03-23T12:04:16.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.530+0000 7f70f5000700 1 -- 172.21.15.63:0/819653085 wait complete. 2024-03-23T12:04:16.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.531+0000 7f70f5000700 1 Processor -- start 2024-03-23T12:04:16.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.531+0000 7f70f5000700 1 -- start start 2024-03-23T12:04:16.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70f5000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70f0102e00 0x7f70f0115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70f5000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f01037a0 0x7f70f01161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70f5000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0116800 con 0x7f70f01037a0 2024-03-23T12:04:16.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70f5000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f010fbd0 con 0x7f70f0102e00 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70ed400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f01037a0 0x7f70f01161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70ed400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f01037a0 0x7f70f01161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39198/0 (socket says 172.21.15.63:39198) 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70ed400700 1 -- 172.21.15.63:0/3867840199 learned_addr learned my addr 172.21.15.63:0/3867840199 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70ed400700 1 -- 172.21.15.63:0/3867840199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70f0102e00 msgr2=0x7f70f0115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70ede00700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70f0102e00 0x7f70f0115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.532+0000 7f70ed400700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70f0102e00 0x7f70f0115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70ed400700 1 -- 172.21.15.63:0/3867840199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e0005040 con 0x7f70f01037a0 2024-03-23T12:04:16.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70ede00700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70f0102e00 0x7f70f0115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:04:16.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70ed400700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f01037a0 0x7f70f01161b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00eb40 tx=0x7f70dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:16.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70e6000700 1 -- 172.21.15.63:0/3867840199 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70dc007cc0 con 0x7f70f01037a0 2024-03-23T12:04:16.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f010fe50 con 0x7f70f01037a0 2024-03-23T12:04:16.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70e6000700 1 -- 172.21.15.63:0/3867840199 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70dc00f040 con 0x7f70f01037a0 2024-03-23T12:04:16.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70e6000700 1 -- 172.21.15.63:0/3867840199 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70dc009570 con 0x7f70f01037a0 2024-03-23T12:04:16.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.533+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f0110410 con 0x7f70f01037a0 2024-03-23T12:04:16.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.534+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f00fe550 con 0x7f70f01037a0 2024-03-23T12:04:16.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.535+0000 7f70e6000700 1 -- 172.21.15.63:0/3867840199 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f70dc0096d0 con 0x7f70f01037a0 2024-03-23T12:04:16.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.536+0000 7f70e6000700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70d406c7d0 0x7f70d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.536+0000 7f70e6000700 1 -- 172.21.15.63:0/3867840199 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f70dc08b530 con 0x7f70f01037a0 2024-03-23T12:04:16.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.536+0000 7f70ede00700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70d406c7d0 0x7f70d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.537+0000 7f70ede00700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70d406c7d0 0x7f70d406ec80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f70e000b0d0 tx=0x7f70e0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:16.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.540+0000 7f70e6000700 1 -- 172.21.15.63:0/3867840199 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f70dc059270 con 0x7f70f01037a0 2024-03-23T12:04:16.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.772+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f70f0109e30 con 0x7f70d406c7d0 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.773+0000 7f70e6000700 1 -- 172.21.15.63:0/3867840199 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f70f0109e30 con 0x7f70d406c7d0 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:04:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:04:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:04:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:04:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:04:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:04:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:04:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:04:16.756 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:04:16.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70d406c7d0 msgr2=0x7f70d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70d406c7d0 0x7f70d406ec80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f70e000b0d0 tx=0x7f70e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f01037a0 msgr2=0x7f70f01161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f01037a0 0x7f70f01161b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00eb40 tx=0x7f70dc00b040 comp rx=0 tx=0).stop 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 shutdown_connections 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70d406c7d0 0x7f70d406ec80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70f0102e00 0x7f70f0115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 --2- 172.21.15.63:0/3867840199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70f01037a0 0x7f70f01161b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 >> 172.21.15.63:0/3867840199 conn(0x7f70f00f89b0 msgr2=0x7f70f00f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 shutdown_connections 2024-03-23T12:04:16.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.776+0000 7f70f5000700 1 -- 172.21.15.63:0/3867840199 wait complete. 2024-03-23T12:04:16.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 -- 172.21.15.63:0/1704811705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8102ca0 msgr2=0x7f72d8103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 --2- 172.21.15.63:0/1704811705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8102ca0 0x7f72d8103070 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f72c8004940 tx=0x7f72c8009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:16.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 -- 172.21.15.63:0/1704811705 shutdown_connections 2024-03-23T12:04:16.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 --2- 172.21.15.63:0/1704811705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8065630 0x7f72d8065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 --2- 172.21.15.63:0/1704811705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8102ca0 0x7f72d8103070 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 -- 172.21.15.63:0/1704811705 >> 172.21.15.63:0/1704811705 conn(0x7f72d8071b70 msgr2=0x7f72d8071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:16.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 -- 172.21.15.63:0/1704811705 shutdown_connections 2024-03-23T12:04:16.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.913+0000 7f72dd200700 1 -- 172.21.15.63:0/1704811705 wait complete. 2024-03-23T12:04:16.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.914+0000 7f72dd200700 1 Processor -- start 2024-03-23T12:04:16.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.914+0000 7f72dd200700 1 -- start start 2024-03-23T12:04:16.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.914+0000 7f72dd200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8065630 0x7f72d81149b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.914+0000 7f72dd200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8102ca0 0x7f72d8114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72dd200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81154f0 con 0x7f72d8065630 2024-03-23T12:04:16.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72dd200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d810fac0 con 0x7f72d8102ca0 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72d7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8065630 0x7f72d81149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72d7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8065630 0x7f72d81149b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39210/0 (socket says 172.21.15.63:39210) 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72d7400700 1 -- 172.21.15.63:0/273113321 learned_addr learned my addr 172.21.15.63:0/273113321 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72d6a00700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8102ca0 0x7f72d8114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72d7400700 1 -- 172.21.15.63:0/273113321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8102ca0 msgr2=0x7f72d8114ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72d7400700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8102ca0 0x7f72d8114ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.915+0000 7f72d7400700 1 -- 172.21.15.63:0/273113321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c8005040 con 0x7f72d8065630 2024-03-23T12:04:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.916+0000 7f72d7400700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8065630 0x7f72d81149b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009390 tx=0x7f72c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:16.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.916+0000 7f72cf400700 1 -- 172.21.15.63:0/273113321 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72c800e040 con 0x7f72d8065630 2024-03-23T12:04:16.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.916+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d810fd40 con 0x7f72d8065630 2024-03-23T12:04:16.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.916+0000 7f72cf400700 1 -- 172.21.15.63:0/273113321 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72c8013420 con 0x7f72d8065630 2024-03-23T12:04:16.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.916+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d8110300 con 0x7f72d8065630 2024-03-23T12:04:16.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.916+0000 7f72cf400700 1 -- 172.21.15.63:0/273113321 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72c80255e0 con 0x7f72d8065630 2024-03-23T12:04:16.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.917+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d8106840 con 0x7f72d8065630 2024-03-23T12:04:16.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.918+0000 7f72cf400700 1 -- 172.21.15.63:0/273113321 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f72c80139e0 con 0x7f72d8065630 2024-03-23T12:04:16.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.919+0000 7f72cf400700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72bc06c7d0 0x7f72bc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:16.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.919+0000 7f72cf400700 1 -- 172.21.15.63:0/273113321 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f72c808ce10 con 0x7f72d8065630 2024-03-23T12:04:16.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.919+0000 7f72d6a00700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72bc06c7d0 0x7f72bc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:16.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.920+0000 7f72d6a00700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72bc06c7d0 0x7f72bc06ec80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f72d80f9e00 tx=0x7f72c4009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:16.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:16.923+0000 7f72cf400700 1 -- 172.21.15.63:0/273113321 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f72c8057010 con 0x7f72d8065630 2024-03-23T12:04:17.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.219+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f72d810fed0 con 0x7f72d8065630 2024-03-23T12:04:17.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.219+0000 7f72cf400700 1 -- 172.21.15.63:0/273113321 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f72d810fed0 con 0x7f72d8065630 2024-03-23T12:04:17.201 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:04:17.201 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:04:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:04:17.201 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:04:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:04:17.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.221+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72bc06c7d0 msgr2=0x7f72bc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.221+0000 7f72dd200700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72bc06c7d0 0x7f72bc06ec80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f72d80f9e00 tx=0x7f72c4009040 comp rx=0 tx=0).stop 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.221+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8065630 msgr2=0x7f72d81149b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.221+0000 7f72dd200700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8065630 0x7f72d81149b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009390 tx=0x7f72c8009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.222+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 shutdown_connections 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.222+0000 7f72dd200700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f72bc06c7d0 0x7f72bc06ec80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.222+0000 7f72dd200700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72d8065630 0x7f72d81149b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.222+0000 7f72dd200700 1 --2- 172.21.15.63:0/273113321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8102ca0 0x7f72d8114ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.222+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 >> 172.21.15.63:0/273113321 conn(0x7f72d8071b70 msgr2=0x7f72d81039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.222+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 shutdown_connections 2024-03-23T12:04:17.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:17.222+0000 7f72dd200700 1 -- 172.21.15.63:0/273113321 wait complete. 2024-03-23T12:04:17.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:16 smithi063 ceph-mon[141124]: from='client.34774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:17.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:16 smithi063 ceph-mon[141124]: from='client.34778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:17.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:16 smithi063 ceph-mon[141124]: pgmap v394: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:17.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:16 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2833411693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:04:17.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:16 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/497829915' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:04:17.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:16 smithi111 ceph-mon[106899]: from='client.34774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:17.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:16 smithi111 ceph-mon[106899]: from='client.34778 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:17.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:16 smithi111 ceph-mon[106899]: pgmap v394: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:17.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:16 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2833411693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:04:17.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:16 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/497829915' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:04:18.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:17 smithi063 ceph-mon[141124]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:18.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:17 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/273113321' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:04:18.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:17 smithi111 ceph-mon[106899]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:18.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:17 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/273113321' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:04:19.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:18 smithi063 ceph-mon[141124]: pgmap v395: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:19.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:18 smithi111 ceph-mon[106899]: pgmap v395: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:21.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:20 smithi063 ceph-mon[141124]: pgmap v396: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:21.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:20 smithi111 ceph-mon[106899]: pgmap v396: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:23.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:22 smithi063 ceph-mon[141124]: pgmap v397: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:23.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:22 smithi111 ceph-mon[106899]: pgmap v397: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:25.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:24 smithi063 ceph-mon[141124]: pgmap v398: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:25.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:24 smithi111 ceph-mon[106899]: pgmap v398: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:26 smithi063 ceph-mon[141124]: pgmap v399: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:26 smithi111 ceph-mon[106899]: pgmap v399: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:29.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:28 smithi063 ceph-mon[141124]: pgmap v400: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:29.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:28 smithi111 ceph-mon[106899]: pgmap v400: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:31.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:30 smithi063 ceph-mon[141124]: pgmap v401: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:31.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:30 smithi111 ceph-mon[106899]: pgmap v401: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:33.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:32 smithi063 ceph-mon[141124]: pgmap v402: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:33.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:32 smithi111 ceph-mon[106899]: pgmap v402: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:35.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:34 smithi063 ceph-mon[141124]: pgmap v403: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:35.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:34 smithi111 ceph-mon[106899]: pgmap v403: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:36.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:36 smithi063 ceph-mon[141124]: pgmap v404: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:36.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:36 smithi111 ceph-mon[106899]: pgmap v404: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:39.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:38 smithi063 ceph-mon[141124]: pgmap v405: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:39.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:38 smithi111 ceph-mon[106899]: pgmap v405: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:40 smithi063 ceph-mon[141124]: pgmap v406: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:41.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:40 smithi111 ceph-mon[106899]: pgmap v406: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:41.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:43.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:42 smithi063 ceph-mon[141124]: pgmap v407: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:43.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:42 smithi111 ceph-mon[106899]: pgmap v407: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:45.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:44 smithi063 ceph-mon[141124]: pgmap v408: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:45.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:44 smithi111 ceph-mon[106899]: pgmap v408: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:46.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:46 smithi063 ceph-mon[141124]: pgmap v409: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:46.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:46 smithi111 ceph-mon[106899]: pgmap v409: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 -- 172.21.15.63:0/291388194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 msgr2=0x7fa2501031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 --2- 172.21.15.63:0/291388194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa2501031d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa240004940 tx=0x7fa240009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 -- 172.21.15.63:0/291388194 shutdown_connections 2024-03-23T12:04:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 --2- 172.21.15.63:0/291388194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2501037a0 0x7fa2500fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 --2- 172.21.15.63:0/291388194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa2501031d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 -- 172.21.15.63:0/291388194 >> 172.21.15.63:0/291388194 conn(0x7fa2500f89b0 msgr2=0x7fa2500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:47.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 -- 172.21.15.63:0/291388194 shutdown_connections 2024-03-23T12:04:47.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.374+0000 7fa256200700 1 -- 172.21.15.63:0/291388194 wait complete. 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.375+0000 7fa256200700 1 Processor -- start 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.375+0000 7fa256200700 1 -- start start 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa256200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa250115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa256200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2501037a0 0x7fa2501161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa256200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa250116820 con 0x7fa250102e00 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa256200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25010fbf0 con 0x7fa2501037a0 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa24f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa250115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:47.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa24f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa250115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56278/0 (socket says 172.21.15.63:56278) 2024-03-23T12:04:47.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa24f400700 1 -- 172.21.15.63:0/3554398171 learned_addr learned my addr 172.21.15.63:0/3554398171 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:47.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa24ea00700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2501037a0 0x7fa2501161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:47.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa24f400700 1 -- 172.21.15.63:0/3554398171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2501037a0 msgr2=0x7fa2501161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:47.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa24f400700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2501037a0 0x7fa2501161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.376+0000 7fa24f400700 1 -- 172.21.15.63:0/3554398171 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240005040 con 0x7fa250102e00 2024-03-23T12:04:47.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.377+0000 7fa24f400700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa250115c90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa240007f50 tx=0x7fa240009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:47.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.377+0000 7fa247400700 1 -- 172.21.15.63:0/3554398171 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa240021070 con 0x7fa250102e00 2024-03-23T12:04:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.377+0000 7fa247400700 1 -- 172.21.15.63:0/3554398171 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa240013e00 con 0x7fa250102e00 2024-03-23T12:04:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.377+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25010fe70 con 0x7fa250102e00 2024-03-23T12:04:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.377+0000 7fa247400700 1 -- 172.21.15.63:0/3554398171 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24000e070 con 0x7fa250102e00 2024-03-23T12:04:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.377+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa250110430 con 0x7fa250102e00 2024-03-23T12:04:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.378+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2500fe550 con 0x7fa250102e00 2024-03-23T12:04:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.379+0000 7fa247400700 1 -- 172.21.15.63:0/3554398171 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa240013950 con 0x7fa250102e00 2024-03-23T12:04:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.380+0000 7fa247400700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa23406c7d0 0x7fa23406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.381+0000 7fa247400700 1 -- 172.21.15.63:0/3554398171 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa24008cd90 con 0x7fa250102e00 2024-03-23T12:04:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.381+0000 7fa24ea00700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa23406c7d0 0x7fa23406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.381+0000 7fa24ea00700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa23406c7d0 0x7fa23406ec80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa23c004600 tx=0x7fa23c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.386+0000 7fa247400700 1 -- 172.21.15.63:0/3554398171 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa24005aa20 con 0x7fa250102e00 2024-03-23T12:04:47.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.621+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa25004bed0 con 0x7fa23406c7d0 2024-03-23T12:04:47.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.622+0000 7fa247400700 1 -- 172.21.15.63:0/3554398171 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa25004bed0 con 0x7fa23406c7d0 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa23406c7d0 msgr2=0x7fa23406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa23406c7d0 0x7fa23406ec80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa23c004600 tx=0x7fa23c00b040 comp rx=0 tx=0).stop 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 msgr2=0x7fa250115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa250115c90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa240007f50 tx=0x7fa240009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 shutdown_connections 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa23406c7d0 0x7fa23406ec80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa250102e00 0x7fa250115c90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 --2- 172.21.15.63:0/3554398171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2501037a0 0x7fa2501161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 >> 172.21.15.63:0/3554398171 conn(0x7fa2500f89b0 msgr2=0x7fa2500f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 shutdown_connections 2024-03-23T12:04:47.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.625+0000 7fa256200700 1 -- 172.21.15.63:0/3554398171 wait complete. 2024-03-23T12:04:47.621 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:04:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 -- 172.21.15.63:0/3511282360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 msgr2=0x7f9970070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 --2- 172.21.15.63:0/3511282360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970070730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9964004970 tx=0x7f9964009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 -- 172.21.15.63:0/3511282360 shutdown_connections 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 --2- 172.21.15.63:0/3511282360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9970070c70 0x7f9970109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 --2- 172.21.15.63:0/3511282360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970070730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 -- 172.21.15.63:0/3511282360 >> 172.21.15.63:0/3511282360 conn(0x7f99700f89b0 msgr2=0x7f99700fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 -- 172.21.15.63:0/3511282360 shutdown_connections 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.764+0000 7f9977600700 1 -- 172.21.15.63:0/3511282360 wait complete. 2024-03-23T12:04:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.765+0000 7f9977600700 1 Processor -- start 2024-03-23T12:04:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.765+0000 7f9977600700 1 -- start start 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.765+0000 7f9977600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.765+0000 7f9977600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9970070c70 0x7f9970116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.765+0000 7f9977600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9970116760 con 0x7f9970070360 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.765+0000 7f9977600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997010fbe0 con 0x7f9970070c70 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.766+0000 7f9974c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.766+0000 7f9974c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970115bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56302/0 (socket says 172.21.15.63:56302) 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.766+0000 7f9974c00700 1 -- 172.21.15.63:0/3460577178 learned_addr learned my addr 172.21.15.63:0/3460577178 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.766+0000 7f9974c00700 1 -- 172.21.15.63:0/3460577178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9970070c70 msgr2=0x7f9970116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.766+0000 7f9974c00700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9970070c70 0x7f9970116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.766+0000 7f9974c00700 1 -- 172.21.15.63:0/3460577178 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9964005040 con 0x7f9970070360 2024-03-23T12:04:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.766+0000 7f9974c00700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970115bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9964004f50 tx=0x7f9964009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.767+0000 7f996cc00700 1 -- 172.21.15.63:0/3460577178 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f996400e040 con 0x7f9970070360 2024-03-23T12:04:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.767+0000 7f996cc00700 1 -- 172.21.15.63:0/3460577178 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9964013420 con 0x7f9970070360 2024-03-23T12:04:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.767+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997010fe60 con 0x7f9970070360 2024-03-23T12:04:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.767+0000 7f996cc00700 1 -- 172.21.15.63:0/3460577178 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f996401c610 con 0x7f9970070360 2024-03-23T12:04:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.767+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99701103a0 con 0x7f9970070360 2024-03-23T12:04:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.768+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9970103b90 con 0x7f9970070360 2024-03-23T12:04:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.769+0000 7f996cc00700 1 -- 172.21.15.63:0/3460577178 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9964013590 con 0x7f9970070360 2024-03-23T12:04:47.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.770+0000 7f996cc00700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f995806c7d0 0x7f995806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:47.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.770+0000 7f996cc00700 1 -- 172.21.15.63:0/3460577178 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f996408cca0 con 0x7f9970070360 2024-03-23T12:04:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.770+0000 7f996fe00700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f995806c7d0 0x7f995806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.771+0000 7f996fe00700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f995806c7d0 0x7f995806ec80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99701064b0 tx=0x7f995c009ce0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:47.775+0000 7f996cc00700 1 -- 172.21.15.63:0/3460577178 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9964056ea0 con 0x7f9970070360 2024-03-23T12:04:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.017+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99701112d0 con 0x7f995806c7d0 2024-03-23T12:04:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.018+0000 7f996cc00700 1 -- 172.21.15.63:0/3460577178 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f99701112d0 con 0x7f995806c7d0 2024-03-23T12:04:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.020+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f995806c7d0 msgr2=0x7f995806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.020+0000 7f9977600700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f995806c7d0 0x7f995806ec80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99701064b0 tx=0x7f995c009ce0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.020+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 msgr2=0x7f9970115bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.020+0000 7f9977600700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970115bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9964004f50 tx=0x7f9964009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.021+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 shutdown_connections 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.021+0000 7f9977600700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f995806c7d0 0x7f995806ec80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.021+0000 7f9977600700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9970070360 0x7f9970115bd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.021+0000 7f9977600700 1 --2- 172.21.15.63:0/3460577178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9970070c70 0x7f9970116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.021+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 >> 172.21.15.63:0/3460577178 conn(0x7f99700f89b0 msgr2=0x7f99701015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.021+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 shutdown_connections 2024-03-23T12:04:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.021+0000 7f9977600700 1 -- 172.21.15.63:0/3460577178 wait complete. 2024-03-23T12:04:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.158+0000 7f3e5e000700 1 -- 172.21.15.63:0/1883216060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 msgr2=0x7f3e580fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.158+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1883216060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e580fe260 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3e48004940 tx=0x7f3e48009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.159+0000 7f3e5e000700 1 -- 172.21.15.63:0/1883216060 shutdown_connections 2024-03-23T12:04:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.159+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1883216060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580fe830 0x7f3e58106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.159+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1883216060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e580fe260 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.159+0000 7f3e5e000700 1 -- 172.21.15.63:0/1883216060 >> 172.21.15.63:0/1883216060 conn(0x7f3e58072bc0 msgr2=0x7f3e58072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.159+0000 7f3e5e000700 1 -- 172.21.15.63:0/1883216060 shutdown_connections 2024-03-23T12:04:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.159+0000 7f3e5e000700 1 -- 172.21.15.63:0/1883216060 wait complete. 2024-03-23T12:04:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.159+0000 7f3e5e000700 1 Processor -- start 2024-03-23T12:04:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e5e000700 1 -- start start 2024-03-23T12:04:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e5e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e58115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e5e000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580fe830 0x7f3e58116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e5e000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581168b0 con 0x7f3e580fde90 2024-03-23T12:04:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e5e000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5810fc80 con 0x7f3e580fe830 2024-03-23T12:04:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e57400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e58115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e57400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e58115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56320/0 (socket says 172.21.15.63:56320) 2024-03-23T12:04:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.160+0000 7f3e57400700 1 -- 172.21.15.63:0/1100252642 learned_addr learned my addr 172.21.15.63:0/1100252642 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e4fe00700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580fe830 0x7f3e58116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e57400700 1 -- 172.21.15.63:0/1100252642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580fe830 msgr2=0x7f3e58116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e57400700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580fe830 0x7f3e58116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e57400700 1 -- 172.21.15.63:0/1100252642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48005040 con 0x7f3e580fde90 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e57400700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e58115d20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000c00 tx=0x7f3e48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e4f400700 1 -- 172.21.15.63:0/1100252642 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e4800e040 con 0x7f3e580fde90 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5810ff00 con 0x7f3e580fde90 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e4f400700 1 -- 172.21.15.63:0/1100252642 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e48013420 con 0x7f3e580fde90 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.161+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e58110440 con 0x7f3e580fde90 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.162+0000 7f3e4f400700 1 -- 172.21.15.63:0/1100252642 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e480255e0 con 0x7f3e580fde90 2024-03-23T12:04:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.162+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e58106a20 con 0x7f3e580fde90 2024-03-23T12:04:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.163+0000 7f3e4f400700 1 -- 172.21.15.63:0/1100252642 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3e48013970 con 0x7f3e580fde90 2024-03-23T12:04:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.164+0000 7f3e4f400700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e3c06c7d0 0x7f3e3c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.164+0000 7f3e4f400700 1 -- 172.21.15.63:0/1100252642 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3e4808cc90 con 0x7f3e580fde90 2024-03-23T12:04:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.165+0000 7f3e4fe00700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e3c06c7d0 0x7f3e3c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.165+0000 7f3e4fe00700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e3c06c7d0 0x7f3e3c06ec80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3e581115c0 tx=0x7f3e40009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.169+0000 7f3e4f400700 1 -- 172.21.15.63:0/1100252642 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3e4805aa50 con 0x7f3e580fde90 2024-03-23T12:04:48.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.392+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e58109dc0 con 0x7f3e3c06c7d0 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.402+0000 7f3e4f400700 1 -- 172.21.15.63:0/1100252642 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f3e58109dc0 con 0x7f3e3c06c7d0 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (14m) 6m ago 27m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (11m) 6m ago 27m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (11m) 6m ago 26m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (13m) 6m ago 26m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (7m) 6m ago 20m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (6m) 6m ago 20m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:04:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (20m) 6m ago 20m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (6m) 6m ago 20m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (16m) 6m ago 28m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (15m) 6m ago 26m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (12m) 6m ago 28m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (12m) 6m ago 26m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (14m) 6m ago 27m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (14m) 6m ago 26m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (10m) 6m ago 25m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (10m) 6m ago 24m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (9m) 6m ago 24m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (8m) 6m ago 23m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (8m) 6m ago 23m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (7m) 6m ago 22m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:04:48.385 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (14m) 6m ago 26m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e3c06c7d0 msgr2=0x7f3e3c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e3c06c7d0 0x7f3e3c06ec80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3e581115c0 tx=0x7f3e40009ee0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 msgr2=0x7f3e58115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e58115d20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000c00 tx=0x7f3e48009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 shutdown_connections 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e3c06c7d0 0x7f3e3c06ec80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e580fde90 0x7f3e58115d20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 --2- 172.21.15.63:0/1100252642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580fe830 0x7f3e58116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 >> 172.21.15.63:0/1100252642 conn(0x7f3e58072bc0 msgr2=0x7f3e580fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 shutdown_connections 2024-03-23T12:04:48.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.405+0000 7f3e5e000700 1 -- 172.21.15.63:0/1100252642 wait complete. 2024-03-23T12:04:48.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.540+0000 7f41b4a00700 1 -- 172.21.15.63:0/525548528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 msgr2=0x7f41b01031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.540+0000 7f41b4a00700 1 --2- 172.21.15.63:0/525548528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b01031f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f41a0004940 tx=0x7f41a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:48.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.540+0000 7f41b4a00700 1 -- 172.21.15.63:0/525548528 shutdown_connections 2024-03-23T12:04:48.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.540+0000 7f41b4a00700 1 --2- 172.21.15.63:0/525548528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b01037c0 0x7f41b00fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.540+0000 7f41b4a00700 1 --2- 172.21.15.63:0/525548528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b01031f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.540+0000 7f41b4a00700 1 -- 172.21.15.63:0/525548528 >> 172.21.15.63:0/525548528 conn(0x7f41b00f8970 msgr2=0x7f41b00fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:48.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.541+0000 7f41b4a00700 1 -- 172.21.15.63:0/525548528 shutdown_connections 2024-03-23T12:04:48.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.541+0000 7f41b4a00700 1 -- 172.21.15.63:0/525548528 wait complete. 2024-03-23T12:04:48.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.541+0000 7f41b4a00700 1 Processor -- start 2024-03-23T12:04:48.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41b4a00700 1 -- start start 2024-03-23T12:04:48.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41b4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b0113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41b4a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b01037c0 0x7f41b0114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41b4a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b0114670 con 0x7f41b0102e20 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41b4a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b010da60 con 0x7f41b01037c0 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41ade00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b0113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41ade00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b0113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56348/0 (socket says 172.21.15.63:56348) 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41ade00700 1 -- 172.21.15.63:0/3305682114 learned_addr learned my addr 172.21.15.63:0/3305682114 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.542+0000 7f41a7e00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b01037c0 0x7f41b0114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41ade00700 1 -- 172.21.15.63:0/3305682114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b01037c0 msgr2=0x7f41b0114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41ade00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b01037c0 0x7f41b0114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41ade00700 1 -- 172.21.15.63:0/3305682114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0005040 con 0x7f41b0102e20 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41a7e00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b01037c0 0x7f41b0114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:04:48.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41ade00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b0113ae0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009360 tx=0x7f41a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:48.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41a6000700 1 -- 172.21.15.63:0/3305682114 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41a000e040 con 0x7f41b0102e20 2024-03-23T12:04:48.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41a6000700 1 -- 172.21.15.63:0/3305682114 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41a0013420 con 0x7f41b0102e20 2024-03-23T12:04:48.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41a6000700 1 -- 172.21.15.63:0/3305682114 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41a001c5f0 con 0x7f41b0102e20 2024-03-23T12:04:48.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b010dce0 con 0x7f41b0102e20 2024-03-23T12:04:48.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.543+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b010e2a0 con 0x7f41b0102e20 2024-03-23T12:04:48.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.544+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b00fe570 con 0x7f41b0102e20 2024-03-23T12:04:48.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.545+0000 7f41a6000700 1 -- 172.21.15.63:0/3305682114 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f41a0013590 con 0x7f41b0102e20 2024-03-23T12:04:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.546+0000 7f41a6000700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f419406c7d0 0x7f419406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.546+0000 7f41a6000700 1 -- 172.21.15.63:0/3305682114 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f41a008cc90 con 0x7f41b0102e20 2024-03-23T12:04:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.547+0000 7f41a7e00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f419406c7d0 0x7f419406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.547+0000 7f41a7e00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f419406c7d0 0x7f419406ec80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4198007fd0 tx=0x7f4198009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:48.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.551+0000 7f41a6000700 1 -- 172.21.15.63:0/3305682114 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f41a005aa50 con 0x7f41b0102e20 2024-03-23T12:04:48.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.847+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f41b010f190 con 0x7f41b0102e20 2024-03-23T12:04:48.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.848+0000 7f41a6000700 1 -- 172.21.15.63:0/3305682114 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f41a005a5e0 con 0x7f41b0102e20 2024-03-23T12:04:48.829 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:04:48.829 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:04:48.829 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:04:48.830 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:04:48.831 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f419406c7d0 msgr2=0x7f419406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f419406c7d0 0x7f419406ec80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4198007fd0 tx=0x7f4198009040 comp rx=0 tx=0).stop 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 msgr2=0x7f41b0113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b0113ae0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009360 tx=0x7f41a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 shutdown_connections 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f419406c7d0 0x7f419406ec80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41b0102e20 0x7f41b0113ae0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 --2- 172.21.15.63:0/3305682114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b01037c0 0x7f41b0114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 >> 172.21.15.63:0/3305682114 conn(0x7f41b00f8970 msgr2=0x7f41b00f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 shutdown_connections 2024-03-23T12:04:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.850+0000 7f41b4a00700 1 -- 172.21.15.63:0/3305682114 wait complete. 2024-03-23T12:04:48.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 -- 172.21.15.63:0/4217686035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 msgr2=0x7fc95c065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 --2- 172.21.15.63:0/4217686035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c065a80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc94c004940 tx=0x7fc94c009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:48.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 -- 172.21.15.63:0/4217686035 shutdown_connections 2024-03-23T12:04:48.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 --2- 172.21.15.63:0/4217686035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c065a80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 --2- 172.21.15.63:0/4217686035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc95c102ca0 0x7fc95c103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 -- 172.21.15.63:0/4217686035 >> 172.21.15.63:0/4217686035 conn(0x7fc95c071b70 msgr2=0x7fc95c071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:48.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 -- 172.21.15.63:0/4217686035 shutdown_connections 2024-03-23T12:04:48.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.989+0000 7fc963200700 1 -- 172.21.15.63:0/4217686035 wait complete. 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.990+0000 7fc963200700 1 Processor -- start 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.990+0000 7fc963200700 1 -- start start 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.990+0000 7fc963200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c0fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc963200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc95c102ca0 0x7fc95c0fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc963200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0fc6f0 con 0x7fc95c065630 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc963200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0fc830 con 0x7fc95c102ca0 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c0fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c0fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56366/0 (socket says 172.21.15.63:56366) 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961e00700 1 -- 172.21.15.63:0/1404516960 learned_addr learned my addr 172.21.15.63:0/1404516960 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961400700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc95c102ca0 0x7fc95c0fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961e00700 1 -- 172.21.15.63:0/1404516960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc95c102ca0 msgr2=0x7fc95c0fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961e00700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc95c102ca0 0x7fc95c0fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:48.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961e00700 1 -- 172.21.15.63:0/1404516960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c005040 con 0x7fc95c065630 2024-03-23T12:04:48.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.991+0000 7fc961e00700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c0fbc20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc95000eb40 tx=0x7fc95000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:48.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.992+0000 7fc956000700 1 -- 172.21.15.63:0/1404516960 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc950007cc0 con 0x7fc95c065630 2024-03-23T12:04:48.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.992+0000 7fc956000700 1 -- 172.21.15.63:0/1404516960 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc95000f040 con 0x7fc95c065630 2024-03-23T12:04:48.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.992+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c0fca80 con 0x7fc95c065630 2024-03-23T12:04:48.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.992+0000 7fc956000700 1 -- 172.21.15.63:0/1404516960 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc950009570 con 0x7fc95c065630 2024-03-23T12:04:48.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.992+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c119f50 con 0x7fc95c065630 2024-03-23T12:04:48.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.993+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c0fe380 con 0x7fc95c065630 2024-03-23T12:04:48.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.994+0000 7fc956000700 1 -- 172.21.15.63:0/1404516960 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc9500096d0 con 0x7fc95c065630 2024-03-23T12:04:48.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.995+0000 7fc956000700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc94406c7d0 0x7fc94406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:48.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.995+0000 7fc956000700 1 -- 172.21.15.63:0/1404516960 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc95008b760 con 0x7fc95c065630 2024-03-23T12:04:48.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.995+0000 7fc961400700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc94406c7d0 0x7fc94406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:48.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.996+0000 7fc961400700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc94406c7d0 0x7fc94406ec80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc94c004560 tx=0x7fc94c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:48.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:48.999+0000 7fc956000700 1 -- 172.21.15.63:0/1404516960 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc950055a10 con 0x7fc95c065630 2024-03-23T12:04:49.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.256+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc95c0fd310 con 0x7fc95c065630 2024-03-23T12:04:49.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.257+0000 7fc956000700 1 -- 172.21.15.63:0/1404516960 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fc950059030 con 0x7fc95c065630 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:04:49.239 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:04:49.240 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:49.241 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:04:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc94406c7d0 msgr2=0x7fc94406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc94406c7d0 0x7fc94406ec80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc94c004560 tx=0x7fc94c009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 msgr2=0x7fc95c0fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c0fbc20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc95000eb40 tx=0x7fc95000b040 comp rx=0 tx=0).stop 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 shutdown_connections 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc94406c7d0 0x7fc94406ec80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc95c065630 0x7fc95c0fbc20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 --2- 172.21.15.63:0/1404516960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc95c102ca0 0x7fc95c0fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 >> 172.21.15.63:0/1404516960 conn(0x7fc95c071b70 msgr2=0x7fc95c1059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.259+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 shutdown_connections 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.260+0000 7fc963200700 1 -- 172.21.15.63:0/1404516960 wait complete. 2024-03-23T12:04:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 -- 172.21.15.63:0/3136072000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fe830 msgr2=0x7fe7e0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 --2- 172.21.15.63:0/3136072000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fe830 0x7fe7e0106f00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc004940 tx=0x7fe7cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 -- 172.21.15.63:0/3136072000 shutdown_connections 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 --2- 172.21.15.63:0/3136072000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fe830 0x7fe7e0106f00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 --2- 172.21.15.63:0/3136072000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e00fde90 0x7fe7e00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 -- 172.21.15.63:0/3136072000 >> 172.21.15.63:0/3136072000 conn(0x7fe7e0072bc0 msgr2=0x7fe7e0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 -- 172.21.15.63:0/3136072000 shutdown_connections 2024-03-23T12:04:49.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.398+0000 7fe7e6200700 1 -- 172.21.15.63:0/3136072000 wait complete. 2024-03-23T12:04:49.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:48 smithi063 ceph-mon[141124]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:49.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:48 smithi063 ceph-mon[141124]: pgmap v410: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:49.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:48 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3305682114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:04:49.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7e6200700 1 Processor -- start 2024-03-23T12:04:49.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7e6200700 1 -- start start 2024-03-23T12:04:49.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7e6200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fde90 0x7fe7e00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:49.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7e6200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e00fe830 0x7fe7e00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:49.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7e6200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00fdb40 con 0x7fe7e00fde90 2024-03-23T12:04:49.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7e6200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00fdcb0 con 0x7fe7e00fe830 2024-03-23T12:04:49.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7df400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fde90 0x7fe7e00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7df400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fde90 0x7fe7e00fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56394/0 (socket says 172.21.15.63:56394) 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.399+0000 7fe7df400700 1 -- 172.21.15.63:0/1943938157 learned_addr learned my addr 172.21.15.63:0/1943938157 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7df400700 1 -- 172.21.15.63:0/1943938157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e00fe830 msgr2=0x7fe7e00fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7df400700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e00fe830 0x7fe7e00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7df400700 1 -- 172.21.15.63:0/1943938157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc005040 con 0x7fe7e00fde90 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7df400700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fde90 0x7fe7e00fbd50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000eb70 tx=0x7fe7d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7d7400700 1 -- 172.21.15.63:0/1943938157 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7d0007cc0 con 0x7fe7e00fde90 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7d7400700 1 -- 172.21.15.63:0/1943938157 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7d000f040 con 0x7fe7e00fde90 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7d7400700 1 -- 172.21.15.63:0/1943938157 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7d00095f0 con 0x7fe7e00fde90 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.400+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e00fc890 con 0x7fe7e00fde90 2024-03-23T12:04:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.401+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e011e730 con 0x7fe7e00fde90 2024-03-23T12:04:49.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.401+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e0101980 con 0x7fe7e00fde90 2024-03-23T12:04:49.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.403+0000 7fe7d7400700 1 -- 172.21.15.63:0/1943938157 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe7d0009750 con 0x7fe7e00fde90 2024-03-23T12:04:49.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.403+0000 7fe7d7400700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe7c406c7d0 0x7fe7c406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:49.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.404+0000 7fe7d7400700 1 -- 172.21.15.63:0/1943938157 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe7d008b6e0 con 0x7fe7e00fde90 2024-03-23T12:04:49.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.404+0000 7fe7dea00700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe7c406c7d0 0x7fe7c406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:49.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.404+0000 7fe7dea00700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe7c406c7d0 0x7fe7c406ec80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00b0d0 tx=0x7fe7cc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.408+0000 7fe7d7400700 1 -- 172.21.15.63:0/1943938157 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe7d0059370 con 0x7fe7e00fde90 2024-03-23T12:04:49.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:48 smithi111 ceph-mon[106899]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:49.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:48 smithi111 ceph-mon[106899]: pgmap v410: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:49.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:48 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3305682114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:04:49.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.638+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7e0109df0 con 0x7fe7c406c7d0 2024-03-23T12:04:49.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.640+0000 7fe7d7400700 1 -- 172.21.15.63:0/1943938157 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe7e0109df0 con 0x7fe7c406c7d0 2024-03-23T12:04:49.621 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:04:49.621 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:04:49.621 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:04:49.622 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe7c406c7d0 msgr2=0x7fe7c406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe7c406c7d0 0x7fe7c406ec80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00b0d0 tx=0x7fe7cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fde90 msgr2=0x7fe7e00fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fde90 0x7fe7e00fbd50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000eb70 tx=0x7fe7d000b040 comp rx=0 tx=0).stop 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 shutdown_connections 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe7c406c7d0 0x7fe7c406ec80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7e00fde90 0x7fe7e00fbd50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 --2- 172.21.15.63:0/1943938157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e00fe830 0x7fe7e00fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 >> 172.21.15.63:0/1943938157 conn(0x7fe7e0072bc0 msgr2=0x7fe7e00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 shutdown_connections 2024-03-23T12:04:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.642+0000 7fe7e6200700 1 -- 172.21.15.63:0/1943938157 wait complete. 2024-03-23T12:04:49.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 -- 172.21.15.63:0/2013110097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 msgr2=0x7fdd480ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 --2- 172.21.15.63:0/2013110097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd480ff0a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdd34004940 tx=0x7fdd34009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:49.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 -- 172.21.15.63:0/2013110097 shutdown_connections 2024-03-23T12:04:49.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 --2- 172.21.15.63:0/2013110097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd480ff0a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 --2- 172.21.15.63:0/2013110097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd48105090 0x7fdd48105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 -- 172.21.15.63:0/2013110097 >> 172.21.15.63:0/2013110097 conn(0x7fdd480fac00 msgr2=0x7fdd480fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:49.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 -- 172.21.15.63:0/2013110097 shutdown_connections 2024-03-23T12:04:49.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.779+0000 7fdd4d800700 1 -- 172.21.15.63:0/2013110097 wait complete. 2024-03-23T12:04:49.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.780+0000 7fdd4d800700 1 Processor -- start 2024-03-23T12:04:49.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.780+0000 7fdd4d800700 1 -- start start 2024-03-23T12:04:49.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd4d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd48105090 0x7fdd48115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd4d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd48115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd4d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd48116600 con 0x7fdd48105a30 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd4d800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4810fbe0 con 0x7fdd48105090 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd46000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd48115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd46000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd48115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56400/0 (socket says 172.21.15.63:56400) 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd46000700 1 -- 172.21.15.63:0/4035733986 learned_addr learned my addr 172.21.15.63:0/4035733986 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd46a00700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd48105090 0x7fdd48115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd46000700 1 -- 172.21.15.63:0/4035733986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd48105090 msgr2=0x7fdd48115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.781+0000 7fdd46000700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd48105090 0x7fdd48115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.782+0000 7fdd46000700 1 -- 172.21.15.63:0/4035733986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34005040 con 0x7fdd48105a30 2024-03-23T12:04:49.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.782+0000 7fdd46000700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd48115fb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009390 tx=0x7fdd34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:49.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.782+0000 7fdd3ea00700 1 -- 172.21.15.63:0/4035733986 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd3400e040 con 0x7fdd48105a30 2024-03-23T12:04:49.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.782+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4810fe60 con 0x7fdd48105a30 2024-03-23T12:04:49.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.782+0000 7fdd3ea00700 1 -- 172.21.15.63:0/4035733986 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd34013420 con 0x7fdd48105a30 2024-03-23T12:04:49.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.782+0000 7fdd3ea00700 1 -- 172.21.15.63:0/4035733986 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd3401c630 con 0x7fdd48105a30 2024-03-23T12:04:49.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.782+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd48110420 con 0x7fdd48105a30 2024-03-23T12:04:49.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.783+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd481023a0 con 0x7fdd48105a30 2024-03-23T12:04:49.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.784+0000 7fdd3ea00700 1 -- 172.21.15.63:0/4035733986 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdd34013a00 con 0x7fdd48105a30 2024-03-23T12:04:49.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.785+0000 7fdd3ea00700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd2c06c7c0 0x7fdd2c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:04:49.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.785+0000 7fdd3ea00700 1 -- 172.21.15.63:0/4035733986 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdd3408cdd0 con 0x7fdd48105a30 2024-03-23T12:04:49.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.785+0000 7fdd46a00700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd2c06c7c0 0x7fdd2c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:04:49.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.786+0000 7fdd46a00700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd2c06c7c0 0x7fdd2c06ec70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd38004990 tx=0x7fdd38009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:04:49.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:49.790+0000 7fdd3ea00700 1 -- 172.21.15.63:0/4035733986 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdd3405aa60 con 0x7fdd48105a30 2024-03-23T12:04:50.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.089+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdd481110c0 con 0x7fdd48105a30 2024-03-23T12:04:50.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.089+0000 7fdd3ea00700 1 -- 172.21.15.63:0/4035733986 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fdd3405a5f0 con 0x7fdd48105a30 2024-03-23T12:04:50.071 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:04:50.071 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:04:50.071 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:04:50.071 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:04:50.071 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:04:50.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd2c06c7c0 msgr2=0x7fdd2c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd2c06c7c0 0x7fdd2c06ec70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd38004990 tx=0x7fdd38009ee0 comp rx=0 tx=0).stop 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 msgr2=0x7fdd48115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd48115fb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009390 tx=0x7fdd34009e30 comp rx=0 tx=0).stop 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 shutdown_connections 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd2c06c7c0 0x7fdd2c06ec70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd48105090 0x7fdd48115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 --2- 172.21.15.63:0/4035733986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd48105a30 0x7fdd48115fb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 >> 172.21.15.63:0/4035733986 conn(0x7fdd480fac00 msgr2=0x7fdd480fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 shutdown_connections 2024-03-23T12:04:50.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:04:50.092+0000 7fdd4d800700 1 -- 172.21.15.63:0/4035733986 wait complete. 2024-03-23T12:04:50.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:49 smithi063 ceph-mon[141124]: from='client.34802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:50.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:49 smithi063 ceph-mon[141124]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:50.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:49 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1404516960' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:04:50.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:49 smithi111 ceph-mon[106899]: from='client.34802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:50.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:49 smithi111 ceph-mon[106899]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:50.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:49 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1404516960' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:04:51.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:50 smithi063 ceph-mon[141124]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:51.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:50 smithi063 ceph-mon[141124]: pgmap v411: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:51.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:50 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4035733986' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:04:51.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:50 smithi111 ceph-mon[106899]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:04:51.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:50 smithi111 ceph-mon[106899]: pgmap v411: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:51.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:50 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4035733986' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:04:53.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:52 smithi063 ceph-mon[141124]: pgmap v412: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:53.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:52 smithi111 ceph-mon[106899]: pgmap v412: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:55.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:54 smithi063 ceph-mon[141124]: pgmap v413: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:54 smithi111 ceph-mon[106899]: pgmap v413: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:56 smithi063 ceph-mon[141124]: pgmap v414: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:04:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:56 smithi111 ceph-mon[106899]: pgmap v414: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:59.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:04:58 smithi063 ceph-mon[141124]: pgmap v415: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:04:59.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:04:58 smithi111 ceph-mon[106899]: pgmap v415: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:00 smithi063 ceph-mon[141124]: pgmap v416: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:01.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:00 smithi111 ceph-mon[106899]: pgmap v416: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:03.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:02 smithi063 ceph-mon[141124]: pgmap v417: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:03.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:02 smithi111 ceph-mon[106899]: pgmap v417: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:05.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:04 smithi063 ceph-mon[141124]: pgmap v418: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:05.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:04 smithi111 ceph-mon[106899]: pgmap v418: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:06.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:06 smithi063 ceph-mon[141124]: pgmap v419: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:06.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:06 smithi111 ceph-mon[106899]: pgmap v419: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:09.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:08 smithi063 ceph-mon[141124]: pgmap v420: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:09.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:08 smithi111 ceph-mon[106899]: pgmap v420: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:10 smithi063 ceph-mon[141124]: pgmap v421: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:11.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:10 smithi111 ceph-mon[106899]: pgmap v421: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:11.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:13.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:12 smithi063 ceph-mon[141124]: pgmap v422: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:13.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:12 smithi111 ceph-mon[106899]: pgmap v422: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:15.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:14 smithi063 ceph-mon[141124]: pgmap v423: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:15.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:14 smithi111 ceph-mon[106899]: pgmap v423: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:16.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:16 smithi063 ceph-mon[141124]: pgmap v424: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:16.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:16 smithi111 ceph-mon[106899]: pgmap v424: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:19.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:18 smithi063 ceph-mon[141124]: pgmap v425: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:19.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:18 smithi111 ceph-mon[106899]: pgmap v425: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:20.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 -- 172.21.15.63:0/1360865567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 msgr2=0x7f2bfc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 --2- 172.21.15.63:0/1360865567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc1031d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0004940 tx=0x7f2bf0009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 -- 172.21.15.63:0/1360865567 shutdown_connections 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 --2- 172.21.15.63:0/1360865567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bfc1037a0 0x7f2bfc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 --2- 172.21.15.63:0/1360865567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc1031d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 -- 172.21.15.63:0/1360865567 >> 172.21.15.63:0/1360865567 conn(0x7f2bfc0f89b0 msgr2=0x7f2bfc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 -- 172.21.15.63:0/1360865567 shutdown_connections 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.247+0000 7f2c03400700 1 -- 172.21.15.63:0/1360865567 wait complete. 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.248+0000 7f2c03400700 1 Processor -- start 2024-03-23T12:05:20.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.248+0000 7f2c03400700 1 -- start start 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.248+0000 7f2c03400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.248+0000 7f2c03400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bfc1037a0 0x7f2bfc116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.248+0000 7f2c03400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc1167d0 con 0x7f2bfc102e00 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.248+0000 7f2c03400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc10fba0 con 0x7f2bfc1037a0 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c00a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c00a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36070/0 (socket says 172.21.15.63:36070) 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c00a00700 1 -- 172.21.15.63:0/3519053693 learned_addr learned my addr 172.21.15.63:0/3519053693 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:20.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c00a00700 1 -- 172.21.15.63:0/3519053693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bfc1037a0 msgr2=0x7f2bfc116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c00a00700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bfc1037a0 0x7f2bfc116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c00a00700 1 -- 172.21.15.63:0/3519053693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf0005040 con 0x7f2bfc102e00 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c00a00700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc115c40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009390 tx=0x7f2bf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2bf8c00700 1 -- 172.21.15.63:0/3519053693 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bf000e040 con 0x7f2bfc102e00 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2bf8c00700 1 -- 172.21.15.63:0/3519053693 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bf0013420 con 0x7f2bfc102e00 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc10fe20 con 0x7f2bfc102e00 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.249+0000 7f2bf8c00700 1 -- 172.21.15.63:0/3519053693 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bf001c630 con 0x7f2bfc102e00 2024-03-23T12:05:20.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.250+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc1103e0 con 0x7f2bfc102e00 2024-03-23T12:05:20.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.251+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc0fe550 con 0x7f2bfc102e00 2024-03-23T12:05:20.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.252+0000 7f2bf8c00700 1 -- 172.21.15.63:0/3519053693 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2bf00139e0 con 0x7f2bfc102e00 2024-03-23T12:05:20.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.252+0000 7f2bf8c00700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2be406c7d0 0x7f2be406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:20.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.253+0000 7f2bf8c00700 1 -- 172.21.15.63:0/3519053693 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2bf008cc90 con 0x7f2bfc102e00 2024-03-23T12:05:20.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.253+0000 7f2bfbe00700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2be406c7d0 0x7f2be406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:20.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.254+0000 7f2bfbe00700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2be406c7d0 0x7f2be406ec80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009b80 tx=0x7f2be8009d60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:20.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.258+0000 7f2bf8c00700 1 -- 172.21.15.63:0/3519053693 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2bf005aa50 con 0x7f2bfc102e00 2024-03-23T12:05:20.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.491+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bfc109e30 con 0x7f2be406c7d0 2024-03-23T12:05:20.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.492+0000 7f2bf8c00700 1 -- 172.21.15.63:0/3519053693 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2bfc109e30 con 0x7f2be406c7d0 2024-03-23T12:05:20.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.494+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2be406c7d0 msgr2=0x7f2be406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.494+0000 7f2c03400700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2be406c7d0 0x7f2be406ec80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009b80 tx=0x7f2be8009d60 comp rx=0 tx=0).stop 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.494+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 msgr2=0x7f2bfc115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.494+0000 7f2c03400700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc115c40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009390 tx=0x7f2bf0009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.495+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 shutdown_connections 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.495+0000 7f2c03400700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2be406c7d0 0x7f2be406ec80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.495+0000 7f2c03400700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bfc102e00 0x7f2bfc115c40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.495+0000 7f2c03400700 1 --2- 172.21.15.63:0/3519053693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bfc1037a0 0x7f2bfc116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.495+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 >> 172.21.15.63:0/3519053693 conn(0x7f2bfc0f89b0 msgr2=0x7f2bfc0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.495+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 shutdown_connections 2024-03-23T12:05:20.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.495+0000 7f2c03400700 1 -- 172.21.15.63:0/3519053693 wait complete. 2024-03-23T12:05:20.490 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:05:20.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.633+0000 7f10e0600700 1 -- 172.21.15.63:0/1851314516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d8102e00 msgr2=0x7f10d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.634+0000 7f10e0600700 1 --2- 172.21.15.63:0/1851314516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d8102e00 0x7f10d81031d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f10cc004940 tx=0x7f10cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:20.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.634+0000 7f10e0600700 1 -- 172.21.15.63:0/1851314516 shutdown_connections 2024-03-23T12:05:20.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.634+0000 7f10e0600700 1 --2- 172.21.15.63:0/1851314516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d81037a0 0x7f10d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.634+0000 7f10e0600700 1 --2- 172.21.15.63:0/1851314516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d8102e00 0x7f10d81031d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.634+0000 7f10e0600700 1 -- 172.21.15.63:0/1851314516 >> 172.21.15.63:0/1851314516 conn(0x7f10d80f89b0 msgr2=0x7f10d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:20.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.634+0000 7f10e0600700 1 -- 172.21.15.63:0/1851314516 shutdown_connections 2024-03-23T12:05:20.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.634+0000 7f10e0600700 1 -- 172.21.15.63:0/1851314516 wait complete. 2024-03-23T12:05:20.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10e0600700 1 Processor -- start 2024-03-23T12:05:20.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10e0600700 1 -- start start 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10e0600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d8102e00 0x7f10d8115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10e0600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d81037a0 0x7f10d8116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10e0600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d81168a0 con 0x7f10d81037a0 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10e0600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d810fc70 con 0x7f10d8102e00 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10dd200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d81037a0 0x7f10d8116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10dd200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d81037a0 0x7f10d8116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36084/0 (socket says 172.21.15.63:36084) 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.635+0000 7f10dd200700 1 -- 172.21.15.63:0/2963605513 learned_addr learned my addr 172.21.15.63:0/2963605513 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10ddc00700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d8102e00 0x7f10d8115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10dd200700 1 -- 172.21.15.63:0/2963605513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d8102e00 msgr2=0x7f10d8115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10dd200700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d8102e00 0x7f10d8115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10dd200700 1 -- 172.21.15.63:0/2963605513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc005040 con 0x7f10d81037a0 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10dd200700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d81037a0 0x7f10d8116250 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f10c400c890 tx=0x7f10c40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10d5600700 1 -- 172.21.15.63:0/2963605513 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10c400a700 con 0x7f10d81037a0 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10d5600700 1 -- 172.21.15.63:0/2963605513 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10c400ad10 con 0x7f10d81037a0 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.636+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d810ff50 con 0x7f10d81037a0 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.637+0000 7f10d5600700 1 -- 172.21.15.63:0/2963605513 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10c4012e70 con 0x7f10d81037a0 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.637+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d8110510 con 0x7f10d81037a0 2024-03-23T12:05:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.638+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d80fe550 con 0x7f10d81037a0 2024-03-23T12:05:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.639+0000 7f10d5600700 1 -- 172.21.15.63:0/2963605513 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f10c400a860 con 0x7f10d81037a0 2024-03-23T12:05:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.639+0000 7f10d5600700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10c006c7d0 0x7f10c006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.640+0000 7f10d5600700 1 -- 172.21.15.63:0/2963605513 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f10c408ba90 con 0x7f10d81037a0 2024-03-23T12:05:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.640+0000 7f10ddc00700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10c006c7d0 0x7f10c006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.640+0000 7f10ddc00700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10c006c7d0 0x7f10c006ec80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:20.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.645+0000 7f10d5600700 1 -- 172.21.15.63:0/2963605513 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f10c40597a0 con 0x7f10d81037a0 2024-03-23T12:05:20.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.880+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10d8109e30 con 0x7f10c006c7d0 2024-03-23T12:05:20.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.882+0000 7f10d5600700 1 -- 172.21.15.63:0/2963605513 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f10d8109e30 con 0x7f10c006c7d0 2024-03-23T12:05:20.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10c006c7d0 msgr2=0x7f10c006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10c006c7d0 0x7f10c006ec80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:20.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d81037a0 msgr2=0x7f10d8116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:20.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d81037a0 0x7f10d8116250 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f10c400c890 tx=0x7f10c40044c0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 shutdown_connections 2024-03-23T12:05:20.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f10c006c7d0 0x7f10c006ec80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10d8102e00 0x7f10d8115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 --2- 172.21.15.63:0/2963605513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f10d81037a0 0x7f10d8116250 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:20.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 >> 172.21.15.63:0/2963605513 conn(0x7f10d80f89b0 msgr2=0x7f10d80704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:20.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 shutdown_connections 2024-03-23T12:05:20.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:20.884+0000 7f10e0600700 1 -- 172.21.15.63:0/2963605513 wait complete. 2024-03-23T12:05:21.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 -- 172.21.15.63:0/2357891384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070c70 msgr2=0x7f54d4108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 --2- 172.21.15.63:0/2357891384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070c70 0x7f54d4108f50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f54c0004970 tx=0x7f54c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:21.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 -- 172.21.15.63:0/2357891384 shutdown_connections 2024-03-23T12:05:21.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 --2- 172.21.15.63:0/2357891384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070c70 0x7f54d4108f50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 --2- 172.21.15.63:0/2357891384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54d4070360 0x7f54d4070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 -- 172.21.15.63:0/2357891384 >> 172.21.15.63:0/2357891384 conn(0x7f54d40f8830 msgr2=0x7f54d40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:21.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 -- 172.21.15.63:0/2357891384 shutdown_connections 2024-03-23T12:05:21.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.022+0000 7f54da400700 1 -- 172.21.15.63:0/2357891384 wait complete. 2024-03-23T12:05:21.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.023+0000 7f54da400700 1 Processor -- start 2024-03-23T12:05:21.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.023+0000 7f54da400700 1 -- start start 2024-03-23T12:05:21.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.023+0000 7f54da400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070360 0x7f54d4115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.023+0000 7f54da400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54d4070c70 0x7f54d4116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.023+0000 7f54da400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4116650 con 0x7f54d4070360 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.023+0000 7f54da400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d410fa40 con 0x7f54d4070c70 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070360 0x7f54d4115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070360 0x7f54d4115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36092/0 (socket says 172.21.15.63:36092) 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d9000700 1 -- 172.21.15.63:0/1377389247 learned_addr learned my addr 172.21.15.63:0/1377389247 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d3e00700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54d4070c70 0x7f54d4116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d9000700 1 -- 172.21.15.63:0/1377389247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54d4070c70 msgr2=0x7f54d4116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d9000700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54d4070c70 0x7f54d4116000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d9000700 1 -- 172.21.15.63:0/1377389247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c0005040 con 0x7f54d4070360 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d9000700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070360 0x7f54d4115ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f54c8002a50 tx=0x7f54c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.024+0000 7f54d0c00700 1 -- 172.21.15.63:0/1377389247 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54c8007cc0 con 0x7f54d4070360 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.025+0000 7f54d0c00700 1 -- 172.21.15.63:0/1377389247 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54c800f040 con 0x7f54d4070360 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.025+0000 7f54d0c00700 1 -- 172.21.15.63:0/1377389247 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54c8009540 con 0x7f54d4070360 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.025+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d410fd20 con 0x7f54d4070360 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.025+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d4110230 con 0x7f54d4070360 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.026+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d4071aa0 con 0x7f54d4070360 2024-03-23T12:05:21.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.027+0000 7f54d0c00700 1 -- 172.21.15.63:0/1377389247 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f54c80096a0 con 0x7f54d4070360 2024-03-23T12:05:21.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.028+0000 7f54d0c00700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54bc06c7d0 0x7f54bc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.028+0000 7f54d0c00700 1 -- 172.21.15.63:0/1377389247 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f54c808b7a0 con 0x7f54d4070360 2024-03-23T12:05:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.029+0000 7f54d3e00700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54bc06c7d0 0x7f54bc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.029+0000 7f54d3e00700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54bc06c7d0 0x7f54bc06ec80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f54d4110d20 tx=0x7f54c0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:21.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.033+0000 7f54d0c00700 1 -- 172.21.15.63:0/1377389247 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f54c80594b0 con 0x7f54d4070360 2024-03-23T12:05:21.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.256+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f54d410bdf0 con 0x7f54bc06c7d0 2024-03-23T12:05:21.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.267+0000 7f54d0c00700 1 -- 172.21.15.63:0/1377389247 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f54d410bdf0 con 0x7f54bc06c7d0 2024-03-23T12:05:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:05:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (14m) 7m ago 27m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (11m) 7m ago 28m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (11m) 7m ago 26m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (14m) 7m ago 27m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (7m) 7m ago 21m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (7m) 7m ago 21m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (21m) 7m ago 21m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (7m) 7m ago 21m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (17m) 7m ago 29m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (15m) 7m ago 26m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (13m) 7m ago 29m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (12m) 7m ago 26m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (15m) 7m ago 27m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:05:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (15m) 7m ago 26m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:05:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (11m) 7m ago 25m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:05:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (10m) 7m ago 25m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:05:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (9m) 7m ago 24m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:05:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (9m) 7m ago 24m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:05:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (8m) 7m ago 23m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:05:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (8m) 7m ago 23m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:05:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (15m) 7m ago 27m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:05:21.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54bc06c7d0 msgr2=0x7f54bc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54bc06c7d0 0x7f54bc06ec80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f54d4110d20 tx=0x7f54c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:21.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070360 msgr2=0x7f54d4115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070360 0x7f54d4115ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f54c8002a50 tx=0x7f54c80046b0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 shutdown_connections 2024-03-23T12:05:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f54bc06c7d0 0x7f54bc06ec80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54d4070360 0x7f54d4115ac0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 --2- 172.21.15.63:0/1377389247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54d4070c70 0x7f54d4116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 >> 172.21.15.63:0/1377389247 conn(0x7f54d40f8830 msgr2=0x7f54d40f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 shutdown_connections 2024-03-23T12:05:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.269+0000 7f54da400700 1 -- 172.21.15.63:0/1377389247 wait complete. 2024-03-23T12:05:21.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:20 smithi063 ceph-mon[141124]: pgmap v426: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:21.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.406+0000 7f6958c00700 1 -- 172.21.15.63:0/3781859438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954102de0 msgr2=0x7f69541031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.406+0000 7f6958c00700 1 --2- 172.21.15.63:0/3781859438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954102de0 0x7f69541031b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6944004940 tx=0x7f6944009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:21.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.406+0000 7f6958c00700 1 -- 172.21.15.63:0/3781859438 shutdown_connections 2024-03-23T12:05:21.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.406+0000 7f6958c00700 1 --2- 172.21.15.63:0/3781859438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6954103780 0x7f69540fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.406+0000 7f6958c00700 1 --2- 172.21.15.63:0/3781859438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954102de0 0x7f69541031b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.406+0000 7f6958c00700 1 -- 172.21.15.63:0/3781859438 >> 172.21.15.63:0/3781859438 conn(0x7f69540f8970 msgr2=0x7f69540fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:21.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.407+0000 7f6958c00700 1 -- 172.21.15.63:0/3781859438 shutdown_connections 2024-03-23T12:05:21.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.407+0000 7f6958c00700 1 -- 172.21.15.63:0/3781859438 wait complete. 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.407+0000 7f6958c00700 1 Processor -- start 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6958c00700 1 -- start start 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6958c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6954102de0 0x7f6954115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6958c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954103780 0x7f69541161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6958c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6954116840 con 0x7f6954103780 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6958c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695410fc10 con 0x7f6954102de0 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6951400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954103780 0x7f69541161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6951400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954103780 0x7f69541161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36106/0 (socket says 172.21.15.63:36106) 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.408+0000 7f6951400700 1 -- 172.21.15.63:0/1888714705 learned_addr learned my addr 172.21.15.63:0/1888714705 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f6951e00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6954102de0 0x7f6954115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f6951400700 1 -- 172.21.15.63:0/1888714705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6954102de0 msgr2=0x7f6954115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f6951400700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6954102de0 0x7f6954115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f6951400700 1 -- 172.21.15.63:0/1888714705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944005040 con 0x7f6954103780 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f6951400700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954103780 0x7f69541161f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f694000c890 tx=0x7f69400044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f694a000700 1 -- 172.21.15.63:0/1888714705 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f694000a7d0 con 0x7f6954103780 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f694a000700 1 -- 172.21.15.63:0/1888714705 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f694000ade0 con 0x7f6954103780 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f694a000700 1 -- 172.21.15.63:0/1888714705 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69400113a0 con 0x7f6954103780 2024-03-23T12:05:21.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695410fef0 con 0x7f6954103780 2024-03-23T12:05:21.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.409+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69541104b0 con 0x7f6954103780 2024-03-23T12:05:21.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.410+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69540fe530 con 0x7f6954103780 2024-03-23T12:05:21.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.411+0000 7f694a000700 1 -- 172.21.15.63:0/1888714705 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6940012400 con 0x7f6954103780 2024-03-23T12:05:21.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.412+0000 7f694a000700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f693806c7d0 0x7f693806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.412+0000 7f694a000700 1 -- 172.21.15.63:0/1888714705 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f694008bb10 con 0x7f6954103780 2024-03-23T12:05:21.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.413+0000 7f6951e00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f693806c7d0 0x7f693806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.413+0000 7f6951e00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f693806c7d0 0x7f693806ec80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f694400b3b0 tx=0x7f6944009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:21.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.417+0000 7f694a000700 1 -- 172.21.15.63:0/1888714705 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6940059820 con 0x7f6954103780 2024-03-23T12:05:21.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:20 smithi111 ceph-mon[106899]: pgmap v426: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:21.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.717+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f695404bed0 con 0x7f6954103780 2024-03-23T12:05:21.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.717+0000 7f694a000700 1 -- 172.21.15.63:0/1888714705 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f69400593b0 con 0x7f6954103780 2024-03-23T12:05:21.698 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:05:21.698 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:05:21.698 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:05:21.699 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:05:21.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:05:21.700 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f693806c7d0 msgr2=0x7f693806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f693806c7d0 0x7f693806ec80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f694400b3b0 tx=0x7f6944009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954103780 msgr2=0x7f69541161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954103780 0x7f69541161f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f694000c890 tx=0x7f69400044c0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 shutdown_connections 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f693806c7d0 0x7f693806ec80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6954102de0 0x7f6954115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 --2- 172.21.15.63:0/1888714705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954103780 0x7f69541161f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 >> 172.21.15.63:0/1888714705 conn(0x7f69540f8970 msgr2=0x7f69540f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 shutdown_connections 2024-03-23T12:05:21.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.720+0000 7f6958c00700 1 -- 172.21.15.63:0/1888714705 wait complete. 2024-03-23T12:05:21.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 -- 172.21.15.63:0/2943121043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 msgr2=0x7f46d80fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 --2- 172.21.15.63:0/2943121043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d80fcc50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f46c4004940 tx=0x7f46c4009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:21.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 -- 172.21.15.63:0/2943121043 shutdown_connections 2024-03-23T12:05:21.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 --2- 172.21.15.63:0/2943121043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d80fcc50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 --2- 172.21.15.63:0/2943121043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8102c40 0x7f46d8103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 -- 172.21.15.63:0/2943121043 >> 172.21.15.63:0/2943121043 conn(0x7f46d80f8830 msgr2=0x7f46d80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:21.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 -- 172.21.15.63:0/2943121043 shutdown_connections 2024-03-23T12:05:21.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.858+0000 7f46dec00700 1 -- 172.21.15.63:0/2943121043 wait complete. 2024-03-23T12:05:21.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.859+0000 7f46dec00700 1 Processor -- start 2024-03-23T12:05:21.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.860+0000 7f46dec00700 1 -- start start 2024-03-23T12:05:21.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.860+0000 7f46dec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8102c40 0x7f46d8113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.860+0000 7f46dec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d8113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.860+0000 7f46dec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d81143f0 con 0x7f46d81035e0 2024-03-23T12:05:21.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.860+0000 7f46dec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d810d7e0 con 0x7f46d8102c40 2024-03-23T12:05:21.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d8113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d8113da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37018/0 (socket says 172.21.15.63:37018) 2024-03-23T12:05:21.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dce00700 1 -- 172.21.15.63:0/1477858036 learned_addr learned my addr 172.21.15.63:0/1477858036 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dd800700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8102c40 0x7f46d8113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dce00700 1 -- 172.21.15.63:0/1477858036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8102c40 msgr2=0x7f46d8113860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dce00700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8102c40 0x7f46d8113860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dce00700 1 -- 172.21.15.63:0/1477858036 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c4005040 con 0x7f46d81035e0 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dd800700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8102c40 0x7f46d8113860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.861+0000 7f46dce00700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d8113da0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009fd0 tx=0x7f46c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.862+0000 7f46d5600700 1 -- 172.21.15.63:0/1477858036 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c400e030 con 0x7f46d81035e0 2024-03-23T12:05:21.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.862+0000 7f46d5600700 1 -- 172.21.15.63:0/1477858036 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46c4013e00 con 0x7f46d81035e0 2024-03-23T12:05:21.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.862+0000 7f46d5600700 1 -- 172.21.15.63:0/1477858036 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c401ccd0 con 0x7f46d81035e0 2024-03-23T12:05:21.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.862+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d810da60 con 0x7f46d81035e0 2024-03-23T12:05:21.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.862+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d810e020 con 0x7f46d81035e0 2024-03-23T12:05:21.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.863+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d80fe390 con 0x7f46d81035e0 2024-03-23T12:05:21.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.864+0000 7f46d5600700 1 -- 172.21.15.63:0/1477858036 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f46c401c500 con 0x7f46d81035e0 2024-03-23T12:05:21.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.865+0000 7f46d5600700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f46c006c7d0 0x7f46c006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:21.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.865+0000 7f46d5600700 1 -- 172.21.15.63:0/1477858036 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f46c408d230 con 0x7f46d81035e0 2024-03-23T12:05:21.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.865+0000 7f46dd800700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f46c006c7d0 0x7f46c006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:21.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.866+0000 7f46dd800700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f46c006c7d0 0x7f46c006ec80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009fd0 tx=0x7f46cc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:21.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:21.869+0000 7f46d5600700 1 -- 172.21.15.63:0/1477858036 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f46c405b000 con 0x7f46d81035e0 2024-03-23T12:05:22.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.127+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f46d8109bf0 con 0x7f46d81035e0 2024-03-23T12:05:22.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.128+0000 7f46d5600700 1 -- 172.21.15.63:0/1477858036 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f46c405ab90 con 0x7f46d81035e0 2024-03-23T12:05:22.108 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:05:22.109 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:05:22.110 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:22.111 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f46c006c7d0 msgr2=0x7f46c006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f46c006c7d0 0x7f46c006ec80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009fd0 tx=0x7f46cc00b040 comp rx=0 tx=0).stop 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 msgr2=0x7f46d8113da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d8113da0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009fd0 tx=0x7f46c4009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 shutdown_connections 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f46c006c7d0 0x7f46c006ec80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8102c40 0x7f46d8113860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 --2- 172.21.15.63:0/1477858036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46d81035e0 0x7f46d8113da0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 >> 172.21.15.63:0/1477858036 conn(0x7f46d80f8830 msgr2=0x7f46d80f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.130+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 shutdown_connections 2024-03-23T12:05:22.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.131+0000 7f46dec00700 1 -- 172.21.15.63:0/1477858036 wait complete. 2024-03-23T12:05:22.113 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:05:22.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 -- 172.21.15.63:0/4220618632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b601037a0 msgr2=0x7f5b600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 --2- 172.21.15.63:0/4220618632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b601037a0 0x7f5b600fce10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c004940 tx=0x7f5b4c009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:22.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 -- 172.21.15.63:0/4220618632 shutdown_connections 2024-03-23T12:05:22.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 --2- 172.21.15.63:0/4220618632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b601037a0 0x7f5b600fce10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 --2- 172.21.15.63:0/4220618632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b60102e00 0x7f5b601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 -- 172.21.15.63:0/4220618632 >> 172.21.15.63:0/4220618632 conn(0x7f5b600f89b0 msgr2=0x7f5b600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:22.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 -- 172.21.15.63:0/4220618632 shutdown_connections 2024-03-23T12:05:22.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.269+0000 7f5b66800700 1 -- 172.21.15.63:0/4220618632 wait complete. 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.270+0000 7f5b66800700 1 Processor -- start 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.270+0000 7f5b66800700 1 -- start start 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.270+0000 7f5b66800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b60102e00 0x7f5b60115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.270+0000 7f5b66800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b601037a0 0x7f5b60115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.270+0000 7f5b66800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b601165d0 con 0x7f5b60102e00 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.270+0000 7f5b66800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6010fbb0 con 0x7f5b601037a0 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b60102e00 0x7f5b60115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b60102e00 0x7f5b60115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37038/0 (socket says 172.21.15.63:37038) 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5f400700 1 -- 172.21.15.63:0/2111109220 learned_addr learned my addr 172.21.15.63:0/2111109220 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5ea00700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b601037a0 0x7f5b60115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5f400700 1 -- 172.21.15.63:0/2111109220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b601037a0 msgr2=0x7f5b60115f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5f400700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b601037a0 0x7f5b60115f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5f400700 1 -- 172.21.15.63:0/2111109220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4c005040 con 0x7f5b60102e00 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b5f400700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b60102e00 0x7f5b60115a40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5b50002a80 tx=0x7f5b500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.271+0000 7f5b57400700 1 -- 172.21.15.63:0/2111109220 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b50007cc0 con 0x7f5b60102e00 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.272+0000 7f5b57400700 1 -- 172.21.15.63:0/2111109220 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b5000f040 con 0x7f5b60102e00 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.272+0000 7f5b57400700 1 -- 172.21.15.63:0/2111109220 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b50009640 con 0x7f5b60102e00 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.272+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6010fe90 con 0x7f5b60102e00 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.272+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b60110450 con 0x7f5b60102e00 2024-03-23T12:05:22.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.273+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b600fe550 con 0x7f5b60102e00 2024-03-23T12:05:22.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.275+0000 7f5b57400700 1 -- 172.21.15.63:0/2111109220 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5b500097a0 con 0x7f5b60102e00 2024-03-23T12:05:22.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.276+0000 7f5b57400700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5b4406c7c0 0x7f5b4406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:22.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.276+0000 7f5b57400700 1 -- 172.21.15.63:0/2111109220 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b5008b650 con 0x7f5b60102e00 2024-03-23T12:05:22.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.277+0000 7f5b5ea00700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5b4406c7c0 0x7f5b4406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:22.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.277+0000 7f5b5ea00700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5b4406c7c0 0x7f5b4406ec70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00b3b0 tx=0x7f5b4c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:22.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.278+0000 7f5b57400700 1 -- 172.21.15.63:0/2111109220 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5b50059420 con 0x7f5b60102e00 2024-03-23T12:05:22.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:21 smithi063 ceph-mon[141124]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:22.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:21 smithi063 ceph-mon[141124]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:22.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:21 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1888714705' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:05:22.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:21 smithi111 ceph-mon[106899]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:22.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:21 smithi111 ceph-mon[106899]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:22.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:21 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1888714705' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:05:22.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.517+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b60111100 con 0x7f5b4406c7c0 2024-03-23T12:05:22.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.518+0000 7f5b57400700 1 -- 172.21.15.63:0/2111109220 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5b60111100 con 0x7f5b4406c7c0 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:05:22.499 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5b4406c7c0 msgr2=0x7f5b4406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5b4406c7c0 0x7f5b4406ec70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00b3b0 tx=0x7f5b4c009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b60102e00 msgr2=0x7f5b60115a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b60102e00 0x7f5b60115a40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5b50002a80 tx=0x7f5b500046b0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 shutdown_connections 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5b4406c7c0 0x7f5b4406ec70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5b60102e00 0x7f5b60115a40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 --2- 172.21.15.63:0/2111109220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b601037a0 0x7f5b60115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 >> 172.21.15.63:0/2111109220 conn(0x7f5b600f89b0 msgr2=0x7f5b60070630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 shutdown_connections 2024-03-23T12:05:22.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.521+0000 7f5b66800700 1 -- 172.21.15.63:0/2111109220 wait complete. 2024-03-23T12:05:22.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.657+0000 7fc20f000700 1 -- 172.21.15.63:0/3479939332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 msgr2=0x7fc2081031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.657+0000 7fc20f000700 1 --2- 172.21.15.63:0/3479939332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc2081031d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8004940 tx=0x7fc1f8009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:22.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.657+0000 7fc20f000700 1 -- 172.21.15.63:0/3479939332 shutdown_connections 2024-03-23T12:05:22.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.657+0000 7fc20f000700 1 --2- 172.21.15.63:0/3479939332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2081037a0 0x7fc2080fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.657+0000 7fc20f000700 1 --2- 172.21.15.63:0/3479939332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc2081031d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.657+0000 7fc20f000700 1 -- 172.21.15.63:0/3479939332 >> 172.21.15.63:0/3479939332 conn(0x7fc2080f89b0 msgr2=0x7fc2080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:22.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.658+0000 7fc20f000700 1 -- 172.21.15.63:0/3479939332 shutdown_connections 2024-03-23T12:05:22.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.658+0000 7fc20f000700 1 -- 172.21.15.63:0/3479939332 wait complete. 2024-03-23T12:05:22.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.658+0000 7fc20f000700 1 Processor -- start 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc20f000700 1 -- start start 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc20f000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc208113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc20f000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2081037a0 0x7fc208113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc20f000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2081146c0 con 0x7fc208102e00 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc20f000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20810da20 con 0x7fc2081037a0 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc207e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc208113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc207e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc208113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37054/0 (socket says 172.21.15.63:37054) 2024-03-23T12:05:22.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc207e00700 1 -- 172.21.15.63:0/3593358311 learned_addr learned my addr 172.21.15.63:0/3593358311 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:22.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc207e00700 1 -- 172.21.15.63:0/3593358311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2081037a0 msgr2=0x7fc208113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.659+0000 7fc1ffe00700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2081037a0 0x7fc208113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:22.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc207e00700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2081037a0 0x7fc208113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc207e00700 1 -- 172.21.15.63:0/3593358311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8005040 con 0x7fc208102e00 2024-03-23T12:05:22.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc1ffe00700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2081037a0 0x7fc208113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:05:22.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc207e00700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc208113aa0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8009390 tx=0x7fc1f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:22.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc204c00700 1 -- 172.21.15.63:0/3593358311 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1f800e040 con 0x7fc208102e00 2024-03-23T12:05:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc204c00700 1 -- 172.21.15.63:0/3593358311 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1f8013420 con 0x7fc208102e00 2024-03-23T12:05:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20810dca0 con 0x7fc208102e00 2024-03-23T12:05:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc204c00700 1 -- 172.21.15.63:0/3593358311 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1f801c630 con 0x7fc208102e00 2024-03-23T12:05:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.660+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20810e260 con 0x7fc208102e00 2024-03-23T12:05:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.661+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2080fe550 con 0x7fc208102e00 2024-03-23T12:05:22.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.662+0000 7fc204c00700 1 -- 172.21.15.63:0/3593358311 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc1f80139e0 con 0x7fc208102e00 2024-03-23T12:05:22.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.664+0000 7fc204c00700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1ec06c7d0 0x7fc1ec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:22.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.664+0000 7fc204c00700 1 -- 172.21.15.63:0/3593358311 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc1f808cd60 con 0x7fc208102e00 2024-03-23T12:05:22.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.664+0000 7fc1ffe00700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1ec06c7d0 0x7fc1ec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:22.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.665+0000 7fc1ffe00700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1ec06c7d0 0x7fc1ec06ec80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0006fd0 tx=0x7fc1f0008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:22.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.668+0000 7fc204c00700 1 -- 172.21.15.63:0/3593358311 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc1f8091070 con 0x7fc208102e00 2024-03-23T12:05:22.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.969+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc20804bed0 con 0x7fc208102e00 2024-03-23T12:05:22.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.969+0000 7fc204c00700 1 -- 172.21.15.63:0/3593358311 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc1f805aaa0 con 0x7fc208102e00 2024-03-23T12:05:22.950 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:05:22.950 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:05:22.950 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:05:22.950 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:05:22.950 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:05:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.971+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1ec06c7d0 msgr2=0x7fc1ec06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.971+0000 7fc20f000700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1ec06c7d0 0x7fc1ec06ec80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0006fd0 tx=0x7fc1f0008040 comp rx=0 tx=0).stop 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.971+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 msgr2=0x7fc208113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.971+0000 7fc20f000700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc208113aa0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8009390 tx=0x7fc1f8009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.972+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 shutdown_connections 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.972+0000 7fc20f000700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc1ec06c7d0 0x7fc1ec06ec80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.972+0000 7fc20f000700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc208102e00 0x7fc208113aa0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.972+0000 7fc20f000700 1 --2- 172.21.15.63:0/3593358311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2081037a0 0x7fc208113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.972+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 >> 172.21.15.63:0/3593358311 conn(0x7fc2080f89b0 msgr2=0x7fc2080f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.972+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 shutdown_connections 2024-03-23T12:05:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:22.972+0000 7fc20f000700 1 -- 172.21.15.63:0/3593358311 wait complete. 2024-03-23T12:05:23.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:22 smithi063 ceph-mon[141124]: from='client.34834 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:23.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:22 smithi063 ceph-mon[141124]: pgmap v427: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:23.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:22 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1477858036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:05:23.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:22 smithi111 ceph-mon[106899]: from='client.34834 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:23.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:22 smithi111 ceph-mon[106899]: pgmap v427: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:23.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:22 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1477858036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:05:24.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:23 smithi063 ceph-mon[141124]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:24.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:23 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3593358311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:05:24.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:23 smithi111 ceph-mon[106899]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:24.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:23 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3593358311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:05:25.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:24 smithi063 ceph-mon[141124]: pgmap v428: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:25.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:24 smithi111 ceph-mon[106899]: pgmap v428: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:26 smithi063 ceph-mon[141124]: pgmap v429: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:26 smithi111 ceph-mon[106899]: pgmap v429: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:29.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:28 smithi063 ceph-mon[141124]: pgmap v430: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:29.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:28 smithi111 ceph-mon[106899]: pgmap v430: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:31.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:30 smithi063 ceph-mon[141124]: pgmap v431: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:31.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:30 smithi111 ceph-mon[106899]: pgmap v431: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:33.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:32 smithi063 ceph-mon[141124]: pgmap v432: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:33.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:32 smithi111 ceph-mon[106899]: pgmap v432: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:35.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:34 smithi063 ceph-mon[141124]: pgmap v433: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:35.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:34 smithi111 ceph-mon[106899]: pgmap v433: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:36.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:36 smithi063 ceph-mon[141124]: pgmap v434: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:36.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:36 smithi111 ceph-mon[106899]: pgmap v434: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:39.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:38 smithi063 ceph-mon[141124]: pgmap v435: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:39.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:38 smithi111 ceph-mon[106899]: pgmap v435: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:40 smithi063 ceph-mon[141124]: pgmap v436: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:41.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:40 smithi111 ceph-mon[106899]: pgmap v436: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:41.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:43.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:42 smithi063 ceph-mon[141124]: pgmap v437: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:43.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:42 smithi111 ceph-mon[106899]: pgmap v437: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:45.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:44 smithi063 ceph-mon[141124]: pgmap v438: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:45.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:44 smithi111 ceph-mon[106899]: pgmap v438: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:46.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:46 smithi063 ceph-mon[141124]: pgmap v439: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:46.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:46 smithi111 ceph-mon[106899]: pgmap v439: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:49.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:48 smithi063 ceph-mon[141124]: pgmap v440: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:49.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:48 smithi111 ceph-mon[106899]: pgmap v440: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:51.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:50 smithi063 ceph-mon[141124]: pgmap v441: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:51.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:50 smithi111 ceph-mon[106899]: pgmap v441: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:53.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.123+0000 7f5253a00700 1 -- 172.21.15.63:0/990571521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 msgr2=0x7f524c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.123+0000 7f5253a00700 1 --2- 172.21.15.63:0/990571521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c0fce10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5238004940 tx=0x7f5238009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:53.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.123+0000 7f5253a00700 1 -- 172.21.15.63:0/990571521 shutdown_connections 2024-03-23T12:05:53.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.123+0000 7f5253a00700 1 --2- 172.21.15.63:0/990571521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c0fce10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.123+0000 7f5253a00700 1 --2- 172.21.15.63:0/990571521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102e00 0x7f524c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.123+0000 7f5253a00700 1 -- 172.21.15.63:0/990571521 >> 172.21.15.63:0/990571521 conn(0x7f524c0f89b0 msgr2=0x7f524c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:53.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.124+0000 7f5253a00700 1 -- 172.21.15.63:0/990571521 shutdown_connections 2024-03-23T12:05:53.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.124+0000 7f5253a00700 1 -- 172.21.15.63:0/990571521 wait complete. 2024-03-23T12:05:53.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.124+0000 7f5253a00700 1 Processor -- start 2024-03-23T12:05:53.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f5253a00700 1 -- start start 2024-03-23T12:05:53.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f5253a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102e00 0x7f524c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f5253a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f5253a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c116890 con 0x7f524c1037a0 2024-03-23T12:05:53.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f5253a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c10fc60 con 0x7f524c102e00 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f524be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f524be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34028/0 (socket says 172.21.15.63:34028) 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f524be00700 1 -- 172.21.15.63:0/2370905400 learned_addr learned my addr 172.21.15.63:0/2370905400 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.125+0000 7f5251000700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102e00 0x7f524c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f524be00700 1 -- 172.21.15.63:0/2370905400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102e00 msgr2=0x7f524c115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f524be00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102e00 0x7f524c115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f524be00700 1 -- 172.21.15.63:0/2370905400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5238005040 con 0x7f524c1037a0 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f524be00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c116240 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5238009390 tx=0x7f5238009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:53.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f5248c00700 1 -- 172.21.15.63:0/2370905400 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f523800e040 con 0x7f524c1037a0 2024-03-23T12:05:53.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f5248c00700 1 -- 172.21.15.63:0/2370905400 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5238013420 con 0x7f524c1037a0 2024-03-23T12:05:53.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f5248c00700 1 -- 172.21.15.63:0/2370905400 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f523801c630 con 0x7f524c1037a0 2024-03-23T12:05:53.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524c10fee0 con 0x7f524c1037a0 2024-03-23T12:05:53.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.126+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524c1104a0 con 0x7f524c1037a0 2024-03-23T12:05:53.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.127+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524c0fe550 con 0x7f524c1037a0 2024-03-23T12:05:53.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.128+0000 7f5248c00700 1 -- 172.21.15.63:0/2370905400 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5238013a00 con 0x7f524c1037a0 2024-03-23T12:05:53.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.129+0000 7f5248c00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f523406c7d0 0x7f523406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.129+0000 7f5248c00700 1 -- 172.21.15.63:0/2370905400 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f523808caa0 con 0x7f524c1037a0 2024-03-23T12:05:53.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.130+0000 7f5251000700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f523406c7d0 0x7f523406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.130+0000 7f5251000700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f523406c7d0 0x7f523406ec80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5240004f40 tx=0x7f524000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:53.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.134+0000 7f5248c00700 1 -- 172.21.15.63:0/2370905400 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f523805a870 con 0x7f524c1037a0 2024-03-23T12:05:53.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:52 smithi063 ceph-mon[141124]: pgmap v442: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.365+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f524c109e10 con 0x7f523406c7d0 2024-03-23T12:05:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.366+0000 7f5248c00700 1 -- 172.21.15.63:0/2370905400 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f524c109e10 con 0x7f523406c7d0 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.368+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f523406c7d0 msgr2=0x7f523406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.368+0000 7f5253a00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f523406c7d0 0x7f523406ec80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5240004f40 tx=0x7f524000b040 comp rx=0 tx=0).stop 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.368+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 msgr2=0x7f524c116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.368+0000 7f5253a00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c116240 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5238009390 tx=0x7f5238009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.369+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 shutdown_connections 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.369+0000 7f5253a00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f523406c7d0 0x7f523406ec80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.369+0000 7f5253a00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102e00 0x7f524c115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.369+0000 7f5253a00700 1 --2- 172.21.15.63:0/2370905400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f524c1037a0 0x7f524c116240 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.369+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 >> 172.21.15.63:0/2370905400 conn(0x7f524c0f89b0 msgr2=0x7f524c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:53.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.369+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 shutdown_connections 2024-03-23T12:05:53.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.369+0000 7f5253a00700 1 -- 172.21.15.63:0/2370905400 wait complete. 2024-03-23T12:05:53.365 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:05:53.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:52 smithi111 ceph-mon[106899]: pgmap v442: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:53.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.507+0000 7fb811c00700 1 -- 172.21.15.63:0/2483884443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 msgr2=0x7fb80c070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.507+0000 7fb811c00700 1 --2- 172.21.15.63:0/2483884443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c070360 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc004970 tx=0x7fb7fc009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:53.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.508+0000 7fb811c00700 1 -- 172.21.15.63:0/2483884443 shutdown_connections 2024-03-23T12:05:53.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.508+0000 7fb811c00700 1 --2- 172.21.15.63:0/2483884443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb80c0708a0 0x7fb80c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.508+0000 7fb811c00700 1 --2- 172.21.15.63:0/2483884443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c070360 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.508+0000 7fb811c00700 1 -- 172.21.15.63:0/2483884443 >> 172.21.15.63:0/2483884443 conn(0x7fb80c0f89b0 msgr2=0x7fb80c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:53.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.508+0000 7fb811c00700 1 -- 172.21.15.63:0/2483884443 shutdown_connections 2024-03-23T12:05:53.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.508+0000 7fb811c00700 1 -- 172.21.15.63:0/2483884443 wait complete. 2024-03-23T12:05:53.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb811c00700 1 Processor -- start 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb811c00700 1 -- start start 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb811c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb80c0708a0 0x7fb80c11a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb811c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c11a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb811c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c11aba0 con 0x7fb80c071f90 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb811c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c114020 con 0x7fb80c0708a0 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb80a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c11a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb80a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c11a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34042/0 (socket says 172.21.15.63:34042) 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb80a000700 1 -- 172.21.15.63:0/3299795679 learned_addr learned my addr 172.21.15.63:0/3299795679 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:53.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.509+0000 7fb80aa00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb80c0708a0 0x7fb80c11a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb80a000700 1 -- 172.21.15.63:0/3299795679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb80c0708a0 msgr2=0x7fb80c11a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb80a000700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb80c0708a0 0x7fb80c11a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb80a000700 1 -- 172.21.15.63:0/3299795679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc005040 con 0x7fb80c071f90 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb80a000700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c11a550 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c8c0 tx=0x7fb7f80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb802a00700 1 -- 172.21.15.63:0/3299795679 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f800a760 con 0x7fb80c071f90 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c114300 con 0x7fb80c071f90 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb802a00700 1 -- 172.21.15.63:0/3299795679 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7f800ad70 con 0x7fb80c071f90 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb802a00700 1 -- 172.21.15.63:0/3299795679 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80113a0 con 0x7fb80c071f90 2024-03-23T12:05:53.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.510+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c114840 con 0x7fb80c071f90 2024-03-23T12:05:53.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.511+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c071aa0 con 0x7fb80c071f90 2024-03-23T12:05:53.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.512+0000 7fb802a00700 1 -- 172.21.15.63:0/3299795679 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb7f8012400 con 0x7fb80c071f90 2024-03-23T12:05:53.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.513+0000 7fb802a00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7f006c7d0 0x7fb7f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.513+0000 7fb802a00700 1 -- 172.21.15.63:0/3299795679 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb7f808bac0 con 0x7fb80c071f90 2024-03-23T12:05:53.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.514+0000 7fb80aa00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7f006c7d0 0x7fb7f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.514+0000 7fb80aa00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7f006c7d0 0x7fb7f006ec80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009fd0 tx=0x7fb7fc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:53.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.518+0000 7fb802a00700 1 -- 172.21.15.63:0/3299795679 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7f8055d40 con 0x7fb80c071f90 2024-03-23T12:05:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.750+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb80c1103e0 con 0x7fb7f006c7d0 2024-03-23T12:05:53.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.752+0000 7fb802a00700 1 -- 172.21.15.63:0/3299795679 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb80c1103e0 con 0x7fb7f006c7d0 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7f006c7d0 msgr2=0x7fb7f006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7f006c7d0 0x7fb7f006ec80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009fd0 tx=0x7fb7fc009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 msgr2=0x7fb80c11a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c11a550 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c8c0 tx=0x7fb7f80044c0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 shutdown_connections 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb7f006c7d0 0x7fb7f006ec80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb80c0708a0 0x7fb80c11a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 --2- 172.21.15.63:0/3299795679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb80c071f90 0x7fb80c11a550 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 >> 172.21.15.63:0/3299795679 conn(0x7fb80c0f89b0 msgr2=0x7fb80c0f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 shutdown_connections 2024-03-23T12:05:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.754+0000 7fb811c00700 1 -- 172.21.15.63:0/3299795679 wait complete. 2024-03-23T12:05:53.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 -- 172.21.15.63:0/1558940121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99501037a0 msgr2=0x7f99500fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 --2- 172.21.15.63:0/1558940121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99501037a0 0x7f99500fce10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f993c004940 tx=0x7f993c009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:53.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 -- 172.21.15.63:0/1558940121 shutdown_connections 2024-03-23T12:05:53.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 --2- 172.21.15.63:0/1558940121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99501037a0 0x7f99500fce10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 --2- 172.21.15.63:0/1558940121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9950102e00 0x7f99501031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 -- 172.21.15.63:0/1558940121 >> 172.21.15.63:0/1558940121 conn(0x7f99500f89b0 msgr2=0x7f99500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:53.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 -- 172.21.15.63:0/1558940121 shutdown_connections 2024-03-23T12:05:53.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.893+0000 7f9955400700 1 -- 172.21.15.63:0/1558940121 wait complete. 2024-03-23T12:05:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.894+0000 7f9955400700 1 Processor -- start 2024-03-23T12:05:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.894+0000 7f9955400700 1 -- start start 2024-03-23T12:05:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.894+0000 7f9955400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9950102e00 0x7f9950113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.894+0000 7f9955400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99501037a0 0x7f9950114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.894+0000 7f9955400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9950114690 con 0x7f9950102e00 2024-03-23T12:05:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.894+0000 7f9955400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995010da80 con 0x7f99501037a0 2024-03-23T12:05:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9950102e00 0x7f9950113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9950102e00 0x7f9950113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34054/0 (socket says 172.21.15.63:34054) 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994e800700 1 -- 172.21.15.63:0/1634788414 learned_addr learned my addr 172.21.15.63:0/1634788414 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994de00700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99501037a0 0x7f9950114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994e800700 1 -- 172.21.15.63:0/1634788414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99501037a0 msgr2=0x7f9950114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994e800700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99501037a0 0x7f9950114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994e800700 1 -- 172.21.15.63:0/1634788414 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c005040 con 0x7f9950102e00 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f994e800700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9950102e00 0x7f9950113b00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f994000c8c0 tx=0x7f99400044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.895+0000 7f9946a00700 1 -- 172.21.15.63:0/1634788414 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994000a980 con 0x7f9950102e00 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.896+0000 7f9946a00700 1 -- 172.21.15.63:0/1634788414 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f994000ce60 con 0x7f9950102e00 2024-03-23T12:05:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.896+0000 7f9946a00700 1 -- 172.21.15.63:0/1634788414 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99400113a0 con 0x7f9950102e00 2024-03-23T12:05:53.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.896+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995010dd60 con 0x7f9950102e00 2024-03-23T12:05:53.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.896+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995010e2f0 con 0x7f9950102e00 2024-03-23T12:05:53.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.897+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99500fe550 con 0x7f9950102e00 2024-03-23T12:05:53.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.898+0000 7f9946a00700 1 -- 172.21.15.63:0/1634788414 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9940012400 con 0x7f9950102e00 2024-03-23T12:05:53.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.899+0000 7f9946a00700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f993406c7d0 0x7f993406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:53.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.899+0000 7f9946a00700 1 -- 172.21.15.63:0/1634788414 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f994008bb30 con 0x7f9950102e00 2024-03-23T12:05:53.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.899+0000 7f994de00700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f993406c7d0 0x7f993406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:53.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.900+0000 7f994de00700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f993406c7d0 0x7f993406ec80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f995010ed60 tx=0x7f993c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:53.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:53.904+0000 7f9946a00700 1 -- 172.21.15.63:0/1634788414 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9940059840 con 0x7f9950102e00 2024-03-23T12:05:54.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.126+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9950109da0 con 0x7f993406c7d0 2024-03-23T12:05:54.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.136+0000 7f9946a00700 1 -- 172.21.15.63:0/1634788414 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f9950109da0 con 0x7f993406c7d0 2024-03-23T12:05:54.118 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (15m) 7m ago 28m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (12m) 7m ago 28m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (12m) 7m ago 27m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (14m) 7m ago 28m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (8m) 7m ago 22m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (7m) 7m ago 21m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (21m) 7m ago 21m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (7m) 7m ago 21m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:05:54.119 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (17m) 7m ago 29m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (16m) 7m ago 27m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (13m) 7m ago 29m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (13m) 7m ago 27m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (16m) 7m ago 28m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (15m) 7m ago 27m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (11m) 7m ago 26m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (11m) 7m ago 25m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (10m) 7m ago 25m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (9m) 7m ago 24m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:05:54.120 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (9m) 7m ago 24m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:05:54.121 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (8m) 7m ago 23m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:05:54.121 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (15m) 7m ago 27m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f993406c7d0 msgr2=0x7f993406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f993406c7d0 0x7f993406ec80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f995010ed60 tx=0x7f993c009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9950102e00 msgr2=0x7f9950113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9950102e00 0x7f9950113b00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f994000c8c0 tx=0x7f99400044c0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 shutdown_connections 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f993406c7d0 0x7f993406ec80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9950102e00 0x7f9950113b00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 --2- 172.21.15.63:0/1634788414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99501037a0 0x7f9950114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 >> 172.21.15.63:0/1634788414 conn(0x7f99500f89b0 msgr2=0x7f99500f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:54.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 shutdown_connections 2024-03-23T12:05:54.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.139+0000 7f9955400700 1 -- 172.21.15.63:0/1634788414 wait complete. 2024-03-23T12:05:54.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 -- 172.21.15.63:0/153819419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 msgr2=0x7fee100fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 --2- 172.21.15.63:0/153819419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee100fce10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fedfc004940 tx=0x7fedfc009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:54.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 -- 172.21.15.63:0/153819419 shutdown_connections 2024-03-23T12:05:54.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 --2- 172.21.15.63:0/153819419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee100fce10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 --2- 172.21.15.63:0/153819419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10102e00 0x7fee101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 -- 172.21.15.63:0/153819419 >> 172.21.15.63:0/153819419 conn(0x7fee100f89b0 msgr2=0x7fee100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:54.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 -- 172.21.15.63:0/153819419 shutdown_connections 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.275+0000 7fee16000700 1 -- 172.21.15.63:0/153819419 wait complete. 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.276+0000 7fee16000700 1 Processor -- start 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.276+0000 7fee16000700 1 -- start start 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.276+0000 7fee16000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10102e00 0x7fee10113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.276+0000 7fee16000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee10114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.276+0000 7fee16000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee10114690 con 0x7fee101037a0 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.276+0000 7fee16000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1010da80 con 0x7fee10102e00 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.277+0000 7fee0ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee10114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:54.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.277+0000 7fee0ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee10114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34086/0 (socket says 172.21.15.63:34086) 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.277+0000 7fee0ea00700 1 -- 172.21.15.63:0/3930922925 learned_addr learned my addr 172.21.15.63:0/3930922925 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.277+0000 7fee0f400700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10102e00 0x7fee10113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.277+0000 7fee0ea00700 1 -- 172.21.15.63:0/3930922925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10102e00 msgr2=0x7fee10113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.277+0000 7fee0ea00700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10102e00 0x7fee10113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.277+0000 7fee0ea00700 1 -- 172.21.15.63:0/3930922925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc005040 con 0x7fee101037a0 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.278+0000 7fee0ea00700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee10114040 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009390 tx=0x7fedfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.278+0000 7fee07400700 1 -- 172.21.15.63:0/3930922925 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc00e040 con 0x7fee101037a0 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.278+0000 7fee07400700 1 -- 172.21.15.63:0/3930922925 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedfc013420 con 0x7fee101037a0 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.278+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1010dd00 con 0x7fee101037a0 2024-03-23T12:05:54.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.278+0000 7fee07400700 1 -- 172.21.15.63:0/3930922925 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc01c630 con 0x7fee101037a0 2024-03-23T12:05:54.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.278+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1010e2c0 con 0x7fee101037a0 2024-03-23T12:05:54.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.279+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee100fe550 con 0x7fee101037a0 2024-03-23T12:05:54.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.280+0000 7fee07400700 1 -- 172.21.15.63:0/3930922925 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fedfc013a00 con 0x7fee101037a0 2024-03-23T12:05:54.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.281+0000 7fee07400700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf406c7d0 0x7fedf406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:54.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.282+0000 7fee07400700 1 -- 172.21.15.63:0/3930922925 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fedfc08cdd0 con 0x7fee101037a0 2024-03-23T12:05:54.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.282+0000 7fee0f400700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf406c7d0 0x7fedf406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:54.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.283+0000 7fee0f400700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf406c7d0 0x7fedf406ec80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fee00004990 tx=0x7fee00009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:54.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.286+0000 7fee07400700 1 -- 172.21.15.63:0/3930922925 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fedfc05aa60 con 0x7fee101037a0 2024-03-23T12:05:54.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.587+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fee10109da0 con 0x7fee101037a0 2024-03-23T12:05:54.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.588+0000 7fee07400700 1 -- 172.21.15.63:0/3930922925 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fedfc05a5f0 con 0x7fee101037a0 2024-03-23T12:05:54.570 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:05:54.570 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:05:54.570 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:05:54.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:05:54.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:05:54.572 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:05:54.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:05:54.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:05:54.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:05:54.572 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:05:54.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf406c7d0 msgr2=0x7fedf406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf406c7d0 0x7fedf406ec80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fee00004990 tx=0x7fee00009ee0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 msgr2=0x7fee10114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee10114040 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009390 tx=0x7fedfc009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:54.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 shutdown_connections 2024-03-23T12:05:54.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedf406c7d0 0x7fedf406ec80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee10102e00 0x7fee10113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 --2- 172.21.15.63:0/3930922925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee101037a0 0x7fee10114040 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.590+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 >> 172.21.15.63:0/3930922925 conn(0x7fee100f89b0 msgr2=0x7fee100f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:54.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.591+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 shutdown_connections 2024-03-23T12:05:54.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.591+0000 7fee16000700 1 -- 172.21.15.63:0/3930922925 wait complete. 2024-03-23T12:05:54.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 -- 172.21.15.63:0/153138244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 msgr2=0x7f7404106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 --2- 172.21.15.63:0/153138244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f7404106f00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f73f0004940 tx=0x7f73f0009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 -- 172.21.15.63:0/153138244 shutdown_connections 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 --2- 172.21.15.63:0/153138244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f7404106f00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 --2- 172.21.15.63:0/153138244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74040fde90 0x7f74040fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 -- 172.21.15.63:0/153138244 >> 172.21.15.63:0/153138244 conn(0x7f7404072bc0 msgr2=0x7f7404072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 -- 172.21.15.63:0/153138244 shutdown_connections 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.728+0000 7f740a800700 1 -- 172.21.15.63:0/153138244 wait complete. 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f740a800700 1 Processor -- start 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f740a800700 1 -- start start 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f740a800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74040fde90 0x7f7404115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f740a800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f74041161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:54.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f740a800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7404116810 con 0x7f74040fe830 2024-03-23T12:05:54.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f740a800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740410fc90 con 0x7f74040fde90 2024-03-23T12:05:54.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f7402a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f74041161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:54.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.729+0000 7f7402a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f74041161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34098/0 (socket says 172.21.15.63:34098) 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f7402a00700 1 -- 172.21.15.63:0/3097522456 learned_addr learned my addr 172.21.15.63:0/3097522456 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f7403400700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74040fde90 0x7f7404115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f7402a00700 1 -- 172.21.15.63:0/3097522456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74040fde90 msgr2=0x7f7404115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f7402a00700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74040fde90 0x7f7404115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f7402a00700 1 -- 172.21.15.63:0/3097522456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f0005040 con 0x7f74040fe830 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f7402a00700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f74041161c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000c00 tx=0x7f73f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f73fb400700 1 -- 172.21.15.63:0/3097522456 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73f000e040 con 0x7f74040fe830 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740410ff10 con 0x7f74040fe830 2024-03-23T12:05:54.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.730+0000 7f73fb400700 1 -- 172.21.15.63:0/3097522456 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73f000bed0 con 0x7f74040fe830 2024-03-23T12:05:54.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.731+0000 7f73fb400700 1 -- 172.21.15.63:0/3097522456 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73f001c3f0 con 0x7f74040fe830 2024-03-23T12:05:54.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.731+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7404110450 con 0x7f74040fe830 2024-03-23T12:05:54.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.732+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7404106a20 con 0x7f74040fe830 2024-03-23T12:05:54.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.733+0000 7f73fb400700 1 -- 172.21.15.63:0/3097522456 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f73f0013420 con 0x7f74040fe830 2024-03-23T12:05:54.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.733+0000 7f73fb400700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f73e806c7d0 0x7f73e806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:54.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.734+0000 7f73fb400700 1 -- 172.21.15.63:0/3097522456 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f73f008cf40 con 0x7f74040fe830 2024-03-23T12:05:54.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.734+0000 7f7403400700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f73e806c7d0 0x7f73e806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:54.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.734+0000 7f7403400700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f73e806c7d0 0x7f73e806ec80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f73f4004f40 tx=0x7f73f400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:54.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.738+0000 7f73fb400700 1 -- 172.21.15.63:0/3097522456 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f73f0057140 con 0x7f74040fe830 2024-03-23T12:05:54.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.999+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7404111380 con 0x7f74040fe830 2024-03-23T12:05:54.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:54.999+0000 7f73fb400700 1 -- 172.21.15.63:0/3097522456 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f73f005a760 con 0x7f74040fe830 2024-03-23T12:05:54.981 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:05:54.981 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:05:54.981 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:05:54.981 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:05:54.981 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:05:54.982 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:05:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:54.984 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:05:54.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f73e806c7d0 msgr2=0x7f73e806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f73e806c7d0 0x7f73e806ec80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f73f4004f40 tx=0x7f73f400b040 comp rx=0 tx=0).stop 2024-03-23T12:05:54.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 msgr2=0x7f74041161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:54.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f74041161c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f73f0000c00 tx=0x7f73f0009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:54.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 shutdown_connections 2024-03-23T12:05:54.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f73e806c7d0 0x7f73e806ec80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74040fde90 0x7f7404115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 --2- 172.21.15.63:0/3097522456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74040fe830 0x7f74041161c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:54.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 >> 172.21.15.63:0/3097522456 conn(0x7f7404072bc0 msgr2=0x7f74040fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:54.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 shutdown_connections 2024-03-23T12:05:54.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.002+0000 7f740a800700 1 -- 172.21.15.63:0/3097522456 wait complete. 2024-03-23T12:05:54.986 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:05:55.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 -- 172.21.15.63:0/155351118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d901037a0 msgr2=0x7f7d900fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 --2- 172.21.15.63:0/155351118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d901037a0 0x7f7d900fce10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c004940 tx=0x7f7d7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:55.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 -- 172.21.15.63:0/155351118 shutdown_connections 2024-03-23T12:05:55.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 --2- 172.21.15.63:0/155351118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d901037a0 0x7f7d900fce10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 --2- 172.21.15.63:0/155351118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d90102e00 0x7f7d901031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 -- 172.21.15.63:0/155351118 >> 172.21.15.63:0/155351118 conn(0x7f7d900f89b0 msgr2=0x7f7d900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:55.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 -- 172.21.15.63:0/155351118 shutdown_connections 2024-03-23T12:05:55.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.142+0000 7f7d97a00700 1 -- 172.21.15.63:0/155351118 wait complete. 2024-03-23T12:05:55.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d97a00700 1 Processor -- start 2024-03-23T12:05:55.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d97a00700 1 -- start start 2024-03-23T12:05:55.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d97a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90102e00 0x7f7d90115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:55.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d97a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d901037a0 0x7f7d90116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:55.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d97a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d901168a0 con 0x7f7d90102e00 2024-03-23T12:05:55.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d97a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9010fc70 con 0x7f7d901037a0 2024-03-23T12:05:55.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d95000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90102e00 0x7f7d90115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:55.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d95000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90102e00 0x7f7d90115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34124/0 (socket says 172.21.15.63:34124) 2024-03-23T12:05:55.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.143+0000 7f7d95000700 1 -- 172.21.15.63:0/29410657 learned_addr learned my addr 172.21.15.63:0/29410657 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:55.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d8fe00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d901037a0 0x7f7d90116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:55.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d95000700 1 -- 172.21.15.63:0/29410657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d901037a0 msgr2=0x7f7d90116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d95000700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d901037a0 0x7f7d90116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d95000700 1 -- 172.21.15.63:0/29410657 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c005040 con 0x7f7d90102e00 2024-03-23T12:05:55.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d95000700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90102e00 0x7f7d90115d10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400eb70 tx=0x7f7d8400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:55.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d8cc00700 1 -- 172.21.15.63:0/29410657 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d84007cc0 con 0x7f7d90102e00 2024-03-23T12:05:55.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d8cc00700 1 -- 172.21.15.63:0/29410657 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d8400f040 con 0x7f7d90102e00 2024-03-23T12:05:55.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9010fef0 con 0x7f7d90102e00 2024-03-23T12:05:55.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.144+0000 7f7d8cc00700 1 -- 172.21.15.63:0/29410657 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8400c9f0 con 0x7f7d90102e00 2024-03-23T12:05:55.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.145+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d90110430 con 0x7f7d90102e00 2024-03-23T12:05:55.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.146+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d900fe550 con 0x7f7d90102e00 2024-03-23T12:05:55.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.146+0000 7f7d8cc00700 1 -- 172.21.15.63:0/29410657 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7d8400cb50 con 0x7f7d90102e00 2024-03-23T12:05:55.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.147+0000 7f7d8cc00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d7806c7d0 0x7f7d7806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:55.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.147+0000 7f7d8cc00700 1 -- 172.21.15.63:0/29410657 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d8408b660 con 0x7f7d90102e00 2024-03-23T12:05:55.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.148+0000 7f7d8fe00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d7806c7d0 0x7f7d7806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:55.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.148+0000 7f7d8fe00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d7806c7d0 0x7f7d7806ec80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7d901115b0 tx=0x7f7d7c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:55.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.152+0000 7f7d8cc00700 1 -- 172.21.15.63:0/29410657 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d840592f0 con 0x7f7d90102e00 2024-03-23T12:05:55.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.386+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d9004bf40 con 0x7f7d7806c7d0 2024-03-23T12:05:55.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.388+0000 7f7d8cc00700 1 -- 172.21.15.63:0/29410657 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7d9004bf40 con 0x7f7d7806c7d0 2024-03-23T12:05:55.369 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:05:55.369 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:05:55.369 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:05:55.370 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d7806c7d0 msgr2=0x7f7d7806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d7806c7d0 0x7f7d7806ec80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7d901115b0 tx=0x7f7d7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90102e00 msgr2=0x7f7d90115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90102e00 0x7f7d90115d10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400eb70 tx=0x7f7d8400b040 comp rx=0 tx=0).stop 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 shutdown_connections 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7d7806c7d0 0x7f7d7806ec80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90102e00 0x7f7d90115d10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 --2- 172.21.15.63:0/29410657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d901037a0 0x7f7d90116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 >> 172.21.15.63:0/29410657 conn(0x7f7d900f89b0 msgr2=0x7f7d900f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:55.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 shutdown_connections 2024-03-23T12:05:55.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.390+0000 7f7d97a00700 1 -- 172.21.15.63:0/29410657 wait complete. 2024-03-23T12:05:55.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:54 smithi063 ceph-mon[141124]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:55.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:54 smithi063 ceph-mon[141124]: from='client.34858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:55.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:54 smithi063 ceph-mon[141124]: pgmap v443: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:55.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:54 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3930922925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:05:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:54 smithi111 ceph-mon[106899]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:54 smithi111 ceph-mon[106899]: from='client.34858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:54 smithi111 ceph-mon[106899]: pgmap v443: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:54 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3930922925' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 -- 172.21.15.63:0/2572787093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 msgr2=0x7fa4201031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 --2- 172.21.15.63:0/2572787093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa4201031f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa410004940 tx=0x7fa410009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 -- 172.21.15.63:0/2572787093 shutdown_connections 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 --2- 172.21.15.63:0/2572787093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4201037c0 0x7fa4200fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 --2- 172.21.15.63:0/2572787093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa4201031f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 -- 172.21.15.63:0/2572787093 >> 172.21.15.63:0/2572787093 conn(0x7fa4200f89d0 msgr2=0x7fa4200fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 -- 172.21.15.63:0/2572787093 shutdown_connections 2024-03-23T12:05:55.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.528+0000 7fa426400700 1 -- 172.21.15.63:0/2572787093 wait complete. 2024-03-23T12:05:55.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.529+0000 7fa426400700 1 Processor -- start 2024-03-23T12:05:55.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.529+0000 7fa426400700 1 -- start start 2024-03-23T12:05:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.530+0000 7fa426400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa420115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.530+0000 7fa426400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4201037c0 0x7fa4201161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.530+0000 7fa426400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa420116820 con 0x7fa420102e20 2024-03-23T12:05:55.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.530+0000 7fa426400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42010fbf0 con 0x7fa4201037c0 2024-03-23T12:05:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.531+0000 7fa41f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa420115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.531+0000 7fa41f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa420115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34134/0 (socket says 172.21.15.63:34134) 2024-03-23T12:05:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.531+0000 7fa41f400700 1 -- 172.21.15.63:0/120797378 learned_addr learned my addr 172.21.15.63:0/120797378 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:05:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.531+0000 7fa417e00700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4201037c0 0x7fa4201161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.531+0000 7fa41f400700 1 -- 172.21.15.63:0/120797378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4201037c0 msgr2=0x7fa4201161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.531+0000 7fa41f400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4201037c0 0x7fa4201161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.531+0000 7fa41f400700 1 -- 172.21.15.63:0/120797378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa410005040 con 0x7fa420102e20 2024-03-23T12:05:55.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.532+0000 7fa41f400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa420115c90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa410008d80 tx=0x7fa410009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:55.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.532+0000 7fa417400700 1 -- 172.21.15.63:0/120797378 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa41000e030 con 0x7fa420102e20 2024-03-23T12:05:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.532+0000 7fa417400700 1 -- 172.21.15.63:0/120797378 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa410013de0 con 0x7fa420102e20 2024-03-23T12:05:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.532+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42010fe70 con 0x7fa420102e20 2024-03-23T12:05:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.532+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa420110430 con 0x7fa420102e20 2024-03-23T12:05:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.532+0000 7fa417400700 1 -- 172.21.15.63:0/120797378 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa41001cd00 con 0x7fa420102e20 2024-03-23T12:05:55.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.533+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc0052b0 con 0x7fa420102e20 2024-03-23T12:05:55.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.534+0000 7fa417400700 1 -- 172.21.15.63:0/120797378 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa410013930 con 0x7fa420102e20 2024-03-23T12:05:55.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.535+0000 7fa417400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa40406c7d0 0x7fa40406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:05:55.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.535+0000 7fa417400700 1 -- 172.21.15.63:0/120797378 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa41008d290 con 0x7fa420102e20 2024-03-23T12:05:55.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.536+0000 7fa417e00700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa40406c7d0 0x7fa40406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:05:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.536+0000 7fa417e00700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa40406c7d0 0x7fa40406ec80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa420111530 tx=0x7fa40800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:05:55.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.540+0000 7fa417400700 1 -- 172.21.15.63:0/120797378 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa41005afa0 con 0x7fa420102e20 2024-03-23T12:05:55.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.838+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa3fc005560 con 0x7fa420102e20 2024-03-23T12:05:55.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.838+0000 7fa417400700 1 -- 172.21.15.63:0/120797378 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa41001ce60 con 0x7fa420102e20 2024-03-23T12:05:55.820 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:05:55.820 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:05:55.820 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:05:55.820 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:05:55.820 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:05:55.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.840+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa40406c7d0 msgr2=0x7fa40406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.840+0000 7fa426400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa40406c7d0 0x7fa40406ec80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa420111530 tx=0x7fa40800b040 comp rx=0 tx=0).stop 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.840+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 msgr2=0x7fa420115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.840+0000 7fa426400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa420115c90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa410008d80 tx=0x7fa410009e30 comp rx=0 tx=0).stop 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.841+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 shutdown_connections 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.841+0000 7fa426400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa40406c7d0 0x7fa40406ec80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.841+0000 7fa426400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa420102e20 0x7fa420115c90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.841+0000 7fa426400700 1 --2- 172.21.15.63:0/120797378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4201037c0 0x7fa4201161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.841+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 >> 172.21.15.63:0/120797378 conn(0x7fa4200f89d0 msgr2=0x7fa4200f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.841+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 shutdown_connections 2024-03-23T12:05:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:05:55.841+0000 7fa426400700 1 -- 172.21.15.63:0/120797378 wait complete. 2024-03-23T12:05:56.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:55 smithi063 ceph-mon[141124]: from='client.34862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:56.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:55 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3097522456' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:05:56.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:56.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:55 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/120797378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:05:56.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:55 smithi111 ceph-mon[106899]: from='client.34862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:56.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:55 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3097522456' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:05:56.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:05:56.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:55 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/120797378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:05:57.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:56 smithi063 ceph-mon[141124]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:57.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:56 smithi063 ceph-mon[141124]: pgmap v444: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:57.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:56 smithi111 ceph-mon[106899]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:05:57.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:56 smithi111 ceph-mon[106899]: pgmap v444: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:59.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:05:58 smithi063 ceph-mon[141124]: pgmap v445: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:05:59.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:05:58 smithi111 ceph-mon[106899]: pgmap v445: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:00 smithi063 ceph-mon[141124]: pgmap v446: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:01.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:00 smithi111 ceph-mon[106899]: pgmap v446: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:03.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:02 smithi063 ceph-mon[141124]: pgmap v447: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:03.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:02 smithi111 ceph-mon[106899]: pgmap v447: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:05.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:04 smithi063 ceph-mon[141124]: pgmap v448: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:05.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:04 smithi111 ceph-mon[106899]: pgmap v448: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:06.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:06 smithi063 ceph-mon[141124]: pgmap v449: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:06.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:06 smithi111 ceph-mon[106899]: pgmap v449: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:09.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:08 smithi063 ceph-mon[141124]: pgmap v450: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:09.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:08 smithi111 ceph-mon[106899]: pgmap v450: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:10 smithi063 ceph-mon[141124]: pgmap v451: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:11.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:10 smithi111 ceph-mon[106899]: pgmap v451: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:11.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:13.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:12 smithi063 ceph-mon[141124]: pgmap v452: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:13.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:12 smithi111 ceph-mon[106899]: pgmap v452: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:15.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:14 smithi063 ceph-mon[141124]: pgmap v453: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:15.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:14 smithi111 ceph-mon[106899]: pgmap v453: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:16.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:16 smithi063 ceph-mon[141124]: pgmap v454: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:16.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:16 smithi111 ceph-mon[106899]: pgmap v454: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:19.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:18 smithi063 ceph-mon[141124]: pgmap v455: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:19.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:18 smithi111 ceph-mon[106899]: pgmap v455: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:21.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:20 smithi063 ceph-mon[141124]: pgmap v456: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:21.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:20 smithi111 ceph-mon[106899]: pgmap v456: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:23.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:22 smithi063 ceph-mon[141124]: pgmap v457: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:22 smithi111 ceph-mon[106899]: pgmap v457: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:25.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:24 smithi063 ceph-mon[141124]: pgmap v458: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:25.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:24 smithi111 ceph-mon[106899]: pgmap v458: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:25.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 -- 172.21.15.63:0/2688043958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f41037a0 msgr2=0x7f07f40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:25.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 --2- 172.21.15.63:0/2688043958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f41037a0 0x7f07f40fce10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07dc004940 tx=0x7f07dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:25.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 -- 172.21.15.63:0/2688043958 shutdown_connections 2024-03-23T12:06:25.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 --2- 172.21.15.63:0/2688043958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f41037a0 0x7f07f40fce10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:25.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 --2- 172.21.15.63:0/2688043958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07f4102e00 0x7f07f41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:25.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 -- 172.21.15.63:0/2688043958 >> 172.21.15.63:0/2688043958 conn(0x7f07f40f89b0 msgr2=0x7f07f40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:25.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 -- 172.21.15.63:0/2688043958 shutdown_connections 2024-03-23T12:06:25.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.996+0000 7f07faa00700 1 -- 172.21.15.63:0/2688043958 wait complete. 2024-03-23T12:06:25.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.997+0000 7f07faa00700 1 Processor -- start 2024-03-23T12:06:25.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.997+0000 7f07faa00700 1 -- start start 2024-03-23T12:06:25.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f4102e00 0x7f07f4115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07faa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07f41037a0 0x7f07f41162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07faa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f4116930 con 0x7f07f4102e00 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07faa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f410fd00 con 0x7f07f41037a0 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f4102e00 0x7f07f4115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f4102e00 0x7f07f4115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35402/0 (socket says 172.21.15.63:35402) 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07f3e00700 1 -- 172.21.15.63:0/1390160269 learned_addr learned my addr 172.21.15.63:0/1390160269 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07ebe00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07f41037a0 0x7f07f41162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07f3e00700 1 -- 172.21.15.63:0/1390160269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07f41037a0 msgr2=0x7f07f41162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07f3e00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07f41037a0 0x7f07f41162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.998+0000 7f07f3e00700 1 -- 172.21.15.63:0/1390160269 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc005040 con 0x7f07f4102e00 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.999+0000 7f07f3e00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f4102e00 0x7f07f4115da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f07e4002a50 tx=0x7f07e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.999+0000 7f07f0c00700 1 -- 172.21.15.63:0/1390160269 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07e4007cc0 con 0x7f07f4102e00 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.999+0000 7f07f0c00700 1 -- 172.21.15.63:0/1390160269 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07e400f040 con 0x7f07f4102e00 2024-03-23T12:06:25.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.999+0000 7f07f0c00700 1 -- 172.21.15.63:0/1390160269 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07e400d9c0 con 0x7f07f4102e00 2024-03-23T12:06:25.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.999+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f410ffe0 con 0x7f07f4102e00 2024-03-23T12:06:25.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:25.999+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f4110520 con 0x7f07f4102e00 2024-03-23T12:06:25.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.000+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f40fe550 con 0x7f07f4102e00 2024-03-23T12:06:25.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.001+0000 7f07f0c00700 1 -- 172.21.15.63:0/1390160269 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f07e400db20 con 0x7f07f4102e00 2024-03-23T12:06:25.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.002+0000 7f07f0c00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d806c7d0 0x7f07d806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:25.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.002+0000 7f07f0c00700 1 -- 172.21.15.63:0/1390160269 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f07e408b760 con 0x7f07f4102e00 2024-03-23T12:06:25.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.002+0000 7f07ebe00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d806c7d0 0x7f07d806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:25.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.003+0000 7f07ebe00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d806c7d0 0x7f07d806ec80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f07f4111640 tx=0x7f07dc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:25.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.007+0000 7f07f0c00700 1 -- 172.21.15.63:0/1390160269 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07e40593f0 con 0x7f07f4102e00 2024-03-23T12:06:26.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.240+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07f404bf40 con 0x7f07d806c7d0 2024-03-23T12:06:26.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.241+0000 7f07f0c00700 1 -- 172.21.15.63:0/1390160269 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f07f404bf40 con 0x7f07d806c7d0 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d806c7d0 msgr2=0x7f07d806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d806c7d0 0x7f07d806ec80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f07f4111640 tx=0x7f07dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f4102e00 msgr2=0x7f07f4115da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f4102e00 0x7f07f4115da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f07e4002a50 tx=0x7f07e40046b0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 shutdown_connections 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d806c7d0 0x7f07d806ec80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07f4102e00 0x7f07f4115da0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 --2- 172.21.15.63:0/1390160269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07f41037a0 0x7f07f41162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.243+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 >> 172.21.15.63:0/1390160269 conn(0x7f07f40f89b0 msgr2=0x7f07f40f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.244+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 shutdown_connections 2024-03-23T12:06:26.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.244+0000 7f07faa00700 1 -- 172.21.15.63:0/1390160269 wait complete. 2024-03-23T12:06:26.239 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:06:26.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 -- 172.21.15.63:0/1939466983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6564102e00 msgr2=0x7f65641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 --2- 172.21.15.63:0/1939466983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6564102e00 0x7f65641031d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6558004940 tx=0x7f6558009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 -- 172.21.15.63:0/1939466983 shutdown_connections 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 --2- 172.21.15.63:0/1939466983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65641037a0 0x7f65640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 --2- 172.21.15.63:0/1939466983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6564102e00 0x7f65641031d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 -- 172.21.15.63:0/1939466983 >> 172.21.15.63:0/1939466983 conn(0x7f65640f89b0 msgr2=0x7f65640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 -- 172.21.15.63:0/1939466983 shutdown_connections 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.382+0000 7f656be00700 1 -- 172.21.15.63:0/1939466983 wait complete. 2024-03-23T12:06:26.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.383+0000 7f656be00700 1 Processor -- start 2024-03-23T12:06:26.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.383+0000 7f656be00700 1 -- start start 2024-03-23T12:06:26.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f656be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f65640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:26.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f656be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f6564070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:26.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f656be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6564070d90 con 0x7f65641037a0 2024-03-23T12:06:26.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f656be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6564070ed0 con 0x7f6564102e00 2024-03-23T12:06:26.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f6568a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f6564070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:26.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f6568a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f6564070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35430/0 (socket says 172.21.15.63:35430) 2024-03-23T12:06:26.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f6568a00700 1 -- 172.21.15.63:0/2142908491 learned_addr learned my addr 172.21.15.63:0/2142908491 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:26.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.384+0000 7f6569400700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f65640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:26.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f6568a00700 1 -- 172.21.15.63:0/2142908491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 msgr2=0x7f65640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f6568a00700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f65640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f6568a00700 1 -- 172.21.15.63:0/2142908491 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6558005040 con 0x7f65641037a0 2024-03-23T12:06:26.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f6568a00700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f6564070800 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f655000eb40 tx=0x7f655000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:26.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f6561600700 1 -- 172.21.15.63:0/2142908491 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6550007cc0 con 0x7f65641037a0 2024-03-23T12:06:26.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f6561600700 1 -- 172.21.15.63:0/2142908491 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f655000f040 con 0x7f65641037a0 2024-03-23T12:06:26.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6564071120 con 0x7f65641037a0 2024-03-23T12:06:26.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f6561600700 1 -- 172.21.15.63:0/2142908491 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f655000c8f0 con 0x7f65641037a0 2024-03-23T12:06:26.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.385+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656411a170 con 0x7f65641037a0 2024-03-23T12:06:26.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.386+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65640fe550 con 0x7f65641037a0 2024-03-23T12:06:26.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.387+0000 7f6561600700 1 -- 172.21.15.63:0/2142908491 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f65500090d0 con 0x7f65641037a0 2024-03-23T12:06:26.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.388+0000 7f6561600700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654c06c7d0 0x7f654c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:26.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.388+0000 7f6561600700 1 -- 172.21.15.63:0/2142908491 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f655008b470 con 0x7f65641037a0 2024-03-23T12:06:26.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.389+0000 7f6569400700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654c06c7d0 0x7f654c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:26.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.389+0000 7f6569400700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654c06c7d0 0x7f654c06ec80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f65580045a0 tx=0x7f6558009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:26.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.393+0000 7f6561600700 1 -- 172.21.15.63:0/2142908491 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6550055720 con 0x7f65641037a0 2024-03-23T12:06:26.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.626+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f656404bed0 con 0x7f654c06c7d0 2024-03-23T12:06:26.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.627+0000 7f6561600700 1 -- 172.21.15.63:0/2142908491 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f656404bed0 con 0x7f654c06c7d0 2024-03-23T12:06:26.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654c06c7d0 msgr2=0x7f654c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654c06c7d0 0x7f654c06ec80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f65580045a0 tx=0x7f6558009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:26.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 msgr2=0x7f6564070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f6564070800 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f655000eb40 tx=0x7f655000b040 comp rx=0 tx=0).stop 2024-03-23T12:06:26.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 shutdown_connections 2024-03-23T12:06:26.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654c06c7d0 0x7f654c06ec80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f65640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 --2- 172.21.15.63:0/2142908491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f6564070800 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 >> 172.21.15.63:0/2142908491 conn(0x7f65640f89b0 msgr2=0x7f65640f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:26.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 shutdown_connections 2024-03-23T12:06:26.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.630+0000 7f656be00700 1 -- 172.21.15.63:0/2142908491 wait complete. 2024-03-23T12:06:26.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.768+0000 7f2dbc800700 1 -- 172.21.15.63:0/890576855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 msgr2=0x7f2db41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.768+0000 7f2dbc800700 1 --2- 172.21.15.63:0/890576855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db41031d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2da8004940 tx=0x7f2da8009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:26.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.768+0000 7f2dbc800700 1 -- 172.21.15.63:0/890576855 shutdown_connections 2024-03-23T12:06:26.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.768+0000 7f2dbc800700 1 --2- 172.21.15.63:0/890576855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db41037a0 0x7f2db40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.768+0000 7f2dbc800700 1 --2- 172.21.15.63:0/890576855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db41031d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.769+0000 7f2dbc800700 1 -- 172.21.15.63:0/890576855 >> 172.21.15.63:0/890576855 conn(0x7f2db40f89b0 msgr2=0x7f2db40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:26.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.769+0000 7f2dbc800700 1 -- 172.21.15.63:0/890576855 shutdown_connections 2024-03-23T12:06:26.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.769+0000 7f2dbc800700 1 -- 172.21.15.63:0/890576855 wait complete. 2024-03-23T12:06:26.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.770+0000 7f2dbc800700 1 Processor -- start 2024-03-23T12:06:26.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.770+0000 7f2dbc800700 1 -- start start 2024-03-23T12:06:26.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2dbc800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db4115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:26.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2dbc800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db41037a0 0x7f2db4116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:26.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2dbc800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db41168e0 con 0x7f2db4102e00 2024-03-23T12:06:26.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2dbc800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db410fc20 con 0x7f2db41037a0 2024-03-23T12:06:26.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db4115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:26.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db4115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35460/0 (socket says 172.21.15.63:35460) 2024-03-23T12:06:26.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9e00700 1 -- 172.21.15.63:0/2517875480 learned_addr learned my addr 172.21.15.63:0/2517875480 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:26.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9400700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db41037a0 0x7f2db4116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:26.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9e00700 1 -- 172.21.15.63:0/2517875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db41037a0 msgr2=0x7f2db4116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9e00700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db41037a0 0x7f2db4116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9e00700 1 -- 172.21.15.63:0/2517875480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8005040 con 0x7f2db4102e00 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9400700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db41037a0 0x7f2db4116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.771+0000 7f2db9e00700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db4115cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009390 tx=0x7f2da8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.772+0000 7f2dae000700 1 -- 172.21.15.63:0/2517875480 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2da800e040 con 0x7f2db4102e00 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.772+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db410fe40 con 0x7f2db4102e00 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.772+0000 7f2dae000700 1 -- 172.21.15.63:0/2517875480 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2da8013420 con 0x7f2db4102e00 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.772+0000 7f2dae000700 1 -- 172.21.15.63:0/2517875480 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2da801c630 con 0x7f2db4102e00 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.772+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db4110400 con 0x7f2db4102e00 2024-03-23T12:06:26.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.773+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db40fe550 con 0x7f2db4102e00 2024-03-23T12:06:26.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.774+0000 7f2dae000700 1 -- 172.21.15.63:0/2517875480 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2da80139e0 con 0x7f2db4102e00 2024-03-23T12:06:26.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.775+0000 7f2dae000700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d9c06c7d0 0x7f2d9c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:26.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.775+0000 7f2dae000700 1 -- 172.21.15.63:0/2517875480 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2da808cce0 con 0x7f2db4102e00 2024-03-23T12:06:26.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.775+0000 7f2db9400700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d9c06c7d0 0x7f2d9c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:26.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.776+0000 7f2db9400700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d9c06c7d0 0x7f2d9c06ec80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2da4006fd0 tx=0x7f2da4008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:26.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:26.780+0000 7f2dae000700 1 -- 172.21.15.63:0/2517875480 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2da8091070 con 0x7f2db4102e00 2024-03-23T12:06:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:26 smithi063 ceph-mon[141124]: pgmap v459: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:26.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:26.902 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:26 smithi111 ceph-mon[106899]: pgmap v459: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:26.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.004+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2db404bf70 con 0x7f2d9c06c7d0 2024-03-23T12:06:26.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.014+0000 7f2dae000700 1 -- 172.21.15.63:0/2517875480 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f2da400c660 con 0x7f2d9c06c7d0 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (15m) 8m ago 28m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (12m) 8m ago 29m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (12m) 8m ago 27m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (15m) 8m ago 28m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (8m) 8m ago 22m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (8m) 8m ago 22m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (22m) 8m ago 22m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (8m) 8m ago 22m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (18m) 8m ago 30m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:06:26.996 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (16m) 8m ago 27m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (14m) 8m ago 30m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (13m) 8m ago 27m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (16m) 8m ago 29m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (16m) 8m ago 27m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (12m) 8m ago 26m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (11m) 8m ago 26m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (11m) 8m ago 25m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (10m) 8m ago 25m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (9m) 8m ago 24m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (9m) 8m ago 24m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:06:26.997 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (16m) 8m ago 28m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:06:26.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.016+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d9c06c7d0 msgr2=0x7f2d9c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.016+0000 7f2dbc800700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d9c06c7d0 0x7f2d9c06ec80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2da4006fd0 tx=0x7f2da4008040 comp rx=0 tx=0).stop 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.016+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 msgr2=0x7f2db4115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.016+0000 7f2dbc800700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db4115cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009390 tx=0x7f2da8009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.016+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 shutdown_connections 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.016+0000 7f2dbc800700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d9c06c7d0 0x7f2d9c06ec80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.016+0000 7f2dbc800700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2db4102e00 0x7f2db4115cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.017+0000 7f2dbc800700 1 --2- 172.21.15.63:0/2517875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db41037a0 0x7f2db4116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.017+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 >> 172.21.15.63:0/2517875480 conn(0x7f2db40f89b0 msgr2=0x7f2db4071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.017+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 shutdown_connections 2024-03-23T12:06:26.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.017+0000 7f2dbc800700 1 -- 172.21.15.63:0/2517875480 wait complete. 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 -- 172.21.15.63:0/370390054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 msgr2=0x7f176c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 --2- 172.21.15.63:0/370390054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c103010 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f175c004940 tx=0x7f175c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 -- 172.21.15.63:0/370390054 shutdown_connections 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 --2- 172.21.15.63:0/370390054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f176c1035e0 0x7f176c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 --2- 172.21.15.63:0/370390054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c103010 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 -- 172.21.15.63:0/370390054 >> 172.21.15.63:0/370390054 conn(0x7f176c0f8830 msgr2=0x7f176c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 -- 172.21.15.63:0/370390054 shutdown_connections 2024-03-23T12:06:27.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.154+0000 7f1771400700 1 -- 172.21.15.63:0/370390054 wait complete. 2024-03-23T12:06:27.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.155+0000 7f1771400700 1 Processor -- start 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.155+0000 7f1771400700 1 -- start start 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.155+0000 7f1771400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.155+0000 7f1771400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f176c1035e0 0x7f176c116070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.155+0000 7f1771400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c1166c0 con 0x7f176c102c40 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.155+0000 7f1771400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c10fab0 con 0x7f176c1035e0 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f176be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f176be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c115b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35480/0 (socket says 172.21.15.63:35480) 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f176be00700 1 -- 172.21.15.63:0/3517408048 learned_addr learned my addr 172.21.15.63:0/3517408048 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f1763e00700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f176c1035e0 0x7f176c116070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f176be00700 1 -- 172.21.15.63:0/3517408048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f176c1035e0 msgr2=0x7f176c116070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f176be00700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f176c1035e0 0x7f176c116070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f176be00700 1 -- 172.21.15.63:0/3517408048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c005040 con 0x7f176c102c40 2024-03-23T12:06:27.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.156+0000 7f176be00700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c115b30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f175c00b3b0 tx=0x7f175c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:27.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.157+0000 7f1768c00700 1 -- 172.21.15.63:0/3517408048 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f175c00e040 con 0x7f176c102c40 2024-03-23T12:06:27.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.157+0000 7f1768c00700 1 -- 172.21.15.63:0/3517408048 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f175c013420 con 0x7f176c102c40 2024-03-23T12:06:27.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.157+0000 7f1768c00700 1 -- 172.21.15.63:0/3517408048 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f175c01c630 con 0x7f176c102c40 2024-03-23T12:06:27.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.157+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c10fd30 con 0x7f176c102c40 2024-03-23T12:06:27.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.157+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c1102a0 con 0x7f176c102c40 2024-03-23T12:06:27.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.158+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c0fe390 con 0x7f176c102c40 2024-03-23T12:06:27.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.159+0000 7f1768c00700 1 -- 172.21.15.63:0/3517408048 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f175c013970 con 0x7f176c102c40 2024-03-23T12:06:27.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.160+0000 7f1768c00700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f175006c7c0 0x7f175006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:27.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.160+0000 7f1768c00700 1 -- 172.21.15.63:0/3517408048 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f175c08cce0 con 0x7f176c102c40 2024-03-23T12:06:27.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.160+0000 7f1763e00700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f175006c7c0 0x7f175006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:27.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.161+0000 7f1763e00700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f175006c7c0 0x7f175006ec70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f176c110c70 tx=0x7f1754009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:27.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.164+0000 7f1768c00700 1 -- 172.21.15.63:0/3517408048 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f175c091070 con 0x7f176c102c40 2024-03-23T12:06:27.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.468+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f176c109c40 con 0x7f176c102c40 2024-03-23T12:06:27.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.469+0000 7f1768c00700 1 -- 172.21.15.63:0/3517408048 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f175c05a9f0 con 0x7f176c102c40 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:06:27.451 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:06:27.452 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:06:27.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.471+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f175006c7c0 msgr2=0x7f175006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.471+0000 7f1771400700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f175006c7c0 0x7f175006ec70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f176c110c70 tx=0x7f1754009ee0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.471+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 msgr2=0x7f176c115b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.471+0000 7f1771400700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c115b30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f175c00b3b0 tx=0x7f175c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:27.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.472+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 shutdown_connections 2024-03-23T12:06:27.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.472+0000 7f1771400700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f175006c7c0 0x7f175006ec70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.472+0000 7f1771400700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c102c40 0x7f176c115b30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.472+0000 7f1771400700 1 --2- 172.21.15.63:0/3517408048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f176c1035e0 0x7f176c116070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.472+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 >> 172.21.15.63:0/3517408048 conn(0x7f176c0f8830 msgr2=0x7f176c0706b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:27.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.472+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 shutdown_connections 2024-03-23T12:06:27.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.472+0000 7f1771400700 1 -- 172.21.15.63:0/3517408048 wait complete. 2024-03-23T12:06:27.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 -- 172.21.15.63:0/787648621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 msgr2=0x7ff6681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 --2- 172.21.15.63:0/787648621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff6681031d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff65c004940 tx=0x7ff65c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:27.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 -- 172.21.15.63:0/787648621 shutdown_connections 2024-03-23T12:06:27.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 --2- 172.21.15.63:0/787648621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6681037a0 0x7ff6680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 --2- 172.21.15.63:0/787648621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff6681031d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 -- 172.21.15.63:0/787648621 >> 172.21.15.63:0/787648621 conn(0x7ff6680f89b0 msgr2=0x7ff6680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:27.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 -- 172.21.15.63:0/787648621 shutdown_connections 2024-03-23T12:06:27.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.611+0000 7ff670800700 1 -- 172.21.15.63:0/787648621 wait complete. 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff670800700 1 Processor -- start 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff670800700 1 -- start start 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff670800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff668113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff670800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6681037a0 0x7ff668113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff670800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681146c0 con 0x7ff668102e00 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff670800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66810da20 con 0x7ff6681037a0 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff66de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff668113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.612+0000 7ff66de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff668113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35486/0 (socket says 172.21.15.63:35486) 2024-03-23T12:06:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff66de00700 1 -- 172.21.15.63:0/3202167399 learned_addr learned my addr 172.21.15.63:0/3202167399 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff66de00700 1 -- 172.21.15.63:0/3202167399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6681037a0 msgr2=0x7ff668113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff66de00700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6681037a0 0x7ff668113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff66de00700 1 -- 172.21.15.63:0/3202167399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c005040 con 0x7ff668102e00 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff66de00700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff668113aa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009390 tx=0x7ff65c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff662000700 1 -- 172.21.15.63:0/3202167399 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65c00e040 con 0x7ff668102e00 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66810dca0 con 0x7ff668102e00 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff662000700 1 -- 172.21.15.63:0/3202167399 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff65c013420 con 0x7ff668102e00 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.613+0000 7ff662000700 1 -- 172.21.15.63:0/3202167399 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65c01c630 con 0x7ff668102e00 2024-03-23T12:06:27.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.614+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66810e260 con 0x7ff668102e00 2024-03-23T12:06:27.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.615+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6680fe550 con 0x7ff668102e00 2024-03-23T12:06:27.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.616+0000 7ff662000700 1 -- 172.21.15.63:0/3202167399 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff65c0139e0 con 0x7ff668102e00 2024-03-23T12:06:27.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.616+0000 7ff662000700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff65006c7d0 0x7ff65006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:27.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.617+0000 7ff662000700 1 -- 172.21.15.63:0/3202167399 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff65c08cce0 con 0x7ff668102e00 2024-03-23T12:06:27.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.617+0000 7ff66d400700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff65006c7d0 0x7ff65006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:27.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.617+0000 7ff66d400700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff65006c7d0 0x7ff65006ec80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff658006fd0 tx=0x7ff658006db0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:27.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.621+0000 7ff662000700 1 -- 172.21.15.63:0/3202167399 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff65c091070 con 0x7ff668102e00 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.880+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff668109e90 con 0x7ff668102e00 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.880+0000 7ff662000700 1 -- 172.21.15.63:0/3202167399 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7ff65c05aaa0 con 0x7ff668102e00 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:06:27.862 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:06:27.863 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:06:27.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:06:27.865 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:06:27.865 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:06:27.865 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:06:27.865 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:06:27.865 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:06:27.865 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff65006c7d0 msgr2=0x7ff65006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff65006c7d0 0x7ff65006ec80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff658006fd0 tx=0x7ff658006db0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 msgr2=0x7ff668113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff668113aa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009390 tx=0x7ff65c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 shutdown_connections 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff65006c7d0 0x7ff65006ec80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff668102e00 0x7ff668113aa0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 --2- 172.21.15.63:0/3202167399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6681037a0 0x7ff668113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 >> 172.21.15.63:0/3202167399 conn(0x7ff6680f89b0 msgr2=0x7ff6680f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 shutdown_connections 2024-03-23T12:06:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:27.883+0000 7ff670800700 1 -- 172.21.15.63:0/3202167399 wait complete. 2024-03-23T12:06:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:06:27.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:27 smithi063 ceph-mon[141124]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:27.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:27 smithi063 ceph-mon[141124]: from='client.34886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:27.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:27 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3517408048' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:06:27.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:27 smithi111 ceph-mon[106899]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:27.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:27 smithi111 ceph-mon[106899]: from='client.34886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:27.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:27 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3517408048' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3813976030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 msgr2=0x7ffa680706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3813976030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa680706d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c004970 tx=0x7ffa5c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3813976030 shutdown_connections 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3813976030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa68070c10 0x7ffa68109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3813976030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa680706d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3813976030 >> 172.21.15.63:0/3813976030 conn(0x7ffa680f8970 msgr2=0x7ffa680fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3813976030 shutdown_connections 2024-03-23T12:06:28.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.022+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3813976030 wait complete. 2024-03-23T12:06:28.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.023+0000 7ffa6fe00700 1 Processor -- start 2024-03-23T12:06:28.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.024+0000 7ffa6fe00700 1 -- start start 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.024+0000 7ffa6fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa68115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.024+0000 7ffa6fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa68070c10 0x7ffa681161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.024+0000 7ffa6fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa68116820 con 0x7ffa68070300 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.024+0000 7ffa6fe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6810fc30 con 0x7ffa68070c10 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa68115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa68115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35512/0 (socket says 172.21.15.63:35512) 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6d400700 1 -- 172.21.15.63:0/3657264422 learned_addr learned my addr 172.21.15.63:0/3657264422 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:28.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6ca00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa68070c10 0x7ffa681161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:28.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6d400700 1 -- 172.21.15.63:0/3657264422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa68070c10 msgr2=0x7ffa681161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6d400700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa68070c10 0x7ffa681161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6d400700 1 -- 172.21.15.63:0/3657264422 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c005040 con 0x7ffa68070300 2024-03-23T12:06:28.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6ca00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa68070c10 0x7ffa681161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:06:28.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6d400700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa68115c90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005010 tx=0x7ffa5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:28.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa65600700 1 -- 172.21.15.63:0/3657264422 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5c00e040 con 0x7ffa68070300 2024-03-23T12:06:28.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa65600700 1 -- 172.21.15.63:0/3657264422 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa5c009930 con 0x7ffa68070300 2024-03-23T12:06:28.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.025+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6810feb0 con 0x7ffa68070300 2024-03-23T12:06:28.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.026+0000 7ffa65600700 1 -- 172.21.15.63:0/3657264422 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5c01c3f0 con 0x7ffa68070300 2024-03-23T12:06:28.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.026+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa681103f0 con 0x7ffa68070300 2024-03-23T12:06:28.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.027+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa68071a40 con 0x7ffa68070300 2024-03-23T12:06:28.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.028+0000 7ffa65600700 1 -- 172.21.15.63:0/3657264422 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ffa5c0136b0 con 0x7ffa68070300 2024-03-23T12:06:28.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.028+0000 7ffa65600700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffa5006c7d0 0x7ffa5006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:28.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.029+0000 7ffa65600700 1 -- 172.21.15.63:0/3657264422 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffa5c08cba0 con 0x7ffa68070300 2024-03-23T12:06:28.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.029+0000 7ffa6ca00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffa5006c7d0 0x7ffa5006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:28.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.029+0000 7ffa6ca00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffa5006c7d0 0x7ffa5006ec80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa54004600 tx=0x7ffa5400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:28.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.033+0000 7ffa65600700 1 -- 172.21.15.63:0/3657264422 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffa5c056ee0 con 0x7ffa68070300 2024-03-23T12:06:28.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.264+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa68071a40 con 0x7ffa5006c7d0 2024-03-23T12:06:28.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.265+0000 7ffa65600700 1 -- 172.21.15.63:0/3657264422 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffa68071a40 con 0x7ffa5006c7d0 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:06:28.247 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:06:28.248 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:06:28.248 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:06:28.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffa5006c7d0 msgr2=0x7ffa5006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffa5006c7d0 0x7ffa5006ec80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa54004600 tx=0x7ffa5400b040 comp rx=0 tx=0).stop 2024-03-23T12:06:28.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 msgr2=0x7ffa68115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa68115c90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005010 tx=0x7ffa5c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:28.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 shutdown_connections 2024-03-23T12:06:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffa5006c7d0 0x7ffa5006ec80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa68070300 0x7ffa68115c90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 --2- 172.21.15.63:0/3657264422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa68070c10 0x7ffa681161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 >> 172.21.15.63:0/3657264422 conn(0x7ffa680f8970 msgr2=0x7ffa680f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 shutdown_connections 2024-03-23T12:06:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.268+0000 7ffa6fe00700 1 -- 172.21.15.63:0/3657264422 wait complete. 2024-03-23T12:06:28.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 -- 172.21.15.63:0/982011899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 msgr2=0x7f09181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 --2- 172.21.15.63:0/982011899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f09181031d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0908004940 tx=0x7f0908009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:28.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 -- 172.21.15.63:0/982011899 shutdown_connections 2024-03-23T12:06:28.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 --2- 172.21.15.63:0/982011899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09181037a0 0x7f09180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 --2- 172.21.15.63:0/982011899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f09181031d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 -- 172.21.15.63:0/982011899 >> 172.21.15.63:0/982011899 conn(0x7f09180f89b0 msgr2=0x7f09180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:28.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 -- 172.21.15.63:0/982011899 shutdown_connections 2024-03-23T12:06:28.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.405+0000 7f091ce00700 1 -- 172.21.15.63:0/982011899 wait complete. 2024-03-23T12:06:28.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.406+0000 7f091ce00700 1 Processor -- start 2024-03-23T12:06:28.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.406+0000 7f091ce00700 1 -- start start 2024-03-23T12:06:28.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.406+0000 7f091ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f0918115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:28.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.406+0000 7f091ce00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09181037a0 0x7f09181161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.406+0000 7f091ce00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0918116810 con 0x7f0918102e00 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f091ce00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091810fbe0 con 0x7f09181037a0 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f0918115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f0918115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35522/0 (socket says 172.21.15.63:35522) 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915e00700 1 -- 172.21.15.63:0/3886128275 learned_addr learned my addr 172.21.15.63:0/3886128275 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915400700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09181037a0 0x7f09181161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915e00700 1 -- 172.21.15.63:0/3886128275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09181037a0 msgr2=0x7f09181161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915e00700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09181037a0 0x7f09181161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915e00700 1 -- 172.21.15.63:0/3886128275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908005040 con 0x7f0918102e00 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.407+0000 7f0915e00700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f0918115c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0908007f50 tx=0x7f0908009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.408+0000 7f090e000700 1 -- 172.21.15.63:0/3886128275 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0908021070 con 0x7f0918102e00 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.408+0000 7f090e000700 1 -- 172.21.15.63:0/3886128275 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0908013e00 con 0x7f0918102e00 2024-03-23T12:06:28.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.408+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091810fe60 con 0x7f0918102e00 2024-03-23T12:06:28.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.408+0000 7f090e000700 1 -- 172.21.15.63:0/3886128275 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f090800e070 con 0x7f0918102e00 2024-03-23T12:06:28.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.408+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0918110420 con 0x7f0918102e00 2024-03-23T12:06:28.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.409+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09180fe550 con 0x7f0918102e00 2024-03-23T12:06:28.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.410+0000 7f090e000700 1 -- 172.21.15.63:0/3886128275 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0908013950 con 0x7f0918102e00 2024-03-23T12:06:28.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.411+0000 7f090e000700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f08fc06c7d0 0x7f08fc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:28.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.411+0000 7f090e000700 1 -- 172.21.15.63:0/3886128275 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f090808ccb0 con 0x7f0918102e00 2024-03-23T12:06:28.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.411+0000 7f0915400700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f08fc06c7d0 0x7f08fc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:28.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.412+0000 7f0915400700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f08fc06c7d0 0x7f08fc06ec80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0904004600 tx=0x7f090400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:28.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.415+0000 7f090e000700 1 -- 172.21.15.63:0/3886128275 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f090805a9c0 con 0x7f0918102e00 2024-03-23T12:06:28.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.711+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f091805f5f0 con 0x7f0918102e00 2024-03-23T12:06:28.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.711+0000 7f090e000700 1 -- 172.21.15.63:0/3886128275 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f090805a550 con 0x7f0918102e00 2024-03-23T12:06:28.693 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:06:28.693 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:06:28.693 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:06:28.693 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:06:28.693 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f08fc06c7d0 msgr2=0x7f08fc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f08fc06c7d0 0x7f08fc06ec80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0904004600 tx=0x7f090400b040 comp rx=0 tx=0).stop 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 msgr2=0x7f0918115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f0918115c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0908007f50 tx=0x7f0908009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 shutdown_connections 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f08fc06c7d0 0x7f08fc06ec80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0918102e00 0x7f0918115c80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 --2- 172.21.15.63:0/3886128275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09181037a0 0x7f09181161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:28.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.713+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 >> 172.21.15.63:0/3886128275 conn(0x7f09180f89b0 msgr2=0x7f09180f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:28.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.714+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 shutdown_connections 2024-03-23T12:06:28.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:28.714+0000 7f091ce00700 1 -- 172.21.15.63:0/3886128275 wait complete. 2024-03-23T12:06:28.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:28 smithi063 ceph-mon[141124]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:28.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:28 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3202167399' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:06:28.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:28 smithi063 ceph-mon[141124]: pgmap v460: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:28.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:28 smithi111 ceph-mon[106899]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:28.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:28 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3202167399' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:06:28.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:28 smithi111 ceph-mon[106899]: pgmap v460: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:29.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:29 smithi063 ceph-mon[141124]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:29.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:29 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3886128275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:06:29.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:29 smithi111 ceph-mon[106899]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:06:29.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:29 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3886128275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:06:30.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:30 smithi063 ceph-mon[141124]: pgmap v461: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:30.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:30 smithi111 ceph-mon[106899]: pgmap v461: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:32 smithi063 ceph-mon[141124]: pgmap v462: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:33.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:32 smithi111 ceph-mon[106899]: pgmap v462: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:35.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:34 smithi063 ceph-mon[141124]: pgmap v463: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:35.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:34 smithi111 ceph-mon[106899]: pgmap v463: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:36.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:36 smithi063 ceph-mon[141124]: pgmap v464: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:36.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:36 smithi111 ceph-mon[106899]: pgmap v464: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:39.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:38 smithi063 ceph-mon[141124]: pgmap v465: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:39.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:38 smithi111 ceph-mon[106899]: pgmap v465: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:40 smithi063 ceph-mon[141124]: pgmap v466: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:40 smithi111 ceph-mon[106899]: pgmap v466: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:43.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:42 smithi063 ceph-mon[141124]: pgmap v467: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:43.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:42 smithi111 ceph-mon[106899]: pgmap v467: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:45.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:44 smithi063 ceph-mon[141124]: pgmap v468: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:45.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:44 smithi111 ceph-mon[106899]: pgmap v468: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:46.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:46 smithi063 ceph-mon[141124]: pgmap v469: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:46.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:46 smithi111 ceph-mon[106899]: pgmap v469: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:49.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:48 smithi063 ceph-mon[141124]: pgmap v470: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:49.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:48 smithi111 ceph-mon[106899]: pgmap v470: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:51.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:50 smithi063 ceph-mon[141124]: pgmap v471: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:51.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:50 smithi111 ceph-mon[106899]: pgmap v471: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:53.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:52 smithi063 ceph-mon[141124]: pgmap v472: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:53.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:52 smithi111 ceph-mon[106899]: pgmap v472: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:55.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:54 smithi063 ceph-mon[141124]: pgmap v473: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:54 smithi111 ceph-mon[106899]: pgmap v473: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:56.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:56 smithi063 ceph-mon[141124]: pgmap v474: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:06:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:56 smithi111 ceph-mon[106899]: pgmap v474: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:58.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.868+0000 7f183e400700 1 -- 172.21.15.63:0/2784635329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18381037a0 msgr2=0x7f18380fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:58.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.868+0000 7f183e400700 1 --2- 172.21.15.63:0/2784635329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18381037a0 0x7f18380fce10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1824004940 tx=0x7f1824009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:58.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.868+0000 7f183e400700 1 -- 172.21.15.63:0/2784635329 shutdown_connections 2024-03-23T12:06:58.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.868+0000 7f183e400700 1 --2- 172.21.15.63:0/2784635329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18381037a0 0x7f18380fce10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:58.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.868+0000 7f183e400700 1 --2- 172.21.15.63:0/2784635329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1838102e00 0x7f18381031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:58.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.868+0000 7f183e400700 1 -- 172.21.15.63:0/2784635329 >> 172.21.15.63:0/2784635329 conn(0x7f18380f89b0 msgr2=0x7f18380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:58.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.869+0000 7f183e400700 1 -- 172.21.15.63:0/2784635329 shutdown_connections 2024-03-23T12:06:58.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.869+0000 7f183e400700 1 -- 172.21.15.63:0/2784635329 wait complete. 2024-03-23T12:06:58.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.869+0000 7f183e400700 1 Processor -- start 2024-03-23T12:06:58.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.869+0000 7f183e400700 1 -- start start 2024-03-23T12:06:58.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f183e400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1838102e00 0x7f1838115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f183e400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18381037a0 0x7f1838116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f183e400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18381168b0 con 0x7f1838102e00 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f183e400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183810fc80 con 0x7f18381037a0 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f1837400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1838102e00 0x7f1838115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f1837400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1838102e00 0x7f1838115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49952/0 (socket says 172.21.15.63:49952) 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f1837400700 1 -- 172.21.15.63:0/973898245 learned_addr learned my addr 172.21.15.63:0/973898245 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.870+0000 7f1836a00700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18381037a0 0x7f1838116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f1837400700 1 -- 172.21.15.63:0/973898245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18381037a0 msgr2=0x7f1838116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f1837400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18381037a0 0x7f1838116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f1837400700 1 -- 172.21.15.63:0/973898245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824005040 con 0x7f1838102e00 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f1836a00700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18381037a0 0x7f1838116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f1837400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1838102e00 0x7f1838115d20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1828002a50 tx=0x7f18280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:58.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f182f400700 1 -- 172.21.15.63:0/973898245 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1828007cc0 con 0x7f1838102e00 2024-03-23T12:06:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f182f400700 1 -- 172.21.15.63:0/973898245 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f182800f040 con 0x7f1838102e00 2024-03-23T12:06:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f182f400700 1 -- 172.21.15.63:0/973898245 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1828009540 con 0x7f1838102e00 2024-03-23T12:06:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.871+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183810ff60 con 0x7f1838102e00 2024-03-23T12:06:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.872+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1838110520 con 0x7f1838102e00 2024-03-23T12:06:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.873+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18380fe550 con 0x7f1838102e00 2024-03-23T12:06:58.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.874+0000 7f182f400700 1 -- 172.21.15.63:0/973898245 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f18280096a0 con 0x7f1838102e00 2024-03-23T12:06:58.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.874+0000 7f182f400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181c06c7c0 0x7f181c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:58.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.875+0000 7f182f400700 1 -- 172.21.15.63:0/973898245 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f182808b820 con 0x7f1838102e00 2024-03-23T12:06:58.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.875+0000 7f1836a00700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181c06c7c0 0x7f181c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:58.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.875+0000 7f1836a00700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181c06c7c0 0x7f181c06ec70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f182400b3b0 tx=0x7f1824009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:58.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:58.879+0000 7f182f400700 1 -- 172.21.15.63:0/973898245 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f18280594b0 con 0x7f1838102e00 2024-03-23T12:06:59.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.111+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f183804bf40 con 0x7f181c06c7c0 2024-03-23T12:06:59.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.112+0000 7f182f400700 1 -- 172.21.15.63:0/973898245 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f183804bf40 con 0x7f181c06c7c0 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181c06c7c0 msgr2=0x7f181c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181c06c7c0 0x7f181c06ec70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f182400b3b0 tx=0x7f1824009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1838102e00 msgr2=0x7f1838115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1838102e00 0x7f1838115d20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1828002a50 tx=0x7f18280046b0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 shutdown_connections 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f181c06c7c0 0x7f181c06ec70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1838102e00 0x7f1838115d20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 --2- 172.21.15.63:0/973898245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18381037a0 0x7f1838116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.114+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 >> 172.21.15.63:0/973898245 conn(0x7f18380f89b0 msgr2=0x7f18380f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:59.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.115+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 shutdown_connections 2024-03-23T12:06:59.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.115+0000 7f183e400700 1 -- 172.21.15.63:0/973898245 wait complete. 2024-03-23T12:06:59.109 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:06:59.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.252+0000 7facb2200700 1 -- 172.21.15.63:0/2897850089 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 msgr2=0x7facac0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.252+0000 7facb2200700 1 --2- 172.21.15.63:0/2897850089 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac0fe260 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fac9c004940 tx=0x7fac9c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:59.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.253+0000 7facb2200700 1 -- 172.21.15.63:0/2897850089 shutdown_connections 2024-03-23T12:06:59.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.253+0000 7facb2200700 1 --2- 172.21.15.63:0/2897850089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facac0fe830 0x7facac106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.253+0000 7facb2200700 1 --2- 172.21.15.63:0/2897850089 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac0fe260 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.253+0000 7facb2200700 1 -- 172.21.15.63:0/2897850089 >> 172.21.15.63:0/2897850089 conn(0x7facac072bc0 msgr2=0x7facac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:59.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.253+0000 7facb2200700 1 -- 172.21.15.63:0/2897850089 shutdown_connections 2024-03-23T12:06:59.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.253+0000 7facb2200700 1 -- 172.21.15.63:0/2897850089 wait complete. 2024-03-23T12:06:59.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facb2200700 1 Processor -- start 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facb2200700 1 -- start start 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facb2200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facb2200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facac0fe830 0x7facac116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facb2200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac1168b0 con 0x7facac0fde90 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facb2200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac10fc80 con 0x7facac0fe830 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49958/0 (socket says 172.21.15.63:49958) 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.254+0000 7facab400700 1 -- 172.21.15.63:0/3288668506 learned_addr learned my addr 172.21.15.63:0/3288668506 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7facaaa00700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facac0fe830 0x7facac116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7facab400700 1 -- 172.21.15.63:0/3288668506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facac0fe830 msgr2=0x7facac116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7facab400700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facac0fe830 0x7facac116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7facab400700 1 -- 172.21.15.63:0/3288668506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c005040 con 0x7facac0fde90 2024-03-23T12:06:59.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7facab400700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac115d20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:59.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7faca3400700 1 -- 172.21.15.63:0/3288668506 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac9c00e040 con 0x7facac0fde90 2024-03-23T12:06:59.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7faca3400700 1 -- 172.21.15.63:0/3288668506 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac9c013420 con 0x7facac0fde90 2024-03-23T12:06:59.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac10ff00 con 0x7facac0fde90 2024-03-23T12:06:59.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7faca3400700 1 -- 172.21.15.63:0/3288668506 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac9c01c630 con 0x7facac0fde90 2024-03-23T12:06:59.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.255+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac110440 con 0x7facac0fde90 2024-03-23T12:06:59.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.257+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac106a20 con 0x7facac0fde90 2024-03-23T12:06:59.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.257+0000 7faca3400700 1 -- 172.21.15.63:0/3288668506 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fac9c013970 con 0x7facac0fde90 2024-03-23T12:06:59.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.258+0000 7faca3400700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac9006c7d0 0x7fac9006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:59.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.258+0000 7faca3400700 1 -- 172.21.15.63:0/3288668506 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fac9c08cc80 con 0x7facac0fde90 2024-03-23T12:06:59.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.259+0000 7facaaa00700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac9006c7d0 0x7fac9006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:59.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.259+0000 7facaaa00700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac9006c7d0 0x7fac9006ec80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7facac1115c0 tx=0x7fac98009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:59.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.263+0000 7faca3400700 1 -- 172.21.15.63:0/3288668506 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac9c05a990 con 0x7facac0fde90 2024-03-23T12:06:59.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:06:58 smithi063 ceph-mon[141124]: pgmap v475: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:59.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:06:58 smithi111 ceph-mon[106899]: pgmap v475: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:06:59.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.498+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7facac109dc0 con 0x7fac9006c7d0 2024-03-23T12:06:59.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.499+0000 7faca3400700 1 -- 172.21.15.63:0/3288668506 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7facac109dc0 con 0x7fac9006c7d0 2024-03-23T12:06:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.501+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac9006c7d0 msgr2=0x7fac9006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.501+0000 7facb2200700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac9006c7d0 0x7fac9006ec80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7facac1115c0 tx=0x7fac98009ee0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.501+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 msgr2=0x7facac115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.501+0000 7facb2200700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac115d20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.502+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 shutdown_connections 2024-03-23T12:06:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.502+0000 7facb2200700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fac9006c7d0 0x7fac9006ec80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.502+0000 7facb2200700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7facac0fde90 0x7facac115d20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.502+0000 7facb2200700 1 --2- 172.21.15.63:0/3288668506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facac0fe830 0x7facac116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.502+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 >> 172.21.15.63:0/3288668506 conn(0x7facac072bc0 msgr2=0x7facac0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.502+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 shutdown_connections 2024-03-23T12:06:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.502+0000 7facb2200700 1 -- 172.21.15.63:0/3288668506 wait complete. 2024-03-23T12:06:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 -- 172.21.15.63:0/2653688936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a60102e00 msgr2=0x7f7a601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 --2- 172.21.15.63:0/2653688936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a60102e00 0x7f7a601031d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7a54004940 tx=0x7f7a54009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 -- 172.21.15.63:0/2653688936 shutdown_connections 2024-03-23T12:06:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 --2- 172.21.15.63:0/2653688936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a601037a0 0x7f7a600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 --2- 172.21.15.63:0/2653688936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a60102e00 0x7f7a601031d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 -- 172.21.15.63:0/2653688936 >> 172.21.15.63:0/2653688936 conn(0x7f7a600f89b0 msgr2=0x7f7a600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 -- 172.21.15.63:0/2653688936 shutdown_connections 2024-03-23T12:06:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.642+0000 7f7a68200700 1 -- 172.21.15.63:0/2653688936 wait complete. 2024-03-23T12:06:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.643+0000 7f7a68200700 1 Processor -- start 2024-03-23T12:06:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.643+0000 7f7a68200700 1 -- start start 2024-03-23T12:06:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.643+0000 7f7a68200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a60102e00 0x7f7a60115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.643+0000 7f7a68200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a601037a0 0x7f7a60116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.643+0000 7f7a68200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60116860 con 0x7f7a601037a0 2024-03-23T12:06:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.643+0000 7f7a68200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6010fc30 con 0x7f7a60102e00 2024-03-23T12:06:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a64e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a601037a0 0x7f7a60116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a64e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a601037a0 0x7f7a60116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49974/0 (socket says 172.21.15.63:49974) 2024-03-23T12:06:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a64e00700 1 -- 172.21.15.63:0/4253014629 learned_addr learned my addr 172.21.15.63:0/4253014629 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:06:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a65800700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a60102e00 0x7f7a60115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a64e00700 1 -- 172.21.15.63:0/4253014629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a60102e00 msgr2=0x7f7a60115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a64e00700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a60102e00 0x7f7a60115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a64e00700 1 -- 172.21.15.63:0/4253014629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a54005040 con 0x7f7a601037a0 2024-03-23T12:06:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a64e00700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a601037a0 0x7f7a60116210 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c002a50 tx=0x7f7a4c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.644+0000 7f7a5d600700 1 -- 172.21.15.63:0/4253014629 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4c007cc0 con 0x7f7a601037a0 2024-03-23T12:06:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.645+0000 7f7a5d600700 1 -- 172.21.15.63:0/4253014629 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a4c00f040 con 0x7f7a601037a0 2024-03-23T12:06:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.645+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6010ff10 con 0x7f7a601037a0 2024-03-23T12:06:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.645+0000 7f7a5d600700 1 -- 172.21.15.63:0/4253014629 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4c00d9a0 con 0x7f7a601037a0 2024-03-23T12:06:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.645+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a601104d0 con 0x7f7a601037a0 2024-03-23T12:06:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.646+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a600fe550 con 0x7f7a601037a0 2024-03-23T12:06:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.647+0000 7f7a5d600700 1 -- 172.21.15.63:0/4253014629 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7a4c00dc10 con 0x7f7a601037a0 2024-03-23T12:06:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.648+0000 7f7a5d600700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a4806c7d0 0x7f7a4806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:06:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.648+0000 7f7a5d600700 1 -- 172.21.15.63:0/4253014629 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7a4c08b7f0 con 0x7f7a601037a0 2024-03-23T12:06:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.648+0000 7f7a65800700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a4806c7d0 0x7f7a4806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:06:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.649+0000 7f7a65800700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a4806c7d0 0x7f7a4806ec80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400b3b0 tx=0x7f7a54009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:06:59.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.652+0000 7f7a5d600700 1 -- 172.21.15.63:0/4253014629 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7a4c059500 con 0x7f7a601037a0 2024-03-23T12:06:59.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.874+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a60109e30 con 0x7f7a4806c7d0 2024-03-23T12:06:59.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.884+0000 7f7a5d600700 1 -- 172.21.15.63:0/4253014629 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f7a60109e30 con 0x7f7a4806c7d0 2024-03-23T12:06:59.864 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (16m) 8m ago 29m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (13m) 8m ago 29m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (13m) 8m ago 28m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (15m) 8m ago 29m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (9m) 8m ago 23m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (9m) 8m ago 22m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (22m) 8m ago 22m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (8m) 8m ago 23m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (18m) 8m ago 30m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (17m) 8m ago 28m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (14m) 8m ago 31m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (14m) 8m ago 28m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:06:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (17m) 8m ago 29m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (17m) 8m ago 28m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (12m) 8m ago 27m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (12m) 8m ago 26m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (11m) 8m ago 26m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (10m) 8m ago 25m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (10m) 8m ago 25m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (9m) 8m ago 24m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:06:59.866 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (16m) 8m ago 28m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a4806c7d0 msgr2=0x7f7a4806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a4806c7d0 0x7f7a4806ec80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400b3b0 tx=0x7f7a54009e30 comp rx=0 tx=0).stop 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a601037a0 msgr2=0x7f7a60116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a601037a0 0x7f7a60116210 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c002a50 tx=0x7f7a4c004600 comp rx=0 tx=0).stop 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 shutdown_connections 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a4806c7d0 0x7f7a4806ec80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a60102e00 0x7f7a60115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 --2- 172.21.15.63:0/4253014629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a601037a0 0x7f7a60116210 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:06:59.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.886+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 >> 172.21.15.63:0/4253014629 conn(0x7f7a600f89b0 msgr2=0x7f7a600f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:06:59.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.887+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 shutdown_connections 2024-03-23T12:06:59.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:06:59.887+0000 7f7a68200700 1 -- 172.21.15.63:0/4253014629 wait complete. 2024-03-23T12:07:00.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 -- 172.21.15.63:0/3193713581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc1037a0 msgr2=0x7fb3cc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 --2- 172.21.15.63:0/3193713581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc1037a0 0x7fb3cc0fce10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8004940 tx=0x7fb3b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:00.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 -- 172.21.15.63:0/3193713581 shutdown_connections 2024-03-23T12:07:00.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 --2- 172.21.15.63:0/3193713581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc1037a0 0x7fb3cc0fce10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 --2- 172.21.15.63:0/3193713581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3cc102e00 0x7fb3cc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 -- 172.21.15.63:0/3193713581 >> 172.21.15.63:0/3193713581 conn(0x7fb3cc0f89b0 msgr2=0x7fb3cc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:00.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 -- 172.21.15.63:0/3193713581 shutdown_connections 2024-03-23T12:07:00.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.023+0000 7fb3d1600700 1 -- 172.21.15.63:0/3193713581 wait complete. 2024-03-23T12:07:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.024+0000 7fb3d1600700 1 Processor -- start 2024-03-23T12:07:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.024+0000 7fb3d1600700 1 -- start start 2024-03-23T12:07:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.024+0000 7fb3d1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc102e00 0x7fb3cc115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.024+0000 7fb3d1600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3cc1037a0 0x7fb3cc1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.024+0000 7fb3d1600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc116840 con 0x7fb3cc102e00 2024-03-23T12:07:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.024+0000 7fb3d1600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc10fc10 con 0x7fb3cc1037a0 2024-03-23T12:07:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3caa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc102e00 0x7fb3cc115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3caa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc102e00 0x7fb3cc115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50002/0 (socket says 172.21.15.63:50002) 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3caa00700 1 -- 172.21.15.63:0/1796426808 learned_addr learned my addr 172.21.15.63:0/1796426808 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3ca000700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3cc1037a0 0x7fb3cc1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3caa00700 1 -- 172.21.15.63:0/1796426808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3cc1037a0 msgr2=0x7fb3cc1161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3caa00700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3cc1037a0 0x7fb3cc1161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3caa00700 1 -- 172.21.15.63:0/1796426808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b8005040 con 0x7fb3cc102e00 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3caa00700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc102e00 0x7fb3cc115cb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00eb40 tx=0x7fb3bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.025+0000 7fb3c2a00700 1 -- 172.21.15.63:0/1796426808 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3bc007cc0 con 0x7fb3cc102e00 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.026+0000 7fb3c2a00700 1 -- 172.21.15.63:0/1796426808 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3bc00f040 con 0x7fb3cc102e00 2024-03-23T12:07:00.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.026+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3cc10fe90 con 0x7fb3cc102e00 2024-03-23T12:07:00.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.026+0000 7fb3c2a00700 1 -- 172.21.15.63:0/1796426808 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3bc009570 con 0x7fb3cc102e00 2024-03-23T12:07:00.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.026+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3cc110450 con 0x7fb3cc102e00 2024-03-23T12:07:00.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.027+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3cc0fe550 con 0x7fb3cc102e00 2024-03-23T12:07:00.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.028+0000 7fb3c2a00700 1 -- 172.21.15.63:0/1796426808 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb3bc0096d0 con 0x7fb3cc102e00 2024-03-23T12:07:00.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.029+0000 7fb3c2a00700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb3b006c7d0 0x7fb3b006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.029+0000 7fb3c2a00700 1 -- 172.21.15.63:0/1796426808 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3bc08b760 con 0x7fb3cc102e00 2024-03-23T12:07:00.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.029+0000 7fb3ca000700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb3b006c7d0 0x7fb3b006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.030+0000 7fb3ca000700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb3b006c7d0 0x7fb3b006ec80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc111550 tx=0x7fb3b8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:00.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.034+0000 7fb3c2a00700 1 -- 172.21.15.63:0/1796426808 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb3bc0594a0 con 0x7fb3cc102e00 2024-03-23T12:07:00.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.331+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb3cc04bed0 con 0x7fb3cc102e00 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.332+0000 7fb3c2a00700 1 -- 172.21.15.63:0/1796426808 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fb3bc059030 con 0x7fb3cc102e00 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:07:00.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:07:00.313 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb3b006c7d0 msgr2=0x7fb3b006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb3b006c7d0 0x7fb3b006ec80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc111550 tx=0x7fb3b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc102e00 msgr2=0x7fb3cc115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc102e00 0x7fb3cc115cb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00eb40 tx=0x7fb3bc00b040 comp rx=0 tx=0).stop 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 shutdown_connections 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb3b006c7d0 0x7fb3b006ec80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3cc102e00 0x7fb3cc115cb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 --2- 172.21.15.63:0/1796426808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3cc1037a0 0x7fb3cc1161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 >> 172.21.15.63:0/1796426808 conn(0x7fb3cc0f89b0 msgr2=0x7fb3cc070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.334+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 shutdown_connections 2024-03-23T12:07:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.335+0000 7fb3d1600700 1 -- 172.21.15.63:0/1796426808 wait complete. 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 -- 172.21.15.63:0/3450470647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 msgr2=0x7f11240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 --2- 172.21.15.63:0/3450470647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f11240fce10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1110004940 tx=0x7f1110009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 -- 172.21.15.63:0/3450470647 shutdown_connections 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 --2- 172.21.15.63:0/3450470647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f11240fce10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 --2- 172.21.15.63:0/3450470647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1124102e00 0x7f11241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 -- 172.21.15.63:0/3450470647 >> 172.21.15.63:0/3450470647 conn(0x7f11240f89b0 msgr2=0x7f11240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 -- 172.21.15.63:0/3450470647 shutdown_connections 2024-03-23T12:07:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.471+0000 7f112b400700 1 -- 172.21.15.63:0/3450470647 wait complete. 2024-03-23T12:07:00.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.472+0000 7f112b400700 1 Processor -- start 2024-03-23T12:07:00.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.472+0000 7f112b400700 1 -- start start 2024-03-23T12:07:00.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f112b400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1124102e00 0x7f1124115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f112b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f1124116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f112b400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1124116890 con 0x7f11241037a0 2024-03-23T12:07:00.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f112b400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112410fc60 con 0x7f1124102e00 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1123e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f1124116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1123e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f1124116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50026/0 (socket says 172.21.15.63:50026) 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1123e00700 1 -- 172.21.15.63:0/821329980 learned_addr learned my addr 172.21.15.63:0/821329980 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1128a00700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1124102e00 0x7f1124115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1123e00700 1 -- 172.21.15.63:0/821329980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1124102e00 msgr2=0x7f1124115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1123e00700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1124102e00 0x7f1124115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1123e00700 1 -- 172.21.15.63:0/821329980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1110005040 con 0x7f11241037a0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.473+0000 7f1128a00700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1124102e00 0x7f1124115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.474+0000 7f1123e00700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f1124116240 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1110009390 tx=0x7f1110009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.474+0000 7f1120c00700 1 -- 172.21.15.63:0/821329980 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111000e040 con 0x7f11241037a0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.474+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112410fee0 con 0x7f11241037a0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.474+0000 7f1120c00700 1 -- 172.21.15.63:0/821329980 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1110013420 con 0x7f11241037a0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.474+0000 7f1120c00700 1 -- 172.21.15.63:0/821329980 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111001c630 con 0x7f11241037a0 2024-03-23T12:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.474+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11241104a0 con 0x7f11241037a0 2024-03-23T12:07:00.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.475+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11240fe550 con 0x7f11241037a0 2024-03-23T12:07:00.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.476+0000 7f1120c00700 1 -- 172.21.15.63:0/821329980 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f1110013a00 con 0x7f11241037a0 2024-03-23T12:07:00.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.477+0000 7f1120c00700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f110c06c7d0 0x7f110c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.477+0000 7f1120c00700 1 -- 172.21.15.63:0/821329980 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f111008cd00 con 0x7f11241037a0 2024-03-23T12:07:00.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.477+0000 7f1128a00700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f110c06c7d0 0x7f110c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.478+0000 7f1128a00700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f110c06c7d0 0x7f110c06ec80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1118007fd0 tx=0x7f1118009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:00.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.482+0000 7f1120c00700 1 -- 172.21.15.63:0/821329980 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f111005aa10 con 0x7f11241037a0 2024-03-23T12:07:00.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.738+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f112404bed0 con 0x7f11241037a0 2024-03-23T12:07:00.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.738+0000 7f1120c00700 1 -- 172.21.15.63:0/821329980 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f111005a5a0 con 0x7f11241037a0 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:07:00.719 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:07:00.720 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:00.721 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f110c06c7d0 msgr2=0x7f110c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f110c06c7d0 0x7f110c06ec80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1118007fd0 tx=0x7f1118009040 comp rx=0 tx=0).stop 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 msgr2=0x7f1124116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f1124116240 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1110009390 tx=0x7f1110009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 shutdown_connections 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f110c06c7d0 0x7f110c06ec80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1124102e00 0x7f1124115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 --2- 172.21.15.63:0/821329980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11241037a0 0x7f1124116240 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.740+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 >> 172.21.15.63:0/821329980 conn(0x7f11240f89b0 msgr2=0x7f11240f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:00.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.741+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 shutdown_connections 2024-03-23T12:07:00.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.741+0000 7f112b400700 1 -- 172.21.15.63:0/821329980 wait complete. 2024-03-23T12:07:00.723 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 -- 172.21.15.63:0/2923509458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fde90 msgr2=0x7f98ec0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 --2- 172.21.15.63:0/2923509458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fde90 0x7f98ec0fe260 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f98e0004940 tx=0x7f98e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 -- 172.21.15.63:0/2923509458 shutdown_connections 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 --2- 172.21.15.63:0/2923509458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fe830 0x7f98ec106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 --2- 172.21.15.63:0/2923509458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fde90 0x7f98ec0fe260 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 -- 172.21.15.63:0/2923509458 >> 172.21.15.63:0/2923509458 conn(0x7f98ec072bc0 msgr2=0x7f98ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 -- 172.21.15.63:0/2923509458 shutdown_connections 2024-03-23T12:07:00.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.878+0000 7f98f4600700 1 -- 172.21.15.63:0/2923509458 wait complete. 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.879+0000 7f98f4600700 1 Processor -- start 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.879+0000 7f98f4600700 1 -- start start 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f4600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fde90 0x7f98ec115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f4600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fe830 0x7f98ec116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f4600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec1168b0 con 0x7f98ec0fe830 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f4600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec10fc80 con 0x7f98ec0fde90 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fe830 0x7f98ec116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fe830 0x7f98ec116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50052/0 (socket says 172.21.15.63:50052) 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f1200700 1 -- 172.21.15.63:0/3475137772 learned_addr learned my addr 172.21.15.63:0/3475137772 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f1c00700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fde90 0x7f98ec115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f1200700 1 -- 172.21.15.63:0/3475137772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fde90 msgr2=0x7f98ec115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f1200700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fde90 0x7f98ec115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:00.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.880+0000 7f98f1200700 1 -- 172.21.15.63:0/3475137772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e0005040 con 0x7f98ec0fe830 2024-03-23T12:07:00.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.881+0000 7f98f1200700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fe830 0x7f98ec116260 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f98d800c8c0 tx=0x7f98d80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:00.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.881+0000 7f98e9600700 1 -- 172.21.15.63:0/3475137772 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d800a760 con 0x7f98ec0fe830 2024-03-23T12:07:00.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.881+0000 7f98e9600700 1 -- 172.21.15.63:0/3475137772 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98d800ad70 con 0x7f98ec0fe830 2024-03-23T12:07:00.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.881+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec10ff60 con 0x7f98ec0fe830 2024-03-23T12:07:00.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.881+0000 7f98e9600700 1 -- 172.21.15.63:0/3475137772 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d80113a0 con 0x7f98ec0fe830 2024-03-23T12:07:00.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.881+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec1104a0 con 0x7f98ec0fe830 2024-03-23T12:07:00.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.882+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec106a20 con 0x7f98ec0fe830 2024-03-23T12:07:00.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.883+0000 7f98e9600700 1 -- 172.21.15.63:0/3475137772 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f98d8012400 con 0x7f98ec0fe830 2024-03-23T12:07:00.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.885+0000 7f98e9600700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98d406c7d0 0x7f98d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:00.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.885+0000 7f98e9600700 1 -- 172.21.15.63:0/3475137772 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f98d808ba60 con 0x7f98ec0fe830 2024-03-23T12:07:00.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.885+0000 7f98f1c00700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98d406c7d0 0x7f98d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:00.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.885+0000 7f98f1c00700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98d406c7d0 0x7f98d406ec80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98e000b3b0 tx=0x7f98e0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:00.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:00.890+0000 7f98e9600700 1 -- 172.21.15.63:0/3475137772 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98d8059770 con 0x7f98ec0fe830 2024-03-23T12:07:01.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.122+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98ec1113e0 con 0x7f98d406c7d0 2024-03-23T12:07:01.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.123+0000 7f98e9600700 1 -- 172.21.15.63:0/3475137772 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f98ec1113e0 con 0x7f98d406c7d0 2024-03-23T12:07:01.103 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:07:01.104 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98d406c7d0 msgr2=0x7f98d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98d406c7d0 0x7f98d406ec80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98e000b3b0 tx=0x7f98e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fe830 msgr2=0x7f98ec116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fe830 0x7f98ec116260 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f98d800c8c0 tx=0x7f98d80044c0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 shutdown_connections 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98d406c7d0 0x7f98d406ec80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fde90 0x7f98ec115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 --2- 172.21.15.63:0/3475137772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec0fe830 0x7f98ec116260 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.125+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 >> 172.21.15.63:0/3475137772 conn(0x7f98ec072bc0 msgr2=0x7f98ec0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:01.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.126+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 shutdown_connections 2024-03-23T12:07:01.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.126+0000 7f98f4600700 1 -- 172.21.15.63:0/3475137772 wait complete. 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 -- 172.21.15.63:0/4012724314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 msgr2=0x7f28440fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 --2- 172.21.15.63:0/4012724314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f28440fc050 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2834004940 tx=0x7f2834009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 -- 172.21.15.63:0/4012724314 shutdown_connections 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 --2- 172.21.15.63:0/4012724314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28440fc590 0x7f28441092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 --2- 172.21.15.63:0/4012724314 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f28440fc050 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 -- 172.21.15.63:0/4012724314 >> 172.21.15.63:0/4012724314 conn(0x7f2844072bc0 msgr2=0x7f2844072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 -- 172.21.15.63:0/4012724314 shutdown_connections 2024-03-23T12:07:01.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.262+0000 7f284a400700 1 -- 172.21.15.63:0/4012724314 wait complete. 2024-03-23T12:07:01.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.263+0000 7f284a400700 1 Processor -- start 2024-03-23T12:07:01.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.263+0000 7f284a400700 1 -- start start 2024-03-23T12:07:01.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.263+0000 7f284a400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f2844115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:01.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.263+0000 7f284a400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28440fc590 0x7f2844116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:01.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f284a400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28441168c0 con 0x7f28440fbc80 2024-03-23T12:07:01.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f284a400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284410fc90 con 0x7f28440fc590 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2843400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f2844115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2843400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f2844115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50066/0 (socket says 172.21.15.63:50066) 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2843400700 1 -- 172.21.15.63:0/3968882941 learned_addr learned my addr 172.21.15.63:0/3968882941 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2842a00700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28440fc590 0x7f2844116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2843400700 1 -- 172.21.15.63:0/3968882941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28440fc590 msgr2=0x7f2844116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2843400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28440fc590 0x7f2844116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2843400700 1 -- 172.21.15.63:0/3968882941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834005040 con 0x7f28440fbc80 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.264+0000 7f2843400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f2844115d30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2834009180 tx=0x7f2834009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.265+0000 7f283b400700 1 -- 172.21.15.63:0/3968882941 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f283400e040 con 0x7f28440fbc80 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.265+0000 7f283b400700 1 -- 172.21.15.63:0/3968882941 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2834004290 con 0x7f28440fbc80 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.265+0000 7f283b400700 1 -- 172.21.15.63:0/3968882941 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f283401c3f0 con 0x7f28440fbc80 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.265+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284410ff10 con 0x7f28440fbc80 2024-03-23T12:07:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.265+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2844110450 con 0x7f28440fbc80 2024-03-23T12:07:01.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.266+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2844065e30 con 0x7f28440fbc80 2024-03-23T12:07:01.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.267+0000 7f283b400700 1 -- 172.21.15.63:0/3968882941 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2834013790 con 0x7f28440fbc80 2024-03-23T12:07:01.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.268+0000 7f283b400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f282806c7c0 0x7f282806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:01.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.268+0000 7f283b400700 1 -- 172.21.15.63:0/3968882941 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f283408ccf0 con 0x7f28440fbc80 2024-03-23T12:07:01.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.268+0000 7f2842a00700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f282806c7c0 0x7f282806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:01.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.269+0000 7f2842a00700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f282806c7c0 0x7f282806ec70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f28441115d0 tx=0x7f2830009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:01.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.272+0000 7f283b400700 1 -- 172.21.15.63:0/3968882941 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f283405aa00 con 0x7f28440fbc80 2024-03-23T12:07:01.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:00 smithi063 ceph-mon[141124]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:01.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:00 smithi063 ceph-mon[141124]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:00 smithi063 ceph-mon[141124]: from='client.34918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:00 smithi063 ceph-mon[141124]: pgmap v476: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:00 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1796426808' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:07:01.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:00 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/821329980' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:07:01.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:00 smithi111 ceph-mon[106899]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:01.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:00 smithi111 ceph-mon[106899]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:01.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:00 smithi111 ceph-mon[106899]: from='client.34918 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:01.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:00 smithi111 ceph-mon[106899]: pgmap v476: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:01.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:00 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1796426808' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:07:01.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:00 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/821329980' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:07:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.574+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f284404bed0 con 0x7f28440fbc80 2024-03-23T12:07:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.575+0000 7f283b400700 1 -- 172.21.15.63:0/3968882941 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2834013420 con 0x7f28440fbc80 2024-03-23T12:07:01.555 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:07:01.555 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:07:01.556 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:07:01.556 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:07:01.556 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:07:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f282806c7c0 msgr2=0x7f282806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f282806c7c0 0x7f282806ec70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f28441115d0 tx=0x7f2830009ee0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 msgr2=0x7f2844115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f2844115d30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2834009180 tx=0x7f2834009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 shutdown_connections 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f282806c7c0 0x7f282806ec70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28440fbc80 0x7f2844115d30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 --2- 172.21.15.63:0/3968882941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28440fc590 0x7f2844116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 >> 172.21.15.63:0/3968882941 conn(0x7f2844072bc0 msgr2=0x7f2844105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 shutdown_connections 2024-03-23T12:07:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:01.577+0000 7f284a400700 1 -- 172.21.15.63:0/3968882941 wait complete. 2024-03-23T12:07:02.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:01 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3968882941' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:07:02.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:01 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3968882941' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:07:03.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:02 smithi063 ceph-mon[141124]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:03.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:02 smithi063 ceph-mon[141124]: pgmap v477: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:03.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:02 smithi111 ceph-mon[106899]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:03.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:02 smithi111 ceph-mon[106899]: pgmap v477: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:05.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:04 smithi063 ceph-mon[141124]: pgmap v478: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:05.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:04 smithi111 ceph-mon[106899]: pgmap v478: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:06.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:06 smithi063 ceph-mon[141124]: pgmap v479: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:06.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:06 smithi111 ceph-mon[106899]: pgmap v479: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:09.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:08 smithi063 ceph-mon[141124]: pgmap v480: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:09.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:08 smithi111 ceph-mon[106899]: pgmap v480: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:11.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:10 smithi063 ceph-mon[141124]: pgmap v481: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:11.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:11.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:10 smithi111 ceph-mon[106899]: pgmap v481: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:11.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:13.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:12 smithi063 ceph-mon[141124]: pgmap v482: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:13.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:12 smithi111 ceph-mon[106899]: pgmap v482: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:15.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:14 smithi063 ceph-mon[141124]: pgmap v483: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:15.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:14 smithi111 ceph-mon[106899]: pgmap v483: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:16.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:16 smithi063 ceph-mon[141124]: pgmap v484: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:16.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:16 smithi111 ceph-mon[106899]: pgmap v484: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:19.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:18 smithi063 ceph-mon[141124]: pgmap v485: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:19.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:18 smithi111 ceph-mon[106899]: pgmap v485: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:21.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:20 smithi063 ceph-mon[141124]: pgmap v486: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:21.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:20 smithi111 ceph-mon[106899]: pgmap v486: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:23.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:22 smithi063 ceph-mon[141124]: pgmap v487: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:22 smithi111 ceph-mon[106899]: pgmap v487: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:25.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:24 smithi063 ceph-mon[141124]: pgmap v488: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:25.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:24 smithi111 ceph-mon[106899]: pgmap v488: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:26.880 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:26 smithi063 ceph-mon[141124]: pgmap v489: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:26 smithi111 ceph-mon[106899]: pgmap v489: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:29.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:28 smithi063 ceph-mon[141124]: pgmap v490: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:29.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:28 smithi111 ceph-mon[106899]: pgmap v490: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:31.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:30 smithi063 ceph-mon[141124]: pgmap v491: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:31.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:30 smithi111 ceph-mon[106899]: pgmap v491: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:31.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.729+0000 7f2750400700 1 -- 172.21.15.63:0/879471283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 msgr2=0x7f2748106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:31.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.729+0000 7f2750400700 1 --2- 172.21.15.63:0/879471283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748106f00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2734004940 tx=0x7f2734009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:31.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.730+0000 7f2750400700 1 -- 172.21.15.63:0/879471283 shutdown_connections 2024-03-23T12:07:31.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.730+0000 7f2750400700 1 --2- 172.21.15.63:0/879471283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748106f00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:31.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.730+0000 7f2750400700 1 --2- 172.21.15.63:0/879471283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27480fde90 0x7f27480fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:31.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.730+0000 7f2750400700 1 -- 172.21.15.63:0/879471283 >> 172.21.15.63:0/879471283 conn(0x7f2748072bc0 msgr2=0x7f2748072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:31.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.730+0000 7f2750400700 1 -- 172.21.15.63:0/879471283 shutdown_connections 2024-03-23T12:07:31.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.730+0000 7f2750400700 1 -- 172.21.15.63:0/879471283 wait complete. 2024-03-23T12:07:31.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.731+0000 7f2750400700 1 Processor -- start 2024-03-23T12:07:31.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.731+0000 7f2750400700 1 -- start start 2024-03-23T12:07:31.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.731+0000 7f2750400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27480fde90 0x7f2748115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:31.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.731+0000 7f2750400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:31.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.731+0000 7f2750400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27481168a0 con 0x7f27480fe830 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.731+0000 7f2750400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274810fc70 con 0x7f27480fde90 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.731+0000 7f274d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.732+0000 7f274d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45804/0 (socket says 172.21.15.63:45804) 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.732+0000 7f274d000700 1 -- 172.21.15.63:0/3524629708 learned_addr learned my addr 172.21.15.63:0/3524629708 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.732+0000 7f274da00700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27480fde90 0x7f2748115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.732+0000 7f274d000700 1 -- 172.21.15.63:0/3524629708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27480fde90 msgr2=0x7f2748115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.732+0000 7f274d000700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27480fde90 0x7f2748115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.732+0000 7f274d000700 1 -- 172.21.15.63:0/3524629708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734005040 con 0x7f27480fe830 2024-03-23T12:07:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.732+0000 7f274d000700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748116250 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2734000c00 tx=0x7f2734009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:31.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.733+0000 7f2745600700 1 -- 172.21.15.63:0/3524629708 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f273400e040 con 0x7f27480fe830 2024-03-23T12:07:31.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.733+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274810fef0 con 0x7f27480fe830 2024-03-23T12:07:31.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.733+0000 7f2745600700 1 -- 172.21.15.63:0/3524629708 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f273400bed0 con 0x7f27480fe830 2024-03-23T12:07:31.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.733+0000 7f2745600700 1 -- 172.21.15.63:0/3524629708 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f273401c3f0 con 0x7f27480fe830 2024-03-23T12:07:31.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.733+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2748110430 con 0x7f27480fe830 2024-03-23T12:07:31.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.734+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2748106a20 con 0x7f27480fe830 2024-03-23T12:07:31.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.735+0000 7f2745600700 1 -- 172.21.15.63:0/3524629708 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2734013420 con 0x7f27480fe830 2024-03-23T12:07:31.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.736+0000 7f2745600700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f273006c7d0 0x7f273006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:31.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.736+0000 7f2745600700 1 -- 172.21.15.63:0/3524629708 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f273408cec0 con 0x7f27480fe830 2024-03-23T12:07:31.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.736+0000 7f274da00700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f273006c7d0 0x7f273006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:31.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.736+0000 7f274da00700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f273006c7d0 0x7f273006ec80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f273c004f40 tx=0x7f273c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:31.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.740+0000 7f2745600700 1 -- 172.21.15.63:0/3524629708 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f273405abd0 con 0x7f27480fe830 2024-03-23T12:07:31.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.972+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27481113d0 con 0x7f273006c7d0 2024-03-23T12:07:31.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.973+0000 7f2745600700 1 -- 172.21.15.63:0/3524629708 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f27481113d0 con 0x7f273006c7d0 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f273006c7d0 msgr2=0x7f273006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f273006c7d0 0x7f273006ec80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f273c004f40 tx=0x7f273c00b040 comp rx=0 tx=0).stop 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 msgr2=0x7f2748116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748116250 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2734000c00 tx=0x7f2734009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 shutdown_connections 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f273006c7d0 0x7f273006ec80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27480fde90 0x7f2748115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:31.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 --2- 172.21.15.63:0/3524629708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27480fe830 0x7f2748116250 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:31.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 >> 172.21.15.63:0/3524629708 conn(0x7f2748072bc0 msgr2=0x7f27480fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:31.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 shutdown_connections 2024-03-23T12:07:31.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:31.976+0000 7f2750400700 1 -- 172.21.15.63:0/3524629708 wait complete. 2024-03-23T12:07:31.972 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.115+0000 7f28b6600700 1 -- 172.21.15.63:0/496750521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 msgr2=0x7f28b00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.115+0000 7f28b6600700 1 --2- 172.21.15.63:0/496750521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b00fce10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f289c004940 tx=0x7f289c009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.115+0000 7f28b6600700 1 -- 172.21.15.63:0/496750521 shutdown_connections 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.115+0000 7f28b6600700 1 --2- 172.21.15.63:0/496750521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b00fce10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.115+0000 7f28b6600700 1 --2- 172.21.15.63:0/496750521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28b0102e00 0x7f28b01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.115+0000 7f28b6600700 1 -- 172.21.15.63:0/496750521 >> 172.21.15.63:0/496750521 conn(0x7f28b00f89b0 msgr2=0x7f28b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.115+0000 7f28b6600700 1 -- 172.21.15.63:0/496750521 shutdown_connections 2024-03-23T12:07:32.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.116+0000 7f28b6600700 1 -- 172.21.15.63:0/496750521 wait complete. 2024-03-23T12:07:32.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.116+0000 7f28b6600700 1 Processor -- start 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.116+0000 7f28b6600700 1 -- start start 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28b6600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28b0102e00 0x7f28b00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28b6600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28b6600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b0070d90 con 0x7f28b01037a0 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28b6600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b0070ed0 con 0x7f28b0102e00 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28aea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28aea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45830/0 (socket says 172.21.15.63:45830) 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28aea00700 1 -- 172.21.15.63:0/4063642748 learned_addr learned my addr 172.21.15.63:0/4063642748 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.117+0000 7f28af400700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28b0102e00 0x7f28b00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28aea00700 1 -- 172.21.15.63:0/4063642748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28b0102e00 msgr2=0x7f28b00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28aea00700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28b0102e00 0x7f28b00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28aea00700 1 -- 172.21.15.63:0/4063642748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c005040 con 0x7f28b01037a0 2024-03-23T12:07:32.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28aea00700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b0070800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f289c009390 tx=0x7f289c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:32.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28a7400700 1 -- 172.21.15.63:0/4063642748 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289c00e040 con 0x7f28b01037a0 2024-03-23T12:07:32.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28a7400700 1 -- 172.21.15.63:0/4063642748 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f289c013420 con 0x7f28b01037a0 2024-03-23T12:07:32.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b0071120 con 0x7f28b01037a0 2024-03-23T12:07:32.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28a7400700 1 -- 172.21.15.63:0/4063642748 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289c01c630 con 0x7f28b01037a0 2024-03-23T12:07:32.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.118+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b011a230 con 0x7f28b01037a0 2024-03-23T12:07:32.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.119+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b00fe550 con 0x7f28b01037a0 2024-03-23T12:07:32.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.120+0000 7f28a7400700 1 -- 172.21.15.63:0/4063642748 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f289c013a00 con 0x7f28b01037a0 2024-03-23T12:07:32.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.121+0000 7f28a7400700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289406c7d0 0x7f289406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.121+0000 7f28a7400700 1 -- 172.21.15.63:0/4063642748 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f289c08caa0 con 0x7f28b01037a0 2024-03-23T12:07:32.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.121+0000 7f28af400700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289406c7d0 0x7f289406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.122+0000 7f28af400700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289406c7d0 0x7f289406ec80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f28a0007fd0 tx=0x7f28a0009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:32.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.126+0000 7f28a7400700 1 -- 172.21.15.63:0/4063642748 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f289c056de0 con 0x7f28b01037a0 2024-03-23T12:07:32.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.357+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28b0109da0 con 0x7f289406c7d0 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.358+0000 7f28a7400700 1 -- 172.21.15.63:0/4063642748 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f28b0109da0 con 0x7f289406c7d0 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289406c7d0 msgr2=0x7f289406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289406c7d0 0x7f289406ec80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f28a0007fd0 tx=0x7f28a0009040 comp rx=0 tx=0).stop 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 msgr2=0x7f28b0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b0070800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f289c009390 tx=0x7f289c009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 shutdown_connections 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289406c7d0 0x7f289406ec80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28b0102e00 0x7f28b00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 --2- 172.21.15.63:0/4063642748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28b01037a0 0x7f28b0070800 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 >> 172.21.15.63:0/4063642748 conn(0x7f28b00f89b0 msgr2=0x7f28b00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:32.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 shutdown_connections 2024-03-23T12:07:32.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.361+0000 7f28b6600700 1 -- 172.21.15.63:0/4063642748 wait complete. 2024-03-23T12:07:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 -- 172.21.15.63:0/2402351543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3241037c0 msgr2=0x7fd3240fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 --2- 172.21.15.63:0/2402351543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3241037c0 0x7fd3240fce30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd310004940 tx=0x7fd310009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 -- 172.21.15.63:0/2402351543 shutdown_connections 2024-03-23T12:07:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 --2- 172.21.15.63:0/2402351543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3241037c0 0x7fd3240fce30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 --2- 172.21.15.63:0/2402351543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd324102e20 0x7fd3241031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 -- 172.21.15.63:0/2402351543 >> 172.21.15.63:0/2402351543 conn(0x7fd3240f89d0 msgr2=0x7fd3240fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 -- 172.21.15.63:0/2402351543 shutdown_connections 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.499+0000 7fd32a000700 1 -- 172.21.15.63:0/2402351543 wait complete. 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd32a000700 1 Processor -- start 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd32a000700 1 -- start start 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd32a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd324102e20 0x7fd3240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd32a000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3241037c0 0x7fd324070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd32a000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd324070d90 con 0x7fd324102e20 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd32a000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd324070ed0 con 0x7fd3241037c0 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd323400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd324102e20 0x7fd3240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd323400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd324102e20 0x7fd3240702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45858/0 (socket says 172.21.15.63:45858) 2024-03-23T12:07:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.500+0000 7fd323400700 1 -- 172.21.15.63:0/1490544643 learned_addr learned my addr 172.21.15.63:0/1490544643 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd322a00700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3241037c0 0x7fd324070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd323400700 1 -- 172.21.15.63:0/1490544643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3241037c0 msgr2=0x7fd324070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd323400700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3241037c0 0x7fd324070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd323400700 1 -- 172.21.15.63:0/1490544643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310005040 con 0x7fd324102e20 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd323400700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd324102e20 0x7fd3240702c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd314002a50 tx=0x7fd3140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd31b400700 1 -- 172.21.15.63:0/1490544643 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd314007cc0 con 0x7fd324102e20 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd31b400700 1 -- 172.21.15.63:0/1490544643 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd31400f040 con 0x7fd324102e20 2024-03-23T12:07:32.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd324119e20 con 0x7fd324102e20 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.501+0000 7fd31b400700 1 -- 172.21.15.63:0/1490544643 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd31400d9c0 con 0x7fd324102e20 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.502+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32411a2a0 con 0x7fd324102e20 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.502+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3240fe570 con 0x7fd324102e20 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.504+0000 7fd31b400700 1 -- 172.21.15.63:0/1490544643 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd31400db60 con 0x7fd324102e20 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.504+0000 7fd31b400700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd30806c7c0 0x7fd30806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.505+0000 7fd31b400700 1 -- 172.21.15.63:0/1490544643 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd31408b1b0 con 0x7fd324102e20 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.505+0000 7fd322a00700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd30806c7c0 0x7fd30806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.505+0000 7fd322a00700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd30806c7c0 0x7fd30806ec70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd310004890 tx=0x7fd310009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:32.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.509+0000 7fd31b400700 1 -- 172.21.15.63:0/1490544643 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd314055330 con 0x7fd324102e20 2024-03-23T12:07:32.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.732+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3240719b0 con 0x7fd30806c7c0 2024-03-23T12:07:32.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.743+0000 7fd31b400700 1 -- 172.21.15.63:0/1490544643 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fd3240719b0 con 0x7fd30806c7c0 2024-03-23T12:07:32.724 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:07:32.724 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (16m) 9m ago 29m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:07:32.724 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (14m) 9m ago 30m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:07:32.724 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (13m) 9m ago 29m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:07:32.724 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (16m) 9m ago 29m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:07:32.724 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (9m) 9m ago 23m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (9m) 9m ago 23m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (23m) 9m ago 23m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (9m) 9m ago 23m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (19m) 9m ago 31m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (17m) 9m ago 28m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (15m) 9m ago 31m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (14m) 9m ago 28m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (17m) 9m ago 30m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (17m) 9m ago 28m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (13m) 9m ago 27m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (12m) 9m ago 27m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (12m) 9m ago 26m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (11m) 9m ago 26m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (10m) 9m ago 25m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:07:32.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (10m) 9m ago 25m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:07:32.726 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (17m) 9m ago 29m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:07:32.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd30806c7c0 msgr2=0x7fd30806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd30806c7c0 0x7fd30806ec70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd310004890 tx=0x7fd310009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:32.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd324102e20 msgr2=0x7fd3240702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd324102e20 0x7fd3240702c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd314002a50 tx=0x7fd3140046b0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 shutdown_connections 2024-03-23T12:07:32.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd30806c7c0 0x7fd30806ec70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd324102e20 0x7fd3240702c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 --2- 172.21.15.63:0/1490544643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3241037c0 0x7fd324070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 >> 172.21.15.63:0/1490544643 conn(0x7fd3240f89d0 msgr2=0x7fd3240f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:32.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.745+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 shutdown_connections 2024-03-23T12:07:32.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.746+0000 7fd32a000700 1 -- 172.21.15.63:0/1490544643 wait complete. 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.882+0000 7f0e93800700 1 -- 172.21.15.63:0/1113874443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c102e00 msgr2=0x7f0e8c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.882+0000 7f0e93800700 1 --2- 172.21.15.63:0/1113874443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c102e00 0x7f0e8c1031d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0e80004940 tx=0x7f0e80009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.882+0000 7f0e93800700 1 -- 172.21.15.63:0/1113874443 shutdown_connections 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.882+0000 7f0e93800700 1 --2- 172.21.15.63:0/1113874443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1037a0 0x7f0e8c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.882+0000 7f0e93800700 1 --2- 172.21.15.63:0/1113874443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c102e00 0x7f0e8c1031d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.882+0000 7f0e93800700 1 -- 172.21.15.63:0/1113874443 >> 172.21.15.63:0/1113874443 conn(0x7f0e8c0f89b0 msgr2=0x7f0e8c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.882+0000 7f0e93800700 1 -- 172.21.15.63:0/1113874443 shutdown_connections 2024-03-23T12:07:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.883+0000 7f0e93800700 1 -- 172.21.15.63:0/1113874443 wait complete. 2024-03-23T12:07:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.883+0000 7f0e93800700 1 Processor -- start 2024-03-23T12:07:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.883+0000 7f0e93800700 1 -- start start 2024-03-23T12:07:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e93800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c102e00 0x7f0e8c115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e93800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c1037a0 0x7f0e8c116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e93800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c116880 con 0x7f0e8c1037a0 2024-03-23T12:07:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e93800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c10fc50 con 0x7f0e8c102e00 2024-03-23T12:07:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e8be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c1037a0 0x7f0e8c116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e8be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c1037a0 0x7f0e8c116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45870/0 (socket says 172.21.15.63:45870) 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e8be00700 1 -- 172.21.15.63:0/3505171889 learned_addr learned my addr 172.21.15.63:0/3505171889 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.884+0000 7f0e90e00700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c102e00 0x7f0e8c115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e8be00700 1 -- 172.21.15.63:0/3505171889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c102e00 msgr2=0x7f0e8c115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e8be00700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c102e00 0x7f0e8c115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e8be00700 1 -- 172.21.15.63:0/3505171889 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e80005040 con 0x7f0e8c1037a0 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e8be00700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c1037a0 0x7f0e8c116230 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c890 tx=0x7f0e780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e88c00700 1 -- 172.21.15.63:0/3505171889 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7800a700 con 0x7f0e8c1037a0 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e88c00700 1 -- 172.21.15.63:0/3505171889 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e7800ad10 con 0x7f0e8c1037a0 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e88c00700 1 -- 172.21.15.63:0/3505171889 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e78012e70 con 0x7f0e8c1037a0 2024-03-23T12:07:32.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c10ff30 con 0x7f0e8c1037a0 2024-03-23T12:07:32.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.885+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c1104f0 con 0x7f0e8c1037a0 2024-03-23T12:07:32.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.886+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c0fe550 con 0x7f0e8c1037a0 2024-03-23T12:07:32.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.887+0000 7f0e88c00700 1 -- 172.21.15.63:0/3505171889 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0e7800a860 con 0x7f0e8c1037a0 2024-03-23T12:07:32.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.888+0000 7f0e88c00700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e7406c7d0 0x7f0e7406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:32.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.888+0000 7f0e88c00700 1 -- 172.21.15.63:0/3505171889 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0e7808ba90 con 0x7f0e8c1037a0 2024-03-23T12:07:32.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.888+0000 7f0e90e00700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e7406c7d0 0x7f0e7406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:32.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.889+0000 7f0e90e00700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e7406c7d0 0x7f0e7406ec80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009fd0 tx=0x7f0e80009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:32.893+0000 7f0e88c00700 1 -- 172.21.15.63:0/3505171889 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0e780597a0 con 0x7f0e8c1037a0 2024-03-23T12:07:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:32 smithi063 ceph-mon[141124]: pgmap v492: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:32 smithi063 ceph-mon[141124]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:33.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.191+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0e8c04bed0 con 0x7f0e8c1037a0 2024-03-23T12:07:33.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.192+0000 7f0e88c00700 1 -- 172.21.15.63:0/3505171889 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f0e78059330 con 0x7f0e8c1037a0 2024-03-23T12:07:33.173 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:07:33.173 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:07:33.173 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:07:33.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:07:33.174 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:07:33.175 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:07:33.175 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e7406c7d0 msgr2=0x7f0e7406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e7406c7d0 0x7f0e7406ec80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009fd0 tx=0x7f0e80009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c1037a0 msgr2=0x7f0e8c116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c1037a0 0x7f0e8c116230 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c890 tx=0x7f0e780044c0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 shutdown_connections 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e7406c7d0 0x7f0e7406ec80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c102e00 0x7f0e8c115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 --2- 172.21.15.63:0/3505171889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e8c1037a0 0x7f0e8c116230 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.194+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 >> 172.21.15.63:0/3505171889 conn(0x7f0e8c0f89b0 msgr2=0x7f0e8c0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.195+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 shutdown_connections 2024-03-23T12:07:33.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.195+0000 7f0e93800700 1 -- 172.21.15.63:0/3505171889 wait complete. 2024-03-23T12:07:33.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 -- 172.21.15.63:0/760470175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 msgr2=0x7fa7ec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/760470175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec1031d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc004940 tx=0x7fa7dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:33.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 -- 172.21.15.63:0/760470175 shutdown_connections 2024-03-23T12:07:33.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/760470175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7ec1037a0 0x7fa7ec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/760470175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec1031d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 -- 172.21.15.63:0/760470175 >> 172.21.15.63:0/760470175 conn(0x7fa7ec0f89b0 msgr2=0x7fa7ec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:33.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 -- 172.21.15.63:0/760470175 shutdown_connections 2024-03-23T12:07:33.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.331+0000 7fa7f2a00700 1 -- 172.21.15.63:0/760470175 wait complete. 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.332+0000 7fa7f2a00700 1 Processor -- start 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.332+0000 7fa7f2a00700 1 -- start start 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.332+0000 7fa7f2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.332+0000 7fa7f2a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7ec1037a0 0x7fa7ec116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.332+0000 7fa7f2a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1167d0 con 0x7fa7ec102e00 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.332+0000 7fa7f2a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec10fba0 con 0x7fa7ec1037a0 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:33.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45886/0 (socket says 172.21.15.63:45886) 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7ebe00700 1 -- 172.21.15.63:0/3865772093 learned_addr learned my addr 172.21.15.63:0/3865772093 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7eb400700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7ec1037a0 0x7fa7ec116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7ebe00700 1 -- 172.21.15.63:0/3865772093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7ec1037a0 msgr2=0x7fa7ec116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7ebe00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7ec1037a0 0x7fa7ec116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7ebe00700 1 -- 172.21.15.63:0/3865772093 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc005040 con 0x7fa7ec102e00 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.333+0000 7fa7ebe00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec115c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc007f50 tx=0x7fa7dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.334+0000 7fa7e3e00700 1 -- 172.21.15.63:0/3865772093 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc021070 con 0x7fa7ec102e00 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.334+0000 7fa7e3e00700 1 -- 172.21.15.63:0/3865772093 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7dc0093b0 con 0x7fa7ec102e00 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.334+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec10fe20 con 0x7fa7ec102e00 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.334+0000 7fa7e3e00700 1 -- 172.21.15.63:0/3865772093 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc00e070 con 0x7fa7ec102e00 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.334+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec110380 con 0x7fa7ec102e00 2024-03-23T12:07:33.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.335+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec0fe550 con 0x7fa7ec102e00 2024-03-23T12:07:33.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.336+0000 7fa7e3e00700 1 -- 172.21.15.63:0/3865772093 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa7dc01c400 con 0x7fa7ec102e00 2024-03-23T12:07:33.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.337+0000 7fa7e3e00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:33.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.337+0000 7fa7e3e00700 1 -- 172.21.15.63:0/3865772093 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa7dc08caa0 con 0x7fa7ec102e00 2024-03-23T12:07:33.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.337+0000 7fa7eb400700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:33.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.338+0000 7fa7eb400700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec1114e0 tx=0x7fa7d800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:33.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.341+0000 7fa7e3e00700 1 -- 172.21.15.63:0/3865772093 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa7dc05a870 con 0x7fa7ec102e00 2024-03-23T12:07:33.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:32 smithi111 ceph-mon[106899]: pgmap v492: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:33.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:32 smithi111 ceph-mon[106899]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:33.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.602+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa7ec04bed0 con 0x7fa7ec102e00 2024-03-23T12:07:33.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.603+0000 7fa7e3e00700 1 -- 172.21.15.63:0/3865772093 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fa7dc05a400 con 0x7fa7ec102e00 2024-03-23T12:07:33.584 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:07:33.584 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:07:33.584 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:07:33.584 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:07:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:33.584 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:07:33.585 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:33.587 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:07:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:07:33.587 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:33.587 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:33.587 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:33.587 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 msgr2=0x7fa7d006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec1114e0 tx=0x7fa7d800b040 comp rx=0 tx=0).stop 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 msgr2=0x7fa7ec115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec115c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc007f50 tx=0x7fa7dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 shutdown_connections 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7ec102e00 0x7fa7ec115c40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 --2- 172.21.15.63:0/3865772093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7ec1037a0 0x7fa7ec116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 >> 172.21.15.63:0/3865772093 conn(0x7fa7ec0f89b0 msgr2=0x7fa7ec0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 shutdown_connections 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.605+0000 7fa7f2a00700 1 -- 172.21.15.63:0/3865772093 wait complete. 2024-03-23T12:07:33.588 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:07:33.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 -- 172.21.15.63:0/374509535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01281037a0 msgr2=0x7f01280fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 --2- 172.21.15.63:0/374509535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01281037a0 0x7f01280fce10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0114004940 tx=0x7f0114009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 -- 172.21.15.63:0/374509535 shutdown_connections 2024-03-23T12:07:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 --2- 172.21.15.63:0/374509535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01281037a0 0x7f01280fce10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 --2- 172.21.15.63:0/374509535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0128102e00 0x7f01281031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 -- 172.21.15.63:0/374509535 >> 172.21.15.63:0/374509535 conn(0x7f01280f89b0 msgr2=0x7f01280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 -- 172.21.15.63:0/374509535 shutdown_connections 2024-03-23T12:07:33.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.744+0000 7f012fc00700 1 -- 172.21.15.63:0/374509535 wait complete. 2024-03-23T12:07:33.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.745+0000 7f012fc00700 1 Processor -- start 2024-03-23T12:07:33.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.745+0000 7f012fc00700 1 -- start start 2024-03-23T12:07:33.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.745+0000 7f012fc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128102e00 0x7f0128115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:33.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.745+0000 7f012fc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01281037a0 0x7f01281161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:33.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.745+0000 7f012fc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128116810 con 0x7f0128102e00 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012fc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012810fbe0 con 0x7f01281037a0 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128102e00 0x7f0128115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128102e00 0x7f0128115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45898/0 (socket says 172.21.15.63:45898) 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012d200700 1 -- 172.21.15.63:0/2749450298 learned_addr learned my addr 172.21.15.63:0/2749450298 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f0127e00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01281037a0 0x7f01281161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012d200700 1 -- 172.21.15.63:0/2749450298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01281037a0 msgr2=0x7f01281161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012d200700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01281037a0 0x7f01281161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012d200700 1 -- 172.21.15.63:0/2749450298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0114005040 con 0x7f0128102e00 2024-03-23T12:07:33.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.746+0000 7f012d200700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128102e00 0x7f0128115c80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f011c002a80 tx=0x7f011c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:33.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.747+0000 7f0124c00700 1 -- 172.21.15.63:0/2749450298 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f011c007cc0 con 0x7f0128102e00 2024-03-23T12:07:33.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.747+0000 7f0124c00700 1 -- 172.21.15.63:0/2749450298 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f011c00f040 con 0x7f0128102e00 2024-03-23T12:07:33.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.747+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012810fec0 con 0x7f0128102e00 2024-03-23T12:07:33.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.747+0000 7f0124c00700 1 -- 172.21.15.63:0/2749450298 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f011c00da60 con 0x7f0128102e00 2024-03-23T12:07:33.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.747+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0128110480 con 0x7f0128102e00 2024-03-23T12:07:33.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.748+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01280fe550 con 0x7f0128102e00 2024-03-23T12:07:33.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.749+0000 7f0124c00700 1 -- 172.21.15.63:0/2749450298 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f011c00dbc0 con 0x7f0128102e00 2024-03-23T12:07:33.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.750+0000 7f0124c00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f011006c7d0 0x7f011006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:33.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.750+0000 7f0124c00700 1 -- 172.21.15.63:0/2749450298 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f011c08b630 con 0x7f0128102e00 2024-03-23T12:07:33.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.750+0000 7f0127e00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f011006c7d0 0x7f011006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:33.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.751+0000 7f0127e00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f011006c7d0 0x7f011006ec80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0128111520 tx=0x7f0114009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:33.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.755+0000 7f0124c00700 1 -- 172.21.15.63:0/2749450298 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f011c059340 con 0x7f0128102e00 2024-03-23T12:07:33.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.988+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f012804bf40 con 0x7f011006c7d0 2024-03-23T12:07:33.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.989+0000 7f0124c00700 1 -- 172.21.15.63:0/2749450298 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f012804bf40 con 0x7f011006c7d0 2024-03-23T12:07:33.970 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:07:33.970 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:07:33.970 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:07:33.971 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f011006c7d0 msgr2=0x7f011006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f011006c7d0 0x7f011006ec80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0128111520 tx=0x7f0114009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128102e00 msgr2=0x7f0128115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128102e00 0x7f0128115c80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f011c002a80 tx=0x7f011c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 shutdown_connections 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f011006c7d0 0x7f011006ec80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128102e00 0x7f0128115c80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 --2- 172.21.15.63:0/2749450298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01281037a0 0x7f01281161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.991+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 >> 172.21.15.63:0/2749450298 conn(0x7f01280f89b0 msgr2=0x7f01280f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:33.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.992+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 shutdown_connections 2024-03-23T12:07:33.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:33.992+0000 7f012fc00700 1 -- 172.21.15.63:0/2749450298 wait complete. 2024-03-23T12:07:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 -- 172.21.15.63:0/785411964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c102e00 msgr2=0x7f477c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 --2- 172.21.15.63:0/785411964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c102e00 0x7f477c1031d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f476c004940 tx=0x7f476c009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 -- 172.21.15.63:0/785411964 shutdown_connections 2024-03-23T12:07:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 --2- 172.21.15.63:0/785411964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f477c1037a0 0x7f477c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 --2- 172.21.15.63:0/785411964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c102e00 0x7f477c1031d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 -- 172.21.15.63:0/785411964 >> 172.21.15.63:0/785411964 conn(0x7f477c0f89b0 msgr2=0x7f477c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 -- 172.21.15.63:0/785411964 shutdown_connections 2024-03-23T12:07:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.129+0000 7f4782400700 1 -- 172.21.15.63:0/785411964 wait complete. 2024-03-23T12:07:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.130+0000 7f4782400700 1 Processor -- start 2024-03-23T12:07:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.130+0000 7f4782400700 1 -- start start 2024-03-23T12:07:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.130+0000 7f4782400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f477c102e00 0x7f477c115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.130+0000 7f4782400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c1037a0 0x7f477c115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.130+0000 7f4782400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c1165a0 con 0x7f477c1037a0 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.130+0000 7f4782400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c10fba0 con 0x7f477c102e00 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c1037a0 0x7f477c115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c1037a0 0x7f477c115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45920/0 (socket says 172.21.15.63:45920) 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477aa00700 1 -- 172.21.15.63:0/2869096632 learned_addr learned my addr 172.21.15.63:0/2869096632 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477b400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f477c102e00 0x7f477c115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477aa00700 1 -- 172.21.15.63:0/2869096632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f477c102e00 msgr2=0x7f477c115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477aa00700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f477c102e00 0x7f477c115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477aa00700 1 -- 172.21.15.63:0/2869096632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476c005040 con 0x7f477c1037a0 2024-03-23T12:07:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f477aa00700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c1037a0 0x7f477c115f50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4768002a50 tx=0x7f47680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.131+0000 7f4773400700 1 -- 172.21.15.63:0/2869096632 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4768007cc0 con 0x7f477c1037a0 2024-03-23T12:07:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.132+0000 7f4773400700 1 -- 172.21.15.63:0/2869096632 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f476800f040 con 0x7f477c1037a0 2024-03-23T12:07:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.132+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477c10fe80 con 0x7f477c1037a0 2024-03-23T12:07:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.132+0000 7f4773400700 1 -- 172.21.15.63:0/2869096632 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47680095d0 con 0x7f477c1037a0 2024-03-23T12:07:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.132+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c110440 con 0x7f477c1037a0 2024-03-23T12:07:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.133+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477c0fe550 con 0x7f477c1037a0 2024-03-23T12:07:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.134+0000 7f4773400700 1 -- 172.21.15.63:0/2869096632 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4768009730 con 0x7f477c1037a0 2024-03-23T12:07:34.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.135+0000 7f4773400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f476006c7c0 0x7f476006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:07:34.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.135+0000 7f4773400700 1 -- 172.21.15.63:0/2869096632 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f476808b7f0 con 0x7f477c1037a0 2024-03-23T12:07:34.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.135+0000 7f477b400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f476006c7c0 0x7f476006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:07:34.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.135+0000 7f477b400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f476006c7c0 0x7f476006ec70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f476c00b020 tx=0x7f476c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:07:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.139+0000 7f4773400700 1 -- 172.21.15.63:0/2869096632 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4768059400 con 0x7f477c1037a0 2024-03-23T12:07:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:33 smithi063 ceph-mon[141124]: from='client.34942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:33 smithi063 ceph-mon[141124]: from='client.34946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:33 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3505171889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:07:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:33 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3865772093' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:07:34.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:33 smithi111 ceph-mon[106899]: from='client.34942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:34.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:33 smithi111 ceph-mon[106899]: from='client.34946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:34.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:33 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3505171889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:07:34.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:33 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3865772093' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:07:34.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.435+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f477c109dc0 con 0x7f477c1037a0 2024-03-23T12:07:34.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.436+0000 7f4773400700 1 -- 172.21.15.63:0/2869096632 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4768058f90 con 0x7f477c1037a0 2024-03-23T12:07:34.417 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:07:34.418 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:07:34.418 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:07:34.418 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:07:34.418 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:07:34.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f476006c7c0 msgr2=0x7f476006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:34.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f476006c7c0 0x7f476006ec70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f476c00b020 tx=0x7f476c009e30 comp rx=0 tx=0).stop 2024-03-23T12:07:34.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c1037a0 msgr2=0x7f477c115f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:07:34.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c1037a0 0x7f477c115f50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4768002a50 tx=0x7f47680046b0 comp rx=0 tx=0).stop 2024-03-23T12:07:34.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 shutdown_connections 2024-03-23T12:07:34.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f476006c7c0 0x7f476006ec70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:34.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f477c102e00 0x7f477c115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:34.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 --2- 172.21.15.63:0/2869096632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f477c1037a0 0x7f477c115f50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:07:34.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 >> 172.21.15.63:0/2869096632 conn(0x7f477c0f89b0 msgr2=0x7f477c071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:07:34.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 shutdown_connections 2024-03-23T12:07:34.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:07:34.438+0000 7f4782400700 1 -- 172.21.15.63:0/2869096632 wait complete. 2024-03-23T12:07:35.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:34 smithi063 ceph-mon[141124]: pgmap v493: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:35.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:34 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2869096632' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:07:35.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:34 smithi111 ceph-mon[106899]: pgmap v493: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:35.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:34 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2869096632' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:07:36.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:35 smithi063 ceph-mon[141124]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:36.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:35 smithi111 ceph-mon[106899]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:07:37.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:36 smithi063 ceph-mon[141124]: pgmap v494: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:37.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:36 smithi111 ceph-mon[106899]: pgmap v494: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:38.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:37 smithi063 ceph-mon[141124]: pgmap v495: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:38.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:38 smithi111 ceph-mon[106899]: pgmap v495: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:41.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:40 smithi063 ceph-mon[141124]: pgmap v496: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:41.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:40 smithi111 ceph-mon[106899]: pgmap v496: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:43.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:42 smithi063 ceph-mon[141124]: pgmap v497: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:43.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:42 smithi111 ceph-mon[106899]: pgmap v497: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:45.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:44 smithi063 ceph-mon[141124]: pgmap v498: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:45.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:44 smithi111 ceph-mon[106899]: pgmap v498: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:46.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:46 smithi063 ceph-mon[141124]: pgmap v499: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:46.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:46 smithi111 ceph-mon[106899]: pgmap v499: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:49.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:48 smithi063 ceph-mon[141124]: pgmap v500: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:49.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:48 smithi111 ceph-mon[106899]: pgmap v500: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:51.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:50 smithi063 ceph-mon[141124]: pgmap v501: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:51.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:50 smithi111 ceph-mon[106899]: pgmap v501: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:53.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:52 smithi063 ceph-mon[141124]: pgmap v502: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:53.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:52 smithi111 ceph-mon[106899]: pgmap v502: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:55.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:54 smithi063 ceph-mon[141124]: pgmap v503: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:54 smithi111 ceph-mon[106899]: pgmap v503: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:56 smithi063 ceph-mon[141124]: pgmap v504: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:07:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:56 smithi111 ceph-mon[106899]: pgmap v504: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:59.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:07:58 smithi063 ceph-mon[141124]: pgmap v505: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:07:59.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:07:58 smithi111 ceph-mon[106899]: pgmap v505: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:01.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:00 smithi063 ceph-mon[141124]: pgmap v506: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:01.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:00 smithi111 ceph-mon[106899]: pgmap v506: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:03.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:02 smithi063 ceph-mon[141124]: pgmap v507: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:03.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:02 smithi111 ceph-mon[106899]: pgmap v507: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:04.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 -- 172.21.15.63:0/744899100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070360 msgr2=0x7ffbb0070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:04.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/744899100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070360 0x7ffbb0070730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffba0004970 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:04.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 -- 172.21.15.63:0/744899100 shutdown_connections 2024-03-23T12:08:04.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/744899100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0070c70 0x7ffbb0109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/744899100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070360 0x7ffbb0070730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 -- 172.21.15.63:0/744899100 >> 172.21.15.63:0/744899100 conn(0x7ffbb00f89b0 msgr2=0x7ffbb00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:04.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 -- 172.21.15.63:0/744899100 shutdown_connections 2024-03-23T12:08:04.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.592+0000 7ffbb6e00700 1 -- 172.21.15.63:0/744899100 wait complete. 2024-03-23T12:08:04.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.593+0000 7ffbb6e00700 1 Processor -- start 2024-03-23T12:08:04.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.593+0000 7ffbb6e00700 1 -- start start 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.593+0000 7ffbb6e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0070360 0x7ffbb0115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbb6e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070c70 0x7ffbb0116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbb6e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0116760 con 0x7ffbb0070c70 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbb6e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb010fbe0 con 0x7ffbb0070360 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbaf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070c70 0x7ffbb0116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbaf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070c70 0x7ffbb0116110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48270/0 (socket says 172.21.15.63:48270) 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbaf400700 1 -- 172.21.15.63:0/1509004460 learned_addr learned my addr 172.21.15.63:0/1509004460 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbafe00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0070360 0x7ffbb0115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbaf400700 1 -- 172.21.15.63:0/1509004460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0070360 msgr2=0x7ffbb0115bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbaf400700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0070360 0x7ffbb0115bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.594+0000 7ffbaf400700 1 -- 172.21.15.63:0/1509004460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0005040 con 0x7ffbb0070c70 2024-03-23T12:08:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.595+0000 7ffbaf400700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070c70 0x7ffbb0116110 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c002a50 tx=0x7ffb9c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.595+0000 7ffba7e00700 1 -- 172.21.15.63:0/1509004460 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb9c007cc0 con 0x7ffbb0070c70 2024-03-23T12:08:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.595+0000 7ffba7e00700 1 -- 172.21.15.63:0/1509004460 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb9c00f040 con 0x7ffbb0070c70 2024-03-23T12:08:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.595+0000 7ffba7e00700 1 -- 172.21.15.63:0/1509004460 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb9c00d9c0 con 0x7ffbb0070c70 2024-03-23T12:08:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.595+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb010fec0 con 0x7ffbb0070c70 2024-03-23T12:08:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.595+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb0110400 con 0x7ffbb0070c70 2024-03-23T12:08:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.596+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0071aa0 con 0x7ffbb0070c70 2024-03-23T12:08:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.597+0000 7ffba7e00700 1 -- 172.21.15.63:0/1509004460 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ffb9c00db20 con 0x7ffbb0070c70 2024-03-23T12:08:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.598+0000 7ffba7e00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb9406c7d0 0x7ffb9406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.598+0000 7ffba7e00700 1 -- 172.21.15.63:0/1509004460 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffb9c08b6e0 con 0x7ffbb0070c70 2024-03-23T12:08:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.599+0000 7ffbafe00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb9406c7d0 0x7ffb9406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.599+0000 7ffbafe00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb9406c7d0 0x7ffb9406ec80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffba000b370 tx=0x7ffba0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:04.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.603+0000 7ffba7e00700 1 -- 172.21.15.63:0/1509004460 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb9c055960 con 0x7ffbb0070c70 2024-03-23T12:08:04.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.834+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbb010bfb0 con 0x7ffb9406c7d0 2024-03-23T12:08:04.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.835+0000 7ffba7e00700 1 -- 172.21.15.63:0/1509004460 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffbb010bfb0 con 0x7ffb9406c7d0 2024-03-23T12:08:04.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb9406c7d0 msgr2=0x7ffb9406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:04.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb9406c7d0 0x7ffb9406ec80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffba000b370 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:04.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070c70 msgr2=0x7ffbb0116110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:04.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070c70 0x7ffbb0116110 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c002a50 tx=0x7ffb9c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 shutdown_connections 2024-03-23T12:08:04.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ffb9406c7d0 0x7ffb9406ec80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0070360 0x7ffbb0115bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 --2- 172.21.15.63:0/1509004460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbb0070c70 0x7ffbb0116110 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 >> 172.21.15.63:0/1509004460 conn(0x7ffbb00f89b0 msgr2=0x7ffbb01015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:04.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 shutdown_connections 2024-03-23T12:08:04.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.838+0000 7ffbb6e00700 1 -- 172.21.15.63:0/1509004460 wait complete. 2024-03-23T12:08:04.834 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:08:04.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 -- 172.21.15.63:0/1139537373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e0071f90 msgr2=0x7f09e0070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:04.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 --2- 172.21.15.63:0/1139537373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e0071f90 0x7f09e0070360 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f09d4004970 tx=0x7f09d4009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:04.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 -- 172.21.15.63:0/1139537373 shutdown_connections 2024-03-23T12:08:04.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 --2- 172.21.15.63:0/1139537373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e00708a0 0x7f09e0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 --2- 172.21.15.63:0/1139537373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e0071f90 0x7f09e0070360 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 -- 172.21.15.63:0/1139537373 >> 172.21.15.63:0/1139537373 conn(0x7f09e00f89b0 msgr2=0x7f09e00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:04.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 -- 172.21.15.63:0/1139537373 shutdown_connections 2024-03-23T12:08:04.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.976+0000 7f09e7e00700 1 -- 172.21.15.63:0/1139537373 wait complete. 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.977+0000 7f09e7e00700 1 Processor -- start 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.977+0000 7f09e7e00700 1 -- start start 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.977+0000 7f09e7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e00708a0 0x7f09e011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.977+0000 7f09e7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0071f90 0x7f09e011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.977+0000 7f09e7e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e011ac50 con 0x7f09e00708a0 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.977+0000 7f09e7e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e0114020 con 0x7f09e0071f90 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e00708a0 0x7f09e011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e00708a0 0x7f09e011a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48284/0 (socket says 172.21.15.63:48284) 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e5400700 1 -- 172.21.15.63:0/501898875 learned_addr learned my addr 172.21.15.63:0/501898875 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e4a00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0071f90 0x7f09e011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e5400700 1 -- 172.21.15.63:0/501898875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0071f90 msgr2=0x7f09e011a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e5400700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0071f90 0x7f09e011a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e5400700 1 -- 172.21.15.63:0/501898875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d4005040 con 0x7f09e00708a0 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09e5400700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e00708a0 0x7f09e011a0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f09d4004f50 tx=0x7f09d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.978+0000 7f09dd600700 1 -- 172.21.15.63:0/501898875 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09d400e040 con 0x7f09e00708a0 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.979+0000 7f09dd600700 1 -- 172.21.15.63:0/501898875 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09d4013420 con 0x7f09e00708a0 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.979+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e01142a0 con 0x7f09e00708a0 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.979+0000 7f09dd600700 1 -- 172.21.15.63:0/501898875 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09d401c610 con 0x7f09e00708a0 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.979+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e01147e0 con 0x7f09e00708a0 2024-03-23T12:08:04.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.980+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e0071aa0 con 0x7f09e00708a0 2024-03-23T12:08:04.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.981+0000 7f09dd600700 1 -- 172.21.15.63:0/501898875 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f09d4013590 con 0x7f09e00708a0 2024-03-23T12:08:04.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.981+0000 7f09dd600700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09c806c7d0 0x7f09c806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:04.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.982+0000 7f09dd600700 1 -- 172.21.15.63:0/501898875 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f09d408cd50 con 0x7f09e00708a0 2024-03-23T12:08:04.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.982+0000 7f09e4a00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09c806c7d0 0x7f09c806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:04.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.982+0000 7f09e4a00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09c806c7d0 0x7f09c806ec80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f09cc007fd0 tx=0x7f09cc009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:04.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:04.986+0000 7f09dd600700 1 -- 172.21.15.63:0/501898875 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f09d405aa90 con 0x7f09e00708a0 2024-03-23T12:08:05.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.219+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09e0071aa0 con 0x7f09c806c7d0 2024-03-23T12:08:05.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.220+0000 7f09dd600700 1 -- 172.21.15.63:0/501898875 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f09e0071aa0 con 0x7f09c806c7d0 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.222+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09c806c7d0 msgr2=0x7f09c806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.222+0000 7f09e7e00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09c806c7d0 0x7f09c806ec80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f09cc007fd0 tx=0x7f09cc009040 comp rx=0 tx=0).stop 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.222+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e00708a0 msgr2=0x7f09e011a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.222+0000 7f09e7e00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e00708a0 0x7f09e011a0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f09d4004f50 tx=0x7f09d4009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.223+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 shutdown_connections 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.223+0000 7f09e7e00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09c806c7d0 0x7f09c806ec80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.223+0000 7f09e7e00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09e00708a0 0x7f09e011a0c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.223+0000 7f09e7e00700 1 --2- 172.21.15.63:0/501898875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0071f90 0x7f09e011a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.223+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 >> 172.21.15.63:0/501898875 conn(0x7f09e00f89b0 msgr2=0x7f09e00f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.223+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 shutdown_connections 2024-03-23T12:08:05.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.223+0000 7f09e7e00700 1 -- 172.21.15.63:0/501898875 wait complete. 2024-03-23T12:08:05.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 -- 172.21.15.63:0/3129830823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d58102e00 msgr2=0x7f6d581031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/3129830823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d58102e00 0x7f6d581031d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6d48004940 tx=0x7f6d48009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 -- 172.21.15.63:0/3129830823 shutdown_connections 2024-03-23T12:08:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/3129830823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d581037a0 0x7f6d580fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/3129830823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d58102e00 0x7f6d581031d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 -- 172.21.15.63:0/3129830823 >> 172.21.15.63:0/3129830823 conn(0x7f6d580f89b0 msgr2=0x7f6d580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 -- 172.21.15.63:0/3129830823 shutdown_connections 2024-03-23T12:08:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.362+0000 7f6d5ca00700 1 -- 172.21.15.63:0/3129830823 wait complete. 2024-03-23T12:08:05.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.363+0000 7f6d5ca00700 1 Processor -- start 2024-03-23T12:08:05.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.363+0000 7f6d5ca00700 1 -- start start 2024-03-23T12:08:05.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.363+0000 7f6d5ca00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d58102e00 0x7f6d58113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:05.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.363+0000 7f6d5ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d581037a0 0x7f6d58113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:05.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d5ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d581145d0 con 0x7f6d581037a0 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d5ca00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5810d9c0 con 0x7f6d58102e00 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d4fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d581037a0 0x7f6d58113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d4fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d581037a0 0x7f6d58113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48300/0 (socket says 172.21.15.63:48300) 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d4fe00700 1 -- 172.21.15.63:0/203541045 learned_addr learned my addr 172.21.15.63:0/203541045 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d55e00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d58102e00 0x7f6d58113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d4fe00700 1 -- 172.21.15.63:0/203541045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d58102e00 msgr2=0x7f6d58113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d4fe00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d58102e00 0x7f6d58113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.364+0000 7f6d4fe00700 1 -- 172.21.15.63:0/203541045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d48005040 con 0x7f6d581037a0 2024-03-23T12:08:05.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.365+0000 7f6d4fe00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d581037a0 0x7f6d58113f80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6d40002a50 tx=0x7f6d400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:05.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.365+0000 7f6d4e000700 1 -- 172.21.15.63:0/203541045 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d40007cc0 con 0x7f6d581037a0 2024-03-23T12:08:05.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.365+0000 7f6d4e000700 1 -- 172.21.15.63:0/203541045 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d4000f040 con 0x7f6d581037a0 2024-03-23T12:08:05.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.365+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5810dca0 con 0x7f6d581037a0 2024-03-23T12:08:05.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.365+0000 7f6d4e000700 1 -- 172.21.15.63:0/203541045 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d4000da30 con 0x7f6d581037a0 2024-03-23T12:08:05.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.365+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5810e260 con 0x7f6d581037a0 2024-03-23T12:08:05.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.366+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d580fe550 con 0x7f6d581037a0 2024-03-23T12:08:05.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.367+0000 7f6d4e000700 1 -- 172.21.15.63:0/203541045 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6d4000db90 con 0x7f6d581037a0 2024-03-23T12:08:05.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.368+0000 7f6d4e000700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d3c06c7d0 0x7f6d3c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:05.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.368+0000 7f6d4e000700 1 -- 172.21.15.63:0/203541045 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d4008b6e0 con 0x7f6d581037a0 2024-03-23T12:08:05.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.368+0000 7f6d55e00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d3c06c7d0 0x7f6d3c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:05.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.369+0000 7f6d55e00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d3c06c7d0 0x7f6d3c06ec80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009fd0 tx=0x7f6d48009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:05.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.373+0000 7f6d4e000700 1 -- 172.21.15.63:0/203541045 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d40059370 con 0x7f6d581037a0 2024-03-23T12:08:05.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:04 smithi063 ceph-mon[141124]: pgmap v508: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:05.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:04 smithi111 ceph-mon[106899]: pgmap v508: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:05.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.598+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d58109e60 con 0x7f6d3c06c7d0 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.608+0000 7f6d4e000700 1 -- 172.21.15.63:0/203541045 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f6d48008ee0 con 0x7f6d3c06c7d0 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (17m) 10m ago 30m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (14m) 10m ago 30m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (14m) 9m ago 29m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (17m) 10m ago 30m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (10m) 10m ago 24m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (10m) 10m ago 24m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:08:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (24m) 9m ago 24m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (9m) 9m ago 24m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (19m) 10m ago 32m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (18m) 9m ago 29m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (15m) 10m ago 32m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (15m) 9m ago 29m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (18m) 10m ago 30m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (18m) 9m ago 29m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (14m) 10m ago 28m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (13m) 10m ago 27m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (12m) 10m ago 27m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (12m) 9m ago 26m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (11m) 9m ago 26m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:08:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (10m) 9m ago 25m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:08:05.591 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (17m) 10m ago 30m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:08:05.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d3c06c7d0 msgr2=0x7f6d3c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d3c06c7d0 0x7f6d3c06ec80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009fd0 tx=0x7f6d48009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:05.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d581037a0 msgr2=0x7f6d58113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d581037a0 0x7f6d58113f80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6d40002a50 tx=0x7f6d400046b0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 shutdown_connections 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d3c06c7d0 0x7f6d3c06ec80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d58102e00 0x7f6d58113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 --2- 172.21.15.63:0/203541045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d581037a0 0x7f6d58113f80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 >> 172.21.15.63:0/203541045 conn(0x7f6d580f89b0 msgr2=0x7f6d580f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 shutdown_connections 2024-03-23T12:08:05.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.610+0000 7f6d5ca00700 1 -- 172.21.15.63:0/203541045 wait complete. 2024-03-23T12:08:05.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 -- 172.21.15.63:0/4014385127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 msgr2=0x7f4a400fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 --2- 172.21.15.63:0/4014385127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a400fe260 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4a30004940 tx=0x7f4a30009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:05.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 -- 172.21.15.63:0/4014385127 shutdown_connections 2024-03-23T12:08:05.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 --2- 172.21.15.63:0/4014385127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a400fe830 0x7f4a40106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 --2- 172.21.15.63:0/4014385127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a400fe260 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 -- 172.21.15.63:0/4014385127 >> 172.21.15.63:0/4014385127 conn(0x7f4a40072bc0 msgr2=0x7f4a40072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:05.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 -- 172.21.15.63:0/4014385127 shutdown_connections 2024-03-23T12:08:05.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.746+0000 7f4a45e00700 1 -- 172.21.15.63:0/4014385127 wait complete. 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.747+0000 7f4a45e00700 1 Processor -- start 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.747+0000 7f4a45e00700 1 -- start start 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.747+0000 7f4a45e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a40115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.747+0000 7f4a45e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a400fe830 0x7f4a40116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.747+0000 7f4a45e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a401168b0 con 0x7f4a400fde90 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.747+0000 7f4a45e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4010fc80 con 0x7f4a400fe830 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a40115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a40115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48322/0 (socket says 172.21.15.63:48322) 2024-03-23T12:08:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3ea00700 1 -- 172.21.15.63:0/2602963272 learned_addr learned my addr 172.21.15.63:0/2602963272 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3e000700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a400fe830 0x7f4a40116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3ea00700 1 -- 172.21.15.63:0/2602963272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a400fe830 msgr2=0x7f4a40116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3ea00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a400fe830 0x7f4a40116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3ea00700 1 -- 172.21.15.63:0/2602963272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30005040 con 0x7f4a400fde90 2024-03-23T12:08:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.748+0000 7f4a3ea00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a40115d20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4a300093b0 tx=0x7f4a30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:05.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.749+0000 7f4a36a00700 1 -- 172.21.15.63:0/2602963272 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a3000e040 con 0x7f4a400fde90 2024-03-23T12:08:05.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.749+0000 7f4a36a00700 1 -- 172.21.15.63:0/2602963272 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a3000bed0 con 0x7f4a400fde90 2024-03-23T12:08:05.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.749+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4010ff00 con 0x7f4a400fde90 2024-03-23T12:08:05.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.749+0000 7f4a36a00700 1 -- 172.21.15.63:0/2602963272 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a3001c3f0 con 0x7f4a400fde90 2024-03-23T12:08:05.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.749+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a40110440 con 0x7f4a400fde90 2024-03-23T12:08:05.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.750+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a40106a20 con 0x7f4a400fde90 2024-03-23T12:08:05.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.751+0000 7f4a36a00700 1 -- 172.21.15.63:0/2602963272 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4a30013790 con 0x7f4a400fde90 2024-03-23T12:08:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.752+0000 7f4a36a00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a2406c7d0 0x7f4a2406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.752+0000 7f4a36a00700 1 -- 172.21.15.63:0/2602963272 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4a3008ce10 con 0x7f4a400fde90 2024-03-23T12:08:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.752+0000 7f4a3e000700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a2406c7d0 0x7f4a2406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:05.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.753+0000 7f4a3e000700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a2406c7d0 0x7f4a2406ec80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4a401115c0 tx=0x7f4a2c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:05.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:05.757+0000 7f4a36a00700 1 -- 172.21.15.63:0/2602963272 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a3005aaa0 con 0x7f4a400fde90 2024-03-23T12:08:06.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.063+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4a4004bed0 con 0x7f4a400fde90 2024-03-23T12:08:06.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.063+0000 7f4a36a00700 1 -- 172.21.15.63:0/2602963272 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f4a30013420 con 0x7f4a400fde90 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:08:06.045 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:08:06.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:06.046 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:08:06.046 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:08:06.046 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:08:06.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:08:06.046 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.065+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a2406c7d0 msgr2=0x7f4a2406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.065+0000 7f4a45e00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a2406c7d0 0x7f4a2406ec80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4a401115c0 tx=0x7f4a2c00b040 comp rx=0 tx=0).stop 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.065+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 msgr2=0x7f4a40115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.065+0000 7f4a45e00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a40115d20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4a300093b0 tx=0x7f4a30009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.066+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 shutdown_connections 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.066+0000 7f4a45e00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a2406c7d0 0x7f4a2406ec80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.066+0000 7f4a45e00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a400fde90 0x7f4a40115d20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.066+0000 7f4a45e00700 1 --2- 172.21.15.63:0/2602963272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a400fe830 0x7f4a40116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.066+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 >> 172.21.15.63:0/2602963272 conn(0x7f4a40072bc0 msgr2=0x7f4a400fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.066+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 shutdown_connections 2024-03-23T12:08:06.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.066+0000 7f4a45e00700 1 -- 172.21.15.63:0/2602963272 wait complete. 2024-03-23T12:08:06.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 -- 172.21.15.63:0/3991701257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 msgr2=0x7fbfc80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 --2- 172.21.15.63:0/3991701257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc80fce10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4004940 tx=0x7fbfb4009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:06.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 -- 172.21.15.63:0/3991701257 shutdown_connections 2024-03-23T12:08:06.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 --2- 172.21.15.63:0/3991701257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc80fce10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 --2- 172.21.15.63:0/3991701257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc8102e00 0x7fbfc81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 -- 172.21.15.63:0/3991701257 >> 172.21.15.63:0/3991701257 conn(0x7fbfc80f89b0 msgr2=0x7fbfc80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:06.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 -- 172.21.15.63:0/3991701257 shutdown_connections 2024-03-23T12:08:06.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.203+0000 7fbfccc00700 1 -- 172.21.15.63:0/3991701257 wait complete. 2024-03-23T12:08:06.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.204+0000 7fbfccc00700 1 Processor -- start 2024-03-23T12:08:06.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.204+0000 7fbfccc00700 1 -- start start 2024-03-23T12:08:06.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfccc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc8102e00 0x7fbfc8113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfccc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfccc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc8114600 con 0x7fbfc81037a0 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfccc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc810d9f0 con 0x7fbfc8102e00 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfc5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfc5e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc8102e00 0x7fbfc8113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfc5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc8113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48336/0 (socket says 172.21.15.63:48336) 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfc5400700 1 -- 172.21.15.63:0/1052843343 learned_addr learned my addr 172.21.15.63:0/1052843343 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfc5400700 1 -- 172.21.15.63:0/1052843343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc8102e00 msgr2=0x7fbfc8113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfc5400700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc8102e00 0x7fbfc8113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.205+0000 7fbfc5400700 1 -- 172.21.15.63:0/1052843343 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb4005040 con 0x7fbfc81037a0 2024-03-23T12:08:06.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.206+0000 7fbfc5400700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc8113fb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4009fd0 tx=0x7fbfb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:06.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.206+0000 7fbfbe000700 1 -- 172.21.15.63:0/1052843343 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfb400e030 con 0x7fbfc81037a0 2024-03-23T12:08:06.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.206+0000 7fbfbe000700 1 -- 172.21.15.63:0/1052843343 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfb40093b0 con 0x7fbfc81037a0 2024-03-23T12:08:06.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.206+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc810dc70 con 0x7fbfc81037a0 2024-03-23T12:08:06.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.206+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc810e230 con 0x7fbfc81037a0 2024-03-23T12:08:06.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.206+0000 7fbfbe000700 1 -- 172.21.15.63:0/1052843343 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfb401cc70 con 0x7fbfc81037a0 2024-03-23T12:08:06.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.207+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc80fe550 con 0x7fbfc81037a0 2024-03-23T12:08:06.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.208+0000 7fbfbe000700 1 -- 172.21.15.63:0/1052843343 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbfb401c400 con 0x7fbfc81037a0 2024-03-23T12:08:06.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.210+0000 7fbfbe000700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbfac06c7d0 0x7fbfac06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.210+0000 7fbfbe000700 1 -- 172.21.15.63:0/1052843343 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbfb408d230 con 0x7fbfc81037a0 2024-03-23T12:08:06.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.210+0000 7fbfc5e00700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbfac06c7d0 0x7fbfac06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.210+0000 7fbfc5e00700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbfac06c7d0 0x7fbfac06ec80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009b80 tx=0x7fbfb800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:06.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.214+0000 7fbfbe000700 1 -- 172.21.15.63:0/1052843343 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbfb405af40 con 0x7fbfc81037a0 2024-03-23T12:08:06.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:05 smithi063 ceph-mon[141124]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:06.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:05 smithi111 ceph-mon[106899]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:06.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.473+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbfc8109da0 con 0x7fbfc81037a0 2024-03-23T12:08:06.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.473+0000 7fbfbe000700 1 -- 172.21.15.63:0/1052843343 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fbfb405aad0 con 0x7fbfc81037a0 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:08:06.455 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:08:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:06.457 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:08:06.458 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:06.458 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:06.458 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:06.458 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:06.458 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:06.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbfac06c7d0 msgr2=0x7fbfac06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbfac06c7d0 0x7fbfac06ec80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009b80 tx=0x7fbfb800b040 comp rx=0 tx=0).stop 2024-03-23T12:08:06.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 msgr2=0x7fbfc8113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc8113fb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4009fd0 tx=0x7fbfb4009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 shutdown_connections 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbfac06c7d0 0x7fbfac06ec80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc8102e00 0x7fbfc8113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 --2- 172.21.15.63:0/1052843343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfc81037a0 0x7fbfc8113fb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 >> 172.21.15.63:0/1052843343 conn(0x7fbfc80f89b0 msgr2=0x7fbfc80f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 shutdown_connections 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.476+0000 7fbfccc00700 1 -- 172.21.15.63:0/1052843343 wait complete. 2024-03-23T12:08:06.460 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:08:06.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 -- 172.21.15.63:0/1372950103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 msgr2=0x7fe1fc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 --2- 172.21.15.63:0/1372950103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc1031d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004940 tx=0x7fe1ec009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:06.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 -- 172.21.15.63:0/1372950103 shutdown_connections 2024-03-23T12:08:06.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 --2- 172.21.15.63:0/1372950103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1fc1037a0 0x7fe1fc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 --2- 172.21.15.63:0/1372950103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc1031d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 -- 172.21.15.63:0/1372950103 >> 172.21.15.63:0/1372950103 conn(0x7fe1fc0f89b0 msgr2=0x7fe1fc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:06.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 -- 172.21.15.63:0/1372950103 shutdown_connections 2024-03-23T12:08:06.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.615+0000 7fe201000700 1 -- 172.21.15.63:0/1372950103 wait complete. 2024-03-23T12:08:06.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.616+0000 7fe201000700 1 Processor -- start 2024-03-23T12:08:06.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.616+0000 7fe201000700 1 -- start start 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe201000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe201000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1fc1037a0 0x7fe1fc113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe201000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc1145d0 con 0x7fe1fc102e00 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe201000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc10d9c0 con 0x7fe1fc1037a0 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe1f9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe1f9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48356/0 (socket says 172.21.15.63:48356) 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe1f9e00700 1 -- 172.21.15.63:0/608725214 learned_addr learned my addr 172.21.15.63:0/608725214 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe1f9400700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1fc1037a0 0x7fe1fc113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe1f9e00700 1 -- 172.21.15.63:0/608725214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1fc1037a0 msgr2=0x7fe1fc113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.617+0000 7fe1f9e00700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1fc1037a0 0x7fe1fc113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe1f9e00700 1 -- 172.21.15.63:0/608725214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec005040 con 0x7fe1fc102e00 2024-03-23T12:08:06.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe1f9400700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1fc1037a0 0x7fe1fc113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:08:06.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe1f9e00700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc113a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009390 tx=0x7fe1ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:06.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe1f2000700 1 -- 172.21.15.63:0/608725214 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ec00e040 con 0x7fe1fc102e00 2024-03-23T12:08:06.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe1f2000700 1 -- 172.21.15.63:0/608725214 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1ec013420 con 0x7fe1fc102e00 2024-03-23T12:08:06.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe1f2000700 1 -- 172.21.15.63:0/608725214 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ec01c630 con 0x7fe1fc102e00 2024-03-23T12:08:06.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc10dc40 con 0x7fe1fc102e00 2024-03-23T12:08:06.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.618+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc10e200 con 0x7fe1fc102e00 2024-03-23T12:08:06.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.619+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc0fe550 con 0x7fe1fc102e00 2024-03-23T12:08:06.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.620+0000 7fe1f2000700 1 -- 172.21.15.63:0/608725214 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe1ec0139e0 con 0x7fe1fc102e00 2024-03-23T12:08:06.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.621+0000 7fe1f2000700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1e006c7d0 0x7fe1e006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.621+0000 7fe1f2000700 1 -- 172.21.15.63:0/608725214 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe1ec08cd60 con 0x7fe1fc102e00 2024-03-23T12:08:06.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.621+0000 7fe1f9400700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1e006c7d0 0x7fe1e006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.622+0000 7fe1f9400700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1e006c7d0 0x7fe1e006ec80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009b80 tx=0x7fe1e800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:06.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.626+0000 7fe1f2000700 1 -- 172.21.15.63:0/608725214 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe1ec091070 con 0x7fe1fc102e00 2024-03-23T12:08:06.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.858+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1fc109e30 con 0x7fe1e006c7d0 2024-03-23T12:08:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.859+0000 7fe1f2000700 1 -- 172.21.15.63:0/608725214 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe1fc109e30 con 0x7fe1e006c7d0 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:08:06.841 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:08:06.842 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:08:06.842 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:08:06.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1e006c7d0 msgr2=0x7fe1e006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1e006c7d0 0x7fe1e006ec80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009b80 tx=0x7fe1e800b040 comp rx=0 tx=0).stop 2024-03-23T12:08:06.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 msgr2=0x7fe1fc113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc113a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009390 tx=0x7fe1ec009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:06.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 shutdown_connections 2024-03-23T12:08:06.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1e006c7d0 0x7fe1e006ec80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1fc102e00 0x7fe1fc113a40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 --2- 172.21.15.63:0/608725214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1fc1037a0 0x7fe1fc113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 >> 172.21.15.63:0/608725214 conn(0x7fe1fc0f89b0 msgr2=0x7fe1fc0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:06.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 shutdown_connections 2024-03-23T12:08:06.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.862+0000 7fe201000700 1 -- 172.21.15.63:0/608725214 wait complete. 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.999+0000 7f15bf800700 1 -- 172.21.15.63:0/3955268395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 msgr2=0x7f15b80706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.999+0000 7f15bf800700 1 --2- 172.21.15.63:0/3955268395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b80706d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f15ac004970 tx=0x7f15ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.999+0000 7f15bf800700 1 -- 172.21.15.63:0/3955268395 shutdown_connections 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.999+0000 7f15bf800700 1 --2- 172.21.15.63:0/3955268395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b8070c10 0x7f15b8109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.999+0000 7f15bf800700 1 --2- 172.21.15.63:0/3955268395 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b80706d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.999+0000 7f15bf800700 1 -- 172.21.15.63:0/3955268395 >> 172.21.15.63:0/3955268395 conn(0x7f15b80f8970 msgr2=0x7f15b80fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:06.999+0000 7f15bf800700 1 -- 172.21.15.63:0/3955268395 shutdown_connections 2024-03-23T12:08:06.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.000+0000 7f15bf800700 1 -- 172.21.15.63:0/3955268395 wait complete. 2024-03-23T12:08:06.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.000+0000 7f15bf800700 1 Processor -- start 2024-03-23T12:08:06.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.000+0000 7f15bf800700 1 -- start start 2024-03-23T12:08:06.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15bf800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b8115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15bf800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b8070c10 0x7f15b81161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15bf800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8116820 con 0x7f15b8070300 2024-03-23T12:08:06.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15bf800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b810fc30 con 0x7f15b8070c10 2024-03-23T12:08:06.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15bce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b8115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15bce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b8115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48360/0 (socket says 172.21.15.63:48360) 2024-03-23T12:08:06.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15bce00700 1 -- 172.21.15.63:0/4186273832 learned_addr learned my addr 172.21.15.63:0/4186273832 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:06.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.001+0000 7f15b7e00700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b8070c10 0x7f15b81161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15bce00700 1 -- 172.21.15.63:0/4186273832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b8070c10 msgr2=0x7f15b81161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15bce00700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b8070c10 0x7f15b81161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15bce00700 1 -- 172.21.15.63:0/4186273832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac005040 con 0x7f15b8070300 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15bce00700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b8115c90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f15ac005010 tx=0x7f15ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15b4c00700 1 -- 172.21.15.63:0/4186273832 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15ac00e040 con 0x7f15b8070300 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15b4c00700 1 -- 172.21.15.63:0/4186273832 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15ac0098b0 con 0x7f15b8070300 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b810feb0 con 0x7f15b8070300 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15b4c00700 1 -- 172.21.15.63:0/4186273832 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15ac013e80 con 0x7f15b8070300 2024-03-23T12:08:06.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.002+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b81103f0 con 0x7f15b8070300 2024-03-23T12:08:06.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.003+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b8071a40 con 0x7f15b8070300 2024-03-23T12:08:06.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.004+0000 7f15b4c00700 1 -- 172.21.15.63:0/4186273832 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f15ac0136b0 con 0x7f15b8070300 2024-03-23T12:08:06.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.005+0000 7f15b4c00700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f15a006c7d0 0x7f15a006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:06.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.005+0000 7f15b4c00700 1 -- 172.21.15.63:0/4186273832 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f15ac08cbc0 con 0x7f15b8070300 2024-03-23T12:08:06.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.006+0000 7f15b7e00700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f15a006c7d0 0x7f15a006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:06.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.006+0000 7f15b7e00700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f15a006c7d0 0x7f15a006ec80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f15b8111570 tx=0x7f15a400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:06.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.010+0000 7f15b4c00700 1 -- 172.21.15.63:0/4186273832 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f15ac056ef0 con 0x7f15b8070300 2024-03-23T12:08:07.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.308+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f15b804bed0 con 0x7f15b8070300 2024-03-23T12:08:07.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.309+0000 7f15b4c00700 1 -- 172.21.15.63:0/4186273832 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f15ac05a510 con 0x7f15b8070300 2024-03-23T12:08:07.290 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:08:07.290 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:08:07.290 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:08:07.290 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:08:07.290 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:08:07.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f15a006c7d0 msgr2=0x7f15a006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:07.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f15a006c7d0 0x7f15a006ec80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f15b8111570 tx=0x7f15a400b040 comp rx=0 tx=0).stop 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 msgr2=0x7f15b8115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b8115c90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f15ac005010 tx=0x7f15ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 shutdown_connections 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f15a006c7d0 0x7f15a006ec80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15b8070300 0x7f15b8115c90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 --2- 172.21.15.63:0/4186273832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15b8070c10 0x7f15b81161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 >> 172.21.15.63:0/4186273832 conn(0x7f15b80f8970 msgr2=0x7f15b80f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 shutdown_connections 2024-03-23T12:08:07.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:07.311+0000 7f15bf800700 1 -- 172.21.15.63:0/4186273832 wait complete. 2024-03-23T12:08:07.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:06 smithi063 ceph-mon[141124]: from='client.34970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:07.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:06 smithi063 ceph-mon[141124]: from='client.34974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:07.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:06 smithi063 ceph-mon[141124]: pgmap v509: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:07.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:06 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2602963272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:08:07.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:06 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1052843343' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:08:07.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:06 smithi111 ceph-mon[106899]: from='client.34970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:07.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:06 smithi111 ceph-mon[106899]: from='client.34974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:07.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:06 smithi111 ceph-mon[106899]: pgmap v509: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:07.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:06 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2602963272' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:08:07.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:06 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1052843343' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:08:08.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:08 smithi063 ceph-mon[141124]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:08.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:08 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4186273832' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:08:08.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:08 smithi111 ceph-mon[106899]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:08.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:08 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4186273832' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:08:09.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:09 smithi063 ceph-mon[141124]: pgmap v510: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:09.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:09 smithi111 ceph-mon[106899]: pgmap v510: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:10.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:10 smithi063 ceph-mon[141124]: pgmap v511: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:10.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:10 smithi111 ceph-mon[106899]: pgmap v511: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:11.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:11.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:12.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:12 smithi063 ceph-mon[141124]: pgmap v512: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:12.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:12 smithi111 ceph-mon[106899]: pgmap v512: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:15.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:14 smithi063 ceph-mon[141124]: pgmap v513: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:15.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:14 smithi111 ceph-mon[106899]: pgmap v513: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:16.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:16 smithi063 ceph-mon[141124]: pgmap v514: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:16.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:16 smithi111 ceph-mon[106899]: pgmap v514: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:19.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:18 smithi063 ceph-mon[141124]: pgmap v515: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:19.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:18 smithi111 ceph-mon[106899]: pgmap v515: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:21.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:20 smithi063 ceph-mon[141124]: pgmap v516: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:21.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:20 smithi111 ceph-mon[106899]: pgmap v516: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:23.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:22 smithi063 ceph-mon[141124]: pgmap v517: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:22 smithi111 ceph-mon[106899]: pgmap v517: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:25.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:24 smithi063 ceph-mon[141124]: pgmap v518: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:25.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:24 smithi111 ceph-mon[106899]: pgmap v518: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:26 smithi063 ceph-mon[141124]: pgmap v519: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:26 smithi111 ceph-mon[106899]: pgmap v519: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:29.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:28 smithi063 ceph-mon[141124]: pgmap v520: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:29.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:28 smithi111 ceph-mon[106899]: pgmap v520: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:31.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:30 smithi063 ceph-mon[141124]: pgmap v521: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:31.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:30 smithi111 ceph-mon[106899]: pgmap v521: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:32 smithi063 ceph-mon[141124]: pgmap v522: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:33.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:32 smithi111 ceph-mon[106899]: pgmap v522: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:35.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:35 smithi063 ceph-mon[141124]: pgmap v523: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:35.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:35 smithi111 ceph-mon[106899]: pgmap v523: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:36.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:36 smithi063 ceph-mon[141124]: pgmap v524: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:36.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:36 smithi111 ceph-mon[106899]: pgmap v524: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 -- 172.21.15.63:0/1647239833 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070300 msgr2=0x7fdac00706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 --2- 172.21.15.63:0/1647239833 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070300 0x7fdac00706d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdab0004970 tx=0x7fdab0009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 -- 172.21.15.63:0/1647239833 shutdown_connections 2024-03-23T12:08:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 --2- 172.21.15.63:0/1647239833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdac0070c10 0x7fdac0109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 --2- 172.21.15.63:0/1647239833 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070300 0x7fdac00706d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 -- 172.21.15.63:0/1647239833 >> 172.21.15.63:0/1647239833 conn(0x7fdac00f89a0 msgr2=0x7fdac00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 -- 172.21.15.63:0/1647239833 shutdown_connections 2024-03-23T12:08:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.466+0000 7fdac6600700 1 -- 172.21.15.63:0/1647239833 wait complete. 2024-03-23T12:08:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.467+0000 7fdac6600700 1 Processor -- start 2024-03-23T12:08:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.467+0000 7fdac6600700 1 -- start start 2024-03-23T12:08:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdac6600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdac0070300 0x7fdac0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdac6600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070c10 0x7fdac0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdac6600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0116870 con 0x7fdac0070c10 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdac6600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac010fc40 con 0x7fdac0070300 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdabea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070c10 0x7fdac0116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdabea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070c10 0x7fdac0116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40338/0 (socket says 172.21.15.63:40338) 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdabea00700 1 -- 172.21.15.63:0/251595978 learned_addr learned my addr 172.21.15.63:0/251595978 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.468+0000 7fdabf400700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdac0070300 0x7fdac0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdabea00700 1 -- 172.21.15.63:0/251595978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdac0070300 msgr2=0x7fdac0115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdabea00700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdac0070300 0x7fdac0115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdabea00700 1 -- 172.21.15.63:0/251595978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0005040 con 0x7fdac0070c10 2024-03-23T12:08:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdabea00700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070c10 0x7fdac0116220 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdaac002a50 tx=0x7fdaac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdab7400700 1 -- 172.21.15.63:0/251595978 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaac007cc0 con 0x7fdac0070c10 2024-03-23T12:08:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdab7400700 1 -- 172.21.15.63:0/251595978 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaac00f040 con 0x7fdac0070c10 2024-03-23T12:08:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac010ff20 con 0x7fdac0070c10 2024-03-23T12:08:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdab7400700 1 -- 172.21.15.63:0/251595978 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaac00d9c0 con 0x7fdac0070c10 2024-03-23T12:08:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.469+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac0110460 con 0x7fdac0070c10 2024-03-23T12:08:37.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.470+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac0071a40 con 0x7fdac0070c10 2024-03-23T12:08:37.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.472+0000 7fdab7400700 1 -- 172.21.15.63:0/251595978 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdaac00db60 con 0x7fdac0070c10 2024-03-23T12:08:37.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.472+0000 7fdab7400700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaa406c7c0 0x7fdaa406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:37.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.472+0000 7fdab7400700 1 -- 172.21.15.63:0/251595978 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdaac08b160 con 0x7fdac0070c10 2024-03-23T12:08:37.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.473+0000 7fdabf400700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaa406c7c0 0x7fdaa406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:37.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.473+0000 7fdabf400700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaa406c7c0 0x7fdaa406ec70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdab0004f50 tx=0x7fdab0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:37.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.477+0000 7fdab7400700 1 -- 172.21.15.63:0/251595978 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdaac058db0 con 0x7fdac0070c10 2024-03-23T12:08:37.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.709+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdac010c050 con 0x7fdaa406c7c0 2024-03-23T12:08:37.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.711+0000 7fdab7400700 1 -- 172.21.15.63:0/251595978 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdac010c050 con 0x7fdaa406c7c0 2024-03-23T12:08:37.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaa406c7c0 msgr2=0x7fdaa406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:37.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaa406c7c0 0x7fdaa406ec70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdab0004f50 tx=0x7fdab0009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:37.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070c10 msgr2=0x7fdac0116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:37.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070c10 0x7fdac0116220 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdaac002a50 tx=0x7fdaac0046b0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 shutdown_connections 2024-03-23T12:08:37.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaa406c7c0 0x7fdaa406ec70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdac0070300 0x7fdac0115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 --2- 172.21.15.63:0/251595978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdac0070c10 0x7fdac0116220 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 >> 172.21.15.63:0/251595978 conn(0x7fdac00f89a0 msgr2=0x7fdac00f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:37.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 shutdown_connections 2024-03-23T12:08:37.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.713+0000 7fdac6600700 1 -- 172.21.15.63:0/251595978 wait complete. 2024-03-23T12:08:37.708 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:08:37.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 -- 172.21.15.63:0/4104222192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070360 msgr2=0x7f492c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:37.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 --2- 172.21.15.63:0/4104222192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070360 0x7f492c070730 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f491c004970 tx=0x7f491c009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:37.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 -- 172.21.15.63:0/4104222192 shutdown_connections 2024-03-23T12:08:37.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 --2- 172.21.15.63:0/4104222192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f492c070c70 0x7f492c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 --2- 172.21.15.63:0/4104222192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070360 0x7f492c070730 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 -- 172.21.15.63:0/4104222192 >> 172.21.15.63:0/4104222192 conn(0x7f492c0f89b0 msgr2=0x7f492c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:37.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 -- 172.21.15.63:0/4104222192 shutdown_connections 2024-03-23T12:08:37.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.852+0000 7f4931e00700 1 -- 172.21.15.63:0/4104222192 wait complete. 2024-03-23T12:08:37.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.853+0000 7f4931e00700 1 Processor -- start 2024-03-23T12:08:37.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.853+0000 7f4931e00700 1 -- start start 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.853+0000 7f4931e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f492c070360 0x7f492c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.853+0000 7f4931e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070c70 0x7f492c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.853+0000 7f4931e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c116810 con 0x7f492c070c70 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.853+0000 7f4931e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c10fbe0 con 0x7f492c070360 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.853+0000 7f492a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070c70 0x7f492c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f492a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070c70 0x7f492c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40348/0 (socket says 172.21.15.63:40348) 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f492a000700 1 -- 172.21.15.63:0/3701712562 learned_addr learned my addr 172.21.15.63:0/3701712562 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f492aa00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f492c070360 0x7f492c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f492a000700 1 -- 172.21.15.63:0/3701712562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f492c070360 msgr2=0x7f492c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f492a000700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f492c070360 0x7f492c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f492a000700 1 -- 172.21.15.63:0/3701712562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c005040 con 0x7f492c070c70 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f492a000700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070c70 0x7f492c1161c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f491800c8c0 tx=0x7f49180044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f4922a00700 1 -- 172.21.15.63:0/3701712562 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f491800a980 con 0x7f492c070c70 2024-03-23T12:08:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f4922a00700 1 -- 172.21.15.63:0/3701712562 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f491800ce60 con 0x7f492c070c70 2024-03-23T12:08:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.854+0000 7f4922a00700 1 -- 172.21.15.63:0/3701712562 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49180113a0 con 0x7f492c070c70 2024-03-23T12:08:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.855+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c10fec0 con 0x7f492c070c70 2024-03-23T12:08:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.855+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c1103b0 con 0x7f492c070c70 2024-03-23T12:08:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.856+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c103b90 con 0x7f492c070c70 2024-03-23T12:08:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.857+0000 7f4922a00700 1 -- 172.21.15.63:0/3701712562 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4918012400 con 0x7f492c070c70 2024-03-23T12:08:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.857+0000 7f4922a00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f491006c7d0 0x7f491006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:37.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.858+0000 7f4922a00700 1 -- 172.21.15.63:0/3701712562 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f491808bc10 con 0x7f492c070c70 2024-03-23T12:08:37.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.858+0000 7f492aa00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f491006c7d0 0x7f491006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:37.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.858+0000 7f492aa00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f491006c7d0 0x7f491006ec80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f491c009fd0 tx=0x7f491c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:37.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:37.862+0000 7f4922a00700 1 -- 172.21.15.63:0/3701712562 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49180598a0 con 0x7f492c070c70 2024-03-23T12:08:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.094+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f492c111340 con 0x7f491006c7d0 2024-03-23T12:08:38.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.095+0000 7f4922a00700 1 -- 172.21.15.63:0/3701712562 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f492c111340 con 0x7f491006c7d0 2024-03-23T12:08:38.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f491006c7d0 msgr2=0x7f491006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f491006c7d0 0x7f491006ec80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f491c009fd0 tx=0x7f491c009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070c70 msgr2=0x7f492c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070c70 0x7f492c1161c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f491800c8c0 tx=0x7f49180044c0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 shutdown_connections 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f491006c7d0 0x7f491006ec80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f492c070360 0x7f492c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 --2- 172.21.15.63:0/3701712562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f492c070c70 0x7f492c1161c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 >> 172.21.15.63:0/3701712562 conn(0x7f492c0f89b0 msgr2=0x7f492c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 shutdown_connections 2024-03-23T12:08:38.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.097+0000 7f4931e00700 1 -- 172.21.15.63:0/3701712562 wait complete. 2024-03-23T12:08:38.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.234+0000 7f7b72a00700 1 -- 172.21.15.63:0/1502254258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 msgr2=0x7f7b6c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.234+0000 7f7b72a00700 1 --2- 172.21.15.63:0/1502254258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c106f20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7b58004970 tx=0x7f7b58009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:38.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.235+0000 7f7b72a00700 1 -- 172.21.15.63:0/1502254258 shutdown_connections 2024-03-23T12:08:38.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.235+0000 7f7b72a00700 1 --2- 172.21.15.63:0/1502254258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c106f20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.235+0000 7f7b72a00700 1 --2- 172.21.15.63:0/1502254258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b6c0fdeb0 0x7f7b6c0fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.235+0000 7f7b72a00700 1 -- 172.21.15.63:0/1502254258 >> 172.21.15.63:0/1502254258 conn(0x7f7b6c072bc0 msgr2=0x7f7b6c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:38.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.235+0000 7f7b72a00700 1 -- 172.21.15.63:0/1502254258 shutdown_connections 2024-03-23T12:08:38.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.235+0000 7f7b72a00700 1 -- 172.21.15.63:0/1502254258 wait complete. 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b72a00700 1 Processor -- start 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b72a00700 1 -- start start 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b72a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b6c0fdeb0 0x7f7b6c113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b72a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b72a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c1145a0 con 0x7f7b6c0fe850 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b72a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c10d990 con 0x7f7b6c0fdeb0 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b6b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b6b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40366/0 (socket says 172.21.15.63:40366) 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.236+0000 7f7b6b400700 1 -- 172.21.15.63:0/742413696 learned_addr learned my addr 172.21.15.63:0/742413696 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b6be00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b6c0fdeb0 0x7f7b6c113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:38.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b6b400700 1 -- 172.21.15.63:0/742413696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b6c0fdeb0 msgr2=0x7f7b6c113a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b6b400700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b6c0fdeb0 0x7f7b6c113a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b6b400700 1 -- 172.21.15.63:0/742413696 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58005040 con 0x7f7b6c0fe850 2024-03-23T12:08:38.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b6b400700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c113f50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009360 tx=0x7f7b58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:38.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b63e00700 1 -- 172.21.15.63:0/742413696 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5800e040 con 0x7f7b6c0fe850 2024-03-23T12:08:38.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b63e00700 1 -- 172.21.15.63:0/742413696 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b58004290 con 0x7f7b6c0fe850 2024-03-23T12:08:38.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c10dc10 con 0x7f7b6c0fe850 2024-03-23T12:08:38.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.237+0000 7f7b63e00700 1 -- 172.21.15.63:0/742413696 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b58013d90 con 0x7f7b6c0fe850 2024-03-23T12:08:38.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.238+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c10e150 con 0x7f7b6c0fe850 2024-03-23T12:08:38.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.239+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c065e10 con 0x7f7b6c0fe850 2024-03-23T12:08:38.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.240+0000 7f7b63e00700 1 -- 172.21.15.63:0/742413696 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7b5800ba40 con 0x7f7b6c0fe850 2024-03-23T12:08:38.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.240+0000 7f7b63e00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b5006c7d0 0x7f7b5006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.241+0000 7f7b63e00700 1 -- 172.21.15.63:0/742413696 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b5808ce40 con 0x7f7b6c0fe850 2024-03-23T12:08:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.241+0000 7f7b6be00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b5006c7d0 0x7f7b5006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.241+0000 7f7b6be00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b5006c7d0 0x7f7b5006ec80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c004f40 tx=0x7f7b5c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:38.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.245+0000 7f7b63e00700 1 -- 172.21.15.63:0/742413696 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b5805aa50 con 0x7f7b6c0fe850 2024-03-23T12:08:38.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.469+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b6c109e20 con 0x7f7b5006c7d0 2024-03-23T12:08:38.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.479+0000 7f7b63e00700 1 -- 172.21.15.63:0/742413696 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f7b6c109e20 con 0x7f7b5006c7d0 2024-03-23T12:08:38.460 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:08:38.460 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (17m) 10m ago 31m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:08:38.460 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (15m) 10m ago 31m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:08:38.460 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (15m) 10m ago 30m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:08:38.460 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (17m) 10m ago 30m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:08:38.460 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (11m) 10m ago 24m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:08:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (10m) 10m ago 24m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:08:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (24m) 10m ago 24m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:08:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (10m) 10m ago 24m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:08:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (20m) 10m ago 32m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:08:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (19m) 10m ago 29m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:08:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (16m) 10m ago 32m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:08:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (15m) 10m ago 29m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (18m) 10m ago 31m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (18m) 10m ago 30m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (14m) 10m ago 29m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (13m) 10m ago 28m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (13m) 10m ago 27m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (12m) 10m ago 27m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (12m) 10m ago 26m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:08:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (11m) 10m ago 26m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:08:38.463 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (18m) 10m ago 30m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:08:38.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b5006c7d0 msgr2=0x7f7b5006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b5006c7d0 0x7f7b5006ec80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c004f40 tx=0x7f7b5c00b040 comp rx=0 tx=0).stop 2024-03-23T12:08:38.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 msgr2=0x7f7b6c113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c113f50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009360 tx=0x7f7b58009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:38.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 shutdown_connections 2024-03-23T12:08:38.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b5006c7d0 0x7f7b5006ec80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b6c0fdeb0 0x7f7b6c113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 --2- 172.21.15.63:0/742413696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b6c0fe850 0x7f7b6c113f50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.481+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 >> 172.21.15.63:0/742413696 conn(0x7f7b6c072bc0 msgr2=0x7f7b6c103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:38.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.482+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 shutdown_connections 2024-03-23T12:08:38.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.482+0000 7f7b72a00700 1 -- 172.21.15.63:0/742413696 wait complete. 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 -- 172.21.15.63:0/3419131698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 msgr2=0x7fb8c00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3419131698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c00fce10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac004940 tx=0x7fb8ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 -- 172.21.15.63:0/3419131698 shutdown_connections 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3419131698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c00fce10 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3419131698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0102e00 0x7fb8c01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 -- 172.21.15.63:0/3419131698 >> 172.21.15.63:0/3419131698 conn(0x7fb8c00f89b0 msgr2=0x7fb8c00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 -- 172.21.15.63:0/3419131698 shutdown_connections 2024-03-23T12:08:38.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.617+0000 7fb8c8200700 1 -- 172.21.15.63:0/3419131698 wait complete. 2024-03-23T12:08:38.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.618+0000 7fb8c8200700 1 Processor -- start 2024-03-23T12:08:38.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.618+0000 7fb8c8200700 1 -- start start 2024-03-23T12:08:38.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c8200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0102e00 0x7fb8c0113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:38.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c8200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c0114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:38.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c8200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c01146b0 con 0x7fb8c01037a0 2024-03-23T12:08:38.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c8200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c010daa0 con 0x7fb8c0102e00 2024-03-23T12:08:38.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c0114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c0114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40384/0 (socket says 172.21.15.63:40384) 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c4e00700 1 -- 172.21.15.63:0/3733374177 learned_addr learned my addr 172.21.15.63:0/3733374177 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c5800700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0102e00 0x7fb8c0113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c4e00700 1 -- 172.21.15.63:0/3733374177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0102e00 msgr2=0x7fb8c0113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c4e00700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0102e00 0x7fb8c0113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.619+0000 7fb8c4e00700 1 -- 172.21.15.63:0/3733374177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac005040 con 0x7fb8c01037a0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.620+0000 7fb8c4e00700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c0114060 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009390 tx=0x7fb8ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.620+0000 7fb8bd600700 1 -- 172.21.15.63:0/3733374177 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8ac00e040 con 0x7fb8c01037a0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.620+0000 7fb8bd600700 1 -- 172.21.15.63:0/3733374177 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8ac013420 con 0x7fb8c01037a0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.620+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c010dd20 con 0x7fb8c01037a0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.620+0000 7fb8bd600700 1 -- 172.21.15.63:0/3733374177 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8ac01c630 con 0x7fb8c01037a0 2024-03-23T12:08:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.620+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c010e2e0 con 0x7fb8c01037a0 2024-03-23T12:08:38.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.621+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c00fe550 con 0x7fb8c01037a0 2024-03-23T12:08:38.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.622+0000 7fb8bd600700 1 -- 172.21.15.63:0/3733374177 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb8ac013a00 con 0x7fb8c01037a0 2024-03-23T12:08:38.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.624+0000 7fb8bd600700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8a806c7d0 0x7fb8a806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:38.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.624+0000 7fb8bd600700 1 -- 172.21.15.63:0/3733374177 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb8ac08cc20 con 0x7fb8c01037a0 2024-03-23T12:08:38.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.624+0000 7fb8c5800700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8a806c7d0 0x7fb8a806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:38.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.625+0000 7fb8c5800700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8a806c7d0 0x7fb8a806ec80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4004f60 tx=0x7fb8b400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:38.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.628+0000 7fb8bd600700 1 -- 172.21.15.63:0/3733374177 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb8ac05a8b0 con 0x7fb8c01037a0 2024-03-23T12:08:38.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.934+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb8c0109dc0 con 0x7fb8c01037a0 2024-03-23T12:08:38.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.935+0000 7fb8bd600700 1 -- 172.21.15.63:0/3733374177 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fb8ac05a440 con 0x7fb8c01037a0 2024-03-23T12:08:38.915 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:08:38.915 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:08:38.915 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:08:38.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:38.915 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:08:38.916 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8a806c7d0 msgr2=0x7fb8a806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8a806c7d0 0x7fb8a806ec80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4004f60 tx=0x7fb8b400b040 comp rx=0 tx=0).stop 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 msgr2=0x7fb8c0114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c0114060 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009390 tx=0x7fb8ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 shutdown_connections 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8a806c7d0 0x7fb8a806ec80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0102e00 0x7fb8c0113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 --2- 172.21.15.63:0/3733374177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8c01037a0 0x7fb8c0114060 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 >> 172.21.15.63:0/3733374177 conn(0x7fb8c00f89b0 msgr2=0x7fb8c00f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 shutdown_connections 2024-03-23T12:08:38.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:38.938+0000 7fb8c8200700 1 -- 172.21.15.63:0/3733374177 wait complete. 2024-03-23T12:08:39.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 -- 172.21.15.63:0/1466922719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b4071f90 msgr2=0x7f32b4070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1466922719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b4071f90 0x7f32b4070360 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f32a4004970 tx=0x7f32a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:39.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 -- 172.21.15.63:0/1466922719 shutdown_connections 2024-03-23T12:08:39.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1466922719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b40708a0 0x7f32b4070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1466922719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b4071f90 0x7f32b4070360 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 -- 172.21.15.63:0/1466922719 >> 172.21.15.63:0/1466922719 conn(0x7f32b40f89b0 msgr2=0x7f32b40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:39.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 -- 172.21.15.63:0/1466922719 shutdown_connections 2024-03-23T12:08:39.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.074+0000 7f32b8a00700 1 -- 172.21.15.63:0/1466922719 wait complete. 2024-03-23T12:08:39.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.075+0000 7f32b8a00700 1 Processor -- start 2024-03-23T12:08:39.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.075+0000 7f32b8a00700 1 -- start start 2024-03-23T12:08:39.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b40708a0 0x7f32b411a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b8a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b4071f90 0x7f32b411a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b8a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b411ac50 con 0x7f32b40708a0 2024-03-23T12:08:39.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b8a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b4114020 con 0x7f32b4071f90 2024-03-23T12:08:39.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b40708a0 0x7f32b411a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b40708a0 0x7f32b411a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40400/0 (socket says 172.21.15.63:40400) 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b1e00700 1 -- 172.21.15.63:0/1937760406 learned_addr learned my addr 172.21.15.63:0/1937760406 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b1400700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b4071f90 0x7f32b411a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b1e00700 1 -- 172.21.15.63:0/1937760406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b4071f90 msgr2=0x7f32b411a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b1e00700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b4071f90 0x7f32b411a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.076+0000 7f32b1e00700 1 -- 172.21.15.63:0/1937760406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a4005040 con 0x7f32b40708a0 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.077+0000 7f32b1e00700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b40708a0 0x7f32b411a0c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f32a4005010 tx=0x7f32a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.077+0000 7f32aa000700 1 -- 172.21.15.63:0/1937760406 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a400e040 con 0x7f32b40708a0 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.077+0000 7f32aa000700 1 -- 172.21.15.63:0/1937760406 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32a40098b0 con 0x7f32b40708a0 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.077+0000 7f32aa000700 1 -- 172.21.15.63:0/1937760406 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a4013e80 con 0x7f32b40708a0 2024-03-23T12:08:39.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.077+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b41142a0 con 0x7f32b40708a0 2024-03-23T12:08:39.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.077+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b41147e0 con 0x7f32b40708a0 2024-03-23T12:08:39.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.078+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b4071aa0 con 0x7f32b40708a0 2024-03-23T12:08:39.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.079+0000 7f32aa000700 1 -- 172.21.15.63:0/1937760406 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f32a40136b0 con 0x7f32b40708a0 2024-03-23T12:08:39.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.080+0000 7f32aa000700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f329806c7d0 0x7f329806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.080+0000 7f32aa000700 1 -- 172.21.15.63:0/1937760406 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f32a408cc20 con 0x7f32b40708a0 2024-03-23T12:08:39.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.080+0000 7f32b1400700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f329806c7d0 0x7f329806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.081+0000 7f32b1400700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f329806c7d0 0x7f329806ec80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32b4115960 tx=0x7f32a000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:39.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.084+0000 7f32aa000700 1 -- 172.21.15.63:0/1937760406 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f32a405a9f0 con 0x7f32b40708a0 2024-03-23T12:08:39.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.344+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f32b404bed0 con 0x7f32b40708a0 2024-03-23T12:08:39.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.344+0000 7f32aa000700 1 -- 172.21.15.63:0/1937760406 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f32a405a580 con 0x7f32b40708a0 2024-03-23T12:08:39.324 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:08:39.324 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:08:39.324 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:08:39.325 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:08:39.326 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:39.327 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:08:39.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.346+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f329806c7d0 msgr2=0x7f329806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.346+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f329806c7d0 0x7f329806ec80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32b4115960 tx=0x7f32a000b040 comp rx=0 tx=0).stop 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.346+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b40708a0 msgr2=0x7f32b411a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.346+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b40708a0 0x7f32b411a0c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f32a4005010 tx=0x7f32a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.347+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 shutdown_connections 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.347+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f329806c7d0 0x7f329806ec80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.347+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b40708a0 0x7f32b411a0c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.347+0000 7f32b8a00700 1 --2- 172.21.15.63:0/1937760406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b4071f90 0x7f32b411a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.347+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 >> 172.21.15.63:0/1937760406 conn(0x7f32b40f89b0 msgr2=0x7f32b40f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.347+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 shutdown_connections 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.347+0000 7f32b8a00700 1 -- 172.21.15.63:0/1937760406 wait complete. 2024-03-23T12:08:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:08:39.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:38 smithi063 ceph-mon[141124]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:39.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:38 smithi063 ceph-mon[141124]: pgmap v525: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:39.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:38 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3733374177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:08:39.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:38 smithi111 ceph-mon[106899]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:39.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:38 smithi111 ceph-mon[106899]: pgmap v525: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:39.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:38 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3733374177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:08:39.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.484+0000 7fea54000700 1 -- 172.21.15.63:0/2540594300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c1037a0 msgr2=0x7fea4c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.484+0000 7fea54000700 1 --2- 172.21.15.63:0/2540594300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c1037a0 0x7fea4c0fce10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fea38004940 tx=0x7fea38009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:39.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.485+0000 7fea54000700 1 -- 172.21.15.63:0/2540594300 shutdown_connections 2024-03-23T12:08:39.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.485+0000 7fea54000700 1 --2- 172.21.15.63:0/2540594300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c1037a0 0x7fea4c0fce10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.485+0000 7fea54000700 1 --2- 172.21.15.63:0/2540594300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea4c102e00 0x7fea4c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.485+0000 7fea54000700 1 -- 172.21.15.63:0/2540594300 >> 172.21.15.63:0/2540594300 conn(0x7fea4c0f89b0 msgr2=0x7fea4c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:39.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.485+0000 7fea54000700 1 -- 172.21.15.63:0/2540594300 shutdown_connections 2024-03-23T12:08:39.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.485+0000 7fea54000700 1 -- 172.21.15.63:0/2540594300 wait complete. 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea54000700 1 Processor -- start 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea54000700 1 -- start start 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea54000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c102e00 0x7fea4c115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea54000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea4c1037a0 0x7fea4c1162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea54000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c116930 con 0x7fea4c102e00 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea54000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c10fd00 con 0x7fea4c1037a0 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea51600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c102e00 0x7fea4c115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea51600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c102e00 0x7fea4c115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40416/0 (socket says 172.21.15.63:40416) 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.486+0000 7fea51600700 1 -- 172.21.15.63:0/3445842199 learned_addr learned my addr 172.21.15.63:0/3445842199 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea50c00700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea4c1037a0 0x7fea4c1162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea51600700 1 -- 172.21.15.63:0/3445842199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea4c1037a0 msgr2=0x7fea4c1162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea51600700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea4c1037a0 0x7fea4c1162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea51600700 1 -- 172.21.15.63:0/3445842199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38005040 con 0x7fea4c102e00 2024-03-23T12:08:39.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea51600700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c102e00 0x7fea4c115da0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fea40002a50 tx=0x7fea400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:39.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea49600700 1 -- 172.21.15.63:0/3445842199 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea40007cc0 con 0x7fea4c102e00 2024-03-23T12:08:39.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea49600700 1 -- 172.21.15.63:0/3445842199 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea4000f040 con 0x7fea4c102e00 2024-03-23T12:08:39.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c10ffe0 con 0x7fea4c102e00 2024-03-23T12:08:39.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.487+0000 7fea49600700 1 -- 172.21.15.63:0/3445842199 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea4000d9c0 con 0x7fea4c102e00 2024-03-23T12:08:39.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.488+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c110520 con 0x7fea4c102e00 2024-03-23T12:08:39.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.488+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c0fe550 con 0x7fea4c102e00 2024-03-23T12:08:39.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.489+0000 7fea49600700 1 -- 172.21.15.63:0/3445842199 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fea4000db20 con 0x7fea4c102e00 2024-03-23T12:08:39.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.490+0000 7fea49600700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea3406c7d0 0x7fea3406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.490+0000 7fea49600700 1 -- 172.21.15.63:0/3445842199 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fea4008b760 con 0x7fea4c102e00 2024-03-23T12:08:39.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.491+0000 7fea50c00700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea3406c7d0 0x7fea3406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.491+0000 7fea50c00700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea3406c7d0 0x7fea3406ec80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fea4c111640 tx=0x7fea38009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:39.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.495+0000 7fea49600700 1 -- 172.21.15.63:0/3445842199 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fea400593f0 con 0x7fea4c102e00 2024-03-23T12:08:39.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.730+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea4c04bf40 con 0x7fea3406c7d0 2024-03-23T12:08:39.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.732+0000 7fea49600700 1 -- 172.21.15.63:0/3445842199 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fea4c04bf40 con 0x7fea3406c7d0 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:08:39.712 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:08:39.713 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:08:39.713 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:08:39.713 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:08:39.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea3406c7d0 msgr2=0x7fea3406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea3406c7d0 0x7fea3406ec80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fea4c111640 tx=0x7fea38009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:39.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c102e00 msgr2=0x7fea4c115da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c102e00 0x7fea4c115da0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fea40002a50 tx=0x7fea400046b0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 shutdown_connections 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fea3406c7d0 0x7fea3406ec80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c102e00 0x7fea4c115da0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 --2- 172.21.15.63:0/3445842199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea4c1037a0 0x7fea4c1162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 >> 172.21.15.63:0/3445842199 conn(0x7fea4c0f89b0 msgr2=0x7fea4c0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 shutdown_connections 2024-03-23T12:08:39.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.734+0000 7fea54000700 1 -- 172.21.15.63:0/3445842199 wait complete. 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 -- 172.21.15.63:0/3491465228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 msgr2=0x7fe6341031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 --2- 172.21.15.63:0/3491465228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe6341031f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe628004940 tx=0x7fe628009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 -- 172.21.15.63:0/3491465228 shutdown_connections 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 --2- 172.21.15.63:0/3491465228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6341037c0 0x7fe6340fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 --2- 172.21.15.63:0/3491465228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe6341031f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 -- 172.21.15.63:0/3491465228 >> 172.21.15.63:0/3491465228 conn(0x7fe6340f89d0 msgr2=0x7fe6340fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 -- 172.21.15.63:0/3491465228 shutdown_connections 2024-03-23T12:08:39.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.871+0000 7fe63bc00700 1 -- 172.21.15.63:0/3491465228 wait complete. 2024-03-23T12:08:39.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.872+0000 7fe63bc00700 1 Processor -- start 2024-03-23T12:08:39.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.872+0000 7fe63bc00700 1 -- start start 2024-03-23T12:08:39.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe63bc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe634113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe63bc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6341037c0 0x7fe634114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe63bc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe634114690 con 0x7fe634102e20 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe63bc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63410da80 con 0x7fe6341037c0 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe639200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe634113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe639200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe634113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40436/0 (socket says 172.21.15.63:40436) 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe639200700 1 -- 172.21.15.63:0/2055615669 learned_addr learned my addr 172.21.15.63:0/2055615669 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe633e00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6341037c0 0x7fe634114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe639200700 1 -- 172.21.15.63:0/2055615669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6341037c0 msgr2=0x7fe634114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe639200700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6341037c0 0x7fe634114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe639200700 1 -- 172.21.15.63:0/2055615669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe628005040 con 0x7fe634102e20 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.873+0000 7fe633e00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6341037c0 0x7fe634114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:08:39.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.874+0000 7fe639200700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe634113b00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe628009390 tx=0x7fe628009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.874+0000 7fe630c00700 1 -- 172.21.15.63:0/2055615669 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe62800e040 con 0x7fe634102e20 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.874+0000 7fe630c00700 1 -- 172.21.15.63:0/2055615669 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe628013420 con 0x7fe634102e20 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.874+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63410dd00 con 0x7fe634102e20 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.874+0000 7fe630c00700 1 -- 172.21.15.63:0/2055615669 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe62801c630 con 0x7fe634102e20 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.874+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63410e2c0 con 0x7fe634102e20 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.875+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6340fe570 con 0x7fe634102e20 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.876+0000 7fe630c00700 1 -- 172.21.15.63:0/2055615669 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe6280139e0 con 0x7fe634102e20 2024-03-23T12:08:39.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.877+0000 7fe630c00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe61c06c7d0 0x7fe61c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:08:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.877+0000 7fe630c00700 1 -- 172.21.15.63:0/2055615669 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe62808cbd0 con 0x7fe634102e20 2024-03-23T12:08:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.877+0000 7fe633e00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe61c06c7d0 0x7fe61c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:08:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.878+0000 7fe633e00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe61c06c7d0 0x7fe61c06ec80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe620009fd0 tx=0x7fe62000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:08:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:39.881+0000 7fe630c00700 1 -- 172.21.15.63:0/2055615669 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe62805aa50 con 0x7fe634102e20 2024-03-23T12:08:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.181+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe63404bed0 con 0x7fe634102e20 2024-03-23T12:08:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.181+0000 7fe630c00700 1 -- 172.21.15.63:0/2055615669 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe62805a5e0 con 0x7fe634102e20 2024-03-23T12:08:40.164 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:08:40.164 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:08:40.164 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:08:40.164 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:08:40.165 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:08:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe61c06c7d0 msgr2=0x7fe61c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe61c06c7d0 0x7fe61c06ec80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe620009fd0 tx=0x7fe62000b040 comp rx=0 tx=0).stop 2024-03-23T12:08:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 msgr2=0x7fe634113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:08:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe634113b00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe628009390 tx=0x7fe628009e30 comp rx=0 tx=0).stop 2024-03-23T12:08:40.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 shutdown_connections 2024-03-23T12:08:40.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe61c06c7d0 0x7fe61c06ec80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:40.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe634102e20 0x7fe634113b00 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:40.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 --2- 172.21.15.63:0/2055615669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6341037c0 0x7fe634114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:08:40.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 >> 172.21.15.63:0/2055615669 conn(0x7fe6340f89d0 msgr2=0x7fe6340f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:08:40.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 shutdown_connections 2024-03-23T12:08:40.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:08:40.184+0000 7fe63bc00700 1 -- 172.21.15.63:0/2055615669 wait complete. 2024-03-23T12:08:40.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:39 smithi063 ceph-mon[141124]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:40.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:39 smithi063 ceph-mon[141124]: from='client.35002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:40.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:39 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1937760406' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:08:40.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:39 smithi111 ceph-mon[106899]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:40.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:39 smithi111 ceph-mon[106899]: from='client.35002 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:40.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:39 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1937760406' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:08:41.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:40 smithi063 ceph-mon[141124]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:40 smithi063 ceph-mon[141124]: pgmap v526: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:40 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2055615669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:08:41.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:40 smithi111 ceph-mon[106899]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:08:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:40 smithi111 ceph-mon[106899]: pgmap v526: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:41.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:40 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2055615669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:08:41.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:43.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:42 smithi063 ceph-mon[141124]: pgmap v527: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:43.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:43 smithi111 ceph-mon[106899]: pgmap v527: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:44.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:44 smithi063 ceph-mon[141124]: pgmap v528: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:44.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:44 smithi111 ceph-mon[106899]: pgmap v528: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:46.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:46 smithi063 ceph-mon[141124]: pgmap v529: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:46.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:46 smithi111 ceph-mon[106899]: pgmap v529: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:49.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:48 smithi063 ceph-mon[141124]: pgmap v530: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:49.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:48 smithi111 ceph-mon[106899]: pgmap v530: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:51.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:50 smithi063 ceph-mon[141124]: pgmap v531: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:51.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:50 smithi111 ceph-mon[106899]: pgmap v531: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:53.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:53 smithi063 ceph-mon[141124]: pgmap v532: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:53.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:53 smithi111 ceph-mon[106899]: pgmap v532: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:55.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:55 smithi063 ceph-mon[141124]: pgmap v533: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:55 smithi111 ceph-mon[106899]: pgmap v533: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:56 smithi063 ceph-mon[141124]: pgmap v534: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:08:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:56 smithi111 ceph-mon[106899]: pgmap v534: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:59.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:08:58 smithi063 ceph-mon[141124]: pgmap v535: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:08:59.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:08:58 smithi111 ceph-mon[106899]: pgmap v535: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:01.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:00 smithi063 ceph-mon[141124]: pgmap v536: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:01.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:01 smithi111 ceph-mon[106899]: pgmap v536: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:03.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:03 smithi063 ceph-mon[141124]: pgmap v537: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:03.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:03 smithi111 ceph-mon[106899]: pgmap v537: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:04.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:04 smithi063 ceph-mon[141124]: pgmap v538: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:04.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:04 smithi111 ceph-mon[106899]: pgmap v538: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:06.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:06 smithi063 ceph-mon[141124]: pgmap v539: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:06.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:06 smithi111 ceph-mon[106899]: pgmap v539: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:09.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:08 smithi063 ceph-mon[141124]: pgmap v540: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:09.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:08 smithi111 ceph-mon[106899]: pgmap v540: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:10.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 -- 172.21.15.63:0/4221101451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 msgr2=0x7f624c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 --2- 172.21.15.63:0/4221101451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c1031d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f623c004940 tx=0x7f623c009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 -- 172.21.15.63:0/4221101451 shutdown_connections 2024-03-23T12:09:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 --2- 172.21.15.63:0/4221101451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f624c1037a0 0x7f624c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 --2- 172.21.15.63:0/4221101451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c1031d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 -- 172.21.15.63:0/4221101451 >> 172.21.15.63:0/4221101451 conn(0x7f624c0f89b0 msgr2=0x7f624c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 -- 172.21.15.63:0/4221101451 shutdown_connections 2024-03-23T12:09:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.337+0000 7f6252600700 1 -- 172.21.15.63:0/4221101451 wait complete. 2024-03-23T12:09:10.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.338+0000 7f6252600700 1 Processor -- start 2024-03-23T12:09:10.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.338+0000 7f6252600700 1 -- start start 2024-03-23T12:09:10.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f6252600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:10.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f6252600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f624c1037a0 0x7f624c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f6252600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c070d90 con 0x7f624c102e00 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f6252600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c070ed0 con 0x7f624c1037a0 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f624b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f624b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40204/0 (socket says 172.21.15.63:40204) 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f624b400700 1 -- 172.21.15.63:0/1789956389 learned_addr learned my addr 172.21.15.63:0/1789956389 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.339+0000 7f6243e00700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f624c1037a0 0x7f624c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f624b400700 1 -- 172.21.15.63:0/1789956389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f624c1037a0 msgr2=0x7f624c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f624b400700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f624c1037a0 0x7f624c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f624b400700 1 -- 172.21.15.63:0/1789956389 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c005040 con 0x7f624c102e00 2024-03-23T12:09:10.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f6243e00700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f624c1037a0 0x7f624c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:09:10.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f624b400700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c0702c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f623c009390 tx=0x7f623c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:10.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f6243400700 1 -- 172.21.15.63:0/1789956389 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623c00e040 con 0x7f624c102e00 2024-03-23T12:09:10.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f6243400700 1 -- 172.21.15.63:0/1789956389 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f623c013420 con 0x7f624c102e00 2024-03-23T12:09:10.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f6243400700 1 -- 172.21.15.63:0/1789956389 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623c01c630 con 0x7f624c102e00 2024-03-23T12:09:10.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624c071120 con 0x7f624c102e00 2024-03-23T12:09:10.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.340+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624c11a170 con 0x7f624c102e00 2024-03-23T12:09:10.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.341+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624c0fe550 con 0x7f624c102e00 2024-03-23T12:09:10.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.342+0000 7f6243400700 1 -- 172.21.15.63:0/1789956389 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f623c0139e0 con 0x7f624c102e00 2024-03-23T12:09:10.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.343+0000 7f6243400700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623006c7d0 0x7f623006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:10.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.344+0000 7f6243400700 1 -- 172.21.15.63:0/1789956389 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f623c08cca0 con 0x7f624c102e00 2024-03-23T12:09:10.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.344+0000 7f6243e00700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623006c7d0 0x7f623006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:10.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.344+0000 7f6243e00700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623006c7d0 0x7f623006ec80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6234007fd0 tx=0x7f6234009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:10.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.348+0000 7f6243400700 1 -- 172.21.15.63:0/1789956389 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f623c056f20 con 0x7f624c102e00 2024-03-23T12:09:10.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.577+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f624c0719b0 con 0x7f623006c7d0 2024-03-23T12:09:10.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.578+0000 7f6243400700 1 -- 172.21.15.63:0/1789956389 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f624c0719b0 con 0x7f623006c7d0 2024-03-23T12:09:10.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.580+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623006c7d0 msgr2=0x7f623006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.580+0000 7f6252600700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623006c7d0 0x7f623006ec80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6234007fd0 tx=0x7f6234009f60 comp rx=0 tx=0).stop 2024-03-23T12:09:10.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.580+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 msgr2=0x7f624c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.580+0000 7f6252600700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c0702c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f623c009390 tx=0x7f623c009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.581+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 shutdown_connections 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.581+0000 7f6252600700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f623006c7d0 0x7f623006ec80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.581+0000 7f6252600700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f624c102e00 0x7f624c0702c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.581+0000 7f6252600700 1 --2- 172.21.15.63:0/1789956389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f624c1037a0 0x7f624c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.581+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 >> 172.21.15.63:0/1789956389 conn(0x7f624c0f89b0 msgr2=0x7f624c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.581+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 shutdown_connections 2024-03-23T12:09:10.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.581+0000 7f6252600700 1 -- 172.21.15.63:0/1789956389 wait complete. 2024-03-23T12:09:10.576 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:09:10.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 -- 172.21.15.63:0/2246288633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde01037a0 msgr2=0x7fcde00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 --2- 172.21.15.63:0/2246288633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde01037a0 0x7fcde00fce10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc004940 tx=0x7fcdcc009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:10.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 -- 172.21.15.63:0/2246288633 shutdown_connections 2024-03-23T12:09:10.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 --2- 172.21.15.63:0/2246288633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde01037a0 0x7fcde00fce10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 --2- 172.21.15.63:0/2246288633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcde0102e00 0x7fcde01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 -- 172.21.15.63:0/2246288633 >> 172.21.15.63:0/2246288633 conn(0x7fcde00f89b0 msgr2=0x7fcde00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:10.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 -- 172.21.15.63:0/2246288633 shutdown_connections 2024-03-23T12:09:10.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.720+0000 7fcde6200700 1 -- 172.21.15.63:0/2246288633 wait complete. 2024-03-23T12:09:10.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.721+0000 7fcde6200700 1 Processor -- start 2024-03-23T12:09:10.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.721+0000 7fcde6200700 1 -- start start 2024-03-23T12:09:10.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.721+0000 7fcde6200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde0102e00 0x7fcde0115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:10.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.721+0000 7fcde6200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcde01037a0 0x7fcde0116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:10.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.721+0000 7fcde6200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01168b0 con 0x7fcde0102e00 2024-03-23T12:09:10.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.721+0000 7fcde6200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde010fc80 con 0x7fcde01037a0 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde0102e00 0x7fcde0115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde0102e00 0x7fcde0115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40214/0 (socket says 172.21.15.63:40214) 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddf400700 1 -- 172.21.15.63:0/3097228687 learned_addr learned my addr 172.21.15.63:0/3097228687 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddea00700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcde01037a0 0x7fcde0116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddf400700 1 -- 172.21.15.63:0/3097228687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcde01037a0 msgr2=0x7fcde0116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddf400700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcde01037a0 0x7fcde0116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddf400700 1 -- 172.21.15.63:0/3097228687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc005040 con 0x7fcde0102e00 2024-03-23T12:09:10.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.722+0000 7fcddf400700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde0102e00 0x7fcde0115d20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0002a50 tx=0x7fcdd00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:10.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.723+0000 7fcdd7400700 1 -- 172.21.15.63:0/3097228687 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdd0007cc0 con 0x7fcde0102e00 2024-03-23T12:09:10.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.723+0000 7fcdd7400700 1 -- 172.21.15.63:0/3097228687 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdd000f040 con 0x7fcde0102e00 2024-03-23T12:09:10.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.723+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde010ff60 con 0x7fcde0102e00 2024-03-23T12:09:10.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.723+0000 7fcdd7400700 1 -- 172.21.15.63:0/3097228687 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdd000d9c0 con 0x7fcde0102e00 2024-03-23T12:09:10.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.723+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde0110520 con 0x7fcde0102e00 2024-03-23T12:09:10.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.724+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde00fe550 con 0x7fcde0102e00 2024-03-23T12:09:10.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.725+0000 7fcdd7400700 1 -- 172.21.15.63:0/3097228687 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fcdd000db20 con 0x7fcde0102e00 2024-03-23T12:09:10.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.726+0000 7fcdd7400700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcdc406c7d0 0x7fcdc406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:10.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.726+0000 7fcdd7400700 1 -- 172.21.15.63:0/3097228687 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcdd008b7a0 con 0x7fcde0102e00 2024-03-23T12:09:10.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.726+0000 7fcddea00700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcdc406c7d0 0x7fcdc406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:10.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.726+0000 7fcddea00700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcdc406c7d0 0x7fcdc406ec80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcde01115c0 tx=0x7fcdcc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:10.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.730+0000 7fcdd7400700 1 -- 172.21.15.63:0/3097228687 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcdd0059430 con 0x7fcde0102e00 2024-03-23T12:09:10.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.965+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcde004bf40 con 0x7fcdc406c7d0 2024-03-23T12:09:10.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.967+0000 7fcdd7400700 1 -- 172.21.15.63:0/3097228687 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcde004bf40 con 0x7fcdc406c7d0 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcdc406c7d0 msgr2=0x7fcdc406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcdc406c7d0 0x7fcdc406ec80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcde01115c0 tx=0x7fcdcc009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde0102e00 msgr2=0x7fcde0115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde0102e00 0x7fcde0115d20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0002a50 tx=0x7fcdd00046b0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 shutdown_connections 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcdc406c7d0 0x7fcdc406ec80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcde0102e00 0x7fcde0115d20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 --2- 172.21.15.63:0/3097228687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcde01037a0 0x7fcde0116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:10.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 >> 172.21.15.63:0/3097228687 conn(0x7fcde00f89b0 msgr2=0x7fcde00f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:10.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 shutdown_connections 2024-03-23T12:09:10.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:10.969+0000 7fcde6200700 1 -- 172.21.15.63:0/3097228687 wait complete. 2024-03-23T12:09:11.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 -- 172.21.15.63:0/446060319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c102c40 msgr2=0x7f5f0c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 --2- 172.21.15.63:0/446060319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c102c40 0x7f5f0c103010 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5f00004940 tx=0x7f5f00009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:11.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 -- 172.21.15.63:0/446060319 shutdown_connections 2024-03-23T12:09:11.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 --2- 172.21.15.63:0/446060319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f0c1035e0 0x7f5f0c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 --2- 172.21.15.63:0/446060319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c102c40 0x7f5f0c103010 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 -- 172.21.15.63:0/446060319 >> 172.21.15.63:0/446060319 conn(0x7f5f0c0f8830 msgr2=0x7f5f0c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:11.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 -- 172.21.15.63:0/446060319 shutdown_connections 2024-03-23T12:09:11.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.107+0000 7f5f12a00700 1 -- 172.21.15.63:0/446060319 wait complete. 2024-03-23T12:09:11.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.108+0000 7f5f12a00700 1 Processor -- start 2024-03-23T12:09:11.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.109+0000 7f5f12a00700 1 -- start start 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.109+0000 7f5f12a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f0c102c40 0x7f5f0c113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.109+0000 7f5f12a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c1035e0 0x7f5f0c113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.109+0000 7f5f12a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c114400 con 0x7f5f0c1035e0 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.109+0000 7f5f12a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c10d7f0 con 0x7f5f0c102c40 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f10c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c1035e0 0x7f5f0c113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f10c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c1035e0 0x7f5f0c113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40236/0 (socket says 172.21.15.63:40236) 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f10c00700 1 -- 172.21.15.63:0/1122666590 learned_addr learned my addr 172.21.15.63:0/1122666590 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:11.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f11600700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f0c102c40 0x7f5f0c113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f10c00700 1 -- 172.21.15.63:0/1122666590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f0c102c40 msgr2=0x7f5f0c113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f10c00700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f0c102c40 0x7f5f0c113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f10c00700 1 -- 172.21.15.63:0/1122666590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f00005040 con 0x7f5f0c1035e0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f10c00700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c1035e0 0x7f5f0c113db0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800eb40 tx=0x7f5ef800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f09600700 1 -- 172.21.15.63:0/1122666590 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef8007cc0 con 0x7f5f0c1035e0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f09600700 1 -- 172.21.15.63:0/1122666590 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ef800f040 con 0x7f5f0c1035e0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c10da70 con 0x7f5f0c1035e0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.110+0000 7f5f09600700 1 -- 172.21.15.63:0/1122666590 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef8009570 con 0x7f5f0c1035e0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.111+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c10e030 con 0x7f5f0c1035e0 2024-03-23T12:09:11.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.112+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c0fe390 con 0x7f5f0c1035e0 2024-03-23T12:09:11.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.113+0000 7f5f09600700 1 -- 172.21.15.63:0/1122666590 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5ef80096d0 con 0x7f5f0c1035e0 2024-03-23T12:09:11.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.113+0000 7f5f09600700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5ef406c7d0 0x7f5ef406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.114+0000 7f5f09600700 1 -- 172.21.15.63:0/1122666590 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5ef808b650 con 0x7f5f0c1035e0 2024-03-23T12:09:11.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.114+0000 7f5f11600700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5ef406c7d0 0x7f5ef406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.114+0000 7f5f11600700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5ef406c7d0 0x7f5ef406ec80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5f00004560 tx=0x7f5f00009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:11.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.118+0000 7f5f09600700 1 -- 172.21.15.63:0/1122666590 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5ef8059310 con 0x7f5f0c1035e0 2024-03-23T12:09:11.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.341+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f0c109bf0 con 0x7f5ef406c7d0 2024-03-23T12:09:11.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.351+0000 7f5f09600700 1 -- 172.21.15.63:0/1122666590 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f5f0c109bf0 con 0x7f5ef406c7d0 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (18m) 11m ago 31m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (15m) 11m ago 31m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (15m) 10m ago 30m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (18m) 11m ago 31m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (11m) 11m ago 25m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (11m) 11m ago 25m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:09:11.333 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (25m) 10m ago 25m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (10m) 10m ago 25m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (20m) 11m ago 33m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (19m) 10m ago 30m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (16m) 11m ago 33m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (16m) 10m ago 30m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (19m) 11m ago 31m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (19m) 10m ago 30m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:09:11.334 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (15m) 11m ago 29m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:09:11.335 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (14m) 11m ago 28m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:09:11.335 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (13m) 11m ago 28m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:09:11.335 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (13m) 10m ago 27m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:09:11.335 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (12m) 10m ago 27m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:09:11.335 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (11m) 10m ago 26m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:09:11.335 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (18m) 11m ago 31m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:09:11.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5ef406c7d0 msgr2=0x7f5ef406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5ef406c7d0 0x7f5ef406ec80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5f00004560 tx=0x7f5f00009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:11.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c1035e0 msgr2=0x7f5f0c113db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c1035e0 0x7f5f0c113db0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800eb40 tx=0x7f5ef800b040 comp rx=0 tx=0).stop 2024-03-23T12:09:11.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 shutdown_connections 2024-03-23T12:09:11.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5ef406c7d0 0x7f5ef406ec80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f0c102c40 0x7f5f0c113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 --2- 172.21.15.63:0/1122666590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f0c1035e0 0x7f5f0c113db0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 >> 172.21.15.63:0/1122666590 conn(0x7f5f0c0f8830 msgr2=0x7f5f0c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:11.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 shutdown_connections 2024-03-23T12:09:11.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.353+0000 7f5f12a00700 1 -- 172.21.15.63:0/1122666590 wait complete. 2024-03-23T12:09:11.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:11 smithi063 ceph-mon[141124]: pgmap v541: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:11.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:11.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:11 smithi111 ceph-mon[106899]: pgmap v541: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:11.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 -- 172.21.15.63:0/1066245801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c1059a0 msgr2=0x7f3d1c0ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 --2- 172.21.15.63:0/1066245801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c1059a0 0x7f3d1c0ff010 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3d08004940 tx=0x7f3d08009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 -- 172.21.15.63:0/1066245801 shutdown_connections 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 --2- 172.21.15.63:0/1066245801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c1059a0 0x7f3d1c0ff010 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 --2- 172.21.15.63:0/1066245801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c105000 0x7f3d1c1053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 -- 172.21.15.63:0/1066245801 >> 172.21.15.63:0/1066245801 conn(0x7f3d1c0fabd0 msgr2=0x7f3d1c0fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 -- 172.21.15.63:0/1066245801 shutdown_connections 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.490+0000 7f3d23200700 1 -- 172.21.15.63:0/1066245801 wait complete. 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.491+0000 7f3d23200700 1 Processor -- start 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.491+0000 7f3d23200700 1 -- start start 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d23200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c105000 0x7f3d1c115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d23200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c1059a0 0x7f3d1c1161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d23200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c116820 con 0x7f3d1c105000 2024-03-23T12:09:11.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d23200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c10fbf0 con 0x7f3d1c1059a0 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d1be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c105000 0x7f3d1c115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d1be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c105000 0x7f3d1c115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39676/0 (socket says 172.21.15.63:39676) 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d1be00700 1 -- 172.21.15.63:0/3060661927 learned_addr learned my addr 172.21.15.63:0/3060661927 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d1b400700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c1059a0 0x7f3d1c1161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d1be00700 1 -- 172.21.15.63:0/3060661927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c1059a0 msgr2=0x7f3d1c1161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d1be00700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c1059a0 0x7f3d1c1161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.492+0000 7f3d1be00700 1 -- 172.21.15.63:0/3060661927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08005040 con 0x7f3d1c105000 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.493+0000 7f3d1be00700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c105000 0x7f3d1c115c90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00eb40 tx=0x7f3d0c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.493+0000 7f3d13e00700 1 -- 172.21.15.63:0/3060661927 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d0c007cc0 con 0x7f3d1c105000 2024-03-23T12:09:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.493+0000 7f3d13e00700 1 -- 172.21.15.63:0/3060661927 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d0c00f040 con 0x7f3d1c105000 2024-03-23T12:09:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.493+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c10fe70 con 0x7f3d1c105000 2024-03-23T12:09:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.493+0000 7f3d13e00700 1 -- 172.21.15.63:0/3060661927 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d0c009570 con 0x7f3d1c105000 2024-03-23T12:09:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.493+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c110430 con 0x7f3d1c105000 2024-03-23T12:09:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.494+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c102310 con 0x7f3d1c105000 2024-03-23T12:09:11.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.495+0000 7f3d13e00700 1 -- 172.21.15.63:0/3060661927 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3d0c0096d0 con 0x7f3d1c105000 2024-03-23T12:09:11.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.496+0000 7f3d13e00700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0006c7d0 0x7f3d0006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.496+0000 7f3d13e00700 1 -- 172.21.15.63:0/3060661927 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3d0c08b6f0 con 0x7f3d1c105000 2024-03-23T12:09:11.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.496+0000 7f3d1b400700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0006c7d0 0x7f3d0006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.497+0000 7f3d1b400700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0006c7d0 0x7f3d0006ec80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3d08009fd0 tx=0x7f3d08009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:11.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.500+0000 7f3d13e00700 1 -- 172.21.15.63:0/3060661927 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3d0c059430 con 0x7f3d1c105000 2024-03-23T12:09:11.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.803+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3d1c04bed0 con 0x7f3d1c105000 2024-03-23T12:09:11.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.803+0000 7f3d13e00700 1 -- 172.21.15.63:0/3060661927 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f3d0c058fc0 con 0x7f3d1c105000 2024-03-23T12:09:11.784 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:09:11.784 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:09:11.784 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:09:11.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:11.784 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:09:11.785 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0006c7d0 msgr2=0x7f3d0006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0006c7d0 0x7f3d0006ec80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3d08009fd0 tx=0x7f3d08009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c105000 msgr2=0x7f3d1c115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c105000 0x7f3d1c115c90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00eb40 tx=0x7f3d0c00b040 comp rx=0 tx=0).stop 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 shutdown_connections 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0006c7d0 0x7f3d0006ec80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d1c105000 0x7f3d1c115c90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 --2- 172.21.15.63:0/3060661927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c1059a0 0x7f3d1c1161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 >> 172.21.15.63:0/3060661927 conn(0x7f3d1c0fabd0 msgr2=0x7f3d1c1095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 shutdown_connections 2024-03-23T12:09:11.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.806+0000 7f3d23200700 1 -- 172.21.15.63:0/3060661927 wait complete. 2024-03-23T12:09:11.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 -- 172.21.15.63:0/276742684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fde90 msgr2=0x7ff2100fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 --2- 172.21.15.63:0/276742684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fde90 0x7ff2100fe260 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff200004940 tx=0x7ff200009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:11.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 -- 172.21.15.63:0/276742684 shutdown_connections 2024-03-23T12:09:11.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 --2- 172.21.15.63:0/276742684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2100fe830 0x7ff210106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 --2- 172.21.15.63:0/276742684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fde90 0x7ff2100fe260 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 -- 172.21.15.63:0/276742684 >> 172.21.15.63:0/276742684 conn(0x7ff210072bc0 msgr2=0x7ff210072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:11.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 -- 172.21.15.63:0/276742684 shutdown_connections 2024-03-23T12:09:11.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.942+0000 7ff215e00700 1 -- 172.21.15.63:0/276742684 wait complete. 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.943+0000 7ff215e00700 1 Processor -- start 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.943+0000 7ff215e00700 1 -- start start 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff215e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2100fde90 0x7ff2100fef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff215e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fe830 0x7ff2100ff4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff215e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210103100 con 0x7ff2100fe830 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff215e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100ff9f0 con 0x7ff2100fde90 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff20e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fe830 0x7ff2100ff4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff20e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fe830 0x7ff2100ff4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39694/0 (socket says 172.21.15.63:39694) 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff20e000700 1 -- 172.21.15.63:0/2857149166 learned_addr learned my addr 172.21.15.63:0/2857149166 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff20ea00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2100fde90 0x7ff2100fef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff20e000700 1 -- 172.21.15.63:0/2857149166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2100fde90 msgr2=0x7ff2100fef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff20e000700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2100fde90 0x7ff2100fef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.944+0000 7ff20e000700 1 -- 172.21.15.63:0/2857149166 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200005040 con 0x7ff2100fe830 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.945+0000 7ff20e000700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fe830 0x7ff2100ff4b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc002a50 tx=0x7ff1fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.945+0000 7ff206a00700 1 -- 172.21.15.63:0/2857149166 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1fc007cc0 con 0x7ff2100fe830 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.945+0000 7ff206a00700 1 -- 172.21.15.63:0/2857149166 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1fc00f040 con 0x7ff2100fe830 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.945+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2100ffcd0 con 0x7ff2100fe830 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.945+0000 7ff206a00700 1 -- 172.21.15.63:0/2857149166 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1fc00d9c0 con 0x7ff2100fe830 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.945+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21011e650 con 0x7ff2100fe830 2024-03-23T12:09:11.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.946+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff210106a20 con 0x7ff2100fe830 2024-03-23T12:09:11.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.947+0000 7ff206a00700 1 -- 172.21.15.63:0/2857149166 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff1fc00db20 con 0x7ff2100fe830 2024-03-23T12:09:11.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.948+0000 7ff206a00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff1f406c7d0 0x7ff1f406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:11.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.948+0000 7ff206a00700 1 -- 172.21.15.63:0/2857149166 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff1fc08b6c0 con 0x7ff2100fe830 2024-03-23T12:09:11.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.948+0000 7ff20ea00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff1f406c7d0 0x7ff1f406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:11.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.949+0000 7ff20ea00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff1f406c7d0 0x7ff1f406ec80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff20000b3b0 tx=0x7ff200009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:11.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:11.953+0000 7ff206a00700 1 -- 172.21.15.63:0/2857149166 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff1fc055940 con 0x7ff2100fe830 2024-03-23T12:09:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.213+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff210065e30 con 0x7ff2100fe830 2024-03-23T12:09:12.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.213+0000 7ff206a00700 1 -- 172.21.15.63:0/2857149166 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7ff1fc058f60 con 0x7ff2100fe830 2024-03-23T12:09:12.194 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:09:12.194 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:09:12.194 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:09:12.194 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:09:12.195 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:12.196 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:12.197 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:09:12.197 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:12.197 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:12.197 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:12.197 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:12.197 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff1f406c7d0 msgr2=0x7ff1f406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff1f406c7d0 0x7ff1f406ec80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff20000b3b0 tx=0x7ff200009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fe830 msgr2=0x7ff2100ff4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fe830 0x7ff2100ff4b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc002a50 tx=0x7ff1fc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 shutdown_connections 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff1f406c7d0 0x7ff1f406ec80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2100fde90 0x7ff2100fef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 --2- 172.21.15.63:0/2857149166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2100fe830 0x7ff2100ff4b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.215+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 >> 172.21.15.63:0/2857149166 conn(0x7ff210072bc0 msgr2=0x7ff2100fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.216+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 shutdown_connections 2024-03-23T12:09:12.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.216+0000 7ff215e00700 1 -- 172.21.15.63:0/2857149166 wait complete. 2024-03-23T12:09:12.199 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:09:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 -- 172.21.15.63:0/1985260677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 msgr2=0x7f92800fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 --2- 172.21.15.63:0/1985260677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f92800fce10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f926c004940 tx=0x7f926c009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 -- 172.21.15.63:0/1985260677 shutdown_connections 2024-03-23T12:09:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 --2- 172.21.15.63:0/1985260677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f92800fce10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 --2- 172.21.15.63:0/1985260677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9280102e00 0x7f92801031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 -- 172.21.15.63:0/1985260677 >> 172.21.15.63:0/1985260677 conn(0x7f92800f89b0 msgr2=0x7f92800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 -- 172.21.15.63:0/1985260677 shutdown_connections 2024-03-23T12:09:12.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.354+0000 7f9287e00700 1 -- 172.21.15.63:0/1985260677 wait complete. 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.355+0000 7f9287e00700 1 Processor -- start 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.355+0000 7f9287e00700 1 -- start start 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.355+0000 7f9287e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9280102e00 0x7f92800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.355+0000 7f9287e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f9280070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.355+0000 7f9287e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9280070d90 con 0x7f92801037a0 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.355+0000 7f9287e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9280070ed0 con 0x7f9280102e00 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9284a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f9280070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9284a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f9280070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39720/0 (socket says 172.21.15.63:39720) 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9284a00700 1 -- 172.21.15.63:0/4095767315 learned_addr learned my addr 172.21.15.63:0/4095767315 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:12.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9285400700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9280102e00 0x7f92800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:12.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9284a00700 1 -- 172.21.15.63:0/4095767315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9280102e00 msgr2=0x7f92800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9284a00700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9280102e00 0x7f92800702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9284a00700 1 -- 172.21.15.63:0/4095767315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c005040 con 0x7f92801037a0 2024-03-23T12:09:12.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.356+0000 7f9284a00700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f9280070800 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f926c009390 tx=0x7f926c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.357+0000 7f927d600700 1 -- 172.21.15.63:0/4095767315 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f926c00e040 con 0x7f92801037a0 2024-03-23T12:09:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.357+0000 7f927d600700 1 -- 172.21.15.63:0/4095767315 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f926c004290 con 0x7f92801037a0 2024-03-23T12:09:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.357+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9280071120 con 0x7f92801037a0 2024-03-23T12:09:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.357+0000 7f927d600700 1 -- 172.21.15.63:0/4095767315 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f926c013e70 con 0x7f92801037a0 2024-03-23T12:09:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.357+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928011a230 con 0x7f92801037a0 2024-03-23T12:09:12.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.358+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92800fe550 con 0x7f92801037a0 2024-03-23T12:09:12.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.359+0000 7f927d600700 1 -- 172.21.15.63:0/4095767315 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f926c013690 con 0x7f92801037a0 2024-03-23T12:09:12.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.360+0000 7f927d600700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926806c760 0x7f926806ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:12.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.360+0000 7f927d600700 1 -- 172.21.15.63:0/4095767315 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f926c08cc60 con 0x7f92801037a0 2024-03-23T12:09:12.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.360+0000 7f9285400700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926806c760 0x7f926806ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:12.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.361+0000 7f9285400700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926806c760 0x7f926806ec10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9274007fd0 tx=0x7f9274009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:12.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.364+0000 7f927d600700 1 -- 172.21.15.63:0/4095767315 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f926c056f90 con 0x7f92801037a0 2024-03-23T12:09:12.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:12 smithi063 ceph-mon[141124]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:12.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:12 smithi063 ceph-mon[141124]: from='client.35026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:12.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:12 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3060661927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:09:12.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:12 smithi111 ceph-mon[106899]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:12.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:12 smithi111 ceph-mon[106899]: from='client.35026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:12.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:12 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3060661927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:09:12.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.597+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9280109da0 con 0x7f926806c760 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.598+0000 7f927d600700 1 -- 172.21.15.63:0/4095767315 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9280109da0 con 0x7f926806c760 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:09:12.579 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:09:12.580 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:09:12.580 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:09:12.580 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:09:12.580 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:09:12.580 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:09:12.580 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:09:12.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926806c760 msgr2=0x7f926806ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926806c760 0x7f926806ec10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9274007fd0 tx=0x7f9274009040 comp rx=0 tx=0).stop 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 msgr2=0x7f9280070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f9280070800 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f926c009390 tx=0x7f926c009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 shutdown_connections 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f926806c760 0x7f926806ec10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9280102e00 0x7f92800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 --2- 172.21.15.63:0/4095767315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92801037a0 0x7f9280070800 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.600+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 >> 172.21.15.63:0/4095767315 conn(0x7f92800f89b0 msgr2=0x7f92800f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.601+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 shutdown_connections 2024-03-23T12:09:12.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.601+0000 7f9287e00700 1 -- 172.21.15.63:0/4095767315 wait complete. 2024-03-23T12:09:12.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 -- 172.21.15.63:0/3230942723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c1037a0 msgr2=0x7ff71c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 --2- 172.21.15.63:0/3230942723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c1037a0 0x7ff71c0fce10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff708004940 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:12.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 -- 172.21.15.63:0/3230942723 shutdown_connections 2024-03-23T12:09:12.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 --2- 172.21.15.63:0/3230942723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c1037a0 0x7ff71c0fce10 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 --2- 172.21.15.63:0/3230942723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff71c102e00 0x7ff71c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 -- 172.21.15.63:0/3230942723 >> 172.21.15.63:0/3230942723 conn(0x7ff71c0f89b0 msgr2=0x7ff71c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:12.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 -- 172.21.15.63:0/3230942723 shutdown_connections 2024-03-23T12:09:12.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.737+0000 7ff724200700 1 -- 172.21.15.63:0/3230942723 wait complete. 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.738+0000 7ff724200700 1 Processor -- start 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.738+0000 7ff724200700 1 -- start start 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.738+0000 7ff724200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c102e00 0x7ff71c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.738+0000 7ff724200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff71c1037a0 0x7ff71c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.738+0000 7ff724200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c1167f0 con 0x7ff71c102e00 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.738+0000 7ff724200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c10fbc0 con 0x7ff71c1037a0 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.739+0000 7ff721800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c102e00 0x7ff71c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.739+0000 7ff721800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c102e00 0x7ff71c115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39748/0 (socket says 172.21.15.63:39748) 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.739+0000 7ff721800700 1 -- 172.21.15.63:0/1395992737 learned_addr learned my addr 172.21.15.63:0/1395992737 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.739+0000 7ff720e00700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff71c1037a0 0x7ff71c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.739+0000 7ff721800700 1 -- 172.21.15.63:0/1395992737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff71c1037a0 msgr2=0x7ff71c1161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.739+0000 7ff721800700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff71c1037a0 0x7ff71c1161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.739+0000 7ff721800700 1 -- 172.21.15.63:0/1395992737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708005040 con 0x7ff71c102e00 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.740+0000 7ff721800700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c102e00 0x7ff71c115c60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff710002a80 tx=0x7ff7100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.740+0000 7ff719600700 1 -- 172.21.15.63:0/1395992737 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff710007cc0 con 0x7ff71c102e00 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.740+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c10fea0 con 0x7ff71c102e00 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.740+0000 7ff719600700 1 -- 172.21.15.63:0/1395992737 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff71000f040 con 0x7ff71c102e00 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.740+0000 7ff719600700 1 -- 172.21.15.63:0/1395992737 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71000da60 con 0x7ff71c102e00 2024-03-23T12:09:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.740+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c110460 con 0x7ff71c102e00 2024-03-23T12:09:12.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.741+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c0fe550 con 0x7ff71c102e00 2024-03-23T12:09:12.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.742+0000 7ff719600700 1 -- 172.21.15.63:0/1395992737 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff71000dbc0 con 0x7ff71c102e00 2024-03-23T12:09:12.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.743+0000 7ff719600700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70406c7d0 0x7ff70406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:12.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.743+0000 7ff719600700 1 -- 172.21.15.63:0/1395992737 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff71008b760 con 0x7ff71c102e00 2024-03-23T12:09:12.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.743+0000 7ff720e00700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70406c7d0 0x7ff70406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:12.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.744+0000 7ff720e00700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70406c7d0 0x7ff70406ec80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff71c111500 tx=0x7ff708009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:12.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:12.747+0000 7ff719600700 1 -- 172.21.15.63:0/1395992737 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff710059370 con 0x7ff71c102e00 2024-03-23T12:09:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.049+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff71c04bed0 con 0x7ff71c102e00 2024-03-23T12:09:13.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.049+0000 7ff719600700 1 -- 172.21.15.63:0/1395992737 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff710058f00 con 0x7ff71c102e00 2024-03-23T12:09:13.031 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:09:13.031 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:09:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:09:13.031 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:09:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:09:13.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70406c7d0 msgr2=0x7ff70406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:13.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70406c7d0 0x7ff70406ec80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff71c111500 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:13.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c102e00 msgr2=0x7ff71c115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:13.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c102e00 0x7ff71c115c60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff710002a80 tx=0x7ff7100046b0 comp rx=0 tx=0).stop 2024-03-23T12:09:13.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 shutdown_connections 2024-03-23T12:09:13.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70406c7d0 0x7ff70406ec80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:13.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff71c102e00 0x7ff71c115c60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:13.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 --2- 172.21.15.63:0/1395992737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff71c1037a0 0x7ff71c1161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:13.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 >> 172.21.15.63:0/1395992737 conn(0x7ff71c0f89b0 msgr2=0x7ff71c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:13.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 shutdown_connections 2024-03-23T12:09:13.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:13.052+0000 7ff724200700 1 -- 172.21.15.63:0/1395992737 wait complete. 2024-03-23T12:09:13.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:13 smithi063 ceph-mon[141124]: from='client.35030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:13.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:13 smithi063 ceph-mon[141124]: pgmap v542: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:13.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:13 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2857149166' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:09:13.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:13 smithi063 ceph-mon[141124]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:13.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:13 smithi111 ceph-mon[106899]: from='client.35030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:13.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:13 smithi111 ceph-mon[106899]: pgmap v542: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:13.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:13 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2857149166' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:09:13.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:13 smithi111 ceph-mon[106899]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:14.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:14 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1395992737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:09:14.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:14 smithi063 ceph-mon[141124]: pgmap v543: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:14.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:14 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1395992737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:09:14.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:14 smithi111 ceph-mon[106899]: pgmap v543: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:16.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:16 smithi063 ceph-mon[141124]: pgmap v544: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:16.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:16 smithi111 ceph-mon[106899]: pgmap v544: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:19.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:19 smithi063 ceph-mon[141124]: pgmap v545: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:19.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:19 smithi111 ceph-mon[106899]: pgmap v545: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:21.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:21 smithi063 ceph-mon[141124]: pgmap v546: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:21.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:21 smithi111 ceph-mon[106899]: pgmap v546: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:23.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:23 smithi063 ceph-mon[141124]: pgmap v547: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:23 smithi111 ceph-mon[106899]: pgmap v547: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:24.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:24 smithi063 ceph-mon[141124]: pgmap v548: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:24.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:24 smithi111 ceph-mon[106899]: pgmap v548: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:26 smithi063 ceph-mon[141124]: pgmap v549: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:26.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:26 smithi111 ceph-mon[106899]: pgmap v549: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:29.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:29 smithi063 ceph-mon[141124]: pgmap v550: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:29.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:29 smithi111 ceph-mon[106899]: pgmap v550: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:31.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:31 smithi063 ceph-mon[141124]: pgmap v551: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:31.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:31 smithi111 ceph-mon[106899]: pgmap v551: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:33 smithi063 ceph-mon[141124]: pgmap v552: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:33.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:33 smithi111 ceph-mon[106899]: pgmap v552: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:34 smithi063 ceph-mon[141124]: pgmap v553: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:34.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:34 smithi111 ceph-mon[106899]: pgmap v553: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:36.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:36 smithi063 ceph-mon[141124]: pgmap v554: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:36.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:36 smithi111 ceph-mon[106899]: pgmap v554: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:39.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:39 smithi063 ceph-mon[141124]: pgmap v555: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:39.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:39 smithi111 ceph-mon[106899]: pgmap v555: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:41.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:41 smithi063 ceph-mon[141124]: pgmap v556: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:41.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:41 smithi111 ceph-mon[106899]: pgmap v556: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:41.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:42.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:42 smithi063 ceph-mon[141124]: pgmap v557: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:42.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:42 smithi111 ceph-mon[106899]: pgmap v557: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:43.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 -- 172.21.15.63:0/4033739922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 msgr2=0x7fcecc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 --2- 172.21.15.63:0/4033739922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc106f00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fceb8004940 tx=0x7fceb8009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:43.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 -- 172.21.15.63:0/4033739922 shutdown_connections 2024-03-23T12:09:43.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 --2- 172.21.15.63:0/4033739922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc106f00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 --2- 172.21.15.63:0/4033739922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcecc0fde90 0x7fcecc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 -- 172.21.15.63:0/4033739922 >> 172.21.15.63:0/4033739922 conn(0x7fcecc072bc0 msgr2=0x7fcecc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:43.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 -- 172.21.15.63:0/4033739922 shutdown_connections 2024-03-23T12:09:43.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.206+0000 7fced2400700 1 -- 172.21.15.63:0/4033739922 wait complete. 2024-03-23T12:09:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.207+0000 7fced2400700 1 Processor -- start 2024-03-23T12:09:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.207+0000 7fced2400700 1 -- start start 2024-03-23T12:09:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fced2400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcecc0fde90 0x7fcecc0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fced2400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fced2400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0fc820 con 0x7fcecc0fe830 2024-03-23T12:09:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fced2400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0fc960 con 0x7fcecc0fde90 2024-03-23T12:09:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fcecaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fcecaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55912/0 (socket says 172.21.15.63:55912) 2024-03-23T12:09:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fcecaa00700 1 -- 172.21.15.63:0/2463082652 learned_addr learned my addr 172.21.15.63:0/2463082652 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.208+0000 7fcecb400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcecc0fde90 0x7fcecc0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fcecaa00700 1 -- 172.21.15.63:0/2463082652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcecc0fde90 msgr2=0x7fcecc0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fcecaa00700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcecc0fde90 0x7fcecc0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fcecaa00700 1 -- 172.21.15.63:0/2463082652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb8005040 con 0x7fcecc0fe830 2024-03-23T12:09:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fcecaa00700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc0fc290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fceb8000c00 tx=0x7fceb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fcec3400700 1 -- 172.21.15.63:0/2463082652 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceb800e040 con 0x7fcecc0fe830 2024-03-23T12:09:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fcec3400700 1 -- 172.21.15.63:0/2463082652 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fceb800bed0 con 0x7fcecc0fe830 2024-03-23T12:09:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcecc0fcbb0 con 0x7fcecc0fe830 2024-03-23T12:09:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fcec3400700 1 -- 172.21.15.63:0/2463082652 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceb801c3f0 con 0x7fcecc0fe830 2024-03-23T12:09:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.209+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcecc11a190 con 0x7fcecc0fe830 2024-03-23T12:09:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.210+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcecc065e30 con 0x7fcecc0fe830 2024-03-23T12:09:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.212+0000 7fcec3400700 1 -- 172.21.15.63:0/2463082652 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fceb8013420 con 0x7fcecc0fe830 2024-03-23T12:09:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.212+0000 7fcec3400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fceb006c7d0 0x7fceb006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.212+0000 7fcec3400700 1 -- 172.21.15.63:0/2463082652 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fceb808cee0 con 0x7fcecc0fe830 2024-03-23T12:09:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.213+0000 7fcecb400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fceb006c7d0 0x7fceb006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.213+0000 7fcecb400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fceb006c7d0 0x7fceb006ec80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcebc004f40 tx=0x7fcebc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:43.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.217+0000 7fcec3400700 1 -- 172.21.15.63:0/2463082652 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fceb8057120 con 0x7fcecc0fe830 2024-03-23T12:09:43.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.451+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcecc109e40 con 0x7fceb006c7d0 2024-03-23T12:09:43.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.453+0000 7fcec3400700 1 -- 172.21.15.63:0/2463082652 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcecc109e40 con 0x7fceb006c7d0 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fceb006c7d0 msgr2=0x7fceb006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fceb006c7d0 0x7fceb006ec80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcebc004f40 tx=0x7fcebc00b040 comp rx=0 tx=0).stop 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 msgr2=0x7fcecc0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc0fc290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fceb8000c00 tx=0x7fceb8009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 shutdown_connections 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fceb006c7d0 0x7fceb006ec80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcecc0fde90 0x7fcecc0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 --2- 172.21.15.63:0/2463082652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcecc0fe830 0x7fcecc0fc290 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 >> 172.21.15.63:0/2463082652 conn(0x7fcecc072bc0 msgr2=0x7fcecc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 shutdown_connections 2024-03-23T12:09:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.455+0000 7fced2400700 1 -- 172.21.15.63:0/2463082652 wait complete. 2024-03-23T12:09:43.450 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.595+0000 7f5977400700 1 -- 172.21.15.63:0/2393876778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 msgr2=0x7f59700fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.595+0000 7f5977400700 1 --2- 172.21.15.63:0/2393876778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f59700fce10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f595c004940 tx=0x7f595c009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.595+0000 7f5977400700 1 -- 172.21.15.63:0/2393876778 shutdown_connections 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.596+0000 7f5977400700 1 --2- 172.21.15.63:0/2393876778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f59700fce10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.596+0000 7f5977400700 1 --2- 172.21.15.63:0/2393876778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5970102e00 0x7f59701031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.596+0000 7f5977400700 1 -- 172.21.15.63:0/2393876778 >> 172.21.15.63:0/2393876778 conn(0x7f59700f89b0 msgr2=0x7f59700fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.596+0000 7f5977400700 1 -- 172.21.15.63:0/2393876778 shutdown_connections 2024-03-23T12:09:43.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.596+0000 7f5977400700 1 -- 172.21.15.63:0/2393876778 wait complete. 2024-03-23T12:09:43.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.596+0000 7f5977400700 1 Processor -- start 2024-03-23T12:09:43.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f5977400700 1 -- start start 2024-03-23T12:09:43.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f5977400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5970102e00 0x7f5970115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f5977400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f5970116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f5977400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5970116890 con 0x7f59701037a0 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f5977400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597010fc60 con 0x7f5970102e00 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f596fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f5970116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f596fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f5970116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55936/0 (socket says 172.21.15.63:55936) 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f596fe00700 1 -- 172.21.15.63:0/1770040818 learned_addr learned my addr 172.21.15.63:0/1770040818 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.597+0000 7f5974a00700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5970102e00 0x7f5970115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f596fe00700 1 -- 172.21.15.63:0/1770040818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5970102e00 msgr2=0x7f5970115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f596fe00700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5970102e00 0x7f5970115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f596fe00700 1 -- 172.21.15.63:0/1770040818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c005040 con 0x7f59701037a0 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f596fe00700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f5970116240 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f595c009390 tx=0x7f595c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:43.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f596cc00700 1 -- 172.21.15.63:0/1770040818 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f595c00e040 con 0x7f59701037a0 2024-03-23T12:09:43.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f596cc00700 1 -- 172.21.15.63:0/1770040818 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f595c013420 con 0x7f59701037a0 2024-03-23T12:09:43.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f596cc00700 1 -- 172.21.15.63:0/1770040818 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f595c01c630 con 0x7f59701037a0 2024-03-23T12:09:43.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597010fee0 con 0x7f59701037a0 2024-03-23T12:09:43.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.598+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59701104a0 con 0x7f59701037a0 2024-03-23T12:09:43.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.599+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59700fe550 con 0x7f59701037a0 2024-03-23T12:09:43.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.600+0000 7f596cc00700 1 -- 172.21.15.63:0/1770040818 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f595c013a00 con 0x7f59701037a0 2024-03-23T12:09:43.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.601+0000 7f596cc00700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f595806c7d0 0x7f595806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.601+0000 7f596cc00700 1 -- 172.21.15.63:0/1770040818 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f595c08cb20 con 0x7f59701037a0 2024-03-23T12:09:43.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.601+0000 7f5974a00700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f595806c7d0 0x7f595806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.602+0000 7f5974a00700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f595806c7d0 0x7f595806ec80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5964007fd0 tx=0x7f5964009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:43.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.606+0000 7f596cc00700 1 -- 172.21.15.63:0/1770040818 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f595c05a830 con 0x7f59701037a0 2024-03-23T12:09:43.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.839+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5970109da0 con 0x7f595806c7d0 2024-03-23T12:09:43.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.841+0000 7f596cc00700 1 -- 172.21.15.63:0/1770040818 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5970109da0 con 0x7f595806c7d0 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f595806c7d0 msgr2=0x7f595806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f595806c7d0 0x7f595806ec80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5964007fd0 tx=0x7f5964009040 comp rx=0 tx=0).stop 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 msgr2=0x7f5970116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f5970116240 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f595c009390 tx=0x7f595c009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 shutdown_connections 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f595806c7d0 0x7f595806ec80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5970102e00 0x7f5970115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 --2- 172.21.15.63:0/1770040818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701037a0 0x7f5970116240 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 >> 172.21.15.63:0/1770040818 conn(0x7f59700f89b0 msgr2=0x7f59700f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:43.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 shutdown_connections 2024-03-23T12:09:43.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.843+0000 7f5977400700 1 -- 172.21.15.63:0/1770040818 wait complete. 2024-03-23T12:09:43.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 -- 172.21.15.63:0/2371350730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12141059b0 msgr2=0x7f12140ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 --2- 172.21.15.63:0/2371350730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12141059b0 0x7f12140ff020 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1200004940 tx=0x7f1200009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:43.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 -- 172.21.15.63:0/2371350730 shutdown_connections 2024-03-23T12:09:43.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 --2- 172.21.15.63:0/2371350730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12141059b0 0x7f12140ff020 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 --2- 172.21.15.63:0/2371350730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1214105010 0x7f12141053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 -- 172.21.15.63:0/2371350730 >> 172.21.15.63:0/2371350730 conn(0x7f12140faba0 msgr2=0x7f12140fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:43.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 -- 172.21.15.63:0/2371350730 shutdown_connections 2024-03-23T12:09:43.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.982+0000 7f1219200700 1 -- 172.21.15.63:0/2371350730 wait complete. 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.983+0000 7f1219200700 1 Processor -- start 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.983+0000 7f1219200700 1 -- start start 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.983+0000 7f1219200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105010 0x7f1214115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.983+0000 7f1219200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12141059b0 0x7f1214116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.983+0000 7f1219200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12141168c0 con 0x7f1214105010 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1219200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121410fc90 con 0x7f12141059b0 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105010 0x7f1214115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105010 0x7f1214115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55956/0 (socket says 172.21.15.63:55956) 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211e00700 1 -- 172.21.15.63:0/3214764095 learned_addr learned my addr 172.21.15.63:0/3214764095 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211400700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12141059b0 0x7f1214116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211e00700 1 -- 172.21.15.63:0/3214764095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12141059b0 msgr2=0x7f1214116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211e00700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12141059b0 0x7f1214116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:43.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211e00700 1 -- 172.21.15.63:0/3214764095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1200005040 con 0x7f1214105010 2024-03-23T12:09:43.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.984+0000 7f1211e00700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105010 0x7f1214115d30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1204002a50 tx=0x7f12040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:43.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.985+0000 7f120a000700 1 -- 172.21.15.63:0/3214764095 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1204007cc0 con 0x7f1214105010 2024-03-23T12:09:43.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.985+0000 7f120a000700 1 -- 172.21.15.63:0/3214764095 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f120400f040 con 0x7f1214105010 2024-03-23T12:09:43.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.985+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121410ff70 con 0x7f1214105010 2024-03-23T12:09:43.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.985+0000 7f120a000700 1 -- 172.21.15.63:0/3214764095 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f120400da70 con 0x7f1214105010 2024-03-23T12:09:43.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.985+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1214110530 con 0x7f1214105010 2024-03-23T12:09:43.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.986+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1214100760 con 0x7f1214105010 2024-03-23T12:09:43.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.987+0000 7f120a000700 1 -- 172.21.15.63:0/3214764095 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f120400dc10 con 0x7f1214105010 2024-03-23T12:09:43.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.988+0000 7f120a000700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f806c7d0 0x7f11f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:43.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.988+0000 7f120a000700 1 -- 172.21.15.63:0/3214764095 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f120408b100 con 0x7f1214105010 2024-03-23T12:09:43.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.988+0000 7f1211400700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f806c7d0 0x7f11f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:43.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.989+0000 7f1211400700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f806c7d0 0x7f11f806ec80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f12141115d0 tx=0x7f1200009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:43.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:43.992+0000 7f120a000700 1 -- 172.21.15.63:0/3214764095 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1204058e10 con 0x7f1214105010 2024-03-23T12:09:44.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.217+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f121404bf40 con 0x7f11f806c7d0 2024-03-23T12:09:44.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.227+0000 7f120a000700 1 -- 172.21.15.63:0/3214764095 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f121404bf40 con 0x7f11f806c7d0 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (19m) 11m ago 32m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (16m) 11m ago 32m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (16m) 11m ago 31m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (18m) 11m ago 31m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (12m) 11m ago 25m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (11m) 11m ago 25m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (25m) 11m ago 25m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (11m) 11m ago 25m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (21m) 11m ago 33m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (20m) 11m ago 31m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (17m) 11m ago 33m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:09:44.208 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (16m) 11m ago 30m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (19m) 11m ago 32m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (19m) 11m ago 31m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (15m) 11m ago 30m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (15m) 11m ago 29m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (14m) 11m ago 28m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (13m) 11m ago 28m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (13m) 11m ago 27m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (12m) 11m ago 27m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:09:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (19m) 11m ago 31m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f806c7d0 msgr2=0x7f11f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f806c7d0 0x7f11f806ec80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f12141115d0 tx=0x7f1200009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105010 msgr2=0x7f1214115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105010 0x7f1214115d30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1204002a50 tx=0x7f12040046b0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 shutdown_connections 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f806c7d0 0x7f11f806ec80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105010 0x7f1214115d30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 --2- 172.21.15.63:0/3214764095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12141059b0 0x7f1214116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 >> 172.21.15.63:0/3214764095 conn(0x7f12140faba0 msgr2=0x7f12140fbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 shutdown_connections 2024-03-23T12:09:44.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.229+0000 7f1219200700 1 -- 172.21.15.63:0/3214764095 wait complete. 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.365+0000 7faead800700 1 -- 172.21.15.63:0/4006307186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea81037a0 msgr2=0x7faea80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.365+0000 7faead800700 1 --2- 172.21.15.63:0/4006307186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea81037a0 0x7faea80fce10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fae94004940 tx=0x7fae94009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.366+0000 7faead800700 1 -- 172.21.15.63:0/4006307186 shutdown_connections 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.366+0000 7faead800700 1 --2- 172.21.15.63:0/4006307186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea81037a0 0x7faea80fce10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.366+0000 7faead800700 1 --2- 172.21.15.63:0/4006307186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea8102e00 0x7faea81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.366+0000 7faead800700 1 -- 172.21.15.63:0/4006307186 >> 172.21.15.63:0/4006307186 conn(0x7faea80f89b0 msgr2=0x7faea80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.366+0000 7faead800700 1 -- 172.21.15.63:0/4006307186 shutdown_connections 2024-03-23T12:09:44.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.366+0000 7faead800700 1 -- 172.21.15.63:0/4006307186 wait complete. 2024-03-23T12:09:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.366+0000 7faead800700 1 Processor -- start 2024-03-23T12:09:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faead800700 1 -- start start 2024-03-23T12:09:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faead800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea8102e00 0x7faea8115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faead800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea81037a0 0x7faea81162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faead800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea8116930 con 0x7faea8102e00 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faead800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea810fd00 con 0x7faea81037a0 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faea6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea8102e00 0x7faea8115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faea6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea8102e00 0x7faea8115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55978/0 (socket says 172.21.15.63:55978) 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faea6a00700 1 -- 172.21.15.63:0/3689877347 learned_addr learned my addr 172.21.15.63:0/3689877347 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.367+0000 7faea6000700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea81037a0 0x7faea81162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7faea6a00700 1 -- 172.21.15.63:0/3689877347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea81037a0 msgr2=0x7faea81162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7faea6a00700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea81037a0 0x7faea81162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7faea6a00700 1 -- 172.21.15.63:0/3689877347 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94005040 con 0x7faea8102e00 2024-03-23T12:09:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7faea6a00700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea8102e00 0x7faea8115da0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fae98002a50 tx=0x7fae980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7fae9ea00700 1 -- 172.21.15.63:0/3689877347 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae98007cc0 con 0x7faea8102e00 2024-03-23T12:09:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7fae9ea00700 1 -- 172.21.15.63:0/3689877347 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae9800f040 con 0x7faea8102e00 2024-03-23T12:09:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea810ffe0 con 0x7faea8102e00 2024-03-23T12:09:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7fae9ea00700 1 -- 172.21.15.63:0/3689877347 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae9800d9c0 con 0x7faea8102e00 2024-03-23T12:09:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.368+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea8110520 con 0x7faea8102e00 2024-03-23T12:09:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.370+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea80fe550 con 0x7faea8102e00 2024-03-23T12:09:44.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.371+0000 7fae9ea00700 1 -- 172.21.15.63:0/3689877347 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fae9800db20 con 0x7faea8102e00 2024-03-23T12:09:44.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.372+0000 7fae9ea00700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae8c06c7d0 0x7fae8c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:44.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.372+0000 7fae9ea00700 1 -- 172.21.15.63:0/3689877347 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fae9808b760 con 0x7faea8102e00 2024-03-23T12:09:44.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.372+0000 7faea6000700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae8c06c7d0 0x7fae8c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:44.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.373+0000 7faea6000700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae8c06c7d0 0x7fae8c06ec80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faea8111640 tx=0x7fae94009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.377+0000 7fae9ea00700 1 -- 172.21.15.63:0/3689877347 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fae980593f0 con 0x7faea8102e00 2024-03-23T12:09:44.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.678+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faea804bed0 con 0x7faea8102e00 2024-03-23T12:09:44.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.678+0000 7fae9ea00700 1 -- 172.21.15.63:0/3689877347 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7fae98058f80 con 0x7faea8102e00 2024-03-23T12:09:44.659 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:09:44.659 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:09:44.659 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:09:44.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:09:44.660 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae8c06c7d0 msgr2=0x7fae8c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae8c06c7d0 0x7fae8c06ec80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faea8111640 tx=0x7fae94009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea8102e00 msgr2=0x7faea8115da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea8102e00 0x7faea8115da0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fae98002a50 tx=0x7fae980046b0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 shutdown_connections 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fae8c06c7d0 0x7fae8c06ec80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faea8102e00 0x7faea8115da0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 --2- 172.21.15.63:0/3689877347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea81037a0 0x7faea81162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 >> 172.21.15.63:0/3689877347 conn(0x7faea80f89b0 msgr2=0x7faea80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:44.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 shutdown_connections 2024-03-23T12:09:44.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.681+0000 7faead800700 1 -- 172.21.15.63:0/3689877347 wait complete. 2024-03-23T12:09:44.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.819+0000 7f771ac00700 1 -- 172.21.15.63:0/3013088568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7714102e00 msgr2=0x7f77141031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.819+0000 7f771ac00700 1 --2- 172.21.15.63:0/3013088568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7714102e00 0x7f77141031d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7704004940 tx=0x7f7704009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:44.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.819+0000 7f771ac00700 1 -- 172.21.15.63:0/3013088568 shutdown_connections 2024-03-23T12:09:44.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.819+0000 7f771ac00700 1 --2- 172.21.15.63:0/3013088568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77141037a0 0x7f77140fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.819+0000 7f771ac00700 1 --2- 172.21.15.63:0/3013088568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7714102e00 0x7f77141031d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.819+0000 7f771ac00700 1 -- 172.21.15.63:0/3013088568 >> 172.21.15.63:0/3013088568 conn(0x7f77140f89b0 msgr2=0x7f77140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.820+0000 7f771ac00700 1 -- 172.21.15.63:0/3013088568 shutdown_connections 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.820+0000 7f771ac00700 1 -- 172.21.15.63:0/3013088568 wait complete. 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.820+0000 7f771ac00700 1 Processor -- start 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f771ac00700 1 -- start start 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f771ac00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7714102e00 0x7f7714113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f771ac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77141037a0 0x7f7714114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f771ac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141146b0 con 0x7f77141037a0 2024-03-23T12:09:44.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f771ac00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771410daa0 con 0x7f7714102e00 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f7713400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77141037a0 0x7f7714114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f7713400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77141037a0 0x7f7714114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56008/0 (socket says 172.21.15.63:56008) 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f7713400700 1 -- 172.21.15.63:0/4062462357 learned_addr learned my addr 172.21.15.63:0/4062462357 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.821+0000 7f7713e00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7714102e00 0x7f7714113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f7713400700 1 -- 172.21.15.63:0/4062462357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7714102e00 msgr2=0x7f7714113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f7713400700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7714102e00 0x7f7714113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f7713400700 1 -- 172.21.15.63:0/4062462357 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7704005040 con 0x7f77141037a0 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f7713400700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77141037a0 0x7f7714114060 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f770000c890 tx=0x7f77000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:44.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f770be00700 1 -- 172.21.15.63:0/4062462357 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f770000a700 con 0x7f77141037a0 2024-03-23T12:09:44.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f770be00700 1 -- 172.21.15.63:0/4062462357 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f770000ad10 con 0x7f77141037a0 2024-03-23T12:09:44.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f770be00700 1 -- 172.21.15.63:0/4062462357 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7700012e70 con 0x7f77141037a0 2024-03-23T12:09:44.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.822+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771410dd80 con 0x7f77141037a0 2024-03-23T12:09:44.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.823+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771410e2f0 con 0x7f77141037a0 2024-03-23T12:09:44.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.824+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77140fe550 con 0x7f77141037a0 2024-03-23T12:09:44.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.824+0000 7f770be00700 1 -- 172.21.15.63:0/4062462357 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f770000a860 con 0x7f77141037a0 2024-03-23T12:09:44.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.825+0000 7f770be00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f76f806c7d0 0x7f76f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:44.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.825+0000 7f770be00700 1 -- 172.21.15.63:0/4062462357 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f770008bb10 con 0x7f77141037a0 2024-03-23T12:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.826+0000 7f7713e00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f76f806c7d0 0x7f76f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.826+0000 7f7713e00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f76f806c7d0 0x7f76f806ec80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7704009fd0 tx=0x7f7704009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:44.830+0000 7f770be00700 1 -- 172.21.15.63:0/4062462357 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f77000597a0 con 0x7f77141037a0 2024-03-23T12:09:45.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.089+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7714109dc0 con 0x7f77141037a0 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.090+0000 7f770be00700 1 -- 172.21.15.63:0/4062462357 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f7700059330 con 0x7f77141037a0 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:09:45.071 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:09:45.072 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:09:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f76f806c7d0 msgr2=0x7f76f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f76f806c7d0 0x7f76f806ec80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7704009fd0 tx=0x7f7704009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77141037a0 msgr2=0x7f7714114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77141037a0 0x7f7714114060 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f770000c890 tx=0x7f77000044c0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 shutdown_connections 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f76f806c7d0 0x7f76f806ec80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7714102e00 0x7f7714113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 --2- 172.21.15.63:0/4062462357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77141037a0 0x7f7714114060 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 >> 172.21.15.63:0/4062462357 conn(0x7f77140f89b0 msgr2=0x7f77140f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 shutdown_connections 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.092+0000 7f771ac00700 1 -- 172.21.15.63:0/4062462357 wait complete. 2024-03-23T12:09:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.230+0000 7fe192800700 1 -- 172.21.15.63:0/3613300261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c1037a0 msgr2=0x7fe18c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.230+0000 7fe192800700 1 --2- 172.21.15.63:0/3613300261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c1037a0 0x7fe18c0fce10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe178004940 tx=0x7fe178009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.230+0000 7fe192800700 1 -- 172.21.15.63:0/3613300261 shutdown_connections 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.231+0000 7fe192800700 1 --2- 172.21.15.63:0/3613300261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c1037a0 0x7fe18c0fce10 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.231+0000 7fe192800700 1 --2- 172.21.15.63:0/3613300261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c102e00 0x7fe18c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.231+0000 7fe192800700 1 -- 172.21.15.63:0/3613300261 >> 172.21.15.63:0/3613300261 conn(0x7fe18c0f89b0 msgr2=0x7fe18c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.231+0000 7fe192800700 1 -- 172.21.15.63:0/3613300261 shutdown_connections 2024-03-23T12:09:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.231+0000 7fe192800700 1 -- 172.21.15.63:0/3613300261 wait complete. 2024-03-23T12:09:45.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.231+0000 7fe192800700 1 Processor -- start 2024-03-23T12:09:45.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe192800700 1 -- start start 2024-03-23T12:09:45.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe192800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c102e00 0x7fe18c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:45.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe192800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c1037a0 0x7fe18c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:45.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe192800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c116890 con 0x7fe18c102e00 2024-03-23T12:09:45.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe192800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c10fc60 con 0x7fe18c1037a0 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe18b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c102e00 0x7fe18c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe18b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c102e00 0x7fe18c115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56024/0 (socket says 172.21.15.63:56024) 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe18b400700 1 -- 172.21.15.63:0/1540036201 learned_addr learned my addr 172.21.15.63:0/1540036201 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.232+0000 7fe18aa00700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c1037a0 0x7fe18c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe18b400700 1 -- 172.21.15.63:0/1540036201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c1037a0 msgr2=0x7fe18c116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe18b400700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c1037a0 0x7fe18c116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe18b400700 1 -- 172.21.15.63:0/1540036201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178005040 con 0x7fe18c102e00 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe18b400700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c102e00 0x7fe18c115d00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe17c002a50 tx=0x7fe17c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe183400700 1 -- 172.21.15.63:0/1540036201 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe17c007cc0 con 0x7fe18c102e00 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe183400700 1 -- 172.21.15.63:0/1540036201 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe17c00f040 con 0x7fe18c102e00 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe183400700 1 -- 172.21.15.63:0/1540036201 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe17c00d9c0 con 0x7fe18c102e00 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c10ff40 con 0x7fe18c102e00 2024-03-23T12:09:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.233+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c110500 con 0x7fe18c102e00 2024-03-23T12:09:45.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.234+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c0fe550 con 0x7fe18c102e00 2024-03-23T12:09:45.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.235+0000 7fe183400700 1 -- 172.21.15.63:0/1540036201 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe17c00db20 con 0x7fe18c102e00 2024-03-23T12:09:45.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.236+0000 7fe183400700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe17006c7d0 0x7fe17006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:45.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.236+0000 7fe183400700 1 -- 172.21.15.63:0/1540036201 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe17c08b6e0 con 0x7fe18c102e00 2024-03-23T12:09:45.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.237+0000 7fe18aa00700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe17006c7d0 0x7fe17006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:45.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.237+0000 7fe18aa00700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe17006c7d0 0x7fe17006ec80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe18c1115a0 tx=0x7fe178009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:45.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.241+0000 7fe183400700 1 -- 172.21.15.63:0/1540036201 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe17c0593f0 con 0x7fe18c102e00 2024-03-23T12:09:45.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:45 smithi063 ceph-mon[141124]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:45.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:45 smithi063 ceph-mon[141124]: from='client.35054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:45.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:45 smithi063 ceph-mon[141124]: pgmap v558: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:45.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:45 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3689877347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:09:45.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:45 smithi111 ceph-mon[106899]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:45.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:45 smithi111 ceph-mon[106899]: from='client.35054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:45.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:45 smithi111 ceph-mon[106899]: pgmap v558: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:45.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:45 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3689877347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:09:45.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.473+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe18c04bf40 con 0x7fe17006c7d0 2024-03-23T12:09:45.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.475+0000 7fe183400700 1 -- 172.21.15.63:0/1540036201 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe18c04bf40 con 0x7fe17006c7d0 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:09:45.456 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:09:45.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe17006c7d0 msgr2=0x7fe17006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe17006c7d0 0x7fe17006ec80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe18c1115a0 tx=0x7fe178009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:45.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c102e00 msgr2=0x7fe18c115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c102e00 0x7fe18c115d00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe17c002a50 tx=0x7fe17c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 shutdown_connections 2024-03-23T12:09:45.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe17006c7d0 0x7fe17006ec80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18c102e00 0x7fe18c115d00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 --2- 172.21.15.63:0/1540036201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18c1037a0 0x7fe18c116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 >> 172.21.15.63:0/1540036201 conn(0x7fe18c0f89b0 msgr2=0x7fe18c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:45.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 shutdown_connections 2024-03-23T12:09:45.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.477+0000 7fe192800700 1 -- 172.21.15.63:0/1540036201 wait complete. 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 -- 172.21.15.63:0/2839599324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06941037c0 msgr2=0x7f06940fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 --2- 172.21.15.63:0/2839599324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06941037c0 0x7f06940fce30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0680004940 tx=0x7f0680009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 -- 172.21.15.63:0/2839599324 shutdown_connections 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 --2- 172.21.15.63:0/2839599324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06941037c0 0x7f06940fce30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 --2- 172.21.15.63:0/2839599324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0694102e20 0x7f06941031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 -- 172.21.15.63:0/2839599324 >> 172.21.15.63:0/2839599324 conn(0x7f06940f89d0 msgr2=0x7f06940fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 -- 172.21.15.63:0/2839599324 shutdown_connections 2024-03-23T12:09:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.615+0000 7f069b200700 1 -- 172.21.15.63:0/2839599324 wait complete. 2024-03-23T12:09:45.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f069b200700 1 Processor -- start 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f069b200700 1 -- start start 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f069b200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0694102e20 0x7f0694115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f069b200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06941037c0 0x7f0694116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f069b200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0694116880 con 0x7f0694102e20 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f069b200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069410fc50 con 0x7f06941037c0 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f0693e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0694102e20 0x7f0694115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f0693e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0694102e20 0x7f0694115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56036/0 (socket says 172.21.15.63:56036) 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.616+0000 7f0693e00700 1 -- 172.21.15.63:0/2887299536 learned_addr learned my addr 172.21.15.63:0/2887299536 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f0693400700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06941037c0 0x7f0694116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f0693e00700 1 -- 172.21.15.63:0/2887299536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06941037c0 msgr2=0x7f0694116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f0693e00700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06941037c0 0x7f0694116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f0693e00700 1 -- 172.21.15.63:0/2887299536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680005040 con 0x7f0694102e20 2024-03-23T12:09:45.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f0693400700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06941037c0 0x7f0694116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:09:45.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f0693e00700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0694102e20 0x7f0694115cf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0684002a50 tx=0x7f06840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:45.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f068be00700 1 -- 172.21.15.63:0/2887299536 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0684007cc0 con 0x7f0694102e20 2024-03-23T12:09:45.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f068be00700 1 -- 172.21.15.63:0/2887299536 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f068400f040 con 0x7f0694102e20 2024-03-23T12:09:45.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069410ff30 con 0x7f0694102e20 2024-03-23T12:09:45.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.617+0000 7f068be00700 1 -- 172.21.15.63:0/2887299536 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0684009540 con 0x7f0694102e20 2024-03-23T12:09:45.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.618+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06941104f0 con 0x7f0694102e20 2024-03-23T12:09:45.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.619+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06940fe570 con 0x7f0694102e20 2024-03-23T12:09:45.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.620+0000 7f068be00700 1 -- 172.21.15.63:0/2887299536 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f06840096a0 con 0x7f0694102e20 2024-03-23T12:09:45.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.620+0000 7f068be00700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f067806c7d0 0x7f067806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:09:45.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.621+0000 7f068be00700 1 -- 172.21.15.63:0/2887299536 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f068408b820 con 0x7f0694102e20 2024-03-23T12:09:45.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.621+0000 7f0693400700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f067806c7d0 0x7f067806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:09:45.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.621+0000 7f0693400700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f067806c7d0 0x7f067806ec80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0694111590 tx=0x7f0680009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:09:45.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.625+0000 7f068be00700 1 -- 172.21.15.63:0/2887299536 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f06840594b0 con 0x7f0694102e20 2024-03-23T12:09:45.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.920+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f069404bed0 con 0x7f0694102e20 2024-03-23T12:09:45.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.920+0000 7f068be00700 1 -- 172.21.15.63:0/2887299536 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f0684059040 con 0x7f0694102e20 2024-03-23T12:09:45.901 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:09:45.902 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:09:45.902 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:09:45.902 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:09:45.902 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f067806c7d0 msgr2=0x7f067806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f067806c7d0 0x7f067806ec80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0694111590 tx=0x7f0680009e30 comp rx=0 tx=0).stop 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0694102e20 msgr2=0x7f0694115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0694102e20 0x7f0694115cf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0684002a50 tx=0x7f06840046b0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 shutdown_connections 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f067806c7d0 0x7f067806ec80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0694102e20 0x7f0694115cf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 --2- 172.21.15.63:0/2887299536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06941037c0 0x7f0694116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 >> 172.21.15.63:0/2887299536 conn(0x7f06940f89d0 msgr2=0x7f06940706c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 shutdown_connections 2024-03-23T12:09:45.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:09:45.923+0000 7f069b200700 1 -- 172.21.15.63:0/2887299536 wait complete. 2024-03-23T12:09:46.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:46 smithi063 ceph-mon[141124]: from='client.35058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:46.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:46 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4062462357' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:09:46.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:46 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2887299536' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:09:46.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:46 smithi111 ceph-mon[106899]: from='client.35058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:46.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:46 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4062462357' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:09:46.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:46 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2887299536' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:09:47.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:47 smithi063 ceph-mon[141124]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:47.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:47 smithi063 ceph-mon[141124]: pgmap v559: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:47.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:47 smithi111 ceph-mon[106899]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:09:47.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:47 smithi111 ceph-mon[106899]: pgmap v559: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:48.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:48 smithi063 ceph-mon[141124]: pgmap v560: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:48.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:48 smithi111 ceph-mon[106899]: pgmap v560: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:51.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:51 smithi063 ceph-mon[141124]: pgmap v561: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:51.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:51 smithi111 ceph-mon[106899]: pgmap v561: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:53.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:53 smithi063 ceph-mon[141124]: pgmap v562: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:53.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:53 smithi111 ceph-mon[106899]: pgmap v562: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:54.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:54 smithi063 ceph-mon[141124]: pgmap v563: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:54.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:54 smithi111 ceph-mon[106899]: pgmap v563: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:56 smithi063 ceph-mon[141124]: pgmap v564: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:09:56.900 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:56 smithi111 ceph-mon[106899]: pgmap v564: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:59.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:09:59 smithi063 ceph-mon[141124]: pgmap v565: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:09:59.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:09:59 smithi111 ceph-mon[106899]: pgmap v565: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:09:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:10:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:00 smithi063 ceph-mon[141124]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:10:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:00 smithi111 ceph-mon[106899]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:10:01.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:01 smithi063 ceph-mon[141124]: pgmap v566: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:01.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:01 smithi111 ceph-mon[106899]: pgmap v566: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:02.354 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:02 smithi111 ceph-mon[106899]: pgmap v567: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:02.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:02 smithi063 ceph-mon[141124]: pgmap v567: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:05.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:05 smithi063 ceph-mon[141124]: pgmap v568: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:05.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:05 smithi111 ceph-mon[106899]: pgmap v568: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:06.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:06 smithi063 ceph-mon[141124]: pgmap v569: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:06.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:06 smithi111 ceph-mon[106899]: pgmap v569: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:09.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:09 smithi063 ceph-mon[141124]: pgmap v570: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:09.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:09 smithi111 ceph-mon[106899]: pgmap v570: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:10.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:10 smithi063 ceph-mon[141124]: pgmap v571: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:10.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:10 smithi111 ceph-mon[106899]: pgmap v571: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:11.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:11.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:12.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:12 smithi063 ceph-mon[141124]: pgmap v572: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:12.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:12 smithi111 ceph-mon[106899]: pgmap v572: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:15.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:15 smithi063 ceph-mon[141124]: pgmap v573: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:15.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:15 smithi111 ceph-mon[106899]: pgmap v573: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:16.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 -- 172.21.15.63:0/1585982490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 msgr2=0x7f61b41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 --2- 172.21.15.63:0/1585982490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b41031d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f61a4004940 tx=0x7f61a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:16.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 -- 172.21.15.63:0/1585982490 shutdown_connections 2024-03-23T12:10:16.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 --2- 172.21.15.63:0/1585982490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41037a0 0x7f61b40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 --2- 172.21.15.63:0/1585982490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b41031d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 -- 172.21.15.63:0/1585982490 >> 172.21.15.63:0/1585982490 conn(0x7f61b40f89b0 msgr2=0x7f61b40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:16.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 -- 172.21.15.63:0/1585982490 shutdown_connections 2024-03-23T12:10:16.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.079+0000 7f61ba000700 1 -- 172.21.15.63:0/1585982490 wait complete. 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.080+0000 7f61ba000700 1 Processor -- start 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.080+0000 7f61ba000700 1 -- start start 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.080+0000 7f61ba000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b4115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.080+0000 7f61ba000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41037a0 0x7f61b4116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.080+0000 7f61ba000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41167d0 con 0x7f61b4102e00 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.080+0000 7f61ba000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b410fba0 con 0x7f61b41037a0 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61b3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b4115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61b3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b4115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36658/0 (socket says 172.21.15.63:36658) 2024-03-23T12:10:16.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61b3400700 1 -- 172.21.15.63:0/3428512927 learned_addr learned my addr 172.21.15.63:0/3428512927 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:16.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61abe00700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41037a0 0x7f61b4116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61b3400700 1 -- 172.21.15.63:0/3428512927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41037a0 msgr2=0x7f61b4116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61b3400700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41037a0 0x7f61b4116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61b3400700 1 -- 172.21.15.63:0/3428512927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a4005040 con 0x7f61b4102e00 2024-03-23T12:10:16.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.081+0000 7f61b3400700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b4115c40 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009180 tx=0x7f61a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:16.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.082+0000 7f61ab400700 1 -- 172.21.15.63:0/3428512927 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a400e040 con 0x7f61b4102e00 2024-03-23T12:10:16.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.082+0000 7f61ab400700 1 -- 172.21.15.63:0/3428512927 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61a4004290 con 0x7f61b4102e00 2024-03-23T12:10:16.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.082+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b410fe20 con 0x7f61b4102e00 2024-03-23T12:10:16.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.082+0000 7f61ab400700 1 -- 172.21.15.63:0/3428512927 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a4013e60 con 0x7f61b4102e00 2024-03-23T12:10:16.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.082+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b41103e0 con 0x7f61b4102e00 2024-03-23T12:10:16.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.083+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b40fe550 con 0x7f61b4102e00 2024-03-23T12:10:16.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.084+0000 7f61ab400700 1 -- 172.21.15.63:0/3428512927 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f61a40136b0 con 0x7f61b4102e00 2024-03-23T12:10:16.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.085+0000 7f61ab400700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f619806c750 0x7f619806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.085+0000 7f61ab400700 1 -- 172.21.15.63:0/3428512927 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f61a408ccd0 con 0x7f61b4102e00 2024-03-23T12:10:16.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.085+0000 7f61abe00700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f619806c750 0x7f619806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.086+0000 7f61abe00700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f619806c750 0x7f619806ec00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f61b41114e0 tx=0x7f619c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:16.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.090+0000 7f61ab400700 1 -- 172.21.15.63:0/3428512927 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61a405a9e0 con 0x7f61b4102e00 2024-03-23T12:10:16.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.323+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f61b40fe550 con 0x7f619806c750 2024-03-23T12:10:16.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.324+0000 7f61ab400700 1 -- 172.21.15.63:0/3428512927 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f61b40fe550 con 0x7f619806c750 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f619806c750 msgr2=0x7f619806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f619806c750 0x7f619806ec00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f61b41114e0 tx=0x7f619c00b040 comp rx=0 tx=0).stop 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 msgr2=0x7f61b4115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b4115c40 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009180 tx=0x7f61a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 shutdown_connections 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f619806c750 0x7f619806ec00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61b4102e00 0x7f61b4115c40 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 --2- 172.21.15.63:0/3428512927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61b41037a0 0x7f61b4116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 >> 172.21.15.63:0/3428512927 conn(0x7f61b40f89b0 msgr2=0x7f61b40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 shutdown_connections 2024-03-23T12:10:16.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.327+0000 7f61ba000700 1 -- 172.21.15.63:0/3428512927 wait complete. 2024-03-23T12:10:16.321 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.464+0000 7f2f46200700 1 -- 172.21.15.63:0/1589550335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 msgr2=0x7f2f400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.464+0000 7f2f46200700 1 --2- 172.21.15.63:0/1589550335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f400fce10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c004940 tx=0x7f2f2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.465+0000 7f2f46200700 1 -- 172.21.15.63:0/1589550335 shutdown_connections 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.465+0000 7f2f46200700 1 --2- 172.21.15.63:0/1589550335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f400fce10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.465+0000 7f2f46200700 1 --2- 172.21.15.63:0/1589550335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.465+0000 7f2f46200700 1 -- 172.21.15.63:0/1589550335 >> 172.21.15.63:0/1589550335 conn(0x7f2f400f89b0 msgr2=0x7f2f400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.465+0000 7f2f46200700 1 -- 172.21.15.63:0/1589550335 shutdown_connections 2024-03-23T12:10:16.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.465+0000 7f2f46200700 1 -- 172.21.15.63:0/1589550335 wait complete. 2024-03-23T12:10:16.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.465+0000 7f2f46200700 1 Processor -- start 2024-03-23T12:10:16.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f46200700 1 -- start start 2024-03-23T12:10:16.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f46200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f46200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f46200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f401167f0 con 0x7f2f401037a0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f46200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4010fbc0 con 0x7f2f40102e00 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f3ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f3ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36684/0 (socket says 172.21.15.63:36684) 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.466+0000 7f2f3ea00700 1 -- 172.21.15.63:0/3150025184 learned_addr learned my addr 172.21.15.63:0/3150025184 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f3ea00700 1 -- 172.21.15.63:0/3150025184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 msgr2=0x7f2f40115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f3ea00700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f3ea00700 1 -- 172.21.15.63:0/3150025184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c005040 con 0x7f2f401037a0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f3ea00700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401161a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c008d50 tx=0x7f2f2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f37400700 1 -- 172.21.15.63:0/3150025184 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2c00e030 con 0x7f2f401037a0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f37400700 1 -- 172.21.15.63:0/3150025184 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f2c013de0 con 0x7f2f401037a0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4010fe40 con 0x7f2f401037a0 2024-03-23T12:10:16.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f37400700 1 -- 172.21.15.63:0/3150025184 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2c01cd90 con 0x7f2f401037a0 2024-03-23T12:10:16.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.467+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f40110400 con 0x7f2f401037a0 2024-03-23T12:10:16.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.468+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f400fe550 con 0x7f2f401037a0 2024-03-23T12:10:16.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.470+0000 7f2f37400700 1 -- 172.21.15.63:0/3150025184 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2f2c013930 con 0x7f2f401037a0 2024-03-23T12:10:16.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.470+0000 7f2f37400700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.471+0000 7f2f37400700 1 -- 172.21.15.63:0/3150025184 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f2c08d300 con 0x7f2f401037a0 2024-03-23T12:10:16.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.471+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.471+0000 7f2f3f400700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2f30004600 tx=0x7f2f3000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:16.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.475+0000 7f2f37400700 1 -- 172.21.15.63:0/3150025184 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f2c05b010 con 0x7f2f401037a0 2024-03-23T12:10:16.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.705+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f40109e10 con 0x7f2f2406c7d0 2024-03-23T12:10:16.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.706+0000 7f2f37400700 1 -- 172.21.15.63:0/3150025184 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2f40109e10 con 0x7f2f2406c7d0 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 msgr2=0x7f2f2406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2f30004600 tx=0x7f2f3000b040 comp rx=0 tx=0).stop 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 msgr2=0x7f2f401161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401161a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c008d50 tx=0x7f2f2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 shutdown_connections 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2406c7d0 0x7f2f2406ec80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f40102e00 0x7f2f40115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 --2- 172.21.15.63:0/3150025184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f401037a0 0x7f2f401161a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 >> 172.21.15.63:0/3150025184 conn(0x7f2f400f89b0 msgr2=0x7f2f400f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 shutdown_connections 2024-03-23T12:10:16.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.709+0000 7f2f46200700 1 -- 172.21.15.63:0/3150025184 wait complete. 2024-03-23T12:10:16.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 -- 172.21.15.63:0/3764850379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fde90 msgr2=0x7ff0300fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 --2- 172.21.15.63:0/3764850379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fde90 0x7ff0300fe260 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff024004940 tx=0x7ff024009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:16.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 -- 172.21.15.63:0/3764850379 shutdown_connections 2024-03-23T12:10:16.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 --2- 172.21.15.63:0/3764850379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0300fe830 0x7ff030106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 --2- 172.21.15.63:0/3764850379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fde90 0x7ff0300fe260 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 -- 172.21.15.63:0/3764850379 >> 172.21.15.63:0/3764850379 conn(0x7ff030072bc0 msgr2=0x7ff030072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:16.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 -- 172.21.15.63:0/3764850379 shutdown_connections 2024-03-23T12:10:16.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.846+0000 7ff037400700 1 -- 172.21.15.63:0/3764850379 wait complete. 2024-03-23T12:10:16.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.847+0000 7ff037400700 1 Processor -- start 2024-03-23T12:10:16.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.847+0000 7ff037400700 1 -- start start 2024-03-23T12:10:16.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.847+0000 7ff037400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0300fde90 0x7ff030115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.847+0000 7ff037400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fe830 0x7ff030116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.847+0000 7ff037400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0301168b0 con 0x7ff0300fe830 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.847+0000 7ff037400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03010fc80 con 0x7ff0300fde90 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fe830 0x7ff030116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fe830 0x7ff030116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36714/0 (socket says 172.21.15.63:36714) 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02fe00700 1 -- 172.21.15.63:0/1230774627 learned_addr learned my addr 172.21.15.63:0/1230774627 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff034a00700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0300fde90 0x7ff030115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02fe00700 1 -- 172.21.15.63:0/1230774627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0300fde90 msgr2=0x7ff030115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02fe00700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0300fde90 0x7ff030115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02fe00700 1 -- 172.21.15.63:0/1230774627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024005040 con 0x7ff0300fe830 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02fe00700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fe830 0x7ff030116260 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff01c002a50 tx=0x7ff01c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02cc00700 1 -- 172.21.15.63:0/1230774627 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff01c007cc0 con 0x7ff0300fe830 2024-03-23T12:10:16.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02cc00700 1 -- 172.21.15.63:0/1230774627 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff01c00f040 con 0x7ff0300fe830 2024-03-23T12:10:16.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.848+0000 7ff02cc00700 1 -- 172.21.15.63:0/1230774627 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff01c009540 con 0x7ff0300fe830 2024-03-23T12:10:16.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.849+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03010ff60 con 0x7ff0300fe830 2024-03-23T12:10:16.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.849+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0301104a0 con 0x7ff0300fe830 2024-03-23T12:10:16.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.850+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff030106a20 con 0x7ff0300fe830 2024-03-23T12:10:16.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.851+0000 7ff02cc00700 1 -- 172.21.15.63:0/1230774627 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff01c0096a0 con 0x7ff0300fe830 2024-03-23T12:10:16.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.852+0000 7ff02cc00700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff01806c7d0 0x7ff01806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:16.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.852+0000 7ff02cc00700 1 -- 172.21.15.63:0/1230774627 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff01c08b7a0 con 0x7ff0300fe830 2024-03-23T12:10:16.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.852+0000 7ff034a00700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff01806c7d0 0x7ff01806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:16.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.853+0000 7ff034a00700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff01806c7d0 0x7ff01806ec80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff02400b3b0 tx=0x7ff024009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:16.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:16.857+0000 7ff02cc00700 1 -- 172.21.15.63:0/1230774627 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff01c0594b0 con 0x7ff0300fe830 2024-03-23T12:10:16.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:16 smithi063 ceph-mon[141124]: pgmap v574: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:16.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:16 smithi111 ceph-mon[106899]: pgmap v574: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:17.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.082+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff030109dc0 con 0x7ff01806c7d0 2024-03-23T12:10:17.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.092+0000 7ff02cc00700 1 -- 172.21.15.63:0/1230774627 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7ff030109dc0 con 0x7ff01806c7d0 2024-03-23T12:10:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:10:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (19m) 12m ago 32m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:10:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (16m) 12m ago 32m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:10:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (16m) 11m ago 31m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:10:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (19m) 12m ago 32m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:10:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (12m) 12m ago 26m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:10:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (12m) 12m ago 26m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (26m) 11m ago 26m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (12m) 11m ago 26m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (21m) 12m ago 34m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (20m) 11m ago 31m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (18m) 12m ago 34m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (17m) 11m ago 31m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (20m) 12m ago 32m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (20m) 11m ago 31m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (16m) 12m ago 30m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (15m) 12m ago 30m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (14m) 12m ago 29m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (14m) 11m ago 29m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:10:17.073 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (13m) 11m ago 28m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:10:17.074 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (13m) 11m ago 28m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:10:17.074 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (19m) 12m ago 32m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff01806c7d0 msgr2=0x7ff01806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff01806c7d0 0x7ff01806ec80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff02400b3b0 tx=0x7ff024009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fe830 msgr2=0x7ff030116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fe830 0x7ff030116260 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff01c002a50 tx=0x7ff01c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 shutdown_connections 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff01806c7d0 0x7ff01806ec80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0300fde90 0x7ff030115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 --2- 172.21.15.63:0/1230774627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0300fe830 0x7ff030116260 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.094+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 >> 172.21.15.63:0/1230774627 conn(0x7ff030072bc0 msgr2=0x7ff0300fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.095+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 shutdown_connections 2024-03-23T12:10:17.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.095+0000 7ff037400700 1 -- 172.21.15.63:0/1230774627 wait complete. 2024-03-23T12:10:17.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 -- 172.21.15.63:0/1574774756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c1037a0 msgr2=0x7f698c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 --2- 172.21.15.63:0/1574774756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c1037a0 0x7f698c0fce10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6978004940 tx=0x7f6978009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:17.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 -- 172.21.15.63:0/1574774756 shutdown_connections 2024-03-23T12:10:17.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 --2- 172.21.15.63:0/1574774756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c1037a0 0x7f698c0fce10 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 --2- 172.21.15.63:0/1574774756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f698c102e00 0x7f698c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 -- 172.21.15.63:0/1574774756 >> 172.21.15.63:0/1574774756 conn(0x7f698c0f89b0 msgr2=0x7f698c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:17.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 -- 172.21.15.63:0/1574774756 shutdown_connections 2024-03-23T12:10:17.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.231+0000 7f6993e00700 1 -- 172.21.15.63:0/1574774756 wait complete. 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.232+0000 7f6993e00700 1 Processor -- start 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.232+0000 7f6993e00700 1 -- start start 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6993e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c102e00 0x7f698c115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6993e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f698c1037a0 0x7f698c115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6993e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c1165b0 con 0x7f698c102e00 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6993e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c10fb90 con 0x7f698c1037a0 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6991400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c102e00 0x7f698c115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6991400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c102e00 0x7f698c115a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36726/0 (socket says 172.21.15.63:36726) 2024-03-23T12:10:17.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6991400700 1 -- 172.21.15.63:0/673258300 learned_addr learned my addr 172.21.15.63:0/673258300 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6991400700 1 -- 172.21.15.63:0/673258300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f698c1037a0 msgr2=0x7f698c115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6990a00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f698c1037a0 0x7f698c115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6991400700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f698c1037a0 0x7f698c115f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6991400700 1 -- 172.21.15.63:0/673258300 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6978005040 con 0x7f698c102e00 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.233+0000 7f6990a00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f698c1037a0 0x7f698c115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.234+0000 7f6991400700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c102e00 0x7f698c115a20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6980002a80 tx=0x7f69800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.234+0000 7f6989600700 1 -- 172.21.15.63:0/673258300 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6980007cc0 con 0x7f698c102e00 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.234+0000 7f6989600700 1 -- 172.21.15.63:0/673258300 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f698000f040 con 0x7f698c102e00 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.234+0000 7f6989600700 1 -- 172.21.15.63:0/673258300 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6980009640 con 0x7f698c102e00 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.234+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c10fe70 con 0x7f698c102e00 2024-03-23T12:10:17.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.234+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c110430 con 0x7f698c102e00 2024-03-23T12:10:17.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.235+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c0fe550 con 0x7f698c102e00 2024-03-23T12:10:17.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.236+0000 7f6989600700 1 -- 172.21.15.63:0/673258300 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f69800097a0 con 0x7f698c102e00 2024-03-23T12:10:17.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.237+0000 7f6989600700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f697406c7c0 0x7f697406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:17.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.237+0000 7f6989600700 1 -- 172.21.15.63:0/673258300 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f698008b220 con 0x7f698c102e00 2024-03-23T12:10:17.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.238+0000 7f6990a00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f697406c7c0 0x7f697406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:17.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.238+0000 7f6990a00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f697406c7c0 0x7f697406ec70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f697800b3b0 tx=0x7f6978009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:17.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.242+0000 7f6989600700 1 -- 172.21.15.63:0/673258300 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6980058eb0 con 0x7f698c102e00 2024-03-23T12:10:17.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.538+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f698c111070 con 0x7f698c102e00 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.538+0000 7f6989600700 1 -- 172.21.15.63:0/673258300 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f6980058a40 con 0x7f698c102e00 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:10:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:10:17.520 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f697406c7c0 msgr2=0x7f697406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f697406c7c0 0x7f697406ec70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f697800b3b0 tx=0x7f6978009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c102e00 msgr2=0x7f698c115a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c102e00 0x7f698c115a20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6980002a80 tx=0x7f69800046b0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 shutdown_connections 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f697406c7c0 0x7f697406ec70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f698c102e00 0x7f698c115a20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 --2- 172.21.15.63:0/673258300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f698c1037a0 0x7f698c115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 >> 172.21.15.63:0/673258300 conn(0x7f698c0f89b0 msgr2=0x7f698c071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:17.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 shutdown_connections 2024-03-23T12:10:17.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.541+0000 7f6993e00700 1 -- 172.21.15.63:0/673258300 wait complete. 2024-03-23T12:10:17.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 -- 172.21.15.63:0/1200512958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 msgr2=0x7fbbbc0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1200512958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc0fcc50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbba8004940 tx=0x7fbba8009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 -- 172.21.15.63:0/1200512958 shutdown_connections 2024-03-23T12:10:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1200512958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc0fcc50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1200512958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbbc102c40 0x7fbbbc103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 -- 172.21.15.63:0/1200512958 >> 172.21.15.63:0/1200512958 conn(0x7fbbbc0f8830 msgr2=0x7fbbbc0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 -- 172.21.15.63:0/1200512958 shutdown_connections 2024-03-23T12:10:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.678+0000 7fbbc1200700 1 -- 172.21.15.63:0/1200512958 wait complete. 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.679+0000 7fbbc1200700 1 Processor -- start 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.679+0000 7fbbc1200700 1 -- start start 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbc1200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbbc102c40 0x7fbbbc115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbc1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbc1200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc116670 con 0x7fbbbc1035e0 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbc1200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc10fa60 con 0x7fbbbc102c40 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbbaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbbaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc116020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36742/0 (socket says 172.21.15.63:36742) 2024-03-23T12:10:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbbaa00700 1 -- 172.21.15.63:0/1970728033 learned_addr learned my addr 172.21.15.63:0/1970728033 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.680+0000 7fbbbb400700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbbc102c40 0x7fbbbc115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbbaa00700 1 -- 172.21.15.63:0/1970728033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbbc102c40 msgr2=0x7fbbbc115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbbaa00700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbbc102c40 0x7fbbbc115ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbbaa00700 1 -- 172.21.15.63:0/1970728033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba8005040 con 0x7fbbbc1035e0 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbbaa00700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc116020 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbba80093b0 tx=0x7fbba8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbb3400700 1 -- 172.21.15.63:0/1970728033 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba800e040 con 0x7fbbbc1035e0 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbb3400700 1 -- 172.21.15.63:0/1970728033 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbba8013420 con 0x7fbbbc1035e0 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc10fce0 con 0x7fbbbc1035e0 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbb3400700 1 -- 172.21.15.63:0/1970728033 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba801c630 con 0x7fbbbc1035e0 2024-03-23T12:10:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.681+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc1102a0 con 0x7fbbbc1035e0 2024-03-23T12:10:17.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.682+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc0fe390 con 0x7fbbbc1035e0 2024-03-23T12:10:17.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.683+0000 7fbbb3400700 1 -- 172.21.15.63:0/1970728033 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbba80139e0 con 0x7fbbbc1035e0 2024-03-23T12:10:17.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.684+0000 7fbbb3400700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbba006c7c0 0x7fbba006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:17.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.684+0000 7fbbb3400700 1 -- 172.21.15.63:0/1970728033 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbba808cc40 con 0x7fbbbc1035e0 2024-03-23T12:10:17.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.684+0000 7fbbbb400700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbba006c7c0 0x7fbba006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:17.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.685+0000 7fbbbb400700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbba006c7c0 0x7fbba006ec70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbbac004f60 tx=0x7fbbac00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:17.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.689+0000 7fbbb3400700 1 -- 172.21.15.63:0/1970728033 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbba805a980 con 0x7fbbbc1035e0 2024-03-23T12:10:17.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:17 smithi063 ceph-mon[141124]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:17.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:17 smithi063 ceph-mon[141124]: from='client.35082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:17.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:17 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/673258300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:10:17.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:17 smithi111 ceph-mon[106899]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:17.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:17 smithi111 ceph-mon[106899]: from='client.35082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:17.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:17 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/673258300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:10:17.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.948+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbbbc109bf0 con 0x7fbbbc1035e0 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.949+0000 7fbbb3400700 1 -- 172.21.15.63:0/1970728033 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fbba805a510 con 0x7fbbbc1035e0 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:10:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:10:17.930 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:17.931 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:10:17.932 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:17.932 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:17.932 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:17.932 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:17.932 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:17.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.951+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbba006c7c0 msgr2=0x7fbba006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.951+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbba006c7c0 0x7fbba006ec70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbbac004f60 tx=0x7fbbac00b040 comp rx=0 tx=0).stop 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.951+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 msgr2=0x7fbbbc116020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.951+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc116020 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbba80093b0 tx=0x7fbba8009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.951+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 shutdown_connections 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.951+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbba006c7c0 0x7fbba006ec70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.951+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbbc102c40 0x7fbbbc115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.952+0000 7fbbc1200700 1 --2- 172.21.15.63:0/1970728033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbbc1035e0 0x7fbbbc116020 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.952+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 >> 172.21.15.63:0/1970728033 conn(0x7fbbbc0f8830 msgr2=0x7fbbbc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.952+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 shutdown_connections 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:17.952+0000 7fbbc1200700 1 -- 172.21.15.63:0/1970728033 wait complete. 2024-03-23T12:10:17.933 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:10:18.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 -- 172.21.15.63:0/4027969021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf0102e00 msgr2=0x7fecf01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4027969021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf0102e00 0x7fecf01031d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fece0004940 tx=0x7fece0009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:18.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 -- 172.21.15.63:0/4027969021 shutdown_connections 2024-03-23T12:10:18.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4027969021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf01037a0 0x7fecf00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4027969021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf0102e00 0x7fecf01031d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 -- 172.21.15.63:0/4027969021 >> 172.21.15.63:0/4027969021 conn(0x7fecf00f89b0 msgr2=0x7fecf00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:18.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 -- 172.21.15.63:0/4027969021 shutdown_connections 2024-03-23T12:10:18.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.090+0000 7fecf4a00700 1 -- 172.21.15.63:0/4027969021 wait complete. 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.091+0000 7fecf4a00700 1 Processor -- start 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.091+0000 7fecf4a00700 1 -- start start 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7fecf4a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf0102e00 0x7fecf0115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7fecf4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf01037a0 0x7fecf0116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7fecf4a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf01167e0 con 0x7fecf01037a0 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7fecf4a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf010fbb0 con 0x7fecf0102e00 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7feced400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf01037a0 0x7fecf0116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7feced400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf01037a0 0x7fecf0116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36756/0 (socket says 172.21.15.63:36756) 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7feced400700 1 -- 172.21.15.63:0/4041341477 learned_addr learned my addr 172.21.15.63:0/4041341477 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7fecede00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf0102e00 0x7fecf0115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:18.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7feced400700 1 -- 172.21.15.63:0/4041341477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf0102e00 msgr2=0x7fecf0115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7feced400700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf0102e00 0x7fecf0115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.092+0000 7feced400700 1 -- 172.21.15.63:0/4041341477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0005040 con 0x7fecf01037a0 2024-03-23T12:10:18.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.093+0000 7feced400700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf01037a0 0x7fecf0116190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00eb40 tx=0x7fecdc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:18.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.093+0000 7fece6000700 1 -- 172.21.15.63:0/4041341477 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecdc007cc0 con 0x7fecf01037a0 2024-03-23T12:10:18.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.093+0000 7fece6000700 1 -- 172.21.15.63:0/4041341477 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecdc00f040 con 0x7fecf01037a0 2024-03-23T12:10:18.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.093+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf010fe30 con 0x7fecf01037a0 2024-03-23T12:10:18.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.093+0000 7fece6000700 1 -- 172.21.15.63:0/4041341477 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecdc00c8f0 con 0x7fecf01037a0 2024-03-23T12:10:18.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.093+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf01103f0 con 0x7fecf01037a0 2024-03-23T12:10:18.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.094+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf00fe550 con 0x7fecf01037a0 2024-03-23T12:10:18.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.095+0000 7fece6000700 1 -- 172.21.15.63:0/4041341477 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fecdc0090d0 con 0x7fecf01037a0 2024-03-23T12:10:18.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.096+0000 7fece6000700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fecd406c7d0 0x7fecd406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:18.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.096+0000 7fece6000700 1 -- 172.21.15.63:0/4041341477 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fecdc08b410 con 0x7fecf01037a0 2024-03-23T12:10:18.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.096+0000 7fecede00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fecd406c7d0 0x7fecd406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:18.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.097+0000 7fecede00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fecd406c7d0 0x7fecd406ec80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fece00045a0 tx=0x7fece0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:18.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.101+0000 7fece6000700 1 -- 172.21.15.63:0/4041341477 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fecdc0591d0 con 0x7fecf01037a0 2024-03-23T12:10:18.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.332+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fecf0109e30 con 0x7fecd406c7d0 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.333+0000 7fece6000700 1 -- 172.21.15.63:0/4041341477 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fecf0109e30 con 0x7fecd406c7d0 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:10:18.313 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:10:18.314 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:10:18.314 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:10:18.314 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:10:18.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fecd406c7d0 msgr2=0x7fecd406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fecd406c7d0 0x7fecd406ec80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fece00045a0 tx=0x7fece0009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:18.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf01037a0 msgr2=0x7fecf0116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf01037a0 0x7fecf0116190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00eb40 tx=0x7fecdc00b040 comp rx=0 tx=0).stop 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 shutdown_connections 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fecd406c7d0 0x7fecd406ec80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf0102e00 0x7fecf0115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 --2- 172.21.15.63:0/4041341477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fecf01037a0 0x7fecf0116190 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.335+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 >> 172.21.15.63:0/4041341477 conn(0x7fecf00f89b0 msgr2=0x7fecf00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.336+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 shutdown_connections 2024-03-23T12:10:18.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.336+0000 7fecf4a00700 1 -- 172.21.15.63:0/4041341477 wait complete. 2024-03-23T12:10:18.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 -- 172.21.15.63:0/715579988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 msgr2=0x7fd0e00fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 --2- 172.21.15.63:0/715579988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e00fce30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc004940 tx=0x7fd0cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:18.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 -- 172.21.15.63:0/715579988 shutdown_connections 2024-03-23T12:10:18.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 --2- 172.21.15.63:0/715579988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e00fce30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 --2- 172.21.15.63:0/715579988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0102e20 0x7fd0e01031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 -- 172.21.15.63:0/715579988 >> 172.21.15.63:0/715579988 conn(0x7fd0e00f89d0 msgr2=0x7fd0e00fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:18.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 -- 172.21.15.63:0/715579988 shutdown_connections 2024-03-23T12:10:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.474+0000 7fd0e7200700 1 -- 172.21.15.63:0/715579988 wait complete. 2024-03-23T12:10:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.475+0000 7fd0e7200700 1 Processor -- start 2024-03-23T12:10:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.475+0000 7fd0e7200700 1 -- start start 2024-03-23T12:10:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0e7200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0102e20 0x7fd0e0115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0e7200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e0116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0e7200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e01167e0 con 0x7fd0e01037c0 2024-03-23T12:10:18.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0e7200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e010fbb0 con 0x7fd0e0102e20 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0df400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e0116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0df400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e0116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36780/0 (socket says 172.21.15.63:36780) 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0df400700 1 -- 172.21.15.63:0/2409996447 learned_addr learned my addr 172.21.15.63:0/2409996447 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0dfe00700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0102e20 0x7fd0e0115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0df400700 1 -- 172.21.15.63:0/2409996447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0102e20 msgr2=0x7fd0e0115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0df400700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0102e20 0x7fd0e0115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.476+0000 7fd0df400700 1 -- 172.21.15.63:0/2409996447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc005040 con 0x7fd0e01037c0 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.477+0000 7fd0df400700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e0116190 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009fd0 tx=0x7fd0cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.477+0000 7fd0d7e00700 1 -- 172.21.15.63:0/2409996447 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0cc00e030 con 0x7fd0e01037c0 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.477+0000 7fd0d7e00700 1 -- 172.21.15.63:0/2409996447 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0cc013e00 con 0x7fd0e01037c0 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.477+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e010fe30 con 0x7fd0e01037c0 2024-03-23T12:10:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.477+0000 7fd0d7e00700 1 -- 172.21.15.63:0/2409996447 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0cc01ccd0 con 0x7fd0e01037c0 2024-03-23T12:10:18.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.477+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e01103f0 con 0x7fd0e01037c0 2024-03-23T12:10:18.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.478+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e00fe570 con 0x7fd0e01037c0 2024-03-23T12:10:18.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.479+0000 7fd0d7e00700 1 -- 172.21.15.63:0/2409996447 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd0cc01c500 con 0x7fd0e01037c0 2024-03-23T12:10:18.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.480+0000 7fd0d7e00700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd0c406c7d0 0x7fd0c406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:18.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.480+0000 7fd0d7e00700 1 -- 172.21.15.63:0/2409996447 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd0cc08d3e0 con 0x7fd0e01037c0 2024-03-23T12:10:18.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.480+0000 7fd0dfe00700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd0c406c7d0 0x7fd0c406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:18.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.481+0000 7fd0dfe00700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd0c406c7d0 0x7fd0c406ec80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009fd0 tx=0x7fd0d000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:18.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.485+0000 7fd0d7e00700 1 -- 172.21.15.63:0/2409996447 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd0cc05b070 con 0x7fd0e01037c0 2024-03-23T12:10:18.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.789+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd0e005f5f0 con 0x7fd0e01037c0 2024-03-23T12:10:18.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.790+0000 7fd0d7e00700 1 -- 172.21.15.63:0/2409996447 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fd0cc004490 con 0x7fd0e01037c0 2024-03-23T12:10:18.769 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:10:18.770 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:10:18.770 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:10:18.770 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:10:18.770 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd0c406c7d0 msgr2=0x7fd0c406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd0c406c7d0 0x7fd0c406ec80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009fd0 tx=0x7fd0d000b040 comp rx=0 tx=0).stop 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 msgr2=0x7fd0e0116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e0116190 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009fd0 tx=0x7fd0cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 shutdown_connections 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd0c406c7d0 0x7fd0c406ec80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0e0102e20 0x7fd0e0115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 --2- 172.21.15.63:0/2409996447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0e01037c0 0x7fd0e0116190 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 >> 172.21.15.63:0/2409996447 conn(0x7fd0e00f89d0 msgr2=0x7fd0e00f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 shutdown_connections 2024-03-23T12:10:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:18.792+0000 7fd0e7200700 1 -- 172.21.15.63:0/2409996447 wait complete. 2024-03-23T12:10:18.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:18 smithi063 ceph-mon[141124]: from='client.35086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:18.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:18 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1970728033' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:10:18.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:18 smithi063 ceph-mon[141124]: pgmap v575: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:18.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:18 smithi111 ceph-mon[106899]: from='client.35086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:18.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:18 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1970728033' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:10:18.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:18 smithi111 ceph-mon[106899]: pgmap v575: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:19.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:19 smithi063 ceph-mon[141124]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:19.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:19 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2409996447' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:10:19.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:19 smithi111 ceph-mon[106899]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:19.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:19 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2409996447' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:10:20.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:20 smithi063 ceph-mon[141124]: pgmap v576: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:20.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:20 smithi111 ceph-mon[106899]: pgmap v576: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:23.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:23 smithi063 ceph-mon[141124]: pgmap v577: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:23 smithi111 ceph-mon[106899]: pgmap v577: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:24.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:24 smithi063 ceph-mon[141124]: pgmap v578: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:24.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:24 smithi111 ceph-mon[106899]: pgmap v578: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:26 smithi063 ceph-mon[141124]: pgmap v579: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:26 smithi111 ceph-mon[106899]: pgmap v579: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:29.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:29 smithi063 ceph-mon[141124]: pgmap v580: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:29.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:29 smithi111 ceph-mon[106899]: pgmap v580: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:30 smithi063 ceph-mon[141124]: pgmap v581: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:30 smithi111 ceph-mon[106899]: pgmap v581: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:33 smithi063 ceph-mon[141124]: pgmap v582: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:33.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:33 smithi111 ceph-mon[106899]: pgmap v582: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:34 smithi063 ceph-mon[141124]: pgmap v583: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:34.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:34 smithi111 ceph-mon[106899]: pgmap v583: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:36.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:36 smithi063 ceph-mon[141124]: pgmap v584: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:36.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:36 smithi111 ceph-mon[106899]: pgmap v584: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:39.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:39 smithi063 ceph-mon[141124]: pgmap v585: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:39.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:39 smithi111 ceph-mon[106899]: pgmap v585: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:40.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:40 smithi063 ceph-mon[141124]: pgmap v586: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:40.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:40 smithi111 ceph-mon[106899]: pgmap v586: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:41.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:41.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:42.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:42 smithi063 ceph-mon[141124]: pgmap v587: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:42.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:42 smithi111 ceph-mon[106899]: pgmap v587: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:44.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:44 smithi063 ceph-mon[141124]: pgmap v588: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:44.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:44 smithi111 ceph-mon[106899]: pgmap v588: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:46.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:46 smithi063 ceph-mon[141124]: pgmap v589: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:46.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:46 smithi111 ceph-mon[106899]: pgmap v589: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:48.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:48 smithi063 ceph-mon[141124]: pgmap v590: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:48.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:48 smithi111 ceph-mon[106899]: pgmap v590: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:48.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.945+0000 7f5f1fc00700 1 -- 172.21.15.63:0/752164896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f181037a0 msgr2=0x7f5f180fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:48.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.945+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/752164896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f181037a0 0x7f5f180fce10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5f04004940 tx=0x7f5f04009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:48.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.945+0000 7f5f1fc00700 1 -- 172.21.15.63:0/752164896 shutdown_connections 2024-03-23T12:10:48.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.945+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/752164896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f181037a0 0x7f5f180fce10 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:48.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.945+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/752164896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f18102e00 0x7f5f181031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:48.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.945+0000 7f5f1fc00700 1 -- 172.21.15.63:0/752164896 >> 172.21.15.63:0/752164896 conn(0x7f5f180f89b0 msgr2=0x7f5f180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:48.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.946+0000 7f5f1fc00700 1 -- 172.21.15.63:0/752164896 shutdown_connections 2024-03-23T12:10:48.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.946+0000 7f5f1fc00700 1 -- 172.21.15.63:0/752164896 wait complete. 2024-03-23T12:10:48.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.946+0000 7f5f1fc00700 1 Processor -- start 2024-03-23T12:10:48.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1fc00700 1 -- start start 2024-03-23T12:10:48.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1fc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f18102e00 0x7f5f18115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:48.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1fc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f181037a0 0x7f5f18116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:48.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1fc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f18116890 con 0x7f5f18102e00 2024-03-23T12:10:48.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1fc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1810fc60 con 0x7f5f181037a0 2024-03-23T12:10:48.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f18102e00 0x7f5f18115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:48.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f18102e00 0x7f5f18115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40488/0 (socket says 172.21.15.63:40488) 2024-03-23T12:10:48.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f1d200700 1 -- 172.21.15.63:0/160337609 learned_addr learned my addr 172.21.15.63:0/160337609 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:48.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.947+0000 7f5f17e00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f181037a0 0x7f5f18116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:48.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f1d200700 1 -- 172.21.15.63:0/160337609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f181037a0 msgr2=0x7f5f18116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:48.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f1d200700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f181037a0 0x7f5f18116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:48.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f1d200700 1 -- 172.21.15.63:0/160337609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f04005040 con 0x7f5f18102e00 2024-03-23T12:10:48.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f1d200700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f18102e00 0x7f5f18115d00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c002a50 tx=0x7f5f0c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:48.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f14c00700 1 -- 172.21.15.63:0/160337609 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0c007cc0 con 0x7f5f18102e00 2024-03-23T12:10:48.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f14c00700 1 -- 172.21.15.63:0/160337609 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f0c00f040 con 0x7f5f18102e00 2024-03-23T12:10:48.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1810ff40 con 0x7f5f18102e00 2024-03-23T12:10:48.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f14c00700 1 -- 172.21.15.63:0/160337609 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0c009540 con 0x7f5f18102e00 2024-03-23T12:10:48.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.948+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f18110500 con 0x7f5f18102e00 2024-03-23T12:10:48.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.949+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f180fe550 con 0x7f5f18102e00 2024-03-23T12:10:48.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.950+0000 7f5f14c00700 1 -- 172.21.15.63:0/160337609 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5f0c0096a0 con 0x7f5f18102e00 2024-03-23T12:10:48.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.951+0000 7f5f14c00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0006c7c0 0x7f5f0006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:48.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.951+0000 7f5f14c00700 1 -- 172.21.15.63:0/160337609 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5f0c08b530 con 0x7f5f18102e00 2024-03-23T12:10:48.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.952+0000 7f5f17e00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0006c7c0 0x7f5f0006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:48.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.952+0000 7f5f17e00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0006c7c0 0x7f5f0006ec70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b3b0 tx=0x7f5f04009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:48.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:48.956+0000 7f5f14c00700 1 -- 172.21.15.63:0/160337609 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5f0c059240 con 0x7f5f18102e00 2024-03-23T12:10:49.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.189+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f1804bf40 con 0x7f5f0006c7c0 2024-03-23T12:10:49.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.190+0000 7f5f14c00700 1 -- 172.21.15.63:0/160337609 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5f1804bf40 con 0x7f5f0006c7c0 2024-03-23T12:10:49.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0006c7c0 msgr2=0x7f5f0006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0006c7c0 0x7f5f0006ec70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b3b0 tx=0x7f5f04009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:49.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f18102e00 msgr2=0x7f5f18115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f18102e00 0x7f5f18115d00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c002a50 tx=0x7f5f0c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 shutdown_connections 2024-03-23T12:10:49.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f0006c7c0 0x7f5f0006ec70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f18102e00 0x7f5f18115d00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 --2- 172.21.15.63:0/160337609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f181037a0 0x7f5f18116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.192+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 >> 172.21.15.63:0/160337609 conn(0x7f5f180f89b0 msgr2=0x7f5f180f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:49.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.193+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 shutdown_connections 2024-03-23T12:10:49.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.193+0000 7f5f1fc00700 1 -- 172.21.15.63:0/160337609 wait complete. 2024-03-23T12:10:49.188 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:10:49.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.332+0000 7fbc9f400700 1 -- 172.21.15.63:0/2325655159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 msgr2=0x7fbc98070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.332+0000 7fbc9f400700 1 --2- 172.21.15.63:0/2325655159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc98070360 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c004970 tx=0x7fbc8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:49.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.332+0000 7fbc9f400700 1 -- 172.21.15.63:0/2325655159 shutdown_connections 2024-03-23T12:10:49.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.332+0000 7fbc9f400700 1 --2- 172.21.15.63:0/2325655159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980708a0 0x7fbc98070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.332+0000 7fbc9f400700 1 --2- 172.21.15.63:0/2325655159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc98070360 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.332+0000 7fbc9f400700 1 -- 172.21.15.63:0/2325655159 >> 172.21.15.63:0/2325655159 conn(0x7fbc980f89b0 msgr2=0x7fbc980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:49.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.333+0000 7fbc9f400700 1 -- 172.21.15.63:0/2325655159 shutdown_connections 2024-03-23T12:10:49.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.333+0000 7fbc9f400700 1 -- 172.21.15.63:0/2325655159 wait complete. 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.333+0000 7fbc9f400700 1 Processor -- start 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc9f400700 1 -- start start 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc9f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980708a0 0x7fbc9811a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc9f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc9811a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc9f400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9811ac50 con 0x7fbc98071f90 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc9f400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc98114020 con 0x7fbc980708a0 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc97e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc9811a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc97e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc9811a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40504/0 (socket says 172.21.15.63:40504) 2024-03-23T12:10:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.334+0000 7fbc97e00700 1 -- 172.21.15.63:0/899962481 learned_addr learned my addr 172.21.15.63:0/899962481 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc9ca00700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980708a0 0x7fbc9811a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc97e00700 1 -- 172.21.15.63:0/899962481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980708a0 msgr2=0x7fbc9811a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc97e00700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980708a0 0x7fbc9811a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc97e00700 1 -- 172.21.15.63:0/899962481 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c005040 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc97e00700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc9811a600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbc84002a50 tx=0x7fbc840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc94c00700 1 -- 172.21.15.63:0/899962481 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc84007cc0 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc94c00700 1 -- 172.21.15.63:0/899962481 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc8400f040 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc94c00700 1 -- 172.21.15.63:0/899962481 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc84009540 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc98114300 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.335+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc98114840 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.336+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc98071aa0 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.337+0000 7fbc94c00700 1 -- 172.21.15.63:0/899962481 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbc840096a0 con 0x7fbc98071f90 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.338+0000 7fbc94c00700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc8006c7d0 0x7fbc8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.338+0000 7fbc94c00700 1 -- 172.21.15.63:0/899962481 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbc8408b820 con 0x7fbc98071f90 2024-03-23T12:10:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.339+0000 7fbc9ca00700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc8006c7d0 0x7fbc8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:49.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.339+0000 7fbc9ca00700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc8006c7d0 0x7fbc8006ec80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009fd0 tx=0x7fbc8c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.343+0000 7fbc94c00700 1 -- 172.21.15.63:0/899962481 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc840594b0 con 0x7fbc98071f90 2024-03-23T12:10:49.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.575+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc98110450 con 0x7fbc8006c7d0 2024-03-23T12:10:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.576+0000 7fbc94c00700 1 -- 172.21.15.63:0/899962481 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbc98110450 con 0x7fbc8006c7d0 2024-03-23T12:10:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.578+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc8006c7d0 msgr2=0x7fbc8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.578+0000 7fbc9f400700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc8006c7d0 0x7fbc8006ec80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009fd0 tx=0x7fbc8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.578+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 msgr2=0x7fbc9811a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.578+0000 7fbc9f400700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc9811a600 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbc84002a50 tx=0x7fbc840046b0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.579+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 shutdown_connections 2024-03-23T12:10:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.579+0000 7fbc9f400700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc8006c7d0 0x7fbc8006ec80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.579+0000 7fbc9f400700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980708a0 0x7fbc9811a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.579+0000 7fbc9f400700 1 --2- 172.21.15.63:0/899962481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc98071f90 0x7fbc9811a600 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.579+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 >> 172.21.15.63:0/899962481 conn(0x7fbc980f89b0 msgr2=0x7fbc980f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.579+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 shutdown_connections 2024-03-23T12:10:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.579+0000 7fbc9f400700 1 -- 172.21.15.63:0/899962481 wait complete. 2024-03-23T12:10:49.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 -- 172.21.15.63:0/2364353172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 msgr2=0x7fb0f01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2364353172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f01031d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4004880 tx=0x7fb0e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:49.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 -- 172.21.15.63:0/2364353172 shutdown_connections 2024-03-23T12:10:49.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2364353172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f01037a0 0x7fb0f00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2364353172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f01031d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 -- 172.21.15.63:0/2364353172 >> 172.21.15.63:0/2364353172 conn(0x7fb0f00f89b0 msgr2=0x7fb0f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:49.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 -- 172.21.15.63:0/2364353172 shutdown_connections 2024-03-23T12:10:49.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.716+0000 7fb0f8200700 1 -- 172.21.15.63:0/2364353172 wait complete. 2024-03-23T12:10:49.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.717+0000 7fb0f8200700 1 Processor -- start 2024-03-23T12:10:49.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.717+0000 7fb0f8200700 1 -- start start 2024-03-23T12:10:49.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f8200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f0115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:49.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f8200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f01037a0 0x7fb0f01161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:49.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f8200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f0116800 con 0x7fb0f0102e00 2024-03-23T12:10:49.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f8200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f010fbd0 con 0x7fb0f01037a0 2024-03-23T12:10:49.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f0115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:49.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f5800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f0115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40526/0 (socket says 172.21.15.63:40526) 2024-03-23T12:10:49.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f5800700 1 -- 172.21.15.63:0/2624705330 learned_addr learned my addr 172.21.15.63:0/2624705330 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f4e00700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f01037a0 0x7fb0f01161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f5800700 1 -- 172.21.15.63:0/2624705330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f01037a0 msgr2=0x7fb0f01161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f5800700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f01037a0 0x7fb0f01161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f5800700 1 -- 172.21.15.63:0/2624705330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e4005040 con 0x7fb0f0102e00 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.718+0000 7fb0f5800700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f0115c70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.719+0000 7fb0ed600700 1 -- 172.21.15.63:0/2624705330 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0e400e070 con 0x7fb0f0102e00 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.719+0000 7fb0ed600700 1 -- 172.21.15.63:0/2624705330 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0e4013dd0 con 0x7fb0f0102e00 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.719+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f010fe50 con 0x7fb0f0102e00 2024-03-23T12:10:49.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.719+0000 7fb0ed600700 1 -- 172.21.15.63:0/2624705330 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0e401ce00 con 0x7fb0f0102e00 2024-03-23T12:10:49.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.719+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f01103b0 con 0x7fb0f0102e00 2024-03-23T12:10:49.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.720+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f00fe550 con 0x7fb0f0102e00 2024-03-23T12:10:49.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.721+0000 7fb0ed600700 1 -- 172.21.15.63:0/2624705330 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb0e4013920 con 0x7fb0f0102e00 2024-03-23T12:10:49.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.722+0000 7fb0ed600700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:49.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.722+0000 7fb0ed600700 1 -- 172.21.15.63:0/2624705330 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb0e408d0d0 con 0x7fb0f0102e00 2024-03-23T12:10:49.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.722+0000 7fb0f4e00700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:49.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.723+0000 7fb0f4e00700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0046c0 tx=0x7fb0dc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:49.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.726+0000 7fb0ed600700 1 -- 172.21.15.63:0/2624705330 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb0e405ad60 con 0x7fb0f0102e00 2024-03-23T12:10:49.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.954+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0f004bed0 con 0x7fb0d806c7d0 2024-03-23T12:10:49.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.964+0000 7fb0ed600700 1 -- 172.21.15.63:0/2624705330 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fb0f004bed0 con 0x7fb0d806c7d0 2024-03-23T12:10:49.944 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (20m) 12m ago 33m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (17m) 12m ago 33m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (17m) 12m ago 32m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (19m) 12m ago 32m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (13m) 12m ago 26m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (12m) 12m ago 26m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (26m) 12m ago 26m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (12m) 12m ago 26m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (22m) 12m ago 34m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (21m) 12m ago 32m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (18m) 12m ago 34m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (18m) 12m ago 32m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (20m) 12m ago 33m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:10:49.945 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (20m) 12m ago 32m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:10:49.946 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (16m) 12m ago 31m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:10:49.946 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (16m) 12m ago 30m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:10:49.946 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (15m) 12m ago 30m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:10:49.946 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (14m) 12m ago 29m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:10:49.946 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (14m) 12m ago 29m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:10:49.946 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (13m) 12m ago 28m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:10:49.946 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (20m) 12m ago 32m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:10:49.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 msgr2=0x7fb0d806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0046c0 tx=0x7fb0dc00b040 comp rx=0 tx=0).stop 2024-03-23T12:10:49.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 msgr2=0x7fb0f0115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:49.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f0115c70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:49.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 shutdown_connections 2024-03-23T12:10:49.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb0d806c7d0 0x7fb0d806ec80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0f0102e00 0x7fb0f0115c70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.966+0000 7fb0f8200700 1 --2- 172.21.15.63:0/2624705330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f01037a0 0x7fb0f01161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:49.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.967+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 >> 172.21.15.63:0/2624705330 conn(0x7fb0f00f89b0 msgr2=0x7fb0f00f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:49.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.967+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 shutdown_connections 2024-03-23T12:10:49.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:49.967+0000 7fb0f8200700 1 -- 172.21.15.63:0/2624705330 wait complete. 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 -- 172.21.15.63:0/2473769256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 msgr2=0x7f32b80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 --2- 172.21.15.63:0/2473769256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b80fce10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f32a4004940 tx=0x7f32a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 -- 172.21.15.63:0/2473769256 shutdown_connections 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 --2- 172.21.15.63:0/2473769256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b80fce10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 --2- 172.21.15.63:0/2473769256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8102e00 0x7f32b81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 -- 172.21.15.63:0/2473769256 >> 172.21.15.63:0/2473769256 conn(0x7f32b80f89b0 msgr2=0x7f32b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 -- 172.21.15.63:0/2473769256 shutdown_connections 2024-03-23T12:10:50.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.104+0000 7f32bf400700 1 -- 172.21.15.63:0/2473769256 wait complete. 2024-03-23T12:10:50.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.105+0000 7f32bf400700 1 Processor -- start 2024-03-23T12:10:50.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.105+0000 7f32bf400700 1 -- start start 2024-03-23T12:10:50.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.105+0000 7f32bf400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8102e00 0x7f32b8115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.105+0000 7f32bf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b81162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.105+0000 7f32bf400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b8116930 con 0x7f32b81037a0 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.105+0000 7f32bf400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b810fd00 con 0x7f32b8102e00 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.105+0000 7f32b7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b81162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b81162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40552/0 (socket says 172.21.15.63:40552) 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b7e00700 1 -- 172.21.15.63:0/2491069040 learned_addr learned my addr 172.21.15.63:0/2491069040 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32bca00700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8102e00 0x7f32b8115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b7e00700 1 -- 172.21.15.63:0/2491069040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8102e00 msgr2=0x7f32b8115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b7e00700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8102e00 0x7f32b8115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b7e00700 1 -- 172.21.15.63:0/2491069040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a4005040 con 0x7f32b81037a0 2024-03-23T12:10:50.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b7e00700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b81162e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009390 tx=0x7f32a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:50.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b4c00700 1 -- 172.21.15.63:0/2491069040 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a400e040 con 0x7f32b81037a0 2024-03-23T12:10:50.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b810ff80 con 0x7f32b81037a0 2024-03-23T12:10:50.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b4c00700 1 -- 172.21.15.63:0/2491069040 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32a4013420 con 0x7f32b81037a0 2024-03-23T12:10:50.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.106+0000 7f32b4c00700 1 -- 172.21.15.63:0/2491069040 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a401c630 con 0x7f32b81037a0 2024-03-23T12:10:50.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.107+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b81104c0 con 0x7f32b81037a0 2024-03-23T12:10:50.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.108+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b80fe550 con 0x7f32b81037a0 2024-03-23T12:10:50.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.108+0000 7f32b4c00700 1 -- 172.21.15.63:0/2491069040 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f32a4013a00 con 0x7f32b81037a0 2024-03-23T12:10:50.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.109+0000 7f32b4c00700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f32a006c7d0 0x7f32a006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.109+0000 7f32b4c00700 1 -- 172.21.15.63:0/2491069040 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f32a408cd50 con 0x7f32b81037a0 2024-03-23T12:10:50.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.110+0000 7f32bca00700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f32a006c7d0 0x7f32a006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.110+0000 7f32bca00700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f32a006c7d0 0x7f32a006ec80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f32ac004990 tx=0x7f32ac009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:50.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.114+0000 7f32b4c00700 1 -- 172.21.15.63:0/2491069040 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f32a405aa60 con 0x7f32b81037a0 2024-03-23T12:10:50.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:50 smithi063 ceph-mon[141124]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:50.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:50 smithi063 ceph-mon[141124]: from='client.35110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:50.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:50 smithi063 ceph-mon[141124]: from='client.35114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:50.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:50 smithi063 ceph-mon[141124]: pgmap v591: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:50.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.410+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f32b804bed0 con 0x7f32b81037a0 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.410+0000 7f32b4c00700 1 -- 172.21.15.63:0/2491069040 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f32a405a5f0 con 0x7f32b81037a0 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:10:50.391 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:10:50.392 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:10:50.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.412+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f32a006c7d0 msgr2=0x7f32a006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.412+0000 7f32bf400700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f32a006c7d0 0x7f32a006ec80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f32ac004990 tx=0x7f32ac009ee0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.412+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 msgr2=0x7f32b81162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.412+0000 7f32bf400700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b81162e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009390 tx=0x7f32a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.413+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 shutdown_connections 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.413+0000 7f32bf400700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f32a006c7d0 0x7f32a006ec80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.413+0000 7f32bf400700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32b8102e00 0x7f32b8115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.413+0000 7f32bf400700 1 --2- 172.21.15.63:0/2491069040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32b81037a0 0x7f32b81162e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.413+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 >> 172.21.15.63:0/2491069040 conn(0x7f32b80f89b0 msgr2=0x7f32b80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.413+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 shutdown_connections 2024-03-23T12:10:50.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.413+0000 7f32bf400700 1 -- 172.21.15.63:0/2491069040 wait complete. 2024-03-23T12:10:50.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:50 smithi111 ceph-mon[106899]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:50.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:50 smithi111 ceph-mon[106899]: from='client.35110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:50.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:50 smithi111 ceph-mon[106899]: from='client.35114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:50.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:50 smithi111 ceph-mon[106899]: pgmap v591: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:50.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 -- 172.21.15.63:0/2132478637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070360 msgr2=0x7f1208070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 --2- 172.21.15.63:0/2132478637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070360 0x7f1208070730 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f11fc004970 tx=0x7f11fc009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 -- 172.21.15.63:0/2132478637 shutdown_connections 2024-03-23T12:10:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 --2- 172.21.15.63:0/2132478637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208070c70 0x7f1208109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 --2- 172.21.15.63:0/2132478637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070360 0x7f1208070730 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 -- 172.21.15.63:0/2132478637 >> 172.21.15.63:0/2132478637 conn(0x7f12080f89b0 msgr2=0x7f12080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 -- 172.21.15.63:0/2132478637 shutdown_connections 2024-03-23T12:10:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.551+0000 7f120fc00700 1 -- 172.21.15.63:0/2132478637 wait complete. 2024-03-23T12:10:50.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f120fc00700 1 Processor -- start 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f120fc00700 1 -- start start 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f120fc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208070360 0x7f12080fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f120fc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070c70 0x7f12080fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f120fc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1208100f70 con 0x7f1208070c70 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f120fc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080fd7f0 con 0x7f1208070360 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f1207e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070c70 0x7f12080fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f1207e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070c70 0x7f12080fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40572/0 (socket says 172.21.15.63:40572) 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.552+0000 7f1207e00700 1 -- 172.21.15.63:0/633879298 learned_addr learned my addr 172.21.15.63:0/633879298 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:50.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f1207e00700 1 -- 172.21.15.63:0/633879298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208070360 msgr2=0x7f12080fcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f120d200700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208070360 0x7f12080fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f1207e00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208070360 0x7f12080fcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f1207e00700 1 -- 172.21.15.63:0/633879298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc005040 con 0x7f1208070c70 2024-03-23T12:10:50.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f120d200700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208070360 0x7f12080fcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:10:50.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f1207e00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070c70 0x7f12080fd2b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f11f400eb70 tx=0x7f11f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f1204c00700 1 -- 172.21.15.63:0/633879298 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f4007cc0 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f1204c00700 1 -- 172.21.15.63:0/633879298 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11f400f040 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12080fda70 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.553+0000 7f1204c00700 1 -- 172.21.15.63:0/633879298 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f40095f0 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.554+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120811e590 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.554+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1208071aa0 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.555+0000 7f1204c00700 1 -- 172.21.15.63:0/633879298 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f11f4009750 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.556+0000 7f1204c00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f006c7d0 0x7f11f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.556+0000 7f1204c00700 1 -- 172.21.15.63:0/633879298 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f11f408b760 con 0x7f1208070c70 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.557+0000 7f120d200700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f006c7d0 0x7f11f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.557+0000 7f120d200700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f006c7d0 0x7f11f006ec80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f11fc004560 tx=0x7f11fc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:50.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.561+0000 7f1204c00700 1 -- 172.21.15.63:0/633879298 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f11f40558e0 con 0x7f1208070c70 2024-03-23T12:10:50.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.822+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f120804bed0 con 0x7f1208070c70 2024-03-23T12:10:50.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.823+0000 7f1204c00700 1 -- 172.21.15.63:0/633879298 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f11f4058f00 con 0x7f1208070c70 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:10:50.804 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:10:50.805 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:10:50.806 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:50.807 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:50.807 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:10:50.807 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:10:50.807 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:50.807 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:50.807 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:50.807 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:10:50.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f006c7d0 msgr2=0x7f11f006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f006c7d0 0x7f11f006ec80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f11fc004560 tx=0x7f11fc009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:50.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070c70 msgr2=0x7f12080fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070c70 0x7f12080fd2b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f11f400eb70 tx=0x7f11f400b040 comp rx=0 tx=0).stop 2024-03-23T12:10:50.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 shutdown_connections 2024-03-23T12:10:50.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f11f006c7d0 0x7f11f006ec80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208070360 0x7f12080fcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 --2- 172.21.15.63:0/633879298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1208070c70 0x7f12080fd2b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 >> 172.21.15.63:0/633879298 conn(0x7f12080f89b0 msgr2=0x7f12080f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:50.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 shutdown_connections 2024-03-23T12:10:50.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.825+0000 7f120fc00700 1 -- 172.21.15.63:0/633879298 wait complete. 2024-03-23T12:10:50.809 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:10:50.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.963+0000 7fc49ba00700 1 -- 172.21.15.63:0/3716503245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc494102e00 msgr2=0x7fc4941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.963+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3716503245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc494102e00 0x7fc4941031d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc488004940 tx=0x7fc488009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:50.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.963+0000 7fc49ba00700 1 -- 172.21.15.63:0/3716503245 shutdown_connections 2024-03-23T12:10:50.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.963+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3716503245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4941037a0 0x7fc4940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.963+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3716503245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc494102e00 0x7fc4941031d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.963+0000 7fc49ba00700 1 -- 172.21.15.63:0/3716503245 >> 172.21.15.63:0/3716503245 conn(0x7fc4940f89b0 msgr2=0x7fc4940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:50.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.963+0000 7fc49ba00700 1 -- 172.21.15.63:0/3716503245 shutdown_connections 2024-03-23T12:10:50.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.964+0000 7fc49ba00700 1 -- 172.21.15.63:0/3716503245 wait complete. 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.964+0000 7fc49ba00700 1 Processor -- start 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.964+0000 7fc49ba00700 1 -- start start 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.965+0000 7fc49ba00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc494102e00 0x7fc494115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.965+0000 7fc49ba00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4941037a0 0x7fc494116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.965+0000 7fc49ba00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc494116880 con 0x7fc4941037a0 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.965+0000 7fc49ba00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49410fc50 con 0x7fc494102e00 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.965+0000 7fc493e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4941037a0 0x7fc494116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.965+0000 7fc493e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4941037a0 0x7fc494116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40582/0 (socket says 172.21.15.63:40582) 2024-03-23T12:10:50.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.965+0000 7fc493e00700 1 -- 172.21.15.63:0/3313635301 learned_addr learned my addr 172.21.15.63:0/3313635301 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc499000700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc494102e00 0x7fc494115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc493e00700 1 -- 172.21.15.63:0/3313635301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc494102e00 msgr2=0x7fc494115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc493e00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc494102e00 0x7fc494115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc493e00700 1 -- 172.21.15.63:0/3313635301 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc488005040 con 0x7fc4941037a0 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc499000700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc494102e00 0x7fc494115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc493e00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4941037a0 0x7fc494116230 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc480002a50 tx=0x7fc480004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc490c00700 1 -- 172.21.15.63:0/3313635301 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc480007cc0 con 0x7fc4941037a0 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc490c00700 1 -- 172.21.15.63:0/3313635301 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc48000f040 con 0x7fc4941037a0 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49410ff30 con 0x7fc4941037a0 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc490c00700 1 -- 172.21.15.63:0/3313635301 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc48000d9a0 con 0x7fc4941037a0 2024-03-23T12:10:50.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.966+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4941104f0 con 0x7fc4941037a0 2024-03-23T12:10:50.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.967+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4940fe550 con 0x7fc4941037a0 2024-03-23T12:10:50.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.968+0000 7fc490c00700 1 -- 172.21.15.63:0/3313635301 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc48000dc10 con 0x7fc4941037a0 2024-03-23T12:10:50.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.970+0000 7fc490c00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc47c06c7d0 0x7fc47c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.970+0000 7fc490c00700 1 -- 172.21.15.63:0/3313635301 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc48008b870 con 0x7fc4941037a0 2024-03-23T12:10:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.970+0000 7fc499000700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc47c06c7d0 0x7fc47c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:50.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.971+0000 7fc499000700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc47c06c7d0 0x7fc47c06ec80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc48800b840 tx=0x7fc488009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:50.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:50.975+0000 7fc490c00700 1 -- 172.21.15.63:0/3313635301 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc480059500 con 0x7fc4941037a0 2024-03-23T12:10:51.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.208+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc494109e30 con 0x7fc47c06c7d0 2024-03-23T12:10:51.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.210+0000 7fc490c00700 1 -- 172.21.15.63:0/3313635301 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc494109e30 con 0x7fc47c06c7d0 2024-03-23T12:10:51.190 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:10:51.191 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:10:51.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc47c06c7d0 msgr2=0x7fc47c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:51.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc47c06c7d0 0x7fc47c06ec80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc48800b840 tx=0x7fc488009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:51.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4941037a0 msgr2=0x7fc494116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:51.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4941037a0 0x7fc494116230 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc480002a50 tx=0x7fc480004600 comp rx=0 tx=0).stop 2024-03-23T12:10:51.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 shutdown_connections 2024-03-23T12:10:51.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc47c06c7d0 0x7fc47c06ec80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc494102e00 0x7fc494115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 --2- 172.21.15.63:0/3313635301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4941037a0 0x7fc494116230 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 >> 172.21.15.63:0/3313635301 conn(0x7fc4940f89b0 msgr2=0x7fc4940706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:51.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.212+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 shutdown_connections 2024-03-23T12:10:51.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.213+0000 7fc49ba00700 1 -- 172.21.15.63:0/3313635301 wait complete. 2024-03-23T12:10:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.349+0000 7fe0fec00700 1 -- 172.21.15.63:0/3165072634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 msgr2=0x7fe0f81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.349+0000 7fe0fec00700 1 --2- 172.21.15.63:0/3165072634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f81031d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8004940 tx=0x7fe0e8009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.349+0000 7fe0fec00700 1 -- 172.21.15.63:0/3165072634 shutdown_connections 2024-03-23T12:10:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.349+0000 7fe0fec00700 1 --2- 172.21.15.63:0/3165072634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0f81037a0 0x7fe0f80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.349+0000 7fe0fec00700 1 --2- 172.21.15.63:0/3165072634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f81031d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.349+0000 7fe0fec00700 1 -- 172.21.15.63:0/3165072634 >> 172.21.15.63:0/3165072634 conn(0x7fe0f80f89b0 msgr2=0x7fe0f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.349+0000 7fe0fec00700 1 -- 172.21.15.63:0/3165072634 shutdown_connections 2024-03-23T12:10:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.350+0000 7fe0fec00700 1 -- 172.21.15.63:0/3165072634 wait complete. 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.350+0000 7fe0fec00700 1 Processor -- start 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.350+0000 7fe0fec00700 1 -- start start 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0fec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0fec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0f81037a0 0x7fe0f8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0fec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f8116880 con 0x7fe0f8102e00 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0fec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f810fc50 con 0x7fe0f81037a0 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0f7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0f7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40594/0 (socket says 172.21.15.63:40594) 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0f7e00700 1 -- 172.21.15.63:0/252192997 learned_addr learned my addr 172.21.15.63:0/252192997 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0f7400700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0f81037a0 0x7fe0f8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0f7e00700 1 -- 172.21.15.63:0/252192997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0f81037a0 msgr2=0x7fe0f8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:51.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0f7e00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0f81037a0 0x7fe0f8116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.351+0000 7fe0f7e00700 1 -- 172.21.15.63:0/252192997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e8005040 con 0x7fe0f8102e00 2024-03-23T12:10:51.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.352+0000 7fe0f7e00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f8115cf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009360 tx=0x7fe0e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:51.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.352+0000 7fe0efe00700 1 -- 172.21.15.63:0/252192997 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0e800e040 con 0x7fe0f8102e00 2024-03-23T12:10:51.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.352+0000 7fe0efe00700 1 -- 172.21.15.63:0/252192997 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0e8013420 con 0x7fe0f8102e00 2024-03-23T12:10:51.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.352+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f810fed0 con 0x7fe0f8102e00 2024-03-23T12:10:51.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.352+0000 7fe0efe00700 1 -- 172.21.15.63:0/252192997 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0e801c5d0 con 0x7fe0f8102e00 2024-03-23T12:10:51.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.352+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f8110490 con 0x7fe0f8102e00 2024-03-23T12:10:51.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.353+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f80fe550 con 0x7fe0f8102e00 2024-03-23T12:10:51.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.354+0000 7fe0efe00700 1 -- 172.21.15.63:0/252192997 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe0e8013590 con 0x7fe0f8102e00 2024-03-23T12:10:51.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.355+0000 7fe0efe00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe0dc06c7d0 0x7fe0dc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:10:51.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.355+0000 7fe0efe00700 1 -- 172.21.15.63:0/252192997 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe0e808cbb0 con 0x7fe0f8102e00 2024-03-23T12:10:51.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.355+0000 7fe0f7400700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe0dc06c7d0 0x7fe0dc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:10:51.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.356+0000 7fe0f7400700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe0dc06c7d0 0x7fe0dc06ec80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8111590 tx=0x7fe0e400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:10:51.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.359+0000 7fe0efe00700 1 -- 172.21.15.63:0/252192997 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe0e805a8c0 con 0x7fe0f8102e00 2024-03-23T12:10:51.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:51 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2491069040' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:10:51.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:51 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/633879298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:10:51.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:51 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2491069040' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:10:51.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:51 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/633879298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:10:51.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.659+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe0f804bed0 con 0x7fe0f8102e00 2024-03-23T12:10:51.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.659+0000 7fe0efe00700 1 -- 172.21.15.63:0/252192997 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe0e805a450 con 0x7fe0f8102e00 2024-03-23T12:10:51.640 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:10:51.640 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:10:51.640 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:10:51.640 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:10:51.640 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:10:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.661+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe0dc06c7d0 msgr2=0x7fe0dc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.661+0000 7fe0fec00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe0dc06c7d0 0x7fe0dc06ec80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8111590 tx=0x7fe0e400b040 comp rx=0 tx=0).stop 2024-03-23T12:10:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.661+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 msgr2=0x7fe0f8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:10:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.661+0000 7fe0fec00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f8115cf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009360 tx=0x7fe0e8009e30 comp rx=0 tx=0).stop 2024-03-23T12:10:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.662+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 shutdown_connections 2024-03-23T12:10:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.662+0000 7fe0fec00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe0dc06c7d0 0x7fe0dc06ec80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.662+0000 7fe0fec00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8102e00 0x7fe0f8115cf0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.662+0000 7fe0fec00700 1 --2- 172.21.15.63:0/252192997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0f81037a0 0x7fe0f8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:10:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.662+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 >> 172.21.15.63:0/252192997 conn(0x7fe0f80f89b0 msgr2=0x7fe0f80706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:10:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.662+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 shutdown_connections 2024-03-23T12:10:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:10:51.662+0000 7fe0fec00700 1 -- 172.21.15.63:0/252192997 wait complete. 2024-03-23T12:10:52.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:52 smithi063 ceph-mon[141124]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:52.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:52 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/252192997' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:10:52.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:52 smithi063 ceph-mon[141124]: pgmap v592: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:52.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:52 smithi111 ceph-mon[106899]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:10:52.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:52 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/252192997' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:10:52.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:52 smithi111 ceph-mon[106899]: pgmap v592: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:54.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:54 smithi063 ceph-mon[141124]: pgmap v593: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:54.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:54 smithi111 ceph-mon[106899]: pgmap v593: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:56 smithi063 ceph-mon[141124]: pgmap v594: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:10:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:56 smithi111 ceph-mon[106899]: pgmap v594: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:58.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:10:58 smithi063 ceph-mon[141124]: pgmap v595: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:10:58.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:10:58 smithi111 ceph-mon[106899]: pgmap v595: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:00 smithi063 ceph-mon[141124]: pgmap v596: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:00 smithi111 ceph-mon[106899]: pgmap v596: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:02.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:02 smithi063 ceph-mon[141124]: pgmap v597: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:02.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:02 smithi111 ceph-mon[106899]: pgmap v597: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:04.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:04 smithi063 ceph-mon[141124]: pgmap v598: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:04.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:04 smithi111 ceph-mon[106899]: pgmap v598: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:06.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:06 smithi063 ceph-mon[141124]: pgmap v599: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:06.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:06 smithi111 ceph-mon[106899]: pgmap v599: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:08.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:08 smithi063 ceph-mon[141124]: pgmap v600: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:08.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:08 smithi111 ceph-mon[106899]: pgmap v600: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:10.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:10 smithi063 ceph-mon[141124]: pgmap v601: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:10.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:10 smithi111 ceph-mon[106899]: pgmap v601: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:11.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:11.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:12.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:12 smithi063 ceph-mon[141124]: pgmap v602: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:12.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:12 smithi111 ceph-mon[106899]: pgmap v602: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:14.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:14 smithi063 ceph-mon[141124]: pgmap v603: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:14.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:14 smithi111 ceph-mon[106899]: pgmap v603: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:16.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:16 smithi063 ceph-mon[141124]: pgmap v604: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:16.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:16 smithi111 ceph-mon[106899]: pgmap v604: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:18.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:18 smithi063 ceph-mon[141124]: pgmap v605: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:18.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:18 smithi111 ceph-mon[106899]: pgmap v605: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:20 smithi063 ceph-mon[141124]: pgmap v606: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:20.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:20 smithi111 ceph-mon[106899]: pgmap v606: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 -- 172.21.15.63:0/222833409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01037a0 msgr2=0x7f44a00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 --2- 172.21.15.63:0/222833409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01037a0 0x7f44a00fce10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f448c004940 tx=0x7f448c009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 -- 172.21.15.63:0/222833409 shutdown_connections 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 --2- 172.21.15.63:0/222833409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01037a0 0x7f44a00fce10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 --2- 172.21.15.63:0/222833409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44a0102e00 0x7f44a01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 -- 172.21.15.63:0/222833409 >> 172.21.15.63:0/222833409 conn(0x7f44a00f89b0 msgr2=0x7f44a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 -- 172.21.15.63:0/222833409 shutdown_connections 2024-03-23T12:11:21.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.815+0000 7f44a6a00700 1 -- 172.21.15.63:0/222833409 wait complete. 2024-03-23T12:11:21.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.816+0000 7f44a6a00700 1 Processor -- start 2024-03-23T12:11:21.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.816+0000 7f44a6a00700 1 -- start start 2024-03-23T12:11:21.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.817+0000 7f44a6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0102e00 0x7f44a00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:21.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.817+0000 7f44a6a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44a01037a0 0x7f44a0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:21.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.817+0000 7f44a6a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0070d90 con 0x7f44a0102e00 2024-03-23T12:11:21.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.817+0000 7f44a6a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0070ed0 con 0x7f44a01037a0 2024-03-23T12:11:21.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.817+0000 7f449fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0102e00 0x7f44a00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:21.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.817+0000 7f449fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0102e00 0x7f44a00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49648/0 (socket says 172.21.15.63:49648) 2024-03-23T12:11:21.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.817+0000 7f449fe00700 1 -- 172.21.15.63:0/579027494 learned_addr learned my addr 172.21.15.63:0/579027494 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:21.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f449f400700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44a01037a0 0x7f44a0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:21.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f449fe00700 1 -- 172.21.15.63:0/579027494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44a01037a0 msgr2=0x7f44a0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:21.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f449fe00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44a01037a0 0x7f44a0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f449fe00700 1 -- 172.21.15.63:0/579027494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c005040 con 0x7f44a0102e00 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f449fe00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0102e00 0x7f44a00702c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4490002a50 tx=0x7f44900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f4497e00700 1 -- 172.21.15.63:0/579027494 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4490007cc0 con 0x7f44a0102e00 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f4497e00700 1 -- 172.21.15.63:0/579027494 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f449000f040 con 0x7f44a0102e00 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.818+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a0119e20 con 0x7f44a0102e00 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.819+0000 7f4497e00700 1 -- 172.21.15.63:0/579027494 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f449000d9c0 con 0x7f44a0102e00 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.819+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a011a2a0 con 0x7f44a0102e00 2024-03-23T12:11:21.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.820+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a00fe550 con 0x7f44a0102e00 2024-03-23T12:11:21.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.821+0000 7f4497e00700 1 -- 172.21.15.63:0/579027494 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f449000dc00 con 0x7f44a0102e00 2024-03-23T12:11:21.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.822+0000 7f4497e00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f448406c7d0 0x7f448406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:21.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.822+0000 7f4497e00700 1 -- 172.21.15.63:0/579027494 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f449008b1b0 con 0x7f44a0102e00 2024-03-23T12:11:21.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.822+0000 7f449f400700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f448406c7d0 0x7f448406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:21.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.823+0000 7f449f400700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f448406c7d0 0x7f448406ec80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f44a0071c00 tx=0x7f448c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:21.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:21.827+0000 7f4497e00700 1 -- 172.21.15.63:0/579027494 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4490055330 con 0x7f44a0102e00 2024-03-23T12:11:22.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.063+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44a00719b0 con 0x7f448406c7d0 2024-03-23T12:11:22.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.064+0000 7f4497e00700 1 -- 172.21.15.63:0/579027494 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f44a00719b0 con 0x7f448406c7d0 2024-03-23T12:11:22.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f448406c7d0 msgr2=0x7f448406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f448406c7d0 0x7f448406ec80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f44a0071c00 tx=0x7f448c009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:22.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0102e00 msgr2=0x7f44a00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0102e00 0x7f44a00702c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4490002a50 tx=0x7f44900046b0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 shutdown_connections 2024-03-23T12:11:22.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f448406c7d0 0x7f448406ec80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0102e00 0x7f44a00702c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 --2- 172.21.15.63:0/579027494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44a01037a0 0x7f44a0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.066+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 >> 172.21.15.63:0/579027494 conn(0x7f44a00f89b0 msgr2=0x7f44a00f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:22.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.067+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 shutdown_connections 2024-03-23T12:11:22.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.067+0000 7f44a6a00700 1 -- 172.21.15.63:0/579027494 wait complete. 2024-03-23T12:11:22.064 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.206+0000 7f028ec00700 1 -- 172.21.15.63:0/3000684895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 msgr2=0x7f0288109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.206+0000 7f028ec00700 1 --2- 172.21.15.63:0/3000684895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f0288109110 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0274004970 tx=0x7f0274009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.206+0000 7f028ec00700 1 -- 172.21.15.63:0/3000684895 shutdown_connections 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.206+0000 7f028ec00700 1 --2- 172.21.15.63:0/3000684895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f0288109110 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.206+0000 7f028ec00700 1 --2- 172.21.15.63:0/3000684895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0288070360 0x7f0288070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.206+0000 7f028ec00700 1 -- 172.21.15.63:0/3000684895 >> 172.21.15.63:0/3000684895 conn(0x7f02880f89b0 msgr2=0x7f02880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.206+0000 7f028ec00700 1 -- 172.21.15.63:0/3000684895 shutdown_connections 2024-03-23T12:11:22.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.207+0000 7f028ec00700 1 -- 172.21.15.63:0/3000684895 wait complete. 2024-03-23T12:11:22.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.207+0000 7f028ec00700 1 Processor -- start 2024-03-23T12:11:22.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.207+0000 7f028ec00700 1 -- start start 2024-03-23T12:11:22.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f028ec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0288070360 0x7f0288115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f028ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f02881161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f028ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0288116830 con 0x7f0288070c70 2024-03-23T12:11:22.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f028ec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028810fc00 con 0x7f0288070360 2024-03-23T12:11:22.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f0287400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f02881161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f0287e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0288070360 0x7f0288115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f0287400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f02881161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49674/0 (socket says 172.21.15.63:49674) 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f0287400700 1 -- 172.21.15.63:0/2869448815 learned_addr learned my addr 172.21.15.63:0/2869448815 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f0287400700 1 -- 172.21.15.63:0/2869448815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0288070360 msgr2=0x7f0288115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f0287400700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0288070360 0x7f0288115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.208+0000 7f0287400700 1 -- 172.21.15.63:0/2869448815 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0274005040 con 0x7f0288070c70 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.209+0000 7f0287400700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f02881161e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0274005010 tx=0x7f0274009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.209+0000 7f027fe00700 1 -- 172.21.15.63:0/2869448815 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f027400e040 con 0x7f0288070c70 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.209+0000 7f027fe00700 1 -- 172.21.15.63:0/2869448815 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0274004290 con 0x7f0288070c70 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.209+0000 7f027fe00700 1 -- 172.21.15.63:0/2869448815 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0274013d00 con 0x7f0288070c70 2024-03-23T12:11:22.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.209+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028810fe80 con 0x7f0288070c70 2024-03-23T12:11:22.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.209+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02881103c0 con 0x7f0288070c70 2024-03-23T12:11:22.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.210+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0288103bd0 con 0x7f0288070c70 2024-03-23T12:11:22.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.211+0000 7f027fe00700 1 -- 172.21.15.63:0/2869448815 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f02740134b0 con 0x7f0288070c70 2024-03-23T12:11:22.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.212+0000 7f027fe00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f026c06c7d0 0x7f026c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.212+0000 7f027fe00700 1 -- 172.21.15.63:0/2869448815 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f027408cc10 con 0x7f0288070c70 2024-03-23T12:11:22.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.212+0000 7f0287e00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f026c06c7d0 0x7f026c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.213+0000 7f0287e00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f026c06c7d0 0x7f026c06ec80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0278009b80 tx=0x7f027800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:22.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.217+0000 7f027fe00700 1 -- 172.21.15.63:0/2869448815 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f027405a8a0 con 0x7f0288070c70 2024-03-23T12:11:22.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:22 smithi063 ceph-mon[141124]: pgmap v607: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:22.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:22 smithi111 ceph-mon[106899]: pgmap v607: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:22.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.448+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0288111360 con 0x7f026c06c7d0 2024-03-23T12:11:22.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.449+0000 7f027fe00700 1 -- 172.21.15.63:0/2869448815 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0288111360 con 0x7f026c06c7d0 2024-03-23T12:11:22.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f026c06c7d0 msgr2=0x7f026c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f026c06c7d0 0x7f026c06ec80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0278009b80 tx=0x7f027800b040 comp rx=0 tx=0).stop 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 msgr2=0x7f02881161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f02881161e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0274005010 tx=0x7f0274009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 shutdown_connections 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f026c06c7d0 0x7f026c06ec80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0288070360 0x7f0288115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 --2- 172.21.15.63:0/2869448815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0288070c70 0x7f02881161e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 >> 172.21.15.63:0/2869448815 conn(0x7f02880f89b0 msgr2=0x7f02881015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 shutdown_connections 2024-03-23T12:11:22.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.452+0000 7f028ec00700 1 -- 172.21.15.63:0/2869448815 wait complete. 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 -- 172.21.15.63:0/2914569668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72781037a0 msgr2=0x7f72780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 --2- 172.21.15.63:0/2914569668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72781037a0 0x7f72780fce10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7264004940 tx=0x7f7264009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 -- 172.21.15.63:0/2914569668 shutdown_connections 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 --2- 172.21.15.63:0/2914569668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72781037a0 0x7f72780fce10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 --2- 172.21.15.63:0/2914569668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7278102e00 0x7f72781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 -- 172.21.15.63:0/2914569668 >> 172.21.15.63:0/2914569668 conn(0x7f72780f89b0 msgr2=0x7f72780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 -- 172.21.15.63:0/2914569668 shutdown_connections 2024-03-23T12:11:22.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.589+0000 7f727f400700 1 -- 172.21.15.63:0/2914569668 wait complete. 2024-03-23T12:11:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.590+0000 7f727f400700 1 Processor -- start 2024-03-23T12:11:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.590+0000 7f727f400700 1 -- start start 2024-03-23T12:11:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.590+0000 7f727f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7278102e00 0x7f7278115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72781037a0 0x7f72781162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727f400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7278116930 con 0x7f7278102e00 2024-03-23T12:11:22.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727f400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727810fd00 con 0x7f72781037a0 2024-03-23T12:11:22.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7278102e00 0x7f7278115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7278102e00 0x7f7278115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49696/0 (socket says 172.21.15.63:49696) 2024-03-23T12:11:22.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727ca00700 1 -- 172.21.15.63:0/4161803667 learned_addr learned my addr 172.21.15.63:0/4161803667 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:22.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f7277e00700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72781037a0 0x7f72781162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727ca00700 1 -- 172.21.15.63:0/4161803667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72781037a0 msgr2=0x7f72781162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727ca00700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72781037a0 0x7f72781162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727ca00700 1 -- 172.21.15.63:0/4161803667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264005040 con 0x7f7278102e00 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.591+0000 7f727ca00700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7278102e00 0x7f7278115da0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f726c002a50 tx=0x7f726c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.592+0000 7f7274c00700 1 -- 172.21.15.63:0/4161803667 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f726c007cc0 con 0x7f7278102e00 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.592+0000 7f7274c00700 1 -- 172.21.15.63:0/4161803667 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f726c00f040 con 0x7f7278102e00 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.592+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727810ffe0 con 0x7f7278102e00 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.592+0000 7f7274c00700 1 -- 172.21.15.63:0/4161803667 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f726c00d9c0 con 0x7f7278102e00 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.592+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7278110520 con 0x7f7278102e00 2024-03-23T12:11:22.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.593+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72780fe550 con 0x7f7278102e00 2024-03-23T12:11:22.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.594+0000 7f7274c00700 1 -- 172.21.15.63:0/4161803667 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f726c00db20 con 0x7f7278102e00 2024-03-23T12:11:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.595+0000 7f7274c00700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726006c7d0 0x7f726006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.595+0000 7f7274c00700 1 -- 172.21.15.63:0/4161803667 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f726c08b760 con 0x7f7278102e00 2024-03-23T12:11:22.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.595+0000 7f7277e00700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726006c7d0 0x7f726006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.596+0000 7f7277e00700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726006c7d0 0x7f726006ec80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7278111640 tx=0x7f7264009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:22.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.599+0000 7f7274c00700 1 -- 172.21.15.63:0/4161803667 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f726c0593f0 con 0x7f7278102e00 2024-03-23T12:11:22.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.821+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f727804bf40 con 0x7f726006c7d0 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.831+0000 7f7274c00700 1 -- 172.21.15.63:0/4161803667 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f727804bf40 con 0x7f726006c7d0 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (20m) 13m ago 33m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (17m) 13m ago 34m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (17m) 13m ago 32m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (20m) 13m ago 33m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (13m) 13m ago 27m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (13m) 13m ago 27m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (27m) 13m ago 27m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:11:22.812 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (13m) 13m ago 27m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (23m) 13m ago 35m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (21m) 13m ago 32m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (19m) 13m ago 35m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (18m) 13m ago 32m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (21m) 13m ago 33m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (21m) 13m ago 32m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (17m) 13m ago 31m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (16m) 13m ago 31m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (16m) 13m ago 30m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (15m) 13m ago 30m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (14m) 13m ago 29m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (14m) 13m ago 29m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:11:22.813 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (21m) 13m ago 33m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:11:22.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726006c7d0 msgr2=0x7f726006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726006c7d0 0x7f726006ec80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7278111640 tx=0x7f7264009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7278102e00 msgr2=0x7f7278115da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7278102e00 0x7f7278115da0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f726c002a50 tx=0x7f726c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 shutdown_connections 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726006c7d0 0x7f726006ec80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7278102e00 0x7f7278115da0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 --2- 172.21.15.63:0/4161803667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72781037a0 0x7f72781162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 >> 172.21.15.63:0/4161803667 conn(0x7f72780f89b0 msgr2=0x7f72780f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 shutdown_connections 2024-03-23T12:11:22.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.834+0000 7f727f400700 1 -- 172.21.15.63:0/4161803667 wait complete. 2024-03-23T12:11:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 -- 172.21.15.63:0/308728697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070c10 msgr2=0x7f4520109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 --2- 172.21.15.63:0/308728697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070c10 0x7f4520109190 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f4508004970 tx=0x7f4508009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 -- 172.21.15.63:0/308728697 shutdown_connections 2024-03-23T12:11:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 --2- 172.21.15.63:0/308728697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070c10 0x7f4520109190 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 --2- 172.21.15.63:0/308728697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4520070300 0x7f45200706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 -- 172.21.15.63:0/308728697 >> 172.21.15.63:0/308728697 conn(0x7f45200f8970 msgr2=0x7f45200fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 -- 172.21.15.63:0/308728697 shutdown_connections 2024-03-23T12:11:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.970+0000 7f4524c00700 1 -- 172.21.15.63:0/308728697 wait complete. 2024-03-23T12:11:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.971+0000 7f4524c00700 1 Processor -- start 2024-03-23T12:11:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.971+0000 7f4524c00700 1 -- start start 2024-03-23T12:11:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f4524c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070300 0x7f4520115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f4524c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4520070c10 0x7f4520116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f4524c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4520116850 con 0x7f4520070300 2024-03-23T12:11:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f4524c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452010fc20 con 0x7f4520070c10 2024-03-23T12:11:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f451de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070300 0x7f4520115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f451de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070300 0x7f4520115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49712/0 (socket says 172.21.15.63:49712) 2024-03-23T12:11:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f451de00700 1 -- 172.21.15.63:0/1599163642 learned_addr learned my addr 172.21.15.63:0/1599163642 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.972+0000 7f4517e00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4520070c10 0x7f4520116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f451de00700 1 -- 172.21.15.63:0/1599163642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4520070c10 msgr2=0x7f4520116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f451de00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4520070c10 0x7f4520116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f451de00700 1 -- 172.21.15.63:0/1599163642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508005040 con 0x7f4520070300 2024-03-23T12:11:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f451de00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070300 0x7f4520115cc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4510002a50 tx=0x7f45100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f4516000700 1 -- 172.21.15.63:0/1599163642 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4510007cc0 con 0x7f4520070300 2024-03-23T12:11:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f4516000700 1 -- 172.21.15.63:0/1599163642 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f451000f040 con 0x7f4520070300 2024-03-23T12:11:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f4516000700 1 -- 172.21.15.63:0/1599163642 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f451000d9c0 con 0x7f4520070300 2024-03-23T12:11:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452010ff00 con 0x7f4520070300 2024-03-23T12:11:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.973+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4520110410 con 0x7f4520070300 2024-03-23T12:11:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.974+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4520071a40 con 0x7f4520070300 2024-03-23T12:11:22.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.975+0000 7f4516000700 1 -- 172.21.15.63:0/1599163642 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f451000db20 con 0x7f4520070300 2024-03-23T12:11:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.976+0000 7f4516000700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f450406c7d0 0x7f450406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.976+0000 7f4516000700 1 -- 172.21.15.63:0/1599163642 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f451008b6e0 con 0x7f4520070300 2024-03-23T12:11:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.977+0000 7f4517e00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f450406c7d0 0x7f450406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.977+0000 7f4517e00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f450406c7d0 0x7f450406ec80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f450800b370 tx=0x7f4508009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:22.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:22.981+0000 7f4516000700 1 -- 172.21.15.63:0/1599163642 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f45100593f0 con 0x7f4520070300 2024-03-23T12:11:23.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.282+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f452004bed0 con 0x7f4520070300 2024-03-23T12:11:23.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.283+0000 7f4516000700 1 -- 172.21.15.63:0/1599163642 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f4510058f80 con 0x7f4520070300 2024-03-23T12:11:23.263 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:11:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:11:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:11:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:11:23.265 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f450406c7d0 msgr2=0x7f450406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f450406c7d0 0x7f450406ec80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f450800b370 tx=0x7f4508009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070300 msgr2=0x7f4520115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070300 0x7f4520115cc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4510002a50 tx=0x7f45100046b0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 shutdown_connections 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f450406c7d0 0x7f450406ec80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4520070300 0x7f4520115cc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 --2- 172.21.15.63:0/1599163642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4520070c10 0x7f4520116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 >> 172.21.15.63:0/1599163642 conn(0x7f45200f8970 msgr2=0x7f45200f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:23.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 shutdown_connections 2024-03-23T12:11:23.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.285+0000 7f4524c00700 1 -- 172.21.15.63:0/1599163642 wait complete. 2024-03-23T12:11:23.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:23 smithi063 ceph-mon[141124]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:23.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:23 smithi063 ceph-mon[141124]: from='client.35138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:23.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:23 smithi063 ceph-mon[141124]: from='client.35142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:23 smithi111 ceph-mon[106899]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:23 smithi111 ceph-mon[106899]: from='client.35138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:23.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:23 smithi111 ceph-mon[106899]: from='client.35142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 -- 172.21.15.63:0/203892995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c80102e00 msgr2=0x7f3c801031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 --2- 172.21.15.63:0/203892995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c80102e00 0x7f3c801031d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3c70004940 tx=0x7f3c70009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 -- 172.21.15.63:0/203892995 shutdown_connections 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 --2- 172.21.15.63:0/203892995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c801037a0 0x7f3c800fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 --2- 172.21.15.63:0/203892995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c80102e00 0x7f3c801031d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 -- 172.21.15.63:0/203892995 >> 172.21.15.63:0/203892995 conn(0x7f3c800f89b0 msgr2=0x7f3c800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 -- 172.21.15.63:0/203892995 shutdown_connections 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.423+0000 7f3c86c00700 1 -- 172.21.15.63:0/203892995 wait complete. 2024-03-23T12:11:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.424+0000 7f3c86c00700 1 Processor -- start 2024-03-23T12:11:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.424+0000 7f3c86c00700 1 -- start start 2024-03-23T12:11:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.424+0000 7f3c86c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c80102e00 0x7f3c800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.424+0000 7f3c86c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c801037a0 0x7f3c80070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.424+0000 7f3c86c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80070d90 con 0x7f3c801037a0 2024-03-23T12:11:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.424+0000 7f3c86c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80070ed0 con 0x7f3c80102e00 2024-03-23T12:11:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c801037a0 0x7f3c80070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c801037a0 0x7f3c80070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49728/0 (socket says 172.21.15.63:49728) 2024-03-23T12:11:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7f400700 1 -- 172.21.15.63:0/3584310541 learned_addr learned my addr 172.21.15.63:0/3584310541 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7fe00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c80102e00 0x7f3c800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7f400700 1 -- 172.21.15.63:0/3584310541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c80102e00 msgr2=0x7f3c800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7f400700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c80102e00 0x7f3c800702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7f400700 1 -- 172.21.15.63:0/3584310541 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70005040 con 0x7f3c801037a0 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7fe00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c80102e00 0x7f3c800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c7f400700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c801037a0 0x7f3c80070800 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c002a50 tx=0x7f3c6c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.425+0000 7f3c77e00700 1 -- 172.21.15.63:0/3584310541 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c6c007cc0 con 0x7f3c801037a0 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.426+0000 7f3c77e00700 1 -- 172.21.15.63:0/3584310541 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c6c00f040 con 0x7f3c801037a0 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.426+0000 7f3c77e00700 1 -- 172.21.15.63:0/3584310541 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c6c00d9a0 con 0x7f3c801037a0 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.426+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c80119e20 con 0x7f3c801037a0 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.426+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8011a2a0 con 0x7f3c801037a0 2024-03-23T12:11:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.427+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c800fe550 con 0x7f3c801037a0 2024-03-23T12:11:23.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.428+0000 7f3c77e00700 1 -- 172.21.15.63:0/3584310541 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3c6c00dc10 con 0x7f3c801037a0 2024-03-23T12:11:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.429+0000 7f3c77e00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c6406c7c0 0x7f3c6406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.429+0000 7f3c77e00700 1 -- 172.21.15.63:0/3584310541 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3c6c08b860 con 0x7f3c801037a0 2024-03-23T12:11:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.429+0000 7f3c7fe00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c6406c7c0 0x7f3c6406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:23.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.430+0000 7f3c7fe00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c6406c7c0 0x7f3c6406ec70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3c70004560 tx=0x7f3c70009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:23.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.433+0000 7f3c77e00700 1 -- 172.21.15.63:0/3584310541 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3c6c055a60 con 0x7f3c801037a0 2024-03-23T12:11:23.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.694+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3c8004bed0 con 0x7f3c801037a0 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.694+0000 7f3c77e00700 1 -- 172.21.15.63:0/3584310541 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f3c6c059080 con 0x7f3c801037a0 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:11:23.675 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:11:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:23.677 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:23.678 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:23.678 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:23.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.696+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c6406c7c0 msgr2=0x7f3c6406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.696+0000 7f3c86c00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c6406c7c0 0x7f3c6406ec70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3c70004560 tx=0x7f3c70009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:23.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.696+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c801037a0 msgr2=0x7f3c80070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.696+0000 7f3c86c00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c801037a0 0x7f3c80070800 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c002a50 tx=0x7f3c6c004600 comp rx=0 tx=0).stop 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.697+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 shutdown_connections 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.697+0000 7f3c86c00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c6406c7c0 0x7f3c6406ec70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.697+0000 7f3c86c00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c80102e00 0x7f3c800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.697+0000 7f3c86c00700 1 --2- 172.21.15.63:0/3584310541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c801037a0 0x7f3c80070800 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.697+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 >> 172.21.15.63:0/3584310541 conn(0x7f3c800f89b0 msgr2=0x7f3c800f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.697+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 shutdown_connections 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.697+0000 7f3c86c00700 1 -- 172.21.15.63:0/3584310541 wait complete. 2024-03-23T12:11:23.679 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 -- 172.21.15.63:0/3771047678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 msgr2=0x7f18b40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 --2- 172.21.15.63:0/3771047678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b40fce10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f18a0004940 tx=0x7f18a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 -- 172.21.15.63:0/3771047678 shutdown_connections 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 --2- 172.21.15.63:0/3771047678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b40fce10 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 --2- 172.21.15.63:0/3771047678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18b4102e00 0x7f18b41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 -- 172.21.15.63:0/3771047678 >> 172.21.15.63:0/3771047678 conn(0x7f18b40f89b0 msgr2=0x7f18b40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 -- 172.21.15.63:0/3771047678 shutdown_connections 2024-03-23T12:11:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.842+0000 7f18ba400700 1 -- 172.21.15.63:0/3771047678 wait complete. 2024-03-23T12:11:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.843+0000 7f18ba400700 1 Processor -- start 2024-03-23T12:11:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.843+0000 7f18ba400700 1 -- start start 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.843+0000 7f18ba400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18b4102e00 0x7f18b4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.843+0000 7f18ba400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.843+0000 7f18ba400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4114690 con 0x7f18b41037a0 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.843+0000 7f18ba400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b410da80 con 0x7f18b4102e00 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b4114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49746/0 (socket says 172.21.15.63:49746) 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b2a00700 1 -- 172.21.15.63:0/1033381905 learned_addr learned my addr 172.21.15.63:0/1033381905 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b3400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18b4102e00 0x7f18b4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b2a00700 1 -- 172.21.15.63:0/1033381905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18b4102e00 msgr2=0x7f18b4113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b2a00700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18b4102e00 0x7f18b4113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b2a00700 1 -- 172.21.15.63:0/1033381905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0005040 con 0x7f18b41037a0 2024-03-23T12:11:23.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18b2a00700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b4114040 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009380 tx=0x7f18a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:23.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18ab400700 1 -- 172.21.15.63:0/1033381905 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a000e040 con 0x7f18b41037a0 2024-03-23T12:11:23.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18ab400700 1 -- 172.21.15.63:0/1033381905 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18a0013420 con 0x7f18b41037a0 2024-03-23T12:11:23.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.844+0000 7f18ab400700 1 -- 172.21.15.63:0/1033381905 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a001c630 con 0x7f18b41037a0 2024-03-23T12:11:23.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.845+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b410dd00 con 0x7f18b41037a0 2024-03-23T12:11:23.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.845+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b410e2c0 con 0x7f18b41037a0 2024-03-23T12:11:23.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.846+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b40fe550 con 0x7f18b41037a0 2024-03-23T12:11:23.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.847+0000 7f18ab400700 1 -- 172.21.15.63:0/1033381905 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f18a00139b0 con 0x7f18b41037a0 2024-03-23T12:11:23.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.847+0000 7f18ab400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f189806c7c0 0x7f189806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:23.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.848+0000 7f18ab400700 1 -- 172.21.15.63:0/1033381905 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f18a008cbb0 con 0x7f18b41037a0 2024-03-23T12:11:23.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.848+0000 7f18b3400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f189806c7c0 0x7f189806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:23.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.848+0000 7f18b3400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f189806c7c0 0x7f189806ec70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f18a4004f40 tx=0x7f18a400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:23.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:23.852+0000 7f18ab400700 1 -- 172.21.15.63:0/1033381905 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f18a005a8c0 con 0x7f18b41037a0 2024-03-23T12:11:24.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.086+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b4109da0 con 0x7f189806c7c0 2024-03-23T12:11:24.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.088+0000 7f18ab400700 1 -- 172.21.15.63:0/1033381905 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f18b4109da0 con 0x7f189806c7c0 2024-03-23T12:11:24.068 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:11:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:11:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:11:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:11:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:11:24.069 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f189806c7c0 msgr2=0x7f189806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f189806c7c0 0x7f189806ec70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f18a4004f40 tx=0x7f18a400b040 comp rx=0 tx=0).stop 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 msgr2=0x7f18b4114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b4114040 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009380 tx=0x7f18a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 shutdown_connections 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f189806c7c0 0x7f189806ec70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18b4102e00 0x7f18b4113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 --2- 172.21.15.63:0/1033381905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18b41037a0 0x7f18b4114040 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 >> 172.21.15.63:0/1033381905 conn(0x7f18b40f89b0 msgr2=0x7f18b40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 shutdown_connections 2024-03-23T12:11:24.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.090+0000 7f18ba400700 1 -- 172.21.15.63:0/1033381905 wait complete. 2024-03-23T12:11:24.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 -- 172.21.15.63:0/4088465809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc102e00 msgr2=0x7f74dc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:24.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 --2- 172.21.15.63:0/4088465809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc102e00 0x7f74dc1031d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f74cc004940 tx=0x7f74cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:24.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 -- 172.21.15.63:0/4088465809 shutdown_connections 2024-03-23T12:11:24.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 --2- 172.21.15.63:0/4088465809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74dc1037a0 0x7f74dc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 --2- 172.21.15.63:0/4088465809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc102e00 0x7f74dc1031d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 -- 172.21.15.63:0/4088465809 >> 172.21.15.63:0/4088465809 conn(0x7f74dc0f89b0 msgr2=0x7f74dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:24.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 -- 172.21.15.63:0/4088465809 shutdown_connections 2024-03-23T12:11:24.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.227+0000 7f74e2600700 1 -- 172.21.15.63:0/4088465809 wait complete. 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.228+0000 7f74e2600700 1 Processor -- start 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.228+0000 7f74e2600700 1 -- start start 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.228+0000 7f74e2600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74dc102e00 0x7f74dc115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.228+0000 7f74e2600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc1037a0 0x7f74dc116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.228+0000 7f74e2600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc116860 con 0x7f74dc1037a0 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.228+0000 7f74e2600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc10fc30 con 0x7f74dc102e00 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.228+0000 7f74daa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc1037a0 0x7f74dc116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74daa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc1037a0 0x7f74dc116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49770/0 (socket says 172.21.15.63:49770) 2024-03-23T12:11:24.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74daa00700 1 -- 172.21.15.63:0/4016655337 learned_addr learned my addr 172.21.15.63:0/4016655337 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:24.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74db400700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74dc102e00 0x7f74dc115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:24.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74daa00700 1 -- 172.21.15.63:0/4016655337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74dc102e00 msgr2=0x7f74dc115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:24.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74daa00700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74dc102e00 0x7f74dc115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74daa00700 1 -- 172.21.15.63:0/4016655337 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc005040 con 0x7f74dc1037a0 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74daa00700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc1037a0 0x7f74dc116210 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f74c8002a50 tx=0x7f74c8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74d3400700 1 -- 172.21.15.63:0/4016655337 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c8007cc0 con 0x7f74dc1037a0 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74d3400700 1 -- 172.21.15.63:0/4016655337 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74c800f040 con 0x7f74dc1037a0 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74d3400700 1 -- 172.21.15.63:0/4016655337 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c800d920 con 0x7f74dc1037a0 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.229+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc10ff10 con 0x7f74dc1037a0 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.230+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc1104d0 con 0x7f74dc1037a0 2024-03-23T12:11:24.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.231+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc0fe550 con 0x7f74dc1037a0 2024-03-23T12:11:24.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.232+0000 7f74d3400700 1 -- 172.21.15.63:0/4016655337 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f74c8009680 con 0x7f74dc1037a0 2024-03-23T12:11:24.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.232+0000 7f74d3400700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74c006c7d0 0x7f74c006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:24.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.232+0000 7f74d3400700 1 -- 172.21.15.63:0/4016655337 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f74c808b750 con 0x7f74dc1037a0 2024-03-23T12:11:24.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.233+0000 7f74db400700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74c006c7d0 0x7f74c006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:24.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.233+0000 7f74db400700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74c006c7d0 0x7f74c006ec80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009fd0 tx=0x7f74cc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:24.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.237+0000 7f74d3400700 1 -- 172.21.15.63:0/4016655337 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f74c80593e0 con 0x7f74dc1037a0 2024-03-23T12:11:24.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:24 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1599163642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:11:24.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:24 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3584310541' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:11:24.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:24 smithi063 ceph-mon[141124]: pgmap v608: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:24.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:24 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1599163642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:11:24.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:24 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3584310541' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:11:24.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:24 smithi111 ceph-mon[106899]: pgmap v608: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:24.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.543+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f74dc04bed0 con 0x7f74dc1037a0 2024-03-23T12:11:24.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.544+0000 7f74d3400700 1 -- 172.21.15.63:0/4016655337 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f74c8058f70 con 0x7f74dc1037a0 2024-03-23T12:11:24.524 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:11:24.525 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:11:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:11:24.525 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:11:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74c006c7d0 msgr2=0x7f74c006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74c006c7d0 0x7f74c006ec80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009fd0 tx=0x7f74cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc1037a0 msgr2=0x7f74dc116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc1037a0 0x7f74dc116210 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f74c8002a50 tx=0x7f74c8004600 comp rx=0 tx=0).stop 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 shutdown_connections 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74c006c7d0 0x7f74c006ec80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74dc102e00 0x7f74dc115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 --2- 172.21.15.63:0/4016655337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74dc1037a0 0x7f74dc116210 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 >> 172.21.15.63:0/4016655337 conn(0x7f74dc0f89b0 msgr2=0x7f74dc0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 shutdown_connections 2024-03-23T12:11:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:24.546+0000 7f74e2600700 1 -- 172.21.15.63:0/4016655337 wait complete. 2024-03-23T12:11:25.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:25 smithi063 ceph-mon[141124]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:25.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:25 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4016655337' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:11:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:25 smithi111 ceph-mon[106899]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:25.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:25 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4016655337' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:11:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:26 smithi063 ceph-mon[141124]: pgmap v609: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:26 smithi111 ceph-mon[106899]: pgmap v609: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:28.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:28 smithi063 ceph-mon[141124]: pgmap v610: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:28.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:28 smithi111 ceph-mon[106899]: pgmap v610: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:30 smithi063 ceph-mon[141124]: pgmap v611: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:30 smithi111 ceph-mon[106899]: pgmap v611: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:32.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:32 smithi063 ceph-mon[141124]: pgmap v612: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:32.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:32 smithi111 ceph-mon[106899]: pgmap v612: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:34 smithi063 ceph-mon[141124]: pgmap v613: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:34.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:34 smithi111 ceph-mon[106899]: pgmap v613: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:36.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:36 smithi063 ceph-mon[141124]: pgmap v614: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:36.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:36 smithi111 ceph-mon[106899]: pgmap v614: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:38.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:38 smithi063 ceph-mon[141124]: pgmap v615: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:38.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:38 smithi111 ceph-mon[106899]: pgmap v615: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:40.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:40 smithi063 ceph-mon[141124]: pgmap v616: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:40.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:40 smithi111 ceph-mon[106899]: pgmap v616: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:41.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:41.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:42.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:42 smithi063 ceph-mon[141124]: pgmap v617: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:42.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:42 smithi111 ceph-mon[106899]: pgmap v617: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:44.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:44 smithi063 ceph-mon[141124]: pgmap v618: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:44.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:44 smithi111 ceph-mon[106899]: pgmap v618: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:46.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:46 smithi063 ceph-mon[141124]: pgmap v619: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:46.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:46 smithi111 ceph-mon[106899]: pgmap v619: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:48.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:48 smithi063 ceph-mon[141124]: pgmap v620: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:48.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:48 smithi111 ceph-mon[106899]: pgmap v620: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:50.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:50 smithi063 ceph-mon[141124]: pgmap v621: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:50.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:50 smithi111 ceph-mon[106899]: pgmap v621: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:52.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:52 smithi063 ceph-mon[141124]: pgmap v622: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:52.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:52 smithi111 ceph-mon[106899]: pgmap v622: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:54.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:54 smithi063 ceph-mon[141124]: pgmap v623: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:54.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:54 smithi111 ceph-mon[106899]: pgmap v623: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 -- 172.21.15.63:0/3862604788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 msgr2=0x7f11980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 --2- 172.21.15.63:0/3862604788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f11980fce10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1184004940 tx=0x7f1184009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 -- 172.21.15.63:0/3862604788 shutdown_connections 2024-03-23T12:11:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 --2- 172.21.15.63:0/3862604788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f11980fce10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 --2- 172.21.15.63:0/3862604788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1198102e00 0x7f11981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 -- 172.21.15.63:0/3862604788 >> 172.21.15.63:0/3862604788 conn(0x7f11980f89b0 msgr2=0x7f11980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 -- 172.21.15.63:0/3862604788 shutdown_connections 2024-03-23T12:11:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.700+0000 7f119fa00700 1 -- 172.21.15.63:0/3862604788 wait complete. 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.701+0000 7f119fa00700 1 Processor -- start 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.701+0000 7f119fa00700 1 -- start start 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f119fa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1198102e00 0x7f1198115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f119fa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f1198116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f119fa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11981168b0 con 0x7f11981037a0 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f119fa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119810fc80 con 0x7f1198102e00 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f1197e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f1198116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f1197e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f1198116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35762/0 (socket says 172.21.15.63:35762) 2024-03-23T12:11:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f119d000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1198102e00 0x7f1198115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f1197e00700 1 -- 172.21.15.63:0/3849933075 learned_addr learned my addr 172.21.15.63:0/3849933075 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f1197e00700 1 -- 172.21.15.63:0/3849933075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1198102e00 msgr2=0x7f1198115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f1197e00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1198102e00 0x7f1198115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.702+0000 7f1197e00700 1 -- 172.21.15.63:0/3849933075 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1184005040 con 0x7f11981037a0 2024-03-23T12:11:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.703+0000 7f1197e00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f1198116260 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1184009390 tx=0x7f1184009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:54.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.703+0000 7f1194c00700 1 -- 172.21.15.63:0/3849933075 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f118400e040 con 0x7f11981037a0 2024-03-23T12:11:54.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.703+0000 7f1194c00700 1 -- 172.21.15.63:0/3849933075 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1184013420 con 0x7f11981037a0 2024-03-23T12:11:54.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.703+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119810ff00 con 0x7f11981037a0 2024-03-23T12:11:54.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.703+0000 7f1194c00700 1 -- 172.21.15.63:0/3849933075 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f118401c630 con 0x7f11981037a0 2024-03-23T12:11:54.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.703+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11981104c0 con 0x7f11981037a0 2024-03-23T12:11:54.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.704+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11980fe550 con 0x7f11981037a0 2024-03-23T12:11:54.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.705+0000 7f1194c00700 1 -- 172.21.15.63:0/3849933075 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f1184013a00 con 0x7f11981037a0 2024-03-23T12:11:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.706+0000 7f1194c00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f118006c7d0 0x7f118006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.706+0000 7f1194c00700 1 -- 172.21.15.63:0/3849933075 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f118408cb20 con 0x7f11981037a0 2024-03-23T12:11:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.707+0000 7f119d000700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f118006c7d0 0x7f118006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.707+0000 7f119d000700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f118006c7d0 0x7f118006ec80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f118c004f60 tx=0x7f118c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.711+0000 7f1194c00700 1 -- 172.21.15.63:0/3849933075 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f118405a830 con 0x7f11981037a0 2024-03-23T12:11:54.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.943+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1198109e30 con 0x7f118006c7d0 2024-03-23T12:11:54.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.944+0000 7f1194c00700 1 -- 172.21.15.63:0/3849933075 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1198109e30 con 0x7f118006c7d0 2024-03-23T12:11:54.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.946+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f118006c7d0 msgr2=0x7f118006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:54.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.946+0000 7f119fa00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f118006c7d0 0x7f118006ec80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f118c004f60 tx=0x7f118c00b040 comp rx=0 tx=0).stop 2024-03-23T12:11:54.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.946+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 msgr2=0x7f1198116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:54.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.946+0000 7f119fa00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f1198116260 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1184009390 tx=0x7f1184009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:54.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.947+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 shutdown_connections 2024-03-23T12:11:54.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.947+0000 7f119fa00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f118006c7d0 0x7f118006ec80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:54.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.947+0000 7f119fa00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1198102e00 0x7f1198115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:54.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.947+0000 7f119fa00700 1 --2- 172.21.15.63:0/3849933075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11981037a0 0x7f1198116260 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:54.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.947+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 >> 172.21.15.63:0/3849933075 conn(0x7f11980f89b0 msgr2=0x7f1198070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:54.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.947+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 shutdown_connections 2024-03-23T12:11:54.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:54.947+0000 7f119fa00700 1 -- 172.21.15.63:0/3849933075 wait complete. 2024-03-23T12:11:54.941 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 -- 172.21.15.63:0/840074574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f4102e20 msgr2=0x7fd2f41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 --2- 172.21.15.63:0/840074574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f4102e20 0x7fd2f41031f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4004940 tx=0x7fd2e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 -- 172.21.15.63:0/840074574 shutdown_connections 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 --2- 172.21.15.63:0/840074574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2f41037c0 0x7fd2f40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 --2- 172.21.15.63:0/840074574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f4102e20 0x7fd2f41031f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 -- 172.21.15.63:0/840074574 >> 172.21.15.63:0/840074574 conn(0x7fd2f40f89d0 msgr2=0x7fd2f40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 -- 172.21.15.63:0/840074574 shutdown_connections 2024-03-23T12:11:55.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.085+0000 7fd2f9400700 1 -- 172.21.15.63:0/840074574 wait complete. 2024-03-23T12:11:55.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.086+0000 7fd2f9400700 1 Processor -- start 2024-03-23T12:11:55.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.086+0000 7fd2f9400700 1 -- start start 2024-03-23T12:11:55.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.086+0000 7fd2f9400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2f4102e20 0x7fd2f4113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.086+0000 7fd2f9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f41037c0 0x7fd2f4113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.086+0000 7fd2f9400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f4114610 con 0x7fd2f41037c0 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.086+0000 7fd2f9400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f410da00 con 0x7fd2f4102e20 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f41037c0 0x7fd2f4113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f41037c0 0x7fd2f4113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35772/0 (socket says 172.21.15.63:35772) 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f1e00700 1 -- 172.21.15.63:0/1238348114 learned_addr learned my addr 172.21.15.63:0/1238348114 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f2800700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2f4102e20 0x7fd2f4113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f1e00700 1 -- 172.21.15.63:0/1238348114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2f4102e20 msgr2=0x7fd2f4113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f1e00700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2f4102e20 0x7fd2f4113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f1e00700 1 -- 172.21.15.63:0/1238348114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e4005040 con 0x7fd2f41037c0 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f2800700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2f4102e20 0x7fd2f4113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.087+0000 7fd2f1e00700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f41037c0 0x7fd2f4113fc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000eb40 tx=0x7fd2e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.088+0000 7fd2eaa00700 1 -- 172.21.15.63:0/1238348114 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e0007cc0 con 0x7fd2f41037c0 2024-03-23T12:11:55.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.088+0000 7fd2eaa00700 1 -- 172.21.15.63:0/1238348114 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2e000f040 con 0x7fd2f41037c0 2024-03-23T12:11:55.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.088+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f410dc80 con 0x7fd2f41037c0 2024-03-23T12:11:55.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.088+0000 7fd2eaa00700 1 -- 172.21.15.63:0/1238348114 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e0009570 con 0x7fd2f41037c0 2024-03-23T12:11:55.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.088+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f410e240 con 0x7fd2f41037c0 2024-03-23T12:11:55.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.089+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f40fe570 con 0x7fd2f41037c0 2024-03-23T12:11:55.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.090+0000 7fd2eaa00700 1 -- 172.21.15.63:0/1238348114 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd2e00096d0 con 0x7fd2f41037c0 2024-03-23T12:11:55.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.092+0000 7fd2eaa00700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd2d806c7d0 0x7fd2d806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.092+0000 7fd2f2800700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd2d806c7d0 0x7fd2d806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.092+0000 7fd2f2800700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd2d806c7d0 0x7fd2d806ec80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400b0d0 tx=0x7fd2e4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:55.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.093+0000 7fd2eaa00700 1 -- 172.21.15.63:0/1238348114 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd2e0007e20 con 0x7fd2f41037c0 2024-03-23T12:11:55.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.098+0000 7fd2eaa00700 1 -- 172.21.15.63:0/1238348114 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd2e0059310 con 0x7fd2f41037c0 2024-03-23T12:11:55.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.330+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f4109e80 con 0x7fd2d806c7d0 2024-03-23T12:11:55.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.332+0000 7fd2eaa00700 1 -- 172.21.15.63:0/1238348114 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd2e4007de0 con 0x7fd2d806c7d0 2024-03-23T12:11:55.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd2d806c7d0 msgr2=0x7fd2d806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd2d806c7d0 0x7fd2d806ec80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400b0d0 tx=0x7fd2e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f41037c0 msgr2=0x7fd2f4113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f41037c0 0x7fd2f4113fc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000eb40 tx=0x7fd2e000b040 comp rx=0 tx=0).stop 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 shutdown_connections 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd2d806c7d0 0x7fd2d806ec80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2f4102e20 0x7fd2f4113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 --2- 172.21.15.63:0/1238348114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2f41037c0 0x7fd2f4113fc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 >> 172.21.15.63:0/1238348114 conn(0x7fd2f40f89d0 msgr2=0x7fd2f4105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 shutdown_connections 2024-03-23T12:11:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.334+0000 7fd2f9400700 1 -- 172.21.15.63:0/1238348114 wait complete. 2024-03-23T12:11:55.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:55 smithi063 ceph-mon[141124]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:55 smithi111 ceph-mon[106899]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 -- 172.21.15.63:0/2436301814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 msgr2=0x7f6fd41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 --2- 172.21.15.63:0/2436301814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd41031d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4004940 tx=0x7f6fc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 -- 172.21.15.63:0/2436301814 shutdown_connections 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 --2- 172.21.15.63:0/2436301814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd41037a0 0x7f6fd40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 --2- 172.21.15.63:0/2436301814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd41031d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 -- 172.21.15.63:0/2436301814 >> 172.21.15.63:0/2436301814 conn(0x7f6fd40f89b0 msgr2=0x7f6fd40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 -- 172.21.15.63:0/2436301814 shutdown_connections 2024-03-23T12:11:55.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.471+0000 7f6fda400700 1 -- 172.21.15.63:0/2436301814 wait complete. 2024-03-23T12:11:55.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.472+0000 7f6fda400700 1 Processor -- start 2024-03-23T12:11:55.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.472+0000 7f6fda400700 1 -- start start 2024-03-23T12:11:55.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.472+0000 7f6fda400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd4113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.472+0000 7f6fda400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd41037a0 0x7f6fd4114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.472+0000 7f6fda400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd41146b0 con 0x7f6fd4102e00 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.472+0000 7f6fda400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd410daa0 con 0x7f6fd41037a0 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd4113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd4113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35804/0 (socket says 172.21.15.63:35804) 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd3400700 1 -- 172.21.15.63:0/3860604549 learned_addr learned my addr 172.21.15.63:0/3860604549 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd2a00700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd41037a0 0x7f6fd4114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd3400700 1 -- 172.21.15.63:0/3860604549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd41037a0 msgr2=0x7f6fd4114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd3400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd41037a0 0x7f6fd4114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd3400700 1 -- 172.21.15.63:0/3860604549 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4005040 con 0x7f6fd4102e00 2024-03-23T12:11:55.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd2a00700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd41037a0 0x7f6fd4114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:11:55.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fd3400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd4113b20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009390 tx=0x7f6fc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:55.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.473+0000 7f6fcb400700 1 -- 172.21.15.63:0/3860604549 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc400e040 con 0x7f6fd4102e00 2024-03-23T12:11:55.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.474+0000 7f6fcb400700 1 -- 172.21.15.63:0/3860604549 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fc4013420 con 0x7f6fd4102e00 2024-03-23T12:11:55.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.474+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd410dd20 con 0x7f6fd4102e00 2024-03-23T12:11:55.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.474+0000 7f6fcb400700 1 -- 172.21.15.63:0/3860604549 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc401c630 con 0x7f6fd4102e00 2024-03-23T12:11:55.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.474+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd410e2e0 con 0x7f6fd4102e00 2024-03-23T12:11:55.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.475+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40fe550 con 0x7f6fd4102e00 2024-03-23T12:11:55.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.476+0000 7f6fcb400700 1 -- 172.21.15.63:0/3860604549 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6fc40139e0 con 0x7f6fd4102e00 2024-03-23T12:11:55.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.477+0000 7f6fcb400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6fb806c7d0 0x7f6fb806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.477+0000 7f6fcb400700 1 -- 172.21.15.63:0/3860604549 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6fc408cc90 con 0x7f6fd4102e00 2024-03-23T12:11:55.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.477+0000 7f6fd2a00700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6fb806c7d0 0x7f6fb806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.478+0000 7f6fd2a00700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6fb806c7d0 0x7f6fb806ec80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0007fd0 tx=0x7f6fc0009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:55.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.481+0000 7f6fcb400700 1 -- 172.21.15.63:0/3860604549 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6fc405aa50 con 0x7f6fd4102e00 2024-03-23T12:11:55.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.709+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fd4109e30 con 0x7f6fb806c7d0 2024-03-23T12:11:55.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.719+0000 7f6fcb400700 1 -- 172.21.15.63:0/3860604549 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f6fd4109e30 con 0x7f6fb806c7d0 2024-03-23T12:11:55.698 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:11:55.698 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (21m) 13m ago 34m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (18m) 13m ago 34m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (18m) 13m ago 33m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (20m) 13m ago 34m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (14m) 13m ago 28m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (13m) 13m ago 27m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (27m) 13m ago 27m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (13m) 13m ago 27m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (23m) 13m ago 35m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (22m) 13m ago 33m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (19m) 13m ago 35m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (19m) 13m ago 33m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (22m) 13m ago 34m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (21m) 13m ago 33m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:11:55.699 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (17m) 13m ago 32m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:11:55.700 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (17m) 13m ago 31m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:11:55.700 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (16m) 13m ago 31m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:11:55.700 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (15m) 13m ago 30m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:11:55.700 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (15m) 13m ago 30m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:11:55.700 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (14m) 13m ago 29m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:11:55.700 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (21m) 13m ago 33m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:11:55.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6fb806c7d0 msgr2=0x7f6fb806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6fb806c7d0 0x7f6fb806ec80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0007fd0 tx=0x7f6fc0009040 comp rx=0 tx=0).stop 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 msgr2=0x7f6fd4113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd4113b20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009390 tx=0x7f6fc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 shutdown_connections 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6fb806c7d0 0x7f6fb806ec80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd4102e00 0x7f6fd4113b20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 --2- 172.21.15.63:0/3860604549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd41037a0 0x7f6fd4114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 >> 172.21.15.63:0/3860604549 conn(0x7f6fd40f89b0 msgr2=0x7f6fd40f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 shutdown_connections 2024-03-23T12:11:55.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.721+0000 7f6fda400700 1 -- 172.21.15.63:0/3860604549 wait complete. 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.858+0000 7ff427000700 1 -- 172.21.15.63:0/937815937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 msgr2=0x7ff4201031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.858+0000 7ff427000700 1 --2- 172.21.15.63:0/937815937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff4201031d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff410004940 tx=0x7ff410009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.858+0000 7ff427000700 1 -- 172.21.15.63:0/937815937 shutdown_connections 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.858+0000 7ff427000700 1 --2- 172.21.15.63:0/937815937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4201037a0 0x7ff4200fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.858+0000 7ff427000700 1 --2- 172.21.15.63:0/937815937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff4201031d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.858+0000 7ff427000700 1 -- 172.21.15.63:0/937815937 >> 172.21.15.63:0/937815937 conn(0x7ff4200f89b0 msgr2=0x7ff4200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.859+0000 7ff427000700 1 -- 172.21.15.63:0/937815937 shutdown_connections 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.859+0000 7ff427000700 1 -- 172.21.15.63:0/937815937 wait complete. 2024-03-23T12:11:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.859+0000 7ff427000700 1 Processor -- start 2024-03-23T12:11:55.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff427000700 1 -- start start 2024-03-23T12:11:55.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff427000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff420113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff427000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4201037a0 0x7ff420114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff427000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4201146b0 con 0x7ff420102e00 2024-03-23T12:11:55.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff427000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42010daa0 con 0x7ff4201037a0 2024-03-23T12:11:55.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff41fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff420113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff41fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff420113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35808/0 (socket says 172.21.15.63:35808) 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff41fe00700 1 -- 172.21.15.63:0/1986129121 learned_addr learned my addr 172.21.15.63:0/1986129121 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.860+0000 7ff41f400700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4201037a0 0x7ff420114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff41fe00700 1 -- 172.21.15.63:0/1986129121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4201037a0 msgr2=0x7ff420114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff41fe00700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4201037a0 0x7ff420114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff41fe00700 1 -- 172.21.15.63:0/1986129121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410005040 con 0x7ff420102e00 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff41f400700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4201037a0 0x7ff420114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff41fe00700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff420113b20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff410009360 tx=0x7ff410009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff417e00700 1 -- 172.21.15.63:0/1986129121 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41000e040 con 0x7ff420102e00 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff417e00700 1 -- 172.21.15.63:0/1986129121 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff410013420 con 0x7ff420102e00 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42010dd20 con 0x7ff420102e00 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff417e00700 1 -- 172.21.15.63:0/1986129121 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41001c5f0 con 0x7ff420102e00 2024-03-23T12:11:55.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.861+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42010e2e0 con 0x7ff420102e00 2024-03-23T12:11:55.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.862+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4200fe550 con 0x7ff420102e00 2024-03-23T12:11:55.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.863+0000 7ff417e00700 1 -- 172.21.15.63:0/1986129121 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff410013590 con 0x7ff420102e00 2024-03-23T12:11:55.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.864+0000 7ff417e00700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff40406c7d0 0x7ff40406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:55.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.864+0000 7ff417e00700 1 -- 172.21.15.63:0/1986129121 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff41008cbc0 con 0x7ff420102e00 2024-03-23T12:11:55.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.864+0000 7ff41f400700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff40406c7d0 0x7ff40406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:55.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.865+0000 7ff41f400700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff40406c7d0 0x7ff40406ec80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009fd0 tx=0x7ff40c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:55.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:55.869+0000 7ff417e00700 1 -- 172.21.15.63:0/1986129121 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff41005a990 con 0x7ff420102e00 2024-03-23T12:11:56.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.172+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff420109e20 con 0x7ff420102e00 2024-03-23T12:11:56.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.173+0000 7ff417e00700 1 -- 172.21.15.63:0/1986129121 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7ff41005a520 con 0x7ff420102e00 2024-03-23T12:11:56.152 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:11:56.152 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:11:56.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:11:56.154 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:11:56.154 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:11:56.154 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:11:56.154 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:11:56.154 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff40406c7d0 msgr2=0x7ff40406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff40406c7d0 0x7ff40406ec80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009fd0 tx=0x7ff40c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 msgr2=0x7ff420113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff420113b20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff410009360 tx=0x7ff410009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 shutdown_connections 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff40406c7d0 0x7ff40406ec80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff420102e00 0x7ff420113b20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 --2- 172.21.15.63:0/1986129121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4201037a0 0x7ff420114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 >> 172.21.15.63:0/1986129121 conn(0x7ff4200f89b0 msgr2=0x7ff4200f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:56.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 shutdown_connections 2024-03-23T12:11:56.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.175+0000 7ff427000700 1 -- 172.21.15.63:0/1986129121 wait complete. 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 -- 172.21.15.63:0/962331410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 msgr2=0x7fa6a0070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 --2- 172.21.15.63:0/962331410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0070730 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa690004970 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 -- 172.21.15.63:0/962331410 shutdown_connections 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 --2- 172.21.15.63:0/962331410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a0070c70 0x7fa6a0109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 --2- 172.21.15.63:0/962331410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0070730 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 -- 172.21.15.63:0/962331410 >> 172.21.15.63:0/962331410 conn(0x7fa6a00f89b0 msgr2=0x7fa6a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 -- 172.21.15.63:0/962331410 shutdown_connections 2024-03-23T12:11:56.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.313+0000 7fa6a5000700 1 -- 172.21.15.63:0/962331410 wait complete. 2024-03-23T12:11:56.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.314+0000 7fa6a5000700 1 Processor -- start 2024-03-23T12:11:56.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.314+0000 7fa6a5000700 1 -- start start 2024-03-23T12:11:56.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.314+0000 7fa6a5000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.314+0000 7fa6a5000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a0070c70 0x7fa6a01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.314+0000 7fa6a5000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0116810 con 0x7fa6a0070360 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.314+0000 7fa6a5000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a010fbe0 con 0x7fa6a0070c70 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.314+0000 7fa69de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa69de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35830/0 (socket says 172.21.15.63:35830) 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa69de00700 1 -- 172.21.15.63:0/3340190493 learned_addr learned my addr 172.21.15.63:0/3340190493 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa69de00700 1 -- 172.21.15.63:0/3340190493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a0070c70 msgr2=0x7fa6a01161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa69de00700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a0070c70 0x7fa6a01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa69de00700 1 -- 172.21.15.63:0/3340190493 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa690005040 con 0x7fa6a0070360 2024-03-23T12:11:56.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa69de00700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0115c80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa690004f50 tx=0x7fa690009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:56.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa696000700 1 -- 172.21.15.63:0/3340190493 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69000e040 con 0x7fa6a0070360 2024-03-23T12:11:56.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa696000700 1 -- 172.21.15.63:0/3340190493 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa690013420 con 0x7fa6a0070360 2024-03-23T12:11:56.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa696000700 1 -- 172.21.15.63:0/3340190493 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69001c610 con 0x7fa6a0070360 2024-03-23T12:11:56.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.315+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a010fe60 con 0x7fa6a0070360 2024-03-23T12:11:56.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.316+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a01103a0 con 0x7fa6a0070360 2024-03-23T12:11:56.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.317+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a0103b90 con 0x7fa6a0070360 2024-03-23T12:11:56.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.318+0000 7fa696000700 1 -- 172.21.15.63:0/3340190493 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa690013590 con 0x7fa6a0070360 2024-03-23T12:11:56.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.318+0000 7fa696000700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa68406c7d0 0x7fa68406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:56.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.319+0000 7fa696000700 1 -- 172.21.15.63:0/3340190493 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa69008cc30 con 0x7fa6a0070360 2024-03-23T12:11:56.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.319+0000 7fa69d400700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa68406c7d0 0x7fa68406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:56.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.319+0000 7fa69d400700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa68406c7d0 0x7fa68406ec80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009b80 tx=0x7fa68c009d60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:56.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.323+0000 7fa696000700 1 -- 172.21.15.63:0/3340190493 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa69005a8c0 con 0x7fa6a0070360 2024-03-23T12:11:56.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.582+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa6a0111330 con 0x7fa6a0070360 2024-03-23T12:11:56.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.583+0000 7fa696000700 1 -- 172.21.15.63:0/3340190493 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fa69005a450 con 0x7fa6a0070360 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:11:56.563 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:11:56.564 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:56.565 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:11:56.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa68406c7d0 msgr2=0x7fa68406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa68406c7d0 0x7fa68406ec80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009b80 tx=0x7fa68c009d60 comp rx=0 tx=0).stop 2024-03-23T12:11:56.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 msgr2=0x7fa6a0115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0115c80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa690004f50 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:56.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 shutdown_connections 2024-03-23T12:11:56.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa68406c7d0 0x7fa68406ec80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a0070360 0x7fa6a0115c80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 --2- 172.21.15.63:0/3340190493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a0070c70 0x7fa6a01161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 >> 172.21.15.63:0/3340190493 conn(0x7fa6a00f89b0 msgr2=0x7fa6a01015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:56.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.585+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 shutdown_connections 2024-03-23T12:11:56.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.586+0000 7fa6a5000700 1 -- 172.21.15.63:0/3340190493 wait complete. 2024-03-23T12:11:56.567 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:11:56.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.724+0000 7fee5dc00700 1 -- 172.21.15.63:0/2729632111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 msgr2=0x7fee580fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.724+0000 7fee5dc00700 1 --2- 172.21.15.63:0/2729632111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee580fce10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fee44004940 tx=0x7fee44009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:56.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.724+0000 7fee5dc00700 1 -- 172.21.15.63:0/2729632111 shutdown_connections 2024-03-23T12:11:56.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.724+0000 7fee5dc00700 1 --2- 172.21.15.63:0/2729632111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee580fce10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.724+0000 7fee5dc00700 1 --2- 172.21.15.63:0/2729632111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee58102e00 0x7fee581031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.724+0000 7fee5dc00700 1 -- 172.21.15.63:0/2729632111 >> 172.21.15.63:0/2729632111 conn(0x7fee580f89b0 msgr2=0x7fee580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:56.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.725+0000 7fee5dc00700 1 -- 172.21.15.63:0/2729632111 shutdown_connections 2024-03-23T12:11:56.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.725+0000 7fee5dc00700 1 -- 172.21.15.63:0/2729632111 wait complete. 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.725+0000 7fee5dc00700 1 Processor -- start 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.725+0000 7fee5dc00700 1 -- start start 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee5dc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee58102e00 0x7fee58115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee5dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee58116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee5dc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee58116880 con 0x7fee581037a0 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee5dc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5810fc50 con 0x7fee58102e00 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee56000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee58116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee56000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee58116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35846/0 (socket says 172.21.15.63:35846) 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee56000700 1 -- 172.21.15.63:0/798829277 learned_addr learned my addr 172.21.15.63:0/798829277 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee56a00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee58102e00 0x7fee58115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:56.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee56000700 1 -- 172.21.15.63:0/798829277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee58102e00 msgr2=0x7fee58115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.726+0000 7fee56000700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee58102e00 0x7fee58115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee56000700 1 -- 172.21.15.63:0/798829277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44005040 con 0x7fee581037a0 2024-03-23T12:11:56.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee56a00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee58102e00 0x7fee58115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:11:56.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee56000700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee58116230 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fee440093b0 tx=0x7fee44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:56.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee4ea00700 1 -- 172.21.15.63:0/798829277 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee4400e040 con 0x7fee581037a0 2024-03-23T12:11:56.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee4ea00700 1 -- 172.21.15.63:0/798829277 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee44013420 con 0x7fee581037a0 2024-03-23T12:11:56.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee4ea00700 1 -- 172.21.15.63:0/798829277 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee4401c630 con 0x7fee581037a0 2024-03-23T12:11:56.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5810fed0 con 0x7fee581037a0 2024-03-23T12:11:56.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.727+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee58110490 con 0x7fee581037a0 2024-03-23T12:11:56.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.728+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee580fe550 con 0x7fee581037a0 2024-03-23T12:11:56.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.729+0000 7fee4ea00700 1 -- 172.21.15.63:0/798829277 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fee440139e0 con 0x7fee581037a0 2024-03-23T12:11:56.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.730+0000 7fee4ea00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee3c06c7c0 0x7fee3c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:56.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.730+0000 7fee4ea00700 1 -- 172.21.15.63:0/798829277 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fee4408cd40 con 0x7fee581037a0 2024-03-23T12:11:56.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.730+0000 7fee56a00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee3c06c7c0 0x7fee3c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:56.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.731+0000 7fee56a00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee3c06c7c0 0x7fee3c06ec70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fee48009fd0 tx=0x7fee48009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:56.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.735+0000 7fee4ea00700 1 -- 172.21.15.63:0/798829277 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fee4405aa80 con 0x7fee581037a0 2024-03-23T12:11:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:56 smithi063 ceph-mon[141124]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:56 smithi063 ceph-mon[141124]: from='client.35170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:56 smithi063 ceph-mon[141124]: pgmap v624: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:56.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:56 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1986129121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:11:56.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:56 smithi111 ceph-mon[106899]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:56 smithi111 ceph-mon[106899]: from='client.35170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:11:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:56 smithi111 ceph-mon[106899]: pgmap v624: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:56.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:56 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1986129121' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:11:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.967+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee58109e10 con 0x7fee3c06c7c0 2024-03-23T12:11:56.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.969+0000 7fee4ea00700 1 -- 172.21.15.63:0/798829277 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fee58109e10 con 0x7fee3c06c7c0 2024-03-23T12:11:56.948 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:11:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:11:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:11:56.949 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:11:56.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee3c06c7c0 msgr2=0x7fee3c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee3c06c7c0 0x7fee3c06ec70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fee48009fd0 tx=0x7fee48009ee0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 msgr2=0x7fee58116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:56.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee58116230 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fee440093b0 tx=0x7fee44009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:56.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 shutdown_connections 2024-03-23T12:11:56.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee3c06c7c0 0x7fee3c06ec70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee58102e00 0x7fee58115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 --2- 172.21.15.63:0/798829277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee581037a0 0x7fee58116230 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 >> 172.21.15.63:0/798829277 conn(0x7fee580f89b0 msgr2=0x7fee580f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 shutdown_connections 2024-03-23T12:11:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:56.972+0000 7fee5dc00700 1 -- 172.21.15.63:0/798829277 wait complete. 2024-03-23T12:11:57.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.107+0000 7fbee1000700 1 -- 172.21.15.63:0/2136842592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fe850 msgr2=0x7fbedc106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:57.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.107+0000 7fbee1000700 1 --2- 172.21.15.63:0/2136842592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fe850 0x7fbedc106f20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbec8004970 tx=0x7fbec8009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:57.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.108+0000 7fbee1000700 1 -- 172.21.15.63:0/2136842592 shutdown_connections 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.108+0000 7fbee1000700 1 --2- 172.21.15.63:0/2136842592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fe850 0x7fbedc106f20 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.108+0000 7fbee1000700 1 --2- 172.21.15.63:0/2136842592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbedc0fdeb0 0x7fbedc0fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.108+0000 7fbee1000700 1 -- 172.21.15.63:0/2136842592 >> 172.21.15.63:0/2136842592 conn(0x7fbedc072bc0 msgr2=0x7fbedc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.108+0000 7fbee1000700 1 -- 172.21.15.63:0/2136842592 shutdown_connections 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.108+0000 7fbee1000700 1 -- 172.21.15.63:0/2136842592 wait complete. 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbee1000700 1 Processor -- start 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbee1000700 1 -- start start 2024-03-23T12:11:57.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbee1000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fdeb0 0x7fbedc115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:57.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbee1000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbedc0fe850 0x7fbedc1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:57.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbee1000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc116810 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbee1000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc10fc20 con 0x7fbedc0fe850 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbed9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fdeb0 0x7fbedc115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbed9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fdeb0 0x7fbedc115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35878/0 (socket says 172.21.15.63:35878) 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbed9400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbedc0fe850 0x7fbedc1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.109+0000 7fbed9e00700 1 -- 172.21.15.63:0/1496837521 learned_addr learned my addr 172.21.15.63:0/1496837521 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbed9e00700 1 -- 172.21.15.63:0/1496837521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbedc0fe850 msgr2=0x7fbedc1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbed9e00700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbedc0fe850 0x7fbedc1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbed9e00700 1 -- 172.21.15.63:0/1496837521 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec8005040 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbed9e00700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fdeb0 0x7fbedc115c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbecc002a50 tx=0x7fbecc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:57.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbed2000700 1 -- 172.21.15.63:0/1496837521 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbecc007cc0 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbed2000700 1 -- 172.21.15.63:0/1496837521 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbecc00f040 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc10ff00 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbed2000700 1 -- 172.21.15.63:0/1496837521 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbecc00d9c0 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.110+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc110410 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.112+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc1069f0 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.113+0000 7fbed2000700 1 -- 172.21.15.63:0/1496837521 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbecc00db60 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.114+0000 7fbed2000700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbec006c7d0 0x7fbec006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:11:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.114+0000 7fbed2000700 1 -- 172.21.15.63:0/1496837521 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbecc08b040 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.114+0000 7fbed9400700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbec006c7d0 0x7fbec006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:11:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.115+0000 7fbed9400700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbec006c7d0 0x7fbec006ec80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbec800b370 tx=0x7fbec8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:11:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.119+0000 7fbed2000700 1 -- 172.21.15.63:0/1496837521 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbecc0552c0 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.416+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbedc109e40 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.417+0000 7fbed2000700 1 -- 172.21.15.63:0/1496837521 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fbecc0588e0 con 0x7fbedc0fdeb0 2024-03-23T12:11:57.397 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:11:57.397 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:11:57.397 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:11:57.397 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:11:57.397 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbec006c7d0 msgr2=0x7fbec006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbec006c7d0 0x7fbec006ec80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbec800b370 tx=0x7fbec8009e30 comp rx=0 tx=0).stop 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fdeb0 msgr2=0x7fbedc115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fdeb0 0x7fbedc115c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbecc002a50 tx=0x7fbecc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 shutdown_connections 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbec006c7d0 0x7fbec006ec80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbedc0fdeb0 0x7fbedc115c80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 --2- 172.21.15.63:0/1496837521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbedc0fe850 0x7fbedc1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:11:57.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.419+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 >> 172.21.15.63:0/1496837521 conn(0x7fbedc072bc0 msgr2=0x7fbedc0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:11:57.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.420+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 shutdown_connections 2024-03-23T12:11:57.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:11:57.420+0000 7fbee1000700 1 -- 172.21.15.63:0/1496837521 wait complete. 2024-03-23T12:11:57.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:57 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3340190493' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:11:57.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:57 smithi063 ceph-mon[141124]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:57.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:57 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1496837521' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:11:57.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:57 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3340190493' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:11:57.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:57 smithi111 ceph-mon[106899]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:11:57.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:57 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1496837521' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:11:58.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:11:58 smithi063 ceph-mon[141124]: pgmap v625: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:11:58.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:11:58 smithi111 ceph-mon[106899]: pgmap v625: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:00 smithi063 ceph-mon[141124]: pgmap v626: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:00 smithi111 ceph-mon[106899]: pgmap v626: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:02.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:02 smithi063 ceph-mon[141124]: pgmap v627: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:02.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:02 smithi111 ceph-mon[106899]: pgmap v627: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:04.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:04 smithi063 ceph-mon[141124]: pgmap v628: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:04.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:04 smithi111 ceph-mon[106899]: pgmap v628: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:06.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:06 smithi063 ceph-mon[141124]: pgmap v629: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:06.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:06 smithi111 ceph-mon[106899]: pgmap v629: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:08.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:08 smithi063 ceph-mon[141124]: pgmap v630: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:08.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:08 smithi111 ceph-mon[106899]: pgmap v630: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:10.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:10 smithi063 ceph-mon[141124]: pgmap v631: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:10.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:10 smithi111 ceph-mon[106899]: pgmap v631: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:11.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:11.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:12.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:12 smithi063 ceph-mon[141124]: pgmap v632: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:12.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:12 smithi111 ceph-mon[106899]: pgmap v632: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:14.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:14 smithi063 ceph-mon[141124]: pgmap v633: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:14.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:14 smithi111 ceph-mon[106899]: pgmap v633: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:16.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:16 smithi063 ceph-mon[141124]: pgmap v634: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:16.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:16 smithi111 ceph-mon[106899]: pgmap v634: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:18.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:18 smithi063 ceph-mon[141124]: pgmap v635: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:18.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:18 smithi111 ceph-mon[106899]: pgmap v635: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:20 smithi063 ceph-mon[141124]: pgmap v636: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:20.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:20 smithi111 ceph-mon[106899]: pgmap v636: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:22.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:22 smithi063 ceph-mon[141124]: pgmap v637: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:22.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:22 smithi111 ceph-mon[106899]: pgmap v637: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:24.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:24 smithi063 ceph-mon[141124]: pgmap v638: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:24.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:24 smithi111 ceph-mon[106899]: pgmap v638: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:26.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:26 smithi063 ceph-mon[141124]: pgmap v639: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:26.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:26 smithi111 ceph-mon[106899]: pgmap v639: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 -- 172.21.15.63:0/3145097235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94105990 msgr2=0x7f5d940ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 --2- 172.21.15.63:0/3145097235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94105990 0x7f5d940ff000 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5d80004940 tx=0x7f5d80009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 -- 172.21.15.63:0/3145097235 shutdown_connections 2024-03-23T12:12:27.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 --2- 172.21.15.63:0/3145097235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94105990 0x7f5d940ff000 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 --2- 172.21.15.63:0/3145097235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d94104ff0 0x7f5d941053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 -- 172.21.15.63:0/3145097235 >> 172.21.15.63:0/3145097235 conn(0x7f5d940faba0 msgr2=0x7f5d940fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:27.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 -- 172.21.15.63:0/3145097235 shutdown_connections 2024-03-23T12:12:27.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.575+0000 7f5d9a400700 1 -- 172.21.15.63:0/3145097235 wait complete. 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.576+0000 7f5d9a400700 1 Processor -- start 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.576+0000 7f5d9a400700 1 -- start start 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d9a400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94104ff0 0x7f5d94115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d9a400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d94105990 0x7f5d94115f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d9a400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d941165c0 con 0x7f5d94104ff0 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d9a400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9410fba0 con 0x7f5d94105990 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d93400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94104ff0 0x7f5d94115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:27.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d93400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94104ff0 0x7f5d94115a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38056/0 (socket says 172.21.15.63:38056) 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d93400700 1 -- 172.21.15.63:0/2786673310 learned_addr learned my addr 172.21.15.63:0/2786673310 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d92a00700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d94105990 0x7f5d94115f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d93400700 1 -- 172.21.15.63:0/2786673310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d94105990 msgr2=0x7f5d94115f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d93400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d94105990 0x7f5d94115f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.577+0000 7f5d93400700 1 -- 172.21.15.63:0/2786673310 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80005040 con 0x7f5d94104ff0 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.578+0000 7f5d93400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94104ff0 0x7f5d94115a30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400eb40 tx=0x7f5d8400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.578+0000 7f5d8b400700 1 -- 172.21.15.63:0/2786673310 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d84007cc0 con 0x7f5d94104ff0 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.578+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9410fe20 con 0x7f5d94104ff0 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.578+0000 7f5d8b400700 1 -- 172.21.15.63:0/2786673310 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d8400f040 con 0x7f5d94104ff0 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.578+0000 7f5d8b400700 1 -- 172.21.15.63:0/2786673310 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d8400c990 con 0x7f5d94104ff0 2024-03-23T12:12:27.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.578+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d941103e0 con 0x7f5d94104ff0 2024-03-23T12:12:27.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.579+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d94102300 con 0x7f5d94104ff0 2024-03-23T12:12:27.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.580+0000 7f5d8b400700 1 -- 172.21.15.63:0/2786673310 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5d8400cc00 con 0x7f5d94104ff0 2024-03-23T12:12:27.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.581+0000 7f5d8b400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5d7806c7c0 0x7f5d7806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:27.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.581+0000 7f5d8b400700 1 -- 172.21.15.63:0/2786673310 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5d8408b690 con 0x7f5d94104ff0 2024-03-23T12:12:27.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.582+0000 7f5d92a00700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5d7806c7c0 0x7f5d7806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:27.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.582+0000 7f5d92a00700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5d7806c7c0 0x7f5d7806ec70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5d941112d0 tx=0x7f5d80009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:27.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.586+0000 7f5d8b400700 1 -- 172.21.15.63:0/2786673310 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5d840593d0 con 0x7f5d94104ff0 2024-03-23T12:12:27.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.820+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d9410c000 con 0x7f5d7806c7c0 2024-03-23T12:12:27.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.821+0000 7f5d8b400700 1 -- 172.21.15.63:0/2786673310 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5d9410c000 con 0x7f5d7806c7c0 2024-03-23T12:12:27.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5d7806c7c0 msgr2=0x7f5d7806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:27.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5d7806c7c0 0x7f5d7806ec70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5d941112d0 tx=0x7f5d80009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:27.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94104ff0 msgr2=0x7f5d94115a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94104ff0 0x7f5d94115a30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400eb40 tx=0x7f5d8400b040 comp rx=0 tx=0).stop 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 shutdown_connections 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5d7806c7c0 0x7f5d7806ec70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d94104ff0 0x7f5d94115a30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 --2- 172.21.15.63:0/2786673310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d94105990 0x7f5d94115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 >> 172.21.15.63:0/2786673310 conn(0x7f5d940faba0 msgr2=0x7f5d940fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 shutdown_connections 2024-03-23T12:12:27.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.824+0000 7f5d9a400700 1 -- 172.21.15.63:0/2786673310 wait complete. 2024-03-23T12:12:27.820 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 -- 172.21.15.63:0/2807035904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 msgr2=0x7f8578106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 --2- 172.21.15.63:0/2807035904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578106f20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8564004970 tx=0x7f8564009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 -- 172.21.15.63:0/2807035904 shutdown_connections 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 --2- 172.21.15.63:0/2807035904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578106f20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 --2- 172.21.15.63:0/2807035904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85780fdeb0 0x7f85780fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 -- 172.21.15.63:0/2807035904 >> 172.21.15.63:0/2807035904 conn(0x7f8578072bc0 msgr2=0x7f8578072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 -- 172.21.15.63:0/2807035904 shutdown_connections 2024-03-23T12:12:27.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.963+0000 7f857d400700 1 -- 172.21.15.63:0/2807035904 wait complete. 2024-03-23T12:12:27.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.964+0000 7f857d400700 1 Processor -- start 2024-03-23T12:12:27.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.964+0000 7f857d400700 1 -- start start 2024-03-23T12:12:27.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.964+0000 7f857d400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85780fdeb0 0x7f8578115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.964+0000 7f857d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f857d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8578116790 con 0x7f85780fe850 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f857d400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857810fb80 con 0x7f85780fdeb0 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f8575e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f8575e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38062/0 (socket says 172.21.15.63:38062) 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f8575e00700 1 -- 172.21.15.63:0/2775728565 learned_addr learned my addr 172.21.15.63:0/2775728565 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f8576800700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85780fdeb0 0x7f8578115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f8575e00700 1 -- 172.21.15.63:0/2775728565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85780fdeb0 msgr2=0x7f8578115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f8575e00700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85780fdeb0 0x7f8578115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.965+0000 7f8575e00700 1 -- 172.21.15.63:0/2775728565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8564005040 con 0x7f85780fe850 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.966+0000 7f8575e00700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578116140 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8564004f50 tx=0x7f8564009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:27.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.966+0000 7f856ea00700 1 -- 172.21.15.63:0/2775728565 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f856400e040 con 0x7f85780fe850 2024-03-23T12:12:27.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.966+0000 7f856ea00700 1 -- 172.21.15.63:0/2775728565 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8564013420 con 0x7f85780fe850 2024-03-23T12:12:27.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.966+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857810fe00 con 0x7f85780fe850 2024-03-23T12:12:27.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.966+0000 7f856ea00700 1 -- 172.21.15.63:0/2775728565 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f856401c630 con 0x7f85780fe850 2024-03-23T12:12:27.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.966+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8578110340 con 0x7f85780fe850 2024-03-23T12:12:27.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.967+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85781069f0 con 0x7f85780fe850 2024-03-23T12:12:27.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.968+0000 7f856ea00700 1 -- 172.21.15.63:0/2775728565 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8564013590 con 0x7f85780fe850 2024-03-23T12:12:27.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.969+0000 7f856ea00700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f855c06c7d0 0x7f855c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:27.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.969+0000 7f856ea00700 1 -- 172.21.15.63:0/2775728565 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f856408cc00 con 0x7f85780fe850 2024-03-23T12:12:27.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.969+0000 7f8576800700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f855c06c7d0 0x7f855c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:27.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.970+0000 7f8576800700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f855c06c7d0 0x7f855c06ec80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8568004f40 tx=0x7f856800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:27.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:27.973+0000 7f856ea00700 1 -- 172.21.15.63:0/2775728565 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f856405a910 con 0x7f85780fe850 2024-03-23T12:12:28.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.205+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f85781158d0 con 0x7f855c06c7d0 2024-03-23T12:12:28.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.206+0000 7f856ea00700 1 -- 172.21.15.63:0/2775728565 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f85781158d0 con 0x7f855c06c7d0 2024-03-23T12:12:28.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f855c06c7d0 msgr2=0x7f855c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f855c06c7d0 0x7f855c06ec80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8568004f40 tx=0x7f856800b040 comp rx=0 tx=0).stop 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 msgr2=0x7f8578116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578116140 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8564004f50 tx=0x7f8564009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 shutdown_connections 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f855c06c7d0 0x7f855c06ec80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85780fdeb0 0x7f8578115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 --2- 172.21.15.63:0/2775728565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85780fe850 0x7f8578116140 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 >> 172.21.15.63:0/2775728565 conn(0x7f8578072bc0 msgr2=0x7f85780fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 shutdown_connections 2024-03-23T12:12:28.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.209+0000 7f857d400700 1 -- 172.21.15.63:0/2775728565 wait complete. 2024-03-23T12:12:28.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 -- 172.21.15.63:0/3426383849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c8102e00 msgr2=0x7f18c81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 --2- 172.21.15.63:0/3426383849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c8102e00 0x7f18c81031d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f18b8004940 tx=0x7f18b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:28.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 -- 172.21.15.63:0/3426383849 shutdown_connections 2024-03-23T12:12:28.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 --2- 172.21.15.63:0/3426383849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18c81037a0 0x7f18c80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 --2- 172.21.15.63:0/3426383849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c8102e00 0x7f18c81031d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 -- 172.21.15.63:0/3426383849 >> 172.21.15.63:0/3426383849 conn(0x7f18c80f89b0 msgr2=0x7f18c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:28.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 -- 172.21.15.63:0/3426383849 shutdown_connections 2024-03-23T12:12:28.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.346+0000 7f18cd400700 1 -- 172.21.15.63:0/3426383849 wait complete. 2024-03-23T12:12:28.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.347+0000 7f18cd400700 1 Processor -- start 2024-03-23T12:12:28.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.347+0000 7f18cd400700 1 -- start start 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.347+0000 7f18cd400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18c8102e00 0x7f18c8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.347+0000 7f18cd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c81037a0 0x7f18c8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.347+0000 7f18cd400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c8116880 con 0x7f18c81037a0 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.347+0000 7f18cd400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c810fc50 con 0x7f18c8102e00 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c81037a0 0x7f18c8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c81037a0 0x7f18c8116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38090/0 (socket says 172.21.15.63:38090) 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c5e00700 1 -- 172.21.15.63:0/1404177601 learned_addr learned my addr 172.21.15.63:0/1404177601 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c6800700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18c8102e00 0x7f18c8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c5e00700 1 -- 172.21.15.63:0/1404177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18c8102e00 msgr2=0x7f18c8115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c5e00700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18c8102e00 0x7f18c8115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c5e00700 1 -- 172.21.15.63:0/1404177601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b8005040 con 0x7f18c81037a0 2024-03-23T12:12:28.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.348+0000 7f18c5e00700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c81037a0 0x7f18c8116230 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f18b400c890 tx=0x7f18b40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:28.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.349+0000 7f18bea00700 1 -- 172.21.15.63:0/1404177601 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18b400a700 con 0x7f18c81037a0 2024-03-23T12:12:28.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.349+0000 7f18bea00700 1 -- 172.21.15.63:0/1404177601 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18b400ad10 con 0x7f18c81037a0 2024-03-23T12:12:28.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.349+0000 7f18bea00700 1 -- 172.21.15.63:0/1404177601 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18b4012e70 con 0x7f18c81037a0 2024-03-23T12:12:28.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.349+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c810ff30 con 0x7f18c81037a0 2024-03-23T12:12:28.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.349+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c81104f0 con 0x7f18c81037a0 2024-03-23T12:12:28.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.350+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c80fe550 con 0x7f18c81037a0 2024-03-23T12:12:28.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.351+0000 7f18bea00700 1 -- 172.21.15.63:0/1404177601 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f18b400a860 con 0x7f18c81037a0 2024-03-23T12:12:28.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.352+0000 7f18bea00700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f18ac06c7c0 0x7f18ac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:28.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.352+0000 7f18bea00700 1 -- 172.21.15.63:0/1404177601 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f18b408ba90 con 0x7f18c81037a0 2024-03-23T12:12:28.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.352+0000 7f18c6800700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f18ac06c7c0 0x7f18ac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:28.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.353+0000 7f18c6800700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f18ac06c7c0 0x7f18ac06ec70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18b8004f90 tx=0x7f18b8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:28.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.356+0000 7f18bea00700 1 -- 172.21.15.63:0/1404177601 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f18b40597a0 con 0x7f18c81037a0 2024-03-23T12:12:28.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:28 smithi063 ceph-mon[141124]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:28.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:28 smithi063 ceph-mon[141124]: pgmap v640: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:28.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:28 smithi111 ceph-mon[106899]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:28.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:28 smithi111 ceph-mon[106899]: pgmap v640: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:28.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.580+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f18c8109e30 con 0x7f18ac06c7c0 2024-03-23T12:12:28.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.590+0000 7f18bea00700 1 -- 172.21.15.63:0/1404177601 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f18c8109e30 con 0x7f18ac06c7c0 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (21m) 14m ago 34m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (18m) 14m ago 35m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (18m) 14m ago 33m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (21m) 14m ago 34m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (14m) 14m ago 28m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (14m) 14m ago 28m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (28m) 14m ago 28m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (14m) 14m ago 28m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (24m) 14m ago 36m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:12:28.571 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (22m) 14m ago 33m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (20m) 14m ago 36m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (19m) 14m ago 33m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (22m) 14m ago 35m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (22m) 14m ago 33m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (18m) 14m ago 32m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (17m) 14m ago 32m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (17m) 14m ago 31m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (16m) 14m ago 31m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (15m) 14m ago 30m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (15m) 14m ago 30m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:12:28.572 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (22m) 14m ago 34m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:12:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f18ac06c7c0 msgr2=0x7f18ac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f18ac06c7c0 0x7f18ac06ec70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18b8004f90 tx=0x7f18b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c81037a0 msgr2=0x7f18c8116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c81037a0 0x7f18c8116230 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f18b400c890 tx=0x7f18b40044c0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 shutdown_connections 2024-03-23T12:12:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f18ac06c7c0 0x7f18ac06ec70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18c8102e00 0x7f18c8115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 --2- 172.21.15.63:0/1404177601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18c81037a0 0x7f18c8116230 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.592+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 >> 172.21.15.63:0/1404177601 conn(0x7f18c80f89b0 msgr2=0x7f18c80706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.593+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 shutdown_connections 2024-03-23T12:12:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.593+0000 7f18cd400700 1 -- 172.21.15.63:0/1404177601 wait complete. 2024-03-23T12:12:28.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.728+0000 7f0d97000700 1 -- 172.21.15.63:0/1457733254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 msgr2=0x7f0d90070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.728+0000 7f0d97000700 1 --2- 172.21.15.63:0/1457733254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90070730 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f0d80004970 tx=0x7f0d80009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.729+0000 7f0d97000700 1 -- 172.21.15.63:0/1457733254 shutdown_connections 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.729+0000 7f0d97000700 1 --2- 172.21.15.63:0/1457733254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90070c70 0x7f0d90109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.729+0000 7f0d97000700 1 --2- 172.21.15.63:0/1457733254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90070730 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.729+0000 7f0d97000700 1 -- 172.21.15.63:0/1457733254 >> 172.21.15.63:0/1457733254 conn(0x7f0d900f89b0 msgr2=0x7f0d900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.729+0000 7f0d97000700 1 -- 172.21.15.63:0/1457733254 shutdown_connections 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.729+0000 7f0d97000700 1 -- 172.21.15.63:0/1457733254 wait complete. 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.730+0000 7f0d97000700 1 Processor -- start 2024-03-23T12:12:28.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.730+0000 7f0d97000700 1 -- start start 2024-03-23T12:12:28.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d97000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:28.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d97000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90070c70 0x7f0d901161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d97000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d90116810 con 0x7f0d90070360 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d97000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9010fbe0 con 0x7f0d90070c70 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d8fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d8fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38110/0 (socket says 172.21.15.63:38110) 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d8fe00700 1 -- 172.21.15.63:0/1185139458 learned_addr learned my addr 172.21.15.63:0/1185139458 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d8f400700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90070c70 0x7f0d901161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d8fe00700 1 -- 172.21.15.63:0/1185139458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90070c70 msgr2=0x7f0d901161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d8fe00700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90070c70 0x7f0d901161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:28.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.731+0000 7f0d8fe00700 1 -- 172.21.15.63:0/1185139458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80005040 con 0x7f0d90070360 2024-03-23T12:12:28.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.732+0000 7f0d8fe00700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90115c80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009360 tx=0x7f0d80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:28.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.732+0000 7f0d87e00700 1 -- 172.21.15.63:0/1185139458 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8000e040 con 0x7f0d90070360 2024-03-23T12:12:28.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.732+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9010fe60 con 0x7f0d90070360 2024-03-23T12:12:28.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.732+0000 7f0d87e00700 1 -- 172.21.15.63:0/1185139458 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d80004290 con 0x7f0d90070360 2024-03-23T12:12:28.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.732+0000 7f0d87e00700 1 -- 172.21.15.63:0/1185139458 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d80013d90 con 0x7f0d90070360 2024-03-23T12:12:28.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.732+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d901103a0 con 0x7f0d90070360 2024-03-23T12:12:28.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.733+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d90103b90 con 0x7f0d90070360 2024-03-23T12:12:28.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.734+0000 7f0d87e00700 1 -- 172.21.15.63:0/1185139458 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0d8000ba40 con 0x7f0d90070360 2024-03-23T12:12:28.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.735+0000 7f0d87e00700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d7406c7d0 0x7f0d7406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:28.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.735+0000 7f0d87e00700 1 -- 172.21.15.63:0/1185139458 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0d8008cdd0 con 0x7f0d90070360 2024-03-23T12:12:28.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.735+0000 7f0d8f400700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d7406c7d0 0x7f0d7406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:28.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.736+0000 7f0d8f400700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d7406c7d0 0x7f0d7406ec80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0d90111520 tx=0x7f0d7c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:28.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:28.740+0000 7f0d87e00700 1 -- 172.21.15.63:0/1185139458 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0d8005a9e0 con 0x7f0d90070360 2024-03-23T12:12:29.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.037+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0d9004bed0 con 0x7f0d90070360 2024-03-23T12:12:29.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.038+0000 7f0d87e00700 1 -- 172.21.15.63:0/1185139458 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f0d8007a020 con 0x7f0d90070360 2024-03-23T12:12:29.018 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:12:29.018 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:12:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:12:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:12:29.020 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:12:29.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d7406c7d0 msgr2=0x7f0d7406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d7406c7d0 0x7f0d7406ec80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0d90111520 tx=0x7f0d7c00b040 comp rx=0 tx=0).stop 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 msgr2=0x7f0d90115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90115c80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009360 tx=0x7f0d80009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 shutdown_connections 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d7406c7d0 0x7f0d7406ec80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d90070360 0x7f0d90115c80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 --2- 172.21.15.63:0/1185139458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d90070c70 0x7f0d901161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 >> 172.21.15.63:0/1185139458 conn(0x7f0d900f89b0 msgr2=0x7f0d901015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 shutdown_connections 2024-03-23T12:12:29.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.040+0000 7f0d97000700 1 -- 172.21.15.63:0/1185139458 wait complete. 2024-03-23T12:12:29.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 -- 172.21.15.63:0/385594646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 msgr2=0x7f99340fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 --2- 172.21.15.63:0/385594646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f99340fce10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f991c004940 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:29.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 -- 172.21.15.63:0/385594646 shutdown_connections 2024-03-23T12:12:29.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 --2- 172.21.15.63:0/385594646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f99340fce10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 --2- 172.21.15.63:0/385594646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9934102e00 0x7f99341031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 -- 172.21.15.63:0/385594646 >> 172.21.15.63:0/385594646 conn(0x7f99340f89b0 msgr2=0x7f99340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:29.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 -- 172.21.15.63:0/385594646 shutdown_connections 2024-03-23T12:12:29.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.178+0000 7f9938e00700 1 -- 172.21.15.63:0/385594646 wait complete. 2024-03-23T12:12:29.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.179+0000 7f9938e00700 1 Processor -- start 2024-03-23T12:12:29.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.180+0000 7f9938e00700 1 -- start start 2024-03-23T12:12:29.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.180+0000 7f9938e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9934102e00 0x7f99340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.180+0000 7f9938e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f9934070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.180+0000 7f9938e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934072210 con 0x7f99341037a0 2024-03-23T12:12:29.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.180+0000 7f9938e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934070d40 con 0x7f9934102e00 2024-03-23T12:12:29.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f992be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f9934070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f992be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f9934070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38122/0 (socket says 172.21.15.63:38122) 2024-03-23T12:12:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f992be00700 1 -- 172.21.15.63:0/1942700879 learned_addr learned my addr 172.21.15.63:0/1942700879 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:12:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f9931e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9934102e00 0x7f99340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f992be00700 1 -- 172.21.15.63:0/1942700879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9934102e00 msgr2=0x7f99340702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f992be00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9934102e00 0x7f99340702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f992be00700 1 -- 172.21.15.63:0/1942700879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c005040 con 0x7f99341037a0 2024-03-23T12:12:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.181+0000 7f992be00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f9934070800 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f991c008d50 tx=0x7f991c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.182+0000 7f992a000700 1 -- 172.21.15.63:0/1942700879 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c00e030 con 0x7f99341037a0 2024-03-23T12:12:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.182+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9934070fc0 con 0x7f99341037a0 2024-03-23T12:12:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.182+0000 7f992a000700 1 -- 172.21.15.63:0/1942700879 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f991c0093b0 con 0x7f99341037a0 2024-03-23T12:12:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.182+0000 7f992a000700 1 -- 172.21.15.63:0/1942700879 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c01cbb0 con 0x7f99341037a0 2024-03-23T12:12:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.182+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9934119f60 con 0x7f99341037a0 2024-03-23T12:12:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.183+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99340fe550 con 0x7f99341037a0 2024-03-23T12:12:29.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.184+0000 7f992a000700 1 -- 172.21.15.63:0/1942700879 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f991c013930 con 0x7f99341037a0 2024-03-23T12:12:29.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.185+0000 7f992a000700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f991806c7c0 0x7f991806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.185+0000 7f992a000700 1 -- 172.21.15.63:0/1942700879 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f991c08d2c0 con 0x7f99341037a0 2024-03-23T12:12:29.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.185+0000 7f9931e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f991806c7c0 0x7f991806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.186+0000 7f9931e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f991806c7c0 0x7f991806ec70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9924009b80 tx=0x7f992400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:29.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.190+0000 7f992a000700 1 -- 172.21.15.63:0/1942700879 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f991c057440 con 0x7f99341037a0 2024-03-23T12:12:29.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:29 smithi063 ceph-mon[141124]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:29.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:29 smithi063 ceph-mon[141124]: from='client.35198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:29.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:29 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1185139458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:12:29.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:29 smithi111 ceph-mon[106899]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:29.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:29 smithi111 ceph-mon[106899]: from='client.35198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:29.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:29 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1185139458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:12:29.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.445+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f993411a150 con 0x7f99341037a0 2024-03-23T12:12:29.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.445+0000 7f992a000700 1 -- 172.21.15.63:0/1942700879 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f991c05aa60 con 0x7f99341037a0 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:12:29.426 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:12:29.427 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:12:29.428 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:12:29.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.447+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f991806c7c0 msgr2=0x7f991806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.447+0000 7f9938e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f991806c7c0 0x7f991806ec70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9924009b80 tx=0x7f992400b040 comp rx=0 tx=0).stop 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.447+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 msgr2=0x7f9934070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.447+0000 7f9938e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f9934070800 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f991c008d50 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.448+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 shutdown_connections 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.448+0000 7f9938e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f991806c7c0 0x7f991806ec70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.448+0000 7f9938e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9934102e00 0x7f99340702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.448+0000 7f9938e00700 1 --2- 172.21.15.63:0/1942700879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99341037a0 0x7f9934070800 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.448+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 >> 172.21.15.63:0/1942700879 conn(0x7f99340f89b0 msgr2=0x7f99340f9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.448+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 shutdown_connections 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.448+0000 7f9938e00700 1 -- 172.21.15.63:0/1942700879 wait complete. 2024-03-23T12:12:29.430 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 -- 172.21.15.63:0/2378314121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 msgr2=0x7f57800fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 --2- 172.21.15.63:0/2378314121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f57800fce10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f576c004940 tx=0x7f576c009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 -- 172.21.15.63:0/2378314121 shutdown_connections 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 --2- 172.21.15.63:0/2378314121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f57800fce10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 --2- 172.21.15.63:0/2378314121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5780102e00 0x7f57801031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 -- 172.21.15.63:0/2378314121 >> 172.21.15.63:0/2378314121 conn(0x7f57800f89b0 msgr2=0x7f57800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 -- 172.21.15.63:0/2378314121 shutdown_connections 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.587+0000 7f5785a00700 1 -- 172.21.15.63:0/2378314121 wait complete. 2024-03-23T12:12:29.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f5785a00700 1 Processor -- start 2024-03-23T12:12:29.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f5785a00700 1 -- start start 2024-03-23T12:12:29.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f5785a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5780102e00 0x7f57800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f5785a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f5780070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f5785a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57800720b0 con 0x7f57801037a0 2024-03-23T12:12:29.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f5785a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5780072220 con 0x7f5780102e00 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f577e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f5780070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f577e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f5780070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38142/0 (socket says 172.21.15.63:38142) 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.588+0000 7f577e000700 1 -- 172.21.15.63:0/2984927077 learned_addr learned my addr 172.21.15.63:0/2984927077 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f577ea00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5780102e00 0x7f57800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f577e000700 1 -- 172.21.15.63:0/2984927077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5780102e00 msgr2=0x7f57800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f577e000700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5780102e00 0x7f57800702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f577e000700 1 -- 172.21.15.63:0/2984927077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c005040 con 0x7f57801037a0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f577e000700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f5780070800 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f576c009390 tx=0x7f576c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f5776a00700 1 -- 172.21.15.63:0/2984927077 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f576c00e040 con 0x7f57801037a0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f5776a00700 1 -- 172.21.15.63:0/2984927077 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f576c013420 con 0x7f57801037a0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5780070e00 con 0x7f57801037a0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.589+0000 7f5776a00700 1 -- 172.21.15.63:0/2984927077 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f576c01c630 con 0x7f57801037a0 2024-03-23T12:12:29.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.590+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578011e700 con 0x7f57801037a0 2024-03-23T12:12:29.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.591+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57800fe550 con 0x7f57801037a0 2024-03-23T12:12:29.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.592+0000 7f5776a00700 1 -- 172.21.15.63:0/2984927077 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f576c013a00 con 0x7f57801037a0 2024-03-23T12:12:29.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.592+0000 7f5776a00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f576406c7d0 0x7f576406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.593+0000 7f5776a00700 1 -- 172.21.15.63:0/2984927077 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f576c08cb20 con 0x7f57801037a0 2024-03-23T12:12:29.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.593+0000 7f577ea00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f576406c7d0 0x7f576406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.593+0000 7f577ea00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f576406c7d0 0x7f576406ec80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5770009fd0 tx=0x7f577000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:29.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.597+0000 7f5776a00700 1 -- 172.21.15.63:0/2984927077 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f576c05a830 con 0x7f57801037a0 2024-03-23T12:12:29.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.828+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f578004bed0 con 0x7f576406c7d0 2024-03-23T12:12:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.830+0000 7f5776a00700 1 -- 172.21.15.63:0/2984927077 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f578004bed0 con 0x7f576406c7d0 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:12:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:12:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:12:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:12:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:12:29.812 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:12:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f576406c7d0 msgr2=0x7f576406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f576406c7d0 0x7f576406ec80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5770009fd0 tx=0x7f577000b040 comp rx=0 tx=0).stop 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 msgr2=0x7f5780070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f5780070800 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f576c009390 tx=0x7f576c009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 shutdown_connections 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f576406c7d0 0x7f576406ec80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5780102e00 0x7f57800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 --2- 172.21.15.63:0/2984927077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57801037a0 0x7f5780070800 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 >> 172.21.15.63:0/2984927077 conn(0x7f57800f89b0 msgr2=0x7f57800f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 shutdown_connections 2024-03-23T12:12:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.832+0000 7f5785a00700 1 -- 172.21.15.63:0/2984927077 wait complete. 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 -- 172.21.15.63:0/1455170819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 msgr2=0x7f70cc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 --2- 172.21.15.63:0/1455170819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc0fce10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f70b4004940 tx=0x7f70b4009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 -- 172.21.15.63:0/1455170819 shutdown_connections 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 --2- 172.21.15.63:0/1455170819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc0fce10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 --2- 172.21.15.63:0/1455170819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70cc102e00 0x7f70cc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 -- 172.21.15.63:0/1455170819 >> 172.21.15.63:0/1455170819 conn(0x7f70cc0f89b0 msgr2=0x7f70cc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 -- 172.21.15.63:0/1455170819 shutdown_connections 2024-03-23T12:12:29.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.969+0000 7f70d1600700 1 -- 172.21.15.63:0/1455170819 wait complete. 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.970+0000 7f70d1600700 1 Processor -- start 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.970+0000 7f70d1600700 1 -- start start 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.970+0000 7f70d1600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70cc102e00 0x7f70cc115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.970+0000 7f70d1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.970+0000 7f70d1600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc116810 con 0x7f70cc1037a0 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.970+0000 7f70d1600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc10fbe0 con 0x7f70cc102e00 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70c3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70c3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38166/0 (socket says 172.21.15.63:38166) 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70c3e00700 1 -- 172.21.15.63:0/2732650219 learned_addr learned my addr 172.21.15.63:0/2732650219 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:12:29.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70caa00700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70cc102e00 0x7f70cc115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70c3e00700 1 -- 172.21.15.63:0/2732650219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70cc102e00 msgr2=0x7f70cc115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70c3e00700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70cc102e00 0x7f70cc115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70c3e00700 1 -- 172.21.15.63:0/2732650219 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b4005040 con 0x7f70cc1037a0 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.971+0000 7f70c3e00700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc1161c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009390 tx=0x7f70b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.972+0000 7f70c2a00700 1 -- 172.21.15.63:0/2732650219 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b400e040 con 0x7f70cc1037a0 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.972+0000 7f70c2a00700 1 -- 172.21.15.63:0/2732650219 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70b4013420 con 0x7f70cc1037a0 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.972+0000 7f70c2a00700 1 -- 172.21.15.63:0/2732650219 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b401c630 con 0x7f70cc1037a0 2024-03-23T12:12:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.972+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70cc10fe60 con 0x7f70cc1037a0 2024-03-23T12:12:29.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.972+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc110420 con 0x7f70cc1037a0 2024-03-23T12:12:29.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.973+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc0fe550 con 0x7f70cc1037a0 2024-03-23T12:12:29.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.974+0000 7f70c2a00700 1 -- 172.21.15.63:0/2732650219 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f70b4013a00 con 0x7f70cc1037a0 2024-03-23T12:12:29.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.975+0000 7f70c2a00700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70b006c7d0 0x7f70b006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:12:29.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.975+0000 7f70c2a00700 1 -- 172.21.15.63:0/2732650219 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f70b408cc50 con 0x7f70cc1037a0 2024-03-23T12:12:29.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.975+0000 7f70caa00700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70b006c7d0 0x7f70b006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:12:29.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.976+0000 7f70caa00700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70b006c7d0 0x7f70b006ec80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009fd0 tx=0x7f70bc009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:12:29.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:29.980+0000 7f70c2a00700 1 -- 172.21.15.63:0/2732650219 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f70b405aa20 con 0x7f70cc1037a0 2024-03-23T12:12:30.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.279+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f70cc04bed0 con 0x7f70cc1037a0 2024-03-23T12:12:30.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.280+0000 7f70c2a00700 1 -- 172.21.15.63:0/2732650219 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f70b405a5b0 con 0x7f70cc1037a0 2024-03-23T12:12:30.261 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:12:30.261 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:12:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:12:30.261 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:12:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70b006c7d0 msgr2=0x7f70b006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70b006c7d0 0x7f70b006ec80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009fd0 tx=0x7f70bc009f60 comp rx=0 tx=0).stop 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 msgr2=0x7f70cc1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc1161c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009390 tx=0x7f70b4009e30 comp rx=0 tx=0).stop 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 shutdown_connections 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f70b006c7d0 0x7f70b006ec80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70cc102e00 0x7f70cc115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 --2- 172.21.15.63:0/2732650219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f70cc1037a0 0x7f70cc1161c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:12:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 >> 172.21.15.63:0/2732650219 conn(0x7f70cc0f89b0 msgr2=0x7f70cc0f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:12:30.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 shutdown_connections 2024-03-23T12:12:30.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:12:30.282+0000 7f70d1600700 1 -- 172.21.15.63:0/2732650219 wait complete. 2024-03-23T12:12:30.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:30 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1942700879' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:12:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:30 smithi063 ceph-mon[141124]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:30 smithi063 ceph-mon[141124]: pgmap v641: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:30.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:30 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1942700879' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:12:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:30 smithi111 ceph-mon[106899]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:12:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:30 smithi111 ceph-mon[106899]: pgmap v641: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:31.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:31 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2732650219' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:12:31.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:31 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2732650219' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:12:32.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:32 smithi063 ceph-mon[141124]: pgmap v642: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:32.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:32 smithi111 ceph-mon[106899]: pgmap v642: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:34.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:34 smithi063 ceph-mon[141124]: pgmap v643: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:34 smithi111 ceph-mon[106899]: pgmap v643: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:36.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:36 smithi063 ceph-mon[141124]: pgmap v644: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:36.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:36 smithi111 ceph-mon[106899]: pgmap v644: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:38.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:38 smithi063 ceph-mon[141124]: pgmap v645: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:38.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:38 smithi111 ceph-mon[106899]: pgmap v645: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:40.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:40 smithi063 ceph-mon[141124]: pgmap v646: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:40.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:40 smithi111 ceph-mon[106899]: pgmap v646: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:42.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:42 smithi063 ceph-mon[141124]: pgmap v647: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:42.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:42 smithi111 ceph-mon[106899]: pgmap v647: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:44.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:44 smithi063 ceph-mon[141124]: pgmap v648: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:44.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:44 smithi111 ceph-mon[106899]: pgmap v648: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:46.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:46 smithi063 ceph-mon[141124]: pgmap v649: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:46.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:46 smithi111 ceph-mon[106899]: pgmap v649: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:48.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:48 smithi063 ceph-mon[141124]: pgmap v650: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:48.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:48 smithi111 ceph-mon[106899]: pgmap v650: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:50.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:50 smithi063 ceph-mon[141124]: pgmap v651: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:50.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:50 smithi111 ceph-mon[106899]: pgmap v651: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:52.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:52 smithi063 ceph-mon[141124]: pgmap v652: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:52.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:52 smithi111 ceph-mon[106899]: pgmap v652: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:54.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:54 smithi063 ceph-mon[141124]: pgmap v653: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:54.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:54 smithi111 ceph-mon[106899]: pgmap v653: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:56.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:56.882 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:56 smithi063 ceph-mon[141124]: pgmap v654: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:56.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:12:56.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:56 smithi111 ceph-mon[106899]: pgmap v654: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:58.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:12:58 smithi063 ceph-mon[141124]: pgmap v655: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:12:58.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:12:58 smithi111 ceph-mon[106899]: pgmap v655: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:00.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:00 smithi063 ceph-mon[141124]: pgmap v656: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:00 smithi111 ceph-mon[106899]: pgmap v656: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:00.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.435+0000 7f6919800700 1 -- 172.21.15.63:0/2370123884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69141037a0 msgr2=0x7f69140fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:00.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.435+0000 7f6919800700 1 --2- 172.21.15.63:0/2370123884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69141037a0 0x7f69140fce10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6900004940 tx=0x7f6900009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:00.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.436+0000 7f6919800700 1 -- 172.21.15.63:0/2370123884 shutdown_connections 2024-03-23T12:13:00.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.436+0000 7f6919800700 1 --2- 172.21.15.63:0/2370123884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69141037a0 0x7f69140fce10 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.436+0000 7f6919800700 1 --2- 172.21.15.63:0/2370123884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6914102e00 0x7f69141031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.436+0000 7f6919800700 1 -- 172.21.15.63:0/2370123884 >> 172.21.15.63:0/2370123884 conn(0x7f69140f89b0 msgr2=0x7f69140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:00.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.436+0000 7f6919800700 1 -- 172.21.15.63:0/2370123884 shutdown_connections 2024-03-23T12:13:00.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.436+0000 7f6919800700 1 -- 172.21.15.63:0/2370123884 wait complete. 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.437+0000 7f6919800700 1 Processor -- start 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.437+0000 7f6919800700 1 -- start start 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.437+0000 7f6919800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6914102e00 0x7f6914113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.437+0000 7f6919800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69141037a0 0x7f6914114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.437+0000 7f6919800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6914114690 con 0x7f6914102e00 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.437+0000 7f6919800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691410da80 con 0x7f69141037a0 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.437+0000 7f6912a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6914102e00 0x7f6914113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.438+0000 7f6912a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6914102e00 0x7f6914113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60160/0 (socket says 172.21.15.63:60160) 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.438+0000 7f6912a00700 1 -- 172.21.15.63:0/3174693003 learned_addr learned my addr 172.21.15.63:0/3174693003 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:00.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.438+0000 7f6912000700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69141037a0 0x7f6914114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:00.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.438+0000 7f6912a00700 1 -- 172.21.15.63:0/3174693003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69141037a0 msgr2=0x7f6914114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:00.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.438+0000 7f6912a00700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69141037a0 0x7f6914114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.438+0000 7f6912a00700 1 -- 172.21.15.63:0/3174693003 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6900005040 con 0x7f6914102e00 2024-03-23T12:13:00.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.438+0000 7f6912a00700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6914102e00 0x7f6914113b00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f690400c8c0 tx=0x7f69040044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:00.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.439+0000 7f690aa00700 1 -- 172.21.15.63:0/3174693003 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f690400a980 con 0x7f6914102e00 2024-03-23T12:13:00.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.439+0000 7f690aa00700 1 -- 172.21.15.63:0/3174693003 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f690400ce60 con 0x7f6914102e00 2024-03-23T12:13:00.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.439+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691410dd60 con 0x7f6914102e00 2024-03-23T12:13:00.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.439+0000 7f690aa00700 1 -- 172.21.15.63:0/3174693003 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69040113a0 con 0x7f6914102e00 2024-03-23T12:13:00.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.439+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691410e2f0 con 0x7f6914102e00 2024-03-23T12:13:00.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.440+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69140fe550 con 0x7f6914102e00 2024-03-23T12:13:00.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.441+0000 7f690aa00700 1 -- 172.21.15.63:0/3174693003 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6904012400 con 0x7f6914102e00 2024-03-23T12:13:00.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.442+0000 7f690aa00700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68f806c7d0 0x7f68f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:00.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.442+0000 7f690aa00700 1 -- 172.21.15.63:0/3174693003 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f690408bb90 con 0x7f6914102e00 2024-03-23T12:13:00.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.442+0000 7f6912000700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68f806c7d0 0x7f68f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:00.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.443+0000 7f6912000700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68f806c7d0 0x7f68f806ec80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f691410ed60 tx=0x7f6900009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.447+0000 7f690aa00700 1 -- 172.21.15.63:0/3174693003 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f69040598a0 con 0x7f6914102e00 2024-03-23T12:13:00.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.678+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6914109da0 con 0x7f68f806c7d0 2024-03-23T12:13:00.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.679+0000 7f690aa00700 1 -- 172.21.15.63:0/3174693003 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6914109da0 con 0x7f68f806c7d0 2024-03-23T12:13:00.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68f806c7d0 msgr2=0x7f68f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68f806c7d0 0x7f68f806ec80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f691410ed60 tx=0x7f6900009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6914102e00 msgr2=0x7f6914113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6914102e00 0x7f6914113b00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f690400c8c0 tx=0x7f69040044c0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 shutdown_connections 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68f806c7d0 0x7f68f806ec80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6914102e00 0x7f6914113b00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 --2- 172.21.15.63:0/3174693003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69141037a0 0x7f6914114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 >> 172.21.15.63:0/3174693003 conn(0x7f69140f89b0 msgr2=0x7f69140f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 shutdown_connections 2024-03-23T12:13:00.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.682+0000 7f6919800700 1 -- 172.21.15.63:0/3174693003 wait complete. 2024-03-23T12:13:00.677 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:13:00.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.820+0000 7f564e600700 1 -- 172.21.15.63:0/2290712107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 msgr2=0x7f56480fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:00.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.820+0000 7f564e600700 1 --2- 172.21.15.63:0/2290712107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f56480fce30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5634004940 tx=0x7f5634009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:00.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.820+0000 7f564e600700 1 -- 172.21.15.63:0/2290712107 shutdown_connections 2024-03-23T12:13:00.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.820+0000 7f564e600700 1 --2- 172.21.15.63:0/2290712107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f56480fce30 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.820+0000 7f564e600700 1 --2- 172.21.15.63:0/2290712107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5648102e20 0x7f56481031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.820+0000 7f564e600700 1 -- 172.21.15.63:0/2290712107 >> 172.21.15.63:0/2290712107 conn(0x7f56480f89d0 msgr2=0x7f56480fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:00.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.821+0000 7f564e600700 1 -- 172.21.15.63:0/2290712107 shutdown_connections 2024-03-23T12:13:00.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.821+0000 7f564e600700 1 -- 172.21.15.63:0/2290712107 wait complete. 2024-03-23T12:13:00.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.821+0000 7f564e600700 1 Processor -- start 2024-03-23T12:13:00.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f564e600700 1 -- start start 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f564e600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5648102e20 0x7f5648113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f564e600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f5648114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f564e600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5648114660 con 0x7f56481037c0 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f564e600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564810da50 con 0x7f5648102e20 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f5646a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f5648114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f5646a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f5648114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60178/0 (socket says 172.21.15.63:60178) 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.822+0000 7f5646a00700 1 -- 172.21.15.63:0/760509613 learned_addr learned my addr 172.21.15.63:0/760509613 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f5646a00700 1 -- 172.21.15.63:0/760509613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5648102e20 msgr2=0x7f5648113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f5646a00700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5648102e20 0x7f5648113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f5646a00700 1 -- 172.21.15.63:0/760509613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634005040 con 0x7f56481037c0 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f5646a00700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f5648114010 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5634009390 tx=0x7f5634009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:00.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f563f400700 1 -- 172.21.15.63:0/760509613 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563400e040 con 0x7f56481037c0 2024-03-23T12:13:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f563f400700 1 -- 172.21.15.63:0/760509613 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5634013420 con 0x7f56481037c0 2024-03-23T12:13:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564810dcd0 con 0x7f56481037c0 2024-03-23T12:13:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f563f400700 1 -- 172.21.15.63:0/760509613 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563401c630 con 0x7f56481037c0 2024-03-23T12:13:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.823+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564810e290 con 0x7f56481037c0 2024-03-23T12:13:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.824+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56480fe570 con 0x7f56481037c0 2024-03-23T12:13:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.825+0000 7f563f400700 1 -- 172.21.15.63:0/760509613 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5634013a00 con 0x7f56481037c0 2024-03-23T12:13:00.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.826+0000 7f563f400700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f562c06c7d0 0x7f562c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:00.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.826+0000 7f563f400700 1 -- 172.21.15.63:0/760509613 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f563408cd10 con 0x7f56481037c0 2024-03-23T12:13:00.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.826+0000 7f5647400700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f562c06c7d0 0x7f562c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:00.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.827+0000 7f5647400700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f562c06c7d0 0x7f562c06ec80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5638004d00 tx=0x7f5638009ce0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:00.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:00.831+0000 7f563f400700 1 -- 172.21.15.63:0/760509613 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f563405aa20 con 0x7f56481037c0 2024-03-23T12:13:01.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.061+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5648109e60 con 0x7f562c06c7d0 2024-03-23T12:13:01.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.063+0000 7f563f400700 1 -- 172.21.15.63:0/760509613 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f563800f040 con 0x7f562c06c7d0 2024-03-23T12:13:01.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f562c06c7d0 msgr2=0x7f562c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f562c06c7d0 0x7f562c06ec80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5638004d00 tx=0x7f5638009ce0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 msgr2=0x7f5648114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f5648114010 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5634009390 tx=0x7f5634009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 shutdown_connections 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f562c06c7d0 0x7f562c06ec80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5648102e20 0x7f5648113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 --2- 172.21.15.63:0/760509613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56481037c0 0x7f5648114010 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 >> 172.21.15.63:0/760509613 conn(0x7f56480f89d0 msgr2=0x7f56480f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 shutdown_connections 2024-03-23T12:13:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.065+0000 7f564e600700 1 -- 172.21.15.63:0/760509613 wait complete. 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 -- 172.21.15.63:0/1564403081 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0981037a0 msgr2=0x7ff0980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 --2- 172.21.15.63:0/1564403081 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0981037a0 0x7ff0980fce10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff084004940 tx=0x7ff084009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 -- 172.21.15.63:0/1564403081 shutdown_connections 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 --2- 172.21.15.63:0/1564403081 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0981037a0 0x7ff0980fce10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 --2- 172.21.15.63:0/1564403081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff098102e00 0x7ff0981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 -- 172.21.15.63:0/1564403081 >> 172.21.15.63:0/1564403081 conn(0x7ff0980f89b0 msgr2=0x7ff0980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 -- 172.21.15.63:0/1564403081 shutdown_connections 2024-03-23T12:13:01.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.203+0000 7ff09fc00700 1 -- 172.21.15.63:0/1564403081 wait complete. 2024-03-23T12:13:01.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.204+0000 7ff09fc00700 1 Processor -- start 2024-03-23T12:13:01.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.205+0000 7ff09fc00700 1 -- start start 2024-03-23T12:13:01.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.205+0000 7ff09fc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff098102e00 0x7ff098113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:01.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.205+0000 7ff09fc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0981037a0 0x7ff098114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:01.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.205+0000 7ff09fc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff098114690 con 0x7ff098102e00 2024-03-23T12:13:01.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.205+0000 7ff09fc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09810da80 con 0x7ff0981037a0 2024-03-23T12:13:01.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff098102e00 0x7ff098113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:01.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff098102e00 0x7ff098113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60210/0 (socket says 172.21.15.63:60210) 2024-03-23T12:13:01.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09d200700 1 -- 172.21.15.63:0/2587105067 learned_addr learned my addr 172.21.15.63:0/2587105067 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:01.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff097e00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0981037a0 0x7ff098114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:01.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09d200700 1 -- 172.21.15.63:0/2587105067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0981037a0 msgr2=0x7ff098114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09d200700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0981037a0 0x7ff098114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09d200700 1 -- 172.21.15.63:0/2587105067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff084005040 con 0x7ff098102e00 2024-03-23T12:13:01.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09d200700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff098102e00 0x7ff098113b00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff08c002a50 tx=0x7ff08c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:01.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff094c00700 1 -- 172.21.15.63:0/2587105067 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff08c007cc0 con 0x7ff098102e00 2024-03-23T12:13:01.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff094c00700 1 -- 172.21.15.63:0/2587105067 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff08c00f040 con 0x7ff098102e00 2024-03-23T12:13:01.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.206+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09810dd60 con 0x7ff098102e00 2024-03-23T12:13:01.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.207+0000 7ff094c00700 1 -- 172.21.15.63:0/2587105067 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff08c00d9c0 con 0x7ff098102e00 2024-03-23T12:13:01.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.207+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09810e2f0 con 0x7ff098102e00 2024-03-23T12:13:01.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.208+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0980fe550 con 0x7ff098102e00 2024-03-23T12:13:01.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.209+0000 7ff094c00700 1 -- 172.21.15.63:0/2587105067 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff08c00db20 con 0x7ff098102e00 2024-03-23T12:13:01.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.210+0000 7ff094c00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff08006c7d0 0x7ff08006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:01.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.210+0000 7ff094c00700 1 -- 172.21.15.63:0/2587105067 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff08c08b820 con 0x7ff098102e00 2024-03-23T12:13:01.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.210+0000 7ff097e00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff08006c7d0 0x7ff08006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:01.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.211+0000 7ff097e00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff08006c7d0 0x7ff08006ec80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff09810ed60 tx=0x7ff084009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:01.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.215+0000 7ff094c00700 1 -- 172.21.15.63:0/2587105067 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff08c059430 con 0x7ff098102e00 2024-03-23T12:13:01.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.436+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff098109da0 con 0x7ff08006c7d0 2024-03-23T12:13:01.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.446+0000 7ff094c00700 1 -- 172.21.15.63:0/2587105067 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7ff098109da0 con 0x7ff08006c7d0 2024-03-23T12:13:01.427 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:13:01.427 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (22m) 14m ago 35m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:13:01.427 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (19m) 14m ago 35m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:13:01.427 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (19m) 14m ago 34m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (22m) 14m ago 35m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (15m) 14m ago 29m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (15m) 14m ago 29m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (28m) 14m ago 28m 10.6M - 17.2.7-726-g9c4e0c34 837124da804f e3a17213ea55 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (14m) 14m ago 29m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (24m) 14m ago 36m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (23m) 14m ago 34m 452M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:13:01.428 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (20m) 14m ago 37m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (20m) 14m ago 34m 44.0M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (23m) 14m ago 35m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (23m) 14m ago 34m 7662k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (18m) 14m ago 33m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (18m) 14m ago 32m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (17m) 14m ago 32m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (16m) 14m ago 31m 97.9M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:13:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (16m) 14m ago 31m 89.7M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:13:01.430 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (15m) 14m ago 30m 85.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:13:01.430 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (22m) 14m ago 34m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:13:01.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff08006c7d0 msgr2=0x7ff08006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff08006c7d0 0x7ff08006ec80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff09810ed60 tx=0x7ff084009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:01.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff098102e00 msgr2=0x7ff098113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff098102e00 0x7ff098113b00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff08c002a50 tx=0x7ff08c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 shutdown_connections 2024-03-23T12:13:01.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff08006c7d0 0x7ff08006ec80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff098102e00 0x7ff098113b00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 --2- 172.21.15.63:0/2587105067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0981037a0 0x7ff098114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 >> 172.21.15.63:0/2587105067 conn(0x7ff0980f89b0 msgr2=0x7ff0980f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:01.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 shutdown_connections 2024-03-23T12:13:01.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.449+0000 7ff09fc00700 1 -- 172.21.15.63:0/2587105067 wait complete. 2024-03-23T12:13:01.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 -- 172.21.15.63:0/2462562052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 msgr2=0x7f112c0fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 --2- 172.21.15.63:0/2462562052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c0fe280 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f111c004970 tx=0x7f111c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:01.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 -- 172.21.15.63:0/2462562052 shutdown_connections 2024-03-23T12:13:01.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 --2- 172.21.15.63:0/2462562052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f112c0fe850 0x7f112c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 --2- 172.21.15.63:0/2462562052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c0fe280 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 -- 172.21.15.63:0/2462562052 >> 172.21.15.63:0/2462562052 conn(0x7f112c072bc0 msgr2=0x7f112c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:01.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 -- 172.21.15.63:0/2462562052 shutdown_connections 2024-03-23T12:13:01.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.586+0000 7f1130e00700 1 -- 172.21.15.63:0/2462562052 wait complete. 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.587+0000 7f1130e00700 1 Processor -- start 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.587+0000 7f1130e00700 1 -- start start 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.587+0000 7f1130e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.587+0000 7f1130e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f112c0fe850 0x7f112c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.587+0000 7f1130e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c116800 con 0x7f112c0fdeb0 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.587+0000 7f1130e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c10fc10 con 0x7f112c0fe850 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.587+0000 7f1129e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1129e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42652/0 (socket says 172.21.15.63:42652) 2024-03-23T12:13:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1129e00700 1 -- 172.21.15.63:0/3038826002 learned_addr learned my addr 172.21.15.63:0/3038826002 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1129e00700 1 -- 172.21.15.63:0/3038826002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f112c0fe850 msgr2=0x7f112c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1129e00700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f112c0fe850 0x7f112c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1129e00700 1 -- 172.21.15.63:0/3038826002 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c005040 con 0x7f112c0fdeb0 2024-03-23T12:13:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1129e00700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c115c70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f111c005010 tx=0x7f111c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:01.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1122000700 1 -- 172.21.15.63:0/3038826002 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111c00e040 con 0x7f112c0fdeb0 2024-03-23T12:13:01.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1122000700 1 -- 172.21.15.63:0/3038826002 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f111c009930 con 0x7f112c0fdeb0 2024-03-23T12:13:01.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c10fe90 con 0x7f112c0fdeb0 2024-03-23T12:13:01.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.588+0000 7f1122000700 1 -- 172.21.15.63:0/3038826002 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111c01c3f0 con 0x7f112c0fdeb0 2024-03-23T12:13:01.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.589+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c1103d0 con 0x7f112c0fdeb0 2024-03-23T12:13:01.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.590+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112c1069f0 con 0x7f112c0fdeb0 2024-03-23T12:13:01.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.591+0000 7f1122000700 1 -- 172.21.15.63:0/3038826002 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f111c009480 con 0x7f112c0fdeb0 2024-03-23T12:13:01.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.591+0000 7f1122000700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f111006c7c0 0x7f111006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:01.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.592+0000 7f1122000700 1 -- 172.21.15.63:0/3038826002 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f111c08ccd0 con 0x7f112c0fdeb0 2024-03-23T12:13:01.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.592+0000 7f1129400700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f111006c7c0 0x7f111006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:01.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.592+0000 7f1129400700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f111006c7c0 0x7f111006ec70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f112c111550 tx=0x7f111800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:01.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.596+0000 7f1122000700 1 -- 172.21.15.63:0/3038826002 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f111c056f50 con 0x7f112c0fdeb0 2024-03-23T12:13:01.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:01 smithi063 ceph-mon[141124]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:01.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.898+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f112c065e10 con 0x7f112c0fdeb0 2024-03-23T12:13:01.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.899+0000 7f1122000700 1 -- 172.21.15.63:0/3038826002 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f111c05a570 con 0x7f112c0fdeb0 2024-03-23T12:13:01.879 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:13:01.879 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:13:01.879 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 3 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 13 2024-03-23T12:13:01.880 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:13:01.881 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f111006c7c0 msgr2=0x7f111006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f111006c7c0 0x7f111006ec70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f112c111550 tx=0x7f111800b040 comp rx=0 tx=0).stop 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 msgr2=0x7f112c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c115c70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f111c005010 tx=0x7f111c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 shutdown_connections 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f111006c7c0 0x7f111006ec70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f112c0fdeb0 0x7f112c115c70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 --2- 172.21.15.63:0/3038826002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f112c0fe850 0x7f112c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:01.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 >> 172.21.15.63:0/3038826002 conn(0x7f112c072bc0 msgr2=0x7f112c0fb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:01.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.901+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 shutdown_connections 2024-03-23T12:13:01.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:01.902+0000 7f1130e00700 1 -- 172.21.15.63:0/3038826002 wait complete. 2024-03-23T12:13:01.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:01 smithi111 ceph-mon[106899]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 -- 172.21.15.63:0/316228200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc065630 msgr2=0x7f3fdc065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 --2- 172.21.15.63:0/316228200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc065630 0x7f3fdc065a80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8004940 tx=0x7f3fc8009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 -- 172.21.15.63:0/316228200 shutdown_connections 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 --2- 172.21.15.63:0/316228200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc065630 0x7f3fdc065a80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 --2- 172.21.15.63:0/316228200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fdc102ca0 0x7f3fdc103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 -- 172.21.15.63:0/316228200 >> 172.21.15.63:0/316228200 conn(0x7f3fdc071b70 msgr2=0x7f3fdc071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 -- 172.21.15.63:0/316228200 shutdown_connections 2024-03-23T12:13:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.040+0000 7f3fe1000700 1 -- 172.21.15.63:0/316228200 wait complete. 2024-03-23T12:13:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.041+0000 7f3fe1000700 1 Processor -- start 2024-03-23T12:13:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.041+0000 7f3fe1000700 1 -- start start 2024-03-23T12:13:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fe1000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fdc065630 0x7f3fdc115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fe1000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc102ca0 0x7f3fdc116070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fe1000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc1166c0 con 0x7f3fdc102ca0 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fe1000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc10fab0 con 0x7f3fdc065630 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc102ca0 0x7f3fdc116070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc102ca0 0x7f3fdc116070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42670/0 (socket says 172.21.15.63:42670) 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdaa00700 1 -- 172.21.15.63:0/3085815380 learned_addr learned my addr 172.21.15.63:0/3085815380 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdb400700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fdc065630 0x7f3fdc115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdaa00700 1 -- 172.21.15.63:0/3085815380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fdc065630 msgr2=0x7f3fdc115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdaa00700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fdc065630 0x7f3fdc115b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdaa00700 1 -- 172.21.15.63:0/3085815380 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8005040 con 0x7f3fdc102ca0 2024-03-23T12:13:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.042+0000 7f3fdb400700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fdc065630 0x7f3fdc115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:13:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.043+0000 7f3fdaa00700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc102ca0 0x7f3fdc116070 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009390 tx=0x7f3fc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.043+0000 7f3fd3400700 1 -- 172.21.15.63:0/3085815380 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fc800e040 con 0x7f3fdc102ca0 2024-03-23T12:13:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.043+0000 7f3fd3400700 1 -- 172.21.15.63:0/3085815380 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fc8013420 con 0x7f3fdc102ca0 2024-03-23T12:13:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.043+0000 7f3fd3400700 1 -- 172.21.15.63:0/3085815380 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fc801c630 con 0x7f3fdc102ca0 2024-03-23T12:13:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.043+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc10fd30 con 0x7f3fdc102ca0 2024-03-23T12:13:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.043+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc110270 con 0x7f3fdc102ca0 2024-03-23T12:13:02.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.044+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc106840 con 0x7f3fdc102ca0 2024-03-23T12:13:02.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.045+0000 7f3fd3400700 1 -- 172.21.15.63:0/3085815380 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3fc80139e0 con 0x7f3fdc102ca0 2024-03-23T12:13:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.046+0000 7f3fd3400700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc006c7d0 0x7f3fc006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.046+0000 7f3fd3400700 1 -- 172.21.15.63:0/3085815380 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3fc808cc80 con 0x7f3fdc102ca0 2024-03-23T12:13:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.046+0000 7f3fdb400700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc006c7d0 0x7f3fc006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.047+0000 7f3fdb400700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc006c7d0 0x7f3fc006ec80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009fd0 tx=0x7f3fcc009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:02.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.051+0000 7f3fd3400700 1 -- 172.21.15.63:0/3085815380 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3fc805aa40 con 0x7f3fdc102ca0 2024-03-23T12:13:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.309+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3fdc111180 con 0x7f3fdc102ca0 2024-03-23T12:13:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.310+0000 7f3fd3400700 1 -- 172.21.15.63:0/3085815380 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f3fc805a5d0 con 0x7f3fdc102ca0 2024-03-23T12:13:02.290 INFO:teuthology.orchestra.run.smithi063.stdout:e33 2024-03-23T12:13:02.290 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:13:02.290 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:13:02.290 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:13:02.291 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:13:02.292 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:44329} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/3285026913,v1:172.21.15.111:6827/3285026913] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc006c7d0 msgr2=0x7f3fc006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc006c7d0 0x7f3fc006ec80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009fd0 tx=0x7f3fcc009ee0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc102ca0 msgr2=0x7f3fdc116070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc102ca0 0x7f3fdc116070 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009390 tx=0x7f3fc8009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 shutdown_connections 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc006c7d0 0x7f3fc006ec80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fdc065630 0x7f3fdc115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 --2- 172.21.15.63:0/3085815380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fdc102ca0 0x7f3fdc116070 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 >> 172.21.15.63:0/3085815380 conn(0x7f3fdc071b70 msgr2=0x7f3fdc1039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 shutdown_connections 2024-03-23T12:13:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.312+0000 7f3fe1000700 1 -- 172.21.15.63:0/3085815380 wait complete. 2024-03-23T12:13:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:13:02.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.450+0000 7f033f600700 1 -- 172.21.15.63:0/2293773690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fde90 msgr2=0x7f03380fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.450+0000 7f033f600700 1 --2- 172.21.15.63:0/2293773690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fde90 0x7f03380fe260 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f032c004940 tx=0x7f032c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:02.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.450+0000 7f033f600700 1 -- 172.21.15.63:0/2293773690 shutdown_connections 2024-03-23T12:13:02.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.450+0000 7f033f600700 1 --2- 172.21.15.63:0/2293773690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03380fe830 0x7f0338106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.450+0000 7f033f600700 1 --2- 172.21.15.63:0/2293773690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fde90 0x7f03380fe260 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.451+0000 7f033f600700 1 -- 172.21.15.63:0/2293773690 >> 172.21.15.63:0/2293773690 conn(0x7f0338072bc0 msgr2=0x7f0338072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:02.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.451+0000 7f033f600700 1 -- 172.21.15.63:0/2293773690 shutdown_connections 2024-03-23T12:13:02.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.451+0000 7f033f600700 1 -- 172.21.15.63:0/2293773690 wait complete. 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.451+0000 7f033f600700 1 Processor -- start 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f033f600700 1 -- start start 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f033f600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03380fde90 0x7f03380fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f033f600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fe830 0x7f03380fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f033f600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03380fc820 con 0x7f03380fe830 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f033f600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03380fc960 con 0x7f03380fde90 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f0337e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fe830 0x7f03380fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f0337e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fe830 0x7f03380fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42694/0 (socket says 172.21.15.63:42694) 2024-03-23T12:13:02.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f0337e00700 1 -- 172.21.15.63:0/3309934452 learned_addr learned my addr 172.21.15.63:0/3309934452 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:02.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.452+0000 7f033cc00700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03380fde90 0x7f03380fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f0337e00700 1 -- 172.21.15.63:0/3309934452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03380fde90 msgr2=0x7f03380fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f0337e00700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03380fde90 0x7f03380fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f0337e00700 1 -- 172.21.15.63:0/3309934452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c005040 con 0x7f03380fe830 2024-03-23T12:13:02.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f0337e00700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fe830 0x7f03380fc290 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0324002a50 tx=0x7f03240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:02.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f0334c00700 1 -- 172.21.15.63:0/3309934452 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0324007cc0 con 0x7f03380fe830 2024-03-23T12:13:02.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f0334c00700 1 -- 172.21.15.63:0/3309934452 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f032400f040 con 0x7f03380fe830 2024-03-23T12:13:02.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0338119df0 con 0x7f03380fe830 2024-03-23T12:13:02.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f0334c00700 1 -- 172.21.15.63:0/3309934452 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f032400d9c0 con 0x7f03380fe830 2024-03-23T12:13:02.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.453+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033811a1f0 con 0x7f03380fe830 2024-03-23T12:13:02.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.454+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0338065e30 con 0x7f03380fe830 2024-03-23T12:13:02.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.455+0000 7f0334c00700 1 -- 172.21.15.63:0/3309934452 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f032400db20 con 0x7f03380fe830 2024-03-23T12:13:02.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.456+0000 7f0334c00700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f032006c7d0 0x7f032006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.456+0000 7f0334c00700 1 -- 172.21.15.63:0/3309934452 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f032408b860 con 0x7f03380fe830 2024-03-23T12:13:02.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.457+0000 7f033cc00700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f032006c7d0 0x7f032006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.457+0000 7f033cc00700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f032006c7d0 0x7f032006ec80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f032c00b3b0 tx=0x7f032c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:02.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.461+0000 7f0334c00700 1 -- 172.21.15.63:0/3309934452 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f03240559e0 con 0x7f03380fe830 2024-03-23T12:13:02.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.690+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f033804bed0 con 0x7f032006c7d0 2024-03-23T12:13:02.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.692+0000 7f0334c00700 1 -- 172.21.15.63:0/3309934452 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f033804bed0 con 0x7f032006c7d0 2024-03-23T12:13:02.672 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:13:02.672 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:13:02.672 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:13:02.672 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:13:02.672 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr", 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: "osd" 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "15/21 daemons upgraded", 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:13:02.673 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:13:02.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f032006c7d0 msgr2=0x7f032006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f032006c7d0 0x7f032006ec80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f032c00b3b0 tx=0x7f032c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fe830 msgr2=0x7f03380fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fe830 0x7f03380fc290 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0324002a50 tx=0x7f03240046b0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 shutdown_connections 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f032006c7d0 0x7f032006ec80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03380fde90 0x7f03380fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 --2- 172.21.15.63:0/3309934452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f03380fe830 0x7f03380fc290 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 >> 172.21.15.63:0/3309934452 conn(0x7f0338072bc0 msgr2=0x7f03380ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 shutdown_connections 2024-03-23T12:13:02.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.694+0000 7f033f600700 1 -- 172.21.15.63:0/3309934452 wait complete. 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 -- 172.21.15.63:0/3178480382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72841037a0 msgr2=0x7f72840fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 --2- 172.21.15.63:0/3178480382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72841037a0 0x7f72840fce10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7270004940 tx=0x7f7270009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 -- 172.21.15.63:0/3178480382 shutdown_connections 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 --2- 172.21.15.63:0/3178480382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72841037a0 0x7f72840fce10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 --2- 172.21.15.63:0/3178480382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7284102e00 0x7f72841031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 -- 172.21.15.63:0/3178480382 >> 172.21.15.63:0/3178480382 conn(0x7f72840f89b0 msgr2=0x7f72840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 -- 172.21.15.63:0/3178480382 shutdown_connections 2024-03-23T12:13:02.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.834+0000 7f7289800700 1 -- 172.21.15.63:0/3178480382 wait complete. 2024-03-23T12:13:02.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.835+0000 7f7289800700 1 Processor -- start 2024-03-23T12:13:02.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.835+0000 7f7289800700 1 -- start start 2024-03-23T12:13:02.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7289800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7284102e00 0x7f7284113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7289800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72841037a0 0x7f7284114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7289800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72841146b0 con 0x7f7284102e00 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7289800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728410daa0 con 0x7f72841037a0 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7282a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7284102e00 0x7f7284113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7282a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7284102e00 0x7f7284113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42712/0 (socket says 172.21.15.63:42712) 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7282a00700 1 -- 172.21.15.63:0/3479979317 learned_addr learned my addr 172.21.15.63:0/3479979317 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7282000700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72841037a0 0x7f7284114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7282a00700 1 -- 172.21.15.63:0/3479979317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72841037a0 msgr2=0x7f7284114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7282a00700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72841037a0 0x7f7284114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.836+0000 7f7282a00700 1 -- 172.21.15.63:0/3479979317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270005040 con 0x7f7284102e00 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.837+0000 7f7282a00700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7284102e00 0x7f7284113b20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7274002a50 tx=0x7f72740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.837+0000 7f727aa00700 1 -- 172.21.15.63:0/3479979317 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7274007cc0 con 0x7f7284102e00 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.837+0000 7f727aa00700 1 -- 172.21.15.63:0/3479979317 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f727400f040 con 0x7f7284102e00 2024-03-23T12:13:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.837+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728410dd80 con 0x7f7284102e00 2024-03-23T12:13:02.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.837+0000 7f727aa00700 1 -- 172.21.15.63:0/3479979317 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f727400d9c0 con 0x7f7284102e00 2024-03-23T12:13:02.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.837+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728410e310 con 0x7f7284102e00 2024-03-23T12:13:02.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.838+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72840fe550 con 0x7f7284102e00 2024-03-23T12:13:02.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.840+0000 7f727aa00700 1 -- 172.21.15.63:0/3479979317 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f727400db60 con 0x7f7284102e00 2024-03-23T12:13:02.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.841+0000 7f727aa00700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726806c7d0 0x7f726806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:02.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.841+0000 7f727aa00700 1 -- 172.21.15.63:0/3479979317 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f727408b1a0 con 0x7f7284102e00 2024-03-23T12:13:02.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.841+0000 7f7282000700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726806c7d0 0x7f726806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:02.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.842+0000 7f7282000700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726806c7d0 0x7f726806ec80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f728410ed80 tx=0x7f7270009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:02.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:02.846+0000 7f727aa00700 1 -- 172.21.15.63:0/3479979317 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7274058db0 con 0x7f7284102e00 2024-03-23T12:13:02.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:02 smithi063 ceph-mon[141124]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:02.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:02 smithi063 ceph-mon[141124]: from='client.35226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:02.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:02 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3038826002' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:02.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:02 smithi063 ceph-mon[141124]: pgmap v657: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:02.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:02 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3085815380' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:13:02.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:02 smithi111 ceph-mon[106899]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:02.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:02 smithi111 ceph-mon[106899]: from='client.35226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:02.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:02 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3038826002' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:02.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:02 smithi111 ceph-mon[106899]: pgmap v657: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:02.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:02 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3085815380' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:13:03.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.144+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f728404bed0 con 0x7f7284102e00 2024-03-23T12:13:03.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.144+0000 7f727aa00700 1 -- 172.21.15.63:0/3479979317 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f7274058940 con 0x7f7284102e00 2024-03-23T12:13:03.124 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:13:03.125 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:13:03.125 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:13:03.125 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:13:03.125 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726806c7d0 msgr2=0x7f726806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726806c7d0 0x7f726806ec80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f728410ed80 tx=0x7f7270009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7284102e00 msgr2=0x7f7284113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7284102e00 0x7f7284113b20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7274002a50 tx=0x7f72740046b0 comp rx=0 tx=0).stop 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 shutdown_connections 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f726806c7d0 0x7f726806ec80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7284102e00 0x7f7284113b20 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 --2- 172.21.15.63:0/3479979317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72841037a0 0x7f7284114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:03.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.146+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 >> 172.21.15.63:0/3479979317 conn(0x7f72840f89b0 msgr2=0x7f72840f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:03.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.147+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 shutdown_connections 2024-03-23T12:13:03.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:03.147+0000 7f7289800700 1 -- 172.21.15.63:0/3479979317 wait complete. 2024-03-23T12:13:03.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:03 smithi063 ceph-mon[141124]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:03.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:03 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3479979317' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:13:03.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:03 smithi111 ceph-mon[106899]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:03.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:03 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3479979317' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:13:04.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:04 smithi063 ceph-mon[141124]: pgmap v658: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:04.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:04 smithi111 ceph-mon[106899]: pgmap v658: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:06.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:06 smithi063 ceph-mon[141124]: pgmap v659: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:06.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:06 smithi111 ceph-mon[106899]: pgmap v659: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:08.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:08 smithi063 ceph-mon[141124]: pgmap v660: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:08.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:08 smithi111 ceph-mon[106899]: pgmap v660: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:10.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:10 smithi063 ceph-mon[141124]: pgmap v661: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:10.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:10 smithi111 ceph-mon[106899]: pgmap v661: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:11.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:11.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:12.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:12 smithi063 ceph-mon[141124]: pgmap v662: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:12.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:12 smithi111 ceph-mon[106899]: pgmap v662: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:14.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:14 smithi063 ceph-mon[141124]: pgmap v663: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:14.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:14 smithi111 ceph-mon[106899]: pgmap v663: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:16.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:16 smithi063 ceph-mon[141124]: pgmap v664: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:16.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:16 smithi111 ceph-mon[106899]: pgmap v664: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:18.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:18 smithi063 ceph-mon[141124]: pgmap v665: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:18.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:18 smithi111 ceph-mon[106899]: pgmap v665: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:19.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:19 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:13:19.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:19 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:13:19.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:19 smithi063 ceph-mon[141124]: pgmap v666: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:19.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:19 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:19 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:13:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:19 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:13:19.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:19 smithi111 ceph-mon[106899]: pgmap v666: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:19.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:19 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:20.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:20 smithi111 ceph-mon[106899]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-23T12:13:20.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:13:20.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:20.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:20.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:20.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:20 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:20 smithi063 ceph-mon[141124]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-23T12:13:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:13:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:20.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:20 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:22.237 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:22 smithi111 ceph-mon[106899]: pgmap v667: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:22.237 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:22 smithi111 ceph-mon[106899]: Upgrade: Updating mds.cephfs.smithi111.atkibb 2024-03-23T12:13:22.237 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:22.237 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T12:13:22.237 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:22 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:13:22.237 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:22 smithi111 ceph-mon[106899]: Deploying daemon mds.cephfs.smithi111.atkibb on smithi111 2024-03-23T12:13:22.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:22 smithi063 ceph-mon[141124]: pgmap v667: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:22.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:22 smithi063 ceph-mon[141124]: Upgrade: Updating mds.cephfs.smithi111.atkibb 2024-03-23T12:13:22.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:22.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.atkibb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-23T12:13:22.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:22 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:13:22.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:22 smithi063 ceph-mon[141124]: Deploying daemon mds.cephfs.smithi111.atkibb on smithi111 2024-03-23T12:13:24.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:24 smithi063 ceph-mon[141124]: pgmap v668: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:24.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:24 smithi063 ceph-mon[141124]: osdmap e86: 6 total, 6 up, 6 in 2024-03-23T12:13:24.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:24 smithi111 ceph-mon[106899]: pgmap v668: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:24.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:24 smithi111 ceph-mon[106899]: osdmap e86: 6 total, 6 up, 6 in 2024-03-23T12:13:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:25 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T12:13:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:25 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-23T12:13:26.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:26 smithi063 ceph-mon[141124]: pgmap v670: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:26.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:26.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:26.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:26 smithi111 ceph-mon[106899]: pgmap v670: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:26.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:26.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:28.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:28 smithi063 ceph-mon[141124]: pgmap v671: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:28.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:28 smithi111 ceph-mon[106899]: pgmap v671: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:30.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: pgmap v672: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:13:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:30 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: pgmap v672: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:13:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:30.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:30 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:31.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:31 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] up:boot 2024-03-23T12:13:31.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:31 smithi063 ceph-mon[141124]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T12:13:31.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T12:13:31.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:31 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] up:boot 2024-03-23T12:13:31.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:31 smithi111 ceph-mon[106899]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-23T12:13:31.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T12:13:32.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:32 smithi063 ceph-mon[141124]: pgmap v673: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:32.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:32 smithi111 ceph-mon[106899]: pgmap v673: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 -- 172.21.15.63:0/3982229292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fc5f0 msgr2=0x7fb72c0fc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 --2- 172.21.15.63:0/3982229292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fc5f0 0x7fb72c0fc9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb71c004940 tx=0x7fb71c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 -- 172.21.15.63:0/3982229292 shutdown_connections 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 --2- 172.21.15.63:0/3982229292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb72c0fcf00 0x7fb72c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 --2- 172.21.15.63:0/3982229292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fc5f0 0x7fb72c0fc9c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 -- 172.21.15.63:0/3982229292 >> 172.21.15.63:0/3982229292 conn(0x7fb72c071b70 msgr2=0x7fb72c071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 -- 172.21.15.63:0/3982229292 shutdown_connections 2024-03-23T12:13:33.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.297+0000 7fb72be00700 1 -- 172.21.15.63:0/3982229292 wait complete. 2024-03-23T12:13:33.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.298+0000 7fb72be00700 1 Processor -- start 2024-03-23T12:13:33.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.298+0000 7fb72be00700 1 -- start start 2024-03-23T12:13:33.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.298+0000 7fb72be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb72c0fc5f0 0x7fb72c115b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:33.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.298+0000 7fb72be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fcf00 0x7fb72c1160b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:33.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.298+0000 7fb72be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c116700 con 0x7fb72c0fcf00 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.298+0000 7fb72be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c10faf0 con 0x7fb72c0fc5f0 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fcf00 0x7fb72c1160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fcf00 0x7fb72c1160b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55028/0 (socket says 172.21.15.63:55028) 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72a000700 1 -- 172.21.15.63:0/4084684038 learned_addr learned my addr 172.21.15.63:0/4084684038 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72aa00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb72c0fc5f0 0x7fb72c115b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72a000700 1 -- 172.21.15.63:0/4084684038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb72c0fc5f0 msgr2=0x7fb72c115b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72a000700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb72c0fc5f0 0x7fb72c115b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72a000700 1 -- 172.21.15.63:0/4084684038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c005040 con 0x7fb72c0fcf00 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.299+0000 7fb72a000700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fcf00 0x7fb72c1160b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb71800eb40 tx=0x7fb71800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.300+0000 7fb722a00700 1 -- 172.21.15.63:0/4084684038 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb718007cc0 con 0x7fb72c0fcf00 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.300+0000 7fb722a00700 1 -- 172.21.15.63:0/4084684038 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb71800f040 con 0x7fb72c0fcf00 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.300+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c10fd70 con 0x7fb72c0fcf00 2024-03-23T12:13:33.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.300+0000 7fb722a00700 1 -- 172.21.15.63:0/4084684038 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb71800c990 con 0x7fb72c0fcf00 2024-03-23T12:13:33.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.300+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c1102b0 con 0x7fb72c0fcf00 2024-03-23T12:13:33.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.301+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c1068f0 con 0x7fb72c0fcf00 2024-03-23T12:13:33.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.302+0000 7fb722a00700 1 -- 172.21.15.63:0/4084684038 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb71800cc00 con 0x7fb72c0fcf00 2024-03-23T12:13:33.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.303+0000 7fb722a00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb71006c7d0 0x7fb71006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:33.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.303+0000 7fb722a00700 1 -- 172.21.15.63:0/4084684038 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb71808b710 con 0x7fb72c0fcf00 2024-03-23T12:13:33.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.303+0000 7fb72aa00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb71006c7d0 0x7fb71006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:33.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.304+0000 7fb72aa00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb71006c7d0 0x7fb71006ec80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00b0d0 tx=0x7fb71c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:33.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.308+0000 7fb722a00700 1 -- 172.21.15.63:0/4084684038 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7180593d0 con 0x7fb72c0fcf00 2024-03-23T12:13:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.570+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb72c115840 con 0x7fb71006c7d0 2024-03-23T12:13:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.572+0000 7fb722a00700 1 -- 172.21.15.63:0/4084684038 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb72c115840 con 0x7fb71006c7d0 2024-03-23T12:13:33.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb71006c7d0 msgr2=0x7fb71006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb71006c7d0 0x7fb71006ec80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00b0d0 tx=0x7fb71c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:33.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fcf00 msgr2=0x7fb72c1160b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fcf00 0x7fb72c1160b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb71800eb40 tx=0x7fb71800b040 comp rx=0 tx=0).stop 2024-03-23T12:13:33.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 shutdown_connections 2024-03-23T12:13:33.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb71006c7d0 0x7fb71006ec80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb72c0fc5f0 0x7fb72c115b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 --2- 172.21.15.63:0/4084684038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb72c0fcf00 0x7fb72c1160b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 >> 172.21.15.63:0/4084684038 conn(0x7fb72c071b70 msgr2=0x7fb72c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:33.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 shutdown_connections 2024-03-23T12:13:33.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.575+0000 7fb72be00700 1 -- 172.21.15.63:0/4084684038 wait complete. 2024-03-23T12:13:33.570 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:13:33.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 -- 172.21.15.63:0/3464220844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd8102e00 msgr2=0x7fdcd81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 --2- 172.21.15.63:0/3464220844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd8102e00 0x7fdcd81031d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fdccc004940 tx=0x7fdccc009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:33.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 -- 172.21.15.63:0/3464220844 shutdown_connections 2024-03-23T12:13:33.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 --2- 172.21.15.63:0/3464220844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd81037a0 0x7fdcd80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 --2- 172.21.15.63:0/3464220844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd8102e00 0x7fdcd81031d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 -- 172.21.15.63:0/3464220844 >> 172.21.15.63:0/3464220844 conn(0x7fdcd80f89b0 msgr2=0x7fdcd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:33.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 -- 172.21.15.63:0/3464220844 shutdown_connections 2024-03-23T12:13:33.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.728+0000 7fdcdf800700 1 -- 172.21.15.63:0/3464220844 wait complete. 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.729+0000 7fdcdf800700 1 Processor -- start 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.729+0000 7fdcdf800700 1 -- start start 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdf800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd8102e00 0x7fdcd8113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdf800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd81037a0 0x7fdcd8113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdf800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd81145d0 con 0x7fdcd81037a0 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdf800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd810d9c0 con 0x7fdcd8102e00 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcd7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd81037a0 0x7fdcd8113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdce00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd8102e00 0x7fdcd8113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcd7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd81037a0 0x7fdcd8113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55054/0 (socket says 172.21.15.63:55054) 2024-03-23T12:13:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdce00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd8102e00 0x7fdcd8113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:55532/0 (socket says 172.21.15.63:55532) 2024-03-23T12:13:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcd7e00700 1 -- 172.21.15.63:0/1494933564 learned_addr learned my addr 172.21.15.63:0/1494933564 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdce00700 1 -- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd81037a0 msgr2=0x7fdcd8113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdce00700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd81037a0 0x7fdcd8113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcdce00700 1 -- 172.21.15.63:0/1494933564 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc005040 con 0x7fdcd8102e00 2024-03-23T12:13:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.730+0000 7fdcd7e00700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd81037a0 0x7fdcd8113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T12:13:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.731+0000 7fdcdce00700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd8102e00 0x7fdcd8113a40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009fd0 tx=0x7fdccc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:33.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.731+0000 7fdcd4c00700 1 -- 172.21.15.63:0/1494933564 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdccc00e070 con 0x7fdcd8102e00 2024-03-23T12:13:33.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.731+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd810dc40 con 0x7fdcd8102e00 2024-03-23T12:13:33.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.731+0000 7fdcd4c00700 1 -- 172.21.15.63:0/1494933564 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdccc013e00 con 0x7fdcd8102e00 2024-03-23T12:13:33.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.731+0000 7fdcd4c00700 1 -- 172.21.15.63:0/1494933564 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdccc01f070 con 0x7fdcd8102e00 2024-03-23T12:13:33.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.731+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd810e200 con 0x7fdcd8102e00 2024-03-23T12:13:33.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.732+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd80fe550 con 0x7fdcd8102e00 2024-03-23T12:13:33.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.733+0000 7fdcd4c00700 1 -- 172.21.15.63:0/1494933564 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdccc01a800 con 0x7fdcd8102e00 2024-03-23T12:13:33.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.734+0000 7fdcd4c00700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc006c7d0 0x7fdcc006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:33.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.734+0000 7fdcd4c00700 1 -- 172.21.15.63:0/1494933564 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdccc090c10 con 0x7fdcd8102e00 2024-03-23T12:13:33.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.734+0000 7fdcd7e00700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc006c7d0 0x7fdcc006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:33.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.735+0000 7fdcd7e00700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc006c7d0 0x7fdcc006ec80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdcd810eb80 tx=0x7fdcc400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:33.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.740+0000 7fdcd4c00700 1 -- 172.21.15.63:0/1494933564 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdccc05e8a0 con 0x7fdcd8102e00 2024-03-23T12:13:33.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:33.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:33.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:33 smithi063 ceph-mon[141124]: pgmap v674: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:33.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:33.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:33.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:33 smithi111 ceph-mon[106899]: pgmap v674: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:33.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.975+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcd8109e30 con 0x7fdcc006c7d0 2024-03-23T12:13:33.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.977+0000 7fdcd4c00700 1 -- 172.21.15.63:0/1494933564 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdcd8109e30 con 0x7fdcc006c7d0 2024-03-23T12:13:33.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc006c7d0 msgr2=0x7fdcc006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc006c7d0 0x7fdcc006ec80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdcd810eb80 tx=0x7fdcc400b040 comp rx=0 tx=0).stop 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd8102e00 msgr2=0x7fdcd8113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd8102e00 0x7fdcd8113a40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009fd0 tx=0x7fdccc009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 shutdown_connections 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdcc006c7d0 0x7fdcc006ec80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcd8102e00 0x7fdcd8113a40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 --2- 172.21.15.63:0/1494933564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcd81037a0 0x7fdcd8113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 >> 172.21.15.63:0/1494933564 conn(0x7fdcd80f89b0 msgr2=0x7fdcd80f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 shutdown_connections 2024-03-23T12:13:33.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:33.979+0000 7fdcdf800700 1 -- 172.21.15.63:0/1494933564 wait complete. 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 -- 172.21.15.63:0/1604909873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6564102e00 msgr2=0x7f65641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 --2- 172.21.15.63:0/1604909873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6564102e00 0x7f65641031d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6554004940 tx=0x7f6554009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 -- 172.21.15.63:0/1604909873 shutdown_connections 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 --2- 172.21.15.63:0/1604909873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65641037a0 0x7f65640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 --2- 172.21.15.63:0/1604909873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6564102e00 0x7f65641031d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 -- 172.21.15.63:0/1604909873 >> 172.21.15.63:0/1604909873 conn(0x7f65640f89b0 msgr2=0x7f65640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 -- 172.21.15.63:0/1604909873 shutdown_connections 2024-03-23T12:13:34.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.129+0000 7f6569400700 1 -- 172.21.15.63:0/1604909873 wait complete. 2024-03-23T12:13:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.130+0000 7f6569400700 1 Processor -- start 2024-03-23T12:13:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6569400700 1 -- start start 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6569400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f656410af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6569400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f656410b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6569400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656410f0f0 con 0x7f65641037a0 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6569400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656410b9e0 con 0x7f6564102e00 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6561e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f656410b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6562800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f656410af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6562800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f656410af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:55548/0 (socket says 172.21.15.63:55548) 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6561e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f656410b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55068/0 (socket says 172.21.15.63:55068) 2024-03-23T12:13:34.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.131+0000 7f6562800700 1 -- 172.21.15.63:0/2370135499 learned_addr learned my addr 172.21.15.63:0/2370135499 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.132+0000 7f6562800700 1 -- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 msgr2=0x7f656410b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.132+0000 7f6562800700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f656410b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.132+0000 7f6562800700 1 -- 172.21.15.63:0/2370135499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554005040 con 0x7f6564102e00 2024-03-23T12:13:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.132+0000 7f6561e00700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f656410b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T12:13:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.132+0000 7f6562800700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f656410af60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6554007f50 tx=0x7f6554009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.132+0000 7f655aa00700 1 -- 172.21.15.63:0/2370135499 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6554021070 con 0x7f6564102e00 2024-03-23T12:13:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.133+0000 7f655aa00700 1 -- 172.21.15.63:0/2370135499 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65540093b0 con 0x7f6564102e00 2024-03-23T12:13:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.133+0000 7f655aa00700 1 -- 172.21.15.63:0/2370135499 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f655400e070 con 0x7f6564102e00 2024-03-23T12:13:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.133+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656410bc60 con 0x7f6564102e00 2024-03-23T12:13:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.133+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656411e5d0 con 0x7f6564102e00 2024-03-23T12:13:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.134+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65640fe550 con 0x7f6564102e00 2024-03-23T12:13:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.135+0000 7f655aa00700 1 -- 172.21.15.63:0/2370135499 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f655401c400 con 0x7f6564102e00 2024-03-23T12:13:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.136+0000 7f655aa00700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654806c7d0 0x7f654806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.136+0000 7f655aa00700 1 -- 172.21.15.63:0/2370135499 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f655408caf0 con 0x7f6564102e00 2024-03-23T12:13:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.136+0000 7f6561e00700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654806c7d0 0x7f654806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.137+0000 7f6561e00700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654806c7d0 0x7f654806ec80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6550004fd0 tx=0x7f655000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:34.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.140+0000 7f655aa00700 1 -- 172.21.15.63:0/2370135499 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6554056d70 con 0x7f6564102e00 2024-03-23T12:13:34.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.367+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f656410c650 con 0x7f654806c7d0 2024-03-23T12:13:34.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.378+0000 7f655aa00700 1 -- 172.21.15.63:0/2370135499 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f656410c650 con 0x7f654806c7d0 2024-03-23T12:13:34.358 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:13:34.358 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (22m) 15m ago 35m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:13:34.358 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (20m) 15m ago 36m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:13:34.358 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (19m) 1s ago 35m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:13:34.358 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (22m) 15m ago 35m 74.3M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (15m) 15m ago 29m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (15m) 15m ago 29m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (5s) 1s ago 29m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (15m) 1s ago 29m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (25m) 15m ago 37m 506M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (23m) 1s ago 34m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (21m) 15m ago 37m 53.4M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (20m) 1s ago 34m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (23m) 15m ago 36m 7671k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (23m) 1s ago 34m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (19m) 15m ago 33m 92.3M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (18m) 15m ago 33m 94.0M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:13:34.359 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (18m) 15m ago 32m 81.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:13:34.360 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (17m) 1s ago 32m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:13:34.360 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (16m) 1s ago 31m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:13:34.360 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (16m) 1s ago 31m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:13:34.360 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (23m) 15m ago 35m 55.0M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:13:34.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654806c7d0 msgr2=0x7f654806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654806c7d0 0x7f654806ec80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6550004fd0 tx=0x7f655000b040 comp rx=0 tx=0).stop 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 msgr2=0x7f656410af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f656410af60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6554007f50 tx=0x7f6554009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 shutdown_connections 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f654806c7d0 0x7f654806ec80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6564102e00 0x7f656410af60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 --2- 172.21.15.63:0/2370135499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65641037a0 0x7f656410b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 >> 172.21.15.63:0/2370135499 conn(0x7f65640f89b0 msgr2=0x7f65640f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 shutdown_connections 2024-03-23T12:13:34.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.382+0000 7f6569400700 1 -- 172.21.15.63:0/2370135499 wait complete. 2024-03-23T12:13:34.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 -- 172.21.15.63:0/4226064732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d24071f90 msgr2=0x7f3d24070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 --2- 172.21.15.63:0/4226064732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d24071f90 0x7f3d24070360 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3d14004970 tx=0x7f3d14009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 -- 172.21.15.63:0/4226064732 shutdown_connections 2024-03-23T12:13:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 --2- 172.21.15.63:0/4226064732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d240708a0 0x7f3d24070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 --2- 172.21.15.63:0/4226064732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d24071f90 0x7f3d24070360 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 -- 172.21.15.63:0/4226064732 >> 172.21.15.63:0/4226064732 conn(0x7f3d240f89b0 msgr2=0x7f3d240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 -- 172.21.15.63:0/4226064732 shutdown_connections 2024-03-23T12:13:34.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.527+0000 7f3d29c00700 1 -- 172.21.15.63:0/4226064732 wait complete. 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.528+0000 7f3d29c00700 1 Processor -- start 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.528+0000 7f3d29c00700 1 -- start start 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.528+0000 7f3d29c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d240708a0 0x7f3d2411a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.528+0000 7f3d29c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d24071f90 0x7f3d2411a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.528+0000 7f3d29c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2411aba0 con 0x7f3d240708a0 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.528+0000 7f3d29c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d24114020 con 0x7f3d24071f90 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.528+0000 7f3d22a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d240708a0 0x7f3d2411a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d22a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d240708a0 0x7f3d2411a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55098/0 (socket says 172.21.15.63:55098) 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d22a00700 1 -- 172.21.15.63:0/2471086869 learned_addr learned my addr 172.21.15.63:0/2471086869 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d22000700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d24071f90 0x7f3d2411a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d22a00700 1 -- 172.21.15.63:0/2471086869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d24071f90 msgr2=0x7f3d2411a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d22a00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d24071f90 0x7f3d2411a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d22a00700 1 -- 172.21.15.63:0/2471086869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d14005040 con 0x7f3d240708a0 2024-03-23T12:13:34.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d22a00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d240708a0 0x7f3d2411a010 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3d14005010 tx=0x7f3d14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:34.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d1aa00700 1 -- 172.21.15.63:0/2471086869 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d1400e040 con 0x7f3d240708a0 2024-03-23T12:13:34.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d1aa00700 1 -- 172.21.15.63:0/2471086869 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d14004290 con 0x7f3d240708a0 2024-03-23T12:13:34.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d1aa00700 1 -- 172.21.15.63:0/2471086869 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d14013d40 con 0x7f3d240708a0 2024-03-23T12:13:34.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.529+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d241142a0 con 0x7f3d240708a0 2024-03-23T12:13:34.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.530+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d241147e0 con 0x7f3d240708a0 2024-03-23T12:13:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.531+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d24071aa0 con 0x7f3d240708a0 2024-03-23T12:13:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.532+0000 7f3d1aa00700 1 -- 172.21.15.63:0/2471086869 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3d14013520 con 0x7f3d240708a0 2024-03-23T12:13:34.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.532+0000 7f3d1aa00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0806c7d0 0x7f3d0806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.533+0000 7f3d1aa00700 1 -- 172.21.15.63:0/2471086869 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3d1408cc40 con 0x7f3d240708a0 2024-03-23T12:13:34.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.533+0000 7f3d22000700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0806c7d0 0x7f3d0806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.533+0000 7f3d22000700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0806c7d0 0x7f3d0806ec80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d2410cfc0 tx=0x7f3d1000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:34.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.537+0000 7f3d1aa00700 1 -- 172.21.15.63:0/2471086869 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3d14056e40 con 0x7f3d240708a0 2024-03-23T12:13:34.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.839+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3d241103e0 con 0x7f3d240708a0 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.839+0000 7f3d1aa00700 1 -- 172.21.15.63:0/2471086869 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f3d1405a460 con 0x7f3d240708a0 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:13:34.819 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:13:34.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:13:34.820 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:13:34.820 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:13:34.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:13:34.820 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:13:34.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0806c7d0 msgr2=0x7f3d0806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0806c7d0 0x7f3d0806ec80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d2410cfc0 tx=0x7f3d1000b040 comp rx=0 tx=0).stop 2024-03-23T12:13:34.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d240708a0 msgr2=0x7f3d2411a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d240708a0 0x7f3d2411a010 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3d14005010 tx=0x7f3d14009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:34.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 shutdown_connections 2024-03-23T12:13:34.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d0806c7d0 0x7f3d0806ec80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d240708a0 0x7f3d2411a010 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 --2- 172.21.15.63:0/2471086869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d24071f90 0x7f3d2411a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 >> 172.21.15.63:0/2471086869 conn(0x7f3d240f89b0 msgr2=0x7f3d240f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:34.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 shutdown_connections 2024-03-23T12:13:34.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.842+0000 7f3d29c00700 1 -- 172.21.15.63:0/2471086869 wait complete. 2024-03-23T12:13:34.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:34 smithi063 ceph-mon[141124]: from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:34.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:34 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:34.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:34 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:34.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:34 smithi063 ceph-mon[141124]: from='client.45009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:34.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:34 smithi111 ceph-mon[106899]: from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:34.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:34 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:34.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:34 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:34.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:34 smithi111 ceph-mon[106899]: from='client.45009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:34.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1100438857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa0070bf0 msgr2=0x7f5fa01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1100438857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa0070bf0 0x7f5fa01091b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c004940 tx=0x7f5f8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:34.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1100438857 shutdown_connections 2024-03-23T12:13:34.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1100438857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa0070bf0 0x7f5fa01091b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1100438857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fa00702e0 0x7f5fa00706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1100438857 >> 172.21.15.63:0/1100438857 conn(0x7f5fa00f8970 msgr2=0x7f5fa00fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:34.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1100438857 shutdown_connections 2024-03-23T12:13:34.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.980+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1100438857 wait complete. 2024-03-23T12:13:34.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.981+0000 7f5fa5e00700 1 Processor -- start 2024-03-23T12:13:34.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.981+0000 7f5fa5e00700 1 -- start start 2024-03-23T12:13:34.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5fa5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00702e0 0x7f5fa0115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5fa5e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fa0070bf0 0x7f5fa0116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5fa5e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0116a00 con 0x7f5fa00702e0 2024-03-23T12:13:34.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5fa5e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa010fd40 con 0x7f5fa0070bf0 2024-03-23T12:13:34.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5f9ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00702e0 0x7f5fa0115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5f9ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00702e0 0x7f5fa0115de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55106/0 (socket says 172.21.15.63:55106) 2024-03-23T12:13:34.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5f9ea00700 1 -- 172.21.15.63:0/1788001615 learned_addr learned my addr 172.21.15.63:0/1788001615 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:34.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.982+0000 7f5f9e000700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fa0070bf0 0x7f5fa0116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5f9ea00700 1 -- 172.21.15.63:0/1788001615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fa0070bf0 msgr2=0x7f5fa0116320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5f9ea00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fa0070bf0 0x7f5fa0116320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5f9ea00700 1 -- 172.21.15.63:0/1788001615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c005040 con 0x7f5fa00702e0 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5f9ea00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00702e0 0x7f5fa0115de0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000b580 tx=0x7f5f900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5f96a00700 1 -- 172.21.15.63:0/1788001615 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f9000f6f0 con 0x7f5fa00702e0 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5f96a00700 1 -- 172.21.15.63:0/1788001615 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f9000fd00 con 0x7f5fa00702e0 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa010ffc0 con 0x7f5fa00702e0 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5f96a00700 1 -- 172.21.15.63:0/1788001615 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f90011e00 con 0x7f5fa00702e0 2024-03-23T12:13:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.983+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa0110500 con 0x7f5fa00702e0 2024-03-23T12:13:34.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.984+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa0071a20 con 0x7f5fa00702e0 2024-03-23T12:13:34.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.986+0000 7f5f96a00700 1 -- 172.21.15.63:0/1788001615 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5f9000f850 con 0x7f5fa00702e0 2024-03-23T12:13:34.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.987+0000 7f5f96a00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f8406c7d0 0x7f5f8406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:34.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.987+0000 7f5f96a00700 1 -- 172.21.15.63:0/1788001615 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5f9008b5b0 con 0x7f5fa00702e0 2024-03-23T12:13:34.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.987+0000 7f5f9e000700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f8406c7d0 0x7f5f8406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:34.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.987+0000 7f5f9e000700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f8406c7d0 0x7f5f8406ec80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0111680 tx=0x7f5f8c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:34.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:34.991+0000 7f5f96a00700 1 -- 172.21.15.63:0/1788001615 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5f90059380 con 0x7f5fa00702e0 2024-03-23T12:13:35.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.274+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5fa004bf00 con 0x7f5fa00702e0 2024-03-23T12:13:35.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.275+0000 7f5f96a00700 1 -- 172.21.15.63:0/1788001615 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f5f90058f10 con 0x7f5fa00702e0 2024-03-23T12:13:35.254 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:13:35.254 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:13:35.254 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:13:35.254 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:13:35.254 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:13:35.255 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:13:35.256 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f8406c7d0 msgr2=0x7f5f8406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f8406c7d0 0x7f5f8406ec80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0111680 tx=0x7f5f8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00702e0 msgr2=0x7f5fa0115de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00702e0 0x7f5fa0115de0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000b580 tx=0x7f5f900091a0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 shutdown_connections 2024-03-23T12:13:35.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f8406c7d0 0x7f5f8406ec80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fa00702e0 0x7f5fa0115de0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 --2- 172.21.15.63:0/1788001615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5fa0070bf0 0x7f5fa0116320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.277+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 >> 172.21.15.63:0/1788001615 conn(0x7f5fa00f8970 msgr2=0x7f5fa00f9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:35.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.278+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 shutdown_connections 2024-03-23T12:13:35.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.278+0000 7f5fa5e00700 1 -- 172.21.15.63:0/1788001615 wait complete. 2024-03-23T12:13:35.258 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 -- 172.21.15.63:0/3035025555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fde90 msgr2=0x7f62080fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 --2- 172.21.15.63:0/3035025555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fde90 0x7f62080fe260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f61fc004940 tx=0x7f61fc009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 -- 172.21.15.63:0/3035025555 shutdown_connections 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 --2- 172.21.15.63:0/3035025555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fe830 0x7f6208106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 --2- 172.21.15.63:0/3035025555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fde90 0x7f62080fe260 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 -- 172.21.15.63:0/3035025555 >> 172.21.15.63:0/3035025555 conn(0x7f6208072bc0 msgr2=0x7f6208072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 -- 172.21.15.63:0/3035025555 shutdown_connections 2024-03-23T12:13:35.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.424+0000 7f620f600700 1 -- 172.21.15.63:0/3035025555 wait complete. 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.425+0000 7f620f600700 1 Processor -- start 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.425+0000 7f620f600700 1 -- start start 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620f600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fde90 0x7f6208115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620f600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fe830 0x7f62081161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620f600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62081167f0 con 0x7f62080fe830 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620f600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620810fbc0 con 0x7f62080fde90 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620cc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fde90 0x7f6208115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:35.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f6207e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fe830 0x7f62081161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:35.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620cc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fde90 0x7f6208115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:55598/0 (socket says 172.21.15.63:55598) 2024-03-23T12:13:35.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620cc00700 1 -- 172.21.15.63:0/2242565635 learned_addr learned my addr 172.21.15.63:0/2242565635 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:35.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620cc00700 1 -- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fe830 msgr2=0x7f62081161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620cc00700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fe830 0x7f62081161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.426+0000 7f620cc00700 1 -- 172.21.15.63:0/2242565635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc005040 con 0x7f62080fde90 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.427+0000 7f620cc00700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fde90 0x7f6208115c60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61fc000c00 tx=0x7f61fc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.427+0000 7f6204c00700 1 -- 172.21.15.63:0/2242565635 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61fc021070 con 0x7f62080fde90 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.427+0000 7f6204c00700 1 -- 172.21.15.63:0/2242565635 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61fc013e00 con 0x7f62080fde90 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.427+0000 7f6204c00700 1 -- 172.21.15.63:0/2242565635 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61fc00e070 con 0x7f62080fde90 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.427+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620810fe40 con 0x7f62080fde90 2024-03-23T12:13:35.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.427+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6208110380 con 0x7f62080fde90 2024-03-23T12:13:35.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.429+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6208106a20 con 0x7f62080fde90 2024-03-23T12:13:35.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.429+0000 7f6204c00700 1 -- 172.21.15.63:0/2242565635 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f61fc013950 con 0x7f62080fde90 2024-03-23T12:13:35.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.430+0000 7f6204c00700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f61f006c7d0 0x7f61f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:35.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.430+0000 7f6204c00700 1 -- 172.21.15.63:0/2242565635 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f61fc08cac0 con 0x7f62080fde90 2024-03-23T12:13:35.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.430+0000 7f6207e00700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f61f006c7d0 0x7f61f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:35.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.431+0000 7f6207e00700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f61f006c7d0 0x7f61f006ec80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6208111500 tx=0x7f61f400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:35.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.435+0000 7f6204c00700 1 -- 172.21.15.63:0/2242565635 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61fc05a750 con 0x7f62080fde90 2024-03-23T12:13:35.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.706+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6208109dc0 con 0x7f61f006c7d0 2024-03-23T12:13:35.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.707+0000 7f6204c00700 1 -- 172.21.15.63:0/2242565635 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6208109dc0 con 0x7f61f006c7d0 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:13:35.687 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:13:35.688 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:13:35.688 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f61f006c7d0 msgr2=0x7f61f006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f61f006c7d0 0x7f61f006ec80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6208111500 tx=0x7f61f400b040 comp rx=0 tx=0).stop 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fde90 msgr2=0x7f6208115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fde90 0x7f6208115c60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61fc000c00 tx=0x7f61fc009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 shutdown_connections 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f61f006c7d0 0x7f61f006ec80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62080fde90 0x7f6208115c60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 --2- 172.21.15.63:0/2242565635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62080fe830 0x7f62081161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 >> 172.21.15.63:0/2242565635 conn(0x7f6208072bc0 msgr2=0x7f62080fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:35.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 shutdown_connections 2024-03-23T12:13:35.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.710+0000 7f620f600700 1 -- 172.21.15.63:0/2242565635 wait complete. 2024-03-23T12:13:35.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.852+0000 7f5de1200700 1 -- 172.21.15.63:0/390115666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc0724f0 msgr2=0x7f5ddc0728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.852+0000 7f5de1200700 1 --2- 172.21.15.63:0/390115666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc0724f0 0x7f5ddc0728c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc004940 tx=0x7f5dcc009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:35.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.853+0000 7f5de1200700 1 -- 172.21.15.63:0/390115666 shutdown_connections 2024-03-23T12:13:35.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.853+0000 7f5de1200700 1 --2- 172.21.15.63:0/390115666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc072e00 0x7f5ddc10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.853+0000 7f5de1200700 1 --2- 172.21.15.63:0/390115666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc0724f0 0x7f5ddc0728c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.853+0000 7f5de1200700 1 -- 172.21.15.63:0/390115666 >> 172.21.15.63:0/390115666 conn(0x7f5ddc0fac00 msgr2=0x7f5ddc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:35.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.853+0000 7f5de1200700 1 -- 172.21.15.63:0/390115666 shutdown_connections 2024-03-23T12:13:35.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.853+0000 7f5de1200700 1 -- 172.21.15.63:0/390115666 wait complete. 2024-03-23T12:13:35.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5de1200700 1 Processor -- start 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5de1200700 1 -- start start 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5de1200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc0724f0 0x7f5ddc119e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5de1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc072e00 0x7f5ddc11a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5de1200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc11a990 con 0x7f5ddc072e00 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5de1200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc113f70 con 0x7f5ddc0724f0 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5dd9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc072e00 0x7f5ddc11a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.854+0000 7f5dd9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc072e00 0x7f5ddc11a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55146/0 (socket says 172.21.15.63:55146) 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.855+0000 7f5dd9400700 1 -- 172.21.15.63:0/3700615465 learned_addr learned my addr 172.21.15.63:0/3700615465 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:13:35.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.855+0000 7f5dd9e00700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc0724f0 0x7f5ddc119e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.855+0000 7f5dd9e00700 1 -- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc072e00 msgr2=0x7f5ddc11a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.855+0000 7f5dd9e00700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc072e00 0x7f5ddc11a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.855+0000 7f5dd9e00700 1 -- 172.21.15.63:0/3700615465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc005040 con 0x7f5ddc0724f0 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.855+0000 7f5dd9400700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc072e00 0x7f5ddc11a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.855+0000 7f5dd9e00700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc0724f0 0x7f5ddc119e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc000c00 tx=0x7f5dcc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.856+0000 7f5dd2000700 1 -- 172.21.15.63:0/3700615465 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dcc00e040 con 0x7f5ddc0724f0 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.856+0000 7f5dd2000700 1 -- 172.21.15.63:0/3700615465 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dcc004290 con 0x7f5ddc0724f0 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.856+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ddc1141f0 con 0x7f5ddc0724f0 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.856+0000 7f5dd2000700 1 -- 172.21.15.63:0/3700615465 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dcc013e30 con 0x7f5ddc0724f0 2024-03-23T12:13:35.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.856+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc1147b0 con 0x7f5ddc0724f0 2024-03-23T12:13:35.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.857+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc073c30 con 0x7f5ddc0724f0 2024-03-23T12:13:35.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.859+0000 7f5dd2000700 1 -- 172.21.15.63:0/3700615465 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5dcc013620 con 0x7f5ddc0724f0 2024-03-23T12:13:35.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.860+0000 7f5dd2000700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dc006c760 0x7f5dc006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:13:35.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.860+0000 7f5dd2000700 1 -- 172.21.15.63:0/3700615465 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5dcc08cbf0 con 0x7f5ddc0724f0 2024-03-23T12:13:35.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.860+0000 7f5dd9400700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dc006c760 0x7f5dc006ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:13:35.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.861+0000 7f5dd9400700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dc006c760 0x7f5dc006ec10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc1156a0 tx=0x7f5dc800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:13:35.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:35.864+0000 7f5dd2000700 1 -- 172.21.15.63:0/3700615465 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5dcc093080 con 0x7f5ddc0724f0 2024-03-23T12:13:35.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:35 smithi111 ceph-mon[106899]: from='client.45013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:35.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:35 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2471086869' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:35.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:35 smithi111 ceph-mon[106899]: pgmap v675: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:35.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:35 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1788001615' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:13:35.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:35.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:35 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:36.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:35 smithi063 ceph-mon[141124]: from='client.45013 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:36.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:35 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2471086869' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:13:36.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:35 smithi063 ceph-mon[141124]: pgmap v675: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:36.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:35 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1788001615' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:13:36.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:36.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:35 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:13:36.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.165+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5ddc115450 con 0x7f5ddc0724f0 2024-03-23T12:13:36.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.165+0000 7f5dd2000700 1 -- 172.21.15.63:0/3700615465 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5dcc05a900 con 0x7f5ddc0724f0 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:13:36.145 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:13:36.146 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:13:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dc006c760 msgr2=0x7f5dc006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dc006c760 0x7f5dc006ec10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc1156a0 tx=0x7f5dc800b040 comp rx=0 tx=0).stop 2024-03-23T12:13:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc0724f0 msgr2=0x7f5ddc119e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:13:36.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc0724f0 0x7f5ddc119e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc000c00 tx=0x7f5dcc009e30 comp rx=0 tx=0).stop 2024-03-23T12:13:36.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 shutdown_connections 2024-03-23T12:13:36.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dc006c760 0x7f5dc006ec10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:36.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ddc0724f0 0x7f5ddc119e00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:36.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 --2- 172.21.15.63:0/3700615465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ddc072e00 0x7f5ddc11a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:13:36.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 >> 172.21.15.63:0/3700615465 conn(0x7f5ddc0fac00 msgr2=0x7f5ddc0fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:13:36.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 shutdown_connections 2024-03-23T12:13:36.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:13:36.168+0000 7f5de1200700 1 -- 172.21.15.63:0/3700615465 wait complete. 2024-03-23T12:13:36.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:36 smithi111 ceph-mon[106899]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:36.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:36 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3700615465' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:13:37.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:36 smithi063 ceph-mon[141124]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:13:37.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:36 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3700615465' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:13:37.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:37 smithi111 ceph-mon[106899]: pgmap v676: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:38.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:37 smithi063 ceph-mon[141124]: pgmap v676: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:40.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:40 smithi063 ceph-mon[141124]: pgmap v677: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:40.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:40 smithi111 ceph-mon[106899]: pgmap v677: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:41.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:41 smithi063 ceph-mon[141124]: pgmap v678: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:41 smithi111 ceph-mon[106899]: pgmap v678: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:44.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:44 smithi063 ceph-mon[141124]: pgmap v679: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:44.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:44 smithi111 ceph-mon[106899]: pgmap v679: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:46.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:46 smithi063 ceph-mon[141124]: pgmap v680: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:46.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:46 smithi111 ceph-mon[106899]: pgmap v680: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:48.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:48 smithi063 ceph-mon[141124]: pgmap v681: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:48.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:48 smithi111 ceph-mon[106899]: pgmap v681: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:50.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:50 smithi063 ceph-mon[141124]: pgmap v682: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:50.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:50 smithi111 ceph-mon[106899]: pgmap v682: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:51.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:51 smithi063 ceph-mon[141124]: pgmap v683: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:51.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:51 smithi111 ceph-mon[106899]: pgmap v683: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:54.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:54 smithi063 ceph-mon[141124]: pgmap v684: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:54.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:54 smithi111 ceph-mon[106899]: pgmap v684: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:56.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:56 smithi063 ceph-mon[141124]: pgmap v685: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:56.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:56.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:56 smithi111 ceph-mon[106899]: pgmap v685: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:56.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:13:58.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:13:58 smithi063 ceph-mon[141124]: pgmap v686: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:13:58.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:13:58 smithi111 ceph-mon[106899]: pgmap v686: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:00.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:00 smithi063 ceph-mon[141124]: pgmap v687: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:00 smithi111 ceph-mon[106899]: pgmap v687: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:01.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:01 smithi063 ceph-mon[141124]: pgmap v688: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:01.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:01 smithi111 ceph-mon[106899]: pgmap v688: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:04.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:04 smithi063 ceph-mon[141124]: pgmap v689: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:04.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:04 smithi111 ceph-mon[106899]: pgmap v689: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:06.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.320+0000 7fe1c4200700 1 -- 172.21.15.63:0/3669875122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 msgr2=0x7fe1bc0728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.320+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3669875122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc0728c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0004940 tx=0x7fe1b0009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.320+0000 7fe1c4200700 1 -- 172.21.15.63:0/3669875122 shutdown_connections 2024-03-23T12:14:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.320+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3669875122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1bc072e00 0x7fe1bc10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.320+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3669875122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc0728c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.320+0000 7fe1c4200700 1 -- 172.21.15.63:0/3669875122 >> 172.21.15.63:0/3669875122 conn(0x7fe1bc0fac00 msgr2=0x7fe1bc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.320+0000 7fe1c4200700 1 -- 172.21.15.63:0/3669875122 shutdown_connections 2024-03-23T12:14:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.321+0000 7fe1c4200700 1 -- 172.21.15.63:0/3669875122 wait complete. 2024-03-23T12:14:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.321+0000 7fe1c4200700 1 Processor -- start 2024-03-23T12:14:06.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c4200700 1 -- start start 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c4200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc119e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c4200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1bc072e00 0x7fe1bc11a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c4200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc11a990 con 0x7fe1bc0724f0 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c4200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc113f70 con 0x7fe1bc072e00 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc119e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc119e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51000/0 (socket says 172.21.15.63:51000) 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c1800700 1 -- 172.21.15.63:0/3015236538 learned_addr learned my addr 172.21.15.63:0/3015236538 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.322+0000 7fe1c0e00700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1bc072e00 0x7fe1bc11a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1c1800700 1 -- 172.21.15.63:0/3015236538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1bc072e00 msgr2=0x7fe1bc11a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1c1800700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1bc072e00 0x7fe1bc11a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1c1800700 1 -- 172.21.15.63:0/3015236538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b0005040 con 0x7fe1bc0724f0 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1c1800700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc119e00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009390 tx=0x7fe1b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1b9600700 1 -- 172.21.15.63:0/3015236538 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1b000e040 con 0x7fe1bc0724f0 2024-03-23T12:14:06.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1bc1141f0 con 0x7fe1bc0724f0 2024-03-23T12:14:06.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1b9600700 1 -- 172.21.15.63:0/3015236538 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1b0013420 con 0x7fe1bc0724f0 2024-03-23T12:14:06.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.323+0000 7fe1b9600700 1 -- 172.21.15.63:0/3015236538 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1b001c630 con 0x7fe1bc0724f0 2024-03-23T12:14:06.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.324+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1bc1147b0 con 0x7fe1bc0724f0 2024-03-23T12:14:06.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.325+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc073c30 con 0x7fe1bc0724f0 2024-03-23T12:14:06.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.326+0000 7fe1b9600700 1 -- 172.21.15.63:0/3015236538 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe1b0013a20 con 0x7fe1bc0724f0 2024-03-23T12:14:06.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.327+0000 7fe1b9600700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1a406c7c0 0x7fe1a406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:06.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.327+0000 7fe1b9600700 1 -- 172.21.15.63:0/3015236538 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe1b008cbb0 con 0x7fe1bc0724f0 2024-03-23T12:14:06.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.327+0000 7fe1c0e00700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1a406c7c0 0x7fe1a406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:06.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.328+0000 7fe1c0e00700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1a406c7c0 0x7fe1a406ec70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc1156a0 tx=0x7fe1a800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:06.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.332+0000 7fe1b9600700 1 -- 172.21.15.63:0/3015236538 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe1b0093080 con 0x7fe1bc0724f0 2024-03-23T12:14:06.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:06 smithi063 ceph-mon[141124]: pgmap v690: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:06.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:06 smithi111 ceph-mon[106899]: pgmap v690: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:06.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.564+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1bc115450 con 0x7fe1a406c7c0 2024-03-23T12:14:06.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.565+0000 7fe1b9600700 1 -- 172.21.15.63:0/3015236538 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe1bc115450 con 0x7fe1a406c7c0 2024-03-23T12:14:06.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1a406c7c0 msgr2=0x7fe1a406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1a406c7c0 0x7fe1a406ec70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc1156a0 tx=0x7fe1a800b040 comp rx=0 tx=0).stop 2024-03-23T12:14:06.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 msgr2=0x7fe1bc119e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc119e00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009390 tx=0x7fe1b0009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:06.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 shutdown_connections 2024-03-23T12:14:06.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe1a406c7c0 0x7fe1a406ec70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1bc0724f0 0x7fe1bc119e00 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 --2- 172.21.15.63:0/3015236538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1bc072e00 0x7fe1bc11a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 >> 172.21.15.63:0/3015236538 conn(0x7fe1bc0fac00 msgr2=0x7fe1bc0fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:06.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 shutdown_connections 2024-03-23T12:14:06.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.568+0000 7fe1c4200700 1 -- 172.21.15.63:0/3015236538 wait complete. 2024-03-23T12:14:06.563 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:14:06.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 -- 172.21.15.63:0/2647070255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 msgr2=0x7fd6980fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 --2- 172.21.15.63:0/2647070255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd6980fe260 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd688004940 tx=0x7fd688009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 -- 172.21.15.63:0/2647070255 shutdown_connections 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 --2- 172.21.15.63:0/2647070255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6980fe830 0x7fd698106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 --2- 172.21.15.63:0/2647070255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd6980fe260 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 -- 172.21.15.63:0/2647070255 >> 172.21.15.63:0/2647070255 conn(0x7fd698072bc0 msgr2=0x7fd698072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 -- 172.21.15.63:0/2647070255 shutdown_connections 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.706+0000 7fd69e600700 1 -- 172.21.15.63:0/2647070255 wait complete. 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.707+0000 7fd69e600700 1 Processor -- start 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.707+0000 7fd69e600700 1 -- start start 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd69e600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd698115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd69e600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6980fe830 0x7fd698116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:06.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd69e600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd698116890 con 0x7fd6980fde90 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd69e600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69810fc60 con 0x7fd6980fe830 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd697400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd698115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd697400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd698115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51024/0 (socket says 172.21.15.63:51024) 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd697400700 1 -- 172.21.15.63:0/2938899739 learned_addr learned my addr 172.21.15.63:0/2938899739 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd697400700 1 -- 172.21.15.63:0/2938899739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6980fe830 msgr2=0x7fd698116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd696a00700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6980fe830 0x7fd698116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd697400700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6980fe830 0x7fd698116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd697400700 1 -- 172.21.15.63:0/2938899739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688005040 con 0x7fd6980fde90 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.708+0000 7fd696a00700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6980fe830 0x7fd698116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.709+0000 7fd697400700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd698115d00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd688000c00 tx=0x7fd688009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.709+0000 7fd68f400700 1 -- 172.21.15.63:0/2938899739 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68800e040 con 0x7fd6980fde90 2024-03-23T12:14:06.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.709+0000 7fd68f400700 1 -- 172.21.15.63:0/2938899739 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd688013420 con 0x7fd6980fde90 2024-03-23T12:14:06.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.709+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69810fee0 con 0x7fd6980fde90 2024-03-23T12:14:06.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.709+0000 7fd68f400700 1 -- 172.21.15.63:0/2938899739 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68801c630 con 0x7fd6980fde90 2024-03-23T12:14:06.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.709+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd698110420 con 0x7fd6980fde90 2024-03-23T12:14:06.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.710+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd698106a20 con 0x7fd6980fde90 2024-03-23T12:14:06.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.711+0000 7fd68f400700 1 -- 172.21.15.63:0/2938899739 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd688013970 con 0x7fd6980fde90 2024-03-23T12:14:06.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.712+0000 7fd68f400700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd67c06c7d0 0x7fd67c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:06.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.712+0000 7fd68f400700 1 -- 172.21.15.63:0/2938899739 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd68808cd20 con 0x7fd6980fde90 2024-03-23T12:14:06.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.712+0000 7fd696a00700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd67c06c7d0 0x7fd67c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:06.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.713+0000 7fd696a00700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd67c06c7d0 0x7fd67c06ec80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd684009fd0 tx=0x7fd68400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:06.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.717+0000 7fd68f400700 1 -- 172.21.15.63:0/2938899739 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd68805a9b0 con 0x7fd6980fde90 2024-03-23T12:14:06.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.949+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd698109dc0 con 0x7fd67c06c7d0 2024-03-23T12:14:06.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.951+0000 7fd68f400700 1 -- 172.21.15.63:0/2938899739 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd698109dc0 con 0x7fd67c06c7d0 2024-03-23T12:14:06.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd67c06c7d0 msgr2=0x7fd67c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd67c06c7d0 0x7fd67c06ec80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd684009fd0 tx=0x7fd68400b040 comp rx=0 tx=0).stop 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 msgr2=0x7fd698115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd698115d00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd688000c00 tx=0x7fd688009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 shutdown_connections 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd67c06c7d0 0x7fd67c06ec80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6980fde90 0x7fd698115d00 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 --2- 172.21.15.63:0/2938899739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6980fe830 0x7fd698116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 >> 172.21.15.63:0/2938899739 conn(0x7fd698072bc0 msgr2=0x7fd6980fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 shutdown_connections 2024-03-23T12:14:06.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:06.953+0000 7fd69e600700 1 -- 172.21.15.63:0/2938899739 wait complete. 2024-03-23T12:14:07.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 -- 172.21.15.63:0/3510583206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 msgr2=0x7fd910105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 --2- 172.21.15.63:0/3510583206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910105220 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd900004940 tx=0x7fd900009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:07.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 -- 172.21.15.63:0/3510583206 shutdown_connections 2024-03-23T12:14:07.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 --2- 172.21.15.63:0/3510583206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9101057f0 0x7fd9100fee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 --2- 172.21.15.63:0/3510583206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910105220 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 -- 172.21.15.63:0/3510583206 >> 172.21.15.63:0/3510583206 conn(0x7fd9100fab10 msgr2=0x7fd9100fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:07.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 -- 172.21.15.63:0/3510583206 shutdown_connections 2024-03-23T12:14:07.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.092+0000 7fd90f200700 1 -- 172.21.15.63:0/3510583206 wait complete. 2024-03-23T12:14:07.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.093+0000 7fd90f200700 1 Processor -- start 2024-03-23T12:14:07.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.093+0000 7fd90f200700 1 -- start start 2024-03-23T12:14:07.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910115b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9101057f0 0x7fd910116050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101166a0 con 0x7fd910104e50 2024-03-23T12:14:07.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91010fa90 con 0x7fd9101057f0 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910115b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910115b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51040/0 (socket says 172.21.15.63:51040) 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90de00700 1 -- 172.21.15.63:0/371225730 learned_addr learned my addr 172.21.15.63:0/371225730 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90d400700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9101057f0 0x7fd910116050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90de00700 1 -- 172.21.15.63:0/371225730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9101057f0 msgr2=0x7fd910116050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90de00700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9101057f0 0x7fd910116050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.094+0000 7fd90de00700 1 -- 172.21.15.63:0/371225730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900005040 con 0x7fd910104e50 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.095+0000 7fd90de00700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910115b10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd900009360 tx=0x7fd900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.095+0000 7fd906000700 1 -- 172.21.15.63:0/371225730 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd90000e040 con 0x7fd910104e50 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.095+0000 7fd906000700 1 -- 172.21.15.63:0/371225730 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd900013420 con 0x7fd910104e50 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.095+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91010fd10 con 0x7fd910104e50 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.095+0000 7fd906000700 1 -- 172.21.15.63:0/371225730 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd90001c5d0 con 0x7fd910104e50 2024-03-23T12:14:07.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.095+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9101102d0 con 0x7fd910104e50 2024-03-23T12:14:07.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.096+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9101005a0 con 0x7fd910104e50 2024-03-23T12:14:07.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.097+0000 7fd906000700 1 -- 172.21.15.63:0/371225730 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd900013590 con 0x7fd910104e50 2024-03-23T12:14:07.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.099+0000 7fd906000700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd8f406c7d0 0x7fd8f406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.099+0000 7fd906000700 1 -- 172.21.15.63:0/371225730 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd90008cbb0 con 0x7fd910104e50 2024-03-23T12:14:07.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.099+0000 7fd90d400700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd8f406c7d0 0x7fd8f406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.100+0000 7fd90d400700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd8f406c7d0 0x7fd8f406ec80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd910110c50 tx=0x7fd8fc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:07.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.103+0000 7fd906000700 1 -- 172.21.15.63:0/371225730 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd90005a8c0 con 0x7fd910104e50 2024-03-23T12:14:07.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.326+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd91010be80 con 0x7fd8f406c7d0 2024-03-23T12:14:07.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.336+0000 7fd906000700 1 -- 172.21.15.63:0/371225730 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fd91010be80 con 0x7fd8f406c7d0 2024-03-23T12:14:07.316 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:14:07.316 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (23m) 31s ago 36m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:14:07.316 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (20m) 31s ago 36m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:14:07.316 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (20m) 34s ago 35m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:14:07.316 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (23m) 31s ago 36m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:14:07.316 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (16m) 31s ago 30m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (16m) 31s ago 30m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (38s) 34s ago 30m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (15m) 34s ago 30m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (25m) 31s ago 38m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (24m) 34s ago 35m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (21m) 31s ago 38m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (21m) 34s ago 35m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (24m) 31s ago 36m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (24m) 34s ago 35m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (20m) 31s ago 34m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (19m) 31s ago 33m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (18m) 31s ago 33m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (18m) 34s ago 32m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (17m) 34s ago 32m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (16m) 34s ago 31m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:14:07.317 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (23m) 31s ago 36m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd8f406c7d0 msgr2=0x7fd8f406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd8f406c7d0 0x7fd8f406ec80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd910110c50 tx=0x7fd8fc00b040 comp rx=0 tx=0).stop 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 msgr2=0x7fd910115b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910115b10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd900009360 tx=0x7fd900009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 shutdown_connections 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd8f406c7d0 0x7fd8f406ec80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd910104e50 0x7fd910115b10 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 --2- 172.21.15.63:0/371225730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9101057f0 0x7fd910116050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.338+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 >> 172.21.15.63:0/371225730 conn(0x7fd9100fab10 msgr2=0x7fd9100fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.339+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 shutdown_connections 2024-03-23T12:14:07.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.339+0000 7fd90f200700 1 -- 172.21.15.63:0/371225730 wait complete. 2024-03-23T12:14:07.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.475+0000 7f8767400700 1 -- 172.21.15.63:0/1516759627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 msgr2=0x7f8760070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.475+0000 7f8767400700 1 --2- 172.21.15.63:0/1516759627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f8760070360 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8754004970 tx=0x7f8754009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:07.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.475+0000 7f8767400700 1 -- 172.21.15.63:0/1516759627 shutdown_connections 2024-03-23T12:14:07.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.475+0000 7f8767400700 1 --2- 172.21.15.63:0/1516759627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87600708a0 0x7f8760070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.475+0000 7f8767400700 1 --2- 172.21.15.63:0/1516759627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f8760070360 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.475+0000 7f8767400700 1 -- 172.21.15.63:0/1516759627 >> 172.21.15.63:0/1516759627 conn(0x7f87600f89b0 msgr2=0x7f87600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:07.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.476+0000 7f8767400700 1 -- 172.21.15.63:0/1516759627 shutdown_connections 2024-03-23T12:14:07.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.476+0000 7f8767400700 1 -- 172.21.15.63:0/1516759627 wait complete. 2024-03-23T12:14:07.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.476+0000 7f8767400700 1 Processor -- start 2024-03-23T12:14:07.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f8767400700 1 -- start start 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f8767400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87600708a0 0x7f876011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f8767400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f876011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f8767400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876011ac50 con 0x7f8760071f90 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f8767400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8760114020 con 0x7f87600708a0 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f875fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f876011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f875fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f876011a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51062/0 (socket says 172.21.15.63:51062) 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f875fe00700 1 -- 172.21.15.63:0/4071947038 learned_addr learned my addr 172.21.15.63:0/4071947038 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.477+0000 7f8764a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87600708a0 0x7f876011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f875fe00700 1 -- 172.21.15.63:0/4071947038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87600708a0 msgr2=0x7f876011a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f875fe00700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87600708a0 0x7f876011a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f875fe00700 1 -- 172.21.15.63:0/4071947038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8754005040 con 0x7f8760071f90 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f875fe00700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f876011a600 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f874c002a80 tx=0x7f874c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:07.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f875cc00700 1 -- 172.21.15.63:0/4071947038 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f874c007cc0 con 0x7f8760071f90 2024-03-23T12:14:07.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f875cc00700 1 -- 172.21.15.63:0/4071947038 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f874c00f040 con 0x7f8760071f90 2024-03-23T12:14:07.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f875cc00700 1 -- 172.21.15.63:0/4071947038 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f874c00da60 con 0x7f8760071f90 2024-03-23T12:14:07.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8760114300 con 0x7f8760071f90 2024-03-23T12:14:07.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.478+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8760114840 con 0x7f8760071f90 2024-03-23T12:14:07.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.479+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8760071aa0 con 0x7f8760071f90 2024-03-23T12:14:07.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.480+0000 7f875cc00700 1 -- 172.21.15.63:0/4071947038 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f874c00dbc0 con 0x7f8760071f90 2024-03-23T12:14:07.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.481+0000 7f875cc00700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f874806c7d0 0x7f874806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.481+0000 7f875cc00700 1 -- 172.21.15.63:0/4071947038 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f874c08b630 con 0x7f8760071f90 2024-03-23T12:14:07.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.482+0000 7f8764a00700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f874806c7d0 0x7f874806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.482+0000 7f8764a00700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f874806c7d0 0x7f874806ec80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f875400b370 tx=0x7f8754009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:07.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.486+0000 7f875cc00700 1 -- 172.21.15.63:0/4071947038 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f874c059340 con 0x7f8760071f90 2024-03-23T12:14:07.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.787+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f876005f5f0 con 0x7f8760071f90 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.788+0000 7f875cc00700 1 -- 172.21.15.63:0/4071947038 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f874c058ed0 con 0x7f8760071f90 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:14:07.769 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:14:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f874806c7d0 msgr2=0x7f874806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f874806c7d0 0x7f874806ec80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f875400b370 tx=0x7f8754009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 msgr2=0x7f876011a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f876011a600 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f874c002a80 tx=0x7f874c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 shutdown_connections 2024-03-23T12:14:07.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f874806c7d0 0x7f874806ec80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87600708a0 0x7f876011a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 --2- 172.21.15.63:0/4071947038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8760071f90 0x7f876011a600 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 >> 172.21.15.63:0/4071947038 conn(0x7f87600f89b0 msgr2=0x7f87600f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:07.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 shutdown_connections 2024-03-23T12:14:07.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.790+0000 7f8767400700 1 -- 172.21.15.63:0/4071947038 wait complete. 2024-03-23T12:14:07.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 -- 172.21.15.63:0/145806869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 msgr2=0x7f2c100fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 --2- 172.21.15.63:0/145806869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c100fce10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc004940 tx=0x7f2bfc009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:07.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 -- 172.21.15.63:0/145806869 shutdown_connections 2024-03-23T12:14:07.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 --2- 172.21.15.63:0/145806869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c100fce10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 --2- 172.21.15.63:0/145806869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c10102e00 0x7f2c101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 -- 172.21.15.63:0/145806869 >> 172.21.15.63:0/145806869 conn(0x7f2c100f89b0 msgr2=0x7f2c100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:07.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 -- 172.21.15.63:0/145806869 shutdown_connections 2024-03-23T12:14:07.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.927+0000 7f2c16400700 1 -- 172.21.15.63:0/145806869 wait complete. 2024-03-23T12:14:07.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.928+0000 7f2c16400700 1 Processor -- start 2024-03-23T12:14:07.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.928+0000 7f2c16400700 1 -- start start 2024-03-23T12:14:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c16400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c10102e00 0x7f2c10115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c16400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c101161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c16400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c101167f0 con 0x7f2c101037a0 2024-03-23T12:14:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c16400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1010fbc0 con 0x7f2c10102e00 2024-03-23T12:14:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c0ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c101161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c0ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c101161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51082/0 (socket says 172.21.15.63:51082) 2024-03-23T12:14:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c0ea00700 1 -- 172.21.15.63:0/3146440821 learned_addr learned my addr 172.21.15.63:0/3146440821 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.929+0000 7f2c0f400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c10102e00 0x7f2c10115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c0ea00700 1 -- 172.21.15.63:0/3146440821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c10102e00 msgr2=0x7f2c10115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c0ea00700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c10102e00 0x7f2c10115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c0ea00700 1 -- 172.21.15.63:0/3146440821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc005040 con 0x7f2c101037a0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c0ea00700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c101161a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009fd0 tx=0x7f2bfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c07400700 1 -- 172.21.15.63:0/3146440821 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bfc00e030 con 0x7f2c101037a0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c07400700 1 -- 172.21.15.63:0/3146440821 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bfc013e00 con 0x7f2c101037a0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c07400700 1 -- 172.21.15.63:0/3146440821 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bfc01ccd0 con 0x7f2c101037a0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1010fe40 con 0x7f2c101037a0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.930+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c10110400 con 0x7f2c101037a0 2024-03-23T12:14:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.931+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c100fe550 con 0x7f2c101037a0 2024-03-23T12:14:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.932+0000 7f2c07400700 1 -- 172.21.15.63:0/3146440821 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2bfc01c500 con 0x7f2c101037a0 2024-03-23T12:14:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.933+0000 7f2c07400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2bf406c7d0 0x7f2bf406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.934+0000 7f2c07400700 1 -- 172.21.15.63:0/3146440821 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2bfc08d2f0 con 0x7f2c101037a0 2024-03-23T12:14:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.934+0000 7f2c0f400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2bf406c7d0 0x7f2bf406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.934+0000 7f2c0f400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2bf406c7d0 0x7f2bf406ec80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2c00009fd0 tx=0x7f2c0000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:07.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:07.938+0000 7f2c07400700 1 -- 172.21.15.63:0/3146440821 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2bfc05b000 con 0x7f2c101037a0 2024-03-23T12:14:08.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.197+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2c1004bed0 con 0x7f2c101037a0 2024-03-23T12:14:08.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.197+0000 7f2c07400700 1 -- 172.21.15.63:0/3146440821 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f2bfc05ab90 con 0x7f2c101037a0 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:14:08.178 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:14:08.179 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:08.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:08.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2bf406c7d0 msgr2=0x7f2bf406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2bf406c7d0 0x7f2bf406ec80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2c00009fd0 tx=0x7f2c0000b040 comp rx=0 tx=0).stop 2024-03-23T12:14:08.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 msgr2=0x7f2c101161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c101161a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009fd0 tx=0x7f2bfc009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:08.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 shutdown_connections 2024-03-23T12:14:08.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2bf406c7d0 0x7f2bf406ec80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c10102e00 0x7f2c10115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 --2- 172.21.15.63:0/3146440821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c101037a0 0x7f2c101161a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 >> 172.21.15.63:0/3146440821 conn(0x7f2c100f89b0 msgr2=0x7f2c100f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:08.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 shutdown_connections 2024-03-23T12:14:08.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.200+0000 7f2c16400700 1 -- 172.21.15.63:0/3146440821 wait complete. 2024-03-23T12:14:08.182 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:14:08.282 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:08 smithi111 ceph-mon[106899]: from='client.35278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:08.282 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:08 smithi111 ceph-mon[106899]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:08.282 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:08 smithi111 ceph-mon[106899]: pgmap v691: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:08.283 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:08 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4071947038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.337+0000 7fee97a00700 1 -- 172.21.15.63:0/1637933097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 msgr2=0x7fee900fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.337+0000 7fee97a00700 1 --2- 172.21.15.63:0/1637933097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fe260 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fee84004940 tx=0x7fee84009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.337+0000 7fee97a00700 1 -- 172.21.15.63:0/1637933097 shutdown_connections 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.337+0000 7fee97a00700 1 --2- 172.21.15.63:0/1637933097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee900fe830 0x7fee90106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.338+0000 7fee97a00700 1 --2- 172.21.15.63:0/1637933097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fe260 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.338+0000 7fee97a00700 1 -- 172.21.15.63:0/1637933097 >> 172.21.15.63:0/1637933097 conn(0x7fee90072bc0 msgr2=0x7fee90072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.338+0000 7fee97a00700 1 -- 172.21.15.63:0/1637933097 shutdown_connections 2024-03-23T12:14:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.338+0000 7fee97a00700 1 -- 172.21.15.63:0/1637933097 wait complete. 2024-03-23T12:14:08.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.339+0000 7fee97a00700 1 Processor -- start 2024-03-23T12:14:08.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.339+0000 7fee97a00700 1 -- start start 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.339+0000 7fee97a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee97a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee900fe830 0x7fee900fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee97a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee900fc8b0 con 0x7fee900fde90 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee97a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee900fc9f0 con 0x7fee900fe830 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee95000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee95000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51096/0 (socket says 172.21.15.63:51096) 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee95000700 1 -- 172.21.15.63:0/2011205187 learned_addr learned my addr 172.21.15.63:0/2011205187 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:08.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee8fe00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee900fe830 0x7fee900fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:08.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee95000700 1 -- 172.21.15.63:0/2011205187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee900fe830 msgr2=0x7fee900fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee95000700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee900fe830 0x7fee900fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee95000700 1 -- 172.21.15.63:0/2011205187 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee84005040 con 0x7fee900fde90 2024-03-23T12:14:08.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.340+0000 7fee95000700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fbd50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fee84009040 tx=0x7fee84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:08.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.341+0000 7fee8cc00700 1 -- 172.21.15.63:0/2011205187 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee8400e030 con 0x7fee900fde90 2024-03-23T12:14:08.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.341+0000 7fee8cc00700 1 -- 172.21.15.63:0/2011205187 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee8400bed0 con 0x7fee900fde90 2024-03-23T12:14:08.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.341+0000 7fee8cc00700 1 -- 172.21.15.63:0/2011205187 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee84013bf0 con 0x7fee900fde90 2024-03-23T12:14:08.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.341+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee90119e90 con 0x7fee900fde90 2024-03-23T12:14:08.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.341+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9011a290 con 0x7fee900fde90 2024-03-23T12:14:08.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.342+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee90065e30 con 0x7fee900fde90 2024-03-23T12:14:08.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.343+0000 7fee8cc00700 1 -- 172.21.15.63:0/2011205187 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fee84013420 con 0x7fee900fde90 2024-03-23T12:14:08.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.344+0000 7fee8cc00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee7806c7d0 0x7fee7806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:08.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.344+0000 7fee8cc00700 1 -- 172.21.15.63:0/2011205187 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fee8408d230 con 0x7fee900fde90 2024-03-23T12:14:08.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.344+0000 7fee8fe00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee7806c7d0 0x7fee7806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:08.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.345+0000 7fee8fe00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee7806c7d0 0x7fee7806ec80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fee900fd690 tx=0x7fee7c0090f0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:08.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.349+0000 7fee8cc00700 1 -- 172.21.15.63:0/2011205187 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fee840574b0 con 0x7fee900fde90 2024-03-23T12:14:08.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:08 smithi063 ceph-mon[141124]: from='client.35278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:08.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:08 smithi063 ceph-mon[141124]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:08.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:08 smithi063 ceph-mon[141124]: pgmap v691: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:08.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:08 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4071947038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:14:08.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.583+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee900fd500 con 0x7fee7806c7d0 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.585+0000 7fee8cc00700 1 -- 172.21.15.63:0/2011205187 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fee900fd500 con 0x7fee7806c7d0 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:14:08.565 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:14:08.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:14:08.566 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:14:08.566 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:14:08.566 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:14:08.566 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:14:08.566 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:14:08.566 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:14:08.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee7806c7d0 msgr2=0x7fee7806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee7806c7d0 0x7fee7806ec80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fee900fd690 tx=0x7fee7c0090f0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 msgr2=0x7fee900fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fbd50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fee84009040 tx=0x7fee84009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 shutdown_connections 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fee7806c7d0 0x7fee7806ec80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee900fde90 0x7fee900fbd50 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 --2- 172.21.15.63:0/2011205187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee900fe830 0x7fee900fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.587+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 >> 172.21.15.63:0/2011205187 conn(0x7fee90072bc0 msgr2=0x7fee900ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.588+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 shutdown_connections 2024-03-23T12:14:08.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.588+0000 7fee97a00700 1 -- 172.21.15.63:0/2011205187 wait complete. 2024-03-23T12:14:08.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 -- 172.21.15.63:0/1557151714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 msgr2=0x7fc7141031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 --2- 172.21.15.63:0/1557151714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc7141031d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc704004940 tx=0x7fc704009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 -- 172.21.15.63:0/1557151714 shutdown_connections 2024-03-23T12:14:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 --2- 172.21.15.63:0/1557151714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7141037a0 0x7fc7140fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 --2- 172.21.15.63:0/1557151714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc7141031d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 -- 172.21.15.63:0/1557151714 >> 172.21.15.63:0/1557151714 conn(0x7fc7140f89b0 msgr2=0x7fc7140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 -- 172.21.15.63:0/1557151714 shutdown_connections 2024-03-23T12:14:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.724+0000 7fc71ae00700 1 -- 172.21.15.63:0/1557151714 wait complete. 2024-03-23T12:14:08.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.725+0000 7fc71ae00700 1 Processor -- start 2024-03-23T12:14:08.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.725+0000 7fc71ae00700 1 -- start start 2024-03-23T12:14:08.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc71ae00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc714113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:08.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc71ae00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7141037a0 0x7fc714113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:08.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc71ae00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141145d0 con 0x7fc714102e00 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc71ae00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71410d9c0 con 0x7fc7141037a0 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc713e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc714113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc713e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc714113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51114/0 (socket says 172.21.15.63:51114) 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc713e00700 1 -- 172.21.15.63:0/3093800028 learned_addr learned my addr 172.21.15.63:0/3093800028 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc713400700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7141037a0 0x7fc714113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc713e00700 1 -- 172.21.15.63:0/3093800028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7141037a0 msgr2=0x7fc714113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc713e00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7141037a0 0x7fc714113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.726+0000 7fc713e00700 1 -- 172.21.15.63:0/3093800028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704005040 con 0x7fc714102e00 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.727+0000 7fc713e00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc714113a40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc704009390 tx=0x7fc704009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.727+0000 7fc70be00700 1 -- 172.21.15.63:0/3093800028 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc70400e040 con 0x7fc714102e00 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.727+0000 7fc70be00700 1 -- 172.21.15.63:0/3093800028 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc704013420 con 0x7fc714102e00 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.727+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71410dc40 con 0x7fc714102e00 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.727+0000 7fc70be00700 1 -- 172.21.15.63:0/3093800028 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc70401c630 con 0x7fc714102e00 2024-03-23T12:14:08.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.727+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71410e200 con 0x7fc714102e00 2024-03-23T12:14:08.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.728+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7140fe550 con 0x7fc714102e00 2024-03-23T12:14:08.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.729+0000 7fc70be00700 1 -- 172.21.15.63:0/3093800028 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc7040139e0 con 0x7fc714102e00 2024-03-23T12:14:08.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.730+0000 7fc70be00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6f806c7d0 0x7fc6f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:08.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.730+0000 7fc70be00700 1 -- 172.21.15.63:0/3093800028 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc70408cd80 con 0x7fc714102e00 2024-03-23T12:14:08.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.730+0000 7fc713400700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6f806c7d0 0x7fc6f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:08.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.731+0000 7fc713400700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6f806c7d0 0x7fc6f806ec80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc71410eb80 tx=0x7fc70000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:08.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:08.734+0000 7fc70be00700 1 -- 172.21.15.63:0/3093800028 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc70405aa90 con 0x7fc714102e00 2024-03-23T12:14:09.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.032+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc71404bed0 con 0x7fc714102e00 2024-03-23T12:14:09.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.032+0000 7fc70be00700 1 -- 172.21.15.63:0/3093800028 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc70405a620 con 0x7fc714102e00 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:14:09.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:14:09.014 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.034+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6f806c7d0 msgr2=0x7fc6f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.034+0000 7fc71ae00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6f806c7d0 0x7fc6f806ec80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc71410eb80 tx=0x7fc70000b040 comp rx=0 tx=0).stop 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.034+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 msgr2=0x7fc714113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.034+0000 7fc71ae00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc714113a40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc704009390 tx=0x7fc704009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.035+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 shutdown_connections 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.035+0000 7fc71ae00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc6f806c7d0 0x7fc6f806ec80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.035+0000 7fc71ae00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc714102e00 0x7fc714113a40 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.035+0000 7fc71ae00700 1 --2- 172.21.15.63:0/3093800028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7141037a0 0x7fc714113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.035+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 >> 172.21.15.63:0/3093800028 conn(0x7fc7140f89b0 msgr2=0x7fc7140f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:09.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.035+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 shutdown_connections 2024-03-23T12:14:09.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:09.035+0000 7fc71ae00700 1 -- 172.21.15.63:0/3093800028 wait complete. 2024-03-23T12:14:09.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:09 smithi063 ceph-mon[141124]: from='client.35286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:09.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:09 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3146440821' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:14:09.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:09 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3093800028' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:14:09.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:09 smithi111 ceph-mon[106899]: from='client.35286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:09.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:09 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3146440821' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:14:09.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:09 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3093800028' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:14:10.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:10 smithi063 ceph-mon[141124]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:10.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:10 smithi063 ceph-mon[141124]: pgmap v692: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:10.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:10 smithi111 ceph-mon[106899]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:10.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:10 smithi111 ceph-mon[106899]: pgmap v692: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:11.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:11.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:11 smithi063 ceph-mon[141124]: pgmap v693: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:11.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:11.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:11 smithi111 ceph-mon[106899]: pgmap v693: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:14.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:14 smithi063 ceph-mon[141124]: pgmap v694: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:14.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:14 smithi111 ceph-mon[106899]: pgmap v694: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:16.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:16 smithi063 ceph-mon[141124]: pgmap v695: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:16.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:16 smithi111 ceph-mon[106899]: pgmap v695: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:18.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:18 smithi063 ceph-mon[141124]: pgmap v696: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:18.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:18 smithi111 ceph-mon[106899]: pgmap v696: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:19.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:19 smithi063 ceph-mon[141124]: pgmap v697: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:19 smithi111 ceph-mon[106899]: pgmap v697: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:21.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:21 smithi063 ceph-mon[141124]: pgmap v698: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:21.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:21 smithi111 ceph-mon[106899]: pgmap v698: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:24.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:24 smithi063 ceph-mon[141124]: pgmap v699: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:24.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:24 smithi111 ceph-mon[106899]: pgmap v699: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:26.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:26 smithi063 ceph-mon[141124]: pgmap v700: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:26.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:26.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:26 smithi111 ceph-mon[106899]: pgmap v700: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:26.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:28.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:28 smithi063 ceph-mon[141124]: pgmap v701: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:28.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:28 smithi111 ceph-mon[106899]: pgmap v701: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:29.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:29 smithi063 ceph-mon[141124]: pgmap v702: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:29.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:29 smithi111 ceph-mon[106899]: pgmap v702: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:31.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:31 smithi063 ceph-mon[141124]: pgmap v703: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:31.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:31 smithi111 ceph-mon[106899]: pgmap v703: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:34.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:34 smithi063 ceph-mon[141124]: pgmap v704: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:34 smithi111 ceph-mon[106899]: pgmap v704: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:36.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:36 smithi063 ceph-mon[141124]: pgmap v705: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:36.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:36 smithi111 ceph-mon[106899]: pgmap v705: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:38.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:38 smithi063 ceph-mon[141124]: pgmap v706: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:38.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:38 smithi111 ceph-mon[106899]: pgmap v706: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 -- 172.21.15.63:0/1947499058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 msgr2=0x7f09c00fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 --2- 172.21.15.63:0/1947499058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c00fce30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f09ac004940 tx=0x7f09ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 -- 172.21.15.63:0/1947499058 shutdown_connections 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 --2- 172.21.15.63:0/1947499058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c00fce30 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 --2- 172.21.15.63:0/1947499058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09c0102e20 0x7f09c01031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 -- 172.21.15.63:0/1947499058 >> 172.21.15.63:0/1947499058 conn(0x7f09c00f89d0 msgr2=0x7f09c00fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 -- 172.21.15.63:0/1947499058 shutdown_connections 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.191+0000 7f09c6400700 1 -- 172.21.15.63:0/1947499058 wait complete. 2024-03-23T12:14:39.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.192+0000 7f09c6400700 1 Processor -- start 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.192+0000 7f09c6400700 1 -- start start 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.192+0000 7f09c6400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09c0102e20 0x7f09c0115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.192+0000 7f09c6400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c0116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.192+0000 7f09c6400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c01167e0 con 0x7f09c01037c0 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.192+0000 7f09c6400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c010fbb0 con 0x7f09c0102e20 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c0116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c0116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41946/0 (socket says 172.21.15.63:41946) 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bea00700 1 -- 172.21.15.63:0/1467596893 learned_addr learned my addr 172.21.15.63:0/1467596893 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bf400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09c0102e20 0x7f09c0115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bea00700 1 -- 172.21.15.63:0/1467596893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09c0102e20 msgr2=0x7f09c0115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bea00700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09c0102e20 0x7f09c0115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bea00700 1 -- 172.21.15.63:0/1467596893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac005040 con 0x7f09c01037c0 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bf400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09c0102e20 0x7f09c0115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.193+0000 7f09bea00700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c0116190 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f09ac008d50 tx=0x7f09ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.194+0000 7f09b7400700 1 -- 172.21.15.63:0/1467596893 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09ac00e030 con 0x7f09c01037c0 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.194+0000 7f09b7400700 1 -- 172.21.15.63:0/1467596893 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09ac013de0 con 0x7f09c01037c0 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.194+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c010fe30 con 0x7f09c01037c0 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.194+0000 7f09b7400700 1 -- 172.21.15.63:0/1467596893 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09ac01cd90 con 0x7f09c01037c0 2024-03-23T12:14:39.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.194+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c01103f0 con 0x7f09c01037c0 2024-03-23T12:14:39.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.195+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c00fe570 con 0x7f09c01037c0 2024-03-23T12:14:39.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.196+0000 7f09b7400700 1 -- 172.21.15.63:0/1467596893 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f09ac013930 con 0x7f09c01037c0 2024-03-23T12:14:39.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.197+0000 7f09b7400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09a406c7d0 0x7f09a406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.197+0000 7f09b7400700 1 -- 172.21.15.63:0/1467596893 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f09ac08d300 con 0x7f09c01037c0 2024-03-23T12:14:39.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.198+0000 7f09bf400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09a406c7d0 0x7f09a406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.198+0000 7f09bf400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09a406c7d0 0x7f09a406ec80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f09b0004600 tx=0x7f09b000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:39.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.202+0000 7f09b7400700 1 -- 172.21.15.63:0/1467596893 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f09ac05b010 con 0x7f09c01037c0 2024-03-23T12:14:39.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:39 smithi063 ceph-mon[141124]: pgmap v707: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:39.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:39 smithi111 ceph-mon[106899]: pgmap v707: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.436+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09c0109e30 con 0x7f09a406c7d0 2024-03-23T12:14:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.437+0000 7f09b7400700 1 -- 172.21.15.63:0/1467596893 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f09c0109e30 con 0x7f09a406c7d0 2024-03-23T12:14:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09a406c7d0 msgr2=0x7f09a406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09a406c7d0 0x7f09a406ec80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f09b0004600 tx=0x7f09b000b040 comp rx=0 tx=0).stop 2024-03-23T12:14:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 msgr2=0x7f09c0116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c0116190 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f09ac008d50 tx=0x7f09ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 shutdown_connections 2024-03-23T12:14:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f09a406c7d0 0x7f09a406ec80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09c0102e20 0x7f09c0115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 --2- 172.21.15.63:0/1467596893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09c01037c0 0x7f09c0116190 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 >> 172.21.15.63:0/1467596893 conn(0x7f09c00f89d0 msgr2=0x7f09c00f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 shutdown_connections 2024-03-23T12:14:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.440+0000 7f09c6400700 1 -- 172.21.15.63:0/1467596893 wait complete. 2024-03-23T12:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:14:39.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 -- 172.21.15.63:0/14205194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 msgr2=0x7f68e40ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 --2- 172.21.15.63:0/14205194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e40ff010 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f68d0004940 tx=0x7f68d0009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:39.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 -- 172.21.15.63:0/14205194 shutdown_connections 2024-03-23T12:14:39.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 --2- 172.21.15.63:0/14205194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e40ff010 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 --2- 172.21.15.63:0/14205194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e4105000 0x7f68e41053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 -- 172.21.15.63:0/14205194 >> 172.21.15.63:0/14205194 conn(0x7f68e40fabd0 msgr2=0x7f68e40fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:39.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 -- 172.21.15.63:0/14205194 shutdown_connections 2024-03-23T12:14:39.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.580+0000 7f68eb800700 1 -- 172.21.15.63:0/14205194 wait complete. 2024-03-23T12:14:39.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.581+0000 7f68eb800700 1 Processor -- start 2024-03-23T12:14:39.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.581+0000 7f68eb800700 1 -- start start 2024-03-23T12:14:39.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.581+0000 7f68eb800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e4105000 0x7f68e4115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.581+0000 7f68eb800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e4116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.581+0000 7f68eb800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e4116880 con 0x7f68e41059a0 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.581+0000 7f68eb800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e410fc50 con 0x7f68e4105000 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.582+0000 7f68e3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e4116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.582+0000 7f68e3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e4116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41962/0 (socket says 172.21.15.63:41962) 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.582+0000 7f68e3e00700 1 -- 172.21.15.63:0/786717257 learned_addr learned my addr 172.21.15.63:0/786717257 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.582+0000 7f68e8e00700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e4105000 0x7f68e4115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.582+0000 7f68e3e00700 1 -- 172.21.15.63:0/786717257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e4105000 msgr2=0x7f68e4115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.582+0000 7f68e3e00700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e4105000 0x7f68e4115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.582+0000 7f68e3e00700 1 -- 172.21.15.63:0/786717257 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d0005040 con 0x7f68e41059a0 2024-03-23T12:14:39.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.583+0000 7f68e3e00700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e4116230 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f68d00093b0 tx=0x7f68d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:39.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.583+0000 7f68e0c00700 1 -- 172.21.15.63:0/786717257 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d000e040 con 0x7f68e41059a0 2024-03-23T12:14:39.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.583+0000 7f68e0c00700 1 -- 172.21.15.63:0/786717257 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68d0013420 con 0x7f68e41059a0 2024-03-23T12:14:39.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.583+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e410fed0 con 0x7f68e41059a0 2024-03-23T12:14:39.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.583+0000 7f68e0c00700 1 -- 172.21.15.63:0/786717257 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d001c630 con 0x7f68e41059a0 2024-03-23T12:14:39.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.583+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e4110410 con 0x7f68e41059a0 2024-03-23T12:14:39.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.584+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e41074d0 con 0x7f68e41059a0 2024-03-23T12:14:39.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.585+0000 7f68e0c00700 1 -- 172.21.15.63:0/786717257 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f68d00139e0 con 0x7f68e41059a0 2024-03-23T12:14:39.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.586+0000 7f68e0c00700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68cc06c7c0 0x7f68cc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.586+0000 7f68e0c00700 1 -- 172.21.15.63:0/786717257 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f68d008cb40 con 0x7f68e41059a0 2024-03-23T12:14:39.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.586+0000 7f68e8e00700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68cc06c7c0 0x7f68cc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.587+0000 7f68e8e00700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68cc06c7c0 0x7f68cc06ec70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f68d8004f60 tx=0x7f68d800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:39.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.590+0000 7f68e0c00700 1 -- 172.21.15.63:0/786717257 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f68d005a850 con 0x7f68e41059a0 2024-03-23T12:14:39.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.824+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68e41113b0 con 0x7f68cc06c7c0 2024-03-23T12:14:39.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.825+0000 7f68e0c00700 1 -- 172.21.15.63:0/786717257 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f68e41113b0 con 0x7f68cc06c7c0 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68cc06c7c0 msgr2=0x7f68cc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68cc06c7c0 0x7f68cc06ec70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f68d8004f60 tx=0x7f68d800b040 comp rx=0 tx=0).stop 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 msgr2=0x7f68e4116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e4116230 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f68d00093b0 tx=0x7f68d0009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 shutdown_connections 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f68cc06c7c0 0x7f68cc06ec70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e4105000 0x7f68e4115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 --2- 172.21.15.63:0/786717257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68e41059a0 0x7f68e4116230 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 >> 172.21.15.63:0/786717257 conn(0x7f68e40fabd0 msgr2=0x7f68e40fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 shutdown_connections 2024-03-23T12:14:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.828+0000 7f68eb800700 1 -- 172.21.15.63:0/786717257 wait complete. 2024-03-23T12:14:39.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 -- 172.21.15.63:0/3546527348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 msgr2=0x7f8314070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 --2- 172.21.15.63:0/3546527348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f8314070360 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8304004970 tx=0x7f8304009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:39.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 -- 172.21.15.63:0/3546527348 shutdown_connections 2024-03-23T12:14:39.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 --2- 172.21.15.63:0/3546527348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83140708a0 0x7f8314070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 --2- 172.21.15.63:0/3546527348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f8314070360 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 -- 172.21.15.63:0/3546527348 >> 172.21.15.63:0/3546527348 conn(0x7f83140f89b0 msgr2=0x7f83140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:39.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 -- 172.21.15.63:0/3546527348 shutdown_connections 2024-03-23T12:14:39.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.966+0000 7f8319800700 1 -- 172.21.15.63:0/3546527348 wait complete. 2024-03-23T12:14:39.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.967+0000 7f8319800700 1 Processor -- start 2024-03-23T12:14:39.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.967+0000 7f8319800700 1 -- start start 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.967+0000 7f8319800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83140708a0 0x7f831411a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.967+0000 7f8319800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f831411a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.967+0000 7f8319800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831411ac50 con 0x7f8314071f90 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.967+0000 7f8319800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8314114020 con 0x7f83140708a0 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f831411a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f831411a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41992/0 (socket says 172.21.15.63:41992) 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312000700 1 -- 172.21.15.63:0/2560274596 learned_addr learned my addr 172.21.15.63:0/2560274596 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312a00700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83140708a0 0x7f831411a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312000700 1 -- 172.21.15.63:0/2560274596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83140708a0 msgr2=0x7f831411a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312000700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83140708a0 0x7f831411a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312000700 1 -- 172.21.15.63:0/2560274596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8304005040 con 0x7f8314071f90 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.968+0000 7f8312000700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f831411a600 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8300002a80 tx=0x7f83000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.969+0000 7f830aa00700 1 -- 172.21.15.63:0/2560274596 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8300007cc0 con 0x7f8314071f90 2024-03-23T12:14:39.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.969+0000 7f830aa00700 1 -- 172.21.15.63:0/2560274596 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f830000f040 con 0x7f8314071f90 2024-03-23T12:14:39.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.969+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8314114300 con 0x7f8314071f90 2024-03-23T12:14:39.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.969+0000 7f830aa00700 1 -- 172.21.15.63:0/2560274596 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f830000da60 con 0x7f8314071f90 2024-03-23T12:14:39.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.969+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8314114840 con 0x7f8314071f90 2024-03-23T12:14:39.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.970+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8314071aa0 con 0x7f8314071f90 2024-03-23T12:14:39.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.971+0000 7f830aa00700 1 -- 172.21.15.63:0/2560274596 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f830000dbc0 con 0x7f8314071f90 2024-03-23T12:14:39.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.972+0000 7f830aa00700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f82f806c7d0 0x7f82f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:39.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.972+0000 7f830aa00700 1 -- 172.21.15.63:0/2560274596 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f830008b6e0 con 0x7f8314071f90 2024-03-23T12:14:39.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.972+0000 7f8312a00700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f82f806c7d0 0x7f82f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:39.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.973+0000 7f8312a00700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f82f806c7d0 0x7f82f806ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f830400b370 tx=0x7f8304009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:39.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:39.976+0000 7f830aa00700 1 -- 172.21.15.63:0/2560274596 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8300059370 con 0x7f8314071f90 2024-03-23T12:14:40.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.201+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8314110450 con 0x7f82f806c7d0 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.211+0000 7f830aa00700 1 -- 172.21.15.63:0/2560274596 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f8314110450 con 0x7f82f806c7d0 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (23m) 64s ago 37m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (21m) 64s ago 37m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (21m) 67s ago 36m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (23m) 64s ago 36m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (17m) 64s ago 30m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:14:40.191 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (16m) 64s ago 30m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (71s) 67s ago 30m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (16m) 67s ago 30m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (26m) 64s ago 38m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (25m) 67s ago 35m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (22m) 64s ago 38m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (21m) 67s ago 35m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (24m) 64s ago 37m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (24m) 67s ago 36m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (20m) 64s ago 35m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (19m) 64s ago 34m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (19m) 64s ago 33m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (18m) 67s ago 33m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:14:40.192 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (18m) 67s ago 32m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:14:40.193 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (17m) 67s ago 32m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:14:40.193 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (24m) 64s ago 36m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:14:40.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f82f806c7d0 msgr2=0x7f82f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f82f806c7d0 0x7f82f806ec80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f830400b370 tx=0x7f8304009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 msgr2=0x7f831411a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f831411a600 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8300002a80 tx=0x7f83000046b0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 shutdown_connections 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f82f806c7d0 0x7f82f806ec80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83140708a0 0x7f831411a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 --2- 172.21.15.63:0/2560274596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8314071f90 0x7f831411a600 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.213+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 >> 172.21.15.63:0/2560274596 conn(0x7f83140f89b0 msgr2=0x7f83140f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.214+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 shutdown_connections 2024-03-23T12:14:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.214+0000 7f8319800700 1 -- 172.21.15.63:0/2560274596 wait complete. 2024-03-23T12:14:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 -- 172.21.15.63:0/3097646935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070c70 msgr2=0x7fa8f8109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/3097646935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070c70 0x7fa8f8109110 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4004970 tx=0x7fa8e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 -- 172.21.15.63:0/3097646935 shutdown_connections 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/3097646935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070c70 0x7fa8f8109110 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/3097646935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8f8070360 0x7fa8f8070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 -- 172.21.15.63:0/3097646935 >> 172.21.15.63:0/3097646935 conn(0x7fa8f80f89b0 msgr2=0x7fa8f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 -- 172.21.15.63:0/3097646935 shutdown_connections 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.349+0000 7fa8fdc00700 1 -- 172.21.15.63:0/3097646935 wait complete. 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8fdc00700 1 Processor -- start 2024-03-23T12:14:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8fdc00700 1 -- start start 2024-03-23T12:14:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8fdc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070360 0x7fa8f8115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8fdc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8f8070c70 0x7fa8f81161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8fdc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f8116830 con 0x7fa8f8070360 2024-03-23T12:14:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8fdc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f810fc00 con 0x7fa8f8070c70 2024-03-23T12:14:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8f6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070360 0x7fa8f8115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8f6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070360 0x7fa8f8115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42010/0 (socket says 172.21.15.63:42010) 2024-03-23T12:14:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.350+0000 7fa8f6a00700 1 -- 172.21.15.63:0/1807364972 learned_addr learned my addr 172.21.15.63:0/1807364972 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8f6000700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8f8070c70 0x7fa8f81161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8f6a00700 1 -- 172.21.15.63:0/1807364972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8f8070c70 msgr2=0x7fa8f81161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8f6a00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8f8070c70 0x7fa8f81161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8f6a00700 1 -- 172.21.15.63:0/1807364972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4005040 con 0x7fa8f8070360 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8f6a00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070360 0x7fa8f8115ca0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800eb40 tx=0x7fa8e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8eea00700 1 -- 172.21.15.63:0/1807364972 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8e8007cc0 con 0x7fa8f8070360 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8eea00700 1 -- 172.21.15.63:0/1807364972 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8e800f040 con 0x7fa8f8070360 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f810fe80 con 0x7fa8f8070360 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.351+0000 7fa8eea00700 1 -- 172.21.15.63:0/1807364972 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8e8009570 con 0x7fa8f8070360 2024-03-23T12:14:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.352+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f81103c0 con 0x7fa8f8070360 2024-03-23T12:14:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.352+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f8103bd0 con 0x7fa8f8070360 2024-03-23T12:14:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.354+0000 7fa8eea00700 1 -- 172.21.15.63:0/1807364972 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa8e80096d0 con 0x7fa8f8070360 2024-03-23T12:14:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.355+0000 7fa8eea00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa8dc06c7d0 0x7fa8dc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.355+0000 7fa8eea00700 1 -- 172.21.15.63:0/1807364972 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa8e808b6f0 con 0x7fa8f8070360 2024-03-23T12:14:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.355+0000 7fa8f6000700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa8dc06c7d0 0x7fa8dc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.356+0000 7fa8f6000700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa8dc06c7d0 0x7fa8dc06ec80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8111540 tx=0x7fa8e4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:40.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.360+0000 7fa8eea00700 1 -- 172.21.15.63:0/1807364972 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa8e8059430 con 0x7fa8f8070360 2024-03-23T12:14:40.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:40 smithi063 ceph-mon[141124]: from='client.35306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:40.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:40 smithi063 ceph-mon[141124]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:40.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:40 smithi111 ceph-mon[106899]: from='client.35306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:40.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:40 smithi111 ceph-mon[106899]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:40.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.658+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa8f810bfe0 con 0x7fa8f8070360 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.659+0000 7fa8eea00700 1 -- 172.21.15.63:0/1807364972 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fa8e8058fc0 con 0x7fa8f8070360 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:14:40.640 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:14:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa8dc06c7d0 msgr2=0x7fa8dc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa8dc06c7d0 0x7fa8dc06ec80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8111540 tx=0x7fa8e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070360 msgr2=0x7fa8f8115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070360 0x7fa8f8115ca0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800eb40 tx=0x7fa8e800b040 comp rx=0 tx=0).stop 2024-03-23T12:14:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 shutdown_connections 2024-03-23T12:14:40.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa8dc06c7d0 0x7fa8dc06ec80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8f8070360 0x7fa8f8115ca0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 --2- 172.21.15.63:0/1807364972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8f8070c70 0x7fa8f81161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 >> 172.21.15.63:0/1807364972 conn(0x7fa8f80f89b0 msgr2=0x7fa8f81015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:40.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 shutdown_connections 2024-03-23T12:14:40.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.661+0000 7fa8fdc00700 1 -- 172.21.15.63:0/1807364972 wait complete. 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 -- 172.21.15.63:0/2308499829 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14441059b0 msgr2=0x7f14440ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 --2- 172.21.15.63:0/2308499829 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14441059b0 0x7f14440ff020 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1430004940 tx=0x7f1430009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 -- 172.21.15.63:0/2308499829 shutdown_connections 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 --2- 172.21.15.63:0/2308499829 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14441059b0 0x7f14440ff020 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 --2- 172.21.15.63:0/2308499829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1444105010 0x7f14441053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 -- 172.21.15.63:0/2308499829 >> 172.21.15.63:0/2308499829 conn(0x7f14440faba0 msgr2=0x7f14440fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 -- 172.21.15.63:0/2308499829 shutdown_connections 2024-03-23T12:14:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.799+0000 7f1448e00700 1 -- 172.21.15.63:0/2308499829 wait complete. 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.800+0000 7f1448e00700 1 Processor -- start 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.800+0000 7f1448e00700 1 -- start start 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1448e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1444105010 0x7f1444115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1448e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14441059b0 0x7f14441161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1448e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1444116840 con 0x7f1444105010 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1448e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144410fc60 con 0x7f14441059b0 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1444105010 0x7f1444115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1444105010 0x7f1444115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42034/0 (socket says 172.21.15.63:42034) 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441e00700 1 -- 172.21.15.63:0/1218725888 learned_addr learned my addr 172.21.15.63:0/1218725888 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441400700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14441059b0 0x7f14441161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441e00700 1 -- 172.21.15.63:0/1218725888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14441059b0 msgr2=0x7f14441161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441e00700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14441059b0 0x7f14441161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441e00700 1 -- 172.21.15.63:0/1218725888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1430005040 con 0x7f1444105010 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.801+0000 7f1441400700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14441059b0 0x7f14441161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.802+0000 7f1441e00700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1444105010 0x7f1444115cb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f143400eb40 tx=0x7f143400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.802+0000 7f143a000700 1 -- 172.21.15.63:0/1218725888 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1434007cc0 con 0x7f1444105010 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.802+0000 7f143a000700 1 -- 172.21.15.63:0/1218725888 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f143400f040 con 0x7f1444105010 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.802+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144410fee0 con 0x7f1444105010 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.802+0000 7f143a000700 1 -- 172.21.15.63:0/1218725888 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1434009570 con 0x7f1444105010 2024-03-23T12:14:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.802+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14441104a0 con 0x7f1444105010 2024-03-23T12:14:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.803+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1444102320 con 0x7f1444105010 2024-03-23T12:14:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.804+0000 7f143a000700 1 -- 172.21.15.63:0/1218725888 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f14340096d0 con 0x7f1444105010 2024-03-23T12:14:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.805+0000 7f143a000700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f142806c7d0 0x7f142806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.805+0000 7f143a000700 1 -- 172.21.15.63:0/1218725888 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f143408b670 con 0x7f1444105010 2024-03-23T12:14:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.805+0000 7f1441400700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f142806c7d0 0x7f142806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.806+0000 7f1441400700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f142806c7d0 0x7f142806ec80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f143000b0d0 tx=0x7f1430009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:40.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:40.809+0000 7f143a000700 1 -- 172.21.15.63:0/1218725888 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1434059430 con 0x7f1444105010 2024-03-23T12:14:41.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.067+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f144404bed0 con 0x7f1444105010 2024-03-23T12:14:41.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.067+0000 7f143a000700 1 -- 172.21.15.63:0/1218725888 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f1434058fc0 con 0x7f1444105010 2024-03-23T12:14:41.047 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:14:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:14:41.049 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:41.050 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:14:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.069+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f142806c7d0 msgr2=0x7f142806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.069+0000 7f1448e00700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f142806c7d0 0x7f142806ec80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f143000b0d0 tx=0x7f1430009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.069+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1444105010 msgr2=0x7f1444115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.069+0000 7f1448e00700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1444105010 0x7f1444115cb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f143400eb40 tx=0x7f143400b040 comp rx=0 tx=0).stop 2024-03-23T12:14:41.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.070+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 shutdown_connections 2024-03-23T12:14:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.070+0000 7f1448e00700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f142806c7d0 0x7f142806ec80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.070+0000 7f1448e00700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1444105010 0x7f1444115cb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.070+0000 7f1448e00700 1 --2- 172.21.15.63:0/1218725888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14441059b0 0x7f14441161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.070+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 >> 172.21.15.63:0/1218725888 conn(0x7f14440faba0 msgr2=0x7f1444109600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.070+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 shutdown_connections 2024-03-23T12:14:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.070+0000 7f1448e00700 1 -- 172.21.15.63:0/1218725888 wait complete. 2024-03-23T12:14:41.052 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.208+0000 7fba68c00700 1 -- 172.21.15.63:0/3401687595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070c10 msgr2=0x7fba64108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.208+0000 7fba68c00700 1 --2- 172.21.15.63:0/3401687595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070c10 0x7fba64108fa0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fba50004970 tx=0x7fba50009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.208+0000 7fba68c00700 1 -- 172.21.15.63:0/3401687595 shutdown_connections 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.208+0000 7fba68c00700 1 --2- 172.21.15.63:0/3401687595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070c10 0x7fba64108fa0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.208+0000 7fba68c00700 1 --2- 172.21.15.63:0/3401687595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba64070300 0x7fba640706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.208+0000 7fba68c00700 1 -- 172.21.15.63:0/3401687595 >> 172.21.15.63:0/3401687595 conn(0x7fba640f87a0 msgr2=0x7fba640fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.209+0000 7fba68c00700 1 -- 172.21.15.63:0/3401687595 shutdown_connections 2024-03-23T12:14:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.209+0000 7fba68c00700 1 -- 172.21.15.63:0/3401687595 wait complete. 2024-03-23T12:14:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.209+0000 7fba68c00700 1 Processor -- start 2024-03-23T12:14:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.209+0000 7fba68c00700 1 -- start start 2024-03-23T12:14:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba68c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070300 0x7fba64115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba68c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba64070c10 0x7fba64115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba68c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba64116610 con 0x7fba64070300 2024-03-23T12:14:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba68c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6410fa00 con 0x7fba64070c10 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba63400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070300 0x7fba64115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba63400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070300 0x7fba64115a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42052/0 (socket says 172.21.15.63:42052) 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba63400700 1 -- 172.21.15.63:0/1392317143 learned_addr learned my addr 172.21.15.63:0/1392317143 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.210+0000 7fba62a00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba64070c10 0x7fba64115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba63400700 1 -- 172.21.15.63:0/1392317143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba64070c10 msgr2=0x7fba64115fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba63400700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba64070c10 0x7fba64115fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba63400700 1 -- 172.21.15.63:0/1392317143 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50005040 con 0x7fba64070300 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba63400700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070300 0x7fba64115a80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fba54002a50 tx=0x7fba540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba5b400700 1 -- 172.21.15.63:0/1392317143 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba54007cc0 con 0x7fba64070300 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba5b400700 1 -- 172.21.15.63:0/1392317143 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba5400f040 con 0x7fba64070300 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6410fce0 con 0x7fba64070300 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba5b400700 1 -- 172.21.15.63:0/1392317143 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba5400d9c0 con 0x7fba64070300 2024-03-23T12:14:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.211+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba641101f0 con 0x7fba64070300 2024-03-23T12:14:41.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.212+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64071a40 con 0x7fba64070300 2024-03-23T12:14:41.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.213+0000 7fba5b400700 1 -- 172.21.15.63:0/1392317143 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fba5400db20 con 0x7fba64070300 2024-03-23T12:14:41.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.214+0000 7fba5b400700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fba4806c7d0 0x7fba4806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:41.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.214+0000 7fba5b400700 1 -- 172.21.15.63:0/1392317143 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fba5408b6e0 con 0x7fba64070300 2024-03-23T12:14:41.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.214+0000 7fba62a00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fba4806c7d0 0x7fba4806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:41.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.215+0000 7fba62a00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fba4806c7d0 0x7fba4806ec80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fba5000b370 tx=0x7fba50009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:41.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.218+0000 7fba5b400700 1 -- 172.21.15.63:0/1392317143 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fba540593f0 con 0x7fba64070300 2024-03-23T12:14:41.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.453+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba6410be60 con 0x7fba4806c7d0 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.455+0000 7fba5b400700 1 -- 172.21.15.63:0/1392317143 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fba6410be60 con 0x7fba4806c7d0 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:14:41.435 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:14:41.436 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:14:41.436 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:14:41.436 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:14:41.436 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:14:41.436 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:14:41.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fba4806c7d0 msgr2=0x7fba4806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fba4806c7d0 0x7fba4806ec80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fba5000b370 tx=0x7fba50009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070300 msgr2=0x7fba64115a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070300 0x7fba64115a80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fba54002a50 tx=0x7fba540046b0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 shutdown_connections 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fba4806c7d0 0x7fba4806ec80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba64070300 0x7fba64115a80 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 --2- 172.21.15.63:0/1392317143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba64070c10 0x7fba64115fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.457+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 >> 172.21.15.63:0/1392317143 conn(0x7fba640f87a0 msgr2=0x7fba640f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.458+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 shutdown_connections 2024-03-23T12:14:41.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.458+0000 7fba68c00700 1 -- 172.21.15.63:0/1392317143 wait complete. 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 -- 172.21.15.63:0/419635720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070360 msgr2=0x7f9b34070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 --2- 172.21.15.63:0/419635720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070360 0x7f9b34070730 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9b24004970 tx=0x7f9b24009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 -- 172.21.15.63:0/419635720 shutdown_connections 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 --2- 172.21.15.63:0/419635720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b34070c70 0x7f9b34108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 --2- 172.21.15.63:0/419635720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070360 0x7f9b34070730 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 -- 172.21.15.63:0/419635720 >> 172.21.15.63:0/419635720 conn(0x7f9b340f8760 msgr2=0x7f9b340fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 -- 172.21.15.63:0/419635720 shutdown_connections 2024-03-23T12:14:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.595+0000 7f9b38a00700 1 -- 172.21.15.63:0/419635720 wait complete. 2024-03-23T12:14:41.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.596+0000 7f9b38a00700 1 Processor -- start 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.596+0000 7f9b38a00700 1 -- start start 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b38a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b34070360 0x7f9b34115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b38a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070c70 0x7f9b34115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b38a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b34116620 con 0x7f9b34070c70 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b38a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3410fa10 con 0x7f9b34070360 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b31e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070c70 0x7f9b34115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b31e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070c70 0x7f9b34115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48006/0 (socket says 172.21.15.63:48006) 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b31e00700 1 -- 172.21.15.63:0/4088667362 learned_addr learned my addr 172.21.15.63:0/4088667362 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b32800700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b34070360 0x7f9b34115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b31e00700 1 -- 172.21.15.63:0/4088667362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b34070360 msgr2=0x7f9b34115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b31e00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b34070360 0x7f9b34115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.597+0000 7f9b31e00700 1 -- 172.21.15.63:0/4088667362 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b24005040 con 0x7f9b34070c70 2024-03-23T12:14:41.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.598+0000 7f9b31e00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070c70 0x7f9b34115fd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9b20002a50 tx=0x7f9b200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:41.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.598+0000 7f9b2aa00700 1 -- 172.21.15.63:0/4088667362 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b20007cc0 con 0x7f9b34070c70 2024-03-23T12:14:41.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.598+0000 7f9b2aa00700 1 -- 172.21.15.63:0/4088667362 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b2000f040 con 0x7f9b34070c70 2024-03-23T12:14:41.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.598+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3410fcf0 con 0x7f9b34070c70 2024-03-23T12:14:41.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.598+0000 7f9b2aa00700 1 -- 172.21.15.63:0/4088667362 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b2000d9c0 con 0x7f9b34070c70 2024-03-23T12:14:41.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.598+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b34110230 con 0x7f9b34070c70 2024-03-23T12:14:41.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.599+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b34071aa0 con 0x7f9b34070c70 2024-03-23T12:14:41.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.600+0000 7f9b2aa00700 1 -- 172.21.15.63:0/4088667362 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9b2000db20 con 0x7f9b34070c70 2024-03-23T12:14:41.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.601+0000 7f9b2aa00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b1806c7c0 0x7f9b1806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:14:41.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.601+0000 7f9b2aa00700 1 -- 172.21.15.63:0/4088667362 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9b2008b760 con 0x7f9b34070c70 2024-03-23T12:14:41.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.601+0000 7f9b32800700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b1806c7c0 0x7f9b1806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:14:41.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.602+0000 7f9b32800700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b1806c7c0 0x7f9b1806ec70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009fd0 tx=0x7f9b24009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:14:41.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.605+0000 7f9b2aa00700 1 -- 172.21.15.63:0/4088667362 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9b200593f0 con 0x7f9b34070c70 2024-03-23T12:14:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:41 smithi063 ceph-mon[141124]: from='client.35314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:41 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1807364972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:14:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:41 smithi063 ceph-mon[141124]: pgmap v708: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:41.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:41 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1218725888' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:14:41.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.907+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9b3410bde0 con 0x7f9b34070c70 2024-03-23T12:14:41.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.908+0000 7f9b2aa00700 1 -- 172.21.15.63:0/4088667362 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f9b20058f80 con 0x7f9b34070c70 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:14:41.888 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:14:41.889 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:14:41.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b1806c7c0 msgr2=0x7f9b1806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b1806c7c0 0x7f9b1806ec70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009fd0 tx=0x7f9b24009e30 comp rx=0 tx=0).stop 2024-03-23T12:14:41.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070c70 msgr2=0x7f9b34115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:14:41.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070c70 0x7f9b34115fd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9b20002a50 tx=0x7f9b200046b0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 shutdown_connections 2024-03-23T12:14:41.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b1806c7c0 0x7f9b1806ec70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b34070360 0x7f9b34115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 --2- 172.21.15.63:0/4088667362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b34070c70 0x7f9b34115fd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:14:41.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 >> 172.21.15.63:0/4088667362 conn(0x7f9b340f8760 msgr2=0x7f9b340f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:14:41.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 shutdown_connections 2024-03-23T12:14:41.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:14:41.910+0000 7f9b38a00700 1 -- 172.21.15.63:0/4088667362 wait complete. 2024-03-23T12:14:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:41 smithi111 ceph-mon[106899]: from='client.35314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:41 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1807364972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:14:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:41 smithi111 ceph-mon[106899]: pgmap v708: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:41 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1218725888' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:14:42.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:42 smithi063 ceph-mon[141124]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:42.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:42 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4088667362' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:14:42.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:42 smithi111 ceph-mon[106899]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:14:42.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:42 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4088667362' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:14:43.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:43 smithi063 ceph-mon[141124]: pgmap v709: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:43.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:43 smithi111 ceph-mon[106899]: pgmap v709: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:46.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:46 smithi063 ceph-mon[141124]: pgmap v710: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:46.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:46 smithi111 ceph-mon[106899]: pgmap v710: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:48.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:48 smithi063 ceph-mon[141124]: pgmap v711: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:48.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:48 smithi111 ceph-mon[106899]: pgmap v711: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:49.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:49 smithi063 ceph-mon[141124]: pgmap v712: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:49.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:49 smithi111 ceph-mon[106899]: pgmap v712: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:51.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:51 smithi063 ceph-mon[141124]: pgmap v713: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:51.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:51 smithi111 ceph-mon[106899]: pgmap v713: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:54.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:54 smithi063 ceph-mon[141124]: pgmap v714: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:54.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:54 smithi111 ceph-mon[106899]: pgmap v714: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:56.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:56 smithi063 ceph-mon[141124]: pgmap v715: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:56.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:56.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:56 smithi111 ceph-mon[106899]: pgmap v715: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:56.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:14:57.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:14:57 smithi063 ceph-mon[141124]: pgmap v716: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:14:57.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:14:57 smithi111 ceph-mon[106899]: pgmap v716: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:00.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:00 smithi063 ceph-mon[141124]: pgmap v717: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:00 smithi111 ceph-mon[106899]: pgmap v717: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:01.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:01 smithi063 ceph-mon[141124]: pgmap v718: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:01.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:01 smithi111 ceph-mon[106899]: pgmap v718: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:04.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:04 smithi063 ceph-mon[141124]: pgmap v719: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:04.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:04 smithi111 ceph-mon[106899]: pgmap v719: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:05.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:05 smithi063 ceph-mon[141124]: pgmap v720: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:05.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:05 smithi111 ceph-mon[106899]: pgmap v720: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:08.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:08 smithi063 ceph-mon[141124]: pgmap v721: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:08.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:08 smithi111 ceph-mon[106899]: pgmap v721: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:09.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:09 smithi063 ceph-mon[141124]: pgmap v722: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:09.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:09 smithi111 ceph-mon[106899]: pgmap v722: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:11.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:11.879 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:11 smithi063 ceph-mon[141124]: pgmap v723: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:11.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:11.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:11 smithi111 ceph-mon[106899]: pgmap v723: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 -- 172.21.15.63:0/2023753732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 msgr2=0x7fc240070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 --2- 172.21.15.63:0/2023753732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc240070360 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc230004970 tx=0x7fc230009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 -- 172.21.15.63:0/2023753732 shutdown_connections 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 --2- 172.21.15.63:0/2023753732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2400708a0 0x7fc240070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 --2- 172.21.15.63:0/2023753732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc240070360 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 -- 172.21.15.63:0/2023753732 >> 172.21.15.63:0/2023753732 conn(0x7fc2400f89b0 msgr2=0x7fc2400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 -- 172.21.15.63:0/2023753732 shutdown_connections 2024-03-23T12:15:12.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.064+0000 7fc245e00700 1 -- 172.21.15.63:0/2023753732 wait complete. 2024-03-23T12:15:12.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.065+0000 7fc245e00700 1 Processor -- start 2024-03-23T12:15:12.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.065+0000 7fc245e00700 1 -- start start 2024-03-23T12:15:12.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc245e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2400708a0 0x7fc24011a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc245e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc24011a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc245e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24011ad50 con 0x7fc240071f90 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc245e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc240114090 con 0x7fc2400708a0 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc23e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc24011a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc23e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc24011a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38192/0 (socket says 172.21.15.63:38192) 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc23e000700 1 -- 172.21.15.63:0/809990233 learned_addr learned my addr 172.21.15.63:0/809990233 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.066+0000 7fc23ea00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2400708a0 0x7fc24011a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc23e000700 1 -- 172.21.15.63:0/809990233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2400708a0 msgr2=0x7fc24011a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc23e000700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2400708a0 0x7fc24011a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc23e000700 1 -- 172.21.15.63:0/809990233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc230005040 con 0x7fc240071f90 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc23e000700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc24011a670 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00b790 tx=0x7fc22c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc236a00700 1 -- 172.21.15.63:0/809990233 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22c00f6f0 con 0x7fc240071f90 2024-03-23T12:15:12.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc236a00700 1 -- 172.21.15.63:0/809990233 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc22c00fd00 con 0x7fc240071f90 2024-03-23T12:15:12.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc240114310 con 0x7fc240071f90 2024-03-23T12:15:12.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc236a00700 1 -- 172.21.15.63:0/809990233 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22c0103a0 con 0x7fc240071f90 2024-03-23T12:15:12.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.067+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc240114850 con 0x7fc240071f90 2024-03-23T12:15:12.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.068+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc240071aa0 con 0x7fc240071f90 2024-03-23T12:15:12.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.069+0000 7fc236a00700 1 -- 172.21.15.63:0/809990233 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc22c00f850 con 0x7fc240071f90 2024-03-23T12:15:12.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.070+0000 7fc236a00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc22406c7d0 0x7fc22406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.070+0000 7fc236a00700 1 -- 172.21.15.63:0/809990233 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc22c08b8a0 con 0x7fc240071f90 2024-03-23T12:15:12.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.071+0000 7fc23ea00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc22406c7d0 0x7fc22406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.071+0000 7fc23ea00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc22406c7d0 0x7fc22406ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc23000b370 tx=0x7fc230009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:12.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.075+0000 7fc236a00700 1 -- 172.21.15.63:0/809990233 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc22c0595b0 con 0x7fc240071f90 2024-03-23T12:15:12.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.308+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2401104d0 con 0x7fc22406c7d0 2024-03-23T12:15:12.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.309+0000 7fc236a00700 1 -- 172.21.15.63:0/809990233 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc2401104d0 con 0x7fc22406c7d0 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc22406c7d0 msgr2=0x7fc22406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc22406c7d0 0x7fc22406ec80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc23000b370 tx=0x7fc230009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 msgr2=0x7fc24011a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc24011a670 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00b790 tx=0x7fc22c009250 comp rx=0 tx=0).stop 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 shutdown_connections 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc22406c7d0 0x7fc22406ec80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2400708a0 0x7fc24011a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 --2- 172.21.15.63:0/809990233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc240071f90 0x7fc24011a670 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 >> 172.21.15.63:0/809990233 conn(0x7fc2400f89b0 msgr2=0x7fc2400f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:12.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 shutdown_connections 2024-03-23T12:15:12.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.312+0000 7fc245e00700 1 -- 172.21.15.63:0/809990233 wait complete. 2024-03-23T12:15:12.306 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:15:12.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.451+0000 7f6ba9c00700 1 -- 172.21.15.63:0/29995572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba4102e00 msgr2=0x7f6ba41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.451+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/29995572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba4102e00 0x7f6ba41031d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6b94004940 tx=0x7f6b94009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:12.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.452+0000 7f6ba9c00700 1 -- 172.21.15.63:0/29995572 shutdown_connections 2024-03-23T12:15:12.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.452+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/29995572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ba41037a0 0x7f6ba40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.452+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/29995572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba4102e00 0x7f6ba41031d0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.452+0000 7f6ba9c00700 1 -- 172.21.15.63:0/29995572 >> 172.21.15.63:0/29995572 conn(0x7f6ba40f89b0 msgr2=0x7f6ba40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:12.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.452+0000 7f6ba9c00700 1 -- 172.21.15.63:0/29995572 shutdown_connections 2024-03-23T12:15:12.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.452+0000 7f6ba9c00700 1 -- 172.21.15.63:0/29995572 wait complete. 2024-03-23T12:15:12.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba9c00700 1 Processor -- start 2024-03-23T12:15:12.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba9c00700 1 -- start start 2024-03-23T12:15:12.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba9c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ba4102e00 0x7f6ba41138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba9c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba41037a0 0x7f6ba4113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba9c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba4114440 con 0x7f6ba41037a0 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba9c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba410da40 con 0x7f6ba4102e00 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba41037a0 0x7f6ba4113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba41037a0 0x7f6ba4113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38216/0 (socket says 172.21.15.63:38216) 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.453+0000 7f6ba2000700 1 -- 172.21.15.63:0/669697900 learned_addr learned my addr 172.21.15.63:0/669697900 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba2a00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ba4102e00 0x7f6ba41138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba2000700 1 -- 172.21.15.63:0/669697900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ba4102e00 msgr2=0x7f6ba41138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba2000700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ba4102e00 0x7f6ba41138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba2000700 1 -- 172.21.15.63:0/669697900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94005040 con 0x7f6ba41037a0 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba2a00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ba4102e00 0x7f6ba41138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba2000700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba41037a0 0x7f6ba4113df0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000c890 tx=0x7f6b900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:12.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6b9aa00700 1 -- 172.21.15.63:0/669697900 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b9000a7d0 con 0x7f6ba41037a0 2024-03-23T12:15:12.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6b9aa00700 1 -- 172.21.15.63:0/669697900 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b9000ade0 con 0x7f6ba41037a0 2024-03-23T12:15:12.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6b9aa00700 1 -- 172.21.15.63:0/669697900 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b900113a0 con 0x7f6ba41037a0 2024-03-23T12:15:12.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba410dd20 con 0x7f6ba41037a0 2024-03-23T12:15:12.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.454+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba410e2e0 con 0x7f6ba41037a0 2024-03-23T12:15:12.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.455+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba40fe550 con 0x7f6ba41037a0 2024-03-23T12:15:12.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.456+0000 7f6b9aa00700 1 -- 172.21.15.63:0/669697900 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6b90012400 con 0x7f6ba41037a0 2024-03-23T12:15:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.458+0000 7f6b9aa00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6b8806c7c0 0x7f6b8806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.458+0000 7f6b9aa00700 1 -- 172.21.15.63:0/669697900 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6b9008bb10 con 0x7f6ba41037a0 2024-03-23T12:15:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.458+0000 7f6ba2a00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6b8806c7c0 0x7f6b8806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.459+0000 7f6ba2a00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6b8806c7c0 0x7f6b8806ec70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400b840 tx=0x7f6b94009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:12.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.462+0000 7f6b9aa00700 1 -- 172.21.15.63:0/669697900 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6b90059820 con 0x7f6ba41037a0 2024-03-23T12:15:12.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.694+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ba4109dc0 con 0x7f6b8806c7c0 2024-03-23T12:15:12.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.696+0000 7f6b9aa00700 1 -- 172.21.15.63:0/669697900 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6ba4109dc0 con 0x7f6b8806c7c0 2024-03-23T12:15:12.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6b8806c7c0 msgr2=0x7f6b8806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6b8806c7c0 0x7f6b8806ec70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400b840 tx=0x7f6b94009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:12.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba41037a0 msgr2=0x7f6ba4113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba41037a0 0x7f6ba4113df0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000c890 tx=0x7f6b900044c0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 shutdown_connections 2024-03-23T12:15:12.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6b8806c7c0 0x7f6b8806ec70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ba4102e00 0x7f6ba41138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 --2- 172.21.15.63:0/669697900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba41037a0 0x7f6ba4113df0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 >> 172.21.15.63:0/669697900 conn(0x7f6ba40f89b0 msgr2=0x7f6ba40f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:12.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 shutdown_connections 2024-03-23T12:15:12.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.698+0000 7f6ba9c00700 1 -- 172.21.15.63:0/669697900 wait complete. 2024-03-23T12:15:12.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.835+0000 7fbeb8600700 1 -- 172.21.15.63:0/1082130513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 msgr2=0x7fbeb00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.835+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1082130513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb00fce10 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c004940 tx=0x7fbe9c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:12.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.835+0000 7fbeb8600700 1 -- 172.21.15.63:0/1082130513 shutdown_connections 2024-03-23T12:15:12.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.835+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1082130513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb00fce10 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.835+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1082130513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0102e00 0x7fbeb01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.835+0000 7fbeb8600700 1 -- 172.21.15.63:0/1082130513 >> 172.21.15.63:0/1082130513 conn(0x7fbeb00f89b0 msgr2=0x7fbeb00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:12.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.835+0000 7fbeb8600700 1 -- 172.21.15.63:0/1082130513 shutdown_connections 2024-03-23T12:15:12.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.836+0000 7fbeb8600700 1 -- 172.21.15.63:0/1082130513 wait complete. 2024-03-23T12:15:12.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.837+0000 7fbeb8600700 1 Processor -- start 2024-03-23T12:15:12.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.837+0000 7fbeb8600700 1 -- start start 2024-03-23T12:15:12.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.837+0000 7fbeb8600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0102e00 0x7fbeb0113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.837+0000 7fbeb8600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb0113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.837+0000 7fbeb8600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0114600 con 0x7fbeb01037a0 2024-03-23T12:15:12.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb8600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb010d9f0 con 0x7fbeb0102e00 2024-03-23T12:15:12.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb0113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb0113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38232/0 (socket says 172.21.15.63:38232) 2024-03-23T12:15:12.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5200700 1 -- 172.21.15.63:0/1381270809 learned_addr learned my addr 172.21.15.63:0/1381270809 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:12.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5c00700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0102e00 0x7fbeb0113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5200700 1 -- 172.21.15.63:0/1381270809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0102e00 msgr2=0x7fbeb0113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:12.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5200700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0102e00 0x7fbeb0113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:12.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5200700 1 -- 172.21.15.63:0/1381270809 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c005040 con 0x7fbeb01037a0 2024-03-23T12:15:12.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.838+0000 7fbeb5200700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb0113fb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009fd0 tx=0x7fbe9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:12.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.839+0000 7fbead600700 1 -- 172.21.15.63:0/1381270809 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe9c00e030 con 0x7fbeb01037a0 2024-03-23T12:15:12.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.839+0000 7fbead600700 1 -- 172.21.15.63:0/1381270809 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe9c0093b0 con 0x7fbeb01037a0 2024-03-23T12:15:12.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.839+0000 7fbead600700 1 -- 172.21.15.63:0/1381270809 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe9c01cc80 con 0x7fbeb01037a0 2024-03-23T12:15:12.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.839+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb010dc70 con 0x7fbeb01037a0 2024-03-23T12:15:12.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.839+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb010e230 con 0x7fbeb01037a0 2024-03-23T12:15:12.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.840+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb00fe550 con 0x7fbeb01037a0 2024-03-23T12:15:12.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.841+0000 7fbead600700 1 -- 172.21.15.63:0/1381270809 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbe9c01c400 con 0x7fbeb01037a0 2024-03-23T12:15:12.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.842+0000 7fbead600700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe9806c7d0 0x7fbe9806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:12.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.842+0000 7fbead600700 1 -- 172.21.15.63:0/1381270809 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbe9c08d140 con 0x7fbeb01037a0 2024-03-23T12:15:12.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.842+0000 7fbeb5c00700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe9806c7d0 0x7fbe9806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:12.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.843+0000 7fbeb5c00700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe9806c7d0 0x7fbe9806ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbea4009b80 tx=0x7fbea400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:12.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:12.847+0000 7fbead600700 1 -- 172.21.15.63:0/1381270809 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbe9c05af10 con 0x7fbeb01037a0 2024-03-23T12:15:13.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.070+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbeb0109dc0 con 0x7fbe9806c7d0 2024-03-23T12:15:13.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.080+0000 7fbead600700 1 -- 172.21.15.63:0/1381270809 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbeb0109dc0 con 0x7fbe9806c7d0 2024-03-23T12:15:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:15:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (24m) 97s ago 37m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:15:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (21m) 97s ago 37m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (21m) 100s ago 36m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (24m) 97s ago 37m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (17m) 97s ago 31m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (17m) 97s ago 31m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (104s) 100s ago 31m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (16m) 100s ago 31m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (26m) 97s ago 39m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (25m) 100s ago 36m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (22m) 97s ago 39m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (22m) 100s ago 36m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (25m) 97s ago 37m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (25m) 100s ago 36m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (21m) 97s ago 35m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (20m) 97s ago 35m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:15:13.060 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (19m) 97s ago 34m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:15:13.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (19m) 100s ago 33m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:15:13.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (18m) 100s ago 33m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:15:13.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (17m) 100s ago 32m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:15:13.061 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (24m) 97s ago 37m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe9806c7d0 msgr2=0x7fbe9806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe9806c7d0 0x7fbe9806ec80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbea4009b80 tx=0x7fbea400b040 comp rx=0 tx=0).stop 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 msgr2=0x7fbeb0113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb0113fb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009fd0 tx=0x7fbe9c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 shutdown_connections 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe9806c7d0 0x7fbe9806ec80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0102e00 0x7fbeb0113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 --2- 172.21.15.63:0/1381270809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb01037a0 0x7fbeb0113fb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 >> 172.21.15.63:0/1381270809 conn(0x7fbeb00f89b0 msgr2=0x7fbeb00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:13.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 shutdown_connections 2024-03-23T12:15:13.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.083+0000 7fbeb8600700 1 -- 172.21.15.63:0/1381270809 wait complete. 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 -- 172.21.15.63:0/1482676799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a141037a0 msgr2=0x7f4a140fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 --2- 172.21.15.63:0/1482676799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a141037a0 0x7f4a140fce10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4a00004940 tx=0x7f4a00009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 -- 172.21.15.63:0/1482676799 shutdown_connections 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 --2- 172.21.15.63:0/1482676799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a141037a0 0x7f4a140fce10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 --2- 172.21.15.63:0/1482676799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a14102e00 0x7f4a141031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 -- 172.21.15.63:0/1482676799 >> 172.21.15.63:0/1482676799 conn(0x7f4a140f89b0 msgr2=0x7f4a140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 -- 172.21.15.63:0/1482676799 shutdown_connections 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.218+0000 7f4a18e00700 1 -- 172.21.15.63:0/1482676799 wait complete. 2024-03-23T12:15:13.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.219+0000 7f4a18e00700 1 Processor -- start 2024-03-23T12:15:13.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.219+0000 7f4a18e00700 1 -- start start 2024-03-23T12:15:13.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a18e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a14102e00 0x7f4a14113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:13.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a18e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a141037a0 0x7f4a14114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:13.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a18e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14114690 con 0x7f4a14102e00 2024-03-23T12:15:13.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a18e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1410da80 con 0x7f4a141037a0 2024-03-23T12:15:13.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a14102e00 0x7f4a14113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a14102e00 0x7f4a14113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38246/0 (socket says 172.21.15.63:38246) 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11e00700 1 -- 172.21.15.63:0/4001711867 learned_addr learned my addr 172.21.15.63:0/4001711867 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11400700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a141037a0 0x7f4a14114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11e00700 1 -- 172.21.15.63:0/4001711867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a141037a0 msgr2=0x7f4a14114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11e00700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a141037a0 0x7f4a14114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11e00700 1 -- 172.21.15.63:0/4001711867 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a00005040 con 0x7f4a14102e00 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.220+0000 7f4a11e00700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a14102e00 0x7f4a14113b00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4a04002a50 tx=0x7f4a040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.221+0000 7f4a0a000700 1 -- 172.21.15.63:0/4001711867 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a04007cc0 con 0x7f4a14102e00 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.221+0000 7f4a0a000700 1 -- 172.21.15.63:0/4001711867 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a0400f040 con 0x7f4a14102e00 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.221+0000 7f4a0a000700 1 -- 172.21.15.63:0/4001711867 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a04009540 con 0x7f4a14102e00 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.221+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1410dd60 con 0x7f4a14102e00 2024-03-23T12:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.221+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1410e320 con 0x7f4a14102e00 2024-03-23T12:15:13.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.222+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a140fe550 con 0x7f4a14102e00 2024-03-23T12:15:13.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.223+0000 7f4a0a000700 1 -- 172.21.15.63:0/4001711867 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4a040096a0 con 0x7f4a14102e00 2024-03-23T12:15:13.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.224+0000 7f4a0a000700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f49f806c7c0 0x7f49f806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:13.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.224+0000 7f4a0a000700 1 -- 172.21.15.63:0/4001711867 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4a0408b820 con 0x7f4a14102e00 2024-03-23T12:15:13.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.225+0000 7f4a11400700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f49f806c7c0 0x7f49f806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:13.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.225+0000 7f4a11400700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f49f806c7c0 0x7f49f806ec70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a1410ec40 tx=0x7f4a00009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:13.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.229+0000 7f4a0a000700 1 -- 172.21.15.63:0/4001711867 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a040594b0 con 0x7f4a14102e00 2024-03-23T12:15:13.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.527+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4a14109dd0 con 0x7f4a14102e00 2024-03-23T12:15:13.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.528+0000 7f4a0a000700 1 -- 172.21.15.63:0/4001711867 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f4a04059040 con 0x7f4a14102e00 2024-03-23T12:15:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:15:13.507 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:15:13.508 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:15:13.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f49f806c7c0 msgr2=0x7f49f806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f49f806c7c0 0x7f49f806ec70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a1410ec40 tx=0x7f4a00009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:13.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a14102e00 msgr2=0x7f4a14113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a14102e00 0x7f4a14113b00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4a04002a50 tx=0x7f4a040046b0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 shutdown_connections 2024-03-23T12:15:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f49f806c7c0 0x7f49f806ec70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a14102e00 0x7f4a14113b00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 --2- 172.21.15.63:0/4001711867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a141037a0 0x7f4a14114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 >> 172.21.15.63:0/4001711867 conn(0x7f4a140f89b0 msgr2=0x7f4a140f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.530+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 shutdown_connections 2024-03-23T12:15:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.531+0000 7f4a18e00700 1 -- 172.21.15.63:0/4001711867 wait complete. 2024-03-23T12:15:13.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.668+0000 7f750be00700 1 -- 172.21.15.63:0/203897651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 msgr2=0x7f75040fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.668+0000 7f750be00700 1 --2- 172.21.15.63:0/203897651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f75040fce10 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f74f0004940 tx=0x7f74f0009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:13.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.669+0000 7f750be00700 1 -- 172.21.15.63:0/203897651 shutdown_connections 2024-03-23T12:15:13.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.669+0000 7f750be00700 1 --2- 172.21.15.63:0/203897651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f75040fce10 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.669+0000 7f750be00700 1 --2- 172.21.15.63:0/203897651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102e00 0x7f75041031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.669+0000 7f750be00700 1 -- 172.21.15.63:0/203897651 >> 172.21.15.63:0/203897651 conn(0x7f75040f89b0 msgr2=0x7f75040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:13.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.669+0000 7f750be00700 1 -- 172.21.15.63:0/203897651 shutdown_connections 2024-03-23T12:15:13.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.669+0000 7f750be00700 1 -- 172.21.15.63:0/203897651 wait complete. 2024-03-23T12:15:13.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.669+0000 7f750be00700 1 Processor -- start 2024-03-23T12:15:13.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f750be00700 1 -- start start 2024-03-23T12:15:13.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f750be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102e00 0x7f7504115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:13.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f750be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f7504116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f750be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504116890 con 0x7f75041037a0 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f750be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750410fc60 con 0x7f7504102e00 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f7508a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f7504116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f7508a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f7504116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38262/0 (socket says 172.21.15.63:38262) 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.670+0000 7f7508a00700 1 -- 172.21.15.63:0/1123263053 learned_addr learned my addr 172.21.15.63:0/1123263053 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7509400700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102e00 0x7f7504115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7508a00700 1 -- 172.21.15.63:0/1123263053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102e00 msgr2=0x7f7504115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7508a00700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102e00 0x7f7504115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7508a00700 1 -- 172.21.15.63:0/1123263053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f0005040 con 0x7f75041037a0 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7508a00700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f7504116240 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009390 tx=0x7f74f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7501600700 1 -- 172.21.15.63:0/1123263053 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f000e040 con 0x7f75041037a0 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7501600700 1 -- 172.21.15.63:0/1123263053 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74f0013420 con 0x7f75041037a0 2024-03-23T12:15:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f7501600700 1 -- 172.21.15.63:0/1123263053 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f001c630 con 0x7f75041037a0 2024-03-23T12:15:13.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750410fee0 con 0x7f75041037a0 2024-03-23T12:15:13.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.671+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75041104a0 con 0x7f75041037a0 2024-03-23T12:15:13.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.672+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75040fe550 con 0x7f75041037a0 2024-03-23T12:15:13.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.673+0000 7f7501600700 1 -- 172.21.15.63:0/1123263053 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f74f00139e0 con 0x7f75041037a0 2024-03-23T12:15:13.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.674+0000 7f7501600700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74ec06c7d0 0x7f74ec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:13.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.674+0000 7f7501600700 1 -- 172.21.15.63:0/1123263053 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f74f008ccf0 con 0x7f75041037a0 2024-03-23T12:15:13.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.675+0000 7f7509400700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74ec06c7d0 0x7f74ec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:13.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.675+0000 7f7509400700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74ec06c7d0 0x7f74ec06ec80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f74f8004990 tx=0x7f74f8009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:13.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.679+0000 7f7501600700 1 -- 172.21.15.63:0/1123263053 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f74f005aa00 con 0x7f75041037a0 2024-03-23T12:15:13.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.937+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f750404bed0 con 0x7f75041037a0 2024-03-23T12:15:13.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.937+0000 7f7501600700 1 -- 172.21.15.63:0/1123263053 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f74f005a590 con 0x7f75041037a0 2024-03-23T12:15:13.916 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:15:13.916 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:15:13.916 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:15:13.917 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:15:13.918 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.939+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74ec06c7d0 msgr2=0x7f74ec06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.939+0000 7f750be00700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74ec06c7d0 0x7f74ec06ec80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f74f8004990 tx=0x7f74f8009ee0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.939+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 msgr2=0x7f7504116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.939+0000 7f750be00700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f7504116240 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009390 tx=0x7f74f0009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.940+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 shutdown_connections 2024-03-23T12:15:13.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.940+0000 7f750be00700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f74ec06c7d0 0x7f74ec06ec80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.940+0000 7f750be00700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7504102e00 0x7f7504115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.940+0000 7f750be00700 1 --2- 172.21.15.63:0/1123263053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75041037a0 0x7f7504116240 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:13.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.940+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 >> 172.21.15.63:0/1123263053 conn(0x7f75040f89b0 msgr2=0x7f75040f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:13.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.940+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 shutdown_connections 2024-03-23T12:15:13.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:13.940+0000 7f750be00700 1 -- 172.21.15.63:0/1123263053 wait complete. 2024-03-23T12:15:13.921 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:15:14.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.078+0000 7fe33fe00700 1 -- 172.21.15.63:0/3477165721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3401035d0 msgr2=0x7fe3400fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.078+0000 7fe33fe00700 1 --2- 172.21.15.63:0/3477165721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3401035d0 0x7fe3400fcc40 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe32c004940 tx=0x7fe32c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.078+0000 7fe33fe00700 1 -- 172.21.15.63:0/3477165721 shutdown_connections 2024-03-23T12:15:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.078+0000 7fe33fe00700 1 --2- 172.21.15.63:0/3477165721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3401035d0 0x7fe3400fcc40 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.078+0000 7fe33fe00700 1 --2- 172.21.15.63:0/3477165721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe340102c30 0x7fe340103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.078+0000 7fe33fe00700 1 -- 172.21.15.63:0/3477165721 >> 172.21.15.63:0/3477165721 conn(0x7fe3400f8760 msgr2=0x7fe3400fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.078+0000 7fe33fe00700 1 -- 172.21.15.63:0/3477165721 shutdown_connections 2024-03-23T12:15:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.079+0000 7fe33fe00700 1 -- 172.21.15.63:0/3477165721 wait complete. 2024-03-23T12:15:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.079+0000 7fe33fe00700 1 Processor -- start 2024-03-23T12:15:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.079+0000 7fe33fe00700 1 -- start start 2024-03-23T12:15:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe340102c30 0x7fe3401138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3401035d0 0x7fe340113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340114480 con 0x7fe340102c30 2024-03-23T12:15:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33fe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34010d870 con 0x7fe3401035d0 2024-03-23T12:15:14.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe340102c30 0x7fe3401138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe340102c30 0x7fe3401138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38284/0 (socket says 172.21.15.63:38284) 2024-03-23T12:15:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33e000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3401035d0 0x7fe340113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.080+0000 7fe33ea00700 1 -- 172.21.15.63:0/1401217543 learned_addr learned my addr 172.21.15.63:0/1401217543 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:14.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe33ea00700 1 -- 172.21.15.63:0/1401217543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3401035d0 msgr2=0x7fe340113e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe33ea00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3401035d0 0x7fe340113e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe33ea00700 1 -- 172.21.15.63:0/1401217543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c005040 con 0x7fe340102c30 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe33ea00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe340102c30 0x7fe3401138f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe33000c8c0 tx=0x7fe3300044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe336a00700 1 -- 172.21.15.63:0/1401217543 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33000a760 con 0x7fe340102c30 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe336a00700 1 -- 172.21.15.63:0/1401217543 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe33000ad70 con 0x7fe340102c30 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe336a00700 1 -- 172.21.15.63:0/1401217543 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3300113a0 con 0x7fe340102c30 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34010db50 con 0x7fe340102c30 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.081+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34010e0e0 con 0x7fe340102c30 2024-03-23T12:15:14.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.082+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3400fe380 con 0x7fe340102c30 2024-03-23T12:15:14.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.083+0000 7fe336a00700 1 -- 172.21.15.63:0/1401217543 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe330012400 con 0x7fe340102c30 2024-03-23T12:15:14.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.084+0000 7fe336a00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32406c7c0 0x7fe32406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:14.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.084+0000 7fe336a00700 1 -- 172.21.15.63:0/1401217543 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe33008ba60 con 0x7fe340102c30 2024-03-23T12:15:14.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.084+0000 7fe33e000700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32406c7c0 0x7fe32406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:14.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.085+0000 7fe33e000700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32406c7c0 0x7fe32406ec70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe34010eb50 tx=0x7fe32c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:14.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.089+0000 7fe336a00700 1 -- 172.21.15.63:0/1401217543 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe330059770 con 0x7fe340102c30 2024-03-23T12:15:14.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.324+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe340109bc0 con 0x7fe32406c7c0 2024-03-23T12:15:14.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.325+0000 7fe336a00700 1 -- 172.21.15.63:0/1401217543 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe340109bc0 con 0x7fe32406c7c0 2024-03-23T12:15:14.304 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:15:14.305 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32406c7c0 msgr2=0x7fe32406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32406c7c0 0x7fe32406ec70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe34010eb50 tx=0x7fe32c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe340102c30 msgr2=0x7fe3401138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe340102c30 0x7fe3401138f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe33000c8c0 tx=0x7fe3300044c0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 shutdown_connections 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe32406c7c0 0x7fe32406ec70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe340102c30 0x7fe3401138f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 --2- 172.21.15.63:0/1401217543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3401035d0 0x7fe340113e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 >> 172.21.15.63:0/1401217543 conn(0x7fe3400f8760 msgr2=0x7fe3400f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:14.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 shutdown_connections 2024-03-23T12:15:14.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.328+0000 7fe33fe00700 1 -- 172.21.15.63:0/1401217543 wait complete. 2024-03-23T12:15:14.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:14 smithi063 ceph-mon[141124]: from='client.35334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:14.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:14 smithi063 ceph-mon[141124]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:14.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:14 smithi063 ceph-mon[141124]: pgmap v724: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:14.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:14 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4001711867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:15:14.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:14 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1123263053' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:15:14.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:14 smithi111 ceph-mon[106899]: from='client.35334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:14.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:14 smithi111 ceph-mon[106899]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:14.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:14 smithi111 ceph-mon[106899]: pgmap v724: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:14.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:14 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4001711867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:15:14.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:14 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1123263053' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 -- 172.21.15.63:0/368424974 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c8102e00 msgr2=0x7f75c81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 --2- 172.21.15.63:0/368424974 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c8102e00 0x7f75c81031d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f75bc004940 tx=0x7f75bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 -- 172.21.15.63:0/368424974 shutdown_connections 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 --2- 172.21.15.63:0/368424974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c81037a0 0x7f75c80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 --2- 172.21.15.63:0/368424974 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c8102e00 0x7f75c81031d0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 -- 172.21.15.63:0/368424974 >> 172.21.15.63:0/368424974 conn(0x7f75c80f89b0 msgr2=0x7f75c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 -- 172.21.15.63:0/368424974 shutdown_connections 2024-03-23T12:15:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.466+0000 7f75cfa00700 1 -- 172.21.15.63:0/368424974 wait complete. 2024-03-23T12:15:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.467+0000 7f75cfa00700 1 Processor -- start 2024-03-23T12:15:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.467+0000 7f75cfa00700 1 -- start start 2024-03-23T12:15:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.467+0000 7f75cfa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8102e00 0x7f75c8115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.467+0000 7f75cfa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c81037a0 0x7f75c81161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.467+0000 7f75cfa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8116810 con 0x7f75c81037a0 2024-03-23T12:15:14.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.467+0000 7f75cfa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c810fbe0 con 0x7f75c8102e00 2024-03-23T12:15:14.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75c7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c81037a0 0x7f75c81161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75c7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c81037a0 0x7f75c81161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38300/0 (socket says 172.21.15.63:38300) 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75c7e00700 1 -- 172.21.15.63:0/2922934739 learned_addr learned my addr 172.21.15.63:0/2922934739 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75cd000700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8102e00 0x7f75c8115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75c7e00700 1 -- 172.21.15.63:0/2922934739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8102e00 msgr2=0x7f75c8115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75c7e00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8102e00 0x7f75c8115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75c7e00700 1 -- 172.21.15.63:0/2922934739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc005040 con 0x7f75c81037a0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.468+0000 7f75c7e00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c81037a0 0x7f75c81161c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f75b400eb40 tx=0x7f75b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.469+0000 7f75c4c00700 1 -- 172.21.15.63:0/2922934739 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b4007cc0 con 0x7f75c81037a0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.469+0000 7f75c4c00700 1 -- 172.21.15.63:0/2922934739 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75b400f040 con 0x7f75c81037a0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.469+0000 7f75c4c00700 1 -- 172.21.15.63:0/2922934739 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b4009570 con 0x7f75c81037a0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.469+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c810fe60 con 0x7f75c81037a0 2024-03-23T12:15:14.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.469+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81103d0 con 0x7f75c81037a0 2024-03-23T12:15:14.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.470+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c80fe550 con 0x7f75c81037a0 2024-03-23T12:15:14.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.471+0000 7f75c4c00700 1 -- 172.21.15.63:0/2922934739 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f75b40096d0 con 0x7f75c81037a0 2024-03-23T12:15:14.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.472+0000 7f75c4c00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f75b006c7c0 0x7f75b006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:14.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.472+0000 7f75c4c00700 1 -- 172.21.15.63:0/2922934739 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f75b408b650 con 0x7f75c81037a0 2024-03-23T12:15:14.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.472+0000 7f75cd000700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f75b006c7c0 0x7f75b006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:14.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.473+0000 7f75cd000700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f75b006c7c0 0x7f75b006ec70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00b0d0 tx=0x7f75bc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:14.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.476+0000 7f75c4c00700 1 -- 172.21.15.63:0/2922934739 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f75b4059310 con 0x7f75c81037a0 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.774+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f75c805f5f0 con 0x7f75c81037a0 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.775+0000 7f75c4c00700 1 -- 172.21.15.63:0/2922934739 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f75b4019e10 con 0x7f75c81037a0 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:15:14.754 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:15:14.755 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:15:14.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.777+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f75b006c7c0 msgr2=0x7f75b006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.777+0000 7f75cfa00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f75b006c7c0 0x7f75b006ec70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00b0d0 tx=0x7f75bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.777+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c81037a0 msgr2=0x7f75c81161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.777+0000 7f75cfa00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c81037a0 0x7f75c81161c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f75b400eb40 tx=0x7f75b400b040 comp rx=0 tx=0).stop 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.778+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 shutdown_connections 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.778+0000 7f75cfa00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f75b006c7c0 0x7f75b006ec70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.778+0000 7f75cfa00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c8102e00 0x7f75c8115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.778+0000 7f75cfa00700 1 --2- 172.21.15.63:0/2922934739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75c81037a0 0x7f75c81161c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.778+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 >> 172.21.15.63:0/2922934739 conn(0x7f75c80f89b0 msgr2=0x7f75c80f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.778+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 shutdown_connections 2024-03-23T12:15:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:14.778+0000 7f75cfa00700 1 -- 172.21.15.63:0/2922934739 wait complete. 2024-03-23T12:15:15.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:15 smithi063 ceph-mon[141124]: from='client.35342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:15.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:15 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2922934739' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:15:15.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:15 smithi111 ceph-mon[106899]: from='client.35342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:15.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:15 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2922934739' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:15:16.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:16 smithi063 ceph-mon[141124]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:16.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:16 smithi063 ceph-mon[141124]: pgmap v725: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:16.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:16 smithi111 ceph-mon[106899]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:16.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:16 smithi111 ceph-mon[106899]: pgmap v725: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:17.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:17 smithi063 ceph-mon[141124]: pgmap v726: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:17.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:17 smithi111 ceph-mon[106899]: pgmap v726: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:20.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:20 smithi063 ceph-mon[141124]: pgmap v727: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:20.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:20 smithi111 ceph-mon[106899]: pgmap v727: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:21.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:21 smithi063 ceph-mon[141124]: pgmap v728: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:21.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:21 smithi111 ceph-mon[106899]: pgmap v728: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:24.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:24 smithi063 ceph-mon[141124]: pgmap v729: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:24.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:24 smithi111 ceph-mon[106899]: pgmap v729: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:25 smithi063 ceph-mon[141124]: pgmap v730: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:25 smithi111 ceph-mon[106899]: pgmap v730: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:26.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:26.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:27.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:27 smithi063 ceph-mon[141124]: pgmap v731: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:27.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:27 smithi111 ceph-mon[106899]: pgmap v731: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:29.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:29 smithi063 ceph-mon[141124]: pgmap v732: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:29.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:29 smithi111 ceph-mon[106899]: pgmap v732: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:31.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:31 smithi063 ceph-mon[141124]: pgmap v733: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:31.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:31 smithi111 ceph-mon[106899]: pgmap v733: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:33.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:33 smithi063 ceph-mon[141124]: pgmap v734: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:33.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:33 smithi111 ceph-mon[106899]: pgmap v734: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:35.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:35 smithi063 ceph-mon[141124]: pgmap v735: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:35.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:35 smithi111 ceph-mon[106899]: pgmap v735: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:37.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:37 smithi063 ceph-mon[141124]: pgmap v736: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:37.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:37 smithi111 ceph-mon[106899]: pgmap v736: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:39.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:39 smithi063 ceph-mon[141124]: pgmap v737: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:39.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:39 smithi111 ceph-mon[106899]: pgmap v737: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:41.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:41 smithi063 ceph-mon[141124]: pgmap v738: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:41 smithi111 ceph-mon[106899]: pgmap v738: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:43.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:43 smithi063 ceph-mon[141124]: pgmap v739: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:43.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:43 smithi111 ceph-mon[106899]: pgmap v739: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:44.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 -- 172.21.15.63:0/2921516414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 msgr2=0x7f33fc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:44.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 --2- 172.21.15.63:0/2921516414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc0fce10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f33e4004940 tx=0x7f33e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:44.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 -- 172.21.15.63:0/2921516414 shutdown_connections 2024-03-23T12:15:44.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 --2- 172.21.15.63:0/2921516414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc0fce10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:44.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 --2- 172.21.15.63:0/2921516414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33fc102e00 0x7f33fc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:44.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 -- 172.21.15.63:0/2921516414 >> 172.21.15.63:0/2921516414 conn(0x7f33fc0f89b0 msgr2=0x7f33fc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:44.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 -- 172.21.15.63:0/2921516414 shutdown_connections 2024-03-23T12:15:44.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.931+0000 7f3400a00700 1 -- 172.21.15.63:0/2921516414 wait complete. 2024-03-23T12:15:44.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.932+0000 7f3400a00700 1 Processor -- start 2024-03-23T12:15:44.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.932+0000 7f3400a00700 1 -- start start 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f3400a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33fc102e00 0x7f33fc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f3400a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f3400a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc070d90 con 0x7f33fc1037a0 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f3400a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc070ed0 con 0x7f33fc102e00 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f33f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f33f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46502/0 (socket says 172.21.15.63:46502) 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f33f3e00700 1 -- 172.21.15.63:0/4218663456 learned_addr learned my addr 172.21.15.63:0/4218663456 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f33f9e00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33fc102e00 0x7f33fc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f33f3e00700 1 -- 172.21.15.63:0/4218663456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33fc102e00 msgr2=0x7f33fc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f33f3e00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33fc102e00 0x7f33fc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.933+0000 7f33f3e00700 1 -- 172.21.15.63:0/4218663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4005040 con 0x7f33fc1037a0 2024-03-23T12:15:44.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.934+0000 7f33f3e00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc070800 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009390 tx=0x7f33e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:44.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.934+0000 7f33f2000700 1 -- 172.21.15.63:0/4218663456 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e400e040 con 0x7f33fc1037a0 2024-03-23T12:15:44.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.934+0000 7f33f2000700 1 -- 172.21.15.63:0/4218663456 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33e4013420 con 0x7f33fc1037a0 2024-03-23T12:15:44.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.934+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc071120 con 0x7f33fc1037a0 2024-03-23T12:15:44.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.934+0000 7f33f2000700 1 -- 172.21.15.63:0/4218663456 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e401c630 con 0x7f33fc1037a0 2024-03-23T12:15:44.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.934+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc11a230 con 0x7f33fc1037a0 2024-03-23T12:15:44.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.935+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc0fe550 con 0x7f33fc1037a0 2024-03-23T12:15:44.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.936+0000 7f33f2000700 1 -- 172.21.15.63:0/4218663456 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f33e4013a00 con 0x7f33fc1037a0 2024-03-23T12:15:44.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.937+0000 7f33f2000700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f33e006c7d0 0x7f33e006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:44.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.937+0000 7f33f2000700 1 -- 172.21.15.63:0/4218663456 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f33e408caa0 con 0x7f33fc1037a0 2024-03-23T12:15:44.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.937+0000 7f33f9e00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f33e006c7d0 0x7f33e006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:44.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.938+0000 7f33f9e00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f33e006c7d0 0x7f33e006ec80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f33ec004f40 tx=0x7f33ec00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:44.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:44.942+0000 7f33f2000700 1 -- 172.21.15.63:0/4218663456 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f33e4056de0 con 0x7f33fc1037a0 2024-03-23T12:15:45.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.173+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f33fc109da0 con 0x7f33e006c7d0 2024-03-23T12:15:45.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.174+0000 7f33f2000700 1 -- 172.21.15.63:0/4218663456 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f33fc109da0 con 0x7f33e006c7d0 2024-03-23T12:15:45.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f33e006c7d0 msgr2=0x7f33e006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f33e006c7d0 0x7f33e006ec80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f33ec004f40 tx=0x7f33ec00b040 comp rx=0 tx=0).stop 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 msgr2=0x7f33fc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc070800 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009390 tx=0x7f33e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 shutdown_connections 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f33e006c7d0 0x7f33e006ec80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33fc102e00 0x7f33fc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 --2- 172.21.15.63:0/4218663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33fc1037a0 0x7f33fc070800 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 >> 172.21.15.63:0/4218663456 conn(0x7f33fc0f89b0 msgr2=0x7f33fc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 shutdown_connections 2024-03-23T12:15:45.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.177+0000 7f3400a00700 1 -- 172.21.15.63:0/4218663456 wait complete. 2024-03-23T12:15:45.171 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:15:45.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 -- 172.21.15.63:0/2104016394 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fdeb0 msgr2=0x7f07e80fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 --2- 172.21.15.63:0/2104016394 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fdeb0 0x7f07e80fe280 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f07dc004970 tx=0x7f07dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:45.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 -- 172.21.15.63:0/2104016394 shutdown_connections 2024-03-23T12:15:45.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 --2- 172.21.15.63:0/2104016394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07e80fe850 0x7f07e8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 --2- 172.21.15.63:0/2104016394 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fdeb0 0x7f07e80fe280 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 -- 172.21.15.63:0/2104016394 >> 172.21.15.63:0/2104016394 conn(0x7f07e8072bc0 msgr2=0x7f07e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 -- 172.21.15.63:0/2104016394 shutdown_connections 2024-03-23T12:15:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.315+0000 7f07ef400700 1 -- 172.21.15.63:0/2104016394 wait complete. 2024-03-23T12:15:45.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.316+0000 7f07ef400700 1 Processor -- start 2024-03-23T12:15:45.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.316+0000 7f07ef400700 1 -- start start 2024-03-23T12:15:45.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.316+0000 7f07ef400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07e80fdeb0 0x7f07e80fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.316+0000 7f07ef400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fe850 0x7f07e80fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.316+0000 7f07ef400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80fc840 con 0x7f07e80fe850 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.316+0000 7f07ef400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80fc980 con 0x7f07e80fdeb0 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.316+0000 7f07e7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fe850 0x7f07e80fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07e7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fe850 0x7f07e80fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46520/0 (socket says 172.21.15.63:46520) 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07e7e00700 1 -- 172.21.15.63:0/2678397988 learned_addr learned my addr 172.21.15.63:0/2678397988 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07e7e00700 1 -- 172.21.15.63:0/2678397988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07e80fdeb0 msgr2=0x7f07e80fbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07eca00700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07e80fdeb0 0x7f07e80fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07e7e00700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07e80fdeb0 0x7f07e80fbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07e7e00700 1 -- 172.21.15.63:0/2678397988 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc005040 con 0x7f07e80fe850 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07eca00700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07e80fdeb0 0x7f07e80fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07e7e00700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fe850 0x7f07e80fc2b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f07d4002a50 tx=0x7f07d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:45.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.317+0000 7f07e4c00700 1 -- 172.21.15.63:0/2678397988 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d4007cc0 con 0x7f07e80fe850 2024-03-23T12:15:45.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.318+0000 7f07e4c00700 1 -- 172.21.15.63:0/2678397988 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07d400f040 con 0x7f07e80fe850 2024-03-23T12:15:45.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.318+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e8119d70 con 0x7f07e80fe850 2024-03-23T12:15:45.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.318+0000 7f07e4c00700 1 -- 172.21.15.63:0/2678397988 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d4009540 con 0x7f07e80fe850 2024-03-23T12:15:45.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.318+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e811a1d0 con 0x7f07e80fe850 2024-03-23T12:15:45.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.319+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e8065e10 con 0x7f07e80fe850 2024-03-23T12:15:45.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.320+0000 7f07e4c00700 1 -- 172.21.15.63:0/2678397988 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f07d40096a0 con 0x7f07e80fe850 2024-03-23T12:15:45.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.321+0000 7f07e4c00700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d006c7d0 0x7f07d006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:45.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.321+0000 7f07e4c00700 1 -- 172.21.15.63:0/2678397988 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f07d408b910 con 0x7f07e80fe850 2024-03-23T12:15:45.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.321+0000 7f07eca00700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d006c7d0 0x7f07d006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:45.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.322+0000 7f07eca00700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d006c7d0 0x7f07d006ec80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00b800 tx=0x7f07dc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:45.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.325+0000 7f07e4c00700 1 -- 172.21.15.63:0/2678397988 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07d4055a90 con 0x7f07e80fe850 2024-03-23T12:15:45.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:45 smithi063 ceph-mon[141124]: pgmap v740: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:45.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:45 smithi111 ceph-mon[106899]: pgmap v740: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:45.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.556+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07e804bed0 con 0x7f07d006c7d0 2024-03-23T12:15:45.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.557+0000 7f07e4c00700 1 -- 172.21.15.63:0/2678397988 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f07e804bed0 con 0x7f07d006c7d0 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d006c7d0 msgr2=0x7f07d006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d006c7d0 0x7f07d006ec80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00b800 tx=0x7f07dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fe850 msgr2=0x7f07e80fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fe850 0x7f07e80fc2b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f07d4002a50 tx=0x7f07d40046b0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 shutdown_connections 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07d006c7d0 0x7f07d006ec80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07e80fdeb0 0x7f07e80fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 --2- 172.21.15.63:0/2678397988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07e80fe850 0x7f07e80fc2b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 >> 172.21.15.63:0/2678397988 conn(0x7f07e8072bc0 msgr2=0x7f07e80ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 shutdown_connections 2024-03-23T12:15:45.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.560+0000 7f07ef400700 1 -- 172.21.15.63:0/2678397988 wait complete. 2024-03-23T12:15:45.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 -- 172.21.15.63:0/2770715316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6218071f90 msgr2=0x7f6218070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 --2- 172.21.15.63:0/2770715316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6218071f90 0x7f6218070360 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f620c004970 tx=0x7f620c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:45.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 -- 172.21.15.63:0/2770715316 shutdown_connections 2024-03-23T12:15:45.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 --2- 172.21.15.63:0/2770715316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62180708a0 0x7f6218070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 --2- 172.21.15.63:0/2770715316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6218071f90 0x7f6218070360 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 -- 172.21.15.63:0/2770715316 >> 172.21.15.63:0/2770715316 conn(0x7f62180f89b0 msgr2=0x7f62180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:45.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 -- 172.21.15.63:0/2770715316 shutdown_connections 2024-03-23T12:15:45.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.698+0000 7f6220200700 1 -- 172.21.15.63:0/2770715316 wait complete. 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.699+0000 7f6220200700 1 Processor -- start 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.699+0000 7f6220200700 1 -- start start 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f6220200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62180708a0 0x7f621811a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f6220200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218071f90 0x7f621811a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f6220200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621811ac50 con 0x7f62180708a0 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f6220200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218114020 con 0x7f6218071f90 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62180708a0 0x7f621811a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:45.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62180708a0 0x7f621811a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46544/0 (socket says 172.21.15.63:46544) 2024-03-23T12:15:45.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621d800700 1 -- 172.21.15.63:0/2221511733 learned_addr learned my addr 172.21.15.63:0/2221511733 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:45.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621ce00700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218071f90 0x7f621811a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:45.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621d800700 1 -- 172.21.15.63:0/2221511733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218071f90 msgr2=0x7f621811a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621d800700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218071f90 0x7f621811a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621d800700 1 -- 172.21.15.63:0/2221511733 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c005040 con 0x7f62180708a0 2024-03-23T12:15:45.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.700+0000 7f621ce00700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218071f90 0x7f621811a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:15:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.701+0000 7f621d800700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62180708a0 0x7f621811a0c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f620c004f50 tx=0x7f620c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.701+0000 7f6215600700 1 -- 172.21.15.63:0/2221511733 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f620c00e040 con 0x7f62180708a0 2024-03-23T12:15:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.701+0000 7f6215600700 1 -- 172.21.15.63:0/2221511733 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f620c013420 con 0x7f62180708a0 2024-03-23T12:15:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.701+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62181142a0 con 0x7f62180708a0 2024-03-23T12:15:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.701+0000 7f6215600700 1 -- 172.21.15.63:0/2221511733 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f620c01c610 con 0x7f62180708a0 2024-03-23T12:15:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.701+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62181147e0 con 0x7f62180708a0 2024-03-23T12:15:45.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.702+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6218071aa0 con 0x7f62180708a0 2024-03-23T12:15:45.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.703+0000 7f6215600700 1 -- 172.21.15.63:0/2221511733 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f620c013590 con 0x7f62180708a0 2024-03-23T12:15:45.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.704+0000 7f6215600700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f620006c7d0 0x7f620006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:45.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.704+0000 7f6215600700 1 -- 172.21.15.63:0/2221511733 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f620c08cd30 con 0x7f62180708a0 2024-03-23T12:15:45.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.704+0000 7f621ce00700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f620006c7d0 0x7f620006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.705+0000 7f621ce00700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f620006c7d0 0x7f620006ec80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6204007fd0 tx=0x7f6204009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:45.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.709+0000 7f6215600700 1 -- 172.21.15.63:0/2221511733 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f620c05a9c0 con 0x7f62180708a0 2024-03-23T12:15:45.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.933+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6218071aa0 con 0x7f620006c7d0 2024-03-23T12:15:45.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.943+0000 7f6215600700 1 -- 172.21.15.63:0/2221511733 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f6218071aa0 con 0x7f620006c7d0 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (25m) 2m ago 38m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (22m) 2m ago 38m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (22m) 2m ago 37m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (24m) 2m ago 37m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (18m) 2m ago 31m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (17m) 2m ago 31m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (2m) 2m ago 31m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (17m) 2m ago 31m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (27m) 2m ago 39m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (26m) 2m ago 37m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (23m) 2m ago 39m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:15:45.923 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (23m) 2m ago 36m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (25m) 2m ago 38m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (25m) 2m ago 37m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (21m) 2m ago 36m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (21m) 2m ago 35m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (20m) 2m ago 35m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (19m) 2m ago 34m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (19m) 2m ago 34m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (18m) 2m ago 33m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:15:45.924 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (25m) 2m ago 37m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f620006c7d0 msgr2=0x7f620006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f620006c7d0 0x7f620006ec80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6204007fd0 tx=0x7f6204009f60 comp rx=0 tx=0).stop 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62180708a0 msgr2=0x7f621811a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62180708a0 0x7f621811a0c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f620c004f50 tx=0x7f620c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 shutdown_connections 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f620006c7d0 0x7f620006ec80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62180708a0 0x7f621811a0c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 --2- 172.21.15.63:0/2221511733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218071f90 0x7f621811a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 >> 172.21.15.63:0/2221511733 conn(0x7f62180f89b0 msgr2=0x7f62180f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:45.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 shutdown_connections 2024-03-23T12:15:45.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:45.945+0000 7f6220200700 1 -- 172.21.15.63:0/2221511733 wait complete. 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.082+0000 7f14a6c00700 1 -- 172.21.15.63:0/1441203838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 msgr2=0x7f14a00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.082+0000 7f14a6c00700 1 --2- 172.21.15.63:0/1441203838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a00fce10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f148c004940 tx=0x7f148c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.083+0000 7f14a6c00700 1 -- 172.21.15.63:0/1441203838 shutdown_connections 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.083+0000 7f14a6c00700 1 --2- 172.21.15.63:0/1441203838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a00fce10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.083+0000 7f14a6c00700 1 --2- 172.21.15.63:0/1441203838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a0102e00 0x7f14a01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.083+0000 7f14a6c00700 1 -- 172.21.15.63:0/1441203838 >> 172.21.15.63:0/1441203838 conn(0x7f14a00f89b0 msgr2=0x7f14a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.083+0000 7f14a6c00700 1 -- 172.21.15.63:0/1441203838 shutdown_connections 2024-03-23T12:15:46.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.083+0000 7f14a6c00700 1 -- 172.21.15.63:0/1441203838 wait complete. 2024-03-23T12:15:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.083+0000 7f14a6c00700 1 Processor -- start 2024-03-23T12:15:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f14a6c00700 1 -- start start 2024-03-23T12:15:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f14a6c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a0102e00 0x7f14a0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f14a6c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f14a6c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0116810 con 0x7f14a01037a0 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f14a6c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a010fbe0 con 0x7f14a0102e00 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f149f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f149f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a01161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46554/0 (socket says 172.21.15.63:46554) 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.084+0000 7f149f400700 1 -- 172.21.15.63:0/105333922 learned_addr learned my addr 172.21.15.63:0/105333922 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f149fe00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a0102e00 0x7f14a0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f149f400700 1 -- 172.21.15.63:0/105333922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a0102e00 msgr2=0x7f14a0115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f149f400700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a0102e00 0x7f14a0115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f149f400700 1 -- 172.21.15.63:0/105333922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c005040 con 0x7f14a01037a0 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f149f400700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a01161c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f148c00b0d0 tx=0x7f148c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f1497e00700 1 -- 172.21.15.63:0/105333922 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f148c021070 con 0x7f14a01037a0 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a010fe60 con 0x7f14a01037a0 2024-03-23T12:15:46.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.085+0000 7f1497e00700 1 -- 172.21.15.63:0/105333922 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f148c013e00 con 0x7f14a01037a0 2024-03-23T12:15:46.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.086+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a01103c0 con 0x7f14a01037a0 2024-03-23T12:15:46.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.086+0000 7f1497e00700 1 -- 172.21.15.63:0/105333922 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f148c00e070 con 0x7f14a01037a0 2024-03-23T12:15:46.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.087+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a00fe550 con 0x7f14a01037a0 2024-03-23T12:15:46.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.087+0000 7f1497e00700 1 -- 172.21.15.63:0/105333922 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f148c013950 con 0x7f14a01037a0 2024-03-23T12:15:46.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.089+0000 7f1497e00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f148406c7c0 0x7f148406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.089+0000 7f1497e00700 1 -- 172.21.15.63:0/105333922 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f148c08cb00 con 0x7f14a01037a0 2024-03-23T12:15:46.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.089+0000 7f149fe00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f148406c7c0 0x7f148406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.090+0000 7f149fe00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f148406c7c0 0x7f148406ec70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1490009b80 tx=0x7f149000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.093+0000 7f1497e00700 1 -- 172.21.15.63:0/105333922 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f148c05a790 con 0x7f14a01037a0 2024-03-23T12:15:46.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.400+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f14a004bed0 con 0x7f14a01037a0 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.401+0000 7f1497e00700 1 -- 172.21.15.63:0/105333922 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f148c05a320 con 0x7f14a01037a0 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f148406c7c0 msgr2=0x7f148406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f148406c7c0 0x7f148406ec70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1490009b80 tx=0x7f149000b040 comp rx=0 tx=0).stop 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 msgr2=0x7f14a01161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a01161c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f148c00b0d0 tx=0x7f148c009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 shutdown_connections 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f148406c7c0 0x7f148406ec70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a0102e00 0x7f14a0115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 --2- 172.21.15.63:0/105333922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14a01037a0 0x7f14a01161c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 >> 172.21.15.63:0/105333922 conn(0x7f14a00f89b0 msgr2=0x7f14a00f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 shutdown_connections 2024-03-23T12:15:46.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.404+0000 7f14a6c00700 1 -- 172.21.15.63:0/105333922 wait complete. 2024-03-23T12:15:46.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 -- 172.21.15.63:0/1314720879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c1035e0 msgr2=0x7fed9c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 --2- 172.21.15.63:0/1314720879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c1035e0 0x7fed9c0fcc50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fed88004940 tx=0x7fed88009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:46.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 -- 172.21.15.63:0/1314720879 shutdown_connections 2024-03-23T12:15:46.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 --2- 172.21.15.63:0/1314720879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c1035e0 0x7fed9c0fcc50 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 --2- 172.21.15.63:0/1314720879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed9c102c40 0x7fed9c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 -- 172.21.15.63:0/1314720879 >> 172.21.15.63:0/1314720879 conn(0x7fed9c0f8830 msgr2=0x7fed9c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:46.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 -- 172.21.15.63:0/1314720879 shutdown_connections 2024-03-23T12:15:46.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.540+0000 7feda2a00700 1 -- 172.21.15.63:0/1314720879 wait complete. 2024-03-23T12:15:46.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.541+0000 7feda2a00700 1 Processor -- start 2024-03-23T12:15:46.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.541+0000 7feda2a00700 1 -- start start 2024-03-23T12:15:46.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.541+0000 7feda2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c102c40 0x7fed9c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda2a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed9c1035e0 0x7fed9c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda2a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c070d90 con 0x7fed9c102c40 2024-03-23T12:15:46.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda2a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c070ed0 con 0x7fed9c1035e0 2024-03-23T12:15:46.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c102c40 0x7fed9c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c102c40 0x7fed9c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46584/0 (socket says 172.21.15.63:46584) 2024-03-23T12:15:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda1600700 1 -- 172.21.15.63:0/1803242808 learned_addr learned my addr 172.21.15.63:0/1803242808 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda0c00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed9c1035e0 0x7fed9c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda1600700 1 -- 172.21.15.63:0/1803242808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed9c1035e0 msgr2=0x7fed9c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda1600700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed9c1035e0 0x7fed9c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda1600700 1 -- 172.21.15.63:0/1803242808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88005040 con 0x7fed9c102c40 2024-03-23T12:15:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.542+0000 7feda1600700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c102c40 0x7fed9c0702c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fed90002a50 tx=0x7fed900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.543+0000 7fed99600700 1 -- 172.21.15.63:0/1803242808 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed90007cc0 con 0x7fed9c102c40 2024-03-23T12:15:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.543+0000 7fed99600700 1 -- 172.21.15.63:0/1803242808 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed9000f040 con 0x7fed9c102c40 2024-03-23T12:15:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.543+0000 7fed99600700 1 -- 172.21.15.63:0/1803242808 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed9000d9c0 con 0x7fed9c102c40 2024-03-23T12:15:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.543+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c119be0 con 0x7fed9c102c40 2024-03-23T12:15:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.543+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c11a060 con 0x7fed9c102c40 2024-03-23T12:15:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.544+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c0fe390 con 0x7fed9c102c40 2024-03-23T12:15:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.545+0000 7fed99600700 1 -- 172.21.15.63:0/1803242808 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fed9000db20 con 0x7fed9c102c40 2024-03-23T12:15:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.546+0000 7fed99600700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fed8406c7d0 0x7fed8406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.546+0000 7fed99600700 1 -- 172.21.15.63:0/1803242808 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fed9008b6f0 con 0x7fed9c102c40 2024-03-23T12:15:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.546+0000 7feda0c00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fed8406c7d0 0x7fed8406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.546+0000 7feda0c00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fed8406c7d0 0x7fed8406ec80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fed9c071c00 tx=0x7fed88009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:46.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.550+0000 7fed99600700 1 -- 172.21.15.63:0/1803242808 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fed90055970 con 0x7fed9c102c40 2024-03-23T12:15:46.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.810+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fed9c0719b0 con 0x7fed9c102c40 2024-03-23T12:15:46.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.810+0000 7fed99600700 1 -- 172.21.15.63:0/1803242808 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fed90058f90 con 0x7fed9c102c40 2024-03-23T12:15:46.790 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:15:46.791 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:15:46.792 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:46.793 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:15:46.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fed8406c7d0 msgr2=0x7fed8406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fed8406c7d0 0x7fed8406ec80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fed9c071c00 tx=0x7fed88009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:46.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c102c40 msgr2=0x7fed9c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c102c40 0x7fed9c0702c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fed90002a50 tx=0x7fed900046b0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 shutdown_connections 2024-03-23T12:15:46.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fed8406c7d0 0x7fed8406ec80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed9c102c40 0x7fed9c0702c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 --2- 172.21.15.63:0/1803242808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed9c1035e0 0x7fed9c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 >> 172.21.15.63:0/1803242808 conn(0x7fed9c0f8830 msgr2=0x7fed9c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:46.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 shutdown_connections 2024-03-23T12:15:46.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.813+0000 7feda2a00700 1 -- 172.21.15.63:0/1803242808 wait complete. 2024-03-23T12:15:46.795 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:15:46.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:46 smithi063 ceph-mon[141124]: from='client.35362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:46.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:46 smithi063 ceph-mon[141124]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:46.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:46 smithi063 ceph-mon[141124]: from='client.35370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:46.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:46 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/105333922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:15:46.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:46 smithi111 ceph-mon[106899]: from='client.35362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:46.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:46 smithi111 ceph-mon[106899]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:46.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:46 smithi111 ceph-mon[106899]: from='client.35370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:46.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:46 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/105333922' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:15:46.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.951+0000 7fc9bec00700 1 -- 172.21.15.63:0/37206783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 msgr2=0x7fc9b80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.951+0000 7fc9bec00700 1 --2- 172.21.15.63:0/37206783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b80fce10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4004940 tx=0x7fc9a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.951+0000 7fc9bec00700 1 -- 172.21.15.63:0/37206783 shutdown_connections 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.951+0000 7fc9bec00700 1 --2- 172.21.15.63:0/37206783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b80fce10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.951+0000 7fc9bec00700 1 --2- 172.21.15.63:0/37206783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9b8102e00 0x7fc9b81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.951+0000 7fc9bec00700 1 -- 172.21.15.63:0/37206783 >> 172.21.15.63:0/37206783 conn(0x7fc9b80f89b0 msgr2=0x7fc9b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.952+0000 7fc9bec00700 1 -- 172.21.15.63:0/37206783 shutdown_connections 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.952+0000 7fc9bec00700 1 -- 172.21.15.63:0/37206783 wait complete. 2024-03-23T12:15:46.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.953+0000 7fc9bec00700 1 Processor -- start 2024-03-23T12:15:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.953+0000 7fc9bec00700 1 -- start start 2024-03-23T12:15:46.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.953+0000 7fc9bec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9b8102e00 0x7fc9b8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9bec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b81161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9bec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b81167f0 con 0x7fc9b81037a0 2024-03-23T12:15:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9bec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b810fbc0 con 0x7fc9b8102e00 2024-03-23T12:15:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b81161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b81161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46596/0 (socket says 172.21.15.63:46596) 2024-03-23T12:15:46.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7400700 1 -- 172.21.15.63:0/1075782715 learned_addr learned my addr 172.21.15.63:0/1075782715 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7e00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9b8102e00 0x7fc9b8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7400700 1 -- 172.21.15.63:0/1075782715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9b8102e00 msgr2=0x7fc9b8115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7400700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9b8102e00 0x7fc9b8115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7400700 1 -- 172.21.15.63:0/1075782715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4005040 con 0x7fc9b81037a0 2024-03-23T12:15:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7e00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9b8102e00 0x7fc9b8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:15:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.954+0000 7fc9b7400700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b81161a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009fd0 tx=0x7fc9a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.955+0000 7fc9afe00700 1 -- 172.21.15.63:0/1075782715 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9a400e030 con 0x7fc9b81037a0 2024-03-23T12:15:46.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.955+0000 7fc9afe00700 1 -- 172.21.15.63:0/1075782715 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9a4013e00 con 0x7fc9b81037a0 2024-03-23T12:15:46.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.955+0000 7fc9afe00700 1 -- 172.21.15.63:0/1075782715 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9a401ccd0 con 0x7fc9b81037a0 2024-03-23T12:15:46.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.955+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b810fe40 con 0x7fc9b81037a0 2024-03-23T12:15:46.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.955+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b8110400 con 0x7fc9b81037a0 2024-03-23T12:15:46.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.956+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b80fe550 con 0x7fc9b81037a0 2024-03-23T12:15:46.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.957+0000 7fc9afe00700 1 -- 172.21.15.63:0/1075782715 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc9a401c500 con 0x7fc9b81037a0 2024-03-23T12:15:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.958+0000 7fc9afe00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc99c06c7d0 0x7fc99c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.958+0000 7fc9afe00700 1 -- 172.21.15.63:0/1075782715 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc9a408d220 con 0x7fc9b81037a0 2024-03-23T12:15:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.958+0000 7fc9b7e00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc99c06c7d0 0x7fc99c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:46.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.959+0000 7fc9b7e00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc99c06c7d0 0x7fc99c06ec80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009fd0 tx=0x7fc9a800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:46.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:46.962+0000 7fc9afe00700 1 -- 172.21.15.63:0/1075782715 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc9a405aff0 con 0x7fc9b81037a0 2024-03-23T12:15:47.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.195+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9b8109da0 con 0x7fc99c06c7d0 2024-03-23T12:15:47.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.196+0000 7fc9afe00700 1 -- 172.21.15.63:0/1075782715 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc9b8109da0 con 0x7fc99c06c7d0 2024-03-23T12:15:47.176 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:15:47.176 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:15:47.176 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:15:47.176 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:15:47.176 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:15:47.177 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc99c06c7d0 msgr2=0x7fc99c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc99c06c7d0 0x7fc99c06ec80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009fd0 tx=0x7fc9a800b040 comp rx=0 tx=0).stop 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 msgr2=0x7fc9b81161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b81161a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009fd0 tx=0x7fc9a4009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 shutdown_connections 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc99c06c7d0 0x7fc99c06ec80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9b8102e00 0x7fc9b8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 --2- 172.21.15.63:0/1075782715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9b81037a0 0x7fc9b81161a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 >> 172.21.15.63:0/1075782715 conn(0x7fc9b80f89b0 msgr2=0x7fc9b80f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:47.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 shutdown_connections 2024-03-23T12:15:47.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.199+0000 7fc9bec00700 1 -- 172.21.15.63:0/1075782715 wait complete. 2024-03-23T12:15:47.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 -- 172.21.15.63:0/1169224430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 msgr2=0x7f32880fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:47.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 --2- 172.21.15.63:0/1169224430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f32880fce10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3274004940 tx=0x7f3274009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:47.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 -- 172.21.15.63:0/1169224430 shutdown_connections 2024-03-23T12:15:47.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 --2- 172.21.15.63:0/1169224430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f32880fce10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 --2- 172.21.15.63:0/1169224430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3288102e00 0x7f32881031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 -- 172.21.15.63:0/1169224430 >> 172.21.15.63:0/1169224430 conn(0x7f32880f89b0 msgr2=0x7f32880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:47.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 -- 172.21.15.63:0/1169224430 shutdown_connections 2024-03-23T12:15:47.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.336+0000 7f328ea00700 1 -- 172.21.15.63:0/1169224430 wait complete. 2024-03-23T12:15:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.337+0000 7f328ea00700 1 Processor -- start 2024-03-23T12:15:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.337+0000 7f328ea00700 1 -- start start 2024-03-23T12:15:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f328ea00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3288102e00 0x7f3288115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f328ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f3288116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f328ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32881168b0 con 0x7f32881037a0 2024-03-23T12:15:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f328ea00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328810fc80 con 0x7f3288102e00 2024-03-23T12:15:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f3287400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f3288116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:47.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f3287400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f3288116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46610/0 (socket says 172.21.15.63:46610) 2024-03-23T12:15:47.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f3287400700 1 -- 172.21.15.63:0/1368469855 learned_addr learned my addr 172.21.15.63:0/1368469855 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:15:47.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f3287e00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3288102e00 0x7f3288115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:47.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f3287400700 1 -- 172.21.15.63:0/1368469855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3288102e00 msgr2=0x7f3288115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:47.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f3287400700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3288102e00 0x7f3288115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.338+0000 7f3287400700 1 -- 172.21.15.63:0/1368469855 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274005040 con 0x7f32881037a0 2024-03-23T12:15:47.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.339+0000 7f3287400700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f3288116260 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3274009390 tx=0x7f3274009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:47.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.339+0000 7f327fe00700 1 -- 172.21.15.63:0/1368469855 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f327400e040 con 0x7f32881037a0 2024-03-23T12:15:47.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.339+0000 7f327fe00700 1 -- 172.21.15.63:0/1368469855 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3274013420 con 0x7f32881037a0 2024-03-23T12:15:47.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.339+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328810ff00 con 0x7f32881037a0 2024-03-23T12:15:47.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.339+0000 7f327fe00700 1 -- 172.21.15.63:0/1368469855 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f327401c630 con 0x7f32881037a0 2024-03-23T12:15:47.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.339+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32881104c0 con 0x7f32881037a0 2024-03-23T12:15:47.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.340+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32880fe550 con 0x7f32881037a0 2024-03-23T12:15:47.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.341+0000 7f327fe00700 1 -- 172.21.15.63:0/1368469855 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3274013a00 con 0x7f32881037a0 2024-03-23T12:15:47.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.342+0000 7f327fe00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f326c06c7d0 0x7f326c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:15:47.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.342+0000 7f327fe00700 1 -- 172.21.15.63:0/1368469855 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f327408cb70 con 0x7f32881037a0 2024-03-23T12:15:47.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.342+0000 7f3287e00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f326c06c7d0 0x7f326c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:15:47.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.343+0000 7f3287e00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f326c06c7d0 0x7f326c06ec80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3278004f40 tx=0x7f327800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:15:47.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.346+0000 7f327fe00700 1 -- 172.21.15.63:0/1368469855 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f327405a880 con 0x7f32881037a0 2024-03-23T12:15:47.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.649+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f328804bed0 con 0x7f32881037a0 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.650+0000 7f327fe00700 1 -- 172.21.15.63:0/1368469855 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f327405a410 con 0x7f32881037a0 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:15:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:15:47.631 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:15:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:15:47.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f326c06c7d0 msgr2=0x7f326c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:47.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f326c06c7d0 0x7f326c06ec80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3278004f40 tx=0x7f327800b040 comp rx=0 tx=0).stop 2024-03-23T12:15:47.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 msgr2=0x7f3288116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f3288116260 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3274009390 tx=0x7f3274009e30 comp rx=0 tx=0).stop 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 shutdown_connections 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f326c06c7d0 0x7f326c06ec80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3288102e00 0x7f3288115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 --2- 172.21.15.63:0/1368469855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32881037a0 0x7f3288116260 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 >> 172.21.15.63:0/1368469855 conn(0x7f32880f89b0 msgr2=0x7f32880f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 shutdown_connections 2024-03-23T12:15:47.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:15:47.652+0000 7f328ea00700 1 -- 172.21.15.63:0/1368469855 wait complete. 2024-03-23T12:15:47.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:47 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1803242808' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:15:47.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:47 smithi063 ceph-mon[141124]: pgmap v741: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:47.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:47 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1803242808' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:15:47.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:47 smithi111 ceph-mon[106899]: pgmap v741: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:48.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:48 smithi063 ceph-mon[141124]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:48.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:48 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1368469855' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:15:48.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:48 smithi111 ceph-mon[106899]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:15:48.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:48 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1368469855' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:15:49.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:49 smithi111 ceph-mon[106899]: pgmap v742: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:50.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:49 smithi063 ceph-mon[141124]: pgmap v742: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:51.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:51 smithi063 ceph-mon[141124]: pgmap v743: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:51.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:51 smithi111 ceph-mon[106899]: pgmap v743: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:53.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:53 smithi063 ceph-mon[141124]: pgmap v744: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:53.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:53 smithi111 ceph-mon[106899]: pgmap v744: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:55.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:55 smithi063 ceph-mon[141124]: pgmap v745: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:55 smithi111 ceph-mon[106899]: pgmap v745: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:56.878 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:56.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:15:57.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:57 smithi111 ceph-mon[106899]: pgmap v746: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:58.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:57 smithi063 ceph-mon[141124]: pgmap v746: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:59.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:15:59 smithi063 ceph-mon[141124]: pgmap v747: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:15:59.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:15:59 smithi111 ceph-mon[106899]: pgmap v747: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:01.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:01 smithi111 ceph-mon[106899]: pgmap v748: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:02.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:01 smithi063 ceph-mon[141124]: pgmap v748: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:03.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:03 smithi063 ceph-mon[141124]: pgmap v749: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:03.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:03 smithi111 ceph-mon[106899]: pgmap v749: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:05.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:05 smithi063 ceph-mon[141124]: pgmap v750: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:05.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:05 smithi111 ceph-mon[106899]: pgmap v750: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:07.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:07 smithi063 ceph-mon[141124]: pgmap v751: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:07.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:07 smithi111 ceph-mon[106899]: pgmap v751: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:09.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:09 smithi063 ceph-mon[141124]: pgmap v752: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:09.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:09 smithi111 ceph-mon[106899]: pgmap v752: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:11.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:11.899 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:11 smithi111 ceph-mon[106899]: pgmap v753: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:11 smithi063 ceph-mon[141124]: pgmap v753: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:13.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:13 smithi063 ceph-mon[141124]: pgmap v754: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:13.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:13 smithi111 ceph-mon[106899]: pgmap v754: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:15.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:15 smithi063 ceph-mon[141124]: pgmap v755: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:15.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:15 smithi111 ceph-mon[106899]: pgmap v755: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:17.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:17 smithi063 ceph-mon[141124]: pgmap v756: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:17.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:17 smithi111 ceph-mon[106899]: pgmap v756: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:17.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 -- 172.21.15.63:0/2660921576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 msgr2=0x7f046c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:17.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 --2- 172.21.15.63:0/2660921576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c0fcc50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0458004940 tx=0x7f0458009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:17.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 -- 172.21.15.63:0/2660921576 shutdown_connections 2024-03-23T12:16:17.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 --2- 172.21.15.63:0/2660921576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c0fcc50 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:17.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 --2- 172.21.15.63:0/2660921576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f046c102c40 0x7f046c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:17.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 -- 172.21.15.63:0/2660921576 >> 172.21.15.63:0/2660921576 conn(0x7f046c0f8830 msgr2=0x7f046c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:17.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 -- 172.21.15.63:0/2660921576 shutdown_connections 2024-03-23T12:16:17.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.813+0000 7f046be00700 1 -- 172.21.15.63:0/2660921576 wait complete. 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.814+0000 7f046be00700 1 Processor -- start 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.814+0000 7f046be00700 1 -- start start 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f046c102c40 0x7f046c1138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c114480 con 0x7f046c1035e0 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c10d870 con 0x7f046c102c40 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c113e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43604/0 (socket says 172.21.15.63:43604) 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046a000700 1 -- 172.21.15.63:0/982553339 learned_addr learned my addr 172.21.15.63:0/982553339 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:17.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046aa00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f046c102c40 0x7f046c1138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046a000700 1 -- 172.21.15.63:0/982553339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f046c102c40 msgr2=0x7f046c1138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046a000700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f046c102c40 0x7f046c1138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046a000700 1 -- 172.21.15.63:0/982553339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0458005040 con 0x7f046c1035e0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.815+0000 7f046aa00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f046c102c40 0x7f046c1138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.816+0000 7f046a000700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c113e30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0458009390 tx=0x7f0458009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.816+0000 7f0462a00700 1 -- 172.21.15.63:0/982553339 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f045800e040 con 0x7f046c1035e0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.816+0000 7f0462a00700 1 -- 172.21.15.63:0/982553339 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0458013420 con 0x7f046c1035e0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.816+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046c10daf0 con 0x7f046c1035e0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.816+0000 7f0462a00700 1 -- 172.21.15.63:0/982553339 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f045801c630 con 0x7f046c1035e0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.816+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c10e0b0 con 0x7f046c1035e0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.817+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c0fe390 con 0x7f046c1035e0 2024-03-23T12:16:17.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.818+0000 7f0462a00700 1 -- 172.21.15.63:0/982553339 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0458013a00 con 0x7f046c1035e0 2024-03-23T12:16:17.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.820+0000 7f0462a00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f045006c7d0 0x7f045006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:17.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.820+0000 7f0462a00700 1 -- 172.21.15.63:0/982553339 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f045808cd00 con 0x7f046c1035e0 2024-03-23T12:16:17.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.820+0000 7f046aa00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f045006c7d0 0x7f045006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:17.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.821+0000 7f046aa00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f045006c7d0 0x7f045006ec80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f045c009fd0 tx=0x7f045c009f20 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:17.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:17.825+0000 7f0462a00700 1 -- 172.21.15.63:0/982553339 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f045805aa10 con 0x7f046c1035e0 2024-03-23T12:16:18.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.056+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f046c109bf0 con 0x7f045006c7d0 2024-03-23T12:16:18.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.058+0000 7f0462a00700 1 -- 172.21.15.63:0/982553339 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f046c109bf0 con 0x7f045006c7d0 2024-03-23T12:16:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f045006c7d0 msgr2=0x7f045006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f045006c7d0 0x7f045006ec80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f045c009fd0 tx=0x7f045c009f20 comp rx=0 tx=0).stop 2024-03-23T12:16:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 msgr2=0x7f046c113e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c113e30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0458009390 tx=0x7f0458009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 shutdown_connections 2024-03-23T12:16:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f045006c7d0 0x7f045006ec80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f046c102c40 0x7f046c1138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 --2- 172.21.15.63:0/982553339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f046c1035e0 0x7f046c113e30 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 >> 172.21.15.63:0/982553339 conn(0x7f046c0f8830 msgr2=0x7f046c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:18.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 shutdown_connections 2024-03-23T12:16:18.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.060+0000 7f046be00700 1 -- 172.21.15.63:0/982553339 wait complete. 2024-03-23T12:16:18.054 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:16:18.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 -- 172.21.15.63:0/2756445000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105090 msgr2=0x7f98c8105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 --2- 172.21.15.63:0/2756445000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105090 0x7f98c8105460 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f98b8004940 tx=0x7f98b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:18.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 -- 172.21.15.63:0/2756445000 shutdown_connections 2024-03-23T12:16:18.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 --2- 172.21.15.63:0/2756445000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98c8105a30 0x7f98c80ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 --2- 172.21.15.63:0/2756445000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105090 0x7f98c8105460 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 -- 172.21.15.63:0/2756445000 >> 172.21.15.63:0/2756445000 conn(0x7f98c80fac00 msgr2=0x7f98c80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:18.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 -- 172.21.15.63:0/2756445000 shutdown_connections 2024-03-23T12:16:18.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.199+0000 7f98cea00700 1 -- 172.21.15.63:0/2756445000 wait complete. 2024-03-23T12:16:18.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.200+0000 7f98cea00700 1 Processor -- start 2024-03-23T12:16:18.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.200+0000 7f98cea00700 1 -- start start 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.200+0000 7f98cea00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98c8105090 0x7f98c8115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.200+0000 7f98cea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105a30 0x7f98c8115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98cea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8116640 con 0x7f98c8105a30 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98cea00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c810fc20 con 0x7f98c8105090 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105a30 0x7f98c8115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105a30 0x7f98c8115ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43626/0 (socket says 172.21.15.63:43626) 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7400700 1 -- 172.21.15.63:0/1142425581 learned_addr learned my addr 172.21.15.63:0/1142425581 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7e00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98c8105090 0x7f98c8115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7400700 1 -- 172.21.15.63:0/1142425581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98c8105090 msgr2=0x7f98c8115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7400700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98c8105090 0x7f98c8115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7400700 1 -- 172.21.15.63:0/1142425581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8005040 con 0x7f98c8105a30 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.201+0000 7f98c7e00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98c8105090 0x7f98c8115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:16:18.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.202+0000 7f98c7400700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105a30 0x7f98c8115ff0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f98b4002a80 tx=0x7f98b4004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:18.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.202+0000 7f98bfe00700 1 -- 172.21.15.63:0/1142425581 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98b4007cc0 con 0x7f98c8105a30 2024-03-23T12:16:18.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.202+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c810ff00 con 0x7f98c8105a30 2024-03-23T12:16:18.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.202+0000 7f98bfe00700 1 -- 172.21.15.63:0/1142425581 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98b400f040 con 0x7f98c8105a30 2024-03-23T12:16:18.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.202+0000 7f98bfe00700 1 -- 172.21.15.63:0/1142425581 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98b400da40 con 0x7f98c8105a30 2024-03-23T12:16:18.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.202+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c81104c0 con 0x7f98c8105a30 2024-03-23T12:16:18.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.203+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c81023a0 con 0x7f98c8105a30 2024-03-23T12:16:18.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.204+0000 7f98bfe00700 1 -- 172.21.15.63:0/1142425581 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f98b400dbe0 con 0x7f98c8105a30 2024-03-23T12:16:18.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.205+0000 7f98bfe00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98ac06c7c0 0x7f98ac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.205+0000 7f98bfe00700 1 -- 172.21.15.63:0/1142425581 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f98b408b270 con 0x7f98c8105a30 2024-03-23T12:16:18.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.205+0000 7f98c7e00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98ac06c7c0 0x7f98ac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.206+0000 7f98c7e00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98ac06c7c0 0x7f98ac06ec70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f98b8000c00 tx=0x7f98b8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:18.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.210+0000 7f98bfe00700 1 -- 172.21.15.63:0/1142425581 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98b4058e80 con 0x7f98c8105a30 2024-03-23T12:16:18.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:18 smithi063 ceph-mon[141124]: from='client.35390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:18.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:18 smithi111 ceph-mon[106899]: from='client.35390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:18.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.446+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98c8111170 con 0x7f98ac06c7c0 2024-03-23T12:16:18.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.447+0000 7f98bfe00700 1 -- 172.21.15.63:0/1142425581 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f98c8111170 con 0x7f98ac06c7c0 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98ac06c7c0 msgr2=0x7f98ac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98ac06c7c0 0x7f98ac06ec70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f98b8000c00 tx=0x7f98b8009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105a30 msgr2=0x7f98c8115ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105a30 0x7f98c8115ff0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f98b4002a80 tx=0x7f98b4004630 comp rx=0 tx=0).stop 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 shutdown_connections 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f98ac06c7c0 0x7f98ac06ec70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98c8105090 0x7f98c8115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 --2- 172.21.15.63:0/1142425581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98c8105a30 0x7f98c8115ff0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 >> 172.21.15.63:0/1142425581 conn(0x7f98c80fac00 msgr2=0x7f98c80fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 shutdown_connections 2024-03-23T12:16:18.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.450+0000 7f98cea00700 1 -- 172.21.15.63:0/1142425581 wait complete. 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 -- 172.21.15.63:0/2926594842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fe830 msgr2=0x7f2320106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 --2- 172.21.15.63:0/2926594842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fe830 0x7f2320106f00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f230c004940 tx=0x7f230c009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 -- 172.21.15.63:0/2926594842 shutdown_connections 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 --2- 172.21.15.63:0/2926594842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fe830 0x7f2320106f00 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 --2- 172.21.15.63:0/2926594842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23200fde90 0x7f23200fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 -- 172.21.15.63:0/2926594842 >> 172.21.15.63:0/2926594842 conn(0x7f2320072bc0 msgr2=0x7f2320072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 -- 172.21.15.63:0/2926594842 shutdown_connections 2024-03-23T12:16:18.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.587+0000 7f2327400700 1 -- 172.21.15.63:0/2926594842 wait complete. 2024-03-23T12:16:18.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.588+0000 7f2327400700 1 Processor -- start 2024-03-23T12:16:18.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.588+0000 7f2327400700 1 -- start start 2024-03-23T12:16:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f2327400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fde90 0x7f2320113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f2327400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23200fe830 0x7f2320114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f2327400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2320114670 con 0x7f23200fde90 2024-03-23T12:16:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f2327400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232010da60 con 0x7f23200fe830 2024-03-23T12:16:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f2324a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fde90 0x7f2320113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f2324a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fde90 0x7f2320113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43642/0 (socket says 172.21.15.63:43642) 2024-03-23T12:16:18.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f2324a00700 1 -- 172.21.15.63:0/4273708140 learned_addr learned my addr 172.21.15.63:0/4273708140 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.589+0000 7f231fe00700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23200fe830 0x7f2320114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f2324a00700 1 -- 172.21.15.63:0/4273708140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23200fe830 msgr2=0x7f2320114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f2324a00700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23200fe830 0x7f2320114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f2324a00700 1 -- 172.21.15.63:0/4273708140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c005040 con 0x7f23200fde90 2024-03-23T12:16:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f2324a00700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fde90 0x7f2320113ae0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2314002a50 tx=0x7f23140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f231cc00700 1 -- 172.21.15.63:0/4273708140 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2314007cc0 con 0x7f23200fde90 2024-03-23T12:16:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f231cc00700 1 -- 172.21.15.63:0/4273708140 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f231400f040 con 0x7f23200fde90 2024-03-23T12:16:18.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232010dd40 con 0x7f23200fde90 2024-03-23T12:16:18.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f231cc00700 1 -- 172.21.15.63:0/4273708140 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2314009540 con 0x7f23200fde90 2024-03-23T12:16:18.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.590+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232010e250 con 0x7f23200fde90 2024-03-23T12:16:18.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.591+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2320065e30 con 0x7f23200fde90 2024-03-23T12:16:18.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.592+0000 7f231cc00700 1 -- 172.21.15.63:0/4273708140 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f23140096a0 con 0x7f23200fde90 2024-03-23T12:16:18.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.594+0000 7f231cc00700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f230806c7d0 0x7f230806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.594+0000 7f231cc00700 1 -- 172.21.15.63:0/4273708140 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f231408b820 con 0x7f23200fde90 2024-03-23T12:16:18.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.594+0000 7f231fe00700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f230806c7d0 0x7f230806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.595+0000 7f231fe00700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f230806c7d0 0x7f230806ec80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f232010ed40 tx=0x7f230c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.598+0000 7f231cc00700 1 -- 172.21.15.63:0/4273708140 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f23140594b0 con 0x7f23200fde90 2024-03-23T12:16:18.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.821+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f232004bed0 con 0x7f230806c7d0 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.831+0000 7f231cc00700 1 -- 172.21.15.63:0/4273708140 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f232004bed0 con 0x7f230806c7d0 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (25m) 2m ago 38m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (22m) 2m ago 38m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (22m) 2m ago 37m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (25m) 2m ago 38m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (18m) 2m ago 32m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (18m) 2m ago 32m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (2m) 2m ago 32m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (18m) 2m ago 32m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:16:18.811 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (27m) 2m ago 40m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (26m) 2m ago 37m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (24m) 2m ago 40m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (23m) 2m ago 37m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (26m) 2m ago 38m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (26m) 2m ago 37m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (22m) 2m ago 36m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (21m) 2m ago 36m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (20m) 2m ago 35m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (20m) 2m ago 35m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (19m) 2m ago 34m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (19m) 2m ago 34m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:16:18.812 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (25m) 2m ago 38m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:16:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f230806c7d0 msgr2=0x7f230806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f230806c7d0 0x7f230806ec80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f232010ed40 tx=0x7f230c009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fde90 msgr2=0x7f2320113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fde90 0x7f2320113ae0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2314002a50 tx=0x7f23140046b0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 shutdown_connections 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f230806c7d0 0x7f230806ec80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23200fde90 0x7f2320113ae0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 --2- 172.21.15.63:0/4273708140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23200fe830 0x7f2320114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.833+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 >> 172.21.15.63:0/4273708140 conn(0x7f2320072bc0 msgr2=0x7f2320103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.834+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 shutdown_connections 2024-03-23T12:16:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.834+0000 7f2327400700 1 -- 172.21.15.63:0/4273708140 wait complete. 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.970+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1867290496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 msgr2=0x7f9eac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.970+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1867290496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac0fce10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9e98004940 tx=0x7f9e98009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.971+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1867290496 shutdown_connections 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.971+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1867290496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac0fce10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.971+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1867290496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eac102e00 0x7f9eac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.971+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1867290496 >> 172.21.15.63:0/1867290496 conn(0x7f9eac0f89b0 msgr2=0x7f9eac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.971+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1867290496 shutdown_connections 2024-03-23T12:16:18.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.971+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1867290496 wait complete. 2024-03-23T12:16:18.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.971+0000 7f9eb0c00700 1 Processor -- start 2024-03-23T12:16:18.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9eb0c00700 1 -- start start 2024-03-23T12:16:18.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9eb0c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eac102e00 0x7f9eac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9eb0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9eb0c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac070d90 con 0x7f9eac1037a0 2024-03-23T12:16:18.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9eb0c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac070ed0 con 0x7f9eac102e00 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9ea9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9ea9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43662/0 (socket says 172.21.15.63:43662) 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9ea9400700 1 -- 172.21.15.63:0/1229576450 learned_addr learned my addr 172.21.15.63:0/1229576450 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.972+0000 7f9ea9e00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eac102e00 0x7f9eac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9ea9400700 1 -- 172.21.15.63:0/1229576450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eac102e00 msgr2=0x7f9eac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9ea9400700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eac102e00 0x7f9eac0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9ea9400700 1 -- 172.21.15.63:0/1229576450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98005040 con 0x7f9eac1037a0 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9ea9400700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac070800 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009390 tx=0x7f9e98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9ea2000700 1 -- 172.21.15.63:0/1229576450 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9800e040 con 0x7f9eac1037a0 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9ea2000700 1 -- 172.21.15.63:0/1229576450 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e98004290 con 0x7f9eac1037a0 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9ea2000700 1 -- 172.21.15.63:0/1229576450 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e98013e70 con 0x7f9eac1037a0 2024-03-23T12:16:18.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac071120 con 0x7f9eac1037a0 2024-03-23T12:16:18.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.973+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac11a230 con 0x7f9eac1037a0 2024-03-23T12:16:18.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.974+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac0fe550 con 0x7f9eac1037a0 2024-03-23T12:16:18.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.975+0000 7f9ea2000700 1 -- 172.21.15.63:0/1229576450 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9e98013690 con 0x7f9eac1037a0 2024-03-23T12:16:18.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.976+0000 7f9ea2000700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9006c7d0 0x7f9e9006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:18.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.976+0000 7f9ea2000700 1 -- 172.21.15.63:0/1229576450 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9e9808cbf0 con 0x7f9eac1037a0 2024-03-23T12:16:18.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.976+0000 7f9ea9e00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9006c7d0 0x7f9e9006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:18.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.977+0000 7f9ea9e00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9006c7d0 0x7f9e9006ec80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c004f40 tx=0x7f9e9c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:18.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:18.980+0000 7f9ea2000700 1 -- 172.21.15.63:0/1229576450 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e98056f20 con 0x7f9eac1037a0 2024-03-23T12:16:19.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.283+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9eac109dd0 con 0x7f9eac1037a0 2024-03-23T12:16:19.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.283+0000 7f9ea2000700 1 -- 172.21.15.63:0/1229576450 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f9e98013420 con 0x7f9eac1037a0 2024-03-23T12:16:19.263 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:16:19.263 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:16:19.263 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:16:19.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:16:19.265 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:16:19.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.285+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9006c7d0 msgr2=0x7f9e9006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.285+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9006c7d0 0x7f9e9006ec80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c004f40 tx=0x7f9e9c00b040 comp rx=0 tx=0).stop 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.285+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 msgr2=0x7f9eac070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.285+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac070800 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009390 tx=0x7f9e98009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.286+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 shutdown_connections 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.286+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9006c7d0 0x7f9e9006ec80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.286+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eac102e00 0x7f9eac0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.286+0000 7f9eb0c00700 1 --2- 172.21.15.63:0/1229576450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eac1037a0 0x7f9eac070800 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.286+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 >> 172.21.15.63:0/1229576450 conn(0x7f9eac0f89b0 msgr2=0x7f9eac0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.286+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 shutdown_connections 2024-03-23T12:16:19.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.286+0000 7f9eb0c00700 1 -- 172.21.15.63:0/1229576450 wait complete. 2024-03-23T12:16:19.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:19 smithi063 ceph-mon[141124]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:19.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:19 smithi063 ceph-mon[141124]: from='client.35398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:19.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:19 smithi063 ceph-mon[141124]: pgmap v757: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:19 smithi111 ceph-mon[106899]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:19 smithi111 ceph-mon[106899]: from='client.35398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:19 smithi111 ceph-mon[106899]: pgmap v757: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 -- 172.21.15.63:0/2921844562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 msgr2=0x7f9088106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 --2- 172.21.15.63:0/2921844562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f9088106f20 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9074004970 tx=0x7f9074009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 -- 172.21.15.63:0/2921844562 shutdown_connections 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 --2- 172.21.15.63:0/2921844562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f9088106f20 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 --2- 172.21.15.63:0/2921844562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90880fdeb0 0x7f90880fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 -- 172.21.15.63:0/2921844562 >> 172.21.15.63:0/2921844562 conn(0x7f9088072bc0 msgr2=0x7f9088072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 -- 172.21.15.63:0/2921844562 shutdown_connections 2024-03-23T12:16:19.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.424+0000 7f9090400700 1 -- 172.21.15.63:0/2921844562 wait complete. 2024-03-23T12:16:19.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.425+0000 7f9090400700 1 Processor -- start 2024-03-23T12:16:19.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.425+0000 7f9090400700 1 -- start start 2024-03-23T12:16:19.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.425+0000 7f9090400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90880fdeb0 0x7f90880fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:19.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.425+0000 7f9090400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f90880fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:19.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.425+0000 7f9090400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90880fc840 con 0x7f90880fe850 2024-03-23T12:16:19.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.425+0000 7f9090400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90880fc980 con 0x7f90880fdeb0 2024-03-23T12:16:19.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f90880fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908da00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90880fdeb0 0x7f90880fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f90880fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43690/0 (socket says 172.21.15.63:43690) 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908d000700 1 -- 172.21.15.63:0/2539123885 learned_addr learned my addr 172.21.15.63:0/2539123885 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908d000700 1 -- 172.21.15.63:0/2539123885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90880fdeb0 msgr2=0x7f90880fbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908d000700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90880fdeb0 0x7f90880fbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908d000700 1 -- 172.21.15.63:0/2539123885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074005040 con 0x7f90880fe850 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.426+0000 7f908d000700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f90880fc2b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9074004f50 tx=0x7f9074009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.427+0000 7f9085600700 1 -- 172.21.15.63:0/2539123885 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f907400e040 con 0x7f90880fe850 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.427+0000 7f9085600700 1 -- 172.21.15.63:0/2539123885 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9074013420 con 0x7f90880fe850 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.427+0000 7f9085600700 1 -- 172.21.15.63:0/2539123885 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f907401c630 con 0x7f90880fe850 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.427+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90880fcbd0 con 0x7f90880fe850 2024-03-23T12:16:19.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.427+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908811a0c0 con 0x7f90880fe850 2024-03-23T12:16:19.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.428+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9088065e10 con 0x7f90880fe850 2024-03-23T12:16:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.429+0000 7f9085600700 1 -- 172.21.15.63:0/2539123885 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9074013590 con 0x7f90880fe850 2024-03-23T12:16:19.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.430+0000 7f9085600700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f907006c8a0 0x7f907006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:19.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.430+0000 7f9085600700 1 -- 172.21.15.63:0/2539123885 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f907408cc40 con 0x7f90880fe850 2024-03-23T12:16:19.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.430+0000 7f908da00700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f907006c8a0 0x7f907006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:19.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.431+0000 7f908da00700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f907006c8a0 0x7f907006ed50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f907c007fd0 tx=0x7f907c009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:19.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.434+0000 7f9085600700 1 -- 172.21.15.63:0/2539123885 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9074056ec0 con 0x7f90880fe850 2024-03-23T12:16:19.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.691+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f908804bed0 con 0x7f90880fe850 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.692+0000 7f9085600700 1 -- 172.21.15.63:0/2539123885 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f907405a4e0 con 0x7f90880fe850 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:16:19.672 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:16:19.673 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f907006c8a0 msgr2=0x7f907006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f907006c8a0 0x7f907006ed50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f907c007fd0 tx=0x7f907c009040 comp rx=0 tx=0).stop 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 msgr2=0x7f90880fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f90880fc2b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9074004f50 tx=0x7f9074009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 shutdown_connections 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f907006c8a0 0x7f907006ed50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90880fdeb0 0x7f90880fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 --2- 172.21.15.63:0/2539123885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90880fe850 0x7f90880fc2b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.694+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 >> 172.21.15.63:0/2539123885 conn(0x7f9088072bc0 msgr2=0x7f9088103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:19.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.695+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 shutdown_connections 2024-03-23T12:16:19.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.695+0000 7f9090400700 1 -- 172.21.15.63:0/2539123885 wait complete. 2024-03-23T12:16:19.676 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:16:19.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 -- 172.21.15.63:0/2377617524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 msgr2=0x7f93cc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 --2- 172.21.15.63:0/2377617524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc1031d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f93bc004940 tx=0x7f93bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 -- 172.21.15.63:0/2377617524 shutdown_connections 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 --2- 172.21.15.63:0/2377617524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 --2- 172.21.15.63:0/2377617524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc1031d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 -- 172.21.15.63:0/2377617524 >> 172.21.15.63:0/2377617524 conn(0x7f93cc0f89b0 msgr2=0x7f93cc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 -- 172.21.15.63:0/2377617524 shutdown_connections 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.833+0000 7f93d2400700 1 -- 172.21.15.63:0/2377617524 wait complete. 2024-03-23T12:16:19.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.834+0000 7f93d2400700 1 Processor -- start 2024-03-23T12:16:19.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.834+0000 7f93d2400700 1 -- start start 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93d2400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93d2400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93d2400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc116860 con 0x7f93cc102e00 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93d2400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc10fc30 con 0x7f93cc1037a0 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93cb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93cb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43708/0 (socket says 172.21.15.63:43708) 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93cb400700 1 -- 172.21.15.63:0/3650942620 learned_addr learned my addr 172.21.15.63:0/3650942620 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93c3e00700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93cb400700 1 -- 172.21.15.63:0/3650942620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 msgr2=0x7f93cc116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93cb400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.835+0000 7f93cb400700 1 -- 172.21.15.63:0/3650942620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc005040 con 0x7f93cc102e00 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.836+0000 7f93cb400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009390 tx=0x7f93bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.836+0000 7f93c3400700 1 -- 172.21.15.63:0/3650942620 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93bc00e040 con 0x7f93cc102e00 2024-03-23T12:16:19.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.836+0000 7f93c3400700 1 -- 172.21.15.63:0/3650942620 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93bc013420 con 0x7f93cc102e00 2024-03-23T12:16:19.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.836+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc10feb0 con 0x7f93cc102e00 2024-03-23T12:16:19.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.836+0000 7f93c3400700 1 -- 172.21.15.63:0/3650942620 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93bc01c630 con 0x7f93cc102e00 2024-03-23T12:16:19.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.836+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc110470 con 0x7f93cc102e00 2024-03-23T12:16:19.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.837+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc0fe550 con 0x7f93cc102e00 2024-03-23T12:16:19.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.838+0000 7f93c3400700 1 -- 172.21.15.63:0/3650942620 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f93bc0139e0 con 0x7f93cc102e00 2024-03-23T12:16:19.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.839+0000 7f93c3400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:19.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.839+0000 7f93c3400700 1 -- 172.21.15.63:0/3650942620 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f93bc08cd30 con 0x7f93cc102e00 2024-03-23T12:16:19.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.839+0000 7f93c3e00700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:19.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.840+0000 7f93c3e00700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f93cc111570 tx=0x7f93b4009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:19.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:19.843+0000 7f93c3400700 1 -- 172.21.15.63:0/3650942620 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f93bc05aaf0 con 0x7f93cc102e00 2024-03-23T12:16:20.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.075+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f93cc04bed0 con 0x7f93b006c7d0 2024-03-23T12:16:20.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.076+0000 7f93c3400700 1 -- 172.21.15.63:0/3650942620 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f93cc04bed0 con 0x7f93b006c7d0 2024-03-23T12:16:20.056 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:16:20.056 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:16:20.056 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:16:20.056 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:16:20.056 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:16:20.056 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:16:20.057 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 msgr2=0x7f93b006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f93cc111570 tx=0x7f93b4009ee0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 msgr2=0x7f93cc115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009390 tx=0x7f93bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 shutdown_connections 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f93b006c7d0 0x7f93b006ec80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93cc102e00 0x7f93cc115cd0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 --2- 172.21.15.63:0/3650942620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93cc1037a0 0x7f93cc116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 >> 172.21.15.63:0/3650942620 conn(0x7f93cc0f89b0 msgr2=0x7f93cc0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 shutdown_connections 2024-03-23T12:16:20.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.079+0000 7f93d2400700 1 -- 172.21.15.63:0/3650942620 wait complete. 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 -- 172.21.15.63:0/1042645755 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 msgr2=0x7f0ab41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 --2- 172.21.15.63:0/1042645755 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab41031d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4004940 tx=0x7f0aa4009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 -- 172.21.15.63:0/1042645755 shutdown_connections 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 --2- 172.21.15.63:0/1042645755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ab41037a0 0x7f0ab40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 --2- 172.21.15.63:0/1042645755 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab41031d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 -- 172.21.15.63:0/1042645755 >> 172.21.15.63:0/1042645755 conn(0x7f0ab40f89b0 msgr2=0x7f0ab40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 -- 172.21.15.63:0/1042645755 shutdown_connections 2024-03-23T12:16:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.219+0000 7f0abac00700 1 -- 172.21.15.63:0/1042645755 wait complete. 2024-03-23T12:16:20.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.220+0000 7f0abac00700 1 Processor -- start 2024-03-23T12:16:20.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.220+0000 7f0abac00700 1 -- start start 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.220+0000 7f0abac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab4113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.220+0000 7f0abac00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ab41037a0 0x7f0ab4113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.220+0000 7f0abac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab41145d0 con 0x7f0ab4102e00 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.220+0000 7f0abac00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab410d9c0 con 0x7f0ab41037a0 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab4113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab4113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43732/0 (socket says 172.21.15.63:43732) 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3e00700 1 -- 172.21.15.63:0/350475881 learned_addr learned my addr 172.21.15.63:0/350475881 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3400700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ab41037a0 0x7f0ab4113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3e00700 1 -- 172.21.15.63:0/350475881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ab41037a0 msgr2=0x7f0ab4113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3e00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ab41037a0 0x7f0ab4113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3e00700 1 -- 172.21.15.63:0/350475881 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4005040 con 0x7f0ab4102e00 2024-03-23T12:16:20.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3400700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ab41037a0 0x7f0ab4113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:16:20.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0ab3e00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab4113a40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009390 tx=0x7f0aa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:20.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0aabe00700 1 -- 172.21.15.63:0/350475881 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa400e040 con 0x7f0ab4102e00 2024-03-23T12:16:20.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0aabe00700 1 -- 172.21.15.63:0/350475881 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aa4013420 con 0x7f0ab4102e00 2024-03-23T12:16:20.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0aabe00700 1 -- 172.21.15.63:0/350475881 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa401c630 con 0x7f0ab4102e00 2024-03-23T12:16:20.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.221+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab410dc40 con 0x7f0ab4102e00 2024-03-23T12:16:20.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.222+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab410e200 con 0x7f0ab4102e00 2024-03-23T12:16:20.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.223+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab40fe550 con 0x7f0ab4102e00 2024-03-23T12:16:20.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.224+0000 7f0aabe00700 1 -- 172.21.15.63:0/350475881 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0aa40139e0 con 0x7f0ab4102e00 2024-03-23T12:16:20.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.224+0000 7f0aabe00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0a9806c7d0 0x7f0a9806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:20.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.225+0000 7f0aabe00700 1 -- 172.21.15.63:0/350475881 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0aa408cd30 con 0x7f0ab4102e00 2024-03-23T12:16:20.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.225+0000 7f0ab3400700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0a9806c7d0 0x7f0a9806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:20.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.225+0000 7f0ab3400700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0a9806c7d0 0x7f0a9806ec80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007fd0 tx=0x7f0aa0009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:20.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.229+0000 7f0aabe00700 1 -- 172.21.15.63:0/350475881 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0aa405aaf0 con 0x7f0ab4102e00 2024-03-23T12:16:20.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:20 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1229576450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:16:20.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:20 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2539123885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:16:20.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:20 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1229576450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:16:20.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:20 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2539123885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:16:20.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.528+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0ab4109dc0 con 0x7f0ab4102e00 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.528+0000 7f0aabe00700 1 -- 172.21.15.63:0/350475881 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f0aa405a680 con 0x7f0ab4102e00 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:16:20.508 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:16:20.509 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:16:20.510 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:16:20.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0a9806c7d0 msgr2=0x7f0a9806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:20.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0a9806c7d0 0x7f0a9806ec80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007fd0 tx=0x7f0aa0009f60 comp rx=0 tx=0).stop 2024-03-23T12:16:20.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 msgr2=0x7f0ab4113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:20.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab4113a40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009390 tx=0x7f0aa4009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:20.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 shutdown_connections 2024-03-23T12:16:20.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0a9806c7d0 0x7f0a9806ec80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ab4102e00 0x7f0ab4113a40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 --2- 172.21.15.63:0/350475881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ab41037a0 0x7f0ab4113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:20.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 >> 172.21.15.63:0/350475881 conn(0x7f0ab40f89b0 msgr2=0x7f0ab40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:20.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 shutdown_connections 2024-03-23T12:16:20.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:20.531+0000 7f0abac00700 1 -- 172.21.15.63:0/350475881 wait complete. 2024-03-23T12:16:21.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:21 smithi063 ceph-mon[141124]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:21.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:21 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/350475881' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:16:21.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:21 smithi063 ceph-mon[141124]: pgmap v758: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:21.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:21 smithi111 ceph-mon[106899]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:21.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:21 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/350475881' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:16:21.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:21 smithi111 ceph-mon[106899]: pgmap v758: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:23.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:23 smithi063 ceph-mon[141124]: pgmap v759: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:23.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:23 smithi111 ceph-mon[106899]: pgmap v759: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:25 smithi063 ceph-mon[141124]: pgmap v760: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:25 smithi111 ceph-mon[106899]: pgmap v760: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:26.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:27.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:27.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:27 smithi111 ceph-mon[106899]: pgmap v761: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:28.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:27 smithi063 ceph-mon[141124]: pgmap v761: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:29.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:29 smithi063 ceph-mon[141124]: pgmap v762: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:29.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:29 smithi111 ceph-mon[106899]: pgmap v762: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:31.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:31 smithi111 ceph-mon[106899]: pgmap v763: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:32.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:31 smithi063 ceph-mon[141124]: pgmap v763: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:33.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:33 smithi063 ceph-mon[141124]: pgmap v764: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:33.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:33 smithi111 ceph-mon[106899]: pgmap v764: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:35.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:35 smithi063 ceph-mon[141124]: pgmap v765: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:35.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:35 smithi111 ceph-mon[106899]: pgmap v765: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:37.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:37 smithi063 ceph-mon[141124]: pgmap v766: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:37.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:37 smithi111 ceph-mon[106899]: pgmap v766: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:39.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:39 smithi063 ceph-mon[141124]: pgmap v767: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:39.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:39 smithi111 ceph-mon[106899]: pgmap v767: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:41.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:41 smithi111 ceph-mon[106899]: pgmap v768: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:41 smithi063 ceph-mon[141124]: pgmap v768: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:43.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:43 smithi063 ceph-mon[141124]: pgmap v769: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:43.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:43 smithi111 ceph-mon[106899]: pgmap v769: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:45.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:45 smithi063 ceph-mon[141124]: pgmap v770: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:45.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:45 smithi111 ceph-mon[106899]: pgmap v770: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:47.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:47 smithi063 ceph-mon[141124]: pgmap v771: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:47.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:47 smithi111 ceph-mon[106899]: pgmap v771: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:49.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:49 smithi063 ceph-mon[141124]: pgmap v772: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:49.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:49 smithi111 ceph-mon[106899]: pgmap v772: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.685+0000 7faecbc00700 1 -- 172.21.15.63:0/3832818659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 msgr2=0x7faec4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.685+0000 7faecbc00700 1 --2- 172.21.15.63:0/3832818659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4106f20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7faeb0004970 tx=0x7faeb0009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.685+0000 7faecbc00700 1 -- 172.21.15.63:0/3832818659 shutdown_connections 2024-03-23T12:16:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.685+0000 7faecbc00700 1 --2- 172.21.15.63:0/3832818659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4106f20 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.685+0000 7faecbc00700 1 --2- 172.21.15.63:0/3832818659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faec40fdeb0 0x7faec40fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.685+0000 7faecbc00700 1 -- 172.21.15.63:0/3832818659 >> 172.21.15.63:0/3832818659 conn(0x7faec4072bc0 msgr2=0x7faec4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.685+0000 7faecbc00700 1 -- 172.21.15.63:0/3832818659 shutdown_connections 2024-03-23T12:16:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.686+0000 7faecbc00700 1 -- 172.21.15.63:0/3832818659 wait complete. 2024-03-23T12:16:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.686+0000 7faecbc00700 1 Processor -- start 2024-03-23T12:16:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.686+0000 7faecbc00700 1 -- start start 2024-03-23T12:16:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faecbc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faec40fdeb0 0x7faec4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faecbc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faecbc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec4116790 con 0x7faec40fe850 2024-03-23T12:16:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faecbc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec410fb80 con 0x7faec40fdeb0 2024-03-23T12:16:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faec3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faec3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35592/0 (socket says 172.21.15.63:35592) 2024-03-23T12:16:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faec3e00700 1 -- 172.21.15.63:0/278255240 learned_addr learned my addr 172.21.15.63:0/278255240 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.687+0000 7faec9200700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faec40fdeb0 0x7faec4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec3e00700 1 -- 172.21.15.63:0/278255240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faec40fdeb0 msgr2=0x7faec4115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec3e00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faec40fdeb0 0x7faec4115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec3e00700 1 -- 172.21.15.63:0/278255240 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb0005040 con 0x7faec40fe850 2024-03-23T12:16:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec9200700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faec40fdeb0 0x7faec4115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:16:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec3e00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4116140 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7faeb0005010 tx=0x7faeb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec0c00700 1 -- 172.21.15.63:0/278255240 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb000e040 con 0x7faec40fe850 2024-03-23T12:16:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec0c00700 1 -- 172.21.15.63:0/278255240 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faeb0004290 con 0x7faec40fe850 2024-03-23T12:16:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec410fe00 con 0x7faec40fe850 2024-03-23T12:16:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faec0c00700 1 -- 172.21.15.63:0/278255240 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb0013e40 con 0x7faec40fe850 2024-03-23T12:16:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.688+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec4110340 con 0x7faec40fe850 2024-03-23T12:16:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.689+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec41069f0 con 0x7faec40fe850 2024-03-23T12:16:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.690+0000 7faec0c00700 1 -- 172.21.15.63:0/278255240 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7faeb0013620 con 0x7faec40fe850 2024-03-23T12:16:50.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.691+0000 7faec0c00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faeac06c760 0x7faeac06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:50.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.691+0000 7faec0c00700 1 -- 172.21.15.63:0/278255240 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faeb008cda0 con 0x7faec40fe850 2024-03-23T12:16:50.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.691+0000 7faec9200700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faeac06c760 0x7faeac06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:50.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.692+0000 7faec9200700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faeac06c760 0x7faeac06ec10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004600 tx=0x7faeb800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.696+0000 7faec0c00700 1 -- 172.21.15.63:0/278255240 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faeb005aa30 con 0x7faec40fe850 2024-03-23T12:16:50.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.935+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faec41111e0 con 0x7faeac06c760 2024-03-23T12:16:50.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.936+0000 7faec0c00700 1 -- 172.21.15.63:0/278255240 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faec41111e0 con 0x7faeac06c760 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faeac06c760 msgr2=0x7faeac06ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faeac06c760 0x7faeac06ec10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004600 tx=0x7faeb800b040 comp rx=0 tx=0).stop 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 msgr2=0x7faec4116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4116140 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7faeb0005010 tx=0x7faeb0009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 shutdown_connections 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faeac06c760 0x7faeac06ec10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faec40fdeb0 0x7faec4115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 --2- 172.21.15.63:0/278255240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faec40fe850 0x7faec4116140 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 >> 172.21.15.63:0/278255240 conn(0x7faec4072bc0 msgr2=0x7faec40fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 shutdown_connections 2024-03-23T12:16:50.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:50.939+0000 7faecbc00700 1 -- 172.21.15.63:0/278255240 wait complete. 2024-03-23T12:16:50.932 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 -- 172.21.15.63:0/831602714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 msgr2=0x7fa5801031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 --2- 172.21.15.63:0/831602714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa5801031d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa574004940 tx=0x7fa574009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 -- 172.21.15.63:0/831602714 shutdown_connections 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 --2- 172.21.15.63:0/831602714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5801037a0 0x7fa5800fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 --2- 172.21.15.63:0/831602714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa5801031d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 -- 172.21.15.63:0/831602714 >> 172.21.15.63:0/831602714 conn(0x7fa5800f89b0 msgr2=0x7fa5800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 -- 172.21.15.63:0/831602714 shutdown_connections 2024-03-23T12:16:51.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.077+0000 7fa588400700 1 -- 172.21.15.63:0/831602714 wait complete. 2024-03-23T12:16:51.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.078+0000 7fa588400700 1 Processor -- start 2024-03-23T12:16:51.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.078+0000 7fa588400700 1 -- start start 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa588400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa580115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa588400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5801037a0 0x7fa580116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa588400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa580116880 con 0x7fa580102e00 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa588400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58010fc50 con 0x7fa5801037a0 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa585a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa580115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa585a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa580115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35612/0 (socket says 172.21.15.63:35612) 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa585a00700 1 -- 172.21.15.63:0/1788578152 learned_addr learned my addr 172.21.15.63:0/1788578152 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa585000700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5801037a0 0x7fa580116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa585a00700 1 -- 172.21.15.63:0/1788578152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5801037a0 msgr2=0x7fa580116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa585a00700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5801037a0 0x7fa580116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.079+0000 7fa585a00700 1 -- 172.21.15.63:0/1788578152 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa574005040 con 0x7fa580102e00 2024-03-23T12:16:51.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.080+0000 7fa585a00700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa580115cf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa574009360 tx=0x7fa574009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:51.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.080+0000 7fa57d600700 1 -- 172.21.15.63:0/1788578152 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa57400e040 con 0x7fa580102e00 2024-03-23T12:16:51.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.080+0000 7fa57d600700 1 -- 172.21.15.63:0/1788578152 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa574013420 con 0x7fa580102e00 2024-03-23T12:16:51.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.080+0000 7fa57d600700 1 -- 172.21.15.63:0/1788578152 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa57401c5f0 con 0x7fa580102e00 2024-03-23T12:16:51.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.080+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58010fed0 con 0x7fa580102e00 2024-03-23T12:16:51.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.080+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa580110490 con 0x7fa580102e00 2024-03-23T12:16:51.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.081+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5800fe550 con 0x7fa580102e00 2024-03-23T12:16:51.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.082+0000 7fa57d600700 1 -- 172.21.15.63:0/1788578152 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa574013590 con 0x7fa580102e00 2024-03-23T12:16:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.083+0000 7fa57d600700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa56806c7d0 0x7fa56806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.083+0000 7fa57d600700 1 -- 172.21.15.63:0/1788578152 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa57408cbc0 con 0x7fa580102e00 2024-03-23T12:16:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.083+0000 7fa585000700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa56806c7d0 0x7fa56806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.084+0000 7fa585000700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa56806c7d0 0x7fa56806ec80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa580111590 tx=0x7fa56c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:51.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.087+0000 7fa57d600700 1 -- 172.21.15.63:0/1788578152 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa57405a990 con 0x7fa580102e00 2024-03-23T12:16:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.324+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa58004bed0 con 0x7fa56806c7d0 2024-03-23T12:16:51.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.325+0000 7fa57d600700 1 -- 172.21.15.63:0/1788578152 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa58004bed0 con 0x7fa56806c7d0 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa56806c7d0 msgr2=0x7fa56806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa56806c7d0 0x7fa56806ec80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa580111590 tx=0x7fa56c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 msgr2=0x7fa580115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa580115cf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa574009360 tx=0x7fa574009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 shutdown_connections 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa56806c7d0 0x7fa56806ec80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa580102e00 0x7fa580115cf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 --2- 172.21.15.63:0/1788578152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5801037a0 0x7fa580116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 >> 172.21.15.63:0/1788578152 conn(0x7fa5800f89b0 msgr2=0x7fa5800706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 shutdown_connections 2024-03-23T12:16:51.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.328+0000 7fa588400700 1 -- 172.21.15.63:0/1788578152 wait complete. 2024-03-23T12:16:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 -- 172.21.15.63:0/3807653724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 msgr2=0x7f07d40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 --2- 172.21.15.63:0/3807653724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d40fce10 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f07c0004940 tx=0x7f07c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 -- 172.21.15.63:0/3807653724 shutdown_connections 2024-03-23T12:16:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 --2- 172.21.15.63:0/3807653724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d40fce10 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 --2- 172.21.15.63:0/3807653724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07d4102e00 0x7f07d41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 -- 172.21.15.63:0/3807653724 >> 172.21.15.63:0/3807653724 conn(0x7f07d40f89b0 msgr2=0x7f07d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 -- 172.21.15.63:0/3807653724 shutdown_connections 2024-03-23T12:16:51.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.466+0000 7f07da400700 1 -- 172.21.15.63:0/3807653724 wait complete. 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.467+0000 7f07da400700 1 Processor -- start 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.467+0000 7f07da400700 1 -- start start 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.467+0000 7f07da400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07d4102e00 0x7f07d4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.467+0000 7f07da400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.467+0000 7f07da400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4116810 con 0x7f07d41037a0 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.467+0000 7f07da400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d410fbe0 con 0x7f07d4102e00 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d41161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56866/0 (socket says 172.21.15.63:56866) 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d2a00700 1 -- 172.21.15.63:0/1949307057 learned_addr learned my addr 172.21.15.63:0/1949307057 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d3400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07d4102e00 0x7f07d4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d2a00700 1 -- 172.21.15.63:0/1949307057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07d4102e00 msgr2=0x7f07d4115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d2a00700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07d4102e00 0x7f07d4115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d2a00700 1 -- 172.21.15.63:0/1949307057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0005040 con 0x7f07d41037a0 2024-03-23T12:16:51.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.468+0000 7f07d2a00700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d41161c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f07c0009fd0 tx=0x7f07c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:51.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.469+0000 7f07cb400700 1 -- 172.21.15.63:0/1949307057 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07c000e030 con 0x7f07d41037a0 2024-03-23T12:16:51.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.469+0000 7f07cb400700 1 -- 172.21.15.63:0/1949307057 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07c00093b0 con 0x7f07d41037a0 2024-03-23T12:16:51.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.469+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d410fe60 con 0x7f07d41037a0 2024-03-23T12:16:51.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.469+0000 7f07cb400700 1 -- 172.21.15.63:0/1949307057 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07c001cc80 con 0x7f07d41037a0 2024-03-23T12:16:51.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.469+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d4110420 con 0x7f07d41037a0 2024-03-23T12:16:51.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.470+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d40fe550 con 0x7f07d41037a0 2024-03-23T12:16:51.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.471+0000 7f07cb400700 1 -- 172.21.15.63:0/1949307057 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f07c001c400 con 0x7f07d41037a0 2024-03-23T12:16:51.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.472+0000 7f07cb400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07b806c7d0 0x7f07b806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.472+0000 7f07cb400700 1 -- 172.21.15.63:0/1949307057 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f07c008d200 con 0x7f07d41037a0 2024-03-23T12:16:51.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.472+0000 7f07d3400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07b806c7d0 0x7f07b806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.473+0000 7f07d3400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07b806c7d0 0x7f07b806ec80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009b80 tx=0x7f07c400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:51.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.476+0000 7f07cb400700 1 -- 172.21.15.63:0/1949307057 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07c005af10 con 0x7f07d41037a0 2024-03-23T12:16:51.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.703+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f07d4109e10 con 0x7f07b806c7d0 2024-03-23T12:16:51.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.713+0000 7f07cb400700 1 -- 172.21.15.63:0/1949307057 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f07d4109e10 con 0x7f07b806c7d0 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (26m) 3m ago 39m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (23m) 3m ago 39m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (23m) 3m ago 38m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (25m) 3m ago 38m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (19m) 3m ago 32m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (18m) 3m ago 32m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (3m) 3m ago 32m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (18m) 3m ago 32m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (28m) 3m ago 40m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (27m) 3m ago 38m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:16:51.692 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (24m) 3m ago 40m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (24m) 3m ago 38m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (26m) 3m ago 39m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (26m) 3m ago 38m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (22m) 3m ago 37m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (22m) 3m ago 36m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (21m) 3m ago 36m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (20m) 3m ago 35m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (20m) 3m ago 35m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (19m) 3m ago 34m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:16:51.693 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (26m) 3m ago 38m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07b806c7d0 msgr2=0x7f07b806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07b806c7d0 0x7f07b806ec80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009b80 tx=0x7f07c400b040 comp rx=0 tx=0).stop 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 msgr2=0x7f07d41161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d41161c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f07c0009fd0 tx=0x7f07c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 shutdown_connections 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07b806c7d0 0x7f07b806ec80 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07d4102e00 0x7f07d4115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 --2- 172.21.15.63:0/1949307057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d41037a0 0x7f07d41161c0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.715+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 >> 172.21.15.63:0/1949307057 conn(0x7f07d40f89b0 msgr2=0x7f07d40f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:51.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.716+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 shutdown_connections 2024-03-23T12:16:51.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.716+0000 7f07da400700 1 -- 172.21.15.63:0/1949307057 wait complete. 2024-03-23T12:16:51.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3451556574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88102e00 msgr2=0x7f3f881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3451556574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88102e00 0x7f3f881031d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c004940 tx=0x7f3f7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:51.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3451556574 shutdown_connections 2024-03-23T12:16:51.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3451556574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f881037a0 0x7f3f880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3451556574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88102e00 0x7f3f881031d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3451556574 >> 172.21.15.63:0/3451556574 conn(0x7f3f880f89b0 msgr2=0x7f3f880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:51.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3451556574 shutdown_connections 2024-03-23T12:16:51.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.852+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3451556574 wait complete. 2024-03-23T12:16:51.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.853+0000 7f3f8fc00700 1 Processor -- start 2024-03-23T12:16:51.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.853+0000 7f3f8fc00700 1 -- start start 2024-03-23T12:16:51.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.853+0000 7f3f8fc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f88102e00 0x7f3f88115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.853+0000 7f3f8fc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f881037a0 0x7f3f88116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.853+0000 7f3f8fc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f88116880 con 0x7f3f881037a0 2024-03-23T12:16:51.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.853+0000 7f3f8fc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8810fc50 con 0x7f3f88102e00 2024-03-23T12:16:51.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f8d200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f88102e00 0x7f3f88115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f87e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f881037a0 0x7f3f88116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f8d200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f88102e00 0x7f3f88115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:43304/0 (socket says 172.21.15.63:43304) 2024-03-23T12:16:51.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f8d200700 1 -- 172.21.15.63:0/3946162723 learned_addr learned my addr 172.21.15.63:0/3946162723 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f87e00700 1 -- 172.21.15.63:0/3946162723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f88102e00 msgr2=0x7f3f88115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f87e00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f88102e00 0x7f3f88115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f87e00700 1 -- 172.21.15.63:0/3946162723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c005040 con 0x7f3f881037a0 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f8d200700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f88102e00 0x7f3f88115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.854+0000 7f3f87e00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f881037a0 0x7f3f88116230 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3f74002840 tx=0x7f3f74004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.855+0000 7f3f84c00700 1 -- 172.21.15.63:0/3946162723 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f74007cc0 con 0x7f3f881037a0 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.855+0000 7f3f84c00700 1 -- 172.21.15.63:0/3946162723 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f7400f040 con 0x7f3f881037a0 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.855+0000 7f3f84c00700 1 -- 172.21.15.63:0/3946162723 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f74013910 con 0x7f3f881037a0 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.855+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8810ff30 con 0x7f3f881037a0 2024-03-23T12:16:51.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.855+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f881104f0 con 0x7f3f881037a0 2024-03-23T12:16:51.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.856+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f880fe550 con 0x7f3f881037a0 2024-03-23T12:16:51.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.857+0000 7f3f84c00700 1 -- 172.21.15.63:0/3946162723 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3f7401c400 con 0x7f3f881037a0 2024-03-23T12:16:51.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.858+0000 7f3f84c00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f7006c7d0 0x7f3f7006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:51.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.858+0000 7f3f84c00700 1 -- 172.21.15.63:0/3946162723 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f7408b060 con 0x7f3f881037a0 2024-03-23T12:16:51.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.858+0000 7f3f8d200700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f7006c7d0 0x7f3f7006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:51.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.859+0000 7f3f8d200700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f7006c7d0 0x7f3f7006ec80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c000c00 tx=0x7f3f7c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:51.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:51.862+0000 7f3f84c00700 1 -- 172.21.15.63:0/3946162723 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f74058cf0 con 0x7f3f881037a0 2024-03-23T12:16:51.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:51 smithi111 ceph-mon[106899]: from='client.35418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:51.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:51 smithi111 ceph-mon[106899]: pgmap v773: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:52.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:51 smithi063 ceph-mon[141124]: from='client.35418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:52.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:51 smithi063 ceph-mon[141124]: pgmap v773: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:52.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.158+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f88109e10 con 0x7f3f881037a0 2024-03-23T12:16:52.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.159+0000 7f3f84c00700 1 -- 172.21.15.63:0/3946162723 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f3f74058880 con 0x7f3f881037a0 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:16:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:16:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:16:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:16:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:16:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:16:52.139 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:16:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f7006c7d0 msgr2=0x7f3f7006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f7006c7d0 0x7f3f7006ec80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c000c00 tx=0x7f3f7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f881037a0 msgr2=0x7f3f88116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f881037a0 0x7f3f88116230 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3f74002840 tx=0x7f3f74004570 comp rx=0 tx=0).stop 2024-03-23T12:16:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 shutdown_connections 2024-03-23T12:16:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3f7006c7d0 0x7f3f7006ec80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f88102e00 0x7f3f88115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 --2- 172.21.15.63:0/3946162723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f881037a0 0x7f3f88116230 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 >> 172.21.15.63:0/3946162723 conn(0x7f3f880f89b0 msgr2=0x7f3f880f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:52.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.161+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 shutdown_connections 2024-03-23T12:16:52.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.162+0000 7f3f8fc00700 1 -- 172.21.15.63:0/3946162723 wait complete. 2024-03-23T12:16:52.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.299+0000 7f088e200700 1 -- 172.21.15.63:0/1397791188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 msgr2=0x7f08881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.299+0000 7f088e200700 1 --2- 172.21.15.63:0/1397791188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f08881031d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f0878004940 tx=0x7f0878009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:52.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.299+0000 7f088e200700 1 -- 172.21.15.63:0/1397791188 shutdown_connections 2024-03-23T12:16:52.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.299+0000 7f088e200700 1 --2- 172.21.15.63:0/1397791188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08881037a0 0x7f08880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.299+0000 7f088e200700 1 --2- 172.21.15.63:0/1397791188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f08881031d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.299+0000 7f088e200700 1 -- 172.21.15.63:0/1397791188 >> 172.21.15.63:0/1397791188 conn(0x7f08880f89b0 msgr2=0x7f08880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:52.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.300+0000 7f088e200700 1 -- 172.21.15.63:0/1397791188 shutdown_connections 2024-03-23T12:16:52.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.300+0000 7f088e200700 1 -- 172.21.15.63:0/1397791188 wait complete. 2024-03-23T12:16:52.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.300+0000 7f088e200700 1 Processor -- start 2024-03-23T12:16:52.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.300+0000 7f088e200700 1 -- start start 2024-03-23T12:16:52.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f088e200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f0888115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:52.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f088e200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08881037a0 0x7f0888116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:52.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f088e200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08881167e0 con 0x7f0888102e00 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f088e200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088810fbb0 con 0x7f08881037a0 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f0887400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f0888115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f0887400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f0888115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56904/0 (socket says 172.21.15.63:56904) 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f0887400700 1 -- 172.21.15.63:0/316214980 learned_addr learned my addr 172.21.15.63:0/316214980 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.301+0000 7f0886a00700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08881037a0 0x7f0888116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f0887400700 1 -- 172.21.15.63:0/316214980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08881037a0 msgr2=0x7f0888116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f0887400700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08881037a0 0x7f0888116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f0887400700 1 -- 172.21.15.63:0/316214980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878005040 con 0x7f0888102e00 2024-03-23T12:16:52.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f0887400700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f0888115c50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0878009fd0 tx=0x7f0878009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:52.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f087f400700 1 -- 172.21.15.63:0/316214980 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0878021070 con 0x7f0888102e00 2024-03-23T12:16:52.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f087f400700 1 -- 172.21.15.63:0/316214980 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0878013e00 con 0x7f0888102e00 2024-03-23T12:16:52.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088810fe30 con 0x7f0888102e00 2024-03-23T12:16:52.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f087f400700 1 -- 172.21.15.63:0/316214980 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f087800e070 con 0x7f0888102e00 2024-03-23T12:16:52.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.302+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0888110390 con 0x7f0888102e00 2024-03-23T12:16:52.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.303+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08880fe550 con 0x7f0888102e00 2024-03-23T12:16:52.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.304+0000 7f087f400700 1 -- 172.21.15.63:0/316214980 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f087801c500 con 0x7f0888102e00 2024-03-23T12:16:52.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.305+0000 7f087f400700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f086c06c7c0 0x7f086c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:52.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.305+0000 7f087f400700 1 -- 172.21.15.63:0/316214980 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f087808cac0 con 0x7f0888102e00 2024-03-23T12:16:52.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.306+0000 7f0886a00700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f086c06c7c0 0x7f086c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:52.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.306+0000 7f0886a00700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f086c06c7c0 0x7f086c06ec70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f08881114f0 tx=0x7f087400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:52.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.310+0000 7f087f400700 1 -- 172.21.15.63:0/316214980 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f087805a7d0 con 0x7f0888102e00 2024-03-23T12:16:52.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.569+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f08880fe550 con 0x7f0888102e00 2024-03-23T12:16:52.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.570+0000 7f087f400700 1 -- 172.21.15.63:0/316214980 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f087805a360 con 0x7f0888102e00 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:16:52.549 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:16:52.550 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:52.551 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:52.552 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:16:52.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f086c06c7c0 msgr2=0x7f086c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f086c06c7c0 0x7f086c06ec70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f08881114f0 tx=0x7f087400b040 comp rx=0 tx=0).stop 2024-03-23T12:16:52.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 msgr2=0x7f0888115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f0888115c50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0878009fd0 tx=0x7f0878009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:52.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 shutdown_connections 2024-03-23T12:16:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f086c06c7c0 0x7f086c06ec70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0888102e00 0x7f0888115c50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 --2- 172.21.15.63:0/316214980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08881037a0 0x7f0888116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 >> 172.21.15.63:0/316214980 conn(0x7f08880f89b0 msgr2=0x7f08880f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 shutdown_connections 2024-03-23T12:16:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.572+0000 7f088e200700 1 -- 172.21.15.63:0/316214980 wait complete. 2024-03-23T12:16:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:16:52.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 -- 172.21.15.63:0/3644083295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 msgr2=0x7f4c2c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 --2- 172.21.15.63:0/3644083295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c1031d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c004940 tx=0x7f4c1c009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:52.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 -- 172.21.15.63:0/3644083295 shutdown_connections 2024-03-23T12:16:52.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 --2- 172.21.15.63:0/3644083295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c1037a0 0x7f4c2c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 --2- 172.21.15.63:0/3644083295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c1031d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 -- 172.21.15.63:0/3644083295 >> 172.21.15.63:0/3644083295 conn(0x7f4c2c0f89b0 msgr2=0x7f4c2c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:52.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 -- 172.21.15.63:0/3644083295 shutdown_connections 2024-03-23T12:16:52.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.712+0000 7f4c30e00700 1 -- 172.21.15.63:0/3644083295 wait complete. 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.713+0000 7f4c30e00700 1 Processor -- start 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.713+0000 7f4c30e00700 1 -- start start 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c30e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c10af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c30e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c1037a0 0x7f4c2c10b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c30e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c10f0d0 con 0x7f4c2c102e00 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c30e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c10b9c0 con 0x7f4c2c1037a0 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c29e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c10af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c29e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c10af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56910/0 (socket says 172.21.15.63:56910) 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c29e00700 1 -- 172.21.15.63:0/4203049308 learned_addr learned my addr 172.21.15.63:0/4203049308 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c29400700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c1037a0 0x7f4c2c10b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c29e00700 1 -- 172.21.15.63:0/4203049308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c1037a0 msgr2=0x7f4c2c10b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c29e00700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c1037a0 0x7f4c2c10b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.714+0000 7f4c29e00700 1 -- 172.21.15.63:0/4203049308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c005040 con 0x7f4c2c102e00 2024-03-23T12:16:52.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.715+0000 7f4c29e00700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c10af40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c008d80 tx=0x7f4c1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:52.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.715+0000 7f4c22000700 1 -- 172.21.15.63:0/4203049308 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c1c00e030 con 0x7f4c2c102e00 2024-03-23T12:16:52.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.715+0000 7f4c22000700 1 -- 172.21.15.63:0/4203049308 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c1c013de0 con 0x7f4c2c102e00 2024-03-23T12:16:52.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.715+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c10bc40 con 0x7f4c2c102e00 2024-03-23T12:16:52.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.715+0000 7f4c22000700 1 -- 172.21.15.63:0/4203049308 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c1c01cd00 con 0x7f4c2c102e00 2024-03-23T12:16:52.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.715+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c11e5b0 con 0x7f4c2c102e00 2024-03-23T12:16:52.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.716+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c0fe550 con 0x7f4c2c102e00 2024-03-23T12:16:52.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.717+0000 7f4c22000700 1 -- 172.21.15.63:0/4203049308 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4c1c013930 con 0x7f4c2c102e00 2024-03-23T12:16:52.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.719+0000 7f4c22000700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4c1006c7d0 0x7f4c1006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:52.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.719+0000 7f4c22000700 1 -- 172.21.15.63:0/4203049308 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4c1c08d2e0 con 0x7f4c2c102e00 2024-03-23T12:16:52.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.719+0000 7f4c29400700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4c1006c7d0 0x7f4c1006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:52.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.719+0000 7f4c29400700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4c1006c7d0 0x7f4c1006ec80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c10c880 tx=0x7f4c1800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:52.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.723+0000 7f4c22000700 1 -- 172.21.15.63:0/4203049308 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4c1c057560 con 0x7f4c2c102e00 2024-03-23T12:16:52.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:52 smithi111 ceph-mon[106899]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:52.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:52 smithi111 ceph-mon[106899]: from='client.35426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:52.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:52 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3946162723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:16:52.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:52 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/316214980' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:16:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.955+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c2c10c630 con 0x7f4c1006c7d0 2024-03-23T12:16:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.956+0000 7f4c22000700 1 -- 172.21.15.63:0/4203049308 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4c2c10c630 con 0x7f4c1006c7d0 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:16:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:16:52.936 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:16:52.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4c1006c7d0 msgr2=0x7f4c1006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4c1006c7d0 0x7f4c1006ec80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c10c880 tx=0x7f4c1800b040 comp rx=0 tx=0).stop 2024-03-23T12:16:52.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 msgr2=0x7f4c2c10af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:52.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c10af40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c008d80 tx=0x7f4c1c009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:52.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 shutdown_connections 2024-03-23T12:16:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4c1006c7d0 0x7f4c1006ec80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c2c102e00 0x7f4c2c10af40 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 --2- 172.21.15.63:0/4203049308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c1037a0 0x7f4c2c10b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 >> 172.21.15.63:0/4203049308 conn(0x7f4c2c0f89b0 msgr2=0x7f4c2c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 shutdown_connections 2024-03-23T12:16:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:52.958+0000 7f4c30e00700 1 -- 172.21.15.63:0/4203049308 wait complete. 2024-03-23T12:16:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 -- 172.21.15.63:0/472867201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7150102e20 msgr2=0x7f71501031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 --2- 172.21.15.63:0/472867201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7150102e20 0x7f71501031f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7140004940 tx=0x7f7140009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 -- 172.21.15.63:0/472867201 shutdown_connections 2024-03-23T12:16:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 --2- 172.21.15.63:0/472867201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71501037c0 0x7f71500fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 --2- 172.21.15.63:0/472867201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7150102e20 0x7f71501031f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:53.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 -- 172.21.15.63:0/472867201 >> 172.21.15.63:0/472867201 conn(0x7f71500f89d0 msgr2=0x7f71500fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:53.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 -- 172.21.15.63:0/472867201 shutdown_connections 2024-03-23T12:16:53.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.096+0000 7f7156600700 1 -- 172.21.15.63:0/472867201 wait complete. 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.097+0000 7f7156600700 1 Processor -- start 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.097+0000 7f7156600700 1 -- start start 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f7156600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150102e20 0x7f7150113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f7156600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71501037c0 0x7f7150114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f7156600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71501146b0 con 0x7f71501037c0 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f7156600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715010daa0 con 0x7f7150102e20 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f714ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71501037c0 0x7f7150114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f714ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71501037c0 0x7f7150114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56926/0 (socket says 172.21.15.63:56926) 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f714ea00700 1 -- 172.21.15.63:0/2696934305 learned_addr learned my addr 172.21.15.63:0/2696934305 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:16:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f714f400700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150102e20 0x7f7150113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f714ea00700 1 -- 172.21.15.63:0/2696934305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150102e20 msgr2=0x7f7150113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f714ea00700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150102e20 0x7f7150113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.098+0000 7f714ea00700 1 -- 172.21.15.63:0/2696934305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7140005040 con 0x7f71501037c0 2024-03-23T12:16:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.099+0000 7f714ea00700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71501037c0 0x7f7150114060 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f713c002a80 tx=0x7f713c004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.099+0000 7f7147400700 1 -- 172.21.15.63:0/2696934305 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713c004e80 con 0x7f71501037c0 2024-03-23T12:16:53.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.099+0000 7f7147400700 1 -- 172.21.15.63:0/2696934305 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f713c00f040 con 0x7f71501037c0 2024-03-23T12:16:53.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.099+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715010dd80 con 0x7f71501037c0 2024-03-23T12:16:53.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.099+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715010e340 con 0x7f71501037c0 2024-03-23T12:16:53.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.099+0000 7f7147400700 1 -- 172.21.15.63:0/2696934305 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713c00da00 con 0x7f71501037c0 2024-03-23T12:16:53.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.100+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71500fe570 con 0x7f71501037c0 2024-03-23T12:16:53.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.101+0000 7f7147400700 1 -- 172.21.15.63:0/2696934305 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f713c007cc0 con 0x7f71501037c0 2024-03-23T12:16:53.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.102+0000 7f7147400700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f713406c7d0 0x7f713406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:16:53.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.102+0000 7f7147400700 1 -- 172.21.15.63:0/2696934305 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f713c08b0f0 con 0x7f71501037c0 2024-03-23T12:16:53.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.102+0000 7f714f400700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f713406c7d0 0x7f713406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:16:53.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.103+0000 7f714f400700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f713406c7d0 0x7f713406ec80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7140009fd0 tx=0x7f7140009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:16:53.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.106+0000 7f7147400700 1 -- 172.21.15.63:0/2696934305 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f713c058d80 con 0x7f71501037c0 2024-03-23T12:16:53.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:52 smithi063 ceph-mon[141124]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:53.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:52 smithi063 ceph-mon[141124]: from='client.35426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:53.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:52 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3946162723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:16:53.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:52 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/316214980' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:16:53.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.401+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f715005f5f0 con 0x7f71501037c0 2024-03-23T12:16:53.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.401+0000 7f7147400700 1 -- 172.21.15.63:0/2696934305 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f713c017e10 con 0x7f71501037c0 2024-03-23T12:16:53.380 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:16:53.380 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:16:53.380 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:16:53.381 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:16:53.382 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:16:53.382 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:16:53.382 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:16:53.382 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:16:53.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.403+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f713406c7d0 msgr2=0x7f713406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.403+0000 7f7156600700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f713406c7d0 0x7f713406ec80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7140009fd0 tx=0x7f7140009e30 comp rx=0 tx=0).stop 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.403+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71501037c0 msgr2=0x7f7150114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.403+0000 7f7156600700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71501037c0 0x7f7150114060 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f713c002a80 tx=0x7f713c004630 comp rx=0 tx=0).stop 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.404+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 shutdown_connections 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.404+0000 7f7156600700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f713406c7d0 0x7f713406ec80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.404+0000 7f7156600700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150102e20 0x7f7150113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.404+0000 7f7156600700 1 --2- 172.21.15.63:0/2696934305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71501037c0 0x7f7150114060 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.404+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 >> 172.21.15.63:0/2696934305 conn(0x7f71500f89d0 msgr2=0x7f71500f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.404+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 shutdown_connections 2024-03-23T12:16:53.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:16:53.404+0000 7f7156600700 1 -- 172.21.15.63:0/2696934305 wait complete. 2024-03-23T12:16:54.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:53 smithi063 ceph-mon[141124]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:54.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:53 smithi063 ceph-mon[141124]: pgmap v774: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:54.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:53 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2696934305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:16:54.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:53 smithi111 ceph-mon[106899]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:16:54.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:53 smithi111 ceph-mon[106899]: pgmap v774: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:54.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:53 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2696934305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:16:55.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:55 smithi063 ceph-mon[141124]: pgmap v775: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:55 smithi111 ceph-mon[106899]: pgmap v775: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:56.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:57.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:16:57.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:57 smithi111 ceph-mon[106899]: pgmap v776: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:58.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:57 smithi063 ceph-mon[141124]: pgmap v776: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:59.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:16:59 smithi063 ceph-mon[141124]: pgmap v777: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:16:59.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:16:59 smithi111 ceph-mon[106899]: pgmap v777: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:01.898 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:01 smithi111 ceph-mon[106899]: pgmap v778: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:02.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:01 smithi063 ceph-mon[141124]: pgmap v778: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:03.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:03 smithi063 ceph-mon[141124]: pgmap v779: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:03.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:03 smithi111 ceph-mon[106899]: pgmap v779: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:05.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:05 smithi063 ceph-mon[141124]: pgmap v780: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:05.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:05 smithi111 ceph-mon[106899]: pgmap v780: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:07.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:07 smithi063 ceph-mon[141124]: pgmap v781: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:07.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:07 smithi111 ceph-mon[106899]: pgmap v781: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:09.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:09 smithi063 ceph-mon[141124]: pgmap v782: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:09.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:09 smithi111 ceph-mon[106899]: pgmap v782: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:11 smithi063 ceph-mon[141124]: pgmap v783: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:12.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:12.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:11 smithi111 ceph-mon[106899]: pgmap v783: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:13.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:13 smithi063 ceph-mon[141124]: pgmap v784: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:13.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:13 smithi111 ceph-mon[106899]: pgmap v784: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:15.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:15 smithi063 ceph-mon[141124]: pgmap v785: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:15.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:15 smithi111 ceph-mon[106899]: pgmap v785: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:17.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:17 smithi063 ceph-mon[141124]: pgmap v786: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:17.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:17 smithi111 ceph-mon[106899]: pgmap v786: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:19.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:19 smithi063 ceph-mon[141124]: pgmap v787: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:19 smithi111 ceph-mon[106899]: pgmap v787: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:22.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:21 smithi063 ceph-mon[141124]: pgmap v788: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:22.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:21 smithi111 ceph-mon[106899]: pgmap v788: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:23.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:23 smithi063 ceph-mon[141124]: pgmap v789: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:23.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:23 smithi111 ceph-mon[106899]: pgmap v789: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:23.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 -- 172.21.15.63:0/3263659589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede41037a0 msgr2=0x7fede40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:23.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 --2- 172.21.15.63:0/3263659589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede41037a0 0x7fede40fce10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fedd0004940 tx=0x7fedd0009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:23.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 -- 172.21.15.63:0/3263659589 shutdown_connections 2024-03-23T12:17:23.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 --2- 172.21.15.63:0/3263659589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede41037a0 0x7fede40fce10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 --2- 172.21.15.63:0/3263659589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fede4102e00 0x7fede41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 -- 172.21.15.63:0/3263659589 >> 172.21.15.63:0/3263659589 conn(0x7fede40f89b0 msgr2=0x7fede40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:23.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 -- 172.21.15.63:0/3263659589 shutdown_connections 2024-03-23T12:17:23.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.557+0000 7fedeb600700 1 -- 172.21.15.63:0/3263659589 wait complete. 2024-03-23T12:17:23.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.558+0000 7fedeb600700 1 Processor -- start 2024-03-23T12:17:23.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.559+0000 7fedeb600700 1 -- start start 2024-03-23T12:17:23.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.559+0000 7fedeb600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede4102e00 0x7fede4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:23.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.559+0000 7fedeb600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fede41037a0 0x7fede4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.559+0000 7fedeb600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede4114600 con 0x7fede4102e00 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.559+0000 7fedeb600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede410d9f0 con 0x7fede41037a0 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.560+0000 7fede8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede4102e00 0x7fede4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.560+0000 7fede8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede4102e00 0x7fede4113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51944/0 (socket says 172.21.15.63:51944) 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.560+0000 7fede8c00700 1 -- 172.21.15.63:0/3704842015 learned_addr learned my addr 172.21.15.63:0/3704842015 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.560+0000 7fede3e00700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fede41037a0 0x7fede4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.560+0000 7fede8c00700 1 -- 172.21.15.63:0/3704842015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fede41037a0 msgr2=0x7fede4113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.560+0000 7fede8c00700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fede41037a0 0x7fede4113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.560+0000 7fede8c00700 1 -- 172.21.15.63:0/3704842015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0005040 con 0x7fede4102e00 2024-03-23T12:17:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.561+0000 7fede8c00700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede4102e00 0x7fede4113a70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fedd8002a80 tx=0x7fedd80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.561+0000 7fede0c00700 1 -- 172.21.15.63:0/3704842015 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd8007cc0 con 0x7fede4102e00 2024-03-23T12:17:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.561+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede410dcd0 con 0x7fede4102e00 2024-03-23T12:17:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.561+0000 7fede0c00700 1 -- 172.21.15.63:0/3704842015 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedd800f040 con 0x7fede4102e00 2024-03-23T12:17:23.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.562+0000 7fede0c00700 1 -- 172.21.15.63:0/3704842015 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd800da60 con 0x7fede4102e00 2024-03-23T12:17:23.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.562+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede410e260 con 0x7fede4102e00 2024-03-23T12:17:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.563+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede40fe550 con 0x7fede4102e00 2024-03-23T12:17:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.564+0000 7fede0c00700 1 -- 172.21.15.63:0/3704842015 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fedd800dc60 con 0x7fede4102e00 2024-03-23T12:17:23.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.565+0000 7fede0c00700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedcc06c7d0 0x7fedcc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:23.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.565+0000 7fede0c00700 1 -- 172.21.15.63:0/3704842015 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fedd808b620 con 0x7fede4102e00 2024-03-23T12:17:23.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.565+0000 7fede3e00700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedcc06c7d0 0x7fedcc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:23.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.566+0000 7fede3e00700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedcc06c7d0 0x7fedcc06ec80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fede410ecd0 tx=0x7fedd0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:23.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.570+0000 7fede0c00700 1 -- 172.21.15.63:0/3704842015 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fedd8059330 con 0x7fede4102e00 2024-03-23T12:17:23.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.809+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fede4109df0 con 0x7fedcc06c7d0 2024-03-23T12:17:23.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.810+0000 7fede0c00700 1 -- 172.21.15.63:0/3704842015 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fede4109df0 con 0x7fedcc06c7d0 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedcc06c7d0 msgr2=0x7fedcc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedcc06c7d0 0x7fedcc06ec80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fede410ecd0 tx=0x7fedd0009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede4102e00 msgr2=0x7fede4113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede4102e00 0x7fede4113a70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fedd8002a80 tx=0x7fedd80046b0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 shutdown_connections 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fedcc06c7d0 0x7fedcc06ec80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fede4102e00 0x7fede4113a70 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 --2- 172.21.15.63:0/3704842015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fede41037a0 0x7fede4113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 >> 172.21.15.63:0/3704842015 conn(0x7fede40f89b0 msgr2=0x7fede40f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 shutdown_connections 2024-03-23T12:17:23.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.813+0000 7fedeb600700 1 -- 172.21.15.63:0/3704842015 wait complete. 2024-03-23T12:17:23.807 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:17:23.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 -- 172.21.15.63:0/1925877889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 msgr2=0x7fca801031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:23.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 --2- 172.21.15.63:0/1925877889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca801031d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fca74004940 tx=0x7fca74009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:23.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 -- 172.21.15.63:0/1925877889 shutdown_connections 2024-03-23T12:17:23.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 --2- 172.21.15.63:0/1925877889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca801037a0 0x7fca800fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 --2- 172.21.15.63:0/1925877889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca801031d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 -- 172.21.15.63:0/1925877889 >> 172.21.15.63:0/1925877889 conn(0x7fca800f89b0 msgr2=0x7fca800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:23.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 -- 172.21.15.63:0/1925877889 shutdown_connections 2024-03-23T12:17:23.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.952+0000 7fca87600700 1 -- 172.21.15.63:0/1925877889 wait complete. 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.953+0000 7fca87600700 1 Processor -- start 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.953+0000 7fca87600700 1 -- start start 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca87600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca80113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca87600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca801037a0 0x7fca80114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca87600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801146b0 con 0x7fca80102e00 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca87600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8010daa0 con 0x7fca801037a0 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca84c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca80113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca84c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca80113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51950/0 (socket says 172.21.15.63:51950) 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca84c00700 1 -- 172.21.15.63:0/3300463718 learned_addr learned my addr 172.21.15.63:0/3300463718 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca7fe00700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca801037a0 0x7fca80114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca84c00700 1 -- 172.21.15.63:0/3300463718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca801037a0 msgr2=0x7fca80114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca84c00700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca801037a0 0x7fca80114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.954+0000 7fca84c00700 1 -- 172.21.15.63:0/3300463718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca74005040 con 0x7fca80102e00 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.955+0000 7fca84c00700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca80113b20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fca74009390 tx=0x7fca74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.955+0000 7fca7cc00700 1 -- 172.21.15.63:0/3300463718 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca7400e040 con 0x7fca80102e00 2024-03-23T12:17:23.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.955+0000 7fca7cc00700 1 -- 172.21.15.63:0/3300463718 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca74013420 con 0x7fca80102e00 2024-03-23T12:17:23.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.955+0000 7fca7cc00700 1 -- 172.21.15.63:0/3300463718 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca7401c630 con 0x7fca80102e00 2024-03-23T12:17:23.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.955+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8010dd20 con 0x7fca80102e00 2024-03-23T12:17:23.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.955+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8010e2e0 con 0x7fca80102e00 2024-03-23T12:17:23.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.956+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca800fe550 con 0x7fca80102e00 2024-03-23T12:17:23.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.957+0000 7fca7cc00700 1 -- 172.21.15.63:0/3300463718 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fca740139e0 con 0x7fca80102e00 2024-03-23T12:17:23.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.958+0000 7fca7cc00700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca6806c7d0 0x7fca6806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:23.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.958+0000 7fca7cc00700 1 -- 172.21.15.63:0/3300463718 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fca7408ce10 con 0x7fca80102e00 2024-03-23T12:17:23.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.958+0000 7fca7fe00700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca6806c7d0 0x7fca6806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:23.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.959+0000 7fca7fe00700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca6806c7d0 0x7fca6806ec80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fca8010ec60 tx=0x7fca6c009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:23.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:23.963+0000 7fca7cc00700 1 -- 172.21.15.63:0/3300463718 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fca7405aaa0 con 0x7fca80102e00 2024-03-23T12:17:24.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.198+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca80109dc0 con 0x7fca6806c7d0 2024-03-23T12:17:24.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.199+0000 7fca7cc00700 1 -- 172.21.15.63:0/3300463718 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fca80109dc0 con 0x7fca6806c7d0 2024-03-23T12:17:24.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca6806c7d0 msgr2=0x7fca6806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca6806c7d0 0x7fca6806ec80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fca8010ec60 tx=0x7fca6c009040 comp rx=0 tx=0).stop 2024-03-23T12:17:24.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 msgr2=0x7fca80113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca80113b20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fca74009390 tx=0x7fca74009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 shutdown_connections 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca6806c7d0 0x7fca6806ec80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80102e00 0x7fca80113b20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 --2- 172.21.15.63:0/3300463718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca801037a0 0x7fca80114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.201+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 >> 172.21.15.63:0/3300463718 conn(0x7fca800f89b0 msgr2=0x7fca800f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.202+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 shutdown_connections 2024-03-23T12:17:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.202+0000 7fca87600700 1 -- 172.21.15.63:0/3300463718 wait complete. 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 -- 172.21.15.63:0/2820400787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fde90 msgr2=0x7f22b40fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 --2- 172.21.15.63:0/2820400787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fde90 0x7f22b40fe260 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f22a8004940 tx=0x7f22a8009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 -- 172.21.15.63:0/2820400787 shutdown_connections 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 --2- 172.21.15.63:0/2820400787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b40fe830 0x7f22b4106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 --2- 172.21.15.63:0/2820400787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fde90 0x7f22b40fe260 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 -- 172.21.15.63:0/2820400787 >> 172.21.15.63:0/2820400787 conn(0x7f22b4072bc0 msgr2=0x7f22b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 -- 172.21.15.63:0/2820400787 shutdown_connections 2024-03-23T12:17:24.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.339+0000 7f22bbc00700 1 -- 172.21.15.63:0/2820400787 wait complete. 2024-03-23T12:17:24.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.340+0000 7f22bbc00700 1 Processor -- start 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.340+0000 7f22bbc00700 1 -- start start 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.340+0000 7f22bbc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b40fde90 0x7f22b40fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.340+0000 7f22bbc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fe830 0x7f22b40fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.340+0000 7f22bbc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b40fc820 con 0x7f22b40fe830 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.340+0000 7f22bbc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b40fc960 con 0x7f22b40fde90 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.340+0000 7f22b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fe830 0x7f22b40fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fe830 0x7f22b40fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51964/0 (socket says 172.21.15.63:51964) 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b3e00700 1 -- 172.21.15.63:0/3917454839 learned_addr learned my addr 172.21.15.63:0/3917454839 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b3e00700 1 -- 172.21.15.63:0/3917454839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b40fde90 msgr2=0x7f22b40fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b3e00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b40fde90 0x7f22b40fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b3e00700 1 -- 172.21.15.63:0/3917454839 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a8005040 con 0x7f22b40fe830 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b3e00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fe830 0x7f22b40fc290 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f22a000eb40 tx=0x7f22a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b0c00700 1 -- 172.21.15.63:0/3917454839 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22a0007cc0 con 0x7f22b40fe830 2024-03-23T12:17:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b0c00700 1 -- 172.21.15.63:0/3917454839 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22a000f040 con 0x7f22b40fe830 2024-03-23T12:17:24.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22b0c00700 1 -- 172.21.15.63:0/3917454839 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22a0009570 con 0x7f22b40fe830 2024-03-23T12:17:24.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.341+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b40fcbb0 con 0x7f22b40fe830 2024-03-23T12:17:24.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.342+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b411a120 con 0x7f22b40fe830 2024-03-23T12:17:24.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.343+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b4065e30 con 0x7f22b40fe830 2024-03-23T12:17:24.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.344+0000 7f22b0c00700 1 -- 172.21.15.63:0/3917454839 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f22a00096d0 con 0x7f22b40fe830 2024-03-23T12:17:24.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.344+0000 7f22b0c00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f229c06c7c0 0x7f229c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:24.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.344+0000 7f22b0c00700 1 -- 172.21.15.63:0/3917454839 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f22a008b7c0 con 0x7f22b40fe830 2024-03-23T12:17:24.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.345+0000 7f22b9200700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f229c06c7c0 0x7f229c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:24.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.345+0000 7f22b9200700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f229c06c7c0 0x7f229c06ec70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f22a800b0d0 tx=0x7f22a8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:24.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.349+0000 7f22b0c00700 1 -- 172.21.15.63:0/3917454839 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f22a00559f0 con 0x7f22b40fe830 2024-03-23T12:17:24.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:24 smithi063 ceph-mon[141124]: from='client.35446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:24.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:24 smithi111 ceph-mon[106899]: from='client.35446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:24.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.574+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f22b404bed0 con 0x7f229c06c7c0 2024-03-23T12:17:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.584+0000 7f22b0c00700 1 -- 172.21.15.63:0/3917454839 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f22b404bed0 con 0x7f229c06c7c0 2024-03-23T12:17:24.563 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:17:24.563 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (26m) 3m ago 39m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:17:24.563 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (23m) 3m ago 40m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:17:24.563 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (23m) 3m ago 38m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:17:24.563 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (26m) 3m ago 39m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (19m) 3m ago 33m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (19m) 3m ago 33m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (3m) 3m ago 33m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (19m) 3m ago 33m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (29m) 3m ago 41m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (27m) 3m ago 38m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (25m) 3m ago 41m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (24m) 3m ago 38m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (27m) 3m ago 39m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (27m) 3m ago 38m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (23m) 3m ago 37m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (22m) 3m ago 37m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (22m) 3m ago 36m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (21m) 3m ago 36m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (20m) 3m ago 35m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:17:24.564 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (20m) 3m ago 35m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:17:24.565 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (27m) 3m ago 39m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:17:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f229c06c7c0 msgr2=0x7f229c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f229c06c7c0 0x7f229c06ec70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f22a800b0d0 tx=0x7f22a8009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fe830 msgr2=0x7f22b40fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fe830 0x7f22b40fc290 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f22a000eb40 tx=0x7f22a000b040 comp rx=0 tx=0).stop 2024-03-23T12:17:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 shutdown_connections 2024-03-23T12:17:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f229c06c7c0 0x7f229c06ec70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b40fde90 0x7f22b40fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 --2- 172.21.15.63:0/3917454839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b40fe830 0x7f22b40fc290 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 >> 172.21.15.63:0/3917454839 conn(0x7f22b4072bc0 msgr2=0x7f22b41037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.586+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 shutdown_connections 2024-03-23T12:17:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.587+0000 7f22bbc00700 1 -- 172.21.15.63:0/3917454839 wait complete. 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 -- 172.21.15.63:0/3161422603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 msgr2=0x7fef340fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 --2- 172.21.15.63:0/3161422603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef340fce10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fef20004940 tx=0x7fef20009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 -- 172.21.15.63:0/3161422603 shutdown_connections 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 --2- 172.21.15.63:0/3161422603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef340fce10 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 --2- 172.21.15.63:0/3161422603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef34102e00 0x7fef341031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 -- 172.21.15.63:0/3161422603 >> 172.21.15.63:0/3161422603 conn(0x7fef340f89b0 msgr2=0x7fef340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 -- 172.21.15.63:0/3161422603 shutdown_connections 2024-03-23T12:17:24.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.724+0000 7fef3be00700 1 -- 172.21.15.63:0/3161422603 wait complete. 2024-03-23T12:17:24.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.725+0000 7fef3be00700 1 Processor -- start 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.725+0000 7fef3be00700 1 -- start start 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.725+0000 7fef3be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef34102e00 0x7fef34115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.725+0000 7fef3be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef34116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.725+0000 7fef3be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341168b0 con 0x7fef341037a0 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.725+0000 7fef3be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3410fc80 con 0x7fef34102e00 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef38a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef34116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef38a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef34116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51978/0 (socket says 172.21.15.63:51978) 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef38a00700 1 -- 172.21.15.63:0/4068187206 learned_addr learned my addr 172.21.15.63:0/4068187206 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef39400700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef34102e00 0x7fef34115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef38a00700 1 -- 172.21.15.63:0/4068187206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef34102e00 msgr2=0x7fef34115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef38a00700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef34102e00 0x7fef34115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:24.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef38a00700 1 -- 172.21.15.63:0/4068187206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef20005040 con 0x7fef341037a0 2024-03-23T12:17:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.726+0000 7fef38a00700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef34116260 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fef20009390 tx=0x7fef20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.727+0000 7fef31600700 1 -- 172.21.15.63:0/4068187206 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2000e040 con 0x7fef341037a0 2024-03-23T12:17:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.727+0000 7fef31600700 1 -- 172.21.15.63:0/4068187206 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef20013420 con 0x7fef341037a0 2024-03-23T12:17:24.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.727+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3410ff00 con 0x7fef341037a0 2024-03-23T12:17:24.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.727+0000 7fef31600700 1 -- 172.21.15.63:0/4068187206 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2001c630 con 0x7fef341037a0 2024-03-23T12:17:24.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.727+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef341104c0 con 0x7fef341037a0 2024-03-23T12:17:24.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.728+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef340fe550 con 0x7fef341037a0 2024-03-23T12:17:24.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.729+0000 7fef31600700 1 -- 172.21.15.63:0/4068187206 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fef20013a00 con 0x7fef341037a0 2024-03-23T12:17:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.730+0000 7fef31600700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fef1c06c7d0 0x7fef1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:24.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.730+0000 7fef31600700 1 -- 172.21.15.63:0/4068187206 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fef2008cb20 con 0x7fef341037a0 2024-03-23T12:17:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.730+0000 7fef39400700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fef1c06c7d0 0x7fef1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:24.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.731+0000 7fef39400700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fef1c06c7d0 0x7fef1c06ec80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef28004f60 tx=0x7fef2800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:24.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:24.734+0000 7fef31600700 1 -- 172.21.15.63:0/4068187206 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef2005a830 con 0x7fef341037a0 2024-03-23T12:17:25.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.029+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fef3404bed0 con 0x7fef341037a0 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.030+0000 7fef31600700 1 -- 172.21.15.63:0/4068187206 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fef2005a3c0 con 0x7fef341037a0 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:17:25.010 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:17:25.011 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:17:25.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fef1c06c7d0 msgr2=0x7fef1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fef1c06c7d0 0x7fef1c06ec80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef28004f60 tx=0x7fef2800b040 comp rx=0 tx=0).stop 2024-03-23T12:17:25.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 msgr2=0x7fef34116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef34116260 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fef20009390 tx=0x7fef20009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:25.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 shutdown_connections 2024-03-23T12:17:25.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fef1c06c7d0 0x7fef1c06ec80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef34102e00 0x7fef34115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 --2- 172.21.15.63:0/4068187206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef341037a0 0x7fef34116260 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.032+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 >> 172.21.15.63:0/4068187206 conn(0x7fef340f89b0 msgr2=0x7fef34070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:25.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.033+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 shutdown_connections 2024-03-23T12:17:25.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.033+0000 7fef3be00700 1 -- 172.21.15.63:0/4068187206 wait complete. 2024-03-23T12:17:25.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 -- 172.21.15.63:0/4151114208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 msgr2=0x7efdcc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 --2- 172.21.15.63:0/4151114208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc0fce10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7efdb8004940 tx=0x7efdb8009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:25.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 -- 172.21.15.63:0/4151114208 shutdown_connections 2024-03-23T12:17:25.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 --2- 172.21.15.63:0/4151114208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc0fce10 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 --2- 172.21.15.63:0/4151114208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdcc102e00 0x7efdcc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 -- 172.21.15.63:0/4151114208 >> 172.21.15.63:0/4151114208 conn(0x7efdcc0f89b0 msgr2=0x7efdcc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:25.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 -- 172.21.15.63:0/4151114208 shutdown_connections 2024-03-23T12:17:25.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.173+0000 7efdd1000700 1 -- 172.21.15.63:0/4151114208 wait complete. 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.174+0000 7efdd1000700 1 Processor -- start 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.174+0000 7efdd1000700 1 -- start start 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.174+0000 7efdd1000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdcc102e00 0x7efdcc113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdd1000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdd1000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc114690 con 0x7efdcc1037a0 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdd1000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc10da80 con 0x7efdcc102e00 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdc9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdc9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51988/0 (socket says 172.21.15.63:51988) 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdc9400700 1 -- 172.21.15.63:0/2250921396 learned_addr learned my addr 172.21.15.63:0/2250921396 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdc9e00700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdcc102e00 0x7efdcc113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdc9400700 1 -- 172.21.15.63:0/2250921396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdcc102e00 msgr2=0x7efdcc113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdc9400700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdcc102e00 0x7efdcc113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.175+0000 7efdc9400700 1 -- 172.21.15.63:0/2250921396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb8005040 con 0x7efdcc1037a0 2024-03-23T12:17:25.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.176+0000 7efdc9400700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc114040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009390 tx=0x7efdb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:25.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.176+0000 7efdc2000700 1 -- 172.21.15.63:0/2250921396 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb800e040 con 0x7efdcc1037a0 2024-03-23T12:17:25.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.176+0000 7efdc2000700 1 -- 172.21.15.63:0/2250921396 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdb8004290 con 0x7efdcc1037a0 2024-03-23T12:17:25.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.176+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc10dd00 con 0x7efdcc1037a0 2024-03-23T12:17:25.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.176+0000 7efdc2000700 1 -- 172.21.15.63:0/2250921396 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb8013e70 con 0x7efdcc1037a0 2024-03-23T12:17:25.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.176+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc10e2c0 con 0x7efdcc1037a0 2024-03-23T12:17:25.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.177+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc0fe550 con 0x7efdcc1037a0 2024-03-23T12:17:25.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.178+0000 7efdc2000700 1 -- 172.21.15.63:0/2250921396 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7efdb8013690 con 0x7efdcc1037a0 2024-03-23T12:17:25.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.179+0000 7efdc2000700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdb006c760 0x7efdb006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.179+0000 7efdc2000700 1 -- 172.21.15.63:0/2250921396 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efdb808cd00 con 0x7efdcc1037a0 2024-03-23T12:17:25.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.180+0000 7efdc9e00700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdb006c760 0x7efdb006ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.180+0000 7efdc9e00700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdb006c760 0x7efdb006ec10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efdbc004f40 tx=0x7efdbc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:25.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.184+0000 7efdc2000700 1 -- 172.21.15.63:0/2250921396 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efdb805aa40 con 0x7efdcc1037a0 2024-03-23T12:17:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:25 smithi063 ceph-mon[141124]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:25 smithi063 ceph-mon[141124]: from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:25 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4068187206' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:17:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:25 smithi063 ceph-mon[141124]: pgmap v790: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:25 smithi111 ceph-mon[106899]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:25 smithi111 ceph-mon[106899]: from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:25 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4068187206' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:17:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:25 smithi111 ceph-mon[106899]: pgmap v790: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:25.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.440+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efdcc109da0 con 0x7efdcc1037a0 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.440+0000 7efdc2000700 1 -- 172.21.15.63:0/2250921396 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7efdb8013420 con 0x7efdcc1037a0 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:17:25.420 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:17:25.421 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdb006c760 msgr2=0x7efdb006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdb006c760 0x7efdb006ec10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efdbc004f40 tx=0x7efdbc00b040 comp rx=0 tx=0).stop 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 msgr2=0x7efdcc114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc114040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009390 tx=0x7efdb8009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 shutdown_connections 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7efdb006c760 0x7efdb006ec10 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdcc102e00 0x7efdcc113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.442+0000 7efdd1000700 1 --2- 172.21.15.63:0/2250921396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc1037a0 0x7efdcc114040 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.443+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 >> 172.21.15.63:0/2250921396 conn(0x7efdcc0f89b0 msgr2=0x7efdcc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:25.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.443+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 shutdown_connections 2024-03-23T12:17:25.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.443+0000 7efdd1000700 1 -- 172.21.15.63:0/2250921396 wait complete. 2024-03-23T12:17:25.424 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:17:25.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.581+0000 7f2d5ea00700 1 -- 172.21.15.63:0/1494951059 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fe870 msgr2=0x7f2d58106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.581+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/1494951059 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fe870 0x7f2d58106f40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2d44004940 tx=0x7f2d44009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:25.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.581+0000 7f2d5ea00700 1 -- 172.21.15.63:0/1494951059 shutdown_connections 2024-03-23T12:17:25.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.581+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/1494951059 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fe870 0x7f2d58106f40 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.581+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/1494951059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d580fded0 0x7f2d580fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.581+0000 7f2d5ea00700 1 -- 172.21.15.63:0/1494951059 >> 172.21.15.63:0/1494951059 conn(0x7f2d58072bc0 msgr2=0x7f2d58072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:25.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.581+0000 7f2d5ea00700 1 -- 172.21.15.63:0/1494951059 shutdown_connections 2024-03-23T12:17:25.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.582+0000 7f2d5ea00700 1 -- 172.21.15.63:0/1494951059 wait complete. 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.582+0000 7f2d5ea00700 1 Processor -- start 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.582+0000 7f2d5ea00700 1 -- start start 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d5ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fded0 0x7f2d58115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d5ea00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d580fe870 0x7f2d581161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d5ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d581167f0 con 0x7f2d580fded0 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d5ea00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5810fbc0 con 0x7f2d580fe870 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d57e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fded0 0x7f2d58115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d57e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fded0 0x7f2d58115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52012/0 (socket says 172.21.15.63:52012) 2024-03-23T12:17:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d57e00700 1 -- 172.21.15.63:0/574282860 learned_addr learned my addr 172.21.15.63:0/574282860 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.583+0000 7f2d57400700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d580fe870 0x7f2d581161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d57e00700 1 -- 172.21.15.63:0/574282860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d580fe870 msgr2=0x7f2d581161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d57e00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d580fe870 0x7f2d581161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d57e00700 1 -- 172.21.15.63:0/574282860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44005040 con 0x7f2d580fded0 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d57e00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fded0 0x7f2d58115c60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2d48002a50 tx=0x7f2d480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d4fe00700 1 -- 172.21.15.63:0/574282860 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d48007cc0 con 0x7f2d580fded0 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d4fe00700 1 -- 172.21.15.63:0/574282860 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d4800f040 con 0x7f2d580fded0 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5810fea0 con 0x7f2d580fded0 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d4fe00700 1 -- 172.21.15.63:0/574282860 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4800d9c0 con 0x7f2d580fded0 2024-03-23T12:17:25.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.584+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d581103e0 con 0x7f2d580fded0 2024-03-23T12:17:25.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.585+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d58106a00 con 0x7f2d580fded0 2024-03-23T12:17:25.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.586+0000 7f2d4fe00700 1 -- 172.21.15.63:0/574282860 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2d4800db20 con 0x7f2d580fded0 2024-03-23T12:17:25.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.587+0000 7f2d4fe00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d3c06c7c0 0x7f2d3c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.587+0000 7f2d4fe00700 1 -- 172.21.15.63:0/574282860 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2d4808b820 con 0x7f2d580fded0 2024-03-23T12:17:25.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.587+0000 7f2d57400700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d3c06c7c0 0x7f2d3c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.588+0000 7f2d57400700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d3c06c7c0 0x7f2d3c06ec70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2d58111500 tx=0x7f2d44009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:25.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.592+0000 7f2d4fe00700 1 -- 172.21.15.63:0/574282860 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2d48059430 con 0x7f2d580fded0 2024-03-23T12:17:25.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.827+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d58109dc0 con 0x7f2d3c06c7c0 2024-03-23T12:17:25.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.828+0000 7f2d4fe00700 1 -- 172.21.15.63:0/574282860 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2d58109dc0 con 0x7f2d3c06c7c0 2024-03-23T12:17:25.808 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:17:25.808 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:17:25.808 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:17:25.808 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:17:25.809 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d3c06c7c0 msgr2=0x7f2d3c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d3c06c7c0 0x7f2d3c06ec70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2d58111500 tx=0x7f2d44009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fded0 msgr2=0x7f2d58115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fded0 0x7f2d58115c60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2d48002a50 tx=0x7f2d480046b0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 shutdown_connections 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2d3c06c7c0 0x7f2d3c06ec70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d580fded0 0x7f2d58115c60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 --2- 172.21.15.63:0/574282860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d580fe870 0x7f2d581161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 >> 172.21.15.63:0/574282860 conn(0x7f2d58072bc0 msgr2=0x7f2d580fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 shutdown_connections 2024-03-23T12:17:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.831+0000 7f2d5ea00700 1 -- 172.21.15.63:0/574282860 wait complete. 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 -- 172.21.15.63:0/4133506650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 msgr2=0x7f34881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 --2- 172.21.15.63:0/4133506650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f34881031d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3478004940 tx=0x7f3478009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 -- 172.21.15.63:0/4133506650 shutdown_connections 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 --2- 172.21.15.63:0/4133506650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34881037a0 0x7f34880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 --2- 172.21.15.63:0/4133506650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f34881031d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 -- 172.21.15.63:0/4133506650 >> 172.21.15.63:0/4133506650 conn(0x7f34880f89b0 msgr2=0x7f34880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 -- 172.21.15.63:0/4133506650 shutdown_connections 2024-03-23T12:17:25.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.972+0000 7f348ee00700 1 -- 172.21.15.63:0/4133506650 wait complete. 2024-03-23T12:17:25.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.973+0000 7f348ee00700 1 Processor -- start 2024-03-23T12:17:25.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.973+0000 7f348ee00700 1 -- start start 2024-03-23T12:17:25.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f348ee00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f3488113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f348ee00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34881037a0 0x7f3488114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f348ee00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34881146b0 con 0x7f3488102e00 2024-03-23T12:17:25.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f348ee00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348810daa0 con 0x7f34881037a0 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f3488113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f3488113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52024/0 (socket says 172.21.15.63:52024) 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487e00700 1 -- 172.21.15.63:0/1625438752 learned_addr learned my addr 172.21.15.63:0/1625438752 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487400700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34881037a0 0x7f3488114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487e00700 1 -- 172.21.15.63:0/1625438752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34881037a0 msgr2=0x7f3488114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487e00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34881037a0 0x7f3488114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487e00700 1 -- 172.21.15.63:0/1625438752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478005040 con 0x7f3488102e00 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.974+0000 7f3487400700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34881037a0 0x7f3488114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.975+0000 7f3487e00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f3488113b20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3478009360 tx=0x7f3478009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.975+0000 7f347fe00700 1 -- 172.21.15.63:0/1625438752 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347800e040 con 0x7f3488102e00 2024-03-23T12:17:25.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.975+0000 7f347fe00700 1 -- 172.21.15.63:0/1625438752 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3478013420 con 0x7f3488102e00 2024-03-23T12:17:25.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.975+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348810dd20 con 0x7f3488102e00 2024-03-23T12:17:25.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.975+0000 7f347fe00700 1 -- 172.21.15.63:0/1625438752 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347801c5f0 con 0x7f3488102e00 2024-03-23T12:17:25.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.975+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348810e2e0 con 0x7f3488102e00 2024-03-23T12:17:25.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.976+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34880fe550 con 0x7f3488102e00 2024-03-23T12:17:25.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.977+0000 7f347fe00700 1 -- 172.21.15.63:0/1625438752 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3478013590 con 0x7f3488102e00 2024-03-23T12:17:25.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.978+0000 7f347fe00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f346c06c7d0 0x7f346c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:25.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.979+0000 7f347fe00700 1 -- 172.21.15.63:0/1625438752 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f347808cbc0 con 0x7f3488102e00 2024-03-23T12:17:25.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.979+0000 7f3487400700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f346c06c7d0 0x7f346c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:25.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.979+0000 7f3487400700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f346c06c7d0 0x7f346c06ec80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fd0 tx=0x7f3474009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:25.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:25.983+0000 7f347fe00700 1 -- 172.21.15.63:0/1625438752 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f347805a990 con 0x7f3488102e00 2024-03-23T12:17:26.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.284+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f348804bed0 con 0x7f3488102e00 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.284+0000 7f347fe00700 1 -- 172.21.15.63:0/1625438752 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f347805a520 con 0x7f3488102e00 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:17:26.264 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:17:26.265 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:17:26.266 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:17:26.266 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:17:26.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f346c06c7d0 msgr2=0x7f346c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:26.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f346c06c7d0 0x7f346c06ec80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fd0 tx=0x7f3474009ee0 comp rx=0 tx=0).stop 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 msgr2=0x7f3488113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f3488113b20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3478009360 tx=0x7f3478009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 shutdown_connections 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f346c06c7d0 0x7f346c06ec80 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3488102e00 0x7f3488113b20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.286+0000 7f348ee00700 1 --2- 172.21.15.63:0/1625438752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34881037a0 0x7f3488114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.287+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 >> 172.21.15.63:0/1625438752 conn(0x7f34880f89b0 msgr2=0x7f34880f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.287+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 shutdown_connections 2024-03-23T12:17:26.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:26.287+0000 7f348ee00700 1 -- 172.21.15.63:0/1625438752 wait complete. 2024-03-23T12:17:26.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:26 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2250921396' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:17:26.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:26.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:26 smithi063 ceph-mon[141124]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:26.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:26 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2250921396' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:17:26.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:26.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:26 smithi111 ceph-mon[106899]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:27.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:27 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1625438752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:17:27.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:27 smithi111 ceph-mon[106899]: pgmap v791: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:27.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:27 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1625438752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:17:27.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:27 smithi063 ceph-mon[141124]: pgmap v791: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:29.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:29 smithi063 ceph-mon[141124]: pgmap v792: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:29.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:29 smithi111 ceph-mon[106899]: pgmap v792: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:32.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:31 smithi063 ceph-mon[141124]: pgmap v793: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:32.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:31 smithi111 ceph-mon[106899]: pgmap v793: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:33.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:33 smithi063 ceph-mon[141124]: pgmap v794: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:33.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:33 smithi111 ceph-mon[106899]: pgmap v794: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:35.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:35 smithi063 ceph-mon[141124]: pgmap v795: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:35.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:35 smithi111 ceph-mon[106899]: pgmap v795: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:37.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:37 smithi063 ceph-mon[141124]: pgmap v796: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:37.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:37 smithi111 ceph-mon[106899]: pgmap v796: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:39.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:39 smithi063 ceph-mon[141124]: pgmap v797: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:39.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:39 smithi111 ceph-mon[106899]: pgmap v797: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:41 smithi063 ceph-mon[141124]: pgmap v798: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:42.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:42.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:41 smithi111 ceph-mon[106899]: pgmap v798: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:43.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:43 smithi063 ceph-mon[141124]: pgmap v799: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:43.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:43 smithi111 ceph-mon[106899]: pgmap v799: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:45.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:45 smithi063 ceph-mon[141124]: pgmap v800: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:45.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:45 smithi111 ceph-mon[106899]: pgmap v800: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:47.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:47 smithi063 ceph-mon[141124]: pgmap v801: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:47.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:47 smithi111 ceph-mon[106899]: pgmap v801: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:49.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:49 smithi063 ceph-mon[141124]: pgmap v802: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:49.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:49 smithi111 ceph-mon[106899]: pgmap v802: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:52.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:51 smithi063 ceph-mon[141124]: pgmap v803: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:52.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:51 smithi111 ceph-mon[106899]: pgmap v803: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:53.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:53 smithi063 ceph-mon[141124]: pgmap v804: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:53.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:53 smithi111 ceph-mon[106899]: pgmap v804: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:55.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:55 smithi063 ceph-mon[141124]: pgmap v805: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:55 smithi111 ceph-mon[106899]: pgmap v805: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:56.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.441+0000 7fda28400700 1 -- 172.21.15.63:0/846607278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 msgr2=0x7fda201031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.441+0000 7fda28400700 1 --2- 172.21.15.63:0/846607278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda201031d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fda14004940 tx=0x7fda14009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.441+0000 7fda28400700 1 -- 172.21.15.63:0/846607278 shutdown_connections 2024-03-23T12:17:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.441+0000 7fda28400700 1 --2- 172.21.15.63:0/846607278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda201037a0 0x7fda200fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.441+0000 7fda28400700 1 --2- 172.21.15.63:0/846607278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda201031d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.441+0000 7fda28400700 1 -- 172.21.15.63:0/846607278 >> 172.21.15.63:0/846607278 conn(0x7fda200f89b0 msgr2=0x7fda200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.442+0000 7fda28400700 1 -- 172.21.15.63:0/846607278 shutdown_connections 2024-03-23T12:17:56.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.442+0000 7fda28400700 1 -- 172.21.15.63:0/846607278 wait complete. 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.442+0000 7fda28400700 1 Processor -- start 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda28400700 1 -- start start 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda28400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda20113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda28400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda201037a0 0x7fda20114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda28400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201146b0 con 0x7fda20102e00 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda28400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2010daa0 con 0x7fda201037a0 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda25a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda20113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda25a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda20113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49330/0 (socket says 172.21.15.63:49330) 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda25a00700 1 -- 172.21.15.63:0/3767097614 learned_addr learned my addr 172.21.15.63:0/3767097614 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.443+0000 7fda25000700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda201037a0 0x7fda20114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda25a00700 1 -- 172.21.15.63:0/3767097614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda201037a0 msgr2=0x7fda20114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda25a00700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda201037a0 0x7fda20114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda25a00700 1 -- 172.21.15.63:0/3767097614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda14005040 con 0x7fda20102e00 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda25a00700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda20113b20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fda1400b3b0 tx=0x7fda14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda1d600700 1 -- 172.21.15.63:0/3767097614 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda1400e040 con 0x7fda20102e00 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda1d600700 1 -- 172.21.15.63:0/3767097614 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda14013420 con 0x7fda20102e00 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2010dd20 con 0x7fda20102e00 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.444+0000 7fda1d600700 1 -- 172.21.15.63:0/3767097614 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda1401c610 con 0x7fda20102e00 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.445+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2010e2e0 con 0x7fda20102e00 2024-03-23T12:17:56.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.446+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda200fe550 con 0x7fda20102e00 2024-03-23T12:17:56.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.447+0000 7fda1d600700 1 -- 172.21.15.63:0/3767097614 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fda14013a00 con 0x7fda20102e00 2024-03-23T12:17:56.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.447+0000 7fda1d600700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda0806c7c0 0x7fda0806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:56.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.448+0000 7fda1d600700 1 -- 172.21.15.63:0/3767097614 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fda1408cb80 con 0x7fda20102e00 2024-03-23T12:17:56.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.448+0000 7fda25000700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda0806c7c0 0x7fda0806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:56.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.448+0000 7fda25000700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda0806c7c0 0x7fda0806ec70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda2010ec60 tx=0x7fda0c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:56.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.452+0000 7fda1d600700 1 -- 172.21.15.63:0/3767097614 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fda1405a890 con 0x7fda20102e00 2024-03-23T12:17:56.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.686+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda20109e30 con 0x7fda0806c7c0 2024-03-23T12:17:56.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.687+0000 7fda1d600700 1 -- 172.21.15.63:0/3767097614 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fda20109e30 con 0x7fda0806c7c0 2024-03-23T12:17:56.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda0806c7c0 msgr2=0x7fda0806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:56.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda0806c7c0 0x7fda0806ec70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda2010ec60 tx=0x7fda0c00b040 comp rx=0 tx=0).stop 2024-03-23T12:17:56.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 msgr2=0x7fda20113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:56.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda20113b20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fda1400b3b0 tx=0x7fda14009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:56.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 shutdown_connections 2024-03-23T12:17:56.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fda0806c7c0 0x7fda0806ec70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda20102e00 0x7fda20113b20 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 --2- 172.21.15.63:0/3767097614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda201037a0 0x7fda20114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 >> 172.21.15.63:0/3767097614 conn(0x7fda200f89b0 msgr2=0x7fda200f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:56.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 shutdown_connections 2024-03-23T12:17:56.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.690+0000 7fda28400700 1 -- 172.21.15.63:0/3767097614 wait complete. 2024-03-23T12:17:56.684 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 -- 172.21.15.63:0/803267573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 msgr2=0x7f5c301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 --2- 172.21.15.63:0/803267573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c301031d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5c24004940 tx=0x7f5c24009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 -- 172.21.15.63:0/803267573 shutdown_connections 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 --2- 172.21.15.63:0/803267573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 0x7f5c300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 --2- 172.21.15.63:0/803267573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c301031d0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 -- 172.21.15.63:0/803267573 >> 172.21.15.63:0/803267573 conn(0x7f5c300f89b0 msgr2=0x7f5c300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 -- 172.21.15.63:0/803267573 shutdown_connections 2024-03-23T12:17:56.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.835+0000 7f5c37800700 1 -- 172.21.15.63:0/803267573 wait complete. 2024-03-23T12:17:56.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.836+0000 7f5c37800700 1 Processor -- start 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.836+0000 7f5c37800700 1 -- start start 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.836+0000 7f5c37800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c30115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.836+0000 7f5c37800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 0x7f5c301161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.836+0000 7f5c37800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c30116810 con 0x7f5c30102e00 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.836+0000 7f5c37800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3010fbe0 con 0x7f5c301037a0 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.836+0000 7f5c34e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c30115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c34e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c30115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49344/0 (socket says 172.21.15.63:49344) 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c34e00700 1 -- 172.21.15.63:0/3328346375 learned_addr learned my addr 172.21.15.63:0/3328346375 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c2fe00700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 0x7f5c301161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c34e00700 1 -- 172.21.15.63:0/3328346375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 msgr2=0x7f5c301161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c34e00700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 0x7f5c301161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c34e00700 1 -- 172.21.15.63:0/3328346375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c24005040 con 0x7f5c30102e00 2024-03-23T12:17:56.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c2fe00700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 0x7f5c301161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:17:56.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c34e00700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c30115c80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009390 tx=0x7f5c24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:56.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c2cc00700 1 -- 172.21.15.63:0/3328346375 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c2400e040 con 0x7f5c30102e00 2024-03-23T12:17:56.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c2cc00700 1 -- 172.21.15.63:0/3328346375 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c24013420 con 0x7f5c30102e00 2024-03-23T12:17:56.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c2cc00700 1 -- 172.21.15.63:0/3328346375 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c2401c630 con 0x7f5c30102e00 2024-03-23T12:17:56.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.837+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3010fe60 con 0x7f5c30102e00 2024-03-23T12:17:56.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.838+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c30110420 con 0x7f5c30102e00 2024-03-23T12:17:56.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.839+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c300fe550 con 0x7f5c30102e00 2024-03-23T12:17:56.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.840+0000 7f5c2cc00700 1 -- 172.21.15.63:0/3328346375 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5c240139e0 con 0x7f5c30102e00 2024-03-23T12:17:56.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.840+0000 7f5c2cc00700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:56.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.841+0000 7f5c2cc00700 1 -- 172.21.15.63:0/3328346375 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5c2408cc90 con 0x7f5c30102e00 2024-03-23T12:17:56.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.841+0000 7f5c2fe00700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:56.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.842+0000 7f5c2fe00700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c007fd0 tx=0x7f5c1c009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:56.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:56.845+0000 7f5c2cc00700 1 -- 172.21.15.63:0/3328346375 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5c2405aa50 con 0x7f5c30102e00 2024-03-23T12:17:57.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.079+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c3004bf40 con 0x7f5c1806c7d0 2024-03-23T12:17:57.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.080+0000 7f5c2cc00700 1 -- 172.21.15.63:0/3328346375 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5c3004bf40 con 0x7f5c1806c7d0 2024-03-23T12:17:57.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 msgr2=0x7f5c1806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c007fd0 tx=0x7f5c1c009f60 comp rx=0 tx=0).stop 2024-03-23T12:17:57.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 msgr2=0x7f5c30115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c30115c80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009390 tx=0x7f5c24009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:57.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 shutdown_connections 2024-03-23T12:17:57.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c30115c80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 --2- 172.21.15.63:0/3328346375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 0x7f5c301161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 >> 172.21.15.63:0/3328346375 conn(0x7f5c300f89b0 msgr2=0x7f5c300f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:57.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 shutdown_connections 2024-03-23T12:17:57.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.083+0000 7f5c37800700 1 -- 172.21.15.63:0/3328346375 wait complete. 2024-03-23T12:17:57.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:57.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:17:57.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 -- 172.21.15.63:0/3292327022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fe850 msgr2=0x7fe008106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 --2- 172.21.15.63:0/3292327022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fe850 0x7fe008106f20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fdff4004970 tx=0x7fdff4009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:57.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 -- 172.21.15.63:0/3292327022 shutdown_connections 2024-03-23T12:17:57.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 --2- 172.21.15.63:0/3292327022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fe850 0x7fe008106f20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 --2- 172.21.15.63:0/3292327022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0080fdeb0 0x7fe0080fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 -- 172.21.15.63:0/3292327022 >> 172.21.15.63:0/3292327022 conn(0x7fe008072bc0 msgr2=0x7fe008072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:57.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 -- 172.21.15.63:0/3292327022 shutdown_connections 2024-03-23T12:17:57.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.220+0000 7fe00ec00700 1 -- 172.21.15.63:0/3292327022 wait complete. 2024-03-23T12:17:57.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.221+0000 7fe00ec00700 1 Processor -- start 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.221+0000 7fe00ec00700 1 -- start start 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe00ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fdeb0 0x7fe008113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe00ec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0080fe850 0x7fe008113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe00ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0081145a0 con 0x7fe0080fdeb0 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe00ec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00810d990 con 0x7fe0080fe850 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe007e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fdeb0 0x7fe008113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe007e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fdeb0 0x7fe008113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49360/0 (socket says 172.21.15.63:49360) 2024-03-23T12:17:57.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe007e00700 1 -- 172.21.15.63:0/2808767887 learned_addr learned my addr 172.21.15.63:0/2808767887 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:57.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe007400700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0080fe850 0x7fe008113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:57.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe007e00700 1 -- 172.21.15.63:0/2808767887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0080fe850 msgr2=0x7fe008113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.222+0000 7fe007e00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0080fe850 0x7fe008113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fe007e00700 1 -- 172.21.15.63:0/2808767887 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff4005040 con 0x7fe0080fdeb0 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fe007400700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0080fe850 0x7fe008113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fe007e00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fdeb0 0x7fe008113a10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdff800c8c0 tx=0x7fdff80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fdfffe00700 1 -- 172.21.15.63:0/2808767887 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdff800a980 con 0x7fe0080fdeb0 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fdfffe00700 1 -- 172.21.15.63:0/2808767887 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdff800ce60 con 0x7fe0080fdeb0 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fdfffe00700 1 -- 172.21.15.63:0/2808767887 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdff80113a0 con 0x7fe0080fdeb0 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00810dc70 con 0x7fe0080fdeb0 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.223+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00810e1b0 con 0x7fe0080fdeb0 2024-03-23T12:17:57.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.224+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe008065e10 con 0x7fe0080fdeb0 2024-03-23T12:17:57.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.225+0000 7fdfffe00700 1 -- 172.21.15.63:0/2808767887 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdff8012400 con 0x7fe0080fdeb0 2024-03-23T12:17:57.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.226+0000 7fdfffe00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfec06c7d0 0x7fdfec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:57.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.226+0000 7fdfffe00700 1 -- 172.21.15.63:0/2808767887 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdff808bb30 con 0x7fe0080fdeb0 2024-03-23T12:17:57.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.226+0000 7fe007400700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfec06c7d0 0x7fdfec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:57.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.227+0000 7fe007400700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfec06c7d0 0x7fdfec06ec80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009fd0 tx=0x7fdff4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:57.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.231+0000 7fdfffe00700 1 -- 172.21.15.63:0/2808767887 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdff8059840 con 0x7fe0080fdeb0 2024-03-23T12:17:57.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.456+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe008109e90 con 0x7fdfec06c7d0 2024-03-23T12:17:57.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.466+0000 7fdfffe00700 1 -- 172.21.15.63:0/2808767887 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fe008109e90 con 0x7fdfec06c7d0 2024-03-23T12:17:57.445 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:17:57.445 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (27m) 4m ago 40m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (24m) 4m ago 40m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (24m) 4m ago 39m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (26m) 4m ago 40m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (20m) 4m ago 34m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (20m) 4m ago 33m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (4m) 4m ago 33m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (19m) 4m ago 34m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (29m) 4m ago 41m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (28m) 4m ago 39m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (25m) 4m ago 42m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (25m) 4m ago 39m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (28m) 4m ago 40m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (27m) 4m ago 39m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (23m) 4m ago 38m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (23m) 4m ago 37m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:17:57.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (22m) 4m ago 37m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:17:57.447 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (21m) 4m ago 36m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:17:57.447 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (21m) 4m ago 36m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:17:57.447 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (20m) 4m ago 35m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:17:57.447 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (27m) 4m ago 39m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfec06c7d0 msgr2=0x7fdfec06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfec06c7d0 0x7fdfec06ec80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009fd0 tx=0x7fdff4009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fdeb0 msgr2=0x7fe008113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fdeb0 0x7fe008113a10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdff800c8c0 tx=0x7fdff80044c0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 shutdown_connections 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfec06c7d0 0x7fdfec06ec80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0080fdeb0 0x7fe008113a10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 --2- 172.21.15.63:0/2808767887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0080fe850 0x7fe008113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 >> 172.21.15.63:0/2808767887 conn(0x7fe008072bc0 msgr2=0x7fe008103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:57.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 shutdown_connections 2024-03-23T12:17:57.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.468+0000 7fe00ec00700 1 -- 172.21.15.63:0/2808767887 wait complete. 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 -- 172.21.15.63:0/1649595471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fe830 msgr2=0x7f9fc4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 --2- 172.21.15.63:0/1649595471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fe830 0x7f9fc4106f00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0004940 tx=0x7f9fb0009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 -- 172.21.15.63:0/1649595471 shutdown_connections 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 --2- 172.21.15.63:0/1649595471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fe830 0x7f9fc4106f00 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 --2- 172.21.15.63:0/1649595471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc40fde90 0x7f9fc40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 -- 172.21.15.63:0/1649595471 >> 172.21.15.63:0/1649595471 conn(0x7f9fc4072bc0 msgr2=0x7f9fc4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 -- 172.21.15.63:0/1649595471 shutdown_connections 2024-03-23T12:17:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.604+0000 7f9fc9000700 1 -- 172.21.15.63:0/1649595471 wait complete. 2024-03-23T12:17:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.605+0000 7f9fc9000700 1 Processor -- start 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.605+0000 7f9fc9000700 1 -- start start 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.605+0000 7f9fc9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fde90 0x7f9fc4115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.605+0000 7f9fc9000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc40fe830 0x7f9fc4116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.605+0000 7f9fc9000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc41168c0 con 0x7f9fc40fde90 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.605+0000 7f9fc9000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc410fc90 con 0x7f9fc40fe830 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fde90 0x7f9fc4115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fde90 0x7f9fc4115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49372/0 (socket says 172.21.15.63:49372) 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1e00700 1 -- 172.21.15.63:0/2291763066 learned_addr learned my addr 172.21.15.63:0/2291763066 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1400700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc40fe830 0x7f9fc4116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1e00700 1 -- 172.21.15.63:0/2291763066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc40fe830 msgr2=0x7f9fc4116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1e00700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc40fe830 0x7f9fc4116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1e00700 1 -- 172.21.15.63:0/2291763066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb0005040 con 0x7f9fc40fde90 2024-03-23T12:17:57.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc1e00700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fde90 0x7f9fc4115d30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4002a50 tx=0x7f9fb40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:57.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fba000700 1 -- 172.21.15.63:0/2291763066 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fb4007cc0 con 0x7f9fc40fde90 2024-03-23T12:17:57.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fba000700 1 -- 172.21.15.63:0/2291763066 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fb400f040 con 0x7f9fc40fde90 2024-03-23T12:17:57.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc410ff70 con 0x7f9fc40fde90 2024-03-23T12:17:57.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.606+0000 7f9fba000700 1 -- 172.21.15.63:0/2291763066 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fb4009540 con 0x7f9fc40fde90 2024-03-23T12:17:57.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.607+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc41104b0 con 0x7f9fc40fde90 2024-03-23T12:17:57.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.608+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc4106a20 con 0x7f9fc40fde90 2024-03-23T12:17:57.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.609+0000 7f9fba000700 1 -- 172.21.15.63:0/2291763066 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9fb40096a0 con 0x7f9fc40fde90 2024-03-23T12:17:57.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.609+0000 7f9fba000700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9fa806c7d0 0x7f9fa806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:57.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.610+0000 7f9fba000700 1 -- 172.21.15.63:0/2291763066 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9fb408b920 con 0x7f9fc40fde90 2024-03-23T12:17:57.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.610+0000 7f9fc1400700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9fa806c7d0 0x7f9fa806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:57.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.610+0000 7f9fc1400700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9fa806c7d0 0x7f9fa806ec80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9fc41115d0 tx=0x7f9fb0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:57.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.614+0000 7f9fba000700 1 -- 172.21.15.63:0/2291763066 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9fb4059530 con 0x7f9fc40fde90 2024-03-23T12:17:57.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.917+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9fc404bed0 con 0x7f9fc40fde90 2024-03-23T12:17:57.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.918+0000 7f9fba000700 1 -- 172.21.15.63:0/2291763066 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f9fb40590c0 con 0x7f9fc40fde90 2024-03-23T12:17:57.897 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:17:57.897 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:17:57.897 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:17:57.898 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:17:57.899 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:17:57.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9fa806c7d0 msgr2=0x7f9fa806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9fa806c7d0 0x7f9fa806ec80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9fc41115d0 tx=0x7f9fb0009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fde90 msgr2=0x7f9fc4115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fde90 0x7f9fc4115d30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4002a50 tx=0x7f9fb40046b0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 shutdown_connections 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9fa806c7d0 0x7f9fa806ec80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fc40fde90 0x7f9fc4115d30 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 --2- 172.21.15.63:0/2291763066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc40fe830 0x7f9fc4116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 >> 172.21.15.63:0/2291763066 conn(0x7f9fc4072bc0 msgr2=0x7f9fc40fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 shutdown_connections 2024-03-23T12:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:57.920+0000 7f9fc9000700 1 -- 172.21.15.63:0/2291763066 wait complete. 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 -- 172.21.15.63:0/2518053463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 msgr2=0x7ff3301031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 --2- 172.21.15.63:0/2518053463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff3301031f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff320004940 tx=0x7ff320009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 -- 172.21.15.63:0/2518053463 shutdown_connections 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 --2- 172.21.15.63:0/2518053463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3301037c0 0x7ff3300fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 --2- 172.21.15.63:0/2518053463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff3301031f0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 -- 172.21.15.63:0/2518053463 >> 172.21.15.63:0/2518053463 conn(0x7ff3300f89d0 msgr2=0x7ff3300fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 -- 172.21.15.63:0/2518053463 shutdown_connections 2024-03-23T12:17:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.058+0000 7ff336e00700 1 -- 172.21.15.63:0/2518053463 wait complete. 2024-03-23T12:17:58.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.059+0000 7ff336e00700 1 Processor -- start 2024-03-23T12:17:58.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.059+0000 7ff336e00700 1 -- start start 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.059+0000 7ff336e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff330115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.059+0000 7ff336e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3301037c0 0x7ff330116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.059+0000 7ff336e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3301168a0 con 0x7ff330102e20 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.059+0000 7ff336e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33010fc70 con 0x7ff3301037c0 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.059+0000 7ff32fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff330115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff32fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff330115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49396/0 (socket says 172.21.15.63:49396) 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff32fe00700 1 -- 172.21.15.63:0/1938695916 learned_addr learned my addr 172.21.15.63:0/1938695916 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff32f400700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3301037c0 0x7ff330116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff32fe00700 1 -- 172.21.15.63:0/1938695916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3301037c0 msgr2=0x7ff330116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff32fe00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3301037c0 0x7ff330116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff32fe00700 1 -- 172.21.15.63:0/1938695916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320005040 con 0x7ff330102e20 2024-03-23T12:17:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff32fe00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff330115d10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff320009180 tx=0x7ff320009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff327e00700 1 -- 172.21.15.63:0/1938695916 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32000e040 con 0x7ff330102e20 2024-03-23T12:17:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.060+0000 7ff327e00700 1 -- 172.21.15.63:0/1938695916 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff320004290 con 0x7ff330102e20 2024-03-23T12:17:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.061+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33010fef0 con 0x7ff330102e20 2024-03-23T12:17:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.061+0000 7ff327e00700 1 -- 172.21.15.63:0/1938695916 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32001c3f0 con 0x7ff330102e20 2024-03-23T12:17:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.061+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3301104b0 con 0x7ff330102e20 2024-03-23T12:17:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.062+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3300fe570 con 0x7ff330102e20 2024-03-23T12:17:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.063+0000 7ff327e00700 1 -- 172.21.15.63:0/1938695916 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff320013790 con 0x7ff330102e20 2024-03-23T12:17:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.063+0000 7ff327e00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff31406c7c0 0x7ff31406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.064+0000 7ff327e00700 1 -- 172.21.15.63:0/1938695916 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff32008cd30 con 0x7ff330102e20 2024-03-23T12:17:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.064+0000 7ff32f400700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff31406c7c0 0x7ff31406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.064+0000 7ff32f400700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff31406c7c0 0x7ff31406ec70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009fd0 tx=0x7ff31c009f20 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:58.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.068+0000 7ff327e00700 1 -- 172.21.15.63:0/1938695916 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff32005aa40 con 0x7ff330102e20 2024-03-23T12:17:58.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:57 smithi063 ceph-mon[141124]: from='client.35474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:58.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:57 smithi063 ceph-mon[141124]: pgmap v806: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:58.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:57 smithi063 ceph-mon[141124]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:58.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:57 smithi111 ceph-mon[106899]: from='client.35474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:58.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:57 smithi111 ceph-mon[106899]: pgmap v806: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:17:58.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:57 smithi111 ceph-mon[106899]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.329+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff3300fe570 con 0x7ff330102e20 2024-03-23T12:17:58.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.329+0000 7ff327e00700 1 -- 172.21.15.63:0/1938695916 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7ff320013420 con 0x7ff330102e20 2024-03-23T12:17:58.309 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:17:58.309 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:17:58.309 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:17:58.309 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:17:58.309 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:17:58.310 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:17:58.311 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:58.312 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff31406c7c0 msgr2=0x7ff31406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff31406c7c0 0x7ff31406ec70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009fd0 tx=0x7ff31c009f20 comp rx=0 tx=0).stop 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 msgr2=0x7ff330115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff330115d10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff320009180 tx=0x7ff320009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 shutdown_connections 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff31406c7c0 0x7ff31406ec70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff330102e20 0x7ff330115d10 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 --2- 172.21.15.63:0/1938695916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3301037c0 0x7ff330116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 >> 172.21.15.63:0/1938695916 conn(0x7ff3300f89d0 msgr2=0x7ff3300f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:58.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 shutdown_connections 2024-03-23T12:17:58.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.332+0000 7ff336e00700 1 -- 172.21.15.63:0/1938695916 wait complete. 2024-03-23T12:17:58.314 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:17:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.468+0000 7ff74ea00700 1 -- 172.21.15.63:0/470228672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fdeb0 msgr2=0x7ff7480fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.468+0000 7ff74ea00700 1 --2- 172.21.15.63:0/470228672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fdeb0 0x7ff7480fe280 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ff738004970 tx=0x7ff738009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.468+0000 7ff74ea00700 1 -- 172.21.15.63:0/470228672 shutdown_connections 2024-03-23T12:17:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.468+0000 7ff74ea00700 1 --2- 172.21.15.63:0/470228672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7480fe850 0x7ff748106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.468+0000 7ff74ea00700 1 --2- 172.21.15.63:0/470228672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fdeb0 0x7ff7480fe280 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.468+0000 7ff74ea00700 1 -- 172.21.15.63:0/470228672 >> 172.21.15.63:0/470228672 conn(0x7ff748072bc0 msgr2=0x7ff748072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.469+0000 7ff74ea00700 1 -- 172.21.15.63:0/470228672 shutdown_connections 2024-03-23T12:17:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.469+0000 7ff74ea00700 1 -- 172.21.15.63:0/470228672 wait complete. 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.469+0000 7ff74ea00700 1 Processor -- start 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff74ea00700 1 -- start start 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff74ea00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7480fdeb0 0x7ff748115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff74ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fe850 0x7ff748116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff74ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff748116780 con 0x7ff7480fe850 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff74ea00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74810fb90 con 0x7ff7480fdeb0 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff747400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fe850 0x7ff748116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff747400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fe850 0x7ff748116130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49428/0 (socket says 172.21.15.63:49428) 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff747400700 1 -- 172.21.15.63:0/1578711346 learned_addr learned my addr 172.21.15.63:0/1578711346 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.470+0000 7ff747e00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7480fdeb0 0x7ff748115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff747400700 1 -- 172.21.15.63:0/1578711346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7480fdeb0 msgr2=0x7ff748115bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff747400700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7480fdeb0 0x7ff748115bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff747400700 1 -- 172.21.15.63:0/1578711346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff738005040 con 0x7ff7480fe850 2024-03-23T12:17:58.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff747400700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fe850 0x7ff748116130 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff734002a50 tx=0x7ff7340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:58.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff73fe00700 1 -- 172.21.15.63:0/1578711346 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff734007cc0 con 0x7ff7480fe850 2024-03-23T12:17:58.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff73fe00700 1 -- 172.21.15.63:0/1578711346 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff73400f040 con 0x7ff7480fe850 2024-03-23T12:17:58.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff73fe00700 1 -- 172.21.15.63:0/1578711346 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff73400d9c0 con 0x7ff7480fe850 2024-03-23T12:17:58.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74810fe70 con 0x7ff7480fe850 2024-03-23T12:17:58.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.471+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7481103b0 con 0x7ff7480fe850 2024-03-23T12:17:58.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.472+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7481069f0 con 0x7ff7480fe850 2024-03-23T12:17:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.473+0000 7ff73fe00700 1 -- 172.21.15.63:0/1578711346 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff73400db20 con 0x7ff7480fe850 2024-03-23T12:17:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.474+0000 7ff73fe00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff72c06c7d0 0x7ff72c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.474+0000 7ff73fe00700 1 -- 172.21.15.63:0/1578711346 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff73408b760 con 0x7ff7480fe850 2024-03-23T12:17:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.475+0000 7ff747e00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff72c06c7d0 0x7ff72c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.475+0000 7ff747e00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff72c06c7d0 0x7ff72c06ec80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff73800b370 tx=0x7ff738009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.479+0000 7ff73fe00700 1 -- 172.21.15.63:0/1578711346 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff734055960 con 0x7ff7480fe850 2024-03-23T12:17:58.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.709+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7481112f0 con 0x7ff72c06c7d0 2024-03-23T12:17:58.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.710+0000 7ff73fe00700 1 -- 172.21.15.63:0/1578711346 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff7481112f0 con 0x7ff72c06c7d0 2024-03-23T12:17:58.690 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:17:58.690 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:17:58.691 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:17:58.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.712+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff72c06c7d0 msgr2=0x7ff72c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.712+0000 7ff74ea00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff72c06c7d0 0x7ff72c06ec80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff73800b370 tx=0x7ff738009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:58.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.712+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fe850 msgr2=0x7ff748116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.712+0000 7ff74ea00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fe850 0x7ff748116130 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff734002a50 tx=0x7ff7340046b0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.713+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 shutdown_connections 2024-03-23T12:17:58.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.713+0000 7ff74ea00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff72c06c7d0 0x7ff72c06ec80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.713+0000 7ff74ea00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7480fdeb0 0x7ff748115bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.713+0000 7ff74ea00700 1 --2- 172.21.15.63:0/1578711346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7480fe850 0x7ff748116130 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.713+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 >> 172.21.15.63:0/1578711346 conn(0x7ff748072bc0 msgr2=0x7ff7480fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:58.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.713+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 shutdown_connections 2024-03-23T12:17:58.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.713+0000 7ff74ea00700 1 -- 172.21.15.63:0/1578711346 wait complete. 2024-03-23T12:17:58.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.849+0000 7fdd33400700 1 -- 172.21.15.63:0/2284580858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fe830 msgr2=0x7fdd2c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.849+0000 7fdd33400700 1 --2- 172.21.15.63:0/2284580858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fe830 0x7fdd2c106f00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fdd18004940 tx=0x7fdd18009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:58.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.850+0000 7fdd33400700 1 -- 172.21.15.63:0/2284580858 shutdown_connections 2024-03-23T12:17:58.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.850+0000 7fdd33400700 1 --2- 172.21.15.63:0/2284580858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fe830 0x7fdd2c106f00 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.850+0000 7fdd33400700 1 --2- 172.21.15.63:0/2284580858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0fde90 0x7fdd2c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.850+0000 7fdd33400700 1 -- 172.21.15.63:0/2284580858 >> 172.21.15.63:0/2284580858 conn(0x7fdd2c072bc0 msgr2=0x7fdd2c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:58.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.850+0000 7fdd33400700 1 -- 172.21.15.63:0/2284580858 shutdown_connections 2024-03-23T12:17:58.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.850+0000 7fdd33400700 1 -- 172.21.15.63:0/2284580858 wait complete. 2024-03-23T12:17:58.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.850+0000 7fdd33400700 1 Processor -- start 2024-03-23T12:17:58.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd33400700 1 -- start start 2024-03-23T12:17:58.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd33400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fde90 0x7fdd2c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd33400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0fe830 0x7fdd2c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd33400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c116800 con 0x7fdd2c0fde90 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd33400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c10fbd0 con 0x7fdd2c0fe830 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd30a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fde90 0x7fdd2c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd30a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fde90 0x7fdd2c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49446/0 (socket says 172.21.15.63:49446) 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.851+0000 7fdd30a00700 1 -- 172.21.15.63:0/983978 learned_addr learned my addr 172.21.15.63:0/983978 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd2be00700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0fe830 0x7fdd2c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd30a00700 1 -- 172.21.15.63:0/983978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0fe830 msgr2=0x7fdd2c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd30a00700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0fe830 0x7fdd2c1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:58.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd30a00700 1 -- 172.21.15.63:0/983978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd18005040 con 0x7fdd2c0fde90 2024-03-23T12:17:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd30a00700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fde90 0x7fdd2c115c70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000eb40 tx=0x7fdd2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd28c00700 1 -- 172.21.15.63:0/983978 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd20007cc0 con 0x7fdd2c0fde90 2024-03-23T12:17:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd28c00700 1 -- 172.21.15.63:0/983978 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd2000f040 con 0x7fdd2c0fde90 2024-03-23T12:17:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd28c00700 1 -- 172.21.15.63:0/983978 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd20009570 con 0x7fdd2c0fde90 2024-03-23T12:17:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.852+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c10fe50 con 0x7fdd2c0fde90 2024-03-23T12:17:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.853+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c110390 con 0x7fdd2c0fde90 2024-03-23T12:17:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.853+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c106a20 con 0x7fdd2c0fde90 2024-03-23T12:17:58.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.854+0000 7fdd28c00700 1 -- 172.21.15.63:0/983978 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdd200096d0 con 0x7fdd2c0fde90 2024-03-23T12:17:58.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.855+0000 7fdd28c00700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd1406c7d0 0x7fdd1406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:17:58.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.855+0000 7fdd28c00700 1 -- 172.21.15.63:0/983978 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdd2008b6f0 con 0x7fdd2c0fde90 2024-03-23T12:17:58.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.856+0000 7fdd2be00700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd1406c7d0 0x7fdd1406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:17:58.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.856+0000 7fdd2be00700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd1406c7d0 0x7fdd1406ec80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b0d0 tx=0x7fdd18009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:17:58.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:58.860+0000 7fdd28c00700 1 -- 172.21.15.63:0/983978 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdd20059430 con 0x7fdd2c0fde90 2024-03-23T12:17:59.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:58 smithi063 ceph-mon[141124]: from='client.35482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:59.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:58 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2291763066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:17:59.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:58 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1938695916' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:17:59.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.159+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdd2c04bed0 con 0x7fdd2c0fde90 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.159+0000 7fdd28c00700 1 -- 172.21.15.63:0/983978 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fdd20019e10 con 0x7fdd2c0fde90 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:17:59.139 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:17:59.140 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:17:59.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd1406c7d0 msgr2=0x7fdd1406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:59.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd1406c7d0 0x7fdd1406ec80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b0d0 tx=0x7fdd18009e30 comp rx=0 tx=0).stop 2024-03-23T12:17:59.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fde90 msgr2=0x7fdd2c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fde90 0x7fdd2c115c70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000eb40 tx=0x7fdd2000b040 comp rx=0 tx=0).stop 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 shutdown_connections 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdd1406c7d0 0x7fdd1406ec80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd2c0fde90 0x7fdd2c115c70 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 --2- 172.21.15.63:0/983978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0fe830 0x7fdd2c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 >> 172.21.15.63:0/983978 conn(0x7fdd2c072bc0 msgr2=0x7fdd2c0fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 shutdown_connections 2024-03-23T12:17:59.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:17:59.162+0000 7fdd33400700 1 -- 172.21.15.63:0/983978 wait complete. 2024-03-23T12:17:59.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:58 smithi111 ceph-mon[106899]: from='client.35482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:17:59.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:58 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2291763066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:17:59.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:58 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1938695916' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:18:00.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:59 smithi063 ceph-mon[141124]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:00.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:59 smithi063 ceph-mon[141124]: pgmap v807: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:00.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:17:59 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/983978' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:18:00.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:59 smithi111 ceph-mon[106899]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:00.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:59 smithi111 ceph-mon[106899]: pgmap v807: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:00.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:17:59 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/983978' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:18:02.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:01 smithi063 ceph-mon[141124]: pgmap v808: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:02.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:01 smithi111 ceph-mon[106899]: pgmap v808: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:03.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:03 smithi063 ceph-mon[141124]: pgmap v809: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:03.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:03 smithi111 ceph-mon[106899]: pgmap v809: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:05.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:05 smithi063 ceph-mon[141124]: pgmap v810: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:05.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:05 smithi111 ceph-mon[106899]: pgmap v810: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:07.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:07 smithi063 ceph-mon[141124]: pgmap v811: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:07.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:07 smithi111 ceph-mon[106899]: pgmap v811: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:09.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:09 smithi063 ceph-mon[141124]: pgmap v812: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:09.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:09 smithi111 ceph-mon[106899]: pgmap v812: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:11 smithi063 ceph-mon[141124]: pgmap v813: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:12.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:12.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:11 smithi111 ceph-mon[106899]: pgmap v813: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:13.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:13 smithi063 ceph-mon[141124]: pgmap v814: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:13.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:13 smithi111 ceph-mon[106899]: pgmap v814: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:15.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:15 smithi063 ceph-mon[141124]: pgmap v815: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:15.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:15 smithi111 ceph-mon[106899]: pgmap v815: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:17.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:17 smithi063 ceph-mon[141124]: pgmap v816: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:17.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:17 smithi111 ceph-mon[106899]: pgmap v816: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:19.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:19 smithi063 ceph-mon[141124]: pgmap v817: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:19.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:19 smithi111 ceph-mon[106899]: pgmap v817: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:22.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:21 smithi063 ceph-mon[141124]: pgmap v818: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:22.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:21 smithi111 ceph-mon[106899]: pgmap v818: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:23.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:23 smithi063 ceph-mon[141124]: pgmap v819: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:23.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:23 smithi111 ceph-mon[106899]: pgmap v819: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:25.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:25 smithi063 ceph-mon[141124]: pgmap v820: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:25.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:25 smithi111 ceph-mon[106899]: pgmap v820: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:27.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:27.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:28.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:27 smithi063 ceph-mon[141124]: pgmap v821: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:28.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:27 smithi111 ceph-mon[106899]: pgmap v821: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 -- 172.21.15.63:0/761344524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d4102e00 msgr2=0x7f86d41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 --2- 172.21.15.63:0/761344524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d4102e00 0x7f86d41031d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f86c4004940 tx=0x7f86c4009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 -- 172.21.15.63:0/761344524 shutdown_connections 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 --2- 172.21.15.63:0/761344524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d41037a0 0x7f86d40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 --2- 172.21.15.63:0/761344524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d4102e00 0x7f86d41031d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 -- 172.21.15.63:0/761344524 >> 172.21.15.63:0/761344524 conn(0x7f86d40f89b0 msgr2=0x7f86d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 -- 172.21.15.63:0/761344524 shutdown_connections 2024-03-23T12:18:29.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.316+0000 7f86daa00700 1 -- 172.21.15.63:0/761344524 wait complete. 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.317+0000 7f86daa00700 1 Processor -- start 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.317+0000 7f86daa00700 1 -- start start 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.317+0000 7f86daa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d4102e00 0x7f86d4115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.317+0000 7f86daa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d41037a0 0x7f86d4116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86daa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4116880 con 0x7f86d41037a0 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86daa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d410fc50 con 0x7f86d4102e00 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86cbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d41037a0 0x7f86d4116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86cbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d41037a0 0x7f86d4116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53988/0 (socket says 172.21.15.63:53988) 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86cbe00700 1 -- 172.21.15.63:0/1674972180 learned_addr learned my addr 172.21.15.63:0/1674972180 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:18:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86d3e00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d4102e00 0x7f86d4115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:29.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86cbe00700 1 -- 172.21.15.63:0/1674972180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d4102e00 msgr2=0x7f86d4115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86cbe00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d4102e00 0x7f86d4115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.318+0000 7f86cbe00700 1 -- 172.21.15.63:0/1674972180 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4005040 con 0x7f86d41037a0 2024-03-23T12:18:29.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.319+0000 7f86cbe00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d41037a0 0x7f86d4116230 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f86bc002a80 tx=0x7f86bc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:29.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.319+0000 7f86d0c00700 1 -- 172.21.15.63:0/1674972180 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86bc004e80 con 0x7f86d41037a0 2024-03-23T12:18:29.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.319+0000 7f86d0c00700 1 -- 172.21.15.63:0/1674972180 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86bc00f040 con 0x7f86d41037a0 2024-03-23T12:18:29.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.319+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d410ff30 con 0x7f86d41037a0 2024-03-23T12:18:29.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.319+0000 7f86d0c00700 1 -- 172.21.15.63:0/1674972180 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86bc00da00 con 0x7f86d41037a0 2024-03-23T12:18:29.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.319+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d41104f0 con 0x7f86d41037a0 2024-03-23T12:18:29.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.320+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d40fe550 con 0x7f86d41037a0 2024-03-23T12:18:29.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.321+0000 7f86d0c00700 1 -- 172.21.15.63:0/1674972180 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f86bc007cc0 con 0x7f86d41037a0 2024-03-23T12:18:29.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.322+0000 7f86d0c00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86b806c7d0 0x7f86b806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:29.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.322+0000 7f86d0c00700 1 -- 172.21.15.63:0/1674972180 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f86bc08b0f0 con 0x7f86d41037a0 2024-03-23T12:18:29.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.322+0000 7f86d3e00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86b806c7d0 0x7f86b806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:29.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.323+0000 7f86d3e00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86b806c7d0 0x7f86b806ec80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009fd0 tx=0x7f86c4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:29.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.327+0000 7f86d0c00700 1 -- 172.21.15.63:0/1674972180 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f86bc058d80 con 0x7f86d41037a0 2024-03-23T12:18:29.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:29 smithi063 ceph-mon[141124]: pgmap v822: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:29.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:29 smithi111 ceph-mon[106899]: pgmap v822: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:29.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.557+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f86d4109de0 con 0x7f86b806c7d0 2024-03-23T12:18:29.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.559+0000 7f86d0c00700 1 -- 172.21.15.63:0/1674972180 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f86d4109de0 con 0x7f86b806c7d0 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86b806c7d0 msgr2=0x7f86b806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86b806c7d0 0x7f86b806ec80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009fd0 tx=0x7f86c4009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d41037a0 msgr2=0x7f86d4116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d41037a0 0x7f86d4116230 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f86bc002a80 tx=0x7f86bc004630 comp rx=0 tx=0).stop 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 shutdown_connections 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f86b806c7d0 0x7f86b806ec80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d4102e00 0x7f86d4115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 --2- 172.21.15.63:0/1674972180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d41037a0 0x7f86d4116230 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 >> 172.21.15.63:0/1674972180 conn(0x7f86d40f89b0 msgr2=0x7f86d40f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 shutdown_connections 2024-03-23T12:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.561+0000 7f86daa00700 1 -- 172.21.15.63:0/1674972180 wait complete. 2024-03-23T12:18:29.555 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 -- 172.21.15.63:0/3245413552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 msgr2=0x7fce4c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 --2- 172.21.15.63:0/3245413552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c0fcc50 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fce34004940 tx=0x7fce34009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 -- 172.21.15.63:0/3245413552 shutdown_connections 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 --2- 172.21.15.63:0/3245413552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c0fcc50 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 --2- 172.21.15.63:0/3245413552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4c102c40 0x7fce4c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 -- 172.21.15.63:0/3245413552 >> 172.21.15.63:0/3245413552 conn(0x7fce4c0f8830 msgr2=0x7fce4c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 -- 172.21.15.63:0/3245413552 shutdown_connections 2024-03-23T12:18:29.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.700+0000 7fce51400700 1 -- 172.21.15.63:0/3245413552 wait complete. 2024-03-23T12:18:29.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.701+0000 7fce51400700 1 Processor -- start 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.701+0000 7fce51400700 1 -- start start 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce51400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4c102c40 0x7fce4c115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce51400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce51400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4c1165e0 con 0x7fce4c1035e0 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce51400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4c10f9d0 con 0x7fce4c102c40 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce43e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce43e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54010/0 (socket says 172.21.15.63:54010) 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce43e00700 1 -- 172.21.15.63:0/78625202 learned_addr learned my addr 172.21.15.63:0/78625202 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce4be00700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4c102c40 0x7fce4c115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce43e00700 1 -- 172.21.15.63:0/78625202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4c102c40 msgr2=0x7fce4c115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce43e00700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4c102c40 0x7fce4c115a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.702+0000 7fce43e00700 1 -- 172.21.15.63:0/78625202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34005040 con 0x7fce4c1035e0 2024-03-23T12:18:29.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.703+0000 7fce43e00700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c115f90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fce34009390 tx=0x7fce34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:29.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.703+0000 7fce48c00700 1 -- 172.21.15.63:0/78625202 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3400e040 con 0x7fce4c1035e0 2024-03-23T12:18:29.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.703+0000 7fce48c00700 1 -- 172.21.15.63:0/78625202 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce34004290 con 0x7fce4c1035e0 2024-03-23T12:18:29.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.703+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4c10fc50 con 0x7fce4c1035e0 2024-03-23T12:18:29.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.703+0000 7fce48c00700 1 -- 172.21.15.63:0/78625202 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce34013e70 con 0x7fce4c1035e0 2024-03-23T12:18:29.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.703+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4c110210 con 0x7fce4c1035e0 2024-03-23T12:18:29.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.704+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4c0fe390 con 0x7fce4c1035e0 2024-03-23T12:18:29.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.705+0000 7fce48c00700 1 -- 172.21.15.63:0/78625202 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fce34013690 con 0x7fce4c1035e0 2024-03-23T12:18:29.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.706+0000 7fce48c00700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce3006c7d0 0x7fce3006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:29.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.706+0000 7fce48c00700 1 -- 172.21.15.63:0/78625202 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fce3408cc20 con 0x7fce4c1035e0 2024-03-23T12:18:29.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.706+0000 7fce4be00700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce3006c7d0 0x7fce3006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:29.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.707+0000 7fce4be00700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce3006c7d0 0x7fce3006ec80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009fd0 tx=0x7fce3c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:29.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.710+0000 7fce48c00700 1 -- 172.21.15.63:0/78625202 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fce3405a9e0 con 0x7fce4c1035e0 2024-03-23T12:18:29.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.942+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce4c109bd0 con 0x7fce3006c7d0 2024-03-23T12:18:29.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.943+0000 7fce48c00700 1 -- 172.21.15.63:0/78625202 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fce4c109bd0 con 0x7fce3006c7d0 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce3006c7d0 msgr2=0x7fce3006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce3006c7d0 0x7fce3006ec80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009fd0 tx=0x7fce3c00b040 comp rx=0 tx=0).stop 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 msgr2=0x7fce4c115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c115f90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fce34009390 tx=0x7fce34009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 shutdown_connections 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fce3006c7d0 0x7fce3006ec80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4c102c40 0x7fce4c115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 --2- 172.21.15.63:0/78625202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4c1035e0 0x7fce4c115f90 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 >> 172.21.15.63:0/78625202 conn(0x7fce4c0f8830 msgr2=0x7fce4c0f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:29.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 shutdown_connections 2024-03-23T12:18:29.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:29.945+0000 7fce51400700 1 -- 172.21.15.63:0/78625202 wait complete. 2024-03-23T12:18:30.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 -- 172.21.15.63:0/3266152911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 msgr2=0x7fb254070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 --2- 172.21.15.63:0/3266152911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb254070360 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb248004970 tx=0x7fb248009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:30.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 -- 172.21.15.63:0/3266152911 shutdown_connections 2024-03-23T12:18:30.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 --2- 172.21.15.63:0/3266152911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540708a0 0x7fb254070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 --2- 172.21.15.63:0/3266152911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb254070360 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 -- 172.21.15.63:0/3266152911 >> 172.21.15.63:0/3266152911 conn(0x7fb2540f89b0 msgr2=0x7fb2540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:30.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 -- 172.21.15.63:0/3266152911 shutdown_connections 2024-03-23T12:18:30.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.082+0000 7fb25c400700 1 -- 172.21.15.63:0/3266152911 wait complete. 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.083+0000 7fb25c400700 1 Processor -- start 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.083+0000 7fb25c400700 1 -- start start 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb25c400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540708a0 0x7fb254119e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb25c400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb25411a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb25c400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25411aa20 con 0x7fb254071f90 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb25c400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb254114000 con 0x7fb2540708a0 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb25411a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540708a0 0x7fb254119e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb25411a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54026/0 (socket says 172.21.15.63:54026) 2024-03-23T12:18:30.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259000700 1 -- 172.21.15.63:0/4221905096 learned_addr learned my addr 172.21.15.63:0/4221905096 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:18:30.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259a00700 1 -- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 msgr2=0x7fb25411a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259a00700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb25411a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259a00700 1 -- 172.21.15.63:0/4221905096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248005040 con 0x7fb2540708a0 2024-03-23T12:18:30.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.084+0000 7fb259000700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb25411a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T12:18:30.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.085+0000 7fb259a00700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540708a0 0x7fb254119e90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb248005010 tx=0x7fb248009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:30.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.085+0000 7fb251600700 1 -- 172.21.15.63:0/4221905096 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb24800e040 con 0x7fb2540708a0 2024-03-23T12:18:30.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.085+0000 7fb251600700 1 -- 172.21.15.63:0/4221905096 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb248004290 con 0x7fb2540708a0 2024-03-23T12:18:30.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.085+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb254114280 con 0x7fb2540708a0 2024-03-23T12:18:30.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.085+0000 7fb251600700 1 -- 172.21.15.63:0/4221905096 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb248013ca0 con 0x7fb2540708a0 2024-03-23T12:18:30.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.085+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2541147c0 con 0x7fb2540708a0 2024-03-23T12:18:30.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.086+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb254071aa0 con 0x7fb2540708a0 2024-03-23T12:18:30.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.088+0000 7fb251600700 1 -- 172.21.15.63:0/4221905096 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb2480134b0 con 0x7fb2540708a0 2024-03-23T12:18:30.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.088+0000 7fb251600700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb23c06c7c0 0x7fb23c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.089+0000 7fb251600700 1 -- 172.21.15.63:0/4221905096 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb24808cb80 con 0x7fb2540708a0 2024-03-23T12:18:30.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.089+0000 7fb259000700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb23c06c7c0 0x7fb23c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.089+0000 7fb259000700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb23c06c7c0 0x7fb23c06ec70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb254115730 tx=0x7fb24000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:30.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.094+0000 7fb251600700 1 -- 172.21.15.63:0/4221905096 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb24805a810 con 0x7fb2540708a0 2024-03-23T12:18:30.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.317+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2541154e0 con 0x7fb23c06c7c0 2024-03-23T12:18:30.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.328+0000 7fb251600700 1 -- 172.21.15.63:0/4221905096 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fb2541154e0 con 0x7fb23c06c7c0 2024-03-23T12:18:30.306 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:18:30.306 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (27m) 4m ago 40m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:18:30.306 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (24m) 4m ago 41m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:18:30.306 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (24m) 4m ago 39m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (27m) 4m ago 40m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (20m) 4m ago 34m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (20m) 4m ago 34m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (5m) 4m ago 34m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (20m) 4m ago 34m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (30m) 4m ago 42m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (28m) 4m ago 39m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (26m) 4m ago 42m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (25m) 4m ago 39m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (28m) 4m ago 41m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (28m) 4m ago 39m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (24m) 4m ago 38m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (23m) 4m ago 38m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (23m) 4m ago 37m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (22m) 4m ago 37m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:18:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (21m) 4m ago 36m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:18:30.308 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (21m) 4m ago 36m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:18:30.308 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (28m) 4m ago 40m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:18:30.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb23c06c7c0 msgr2=0x7fb23c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb23c06c7c0 0x7fb23c06ec70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb254115730 tx=0x7fb24000b040 comp rx=0 tx=0).stop 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540708a0 msgr2=0x7fb254119e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540708a0 0x7fb254119e90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb248005010 tx=0x7fb248009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 shutdown_connections 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb23c06c7c0 0x7fb23c06ec70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540708a0 0x7fb254119e90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 --2- 172.21.15.63:0/4221905096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb254071f90 0x7fb25411a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 >> 172.21.15.63:0/4221905096 conn(0x7fb2540f89b0 msgr2=0x7fb2540f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 shutdown_connections 2024-03-23T12:18:30.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.330+0000 7fb25c400700 1 -- 172.21.15.63:0/4221905096 wait complete. 2024-03-23T12:18:30.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:30 smithi111 ceph-mon[106899]: from='client.35502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:30.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:30 smithi111 ceph-mon[106899]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:30.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.466+0000 7f0e3d800700 1 -- 172.21.15.63:0/2409372987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e38102e00 msgr2=0x7f0e381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.466+0000 7f0e3d800700 1 --2- 172.21.15.63:0/2409372987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e38102e00 0x7f0e381031d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0e28004940 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.467+0000 7f0e3d800700 1 -- 172.21.15.63:0/2409372987 shutdown_connections 2024-03-23T12:18:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.467+0000 7f0e3d800700 1 --2- 172.21.15.63:0/2409372987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e381037a0 0x7f0e380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.467+0000 7f0e3d800700 1 --2- 172.21.15.63:0/2409372987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e38102e00 0x7f0e381031d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.467+0000 7f0e3d800700 1 -- 172.21.15.63:0/2409372987 >> 172.21.15.63:0/2409372987 conn(0x7f0e380f89b0 msgr2=0x7f0e380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.467+0000 7f0e3d800700 1 -- 172.21.15.63:0/2409372987 shutdown_connections 2024-03-23T12:18:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.467+0000 7f0e3d800700 1 -- 172.21.15.63:0/2409372987 wait complete. 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.467+0000 7f0e3d800700 1 Processor -- start 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e3d800700 1 -- start start 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e3d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e38102e00 0x7f0e38115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e3d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e381037a0 0x7f0e38116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e3d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e38116880 con 0x7f0e381037a0 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e3d800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3810fc50 con 0x7f0e38102e00 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e36000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e381037a0 0x7f0e38116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e36000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e381037a0 0x7f0e38116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54048/0 (socket says 172.21.15.63:54048) 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.468+0000 7f0e36000700 1 -- 172.21.15.63:0/1281001207 learned_addr learned my addr 172.21.15.63:0/1281001207 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:18:30.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e36a00700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e38102e00 0x7f0e38115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e36000700 1 -- 172.21.15.63:0/1281001207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e38102e00 msgr2=0x7f0e38115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e36000700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e38102e00 0x7f0e38115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e36000700 1 -- 172.21.15.63:0/1281001207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28005040 con 0x7f0e381037a0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e36000700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e381037a0 0x7f0e38116230 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0e24002a80 tx=0x7f0e24004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e2ea00700 1 -- 172.21.15.63:0/1281001207 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e24004e80 con 0x7f0e381037a0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e2ea00700 1 -- 172.21.15.63:0/1281001207 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e2400f040 con 0x7f0e381037a0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3810ff30 con 0x7f0e381037a0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.469+0000 7f0e2ea00700 1 -- 172.21.15.63:0/1281001207 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2400da00 con 0x7f0e381037a0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.470+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e381104f0 con 0x7f0e381037a0 2024-03-23T12:18:30.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.471+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e380fe550 con 0x7f0e381037a0 2024-03-23T12:18:30.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.472+0000 7f0e2ea00700 1 -- 172.21.15.63:0/1281001207 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0e24007cc0 con 0x7f0e381037a0 2024-03-23T12:18:30.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.473+0000 7f0e2ea00700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e1c06c7d0 0x7f0e1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.473+0000 7f0e2ea00700 1 -- 172.21.15.63:0/1281001207 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0e2408b020 con 0x7f0e381037a0 2024-03-23T12:18:30.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.473+0000 7f0e36a00700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e1c06c7d0 0x7f0e1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.474+0000 7f0e36a00700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e1c06c7d0 0x7f0e1c06ec80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009fd0 tx=0x7f0e28009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:30.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.478+0000 7f0e2ea00700 1 -- 172.21.15.63:0/1281001207 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0e24058d30 con 0x7f0e381037a0 2024-03-23T12:18:30.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:30 smithi063 ceph-mon[141124]: from='client.35502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:30.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:30 smithi063 ceph-mon[141124]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:30.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.781+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0e38109de0 con 0x7f0e381037a0 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.781+0000 7f0e2ea00700 1 -- 172.21.15.63:0/1281001207 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f0e240588c0 con 0x7f0e381037a0 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:18:30.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:18:30.761 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e1c06c7d0 msgr2=0x7f0e1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e1c06c7d0 0x7f0e1c06ec80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009fd0 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e381037a0 msgr2=0x7f0e38116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e381037a0 0x7f0e38116230 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0e24002a80 tx=0x7f0e24004630 comp rx=0 tx=0).stop 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 shutdown_connections 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0e1c06c7d0 0x7f0e1c06ec80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e38102e00 0x7f0e38115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 --2- 172.21.15.63:0/1281001207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e381037a0 0x7f0e38116230 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 >> 172.21.15.63:0/1281001207 conn(0x7f0e380f89b0 msgr2=0x7f0e380f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 shutdown_connections 2024-03-23T12:18:30.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.784+0000 7f0e3d800700 1 -- 172.21.15.63:0/1281001207 wait complete. 2024-03-23T12:18:30.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.921+0000 7f677ac00700 1 -- 172.21.15.63:0/2164118778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fe830 msgr2=0x7f6774106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.921+0000 7f677ac00700 1 --2- 172.21.15.63:0/2164118778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fe830 0x7f6774106f00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6760004940 tx=0x7f6760009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:30.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.922+0000 7f677ac00700 1 -- 172.21.15.63:0/2164118778 shutdown_connections 2024-03-23T12:18:30.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.922+0000 7f677ac00700 1 --2- 172.21.15.63:0/2164118778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fe830 0x7f6774106f00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.922+0000 7f677ac00700 1 --2- 172.21.15.63:0/2164118778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67740fde90 0x7f67740fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.922+0000 7f677ac00700 1 -- 172.21.15.63:0/2164118778 >> 172.21.15.63:0/2164118778 conn(0x7f6774072bc0 msgr2=0x7f6774072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:30.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.922+0000 7f677ac00700 1 -- 172.21.15.63:0/2164118778 shutdown_connections 2024-03-23T12:18:30.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.922+0000 7f677ac00700 1 -- 172.21.15.63:0/2164118778 wait complete. 2024-03-23T12:18:30.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f677ac00700 1 Processor -- start 2024-03-23T12:18:30.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f677ac00700 1 -- start start 2024-03-23T12:18:30.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f677ac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fde90 0x7f6774113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f677ac00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67740fe830 0x7f6774114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f677ac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774114670 con 0x7f67740fde90 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f677ac00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677410da60 con 0x7f67740fe830 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f6773e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fde90 0x7f6774113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f6773e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fde90 0x7f6774113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54076/0 (socket says 172.21.15.63:54076) 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.923+0000 7f6773e00700 1 -- 172.21.15.63:0/123456430 learned_addr learned my addr 172.21.15.63:0/123456430 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f6773400700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67740fe830 0x7f6774114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f6773e00700 1 -- 172.21.15.63:0/123456430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67740fe830 msgr2=0x7f6774114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f6773e00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67740fe830 0x7f6774114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f6773e00700 1 -- 172.21.15.63:0/123456430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6760005040 con 0x7f67740fde90 2024-03-23T12:18:30.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f6773e00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fde90 0x7f6774113ae0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6764002a50 tx=0x7f67640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:30.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f676be00700 1 -- 172.21.15.63:0/123456430 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6764007cc0 con 0x7f67740fde90 2024-03-23T12:18:30.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f676be00700 1 -- 172.21.15.63:0/123456430 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f676400f040 con 0x7f67740fde90 2024-03-23T12:18:30.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677410dd40 con 0x7f67740fde90 2024-03-23T12:18:30.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.924+0000 7f676be00700 1 -- 172.21.15.63:0/123456430 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f676400d9c0 con 0x7f67740fde90 2024-03-23T12:18:30.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.925+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677410e250 con 0x7f67740fde90 2024-03-23T12:18:30.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.926+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6774065e30 con 0x7f67740fde90 2024-03-23T12:18:30.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.927+0000 7f676be00700 1 -- 172.21.15.63:0/123456430 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f676400db20 con 0x7f67740fde90 2024-03-23T12:18:30.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.927+0000 7f676be00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f675806c7d0 0x7f675806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:30.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.927+0000 7f676be00700 1 -- 172.21.15.63:0/123456430 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f676408b820 con 0x7f67740fde90 2024-03-23T12:18:30.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.928+0000 7f6773400700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f675806c7d0 0x7f675806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:30.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.928+0000 7f6773400700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f675806c7d0 0x7f675806ec80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f677410ed40 tx=0x7f6760009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:30.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:30.932+0000 7f676be00700 1 -- 172.21.15.63:0/123456430 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6764059430 con 0x7f67740fde90 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.196+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6774109dd0 con 0x7f67740fde90 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.196+0000 7f676be00700 1 -- 172.21.15.63:0/123456430 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f6764058fc0 con 0x7f67740fde90 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:18:31.175 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:18:31.176 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:18:31.177 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:18:31.178 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:18:31.178 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:18:31.178 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:18:31.178 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:18:31.178 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:18:31.178 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f675806c7d0 msgr2=0x7f675806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f675806c7d0 0x7f675806ec80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f677410ed40 tx=0x7f6760009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fde90 msgr2=0x7f6774113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fde90 0x7f6774113ae0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6764002a50 tx=0x7f67640046b0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 shutdown_connections 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f675806c7d0 0x7f675806ec80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67740fde90 0x7f6774113ae0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 --2- 172.21.15.63:0/123456430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67740fe830 0x7f6774114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 >> 172.21.15.63:0/123456430 conn(0x7f6774072bc0 msgr2=0x7f6774103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:31.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 shutdown_connections 2024-03-23T12:18:31.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.199+0000 7f677ac00700 1 -- 172.21.15.63:0/123456430 wait complete. 2024-03-23T12:18:31.180 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:18:31.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.337+0000 7f8ddfa00700 1 -- 172.21.15.63:0/70749781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 msgr2=0x7f8dd80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.338+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/70749781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd80fce10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4004940 tx=0x7f8dc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:31.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.338+0000 7f8ddfa00700 1 -- 172.21.15.63:0/70749781 shutdown_connections 2024-03-23T12:18:31.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.338+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/70749781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd80fce10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.338+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/70749781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd8102e00 0x7f8dd81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.338+0000 7f8ddfa00700 1 -- 172.21.15.63:0/70749781 >> 172.21.15.63:0/70749781 conn(0x7f8dd80f89b0 msgr2=0x7f8dd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:31.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.338+0000 7f8ddfa00700 1 -- 172.21.15.63:0/70749781 shutdown_connections 2024-03-23T12:18:31.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.338+0000 7f8ddfa00700 1 -- 172.21.15.63:0/70749781 wait complete. 2024-03-23T12:18:31.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.339+0000 7f8ddfa00700 1 Processor -- start 2024-03-23T12:18:31.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.339+0000 7f8ddfa00700 1 -- start start 2024-03-23T12:18:31.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.339+0000 7f8ddfa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd8102e00 0x7f8dd80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:31.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.339+0000 7f8ddfa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.339+0000 7f8ddfa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd8070d90 con 0x7f8dd81037a0 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.339+0000 7f8ddfa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd8070ed0 con 0x7f8dd8102e00 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.339+0000 7f8dd7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54090/0 (socket says 172.21.15.63:54090) 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd7e00700 1 -- 172.21.15.63:0/3384393102 learned_addr learned my addr 172.21.15.63:0/3384393102 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8ddd000700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd8102e00 0x7f8dd80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd7e00700 1 -- 172.21.15.63:0/3384393102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd8102e00 msgr2=0x7f8dd80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd7e00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd8102e00 0x7f8dd80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd7e00700 1 -- 172.21.15.63:0/3384393102 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc4005040 con 0x7f8dd81037a0 2024-03-23T12:18:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd7e00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd8070800 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009390 tx=0x7f8dc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:31.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd4c00700 1 -- 172.21.15.63:0/3384393102 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc400e040 con 0x7f8dd81037a0 2024-03-23T12:18:31.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8dd4c00700 1 -- 172.21.15.63:0/3384393102 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dc4004290 con 0x7f8dd81037a0 2024-03-23T12:18:31.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.340+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd8071120 con 0x7f8dd81037a0 2024-03-23T12:18:31.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.341+0000 7f8dd4c00700 1 -- 172.21.15.63:0/3384393102 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc4013e70 con 0x7f8dd81037a0 2024-03-23T12:18:31.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.341+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd811a250 con 0x7f8dd81037a0 2024-03-23T12:18:31.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.342+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd80fe550 con 0x7f8dd81037a0 2024-03-23T12:18:31.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.343+0000 7f8dd4c00700 1 -- 172.21.15.63:0/3384393102 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8dc4013690 con 0x7f8dd81037a0 2024-03-23T12:18:31.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.343+0000 7f8dd4c00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8dc006c7d0 0x7f8dc006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:31.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.344+0000 7f8dd4c00700 1 -- 172.21.15.63:0/3384393102 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8dc408cbe0 con 0x7f8dd81037a0 2024-03-23T12:18:31.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.344+0000 7f8ddd000700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8dc006c7d0 0x7f8dc006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:31.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.344+0000 7f8ddd000700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8dc006c7d0 0x7f8dc006ec80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc004f60 tx=0x7f8dcc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:31.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.348+0000 7f8dd4c00700 1 -- 172.21.15.63:0/3384393102 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8dc4056fd0 con 0x7f8dd81037a0 2024-03-23T12:18:31.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.580+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dd8109e30 con 0x7f8dc006c7d0 2024-03-23T12:18:31.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.582+0000 7f8dd4c00700 1 -- 172.21.15.63:0/3384393102 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8dd8109e30 con 0x7f8dc006c7d0 2024-03-23T12:18:31.560 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:18:31.560 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:18:31.560 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:18:31.560 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:18:31.560 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:18:31.561 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8dc006c7d0 msgr2=0x7f8dc006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8dc006c7d0 0x7f8dc006ec80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc004f60 tx=0x7f8dcc00b040 comp rx=0 tx=0).stop 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 msgr2=0x7f8dd8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd8070800 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009390 tx=0x7f8dc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 shutdown_connections 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8dc006c7d0 0x7f8dc006ec80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8dd8102e00 0x7f8dd80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 --2- 172.21.15.63:0/3384393102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8dd81037a0 0x7f8dd8070800 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.584+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 >> 172.21.15.63:0/3384393102 conn(0x7f8dd80f89b0 msgr2=0x7f8dd80f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:31.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.585+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 shutdown_connections 2024-03-23T12:18:31.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.585+0000 7f8ddfa00700 1 -- 172.21.15.63:0/3384393102 wait complete. 2024-03-23T12:18:31.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 -- 172.21.15.63:0/285259821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a38102e00 msgr2=0x7f5a381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/285259821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a38102e00 0x7f5a381031d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5a28004940 tx=0x7f5a28009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:31.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 -- 172.21.15.63:0/285259821 shutdown_connections 2024-03-23T12:18:31.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/285259821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a381037a0 0x7f5a380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/285259821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a38102e00 0x7f5a381031d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 -- 172.21.15.63:0/285259821 >> 172.21.15.63:0/285259821 conn(0x7f5a380f89b0 msgr2=0x7f5a380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:31.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 -- 172.21.15.63:0/285259821 shutdown_connections 2024-03-23T12:18:31.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.726+0000 7f5a3ec00700 1 -- 172.21.15.63:0/285259821 wait complete. 2024-03-23T12:18:31.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.727+0000 7f5a3ec00700 1 Processor -- start 2024-03-23T12:18:31.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.727+0000 7f5a3ec00700 1 -- start start 2024-03-23T12:18:31.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.727+0000 7f5a3ec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a38102e00 0x7f5a38115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:31.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.727+0000 7f5a3ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a381037a0 0x7f5a38116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:31.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.727+0000 7f5a3ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a38116880 con 0x7f5a381037a0 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.727+0000 7f5a3ec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3810fc50 con 0x7f5a38102e00 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a381037a0 0x7f5a38116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a381037a0 0x7f5a38116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40698/0 (socket says 172.21.15.63:40698) 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37400700 1 -- 172.21.15.63:0/2577833323 learned_addr learned my addr 172.21.15.63:0/2577833323 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37e00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a38102e00 0x7f5a38115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37400700 1 -- 172.21.15.63:0/2577833323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a38102e00 msgr2=0x7f5a38115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37400700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a38102e00 0x7f5a38115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:31.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37400700 1 -- 172.21.15.63:0/2577833323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28005040 con 0x7f5a381037a0 2024-03-23T12:18:31.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a37400700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a381037a0 0x7f5a38116230 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400c890 tx=0x7f5a240044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:31.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.728+0000 7f5a2fe00700 1 -- 172.21.15.63:0/2577833323 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a2400a700 con 0x7f5a381037a0 2024-03-23T12:18:31.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.729+0000 7f5a2fe00700 1 -- 172.21.15.63:0/2577833323 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a2400ad10 con 0x7f5a381037a0 2024-03-23T12:18:31.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.729+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3810ff30 con 0x7f5a381037a0 2024-03-23T12:18:31.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.729+0000 7f5a2fe00700 1 -- 172.21.15.63:0/2577833323 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a24012e70 con 0x7f5a381037a0 2024-03-23T12:18:31.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.729+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a381104f0 con 0x7f5a381037a0 2024-03-23T12:18:31.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.730+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a380fe550 con 0x7f5a381037a0 2024-03-23T12:18:31.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.731+0000 7f5a2fe00700 1 -- 172.21.15.63:0/2577833323 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5a2400a860 con 0x7f5a381037a0 2024-03-23T12:18:31.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.732+0000 7f5a2fe00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a1c06c7d0 0x7f5a1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:18:31.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.732+0000 7f5a2fe00700 1 -- 172.21.15.63:0/2577833323 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5a2408ba90 con 0x7f5a381037a0 2024-03-23T12:18:31.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.732+0000 7f5a37e00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a1c06c7d0 0x7f5a1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:18:31.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.732+0000 7f5a37e00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a1c06c7d0 0x7f5a1c06ec80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800b840 tx=0x7f5a28009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:18:31.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:31.736+0000 7f5a2fe00700 1 -- 172.21.15.63:0/2577833323 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5a240597a0 con 0x7f5a381037a0 2024-03-23T12:18:32.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.035+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5a3804bed0 con 0x7f5a381037a0 2024-03-23T12:18:32.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.035+0000 7f5a2fe00700 1 -- 172.21.15.63:0/2577833323 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5a24059330 con 0x7f5a381037a0 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:18:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:18:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:18:32.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a1c06c7d0 msgr2=0x7f5a1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:32.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a1c06c7d0 0x7f5a1c06ec80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800b840 tx=0x7f5a28009e30 comp rx=0 tx=0).stop 2024-03-23T12:18:32.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a381037a0 msgr2=0x7f5a38116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a381037a0 0x7f5a38116230 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400c890 tx=0x7f5a240044c0 comp rx=0 tx=0).stop 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 shutdown_connections 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a1c06c7d0 0x7f5a1c06ec80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a38102e00 0x7f5a38115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 --2- 172.21.15.63:0/2577833323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a381037a0 0x7f5a38116230 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.037+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 >> 172.21.15.63:0/2577833323 conn(0x7f5a380f89b0 msgr2=0x7f5a380706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.038+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 shutdown_connections 2024-03-23T12:18:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:18:32.038+0000 7f5a3ec00700 1 -- 172.21.15.63:0/2577833323 wait complete. 2024-03-23T12:18:32.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:31 smithi063 ceph-mon[141124]: from='client.45231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:32.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:31 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1281001207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:18:32.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:31 smithi063 ceph-mon[141124]: pgmap v823: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:32.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:31 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/123456430' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:18:32.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:31 smithi111 ceph-mon[106899]: from='client.45231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:32.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:31 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1281001207' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:18:32.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:31 smithi111 ceph-mon[106899]: pgmap v823: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:32.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:31 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/123456430' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:18:33.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:32 smithi063 ceph-mon[141124]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:33.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:32 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2577833323' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:18:33.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:32 smithi111 ceph-mon[106899]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:18:33.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:32 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2577833323' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:18:34.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:33 smithi063 ceph-mon[141124]: pgmap v824: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:34.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:33 smithi111 ceph-mon[106899]: pgmap v824: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:35.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:35 smithi111 ceph-mon[106899]: pgmap v825: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:35.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:35 smithi063 ceph-mon[141124]: pgmap v825: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:37.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:37 smithi111 ceph-mon[106899]: pgmap v826: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:37.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:37 smithi063 ceph-mon[141124]: pgmap v826: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:39.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:39 smithi111 ceph-mon[106899]: pgmap v827: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:39.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:39 smithi063 ceph-mon[141124]: pgmap v827: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:41 smithi063 ceph-mon[141124]: pgmap v828: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:42.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:42.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:41 smithi111 ceph-mon[106899]: pgmap v828: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:43.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:43 smithi111 ceph-mon[106899]: pgmap v829: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:43.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:43 smithi063 ceph-mon[141124]: pgmap v829: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:45.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:45 smithi111 ceph-mon[106899]: pgmap v830: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:45.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:45 smithi063 ceph-mon[141124]: pgmap v830: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:47.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:47 smithi111 ceph-mon[106899]: pgmap v831: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:47.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:47 smithi063 ceph-mon[141124]: pgmap v831: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:49.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:49 smithi111 ceph-mon[106899]: pgmap v832: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:49.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:49 smithi063 ceph-mon[141124]: pgmap v832: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:52.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:51 smithi063 ceph-mon[141124]: pgmap v833: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:52.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:51 smithi111 ceph-mon[106899]: pgmap v833: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:53.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:53 smithi111 ceph-mon[106899]: pgmap v834: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:53.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:53 smithi063 ceph-mon[141124]: pgmap v834: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:55 smithi111 ceph-mon[106899]: pgmap v835: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:55.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:55 smithi063 ceph-mon[141124]: pgmap v835: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:57.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:57.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:18:58.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:57 smithi063 ceph-mon[141124]: pgmap v836: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:58.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:57 smithi111 ceph-mon[106899]: pgmap v836: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:59.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:18:59 smithi111 ceph-mon[106899]: pgmap v837: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:18:59.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:18:59 smithi063 ceph-mon[141124]: pgmap v837: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:02.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:01 smithi063 ceph-mon[141124]: pgmap v838: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:02.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:01 smithi111 ceph-mon[106899]: pgmap v838: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 -- 172.21.15.63:0/2989780198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 msgr2=0x7fb9101031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 --2- 172.21.15.63:0/2989780198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9101031d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb900004940 tx=0x7fb900009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 -- 172.21.15.63:0/2989780198 shutdown_connections 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 --2- 172.21.15.63:0/2989780198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9101037a0 0x7fb9100fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 --2- 172.21.15.63:0/2989780198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9101031d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 -- 172.21.15.63:0/2989780198 >> 172.21.15.63:0/2989780198 conn(0x7fb9100f89b0 msgr2=0x7fb9100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 -- 172.21.15.63:0/2989780198 shutdown_connections 2024-03-23T12:19:02.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.191+0000 7fb915e00700 1 -- 172.21.15.63:0/2989780198 wait complete. 2024-03-23T12:19:02.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.192+0000 7fb915e00700 1 Processor -- start 2024-03-23T12:19:02.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb915e00700 1 -- start start 2024-03-23T12:19:02.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb915e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb915e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9101037a0 0x7fb910070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb915e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb910070d90 con 0x7fb910102e00 2024-03-23T12:19:02.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb915e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb910070ed0 con 0x7fb9101037a0 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb90ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb90ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9100702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56340/0 (socket says 172.21.15.63:56340) 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.193+0000 7fb90ea00700 1 -- 172.21.15.63:0/1802697624 learned_addr learned my addr 172.21.15.63:0/1802697624 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb907e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9101037a0 0x7fb910070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb90ea00700 1 -- 172.21.15.63:0/1802697624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9101037a0 msgr2=0x7fb910070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb90ea00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9101037a0 0x7fb910070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb90ea00700 1 -- 172.21.15.63:0/1802697624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb900005040 con 0x7fb910102e00 2024-03-23T12:19:02.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb90ea00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9100702c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb90000b3b0 tx=0x7fb900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:02.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb906a00700 1 -- 172.21.15.63:0/1802697624 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb90000e040 con 0x7fb910102e00 2024-03-23T12:19:02.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb906a00700 1 -- 172.21.15.63:0/1802697624 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb900013420 con 0x7fb910102e00 2024-03-23T12:19:02.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb906a00700 1 -- 172.21.15.63:0/1802697624 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb90001c630 con 0x7fb910102e00 2024-03-23T12:19:02.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.194+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb910071120 con 0x7fb910102e00 2024-03-23T12:19:02.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.195+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91011a250 con 0x7fb910102e00 2024-03-23T12:19:02.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.196+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9100fe550 con 0x7fb910102e00 2024-03-23T12:19:02.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.196+0000 7fb906a00700 1 -- 172.21.15.63:0/1802697624 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb900013970 con 0x7fb910102e00 2024-03-23T12:19:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.197+0000 7fb906a00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8f406c7c0 0x7fb8f406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.197+0000 7fb906a00700 1 -- 172.21.15.63:0/1802697624 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb90008cbb0 con 0x7fb910102e00 2024-03-23T12:19:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.198+0000 7fb907e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8f406c7c0 0x7fb8f406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.198+0000 7fb907e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8f406c7c0 0x7fb8f406ec70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb910071c00 tx=0x7fb8f8009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:02.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.202+0000 7fb906a00700 1 -- 172.21.15.63:0/1802697624 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb900093080 con 0x7fb910102e00 2024-03-23T12:19:02.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.435+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb91004bf70 con 0x7fb8f406c7c0 2024-03-23T12:19:02.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.437+0000 7fb906a00700 1 -- 172.21.15.63:0/1802697624 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb8f8004e30 con 0x7fb8f406c7c0 2024-03-23T12:19:02.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8f406c7c0 msgr2=0x7fb8f406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8f406c7c0 0x7fb8f406ec70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb910071c00 tx=0x7fb8f8009ee0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 msgr2=0x7fb9100702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9100702c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb90000b3b0 tx=0x7fb900009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 shutdown_connections 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8f406c7c0 0x7fb8f406ec70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb910102e00 0x7fb9100702c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 --2- 172.21.15.63:0/1802697624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9101037a0 0x7fb910070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 >> 172.21.15.63:0/1802697624 conn(0x7fb9100f89b0 msgr2=0x7fb9100f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 shutdown_connections 2024-03-23T12:19:02.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.439+0000 7fb915e00700 1 -- 172.21.15.63:0/1802697624 wait complete. 2024-03-23T12:19:02.433 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 -- 172.21.15.63:0/689467985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070c10 msgr2=0x7f5214109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 --2- 172.21.15.63:0/689467985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070c10 0x7f5214109190 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5200004970 tx=0x7f5200009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 -- 172.21.15.63:0/689467985 shutdown_connections 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 --2- 172.21.15.63:0/689467985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070c10 0x7f5214109190 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 --2- 172.21.15.63:0/689467985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5214070300 0x7f52140706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 -- 172.21.15.63:0/689467985 >> 172.21.15.63:0/689467985 conn(0x7f52140f8970 msgr2=0x7f52140fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 -- 172.21.15.63:0/689467985 shutdown_connections 2024-03-23T12:19:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.578+0000 7f521be00700 1 -- 172.21.15.63:0/689467985 wait complete. 2024-03-23T12:19:02.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.579+0000 7f521be00700 1 Processor -- start 2024-03-23T12:19:02.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.579+0000 7f521be00700 1 -- start start 2024-03-23T12:19:02.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.579+0000 7f521be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070300 0x7f5214115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.579+0000 7f521be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5214070c10 0x7f5214116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.579+0000 7f521be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5214116850 con 0x7f5214070300 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.579+0000 7f521be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521410fc20 con 0x7f5214070c10 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5219400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070300 0x7f5214115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5219400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070300 0x7f5214115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56360/0 (socket says 172.21.15.63:56360) 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5219400700 1 -- 172.21.15.63:0/3564496624 learned_addr learned my addr 172.21.15.63:0/3564496624 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5218a00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5214070c10 0x7f5214116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5219400700 1 -- 172.21.15.63:0/3564496624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5214070c10 msgr2=0x7f5214116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5219400700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5214070c10 0x7f5214116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5219400700 1 -- 172.21.15.63:0/3564496624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5200005040 con 0x7f5214070300 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5218a00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5214070c10 0x7f5214116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5219400700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070300 0x7f5214115cc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5208002a80 tx=0x7f52080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:02.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.580+0000 7f5211600700 1 -- 172.21.15.63:0/3564496624 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5208007cc0 con 0x7f5214070300 2024-03-23T12:19:02.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.581+0000 7f5211600700 1 -- 172.21.15.63:0/3564496624 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f520800f040 con 0x7f5214070300 2024-03-23T12:19:02.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.581+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521410ff00 con 0x7f5214070300 2024-03-23T12:19:02.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.581+0000 7f5211600700 1 -- 172.21.15.63:0/3564496624 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5208009640 con 0x7f5214070300 2024-03-23T12:19:02.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.581+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5214110440 con 0x7f5214070300 2024-03-23T12:19:02.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.582+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214071a40 con 0x7f5214070300 2024-03-23T12:19:02.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.583+0000 7f5211600700 1 -- 172.21.15.63:0/3564496624 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f52080097a0 con 0x7f5214070300 2024-03-23T12:19:02.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.584+0000 7f5211600700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f51fc06c7d0 0x7f51fc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.584+0000 7f5211600700 1 -- 172.21.15.63:0/3564496624 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f520808b330 con 0x7f5214070300 2024-03-23T12:19:02.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.584+0000 7f5218a00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f51fc06c7d0 0x7f51fc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.584+0000 7f5218a00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f51fc06c7d0 0x7f51fc06ec80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5200009fd0 tx=0x7f5200009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:02.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.588+0000 7f5211600700 1 -- 172.21.15.63:0/3564496624 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5208058f40 con 0x7f5214070300 2024-03-23T12:19:02.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.822+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f521404bf40 con 0x7f51fc06c7d0 2024-03-23T12:19:02.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.823+0000 7f5211600700 1 -- 172.21.15.63:0/3564496624 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f521404bf40 con 0x7f51fc06c7d0 2024-03-23T12:19:02.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.825+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f51fc06c7d0 msgr2=0x7f51fc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.825+0000 7f521be00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f51fc06c7d0 0x7f51fc06ec80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5200009fd0 tx=0x7f5200009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:02.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.825+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070300 msgr2=0x7f5214115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.825+0000 7f521be00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070300 0x7f5214115cc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5208002a80 tx=0x7f52080046b0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.826+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 shutdown_connections 2024-03-23T12:19:02.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.826+0000 7f521be00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f51fc06c7d0 0x7f51fc06ec80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.826+0000 7f521be00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5214070300 0x7f5214115cc0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.826+0000 7f521be00700 1 --2- 172.21.15.63:0/3564496624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5214070c10 0x7f5214116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.826+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 >> 172.21.15.63:0/3564496624 conn(0x7f52140f8970 msgr2=0x7f52140f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:02.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.826+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 shutdown_connections 2024-03-23T12:19:02.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.826+0000 7f521be00700 1 -- 172.21.15.63:0/3564496624 wait complete. 2024-03-23T12:19:02.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 -- 172.21.15.63:0/1995321618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 msgr2=0x7f8efc1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 --2- 172.21.15.63:0/1995321618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc1053e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8eec004940 tx=0x7f8eec009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:02.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 -- 172.21.15.63:0/1995321618 shutdown_connections 2024-03-23T12:19:02.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 --2- 172.21.15.63:0/1995321618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8efc1059b0 0x7f8efc0ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 --2- 172.21.15.63:0/1995321618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc1053e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 -- 172.21.15.63:0/1995321618 >> 172.21.15.63:0/1995321618 conn(0x7f8efc0faba0 msgr2=0x7f8efc0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:02.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 -- 172.21.15.63:0/1995321618 shutdown_connections 2024-03-23T12:19:02.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.963+0000 7f8f01800700 1 -- 172.21.15.63:0/1995321618 wait complete. 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.964+0000 7f8f01800700 1 Processor -- start 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.964+0000 7f8f01800700 1 -- start start 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.964+0000 7f8f01800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.964+0000 7f8f01800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8efc1059b0 0x7f8efc116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.964+0000 7f8f01800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc1167e0 con 0x7f8efc105010 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.964+0000 7f8f01800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc10fbb0 con 0x7f8efc1059b0 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.964+0000 7f8efaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8efaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56376/0 (socket says 172.21.15.63:56376) 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8efaa00700 1 -- 172.21.15.63:0/1589022942 learned_addr learned my addr 172.21.15.63:0/1589022942 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8efaa00700 1 -- 172.21.15.63:0/1589022942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8efc1059b0 msgr2=0x7f8efc116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8ef3e00700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8efc1059b0 0x7f8efc116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8efaa00700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8efc1059b0 0x7f8efc116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8efaa00700 1 -- 172.21.15.63:0/1589022942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec005040 con 0x7f8efc105010 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8ef3e00700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8efc1059b0 0x7f8efc116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8efaa00700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc115c50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007f50 tx=0x7f8eec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.965+0000 7f8ef2a00700 1 -- 172.21.15.63:0/1589022942 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eec021070 con 0x7f8efc105010 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.966+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc10fe30 con 0x7f8efc105010 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.966+0000 7f8ef2a00700 1 -- 172.21.15.63:0/1589022942 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8eec013e00 con 0x7f8efc105010 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.966+0000 7f8ef2a00700 1 -- 172.21.15.63:0/1589022942 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eec00e070 con 0x7f8efc105010 2024-03-23T12:19:02.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.966+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc1103f0 con 0x7f8efc105010 2024-03-23T12:19:02.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.967+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc100760 con 0x7f8efc105010 2024-03-23T12:19:02.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.968+0000 7f8ef2a00700 1 -- 172.21.15.63:0/1589022942 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8eec013950 con 0x7f8efc105010 2024-03-23T12:19:02.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.969+0000 7f8ef2a00700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ee006c7d0 0x7f8ee006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:02.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.969+0000 7f8ef2a00700 1 -- 172.21.15.63:0/1589022942 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8eec08cd00 con 0x7f8efc105010 2024-03-23T12:19:02.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.969+0000 7f8ef3e00700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ee006c7d0 0x7f8ee006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:02.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.970+0000 7f8ef3e00700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ee006c7d0 0x7f8ee006ec80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4004600 tx=0x7f8ee400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:02.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:02.974+0000 7f8ef2a00700 1 -- 172.21.15.63:0/1589022942 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8eec05aa10 con 0x7f8efc105010 2024-03-23T12:19:03.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.198+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8efc04bf40 con 0x7f8ee006c7d0 2024-03-23T12:19:03.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.208+0000 7f8ef2a00700 1 -- 172.21.15.63:0/1589022942 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f8efc04bf40 con 0x7f8ee006c7d0 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (28m) 5m ago 41m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (25m) 5m ago 41m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (25m) 5m ago 40m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (28m) 5m ago 41m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (21m) 5m ago 35m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (21m) 5m ago 35m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (5m) 5m ago 34m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (20m) 5m ago 35m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (30m) 5m ago 42m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (29m) 5m ago 40m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (26m) 5m ago 43m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (26m) 5m ago 40m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:19:03.188 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (29m) 5m ago 41m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (29m) 5m ago 40m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (25m) 5m ago 39m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (24m) 5m ago 38m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (23m) 5m ago 38m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (23m) 5m ago 37m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (22m) 5m ago 37m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (21m) 5m ago 36m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:19:03.189 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (28m) 5m ago 40m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ee006c7d0 msgr2=0x7f8ee006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ee006c7d0 0x7f8ee006ec80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4004600 tx=0x7f8ee400b040 comp rx=0 tx=0).stop 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 msgr2=0x7f8efc115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc115c50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007f50 tx=0x7f8eec009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 shutdown_connections 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ee006c7d0 0x7f8ee006ec80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8efc105010 0x7f8efc115c50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.210+0000 7f8f01800700 1 --2- 172.21.15.63:0/1589022942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8efc1059b0 0x7f8efc116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.211+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 >> 172.21.15.63:0/1589022942 conn(0x7f8efc0faba0 msgr2=0x7f8efc0fbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:03.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.211+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 shutdown_connections 2024-03-23T12:19:03.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.211+0000 7f8f01800700 1 -- 172.21.15.63:0/1589022942 wait complete. 2024-03-23T12:19:03.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.345+0000 7f31df000700 1 -- 172.21.15.63:0/3482352788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 msgr2=0x7f31d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.345+0000 7f31df000700 1 --2- 172.21.15.63:0/3482352788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d81031d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f31c8004940 tx=0x7f31c8009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:03.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.346+0000 7f31df000700 1 -- 172.21.15.63:0/3482352788 shutdown_connections 2024-03-23T12:19:03.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.346+0000 7f31df000700 1 --2- 172.21.15.63:0/3482352788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81037a0 0x7f31d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.346+0000 7f31df000700 1 --2- 172.21.15.63:0/3482352788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d81031d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.346+0000 7f31df000700 1 -- 172.21.15.63:0/3482352788 >> 172.21.15.63:0/3482352788 conn(0x7f31d80f89b0 msgr2=0x7f31d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:03.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.346+0000 7f31df000700 1 -- 172.21.15.63:0/3482352788 shutdown_connections 2024-03-23T12:19:03.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.346+0000 7f31df000700 1 -- 172.21.15.63:0/3482352788 wait complete. 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.346+0000 7f31df000700 1 Processor -- start 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31df000700 1 -- start start 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31df000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31df000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81037a0 0x7f31d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31df000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d8070d90 con 0x7f31d8102e00 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31df000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d8070ed0 con 0x7f31d81037a0 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31d7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31d7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56394/0 (socket says 172.21.15.63:56394) 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.347+0000 7f31d7e00700 1 -- 172.21.15.63:0/2853296377 learned_addr learned my addr 172.21.15.63:0/2853296377 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31d7400700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81037a0 0x7f31d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:03.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31d7e00700 1 -- 172.21.15.63:0/2853296377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81037a0 msgr2=0x7f31d8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31d7e00700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81037a0 0x7f31d8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31d7e00700 1 -- 172.21.15.63:0/2853296377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c8005040 con 0x7f31d8102e00 2024-03-23T12:19:03.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31d7400700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81037a0 0x7f31d8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:19:03.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31d7e00700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d80702c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f31c8008d80 tx=0x7f31c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:03.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31cfe00700 1 -- 172.21.15.63:0/2853296377 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31c800e030 con 0x7f31d8102e00 2024-03-23T12:19:03.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31cfe00700 1 -- 172.21.15.63:0/2853296377 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31c8013de0 con 0x7f31d8102e00 2024-03-23T12:19:03.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d8071120 con 0x7f31d8102e00 2024-03-23T12:19:03.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31cfe00700 1 -- 172.21.15.63:0/2853296377 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31c801cd60 con 0x7f31d8102e00 2024-03-23T12:19:03.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.348+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d811a170 con 0x7f31d8102e00 2024-03-23T12:19:03.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.349+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d80fe550 con 0x7f31d8102e00 2024-03-23T12:19:03.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.350+0000 7f31cfe00700 1 -- 172.21.15.63:0/2853296377 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f31c801c590 con 0x7f31d8102e00 2024-03-23T12:19:03.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.351+0000 7f31cfe00700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f31bc06c7d0 0x7f31bc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:03.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.351+0000 7f31cfe00700 1 -- 172.21.15.63:0/2853296377 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31c808d330 con 0x7f31d8102e00 2024-03-23T12:19:03.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.352+0000 7f31d7400700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f31bc06c7d0 0x7f31bc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:03.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.352+0000 7f31d7400700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f31bc06c7d0 0x7f31bc06ec80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f31c4004600 tx=0x7f31c400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:03.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.355+0000 7f31cfe00700 1 -- 172.21.15.63:0/2853296377 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31c80575b0 con 0x7f31d8102e00 2024-03-23T12:19:03.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:03 smithi111 ceph-mon[106899]: from='client.35530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:03.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:03 smithi111 ceph-mon[106899]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:03.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:03 smithi111 ceph-mon[106899]: pgmap v839: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:03.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:03 smithi063 ceph-mon[141124]: from='client.35530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:03.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:03 smithi063 ceph-mon[141124]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:03.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:03 smithi063 ceph-mon[141124]: pgmap v839: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:03.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.660+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f31d80719b0 con 0x7f31d8102e00 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.660+0000 7f31cfe00700 1 -- 172.21.15.63:0/2853296377 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f31c805abd0 con 0x7f31d8102e00 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:19:03.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:19:03.641 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:19:03.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f31bc06c7d0 msgr2=0x7f31bc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f31bc06c7d0 0x7f31bc06ec80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f31c4004600 tx=0x7f31c400b040 comp rx=0 tx=0).stop 2024-03-23T12:19:03.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 msgr2=0x7f31d80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d80702c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f31c8008d80 tx=0x7f31c8009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:03.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 shutdown_connections 2024-03-23T12:19:03.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f31bc06c7d0 0x7f31bc06ec80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31d8102e00 0x7f31d80702c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 --2- 172.21.15.63:0/2853296377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81037a0 0x7f31d8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 >> 172.21.15.63:0/2853296377 conn(0x7f31d80f89b0 msgr2=0x7f31d80f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:03.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 shutdown_connections 2024-03-23T12:19:03.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.663+0000 7f31df000700 1 -- 172.21.15.63:0/2853296377 wait complete. 2024-03-23T12:19:03.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.800+0000 7fb9a4e00700 1 -- 172.21.15.63:0/257016839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a01037a0 msgr2=0x7fb9a00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.800+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/257016839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a01037a0 0x7fb9a00fce10 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb98c004940 tx=0x7fb98c009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:03.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.800+0000 7fb9a4e00700 1 -- 172.21.15.63:0/257016839 shutdown_connections 2024-03-23T12:19:03.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.800+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/257016839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a01037a0 0x7fb9a00fce10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.800+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/257016839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a0102e00 0x7fb9a01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.800+0000 7fb9a4e00700 1 -- 172.21.15.63:0/257016839 >> 172.21.15.63:0/257016839 conn(0x7fb9a00f89b0 msgr2=0x7fb9a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:03.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.801+0000 7fb9a4e00700 1 -- 172.21.15.63:0/257016839 shutdown_connections 2024-03-23T12:19:03.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.801+0000 7fb9a4e00700 1 -- 172.21.15.63:0/257016839 wait complete. 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.801+0000 7fb9a4e00700 1 Processor -- start 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb9a4e00700 1 -- start start 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb9a4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a0102e00 0x7fb9a0113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb9a4e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a01037a0 0x7fb9a0114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb9a4e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a0114690 con 0x7fb9a0102e00 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb9a4e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a010da80 con 0x7fb9a01037a0 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb99de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a0102e00 0x7fb9a0113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb99de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a0102e00 0x7fb9a0113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56412/0 (socket says 172.21.15.63:56412) 2024-03-23T12:19:03.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.802+0000 7fb99de00700 1 -- 172.21.15.63:0/1999164125 learned_addr learned my addr 172.21.15.63:0/1999164125 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:03.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb99d400700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a01037a0 0x7fb9a0114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:03.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb99de00700 1 -- 172.21.15.63:0/1999164125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a01037a0 msgr2=0x7fb9a0114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:03.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb99de00700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a01037a0 0x7fb9a0114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:03.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb99de00700 1 -- 172.21.15.63:0/1999164125 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c005040 con 0x7fb9a0102e00 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb99d400700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a01037a0 0x7fb9a0114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb99de00700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a0102e00 0x7fb9a0113b00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb99000c8c0 tx=0x7fb9900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb996000700 1 -- 172.21.15.63:0/1999164125 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb99000a980 con 0x7fb9a0102e00 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb996000700 1 -- 172.21.15.63:0/1999164125 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb99000ce60 con 0x7fb9a0102e00 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb996000700 1 -- 172.21.15.63:0/1999164125 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9900113a0 con 0x7fb9a0102e00 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.803+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a010dd60 con 0x7fb9a0102e00 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.804+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a010e320 con 0x7fb9a0102e00 2024-03-23T12:19:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.805+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a00fe550 con 0x7fb9a0102e00 2024-03-23T12:19:03.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.806+0000 7fb996000700 1 -- 172.21.15.63:0/1999164125 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb990012400 con 0x7fb9a0102e00 2024-03-23T12:19:03.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.806+0000 7fb996000700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb98406c7c0 0x7fb98406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:03.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.807+0000 7fb996000700 1 -- 172.21.15.63:0/1999164125 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb99008ba70 con 0x7fb9a0102e00 2024-03-23T12:19:03.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.807+0000 7fb99d400700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb98406c7c0 0x7fb98406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:03.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.807+0000 7fb99d400700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb98406c7c0 0x7fb98406ec70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00b3b0 tx=0x7fb98c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:03.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:03.811+0000 7fb996000700 1 -- 172.21.15.63:0/1999164125 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb990059840 con 0x7fb9a0102e00 2024-03-23T12:19:04.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.070+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb9a0109da0 con 0x7fb9a0102e00 2024-03-23T12:19:04.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.071+0000 7fb996000700 1 -- 172.21.15.63:0/1999164125 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fb9900593d0 con 0x7fb9a0102e00 2024-03-23T12:19:04.050 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:19:04.050 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:19:04.050 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:19:04.051 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:19:04.052 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:04.053 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb98406c7c0 msgr2=0x7fb98406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb98406c7c0 0x7fb98406ec70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00b3b0 tx=0x7fb98c009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a0102e00 msgr2=0x7fb9a0113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a0102e00 0x7fb9a0113b00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb99000c8c0 tx=0x7fb9900044c0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 shutdown_connections 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb98406c7c0 0x7fb98406ec70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9a0102e00 0x7fb9a0113b00 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 --2- 172.21.15.63:0/1999164125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a01037a0 0x7fb9a0114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 >> 172.21.15.63:0/1999164125 conn(0x7fb9a00f89b0 msgr2=0x7fb9a00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:04.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 shutdown_connections 2024-03-23T12:19:04.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.073+0000 7fb9a4e00700 1 -- 172.21.15.63:0/1999164125 wait complete. 2024-03-23T12:19:04.055 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 -- 172.21.15.63:0/2336557804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fe830 msgr2=0x7fbc44106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 --2- 172.21.15.63:0/2336557804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fe830 0x7fbc44106f00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fbc30004940 tx=0x7fbc30009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 -- 172.21.15.63:0/2336557804 shutdown_connections 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 --2- 172.21.15.63:0/2336557804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fe830 0x7fbc44106f00 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 --2- 172.21.15.63:0/2336557804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440fde90 0x7fbc440fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 -- 172.21.15.63:0/2336557804 >> 172.21.15.63:0/2336557804 conn(0x7fbc44072bc0 msgr2=0x7fbc44072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 -- 172.21.15.63:0/2336557804 shutdown_connections 2024-03-23T12:19:04.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.212+0000 7fbc49600700 1 -- 172.21.15.63:0/2336557804 wait complete. 2024-03-23T12:19:04.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.213+0000 7fbc49600700 1 Processor -- start 2024-03-23T12:19:04.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.213+0000 7fbc49600700 1 -- start start 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc49600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fde90 0x7fbc44115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc49600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440fe830 0x7fbc44116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc49600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc441168a0 con 0x7fbc440fde90 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc49600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4410fc70 con 0x7fbc440fe830 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc42a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fde90 0x7fbc44115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc42a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fde90 0x7fbc44115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56432/0 (socket says 172.21.15.63:56432) 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc42a00700 1 -- 172.21.15.63:0/2171124543 learned_addr learned my addr 172.21.15.63:0/2171124543 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc42000700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440fe830 0x7fbc44116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc42a00700 1 -- 172.21.15.63:0/2171124543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440fe830 msgr2=0x7fbc44116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc42a00700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440fe830 0x7fbc44116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.214+0000 7fbc42a00700 1 -- 172.21.15.63:0/2171124543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30005040 con 0x7fbc440fde90 2024-03-23T12:19:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.215+0000 7fbc42a00700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fde90 0x7fbc44115d10 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fbc34002a50 tx=0x7fbc340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.215+0000 7fbc3aa00700 1 -- 172.21.15.63:0/2171124543 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc34007cc0 con 0x7fbc440fde90 2024-03-23T12:19:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.215+0000 7fbc3aa00700 1 -- 172.21.15.63:0/2171124543 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc3400f040 con 0x7fbc440fde90 2024-03-23T12:19:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.215+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4410ff50 con 0x7fbc440fde90 2024-03-23T12:19:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.215+0000 7fbc3aa00700 1 -- 172.21.15.63:0/2171124543 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc3400d9c0 con 0x7fbc440fde90 2024-03-23T12:19:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.215+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc44110490 con 0x7fbc440fde90 2024-03-23T12:19:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.216+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44106a20 con 0x7fbc440fde90 2024-03-23T12:19:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.217+0000 7fbc3aa00700 1 -- 172.21.15.63:0/2171124543 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbc3400db20 con 0x7fbc440fde90 2024-03-23T12:19:04.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.218+0000 7fbc3aa00700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc2806c7d0 0x7fbc2806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.218+0000 7fbc3aa00700 1 -- 172.21.15.63:0/2171124543 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbc3408b760 con 0x7fbc440fde90 2024-03-23T12:19:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.219+0000 7fbc42000700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc2806c7d0 0x7fbc2806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.219+0000 7fbc42000700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc2806c7d0 0x7fbc2806ec80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbc441115b0 tx=0x7fbc30009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:04.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.223+0000 7fbc3aa00700 1 -- 172.21.15.63:0/2171124543 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc340593f0 con 0x7fbc440fde90 2024-03-23T12:19:04.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:04 smithi111 ceph-mon[106899]: from='client.35538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:04.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:04 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2853296377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:19:04.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:04 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1999164125' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:19:04.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.456+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc44109e40 con 0x7fbc2806c7d0 2024-03-23T12:19:04.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.457+0000 7fbc3aa00700 1 -- 172.21.15.63:0/2171124543 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbc44109e40 con 0x7fbc2806c7d0 2024-03-23T12:19:04.436 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:19:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:19:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:19:04.437 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:19:04.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc2806c7d0 msgr2=0x7fbc2806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc2806c7d0 0x7fbc2806ec80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbc441115b0 tx=0x7fbc30009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fde90 msgr2=0x7fbc44115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fde90 0x7fbc44115d10 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fbc34002a50 tx=0x7fbc340046b0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 shutdown_connections 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc2806c7d0 0x7fbc2806ec80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc440fde90 0x7fbc44115d10 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 --2- 172.21.15.63:0/2171124543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440fe830 0x7fbc44116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.459+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 >> 172.21.15.63:0/2171124543 conn(0x7fbc44072bc0 msgr2=0x7fbc440fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:04.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.460+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 shutdown_connections 2024-03-23T12:19:04.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.460+0000 7fbc49600700 1 -- 172.21.15.63:0/2171124543 wait complete. 2024-03-23T12:19:04.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 -- 172.21.15.63:0/3743677709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 msgr2=0x7faa9c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 --2- 172.21.15.63:0/3743677709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c0fce10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7faa84004940 tx=0x7faa84009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 -- 172.21.15.63:0/3743677709 shutdown_connections 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 --2- 172.21.15.63:0/3743677709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c0fce10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 --2- 172.21.15.63:0/3743677709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 -- 172.21.15.63:0/3743677709 >> 172.21.15.63:0/3743677709 conn(0x7faa9c0f89b0 msgr2=0x7faa9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 -- 172.21.15.63:0/3743677709 shutdown_connections 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.597+0000 7faaa0c00700 1 -- 172.21.15.63:0/3743677709 wait complete. 2024-03-23T12:19:04.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.598+0000 7faaa0c00700 1 Processor -- start 2024-03-23T12:19:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.598+0000 7faaa0c00700 1 -- start start 2024-03-23T12:19:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faaa0c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:04.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faaa0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faaa0c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c114600 con 0x7faa9c1037a0 2024-03-23T12:19:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faaa0c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c10d9f0 con 0x7faa9c102e00 2024-03-23T12:19:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faa93e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faa93e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56450/0 (socket says 172.21.15.63:56450) 2024-03-23T12:19:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faa93e00700 1 -- 172.21.15.63:0/4133550256 learned_addr learned my addr 172.21.15.63:0/4133550256 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.599+0000 7faa99e00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faa93e00700 1 -- 172.21.15.63:0/4133550256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 msgr2=0x7faa9c113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faa93e00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faa93e00700 1 -- 172.21.15.63:0/4133550256 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84005040 con 0x7faa9c1037a0 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faa93e00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c113fb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7faa84008d50 tx=0x7faa84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faa92000700 1 -- 172.21.15.63:0/4133550256 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8400e030 con 0x7faa9c1037a0 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faa92000700 1 -- 172.21.15.63:0/4133550256 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa840093b0 con 0x7faa9c1037a0 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c10dc70 con 0x7faa9c1037a0 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faa92000700 1 -- 172.21.15.63:0/4133550256 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8401cbb0 con 0x7faa9c1037a0 2024-03-23T12:19:04.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.600+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c10e230 con 0x7faa9c1037a0 2024-03-23T12:19:04.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.601+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c0fe550 con 0x7faa9c1037a0 2024-03-23T12:19:04.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.602+0000 7faa92000700 1 -- 172.21.15.63:0/4133550256 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7faa84013930 con 0x7faa9c1037a0 2024-03-23T12:19:04.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.604+0000 7faa92000700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:04.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.604+0000 7faa92000700 1 -- 172.21.15.63:0/4133550256 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa8408d320 con 0x7faa9c1037a0 2024-03-23T12:19:04.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.604+0000 7faa99e00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:04.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.605+0000 7faa99e00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009b80 tx=0x7faa8c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:04.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.608+0000 7faa92000700 1 -- 172.21.15.63:0/4133550256 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faa8405af30 con 0x7faa9c1037a0 2024-03-23T12:19:04.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:04 smithi063 ceph-mon[141124]: from='client.35538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:04.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:04 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2853296377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:19:04.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:04 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1999164125' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:19:04.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.905+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faa9c109da0 con 0x7faa9c1037a0 2024-03-23T12:19:04.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.905+0000 7faa92000700 1 -- 172.21.15.63:0/4133550256 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7faa8405aac0 con 0x7faa9c1037a0 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:19:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:19:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.907+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 msgr2=0x7faa8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.907+0000 7faaa0c00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009b80 tx=0x7faa8c00b040 comp rx=0 tx=0).stop 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.907+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 msgr2=0x7faa9c113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.907+0000 7faaa0c00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c113fb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7faa84008d50 tx=0x7faa84009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.908+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 shutdown_connections 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.908+0000 7faaa0c00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faa8006c7d0 0x7faa8006ec80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.908+0000 7faaa0c00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa9c102e00 0x7faa9c113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.908+0000 7faaa0c00700 1 --2- 172.21.15.63:0/4133550256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa9c1037a0 0x7faa9c113fb0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:04.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.908+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 >> 172.21.15.63:0/4133550256 conn(0x7faa9c0f89b0 msgr2=0x7faa9c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:04.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.908+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 shutdown_connections 2024-03-23T12:19:04.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:04.908+0000 7faaa0c00700 1 -- 172.21.15.63:0/4133550256 wait complete. 2024-03-23T12:19:05.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:05 smithi111 ceph-mon[106899]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:05.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:05 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4133550256' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:19:05.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:05 smithi111 ceph-mon[106899]: pgmap v840: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:05.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:05 smithi063 ceph-mon[141124]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:05.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:05 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4133550256' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:19:05.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:05 smithi063 ceph-mon[141124]: pgmap v840: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:07.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:07 smithi111 ceph-mon[106899]: pgmap v841: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:07.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:07 smithi063 ceph-mon[141124]: pgmap v841: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:09.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:09 smithi111 ceph-mon[106899]: pgmap v842: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:09.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:09 smithi063 ceph-mon[141124]: pgmap v842: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:12.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:11 smithi063 ceph-mon[141124]: pgmap v843: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:12.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:12.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:11 smithi111 ceph-mon[106899]: pgmap v843: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:13.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:13 smithi111 ceph-mon[106899]: pgmap v844: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:13.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:13 smithi063 ceph-mon[141124]: pgmap v844: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:15.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:15 smithi111 ceph-mon[106899]: pgmap v845: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:15.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:15 smithi063 ceph-mon[141124]: pgmap v845: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:17.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:17 smithi111 ceph-mon[106899]: pgmap v846: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:17.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:17 smithi063 ceph-mon[141124]: pgmap v846: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:19.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:19 smithi111 ceph-mon[106899]: pgmap v847: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:19.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:19 smithi063 ceph-mon[141124]: pgmap v847: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:22.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:21 smithi063 ceph-mon[141124]: pgmap v848: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:22.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:21 smithi111 ceph-mon[106899]: pgmap v848: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:23.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:23 smithi111 ceph-mon[106899]: pgmap v849: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:23.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:23 smithi063 ceph-mon[141124]: pgmap v849: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:25.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:25 smithi111 ceph-mon[106899]: pgmap v850: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:25.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:25 smithi063 ceph-mon[141124]: pgmap v850: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:27.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:28.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:27 smithi063 ceph-mon[141124]: pgmap v851: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:28.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:27 smithi111 ceph-mon[106899]: pgmap v851: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:29.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:29 smithi111 ceph-mon[106899]: pgmap v852: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:29.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:29 smithi063 ceph-mon[141124]: pgmap v852: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:32.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:31 smithi063 ceph-mon[141124]: pgmap v853: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:32.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:31 smithi111 ceph-mon[106899]: pgmap v853: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:33.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:33 smithi111 ceph-mon[106899]: pgmap v854: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:33 smithi063 ceph-mon[141124]: pgmap v854: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:35.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 -- 172.21.15.63:0/3099024370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a41037a0 msgr2=0x7f49a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 --2- 172.21.15.63:0/3099024370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a41037a0 0x7f49a40fce10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4990004940 tx=0x7f4990009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:35.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 -- 172.21.15.63:0/3099024370 shutdown_connections 2024-03-23T12:19:35.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 --2- 172.21.15.63:0/3099024370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a41037a0 0x7f49a40fce10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 --2- 172.21.15.63:0/3099024370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a4102e00 0x7f49a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 -- 172.21.15.63:0/3099024370 >> 172.21.15.63:0/3099024370 conn(0x7f49a40f89b0 msgr2=0x7f49a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:35.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 -- 172.21.15.63:0/3099024370 shutdown_connections 2024-03-23T12:19:35.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.061+0000 7f49ab400700 1 -- 172.21.15.63:0/3099024370 wait complete. 2024-03-23T12:19:35.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.062+0000 7f49ab400700 1 Processor -- start 2024-03-23T12:19:35.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.062+0000 7f49ab400700 1 -- start start 2024-03-23T12:19:35.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.062+0000 7f49ab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a4102e00 0x7f49a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.062+0000 7f49ab400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a41037a0 0x7f49a4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.062+0000 7f49ab400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4070d90 con 0x7f49a4102e00 2024-03-23T12:19:35.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.062+0000 7f49ab400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4070ed0 con 0x7f49a41037a0 2024-03-23T12:19:35.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a4102e00 0x7f49a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a4102e00 0x7f49a40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50014/0 (socket says 172.21.15.63:50014) 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a8a00700 1 -- 172.21.15.63:0/2828867109 learned_addr learned my addr 172.21.15.63:0/2828867109 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a3e00700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a41037a0 0x7f49a4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a8a00700 1 -- 172.21.15.63:0/2828867109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a41037a0 msgr2=0x7f49a4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a8a00700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a41037a0 0x7f49a4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a8a00700 1 -- 172.21.15.63:0/2828867109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990005040 con 0x7f49a4102e00 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a8a00700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a4102e00 0x7f49a40702c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4998002a50 tx=0x7f49980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.063+0000 7f49a0c00700 1 -- 172.21.15.63:0/2828867109 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4998007cc0 con 0x7f49a4102e00 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.064+0000 7f49a0c00700 1 -- 172.21.15.63:0/2828867109 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f499800f040 con 0x7f49a4102e00 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.064+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a4119e00 con 0x7f49a4102e00 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.064+0000 7f49a0c00700 1 -- 172.21.15.63:0/2828867109 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499800d9c0 con 0x7f49a4102e00 2024-03-23T12:19:35.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.064+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a411a280 con 0x7f49a4102e00 2024-03-23T12:19:35.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.065+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a40fe550 con 0x7f49a4102e00 2024-03-23T12:19:35.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.066+0000 7f49a0c00700 1 -- 172.21.15.63:0/2828867109 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f499800db20 con 0x7f49a4102e00 2024-03-23T12:19:35.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.067+0000 7f49a0c00700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f498c06c7d0 0x7f498c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.067+0000 7f49a0c00700 1 -- 172.21.15.63:0/2828867109 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f499808b860 con 0x7f49a4102e00 2024-03-23T12:19:35.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.067+0000 7f49a3e00700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f498c06c7d0 0x7f498c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.068+0000 7f49a3e00700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f498c06c7d0 0x7f498c06ec80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f49a4071c00 tx=0x7f4990009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:35.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.072+0000 7f49a0c00700 1 -- 172.21.15.63:0/2828867109 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49980559e0 con 0x7f49a4102e00 2024-03-23T12:19:35.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.306+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49a40719b0 con 0x7f498c06c7d0 2024-03-23T12:19:35.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.308+0000 7f49a0c00700 1 -- 172.21.15.63:0/2828867109 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f49a40719b0 con 0x7f498c06c7d0 2024-03-23T12:19:35.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f498c06c7d0 msgr2=0x7f498c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f498c06c7d0 0x7f498c06ec80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f49a4071c00 tx=0x7f4990009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:35.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a4102e00 msgr2=0x7f49a40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a4102e00 0x7f49a40702c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4998002a50 tx=0x7f49980046b0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 shutdown_connections 2024-03-23T12:19:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f498c06c7d0 0x7f498c06ec80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49a4102e00 0x7f49a40702c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 --2- 172.21.15.63:0/2828867109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a41037a0 0x7f49a4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 >> 172.21.15.63:0/2828867109 conn(0x7f49a40f89b0 msgr2=0x7f49a40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 shutdown_connections 2024-03-23T12:19:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.310+0000 7f49ab400700 1 -- 172.21.15.63:0/2828867109 wait complete. 2024-03-23T12:19:35.304 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:19:35.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:35 smithi111 ceph-mon[106899]: pgmap v855: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:35.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 -- 172.21.15.63:0/675528824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105090 msgr2=0x7f1394105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 --2- 172.21.15.63:0/675528824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105090 0x7f1394105460 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1384004940 tx=0x7f1384009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:35.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 -- 172.21.15.63:0/675528824 shutdown_connections 2024-03-23T12:19:35.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 --2- 172.21.15.63:0/675528824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1394105a30 0x7f13940ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 --2- 172.21.15.63:0/675528824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105090 0x7f1394105460 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 -- 172.21.15.63:0/675528824 >> 172.21.15.63:0/675528824 conn(0x7f13940fac00 msgr2=0x7f13940fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:35.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 -- 172.21.15.63:0/675528824 shutdown_connections 2024-03-23T12:19:35.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.448+0000 7f1399000700 1 -- 172.21.15.63:0/675528824 wait complete. 2024-03-23T12:19:35.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.449+0000 7f1399000700 1 Processor -- start 2024-03-23T12:19:35.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.449+0000 7f1399000700 1 -- start start 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1399000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1394105090 0x7f1394115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1399000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105a30 0x7f1394115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1399000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394116620 con 0x7f1394105a30 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1399000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139410fc00 con 0x7f1394105090 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105a30 0x7f1394115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105a30 0x7f1394115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50028/0 (socket says 172.21.15.63:50028) 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391400700 1 -- 172.21.15.63:0/4071693760 learned_addr learned my addr 172.21.15.63:0/4071693760 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391e00700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1394105090 0x7f1394115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391400700 1 -- 172.21.15.63:0/4071693760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1394105090 msgr2=0x7f1394115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391400700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1394105090 0x7f1394115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391400700 1 -- 172.21.15.63:0/4071693760 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384005040 con 0x7f1394105a30 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.450+0000 7f1391e00700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1394105090 0x7f1394115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.451+0000 7f1391400700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105a30 0x7f1394115fd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1380002a50 tx=0x7f1380004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.451+0000 7f138a000700 1 -- 172.21.15.63:0/4071693760 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1380007cc0 con 0x7f1394105a30 2024-03-23T12:19:35.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.451+0000 7f138a000700 1 -- 172.21.15.63:0/4071693760 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f138000f040 con 0x7f1394105a30 2024-03-23T12:19:35.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.451+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139410fee0 con 0x7f1394105a30 2024-03-23T12:19:35.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.451+0000 7f138a000700 1 -- 172.21.15.63:0/4071693760 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f138000d990 con 0x7f1394105a30 2024-03-23T12:19:35.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.451+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13941104a0 con 0x7f1394105a30 2024-03-23T12:19:35.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.452+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13941023a0 con 0x7f1394105a30 2024-03-23T12:19:35.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.453+0000 7f138a000700 1 -- 172.21.15.63:0/4071693760 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f1380009710 con 0x7f1394105a30 2024-03-23T12:19:35.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.455+0000 7f138a000700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f137806c7c0 0x7f137806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.455+0000 7f138a000700 1 -- 172.21.15.63:0/4071693760 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f138008b910 con 0x7f1394105a30 2024-03-23T12:19:35.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.455+0000 7f1391e00700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f137806c7c0 0x7f137806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.455+0000 7f1391e00700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f137806c7c0 0x7f137806ec70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f138400b840 tx=0x7f1384009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:35.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.459+0000 7f138a000700 1 -- 172.21.15.63:0/4071693760 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1380059520 con 0x7f1394105a30 2024-03-23T12:19:35.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:35 smithi063 ceph-mon[141124]: pgmap v855: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:35.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.693+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13941110e0 con 0x7f137806c7c0 2024-03-23T12:19:35.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.694+0000 7f138a000700 1 -- 172.21.15.63:0/4071693760 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f13941110e0 con 0x7f137806c7c0 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f137806c7c0 msgr2=0x7f137806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f137806c7c0 0x7f137806ec70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f138400b840 tx=0x7f1384009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105a30 msgr2=0x7f1394115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105a30 0x7f1394115fd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1380002a50 tx=0x7f1380004600 comp rx=0 tx=0).stop 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 shutdown_connections 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f137806c7c0 0x7f137806ec70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1394105090 0x7f1394115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 --2- 172.21.15.63:0/4071693760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1394105a30 0x7f1394115fd0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 >> 172.21.15.63:0/4071693760 conn(0x7f13940fac00 msgr2=0x7f13940fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:35.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 shutdown_connections 2024-03-23T12:19:35.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.697+0000 7f1399000700 1 -- 172.21.15.63:0/4071693760 wait complete. 2024-03-23T12:19:35.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 -- 172.21.15.63:0/1804535892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22281037a0 msgr2=0x7f22280fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 --2- 172.21.15.63:0/1804535892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22281037a0 0x7f22280fce10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2214004940 tx=0x7f2214009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:35.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 -- 172.21.15.63:0/1804535892 shutdown_connections 2024-03-23T12:19:35.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 --2- 172.21.15.63:0/1804535892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22281037a0 0x7f22280fce10 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 --2- 172.21.15.63:0/1804535892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2228102e00 0x7f22281031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 -- 172.21.15.63:0/1804535892 >> 172.21.15.63:0/1804535892 conn(0x7f22280f89b0 msgr2=0x7f22280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:35.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 -- 172.21.15.63:0/1804535892 shutdown_connections 2024-03-23T12:19:35.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.835+0000 7f222f800700 1 -- 172.21.15.63:0/1804535892 wait complete. 2024-03-23T12:19:35.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.836+0000 7f222f800700 1 Processor -- start 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.836+0000 7f222f800700 1 -- start start 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222f800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2228102e00 0x7f2228115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222f800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22281037a0 0x7f2228116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222f800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281168b0 con 0x7f2228102e00 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222f800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222810fc80 con 0x7f22281037a0 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2228102e00 0x7f2228115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2228102e00 0x7f2228115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50040/0 (socket says 172.21.15.63:50040) 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222ce00700 1 -- 172.21.15.63:0/2429888368 learned_addr learned my addr 172.21.15.63:0/2429888368 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f2227e00700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22281037a0 0x7f2228116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222ce00700 1 -- 172.21.15.63:0/2429888368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22281037a0 msgr2=0x7f2228116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222ce00700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22281037a0 0x7f2228116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.837+0000 7f222ce00700 1 -- 172.21.15.63:0/2429888368 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2214005040 con 0x7f2228102e00 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.838+0000 7f222ce00700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2228102e00 0x7f2228115d20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f221c002a50 tx=0x7f221c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.838+0000 7f2224c00700 1 -- 172.21.15.63:0/2429888368 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f221c007cc0 con 0x7f2228102e00 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.838+0000 7f2224c00700 1 -- 172.21.15.63:0/2429888368 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f221c00f040 con 0x7f2228102e00 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.838+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222810ff60 con 0x7f2228102e00 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.838+0000 7f2224c00700 1 -- 172.21.15.63:0/2429888368 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f221c00d9c0 con 0x7f2228102e00 2024-03-23T12:19:35.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.838+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2228110520 con 0x7f2228102e00 2024-03-23T12:19:35.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.839+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22280fe550 con 0x7f2228102e00 2024-03-23T12:19:35.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.840+0000 7f2224c00700 1 -- 172.21.15.63:0/2429888368 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f221c00db20 con 0x7f2228102e00 2024-03-23T12:19:35.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.842+0000 7f2224c00700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f221006c7d0 0x7f221006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:35.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.842+0000 7f2224c00700 1 -- 172.21.15.63:0/2429888368 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f221c08b6e0 con 0x7f2228102e00 2024-03-23T12:19:35.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.842+0000 7f2227e00700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f221006c7d0 0x7f221006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:35.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.843+0000 7f2227e00700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f221006c7d0 0x7f221006ec80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f22281115c0 tx=0x7f2214009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:35.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:35.847+0000 7f2224c00700 1 -- 172.21.15.63:0/2429888368 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f221c0593f0 con 0x7f2228102e00 2024-03-23T12:19:36.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.071+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f222804bf40 con 0x7f221006c7d0 2024-03-23T12:19:36.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.081+0000 7f2224c00700 1 -- 172.21.15.63:0/2429888368 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f222804bf40 con 0x7f221006c7d0 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (28m) 6m ago 41m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (26m) 6m ago 42m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (25m) 6m ago 41m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (28m) 6m ago 41m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (21m) 6m ago 35m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (21m) 6m ago 35m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (6m) 6m ago 35m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (21m) 6m ago 35m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (31m) 6m ago 43m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (29m) 6m ago 40m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (27m) 6m ago 43m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (26m) 6m ago 40m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (29m) 6m ago 42m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:19:36.061 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (29m) 6m ago 40m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:19:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (25m) 6m ago 39m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:19:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (24m) 6m ago 39m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:19:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (24m) 6m ago 38m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:19:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (23m) 6m ago 38m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:19:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (22m) 6m ago 37m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:19:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (22m) 6m ago 37m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:19:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (29m) 6m ago 41m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f221006c7d0 msgr2=0x7f221006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f221006c7d0 0x7f221006ec80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f22281115c0 tx=0x7f2214009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2228102e00 msgr2=0x7f2228115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2228102e00 0x7f2228115d20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f221c002a50 tx=0x7f221c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 shutdown_connections 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f221006c7d0 0x7f221006ec80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2228102e00 0x7f2228115d20 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 --2- 172.21.15.63:0/2429888368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22281037a0 0x7f2228116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 >> 172.21.15.63:0/2429888368 conn(0x7f22280f89b0 msgr2=0x7f22280f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 shutdown_connections 2024-03-23T12:19:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.084+0000 7f222f800700 1 -- 172.21.15.63:0/2429888368 wait complete. 2024-03-23T12:19:36.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.220+0000 7f0556c00700 1 -- 172.21.15.63:0/637064244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 msgr2=0x7f05500fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.220+0000 7f0556c00700 1 --2- 172.21.15.63:0/637064244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f05500fe260 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0540004940 tx=0x7f0540009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:36.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.220+0000 7f0556c00700 1 -- 172.21.15.63:0/637064244 shutdown_connections 2024-03-23T12:19:36.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.220+0000 7f0556c00700 1 --2- 172.21.15.63:0/637064244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05500fe830 0x7f0550106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.220+0000 7f0556c00700 1 --2- 172.21.15.63:0/637064244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f05500fe260 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.220+0000 7f0556c00700 1 -- 172.21.15.63:0/637064244 >> 172.21.15.63:0/637064244 conn(0x7f0550072bc0 msgr2=0x7f0550072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:36.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.221+0000 7f0556c00700 1 -- 172.21.15.63:0/637064244 shutdown_connections 2024-03-23T12:19:36.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.221+0000 7f0556c00700 1 -- 172.21.15.63:0/637064244 wait complete. 2024-03-23T12:19:36.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.221+0000 7f0556c00700 1 Processor -- start 2024-03-23T12:19:36.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f0556c00700 1 -- start start 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f0556c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f0550113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f0556c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05500fe830 0x7f0550114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f0556c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550114690 con 0x7f05500fde90 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f0556c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055010da80 con 0x7f05500fe830 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f054fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f0550113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f054fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f0550113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50064/0 (socket says 172.21.15.63:50064) 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.222+0000 7f054fe00700 1 -- 172.21.15.63:0/2270573135 learned_addr learned my addr 172.21.15.63:0/2270573135 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f054f400700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05500fe830 0x7f0550114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f054fe00700 1 -- 172.21.15.63:0/2270573135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05500fe830 msgr2=0x7f0550114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f054fe00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05500fe830 0x7f0550114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f054fe00700 1 -- 172.21.15.63:0/2270573135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0540005040 con 0x7f05500fde90 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f054f400700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05500fe830 0x7f0550114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:19:36.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f054fe00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f0550113b00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f05400093b0 tx=0x7f0540009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:36.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f0547e00700 1 -- 172.21.15.63:0/2270573135 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f054000e040 con 0x7f05500fde90 2024-03-23T12:19:36.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f0547e00700 1 -- 172.21.15.63:0/2270573135 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0540004290 con 0x7f05500fde90 2024-03-23T12:19:36.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055010dd00 con 0x7f05500fde90 2024-03-23T12:19:36.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f0547e00700 1 -- 172.21.15.63:0/2270573135 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f054001c3f0 con 0x7f05500fde90 2024-03-23T12:19:36.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.223+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055010e240 con 0x7f05500fde90 2024-03-23T12:19:36.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.224+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0550065e30 con 0x7f05500fde90 2024-03-23T12:19:36.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.225+0000 7f0547e00700 1 -- 172.21.15.63:0/2270573135 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0540013790 con 0x7f05500fde90 2024-03-23T12:19:36.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.227+0000 7f0547e00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f053406c7d0 0x7f053406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:36.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.227+0000 7f0547e00700 1 -- 172.21.15.63:0/2270573135 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f054008cc20 con 0x7f05500fde90 2024-03-23T12:19:36.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.227+0000 7f054f400700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f053406c7d0 0x7f053406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:36.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.227+0000 7f054f400700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f053406c7d0 0x7f053406ec80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f053c009fd0 tx=0x7f053c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:36.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.231+0000 7f0547e00700 1 -- 172.21.15.63:0/2270573135 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f054005a9f0 con 0x7f05500fde90 2024-03-23T12:19:36.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.531+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f055010f180 con 0x7f05500fde90 2024-03-23T12:19:36.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.532+0000 7f0547e00700 1 -- 172.21.15.63:0/2270573135 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f0540013420 con 0x7f05500fde90 2024-03-23T12:19:36.511 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:19:36.511 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:19:36.511 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:19:36.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:36.511 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:19:36.512 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:19:36.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f053406c7d0 msgr2=0x7f053406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f053406c7d0 0x7f053406ec80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f053c009fd0 tx=0x7f053c00b040 comp rx=0 tx=0).stop 2024-03-23T12:19:36.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 msgr2=0x7f0550113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f0550113b00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f05400093b0 tx=0x7f0540009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 shutdown_connections 2024-03-23T12:19:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f053406c7d0 0x7f053406ec80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05500fde90 0x7f0550113b00 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 --2- 172.21.15.63:0/2270573135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05500fe830 0x7f0550114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 >> 172.21.15.63:0/2270573135 conn(0x7f0550072bc0 msgr2=0x7f0550103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 shutdown_connections 2024-03-23T12:19:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.534+0000 7f0556c00700 1 -- 172.21.15.63:0/2270573135 wait complete. 2024-03-23T12:19:36.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 -- 172.21.15.63:0/3121689370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e38102e00 msgr2=0x7f9e381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3121689370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e38102e00 0x7f9e381031d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9e28004940 tx=0x7f9e28009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 -- 172.21.15.63:0/3121689370 shutdown_connections 2024-03-23T12:19:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3121689370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e381037a0 0x7f9e380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3121689370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e38102e00 0x7f9e381031d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 -- 172.21.15.63:0/3121689370 >> 172.21.15.63:0/3121689370 conn(0x7f9e380f89b0 msgr2=0x7f9e380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 -- 172.21.15.63:0/3121689370 shutdown_connections 2024-03-23T12:19:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.671+0000 7f9e3d400700 1 -- 172.21.15.63:0/3121689370 wait complete. 2024-03-23T12:19:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.672+0000 7f9e3d400700 1 Processor -- start 2024-03-23T12:19:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.672+0000 7f9e3d400700 1 -- start start 2024-03-23T12:19:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.672+0000 7f9e3d400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e38102e00 0x7f9e38115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.672+0000 7f9e3d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e381037a0 0x7f9e38116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.672+0000 7f9e3d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38116860 con 0x7f9e381037a0 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.672+0000 7f9e3d400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3810fc30 con 0x7f9e38102e00 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e35e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e381037a0 0x7f9e38116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e35e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e381037a0 0x7f9e38116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50074/0 (socket says 172.21.15.63:50074) 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e35e00700 1 -- 172.21.15.63:0/3507239761 learned_addr learned my addr 172.21.15.63:0/3507239761 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e36800700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e38102e00 0x7f9e38115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e35e00700 1 -- 172.21.15.63:0/3507239761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e38102e00 msgr2=0x7f9e38115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e35e00700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e38102e00 0x7f9e38115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e35e00700 1 -- 172.21.15.63:0/3507239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e28005040 con 0x7f9e381037a0 2024-03-23T12:19:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e35e00700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e381037a0 0x7f9e38116210 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400c890 tx=0x7f9e240044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:36.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.673+0000 7f9e2ea00700 1 -- 172.21.15.63:0/3507239761 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e2400a7d0 con 0x7f9e381037a0 2024-03-23T12:19:36.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.674+0000 7f9e2ea00700 1 -- 172.21.15.63:0/3507239761 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e2400ade0 con 0x7f9e381037a0 2024-03-23T12:19:36.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.674+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3810ff10 con 0x7f9e381037a0 2024-03-23T12:19:36.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.674+0000 7f9e2ea00700 1 -- 172.21.15.63:0/3507239761 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e240113a0 con 0x7f9e381037a0 2024-03-23T12:19:36.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.674+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381104d0 con 0x7f9e381037a0 2024-03-23T12:19:36.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.675+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e380fe550 con 0x7f9e381037a0 2024-03-23T12:19:36.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.676+0000 7f9e2ea00700 1 -- 172.21.15.63:0/3507239761 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9e24012400 con 0x7f9e381037a0 2024-03-23T12:19:36.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.677+0000 7f9e2ea00700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1c06c7d0 0x7f9e1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:36.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.677+0000 7f9e2ea00700 1 -- 172.21.15.63:0/3507239761 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9e2408bbf0 con 0x7f9e381037a0 2024-03-23T12:19:36.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.677+0000 7f9e36800700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1c06c7d0 0x7f9e1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.678+0000 7f9e36800700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1c06c7d0 0x7f9e1c06ec80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800b3b0 tx=0x7f9e28009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.681+0000 7f9e2ea00700 1 -- 172.21.15.63:0/3507239761 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e24059880 con 0x7f9e381037a0 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.939+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9e3805f5f0 con 0x7f9e381037a0 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.939+0000 7f9e2ea00700 1 -- 172.21.15.63:0/3507239761 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f9e24059410 con 0x7f9e381037a0 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:19:36.919 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:19:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:36.921 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:36.922 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:19:36.922 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:19:36.922 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:36.922 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:36.922 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:36.922 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1c06c7d0 msgr2=0x7f9e1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1c06c7d0 0x7f9e1c06ec80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800b3b0 tx=0x7f9e28009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e381037a0 msgr2=0x7f9e38116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e381037a0 0x7f9e38116210 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400c890 tx=0x7f9e240044c0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 shutdown_connections 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e1c06c7d0 0x7f9e1c06ec80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e38102e00 0x7f9e38115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 --2- 172.21.15.63:0/3507239761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e381037a0 0x7f9e38116210 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 >> 172.21.15.63:0/3507239761 conn(0x7f9e380f89b0 msgr2=0x7f9e380f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 shutdown_connections 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:36.942+0000 7f9e3d400700 1 -- 172.21.15.63:0/3507239761 wait complete. 2024-03-23T12:19:36.923 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 -- 172.21.15.63:0/925209948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 msgr2=0x7fdfc40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 --2- 172.21.15.63:0/925209948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc40fce10 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0004940 tx=0x7fdfb0009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 -- 172.21.15.63:0/925209948 shutdown_connections 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 --2- 172.21.15.63:0/925209948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc40fce10 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 --2- 172.21.15.63:0/925209948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfc4102e00 0x7fdfc41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 -- 172.21.15.63:0/925209948 >> 172.21.15.63:0/925209948 conn(0x7fdfc40f89b0 msgr2=0x7fdfc40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 -- 172.21.15.63:0/925209948 shutdown_connections 2024-03-23T12:19:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.080+0000 7fdfcc400700 1 -- 172.21.15.63:0/925209948 wait complete. 2024-03-23T12:19:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.081+0000 7fdfcc400700 1 Processor -- start 2024-03-23T12:19:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.081+0000 7fdfcc400700 1 -- start start 2024-03-23T12:19:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.081+0000 7fdfcc400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfc4102e00 0x7fdfc4115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.081+0000 7fdfcc400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc4116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfcc400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc4116890 con 0x7fdfc41037a0 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfcc400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc410fc60 con 0x7fdfc4102e00 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfc9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc4116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfc9000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc4116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50088/0 (socket says 172.21.15.63:50088) 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfc9000700 1 -- 172.21.15.63:0/120564115 learned_addr learned my addr 172.21.15.63:0/120564115 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfc9a00700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfc4102e00 0x7fdfc4115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfc9000700 1 -- 172.21.15.63:0/120564115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfc4102e00 msgr2=0x7fdfc4115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfc9000700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfc4102e00 0x7fdfc4115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.082+0000 7fdfc9000700 1 -- 172.21.15.63:0/120564115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0005040 con 0x7fdfc41037a0 2024-03-23T12:19:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.083+0000 7fdfc9000700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc4116240 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009390 tx=0x7fdfb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.083+0000 7fdfc1600700 1 -- 172.21.15.63:0/120564115 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfb000e040 con 0x7fdfc41037a0 2024-03-23T12:19:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.083+0000 7fdfc1600700 1 -- 172.21.15.63:0/120564115 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfb0004290 con 0x7fdfc41037a0 2024-03-23T12:19:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.083+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc410fee0 con 0x7fdfc41037a0 2024-03-23T12:19:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.083+0000 7fdfc1600700 1 -- 172.21.15.63:0/120564115 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfb0013e70 con 0x7fdfc41037a0 2024-03-23T12:19:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.083+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc41104a0 con 0x7fdfc41037a0 2024-03-23T12:19:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.084+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc40fe550 con 0x7fdfc41037a0 2024-03-23T12:19:37.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.085+0000 7fdfc1600700 1 -- 172.21.15.63:0/120564115 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdfb0013690 con 0x7fdfc41037a0 2024-03-23T12:19:37.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.086+0000 7fdfc1600700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfac06c7d0 0x7fdfac06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:37.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.086+0000 7fdfc1600700 1 -- 172.21.15.63:0/120564115 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdfb008cbe0 con 0x7fdfc41037a0 2024-03-23T12:19:37.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.086+0000 7fdfc9a00700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfac06c7d0 0x7fdfac06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:37.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.087+0000 7fdfc9a00700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfac06c7d0 0x7fdfac06ec80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8007fd0 tx=0x7fdfb8009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:37.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.090+0000 7fdfc1600700 1 -- 172.21.15.63:0/120564115 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdfb005a9a0 con 0x7fdfc41037a0 2024-03-23T12:19:37.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:36 smithi063 ceph-mon[141124]: from='client.35558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:37.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:36 smithi063 ceph-mon[141124]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:37.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:36 smithi063 ceph-mon[141124]: from='client.35566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:37.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:36 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2270573135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:19:37.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:36 smithi111 ceph-mon[106899]: from='client.35558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:37.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:36 smithi111 ceph-mon[106899]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:37.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:36 smithi111 ceph-mon[106899]: from='client.35566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:37.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:36 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2270573135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:19:37.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.324+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfc4109da0 con 0x7fdfac06c7d0 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.326+0000 7fdfc1600700 1 -- 172.21.15.63:0/120564115 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdfc4109da0 con 0x7fdfac06c7d0 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:19:37.305 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:19:37.306 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfac06c7d0 msgr2=0x7fdfac06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfac06c7d0 0x7fdfac06ec80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8007fd0 tx=0x7fdfb8009040 comp rx=0 tx=0).stop 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 msgr2=0x7fdfc4116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc4116240 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009390 tx=0x7fdfb0009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 shutdown_connections 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfac06c7d0 0x7fdfac06ec80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfc4102e00 0x7fdfc4115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 --2- 172.21.15.63:0/120564115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfc41037a0 0x7fdfc4116240 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 >> 172.21.15.63:0/120564115 conn(0x7fdfc40f89b0 msgr2=0x7fdfc40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 shutdown_connections 2024-03-23T12:19:37.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.328+0000 7fdfcc400700 1 -- 172.21.15.63:0/120564115 wait complete. 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 -- 172.21.15.63:0/859771923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc514071f90 msgr2=0x7fc514070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 --2- 172.21.15.63:0/859771923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc514071f90 0x7fc514070360 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc508004970 tx=0x7fc508009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 -- 172.21.15.63:0/859771923 shutdown_connections 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 --2- 172.21.15.63:0/859771923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5140708a0 0x7fc514070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 --2- 172.21.15.63:0/859771923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc514071f90 0x7fc514070360 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 -- 172.21.15.63:0/859771923 >> 172.21.15.63:0/859771923 conn(0x7fc5140f89b0 msgr2=0x7fc5140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 -- 172.21.15.63:0/859771923 shutdown_connections 2024-03-23T12:19:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.465+0000 7fc51b600700 1 -- 172.21.15.63:0/859771923 wait complete. 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.466+0000 7fc51b600700 1 Processor -- start 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.466+0000 7fc51b600700 1 -- start start 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.466+0000 7fc51b600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5140708a0 0x7fc51411a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.466+0000 7fc51b600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc514071f90 0x7fc51411a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.466+0000 7fc51b600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51411ac50 con 0x7fc5140708a0 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.466+0000 7fc51b600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc514114020 con 0x7fc514071f90 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc518c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5140708a0 0x7fc51411a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc518c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5140708a0 0x7fc51411a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50112/0 (socket says 172.21.15.63:50112) 2024-03-23T12:19:37.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc518c00700 1 -- 172.21.15.63:0/2918486899 learned_addr learned my addr 172.21.15.63:0/2918486899 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc513e00700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc514071f90 0x7fc51411a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc518c00700 1 -- 172.21.15.63:0/2918486899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc514071f90 msgr2=0x7fc51411a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc518c00700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc514071f90 0x7fc51411a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc518c00700 1 -- 172.21.15.63:0/2918486899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc508005040 con 0x7fc5140708a0 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.467+0000 7fc518c00700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5140708a0 0x7fc51411a0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc508009380 tx=0x7fc508009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.468+0000 7fc510c00700 1 -- 172.21.15.63:0/2918486899 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc50800e040 con 0x7fc5140708a0 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.468+0000 7fc510c00700 1 -- 172.21.15.63:0/2918486899 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc508013420 con 0x7fc5140708a0 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.468+0000 7fc510c00700 1 -- 172.21.15.63:0/2918486899 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc50801c630 con 0x7fc5140708a0 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.468+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5141142a0 con 0x7fc5140708a0 2024-03-23T12:19:37.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.468+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc514114790 con 0x7fc5140708a0 2024-03-23T12:19:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.469+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc514071aa0 con 0x7fc5140708a0 2024-03-23T12:19:37.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.470+0000 7fc510c00700 1 -- 172.21.15.63:0/2918486899 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc508013990 con 0x7fc5140708a0 2024-03-23T12:19:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.471+0000 7fc510c00700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4fc06c7c0 0x7fc4fc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:19:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.471+0000 7fc510c00700 1 -- 172.21.15.63:0/2918486899 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc50808cdb0 con 0x7fc5140708a0 2024-03-23T12:19:37.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.471+0000 7fc513e00700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4fc06c7c0 0x7fc4fc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:19:37.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.472+0000 7fc513e00700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4fc06c7c0 0x7fc4fc06ec70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc514115960 tx=0x7fc50000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:19:37.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.475+0000 7fc510c00700 1 -- 172.21.15.63:0/2918486899 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc50805aa40 con 0x7fc5140708a0 2024-03-23T12:19:37.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.772+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc51405f5f0 con 0x7fc5140708a0 2024-03-23T12:19:37.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.772+0000 7fc510c00700 1 -- 172.21.15.63:0/2918486899 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc508004de0 con 0x7fc5140708a0 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:19:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:19:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:19:37.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.774+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4fc06c7c0 msgr2=0x7fc4fc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.774+0000 7fc51b600700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4fc06c7c0 0x7fc4fc06ec70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc514115960 tx=0x7fc50000b040 comp rx=0 tx=0).stop 2024-03-23T12:19:37.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.774+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5140708a0 msgr2=0x7fc51411a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:19:37.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.774+0000 7fc51b600700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5140708a0 0x7fc51411a0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc508009380 tx=0x7fc508009e30 comp rx=0 tx=0).stop 2024-03-23T12:19:37.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.775+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 shutdown_connections 2024-03-23T12:19:37.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.775+0000 7fc51b600700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4fc06c7c0 0x7fc4fc06ec70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.775+0000 7fc51b600700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5140708a0 0x7fc51411a0c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.775+0000 7fc51b600700 1 --2- 172.21.15.63:0/2918486899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc514071f90 0x7fc51411a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:19:37.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.775+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 >> 172.21.15.63:0/2918486899 conn(0x7fc5140f89b0 msgr2=0x7fc5140f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:19:37.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.775+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 shutdown_connections 2024-03-23T12:19:37.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:19:37.775+0000 7fc51b600700 1 -- 172.21.15.63:0/2918486899 wait complete. 2024-03-23T12:19:38.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:37 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3507239761' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:19:38.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:37 smithi063 ceph-mon[141124]: pgmap v856: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:38.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:37 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3507239761' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:19:38.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:37 smithi111 ceph-mon[106899]: pgmap v856: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:39.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:38 smithi063 ceph-mon[141124]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:39.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:38 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2918486899' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:19:39.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:38 smithi111 ceph-mon[106899]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:19:39.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:38 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2918486899' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:19:40.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:39 smithi063 ceph-mon[141124]: pgmap v857: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:40.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:39 smithi111 ceph-mon[106899]: pgmap v857: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:41 smithi063 ceph-mon[141124]: pgmap v858: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:42.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:42.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:41 smithi111 ceph-mon[106899]: pgmap v858: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:43.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:43 smithi111 ceph-mon[106899]: pgmap v859: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:43.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:43 smithi063 ceph-mon[141124]: pgmap v859: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:45.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:45 smithi111 ceph-mon[106899]: pgmap v860: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:45.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:45 smithi063 ceph-mon[141124]: pgmap v860: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:47.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:47 smithi111 ceph-mon[106899]: pgmap v861: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:47.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:47 smithi063 ceph-mon[141124]: pgmap v861: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:49.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:49 smithi111 ceph-mon[106899]: pgmap v862: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:49.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:49 smithi063 ceph-mon[141124]: pgmap v862: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:51 smithi063 ceph-mon[141124]: pgmap v863: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:51 smithi111 ceph-mon[106899]: pgmap v863: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:53.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:53 smithi111 ceph-mon[106899]: pgmap v864: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:53.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:53 smithi063 ceph-mon[141124]: pgmap v864: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:55 smithi111 ceph-mon[106899]: pgmap v865: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:55.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:55 smithi063 ceph-mon[141124]: pgmap v865: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:57.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:19:58.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:57 smithi063 ceph-mon[141124]: pgmap v866: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:58.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:57 smithi111 ceph-mon[106899]: pgmap v866: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:59.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:19:59 smithi111 ceph-mon[106899]: pgmap v867: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:19:59.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:19:59 smithi063 ceph-mon[141124]: pgmap v867: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:00.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:20:00.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:20:00.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:20:00.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : "__main__", mod_spec) 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:20:00.379 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:19:59.999+0000 7fc328c00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: /usr/bin/podman: timeout after 895 seconds 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: Traceback (most recent call last): 2024-03-23T12:20:00.380 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: "__main__", mod_spec) 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: exec(code, run_globals) 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:20:00.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: fs cephfs is degraded 2024-03-23T12:20:00.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:20:00.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:00 smithi063 ceph-mon[141124]: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: /usr/bin/podman: timeout after 895 seconds 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: Traceback (most recent call last): 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: "__main__", mod_spec) 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:20:00.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: exec(code, run_globals) 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:00.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:20:00.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: fs cephfs is degraded 2024-03-23T12:20:00.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:20:00.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:00 smithi111 ceph-mon[106899]: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:20:02.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:01 smithi063 ceph-mon[141124]: pgmap v868: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:02.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:01 smithi111 ceph-mon[106899]: pgmap v868: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:03.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:03 smithi111 ceph-mon[106899]: pgmap v869: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:03.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:03 smithi063 ceph-mon[141124]: pgmap v869: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:05.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:05 smithi111 ceph-mon[106899]: pgmap v870: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:05.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:05 smithi063 ceph-mon[141124]: pgmap v870: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:07.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:07 smithi111 ceph-mon[106899]: pgmap v871: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:07.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:07 smithi063 ceph-mon[141124]: pgmap v871: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 -- 172.21.15.63:0/3647565434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 msgr2=0x7fcd781031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 --2- 172.21.15.63:0/3647565434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd781031d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcd68004940 tx=0x7fcd68009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 -- 172.21.15.63:0/3647565434 shutdown_connections 2024-03-23T12:20:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 --2- 172.21.15.63:0/3647565434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd781037a0 0x7fcd780fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:07.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 --2- 172.21.15.63:0/3647565434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd781031d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 -- 172.21.15.63:0/3647565434 >> 172.21.15.63:0/3647565434 conn(0x7fcd780f89b0 msgr2=0x7fcd780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 -- 172.21.15.63:0/3647565434 shutdown_connections 2024-03-23T12:20:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.930+0000 7fcd7e600700 1 -- 172.21.15.63:0/3647565434 wait complete. 2024-03-23T12:20:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.931+0000 7fcd7e600700 1 Processor -- start 2024-03-23T12:20:07.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.931+0000 7fcd7e600700 1 -- start start 2024-03-23T12:20:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.931+0000 7fcd7e600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd78115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.931+0000 7fcd7e600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd781037a0 0x7fcd78115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.931+0000 7fcd7e600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78116630 con 0x7fcd78102e00 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.931+0000 7fcd7e600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7810fc10 con 0x7fcd781037a0 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd77400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd78115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd77400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd78115aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36294/0 (socket says 172.21.15.63:36294) 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd77400700 1 -- 172.21.15.63:0/4070235403 learned_addr learned my addr 172.21.15.63:0/4070235403 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd76a00700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd781037a0 0x7fcd78115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd77400700 1 -- 172.21.15.63:0/4070235403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd781037a0 msgr2=0x7fcd78115fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd77400700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd781037a0 0x7fcd78115fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd77400700 1 -- 172.21.15.63:0/4070235403 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68005040 con 0x7fcd78102e00 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.932+0000 7fcd77400700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd78115aa0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcd68009390 tx=0x7fcd68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.933+0000 7fcd6f400700 1 -- 172.21.15.63:0/4070235403 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd6800e040 con 0x7fcd78102e00 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.933+0000 7fcd6f400700 1 -- 172.21.15.63:0/4070235403 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd68013420 con 0x7fcd78102e00 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.933+0000 7fcd6f400700 1 -- 172.21.15.63:0/4070235403 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd6801c630 con 0x7fcd78102e00 2024-03-23T12:20:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.933+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7810fe90 con 0x7fcd78102e00 2024-03-23T12:20:07.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.933+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd78110450 con 0x7fcd78102e00 2024-03-23T12:20:07.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.934+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd780fe550 con 0x7fcd78102e00 2024-03-23T12:20:07.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.935+0000 7fcd6f400700 1 -- 172.21.15.63:0/4070235403 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fcd680139e0 con 0x7fcd78102e00 2024-03-23T12:20:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.936+0000 7fcd6f400700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcd5c06c810 0x7fcd5c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.936+0000 7fcd6f400700 1 -- 172.21.15.63:0/4070235403 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcd6808cd60 con 0x7fcd78102e00 2024-03-23T12:20:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.936+0000 7fcd76a00700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcd5c06c810 0x7fcd5c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:07.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.937+0000 7fcd76a00700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcd5c06c810 0x7fcd5c06ecc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcd64007fd0 tx=0x7fcd64009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:07.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:07.941+0000 7fcd6f400700 1 -- 172.21.15.63:0/4070235403 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcd6805aaa0 con 0x7fcd78102e00 2024-03-23T12:20:08.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.174+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd780fe550 con 0x7fcd5c06c810 2024-03-23T12:20:08.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.175+0000 7fcd6f400700 1 -- 172.21.15.63:0/4070235403 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcd780fe550 con 0x7fcd5c06c810 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.177+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcd5c06c810 msgr2=0x7fcd5c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.177+0000 7fcd7e600700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcd5c06c810 0x7fcd5c06ecc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcd64007fd0 tx=0x7fcd64009040 comp rx=0 tx=0).stop 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.177+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 msgr2=0x7fcd78115aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.177+0000 7fcd7e600700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd78115aa0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcd68009390 tx=0x7fcd68009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.178+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 shutdown_connections 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.178+0000 7fcd7e600700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcd5c06c810 0x7fcd5c06ecc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.178+0000 7fcd7e600700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd78102e00 0x7fcd78115aa0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.178+0000 7fcd7e600700 1 --2- 172.21.15.63:0/4070235403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd781037a0 0x7fcd78115fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.178+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 >> 172.21.15.63:0/4070235403 conn(0x7fcd780f89b0 msgr2=0x7fcd780f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.178+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 shutdown_connections 2024-03-23T12:20:08.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.178+0000 7fcd7e600700 1 -- 172.21.15.63:0/4070235403 wait complete. 2024-03-23T12:20:08.170 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:20:08.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.315+0000 7f432f600700 1 -- 172.21.15.63:0/4067348565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fde90 msgr2=0x7f43280fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.315+0000 7f432f600700 1 --2- 172.21.15.63:0/4067348565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fde90 0x7f43280fe260 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f431c004940 tx=0x7f431c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:08.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.316+0000 7f432f600700 1 -- 172.21.15.63:0/4067348565 shutdown_connections 2024-03-23T12:20:08.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.316+0000 7f432f600700 1 --2- 172.21.15.63:0/4067348565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43280fe830 0x7f4328106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.316+0000 7f432f600700 1 --2- 172.21.15.63:0/4067348565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fde90 0x7f43280fe260 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.316+0000 7f432f600700 1 -- 172.21.15.63:0/4067348565 >> 172.21.15.63:0/4067348565 conn(0x7f4328072bc0 msgr2=0x7f4328072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:08.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.316+0000 7f432f600700 1 -- 172.21.15.63:0/4067348565 shutdown_connections 2024-03-23T12:20:08.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.316+0000 7f432f600700 1 -- 172.21.15.63:0/4067348565 wait complete. 2024-03-23T12:20:08.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.317+0000 7f432f600700 1 Processor -- start 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.317+0000 7f432f600700 1 -- start start 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.317+0000 7f432f600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43280fde90 0x7f4328115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.317+0000 7f432f600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fe830 0x7f4328116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.317+0000 7f432f600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43281168b0 con 0x7f43280fe830 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.317+0000 7f432f600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432810fc80 con 0x7f43280fde90 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.317+0000 7f4327e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fe830 0x7f4328116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f432cc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43280fde90 0x7f4328115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f4327e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fe830 0x7f4328116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36304/0 (socket says 172.21.15.63:36304) 2024-03-23T12:20:08.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f4327e00700 1 -- 172.21.15.63:0/1293324636 learned_addr learned my addr 172.21.15.63:0/1293324636 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:08.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f4327e00700 1 -- 172.21.15.63:0/1293324636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43280fde90 msgr2=0x7f4328115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f4327e00700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43280fde90 0x7f4328115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f4327e00700 1 -- 172.21.15.63:0/1293324636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c005040 con 0x7f43280fe830 2024-03-23T12:20:08.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f4327e00700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fe830 0x7f4328116260 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f431400c8c0 tx=0x7f43140044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:08.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.318+0000 7f4324c00700 1 -- 172.21.15.63:0/1293324636 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f431400a760 con 0x7f43280fe830 2024-03-23T12:20:08.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.319+0000 7f4324c00700 1 -- 172.21.15.63:0/1293324636 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f431400ad70 con 0x7f43280fe830 2024-03-23T12:20:08.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.319+0000 7f4324c00700 1 -- 172.21.15.63:0/1293324636 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43140113a0 con 0x7f43280fe830 2024-03-23T12:20:08.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.319+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432810ff60 con 0x7f43280fe830 2024-03-23T12:20:08.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.319+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43281104a0 con 0x7f43280fe830 2024-03-23T12:20:08.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.320+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4328106a20 con 0x7f43280fe830 2024-03-23T12:20:08.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.321+0000 7f4324c00700 1 -- 172.21.15.63:0/1293324636 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4314012400 con 0x7f43280fe830 2024-03-23T12:20:08.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.322+0000 7f4324c00700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f431006c7d0 0x7f431006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:08.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.322+0000 7f4324c00700 1 -- 172.21.15.63:0/1293324636 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f431408b9a0 con 0x7f43280fe830 2024-03-23T12:20:08.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.322+0000 7f432cc00700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f431006c7d0 0x7f431006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:08.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.323+0000 7f432cc00700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f431006c7d0 0x7f431006ec80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f431c00b3b0 tx=0x7f431c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:08.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.326+0000 7f4324c00700 1 -- 172.21.15.63:0/1293324636 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4314059770 con 0x7f43280fe830 2024-03-23T12:20:08.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.557+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4328111370 con 0x7f431006c7d0 2024-03-23T12:20:08.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.559+0000 7f4324c00700 1 -- 172.21.15.63:0/1293324636 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4328111370 con 0x7f431006c7d0 2024-03-23T12:20:08.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f431006c7d0 msgr2=0x7f431006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f431006c7d0 0x7f431006ec80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f431c00b3b0 tx=0x7f431c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:08.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fe830 msgr2=0x7f4328116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fe830 0x7f4328116260 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f431400c8c0 tx=0x7f43140044c0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 shutdown_connections 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f431006c7d0 0x7f431006ec80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43280fde90 0x7f4328115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 --2- 172.21.15.63:0/1293324636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f43280fe830 0x7f4328116260 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 >> 172.21.15.63:0/1293324636 conn(0x7f4328072bc0 msgr2=0x7f43280fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 shutdown_connections 2024-03-23T12:20:08.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.561+0000 7f432f600700 1 -- 172.21.15.63:0/1293324636 wait complete. 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 -- 172.21.15.63:0/3655186518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac102ca0 msgr2=0x7f62ac103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 --2- 172.21.15.63:0/3655186518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac102ca0 0x7f62ac103070 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f62a0004940 tx=0x7f62a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 -- 172.21.15.63:0/3655186518 shutdown_connections 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 --2- 172.21.15.63:0/3655186518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62ac065630 0x7f62ac065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 --2- 172.21.15.63:0/3655186518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac102ca0 0x7f62ac103070 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 -- 172.21.15.63:0/3655186518 >> 172.21.15.63:0/3655186518 conn(0x7f62ac071b70 msgr2=0x7f62ac071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 -- 172.21.15.63:0/3655186518 shutdown_connections 2024-03-23T12:20:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.699+0000 7f62b1e00700 1 -- 172.21.15.63:0/3655186518 wait complete. 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.700+0000 7f62b1e00700 1 Processor -- start 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.700+0000 7f62b1e00700 1 -- start start 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac065630 0x7f62ac115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b1e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62ac102ca0 0x7f62ac115f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b1e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac1165c0 con 0x7f62ac065630 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b1e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac10f9b0 con 0x7f62ac102ca0 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b0a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac065630 0x7f62ac115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:08.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b0a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac065630 0x7f62ac115a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36324/0 (socket says 172.21.15.63:36324) 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b0a00700 1 -- 172.21.15.63:0/654280459 learned_addr learned my addr 172.21.15.63:0/654280459 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62abe00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62ac102ca0 0x7f62ac115f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b0a00700 1 -- 172.21.15.63:0/654280459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62ac102ca0 msgr2=0x7f62ac115f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b0a00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62ac102ca0 0x7f62ac115f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.701+0000 7f62b0a00700 1 -- 172.21.15.63:0/654280459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a0005040 con 0x7f62ac065630 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.702+0000 7f62b0a00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac065630 0x7f62ac115a30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f62a0007f50 tx=0x7f62a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.702+0000 7f62a8c00700 1 -- 172.21.15.63:0/654280459 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a0021070 con 0x7f62ac065630 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.702+0000 7f62a8c00700 1 -- 172.21.15.63:0/654280459 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62a00093b0 con 0x7f62ac065630 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.702+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac10fc30 con 0x7f62ac065630 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.702+0000 7f62a8c00700 1 -- 172.21.15.63:0/654280459 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a000e070 con 0x7f62ac065630 2024-03-23T12:20:08.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.702+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac110190 con 0x7f62ac065630 2024-03-23T12:20:08.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.703+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac106130 con 0x7f62ac065630 2024-03-23T12:20:08.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.704+0000 7f62a8c00700 1 -- 172.21.15.63:0/654280459 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f62a001c400 con 0x7f62ac065630 2024-03-23T12:20:08.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.705+0000 7f62a8c00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f629406c7d0 0x7f629406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:08.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.705+0000 7f62a8c00700 1 -- 172.21.15.63:0/654280459 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f62a008cb60 con 0x7f62ac065630 2024-03-23T12:20:08.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.705+0000 7f62abe00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f629406c7d0 0x7f629406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:08.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.706+0000 7f62abe00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f629406c7d0 0x7f629406ec80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f62ac110b70 tx=0x7f629800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:08.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.709+0000 7f62a8c00700 1 -- 172.21.15.63:0/654280459 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f62a005a870 con 0x7f62ac065630 2024-03-23T12:20:08.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.941+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f62ac111080 con 0x7f629406c7d0 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.952+0000 7f62a8c00700 1 -- 172.21.15.63:0/654280459 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f62ac111080 con 0x7f629406c7d0 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (29m) 6m ago 42m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (26m) 6m ago 42m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (26m) 6m ago 41m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (29m) 6m ago 42m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (22m) 6m ago 36m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:20:08.930 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (22m) 6m ago 36m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (6m) 6m ago 36m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (21m) 6m ago 36m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (31m) 6m ago 44m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (30m) 6m ago 41m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (27m) 6m ago 44m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (27m) 6m ago 41m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (30m) 6m ago 42m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (30m) 6m ago 41m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (26m) 6m ago 40m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (25m) 6m ago 39m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (24m) 6m ago 39m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (24m) 6m ago 38m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (23m) 6m ago 38m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (22m) 6m ago 37m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:20:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (29m) 6m ago 42m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:20:08.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f629406c7d0 msgr2=0x7f629406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f629406c7d0 0x7f629406ec80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f62ac110b70 tx=0x7f629800b040 comp rx=0 tx=0).stop 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac065630 msgr2=0x7f62ac115a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac065630 0x7f62ac115a30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f62a0007f50 tx=0x7f62a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 shutdown_connections 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f629406c7d0 0x7f629406ec80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62ac065630 0x7f62ac115a30 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 --2- 172.21.15.63:0/654280459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62ac102ca0 0x7f62ac115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 >> 172.21.15.63:0/654280459 conn(0x7f62ac071b70 msgr2=0x7f62ac072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.954+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 shutdown_connections 2024-03-23T12:20:08.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:08.955+0000 7f62b1e00700 1 -- 172.21.15.63:0/654280459 wait complete. 2024-03-23T12:20:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 -- 172.21.15.63:0/1622745088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 msgr2=0x7fa7e81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1622745088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e81031d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc004940 tx=0x7fa7dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 -- 172.21.15.63:0/1622745088 shutdown_connections 2024-03-23T12:20:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1622745088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e81037a0 0x7fa7e80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1622745088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e81031d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 -- 172.21.15.63:0/1622745088 >> 172.21.15.63:0/1622745088 conn(0x7fa7e80f89b0 msgr2=0x7fa7e80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 -- 172.21.15.63:0/1622745088 shutdown_connections 2024-03-23T12:20:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.090+0000 7fa7ef600700 1 -- 172.21.15.63:0/1622745088 wait complete. 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.091+0000 7fa7ef600700 1 Processor -- start 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.091+0000 7fa7ef600700 1 -- start start 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.091+0000 7fa7ef600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.091+0000 7fa7ef600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e81037a0 0x7fa7e8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.091+0000 7fa7ef600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e8070d90 con 0x7fa7e8102e00 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.091+0000 7fa7ef600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e8070ed0 con 0x7fa7e81037a0 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7ecc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7ecc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36342/0 (socket says 172.21.15.63:36342) 2024-03-23T12:20:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7ecc00700 1 -- 172.21.15.63:0/1786401237 learned_addr learned my addr 172.21.15.63:0/1786401237 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7ecc00700 1 -- 172.21.15.63:0/1786401237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e81037a0 msgr2=0x7fa7e8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7e7e00700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e81037a0 0x7fa7e8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7ecc00700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e81037a0 0x7fa7e8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7ecc00700 1 -- 172.21.15.63:0/1786401237 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc005040 con 0x7fa7e8102e00 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7e7e00700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e81037a0 0x7fa7e8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.092+0000 7fa7ecc00700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e80702c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009390 tx=0x7fa7dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.093+0000 7fa7e4c00700 1 -- 172.21.15.63:0/1786401237 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc00e040 con 0x7fa7e8102e00 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.093+0000 7fa7e4c00700 1 -- 172.21.15.63:0/1786401237 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7dc013420 con 0x7fa7e8102e00 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.093+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e8071120 con 0x7fa7e8102e00 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.093+0000 7fa7e4c00700 1 -- 172.21.15.63:0/1786401237 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7dc01c630 con 0x7fa7e8102e00 2024-03-23T12:20:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.093+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e811a250 con 0x7fa7e8102e00 2024-03-23T12:20:09.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.094+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e80fe550 con 0x7fa7e8102e00 2024-03-23T12:20:09.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.095+0000 7fa7e4c00700 1 -- 172.21.15.63:0/1786401237 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa7dc0139e0 con 0x7fa7e8102e00 2024-03-23T12:20:09.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.096+0000 7fa7e4c00700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.096+0000 7fa7e4c00700 1 -- 172.21.15.63:0/1786401237 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa7dc08cbe0 con 0x7fa7e8102e00 2024-03-23T12:20:09.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.096+0000 7fa7e7e00700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.097+0000 7fa7e7e00700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009fd0 tx=0x7fa7d400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.100+0000 7fa7e4c00700 1 -- 172.21.15.63:0/1786401237 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa7dc056f20 con 0x7fa7e8102e00 2024-03-23T12:20:09.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.399+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa7e80719b0 con 0x7fa7e8102e00 2024-03-23T12:20:09.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.400+0000 7fa7e4c00700 1 -- 172.21.15.63:0/1786401237 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fa7dc05a540 con 0x7fa7e8102e00 2024-03-23T12:20:09.378 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:09.379 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:20:09.380 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:20:09.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:20:09.380 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.402+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 msgr2=0x7fa7d006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.402+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009fd0 tx=0x7fa7d400b040 comp rx=0 tx=0).stop 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.402+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 msgr2=0x7fa7e80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.402+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e80702c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009390 tx=0x7fa7dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.403+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 shutdown_connections 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.403+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa7d006c7d0 0x7fa7d006ec80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.403+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7e8102e00 0x7fa7e80702c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.403+0000 7fa7ef600700 1 --2- 172.21.15.63:0/1786401237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e81037a0 0x7fa7e8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.403+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 >> 172.21.15.63:0/1786401237 conn(0x7fa7e80f89b0 msgr2=0x7fa7e80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:09.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.403+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 shutdown_connections 2024-03-23T12:20:09.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.403+0000 7fa7ef600700 1 -- 172.21.15.63:0/1786401237 wait complete. 2024-03-23T12:20:09.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:09 smithi111 ceph-mon[106899]: from='client.35586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:09.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:09 smithi111 ceph-mon[106899]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:09.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:09 smithi111 ceph-mon[106899]: from='client.35594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:09.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:09 smithi111 ceph-mon[106899]: pgmap v872: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 -- 172.21.15.63:0/4104010651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 msgr2=0x7f5a70070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 --2- 172.21.15.63:0/4104010651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70070d10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c004970 tx=0x7f5a5c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 -- 172.21.15.63:0/4104010651 shutdown_connections 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 --2- 172.21.15.63:0/4104010651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70070d10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 --2- 172.21.15.63:0/4104010651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a70071f90 0x7f5a70070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 -- 172.21.15.63:0/4104010651 >> 172.21.15.63:0/4104010651 conn(0x7f5a700f8830 msgr2=0x7f5a700fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 -- 172.21.15.63:0/4104010651 shutdown_connections 2024-03-23T12:20:09.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.540+0000 7f5a74a00700 1 -- 172.21.15.63:0/4104010651 wait complete. 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.541+0000 7f5a74a00700 1 Processor -- start 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a74a00700 1 -- start start 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a74a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70119f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a74a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a70071f90 0x7f5a7011a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a74a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7011aad0 con 0x7f5a700708a0 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a74a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a70113ea0 con 0x7f5a70071f90 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a6e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70119f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a6e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70119f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36358/0 (socket says 172.21.15.63:36358) 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.542+0000 7f5a6e800700 1 -- 172.21.15.63:0/1946958572 learned_addr learned my addr 172.21.15.63:0/1946958572 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:09.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a6e800700 1 -- 172.21.15.63:0/1946958572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a70071f90 msgr2=0x7f5a7011a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a6e800700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a70071f90 0x7f5a7011a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a6e800700 1 -- 172.21.15.63:0/1946958572 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c005040 con 0x7f5a700708a0 2024-03-23T12:20:09.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a6e800700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70119f40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000eb70 tx=0x7f5a6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:09.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a66a00700 1 -- 172.21.15.63:0/1946958572 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a60007cc0 con 0x7f5a700708a0 2024-03-23T12:20:09.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a66a00700 1 -- 172.21.15.63:0/1946958572 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a6000f040 con 0x7f5a700708a0 2024-03-23T12:20:09.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a66a00700 1 -- 172.21.15.63:0/1946958572 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a600095f0 con 0x7f5a700708a0 2024-03-23T12:20:09.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70114120 con 0x7f5a700708a0 2024-03-23T12:20:09.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.543+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a70114660 con 0x7f5a700708a0 2024-03-23T12:20:09.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.545+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a70071aa0 con 0x7f5a700708a0 2024-03-23T12:20:09.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.546+0000 7f5a66a00700 1 -- 172.21.15.63:0/1946958572 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5a60009750 con 0x7f5a700708a0 2024-03-23T12:20:09.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.547+0000 7f5a66a00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a5406c7d0 0x7f5a5406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.547+0000 7f5a66a00700 1 -- 172.21.15.63:0/1946958572 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5a6008b660 con 0x7f5a700708a0 2024-03-23T12:20:09.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.547+0000 7f5a6de00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a5406c7d0 0x7f5a5406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.548+0000 7f5a6de00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a5406c7d0 0x7f5a5406ec80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00b760 tx=0x7f5a5c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:09.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.551+0000 7f5a66a00700 1 -- 172.21.15.63:0/1946958572 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5a60059370 con 0x7f5a700708a0 2024-03-23T12:20:09.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:09 smithi063 ceph-mon[141124]: from='client.35586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:09.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:09 smithi063 ceph-mon[141124]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:09.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:09 smithi063 ceph-mon[141124]: from='client.35594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:09.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:09 smithi063 ceph-mon[141124]: pgmap v872: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:09.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.813+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5a7004bed0 con 0x7f5a700708a0 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.813+0000 7f5a66a00700 1 -- 172.21.15.63:0/1946958572 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f5a60058f00 con 0x7f5a700708a0 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:09.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a5406c7d0 msgr2=0x7f5a5406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a5406c7d0 0x7f5a5406ec80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00b760 tx=0x7f5a5c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:09.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 msgr2=0x7f5a70119f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70119f40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000eb70 tx=0x7f5a6000b040 comp rx=0 tx=0).stop 2024-03-23T12:20:09.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 shutdown_connections 2024-03-23T12:20:09.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5a5406c7d0 0x7f5a5406ec80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a700708a0 0x7f5a70119f40 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 --2- 172.21.15.63:0/1946958572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a70071f90 0x7f5a7011a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 >> 172.21.15.63:0/1946958572 conn(0x7f5a700f8830 msgr2=0x7f5a700f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:09.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 shutdown_connections 2024-03-23T12:20:09.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.816+0000 7f5a74a00700 1 -- 172.21.15.63:0/1946958572 wait complete. 2024-03-23T12:20:09.797 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:20:09.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 -- 172.21.15.63:0/991740377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 msgr2=0x7fb3a8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 --2- 172.21.15.63:0/991740377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a8106f00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb394004940 tx=0x7fb394009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:09.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 -- 172.21.15.63:0/991740377 shutdown_connections 2024-03-23T12:20:09.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 --2- 172.21.15.63:0/991740377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a8106f00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 --2- 172.21.15.63:0/991740377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a80fde90 0x7fb3a80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 -- 172.21.15.63:0/991740377 >> 172.21.15.63:0/991740377 conn(0x7fb3a8072bc0 msgr2=0x7fb3a8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:09.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 -- 172.21.15.63:0/991740377 shutdown_connections 2024-03-23T12:20:09.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.953+0000 7fb3ad800700 1 -- 172.21.15.63:0/991740377 wait complete. 2024-03-23T12:20:09.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.954+0000 7fb3ad800700 1 Processor -- start 2024-03-23T12:20:09.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.954+0000 7fb3ad800700 1 -- start start 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.954+0000 7fb3ad800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a80fde90 0x7fb3a8115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3ad800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a81161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3ad800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8116800 con 0x7fb3a80fe830 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3ad800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a810fbd0 con 0x7fb3a80fde90 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a81161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a81161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36380/0 (socket says 172.21.15.63:36380) 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6000700 1 -- 172.21.15.63:0/1782610624 learned_addr learned my addr 172.21.15.63:0/1782610624 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6a00700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a80fde90 0x7fb3a8115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6000700 1 -- 172.21.15.63:0/1782610624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a80fde90 msgr2=0x7fb3a8115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6000700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a80fde90 0x7fb3a8115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6000700 1 -- 172.21.15.63:0/1782610624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394005040 con 0x7fb3a80fe830 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.955+0000 7fb3a6000700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a81161b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb394000c00 tx=0x7fb394009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.956+0000 7fb39ea00700 1 -- 172.21.15.63:0/1782610624 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb39400e040 con 0x7fb3a80fe830 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.956+0000 7fb39ea00700 1 -- 172.21.15.63:0/1782610624 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb394013420 con 0x7fb3a80fe830 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.956+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a810fe50 con 0x7fb3a80fe830 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.956+0000 7fb39ea00700 1 -- 172.21.15.63:0/1782610624 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb39401c630 con 0x7fb3a80fe830 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.956+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a8110390 con 0x7fb3a80fe830 2024-03-23T12:20:09.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.957+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a8106a20 con 0x7fb3a80fe830 2024-03-23T12:20:09.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.958+0000 7fb39ea00700 1 -- 172.21.15.63:0/1782610624 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb394013970 con 0x7fb3a80fe830 2024-03-23T12:20:09.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.959+0000 7fb39ea00700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb38c06c7d0 0x7fb38c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:09.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.959+0000 7fb39ea00700 1 -- 172.21.15.63:0/1782610624 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb39408cc70 con 0x7fb3a80fe830 2024-03-23T12:20:09.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.959+0000 7fb3a6a00700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb38c06c7d0 0x7fb38c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:09.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.959+0000 7fb3a6a00700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb38c06c7d0 0x7fb38c06ec80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb398009fd0 tx=0x7fb398009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:09.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:09.963+0000 7fb39ea00700 1 -- 172.21.15.63:0/1782610624 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb394093080 con 0x7fb3a80fe830 2024-03-23T12:20:10.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.195+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3a81112c0 con 0x7fb38c06c7d0 2024-03-23T12:20:10.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.196+0000 7fb39ea00700 1 -- 172.21.15.63:0/1782610624 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb3a81112c0 con 0x7fb38c06c7d0 2024-03-23T12:20:10.174 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:20:10.174 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb38c06c7d0 msgr2=0x7fb38c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb38c06c7d0 0x7fb38c06ec80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb398009fd0 tx=0x7fb398009f60 comp rx=0 tx=0).stop 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 msgr2=0x7fb3a81161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a81161b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb394000c00 tx=0x7fb394009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 shutdown_connections 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb38c06c7d0 0x7fb38c06ec80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a80fde90 0x7fb3a8115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.198+0000 7fb3ad800700 1 --2- 172.21.15.63:0/1782610624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3a80fe830 0x7fb3a81161b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.199+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 >> 172.21.15.63:0/1782610624 conn(0x7fb3a8072bc0 msgr2=0x7fb3a80fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.199+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 shutdown_connections 2024-03-23T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.199+0000 7fb3ad800700 1 -- 172.21.15.63:0/1782610624 wait complete. 2024-03-23T12:20:10.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.334+0000 7f162dc00700 1 -- 172.21.15.63:0/3865364038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fe850 msgr2=0x7f1628106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:10.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.334+0000 7f162dc00700 1 --2- 172.21.15.63:0/3865364038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fe850 0x7f1628106f20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1614004970 tx=0x7f1614009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:10.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.335+0000 7f162dc00700 1 -- 172.21.15.63:0/3865364038 shutdown_connections 2024-03-23T12:20:10.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.335+0000 7f162dc00700 1 --2- 172.21.15.63:0/3865364038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fe850 0x7f1628106f20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.335+0000 7f162dc00700 1 --2- 172.21.15.63:0/3865364038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16280fdeb0 0x7f16280fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.335+0000 7f162dc00700 1 -- 172.21.15.63:0/3865364038 >> 172.21.15.63:0/3865364038 conn(0x7f1628072bc0 msgr2=0x7f1628072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:10.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.335+0000 7f162dc00700 1 -- 172.21.15.63:0/3865364038 shutdown_connections 2024-03-23T12:20:10.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.335+0000 7f162dc00700 1 -- 172.21.15.63:0/3865364038 wait complete. 2024-03-23T12:20:10.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.335+0000 7f162dc00700 1 Processor -- start 2024-03-23T12:20:10.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f162dc00700 1 -- start start 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f162dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fdeb0 0x7f1628113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f162dc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16280fe850 0x7f1628113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f162dc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16281145a0 con 0x7f16280fdeb0 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f162dc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162810d990 con 0x7f16280fe850 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f1626a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fdeb0 0x7f1628113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f1626a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fdeb0 0x7f1628113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36406/0 (socket says 172.21.15.63:36406) 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.336+0000 7f1626a00700 1 -- 172.21.15.63:0/1274169254 learned_addr learned my addr 172.21.15.63:0/1274169254 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f1626000700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16280fe850 0x7f1628113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f1626a00700 1 -- 172.21.15.63:0/1274169254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16280fe850 msgr2=0x7f1628113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f1626a00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16280fe850 0x7f1628113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f1626a00700 1 -- 172.21.15.63:0/1274169254 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1614005040 con 0x7f16280fdeb0 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f1626a00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fdeb0 0x7f1628113a10 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1618002a50 tx=0x7f16180046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f161ea00700 1 -- 172.21.15.63:0/1274169254 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1618007cc0 con 0x7f16280fdeb0 2024-03-23T12:20:10.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f161ea00700 1 -- 172.21.15.63:0/1274169254 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f161800f040 con 0x7f16280fdeb0 2024-03-23T12:20:10.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162810dc70 con 0x7f16280fdeb0 2024-03-23T12:20:10.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.337+0000 7f161ea00700 1 -- 172.21.15.63:0/1274169254 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1618009540 con 0x7f16280fdeb0 2024-03-23T12:20:10.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.338+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162810e180 con 0x7f16280fdeb0 2024-03-23T12:20:10.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.339+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1628065e10 con 0x7f16280fdeb0 2024-03-23T12:20:10.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.340+0000 7f161ea00700 1 -- 172.21.15.63:0/1274169254 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f16180096a0 con 0x7f16280fdeb0 2024-03-23T12:20:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.341+0000 7f161ea00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f160c06c7c0 0x7f160c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.341+0000 7f161ea00700 1 -- 172.21.15.63:0/1274169254 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f161808b8a0 con 0x7f16280fdeb0 2024-03-23T12:20:10.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.341+0000 7f1626000700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f160c06c7c0 0x7f160c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:10.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.342+0000 7f1626000700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f160c06c7c0 0x7f160c06ec70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1614004f50 tx=0x7f1614009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:10.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.345+0000 7f161ea00700 1 -- 172.21.15.63:0/1274169254 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1618059530 con 0x7f16280fdeb0 2024-03-23T12:20:10.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:10 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1786401237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:20:10.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:10 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1946958572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:20:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.643+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1628109e20 con 0x7f16280fdeb0 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.643+0000 7f161ea00700 1 -- 172.21.15.63:0/1274169254 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f16180590c0 con 0x7f16280fdeb0 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:20:10.622 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:20:10.623 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:10.624 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:20:10.624 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:20:10.624 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:20:10.624 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:20:10.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f160c06c7c0 msgr2=0x7f160c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f160c06c7c0 0x7f160c06ec70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1614004f50 tx=0x7f1614009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fdeb0 msgr2=0x7f1628113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fdeb0 0x7f1628113a10 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1618002a50 tx=0x7f16180046b0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 shutdown_connections 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f160c06c7c0 0x7f160c06ec70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16280fdeb0 0x7f1628113a10 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 --2- 172.21.15.63:0/1274169254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16280fe850 0x7f1628113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 >> 172.21.15.63:0/1274169254 conn(0x7f1628072bc0 msgr2=0x7f1628103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 shutdown_connections 2024-03-23T12:20:10.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:10.646+0000 7f162dc00700 1 -- 172.21.15.63:0/1274169254 wait complete. 2024-03-23T12:20:10.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:10 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1786401237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:20:10.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:10 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1946958572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:20:11.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:11 smithi111 ceph-mon[106899]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:11.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:11 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1274169254' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:20:11.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:11.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:11 smithi111 ceph-mon[106899]: pgmap v873: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:11.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:11 smithi063 ceph-mon[141124]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:11.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:11 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1274169254' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:20:11.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:11.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:11 smithi063 ceph-mon[141124]: pgmap v873: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:13.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:13 smithi111 ceph-mon[106899]: pgmap v874: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:13.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:13 smithi063 ceph-mon[141124]: pgmap v874: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:15.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:15 smithi111 ceph-mon[106899]: pgmap v875: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:15.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:15 smithi063 ceph-mon[141124]: pgmap v875: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:17.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:17 smithi111 ceph-mon[106899]: pgmap v876: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:17.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:17 smithi063 ceph-mon[141124]: pgmap v876: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:19.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:19 smithi111 ceph-mon[106899]: pgmap v877: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:19.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:19 smithi063 ceph-mon[141124]: pgmap v877: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:22.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:21 smithi063 ceph-mon[141124]: pgmap v878: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:22.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:21 smithi111 ceph-mon[106899]: pgmap v878: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:23.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:23 smithi111 ceph-mon[106899]: pgmap v879: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:23.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:23 smithi063 ceph-mon[141124]: pgmap v879: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:25.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:25 smithi111 ceph-mon[106899]: pgmap v880: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:25.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:25 smithi063 ceph-mon[141124]: pgmap v880: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:28.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:27 smithi063 ceph-mon[141124]: pgmap v881: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:28.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:27 smithi111 ceph-mon[106899]: pgmap v881: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:29.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:29 smithi063 ceph-mon[141124]: pgmap v882: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:29.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:29 smithi111 ceph-mon[106899]: pgmap v882: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:32.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:31 smithi063 ceph-mon[141124]: pgmap v883: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:32.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:31 smithi111 ceph-mon[106899]: pgmap v883: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:33.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:33 smithi063 ceph-mon[141124]: pgmap v884: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:33.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:33 smithi111 ceph-mon[106899]: pgmap v884: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:35.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:35 smithi063 ceph-mon[141124]: pgmap v885: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:35.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:35 smithi111 ceph-mon[106899]: pgmap v885: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:37.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:37 smithi063 ceph-mon[141124]: pgmap v886: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:37.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:37 smithi111 ceph-mon[106899]: pgmap v886: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:39.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:39 smithi063 ceph-mon[141124]: pgmap v887: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:39.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:39 smithi111 ceph-mon[106899]: pgmap v887: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.799+0000 7f676b200700 1 -- 172.21.15.63:0/1776329786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6764100be0 msgr2=0x7f6764101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.799+0000 7f676b200700 1 --2- 172.21.15.63:0/1776329786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6764100be0 0x7f6764101050 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f674c004940 tx=0x7f674c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.799+0000 7f676b200700 1 -- 172.21.15.63:0/1776329786 shutdown_connections 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.799+0000 7f676b200700 1 --2- 172.21.15.63:0/1776329786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6764100be0 0x7f6764101050 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.799+0000 7f676b200700 1 --2- 172.21.15.63:0/1776329786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67640ff3e0 0x7f67640ff7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.799+0000 7f676b200700 1 -- 172.21.15.63:0/1776329786 >> 172.21.15.63:0/1776329786 conn(0x7f67640faba0 msgr2=0x7f67640fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.800+0000 7f676b200700 1 -- 172.21.15.63:0/1776329786 shutdown_connections 2024-03-23T12:20:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.800+0000 7f676b200700 1 -- 172.21.15.63:0/1776329786 wait complete. 2024-03-23T12:20:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.800+0000 7f676b200700 1 Processor -- start 2024-03-23T12:20:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f676b200700 1 -- start start 2024-03-23T12:20:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f676b200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67640ff3e0 0x7f6764115a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f676b200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6764100be0 0x7f6764115f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f676b200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764116590 con 0x7f67640ff3e0 2024-03-23T12:20:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f676b200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676410fb90 con 0x7f6764100be0 2024-03-23T12:20:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f6763e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67640ff3e0 0x7f6764115a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f6763e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67640ff3e0 0x7f6764115a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51804/0 (socket says 172.21.15.63:51804) 2024-03-23T12:20:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f6763e00700 1 -- 172.21.15.63:0/4216596769 learned_addr learned my addr 172.21.15.63:0/4216596769 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.801+0000 7f675be00700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6764100be0 0x7f6764115f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f6763e00700 1 -- 172.21.15.63:0/4216596769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6764100be0 msgr2=0x7f6764115f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f6763e00700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6764100be0 0x7f6764115f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f6763e00700 1 -- 172.21.15.63:0/4216596769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c005040 con 0x7f67640ff3e0 2024-03-23T12:20:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f6763e00700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67640ff3e0 0x7f6764115a00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6754002a80 tx=0x7f67540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f6760c00700 1 -- 172.21.15.63:0/4216596769 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6754007cc0 con 0x7f67640ff3e0 2024-03-23T12:20:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f6760c00700 1 -- 172.21.15.63:0/4216596769 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f675400f040 con 0x7f67640ff3e0 2024-03-23T12:20:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676410fe70 con 0x7f67640ff3e0 2024-03-23T12:20:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.802+0000 7f6760c00700 1 -- 172.21.15.63:0/4216596769 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f675400da60 con 0x7f67640ff3e0 2024-03-23T12:20:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.803+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6764110430 con 0x7f67640ff3e0 2024-03-23T12:20:40.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.804+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6764107620 con 0x7f67640ff3e0 2024-03-23T12:20:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.805+0000 7f6760c00700 1 -- 172.21.15.63:0/4216596769 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f675400dbc0 con 0x7f67640ff3e0 2024-03-23T12:20:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.806+0000 7f6760c00700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f674806c7c0 0x7f674806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.806+0000 7f6760c00700 1 -- 172.21.15.63:0/4216596769 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f675408b500 con 0x7f67640ff3e0 2024-03-23T12:20:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.806+0000 7f675be00700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f674806c7c0 0x7f674806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.807+0000 7f675be00700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f674806c7c0 0x7f674806ec70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6764111230 tx=0x7f674c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:40.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:40.812+0000 7f6760c00700 1 -- 172.21.15.63:0/4216596769 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6754059210 con 0x7f67640ff3e0 2024-03-23T12:20:41.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.046+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f676404c780 con 0x7f674806c7c0 2024-03-23T12:20:41.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.047+0000 7f6760c00700 1 -- 172.21.15.63:0/4216596769 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f676404c780 con 0x7f674806c7c0 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f674806c7c0 msgr2=0x7f674806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f674806c7c0 0x7f674806ec70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6764111230 tx=0x7f674c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67640ff3e0 msgr2=0x7f6764115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67640ff3e0 0x7f6764115a00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6754002a80 tx=0x7f67540046b0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 shutdown_connections 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f674806c7c0 0x7f674806ec70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67640ff3e0 0x7f6764115a00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 --2- 172.21.15.63:0/4216596769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6764100be0 0x7f6764115f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 >> 172.21.15.63:0/4216596769 conn(0x7f67640faba0 msgr2=0x7f67640fb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 shutdown_connections 2024-03-23T12:20:41.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.050+0000 7f676b200700 1 -- 172.21.15.63:0/4216596769 wait complete. 2024-03-23T12:20:41.043 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:20:41.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.188+0000 7f614fe00700 1 -- 172.21.15.63:0/980719682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 msgr2=0x7f6148070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.188+0000 7f614fe00700 1 --2- 172.21.15.63:0/980719682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f6148070d10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f61340048b0 tx=0x7f6134009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:41.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.189+0000 7f614fe00700 1 -- 172.21.15.63:0/980719682 shutdown_connections 2024-03-23T12:20:41.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.189+0000 7f614fe00700 1 --2- 172.21.15.63:0/980719682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f6148070d10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.189+0000 7f614fe00700 1 --2- 172.21.15.63:0/980719682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6148071f90 0x7f6148070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.189+0000 7f614fe00700 1 -- 172.21.15.63:0/980719682 >> 172.21.15.63:0/980719682 conn(0x7f61480f8970 msgr2=0x7f61480fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:41.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.189+0000 7f614fe00700 1 -- 172.21.15.63:0/980719682 shutdown_connections 2024-03-23T12:20:41.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.189+0000 7f614fe00700 1 -- 172.21.15.63:0/980719682 wait complete. 2024-03-23T12:20:41.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.189+0000 7f614fe00700 1 Processor -- start 2024-03-23T12:20:41.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614fe00700 1 -- start start 2024-03-23T12:20:41.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f614811a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6148071f90 0x7f614811a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614811ac40 con 0x7f61480708a0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614fe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6148114010 con 0x7f6148071f90 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f614811a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f614811a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51820/0 (socket says 172.21.15.63:51820) 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614d400700 1 -- 172.21.15.63:0/3698280413 learned_addr learned my addr 172.21.15.63:0/3698280413 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.190+0000 7f614ca00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6148071f90 0x7f614811a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f614d400700 1 -- 172.21.15.63:0/3698280413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6148071f90 msgr2=0x7f614811a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f614d400700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6148071f90 0x7f614811a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f614d400700 1 -- 172.21.15.63:0/3698280413 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6134005040 con 0x7f61480708a0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f614d400700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f614811a0b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f613c0041d0 tx=0x7f613c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f6145600700 1 -- 172.21.15.63:0/3698280413 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613c007cc0 con 0x7f61480708a0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f6145600700 1 -- 172.21.15.63:0/3698280413 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f613c00f040 con 0x7f61480708a0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148114290 con 0x7f61480708a0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f6145600700 1 -- 172.21.15.63:0/3698280413 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613c00c990 con 0x7f61480708a0 2024-03-23T12:20:41.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.191+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61481147d0 con 0x7f61480708a0 2024-03-23T12:20:41.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.192+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6148071aa0 con 0x7f61480708a0 2024-03-23T12:20:41.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.193+0000 7f6145600700 1 -- 172.21.15.63:0/3698280413 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f613c00caf0 con 0x7f61480708a0 2024-03-23T12:20:41.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.194+0000 7f6145600700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f613006c7d0 0x7f613006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.194+0000 7f6145600700 1 -- 172.21.15.63:0/3698280413 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f613c08b4a0 con 0x7f61480708a0 2024-03-23T12:20:41.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.195+0000 7f614ca00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f613006c7d0 0x7f613006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.195+0000 7f614ca00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f613006c7d0 0x7f613006ec80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6148115950 tx=0x7f6134009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:41.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.199+0000 7f6145600700 1 -- 172.21.15.63:0/3698280413 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f613c0591e0 con 0x7f61480708a0 2024-03-23T12:20:41.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.429+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f614804bf40 con 0x7f613006c7d0 2024-03-23T12:20:41.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.431+0000 7f6145600700 1 -- 172.21.15.63:0/3698280413 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f614804bf40 con 0x7f613006c7d0 2024-03-23T12:20:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.433+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f613006c7d0 msgr2=0x7f613006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.433+0000 7f614fe00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f613006c7d0 0x7f613006ec80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6148115950 tx=0x7f6134009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.433+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 msgr2=0x7f614811a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.433+0000 7f614fe00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f614811a0b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f613c0041d0 tx=0x7f613c00b040 comp rx=0 tx=0).stop 2024-03-23T12:20:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.434+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 shutdown_connections 2024-03-23T12:20:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.434+0000 7f614fe00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f613006c7d0 0x7f613006ec80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.434+0000 7f614fe00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61480708a0 0x7f614811a0b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.434+0000 7f614fe00700 1 --2- 172.21.15.63:0/3698280413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6148071f90 0x7f614811a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.434+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 >> 172.21.15.63:0/3698280413 conn(0x7f61480f8970 msgr2=0x7f61480f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:41.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.434+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 shutdown_connections 2024-03-23T12:20:41.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.434+0000 7f614fe00700 1 -- 172.21.15.63:0/3698280413 wait complete. 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.570+0000 7fceede00700 1 -- 172.21.15.63:0/2157437518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee81037a0 msgr2=0x7fcee80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.570+0000 7fceede00700 1 --2- 172.21.15.63:0/2157437518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee81037a0 0x7fcee80fce10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fced4004940 tx=0x7fced4009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.570+0000 7fceede00700 1 -- 172.21.15.63:0/2157437518 shutdown_connections 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.570+0000 7fceede00700 1 --2- 172.21.15.63:0/2157437518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee81037a0 0x7fcee80fce10 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.570+0000 7fceede00700 1 --2- 172.21.15.63:0/2157437518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee8102e00 0x7fcee81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.570+0000 7fceede00700 1 -- 172.21.15.63:0/2157437518 >> 172.21.15.63:0/2157437518 conn(0x7fcee80f89b0 msgr2=0x7fcee80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.571+0000 7fceede00700 1 -- 172.21.15.63:0/2157437518 shutdown_connections 2024-03-23T12:20:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.571+0000 7fceede00700 1 -- 172.21.15.63:0/2157437518 wait complete. 2024-03-23T12:20:41.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.571+0000 7fceede00700 1 Processor -- start 2024-03-23T12:20:41.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.571+0000 7fceede00700 1 -- start start 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fceede00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee8102e00 0x7fcee8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fceede00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee81037a0 0x7fcee8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fceede00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8116880 con 0x7fcee8102e00 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fceede00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee810fc50 con 0x7fcee81037a0 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fcee6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee8102e00 0x7fcee8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fcee6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee8102e00 0x7fcee8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52022/0 (socket says 172.21.15.63:52022) 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fcee6a00700 1 -- 172.21.15.63:0/843600636 learned_addr learned my addr 172.21.15.63:0/843600636 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.572+0000 7fcee6000700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee81037a0 0x7fcee8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fcee6a00700 1 -- 172.21.15.63:0/843600636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee81037a0 msgr2=0x7fcee8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fcee6a00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee81037a0 0x7fcee8116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fcee6a00700 1 -- 172.21.15.63:0/843600636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4005040 con 0x7fcee8102e00 2024-03-23T12:20:41.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fcee6a00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee8102e00 0x7fcee8115cf0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fced8002a50 tx=0x7fced80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:41.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fcedea00700 1 -- 172.21.15.63:0/843600636 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced8007cc0 con 0x7fcee8102e00 2024-03-23T12:20:41.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fcedea00700 1 -- 172.21.15.63:0/843600636 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fced800f040 con 0x7fcee8102e00 2024-03-23T12:20:41.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee810ff30 con 0x7fcee8102e00 2024-03-23T12:20:41.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fcedea00700 1 -- 172.21.15.63:0/843600636 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced800d9c0 con 0x7fcee8102e00 2024-03-23T12:20:41.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.573+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee81104f0 con 0x7fcee8102e00 2024-03-23T12:20:41.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.574+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee80fe550 con 0x7fcee8102e00 2024-03-23T12:20:41.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.576+0000 7fcedea00700 1 -- 172.21.15.63:0/843600636 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fced800db20 con 0x7fcee8102e00 2024-03-23T12:20:41.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.576+0000 7fcedea00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcecc06c7c0 0x7fcecc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.577+0000 7fcedea00700 1 -- 172.21.15.63:0/843600636 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fced808b820 con 0x7fcee8102e00 2024-03-23T12:20:41.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.577+0000 7fcee6000700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcecc06c7c0 0x7fcecc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.578+0000 7fcee6000700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcecc06c7c0 0x7fcecc06ec70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcee8111590 tx=0x7fced4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:41.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.581+0000 7fcedea00700 1 -- 172.21.15.63:0/843600636 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fced8059430 con 0x7fcee8102e00 2024-03-23T12:20:41.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.808+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcee804bf40 con 0x7fcecc06c7c0 2024-03-23T12:20:41.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.818+0000 7fcedea00700 1 -- 172.21.15.63:0/843600636 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fcee804bf40 con 0x7fcecc06c7c0 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (30m) 7m ago 43m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (27m) 7m ago 43m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (27m) 7m ago 42m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (29m) 7m ago 42m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (23m) 7m ago 36m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (22m) 7m ago 36m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (7m) 7m ago 36m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (22m) 7m ago 36m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (32m) 7m ago 44m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (31m) 7m ago 42m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:20:41.798 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (28m) 7m ago 44m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (27m) 7m ago 41m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (30m) 7m ago 43m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (30m) 7m ago 42m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (26m) 7m ago 41m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (25m) 7m ago 40m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (25m) 7m ago 39m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (24m) 7m ago 39m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (24m) 7m ago 38m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (23m) 7m ago 38m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:20:41.799 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (30m) 7m ago 42m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:20:41.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcecc06c7c0 msgr2=0x7fcecc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcecc06c7c0 0x7fcecc06ec70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcee8111590 tx=0x7fced4009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:41.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee8102e00 msgr2=0x7fcee8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee8102e00 0x7fcee8115cf0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fced8002a50 tx=0x7fced80046b0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 shutdown_connections 2024-03-23T12:20:41.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcecc06c7c0 0x7fcecc06ec70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcee8102e00 0x7fcee8115cf0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 --2- 172.21.15.63:0/843600636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee81037a0 0x7fcee8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 >> 172.21.15.63:0/843600636 conn(0x7fcee80f89b0 msgr2=0x7fcee80f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:41.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 shutdown_connections 2024-03-23T12:20:41.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.821+0000 7fceede00700 1 -- 172.21.15.63:0/843600636 wait complete. 2024-03-23T12:20:41.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 -- 172.21.15.63:0/99612567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 msgr2=0x7f20e81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 --2- 172.21.15.63:0/99612567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e81031d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f20dc004940 tx=0x7f20dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:41.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 -- 172.21.15.63:0/99612567 shutdown_connections 2024-03-23T12:20:41.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 --2- 172.21.15.63:0/99612567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e81037a0 0x7f20e80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 --2- 172.21.15.63:0/99612567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e81031d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 -- 172.21.15.63:0/99612567 >> 172.21.15.63:0/99612567 conn(0x7f20e80f89b0 msgr2=0x7f20e80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:41.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 -- 172.21.15.63:0/99612567 shutdown_connections 2024-03-23T12:20:41.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.959+0000 7f20efa00700 1 -- 172.21.15.63:0/99612567 wait complete. 2024-03-23T12:20:41.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.960+0000 7f20efa00700 1 Processor -- start 2024-03-23T12:20:41.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.960+0000 7f20efa00700 1 -- start start 2024-03-23T12:20:41.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.960+0000 7f20efa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.960+0000 7f20efa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e81037a0 0x7f20e8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.960+0000 7f20efa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e8116860 con 0x7f20e8102e00 2024-03-23T12:20:41.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.960+0000 7f20efa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e810fc30 con 0x7f20e81037a0 2024-03-23T12:20:41.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20ed000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20ed000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e8115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52044/0 (socket says 172.21.15.63:52044) 2024-03-23T12:20:41.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20ed000700 1 -- 172.21.15.63:0/58205425 learned_addr learned my addr 172.21.15.63:0/58205425 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:41.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20e7e00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e81037a0 0x7f20e8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20ed000700 1 -- 172.21.15.63:0/58205425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e81037a0 msgr2=0x7f20e8116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:41.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20ed000700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e81037a0 0x7f20e8116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:41.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20ed000700 1 -- 172.21.15.63:0/58205425 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc005040 con 0x7f20e8102e00 2024-03-23T12:20:41.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20ed000700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e8115cd0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f20dc009390 tx=0x7f20dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:41.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20e4c00700 1 -- 172.21.15.63:0/58205425 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20dc00e040 con 0x7f20e8102e00 2024-03-23T12:20:41.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.961+0000 7f20e4c00700 1 -- 172.21.15.63:0/58205425 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20dc013420 con 0x7f20e8102e00 2024-03-23T12:20:41.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.962+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e810feb0 con 0x7f20e8102e00 2024-03-23T12:20:41.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.962+0000 7f20e4c00700 1 -- 172.21.15.63:0/58205425 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20dc01c630 con 0x7f20e8102e00 2024-03-23T12:20:41.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.962+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e8110470 con 0x7f20e8102e00 2024-03-23T12:20:41.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.963+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e80fe550 con 0x7f20e8102e00 2024-03-23T12:20:41.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.964+0000 7f20e4c00700 1 -- 172.21.15.63:0/58205425 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f20dc0139e0 con 0x7f20e8102e00 2024-03-23T12:20:41.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.965+0000 7f20e4c00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20d006c7d0 0x7f20d006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:41.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.965+0000 7f20e4c00700 1 -- 172.21.15.63:0/58205425 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f20dc08cce0 con 0x7f20e8102e00 2024-03-23T12:20:41.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.965+0000 7f20e7e00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20d006c7d0 0x7f20d006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:41.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.966+0000 7f20e7e00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20d006c7d0 0x7f20d006ec80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f20e8111570 tx=0x7f20d4009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:41.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:41.970+0000 7f20e4c00700 1 -- 172.21.15.63:0/58205425 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f20dc091070 con 0x7f20e8102e00 2024-03-23T12:20:42.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:41 smithi063 ceph-mon[141124]: from='client.35614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:41 smithi063 ceph-mon[141124]: pgmap v888: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:42.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:42.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:41 smithi111 ceph-mon[106899]: from='client.35614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:42.148 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:41 smithi111 ceph-mon[106899]: pgmap v888: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:42.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.271+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f20e804bed0 con 0x7f20e8102e00 2024-03-23T12:20:42.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.272+0000 7f20e4c00700 1 -- 172.21.15.63:0/58205425 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f20dc05aaa0 con 0x7f20e8102e00 2024-03-23T12:20:42.251 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:20:42.251 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:20:42.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:20:42.253 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:20:42.253 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:20:42.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:20:42.253 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:20:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.274+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20d006c7d0 msgr2=0x7f20d006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.274+0000 7f20efa00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20d006c7d0 0x7f20d006ec80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f20e8111570 tx=0x7f20d4009ee0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.274+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 msgr2=0x7f20e8115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.274+0000 7f20efa00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e8115cd0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f20dc009390 tx=0x7f20dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.275+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 shutdown_connections 2024-03-23T12:20:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.275+0000 7f20efa00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20d006c7d0 0x7f20d006ec80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.275+0000 7f20efa00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e8102e00 0x7f20e8115cd0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.275+0000 7f20efa00700 1 --2- 172.21.15.63:0/58205425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e81037a0 0x7f20e8116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.275+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 >> 172.21.15.63:0/58205425 conn(0x7f20e80f89b0 msgr2=0x7f20e80f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:42.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.275+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 shutdown_connections 2024-03-23T12:20:42.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.275+0000 7f20efa00700 1 -- 172.21.15.63:0/58205425 wait complete. 2024-03-23T12:20:42.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.411+0000 7f1637e00700 1 -- 172.21.15.63:0/92019958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1630102e20 msgr2=0x7f16301031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.411+0000 7f1637e00700 1 --2- 172.21.15.63:0/92019958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1630102e20 0x7f16301031f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1624004940 tx=0x7f1624009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:42.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.412+0000 7f1637e00700 1 -- 172.21.15.63:0/92019958 shutdown_connections 2024-03-23T12:20:42.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.412+0000 7f1637e00700 1 --2- 172.21.15.63:0/92019958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16301037c0 0x7f16300fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.412+0000 7f1637e00700 1 --2- 172.21.15.63:0/92019958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1630102e20 0x7f16301031f0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.412+0000 7f1637e00700 1 -- 172.21.15.63:0/92019958 >> 172.21.15.63:0/92019958 conn(0x7f16300f8970 msgr2=0x7f16300fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:42.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.412+0000 7f1637e00700 1 -- 172.21.15.63:0/92019958 shutdown_connections 2024-03-23T12:20:42.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.412+0000 7f1637e00700 1 -- 172.21.15.63:0/92019958 wait complete. 2024-03-23T12:20:42.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.412+0000 7f1637e00700 1 Processor -- start 2024-03-23T12:20:42.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1637e00700 1 -- start start 2024-03-23T12:20:42.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1637e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1630102e20 0x7f1630113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1637e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16301037c0 0x7f1630114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1637e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630114670 con 0x7f16301037c0 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1637e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163010da60 con 0x7f1630102e20 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1634a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16301037c0 0x7f1630114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1634a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16301037c0 0x7f1630114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52064/0 (socket says 172.21.15.63:52064) 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1634a00700 1 -- 172.21.15.63:0/2700155976 learned_addr learned my addr 172.21.15.63:0/2700155976 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.413+0000 7f1635400700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1630102e20 0x7f1630113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f1634a00700 1 -- 172.21.15.63:0/2700155976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1630102e20 msgr2=0x7f1630113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f1634a00700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1630102e20 0x7f1630113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f1634a00700 1 -- 172.21.15.63:0/2700155976 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1624005040 con 0x7f16301037c0 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f1634a00700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16301037c0 0x7f1630114020 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c890 tx=0x7f161c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:42.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f162d600700 1 -- 172.21.15.63:0/2700155976 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f161c00a700 con 0x7f16301037c0 2024-03-23T12:20:42.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f162d600700 1 -- 172.21.15.63:0/2700155976 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f161c00ad10 con 0x7f16301037c0 2024-03-23T12:20:42.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f162d600700 1 -- 172.21.15.63:0/2700155976 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f161c012e70 con 0x7f16301037c0 2024-03-23T12:20:42.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163010dd40 con 0x7f16301037c0 2024-03-23T12:20:42.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.414+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163010e2b0 con 0x7f16301037c0 2024-03-23T12:20:42.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.415+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16300fe570 con 0x7f16301037c0 2024-03-23T12:20:42.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.416+0000 7f162d600700 1 -- 172.21.15.63:0/2700155976 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f161c00a860 con 0x7f16301037c0 2024-03-23T12:20:42.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.418+0000 7f162d600700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f161806c7d0 0x7f161806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:42.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.418+0000 7f162d600700 1 -- 172.21.15.63:0/2700155976 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f161c08bae0 con 0x7f16301037c0 2024-03-23T12:20:42.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.418+0000 7f1635400700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f161806c7d0 0x7f161806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:42.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.419+0000 7f1635400700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f161806c7d0 0x7f161806ec80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1624009fd0 tx=0x7f1624009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.422+0000 7f162d600700 1 -- 172.21.15.63:0/2700155976 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f161c0597f0 con 0x7f16301037c0 2024-03-23T12:20:42.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.681+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1630109de0 con 0x7f16301037c0 2024-03-23T12:20:42.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.682+0000 7f162d600700 1 -- 172.21.15.63:0/2700155976 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f161c059380 con 0x7f16301037c0 2024-03-23T12:20:42.661 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:20:42.661 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:20:42.661 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:20:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:20:42.663 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:20:42.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f161806c7d0 msgr2=0x7f161806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f161806c7d0 0x7f161806ec80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1624009fd0 tx=0x7f1624009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:42.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16301037c0 msgr2=0x7f1630114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16301037c0 0x7f1630114020 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c890 tx=0x7f161c0044c0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 shutdown_connections 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f161806c7d0 0x7f161806ec80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1630102e20 0x7f1630113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 --2- 172.21.15.63:0/2700155976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16301037c0 0x7f1630114020 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 >> 172.21.15.63:0/2700155976 conn(0x7f16300f8970 msgr2=0x7f16300f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 shutdown_connections 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.684+0000 7f1637e00700 1 -- 172.21.15.63:0/2700155976 wait complete. 2024-03-23T12:20:42.666 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.823+0000 7f8cc7c00700 1 -- 172.21.15.63:0/874964439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 msgr2=0x7f8cc00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.823+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/874964439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc00fce10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8cac004940 tx=0x7f8cac009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.823+0000 7f8cc7c00700 1 -- 172.21.15.63:0/874964439 shutdown_connections 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.823+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/874964439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc00fce10 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.823+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/874964439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cc0102e00 0x7f8cc01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.823+0000 7f8cc7c00700 1 -- 172.21.15.63:0/874964439 >> 172.21.15.63:0/874964439 conn(0x7f8cc00f89b0 msgr2=0x7f8cc00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.823+0000 7f8cc7c00700 1 -- 172.21.15.63:0/874964439 shutdown_connections 2024-03-23T12:20:42.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.824+0000 7f8cc7c00700 1 -- 172.21.15.63:0/874964439 wait complete. 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.824+0000 7f8cc7c00700 1 Processor -- start 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.824+0000 7f8cc7c00700 1 -- start start 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cc7c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cc0102e00 0x7f8cc00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cc7c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cc7c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc0070d90 con 0x7f8cc01037a0 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cc7c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc0070ed0 con 0x7f8cc0102e00 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cbfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cbfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52096/0 (socket says 172.21.15.63:52096) 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cbfe00700 1 -- 172.21.15.63:0/561173959 learned_addr learned my addr 172.21.15.63:0/561173959 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:42.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.825+0000 7f8cc5200700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cc0102e00 0x7f8cc00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:42.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cbfe00700 1 -- 172.21.15.63:0/561173959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cc0102e00 msgr2=0x7f8cc00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:42.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cbfe00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cc0102e00 0x7f8cc00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:42.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cbfe00700 1 -- 172.21.15.63:0/561173959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac005040 con 0x7f8cc01037a0 2024-03-23T12:20:42.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cbfe00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc0070800 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009390 tx=0x7f8cac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:42.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cbcc00700 1 -- 172.21.15.63:0/561173959 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cac00e040 con 0x7f8cc01037a0 2024-03-23T12:20:42.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cbcc00700 1 -- 172.21.15.63:0/561173959 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cac013420 con 0x7f8cc01037a0 2024-03-23T12:20:42.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc0071120 con 0x7f8cc01037a0 2024-03-23T12:20:42.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cbcc00700 1 -- 172.21.15.63:0/561173959 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cac01c630 con 0x7f8cc01037a0 2024-03-23T12:20:42.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.826+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc011a230 con 0x7f8cc01037a0 2024-03-23T12:20:42.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.828+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc00fe550 con 0x7f8cc01037a0 2024-03-23T12:20:42.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.828+0000 7f8cbcc00700 1 -- 172.21.15.63:0/561173959 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8cac013a00 con 0x7f8cc01037a0 2024-03-23T12:20:42.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.829+0000 7f8cbcc00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca806c7d0 0x7f8ca806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:42.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.829+0000 7f8cbcc00700 1 -- 172.21.15.63:0/561173959 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8cac08ca60 con 0x7f8cc01037a0 2024-03-23T12:20:42.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.830+0000 7f8cc5200700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca806c7d0 0x7f8ca806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:42.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.830+0000 7f8cc5200700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca806c7d0 0x7f8ca806ec80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4004f40 tx=0x7f8cb400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:42.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:42.834+0000 7f8cbcc00700 1 -- 172.21.15.63:0/561173959 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8cac056da0 con 0x7f8cc01037a0 2024-03-23T12:20:43.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.068+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cc0109e40 con 0x7f8ca806c7d0 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.069+0000 7f8cbcc00700 1 -- 172.21.15.63:0/561173959 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8cb400f040 con 0x7f8ca806c7d0 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:20:43.048 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:20:43.049 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:20:43.049 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:20:43.049 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:20:43.049 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:20:43.049 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:20:43.049 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:20:43.049 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca806c7d0 msgr2=0x7f8ca806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca806c7d0 0x7f8ca806ec80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4004f40 tx=0x7f8cb400b040 comp rx=0 tx=0).stop 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 msgr2=0x7f8cc0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc0070800 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009390 tx=0x7f8cac009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 shutdown_connections 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8ca806c7d0 0x7f8ca806ec80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cc0102e00 0x7f8cc00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.071+0000 7f8cc7c00700 1 --2- 172.21.15.63:0/561173959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cc01037a0 0x7f8cc0070800 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.072+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 >> 172.21.15.63:0/561173959 conn(0x7f8cc00f89b0 msgr2=0x7f8cc00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.072+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 shutdown_connections 2024-03-23T12:20:43.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.072+0000 7f8cc7c00700 1 -- 172.21.15.63:0/561173959 wait complete. 2024-03-23T12:20:43.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:42 smithi063 ceph-mon[141124]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:43.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:42 smithi063 ceph-mon[141124]: from='client.35622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:43.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:42 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/58205425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:20:43.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:42 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2700155976' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:20:43.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:42 smithi111 ceph-mon[106899]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:43.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:42 smithi111 ceph-mon[106899]: from='client.35622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:43.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:42 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/58205425' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:20:43.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:42 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2700155976' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:20:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 -- 172.21.15.63:0/745693690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 msgr2=0x7fb5b00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/745693690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b00fce10 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb59c004940 tx=0x7fb59c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 -- 172.21.15.63:0/745693690 shutdown_connections 2024-03-23T12:20:43.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/745693690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b00fce10 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/745693690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0102e00 0x7fb5b01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 -- 172.21.15.63:0/745693690 >> 172.21.15.63:0/745693690 conn(0x7fb5b00f89b0 msgr2=0x7fb5b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 -- 172.21.15.63:0/745693690 shutdown_connections 2024-03-23T12:20:43.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.210+0000 7fb5b7a00700 1 -- 172.21.15.63:0/745693690 wait complete. 2024-03-23T12:20:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.211+0000 7fb5b7a00700 1 Processor -- start 2024-03-23T12:20:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.211+0000 7fb5b7a00700 1 -- start start 2024-03-23T12:20:43.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.211+0000 7fb5b7a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0102e00 0x7fb5b0115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.211+0000 7fb5b7a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b01161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5b7a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b01167f0 con 0x7fb5b01037a0 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5b7a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b010fbc0 con 0x7fb5b0102e00 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5afe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b01161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5afe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b01161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52110/0 (socket says 172.21.15.63:52110) 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5afe00700 1 -- 172.21.15.63:0/2867320183 learned_addr learned my addr 172.21.15.63:0/2867320183 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5b5000700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0102e00 0x7fb5b0115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:43.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5afe00700 1 -- 172.21.15.63:0/2867320183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0102e00 msgr2=0x7fb5b0115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5afe00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0102e00 0x7fb5b0115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5afe00700 1 -- 172.21.15.63:0/2867320183 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c005040 con 0x7fb5b01037a0 2024-03-23T12:20:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5b5000700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0102e00 0x7fb5b0115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:20:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.212+0000 7fb5afe00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b01161a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009390 tx=0x7fb59c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.213+0000 7fb5acc00700 1 -- 172.21.15.63:0/2867320183 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb59c00e040 con 0x7fb5b01037a0 2024-03-23T12:20:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.213+0000 7fb5acc00700 1 -- 172.21.15.63:0/2867320183 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb59c013420 con 0x7fb5b01037a0 2024-03-23T12:20:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.213+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b010fe40 con 0x7fb5b01037a0 2024-03-23T12:20:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.213+0000 7fb5acc00700 1 -- 172.21.15.63:0/2867320183 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb59c01c630 con 0x7fb5b01037a0 2024-03-23T12:20:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.213+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b0110400 con 0x7fb5b01037a0 2024-03-23T12:20:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.214+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b00fe550 con 0x7fb5b01037a0 2024-03-23T12:20:43.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.215+0000 7fb5acc00700 1 -- 172.21.15.63:0/2867320183 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb59c013a00 con 0x7fb5b01037a0 2024-03-23T12:20:43.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.216+0000 7fb5acc00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb59806c7d0 0x7fb59806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:20:43.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.216+0000 7fb5acc00700 1 -- 172.21.15.63:0/2867320183 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb59c08cd10 con 0x7fb5b01037a0 2024-03-23T12:20:43.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.216+0000 7fb5b5000700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb59806c7d0 0x7fb59806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:20:43.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.217+0000 7fb5b5000700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb59806c7d0 0x7fb59806ec80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009b80 tx=0x7fb5a400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:20:43.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.221+0000 7fb5acc00700 1 -- 172.21.15.63:0/2867320183 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb59c05aa20 con 0x7fb5b01037a0 2024-03-23T12:20:43.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.523+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb5b005f5f0 con 0x7fb5b01037a0 2024-03-23T12:20:43.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.524+0000 7fb5acc00700 1 -- 172.21.15.63:0/2867320183 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fb59c05a5b0 con 0x7fb5b01037a0 2024-03-23T12:20:43.503 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:20:43.503 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:20:43.503 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:20:43.503 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:20:43.503 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:20:43.504 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:20:43.505 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:20:43.505 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:20:43.505 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:20:43.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb59806c7d0 msgr2=0x7fb59806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:43.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb59806c7d0 0x7fb59806ec80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009b80 tx=0x7fb5a400b040 comp rx=0 tx=0).stop 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 msgr2=0x7fb5b01161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b01161a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009390 tx=0x7fb59c009e30 comp rx=0 tx=0).stop 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 shutdown_connections 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb59806c7d0 0x7fb59806ec80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0102e00 0x7fb5b0115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 --2- 172.21.15.63:0/2867320183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5b01037a0 0x7fb5b01161a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.526+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 >> 172.21.15.63:0/2867320183 conn(0x7fb5b00f89b0 msgr2=0x7fb5b00f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.527+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 shutdown_connections 2024-03-23T12:20:43.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:20:43.527+0000 7fb5b7a00700 1 -- 172.21.15.63:0/2867320183 wait complete. 2024-03-23T12:20:44.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:43 smithi063 ceph-mon[141124]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:44.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:43 smithi063 ceph-mon[141124]: pgmap v889: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:44.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:43 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2867320183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:20:44.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:43 smithi111 ceph-mon[106899]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:20:44.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:43 smithi111 ceph-mon[106899]: pgmap v889: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:44.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:43 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2867320183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:20:45.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:45 smithi063 ceph-mon[141124]: pgmap v890: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:45.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:45 smithi111 ceph-mon[106899]: pgmap v890: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:47.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:47 smithi063 ceph-mon[141124]: pgmap v891: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:47.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:47 smithi111 ceph-mon[106899]: pgmap v891: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:49.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:49 smithi063 ceph-mon[141124]: pgmap v892: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:49.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:49 smithi111 ceph-mon[106899]: pgmap v892: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:51 smithi063 ceph-mon[141124]: pgmap v893: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:51 smithi111 ceph-mon[106899]: pgmap v893: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:53.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:53 smithi063 ceph-mon[141124]: pgmap v894: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:53.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:53 smithi111 ceph-mon[106899]: pgmap v894: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:55.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:55 smithi063 ceph-mon[141124]: pgmap v895: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:55.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:55 smithi111 ceph-mon[106899]: pgmap v895: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:57.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:20:58.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:57 smithi063 ceph-mon[141124]: pgmap v896: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:58.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:57 smithi111 ceph-mon[106899]: pgmap v896: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:59.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:20:59 smithi063 ceph-mon[141124]: pgmap v897: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:20:59.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:20:59 smithi111 ceph-mon[106899]: pgmap v897: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:02.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:01 smithi063 ceph-mon[141124]: pgmap v898: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:02.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:01 smithi111 ceph-mon[106899]: pgmap v898: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:03.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:03 smithi063 ceph-mon[141124]: pgmap v899: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:03.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:03 smithi111 ceph-mon[106899]: pgmap v899: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:05.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:05 smithi063 ceph-mon[141124]: pgmap v900: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:05.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:05 smithi111 ceph-mon[106899]: pgmap v900: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:07.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:07 smithi063 ceph-mon[141124]: pgmap v901: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:07.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:07 smithi111 ceph-mon[106899]: pgmap v901: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:09.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:09 smithi063 ceph-mon[141124]: pgmap v902: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:09.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:09 smithi111 ceph-mon[106899]: pgmap v902: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:12.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:12.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:11 smithi063 ceph-mon[141124]: pgmap v903: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:12.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:12.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:11 smithi111 ceph-mon[106899]: pgmap v903: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:13.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:13 smithi063 ceph-mon[141124]: pgmap v904: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:13.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:13 smithi111 ceph-mon[106899]: pgmap v904: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:13.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 -- 172.21.15.63:0/536672041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 msgr2=0x7fd7601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:13.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 --2- 172.21.15.63:0/536672041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7601031d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd750004940 tx=0x7fd750009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:13.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 -- 172.21.15.63:0/536672041 shutdown_connections 2024-03-23T12:21:13.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 --2- 172.21.15.63:0/536672041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7601037a0 0x7fd7600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:13.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 --2- 172.21.15.63:0/536672041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7601031d0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:13.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 -- 172.21.15.63:0/536672041 >> 172.21.15.63:0/536672041 conn(0x7fd7600f89b0 msgr2=0x7fd7600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:13.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 -- 172.21.15.63:0/536672041 shutdown_connections 2024-03-23T12:21:13.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.681+0000 7fd765a00700 1 -- 172.21.15.63:0/536672041 wait complete. 2024-03-23T12:21:13.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.682+0000 7fd765a00700 1 Processor -- start 2024-03-23T12:21:13.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.682+0000 7fd765a00700 1 -- start start 2024-03-23T12:21:13.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd765a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:13.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd765a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7601037a0 0x7fd760070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:13.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd765a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760070d90 con 0x7fd760102e00 2024-03-23T12:21:13.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd765a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760070ed0 con 0x7fd7601037a0 2024-03-23T12:21:13.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd75ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:13.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd75ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7600702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47300/0 (socket says 172.21.15.63:47300) 2024-03-23T12:21:13.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd75ea00700 1 -- 172.21.15.63:0/3596701117 learned_addr learned my addr 172.21.15.63:0/3596701117 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:13.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd757e00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7601037a0 0x7fd760070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:13.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd75ea00700 1 -- 172.21.15.63:0/3596701117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7601037a0 msgr2=0x7fd760070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd75ea00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7601037a0 0x7fd760070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd75ea00700 1 -- 172.21.15.63:0/3596701117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd750005040 con 0x7fd760102e00 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.683+0000 7fd757e00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7601037a0 0x7fd760070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.684+0000 7fd75ea00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7600702c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd750009390 tx=0x7fd750009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.684+0000 7fd756a00700 1 -- 172.21.15.63:0/3596701117 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd75000e040 con 0x7fd760102e00 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.684+0000 7fd756a00700 1 -- 172.21.15.63:0/3596701117 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd750013420 con 0x7fd760102e00 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.684+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd760071120 con 0x7fd760102e00 2024-03-23T12:21:13.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.684+0000 7fd756a00700 1 -- 172.21.15.63:0/3596701117 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd75001c630 con 0x7fd760102e00 2024-03-23T12:21:13.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.684+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76011a250 con 0x7fd760102e00 2024-03-23T12:21:13.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.686+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7600fe550 con 0x7fd760102e00 2024-03-23T12:21:13.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.686+0000 7fd756a00700 1 -- 172.21.15.63:0/3596701117 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd7500139e0 con 0x7fd760102e00 2024-03-23T12:21:13.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.687+0000 7fd756a00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd74406c7d0 0x7fd74406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:13.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.687+0000 7fd756a00700 1 -- 172.21.15.63:0/3596701117 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd75008cbf0 con 0x7fd760102e00 2024-03-23T12:21:13.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.688+0000 7fd757e00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd74406c7d0 0x7fd74406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:13.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.688+0000 7fd757e00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd74406c7d0 0x7fd74406ec80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd748009fd0 tx=0x7fd748009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:13.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.692+0000 7fd756a00700 1 -- 172.21.15.63:0/3596701117 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd750056fe0 con 0x7fd760102e00 2024-03-23T12:21:13.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.925+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7600fe550 con 0x7fd74406c7d0 2024-03-23T12:21:13.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.927+0000 7fd756a00700 1 -- 172.21.15.63:0/3596701117 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd7600fe550 con 0x7fd74406c7d0 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd74406c7d0 msgr2=0x7fd74406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd74406c7d0 0x7fd74406ec80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd748009fd0 tx=0x7fd748009ee0 comp rx=0 tx=0).stop 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 msgr2=0x7fd7600702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7600702c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd750009390 tx=0x7fd750009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 shutdown_connections 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd74406c7d0 0x7fd74406ec80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd760102e00 0x7fd7600702c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 --2- 172.21.15.63:0/3596701117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7601037a0 0x7fd760070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.929+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 >> 172.21.15.63:0/3596701117 conn(0x7fd7600f89b0 msgr2=0x7fd7600f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.930+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 shutdown_connections 2024-03-23T12:21:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:13.930+0000 7fd765a00700 1 -- 172.21.15.63:0/3596701117 wait complete. 2024-03-23T12:21:13.923 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 -- 172.21.15.63:0/970265516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fde90 msgr2=0x7fa0a80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 --2- 172.21.15.63:0/970265516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fde90 0x7fa0a80fe260 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa09c004940 tx=0x7fa09c009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 -- 172.21.15.63:0/970265516 shutdown_connections 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 --2- 172.21.15.63:0/970265516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0a80fe830 0x7fa0a8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 --2- 172.21.15.63:0/970265516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fde90 0x7fa0a80fe260 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 -- 172.21.15.63:0/970265516 >> 172.21.15.63:0/970265516 conn(0x7fa0a8072bc0 msgr2=0x7fa0a8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 -- 172.21.15.63:0/970265516 shutdown_connections 2024-03-23T12:21:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.069+0000 7fa0af600700 1 -- 172.21.15.63:0/970265516 wait complete. 2024-03-23T12:21:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0af600700 1 Processor -- start 2024-03-23T12:21:14.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0af600700 1 -- start start 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0af600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0a80fde90 0x7fa0a8115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0af600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fe830 0x7fa0a8116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0af600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81168b0 con 0x7fa0a80fe830 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0af600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a810fc80 con 0x7fa0a80fde90 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0a7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fe830 0x7fa0a8116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0a7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fe830 0x7fa0a8116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47328/0 (socket says 172.21.15.63:47328) 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.070+0000 7fa0a7e00700 1 -- 172.21.15.63:0/2788031279 learned_addr learned my addr 172.21.15.63:0/2788031279 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0acc00700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0a80fde90 0x7fa0a8115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0a7e00700 1 -- 172.21.15.63:0/2788031279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0a80fde90 msgr2=0x7fa0a8115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0a7e00700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0a80fde90 0x7fa0a8115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0a7e00700 1 -- 172.21.15.63:0/2788031279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c005040 con 0x7fa0a80fe830 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0a7e00700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fe830 0x7fa0a8116260 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa094002a50 tx=0x7fa0940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0a4c00700 1 -- 172.21.15.63:0/2788031279 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa094007cc0 con 0x7fa0a80fe830 2024-03-23T12:21:14.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0a4c00700 1 -- 172.21.15.63:0/2788031279 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa09400f040 con 0x7fa0a80fe830 2024-03-23T12:21:14.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a810ff60 con 0x7fa0a80fe830 2024-03-23T12:21:14.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.071+0000 7fa0a4c00700 1 -- 172.21.15.63:0/2788031279 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa094009540 con 0x7fa0a80fe830 2024-03-23T12:21:14.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.072+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a81104a0 con 0x7fa0a80fe830 2024-03-23T12:21:14.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.073+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a8106a20 con 0x7fa0a80fe830 2024-03-23T12:21:14.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.074+0000 7fa0a4c00700 1 -- 172.21.15.63:0/2788031279 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa0940096a0 con 0x7fa0a80fe830 2024-03-23T12:21:14.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.074+0000 7fa0a4c00700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa09006c7d0 0x7fa09006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.075+0000 7fa0a4c00700 1 -- 172.21.15.63:0/2788031279 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa09408b8a0 con 0x7fa0a80fe830 2024-03-23T12:21:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.075+0000 7fa0acc00700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa09006c7d0 0x7fa09006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.075+0000 7fa0acc00700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa09006c7d0 0x7fa09006ec80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00b3b0 tx=0x7fa09c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:14.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.079+0000 7fa0a4c00700 1 -- 172.21.15.63:0/2788031279 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa094059530 con 0x7fa0a80fe830 2024-03-23T12:21:14.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.318+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0a8109dc0 con 0x7fa09006c7d0 2024-03-23T12:21:14.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.320+0000 7fa0a4c00700 1 -- 172.21.15.63:0/2788031279 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa0a8109dc0 con 0x7fa09006c7d0 2024-03-23T12:21:14.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa09006c7d0 msgr2=0x7fa09006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa09006c7d0 0x7fa09006ec80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00b3b0 tx=0x7fa09c009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:14.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fe830 msgr2=0x7fa0a8116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fe830 0x7fa0a8116260 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa094002a50 tx=0x7fa0940046b0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 shutdown_connections 2024-03-23T12:21:14.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa09006c7d0 0x7fa09006ec80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0a80fde90 0x7fa0a8115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 --2- 172.21.15.63:0/2788031279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0a80fe830 0x7fa0a8116260 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 >> 172.21.15.63:0/2788031279 conn(0x7fa0a8072bc0 msgr2=0x7fa0a80fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:14.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 shutdown_connections 2024-03-23T12:21:14.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.322+0000 7fa0af600700 1 -- 172.21.15.63:0/2788031279 wait complete. 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.460+0000 7f2838000700 1 -- 172.21.15.63:0/551291721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 msgr2=0x7f28301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.460+0000 7f2838000700 1 --2- 172.21.15.63:0/551291721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f28301031d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2824004940 tx=0x7f2824009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.460+0000 7f2838000700 1 -- 172.21.15.63:0/551291721 shutdown_connections 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.460+0000 7f2838000700 1 --2- 172.21.15.63:0/551291721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28301037a0 0x7f28300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.460+0000 7f2838000700 1 --2- 172.21.15.63:0/551291721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f28301031d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.460+0000 7f2838000700 1 -- 172.21.15.63:0/551291721 >> 172.21.15.63:0/551291721 conn(0x7f28300f89b0 msgr2=0x7f28300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.461+0000 7f2838000700 1 -- 172.21.15.63:0/551291721 shutdown_connections 2024-03-23T12:21:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.461+0000 7f2838000700 1 -- 172.21.15.63:0/551291721 wait complete. 2024-03-23T12:21:14.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.462+0000 7f2838000700 1 Processor -- start 2024-03-23T12:21:14.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.462+0000 7f2838000700 1 -- start start 2024-03-23T12:21:14.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.462+0000 7f2838000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f2830113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.462+0000 7f2838000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28301037a0 0x7f2830113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2838000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301145d0 con 0x7f2830102e00 2024-03-23T12:21:14.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2838000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283010d9c0 con 0x7f28301037a0 2024-03-23T12:21:14.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2835600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f2830113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2835600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f2830113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47344/0 (socket says 172.21.15.63:47344) 2024-03-23T12:21:14.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2835600700 1 -- 172.21.15.63:0/291154842 learned_addr learned my addr 172.21.15.63:0/291154842 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:14.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2834c00700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28301037a0 0x7f2830113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2835600700 1 -- 172.21.15.63:0/291154842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28301037a0 msgr2=0x7f2830113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2835600700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28301037a0 0x7f2830113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2835600700 1 -- 172.21.15.63:0/291154842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824005040 con 0x7f2830102e00 2024-03-23T12:21:14.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2834c00700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28301037a0 0x7f2830113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:21:14.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.463+0000 7f2835600700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f2830113a40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f2824007f50 tx=0x7f2824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:14.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.464+0000 7f282d600700 1 -- 172.21.15.63:0/291154842 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2824021070 con 0x7f2830102e00 2024-03-23T12:21:14.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.464+0000 7f282d600700 1 -- 172.21.15.63:0/291154842 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2824013e00 con 0x7f2830102e00 2024-03-23T12:21:14.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.464+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283010dc40 con 0x7f2830102e00 2024-03-23T12:21:14.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.464+0000 7f282d600700 1 -- 172.21.15.63:0/291154842 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f282400e070 con 0x7f2830102e00 2024-03-23T12:21:14.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.464+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283010e200 con 0x7f2830102e00 2024-03-23T12:21:14.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.465+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28300fe550 con 0x7f2830102e00 2024-03-23T12:21:14.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.466+0000 7f282d600700 1 -- 172.21.15.63:0/291154842 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2824013950 con 0x7f2830102e00 2024-03-23T12:21:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.467+0000 7f282d600700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f281806c7d0 0x7f281806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.467+0000 7f282d600700 1 -- 172.21.15.63:0/291154842 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f282408cca0 con 0x7f2830102e00 2024-03-23T12:21:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.467+0000 7f2834c00700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f281806c7d0 0x7f281806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.468+0000 7f2834c00700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f281806c7d0 0x7f281806ec80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f281c004600 tx=0x7f281c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:14.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.471+0000 7f282d600700 1 -- 172.21.15.63:0/291154842 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f282405a9b0 con 0x7f2830102e00 2024-03-23T12:21:14.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:14 smithi063 ceph-mon[141124]: from='client.35642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:14 smithi111 ceph-mon[106899]: from='client.35642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:14.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.696+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2830109e30 con 0x7f281806c7d0 2024-03-23T12:21:14.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.706+0000 7f282d600700 1 -- 172.21.15.63:0/291154842 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f2830109e30 con 0x7f281806c7d0 2024-03-23T12:21:14.685 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:21:14.685 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (30m) 7m ago 43m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:21:14.685 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (27m) 7m ago 43m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:21:14.685 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (27m) 7m ago 42m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:21:14.685 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (30m) 7m ago 43m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (23m) 7m ago 37m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (23m) 7m ago 37m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (7m) 7m ago 37m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (22m) 7m ago 37m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (32m) 7m ago 45m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (31m) 7m ago 42m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (28m) 7m ago 45m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (28m) 7m ago 42m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (31m) 7m ago 43m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (31m) 7m ago 42m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (27m) 7m ago 41m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (26m) 7m ago 41m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:21:14.686 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (25m) 7m ago 40m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:21:14.687 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (25m) 7m ago 39m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:21:14.687 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (24m) 7m ago 39m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:21:14.687 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (23m) 7m ago 38m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:21:14.687 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (30m) 7m ago 43m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.708+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f281806c7d0 msgr2=0x7f281806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.708+0000 7f2838000700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f281806c7d0 0x7f281806ec80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f281c004600 tx=0x7f281c00b040 comp rx=0 tx=0).stop 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.708+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 msgr2=0x7f2830113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.708+0000 7f2838000700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f2830113a40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f2824007f50 tx=0x7f2824009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.709+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 shutdown_connections 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.709+0000 7f2838000700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f281806c7d0 0x7f281806ec80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.709+0000 7f2838000700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2830102e00 0x7f2830113a40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.709+0000 7f2838000700 1 --2- 172.21.15.63:0/291154842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28301037a0 0x7f2830113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.709+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 >> 172.21.15.63:0/291154842 conn(0x7f28300f89b0 msgr2=0x7f28300f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:14.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.709+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 shutdown_connections 2024-03-23T12:21:14.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.709+0000 7f2838000700 1 -- 172.21.15.63:0/291154842 wait complete. 2024-03-23T12:21:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 -- 172.21.15.63:0/2829621443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c102c40 msgr2=0x7f6f0c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 --2- 172.21.15.63:0/2829621443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c102c40 0x7f6f0c103010 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6efc004940 tx=0x7f6efc009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 -- 172.21.15.63:0/2829621443 shutdown_connections 2024-03-23T12:21:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 --2- 172.21.15.63:0/2829621443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0c1035e0 0x7f6f0c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 --2- 172.21.15.63:0/2829621443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c102c40 0x7f6f0c103010 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 -- 172.21.15.63:0/2829621443 >> 172.21.15.63:0/2829621443 conn(0x7f6f0c0f8830 msgr2=0x7f6f0c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 -- 172.21.15.63:0/2829621443 shutdown_connections 2024-03-23T12:21:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.845+0000 7f6f10e00700 1 -- 172.21.15.63:0/2829621443 wait complete. 2024-03-23T12:21:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.846+0000 7f6f10e00700 1 Processor -- start 2024-03-23T12:21:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.846+0000 7f6f10e00700 1 -- start start 2024-03-23T12:21:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.846+0000 7f6f10e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0c102c40 0x7f6f0c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.846+0000 7f6f10e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c1035e0 0x7f6f0c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.846+0000 7f6f10e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c070d90 con 0x7f6f0c1035e0 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.846+0000 7f6f10e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c070ed0 con 0x7f6f0c102c40 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c1035e0 0x7f6f0c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c1035e0 0x7f6f0c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47350/0 (socket says 172.21.15.63:47350) 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0aa00700 1 -- 172.21.15.63:0/1753058140 learned_addr learned my addr 172.21.15.63:0/1753058140 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0b400700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0c102c40 0x7f6f0c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0aa00700 1 -- 172.21.15.63:0/1753058140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0c102c40 msgr2=0x7f6f0c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0aa00700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0c102c40 0x7f6f0c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0aa00700 1 -- 172.21.15.63:0/1753058140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc005040 con 0x7f6f0c1035e0 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f0aa00700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c1035e0 0x7f6f0c070800 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800eb40 tx=0x7f6ef800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.847+0000 7f6f03400700 1 -- 172.21.15.63:0/1753058140 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ef8007cc0 con 0x7f6f0c1035e0 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.848+0000 7f6f03400700 1 -- 172.21.15.63:0/1753058140 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ef800f040 con 0x7f6f0c1035e0 2024-03-23T12:21:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.848+0000 7f6f03400700 1 -- 172.21.15.63:0/1753058140 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ef800c8f0 con 0x7f6f0c1035e0 2024-03-23T12:21:14.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.848+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c071120 con 0x7f6f0c1035e0 2024-03-23T12:21:14.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.848+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c119fa0 con 0x7f6f0c1035e0 2024-03-23T12:21:14.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.849+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c0fe390 con 0x7f6f0c1035e0 2024-03-23T12:21:14.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.850+0000 7f6f03400700 1 -- 172.21.15.63:0/1753058140 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6ef800cb60 con 0x7f6f0c1035e0 2024-03-23T12:21:14.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.851+0000 7f6f03400700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6ef006c7d0 0x7f6ef006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:14.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.851+0000 7f6f03400700 1 -- 172.21.15.63:0/1753058140 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ef808b590 con 0x7f6f0c1035e0 2024-03-23T12:21:14.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.851+0000 7f6f0b400700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6ef006c7d0 0x7f6ef006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:14.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.852+0000 7f6f0b400700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6ef006c7d0 0x7f6ef006ec80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00b0d0 tx=0x7f6efc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:14.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:14.855+0000 7f6f03400700 1 -- 172.21.15.63:0/1753058140 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6ef80557c0 con 0x7f6f0c1035e0 2024-03-23T12:21:15.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.161+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6f0c05f5f0 con 0x7f6f0c1035e0 2024-03-23T12:21:15.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.161+0000 7f6f03400700 1 -- 172.21.15.63:0/1753058140 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f6ef8058de0 con 0x7f6f0c1035e0 2024-03-23T12:21:15.140 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:21:15.140 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:21:15.141 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:21:15.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:21:15.142 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:21:15.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.163+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6ef006c7d0 msgr2=0x7f6ef006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.163+0000 7f6f10e00700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6ef006c7d0 0x7f6ef006ec80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00b0d0 tx=0x7f6efc009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.163+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c1035e0 msgr2=0x7f6f0c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.163+0000 7f6f10e00700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c1035e0 0x7f6f0c070800 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800eb40 tx=0x7f6ef800b040 comp rx=0 tx=0).stop 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.164+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 shutdown_connections 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.164+0000 7f6f10e00700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6ef006c7d0 0x7f6ef006ec80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.164+0000 7f6f10e00700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0c102c40 0x7f6f0c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.164+0000 7f6f10e00700 1 --2- 172.21.15.63:0/1753058140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f0c1035e0 0x7f6f0c070800 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.164+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 >> 172.21.15.63:0/1753058140 conn(0x7f6f0c0f8830 msgr2=0x7f6f0c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.164+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 shutdown_connections 2024-03-23T12:21:15.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.164+0000 7f6f10e00700 1 -- 172.21.15.63:0/1753058140 wait complete. 2024-03-23T12:21:15.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 -- 172.21.15.63:0/3986063615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40065630 msgr2=0x7f7b40065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3986063615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40065630 0x7f7b40065a80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c004940 tx=0x7f7b2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:15.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 -- 172.21.15.63:0/3986063615 shutdown_connections 2024-03-23T12:21:15.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3986063615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40065630 0x7f7b40065a80 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3986063615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b40102ca0 0x7f7b40103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 -- 172.21.15.63:0/3986063615 >> 172.21.15.63:0/3986063615 conn(0x7f7b40071b70 msgr2=0x7f7b40071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:15.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 -- 172.21.15.63:0/3986063615 shutdown_connections 2024-03-23T12:21:15.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.302+0000 7f7b3f200700 1 -- 172.21.15.63:0/3986063615 wait complete. 2024-03-23T12:21:15.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.303+0000 7f7b3f200700 1 Processor -- start 2024-03-23T12:21:15.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.303+0000 7f7b3f200700 1 -- start start 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b40065630 0x7f7b401138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40102ca0 0x7f7b40113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b40114450 con 0x7f7b40102ca0 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4010d840 con 0x7f7b40065630 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40102ca0 0x7f7b40113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40102ca0 0x7f7b40113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47364/0 (socket says 172.21.15.63:47364) 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3d400700 1 -- 172.21.15.63:0/3940971617 learned_addr learned my addr 172.21.15.63:0/3940971617 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3de00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b40065630 0x7f7b401138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3d400700 1 -- 172.21.15.63:0/3940971617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b40065630 msgr2=0x7f7b401138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.304+0000 7f7b3d400700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b40065630 0x7f7b401138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.305+0000 7f7b3d400700 1 -- 172.21.15.63:0/3940971617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c005040 con 0x7f7b40102ca0 2024-03-23T12:21:15.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.305+0000 7f7b3d400700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40102ca0 0x7f7b40113e00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009180 tx=0x7f7b2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:15.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.305+0000 7f7b36000700 1 -- 172.21.15.63:0/3940971617 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b2c00e040 con 0x7f7b40102ca0 2024-03-23T12:21:15.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.305+0000 7f7b36000700 1 -- 172.21.15.63:0/3940971617 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b2c004290 con 0x7f7b40102ca0 2024-03-23T12:21:15.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.305+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4010dac0 con 0x7f7b40102ca0 2024-03-23T12:21:15.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.305+0000 7f7b36000700 1 -- 172.21.15.63:0/3940971617 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b2c013e60 con 0x7f7b40102ca0 2024-03-23T12:21:15.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.305+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4010e080 con 0x7f7b40102ca0 2024-03-23T12:21:15.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.306+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b400fe380 con 0x7f7b40102ca0 2024-03-23T12:21:15.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.307+0000 7f7b36000700 1 -- 172.21.15.63:0/3940971617 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7b2c0136b0 con 0x7f7b40102ca0 2024-03-23T12:21:15.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.308+0000 7f7b36000700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2406c7c0 0x7f7b2406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:15.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.308+0000 7f7b36000700 1 -- 172.21.15.63:0/3940971617 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b2c08cc90 con 0x7f7b40102ca0 2024-03-23T12:21:15.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.308+0000 7f7b3de00700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2406c7c0 0x7f7b2406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:15.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.309+0000 7f7b3de00700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2406c7c0 0x7f7b2406ec70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7b30004f40 tx=0x7f7b3000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:15.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.313+0000 7f7b36000700 1 -- 172.21.15.63:0/3940971617 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b2c05a9a0 con 0x7f7b40102ca0 2024-03-23T12:21:15.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.571+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7b4005f5f0 con 0x7f7b40102ca0 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.572+0000 7f7b36000700 1 -- 172.21.15.63:0/3940971617 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f7b2c05a530 con 0x7f7b40102ca0 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:21:15.551 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:21:15.552 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:15.553 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:15.554 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:15.554 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:15.554 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:15.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2406c7c0 msgr2=0x7f7b2406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2406c7c0 0x7f7b2406ec70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7b30004f40 tx=0x7f7b3000b040 comp rx=0 tx=0).stop 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40102ca0 msgr2=0x7f7b40113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40102ca0 0x7f7b40113e00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009180 tx=0x7f7b2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 shutdown_connections 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7b2406c7c0 0x7f7b2406ec70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b40065630 0x7f7b401138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 --2- 172.21.15.63:0/3940971617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b40102ca0 0x7f7b40113e00 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 >> 172.21.15.63:0/3940971617 conn(0x7f7b40071b70 msgr2=0x7f7b401059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 shutdown_connections 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.575+0000 7f7b3f200700 1 -- 172.21.15.63:0/3940971617 wait complete. 2024-03-23T12:21:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:21:15.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:15 smithi063 ceph-mon[141124]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:15.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:15 smithi063 ceph-mon[141124]: from='client.35650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:15.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:15 smithi063 ceph-mon[141124]: pgmap v905: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:15.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:15 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1753058140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:21:15.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:15 smithi111 ceph-mon[106899]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:15.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:15 smithi111 ceph-mon[106899]: from='client.35650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:15.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:15 smithi111 ceph-mon[106899]: pgmap v905: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:15.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:15 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1753058140' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:21:15.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 -- 172.21.15.63:0/3587686503 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 msgr2=0x7effa40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 --2- 172.21.15.63:0/3587686503 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa40fce10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7eff8c004940 tx=0x7eff8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:15.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 -- 172.21.15.63:0/3587686503 shutdown_connections 2024-03-23T12:21:15.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 --2- 172.21.15.63:0/3587686503 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa40fce10 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 --2- 172.21.15.63:0/3587686503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa4102e00 0x7effa41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 -- 172.21.15.63:0/3587686503 >> 172.21.15.63:0/3587686503 conn(0x7effa40f89b0 msgr2=0x7effa40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:15.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 -- 172.21.15.63:0/3587686503 shutdown_connections 2024-03-23T12:21:15.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.714+0000 7effaa000700 1 -- 172.21.15.63:0/3587686503 wait complete. 2024-03-23T12:21:15.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.715+0000 7effaa000700 1 Processor -- start 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.715+0000 7effaa000700 1 -- start start 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.715+0000 7effaa000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa4102e00 0x7effa4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.715+0000 7effaa000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.715+0000 7effaa000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa4114600 con 0x7effa41037a0 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.715+0000 7effaa000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa410d9f0 con 0x7effa4102e00 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7eff9be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7eff9be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa4113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47384/0 (socket says 172.21.15.63:47384) 2024-03-23T12:21:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7eff9be00700 1 -- 172.21.15.63:0/25052029 learned_addr learned my addr 172.21.15.63:0/25052029 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:15.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7effa3400700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa4102e00 0x7effa4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:15.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7eff9be00700 1 -- 172.21.15.63:0/25052029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa4102e00 msgr2=0x7effa4113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7eff9be00700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa4102e00 0x7effa4113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7eff9be00700 1 -- 172.21.15.63:0/25052029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8c005040 con 0x7effa41037a0 2024-03-23T12:21:15.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.716+0000 7eff9be00700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa4113fb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009fd0 tx=0x7eff8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:15.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.717+0000 7eff9b400700 1 -- 172.21.15.63:0/25052029 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff8c00e030 con 0x7effa41037a0 2024-03-23T12:21:15.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.717+0000 7eff9b400700 1 -- 172.21.15.63:0/25052029 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff8c013e00 con 0x7effa41037a0 2024-03-23T12:21:15.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.717+0000 7eff9b400700 1 -- 172.21.15.63:0/25052029 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff8c01ccd0 con 0x7effa41037a0 2024-03-23T12:21:15.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.717+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa410dc70 con 0x7effa41037a0 2024-03-23T12:21:15.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.717+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa410e230 con 0x7effa41037a0 2024-03-23T12:21:15.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.718+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa40fe550 con 0x7effa41037a0 2024-03-23T12:21:15.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.719+0000 7eff9b400700 1 -- 172.21.15.63:0/25052029 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7eff8c01c500 con 0x7effa41037a0 2024-03-23T12:21:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.720+0000 7eff9b400700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7eff8806c7d0 0x7eff8806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.720+0000 7eff9b400700 1 -- 172.21.15.63:0/25052029 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7eff8c08d2a0 con 0x7effa41037a0 2024-03-23T12:21:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.720+0000 7effa3400700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7eff8806c7d0 0x7eff8806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.721+0000 7effa3400700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7eff8806c7d0 0x7eff8806ec80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7eff94009fd0 tx=0x7eff9400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:15.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.724+0000 7eff9b400700 1 -- 172.21.15.63:0/25052029 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff8c05aff0 con 0x7effa41037a0 2024-03-23T12:21:15.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.956+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7effa4109dc0 con 0x7eff8806c7d0 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.958+0000 7eff9b400700 1 -- 172.21.15.63:0/25052029 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7effa4109dc0 con 0x7eff8806c7d0 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:21:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:21:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:21:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:21:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:21:15.938 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:21:15.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7eff8806c7d0 msgr2=0x7eff8806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7eff8806c7d0 0x7eff8806ec80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7eff94009fd0 tx=0x7eff9400b040 comp rx=0 tx=0).stop 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 msgr2=0x7effa4113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa4113fb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009fd0 tx=0x7eff8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 shutdown_connections 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7eff8806c7d0 0x7eff8806ec80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa4102e00 0x7effa4113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 --2- 172.21.15.63:0/25052029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa41037a0 0x7effa4113fb0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 >> 172.21.15.63:0/25052029 conn(0x7effa40f89b0 msgr2=0x7effa40f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 shutdown_connections 2024-03-23T12:21:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:15.960+0000 7effaa000700 1 -- 172.21.15.63:0/25052029 wait complete. 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 -- 172.21.15.63:0/1934138554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 msgr2=0x7f6d980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/1934138554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d980fce10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6d84004940 tx=0x7f6d84009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 -- 172.21.15.63:0/1934138554 shutdown_connections 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/1934138554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d980fce10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/1934138554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d98102e00 0x7f6d981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 -- 172.21.15.63:0/1934138554 >> 172.21.15.63:0/1934138554 conn(0x7f6d980f89b0 msgr2=0x7f6d980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 -- 172.21.15.63:0/1934138554 shutdown_connections 2024-03-23T12:21:16.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.097+0000 7f6d9fe00700 1 -- 172.21.15.63:0/1934138554 wait complete. 2024-03-23T12:21:16.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.098+0000 7f6d9fe00700 1 Processor -- start 2024-03-23T12:21:16.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.098+0000 7f6d9fe00700 1 -- start start 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.098+0000 7f6d9fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d98102e00 0x7f6d98115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.098+0000 7f6d9fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d98116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.098+0000 7f6d9fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d98116890 con 0x7f6d981037a0 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.098+0000 7f6d9fe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9810fc60 con 0x7f6d98102e00 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d98116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d98116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47408/0 (socket says 172.21.15.63:47408) 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9ca00700 1 -- 172.21.15.63:0/2095446724 learned_addr learned my addr 172.21.15.63:0/2095446724 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:16.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9d400700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d98102e00 0x7f6d98115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:16.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9ca00700 1 -- 172.21.15.63:0/2095446724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d98102e00 msgr2=0x7f6d98115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:16.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9ca00700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d98102e00 0x7f6d98115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:16.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9ca00700 1 -- 172.21.15.63:0/2095446724 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d84005040 con 0x7f6d981037a0 2024-03-23T12:21:16.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9d400700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d98102e00 0x7f6d98115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d9ca00700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d98116240 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6d840093b0 tx=0x7f6d84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.099+0000 7f6d95600700 1 -- 172.21.15.63:0/2095446724 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d8400e040 con 0x7f6d981037a0 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.100+0000 7f6d95600700 1 -- 172.21.15.63:0/2095446724 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d84013420 con 0x7f6d981037a0 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.100+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9810fee0 con 0x7f6d981037a0 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.100+0000 7f6d95600700 1 -- 172.21.15.63:0/2095446724 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d8401c630 con 0x7f6d981037a0 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.100+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d981104a0 con 0x7f6d981037a0 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.101+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d980fe550 con 0x7f6d981037a0 2024-03-23T12:21:16.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.102+0000 7f6d95600700 1 -- 172.21.15.63:0/2095446724 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6d840139e0 con 0x7f6d981037a0 2024-03-23T12:21:16.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.102+0000 7f6d95600700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8006c7c0 0x7f6d8006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:16.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.103+0000 7f6d95600700 1 -- 172.21.15.63:0/2095446724 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d8408ccc0 con 0x7f6d981037a0 2024-03-23T12:21:16.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.103+0000 7f6d9d400700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8006c7c0 0x7f6d8006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:16.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.104+0000 7f6d9d400700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8006c7c0 0x7f6d8006ec70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c007fd0 tx=0x7f6d8c009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:16.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.108+0000 7f6d95600700 1 -- 172.21.15.63:0/2095446724 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d8405aa80 con 0x7f6d981037a0 2024-03-23T12:21:16.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.401+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6d98109da0 con 0x7f6d981037a0 2024-03-23T12:21:16.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.401+0000 7f6d95600700 1 -- 172.21.15.63:0/2095446724 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f6d8405a610 con 0x7f6d981037a0 2024-03-23T12:21:16.380 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:21:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:21:16.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.403+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8006c7c0 msgr2=0x7f6d8006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.403+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8006c7c0 0x7f6d8006ec70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c007fd0 tx=0x7f6d8c009040 comp rx=0 tx=0).stop 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.403+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 msgr2=0x7f6d98116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.403+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d98116240 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6d840093b0 tx=0x7f6d84009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.404+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 shutdown_connections 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.404+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8006c7c0 0x7f6d8006ec70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.404+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d98102e00 0x7f6d98115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.404+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/2095446724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d981037a0 0x7f6d98116240 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.404+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 >> 172.21.15.63:0/2095446724 conn(0x7f6d980f89b0 msgr2=0x7f6d980f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.404+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 shutdown_connections 2024-03-23T12:21:16.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:16.404+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2095446724 wait complete. 2024-03-23T12:21:16.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:16 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3940971617' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:21:16.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:16 smithi063 ceph-mon[141124]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:16.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:16 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3940971617' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:21:16.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:16 smithi111 ceph-mon[106899]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:17.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:17 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2095446724' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:21:17.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:17 smithi063 ceph-mon[141124]: pgmap v906: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:17.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:17 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2095446724' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:21:17.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:17 smithi111 ceph-mon[106899]: pgmap v906: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:19.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:19 smithi063 ceph-mon[141124]: pgmap v907: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:19.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:19 smithi111 ceph-mon[106899]: pgmap v907: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:22.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:21 smithi063 ceph-mon[141124]: pgmap v908: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:22.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:21 smithi111 ceph-mon[106899]: pgmap v908: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:23.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:23 smithi063 ceph-mon[141124]: pgmap v909: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:23.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:23 smithi111 ceph-mon[106899]: pgmap v909: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:25.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:25 smithi063 ceph-mon[141124]: pgmap v910: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:25.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:25 smithi111 ceph-mon[106899]: pgmap v910: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:28.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:27 smithi063 ceph-mon[141124]: pgmap v911: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:28.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:27 smithi111 ceph-mon[106899]: pgmap v911: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:29.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:29 smithi063 ceph-mon[141124]: pgmap v912: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:29.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:29 smithi111 ceph-mon[106899]: pgmap v912: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:32.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:31 smithi063 ceph-mon[141124]: pgmap v913: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:32.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:31 smithi111 ceph-mon[106899]: pgmap v913: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:33.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:33 smithi063 ceph-mon[141124]: pgmap v914: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:33.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:33 smithi111 ceph-mon[106899]: pgmap v914: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:35.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:35 smithi063 ceph-mon[141124]: pgmap v915: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:35.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:35 smithi111 ceph-mon[106899]: pgmap v915: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:37.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:37 smithi063 ceph-mon[141124]: pgmap v916: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:37.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:37 smithi111 ceph-mon[106899]: pgmap v916: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:39.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:39 smithi063 ceph-mon[141124]: pgmap v917: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:39.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:39 smithi111 ceph-mon[106899]: pgmap v917: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:42.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:42.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:41 smithi063 ceph-mon[141124]: pgmap v918: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:42.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:42.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:41 smithi111 ceph-mon[106899]: pgmap v918: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:43.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:43 smithi063 ceph-mon[141124]: pgmap v919: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:43.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:43 smithi111 ceph-mon[106899]: pgmap v919: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:45.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:45 smithi063 ceph-mon[141124]: pgmap v920: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:45.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:45 smithi111 ceph-mon[106899]: pgmap v920: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.555+0000 7f4680000700 1 -- 172.21.15.63:0/795681187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46780708a0 msgr2=0x7f4678070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.555+0000 7f4680000700 1 --2- 172.21.15.63:0/795681187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46780708a0 0x7f4678070d10 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4664004970 tx=0x7f4664009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.556+0000 7f4680000700 1 -- 172.21.15.63:0/795681187 shutdown_connections 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.556+0000 7f4680000700 1 --2- 172.21.15.63:0/795681187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46780708a0 0x7f4678070d10 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.556+0000 7f4680000700 1 --2- 172.21.15.63:0/795681187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678071f90 0x7f4678070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.556+0000 7f4680000700 1 -- 172.21.15.63:0/795681187 >> 172.21.15.63:0/795681187 conn(0x7f46780f89b0 msgr2=0x7f46780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.556+0000 7f4680000700 1 -- 172.21.15.63:0/795681187 shutdown_connections 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.556+0000 7f4680000700 1 -- 172.21.15.63:0/795681187 wait complete. 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.556+0000 7f4680000700 1 Processor -- start 2024-03-23T12:21:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f4680000700 1 -- start start 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f4680000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780708a0 0x7f467811a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f4680000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4678071f90 0x7f467811a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f4680000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467811ab90 con 0x7f4678071f90 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f4680000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4678114010 con 0x7f46780708a0 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f467cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4678071f90 0x7f467811a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f467cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4678071f90 0x7f467811a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45652/0 (socket says 172.21.15.63:45652) 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.557+0000 7f467cc00700 1 -- 172.21.15.63:0/191527571 learned_addr learned my addr 172.21.15.63:0/191527571 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f467d600700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780708a0 0x7f467811a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f467cc00700 1 -- 172.21.15.63:0/191527571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780708a0 msgr2=0x7f467811a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f467cc00700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780708a0 0x7f467811a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f467cc00700 1 -- 172.21.15.63:0/191527571 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664005040 con 0x7f4678071f90 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f467cc00700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4678071f90 0x7f467811a540 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4664009360 tx=0x7f4664009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f4675600700 1 -- 172.21.15.63:0/191527571 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f466400e040 con 0x7f4678071f90 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f4675600700 1 -- 172.21.15.63:0/191527571 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4664004290 con 0x7f4678071f90 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.558+0000 7f4675600700 1 -- 172.21.15.63:0/191527571 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4664013e70 con 0x7f4678071f90 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.559+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4678114290 con 0x7f4678071f90 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.559+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4678114780 con 0x7f4678071f90 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.560+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4678071aa0 con 0x7f4678071f90 2024-03-23T12:21:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.561+0000 7f4675600700 1 -- 172.21.15.63:0/191527571 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4664013620 con 0x7f4678071f90 2024-03-23T12:21:46.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.561+0000 7f4675600700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f466006c7d0 0x7f466006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:46.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.562+0000 7f4675600700 1 -- 172.21.15.63:0/191527571 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f466408cde0 con 0x7f4678071f90 2024-03-23T12:21:46.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.562+0000 7f467d600700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f466006c7d0 0x7f466006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:46.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.562+0000 7f467d600700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f466006c7d0 0x7f466006ec80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f466c004f20 tx=0x7f466c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:46.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.566+0000 7f4675600700 1 -- 172.21.15.63:0/191527571 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4664056fe0 con 0x7f4678071f90 2024-03-23T12:21:46.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.799+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f46781103e0 con 0x7f466006c7d0 2024-03-23T12:21:46.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.800+0000 7f4675600700 1 -- 172.21.15.63:0/191527571 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f46781103e0 con 0x7f466006c7d0 2024-03-23T12:21:46.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f466006c7d0 msgr2=0x7f466006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:46.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f466006c7d0 0x7f466006ec80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f466c004f20 tx=0x7f466c00b040 comp rx=0 tx=0).stop 2024-03-23T12:21:46.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4678071f90 msgr2=0x7f467811a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:46.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4678071f90 0x7f467811a540 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4664009360 tx=0x7f4664009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:46.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 shutdown_connections 2024-03-23T12:21:46.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f466006c7d0 0x7f466006ec80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780708a0 0x7f467811a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 --2- 172.21.15.63:0/191527571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4678071f90 0x7f467811a540 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 >> 172.21.15.63:0/191527571 conn(0x7f46780f89b0 msgr2=0x7f46780f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:46.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 shutdown_connections 2024-03-23T12:21:46.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.802+0000 7f4680000700 1 -- 172.21.15.63:0/191527571 wait complete. 2024-03-23T12:21:46.794 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 -- 172.21.15.63:0/3262604633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa0102e00 msgr2=0x7fcaa01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 --2- 172.21.15.63:0/3262604633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa0102e00 0x7fcaa01031d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fca90004940 tx=0x7fca90009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 -- 172.21.15.63:0/3262604633 shutdown_connections 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 --2- 172.21.15.63:0/3262604633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaa01037a0 0x7fcaa00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 --2- 172.21.15.63:0/3262604633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa0102e00 0x7fcaa01031d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 -- 172.21.15.63:0/3262604633 >> 172.21.15.63:0/3262604633 conn(0x7fcaa00f89b0 msgr2=0x7fcaa00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 -- 172.21.15.63:0/3262604633 shutdown_connections 2024-03-23T12:21:46.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.941+0000 7fcaa6200700 1 -- 172.21.15.63:0/3262604633 wait complete. 2024-03-23T12:21:46.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.942+0000 7fcaa6200700 1 Processor -- start 2024-03-23T12:21:46.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.942+0000 7fcaa6200700 1 -- start start 2024-03-23T12:21:46.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.942+0000 7fcaa6200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaa0102e00 0x7fcaa00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:46.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.942+0000 7fcaa6200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa01037a0 0x7fcaa0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:46.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.942+0000 7fcaa6200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa0070e20 con 0x7fcaa01037a0 2024-03-23T12:21:46.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.942+0000 7fcaa6200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa0070f60 con 0x7fcaa0102e00 2024-03-23T12:21:46.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca97e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa01037a0 0x7fcaa0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:46.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca97e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa01037a0 0x7fcaa0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45680/0 (socket says 172.21.15.63:45680) 2024-03-23T12:21:46.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca97e00700 1 -- 172.21.15.63:0/1291313264 learned_addr learned my addr 172.21.15.63:0/1291313264 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:46.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca9f400700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaa0102e00 0x7fcaa00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:46.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca97e00700 1 -- 172.21.15.63:0/1291313264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaa0102e00 msgr2=0x7fcaa00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca97e00700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaa0102e00 0x7fcaa00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca97e00700 1 -- 172.21.15.63:0/1291313264 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca90005040 con 0x7fcaa01037a0 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.943+0000 7fca97e00700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa01037a0 0x7fcaa0070800 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fca8800b790 tx=0x7fca88009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.944+0000 7fca97400700 1 -- 172.21.15.63:0/1291313264 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca8800f6f0 con 0x7fcaa01037a0 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.944+0000 7fca97400700 1 -- 172.21.15.63:0/1291313264 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca8800fd00 con 0x7fcaa01037a0 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.944+0000 7fca97400700 1 -- 172.21.15.63:0/1291313264 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca880103a0 con 0x7fcaa01037a0 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.944+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa0119da0 con 0x7fcaa01037a0 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.944+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa011a2b0 con 0x7fcaa01037a0 2024-03-23T12:21:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.945+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa00fe550 con 0x7fcaa01037a0 2024-03-23T12:21:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.946+0000 7fca97400700 1 -- 172.21.15.63:0/1291313264 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fca8800f850 con 0x7fcaa01037a0 2024-03-23T12:21:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.947+0000 7fca97400700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca8406c7d0 0x7fca8406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.947+0000 7fca97400700 1 -- 172.21.15.63:0/1291313264 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fca8808b840 con 0x7fcaa01037a0 2024-03-23T12:21:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.947+0000 7fca9f400700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca8406c7d0 0x7fca8406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.948+0000 7fca9f400700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca8406c7d0 0x7fca8406ec80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fca90009fd0 tx=0x7fca90009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:46.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:46.951+0000 7fca97400700 1 -- 172.21.15.63:0/1291313264 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fca88055ac0 con 0x7fcaa01037a0 2024-03-23T12:21:47.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.187+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcaa004bed0 con 0x7fca8406c7d0 2024-03-23T12:21:47.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.189+0000 7fca97400700 1 -- 172.21.15.63:0/1291313264 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcaa004bed0 con 0x7fca8406c7d0 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca8406c7d0 msgr2=0x7fca8406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca8406c7d0 0x7fca8406ec80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fca90009fd0 tx=0x7fca90009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa01037a0 msgr2=0x7fcaa0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa01037a0 0x7fcaa0070800 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fca8800b790 tx=0x7fca88009220 comp rx=0 tx=0).stop 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 shutdown_connections 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fca8406c7d0 0x7fca8406ec80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcaa0102e00 0x7fcaa00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 --2- 172.21.15.63:0/1291313264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcaa01037a0 0x7fcaa0070800 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 >> 172.21.15.63:0/1291313264 conn(0x7fcaa00f89b0 msgr2=0x7fcaa00f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 shutdown_connections 2024-03-23T12:21:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.191+0000 7fcaa6200700 1 -- 172.21.15.63:0/1291313264 wait complete. 2024-03-23T12:21:47.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 -- 172.21.15.63:0/4086858815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 msgr2=0x7f69e41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 --2- 172.21.15.63:0/4086858815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e41031d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f69d8004940 tx=0x7f69d8009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:47.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 -- 172.21.15.63:0/4086858815 shutdown_connections 2024-03-23T12:21:47.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 --2- 172.21.15.63:0/4086858815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69e41037a0 0x7f69e40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 --2- 172.21.15.63:0/4086858815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e41031d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 -- 172.21.15.63:0/4086858815 >> 172.21.15.63:0/4086858815 conn(0x7f69e40f89b0 msgr2=0x7f69e40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:47.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 -- 172.21.15.63:0/4086858815 shutdown_connections 2024-03-23T12:21:47.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.329+0000 7f69eb600700 1 -- 172.21.15.63:0/4086858815 wait complete. 2024-03-23T12:21:47.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.330+0000 7f69eb600700 1 Processor -- start 2024-03-23T12:21:47.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.330+0000 7f69eb600700 1 -- start start 2024-03-23T12:21:47.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.330+0000 7f69eb600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:47.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.330+0000 7f69eb600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69e41037a0 0x7f69e4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:47.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.330+0000 7f69eb600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4116860 con 0x7f69e4102e00 2024-03-23T12:21:47.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.330+0000 7f69eb600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e410fc30 con 0x7f69e41037a0 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e4115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45694/0 (socket says 172.21.15.63:45694) 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e8c00700 1 -- 172.21.15.63:0/2973082531 learned_addr learned my addr 172.21.15.63:0/2973082531 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e3e00700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69e41037a0 0x7f69e4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e8c00700 1 -- 172.21.15.63:0/2973082531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69e41037a0 msgr2=0x7f69e4116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e8c00700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69e41037a0 0x7f69e4116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e8c00700 1 -- 172.21.15.63:0/2973082531 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d8005040 con 0x7f69e4102e00 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.331+0000 7f69e3e00700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69e41037a0 0x7f69e4116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:21:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.332+0000 7f69e8c00700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e4115cd0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009390 tx=0x7f69d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:47.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.332+0000 7f69e0c00700 1 -- 172.21.15.63:0/2973082531 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d800e040 con 0x7f69e4102e00 2024-03-23T12:21:47.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.332+0000 7f69e0c00700 1 -- 172.21.15.63:0/2973082531 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d8013420 con 0x7f69e4102e00 2024-03-23T12:21:47.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.332+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e410feb0 con 0x7f69e4102e00 2024-03-23T12:21:47.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.332+0000 7f69e0c00700 1 -- 172.21.15.63:0/2973082531 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d801c630 con 0x7f69e4102e00 2024-03-23T12:21:47.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.332+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e4110470 con 0x7f69e4102e00 2024-03-23T12:21:47.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.333+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e40fe550 con 0x7f69e4102e00 2024-03-23T12:21:47.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.334+0000 7f69e0c00700 1 -- 172.21.15.63:0/2973082531 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f69d80139e0 con 0x7f69e4102e00 2024-03-23T12:21:47.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.335+0000 7f69e0c00700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f69cc06c7d0 0x7f69cc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:47.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.335+0000 7f69e0c00700 1 -- 172.21.15.63:0/2973082531 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f69d808cbd0 con 0x7f69e4102e00 2024-03-23T12:21:47.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.335+0000 7f69e3e00700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f69cc06c7d0 0x7f69cc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:47.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.336+0000 7f69e3e00700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f69cc06c7d0 0x7f69cc06ec80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009fd0 tx=0x7f69d0009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:47.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.339+0000 7f69e0c00700 1 -- 172.21.15.63:0/2973082531 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f69d805aa50 con 0x7f69e4102e00 2024-03-23T12:21:47.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.563+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f69e404bf40 con 0x7f69cc06c7d0 2024-03-23T12:21:47.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.574+0000 7f69e0c00700 1 -- 172.21.15.63:0/2973082531 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f69e404bf40 con 0x7f69cc06c7d0 2024-03-23T12:21:47.552 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:21:47.552 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (31m) 8m ago 44m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:21:47.552 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (28m) 8m ago 44m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:21:47.552 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (28m) 8m ago 43m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:21:47.552 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (30m) 8m ago 43m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:21:47.552 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (24m) 8m ago 37m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (23m) 8m ago 37m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (8m) 8m ago 37m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (23m) 8m ago 37m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (33m) 8m ago 45m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (32m) 8m ago 43m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (29m) 8m ago 45m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (29m) 8m ago 43m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (31m) 8m ago 44m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (31m) 8m ago 43m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (27m) 8m ago 42m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (27m) 8m ago 41m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (26m) 8m ago 41m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:21:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (25m) 8m ago 40m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:21:47.554 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (25m) 8m ago 40m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:21:47.554 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (24m) 8m ago 39m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:21:47.554 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (31m) 8m ago 43m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:21:47.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f69cc06c7d0 msgr2=0x7f69cc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f69cc06c7d0 0x7f69cc06ec80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009fd0 tx=0x7f69d0009ee0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 msgr2=0x7f69e4115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e4115cd0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009390 tx=0x7f69d8009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:47.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 shutdown_connections 2024-03-23T12:21:47.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f69cc06c7d0 0x7f69cc06ec80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69e4102e00 0x7f69e4115cd0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 --2- 172.21.15.63:0/2973082531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69e41037a0 0x7f69e4116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 >> 172.21.15.63:0/2973082531 conn(0x7f69e40f89b0 msgr2=0x7f69e40f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:47.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 shutdown_connections 2024-03-23T12:21:47.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.577+0000 7f69eb600700 1 -- 172.21.15.63:0/2973082531 wait complete. 2024-03-23T12:21:47.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:47 smithi063 ceph-mon[141124]: from='client.35670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:47.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:47 smithi063 ceph-mon[141124]: pgmap v921: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:47.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:47 smithi111 ceph-mon[106899]: from='client.35670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:47.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:47 smithi111 ceph-mon[106899]: pgmap v921: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:47.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 -- 172.21.15.63:0/4155106140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 msgr2=0x7fcc900fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 --2- 172.21.15.63:0/4155106140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc900fce10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fcc78004940 tx=0x7fcc78009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:47.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 -- 172.21.15.63:0/4155106140 shutdown_connections 2024-03-23T12:21:47.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 --2- 172.21.15.63:0/4155106140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc900fce10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 --2- 172.21.15.63:0/4155106140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc90102e00 0x7fcc901031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 -- 172.21.15.63:0/4155106140 >> 172.21.15.63:0/4155106140 conn(0x7fcc900f89b0 msgr2=0x7fcc900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 -- 172.21.15.63:0/4155106140 shutdown_connections 2024-03-23T12:21:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.714+0000 7fcc96200700 1 -- 172.21.15.63:0/4155106140 wait complete. 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.715+0000 7fcc96200700 1 Processor -- start 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.715+0000 7fcc96200700 1 -- start start 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.715+0000 7fcc96200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc90102e00 0x7fcc90113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.715+0000 7fcc96200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc90113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.715+0000 7fcc96200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90114600 con 0x7fcc901037a0 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.715+0000 7fcc96200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9010d9f0 con 0x7fcc90102e00 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc87e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc90113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc87e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc90113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45722/0 (socket says 172.21.15.63:45722) 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc87e00700 1 -- 172.21.15.63:0/1244629316 learned_addr learned my addr 172.21.15.63:0/1244629316 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc87e00700 1 -- 172.21.15.63:0/1244629316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc90102e00 msgr2=0x7fcc90113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc8f400700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc90102e00 0x7fcc90113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc87e00700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc90102e00 0x7fcc90113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc87e00700 1 -- 172.21.15.63:0/1244629316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc78005040 con 0x7fcc901037a0 2024-03-23T12:21:47.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc8f400700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc90102e00 0x7fcc90113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.716+0000 7fcc87e00700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc90113fb0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fcc78008d50 tx=0x7fcc78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.717+0000 7fcc87400700 1 -- 172.21.15.63:0/1244629316 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7800e030 con 0x7fcc901037a0 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.717+0000 7fcc87400700 1 -- 172.21.15.63:0/1244629316 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc78013de0 con 0x7fcc901037a0 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.717+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9010dc70 con 0x7fcc901037a0 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.717+0000 7fcc87400700 1 -- 172.21.15.63:0/1244629316 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7801cd90 con 0x7fcc901037a0 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.717+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9010e230 con 0x7fcc901037a0 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.718+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc900fe550 con 0x7fcc901037a0 2024-03-23T12:21:47.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.719+0000 7fcc87400700 1 -- 172.21.15.63:0/1244629316 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fcc78013930 con 0x7fcc901037a0 2024-03-23T12:21:47.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.720+0000 7fcc87400700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcc7406c7d0 0x7fcc7406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:47.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.720+0000 7fcc87400700 1 -- 172.21.15.63:0/1244629316 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcc7808d300 con 0x7fcc901037a0 2024-03-23T12:21:47.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.720+0000 7fcc8f400700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcc7406c7d0 0x7fcc7406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:47.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.721+0000 7fcc8f400700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcc7406c7d0 0x7fcc7406ec80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcc80004600 tx=0x7fcc8000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:47.724+0000 7fcc87400700 1 -- 172.21.15.63:0/1244629316 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcc7805b010 con 0x7fcc901037a0 2024-03-23T12:21:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.024+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcc9010f0c0 con 0x7fcc901037a0 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.025+0000 7fcc87400700 1 -- 172.21.15.63:0/1244629316 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fcc7805aba0 con 0x7fcc901037a0 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:21:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:21:48.004 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:21:48.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcc7406c7d0 msgr2=0x7fcc7406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcc7406c7d0 0x7fcc7406ec80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcc80004600 tx=0x7fcc8000b040 comp rx=0 tx=0).stop 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 msgr2=0x7fcc90113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc90113fb0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fcc78008d50 tx=0x7fcc78009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 shutdown_connections 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcc7406c7d0 0x7fcc7406ec80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc90102e00 0x7fcc90113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 --2- 172.21.15.63:0/1244629316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc901037a0 0x7fcc90113fb0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.027+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 >> 172.21.15.63:0/1244629316 conn(0x7fcc900f89b0 msgr2=0x7fcc900f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.028+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 shutdown_connections 2024-03-23T12:21:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.028+0000 7fcc96200700 1 -- 172.21.15.63:0/1244629316 wait complete. 2024-03-23T12:21:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 -- 172.21.15.63:0/1275599892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c1037a0 msgr2=0x7fbe5c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1275599892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c1037a0 0x7fbe5c0fce10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbe48004940 tx=0x7fbe48009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 -- 172.21.15.63:0/1275599892 shutdown_connections 2024-03-23T12:21:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1275599892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c1037a0 0x7fbe5c0fce10 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1275599892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe5c102e00 0x7fbe5c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 -- 172.21.15.63:0/1275599892 >> 172.21.15.63:0/1275599892 conn(0x7fbe5c0f89b0 msgr2=0x7fbe5c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 -- 172.21.15.63:0/1275599892 shutdown_connections 2024-03-23T12:21:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.165+0000 7fbe63c00700 1 -- 172.21.15.63:0/1275599892 wait complete. 2024-03-23T12:21:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.166+0000 7fbe63c00700 1 Processor -- start 2024-03-23T12:21:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe63c00700 1 -- start start 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe63c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c102e00 0x7fbe5c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe63c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe5c1037a0 0x7fbe5c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe63c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5c070d90 con 0x7fbe5c102e00 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe63c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5c070ed0 con 0x7fbe5c1037a0 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe61200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c102e00 0x7fbe5c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe61200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c102e00 0x7fbe5c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45734/0 (socket says 172.21.15.63:45734) 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.167+0000 7fbe61200700 1 -- 172.21.15.63:0/1210623298 learned_addr learned my addr 172.21.15.63:0/1210623298 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe5be00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe5c1037a0 0x7fbe5c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe61200700 1 -- 172.21.15.63:0/1210623298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe5c1037a0 msgr2=0x7fbe5c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe61200700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe5c1037a0 0x7fbe5c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe61200700 1 -- 172.21.15.63:0/1210623298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe48005040 con 0x7fbe5c102e00 2024-03-23T12:21:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe61200700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c102e00 0x7fbe5c0702c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbe50002a50 tx=0x7fbe500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe58c00700 1 -- 172.21.15.63:0/1210623298 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe50007cc0 con 0x7fbe5c102e00 2024-03-23T12:21:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe58c00700 1 -- 172.21.15.63:0/1210623298 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe5000f040 con 0x7fbe5c102e00 2024-03-23T12:21:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5c119e00 con 0x7fbe5c102e00 2024-03-23T12:21:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe58c00700 1 -- 172.21.15.63:0/1210623298 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe5000d9c0 con 0x7fbe5c102e00 2024-03-23T12:21:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.168+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5c11a280 con 0x7fbe5c102e00 2024-03-23T12:21:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.169+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5c0fe550 con 0x7fbe5c102e00 2024-03-23T12:21:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.170+0000 7fbe58c00700 1 -- 172.21.15.63:0/1210623298 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbe5000db20 con 0x7fbe5c102e00 2024-03-23T12:21:48.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.171+0000 7fbe58c00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe4406c7d0 0x7fbe4406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.171+0000 7fbe58c00700 1 -- 172.21.15.63:0/1210623298 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbe5008b7e0 con 0x7fbe5c102e00 2024-03-23T12:21:48.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.172+0000 7fbe5be00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe4406c7d0 0x7fbe4406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.172+0000 7fbe5be00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe4406c7d0 0x7fbe4406ec80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c071c00 tx=0x7fbe48009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:48.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.176+0000 7fbe58c00700 1 -- 172.21.15.63:0/1210623298 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbe500559e0 con 0x7fbe5c102e00 2024-03-23T12:21:48.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.436+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbe5c0719b0 con 0x7fbe5c102e00 2024-03-23T12:21:48.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.437+0000 7fbe58c00700 1 -- 172.21.15.63:0/1210623298 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fbe50059000 con 0x7fbe5c102e00 2024-03-23T12:21:48.415 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:21:48.415 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:21:48.415 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:21:48.415 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:21:48.415 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:21:48.416 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:21:48.417 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:48.418 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.439+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe4406c7d0 msgr2=0x7fbe4406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.439+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe4406c7d0 0x7fbe4406ec80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c071c00 tx=0x7fbe48009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.439+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c102e00 msgr2=0x7fbe5c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.439+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c102e00 0x7fbe5c0702c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbe50002a50 tx=0x7fbe500046b0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.440+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 shutdown_connections 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.440+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbe4406c7d0 0x7fbe4406ec80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.440+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe5c102e00 0x7fbe5c0702c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.440+0000 7fbe63c00700 1 --2- 172.21.15.63:0/1210623298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe5c1037a0 0x7fbe5c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.440+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 >> 172.21.15.63:0/1210623298 conn(0x7fbe5c0f89b0 msgr2=0x7fbe5c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:48.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.440+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 shutdown_connections 2024-03-23T12:21:48.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.440+0000 7fbe63c00700 1 -- 172.21.15.63:0/1210623298 wait complete. 2024-03-23T12:21:48.420 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 -- 172.21.15.63:0/1570624707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 msgr2=0x7f4f28109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 --2- 172.21.15.63:0/1570624707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f28109110 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4f14004970 tx=0x7f4f14009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 -- 172.21.15.63:0/1570624707 shutdown_connections 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 --2- 172.21.15.63:0/1570624707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f28109110 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 --2- 172.21.15.63:0/1570624707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f28070360 0x7f4f28070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 -- 172.21.15.63:0/1570624707 >> 172.21.15.63:0/1570624707 conn(0x7f4f280f89b0 msgr2=0x7f4f280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 -- 172.21.15.63:0/1570624707 shutdown_connections 2024-03-23T12:21:48.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.578+0000 7f4f30000700 1 -- 172.21.15.63:0/1570624707 wait complete. 2024-03-23T12:21:48.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f30000700 1 Processor -- start 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f30000700 1 -- start start 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f30000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f28070360 0x7f4f28115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f30000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f281161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f30000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28116840 con 0x7f4f28070c70 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f30000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2810fc10 con 0x7f4f28070360 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f2cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f281161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f2cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f281161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45750/0 (socket says 172.21.15.63:45750) 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.579+0000 7f4f2cc00700 1 -- 172.21.15.63:0/958067246 learned_addr learned my addr 172.21.15.63:0/958067246 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f2d600700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f28070360 0x7f4f28115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f2cc00700 1 -- 172.21.15.63:0/958067246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f28070360 msgr2=0x7f4f28115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f2cc00700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f28070360 0x7f4f28115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f2cc00700 1 -- 172.21.15.63:0/958067246 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14005040 con 0x7f4f28070c70 2024-03-23T12:21:48.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f2cc00700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f281161f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4f14005010 tx=0x7f4f14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:48.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f25600700 1 -- 172.21.15.63:0/958067246 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f1400e040 con 0x7f4f28070c70 2024-03-23T12:21:48.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f25600700 1 -- 172.21.15.63:0/958067246 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f14004290 con 0x7f4f28070c70 2024-03-23T12:21:48.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2810fe90 con 0x7f4f28070c70 2024-03-23T12:21:48.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.580+0000 7f4f25600700 1 -- 172.21.15.63:0/958067246 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f14013d00 con 0x7f4f28070c70 2024-03-23T12:21:48.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.581+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f281103d0 con 0x7f4f28070c70 2024-03-23T12:21:48.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.582+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f28071aa0 con 0x7f4f28070c70 2024-03-23T12:21:48.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.583+0000 7f4f25600700 1 -- 172.21.15.63:0/958067246 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4f140134b0 con 0x7f4f28070c70 2024-03-23T12:21:48.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.583+0000 7f4f25600700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f1006c7d0 0x7f4f1006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.583+0000 7f4f25600700 1 -- 172.21.15.63:0/958067246 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f1408cc10 con 0x7f4f28070c70 2024-03-23T12:21:48.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.584+0000 7f4f2d600700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f1006c7d0 0x7f4f1006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.584+0000 7f4f2d600700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f1006c7d0 0x7f4f1006ec80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009b80 tx=0x7f4f1c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:48.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.588+0000 7f4f25600700 1 -- 172.21.15.63:0/958067246 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f1405a8a0 con 0x7f4f28070c70 2024-03-23T12:21:48.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:48 smithi063 ceph-mon[141124]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:48.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:48 smithi063 ceph-mon[141124]: from='client.35678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:48.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:48 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1244629316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:21:48.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:48 smithi111 ceph-mon[106899]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:48.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:48 smithi111 ceph-mon[106899]: from='client.35678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:48.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:48 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1244629316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:21:48.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.823+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f2810c040 con 0x7f4f1006c7d0 2024-03-23T12:21:48.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.824+0000 7f4f25600700 1 -- 172.21.15.63:0/958067246 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4f2810c040 con 0x7f4f1006c7d0 2024-03-23T12:21:48.802 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:21:48.802 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:21:48.803 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:21:48.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f1006c7d0 msgr2=0x7f4f1006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f1006c7d0 0x7f4f1006ec80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009b80 tx=0x7f4f1c00b040 comp rx=0 tx=0).stop 2024-03-23T12:21:48.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 msgr2=0x7f4f281161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f281161f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4f14005010 tx=0x7f4f14009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:48.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 shutdown_connections 2024-03-23T12:21:48.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f1006c7d0 0x7f4f1006ec80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f28070360 0x7f4f28115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 --2- 172.21.15.63:0/958067246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f28070c70 0x7f4f281161f0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 >> 172.21.15.63:0/958067246 conn(0x7f4f280f89b0 msgr2=0x7f4f280f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:48.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 shutdown_connections 2024-03-23T12:21:48.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.827+0000 7f4f30000700 1 -- 172.21.15.63:0/958067246 wait complete. 2024-03-23T12:21:48.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 -- 172.21.15.63:0/849339720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e80102e00 msgr2=0x7f3e801031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 --2- 172.21.15.63:0/849339720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e80102e00 0x7f3e801031d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3e74004940 tx=0x7f3e74009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:48.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 -- 172.21.15.63:0/849339720 shutdown_connections 2024-03-23T12:21:48.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 --2- 172.21.15.63:0/849339720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e801037a0 0x7f3e800fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 --2- 172.21.15.63:0/849339720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e80102e00 0x7f3e801031d0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 -- 172.21.15.63:0/849339720 >> 172.21.15.63:0/849339720 conn(0x7f3e800f89b0 msgr2=0x7f3e800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:48.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 -- 172.21.15.63:0/849339720 shutdown_connections 2024-03-23T12:21:48.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.965+0000 7f3e87400700 1 -- 172.21.15.63:0/849339720 wait complete. 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.966+0000 7f3e87400700 1 Processor -- start 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.966+0000 7f3e87400700 1 -- start start 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.966+0000 7f3e87400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e80102e00 0x7f3e800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.966+0000 7f3e87400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e801037a0 0x7f3e80070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.966+0000 7f3e87400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80070d90 con 0x7f3e801037a0 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.966+0000 7f3e87400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80070ed0 con 0x7f3e80102e00 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e7fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e801037a0 0x7f3e80070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e7fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e801037a0 0x7f3e80070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45756/0 (socket says 172.21.15.63:45756) 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e7fe00700 1 -- 172.21.15.63:0/3257353322 learned_addr learned my addr 172.21.15.63:0/3257353322 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e84a00700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e80102e00 0x7f3e800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e7fe00700 1 -- 172.21.15.63:0/3257353322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e80102e00 msgr2=0x7f3e800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e7fe00700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e80102e00 0x7f3e800702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e7fe00700 1 -- 172.21.15.63:0/3257353322 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74005040 con 0x7f3e801037a0 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.967+0000 7f3e7fe00700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e801037a0 0x7f3e80070800 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c002a50 tx=0x7f3e6c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.968+0000 7f3e7cc00700 1 -- 172.21.15.63:0/3257353322 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6c007cc0 con 0x7f3e801037a0 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.968+0000 7f3e7cc00700 1 -- 172.21.15.63:0/3257353322 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e6c00f040 con 0x7f3e801037a0 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.968+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e80119e20 con 0x7f3e801037a0 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.968+0000 7f3e7cc00700 1 -- 172.21.15.63:0/3257353322 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6c00d9a0 con 0x7f3e801037a0 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.968+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8011a2a0 con 0x7f3e801037a0 2024-03-23T12:21:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.969+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e800fe550 con 0x7f3e801037a0 2024-03-23T12:21:48.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.970+0000 7f3e7cc00700 1 -- 172.21.15.63:0/3257353322 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3e6c00dc10 con 0x7f3e801037a0 2024-03-23T12:21:48.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.971+0000 7f3e7cc00700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e6806c7d0 0x7f3e6806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:21:48.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.971+0000 7f3e7cc00700 1 -- 172.21.15.63:0/3257353322 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3e6c08b7e0 con 0x7f3e801037a0 2024-03-23T12:21:48.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.971+0000 7f3e84a00700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e6806c7d0 0x7f3e6806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:21:48.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.972+0000 7f3e84a00700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e6806c7d0 0x7f3e6806ec80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400b3b0 tx=0x7f3e74009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:21:48.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:48.975+0000 7f3e7cc00700 1 -- 172.21.15.63:0/3257353322 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3e6c055a60 con 0x7f3e801037a0 2024-03-23T12:21:49.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.273+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3e8005f5f0 con 0x7f3e801037a0 2024-03-23T12:21:49.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.273+0000 7f3e7cc00700 1 -- 172.21.15.63:0/3257353322 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f3e6c059080 con 0x7f3e801037a0 2024-03-23T12:21:49.251 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:21:49.251 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:21:49.251 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:21:49.251 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:21:49.251 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:21:49.252 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:21:49.253 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:21:49.253 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:21:49.253 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.275+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e6806c7d0 msgr2=0x7f3e6806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e6806c7d0 0x7f3e6806ec80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400b3b0 tx=0x7f3e74009e30 comp rx=0 tx=0).stop 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e801037a0 msgr2=0x7f3e80070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e801037a0 0x7f3e80070800 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c002a50 tx=0x7f3e6c004600 comp rx=0 tx=0).stop 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 shutdown_connections 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3e6806c7d0 0x7f3e6806ec80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e80102e00 0x7f3e800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 --2- 172.21.15.63:0/3257353322 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e801037a0 0x7f3e80070800 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 >> 172.21.15.63:0/3257353322 conn(0x7f3e800f89b0 msgr2=0x7f3e800f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:21:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 shutdown_connections 2024-03-23T12:21:49.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:21:49.276+0000 7f3e87400700 1 -- 172.21.15.63:0/3257353322 wait complete. 2024-03-23T12:21:49.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:49 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1210623298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:21:49.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:49 smithi063 ceph-mon[141124]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:49.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:49 smithi063 ceph-mon[141124]: pgmap v922: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:49.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:49 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1210623298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:21:49.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:49 smithi111 ceph-mon[106899]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:21:49.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:49 smithi111 ceph-mon[106899]: pgmap v922: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:50.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:50 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3257353322' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:21:50.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:50 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3257353322' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:21:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:51 smithi063 ceph-mon[141124]: pgmap v923: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:51 smithi111 ceph-mon[106899]: pgmap v923: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:53.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:53 smithi063 ceph-mon[141124]: pgmap v924: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:53.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:53 smithi111 ceph-mon[106899]: pgmap v924: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:56 smithi063 ceph-mon[141124]: pgmap v925: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:57.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:56 smithi111 ceph-mon[106899]: pgmap v925: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:57.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:21:59.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:21:58 smithi063 ceph-mon[141124]: pgmap v926: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:21:59.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:21:58 smithi111 ceph-mon[106899]: pgmap v926: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:01.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:00 smithi063 ceph-mon[141124]: pgmap v927: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:01.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:00 smithi111 ceph-mon[106899]: pgmap v927: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:03.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:02 smithi063 ceph-mon[141124]: pgmap v928: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:03.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:02 smithi111 ceph-mon[106899]: pgmap v928: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:05.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:04 smithi063 ceph-mon[141124]: pgmap v929: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:05.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:04 smithi111 ceph-mon[106899]: pgmap v929: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:07.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:06 smithi063 ceph-mon[141124]: pgmap v930: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:07.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:06 smithi111 ceph-mon[106899]: pgmap v930: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:09.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:08 smithi063 ceph-mon[141124]: pgmap v931: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:09.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:08 smithi111 ceph-mon[106899]: pgmap v931: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:11.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:10 smithi063 ceph-mon[141124]: pgmap v932: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:10 smithi111 ceph-mon[106899]: pgmap v932: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:12.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:12.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:13.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:12 smithi063 ceph-mon[141124]: pgmap v933: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:13.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:12 smithi111 ceph-mon[106899]: pgmap v933: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:15.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:14 smithi063 ceph-mon[141124]: pgmap v934: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:15.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:14 smithi111 ceph-mon[106899]: pgmap v934: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:17.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:16 smithi063 ceph-mon[141124]: pgmap v935: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:17.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:16 smithi111 ceph-mon[106899]: pgmap v935: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:19.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:18 smithi063 ceph-mon[141124]: pgmap v936: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:19.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:18 smithi111 ceph-mon[106899]: pgmap v936: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:19.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.428+0000 7fc860c00700 1 -- 172.21.15.63:0/1661706640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 msgr2=0x7fc85c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:19.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.428+0000 7fc860c00700 1 --2- 172.21.15.63:0/1661706640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c0fce30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc848004940 tx=0x7fc848009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:19.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.429+0000 7fc860c00700 1 -- 172.21.15.63:0/1661706640 shutdown_connections 2024-03-23T12:22:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.429+0000 7fc860c00700 1 --2- 172.21.15.63:0/1661706640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c0fce30 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.429+0000 7fc860c00700 1 --2- 172.21.15.63:0/1661706640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85c102e20 0x7fc85c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.429+0000 7fc860c00700 1 -- 172.21.15.63:0/1661706640 >> 172.21.15.63:0/1661706640 conn(0x7fc85c0f89d0 msgr2=0x7fc85c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.429+0000 7fc860c00700 1 -- 172.21.15.63:0/1661706640 shutdown_connections 2024-03-23T12:22:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.429+0000 7fc860c00700 1 -- 172.21.15.63:0/1661706640 wait complete. 2024-03-23T12:22:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.429+0000 7fc860c00700 1 Processor -- start 2024-03-23T12:22:19.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc860c00700 1 -- start start 2024-03-23T12:22:19.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc860c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85c102e20 0x7fc85c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:19.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc860c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:19.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc860c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c116860 con 0x7fc85c1037c0 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc860c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c10fc30 con 0x7fc85c102e20 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc859400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc859400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41378/0 (socket says 172.21.15.63:41378) 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.430+0000 7fc859400700 1 -- 172.21.15.63:0/3911832243 learned_addr learned my addr 172.21.15.63:0/3911832243 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc859e00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85c102e20 0x7fc85c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc859400700 1 -- 172.21.15.63:0/3911832243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85c102e20 msgr2=0x7fc85c115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc859400700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85c102e20 0x7fc85c115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc859400700 1 -- 172.21.15.63:0/3911832243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848005040 con 0x7fc85c1037c0 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc859400700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c116210 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc848009390 tx=0x7fc848009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc852000700 1 -- 172.21.15.63:0/3911832243 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84800e040 con 0x7fc85c1037c0 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc852000700 1 -- 172.21.15.63:0/3911832243 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc848013420 con 0x7fc85c1037c0 2024-03-23T12:22:19.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c10feb0 con 0x7fc85c1037c0 2024-03-23T12:22:19.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.431+0000 7fc852000700 1 -- 172.21.15.63:0/3911832243 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84801c630 con 0x7fc85c1037c0 2024-03-23T12:22:19.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.432+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c110470 con 0x7fc85c1037c0 2024-03-23T12:22:19.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.433+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85c0fe570 con 0x7fc85c1037c0 2024-03-23T12:22:19.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.434+0000 7fc852000700 1 -- 172.21.15.63:0/3911832243 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc848013a00 con 0x7fc85c1037c0 2024-03-23T12:22:19.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.434+0000 7fc852000700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84006c7d0 0x7fc84006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:19.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.435+0000 7fc852000700 1 -- 172.21.15.63:0/3911832243 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc84808cc20 con 0x7fc85c1037c0 2024-03-23T12:22:19.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.435+0000 7fc859e00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84006c7d0 0x7fc84006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:19.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.435+0000 7fc859e00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84006c7d0 0x7fc84006ec80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc84c004f40 tx=0x7fc84c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:19.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.439+0000 7fc852000700 1 -- 172.21.15.63:0/3911832243 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc84805a8b0 con 0x7fc85c1037c0 2024-03-23T12:22:19.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.677+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc85c1113c0 con 0x7fc84006c7d0 2024-03-23T12:22:19.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.679+0000 7fc852000700 1 -- 172.21.15.63:0/3911832243 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc85c1113c0 con 0x7fc84006c7d0 2024-03-23T12:22:19.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84006c7d0 msgr2=0x7fc84006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:19.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84006c7d0 0x7fc84006ec80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc84c004f40 tx=0x7fc84c00b040 comp rx=0 tx=0).stop 2024-03-23T12:22:19.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 msgr2=0x7fc85c116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:19.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c116210 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc848009390 tx=0x7fc848009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:19.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 shutdown_connections 2024-03-23T12:22:19.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84006c7d0 0x7fc84006ec80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85c102e20 0x7fc85c115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 --2- 172.21.15.63:0/3911832243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc85c1037c0 0x7fc85c116210 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 >> 172.21.15.63:0/3911832243 conn(0x7fc85c0f89d0 msgr2=0x7fc85c0f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:19.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 shutdown_connections 2024-03-23T12:22:19.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.681+0000 7fc860c00700 1 -- 172.21.15.63:0/3911832243 wait complete. 2024-03-23T12:22:19.674 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:22:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 -- 172.21.15.63:0/2711936483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c28102e00 msgr2=0x7f3c281031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 --2- 172.21.15.63:0/2711936483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c28102e00 0x7f3c281031d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3c18004940 tx=0x7f3c18009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 -- 172.21.15.63:0/2711936483 shutdown_connections 2024-03-23T12:22:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 --2- 172.21.15.63:0/2711936483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c281037a0 0x7f3c280fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 --2- 172.21.15.63:0/2711936483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c28102e00 0x7f3c281031d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 -- 172.21.15.63:0/2711936483 >> 172.21.15.63:0/2711936483 conn(0x7f3c280f89b0 msgr2=0x7f3c280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 -- 172.21.15.63:0/2711936483 shutdown_connections 2024-03-23T12:22:19.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.820+0000 7f3c2f200700 1 -- 172.21.15.63:0/2711936483 wait complete. 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.821+0000 7f3c2f200700 1 Processor -- start 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.821+0000 7f3c2f200700 1 -- start start 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.821+0000 7f3c2f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c28102e00 0x7f3c280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.821+0000 7f3c2f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c281037a0 0x7f3c28070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.821+0000 7f3c2f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28070d90 con 0x7f3c281037a0 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.821+0000 7f3c2f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28070ed0 con 0x7f3c28102e00 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c281037a0 0x7f3c28070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c281037a0 0x7f3c28070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41382/0 (socket says 172.21.15.63:41382) 2024-03-23T12:22:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27400700 1 -- 172.21.15.63:0/1450840859 learned_addr learned my addr 172.21.15.63:0/1450840859 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27e00700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c28102e00 0x7f3c280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27400700 1 -- 172.21.15.63:0/1450840859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c28102e00 msgr2=0x7f3c280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27400700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c28102e00 0x7f3c280702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27400700 1 -- 172.21.15.63:0/1450840859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c18005040 con 0x7f3c281037a0 2024-03-23T12:22:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27e00700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c28102e00 0x7f3c280702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:22:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.822+0000 7f3c27400700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c281037a0 0x7f3c28070800 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3c14002a50 tx=0x7f3c14004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.823+0000 7f3c1fe00700 1 -- 172.21.15.63:0/1450840859 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c14007cc0 con 0x7f3c281037a0 2024-03-23T12:22:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.823+0000 7f3c1fe00700 1 -- 172.21.15.63:0/1450840859 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c1400f040 con 0x7f3c281037a0 2024-03-23T12:22:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.823+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c28119e20 con 0x7f3c281037a0 2024-03-23T12:22:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.823+0000 7f3c1fe00700 1 -- 172.21.15.63:0/1450840859 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c1400d9a0 con 0x7f3c281037a0 2024-03-23T12:22:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.823+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2811a2a0 con 0x7f3c281037a0 2024-03-23T12:22:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.824+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c280fe550 con 0x7f3c281037a0 2024-03-23T12:22:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.825+0000 7f3c1fe00700 1 -- 172.21.15.63:0/1450840859 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3c1400dc10 con 0x7f3c281037a0 2024-03-23T12:22:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.826+0000 7f3c1fe00700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0c06c7d0 0x7f3c0c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:19.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.826+0000 7f3c1fe00700 1 -- 172.21.15.63:0/1450840859 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3c1408b860 con 0x7f3c281037a0 2024-03-23T12:22:19.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.826+0000 7f3c27e00700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0c06c7d0 0x7f3c0c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:19.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.827+0000 7f3c27e00700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0c06c7d0 0x7f3c0c06ec80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c18000c00 tx=0x7f3c18009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:19.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:19.830+0000 7f3c1fe00700 1 -- 172.21.15.63:0/1450840859 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3c14055a60 con 0x7f3c281037a0 2024-03-23T12:22:20.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.062+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c2804bed0 con 0x7f3c0c06c7d0 2024-03-23T12:22:20.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.063+0000 7f3c1fe00700 1 -- 172.21.15.63:0/1450840859 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3c2804bed0 con 0x7f3c0c06c7d0 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.065+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0c06c7d0 msgr2=0x7f3c0c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.065+0000 7f3c2f200700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0c06c7d0 0x7f3c0c06ec80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c18000c00 tx=0x7f3c18009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c281037a0 msgr2=0x7f3c28070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c281037a0 0x7f3c28070800 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3c14002a50 tx=0x7f3c14004600 comp rx=0 tx=0).stop 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 shutdown_connections 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0c06c7d0 0x7f3c0c06ec80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c28102e00 0x7f3c280702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 --2- 172.21.15.63:0/1450840859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c281037a0 0x7f3c28070800 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 >> 172.21.15.63:0/1450840859 conn(0x7f3c280f89b0 msgr2=0x7f3c280f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 shutdown_connections 2024-03-23T12:22:20.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.066+0000 7f3c2f200700 1 -- 172.21.15.63:0/1450840859 wait complete. 2024-03-23T12:22:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 -- 172.21.15.63:0/2405182612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c1037a0 msgr2=0x7f215c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 --2- 172.21.15.63:0/2405182612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c1037a0 0x7f215c0fce10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2148004940 tx=0x7f2148009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 -- 172.21.15.63:0/2405182612 shutdown_connections 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 --2- 172.21.15.63:0/2405182612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c1037a0 0x7f215c0fce10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 --2- 172.21.15.63:0/2405182612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f215c102e00 0x7f215c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 -- 172.21.15.63:0/2405182612 >> 172.21.15.63:0/2405182612 conn(0x7f215c0f89b0 msgr2=0x7f215c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 -- 172.21.15.63:0/2405182612 shutdown_connections 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.204+0000 7f2162e00700 1 -- 172.21.15.63:0/2405182612 wait complete. 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.206+0000 7f2162e00700 1 Processor -- start 2024-03-23T12:22:20.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.206+0000 7f2162e00700 1 -- start start 2024-03-23T12:22:20.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.206+0000 7f2162e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c102e00 0x7f215c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:20.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.206+0000 7f2162e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f215c1037a0 0x7f215c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:20.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.206+0000 7f2162e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c116810 con 0x7f215c102e00 2024-03-23T12:22:20.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.206+0000 7f2162e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c10fbe0 con 0x7f215c1037a0 2024-03-23T12:22:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.207+0000 7f215be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c102e00 0x7f215c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.207+0000 7f215be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c102e00 0x7f215c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41402/0 (socket says 172.21.15.63:41402) 2024-03-23T12:22:20.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.207+0000 7f215be00700 1 -- 172.21.15.63:0/2984582470 learned_addr learned my addr 172.21.15.63:0/2984582470 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.207+0000 7f215be00700 1 -- 172.21.15.63:0/2984582470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f215c1037a0 msgr2=0x7f215c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.207+0000 7f215be00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f215c1037a0 0x7f215c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.207+0000 7f215be00700 1 -- 172.21.15.63:0/2984582470 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148005040 con 0x7f215c102e00 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.207+0000 7f215be00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c102e00 0x7f215c115c80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f214c00eb40 tx=0x7f214c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.208+0000 7f2153e00700 1 -- 172.21.15.63:0/2984582470 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f214c007cc0 con 0x7f215c102e00 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.208+0000 7f2153e00700 1 -- 172.21.15.63:0/2984582470 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f214c00f040 con 0x7f215c102e00 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.208+0000 7f2153e00700 1 -- 172.21.15.63:0/2984582470 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f214c009570 con 0x7f215c102e00 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.208+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c10fe60 con 0x7f215c102e00 2024-03-23T12:22:20.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.208+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c110420 con 0x7f215c102e00 2024-03-23T12:22:20.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.209+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c0fe550 con 0x7f215c102e00 2024-03-23T12:22:20.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.210+0000 7f2153e00700 1 -- 172.21.15.63:0/2984582470 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f214c0096d0 con 0x7f215c102e00 2024-03-23T12:22:20.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.211+0000 7f2153e00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f214006c7d0 0x7f214006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:20.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.211+0000 7f2153e00700 1 -- 172.21.15.63:0/2984582470 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f214c08b6f0 con 0x7f215c102e00 2024-03-23T12:22:20.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.211+0000 7f215b400700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f214006c7d0 0x7f214006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:20.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.212+0000 7f215b400700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f214006c7d0 0x7f214006ec80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f214800b0d0 tx=0x7f2148009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:20.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.215+0000 7f2153e00700 1 -- 172.21.15.63:0/2984582470 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f214c059430 con 0x7f215c102e00 2024-03-23T12:22:20.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.440+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f215c109e10 con 0x7f214006c7d0 2024-03-23T12:22:20.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.450+0000 7f2153e00700 1 -- 172.21.15.63:0/2984582470 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f215c109e10 con 0x7f214006c7d0 2024-03-23T12:22:20.429 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:22:20.429 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (31m) 8m ago 44m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:22:20.429 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (28m) 8m ago 44m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:22:20.429 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (28m) 8m ago 43m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:22:20.429 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (31m) 8m ago 44m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:22:20.429 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (24m) 8m ago 38m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (24m) 8m ago 38m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (8m) 8m ago 38m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (24m) 8m ago 38m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (34m) 8m ago 46m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (32m) 8m ago 43m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (30m) 8m ago 46m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (29m) 8m ago 43m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (32m) 8m ago 44m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (32m) 8m ago 43m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (28m) 8m ago 42m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (27m) 8m ago 42m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (26m) 8m ago 41m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:22:20.430 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (26m) 8m ago 41m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:22:20.431 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (25m) 8m ago 40m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:22:20.431 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (25m) 8m ago 40m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:22:20.431 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (31m) 8m ago 44m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:22:20.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.452+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f214006c7d0 msgr2=0x7f214006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.452+0000 7f2162e00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f214006c7d0 0x7f214006ec80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f214800b0d0 tx=0x7f2148009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.452+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c102e00 msgr2=0x7f215c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.452+0000 7f2162e00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c102e00 0x7f215c115c80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f214c00eb40 tx=0x7f214c00b040 comp rx=0 tx=0).stop 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.453+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 shutdown_connections 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.453+0000 7f2162e00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f214006c7d0 0x7f214006ec80 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.453+0000 7f2162e00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f215c102e00 0x7f215c115c80 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.453+0000 7f2162e00700 1 --2- 172.21.15.63:0/2984582470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f215c1037a0 0x7f215c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.453+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 >> 172.21.15.63:0/2984582470 conn(0x7f215c0f89b0 msgr2=0x7f215c0f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.453+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 shutdown_connections 2024-03-23T12:22:20.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.453+0000 7f2162e00700 1 -- 172.21.15.63:0/2984582470 wait complete. 2024-03-23T12:22:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 -- 172.21.15.63:0/2788104201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec102e00 msgr2=0x7ff3ec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/2788104201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec102e00 0x7ff3ec1031d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0004940 tx=0x7ff3e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 -- 172.21.15.63:0/2788104201 shutdown_connections 2024-03-23T12:22:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/2788104201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3ec1037a0 0x7ff3ec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/2788104201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec102e00 0x7ff3ec1031d0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 -- 172.21.15.63:0/2788104201 >> 172.21.15.63:0/2788104201 conn(0x7ff3ec0f89b0 msgr2=0x7ff3ec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 -- 172.21.15.63:0/2788104201 shutdown_connections 2024-03-23T12:22:20.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.589+0000 7ff3f3a00700 1 -- 172.21.15.63:0/2788104201 wait complete. 2024-03-23T12:22:20.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.590+0000 7ff3f3a00700 1 Processor -- start 2024-03-23T12:22:20.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.590+0000 7ff3f3a00700 1 -- start start 2024-03-23T12:22:20.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.590+0000 7ff3f3a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3ec102e00 0x7ff3ec115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:20.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.590+0000 7ff3f3a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec1037a0 0x7ff3ec116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:20.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.590+0000 7ff3f3a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec116860 con 0x7ff3ec1037a0 2024-03-23T12:22:20.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.590+0000 7ff3f3a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec10fc30 con 0x7ff3ec102e00 2024-03-23T12:22:20.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec1037a0 0x7ff3ec116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:20.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec1037a0 0x7ff3ec116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41414/0 (socket says 172.21.15.63:41414) 2024-03-23T12:22:20.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3ebe00700 1 -- 172.21.15.63:0/1189442802 learned_addr learned my addr 172.21.15.63:0/1189442802 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3f1000700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3ec102e00 0x7ff3ec115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3ebe00700 1 -- 172.21.15.63:0/1189442802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3ec102e00 msgr2=0x7ff3ec115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3ebe00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3ec102e00 0x7ff3ec115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3ebe00700 1 -- 172.21.15.63:0/1189442802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e0005040 con 0x7ff3ec1037a0 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3f1000700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3ec102e00 0x7ff3ec115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.591+0000 7ff3ebe00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec1037a0 0x7ff3ec116210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8002a50 tx=0x7ff3d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.592+0000 7ff3e8c00700 1 -- 172.21.15.63:0/1189442802 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d8007cc0 con 0x7ff3ec1037a0 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.592+0000 7ff3e8c00700 1 -- 172.21.15.63:0/1189442802 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3d800f040 con 0x7ff3ec1037a0 2024-03-23T12:22:20.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.592+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec10ff10 con 0x7ff3ec1037a0 2024-03-23T12:22:20.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.592+0000 7ff3e8c00700 1 -- 172.21.15.63:0/1189442802 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d800d9a0 con 0x7ff3ec1037a0 2024-03-23T12:22:20.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.592+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec1104d0 con 0x7ff3ec1037a0 2024-03-23T12:22:20.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.593+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec0fe550 con 0x7ff3ec1037a0 2024-03-23T12:22:20.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.594+0000 7ff3e8c00700 1 -- 172.21.15.63:0/1189442802 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff3d800dc10 con 0x7ff3ec1037a0 2024-03-23T12:22:20.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.595+0000 7ff3e8c00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3d406c7d0 0x7ff3d406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:20.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.595+0000 7ff3e8c00700 1 -- 172.21.15.63:0/1189442802 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff3d808b870 con 0x7ff3ec1037a0 2024-03-23T12:22:20.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.595+0000 7ff3f1000700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3d406c7d0 0x7ff3d406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:20.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.595+0000 7ff3f1000700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3d406c7d0 0x7ff3d406ec80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000b840 tx=0x7ff3e0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.599+0000 7ff3e8c00700 1 -- 172.21.15.63:0/1189442802 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff3d8059500 con 0x7ff3ec1037a0 2024-03-23T12:22:20.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.899+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff3ec04bed0 con 0x7ff3ec1037a0 2024-03-23T12:22:20.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.900+0000 7ff3e8c00700 1 -- 172.21.15.63:0/1189442802 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7ff3d8059090 con 0x7ff3ec1037a0 2024-03-23T12:22:20.879 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:22:20.879 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:22:20.879 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:22:20.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:20.879 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:22:20.880 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:22:20.881 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:22:20.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3d406c7d0 msgr2=0x7ff3d406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3d406c7d0 0x7ff3d406ec80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000b840 tx=0x7ff3e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:20.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec1037a0 msgr2=0x7ff3ec116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec1037a0 0x7ff3ec116210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8002a50 tx=0x7ff3d8004600 comp rx=0 tx=0).stop 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 shutdown_connections 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff3d406c7d0 0x7ff3d406ec80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3ec102e00 0x7ff3ec115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 --2- 172.21.15.63:0/1189442802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3ec1037a0 0x7ff3ec116210 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.902+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 >> 172.21.15.63:0/1189442802 conn(0x7ff3ec0f89b0 msgr2=0x7ff3ec0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.903+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 shutdown_connections 2024-03-23T12:22:20.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:20.903+0000 7ff3f3a00700 1 -- 172.21.15.63:0/1189442802 wait complete. 2024-03-23T12:22:21.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 -- 172.21.15.63:0/3686326495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070c70 msgr2=0x7fab08109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 --2- 172.21.15.63:0/3686326495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070c70 0x7fab08109110 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7faaf4004970 tx=0x7faaf4009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:21.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 -- 172.21.15.63:0/3686326495 shutdown_connections 2024-03-23T12:22:21.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 --2- 172.21.15.63:0/3686326495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070c70 0x7fab08109110 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 --2- 172.21.15.63:0/3686326495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab08070360 0x7fab08070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 -- 172.21.15.63:0/3686326495 >> 172.21.15.63:0/3686326495 conn(0x7fab080f89b0 msgr2=0x7fab080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:21.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 -- 172.21.15.63:0/3686326495 shutdown_connections 2024-03-23T12:22:21.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.041+0000 7fab0d800700 1 -- 172.21.15.63:0/3686326495 wait complete. 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab0d800700 1 Processor -- start 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab0d800700 1 -- start start 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab0d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070360 0x7fab08115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab0d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab08070c70 0x7fab081161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab0d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab08116830 con 0x7fab08070360 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab0d800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0810fc00 con 0x7fab08070c70 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab06a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070360 0x7fab08115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab06a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070360 0x7fab08115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41430/0 (socket says 172.21.15.63:41430) 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.042+0000 7fab06a00700 1 -- 172.21.15.63:0/4087302790 learned_addr learned my addr 172.21.15.63:0/4087302790 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7fab06a00700 1 -- 172.21.15.63:0/4087302790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab08070c70 msgr2=0x7fab081161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7fab06a00700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab08070c70 0x7fab081161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7fab06a00700 1 -- 172.21.15.63:0/4087302790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf4005040 con 0x7fab08070360 2024-03-23T12:22:21.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7fab06a00700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070360 0x7fab08115ca0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7faaf800eb40 tx=0x7faaf800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:21.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7faafea00700 1 -- 172.21.15.63:0/4087302790 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaf8007cc0 con 0x7fab08070360 2024-03-23T12:22:21.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7faafea00700 1 -- 172.21.15.63:0/4087302790 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faaf800f040 con 0x7fab08070360 2024-03-23T12:22:21.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7faafea00700 1 -- 172.21.15.63:0/4087302790 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaf8009570 con 0x7fab08070360 2024-03-23T12:22:21.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.043+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0810fe80 con 0x7fab08070360 2024-03-23T12:22:21.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.044+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab08110370 con 0x7fab08070360 2024-03-23T12:22:21.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.045+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab08103bd0 con 0x7fab08070360 2024-03-23T12:22:21.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.046+0000 7faafea00700 1 -- 172.21.15.63:0/4087302790 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7faaf80096d0 con 0x7fab08070360 2024-03-23T12:22:21.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.047+0000 7faafea00700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faaec06c7d0 0x7faaec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.047+0000 7faafea00700 1 -- 172.21.15.63:0/4087302790 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faaf808b7e0 con 0x7fab08070360 2024-03-23T12:22:21.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.047+0000 7fab06000700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faaec06c7d0 0x7faaec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.048+0000 7fab06000700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faaec06c7d0 0x7faaec06ec80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faaf400b370 tx=0x7faaf4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:21.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.052+0000 7faafea00700 1 -- 172.21.15.63:0/4087302790 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faaf80594a0 con 0x7fab08070360 2024-03-23T12:22:21.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:20 smithi063 ceph-mon[141124]: pgmap v937: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:21.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:20 smithi063 ceph-mon[141124]: from='client.35698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:21.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:20 smithi063 ceph-mon[141124]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:21.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:20 smithi111 ceph-mon[106899]: pgmap v937: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:21.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:20 smithi111 ceph-mon[106899]: from='client.35698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:21.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:20 smithi111 ceph-mon[106899]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:21.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.314+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fab08111320 con 0x7fab08070360 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.315+0000 7faafea00700 1 -- 172.21.15.63:0/4087302790 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7faaf8059030 con 0x7fab08070360 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:22:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:22:21.295 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:21.296 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:21.297 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:21.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faaec06c7d0 msgr2=0x7faaec06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faaec06c7d0 0x7faaec06ec80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faaf400b370 tx=0x7faaf4009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:21.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070360 msgr2=0x7fab08115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070360 0x7fab08115ca0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7faaf800eb40 tx=0x7faaf800b040 comp rx=0 tx=0).stop 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 shutdown_connections 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7faaec06c7d0 0x7faaec06ec80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab08070360 0x7fab08115ca0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 --2- 172.21.15.63:0/4087302790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab08070c70 0x7fab081161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 >> 172.21.15.63:0/4087302790 conn(0x7fab080f89b0 msgr2=0x7fab081015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 shutdown_connections 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.317+0000 7fab0d800700 1 -- 172.21.15.63:0/4087302790 wait complete. 2024-03-23T12:22:21.298 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 -- 172.21.15.63:0/724357316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 msgr2=0x7f9eb00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 --2- 172.21.15.63:0/724357316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb00fe260 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0004940 tx=0x7f9ea0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 -- 172.21.15.63:0/724357316 shutdown_connections 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 --2- 172.21.15.63:0/724357316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eb00fe830 0x7f9eb0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 --2- 172.21.15.63:0/724357316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb00fe260 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 -- 172.21.15.63:0/724357316 >> 172.21.15.63:0/724357316 conn(0x7f9eb0072bc0 msgr2=0x7f9eb0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 -- 172.21.15.63:0/724357316 shutdown_connections 2024-03-23T12:22:21.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.455+0000 7f9eb6200700 1 -- 172.21.15.63:0/724357316 wait complete. 2024-03-23T12:22:21.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.456+0000 7f9eb6200700 1 Processor -- start 2024-03-23T12:22:21.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.456+0000 7f9eb6200700 1 -- start start 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eb6200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eb6200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eb00fe830 0x7f9eb01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eb6200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb0116810 con 0x7f9eb00fde90 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eb6200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb010fbe0 con 0x7f9eb00fe830 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb0115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39554/0 (socket says 172.21.15.63:39554) 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaf400700 1 -- 172.21.15.63:0/2829611029 learned_addr learned my addr 172.21.15.63:0/2829611029 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaea00700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eb00fe830 0x7f9eb01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaf400700 1 -- 172.21.15.63:0/2829611029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eb00fe830 msgr2=0x7f9eb01161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaf400700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eb00fe830 0x7f9eb01161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaf400700 1 -- 172.21.15.63:0/2829611029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea0005040 con 0x7f9eb00fde90 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.457+0000 7f9eaea00700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eb00fe830 0x7f9eb01161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.458+0000 7f9eaf400700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb0115c80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000c00 tx=0x7f9ea0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.458+0000 7f9ea7400700 1 -- 172.21.15.63:0/2829611029 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea000e040 con 0x7f9eb00fde90 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.458+0000 7f9ea7400700 1 -- 172.21.15.63:0/2829611029 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ea0013420 con 0x7f9eb00fde90 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.458+0000 7f9ea7400700 1 -- 172.21.15.63:0/2829611029 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea001c630 con 0x7f9eb00fde90 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.458+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb010fe60 con 0x7f9eb00fde90 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.458+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb01103a0 con 0x7f9eb00fde90 2024-03-23T12:22:21.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.459+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb0106a20 con 0x7f9eb00fde90 2024-03-23T12:22:21.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.460+0000 7f9ea7400700 1 -- 172.21.15.63:0/2829611029 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9ea0013970 con 0x7f9eb00fde90 2024-03-23T12:22:21.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.461+0000 7f9ea7400700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9406c7d0 0x7f9e9406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.461+0000 7f9ea7400700 1 -- 172.21.15.63:0/2829611029 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9ea008cc70 con 0x7f9eb00fde90 2024-03-23T12:22:21.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.461+0000 7f9eaea00700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9406c7d0 0x7f9e9406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.462+0000 7f9eaea00700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9406c7d0 0x7f9e9406ec80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007fd0 tx=0x7f9e9c009f60 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:21.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.465+0000 7f9ea7400700 1 -- 172.21.15.63:0/2829611029 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9ea0093080 con 0x7f9eb00fde90 2024-03-23T12:22:21.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.696+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9eb0109dc0 con 0x7f9e9406c7d0 2024-03-23T12:22:21.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.697+0000 7f9ea7400700 1 -- 172.21.15.63:0/2829611029 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9eb0109dc0 con 0x7f9e9406c7d0 2024-03-23T12:22:21.676 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:22:21.676 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:22:21.676 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:22:21.677 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9406c7d0 msgr2=0x7f9e9406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9406c7d0 0x7f9e9406ec80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007fd0 tx=0x7f9e9c009f60 comp rx=0 tx=0).stop 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 msgr2=0x7f9eb0115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb0115c80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000c00 tx=0x7f9ea0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 shutdown_connections 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e9406c7d0 0x7f9e9406ec80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb00fde90 0x7f9eb0115c80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 --2- 172.21.15.63:0/2829611029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eb00fe830 0x7f9eb01161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 >> 172.21.15.63:0/2829611029 conn(0x7f9eb0072bc0 msgr2=0x7f9eb00fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 shutdown_connections 2024-03-23T12:22:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.700+0000 7f9eb6200700 1 -- 172.21.15.63:0/2829611029 wait complete. 2024-03-23T12:22:21.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 -- 172.21.15.63:0/2982283193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b01035e0 msgr2=0x7f08b00fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 --2- 172.21.15.63:0/2982283193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b01035e0 0x7f08b00fcc50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0898004940 tx=0x7f0898009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:21.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 -- 172.21.15.63:0/2982283193 shutdown_connections 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 --2- 172.21.15.63:0/2982283193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b01035e0 0x7f08b00fcc50 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 --2- 172.21.15.63:0/2982283193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b0102c40 0x7f08b0103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 -- 172.21.15.63:0/2982283193 >> 172.21.15.63:0/2982283193 conn(0x7f08b00f8830 msgr2=0x7f08b00fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 -- 172.21.15.63:0/2982283193 shutdown_connections 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.838+0000 7f08b5c00700 1 -- 172.21.15.63:0/2982283193 wait complete. 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.839+0000 7f08b5c00700 1 Processor -- start 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.839+0000 7f08b5c00700 1 -- start start 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08b5c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b0102c40 0x7f08b0115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08b5c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b01035e0 0x7f08b0116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08b5c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b0116680 con 0x7f08b0102c40 2024-03-23T12:22:21.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08b5c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b010fa70 con 0x7f08b01035e0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08afe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b0102c40 0x7f08b0115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08afe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b0102c40 0x7f08b0115af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39568/0 (socket says 172.21.15.63:39568) 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08afe00700 1 -- 172.21.15.63:0/3461612481 learned_addr learned my addr 172.21.15.63:0/3461612481 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08a7e00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b01035e0 0x7f08b0116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08a7e00700 1 -- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b0102c40 msgr2=0x7f08b0115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08a7e00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b0102c40 0x7f08b0115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08a7e00700 1 -- 172.21.15.63:0/3461612481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898005040 con 0x7f08b01035e0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.840+0000 7f08afe00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b0102c40 0x7f08b0115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.841+0000 7f08a7e00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b01035e0 0x7f08b0116030 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08b0067020 tx=0x7f0898009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.841+0000 7f08acc00700 1 -- 172.21.15.63:0/3461612481 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f089800e040 con 0x7f08b01035e0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.841+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b010fcf0 con 0x7f08b01035e0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.841+0000 7f08acc00700 1 -- 172.21.15.63:0/3461612481 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0898013420 con 0x7f08b01035e0 2024-03-23T12:22:21.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.841+0000 7f08acc00700 1 -- 172.21.15.63:0/3461612481 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f089801c630 con 0x7f08b01035e0 2024-03-23T12:22:21.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.841+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b01102b0 con 0x7f08b01035e0 2024-03-23T12:22:21.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.842+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b00fe390 con 0x7f08b01035e0 2024-03-23T12:22:21.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.843+0000 7f08acc00700 1 -- 172.21.15.63:0/3461612481 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0898013970 con 0x7f08b01035e0 2024-03-23T12:22:21.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.844+0000 7f08acc00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f089406c7d0 0x7f089406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:21.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.844+0000 7f08acc00700 1 -- 172.21.15.63:0/3461612481 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f089808cb90 con 0x7f08b01035e0 2024-03-23T12:22:21.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.844+0000 7f08afe00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f089406c7d0 0x7f089406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:21.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.845+0000 7f08afe00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f089406c7d0 0x7f089406ec80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f08a00041d0 tx=0x7f08a000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:21.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:21.849+0000 7f08acc00700 1 -- 172.21.15.63:0/3461612481 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f089805a820 con 0x7f08b01035e0 2024-03-23T12:22:22.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:21 smithi063 ceph-mon[141124]: from='client.35706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:22.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:21 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1189442802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:22:22.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:21 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4087302790' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:22:22.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.149+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f08b005f5f0 con 0x7f08b01035e0 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.150+0000 7f08acc00700 1 -- 172.21.15.63:0/3461612481 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f089805a3b0 con 0x7f08b01035e0 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:22:22.129 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:22:22.130 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:22:22.131 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:22:22.131 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:22:22.131 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:22:22.131 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:22:22.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f089406c7d0 msgr2=0x7f089406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f089406c7d0 0x7f089406ec80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f08a00041d0 tx=0x7f08a000b040 comp rx=0 tx=0).stop 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b01035e0 msgr2=0x7f08b0116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b01035e0 0x7f08b0116030 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08b0067020 tx=0x7f0898009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 shutdown_connections 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f089406c7d0 0x7f089406ec80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b0102c40 0x7f08b0115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 --2- 172.21.15.63:0/3461612481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b01035e0 0x7f08b0116030 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.152+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 >> 172.21.15.63:0/3461612481 conn(0x7f08b00f8830 msgr2=0x7f08b00f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.153+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 shutdown_connections 2024-03-23T12:22:22.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:22.153+0000 7f08b5c00700 1 -- 172.21.15.63:0/3461612481 wait complete. 2024-03-23T12:22:22.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:21 smithi111 ceph-mon[106899]: from='client.35706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:22.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:21 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1189442802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:22:22.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:21 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4087302790' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:22:23.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:22 smithi063 ceph-mon[141124]: pgmap v938: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:23.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:22 smithi063 ceph-mon[141124]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:23.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:22 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3461612481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:22:23.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:22 smithi111 ceph-mon[106899]: pgmap v938: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:23.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:22 smithi111 ceph-mon[106899]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:23.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:22 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3461612481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:22:25.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:24 smithi063 ceph-mon[141124]: pgmap v939: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:25.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:24 smithi111 ceph-mon[106899]: pgmap v939: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:26 smithi063 ceph-mon[141124]: pgmap v940: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:27.128 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:26 smithi111 ceph-mon[106899]: pgmap v940: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:29.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:28 smithi063 ceph-mon[141124]: pgmap v941: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:29.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:28 smithi111 ceph-mon[106899]: pgmap v941: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:31.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:30 smithi063 ceph-mon[141124]: pgmap v942: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:31.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:30 smithi111 ceph-mon[106899]: pgmap v942: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:33.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:32 smithi063 ceph-mon[141124]: pgmap v943: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:33.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:32 smithi111 ceph-mon[106899]: pgmap v943: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:35.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:34 smithi063 ceph-mon[141124]: pgmap v944: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:35.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:34 smithi111 ceph-mon[106899]: pgmap v944: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:37.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:36 smithi063 ceph-mon[141124]: pgmap v945: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:37.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:36 smithi111 ceph-mon[106899]: pgmap v945: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:39.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:38 smithi063 ceph-mon[141124]: pgmap v946: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:39.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:38 smithi111 ceph-mon[106899]: pgmap v946: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:41.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:40 smithi063 ceph-mon[141124]: pgmap v947: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:41.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:40 smithi111 ceph-mon[106899]: pgmap v947: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:42.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:42.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:43.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:42 smithi063 ceph-mon[141124]: pgmap v948: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:43.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:42 smithi111 ceph-mon[106899]: pgmap v948: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:45.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:44 smithi063 ceph-mon[141124]: pgmap v949: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:45.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:44 smithi111 ceph-mon[106899]: pgmap v949: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:47.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:46 smithi063 ceph-mon[141124]: pgmap v950: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:47.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:46 smithi111 ceph-mon[106899]: pgmap v950: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:49.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:48 smithi063 ceph-mon[141124]: pgmap v951: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:49.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:48 smithi111 ceph-mon[106899]: pgmap v951: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:51.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:50 smithi063 ceph-mon[141124]: pgmap v952: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:51.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:50 smithi111 ceph-mon[106899]: pgmap v952: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 -- 172.21.15.63:0/3628456431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 msgr2=0x7ff7240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 --2- 172.21.15.63:0/3628456431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7240fce10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff710004940 tx=0x7ff710009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 -- 172.21.15.63:0/3628456431 shutdown_connections 2024-03-23T12:22:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 --2- 172.21.15.63:0/3628456431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7240fce10 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 --2- 172.21.15.63:0/3628456431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff724102e00 0x7ff7241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 -- 172.21.15.63:0/3628456431 >> 172.21.15.63:0/3628456431 conn(0x7ff7240f89b0 msgr2=0x7ff7240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 -- 172.21.15.63:0/3628456431 shutdown_connections 2024-03-23T12:22:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.315+0000 7ff729600700 1 -- 172.21.15.63:0/3628456431 wait complete. 2024-03-23T12:22:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.316+0000 7ff729600700 1 Processor -- start 2024-03-23T12:22:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.316+0000 7ff729600700 1 -- start start 2024-03-23T12:22:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.316+0000 7ff729600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff724102e00 0x7ff724115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.316+0000 7ff729600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7241161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.316+0000 7ff729600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff724116800 con 0x7ff7241037a0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.316+0000 7ff729600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72410fbd0 con 0x7ff724102e00 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7241161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7241161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46184/0 (socket says 172.21.15.63:46184) 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722000700 1 -- 172.21.15.63:0/2898199666 learned_addr learned my addr 172.21.15.63:0/2898199666 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722a00700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff724102e00 0x7ff724115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722000700 1 -- 172.21.15.63:0/2898199666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff724102e00 msgr2=0x7ff724115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722000700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff724102e00 0x7ff724115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722000700 1 -- 172.21.15.63:0/2898199666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff710005040 con 0x7ff7241037a0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff722000700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7241161b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff71000b0d0 tx=0x7ff710009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.317+0000 7ff71aa00700 1 -- 172.21.15.63:0/2898199666 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff710021070 con 0x7ff7241037a0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.318+0000 7ff71aa00700 1 -- 172.21.15.63:0/2898199666 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff710013e00 con 0x7ff7241037a0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.318+0000 7ff71aa00700 1 -- 172.21.15.63:0/2898199666 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71000e070 con 0x7ff7241037a0 2024-03-23T12:22:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.318+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72410fe50 con 0x7ff7241037a0 2024-03-23T12:22:52.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.318+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7241103b0 con 0x7ff7241037a0 2024-03-23T12:22:52.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.319+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7240fe550 con 0x7ff7241037a0 2024-03-23T12:22:52.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.320+0000 7ff71aa00700 1 -- 172.21.15.63:0/2898199666 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff71001c400 con 0x7ff7241037a0 2024-03-23T12:22:52.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.320+0000 7ff71aa00700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70806c7c0 0x7ff70806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:52.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.321+0000 7ff71aa00700 1 -- 172.21.15.63:0/2898199666 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff71008cbe0 con 0x7ff7241037a0 2024-03-23T12:22:52.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.321+0000 7ff722a00700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70806c7c0 0x7ff70806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:52.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.321+0000 7ff722a00700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70806c7c0 0x7ff70806ec70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff714009fd0 tx=0x7ff71400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.325+0000 7ff71aa00700 1 -- 172.21.15.63:0/2898199666 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff71005a870 con 0x7ff7241037a0 2024-03-23T12:22:52.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.560+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff724109e30 con 0x7ff70806c7c0 2024-03-23T12:22:52.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.561+0000 7ff71aa00700 1 -- 172.21.15.63:0/2898199666 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff724109e30 con 0x7ff70806c7c0 2024-03-23T12:22:52.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70806c7c0 msgr2=0x7ff70806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70806c7c0 0x7ff70806ec70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff714009fd0 tx=0x7ff71400b040 comp rx=0 tx=0).stop 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 msgr2=0x7ff7241161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7241161b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff71000b0d0 tx=0x7ff710009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 shutdown_connections 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff70806c7c0 0x7ff70806ec70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff724102e00 0x7ff724115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 --2- 172.21.15.63:0/2898199666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7241037a0 0x7ff7241161b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.563+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 >> 172.21.15.63:0/2898199666 conn(0x7ff7240f89b0 msgr2=0x7ff7240f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.564+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 shutdown_connections 2024-03-23T12:22:52.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.564+0000 7ff729600700 1 -- 172.21.15.63:0/2898199666 wait complete. 2024-03-23T12:22:52.556 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 -- 172.21.15.63:0/3059443885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 msgr2=0x7fe4d40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 --2- 172.21.15.63:0/3059443885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d40fce10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0004940 tx=0x7fe4c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 -- 172.21.15.63:0/3059443885 shutdown_connections 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 --2- 172.21.15.63:0/3059443885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d40fce10 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 --2- 172.21.15.63:0/3059443885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d4102e00 0x7fe4d41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 -- 172.21.15.63:0/3059443885 >> 172.21.15.63:0/3059443885 conn(0x7fe4d40f89b0 msgr2=0x7fe4d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 -- 172.21.15.63:0/3059443885 shutdown_connections 2024-03-23T12:22:52.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.702+0000 7fe4daa00700 1 -- 172.21.15.63:0/3059443885 wait complete. 2024-03-23T12:22:52.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.703+0000 7fe4daa00700 1 Processor -- start 2024-03-23T12:22:52.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.703+0000 7fe4daa00700 1 -- start start 2024-03-23T12:22:52.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.703+0000 7fe4daa00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d4102e00 0x7fe4d4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:52.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.703+0000 7fe4daa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:52.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4daa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d4114600 con 0x7fe4d41037a0 2024-03-23T12:22:52.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4daa00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d410d9f0 con 0x7fe4d4102e00 2024-03-23T12:22:52.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d4113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46192/0 (socket says 172.21.15.63:46192) 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3400700 1 -- 172.21.15.63:0/801793162 learned_addr learned my addr 172.21.15.63:0/801793162 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3e00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d4102e00 0x7fe4d4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3400700 1 -- 172.21.15.63:0/801793162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d4102e00 msgr2=0x7fe4d4113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3400700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d4102e00 0x7fe4d4113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3400700 1 -- 172.21.15.63:0/801793162 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c0005040 con 0x7fe4d41037a0 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.704+0000 7fe4d3e00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d4102e00 0x7fe4d4113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:22:52.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.705+0000 7fe4d3400700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d4113fb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0008d50 tx=0x7fe4c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.705+0000 7fe4cbe00700 1 -- 172.21.15.63:0/801793162 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4c000e030 con 0x7fe4d41037a0 2024-03-23T12:22:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.705+0000 7fe4cbe00700 1 -- 172.21.15.63:0/801793162 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4c0013de0 con 0x7fe4d41037a0 2024-03-23T12:22:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.705+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d410dc70 con 0x7fe4d41037a0 2024-03-23T12:22:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.705+0000 7fe4cbe00700 1 -- 172.21.15.63:0/801793162 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4c001cd90 con 0x7fe4d41037a0 2024-03-23T12:22:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.705+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d410e230 con 0x7fe4d41037a0 2024-03-23T12:22:52.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.706+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d40fe550 con 0x7fe4d41037a0 2024-03-23T12:22:52.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.707+0000 7fe4cbe00700 1 -- 172.21.15.63:0/801793162 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe4c0013930 con 0x7fe4d41037a0 2024-03-23T12:22:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.708+0000 7fe4cbe00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe4b806c7d0 0x7fe4b806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.709+0000 7fe4cbe00700 1 -- 172.21.15.63:0/801793162 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe4c008d300 con 0x7fe4d41037a0 2024-03-23T12:22:52.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.709+0000 7fe4d3e00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe4b806c7d0 0x7fe4b806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:52.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.709+0000 7fe4d3e00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe4b806c7d0 0x7fe4b806ec80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4004600 tx=0x7fe4c400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:52.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.713+0000 7fe4cbe00700 1 -- 172.21.15.63:0/801793162 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe4c005b010 con 0x7fe4d41037a0 2024-03-23T12:22:52.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.947+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4d4109da0 con 0x7fe4b806c7d0 2024-03-23T12:22:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.948+0000 7fe4cbe00700 1 -- 172.21.15.63:0/801793162 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe4d4109da0 con 0x7fe4b806c7d0 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe4b806c7d0 msgr2=0x7fe4b806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe4b806c7d0 0x7fe4b806ec80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4004600 tx=0x7fe4c400b040 comp rx=0 tx=0).stop 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 msgr2=0x7fe4d4113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d4113fb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0008d50 tx=0x7fe4c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 shutdown_connections 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe4b806c7d0 0x7fe4b806ec80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d4102e00 0x7fe4d4113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 --2- 172.21.15.63:0/801793162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4d41037a0 0x7fe4d4113fb0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 >> 172.21.15.63:0/801793162 conn(0x7fe4d40f89b0 msgr2=0x7fe4d40f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 shutdown_connections 2024-03-23T12:22:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:52.951+0000 7fe4daa00700 1 -- 172.21.15.63:0/801793162 wait complete. 2024-03-23T12:22:53.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 -- 172.21.15.63:0/303809425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc1037c0 msgr2=0x7f0bcc0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 --2- 172.21.15.63:0/303809425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc1037c0 0x7f0bcc0fce30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8004940 tx=0x7f0bb8009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:53.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 -- 172.21.15.63:0/303809425 shutdown_connections 2024-03-23T12:22:53.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 --2- 172.21.15.63:0/303809425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc1037c0 0x7f0bcc0fce30 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 --2- 172.21.15.63:0/303809425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc102e20 0x7f0bcc1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 -- 172.21.15.63:0/303809425 >> 172.21.15.63:0/303809425 conn(0x7f0bcc0f89d0 msgr2=0x7f0bcc0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:53.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 -- 172.21.15.63:0/303809425 shutdown_connections 2024-03-23T12:22:53.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.089+0000 7f0bd1400700 1 -- 172.21.15.63:0/303809425 wait complete. 2024-03-23T12:22:53.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.090+0000 7f0bd1400700 1 Processor -- start 2024-03-23T12:22:53.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.090+0000 7f0bd1400700 1 -- start start 2024-03-23T12:22:53.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bd1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc102e20 0x7f0bcc115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bd1400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc1037c0 0x7f0bcc116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bd1400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc116860 con 0x7f0bcc102e20 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bd1400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc10fc30 con 0x7f0bcc1037c0 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bca800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc102e20 0x7f0bcc115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bca800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc102e20 0x7f0bcc115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46212/0 (socket says 172.21.15.63:46212) 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bca800700 1 -- 172.21.15.63:0/3618480259 learned_addr learned my addr 172.21.15.63:0/3618480259 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bca800700 1 -- 172.21.15.63:0/3618480259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc1037c0 msgr2=0x7f0bcc116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bca800700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc1037c0 0x7f0bcc116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.091+0000 7f0bca800700 1 -- 172.21.15.63:0/3618480259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8005040 con 0x7f0bcc102e20 2024-03-23T12:22:53.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.092+0000 7f0bca800700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc102e20 0x7f0bcc115cd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc002a50 tx=0x7f0bbc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:53.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.092+0000 7f0bc2a00700 1 -- 172.21.15.63:0/3618480259 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bbc007cc0 con 0x7f0bcc102e20 2024-03-23T12:22:53.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.092+0000 7f0bc2a00700 1 -- 172.21.15.63:0/3618480259 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bbc00f040 con 0x7f0bcc102e20 2024-03-23T12:22:53.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.092+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc10ff10 con 0x7f0bcc102e20 2024-03-23T12:22:53.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.092+0000 7f0bc2a00700 1 -- 172.21.15.63:0/3618480259 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bbc009540 con 0x7f0bcc102e20 2024-03-23T12:22:53.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.092+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc1104d0 con 0x7f0bcc102e20 2024-03-23T12:22:53.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.093+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc0fe570 con 0x7f0bcc102e20 2024-03-23T12:22:53.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.094+0000 7f0bc2a00700 1 -- 172.21.15.63:0/3618480259 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0bbc0096a0 con 0x7f0bcc102e20 2024-03-23T12:22:53.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.095+0000 7f0bc2a00700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0bb006c7d0 0x7f0bb006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.095+0000 7f0bc2a00700 1 -- 172.21.15.63:0/3618480259 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0bbc08b8a0 con 0x7f0bcc102e20 2024-03-23T12:22:53.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.095+0000 7f0bc9e00700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0bb006c7d0 0x7f0bb006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.096+0000 7f0bc9e00700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0bb006c7d0 0x7f0bb006ec80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8004f90 tx=0x7f0bb8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.100+0000 7f0bc2a00700 1 -- 172.21.15.63:0/3618480259 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0bbc059530 con 0x7f0bcc102e20 2024-03-23T12:22:53.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:52 smithi063 ceph-mon[141124]: pgmap v953: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:53.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:52 smithi111 ceph-mon[106899]: pgmap v953: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:53.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.325+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bcc109e30 con 0x7f0bb006c7d0 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.335+0000 7f0bc2a00700 1 -- 172.21.15.63:0/3618480259 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f0bcc109e30 con 0x7f0bb006c7d0 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (32m) 9m ago 45m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (29m) 9m ago 45m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (29m) 9m ago 44m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (31m) 9m ago 44m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (25m) 9m ago 38m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (24m) 9m ago 38m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (9m) 9m ago 38m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (24m) 9m ago 38m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:22:53.314 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (34m) 9m ago 46m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (33m) 9m ago 44m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (30m) 9m ago 46m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (30m) 9m ago 44m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (33m) 9m ago 45m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (32m) 9m ago 44m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (28m) 9m ago 43m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (28m) 9m ago 42m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (27m) 9m ago 42m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (26m) 9m ago 41m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (26m) 9m ago 41m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (25m) 9m ago 40m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:22:53.315 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (32m) 9m ago 44m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:22:53.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0bb006c7d0 msgr2=0x7f0bb006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0bb006c7d0 0x7f0bb006ec80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8004f90 tx=0x7f0bb8009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:53.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc102e20 msgr2=0x7f0bcc115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc102e20 0x7f0bcc115cd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc002a50 tx=0x7f0bbc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 shutdown_connections 2024-03-23T12:22:53.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0bb006c7d0 0x7f0bb006ec80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bcc102e20 0x7f0bcc115cd0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 --2- 172.21.15.63:0/3618480259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc1037c0 0x7f0bcc116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.337+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 >> 172.21.15.63:0/3618480259 conn(0x7f0bcc0f89d0 msgr2=0x7f0bcc0f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:53.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.338+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 shutdown_connections 2024-03-23T12:22:53.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.338+0000 7f0bd1400700 1 -- 172.21.15.63:0/3618480259 wait complete. 2024-03-23T12:22:53.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 -- 172.21.15.63:0/605797946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 msgr2=0x7fe3cc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/605797946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc1031d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0004940 tx=0x7fe3c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 -- 172.21.15.63:0/605797946 shutdown_connections 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/605797946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3cc1037a0 0x7fe3cc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/605797946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc1031d0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 -- 172.21.15.63:0/605797946 >> 172.21.15.63:0/605797946 conn(0x7fe3cc0f89b0 msgr2=0x7fe3cc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 -- 172.21.15.63:0/605797946 shutdown_connections 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.474+0000 7fe3d3e00700 1 -- 172.21.15.63:0/605797946 wait complete. 2024-03-23T12:22:53.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.475+0000 7fe3d3e00700 1 Processor -- start 2024-03-23T12:22:53.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.475+0000 7fe3d3e00700 1 -- start start 2024-03-23T12:22:53.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d3e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3cc1037a0 0x7fe3cc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d3e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc070d90 con 0x7fe3cc102e00 2024-03-23T12:22:53.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d3e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc070ed0 con 0x7fe3cc1037a0 2024-03-23T12:22:53.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46236/0 (socket says 172.21.15.63:46236) 2024-03-23T12:22:53.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d1400700 1 -- 172.21.15.63:0/3422970464 learned_addr learned my addr 172.21.15.63:0/3422970464 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:53.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d1400700 1 -- 172.21.15.63:0/3422970464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3cc1037a0 msgr2=0x7fe3cc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.476+0000 7fe3d0a00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3cc1037a0 0x7fe3cc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3d1400700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3cc1037a0 0x7fe3cc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3d1400700 1 -- 172.21.15.63:0/3422970464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c0005040 con 0x7fe3cc102e00 2024-03-23T12:22:53.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3d0a00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3cc1037a0 0x7fe3cc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:22:53.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3d1400700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc0702c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009390 tx=0x7fe3c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:53.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3c9600700 1 -- 172.21.15.63:0/3422970464 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c000e040 con 0x7fe3cc102e00 2024-03-23T12:22:53.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3c9600700 1 -- 172.21.15.63:0/3422970464 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3c0013420 con 0x7fe3cc102e00 2024-03-23T12:22:53.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc071120 con 0x7fe3cc102e00 2024-03-23T12:22:53.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3c9600700 1 -- 172.21.15.63:0/3422970464 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c001c630 con 0x7fe3cc102e00 2024-03-23T12:22:53.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.477+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc11a250 con 0x7fe3cc102e00 2024-03-23T12:22:53.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.478+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc0fe550 con 0x7fe3cc102e00 2024-03-23T12:22:53.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.479+0000 7fe3c9600700 1 -- 172.21.15.63:0/3422970464 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe3c00139e0 con 0x7fe3cc102e00 2024-03-23T12:22:53.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.480+0000 7fe3c9600700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe3b406c7d0 0x7fe3b406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.480+0000 7fe3c9600700 1 -- 172.21.15.63:0/3422970464 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe3c008cca0 con 0x7fe3cc102e00 2024-03-23T12:22:53.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.481+0000 7fe3d0a00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe3b406c7d0 0x7fe3b406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.481+0000 7fe3d0a00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe3b406c7d0 0x7fe3b406ec80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8007fd0 tx=0x7fe3b8009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:53.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.485+0000 7fe3c9600700 1 -- 172.21.15.63:0/3422970464 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe3c0056f20 con 0x7fe3cc102e00 2024-03-23T12:22:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.786+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe3cc0719b0 con 0x7fe3cc102e00 2024-03-23T12:22:53.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.786+0000 7fe3c9600700 1 -- 172.21.15.63:0/3422970464 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fe3c005a540 con 0x7fe3cc102e00 2024-03-23T12:22:53.765 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:22:53.765 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:22:53.765 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:22:53.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:53.765 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:22:53.765 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:22:53.766 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe3b406c7d0 msgr2=0x7fe3b406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe3b406c7d0 0x7fe3b406ec80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8007fd0 tx=0x7fe3b8009040 comp rx=0 tx=0).stop 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 msgr2=0x7fe3cc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc0702c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009390 tx=0x7fe3c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 shutdown_connections 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe3b406c7d0 0x7fe3b406ec80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3cc102e00 0x7fe3cc0702c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 --2- 172.21.15.63:0/3422970464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3cc1037a0 0x7fe3cc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 >> 172.21.15.63:0/3422970464 conn(0x7fe3cc0f89b0 msgr2=0x7fe3cc0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 shutdown_connections 2024-03-23T12:22:53.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.789+0000 7fe3d3e00700 1 -- 172.21.15.63:0/3422970464 wait complete. 2024-03-23T12:22:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.927+0000 7f8d70400700 1 -- 172.21.15.63:0/690281443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d681037c0 msgr2=0x7f8d680fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.927+0000 7f8d70400700 1 --2- 172.21.15.63:0/690281443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d681037c0 0x7f8d680fce30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8d54004940 tx=0x7f8d54009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.927+0000 7f8d70400700 1 -- 172.21.15.63:0/690281443 shutdown_connections 2024-03-23T12:22:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.927+0000 7f8d70400700 1 --2- 172.21.15.63:0/690281443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d681037c0 0x7f8d680fce30 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.927+0000 7f8d70400700 1 --2- 172.21.15.63:0/690281443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d68102e20 0x7f8d681031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.927+0000 7f8d70400700 1 -- 172.21.15.63:0/690281443 >> 172.21.15.63:0/690281443 conn(0x7f8d680f8970 msgr2=0x7f8d680fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.927+0000 7f8d70400700 1 -- 172.21.15.63:0/690281443 shutdown_connections 2024-03-23T12:22:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.928+0000 7f8d70400700 1 -- 172.21.15.63:0/690281443 wait complete. 2024-03-23T12:22:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.929+0000 7f8d70400700 1 Processor -- start 2024-03-23T12:22:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.929+0000 7f8d70400700 1 -- start start 2024-03-23T12:22:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.929+0000 7f8d70400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d68102e20 0x7f8d68115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.929+0000 7f8d70400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d681037c0 0x7f8d681161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.929+0000 7f8d70400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d68116800 con 0x7f8d68102e20 2024-03-23T12:22:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.929+0000 7f8d70400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6810fbd0 con 0x7f8d681037c0 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6da00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d68102e20 0x7f8d68115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6da00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d68102e20 0x7f8d68115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46262/0 (socket says 172.21.15.63:46262) 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6da00700 1 -- 172.21.15.63:0/688050058 learned_addr learned my addr 172.21.15.63:0/688050058 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6d000700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d681037c0 0x7f8d681161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6da00700 1 -- 172.21.15.63:0/688050058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d681037c0 msgr2=0x7f8d681161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6da00700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d681037c0 0x7f8d681161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6da00700 1 -- 172.21.15.63:0/688050058 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54005040 con 0x7f8d68102e20 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6d000700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d681037c0 0x7f8d681161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:22:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.930+0000 7f8d6da00700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d68102e20 0x7f8d68115c70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00eb40 tx=0x7f8d5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:53.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.931+0000 7f8d65600700 1 -- 172.21.15.63:0/688050058 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d5c007cc0 con 0x7f8d68102e20 2024-03-23T12:22:53.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.931+0000 7f8d65600700 1 -- 172.21.15.63:0/688050058 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d5c00f040 con 0x7f8d68102e20 2024-03-23T12:22:53.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.931+0000 7f8d65600700 1 -- 172.21.15.63:0/688050058 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d5c009570 con 0x7f8d68102e20 2024-03-23T12:22:53.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.931+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6810fe50 con 0x7f8d68102e20 2024-03-23T12:22:53.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.931+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d68110410 con 0x7f8d68102e20 2024-03-23T12:22:53.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.932+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d680fe570 con 0x7f8d68102e20 2024-03-23T12:22:53.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.933+0000 7f8d65600700 1 -- 172.21.15.63:0/688050058 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8d5c0096d0 con 0x7f8d68102e20 2024-03-23T12:22:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.933+0000 7f8d65600700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d5006c7d0 0x7f8d5006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.934+0000 7f8d65600700 1 -- 172.21.15.63:0/688050058 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8d5c08b7e0 con 0x7f8d68102e20 2024-03-23T12:22:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.934+0000 7f8d6d000700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d5006c7d0 0x7f8d5006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.934+0000 7f8d6d000700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d5006c7d0 0x7f8d5006ec80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8d54000c00 tx=0x7f8d54009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:53.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:53.938+0000 7f8d65600700 1 -- 172.21.15.63:0/688050058 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8d5c0594a0 con 0x7f8d68102e20 2024-03-23T12:22:54.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:53 smithi063 ceph-mon[141124]: from='client.35726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:54.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:53 smithi063 ceph-mon[141124]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:54.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:53 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3422970464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:22:54.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:53 smithi111 ceph-mon[106899]: from='client.35726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:54.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:53 smithi111 ceph-mon[106899]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:54.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:53 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3422970464' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:22:54.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.198+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8d6804bed0 con 0x7f8d68102e20 2024-03-23T12:22:54.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.198+0000 7f8d65600700 1 -- 172.21.15.63:0/688050058 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f8d5c059030 con 0x7f8d68102e20 2024-03-23T12:22:54.177 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:22:54.177 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:22:54.177 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:22:54.178 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:22:54.179 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:54.180 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d5006c7d0 msgr2=0x7f8d5006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d5006c7d0 0x7f8d5006ec80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8d54000c00 tx=0x7f8d54009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d68102e20 msgr2=0x7f8d68115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d68102e20 0x7f8d68115c70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00eb40 tx=0x7f8d5c00b040 comp rx=0 tx=0).stop 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 shutdown_connections 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d5006c7d0 0x7f8d5006ec80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d68102e20 0x7f8d68115c70 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 --2- 172.21.15.63:0/688050058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d681037c0 0x7f8d681161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 >> 172.21.15.63:0/688050058 conn(0x7f8d680f8970 msgr2=0x7f8d680f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:54.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 shutdown_connections 2024-03-23T12:22:54.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.201+0000 7f8d70400700 1 -- 172.21.15.63:0/688050058 wait complete. 2024-03-23T12:22:54.182 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 -- 172.21.15.63:0/64930474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d901037a0 msgr2=0x7f8d900fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 --2- 172.21.15.63:0/64930474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d901037a0 0x7f8d900fce10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c004940 tx=0x7f8d7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 -- 172.21.15.63:0/64930474 shutdown_connections 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 --2- 172.21.15.63:0/64930474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d901037a0 0x7f8d900fce10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 --2- 172.21.15.63:0/64930474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d90102e00 0x7f8d901031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 -- 172.21.15.63:0/64930474 >> 172.21.15.63:0/64930474 conn(0x7f8d900f89b0 msgr2=0x7f8d900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 -- 172.21.15.63:0/64930474 shutdown_connections 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.339+0000 7f8d95600700 1 -- 172.21.15.63:0/64930474 wait complete. 2024-03-23T12:22:54.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.340+0000 7f8d95600700 1 Processor -- start 2024-03-23T12:22:54.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.340+0000 7f8d95600700 1 -- start start 2024-03-23T12:22:54.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d95600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90102e00 0x7f8d90113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:54.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d95600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d901037a0 0x7f8d90114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:54.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d95600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90114690 con 0x7f8d90102e00 2024-03-23T12:22:54.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d95600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9010da80 con 0x7f8d901037a0 2024-03-23T12:22:54.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d8ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90102e00 0x7f8d90113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d8ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90102e00 0x7f8d90113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46286/0 (socket says 172.21.15.63:46286) 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d8ea00700 1 -- 172.21.15.63:0/367204687 learned_addr learned my addr 172.21.15.63:0/367204687 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.341+0000 7f8d8e000700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d901037a0 0x7f8d90114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d8ea00700 1 -- 172.21.15.63:0/367204687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d901037a0 msgr2=0x7f8d90114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d8ea00700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d901037a0 0x7f8d90114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d8ea00700 1 -- 172.21.15.63:0/367204687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c005040 con 0x7f8d90102e00 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d8e000700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d901037a0 0x7f8d90114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d8ea00700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90102e00 0x7f8d90113b00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8d80002a50 tx=0x7f8d800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d86a00700 1 -- 172.21.15.63:0/367204687 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d80007cc0 con 0x7f8d90102e00 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d86a00700 1 -- 172.21.15.63:0/367204687 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d8000f040 con 0x7f8d90102e00 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d86a00700 1 -- 172.21.15.63:0/367204687 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d80009540 con 0x7f8d90102e00 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9010dd60 con 0x7f8d90102e00 2024-03-23T12:22:54.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.342+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9010e320 con 0x7f8d90102e00 2024-03-23T12:22:54.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.343+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d900fe550 con 0x7f8d90102e00 2024-03-23T12:22:54.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.344+0000 7f8d86a00700 1 -- 172.21.15.63:0/367204687 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8d800096a0 con 0x7f8d90102e00 2024-03-23T12:22:54.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.345+0000 7f8d86a00700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d7406c7c0 0x7f8d7406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:54.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.345+0000 7f8d86a00700 1 -- 172.21.15.63:0/367204687 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8d8008b8a0 con 0x7f8d90102e00 2024-03-23T12:22:54.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.346+0000 7f8d8e000700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d7406c7c0 0x7f8d7406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:54.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.346+0000 7f8d8e000700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d7406c7c0 0x7f8d7406ec70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8d9010ec40 tx=0x7f8d7c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:54.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.350+0000 7f8d86a00700 1 -- 172.21.15.63:0/367204687 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8d80059530 con 0x7f8d90102e00 2024-03-23T12:22:54.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.581+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d90109da0 con 0x7f8d7406c7c0 2024-03-23T12:22:54.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.582+0000 7f8d86a00700 1 -- 172.21.15.63:0/367204687 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8d90109da0 con 0x7f8d7406c7c0 2024-03-23T12:22:54.561 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:22:54.561 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:22:54.561 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:22:54.562 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d7406c7c0 msgr2=0x7f8d7406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d7406c7c0 0x7f8d7406ec70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8d9010ec40 tx=0x7f8d7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90102e00 msgr2=0x7f8d90113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90102e00 0x7f8d90113b00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8d80002a50 tx=0x7f8d800046b0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 shutdown_connections 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8d7406c7c0 0x7f8d7406ec70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90102e00 0x7f8d90113b00 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 --2- 172.21.15.63:0/367204687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d901037a0 0x7f8d90114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 >> 172.21.15.63:0/367204687 conn(0x7f8d900f89b0 msgr2=0x7f8d900f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 shutdown_connections 2024-03-23T12:22:54.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.585+0000 7f8d95600700 1 -- 172.21.15.63:0/367204687 wait complete. 2024-03-23T12:22:54.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 -- 172.21.15.63:0/1371618260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 msgr2=0x7fa2a40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1371618260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a40706d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa298004970 tx=0x7fa298009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:54.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 -- 172.21.15.63:0/1371618260 shutdown_connections 2024-03-23T12:22:54.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1371618260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a4070c10 0x7fa2a4109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1371618260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a40706d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 -- 172.21.15.63:0/1371618260 >> 172.21.15.63:0/1371618260 conn(0x7fa2a40f8970 msgr2=0x7fa2a40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:54.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 -- 172.21.15.63:0/1371618260 shutdown_connections 2024-03-23T12:22:54.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.723+0000 7fa2ab400700 1 -- 172.21.15.63:0/1371618260 wait complete. 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.724+0000 7fa2ab400700 1 Processor -- start 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.724+0000 7fa2ab400700 1 -- start start 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2ab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a4115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2ab400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a4070c10 0x7fa2a4116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2ab400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a4116880 con 0x7fa2a4070300 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2ab400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a410fc50 con 0x7fa2a4070c10 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a4115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a4115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46304/0 (socket says 172.21.15.63:46304) 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a8a00700 1 -- 172.21.15.63:0/1593988908 learned_addr learned my addr 172.21.15.63:0/1593988908 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a8a00700 1 -- 172.21.15.63:0/1593988908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a4070c10 msgr2=0x7fa2a4116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a3e00700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a4070c10 0x7fa2a4116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:54.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a8a00700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a4070c10 0x7fa2a4116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a8a00700 1 -- 172.21.15.63:0/1593988908 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa298005040 con 0x7fa2a4070300 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.725+0000 7fa2a3e00700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a4070c10 0x7fa2a4116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.726+0000 7fa2a8a00700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a4115cf0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa298005010 tx=0x7fa298009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.726+0000 7fa2a0c00700 1 -- 172.21.15.63:0/1593988908 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa29800e040 con 0x7fa2a4070300 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.726+0000 7fa2a0c00700 1 -- 172.21.15.63:0/1593988908 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa298004290 con 0x7fa2a4070300 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.726+0000 7fa2a0c00700 1 -- 172.21.15.63:0/1593988908 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa298013e40 con 0x7fa2a4070300 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.726+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a410fed0 con 0x7fa2a4070300 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.726+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a4110410 con 0x7fa2a4070300 2024-03-23T12:22:54.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.727+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a4071a40 con 0x7fa2a4070300 2024-03-23T12:22:54.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.728+0000 7fa2a0c00700 1 -- 172.21.15.63:0/1593988908 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa298013620 con 0x7fa2a4070300 2024-03-23T12:22:54.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.729+0000 7fa2a0c00700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa28c06c7d0 0x7fa28c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:22:54.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.729+0000 7fa2a0c00700 1 -- 172.21.15.63:0/1593988908 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa29808cc10 con 0x7fa2a4070300 2024-03-23T12:22:54.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.729+0000 7fa2a3e00700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa28c06c7d0 0x7fa28c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:22:54.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.730+0000 7fa2a3e00700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa28c06c7d0 0x7fa28c06ec80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa290004600 tx=0x7fa29000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:22:54.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:54.733+0000 7fa2a0c00700 1 -- 172.21.15.63:0/1593988908 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa29805a9e0 con 0x7fa2a4070300 2024-03-23T12:22:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.032+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa2a405f5f0 con 0x7fa2a4070300 2024-03-23T12:22:55.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.033+0000 7fa2a0c00700 1 -- 172.21.15.63:0/1593988908 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fa2a405f5f0 con 0x7fa2a4070300 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:22:55.012 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:22:55.013 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:22:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa28c06c7d0 msgr2=0x7fa28c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa28c06c7d0 0x7fa28c06ec80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa290004600 tx=0x7fa29000b040 comp rx=0 tx=0).stop 2024-03-23T12:22:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 msgr2=0x7fa2a4115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:22:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a4115cf0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa298005010 tx=0x7fa298009e30 comp rx=0 tx=0).stop 2024-03-23T12:22:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 shutdown_connections 2024-03-23T12:22:55.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa28c06c7d0 0x7fa28c06ec80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:55.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2a4070300 0x7fa2a4115cf0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:55.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 --2- 172.21.15.63:0/1593988908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2a4070c10 0x7fa2a4116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:22:55.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 >> 172.21.15.63:0/1593988908 conn(0x7fa2a40f8970 msgr2=0x7fa2a40f9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:22:55.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 shutdown_connections 2024-03-23T12:22:55.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:22:55.035+0000 7fa2ab400700 1 -- 172.21.15.63:0/1593988908 wait complete. 2024-03-23T12:22:55.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:54 smithi063 ceph-mon[141124]: pgmap v954: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:55.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:54 smithi063 ceph-mon[141124]: from='client.35734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:55.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:54 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/688050058' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:22:55.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:54 smithi111 ceph-mon[106899]: pgmap v954: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:55.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:54 smithi111 ceph-mon[106899]: from='client.35734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:55.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:54 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/688050058' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:22:56.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:55 smithi063 ceph-mon[141124]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:56.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:55 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1593988908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:22:56.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:56.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:55 smithi111 ceph-mon[106899]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:22:56.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:55 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1593988908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:22:56.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:22:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:56 smithi063 ceph-mon[141124]: pgmap v955: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:57.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:56 smithi111 ceph-mon[106899]: pgmap v955: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:59.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:22:58 smithi063 ceph-mon[141124]: pgmap v956: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:22:59.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:22:58 smithi111 ceph-mon[106899]: pgmap v956: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:01.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:00 smithi063 ceph-mon[141124]: pgmap v957: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:01.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:00 smithi111 ceph-mon[106899]: pgmap v957: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:03.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:02 smithi063 ceph-mon[141124]: pgmap v958: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:03.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:02 smithi111 ceph-mon[106899]: pgmap v958: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:05.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:04 smithi063 ceph-mon[141124]: pgmap v959: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:05.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:04 smithi111 ceph-mon[106899]: pgmap v959: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:07.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:06 smithi063 ceph-mon[141124]: pgmap v960: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:07.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:06 smithi111 ceph-mon[106899]: pgmap v960: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:09.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:08 smithi063 ceph-mon[141124]: pgmap v961: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:09.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:08 smithi111 ceph-mon[106899]: pgmap v961: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:11.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:10 smithi063 ceph-mon[141124]: pgmap v962: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:11.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:10 smithi111 ceph-mon[106899]: pgmap v962: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:13.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:12 smithi063 ceph-mon[141124]: pgmap v963: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:13.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:12 smithi111 ceph-mon[106899]: pgmap v963: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:15.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:14 smithi063 ceph-mon[141124]: pgmap v964: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:15.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:14 smithi111 ceph-mon[106899]: pgmap v964: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:17.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:16 smithi063 ceph-mon[141124]: pgmap v965: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:17.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:16 smithi111 ceph-mon[106899]: pgmap v965: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:19.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:18 smithi063 ceph-mon[141124]: pgmap v966: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:19.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:18 smithi111 ceph-mon[106899]: pgmap v966: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:21.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:20 smithi063 ceph-mon[141124]: pgmap v967: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:21.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:20 smithi111 ceph-mon[106899]: pgmap v967: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:23.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:22 smithi063 ceph-mon[141124]: pgmap v968: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:23.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:22 smithi111 ceph-mon[106899]: pgmap v968: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:25.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:24 smithi063 ceph-mon[141124]: pgmap v969: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:25.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:24 smithi111 ceph-mon[106899]: pgmap v969: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 -- 172.21.15.63:0/3711477470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070c10 msgr2=0x7f7ff4109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 --2- 172.21.15.63:0/3711477470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070c10 0x7f7ff4109190 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0004970 tx=0x7f7fe0009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 -- 172.21.15.63:0/3711477470 shutdown_connections 2024-03-23T12:23:25.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 --2- 172.21.15.63:0/3711477470 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070c10 0x7f7ff4109190 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 --2- 172.21.15.63:0/3711477470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff4070300 0x7f7ff40706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 -- 172.21.15.63:0/3711477470 >> 172.21.15.63:0/3711477470 conn(0x7f7ff40f8970 msgr2=0x7f7ff40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 -- 172.21.15.63:0/3711477470 shutdown_connections 2024-03-23T12:23:25.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.190+0000 7f7ffc600700 1 -- 172.21.15.63:0/3711477470 wait complete. 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.191+0000 7f7ffc600700 1 Processor -- start 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.191+0000 7f7ffc600700 1 -- start start 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ffc600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070300 0x7f7ff4115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ffc600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff4070c10 0x7f7ff4116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ffc600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff4116850 con 0x7f7ff4070300 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ffc600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff410fc20 con 0x7f7ff4070c10 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ff9c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070300 0x7f7ff4115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ff9c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070300 0x7f7ff4115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56044/0 (socket says 172.21.15.63:56044) 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ff9c00700 1 -- 172.21.15.63:0/2683752224 learned_addr learned my addr 172.21.15.63:0/2683752224 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ff9c00700 1 -- 172.21.15.63:0/2683752224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff4070c10 msgr2=0x7f7ff4116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ff9c00700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff4070c10 0x7f7ff4116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.192+0000 7f7ff9c00700 1 -- 172.21.15.63:0/2683752224 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe0005040 con 0x7f7ff4070300 2024-03-23T12:23:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.193+0000 7f7ff9c00700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070300 0x7f7ff4115cc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800eb40 tx=0x7f7fe800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.193+0000 7f7ff1600700 1 -- 172.21.15.63:0/2683752224 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fe8007cc0 con 0x7f7ff4070300 2024-03-23T12:23:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.193+0000 7f7ff1600700 1 -- 172.21.15.63:0/2683752224 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fe800f040 con 0x7f7ff4070300 2024-03-23T12:23:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.193+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff410fea0 con 0x7f7ff4070300 2024-03-23T12:23:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.193+0000 7f7ff1600700 1 -- 172.21.15.63:0/2683752224 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fe8009570 con 0x7f7ff4070300 2024-03-23T12:23:25.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.193+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff41103e0 con 0x7f7ff4070300 2024-03-23T12:23:25.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.194+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff4071a40 con 0x7f7ff4070300 2024-03-23T12:23:25.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.195+0000 7f7ff1600700 1 -- 172.21.15.63:0/2683752224 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7fe80096d0 con 0x7f7ff4070300 2024-03-23T12:23:25.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.196+0000 7f7ff1600700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7fdc06c7d0 0x7f7fdc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.197+0000 7f7ff1600700 1 -- 172.21.15.63:0/2683752224 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7fe808b7e0 con 0x7f7ff4070300 2024-03-23T12:23:25.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.197+0000 7f7ff9200700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7fdc06c7d0 0x7f7fdc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.197+0000 7f7ff9200700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7fdc06c7d0 0x7f7fdc06ec80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000b370 tx=0x7f7fe0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:25.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.202+0000 7f7ff1600700 1 -- 172.21.15.63:0/2683752224 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7fe80594a0 con 0x7f7ff4070300 2024-03-23T12:23:25.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.433+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ff410c050 con 0x7f7fdc06c7d0 2024-03-23T12:23:25.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.434+0000 7f7ff1600700 1 -- 172.21.15.63:0/2683752224 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ff410c050 con 0x7f7fdc06c7d0 2024-03-23T12:23:25.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7fdc06c7d0 msgr2=0x7f7fdc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7fdc06c7d0 0x7f7fdc06ec80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000b370 tx=0x7f7fe0009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070300 msgr2=0x7f7ff4115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070300 0x7f7ff4115cc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800eb40 tx=0x7f7fe800b040 comp rx=0 tx=0).stop 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 shutdown_connections 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7fdc06c7d0 0x7f7fdc06ec80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ff4070300 0x7f7ff4115cc0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 --2- 172.21.15.63:0/2683752224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff4070c10 0x7f7ff4116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 >> 172.21.15.63:0/2683752224 conn(0x7f7ff40f8970 msgr2=0x7f7ff40f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 shutdown_connections 2024-03-23T12:23:25.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.437+0000 7f7ffc600700 1 -- 172.21.15.63:0/2683752224 wait complete. 2024-03-23T12:23:25.429 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 -- 172.21.15.63:0/135724432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c102e20 msgr2=0x7f2f6c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 --2- 172.21.15.63:0/135724432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c102e20 0x7f2f6c1031f0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2f60004940 tx=0x7f2f60009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 -- 172.21.15.63:0/135724432 shutdown_connections 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 --2- 172.21.15.63:0/135724432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c1037c0 0x7f2f6c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 --2- 172.21.15.63:0/135724432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c102e20 0x7f2f6c1031f0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 -- 172.21.15.63:0/135724432 >> 172.21.15.63:0/135724432 conn(0x7f2f6c0f8970 msgr2=0x7f2f6c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 -- 172.21.15.63:0/135724432 shutdown_connections 2024-03-23T12:23:25.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.576+0000 7f2f73600700 1 -- 172.21.15.63:0/135724432 wait complete. 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.577+0000 7f2f73600700 1 Processor -- start 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.577+0000 7f2f73600700 1 -- start start 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.577+0000 7f2f73600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c102e20 0x7f2f6c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.577+0000 7f2f73600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c1037c0 0x7f2f6c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.577+0000 7f2f73600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c116840 con 0x7f2f6c1037c0 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.577+0000 7f2f73600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c10fc10 con 0x7f2f6c102e20 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f6be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c1037c0 0x7f2f6c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f6be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c1037c0 0x7f2f6c1161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56056/0 (socket says 172.21.15.63:56056) 2024-03-23T12:23:25.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f6be00700 1 -- 172.21.15.63:0/844114127 learned_addr learned my addr 172.21.15.63:0/844114127 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f70c00700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c102e20 0x7f2f6c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f6be00700 1 -- 172.21.15.63:0/844114127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c102e20 msgr2=0x7f2f6c115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f6be00700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c102e20 0x7f2f6c115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f6be00700 1 -- 172.21.15.63:0/844114127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60005040 con 0x7f2f6c1037c0 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f70c00700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c102e20 0x7f2f6c115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f6be00700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c1037c0 0x7f2f6c1161f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800eb40 tx=0x7f2f5800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.578+0000 7f2f68c00700 1 -- 172.21.15.63:0/844114127 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f58007cc0 con 0x7f2f6c1037c0 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.579+0000 7f2f68c00700 1 -- 172.21.15.63:0/844114127 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f5800f040 con 0x7f2f6c1037c0 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.579+0000 7f2f68c00700 1 -- 172.21.15.63:0/844114127 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f58009570 con 0x7f2f6c1037c0 2024-03-23T12:23:25.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.579+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c10fe90 con 0x7f2f6c1037c0 2024-03-23T12:23:25.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.579+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c110450 con 0x7f2f6c1037c0 2024-03-23T12:23:25.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.580+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c0fe570 con 0x7f2f6c1037c0 2024-03-23T12:23:25.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.581+0000 7f2f68c00700 1 -- 172.21.15.63:0/844114127 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2f580096d0 con 0x7f2f6c1037c0 2024-03-23T12:23:25.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.581+0000 7f2f68c00700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f5406c7c0 0x7f2f5406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.582+0000 7f2f68c00700 1 -- 172.21.15.63:0/844114127 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f5808b540 con 0x7f2f6c1037c0 2024-03-23T12:23:25.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.582+0000 7f2f70c00700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f5406c7c0 0x7f2f5406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.582+0000 7f2f70c00700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f5406c7c0 0x7f2f5406ec70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009fd0 tx=0x7f2f60009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:25.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.586+0000 7f2f68c00700 1 -- 172.21.15.63:0/844114127 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f58059280 con 0x7f2f6c1037c0 2024-03-23T12:23:25.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.819+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f6c109e50 con 0x7f2f5406c7c0 2024-03-23T12:23:25.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.820+0000 7f2f68c00700 1 -- 172.21.15.63:0/844114127 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2f6c109e50 con 0x7f2f5406c7c0 2024-03-23T12:23:25.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f5406c7c0 msgr2=0x7f2f5406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f5406c7c0 0x7f2f5406ec70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009fd0 tx=0x7f2f60009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c1037c0 msgr2=0x7f2f6c1161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c1037c0 0x7f2f6c1161f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800eb40 tx=0x7f2f5800b040 comp rx=0 tx=0).stop 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 shutdown_connections 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f5406c7c0 0x7f2f5406ec70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c102e20 0x7f2f6c115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 --2- 172.21.15.63:0/844114127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f6c1037c0 0x7f2f6c1161f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 >> 172.21.15.63:0/844114127 conn(0x7f2f6c0f8970 msgr2=0x7f2f6c0703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 shutdown_connections 2024-03-23T12:23:25.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.823+0000 7f2f73600700 1 -- 172.21.15.63:0/844114127 wait complete. 2024-03-23T12:23:25.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 -- 172.21.15.63:0/1578404186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd714102e00 msgr2=0x7fd7141031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 --2- 172.21.15.63:0/1578404186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd714102e00 0x7fd7141031d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd704004940 tx=0x7fd704009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:25.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 -- 172.21.15.63:0/1578404186 shutdown_connections 2024-03-23T12:23:25.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 --2- 172.21.15.63:0/1578404186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7141037a0 0x7fd7140fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 --2- 172.21.15.63:0/1578404186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd714102e00 0x7fd7141031d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 -- 172.21.15.63:0/1578404186 >> 172.21.15.63:0/1578404186 conn(0x7fd7140f89b0 msgr2=0x7fd7140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:25.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 -- 172.21.15.63:0/1578404186 shutdown_connections 2024-03-23T12:23:25.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.960+0000 7fd719400700 1 -- 172.21.15.63:0/1578404186 wait complete. 2024-03-23T12:23:25.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.961+0000 7fd719400700 1 Processor -- start 2024-03-23T12:23:25.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.961+0000 7fd719400700 1 -- start start 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd719400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714102e00 0x7fd714115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd719400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7141037a0 0x7fd714115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd719400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7141165b0 con 0x7fd7141037a0 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd719400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71410fb90 con 0x7fd714102e00 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd711e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7141037a0 0x7fd714115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd711e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7141037a0 0x7fd714115f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56068/0 (socket says 172.21.15.63:56068) 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd711e00700 1 -- 172.21.15.63:0/488084175 learned_addr learned my addr 172.21.15.63:0/488084175 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd712800700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714102e00 0x7fd714115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd711e00700 1 -- 172.21.15.63:0/488084175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714102e00 msgr2=0x7fd714115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd711e00700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714102e00 0x7fd714115a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd711e00700 1 -- 172.21.15.63:0/488084175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd704005040 con 0x7fd7141037a0 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.962+0000 7fd712800700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714102e00 0x7fd714115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:23:25.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.963+0000 7fd711e00700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7141037a0 0x7fd714115f60 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd700002a50 tx=0x7fd7000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:25.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.963+0000 7fd70aa00700 1 -- 172.21.15.63:0/488084175 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd700007cc0 con 0x7fd7141037a0 2024-03-23T12:23:25.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.963+0000 7fd70aa00700 1 -- 172.21.15.63:0/488084175 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd70000f040 con 0x7fd7141037a0 2024-03-23T12:23:25.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.963+0000 7fd70aa00700 1 -- 172.21.15.63:0/488084175 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7000095d0 con 0x7fd7141037a0 2024-03-23T12:23:25.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.963+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71410fe70 con 0x7fd7141037a0 2024-03-23T12:23:25.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.963+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd714110430 con 0x7fd7141037a0 2024-03-23T12:23:25.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.964+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7140fe550 con 0x7fd7141037a0 2024-03-23T12:23:25.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.965+0000 7fd70aa00700 1 -- 172.21.15.63:0/488084175 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd700009730 con 0x7fd7141037a0 2024-03-23T12:23:25.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.966+0000 7fd70aa00700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd6f806c7c0 0x7fd6f806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:25.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.966+0000 7fd70aa00700 1 -- 172.21.15.63:0/488084175 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd70008b720 con 0x7fd7141037a0 2024-03-23T12:23:25.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.967+0000 7fd712800700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd6f806c7c0 0x7fd6f806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:25.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.967+0000 7fd712800700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd6f806c7c0 0x7fd6f806ec70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd70400b840 tx=0x7fd704009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:25.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:25.971+0000 7fd70aa00700 1 -- 172.21.15.63:0/488084175 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd700059460 con 0x7fd7141037a0 2024-03-23T12:23:26.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.193+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd714109e30 con 0x7fd6f806c7c0 2024-03-23T12:23:26.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.203+0000 7fd70aa00700 1 -- 172.21.15.63:0/488084175 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fd714109e30 con 0x7fd6f806c7c0 2024-03-23T12:23:26.181 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:23:26.181 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (32m) 9m ago 45m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:23:26.181 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (29m) 9m ago 46m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:23:26.181 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (29m) 9m ago 44m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:23:26.181 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (32m) 9m ago 45m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (25m) 9m ago 39m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (25m) 9m ago 39m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (9m) 9m ago 39m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (25m) 9m ago 39m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (35m) 9m ago 47m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (33m) 9m ago 44m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (31m) 9m ago 47m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (30m) 9m ago 44m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (33m) 9m ago 46m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (33m) 9m ago 44m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (29m) 9m ago 43m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (28m) 9m ago 43m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (28m) 9m ago 42m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:23:26.182 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (27m) 9m ago 42m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:23:26.183 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (26m) 9m ago 41m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:23:26.183 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (26m) 9m ago 41m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:23:26.183 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (33m) 9m ago 45m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:23:26.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.205+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd6f806c7c0 msgr2=0x7fd6f806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.205+0000 7fd719400700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd6f806c7c0 0x7fd6f806ec70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd70400b840 tx=0x7fd704009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:26.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.205+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7141037a0 msgr2=0x7fd714115f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.205+0000 7fd719400700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7141037a0 0x7fd714115f60 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd700002a50 tx=0x7fd7000046b0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.206+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 shutdown_connections 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.206+0000 7fd719400700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd6f806c7c0 0x7fd6f806ec70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.206+0000 7fd719400700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd714102e00 0x7fd714115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.206+0000 7fd719400700 1 --2- 172.21.15.63:0/488084175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7141037a0 0x7fd714115f60 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.206+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 >> 172.21.15.63:0/488084175 conn(0x7fd7140f89b0 msgr2=0x7fd714071cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.206+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 shutdown_connections 2024-03-23T12:23:26.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.206+0000 7fd719400700 1 -- 172.21.15.63:0/488084175 wait complete. 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 -- 172.21.15.63:0/2840320168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 msgr2=0x7fe9300fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 --2- 172.21.15.63:0/2840320168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe9300fce10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe91c004940 tx=0x7fe91c009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 -- 172.21.15.63:0/2840320168 shutdown_connections 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 --2- 172.21.15.63:0/2840320168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe9300fce10 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 --2- 172.21.15.63:0/2840320168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930102e00 0x7fe9301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 -- 172.21.15.63:0/2840320168 >> 172.21.15.63:0/2840320168 conn(0x7fe9300f89b0 msgr2=0x7fe9300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 -- 172.21.15.63:0/2840320168 shutdown_connections 2024-03-23T12:23:26.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.341+0000 7fe935800700 1 -- 172.21.15.63:0/2840320168 wait complete. 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe935800700 1 Processor -- start 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe935800700 1 -- start start 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe935800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930102e00 0x7fe930115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe935800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe930116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe935800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9301168b0 con 0x7fe9301037a0 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe935800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93010fc80 con 0x7fe930102e00 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe92e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe930116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:26.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe92e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe930116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56096/0 (socket says 172.21.15.63:56096) 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.342+0000 7fe92e000700 1 -- 172.21.15.63:0/836345358 learned_addr learned my addr 172.21.15.63:0/836345358 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe92ea00700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930102e00 0x7fe930115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe92e000700 1 -- 172.21.15.63:0/836345358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930102e00 msgr2=0x7fe930115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe92e000700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930102e00 0x7fe930115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe92e000700 1 -- 172.21.15.63:0/836345358 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c005040 con 0x7fe9301037a0 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe92e000700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe930116260 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009390 tx=0x7fe91c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe926a00700 1 -- 172.21.15.63:0/836345358 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe91c00e040 con 0x7fe9301037a0 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe926a00700 1 -- 172.21.15.63:0/836345358 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe91c004290 con 0x7fe9301037a0 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe926a00700 1 -- 172.21.15.63:0/836345358 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe91c013e70 con 0x7fe9301037a0 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.343+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93010ff00 con 0x7fe9301037a0 2024-03-23T12:23:26.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.344+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9301104c0 con 0x7fe9301037a0 2024-03-23T12:23:26.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.345+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9300fe550 con 0x7fe9301037a0 2024-03-23T12:23:26.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.346+0000 7fe926a00700 1 -- 172.21.15.63:0/836345358 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe91c013690 con 0x7fe9301037a0 2024-03-23T12:23:26.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.347+0000 7fe926a00700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe91406c7d0 0x7fe91406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:26.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.347+0000 7fe926a00700 1 -- 172.21.15.63:0/836345358 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe91c08cb20 con 0x7fe9301037a0 2024-03-23T12:23:26.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.347+0000 7fe92ea00700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe91406c7d0 0x7fe91406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:26.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.348+0000 7fe92ea00700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe91406c7d0 0x7fe91406ec80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe920004f60 tx=0x7fe92000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:26.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.352+0000 7fe926a00700 1 -- 172.21.15.63:0/836345358 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe91c05a9a0 con 0x7fe9301037a0 2024-03-23T12:23:26.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.649+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe930109dc0 con 0x7fe9301037a0 2024-03-23T12:23:26.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.650+0000 7fe926a00700 1 -- 172.21.15.63:0/836345358 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fe91c013420 con 0x7fe9301037a0 2024-03-23T12:23:26.628 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:23:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:23:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:23:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:23:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:23:26.629 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:23:26.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe91406c7d0 msgr2=0x7fe91406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe91406c7d0 0x7fe91406ec80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe920004f60 tx=0x7fe92000b040 comp rx=0 tx=0).stop 2024-03-23T12:23:26.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 msgr2=0x7fe930116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe930116260 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009390 tx=0x7fe91c009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:26.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 shutdown_connections 2024-03-23T12:23:26.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe91406c7d0 0x7fe91406ec80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe930102e00 0x7fe930115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 --2- 172.21.15.63:0/836345358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9301037a0 0x7fe930116260 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.652+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 >> 172.21.15.63:0/836345358 conn(0x7fe9300f89b0 msgr2=0x7fe930070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:26.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.653+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 shutdown_connections 2024-03-23T12:23:26.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.653+0000 7fe935800700 1 -- 172.21.15.63:0/836345358 wait complete. 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 -- 172.21.15.63:0/930021266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 msgr2=0x7f9e9c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/930021266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c070730 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c004970 tx=0x7f9e8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 -- 172.21.15.63:0/930021266 shutdown_connections 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/930021266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e9c070c70 0x7f9e9c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/930021266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c070730 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 -- 172.21.15.63:0/930021266 >> 172.21.15.63:0/930021266 conn(0x7f9e9c0f8930 msgr2=0x7f9e9c0fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 -- 172.21.15.63:0/930021266 shutdown_connections 2024-03-23T12:23:26.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.792+0000 7f9ea2a00700 1 -- 172.21.15.63:0/930021266 wait complete. 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.793+0000 7f9ea2a00700 1 Processor -- start 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.793+0000 7f9ea2a00700 1 -- start start 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.793+0000 7f9ea2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9ea2a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e9c070c70 0x7f9e9c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9ea2a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c1167a0 con 0x7f9e9c070360 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9ea2a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c10fb90 con 0x7f9e9c070c70 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9e9be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9e9be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56118/0 (socket says 172.21.15.63:56118) 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9e9be00700 1 -- 172.21.15.63:0/3791786525 learned_addr learned my addr 172.21.15.63:0/3791786525 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9e9b400700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e9c070c70 0x7f9e9c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:26.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9e9be00700 1 -- 172.21.15.63:0/3791786525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e9c070c70 msgr2=0x7f9e9c116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:26.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9e9be00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e9c070c70 0x7f9e9c116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:26.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.794+0000 7f9e9be00700 1 -- 172.21.15.63:0/3791786525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c005040 con 0x7f9e9c070360 2024-03-23T12:23:26.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.795+0000 7f9e9be00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c115c10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c004f50 tx=0x7f9e8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:26.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.795+0000 7f9e93e00700 1 -- 172.21.15.63:0/3791786525 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e8c00e040 con 0x7f9e9c070360 2024-03-23T12:23:26.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.795+0000 7f9e93e00700 1 -- 172.21.15.63:0/3791786525 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e8c013420 con 0x7f9e9c070360 2024-03-23T12:23:26.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.795+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c10fe10 con 0x7f9e9c070360 2024-03-23T12:23:26.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.795+0000 7f9e93e00700 1 -- 172.21.15.63:0/3791786525 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e8c01c630 con 0x7f9e9c070360 2024-03-23T12:23:26.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.795+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c110350 con 0x7f9e9c070360 2024-03-23T12:23:26.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.796+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c071aa0 con 0x7f9e9c070360 2024-03-23T12:23:26.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.797+0000 7f9e93e00700 1 -- 172.21.15.63:0/3791786525 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9e8c013590 con 0x7f9e9c070360 2024-03-23T12:23:26.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.798+0000 7f9e93e00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e8006c7d0 0x7f9e8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:26.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.799+0000 7f9e93e00700 1 -- 172.21.15.63:0/3791786525 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9e8c08cc00 con 0x7f9e9c070360 2024-03-23T12:23:26.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.799+0000 7f9e9b400700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e8006c7d0 0x7f9e8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:26.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.799+0000 7f9e9b400700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e8006c7d0 0x7f9e8006ec80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c111440 tx=0x7f9e8800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:26.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:26.803+0000 7f9e93e00700 1 -- 172.21.15.63:0/3791786525 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e8c05a910 con 0x7f9e9c070360 2024-03-23T12:23:27.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.060+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9e9c04bf00 con 0x7f9e9c070360 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.061+0000 7f9e93e00700 1 -- 172.21.15.63:0/3791786525 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f9e8c05a4a0 con 0x7f9e9c070360 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:23:27.039 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:23:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:27.041 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:23:27.042 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:27.042 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:27.042 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:27.042 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:27.042 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e8006c7d0 msgr2=0x7f9e8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e8006c7d0 0x7f9e8006ec80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c111440 tx=0x7f9e8800b040 comp rx=0 tx=0).stop 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 msgr2=0x7f9e9c115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c115c10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c004f50 tx=0x7f9e8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 shutdown_connections 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9e8006c7d0 0x7f9e8006ec80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e9c070360 0x7f9e9c115c10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 --2- 172.21.15.63:0/3791786525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e9c070c70 0x7f9e9c116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 >> 172.21.15.63:0/3791786525 conn(0x7f9e9c0f8930 msgr2=0x7f9e9c0f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 shutdown_connections 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.064+0000 7f9ea2a00700 1 -- 172.21.15.63:0/3791786525 wait complete. 2024-03-23T12:23:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:23:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:26 smithi063 ceph-mon[141124]: pgmap v970: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:26 smithi063 ceph-mon[141124]: from='client.35754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:26 smithi063 ceph-mon[141124]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:26 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/836345358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:23:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:26 smithi111 ceph-mon[106899]: pgmap v970: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:26 smithi111 ceph-mon[106899]: from='client.35754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:26 smithi111 ceph-mon[106899]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:26 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/836345358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 -- 172.21.15.63:0/1959123669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 msgr2=0x7f5e101031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 --2- 172.21.15.63:0/1959123669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e101031d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5e00004940 tx=0x7f5e00009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 -- 172.21.15.63:0/1959123669 shutdown_connections 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 --2- 172.21.15.63:0/1959123669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e101037a0 0x7f5e100fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 --2- 172.21.15.63:0/1959123669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e101031d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 -- 172.21.15.63:0/1959123669 >> 172.21.15.63:0/1959123669 conn(0x7f5e100f89b0 msgr2=0x7f5e100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 -- 172.21.15.63:0/1959123669 shutdown_connections 2024-03-23T12:23:27.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.202+0000 7f5e16800700 1 -- 172.21.15.63:0/1959123669 wait complete. 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.203+0000 7f5e16800700 1 Processor -- start 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.203+0000 7f5e16800700 1 -- start start 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.203+0000 7f5e16800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.203+0000 7f5e16800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e101037a0 0x7f5e10070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.203+0000 7f5e16800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10070d90 con 0x7f5e10102e00 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.203+0000 7f5e16800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10070ed0 con 0x7f5e101037a0 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e100702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56128/0 (socket says 172.21.15.63:56128) 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e0f400700 1 -- 172.21.15.63:0/1863682225 learned_addr learned my addr 172.21.15.63:0/1863682225 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e07e00700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e101037a0 0x7f5e10070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e0f400700 1 -- 172.21.15.63:0/1863682225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e101037a0 msgr2=0x7f5e10070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e0f400700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e101037a0 0x7f5e10070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e0f400700 1 -- 172.21.15.63:0/1863682225 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00005040 con 0x7f5e10102e00 2024-03-23T12:23:27.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.204+0000 7f5e0f400700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e100702c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009390 tx=0x7f5e00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:27.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.205+0000 7f5e07400700 1 -- 172.21.15.63:0/1863682225 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0000e040 con 0x7f5e10102e00 2024-03-23T12:23:27.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.205+0000 7f5e07400700 1 -- 172.21.15.63:0/1863682225 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e00013420 con 0x7f5e10102e00 2024-03-23T12:23:27.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.205+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e10071120 con 0x7f5e10102e00 2024-03-23T12:23:27.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.205+0000 7f5e07400700 1 -- 172.21.15.63:0/1863682225 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0001c630 con 0x7f5e10102e00 2024-03-23T12:23:27.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.205+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1011a250 con 0x7f5e10102e00 2024-03-23T12:23:27.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.206+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e100fe550 con 0x7f5e10102e00 2024-03-23T12:23:27.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.207+0000 7f5e07400700 1 -- 172.21.15.63:0/1863682225 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5e000139e0 con 0x7f5e10102e00 2024-03-23T12:23:27.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.208+0000 7f5e07400700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5df406c7d0 0x7f5df406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:27.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.208+0000 7f5e07400700 1 -- 172.21.15.63:0/1863682225 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e0008cbe0 con 0x7f5e10102e00 2024-03-23T12:23:27.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.208+0000 7f5e07e00700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5df406c7d0 0x7f5df406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:27.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.208+0000 7f5e07e00700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5df406c7d0 0x7f5df406ec80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5e10071c00 tx=0x7f5df8009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:27.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.212+0000 7f5e07400700 1 -- 172.21.15.63:0/1863682225 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5e00056f20 con 0x7f5e10102e00 2024-03-23T12:23:27.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.445+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e100719b0 con 0x7f5df406c7d0 2024-03-23T12:23:27.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.446+0000 7f5e07400700 1 -- 172.21.15.63:0/1863682225 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5e100719b0 con 0x7f5df406c7d0 2024-03-23T12:23:27.424 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:23:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:23:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:23:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:23:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:23:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:23:27.425 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:23:27.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5df406c7d0 msgr2=0x7f5df406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5df406c7d0 0x7f5df406ec80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5e10071c00 tx=0x7f5df8009ee0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 msgr2=0x7f5e100702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e100702c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009390 tx=0x7f5e00009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 shutdown_connections 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5df406c7d0 0x7f5df406ec80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e10102e00 0x7f5e100702c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 --2- 172.21.15.63:0/1863682225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e101037a0 0x7f5e10070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 >> 172.21.15.63:0/1863682225 conn(0x7f5e100f89b0 msgr2=0x7f5e100f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 shutdown_connections 2024-03-23T12:23:27.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.449+0000 7f5e16800700 1 -- 172.21.15.63:0/1863682225 wait complete. 2024-03-23T12:23:27.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 -- 172.21.15.63:0/1359586339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e00102e00 msgr2=0x7f2e001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 --2- 172.21.15.63:0/1359586339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e00102e00 0x7f2e001031d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f2df0004940 tx=0x7f2df0009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:27.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 -- 172.21.15.63:0/1359586339 shutdown_connections 2024-03-23T12:23:27.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 --2- 172.21.15.63:0/1359586339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e001037a0 0x7f2e000fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 --2- 172.21.15.63:0/1359586339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e00102e00 0x7f2e001031d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 -- 172.21.15.63:0/1359586339 >> 172.21.15.63:0/1359586339 conn(0x7f2e000f89b0 msgr2=0x7f2e000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:27.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 -- 172.21.15.63:0/1359586339 shutdown_connections 2024-03-23T12:23:27.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.587+0000 7f2e04a00700 1 -- 172.21.15.63:0/1359586339 wait complete. 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.588+0000 7f2e04a00700 1 Processor -- start 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.588+0000 7f2e04a00700 1 -- start start 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2e04a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e00102e00 0x7f2e000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2e04a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e001037a0 0x7f2e00070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2e04a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e00070d90 con 0x7f2e001037a0 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2e04a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e00070ed0 con 0x7f2e00102e00 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2dfd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e001037a0 0x7f2e00070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2dfd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e001037a0 0x7f2e00070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56140/0 (socket says 172.21.15.63:56140) 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2dfd400700 1 -- 172.21.15.63:0/2621755472 learned_addr learned my addr 172.21.15.63:0/2621755472 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2dfde00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e00102e00 0x7f2e000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2dfd400700 1 -- 172.21.15.63:0/2621755472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e00102e00 msgr2=0x7f2e000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2dfd400700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e00102e00 0x7f2e000702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.589+0000 7f2dfd400700 1 -- 172.21.15.63:0/2621755472 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df0005040 con 0x7f2e001037a0 2024-03-23T12:23:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.590+0000 7f2dfd400700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e001037a0 0x7f2e00070800 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00eb40 tx=0x7f2dec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.590+0000 7f2df6000700 1 -- 172.21.15.63:0/2621755472 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dec007cc0 con 0x7f2e001037a0 2024-03-23T12:23:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.590+0000 7f2df6000700 1 -- 172.21.15.63:0/2621755472 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dec00f040 con 0x7f2e001037a0 2024-03-23T12:23:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.590+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e00071120 con 0x7f2e001037a0 2024-03-23T12:23:27.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.590+0000 7f2df6000700 1 -- 172.21.15.63:0/2621755472 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dec00c8f0 con 0x7f2e001037a0 2024-03-23T12:23:27.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.590+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0011a170 con 0x7f2e001037a0 2024-03-23T12:23:27.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.591+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e000fe550 con 0x7f2e001037a0 2024-03-23T12:23:27.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.592+0000 7f2df6000700 1 -- 172.21.15.63:0/2621755472 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2dec0090d0 con 0x7f2e001037a0 2024-03-23T12:23:27.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.593+0000 7f2df6000700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2de406c7d0 0x7f2de406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:27.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.593+0000 7f2df6000700 1 -- 172.21.15.63:0/2621755472 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2dec08b470 con 0x7f2e001037a0 2024-03-23T12:23:27.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.593+0000 7f2dfde00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2de406c7d0 0x7f2de406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:27.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.594+0000 7f2dfde00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2de406c7d0 0x7f2de406ec80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2df00045a0 tx=0x7f2df0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:27.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.598+0000 7f2df6000700 1 -- 172.21.15.63:0/2621755472 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2dec055720 con 0x7f2e001037a0 2024-03-23T12:23:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.895+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2e00109e50 con 0x7f2e001037a0 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.895+0000 7f2df6000700 1 -- 172.21.15.63:0/2621755472 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f2dec019e10 con 0x7f2e001037a0 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:23:27.873 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:23:27.874 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:23:27.875 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:23:27.875 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:23:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.897+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2de406c7d0 msgr2=0x7f2de406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.897+0000 7f2e04a00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2de406c7d0 0x7f2de406ec80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2df00045a0 tx=0x7f2df0009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.897+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e001037a0 msgr2=0x7f2e00070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.897+0000 7f2e04a00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e001037a0 0x7f2e00070800 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00eb40 tx=0x7f2dec00b040 comp rx=0 tx=0).stop 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.898+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 shutdown_connections 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.898+0000 7f2e04a00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2de406c7d0 0x7f2de406ec80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.898+0000 7f2e04a00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e00102e00 0x7f2e000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.898+0000 7f2e04a00700 1 --2- 172.21.15.63:0/2621755472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e001037a0 0x7f2e00070800 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.898+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 >> 172.21.15.63:0/2621755472 conn(0x7f2e000f89b0 msgr2=0x7f2e000f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.898+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 shutdown_connections 2024-03-23T12:23:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:27.898+0000 7f2e04a00700 1 -- 172.21.15.63:0/2621755472 wait complete. 2024-03-23T12:23:28.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:27 smithi063 ceph-mon[141124]: from='client.35762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:28.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:27 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3791786525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:23:28.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:27 smithi111 ceph-mon[106899]: from='client.35762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:28.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:27 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3791786525' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:23:29.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:28 smithi063 ceph-mon[141124]: pgmap v971: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:29.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:28 smithi063 ceph-mon[141124]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:29.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:28 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2621755472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:23:29.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:28 smithi111 ceph-mon[106899]: pgmap v971: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:29.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:28 smithi111 ceph-mon[106899]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:23:29.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:28 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2621755472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:23:31.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:30 smithi063 ceph-mon[141124]: pgmap v972: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:31.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:30 smithi111 ceph-mon[106899]: pgmap v972: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:33.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:32 smithi063 ceph-mon[141124]: pgmap v973: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:33.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:32 smithi111 ceph-mon[106899]: pgmap v973: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:35.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:34 smithi063 ceph-mon[141124]: pgmap v974: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:35.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:34 smithi111 ceph-mon[106899]: pgmap v974: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:37.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:36 smithi063 ceph-mon[141124]: pgmap v975: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:37.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:36 smithi111 ceph-mon[106899]: pgmap v975: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:39.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:38 smithi063 ceph-mon[141124]: pgmap v976: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:39.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:38 smithi111 ceph-mon[106899]: pgmap v976: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:41.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:40 smithi063 ceph-mon[141124]: pgmap v977: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:41.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:40 smithi111 ceph-mon[106899]: pgmap v977: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:42.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:42.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:43.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:42 smithi063 ceph-mon[141124]: pgmap v978: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:43.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:42 smithi111 ceph-mon[106899]: pgmap v978: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:45.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:44 smithi063 ceph-mon[141124]: pgmap v979: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:45.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:44 smithi111 ceph-mon[106899]: pgmap v979: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:47.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:46 smithi063 ceph-mon[141124]: pgmap v980: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:47.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:46 smithi111 ceph-mon[106899]: pgmap v980: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:49.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:48 smithi063 ceph-mon[141124]: pgmap v981: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:49.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:48 smithi111 ceph-mon[106899]: pgmap v981: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:51.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:50 smithi063 ceph-mon[141124]: pgmap v982: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:51.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:50 smithi111 ceph-mon[106899]: pgmap v982: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:53.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:52 smithi063 ceph-mon[141124]: pgmap v983: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:53.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:52 smithi111 ceph-mon[106899]: pgmap v983: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:55.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:54 smithi063 ceph-mon[141124]: pgmap v984: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:55.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:54 smithi111 ceph-mon[106899]: pgmap v984: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:56 smithi063 ceph-mon[141124]: pgmap v985: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:57.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:56 smithi111 ceph-mon[106899]: pgmap v985: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:57.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:23:58.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 -- 172.21.15.63:0/2903780971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 msgr2=0x7f6da00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 --2- 172.21.15.63:0/2903780971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da00fce10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c004940 tx=0x7f6d8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:58.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 -- 172.21.15.63:0/2903780971 shutdown_connections 2024-03-23T12:23:58.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 --2- 172.21.15.63:0/2903780971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da00fce10 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 --2- 172.21.15.63:0/2903780971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6da0102e00 0x7f6da01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 -- 172.21.15.63:0/2903780971 >> 172.21.15.63:0/2903780971 conn(0x7f6da00f89b0 msgr2=0x7f6da00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:58.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 -- 172.21.15.63:0/2903780971 shutdown_connections 2024-03-23T12:23:58.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.049+0000 7f6da7800700 1 -- 172.21.15.63:0/2903780971 wait complete. 2024-03-23T12:23:58.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.050+0000 7f6da7800700 1 Processor -- start 2024-03-23T12:23:58.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.050+0000 7f6da7800700 1 -- start start 2024-03-23T12:23:58.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6da7800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6da0102e00 0x7f6da0115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6da7800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da0116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6da7800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da01168b0 con 0x7f6da01037a0 2024-03-23T12:23:58.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6da7800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da010fc80 con 0x7f6da0102e00 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6d9fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da0116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6d9fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da0116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43910/0 (socket says 172.21.15.63:43910) 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2477852858 learned_addr learned my addr 172.21.15.63:0/2477852858 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6da4e00700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6da0102e00 0x7f6da0115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2477852858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6da0102e00 msgr2=0x7f6da0115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6da0102e00 0x7f6da0115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.051+0000 7f6d9fe00700 1 -- 172.21.15.63:0/2477852858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c005040 con 0x7f6da01037a0 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.052+0000 7f6d9fe00700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da0116260 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009390 tx=0x7f6d8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:58.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.052+0000 7f6d9cc00700 1 -- 172.21.15.63:0/2477852858 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d8c00e040 con 0x7f6da01037a0 2024-03-23T12:23:58.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.052+0000 7f6d9cc00700 1 -- 172.21.15.63:0/2477852858 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d8c013420 con 0x7f6da01037a0 2024-03-23T12:23:58.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.052+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da010ff00 con 0x7f6da01037a0 2024-03-23T12:23:58.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.052+0000 7f6d9cc00700 1 -- 172.21.15.63:0/2477852858 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d8c01c630 con 0x7f6da01037a0 2024-03-23T12:23:58.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.052+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da01104c0 con 0x7f6da01037a0 2024-03-23T12:23:58.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.053+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da00fe550 con 0x7f6da01037a0 2024-03-23T12:23:58.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.054+0000 7f6d9cc00700 1 -- 172.21.15.63:0/2477852858 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6d8c013a00 con 0x7f6da01037a0 2024-03-23T12:23:58.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.055+0000 7f6d9cc00700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8806c7d0 0x7f6d8806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.055+0000 7f6d9cc00700 1 -- 172.21.15.63:0/2477852858 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d8c08caa0 con 0x7f6da01037a0 2024-03-23T12:23:58.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.055+0000 7f6da4e00700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8806c7d0 0x7f6d8806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.056+0000 7f6da4e00700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8806c7d0 0x7f6d8806ec80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6d94004f60 tx=0x7f6d9400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:58.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.060+0000 7f6d9cc00700 1 -- 172.21.15.63:0/2477852858 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d8c05a870 con 0x7f6da01037a0 2024-03-23T12:23:58.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.291+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6da0109dc0 con 0x7f6d8806c7d0 2024-03-23T12:23:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.292+0000 7f6d9cc00700 1 -- 172.21.15.63:0/2477852858 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6da0109dc0 con 0x7f6d8806c7d0 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8806c7d0 msgr2=0x7f6d8806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8806c7d0 0x7f6d8806ec80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6d94004f60 tx=0x7f6d9400b040 comp rx=0 tx=0).stop 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 msgr2=0x7f6da0116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da0116260 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009390 tx=0x7f6d8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 shutdown_connections 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6d8806c7d0 0x7f6d8806ec80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6da0102e00 0x7f6da0115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 --2- 172.21.15.63:0/2477852858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6da01037a0 0x7f6da0116260 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 >> 172.21.15.63:0/2477852858 conn(0x7f6da00f89b0 msgr2=0x7f6da0070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 shutdown_connections 2024-03-23T12:23:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.295+0000 7f6da7800700 1 -- 172.21.15.63:0/2477852858 wait complete. 2024-03-23T12:23:58.289 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:23:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 -- 172.21.15.63:0/1667222870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 msgr2=0x7f7a581053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/1667222870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a581053e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7a48004940 tx=0x7f7a48009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 -- 172.21.15.63:0/1667222870 shutdown_connections 2024-03-23T12:23:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/1667222870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a581059b0 0x7f7a580ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/1667222870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a581053e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 -- 172.21.15.63:0/1667222870 >> 172.21.15.63:0/1667222870 conn(0x7f7a580faba0 msgr2=0x7f7a580fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 -- 172.21.15.63:0/1667222870 shutdown_connections 2024-03-23T12:23:58.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.434+0000 7f7a5ea00700 1 -- 172.21.15.63:0/1667222870 wait complete. 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.435+0000 7f7a5ea00700 1 Processor -- start 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.435+0000 7f7a5ea00700 1 -- start start 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a5ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a58115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a5ea00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a581059b0 0x7f7a581161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a5ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a58116830 con 0x7f7a58105010 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a5ea00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5810fc00 con 0x7f7a581059b0 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a57e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a58115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a57e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a58115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43940/0 (socket says 172.21.15.63:43940) 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a57e00700 1 -- 172.21.15.63:0/646875292 learned_addr learned my addr 172.21.15.63:0/646875292 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a57400700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a581059b0 0x7f7a581161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a57e00700 1 -- 172.21.15.63:0/646875292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a581059b0 msgr2=0x7f7a581161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a57e00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a581059b0 0x7f7a581161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.436+0000 7f7a57e00700 1 -- 172.21.15.63:0/646875292 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48005040 con 0x7f7a58105010 2024-03-23T12:23:58.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.437+0000 7f7a57e00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a58115ca0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009390 tx=0x7f7a48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.437+0000 7f7a4fe00700 1 -- 172.21.15.63:0/646875292 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4800e040 con 0x7f7a58105010 2024-03-23T12:23:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.437+0000 7f7a4fe00700 1 -- 172.21.15.63:0/646875292 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a48013420 con 0x7f7a58105010 2024-03-23T12:23:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.437+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5810fe80 con 0x7f7a58105010 2024-03-23T12:23:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.437+0000 7f7a4fe00700 1 -- 172.21.15.63:0/646875292 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4801c630 con 0x7f7a58105010 2024-03-23T12:23:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.437+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a58110440 con 0x7f7a58105010 2024-03-23T12:23:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.438+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a58100760 con 0x7f7a58105010 2024-03-23T12:23:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.439+0000 7f7a4fe00700 1 -- 172.21.15.63:0/646875292 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7a480139e0 con 0x7f7a58105010 2024-03-23T12:23:58.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.440+0000 7f7a4fe00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a3c06c7d0 0x7f7a3c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.440+0000 7f7a4fe00700 1 -- 172.21.15.63:0/646875292 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7a4808cc90 con 0x7f7a58105010 2024-03-23T12:23:58.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.440+0000 7f7a57400700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a3c06c7d0 0x7f7a3c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.441+0000 7f7a57400700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a3c06c7d0 0x7f7a3c06ec80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7a58111540 tx=0x7f7a44009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:58.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.444+0000 7f7a4fe00700 1 -- 172.21.15.63:0/646875292 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7a4805aa50 con 0x7f7a58105010 2024-03-23T12:23:58.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.676+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a5804bf40 con 0x7f7a3c06c7d0 2024-03-23T12:23:58.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.678+0000 7f7a4fe00700 1 -- 172.21.15.63:0/646875292 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7a5804bf40 con 0x7f7a3c06c7d0 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a3c06c7d0 msgr2=0x7f7a3c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a3c06c7d0 0x7f7a3c06ec80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7a58111540 tx=0x7f7a44009ee0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 msgr2=0x7f7a58115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a58115ca0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009390 tx=0x7f7a48009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 shutdown_connections 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7a3c06c7d0 0x7f7a3c06ec80 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a58105010 0x7f7a58115ca0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 --2- 172.21.15.63:0/646875292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a581059b0 0x7f7a581161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 >> 172.21.15.63:0/646875292 conn(0x7f7a580faba0 msgr2=0x7f7a580fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:58.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 shutdown_connections 2024-03-23T12:23:58.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.680+0000 7f7a5ea00700 1 -- 172.21.15.63:0/646875292 wait complete. 2024-03-23T12:23:58.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.817+0000 7fa044c00700 1 -- 172.21.15.63:0/3746844585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070360 msgr2=0x7fa040070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.817+0000 7fa044c00700 1 --2- 172.21.15.63:0/3746844585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070360 0x7fa040070730 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa030004970 tx=0x7fa030009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:58.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.818+0000 7fa044c00700 1 -- 172.21.15.63:0/3746844585 shutdown_connections 2024-03-23T12:23:58.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.818+0000 7fa044c00700 1 --2- 172.21.15.63:0/3746844585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa040070c70 0x7fa040109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.818+0000 7fa044c00700 1 --2- 172.21.15.63:0/3746844585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070360 0x7fa040070730 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.818+0000 7fa044c00700 1 -- 172.21.15.63:0/3746844585 >> 172.21.15.63:0/3746844585 conn(0x7fa0400f89b0 msgr2=0x7fa0400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:58.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.818+0000 7fa044c00700 1 -- 172.21.15.63:0/3746844585 shutdown_connections 2024-03-23T12:23:58.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.818+0000 7fa044c00700 1 -- 172.21.15.63:0/3746844585 wait complete. 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.818+0000 7fa044c00700 1 Processor -- start 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa044c00700 1 -- start start 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa044c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa040070360 0x7fa040115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa044c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070c70 0x7fa0401161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa044c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa040116810 con 0x7fa040070c70 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa044c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04010fbe0 con 0x7fa040070360 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa037e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070c70 0x7fa0401161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa037e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070c70 0x7fa0401161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43950/0 (socket says 172.21.15.63:43950) 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa037e00700 1 -- 172.21.15.63:0/1185372361 learned_addr learned my addr 172.21.15.63:0/1185372361 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.819+0000 7fa03de00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa040070360 0x7fa040115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa037e00700 1 -- 172.21.15.63:0/1185372361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa040070360 msgr2=0x7fa040115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa037e00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa040070360 0x7fa040115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:58.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa037e00700 1 -- 172.21.15.63:0/1185372361 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa030005040 con 0x7fa040070c70 2024-03-23T12:23:58.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa037e00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070c70 0x7fa0401161c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa028002a80 tx=0x7fa0280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:58.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa036000700 1 -- 172.21.15.63:0/1185372361 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa028007cc0 con 0x7fa040070c70 2024-03-23T12:23:58.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa036000700 1 -- 172.21.15.63:0/1185372361 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa02800f040 con 0x7fa040070c70 2024-03-23T12:23:58.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04010fec0 con 0x7fa040070c70 2024-03-23T12:23:58.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa036000700 1 -- 172.21.15.63:0/1185372361 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa02800da60 con 0x7fa040070c70 2024-03-23T12:23:58.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.820+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa040110400 con 0x7fa040070c70 2024-03-23T12:23:58.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.821+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa040103b90 con 0x7fa040070c70 2024-03-23T12:23:58.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.822+0000 7fa036000700 1 -- 172.21.15.63:0/1185372361 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa02800dbc0 con 0x7fa040070c70 2024-03-23T12:23:58.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.823+0000 7fa036000700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa02406c7d0 0x7fa02406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:58.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.823+0000 7fa036000700 1 -- 172.21.15.63:0/1185372361 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa02808b760 con 0x7fa040070c70 2024-03-23T12:23:58.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.823+0000 7fa03de00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa02406c7d0 0x7fa02406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:58.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.824+0000 7fa03de00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa02406c7d0 0x7fa02406ec80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b370 tx=0x7fa030009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:58.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:58.828+0000 7fa036000700 1 -- 172.21.15.63:0/1185372361 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa028059370 con 0x7fa040070c70 2024-03-23T12:23:59.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.052+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa04010bfb0 con 0x7fa02406c7d0 2024-03-23T12:23:59.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.062+0000 7fa036000700 1 -- 172.21.15.63:0/1185372361 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fa04010bfb0 con 0x7fa02406c7d0 2024-03-23T12:23:59.040 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:23:59.040 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (33m) 10m ago 46m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (30m) 10m ago 46m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (30m) 10m ago 45m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (32m) 10m ago 46m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (26m) 10m ago 40m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (26m) 10m ago 39m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (10m) 10m ago 39m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (25m) 10m ago 40m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (35m) 10m ago 47m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (34m) 10m ago 45m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (31m) 10m ago 48m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (31m) 10m ago 45m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (34m) 10m ago 46m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (33m) 10m ago 45m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:23:59.041 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (29m) 10m ago 44m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:23:59.042 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (29m) 10m ago 43m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:23:59.042 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (28m) 10m ago 43m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:23:59.042 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (27m) 10m ago 42m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:23:59.042 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (27m) 10m ago 42m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:23:59.042 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (26m) 10m ago 41m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:23:59.042 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (33m) 10m ago 45m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa02406c7d0 msgr2=0x7fa02406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa02406c7d0 0x7fa02406ec80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b370 tx=0x7fa030009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070c70 msgr2=0x7fa0401161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070c70 0x7fa0401161c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa028002a80 tx=0x7fa0280046b0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 shutdown_connections 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa02406c7d0 0x7fa02406ec80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa040070360 0x7fa040115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 --2- 172.21.15.63:0/1185372361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa040070c70 0x7fa0401161c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 >> 172.21.15.63:0/1185372361 conn(0x7fa0400f89b0 msgr2=0x7fa0401015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:59.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 shutdown_connections 2024-03-23T12:23:59.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.064+0000 7fa044c00700 1 -- 172.21.15.63:0/1185372361 wait complete. 2024-03-23T12:23:59.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:58 smithi063 ceph-mon[141124]: pgmap v986: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:59.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:58 smithi111 ceph-mon[106899]: pgmap v986: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:23:59.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.200+0000 7fdb16400700 1 -- 172.21.15.63:0/2116341873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb101037a0 msgr2=0x7fdb100fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.200+0000 7fdb16400700 1 --2- 172.21.15.63:0/2116341873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb101037a0 0x7fdb100fce10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fdafc004940 tx=0x7fdafc009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:59.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.200+0000 7fdb16400700 1 -- 172.21.15.63:0/2116341873 shutdown_connections 2024-03-23T12:23:59.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.200+0000 7fdb16400700 1 --2- 172.21.15.63:0/2116341873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb101037a0 0x7fdb100fce10 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.200+0000 7fdb16400700 1 --2- 172.21.15.63:0/2116341873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb10102e00 0x7fdb101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.200+0000 7fdb16400700 1 -- 172.21.15.63:0/2116341873 >> 172.21.15.63:0/2116341873 conn(0x7fdb100f89b0 msgr2=0x7fdb100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:59.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.201+0000 7fdb16400700 1 -- 172.21.15.63:0/2116341873 shutdown_connections 2024-03-23T12:23:59.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.201+0000 7fdb16400700 1 -- 172.21.15.63:0/2116341873 wait complete. 2024-03-23T12:23:59.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.201+0000 7fdb16400700 1 Processor -- start 2024-03-23T12:23:59.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb16400700 1 -- start start 2024-03-23T12:23:59.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb16400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb10102e00 0x7fdb10113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:59.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb16400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb101037a0 0x7fdb10114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb16400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb101146b0 con 0x7fdb10102e00 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb16400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1010daa0 con 0x7fdb101037a0 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb10102e00 0x7fdb10113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb10102e00 0x7fdb10113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43978/0 (socket says 172.21.15.63:43978) 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb0f400700 1 -- 172.21.15.63:0/3245592001 learned_addr learned my addr 172.21.15.63:0/3245592001 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.202+0000 7fdb0ea00700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb101037a0 0x7fdb10114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb0f400700 1 -- 172.21.15.63:0/3245592001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb101037a0 msgr2=0x7fdb10114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb0f400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb101037a0 0x7fdb10114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb0f400700 1 -- 172.21.15.63:0/3245592001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc005040 con 0x7fdb10102e00 2024-03-23T12:23:59.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb0f400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb10102e00 0x7fdb10113b20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdb00002a50 tx=0x7fdb000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:59.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb07400700 1 -- 172.21.15.63:0/3245592001 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb00007cc0 con 0x7fdb10102e00 2024-03-23T12:23:59.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb07400700 1 -- 172.21.15.63:0/3245592001 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb0000f040 con 0x7fdb10102e00 2024-03-23T12:23:59.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb07400700 1 -- 172.21.15.63:0/3245592001 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb0000d9c0 con 0x7fdb10102e00 2024-03-23T12:23:59.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1010dd80 con 0x7fdb10102e00 2024-03-23T12:23:59.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.203+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1010e310 con 0x7fdb10102e00 2024-03-23T12:23:59.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.204+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb100fe550 con 0x7fdb10102e00 2024-03-23T12:23:59.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.205+0000 7fdb07400700 1 -- 172.21.15.63:0/3245592001 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdb0000db60 con 0x7fdb10102e00 2024-03-23T12:23:59.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.206+0000 7fdb07400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaf406c7d0 0x7fdaf406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:59.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.206+0000 7fdb07400700 1 -- 172.21.15.63:0/3245592001 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdb0008b0c0 con 0x7fdb10102e00 2024-03-23T12:23:59.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.207+0000 7fdb0ea00700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaf406c7d0 0x7fdaf406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.207+0000 7fdb0ea00700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaf406c7d0 0x7fdaf406ec80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdb1010ed80 tx=0x7fdafc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:59.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.211+0000 7fdb07400700 1 -- 172.21.15.63:0/3245592001 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdb00058d50 con 0x7fdb10102e00 2024-03-23T12:23:59.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.508+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdb1004bed0 con 0x7fdb10102e00 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.508+0000 7fdb07400700 1 -- 172.21.15.63:0/3245592001 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fdb000588e0 con 0x7fdb10102e00 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:23:59.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:23:59.488 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:23:59.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.510+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaf406c7d0 msgr2=0x7fdaf406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.510+0000 7fdb16400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaf406c7d0 0x7fdaf406ec80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdb1010ed80 tx=0x7fdafc009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:59.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.510+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb10102e00 msgr2=0x7fdb10113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.510+0000 7fdb16400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb10102e00 0x7fdb10113b20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdb00002a50 tx=0x7fdb000046b0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.511+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 shutdown_connections 2024-03-23T12:23:59.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.511+0000 7fdb16400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdaf406c7d0 0x7fdaf406ec80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.511+0000 7fdb16400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb10102e00 0x7fdb10113b20 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.511+0000 7fdb16400700 1 --2- 172.21.15.63:0/3245592001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb101037a0 0x7fdb10114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.511+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 >> 172.21.15.63:0/3245592001 conn(0x7fdb100f89b0 msgr2=0x7fdb100f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:59.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.511+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 shutdown_connections 2024-03-23T12:23:59.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.511+0000 7fdb16400700 1 -- 172.21.15.63:0/3245592001 wait complete. 2024-03-23T12:23:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.649+0000 7fad3d200700 1 -- 172.21.15.63:0/2230973457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 msgr2=0x7fad380fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.649+0000 7fad3d200700 1 --2- 172.21.15.63:0/2230973457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 0x7fad380fce10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fad24004940 tx=0x7fad24009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.650+0000 7fad3d200700 1 -- 172.21.15.63:0/2230973457 shutdown_connections 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.650+0000 7fad3d200700 1 --2- 172.21.15.63:0/2230973457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 0x7fad380fce10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.650+0000 7fad3d200700 1 --2- 172.21.15.63:0/2230973457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 0x7fad381031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.650+0000 7fad3d200700 1 -- 172.21.15.63:0/2230973457 >> 172.21.15.63:0/2230973457 conn(0x7fad380f89b0 msgr2=0x7fad380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.650+0000 7fad3d200700 1 -- 172.21.15.63:0/2230973457 shutdown_connections 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.650+0000 7fad3d200700 1 -- 172.21.15.63:0/2230973457 wait complete. 2024-03-23T12:23:59.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.651+0000 7fad3d200700 1 Processor -- start 2024-03-23T12:23:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.651+0000 7fad3d200700 1 -- start start 2024-03-23T12:23:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad3d200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 0x7fad38113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad3d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 0x7fad38114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad3d200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad38114690 con 0x7fad381037a0 2024-03-23T12:23:59.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad3d200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3810da80 con 0x7fad38102e00 2024-03-23T12:23:59.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad35e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 0x7fad38113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:59.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad35e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 0x7fad38113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.63:59460/0 (socket says 172.21.15.63:59460) 2024-03-23T12:23:59.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad35e00700 1 -- 172.21.15.63:0/2687404374 learned_addr learned my addr 172.21.15.63:0/2687404374 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:23:59.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.652+0000 7fad35400700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 0x7fad38114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:59.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad35400700 1 -- 172.21.15.63:0/2687404374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 msgr2=0x7fad38113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad35400700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 0x7fad38113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad35400700 1 -- 172.21.15.63:0/2687404374 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad24005040 con 0x7fad381037a0 2024-03-23T12:23:59.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad35e00700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 0x7fad38113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T12:23:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad35400700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 0x7fad38114040 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fad38067190 tx=0x7fad24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad2e000700 1 -- 172.21.15.63:0/2687404374 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad2400e040 con 0x7fad381037a0 2024-03-23T12:23:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad2e000700 1 -- 172.21.15.63:0/2687404374 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad24004290 con 0x7fad381037a0 2024-03-23T12:23:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3810dd00 con 0x7fad381037a0 2024-03-23T12:23:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.653+0000 7fad2e000700 1 -- 172.21.15.63:0/2687404374 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad24013d60 con 0x7fad381037a0 2024-03-23T12:23:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.654+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3810e2c0 con 0x7fad381037a0 2024-03-23T12:23:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.655+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad380fe550 con 0x7fad381037a0 2024-03-23T12:23:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.656+0000 7fad2e000700 1 -- 172.21.15.63:0/2687404374 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fad240135b0 con 0x7fad381037a0 2024-03-23T12:23:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.657+0000 7fad2e000700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:23:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.657+0000 7fad2e000700 1 -- 172.21.15.63:0/2687404374 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fad2408cb50 con 0x7fad381037a0 2024-03-23T12:23:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.657+0000 7fad35e00700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:23:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.658+0000 7fad35e00700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fad280041d0 tx=0x7fad2800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:23:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.661+0000 7fad2e000700 1 -- 172.21.15.63:0/2687404374 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fad2405a860 con 0x7fad381037a0 2024-03-23T12:23:59.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.918+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fad3805f5f0 con 0x7fad381037a0 2024-03-23T12:23:59.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.919+0000 7fad2e000700 1 -- 172.21.15.63:0/2687404374 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fad2405a3f0 con 0x7fad381037a0 2024-03-23T12:23:59.897 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:23:59.898 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:23:59.899 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:23:59.900 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:59.901 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:59.901 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:59.901 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:23:59.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 msgr2=0x7fad1c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fad280041d0 tx=0x7fad2800b040 comp rx=0 tx=0).stop 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 msgr2=0x7fad38114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 0x7fad38114040 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fad38067190 tx=0x7fad24009e30 comp rx=0 tx=0).stop 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 shutdown_connections 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fad1c06c7d0 0x7fad1c06ec80 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad38102e00 0x7fad38113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 --2- 172.21.15.63:0/2687404374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad381037a0 0x7fad38114040 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 >> 172.21.15.63:0/2687404374 conn(0x7fad380f89b0 msgr2=0x7fad380f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 shutdown_connections 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:23:59.921+0000 7fad3d200700 1 -- 172.21.15.63:0/2687404374 wait complete. 2024-03-23T12:23:59.902 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 -- 172.21.15.63:0/1013983255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 msgr2=0x7f27601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 --2- 172.21.15.63:0/1013983255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f27601031d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2750004940 tx=0x7f2750009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 -- 172.21.15.63:0/1013983255 shutdown_connections 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 --2- 172.21.15.63:0/1013983255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601037a0 0x7f27600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 --2- 172.21.15.63:0/1013983255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f27601031d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 -- 172.21.15.63:0/1013983255 >> 172.21.15.63:0/1013983255 conn(0x7f27600f89b0 msgr2=0x7f27600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 -- 172.21.15.63:0/1013983255 shutdown_connections 2024-03-23T12:24:00.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.059+0000 7f2767000700 1 -- 172.21.15.63:0/1013983255 wait complete. 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.060+0000 7f2767000700 1 Processor -- start 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.060+0000 7f2767000700 1 -- start start 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.060+0000 7f2767000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f2760115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.060+0000 7f2767000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601037a0 0x7f2760116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.060+0000 7f2767000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27601168e0 con 0x7f2760102e00 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.060+0000 7f2767000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276010fc20 con 0x7f27601037a0 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f2760115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f2760115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44020/0 (socket says 172.21.15.63:44020) 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275fe00700 1 -- 172.21.15.63:0/3400396245 learned_addr learned my addr 172.21.15.63:0/3400396245 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275f400700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601037a0 0x7f2760116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:00.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275fe00700 1 -- 172.21.15.63:0/3400396245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601037a0 msgr2=0x7f2760116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275fe00700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601037a0 0x7f2760116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275fe00700 1 -- 172.21.15.63:0/3400396245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750005040 con 0x7f2760102e00 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f275fe00700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f2760115cc0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2750009360 tx=0x7f2750009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.061+0000 7f2757e00700 1 -- 172.21.15.63:0/3400396245 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f275000e040 con 0x7f2760102e00 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.062+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276010fe40 con 0x7f2760102e00 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.062+0000 7f2757e00700 1 -- 172.21.15.63:0/3400396245 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2750013420 con 0x7f2760102e00 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.062+0000 7f2757e00700 1 -- 172.21.15.63:0/3400396245 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f275001c5f0 con 0x7f2760102e00 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.062+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2760110400 con 0x7f2760102e00 2024-03-23T12:24:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.063+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27600fe550 con 0x7f2760102e00 2024-03-23T12:24:00.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.064+0000 7f2757e00700 1 -- 172.21.15.63:0/3400396245 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2750013590 con 0x7f2760102e00 2024-03-23T12:24:00.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.065+0000 7f2757e00700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f274406c7d0 0x7f274406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:00.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.065+0000 7f2757e00700 1 -- 172.21.15.63:0/3400396245 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f275008cc90 con 0x7f2760102e00 2024-03-23T12:24:00.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.065+0000 7f275f400700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f274406c7d0 0x7f274406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:00.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.066+0000 7f275f400700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f274406c7d0 0x7f274406ec80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f274c006fd0 tx=0x7f274c008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:00.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.069+0000 7f2757e00700 1 -- 172.21.15.63:0/3400396245 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f275005aa50 con 0x7f2760102e00 2024-03-23T12:24:00.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:59 smithi063 ceph-mon[141124]: from='client.35782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:00.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:59 smithi063 ceph-mon[141124]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:00.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:59 smithi063 ceph-mon[141124]: from='client.35790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:00.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:23:59 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3245592001' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:24:00.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:59 smithi111 ceph-mon[106899]: from='client.35782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:00.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:59 smithi111 ceph-mon[106899]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:00.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:59 smithi111 ceph-mon[106899]: from='client.35790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:00.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:23:59 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3245592001' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:24:00.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.306+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f276004bed0 con 0x7f274406c7d0 2024-03-23T12:24:00.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.307+0000 7f2757e00700 1 -- 172.21.15.63:0/3400396245 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f276004bed0 con 0x7f274406c7d0 2024-03-23T12:24:00.286 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:24:00.286 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:24:00.286 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:24:00.286 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:24:00.286 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:24:00.286 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:24:00.287 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:24:00.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.309+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f274406c7d0 msgr2=0x7f274406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.309+0000 7f2767000700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f274406c7d0 0x7f274406ec80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f274c006fd0 tx=0x7f274c008040 comp rx=0 tx=0).stop 2024-03-23T12:24:00.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.309+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 msgr2=0x7f2760115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.309+0000 7f2767000700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f2760115cc0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2750009360 tx=0x7f2750009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:00.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.310+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 shutdown_connections 2024-03-23T12:24:00.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.310+0000 7f2767000700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f274406c7d0 0x7f274406ec80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.310+0000 7f2767000700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2760102e00 0x7f2760115cc0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.310+0000 7f2767000700 1 --2- 172.21.15.63:0/3400396245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601037a0 0x7f2760116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.310+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 >> 172.21.15.63:0/3400396245 conn(0x7f27600f89b0 msgr2=0x7f2760071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:00.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.310+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 shutdown_connections 2024-03-23T12:24:00.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.310+0000 7f2767000700 1 -- 172.21.15.63:0/3400396245 wait complete. 2024-03-23T12:24:00.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 -- 172.21.15.63:0/2757345075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 msgr2=0x7f58a0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 --2- 172.21.15.63:0/2757345075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a0106f20 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f588c004970 tx=0x7f588c009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:00.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 -- 172.21.15.63:0/2757345075 shutdown_connections 2024-03-23T12:24:00.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 --2- 172.21.15.63:0/2757345075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a0106f20 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 --2- 172.21.15.63:0/2757345075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a00fdeb0 0x7f58a00fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 -- 172.21.15.63:0/2757345075 >> 172.21.15.63:0/2757345075 conn(0x7f58a0072bc0 msgr2=0x7f58a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 -- 172.21.15.63:0/2757345075 shutdown_connections 2024-03-23T12:24:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.446+0000 7f58a5a00700 1 -- 172.21.15.63:0/2757345075 wait complete. 2024-03-23T12:24:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.447+0000 7f58a5a00700 1 Processor -- start 2024-03-23T12:24:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.447+0000 7f58a5a00700 1 -- start start 2024-03-23T12:24:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f58a5a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a00fdeb0 0x7f58a0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f58a5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f58a5a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a0116810 con 0x7f58a00fe850 2024-03-23T12:24:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f58a5a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a010fc20 con 0x7f58a00fdeb0 2024-03-23T12:24:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f589e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f589e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a01161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44046/0 (socket says 172.21.15.63:44046) 2024-03-23T12:24:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f589e000700 1 -- 172.21.15.63:0/3240608241 learned_addr learned my addr 172.21.15.63:0/3240608241 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f589ea00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a00fdeb0 0x7f58a0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f589e000700 1 -- 172.21.15.63:0/3240608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a00fdeb0 msgr2=0x7f58a0115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f589e000700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a00fdeb0 0x7f58a0115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.448+0000 7f589e000700 1 -- 172.21.15.63:0/3240608241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c005040 con 0x7f58a00fe850 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.449+0000 7f589e000700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a01161c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f588c004f50 tx=0x7f588c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.449+0000 7f5896a00700 1 -- 172.21.15.63:0/3240608241 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f588c00e040 con 0x7f58a00fe850 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.449+0000 7f5896a00700 1 -- 172.21.15.63:0/3240608241 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f588c013420 con 0x7f58a00fe850 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.449+0000 7f5896a00700 1 -- 172.21.15.63:0/3240608241 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f588c01c630 con 0x7f58a00fe850 2024-03-23T12:24:00.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.449+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a010fea0 con 0x7f58a00fe850 2024-03-23T12:24:00.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.449+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a01103e0 con 0x7f58a00fe850 2024-03-23T12:24:00.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.450+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a01069f0 con 0x7f58a00fe850 2024-03-23T12:24:00.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.451+0000 7f5896a00700 1 -- 172.21.15.63:0/3240608241 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f588c013590 con 0x7f58a00fe850 2024-03-23T12:24:00.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.452+0000 7f5896a00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f588406c7d0 0x7f588406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:00.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.452+0000 7f5896a00700 1 -- 172.21.15.63:0/3240608241 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f588c08cc00 con 0x7f58a00fe850 2024-03-23T12:24:00.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.452+0000 7f589ea00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f588406c7d0 0x7f588406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:00.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.453+0000 7f589ea00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f588406c7d0 0x7f588406ec80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5890004f60 tx=0x7f589000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:00.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.456+0000 7f5896a00700 1 -- 172.21.15.63:0/3240608241 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f588c056e80 con 0x7f58a00fe850 2024-03-23T12:24:00.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.751+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f58a0111310 con 0x7f58a00fe850 2024-03-23T12:24:00.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.751+0000 7f5896a00700 1 -- 172.21.15.63:0/3240608241 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f588c05a4a0 con 0x7f58a00fe850 2024-03-23T12:24:00.730 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:24:00.730 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:24:00.731 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:24:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:24:00.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f588406c7d0 msgr2=0x7f588406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f588406c7d0 0x7f588406ec80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5890004f60 tx=0x7f589000b040 comp rx=0 tx=0).stop 2024-03-23T12:24:00.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 msgr2=0x7f58a01161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:00.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a01161c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f588c004f50 tx=0x7f588c009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:00.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 shutdown_connections 2024-03-23T12:24:00.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f588406c7d0 0x7f588406ec80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58a00fdeb0 0x7f58a0115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 --2- 172.21.15.63:0/3240608241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58a00fe850 0x7f58a01161c0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:00.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 >> 172.21.15.63:0/3240608241 conn(0x7f58a0072bc0 msgr2=0x7f58a00fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:00.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 shutdown_connections 2024-03-23T12:24:00.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:00.754+0000 7f58a5a00700 1 -- 172.21.15.63:0/3240608241 wait complete. 2024-03-23T12:24:01.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:00 smithi063 ceph-mon[141124]: pgmap v987: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:01.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:00 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2687404374' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:24:01.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:00 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3240608241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:24:01.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:00 smithi111 ceph-mon[106899]: pgmap v987: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:01.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:00 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2687404374' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:24:01.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:00 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3240608241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:24:02.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:01 smithi063 ceph-mon[141124]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:02.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:01 smithi111 ceph-mon[106899]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:03.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:02 smithi063 ceph-mon[141124]: pgmap v988: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:03.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:02 smithi111 ceph-mon[106899]: pgmap v988: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:05.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:04 smithi063 ceph-mon[141124]: pgmap v989: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:05.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:04 smithi111 ceph-mon[106899]: pgmap v989: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:07.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:06 smithi063 ceph-mon[141124]: pgmap v990: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:07.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:06 smithi111 ceph-mon[106899]: pgmap v990: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:09.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:08 smithi063 ceph-mon[141124]: pgmap v991: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:09.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:08 smithi111 ceph-mon[106899]: pgmap v991: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:11.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:10 smithi063 ceph-mon[141124]: pgmap v992: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:11.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:11.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:10 smithi111 ceph-mon[106899]: pgmap v992: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:13.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:12 smithi063 ceph-mon[141124]: pgmap v993: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:13.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:12 smithi111 ceph-mon[106899]: pgmap v993: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:15.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:14 smithi063 ceph-mon[141124]: pgmap v994: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:15.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:14 smithi111 ceph-mon[106899]: pgmap v994: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:17.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:16 smithi063 ceph-mon[141124]: pgmap v995: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:17.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:16 smithi111 ceph-mon[106899]: pgmap v995: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:19.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:18 smithi063 ceph-mon[141124]: pgmap v996: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:19.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:18 smithi111 ceph-mon[106899]: pgmap v996: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:21.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:20 smithi063 ceph-mon[141124]: pgmap v997: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:21.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:20 smithi111 ceph-mon[106899]: pgmap v997: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:23.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:22 smithi063 ceph-mon[141124]: pgmap v998: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:23.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:22 smithi111 ceph-mon[106899]: pgmap v998: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:25.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:24 smithi063 ceph-mon[141124]: pgmap v999: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:25.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:24 smithi111 ceph-mon[106899]: pgmap v999: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:26 smithi063 ceph-mon[141124]: pgmap v1000: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:27.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:26 smithi111 ceph-mon[106899]: pgmap v1000: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:29.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:28 smithi063 ceph-mon[141124]: pgmap v1001: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:29.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:28 smithi111 ceph-mon[106899]: pgmap v1001: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:30.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 -- 172.21.15.63:0/2807816868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 msgr2=0x7fc90c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:30.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 --2- 172.21.15.63:0/2807816868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c1031d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc900004940 tx=0x7fc900009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:30.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 -- 172.21.15.63:0/2807816868 shutdown_connections 2024-03-23T12:24:30.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 --2- 172.21.15.63:0/2807816868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc90c1037a0 0x7fc90c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:30.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 --2- 172.21.15.63:0/2807816868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c1031d0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:30.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 -- 172.21.15.63:0/2807816868 >> 172.21.15.63:0/2807816868 conn(0x7fc90c0f89b0 msgr2=0x7fc90c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:30.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 -- 172.21.15.63:0/2807816868 shutdown_connections 2024-03-23T12:24:30.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.908+0000 7fc914800700 1 -- 172.21.15.63:0/2807816868 wait complete. 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.909+0000 7fc914800700 1 Processor -- start 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.909+0000 7fc914800700 1 -- start start 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc914800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc914800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc90c1037a0 0x7fc90c116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc914800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c1167e0 con 0x7fc90c102e00 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc914800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c10fbb0 con 0x7fc90c1037a0 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc911e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:30.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc911e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56628/0 (socket says 172.21.15.63:56628) 2024-03-23T12:24:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc911e00700 1 -- 172.21.15.63:0/1355737718 learned_addr learned my addr 172.21.15.63:0/1355737718 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc911400700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc90c1037a0 0x7fc90c116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc911e00700 1 -- 172.21.15.63:0/1355737718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc90c1037a0 msgr2=0x7fc90c116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc911e00700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc90c1037a0 0x7fc90c116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.910+0000 7fc911e00700 1 -- 172.21.15.63:0/1355737718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900005040 con 0x7fc90c102e00 2024-03-23T12:24:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.911+0000 7fc911e00700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c115c50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc900007f50 tx=0x7fc900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.911+0000 7fc906000700 1 -- 172.21.15.63:0/1355737718 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc900021070 con 0x7fc90c102e00 2024-03-23T12:24:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.911+0000 7fc906000700 1 -- 172.21.15.63:0/1355737718 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc900013e00 con 0x7fc90c102e00 2024-03-23T12:24:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.911+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c10fe30 con 0x7fc90c102e00 2024-03-23T12:24:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.911+0000 7fc906000700 1 -- 172.21.15.63:0/1355737718 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc90000e070 con 0x7fc90c102e00 2024-03-23T12:24:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.911+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c1103f0 con 0x7fc90c102e00 2024-03-23T12:24:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.912+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c0fe550 con 0x7fc90c102e00 2024-03-23T12:24:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.913+0000 7fc906000700 1 -- 172.21.15.63:0/1355737718 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc900013950 con 0x7fc90c102e00 2024-03-23T12:24:30.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.914+0000 7fc906000700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc8f406c7d0 0x7fc8f406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:30.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.914+0000 7fc906000700 1 -- 172.21.15.63:0/1355737718 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc90008cc90 con 0x7fc90c102e00 2024-03-23T12:24:30.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.914+0000 7fc911400700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc8f406c7d0 0x7fc8f406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:30.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.915+0000 7fc911400700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc8f406c7d0 0x7fc8f406ec80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc004600 tx=0x7fc8fc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:30.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:30.919+0000 7fc906000700 1 -- 172.21.15.63:0/1355737718 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc90005a9a0 con 0x7fc90c102e00 2024-03-23T12:24:31.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:30 smithi063 ceph-mon[141124]: pgmap v1002: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:31.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.153+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc90c04bf40 con 0x7fc8f406c7d0 2024-03-23T12:24:31.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.154+0000 7fc906000700 1 -- 172.21.15.63:0/1355737718 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc90c04bf40 con 0x7fc8f406c7d0 2024-03-23T12:24:31.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc8f406c7d0 msgr2=0x7fc8f406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc8f406c7d0 0x7fc8f406ec80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc004600 tx=0x7fc8fc00b040 comp rx=0 tx=0).stop 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 msgr2=0x7fc90c115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c115c50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc900007f50 tx=0x7fc900009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 shutdown_connections 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc8f406c7d0 0x7fc8f406ec80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c102e00 0x7fc90c115c50 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 --2- 172.21.15.63:0/1355737718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc90c1037a0 0x7fc90c116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.156+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 >> 172.21.15.63:0/1355737718 conn(0x7fc90c0f89b0 msgr2=0x7fc90c0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.157+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 shutdown_connections 2024-03-23T12:24:31.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.157+0000 7fc914800700 1 -- 172.21.15.63:0/1355737718 wait complete. 2024-03-23T12:24:31.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:30 smithi111 ceph-mon[106899]: pgmap v1002: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:31.150 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:24:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 -- 172.21.15.63:0/4223645308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 msgr2=0x7feb141031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 --2- 172.21.15.63:0/4223645308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb141031d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7feb08004940 tx=0x7feb08009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 -- 172.21.15.63:0/4223645308 shutdown_connections 2024-03-23T12:24:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 --2- 172.21.15.63:0/4223645308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb141037a0 0x7feb140fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 --2- 172.21.15.63:0/4223645308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb141031d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 -- 172.21.15.63:0/4223645308 >> 172.21.15.63:0/4223645308 conn(0x7feb140f89b0 msgr2=0x7feb140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:31.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 -- 172.21.15.63:0/4223645308 shutdown_connections 2024-03-23T12:24:31.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.296+0000 7feb1bc00700 1 -- 172.21.15.63:0/4223645308 wait complete. 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.297+0000 7feb1bc00700 1 Processor -- start 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb1bc00700 1 -- start start 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb1bc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb14115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb1bc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb141037a0 0x7feb141161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb1bc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb141167f0 con 0x7feb14102e00 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb1bc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1410fbc0 con 0x7feb141037a0 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb19200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb14115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb19200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb14115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56644/0 (socket says 172.21.15.63:56644) 2024-03-23T12:24:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb19200700 1 -- 172.21.15.63:0/976676680 learned_addr learned my addr 172.21.15.63:0/976676680 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.298+0000 7feb13e00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb141037a0 0x7feb141161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb19200700 1 -- 172.21.15.63:0/976676680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb141037a0 msgr2=0x7feb141161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb19200700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb141037a0 0x7feb141161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb19200700 1 -- 172.21.15.63:0/976676680 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb08005040 con 0x7feb14102e00 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb19200700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb14115c60 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7feb08007f50 tx=0x7feb08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb10c00700 1 -- 172.21.15.63:0/976676680 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb08021070 con 0x7feb14102e00 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb10c00700 1 -- 172.21.15.63:0/976676680 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb08013e00 con 0x7feb14102e00 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1410fe40 con 0x7feb14102e00 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb10c00700 1 -- 172.21.15.63:0/976676680 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb0800e070 con 0x7feb14102e00 2024-03-23T12:24:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.299+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb14110400 con 0x7feb14102e00 2024-03-23T12:24:31.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.300+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb140fe550 con 0x7feb14102e00 2024-03-23T12:24:31.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.301+0000 7feb10c00700 1 -- 172.21.15.63:0/976676680 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7feb08013950 con 0x7feb14102e00 2024-03-23T12:24:31.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.302+0000 7feb10c00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feafc06c7d0 0x7feafc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:31.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.302+0000 7feb10c00700 1 -- 172.21.15.63:0/976676680 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb0808cd90 con 0x7feb14102e00 2024-03-23T12:24:31.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.302+0000 7feb13e00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feafc06c7d0 0x7feafc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:31.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.303+0000 7feb13e00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feafc06c7d0 0x7feafc06ec80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7feb00004600 tx=0x7feb0000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:31.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.306+0000 7feb10c00700 1 -- 172.21.15.63:0/976676680 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb0805aa20 con 0x7feb14102e00 2024-03-23T12:24:31.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.543+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb1404bed0 con 0x7feafc06c7d0 2024-03-23T12:24:31.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.544+0000 7feb10c00700 1 -- 172.21.15.63:0/976676680 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feb1404bed0 con 0x7feafc06c7d0 2024-03-23T12:24:31.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feafc06c7d0 msgr2=0x7feafc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feafc06c7d0 0x7feafc06ec80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7feb00004600 tx=0x7feb0000b040 comp rx=0 tx=0).stop 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 msgr2=0x7feb14115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb14115c60 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7feb08007f50 tx=0x7feb08009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 shutdown_connections 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feafc06c7d0 0x7feafc06ec80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb14102e00 0x7feb14115c60 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 --2- 172.21.15.63:0/976676680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb141037a0 0x7feb141161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 >> 172.21.15.63:0/976676680 conn(0x7feb140f89b0 msgr2=0x7feb140f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 shutdown_connections 2024-03-23T12:24:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.547+0000 7feb1bc00700 1 -- 172.21.15.63:0/976676680 wait complete. 2024-03-23T12:24:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.685+0000 7f2abba00700 1 -- 172.21.15.63:0/525783445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fde90 msgr2=0x7f2ab40fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.685+0000 7f2abba00700 1 --2- 172.21.15.63:0/525783445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fde90 0x7f2ab40fe260 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8004940 tx=0x7f2aa8009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.686+0000 7f2abba00700 1 -- 172.21.15.63:0/525783445 shutdown_connections 2024-03-23T12:24:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.686+0000 7f2abba00700 1 --2- 172.21.15.63:0/525783445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ab40fe830 0x7f2ab4106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.686+0000 7f2abba00700 1 --2- 172.21.15.63:0/525783445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fde90 0x7f2ab40fe260 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.686+0000 7f2abba00700 1 -- 172.21.15.63:0/525783445 >> 172.21.15.63:0/525783445 conn(0x7f2ab4072bc0 msgr2=0x7f2ab4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:31.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.686+0000 7f2abba00700 1 -- 172.21.15.63:0/525783445 shutdown_connections 2024-03-23T12:24:31.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.686+0000 7f2abba00700 1 -- 172.21.15.63:0/525783445 wait complete. 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2abba00700 1 Processor -- start 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2abba00700 1 -- start start 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2abba00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ab40fde90 0x7f2ab4115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2abba00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fe830 0x7f2ab4116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2abba00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab41168b0 con 0x7f2ab40fe830 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2abba00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab410fc80 con 0x7f2ab40fde90 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2ab3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fe830 0x7f2ab4116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2ab3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fe830 0x7f2ab4116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54660/0 (socket says 172.21.15.63:54660) 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2ab3e00700 1 -- 172.21.15.63:0/3978919745 learned_addr learned my addr 172.21.15.63:0/3978919745 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.687+0000 7f2ab9000700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ab40fde90 0x7f2ab4115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2ab3e00700 1 -- 172.21.15.63:0/3978919745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ab40fde90 msgr2=0x7f2ab4115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2ab3e00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ab40fde90 0x7f2ab4115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2ab3e00700 1 -- 172.21.15.63:0/3978919745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8005040 con 0x7f2ab40fe830 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2ab3e00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fe830 0x7f2ab4116260 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0002a50 tx=0x7f2aa00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2ab0c00700 1 -- 172.21.15.63:0/3978919745 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa0007cc0 con 0x7f2ab40fe830 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2ab0c00700 1 -- 172.21.15.63:0/3978919745 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aa000f040 con 0x7f2ab40fe830 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab410ff60 con 0x7f2ab40fe830 2024-03-23T12:24:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.688+0000 7f2ab0c00700 1 -- 172.21.15.63:0/3978919745 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa000d9c0 con 0x7f2ab40fe830 2024-03-23T12:24:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.689+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab41104a0 con 0x7f2ab40fe830 2024-03-23T12:24:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.690+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab4106a20 con 0x7f2ab40fe830 2024-03-23T12:24:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.691+0000 7f2ab0c00700 1 -- 172.21.15.63:0/3978919745 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2aa000db20 con 0x7f2ab40fe830 2024-03-23T12:24:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.691+0000 7f2ab0c00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2a9c06c7c0 0x7f2a9c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.692+0000 7f2ab0c00700 1 -- 172.21.15.63:0/3978919745 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2aa008b820 con 0x7f2ab40fe830 2024-03-23T12:24:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.692+0000 7f2ab9000700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2a9c06c7c0 0x7f2a9c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.692+0000 7f2ab9000700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2a9c06c7c0 0x7f2a9c06ec70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800b7e0 tx=0x7f2aa8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:31.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.696+0000 7f2ab0c00700 1 -- 172.21.15.63:0/3978919745 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2aa0059430 con 0x7f2ab40fe830 2024-03-23T12:24:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.918+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ab4109dc0 con 0x7f2a9c06c7c0 2024-03-23T12:24:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.928+0000 7f2ab0c00700 1 -- 172.21.15.63:0/3978919745 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f2ab4109dc0 con 0x7f2a9c06c7c0 2024-03-23T12:24:31.907 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:24:31.907 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (33m) 10m ago 46m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:24:31.907 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (31m) 10m ago 47m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:24:31.907 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (30m) 10m ago 46m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:24:31.907 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (33m) 10m ago 46m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:24:31.907 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (26m) 10m ago 40m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (26m) 10m ago 40m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (11m) 10m ago 40m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (26m) 10m ago 40m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (36m) 10m ago 48m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (34m) 10m ago 45m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (32m) 10m ago 48m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (31m) 10m ago 45m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (34m) 10m ago 47m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (34m) 10m ago 45m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (30m) 10m ago 44m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (29m) 10m ago 44m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (29m) 10m ago 43m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (28m) 10m ago 43m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (27m) 10m ago 42m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:24:31.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (27m) 10m ago 42m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:24:31.909 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (34m) 10m ago 46m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:24:31.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2a9c06c7c0 msgr2=0x7f2a9c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2a9c06c7c0 0x7f2a9c06ec70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800b7e0 tx=0x7f2aa8009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:31.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fe830 msgr2=0x7f2ab4116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fe830 0x7f2ab4116260 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0002a50 tx=0x7f2aa00046b0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 shutdown_connections 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2a9c06c7c0 0x7f2a9c06ec70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ab40fde90 0x7f2ab4115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 --2- 172.21.15.63:0/3978919745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ab40fe830 0x7f2ab4116260 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 >> 172.21.15.63:0/3978919745 conn(0x7f2ab4072bc0 msgr2=0x7f2ab40fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 shutdown_connections 2024-03-23T12:24:31.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:31.931+0000 7f2abba00700 1 -- 172.21.15.63:0/3978919745 wait complete. 2024-03-23T12:24:32.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.068+0000 7f9b90e00700 1 -- 172.21.15.63:0/1941963407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 msgr2=0x7f9b8c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.068+0000 7f9b90e00700 1 --2- 172.21.15.63:0/1941963407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c0fce10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9b78004940 tx=0x7f9b78009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:32.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.069+0000 7f9b90e00700 1 -- 172.21.15.63:0/1941963407 shutdown_connections 2024-03-23T12:24:32.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.069+0000 7f9b90e00700 1 --2- 172.21.15.63:0/1941963407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c0fce10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.069+0000 7f9b90e00700 1 --2- 172.21.15.63:0/1941963407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8c102e00 0x7f9b8c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.069+0000 7f9b90e00700 1 -- 172.21.15.63:0/1941963407 >> 172.21.15.63:0/1941963407 conn(0x7f9b8c0f89b0 msgr2=0x7f9b8c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:32.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.069+0000 7f9b90e00700 1 -- 172.21.15.63:0/1941963407 shutdown_connections 2024-03-23T12:24:32.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.069+0000 7f9b90e00700 1 -- 172.21.15.63:0/1941963407 wait complete. 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.069+0000 7f9b90e00700 1 Processor -- start 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b90e00700 1 -- start start 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b90e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8c102e00 0x7f9b8c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b90e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b90e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c116890 con 0x7f9b8c1037a0 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b90e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c10fc60 con 0x7f9b8c102e00 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b89400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b89400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54690/0 (socket says 172.21.15.63:54690) 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.070+0000 7f9b89400700 1 -- 172.21.15.63:0/2178257647 learned_addr learned my addr 172.21.15.63:0/2178257647 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:32.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b89e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8c102e00 0x7f9b8c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b89400700 1 -- 172.21.15.63:0/2178257647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8c102e00 msgr2=0x7f9b8c115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b89400700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8c102e00 0x7f9b8c115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b89400700 1 -- 172.21.15.63:0/2178257647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78005040 con 0x7f9b8c1037a0 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b89400700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c116240 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009390 tx=0x7f9b78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b82000700 1 -- 172.21.15.63:0/2178257647 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7800e040 con 0x7f9b8c1037a0 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b82000700 1 -- 172.21.15.63:0/2178257647 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b78013420 con 0x7f9b8c1037a0 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b82000700 1 -- 172.21.15.63:0/2178257647 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7801c630 con 0x7f9b8c1037a0 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c10fee0 con 0x7f9b8c1037a0 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.071+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c1104a0 con 0x7f9b8c1037a0 2024-03-23T12:24:32.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.073+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c0fe550 con 0x7f9b8c1037a0 2024-03-23T12:24:32.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.074+0000 7f9b82000700 1 -- 172.21.15.63:0/2178257647 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9b78013a00 con 0x7f9b8c1037a0 2024-03-23T12:24:32.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.074+0000 7f9b82000700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b7006c7d0 0x7f9b7006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.075+0000 7f9b82000700 1 -- 172.21.15.63:0/2178257647 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9b7808cd00 con 0x7f9b8c1037a0 2024-03-23T12:24:32.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.075+0000 7f9b89e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b7006c7d0 0x7f9b7006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.075+0000 7f9b89e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b7006c7d0 0x7f9b7006ec80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004990 tx=0x7f9b7c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:32.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.079+0000 7f9b82000700 1 -- 172.21.15.63:0/2178257647 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9b7805aa10 con 0x7f9b8c1037a0 2024-03-23T12:24:32.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:31 smithi063 ceph-mon[141124]: from='client.35810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:32.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:31 smithi111 ceph-mon[106899]: from='client.35810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:32.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.378+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9b8c04bed0 con 0x7f9b8c1037a0 2024-03-23T12:24:32.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.378+0000 7f9b82000700 1 -- 172.21.15.63:0/2178257647 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f9b7805a5a0 con 0x7f9b8c1037a0 2024-03-23T12:24:32.357 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:24:32.357 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:24:32.357 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:24:32.358 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:24:32.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b7006c7d0 msgr2=0x7f9b7006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b7006c7d0 0x7f9b7006ec80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004990 tx=0x7f9b7c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 msgr2=0x7f9b8c116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c116240 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009390 tx=0x7f9b78009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:32.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 shutdown_connections 2024-03-23T12:24:32.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9b7006c7d0 0x7f9b7006ec80 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8c102e00 0x7f9b8c115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 --2- 172.21.15.63:0/2178257647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b8c1037a0 0x7f9b8c116240 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 >> 172.21.15.63:0/2178257647 conn(0x7f9b8c0f89b0 msgr2=0x7f9b8c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:32.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.381+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 shutdown_connections 2024-03-23T12:24:32.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.382+0000 7f9b90e00700 1 -- 172.21.15.63:0/2178257647 wait complete. 2024-03-23T12:24:32.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.518+0000 7f9421600700 1 -- 172.21.15.63:0/1847582630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c102e00 msgr2=0x7f941c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.518+0000 7f9421600700 1 --2- 172.21.15.63:0/1847582630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c102e00 0x7f941c1031d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f940c004940 tx=0x7f940c009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:32.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.519+0000 7f9421600700 1 -- 172.21.15.63:0/1847582630 shutdown_connections 2024-03-23T12:24:32.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.519+0000 7f9421600700 1 --2- 172.21.15.63:0/1847582630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941c1037a0 0x7f941c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.519+0000 7f9421600700 1 --2- 172.21.15.63:0/1847582630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c102e00 0x7f941c1031d0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.519+0000 7f9421600700 1 -- 172.21.15.63:0/1847582630 >> 172.21.15.63:0/1847582630 conn(0x7f941c0f89b0 msgr2=0x7f941c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:32.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.519+0000 7f9421600700 1 -- 172.21.15.63:0/1847582630 shutdown_connections 2024-03-23T12:24:32.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.519+0000 7f9421600700 1 -- 172.21.15.63:0/1847582630 wait complete. 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.519+0000 7f9421600700 1 Processor -- start 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f9421600700 1 -- start start 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f9421600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941c102e00 0x7f941c115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f9421600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c1037a0 0x7f941c116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f9421600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c116880 con 0x7f941c1037a0 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f9421600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c10fc50 con 0x7f941c102e00 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f941a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c1037a0 0x7f941c116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f941a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c1037a0 0x7f941c116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54708/0 (socket says 172.21.15.63:54708) 2024-03-23T12:24:32.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f941a000700 1 -- 172.21.15.63:0/1469526157 learned_addr learned my addr 172.21.15.63:0/1469526157 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:32.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.520+0000 7f941aa00700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941c102e00 0x7f941c115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f941a000700 1 -- 172.21.15.63:0/1469526157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941c102e00 msgr2=0x7f941c115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f941a000700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941c102e00 0x7f941c115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f941a000700 1 -- 172.21.15.63:0/1469526157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c005040 con 0x7f941c1037a0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f941a000700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c1037a0 0x7f941c116230 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9408002a50 tx=0x7f9408004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f9412a00700 1 -- 172.21.15.63:0/1469526157 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9408007cc0 con 0x7f941c1037a0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f9412a00700 1 -- 172.21.15.63:0/1469526157 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f940800f040 con 0x7f941c1037a0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941c10ff30 con 0x7f941c1037a0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f9412a00700 1 -- 172.21.15.63:0/1469526157 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f940800d920 con 0x7f941c1037a0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.521+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941c1104f0 con 0x7f941c1037a0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.523+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941c0fe550 con 0x7f941c1037a0 2024-03-23T12:24:32.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.523+0000 7f9412a00700 1 -- 172.21.15.63:0/1469526157 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9408009680 con 0x7f941c1037a0 2024-03-23T12:24:32.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.524+0000 7f9412a00700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f940006c7d0 0x7f940006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.524+0000 7f9412a00700 1 -- 172.21.15.63:0/1469526157 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f940808b6d0 con 0x7f941c1037a0 2024-03-23T12:24:32.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.525+0000 7f941aa00700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f940006c7d0 0x7f940006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.525+0000 7f941aa00700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f940006c7d0 0x7f940006ec80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f940c009fd0 tx=0x7f940c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:32.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.529+0000 7f9412a00700 1 -- 172.21.15.63:0/1469526157 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f94080593e0 con 0x7f941c1037a0 2024-03-23T12:24:32.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.788+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f941c04bed0 con 0x7f941c1037a0 2024-03-23T12:24:32.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.789+0000 7f9412a00700 1 -- 172.21.15.63:0/1469526157 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f9408058f70 con 0x7f941c1037a0 2024-03-23T12:24:32.767 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:24:32.767 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:24:32.767 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:24:32.767 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:24:32.768 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:24:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:24:32.770 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f940006c7d0 msgr2=0x7f940006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f940006c7d0 0x7f940006ec80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f940c009fd0 tx=0x7f940c009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c1037a0 msgr2=0x7f941c116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c1037a0 0x7f941c116230 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9408002a50 tx=0x7f9408004600 comp rx=0 tx=0).stop 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 shutdown_connections 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f940006c7d0 0x7f940006ec80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941c102e00 0x7f941c115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 --2- 172.21.15.63:0/1469526157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f941c1037a0 0x7f941c116230 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.791+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 >> 172.21.15.63:0/1469526157 conn(0x7f941c0f89b0 msgr2=0x7f941c0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.792+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 shutdown_connections 2024-03-23T12:24:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.792+0000 7f9421600700 1 -- 172.21.15.63:0/1469526157 wait complete. 2024-03-23T12:24:32.772 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 -- 172.21.15.63:0/1349340938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fe830 msgr2=0x7fc650106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 --2- 172.21.15.63:0/1349340938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fe830 0x7fc650106f00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc63c004940 tx=0x7fc63c009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 -- 172.21.15.63:0/1349340938 shutdown_connections 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 --2- 172.21.15.63:0/1349340938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fe830 0x7fc650106f00 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 --2- 172.21.15.63:0/1349340938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6500fde90 0x7fc6500fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 -- 172.21.15.63:0/1349340938 >> 172.21.15.63:0/1349340938 conn(0x7fc650072bc0 msgr2=0x7fc650072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 -- 172.21.15.63:0/1349340938 shutdown_connections 2024-03-23T12:24:32.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.932+0000 7fc657800700 1 -- 172.21.15.63:0/1349340938 wait complete. 2024-03-23T12:24:32.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.933+0000 7fc657800700 1 Processor -- start 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.933+0000 7fc657800700 1 -- start start 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.933+0000 7fc657800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fde90 0x7fc6500fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.933+0000 7fc657800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6500fe830 0x7fc6500fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.933+0000 7fc657800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500fc820 con 0x7fc6500fde90 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.933+0000 7fc657800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500fc960 con 0x7fc6500fe830 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc654e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fde90 0x7fc6500fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc654e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fde90 0x7fc6500fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54728/0 (socket says 172.21.15.63:54728) 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc654e00700 1 -- 172.21.15.63:0/2017412765 learned_addr learned my addr 172.21.15.63:0/2017412765 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc64fe00700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6500fe830 0x7fc6500fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc654e00700 1 -- 172.21.15.63:0/2017412765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6500fe830 msgr2=0x7fc6500fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc654e00700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6500fe830 0x7fc6500fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc654e00700 1 -- 172.21.15.63:0/2017412765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c005040 con 0x7fc6500fde90 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc64fe00700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6500fe830 0x7fc6500fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:24:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc654e00700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fde90 0x7fc6500fbd50 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc64400eb40 tx=0x7fc64400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:32.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.934+0000 7fc64cc00700 1 -- 172.21.15.63:0/2017412765 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc644007cc0 con 0x7fc6500fde90 2024-03-23T12:24:32.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.935+0000 7fc64cc00700 1 -- 172.21.15.63:0/2017412765 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc64400f040 con 0x7fc6500fde90 2024-03-23T12:24:32.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.935+0000 7fc64cc00700 1 -- 172.21.15.63:0/2017412765 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc644009570 con 0x7fc6500fde90 2024-03-23T12:24:32.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.935+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6500fcbb0 con 0x7fc6500fde90 2024-03-23T12:24:32.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.935+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65011a130 con 0x7fc6500fde90 2024-03-23T12:24:32.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.936+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc650065e30 con 0x7fc6500fde90 2024-03-23T12:24:32.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.937+0000 7fc64cc00700 1 -- 172.21.15.63:0/2017412765 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc6440096d0 con 0x7fc6500fde90 2024-03-23T12:24:32.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.938+0000 7fc64cc00700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc63806c7d0 0x7fc63806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:32.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.938+0000 7fc64cc00700 1 -- 172.21.15.63:0/2017412765 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc64408b6e0 con 0x7fc6500fde90 2024-03-23T12:24:32.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.938+0000 7fc64fe00700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc63806c7d0 0x7fc63806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:32.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.939+0000 7fc64fe00700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc63806c7d0 0x7fc63806ec80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc63c000c00 tx=0x7fc63c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:32.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:32.942+0000 7fc64cc00700 1 -- 172.21.15.63:0/2017412765 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc644055990 con 0x7fc6500fde90 2024-03-23T12:24:33.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:32 smithi063 ceph-mon[141124]: pgmap v1003: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:32 smithi063 ceph-mon[141124]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:32 smithi063 ceph-mon[141124]: from='client.35818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:33.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:32 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2178257647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:24:33.130 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:32 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1469526157' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:24:33.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:32 smithi111 ceph-mon[106899]: pgmap v1003: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:33.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:32 smithi111 ceph-mon[106899]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:33.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:32 smithi111 ceph-mon[106899]: from='client.35818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:33.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:32 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2178257647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:24:33.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:32 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1469526157' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:24:33.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.178+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6500fd440 con 0x7fc63806c7d0 2024-03-23T12:24:33.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.179+0000 7fc64cc00700 1 -- 172.21.15.63:0/2017412765 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc6500fd440 con 0x7fc63806c7d0 2024-03-23T12:24:33.158 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:24:33.158 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:24:33.158 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:24:33.158 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:24:33.158 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:24:33.159 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:24:33.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc63806c7d0 msgr2=0x7fc63806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:33.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc63806c7d0 0x7fc63806ec80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc63c000c00 tx=0x7fc63c009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fde90 msgr2=0x7fc6500fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fde90 0x7fc6500fbd50 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc64400eb40 tx=0x7fc64400b040 comp rx=0 tx=0).stop 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 shutdown_connections 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc63806c7d0 0x7fc63806ec80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6500fde90 0x7fc6500fbd50 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 --2- 172.21.15.63:0/2017412765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6500fe830 0x7fc6500fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 >> 172.21.15.63:0/2017412765 conn(0x7fc650072bc0 msgr2=0x7fc6500ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 shutdown_connections 2024-03-23T12:24:33.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.182+0000 7fc657800700 1 -- 172.21.15.63:0/2017412765 wait complete. 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 -- 172.21.15.63:0/1674367189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fde90 msgr2=0x7f4a780fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 --2- 172.21.15.63:0/1674367189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fde90 0x7f4a780fe260 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4a68004940 tx=0x7f4a68009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 -- 172.21.15.63:0/1674367189 shutdown_connections 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 --2- 172.21.15.63:0/1674367189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a780fe830 0x7f4a78106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 --2- 172.21.15.63:0/1674367189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fde90 0x7f4a780fe260 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 -- 172.21.15.63:0/1674367189 >> 172.21.15.63:0/1674367189 conn(0x7f4a78072bc0 msgr2=0x7f4a78072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 -- 172.21.15.63:0/1674367189 shutdown_connections 2024-03-23T12:24:33.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.320+0000 7f4a7d200700 1 -- 172.21.15.63:0/1674367189 wait complete. 2024-03-23T12:24:33.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.321+0000 7f4a7d200700 1 Processor -- start 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.321+0000 7f4a7d200700 1 -- start start 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a7d200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a780fde90 0x7f4a78115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a7d200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fe830 0x7f4a78116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a7d200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a78116850 con 0x7f4a780fe830 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a7d200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7810fcb0 con 0x7f4a780fde90 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a75400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fe830 0x7f4a78116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a75400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fe830 0x7f4a78116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54748/0 (socket says 172.21.15.63:54748) 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a75400700 1 -- 172.21.15.63:0/2159248751 learned_addr learned my addr 172.21.15.63:0/2159248751 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:24:33.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.322+0000 7f4a75e00700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a780fde90 0x7f4a78115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:33.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a75400700 1 -- 172.21.15.63:0/2159248751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a780fde90 msgr2=0x7f4a78115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:33.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a75400700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a780fde90 0x7f4a78115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a75400700 1 -- 172.21.15.63:0/2159248751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68005040 con 0x7f4a780fe830 2024-03-23T12:24:33.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a75400700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fe830 0x7f4a78116200 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4a64002a50 tx=0x7f4a640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:33.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a6e000700 1 -- 172.21.15.63:0/2159248751 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a64007cc0 con 0x7f4a780fe830 2024-03-23T12:24:33.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a6e000700 1 -- 172.21.15.63:0/2159248751 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a6400f040 con 0x7f4a780fe830 2024-03-23T12:24:33.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7810ff90 con 0x7f4a780fe830 2024-03-23T12:24:33.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a6e000700 1 -- 172.21.15.63:0/2159248751 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a6400d9c0 con 0x7f4a780fe830 2024-03-23T12:24:33.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.323+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a781104d0 con 0x7f4a780fe830 2024-03-23T12:24:33.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.324+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a78106a20 con 0x7f4a780fe830 2024-03-23T12:24:33.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.325+0000 7f4a6e000700 1 -- 172.21.15.63:0/2159248751 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4a6400db60 con 0x7f4a780fe830 2024-03-23T12:24:33.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.327+0000 7f4a6e000700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a5c06c7d0 0x7f4a5c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:24:33.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.327+0000 7f4a6e000700 1 -- 172.21.15.63:0/2159248751 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4a6408b0c0 con 0x7f4a780fe830 2024-03-23T12:24:33.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.327+0000 7f4a75e00700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a5c06c7d0 0x7f4a5c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:24:33.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.327+0000 7f4a75e00700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a5c06c7d0 0x7f4a5c06ec80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800b3b0 tx=0x7f4a68009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:24:33.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.331+0000 7f4a6e000700 1 -- 172.21.15.63:0/2159248751 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a640552c0 con 0x7f4a780fe830 2024-03-23T12:24:33.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.629+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4a78111350 con 0x7f4a780fe830 2024-03-23T12:24:33.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.630+0000 7f4a6e000700 1 -- 172.21.15.63:0/2159248751 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f4a64017e10 con 0x7f4a780fe830 2024-03-23T12:24:33.608 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:24:33.608 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:24:33.608 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:24:33.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:24:33.609 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:24:33.610 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:24:33.610 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:24:33.610 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:24:33.610 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:24:33.610 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:24:33.610 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a5c06c7d0 msgr2=0x7f4a5c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a5c06c7d0 0x7f4a5c06ec80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800b3b0 tx=0x7f4a68009e30 comp rx=0 tx=0).stop 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fe830 msgr2=0x7f4a78116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fe830 0x7f4a78116200 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4a64002a50 tx=0x7f4a640046b0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 shutdown_connections 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4a5c06c7d0 0x7f4a5c06ec80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a780fde90 0x7f4a78115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 --2- 172.21.15.63:0/2159248751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a780fe830 0x7f4a78116200 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:24:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 >> 172.21.15.63:0/2159248751 conn(0x7f4a78072bc0 msgr2=0x7f4a780fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:24:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 shutdown_connections 2024-03-23T12:24:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:24:33.632+0000 7f4a7d200700 1 -- 172.21.15.63:0/2159248751 wait complete. 2024-03-23T12:24:34.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:33 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2159248751' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:24:34.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:33 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2159248751' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:24:35.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:34 smithi063 ceph-mon[141124]: pgmap v1004: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:35.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:34 smithi063 ceph-mon[141124]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:35.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:34 smithi111 ceph-mon[106899]: pgmap v1004: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:35.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:34 smithi111 ceph-mon[106899]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:24:37.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:36 smithi063 ceph-mon[141124]: pgmap v1005: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:37.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:36 smithi111 ceph-mon[106899]: pgmap v1005: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:39.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:38 smithi063 ceph-mon[141124]: pgmap v1006: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:39.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:38 smithi111 ceph-mon[106899]: pgmap v1006: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:41.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:40 smithi063 ceph-mon[141124]: pgmap v1007: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:41.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:41.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:40 smithi111 ceph-mon[106899]: pgmap v1007: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:41.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:43.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:42 smithi063 ceph-mon[141124]: pgmap v1008: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:43.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:42 smithi111 ceph-mon[106899]: pgmap v1008: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:45.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:44 smithi063 ceph-mon[141124]: pgmap v1009: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:45.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:44 smithi111 ceph-mon[106899]: pgmap v1009: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:47.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:46 smithi063 ceph-mon[141124]: pgmap v1010: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:47.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:46 smithi111 ceph-mon[106899]: pgmap v1010: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:49.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:48 smithi063 ceph-mon[141124]: pgmap v1011: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:49.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:48 smithi111 ceph-mon[106899]: pgmap v1011: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:51.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:50 smithi063 ceph-mon[141124]: pgmap v1012: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:51.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:50 smithi111 ceph-mon[106899]: pgmap v1012: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:52 smithi063 ceph-mon[141124]: pgmap v1013: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:53.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:52 smithi111 ceph-mon[106899]: pgmap v1013: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:55.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:54 smithi063 ceph-mon[141124]: pgmap v1014: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:55.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:54 smithi111 ceph-mon[106899]: pgmap v1014: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:56 smithi063 ceph-mon[141124]: pgmap v1015: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:57.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:56 smithi111 ceph-mon[106899]: pgmap v1015: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:57.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:24:59.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:24:58 smithi063 ceph-mon[141124]: pgmap v1016: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:24:59.149 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:24:58 smithi111 ceph-mon[106899]: pgmap v1016: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:01.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:00 smithi063 ceph-mon[141124]: pgmap v1017: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:01.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:00 smithi111 ceph-mon[106899]: pgmap v1017: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:03.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:02 smithi063 ceph-mon[141124]: pgmap v1018: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:03.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:02 smithi111 ceph-mon[106899]: pgmap v1018: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:03.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.787+0000 7f956d800700 1 -- 172.21.15.63:0/1936603333 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9568102e00 msgr2=0x7f95681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:03.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.787+0000 7f956d800700 1 --2- 172.21.15.63:0/1936603333 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9568102e00 0x7f95681031d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9558004940 tx=0x7f9558009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:03.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.787+0000 7f956d800700 1 -- 172.21.15.63:0/1936603333 shutdown_connections 2024-03-23T12:25:03.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.787+0000 7f956d800700 1 --2- 172.21.15.63:0/1936603333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95681037a0 0x7f95680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:03.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.787+0000 7f956d800700 1 --2- 172.21.15.63:0/1936603333 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9568102e00 0x7f95681031d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:03.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.787+0000 7f956d800700 1 -- 172.21.15.63:0/1936603333 >> 172.21.15.63:0/1936603333 conn(0x7f95680f89b0 msgr2=0x7f95680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:03.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.788+0000 7f956d800700 1 -- 172.21.15.63:0/1936603333 shutdown_connections 2024-03-23T12:25:03.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.788+0000 7f956d800700 1 -- 172.21.15.63:0/1936603333 wait complete. 2024-03-23T12:25:03.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.788+0000 7f956d800700 1 Processor -- start 2024-03-23T12:25:03.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f956d800700 1 -- start start 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f956d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9568102e00 0x7f9568115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f956d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95681037a0 0x7f9568116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f956d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9568116860 con 0x7f95681037a0 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f956d800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956810fc30 con 0x7f9568102e00 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f9566000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95681037a0 0x7f9568116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f9566000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95681037a0 0x7f9568116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35524/0 (socket says 172.21.15.63:35524) 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.789+0000 7f9566000700 1 -- 172.21.15.63:0/3919521870 learned_addr learned my addr 172.21.15.63:0/3919521870 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f9566a00700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9568102e00 0x7f9568115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f9566000700 1 -- 172.21.15.63:0/3919521870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9568102e00 msgr2=0x7f9568115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f9566000700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9568102e00 0x7f9568115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f9566000700 1 -- 172.21.15.63:0/3919521870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9558005040 con 0x7f95681037a0 2024-03-23T12:25:03.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f9566000700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95681037a0 0x7f9568116210 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f955400c890 tx=0x7f95540044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:03.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f955ea00700 1 -- 172.21.15.63:0/3919521870 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f955400a700 con 0x7f95681037a0 2024-03-23T12:25:03.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f955ea00700 1 -- 172.21.15.63:0/3919521870 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f955400ad10 con 0x7f95681037a0 2024-03-23T12:25:03.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956810ff10 con 0x7f95681037a0 2024-03-23T12:25:03.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.790+0000 7f955ea00700 1 -- 172.21.15.63:0/3919521870 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9554012e70 con 0x7f95681037a0 2024-03-23T12:25:03.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.791+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95681104d0 con 0x7f95681037a0 2024-03-23T12:25:03.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.792+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95680fe550 con 0x7f95681037a0 2024-03-23T12:25:03.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.793+0000 7f955ea00700 1 -- 172.21.15.63:0/3919521870 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f955400a860 con 0x7f95681037a0 2024-03-23T12:25:03.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.793+0000 7f955ea00700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f954c06c7c0 0x7f954c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:03.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.794+0000 7f955ea00700 1 -- 172.21.15.63:0/3919521870 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f955408ba90 con 0x7f95681037a0 2024-03-23T12:25:03.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.794+0000 7f9566a00700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f954c06c7c0 0x7f954c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:03.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.794+0000 7f9566a00700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f954c06c7c0 0x7f954c06ec70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9558004f90 tx=0x7f9558009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:03.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:03.799+0000 7f955ea00700 1 -- 172.21.15.63:0/3919521870 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f95540597a0 con 0x7f95681037a0 2024-03-23T12:25:04.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.033+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9568109e30 con 0x7f954c06c7c0 2024-03-23T12:25:04.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.035+0000 7f955ea00700 1 -- 172.21.15.63:0/3919521870 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9568109e30 con 0x7f954c06c7c0 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f954c06c7c0 msgr2=0x7f954c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f954c06c7c0 0x7f954c06ec70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9558004f90 tx=0x7f9558009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95681037a0 msgr2=0x7f9568116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95681037a0 0x7f9568116210 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f955400c890 tx=0x7f95540044c0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 shutdown_connections 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f954c06c7c0 0x7f954c06ec70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9568102e00 0x7f9568115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 --2- 172.21.15.63:0/3919521870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95681037a0 0x7f9568116210 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 >> 172.21.15.63:0/3919521870 conn(0x7f95680f89b0 msgr2=0x7f95680f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 shutdown_connections 2024-03-23T12:25:04.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.037+0000 7f956d800700 1 -- 172.21.15.63:0/3919521870 wait complete. 2024-03-23T12:25:04.030 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:25:04.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 -- 172.21.15.63:0/693308421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105090 msgr2=0x7f1a04105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 --2- 172.21.15.63:0/693308421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105090 0x7f1a04105460 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f19f4004940 tx=0x7f19f4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 -- 172.21.15.63:0/693308421 shutdown_connections 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 --2- 172.21.15.63:0/693308421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04105a30 0x7f1a040ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 --2- 172.21.15.63:0/693308421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105090 0x7f1a04105460 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 -- 172.21.15.63:0/693308421 >> 172.21.15.63:0/693308421 conn(0x7f1a040fac00 msgr2=0x7f1a040fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 -- 172.21.15.63:0/693308421 shutdown_connections 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.177+0000 7f1a09a00700 1 -- 172.21.15.63:0/693308421 wait complete. 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.178+0000 7f1a09a00700 1 Processor -- start 2024-03-23T12:25:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.178+0000 7f1a09a00700 1 -- start start 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a09a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04105090 0x7f1a04115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a09a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105a30 0x7f1a04115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a09a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a04116620 con 0x7f1a04105a30 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a09a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0410fc00 con 0x7f1a04105090 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a02000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105a30 0x7f1a04115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a02000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105a30 0x7f1a04115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35550/0 (socket says 172.21.15.63:35550) 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a02000700 1 -- 172.21.15.63:0/588800334 learned_addr learned my addr 172.21.15.63:0/588800334 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:04.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a02a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04105090 0x7f1a04115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a02000700 1 -- 172.21.15.63:0/588800334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04105090 msgr2=0x7f1a04115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a02000700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04105090 0x7f1a04115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.179+0000 7f1a02000700 1 -- 172.21.15.63:0/588800334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4005040 con 0x7f1a04105a30 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.180+0000 7f1a02000700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105a30 0x7f1a04115fd0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f19f0002a50 tx=0x7f19f0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.180+0000 7f19faa00700 1 -- 172.21.15.63:0/588800334 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19f0007cc0 con 0x7f1a04105a30 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.180+0000 7f19faa00700 1 -- 172.21.15.63:0/588800334 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19f000f040 con 0x7f1a04105a30 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.180+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0410fee0 con 0x7f1a04105a30 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.180+0000 7f19faa00700 1 -- 172.21.15.63:0/588800334 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19f000d9a0 con 0x7f1a04105a30 2024-03-23T12:25:04.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.180+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a041104a0 con 0x7f1a04105a30 2024-03-23T12:25:04.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.181+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a041023a0 con 0x7f1a04105a30 2024-03-23T12:25:04.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.182+0000 7f19faa00700 1 -- 172.21.15.63:0/588800334 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f19f000dc60 con 0x7f1a04105a30 2024-03-23T12:25:04.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.183+0000 7f19faa00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f19e806c7c0 0x7f19e806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.183+0000 7f19faa00700 1 -- 172.21.15.63:0/588800334 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f19f008b930 con 0x7f1a04105a30 2024-03-23T12:25:04.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.183+0000 7f1a02a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f19e806c7c0 0x7f19e806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.184+0000 7f1a02a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f19e806c7c0 0x7f19e806ec70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f19f400b020 tx=0x7f19f4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:04.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.188+0000 7f19faa00700 1 -- 172.21.15.63:0/588800334 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f19f0059540 con 0x7f1a04105a30 2024-03-23T12:25:04.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.420+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a04111150 con 0x7f19e806c7c0 2024-03-23T12:25:04.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.422+0000 7f19faa00700 1 -- 172.21.15.63:0/588800334 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1a04111150 con 0x7f19e806c7c0 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f19e806c7c0 msgr2=0x7f19e806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f19e806c7c0 0x7f19e806ec70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f19f400b020 tx=0x7f19f4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105a30 msgr2=0x7f1a04115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105a30 0x7f1a04115fd0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f19f0002a50 tx=0x7f19f0004600 comp rx=0 tx=0).stop 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 shutdown_connections 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f19e806c7c0 0x7f19e806ec70 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a04105090 0x7f1a04115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 --2- 172.21.15.63:0/588800334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a04105a30 0x7f1a04115fd0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.424+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 >> 172.21.15.63:0/588800334 conn(0x7f1a040fac00 msgr2=0x7f1a040fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:04.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.425+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 shutdown_connections 2024-03-23T12:25:04.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.425+0000 7f1a09a00700 1 -- 172.21.15.63:0/588800334 wait complete. 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 -- 172.21.15.63:0/782033680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df4105000 msgr2=0x7f5df41053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 --2- 172.21.15.63:0/782033680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df4105000 0x7f5df41053d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5de4004940 tx=0x7f5de4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 -- 172.21.15.63:0/782033680 shutdown_connections 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 --2- 172.21.15.63:0/782033680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df41059a0 0x7f5df40ff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 --2- 172.21.15.63:0/782033680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df4105000 0x7f5df41053d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 -- 172.21.15.63:0/782033680 >> 172.21.15.63:0/782033680 conn(0x7f5df40fabd0 msgr2=0x7f5df40fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 -- 172.21.15.63:0/782033680 shutdown_connections 2024-03-23T12:25:04.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.563+0000 7f5dfa600700 1 -- 172.21.15.63:0/782033680 wait complete. 2024-03-23T12:25:04.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.564+0000 7f5dfa600700 1 Processor -- start 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.564+0000 7f5dfa600700 1 -- start start 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.564+0000 7f5dfa600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df4105000 0x7f5df4115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.564+0000 7f5dfa600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df41059a0 0x7f5df41161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.564+0000 7f5dfa600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df4116820 con 0x7f5df41059a0 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.564+0000 7f5dfa600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df410fbf0 con 0x7f5df4105000 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.565+0000 7f5df2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df41059a0 0x7f5df41161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.565+0000 7f5df2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df41059a0 0x7f5df41161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35560/0 (socket says 172.21.15.63:35560) 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.565+0000 7f5df2a00700 1 -- 172.21.15.63:0/2550597019 learned_addr learned my addr 172.21.15.63:0/2550597019 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.565+0000 7f5df2a00700 1 -- 172.21.15.63:0/2550597019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df4105000 msgr2=0x7f5df4115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.565+0000 7f5df2a00700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df4105000 0x7f5df4115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.565+0000 7f5df2a00700 1 -- 172.21.15.63:0/2550597019 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de4005040 con 0x7f5df41059a0 2024-03-23T12:25:04.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.565+0000 7f5df2a00700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df41059a0 0x7f5df41161d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5de000eb40 tx=0x7f5de000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:04.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.566+0000 7f5deb400700 1 -- 172.21.15.63:0/2550597019 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5de0007cc0 con 0x7f5df41059a0 2024-03-23T12:25:04.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.566+0000 7f5deb400700 1 -- 172.21.15.63:0/2550597019 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5de000f040 con 0x7f5df41059a0 2024-03-23T12:25:04.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.566+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df410fe70 con 0x7f5df41059a0 2024-03-23T12:25:04.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.566+0000 7f5deb400700 1 -- 172.21.15.63:0/2550597019 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5de0009570 con 0x7f5df41059a0 2024-03-23T12:25:04.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.566+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df4110430 con 0x7f5df41059a0 2024-03-23T12:25:04.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.567+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df4102310 con 0x7f5df41059a0 2024-03-23T12:25:04.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.568+0000 7f5deb400700 1 -- 172.21.15.63:0/2550597019 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5de00096d0 con 0x7f5df41059a0 2024-03-23T12:25:04.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.569+0000 7f5deb400700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dd806c7d0 0x7f5dd806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.569+0000 7f5deb400700 1 -- 172.21.15.63:0/2550597019 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5de008b540 con 0x7f5df41059a0 2024-03-23T12:25:04.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.569+0000 7f5df3400700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dd806c7d0 0x7f5dd806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.570+0000 7f5df3400700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dd806c7d0 0x7f5dd806ec80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5de4000c00 tx=0x7f5de4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:04.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.573+0000 7f5deb400700 1 -- 172.21.15.63:0/2550597019 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5de0059280 con 0x7f5df41059a0 2024-03-23T12:25:04.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.794+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5df4111350 con 0x7f5dd806c7d0 2024-03-23T12:25:04.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.804+0000 7f5deb400700 1 -- 172.21.15.63:0/2550597019 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f5df4111350 con 0x7f5dd806c7d0 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (34m) 11m ago 47m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (31m) 11m ago 47m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (31m) 11m ago 46m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (34m) 11m ago 47m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (27m) 11m ago 41m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (27m) 11m ago 41m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (11m) 11m ago 41m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (26m) 11m ago 41m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:25:04.782 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (36m) 11m ago 49m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (35m) 11m ago 46m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (32m) 11m ago 49m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (32m) 11m ago 46m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (35m) 11m ago 47m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (35m) 11m ago 46m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (31m) 11m ago 45m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (30m) 11m ago 44m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (29m) 11m ago 44m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (29m) 11m ago 43m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (28m) 11m ago 43m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (27m) 11m ago 42m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:25:04.783 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (34m) 11m ago 46m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:25:04.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.806+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dd806c7d0 msgr2=0x7f5dd806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.806+0000 7f5dfa600700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dd806c7d0 0x7f5dd806ec80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5de4000c00 tx=0x7f5de4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:04.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.806+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df41059a0 msgr2=0x7f5df41161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.806+0000 7f5dfa600700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df41059a0 0x7f5df41161d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5de000eb40 tx=0x7f5de000b040 comp rx=0 tx=0).stop 2024-03-23T12:25:04.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.807+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 shutdown_connections 2024-03-23T12:25:04.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.807+0000 7f5dfa600700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5dd806c7d0 0x7f5dd806ec80 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.807+0000 7f5dfa600700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df4105000 0x7f5df4115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.807+0000 7f5dfa600700 1 --2- 172.21.15.63:0/2550597019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5df41059a0 0x7f5df41161d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.807+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 >> 172.21.15.63:0/2550597019 conn(0x7f5df40fabd0 msgr2=0x7f5df41095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:04.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.807+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 shutdown_connections 2024-03-23T12:25:04.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.807+0000 7f5dfa600700 1 -- 172.21.15.63:0/2550597019 wait complete. 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 -- 172.21.15.63:0/227290051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc1037a0 msgr2=0x7f07dc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 --2- 172.21.15.63:0/227290051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc1037a0 0x7f07dc0fce10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f07c4004940 tx=0x7f07c4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 -- 172.21.15.63:0/227290051 shutdown_connections 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 --2- 172.21.15.63:0/227290051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc1037a0 0x7f07dc0fce10 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 --2- 172.21.15.63:0/227290051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07dc102e00 0x7f07dc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 -- 172.21.15.63:0/227290051 >> 172.21.15.63:0/227290051 conn(0x7f07dc0f89b0 msgr2=0x7f07dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 -- 172.21.15.63:0/227290051 shutdown_connections 2024-03-23T12:25:04.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.943+0000 7f07e0c00700 1 -- 172.21.15.63:0/227290051 wait complete. 2024-03-23T12:25:04.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.944+0000 7f07e0c00700 1 Processor -- start 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.944+0000 7f07e0c00700 1 -- start start 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc102e00 0x7f07dc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07e0c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07dc1037a0 0x7f07dc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07e0c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc070d90 con 0x7f07dc102e00 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07e0c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc070ed0 con 0x7f07dc1037a0 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07d9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc102e00 0x7f07dc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07d9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc102e00 0x7f07dc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35588/0 (socket says 172.21.15.63:35588) 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07d9e00700 1 -- 172.21.15.63:0/3936703435 learned_addr learned my addr 172.21.15.63:0/3936703435 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07d3e00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07dc1037a0 0x7f07dc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07d9e00700 1 -- 172.21.15.63:0/3936703435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07dc1037a0 msgr2=0x7f07dc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07d9e00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07dc1037a0 0x7f07dc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.945+0000 7f07d9e00700 1 -- 172.21.15.63:0/3936703435 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4005040 con 0x7f07dc102e00 2024-03-23T12:25:04.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.946+0000 7f07d9e00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc102e00 0x7f07dc0702c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f07cc002a50 tx=0x7f07cc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:04.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.946+0000 7f07d2000700 1 -- 172.21.15.63:0/3936703435 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07cc007cc0 con 0x7f07dc102e00 2024-03-23T12:25:04.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.946+0000 7f07d2000700 1 -- 172.21.15.63:0/3936703435 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07cc00f040 con 0x7f07dc102e00 2024-03-23T12:25:04.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.946+0000 7f07d2000700 1 -- 172.21.15.63:0/3936703435 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07cc00d9c0 con 0x7f07dc102e00 2024-03-23T12:25:04.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.946+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07dc119e00 con 0x7f07dc102e00 2024-03-23T12:25:04.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.946+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07dc11a280 con 0x7f07dc102e00 2024-03-23T12:25:04.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.947+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07dc0fe550 con 0x7f07dc102e00 2024-03-23T12:25:04.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.948+0000 7f07d2000700 1 -- 172.21.15.63:0/3936703435 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f07cc00db20 con 0x7f07dc102e00 2024-03-23T12:25:04.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.949+0000 7f07d2000700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07c006c7d0 0x7f07c006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:04.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.949+0000 7f07d2000700 1 -- 172.21.15.63:0/3936703435 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f07cc08b740 con 0x7f07dc102e00 2024-03-23T12:25:04.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.949+0000 7f07d3e00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07c006c7d0 0x7f07c006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:04.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.950+0000 7f07d3e00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07c006c7d0 0x7f07c006ec80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f07dc071c00 tx=0x7f07c4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:04.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:04.953+0000 7f07d2000700 1 -- 172.21.15.63:0/3936703435 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07cc055940 con 0x7f07dc102e00 2024-03-23T12:25:05.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:04 smithi063 ceph-mon[141124]: pgmap v1019: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:05.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:04 smithi063 ceph-mon[141124]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:05.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:04 smithi111 ceph-mon[106899]: pgmap v1019: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:05.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:04 smithi111 ceph-mon[106899]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.254+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f07dc0719b0 con 0x7f07dc102e00 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.254+0000 7f07d2000700 1 -- 172.21.15.63:0/3936703435 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f07cc058f60 con 0x7f07dc102e00 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:05.232 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:25:05.233 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:25:05.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07c006c7d0 msgr2=0x7f07c006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07c006c7d0 0x7f07c006ec80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f07dc071c00 tx=0x7f07c4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc102e00 msgr2=0x7f07dc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc102e00 0x7f07dc0702c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f07cc002a50 tx=0x7f07cc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 shutdown_connections 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f07c006c7d0 0x7f07c006ec80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07dc102e00 0x7f07dc0702c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 --2- 172.21.15.63:0/3936703435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07dc1037a0 0x7f07dc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 >> 172.21.15.63:0/3936703435 conn(0x7f07dc0f89b0 msgr2=0x7f07dc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 shutdown_connections 2024-03-23T12:25:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.257+0000 7f07e0c00700 1 -- 172.21.15.63:0/3936703435 wait complete. 2024-03-23T12:25:05.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 -- 172.21.15.63:0/2036457182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd130102e00 msgr2=0x7fd1301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 --2- 172.21.15.63:0/2036457182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd130102e00 0x7fd1301031d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd120004940 tx=0x7fd120009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:05.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 -- 172.21.15.63:0/2036457182 shutdown_connections 2024-03-23T12:25:05.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 --2- 172.21.15.63:0/2036457182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1301037a0 0x7fd1300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 --2- 172.21.15.63:0/2036457182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd130102e00 0x7fd1301031d0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 -- 172.21.15.63:0/2036457182 >> 172.21.15.63:0/2036457182 conn(0x7fd1300f89b0 msgr2=0x7fd1300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:05.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 -- 172.21.15.63:0/2036457182 shutdown_connections 2024-03-23T12:25:05.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.393+0000 7fd136a00700 1 -- 172.21.15.63:0/2036457182 wait complete. 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd136a00700 1 Processor -- start 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd136a00700 1 -- start start 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd136a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd130102e00 0x7fd130113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd136a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1301037a0 0x7fd130114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd136a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1301146b0 con 0x7fd1301037a0 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd136a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13010daa0 con 0x7fd130102e00 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd12f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1301037a0 0x7fd130114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd12f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1301037a0 0x7fd130114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35606/0 (socket says 172.21.15.63:35606) 2024-03-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.394+0000 7fd12f400700 1 -- 172.21.15.63:0/1209086796 learned_addr learned my addr 172.21.15.63:0/1209086796 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd12fe00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd130102e00 0x7fd130113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd12f400700 1 -- 172.21.15.63:0/1209086796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd130102e00 msgr2=0x7fd130113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd12f400700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd130102e00 0x7fd130113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd12f400700 1 -- 172.21.15.63:0/1209086796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120005040 con 0x7fd1301037a0 2024-03-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd12f400700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1301037a0 0x7fd130114060 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00c890 tx=0x7fd11c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd127e00700 1 -- 172.21.15.63:0/1209086796 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd11c00a700 con 0x7fd1301037a0 2024-03-23T12:25:05.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd127e00700 1 -- 172.21.15.63:0/1209086796 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd11c00ad10 con 0x7fd1301037a0 2024-03-23T12:25:05.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.395+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13010dd80 con 0x7fd1301037a0 2024-03-23T12:25:05.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.396+0000 7fd127e00700 1 -- 172.21.15.63:0/1209086796 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd11c012e70 con 0x7fd1301037a0 2024-03-23T12:25:05.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.396+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13010e340 con 0x7fd1301037a0 2024-03-23T12:25:05.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.397+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1300fe550 con 0x7fd1301037a0 2024-03-23T12:25:05.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.398+0000 7fd127e00700 1 -- 172.21.15.63:0/1209086796 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd11c00a860 con 0x7fd1301037a0 2024-03-23T12:25:05.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.398+0000 7fd127e00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd11406c7d0 0x7fd11406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:05.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.399+0000 7fd127e00700 1 -- 172.21.15.63:0/1209086796 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd11c08ba90 con 0x7fd1301037a0 2024-03-23T12:25:05.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.399+0000 7fd12fe00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd11406c7d0 0x7fd11406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:05.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.399+0000 7fd12fe00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd11406c7d0 0x7fd11406ec80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd120009fd0 tx=0x7fd120009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:05.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.403+0000 7fd127e00700 1 -- 172.21.15.63:0/1209086796 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd11c0597a0 con 0x7fd1301037a0 2024-03-23T12:25:05.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.661+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd130109dc0 con 0x7fd1301037a0 2024-03-23T12:25:05.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.661+0000 7fd127e00700 1 -- 172.21.15.63:0/1209086796 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fd11c059330 con 0x7fd1301037a0 2024-03-23T12:25:05.639 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:25:05.639 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:25:05.639 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:25:05.639 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:25:05.640 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:25:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:05.642 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:05.643 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:05.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd11406c7d0 msgr2=0x7fd11406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd11406c7d0 0x7fd11406ec80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd120009fd0 tx=0x7fd120009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:05.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1301037a0 msgr2=0x7fd130114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1301037a0 0x7fd130114060 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00c890 tx=0x7fd11c0044c0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 shutdown_connections 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd11406c7d0 0x7fd11406ec80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd130102e00 0x7fd130113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 --2- 172.21.15.63:0/1209086796 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1301037a0 0x7fd130114060 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 >> 172.21.15.63:0/1209086796 conn(0x7fd1300f89b0 msgr2=0x7fd1300f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 shutdown_connections 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.664+0000 7fd136a00700 1 -- 172.21.15.63:0/1209086796 wait complete. 2024-03-23T12:25:05.644 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:25:05.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 -- 172.21.15.63:0/3854138451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e8102c40 msgr2=0x7fc4e8103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 --2- 172.21.15.63:0/3854138451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e8102c40 0x7fc4e8103010 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc004940 tx=0x7fc4dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:05.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 -- 172.21.15.63:0/3854138451 shutdown_connections 2024-03-23T12:25:05.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 --2- 172.21.15.63:0/3854138451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e81035e0 0x7fc4e80fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 --2- 172.21.15.63:0/3854138451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e8102c40 0x7fc4e8103010 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 -- 172.21.15.63:0/3854138451 >> 172.21.15.63:0/3854138451 conn(0x7fc4e80f8830 msgr2=0x7fc4e80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:05.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 -- 172.21.15.63:0/3854138451 shutdown_connections 2024-03-23T12:25:05.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.807+0000 7fc4ee800700 1 -- 172.21.15.63:0/3854138451 wait complete. 2024-03-23T12:25:05.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.808+0000 7fc4ee800700 1 Processor -- start 2024-03-23T12:25:05.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.808+0000 7fc4ee800700 1 -- start start 2024-03-23T12:25:05.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.808+0000 7fc4ee800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e8102c40 0x7fc4e8115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.808+0000 7fc4ee800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e81035e0 0x7fc4e8115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.808+0000 7fc4ee800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e8116640 con 0x7fc4e81035e0 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.808+0000 7fc4ee800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e810fa30 con 0x7fc4e8102c40 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.809+0000 7fc4eca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e81035e0 0x7fc4e8115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.809+0000 7fc4eca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e81035e0 0x7fc4e8115ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35624/0 (socket says 172.21.15.63:35624) 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.809+0000 7fc4eca00700 1 -- 172.21.15.63:0/2163114553 learned_addr learned my addr 172.21.15.63:0/2163114553 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.809+0000 7fc4ed400700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e8102c40 0x7fc4e8115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.809+0000 7fc4eca00700 1 -- 172.21.15.63:0/2163114553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e8102c40 msgr2=0x7fc4e8115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.809+0000 7fc4eca00700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e8102c40 0x7fc4e8115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:05.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.809+0000 7fc4eca00700 1 -- 172.21.15.63:0/2163114553 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc005040 con 0x7fc4e81035e0 2024-03-23T12:25:05.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.810+0000 7fc4eca00700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e81035e0 0x7fc4e8115ff0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400eb40 tx=0x7fc4d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:05.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.810+0000 7fc4e5600700 1 -- 172.21.15.63:0/2163114553 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4d4007cc0 con 0x7fc4e81035e0 2024-03-23T12:25:05.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.810+0000 7fc4e5600700 1 -- 172.21.15.63:0/2163114553 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4d400f040 con 0x7fc4e81035e0 2024-03-23T12:25:05.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.810+0000 7fc4e5600700 1 -- 172.21.15.63:0/2163114553 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4d400c8f0 con 0x7fc4e81035e0 2024-03-23T12:25:05.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.810+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e810fcb0 con 0x7fc4e81035e0 2024-03-23T12:25:05.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.810+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e8110220 con 0x7fc4e81035e0 2024-03-23T12:25:05.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.811+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e80fe390 con 0x7fc4e81035e0 2024-03-23T12:25:05.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.812+0000 7fc4e5600700 1 -- 172.21.15.63:0/2163114553 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc4d400cb60 con 0x7fc4e81035e0 2024-03-23T12:25:05.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.814+0000 7fc4e5600700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4d006c7d0 0x7fc4d006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:05.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.814+0000 7fc4e5600700 1 -- 172.21.15.63:0/2163114553 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc4d408b510 con 0x7fc4e81035e0 2024-03-23T12:25:05.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.814+0000 7fc4ed400700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4d006c7d0 0x7fc4d006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:05.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.814+0000 7fc4ed400700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4d006c7d0 0x7fc4d006ec80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0045a0 tx=0x7fc4dc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:05.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:05.819+0000 7fc4e5600700 1 -- 172.21.15.63:0/2163114553 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc4d4059250 con 0x7fc4e81035e0 2024-03-23T12:25:06.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.051+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4e8109bf0 con 0x7fc4d006c7d0 2024-03-23T12:25:06.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.052+0000 7fc4e5600700 1 -- 172.21.15.63:0/2163114553 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc4e8109bf0 con 0x7fc4d006c7d0 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:25:06.030 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:25:06.031 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:25:06.031 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:25:06.031 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:25:06.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4d006c7d0 msgr2=0x7fc4d006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:06.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4d006c7d0 0x7fc4d006ec80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0045a0 tx=0x7fc4dc009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:06.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e81035e0 msgr2=0x7fc4e8115ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:06.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e81035e0 0x7fc4e8115ff0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400eb40 tx=0x7fc4d400b040 comp rx=0 tx=0).stop 2024-03-23T12:25:06.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 shutdown_connections 2024-03-23T12:25:06.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc4d006c7d0 0x7fc4d006ec80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e8102c40 0x7fc4e8115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 --2- 172.21.15.63:0/2163114553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc4e81035e0 0x7fc4e8115ff0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 >> 172.21.15.63:0/2163114553 conn(0x7fc4e80f8830 msgr2=0x7fc4e80f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:06.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 shutdown_connections 2024-03-23T12:25:06.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.055+0000 7fc4ee800700 1 -- 172.21.15.63:0/2163114553 wait complete. 2024-03-23T12:25:06.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:05 smithi063 ceph-mon[141124]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:06.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:05 smithi063 ceph-mon[141124]: from='client.35846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:06.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:05 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3936703435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:25:06.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:05 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1209086796' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:25:06.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:05 smithi111 ceph-mon[106899]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:06.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:05 smithi111 ceph-mon[106899]: from='client.35846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:06.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:05 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3936703435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:25:06.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:05 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1209086796' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 -- 172.21.15.63:0/660646898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff164071f90 msgr2=0x7ff164070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 --2- 172.21.15.63:0/660646898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff164071f90 0x7ff164070360 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff158004970 tx=0x7ff158009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 -- 172.21.15.63:0/660646898 shutdown_connections 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 --2- 172.21.15.63:0/660646898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1640708a0 0x7ff164070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 --2- 172.21.15.63:0/660646898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff164071f90 0x7ff164070360 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 -- 172.21.15.63:0/660646898 >> 172.21.15.63:0/660646898 conn(0x7ff1640f89b0 msgr2=0x7ff1640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 -- 172.21.15.63:0/660646898 shutdown_connections 2024-03-23T12:25:06.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.193+0000 7ff16b600700 1 -- 172.21.15.63:0/660646898 wait complete. 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.194+0000 7ff16b600700 1 Processor -- start 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.194+0000 7ff16b600700 1 -- start start 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff16b600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1640708a0 0x7ff164119de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff16b600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff164071f90 0x7ff16411a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff16b600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16411a970 con 0x7ff1640708a0 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff16b600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff164114000 con 0x7ff164071f90 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff168c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1640708a0 0x7ff164119de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff168c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1640708a0 0x7ff164119de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35632/0 (socket says 172.21.15.63:35632) 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff168c00700 1 -- 172.21.15.63:0/1523056848 learned_addr learned my addr 172.21.15.63:0/1523056848 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff163e00700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff164071f90 0x7ff16411a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:06.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff168c00700 1 -- 172.21.15.63:0/1523056848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff164071f90 msgr2=0x7ff16411a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:06.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff168c00700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff164071f90 0x7ff16411a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff168c00700 1 -- 172.21.15.63:0/1523056848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff158005040 con 0x7ff1640708a0 2024-03-23T12:25:06.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.195+0000 7ff163e00700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff164071f90 0x7ff16411a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:25:06.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.196+0000 7ff168c00700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1640708a0 0x7ff164119de0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff158004f50 tx=0x7ff158009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:06.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.196+0000 7ff160c00700 1 -- 172.21.15.63:0/1523056848 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff15800e040 con 0x7ff1640708a0 2024-03-23T12:25:06.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.196+0000 7ff160c00700 1 -- 172.21.15.63:0/1523056848 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff158013420 con 0x7ff1640708a0 2024-03-23T12:25:06.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.196+0000 7ff160c00700 1 -- 172.21.15.63:0/1523056848 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff15801c610 con 0x7ff1640708a0 2024-03-23T12:25:06.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.196+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff164114280 con 0x7ff1640708a0 2024-03-23T12:25:06.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.196+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1641147c0 con 0x7ff1640708a0 2024-03-23T12:25:06.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.197+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff164071aa0 con 0x7ff1640708a0 2024-03-23T12:25:06.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.198+0000 7ff160c00700 1 -- 172.21.15.63:0/1523056848 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff158013590 con 0x7ff1640708a0 2024-03-23T12:25:06.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.199+0000 7ff160c00700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff14c06c7c0 0x7ff14c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:06.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.199+0000 7ff160c00700 1 -- 172.21.15.63:0/1523056848 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff15808cd50 con 0x7ff1640708a0 2024-03-23T12:25:06.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.199+0000 7ff163e00700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff14c06c7c0 0x7ff14c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:06.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.200+0000 7ff163e00700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff14c06c7c0 0x7ff14c06ec70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff150009fd0 tx=0x7ff15000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:06.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.204+0000 7ff160c00700 1 -- 172.21.15.63:0/1523056848 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff158057000 con 0x7ff1640708a0 2024-03-23T12:25:06.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.498+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff164071aa0 con 0x7ff1640708a0 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.498+0000 7ff160c00700 1 -- 172.21.15.63:0/1523056848 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7ff15805a620 con 0x7ff1640708a0 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:25:06.476 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:25:06.477 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:25:06.478 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:25:06.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff14c06c7c0 msgr2=0x7ff14c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:06.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff14c06c7c0 0x7ff14c06ec70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff150009fd0 tx=0x7ff15000b040 comp rx=0 tx=0).stop 2024-03-23T12:25:06.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1640708a0 msgr2=0x7ff164119de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:06.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1640708a0 0x7ff164119de0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff158004f50 tx=0x7ff158009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:06.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 shutdown_connections 2024-03-23T12:25:06.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff14c06c7c0 0x7ff14c06ec70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1640708a0 0x7ff164119de0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 --2- 172.21.15.63:0/1523056848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff164071f90 0x7ff16411a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:06.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 >> 172.21.15.63:0/1523056848 conn(0x7ff1640f89b0 msgr2=0x7ff1640f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:06.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 shutdown_connections 2024-03-23T12:25:06.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:06.501+0000 7ff16b600700 1 -- 172.21.15.63:0/1523056848 wait complete. 2024-03-23T12:25:07.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:06 smithi063 ceph-mon[141124]: pgmap v1020: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:07.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:06 smithi063 ceph-mon[141124]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:07.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:06 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1523056848' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:25:07.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:06 smithi111 ceph-mon[106899]: pgmap v1020: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:07.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:06 smithi111 ceph-mon[106899]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:07.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:06 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1523056848' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:25:09.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:08 smithi063 ceph-mon[141124]: pgmap v1021: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:09.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:08 smithi111 ceph-mon[106899]: pgmap v1021: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:11.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:10 smithi063 ceph-mon[141124]: pgmap v1022: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:11.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:11.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:10 smithi111 ceph-mon[106899]: pgmap v1022: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:11.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:13.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:12 smithi063 ceph-mon[141124]: pgmap v1023: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:13.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:12 smithi111 ceph-mon[106899]: pgmap v1023: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:15.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:14 smithi063 ceph-mon[141124]: pgmap v1024: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:15.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:14 smithi111 ceph-mon[106899]: pgmap v1024: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:17.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:16 smithi063 ceph-mon[141124]: pgmap v1025: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:17.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:16 smithi111 ceph-mon[106899]: pgmap v1025: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:19.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:18 smithi063 ceph-mon[141124]: pgmap v1026: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:19.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:18 smithi111 ceph-mon[106899]: pgmap v1026: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:21.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:20 smithi063 ceph-mon[141124]: pgmap v1027: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:21.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:20 smithi111 ceph-mon[106899]: pgmap v1027: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:23.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:22 smithi063 ceph-mon[141124]: pgmap v1028: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:23.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:22 smithi111 ceph-mon[106899]: pgmap v1028: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:25.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:24 smithi063 ceph-mon[141124]: pgmap v1029: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:25.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:24 smithi111 ceph-mon[106899]: pgmap v1029: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:27.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:26 smithi063 ceph-mon[141124]: pgmap v1030: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:27.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:26 smithi111 ceph-mon[106899]: pgmap v1030: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:27.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:29.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:28 smithi063 ceph-mon[141124]: pgmap v1031: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:29.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:28 smithi111 ceph-mon[106899]: pgmap v1031: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:31.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:30 smithi063 ceph-mon[141124]: pgmap v1032: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:31.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:30 smithi111 ceph-mon[106899]: pgmap v1032: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:33.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:32 smithi063 ceph-mon[141124]: pgmap v1033: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:33.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:32 smithi111 ceph-mon[106899]: pgmap v1033: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:35.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:34 smithi063 ceph-mon[141124]: pgmap v1034: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:35.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:34 smithi111 ceph-mon[106899]: pgmap v1034: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:36.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 -- 172.21.15.63:0/72237185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa0102e00 msgr2=0x7f9fa01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:36.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/72237185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa0102e00 0x7f9fa01031d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f9f94004940 tx=0x7f9f94009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:36.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 -- 172.21.15.63:0/72237185 shutdown_connections 2024-03-23T12:25:36.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/72237185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fa01037a0 0x7f9fa00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:36.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/72237185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa0102e00 0x7f9fa01031d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:36.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 -- 172.21.15.63:0/72237185 >> 172.21.15.63:0/72237185 conn(0x7f9fa00f89b0 msgr2=0x7f9fa00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:36.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 -- 172.21.15.63:0/72237185 shutdown_connections 2024-03-23T12:25:36.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.653+0000 7f9fa7e00700 1 -- 172.21.15.63:0/72237185 wait complete. 2024-03-23T12:25:36.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.654+0000 7f9fa7e00700 1 Processor -- start 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.654+0000 7f9fa7e00700 1 -- start start 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fa0102e00 0x7f9fa00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa01037a0 0x7f9fa0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa7e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa0070d90 con 0x7f9fa01037a0 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa7e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa0070ed0 con 0x7f9fa0102e00 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa01037a0 0x7f9fa0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa01037a0 0x7f9fa0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42610/0 (socket says 172.21.15.63:42610) 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa4a00700 1 -- 172.21.15.63:0/1911977718 learned_addr learned my addr 172.21.15.63:0/1911977718 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.655+0000 7f9fa5400700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fa0102e00 0x7f9fa00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9fa4a00700 1 -- 172.21.15.63:0/1911977718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fa0102e00 msgr2=0x7f9fa00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:36.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9fa4a00700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fa0102e00 0x7f9fa00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:36.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9fa4a00700 1 -- 172.21.15.63:0/1911977718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f94005040 con 0x7f9fa01037a0 2024-03-23T12:25:36.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9fa4a00700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa01037a0 0x7f9fa0070800 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c002a50 tx=0x7f9f8c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:36.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9f9d600700 1 -- 172.21.15.63:0/1911977718 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f8c007cc0 con 0x7f9fa01037a0 2024-03-23T12:25:36.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa0119d40 con 0x7f9fa01037a0 2024-03-23T12:25:36.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9f9d600700 1 -- 172.21.15.63:0/1911977718 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f8c00f040 con 0x7f9fa01037a0 2024-03-23T12:25:36.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9f9d600700 1 -- 172.21.15.63:0/1911977718 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f8c009590 con 0x7f9fa01037a0 2024-03-23T12:25:36.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.656+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa011a1c0 con 0x7f9fa01037a0 2024-03-23T12:25:36.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.657+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa00fe550 con 0x7f9fa01037a0 2024-03-23T12:25:36.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.658+0000 7f9f9d600700 1 -- 172.21.15.63:0/1911977718 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9f8c0096f0 con 0x7f9fa01037a0 2024-03-23T12:25:36.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.659+0000 7f9f9d600700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9f8806c7d0 0x7f9f8806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:36.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.659+0000 7f9f9d600700 1 -- 172.21.15.63:0/1911977718 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9f8c08b940 con 0x7f9fa01037a0 2024-03-23T12:25:36.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.660+0000 7f9fa5400700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9f8806c7d0 0x7f9f8806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:36.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.660+0000 7f9fa5400700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9f8806c7d0 0x7f9f8806ec80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400b020 tx=0x7f9f94009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:36.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.664+0000 7f9f9d600700 1 -- 172.21.15.63:0/1911977718 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9f8c055ac0 con 0x7f9fa01037a0 2024-03-23T12:25:36.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.894+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fa004bed0 con 0x7f9f8806c7d0 2024-03-23T12:25:36.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.895+0000 7f9f9d600700 1 -- 172.21.15.63:0/1911977718 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9fa004bed0 con 0x7f9f8806c7d0 2024-03-23T12:25:36.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9f8806c7d0 msgr2=0x7f9f8806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:36.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9f8806c7d0 0x7f9f8806ec80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400b020 tx=0x7f9f94009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:36.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa01037a0 msgr2=0x7f9fa0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:36.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa01037a0 0x7f9fa0070800 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c002a50 tx=0x7f9f8c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:25:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 shutdown_connections 2024-03-23T12:25:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9f8806c7d0 0x7f9f8806ec80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fa0102e00 0x7f9fa00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 --2- 172.21.15.63:0/1911977718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fa01037a0 0x7f9fa0070800 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 >> 172.21.15.63:0/1911977718 conn(0x7f9fa00f89b0 msgr2=0x7f9fa00f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 shutdown_connections 2024-03-23T12:25:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:36.898+0000 7f9fa7e00700 1 -- 172.21.15.63:0/1911977718 wait complete. 2024-03-23T12:25:36.891 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 -- 172.21.15.63:0/19317387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 msgr2=0x7f3394106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 --2- 172.21.15.63:0/19317387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f3394106f40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3380004940 tx=0x7f3380009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 -- 172.21.15.63:0/19317387 shutdown_connections 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 --2- 172.21.15.63:0/19317387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f3394106f40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 --2- 172.21.15.63:0/19317387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940fded0 0x7f33940fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 -- 172.21.15.63:0/19317387 >> 172.21.15.63:0/19317387 conn(0x7f3394072bc0 msgr2=0x7f3394072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 -- 172.21.15.63:0/19317387 shutdown_connections 2024-03-23T12:25:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.036+0000 7f339bc00700 1 -- 172.21.15.63:0/19317387 wait complete. 2024-03-23T12:25:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.037+0000 7f339bc00700 1 Processor -- start 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.037+0000 7f339bc00700 1 -- start start 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f339bc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940fded0 0x7f3394115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f339bc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f33941161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f339bc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33941167f0 con 0x7f33940fe870 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f339bc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339410fbc0 con 0x7f33940fded0 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f3393e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f33941161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f3393e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f33941161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42634/0 (socket says 172.21.15.63:42634) 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f3393e00700 1 -- 172.21.15.63:0/1824559316 learned_addr learned my addr 172.21.15.63:0/1824559316 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f3399200700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940fded0 0x7f3394115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f3393e00700 1 -- 172.21.15.63:0/1824559316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940fded0 msgr2=0x7f3394115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f3393e00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940fded0 0x7f3394115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.038+0000 7f3393e00700 1 -- 172.21.15.63:0/1824559316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3380005040 con 0x7f33940fe870 2024-03-23T12:25:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.039+0000 7f3393e00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f33941161a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3380009390 tx=0x7f3380009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.039+0000 7f3390c00700 1 -- 172.21.15.63:0/1824559316 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f338000e040 con 0x7f33940fe870 2024-03-23T12:25:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.039+0000 7f3390c00700 1 -- 172.21.15.63:0/1824559316 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3380013420 con 0x7f33940fe870 2024-03-23T12:25:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.039+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339410fea0 con 0x7f33940fe870 2024-03-23T12:25:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.039+0000 7f3390c00700 1 -- 172.21.15.63:0/1824559316 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f338001c630 con 0x7f33940fe870 2024-03-23T12:25:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.039+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33941103e0 con 0x7f33940fe870 2024-03-23T12:25:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.040+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394106a00 con 0x7f33940fe870 2024-03-23T12:25:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.041+0000 7f3390c00700 1 -- 172.21.15.63:0/1824559316 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3380013a20 con 0x7f33940fe870 2024-03-23T12:25:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.042+0000 7f3390c00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f337c06c7d0 0x7f337c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.042+0000 7f3390c00700 1 -- 172.21.15.63:0/1824559316 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f338008cc10 con 0x7f33940fe870 2024-03-23T12:25:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.042+0000 7f3399200700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f337c06c7d0 0x7f337c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.043+0000 7f3399200700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f337c06c7d0 0x7f337c06ec80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3388004f40 tx=0x7f338800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.047+0000 7f3390c00700 1 -- 172.21.15.63:0/1824559316 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f338005a920 con 0x7f33940fe870 2024-03-23T12:25:37.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:36 smithi063 ceph-mon[141124]: pgmap v1035: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:37.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:36 smithi111 ceph-mon[106899]: pgmap v1035: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:37.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.281+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3394111320 con 0x7f337c06c7d0 2024-03-23T12:25:37.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.283+0000 7f3390c00700 1 -- 172.21.15.63:0/1824559316 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3394111320 con 0x7f337c06c7d0 2024-03-23T12:25:37.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f337c06c7d0 msgr2=0x7f337c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f337c06c7d0 0x7f337c06ec80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3388004f40 tx=0x7f338800b040 comp rx=0 tx=0).stop 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 msgr2=0x7f33941161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f33941161a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3380009390 tx=0x7f3380009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 shutdown_connections 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f337c06c7d0 0x7f337c06ec80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940fded0 0x7f3394115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 --2- 172.21.15.63:0/1824559316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33940fe870 0x7f33941161a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 >> 172.21.15.63:0/1824559316 conn(0x7f3394072bc0 msgr2=0x7f33940fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 shutdown_connections 2024-03-23T12:25:37.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.285+0000 7f339bc00700 1 -- 172.21.15.63:0/1824559316 wait complete. 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 -- 172.21.15.63:0/1631205553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a01037a0 msgr2=0x7ff1a00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 --2- 172.21.15.63:0/1631205553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a01037a0 0x7ff1a00fce10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff18c004940 tx=0x7ff18c009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 -- 172.21.15.63:0/1631205553 shutdown_connections 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 --2- 172.21.15.63:0/1631205553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a01037a0 0x7ff1a00fce10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 --2- 172.21.15.63:0/1631205553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a0102e00 0x7ff1a01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 -- 172.21.15.63:0/1631205553 >> 172.21.15.63:0/1631205553 conn(0x7ff1a00f89b0 msgr2=0x7ff1a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 -- 172.21.15.63:0/1631205553 shutdown_connections 2024-03-23T12:25:37.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.422+0000 7ff1a7200700 1 -- 172.21.15.63:0/1631205553 wait complete. 2024-03-23T12:25:37.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.423+0000 7ff1a7200700 1 Processor -- start 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.423+0000 7ff1a7200700 1 -- start start 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff1a7200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a0102e00 0x7ff1a00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff1a7200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a01037a0 0x7ff1a0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff1a7200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0070d90 con 0x7ff1a0102e00 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff1a7200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0070ed0 con 0x7ff1a01037a0 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a0102e00 0x7ff1a00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a0102e00 0x7ff1a00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42660/0 (socket says 172.21.15.63:42660) 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19fe00700 1 -- 172.21.15.63:0/2711756828 learned_addr learned my addr 172.21.15.63:0/2711756828 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19f400700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a01037a0 0x7ff1a0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19fe00700 1 -- 172.21.15.63:0/2711756828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a01037a0 msgr2=0x7ff1a0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19fe00700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a01037a0 0x7ff1a0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19fe00700 1 -- 172.21.15.63:0/2711756828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c005040 con 0x7ff1a0102e00 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.424+0000 7ff19f400700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a01037a0 0x7ff1a0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:25:37.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.425+0000 7ff19fe00700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a0102e00 0x7ff1a00702c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff190002a50 tx=0x7ff1900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:37.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.425+0000 7ff197e00700 1 -- 172.21.15.63:0/2711756828 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff190007cc0 con 0x7ff1a0102e00 2024-03-23T12:25:37.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.425+0000 7ff197e00700 1 -- 172.21.15.63:0/2711756828 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff19000f040 con 0x7ff1a0102e00 2024-03-23T12:25:37.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.425+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a0119e00 con 0x7ff1a0102e00 2024-03-23T12:25:37.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.425+0000 7ff197e00700 1 -- 172.21.15.63:0/2711756828 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff190009540 con 0x7ff1a0102e00 2024-03-23T12:25:37.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.425+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a011a280 con 0x7ff1a0102e00 2024-03-23T12:25:37.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.426+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a00fe550 con 0x7ff1a0102e00 2024-03-23T12:25:37.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.427+0000 7ff197e00700 1 -- 172.21.15.63:0/2711756828 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7ff1900096a0 con 0x7ff1a0102e00 2024-03-23T12:25:37.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.428+0000 7ff197e00700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff18406c7d0 0x7ff18406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.428+0000 7ff197e00700 1 -- 172.21.15.63:0/2711756828 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff19008b910 con 0x7ff1a0102e00 2024-03-23T12:25:37.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.428+0000 7ff19f400700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff18406c7d0 0x7ff18406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.429+0000 7ff19f400700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff18406c7d0 0x7ff18406ec80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0071c00 tx=0x7ff18c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:37.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.432+0000 7ff197e00700 1 -- 172.21.15.63:0/2711756828 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff190055a90 con 0x7ff1a0102e00 2024-03-23T12:25:37.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.657+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1a00719b0 con 0x7ff18406c7d0 2024-03-23T12:25:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.667+0000 7ff197e00700 1 -- 172.21.15.63:0/2711756828 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7ff1a00719b0 con 0x7ff18406c7d0 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (34m) 12m ago 48m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (32m) 12m ago 48m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (32m) 12m ago 47m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (34m) 12m ago 47m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (28m) 12m ago 41m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (27m) 12m ago 41m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (12m) 12m ago 41m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (27m) 12m ago 41m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (37m) 12m ago 49m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:25:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (35m) 12m ago 46m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (33m) 12m ago 49m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (32m) 12m ago 46m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (35m) 12m ago 48m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (35m) 12m ago 47m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (31m) 12m ago 45m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (30m) 12m ago 45m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (30m) 12m ago 44m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (29m) 12m ago 44m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (28m) 12m ago 43m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (28m) 12m ago 43m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:25:37.647 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (35m) 12m ago 47m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff18406c7d0 msgr2=0x7ff18406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff18406c7d0 0x7ff18406ec80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0071c00 tx=0x7ff18c009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a0102e00 msgr2=0x7ff1a00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a0102e00 0x7ff1a00702c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff190002a50 tx=0x7ff1900046b0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 shutdown_connections 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7ff18406c7d0 0x7ff18406ec80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1a0102e00 0x7ff1a00702c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 --2- 172.21.15.63:0/2711756828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a01037a0 0x7ff1a0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 >> 172.21.15.63:0/2711756828 conn(0x7ff1a00f89b0 msgr2=0x7ff1a00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 shutdown_connections 2024-03-23T12:25:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.670+0000 7ff1a7200700 1 -- 172.21.15.63:0/2711756828 wait complete. 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 -- 172.21.15.63:0/164909293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 msgr2=0x7fd78c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 --2- 172.21.15.63:0/164909293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c106f00 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd778004940 tx=0x7fd778009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 -- 172.21.15.63:0/164909293 shutdown_connections 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 --2- 172.21.15.63:0/164909293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c106f00 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 --2- 172.21.15.63:0/164909293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd78c0fde90 0x7fd78c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 -- 172.21.15.63:0/164909293 >> 172.21.15.63:0/164909293 conn(0x7fd78c072bc0 msgr2=0x7fd78c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 -- 172.21.15.63:0/164909293 shutdown_connections 2024-03-23T12:25:37.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.805+0000 7fd793a00700 1 -- 172.21.15.63:0/164909293 wait complete. 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.806+0000 7fd793a00700 1 Processor -- start 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.806+0000 7fd793a00700 1 -- start start 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd793a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd78c0fde90 0x7fd78c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd793a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd793a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c1168c0 con 0x7fd78c0fe830 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd793a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c10fc90 con 0x7fd78c0fde90 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd78be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd78be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42666/0 (socket says 172.21.15.63:42666) 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd78be00700 1 -- 172.21.15.63:0/4062492801 learned_addr learned my addr 172.21.15.63:0/4062492801 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:37.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd791000700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd78c0fde90 0x7fd78c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.807+0000 7fd78be00700 1 -- 172.21.15.63:0/4062492801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd78c0fde90 msgr2=0x7fd78c115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:37.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd78be00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd78c0fde90 0x7fd78c115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:37.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd78be00700 1 -- 172.21.15.63:0/4062492801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd778005040 con 0x7fd78c0fe830 2024-03-23T12:25:37.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd78be00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c116270 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd778000c00 tx=0x7fd778009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:37.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd788c00700 1 -- 172.21.15.63:0/4062492801 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd77800e040 con 0x7fd78c0fe830 2024-03-23T12:25:37.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd788c00700 1 -- 172.21.15.63:0/4062492801 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd778013420 con 0x7fd78c0fe830 2024-03-23T12:25:37.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78c10ff10 con 0x7fd78c0fe830 2024-03-23T12:25:37.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd788c00700 1 -- 172.21.15.63:0/4062492801 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd77801c630 con 0x7fd78c0fe830 2024-03-23T12:25:37.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.808+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c110450 con 0x7fd78c0fe830 2024-03-23T12:25:37.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.809+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c106a20 con 0x7fd78c0fe830 2024-03-23T12:25:37.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.810+0000 7fd788c00700 1 -- 172.21.15.63:0/4062492801 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd778013970 con 0x7fd78c0fe830 2024-03-23T12:25:37.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.811+0000 7fd788c00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7d0 0x7fd77406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:37.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.811+0000 7fd788c00700 1 -- 172.21.15.63:0/4062492801 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd77808cc70 con 0x7fd78c0fe830 2024-03-23T12:25:37.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.811+0000 7fd791000700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7d0 0x7fd77406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:37.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.812+0000 7fd791000700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7d0 0x7fd77406ec80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd780004990 tx=0x7fd780009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:37.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:37.815+0000 7fd788c00700 1 -- 172.21.15.63:0/4062492801 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd778093080 con 0x7fd78c0fe830 2024-03-23T12:25:38.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.116+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd78c111380 con 0x7fd78c0fe830 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.116+0000 7fd788c00700 1 -- 172.21.15.63:0/4062492801 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fd77805a980 con 0x7fd78c0fe830 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:38.095 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:25:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:25:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:25:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:25:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:25:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:25:38.096 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:25:38.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7d0 msgr2=0x7fd77406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7d0 0x7fd77406ec80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd780004990 tx=0x7fd780009ee0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 msgr2=0x7fd78c116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c116270 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd778000c00 tx=0x7fd778009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:38.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 shutdown_connections 2024-03-23T12:25:38.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd77406c7d0 0x7fd77406ec80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd78c0fde90 0x7fd78c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 --2- 172.21.15.63:0/4062492801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd78c0fe830 0x7fd78c116270 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 >> 172.21.15.63:0/4062492801 conn(0x7fd78c072bc0 msgr2=0x7fd78c0fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:38.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 shutdown_connections 2024-03-23T12:25:38.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.119+0000 7fd793a00700 1 -- 172.21.15.63:0/4062492801 wait complete. 2024-03-23T12:25:38.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:37 smithi063 ceph-mon[141124]: from='client.35866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:38.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:37 smithi111 ceph-mon[106899]: from='client.35866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:38.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.256+0000 7fbc94600700 1 -- 172.21.15.63:0/3259648704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c1037a0 msgr2=0x7fbc8c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.256+0000 7fbc94600700 1 --2- 172.21.15.63:0/3259648704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c1037a0 0x7fbc8c0fce10 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbc78004940 tx=0x7fbc78009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:38.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.257+0000 7fbc94600700 1 -- 172.21.15.63:0/3259648704 shutdown_connections 2024-03-23T12:25:38.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.257+0000 7fbc94600700 1 --2- 172.21.15.63:0/3259648704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c1037a0 0x7fbc8c0fce10 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.257+0000 7fbc94600700 1 --2- 172.21.15.63:0/3259648704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8c102e00 0x7fbc8c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.257+0000 7fbc94600700 1 -- 172.21.15.63:0/3259648704 >> 172.21.15.63:0/3259648704 conn(0x7fbc8c0f89b0 msgr2=0x7fbc8c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:38.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.257+0000 7fbc94600700 1 -- 172.21.15.63:0/3259648704 shutdown_connections 2024-03-23T12:25:38.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.257+0000 7fbc94600700 1 -- 172.21.15.63:0/3259648704 wait complete. 2024-03-23T12:25:38.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.258+0000 7fbc94600700 1 Processor -- start 2024-03-23T12:25:38.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.258+0000 7fbc94600700 1 -- start start 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc94600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c102e00 0x7fbc8c113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc94600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8c1037a0 0x7fbc8c114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc94600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c114690 con 0x7fbc8c102e00 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc94600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c10da80 con 0x7fbc8c1037a0 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc91c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c102e00 0x7fbc8c113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc91c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c102e00 0x7fbc8c113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42684/0 (socket says 172.21.15.63:42684) 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc91c00700 1 -- 172.21.15.63:0/1968993298 learned_addr learned my addr 172.21.15.63:0/1968993298 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc91200700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8c1037a0 0x7fbc8c114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc91c00700 1 -- 172.21.15.63:0/1968993298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8c1037a0 msgr2=0x7fbc8c114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc91c00700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8c1037a0 0x7fbc8c114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.259+0000 7fbc91c00700 1 -- 172.21.15.63:0/1968993298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78005040 con 0x7fbc8c102e00 2024-03-23T12:25:38.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.260+0000 7fbc91c00700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c102e00 0x7fbc8c113b00 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbc80002a50 tx=0x7fbc800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:38.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.260+0000 7fbc89600700 1 -- 172.21.15.63:0/1968993298 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc80007cc0 con 0x7fbc8c102e00 2024-03-23T12:25:38.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.260+0000 7fbc89600700 1 -- 172.21.15.63:0/1968993298 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc8000f040 con 0x7fbc8c102e00 2024-03-23T12:25:38.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.260+0000 7fbc89600700 1 -- 172.21.15.63:0/1968993298 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc8000d9c0 con 0x7fbc8c102e00 2024-03-23T12:25:38.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.260+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c10dd60 con 0x7fbc8c102e00 2024-03-23T12:25:38.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.260+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c10e2f0 con 0x7fbc8c102e00 2024-03-23T12:25:38.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.261+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c0fe550 con 0x7fbc8c102e00 2024-03-23T12:25:38.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.262+0000 7fbc89600700 1 -- 172.21.15.63:0/1968993298 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbc8000db20 con 0x7fbc8c102e00 2024-03-23T12:25:38.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.263+0000 7fbc89600700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc7406c7d0 0x7fbc7406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:38.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.263+0000 7fbc89600700 1 -- 172.21.15.63:0/1968993298 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbc8008b760 con 0x7fbc8c102e00 2024-03-23T12:25:38.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.263+0000 7fbc91200700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc7406c7d0 0x7fbc7406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:38.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.264+0000 7fbc91200700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc7406c7d0 0x7fbc7406ec80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c10ed60 tx=0x7fbc78009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:38.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.268+0000 7fbc89600700 1 -- 172.21.15.63:0/1968993298 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc800593f0 con 0x7fbc8c102e00 2024-03-23T12:25:38.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.525+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbc8c109df0 con 0x7fbc8c102e00 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.525+0000 7fbc89600700 1 -- 172.21.15.63:0/1968993298 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fbc80058f80 con 0x7fbc8c102e00 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:25:38.504 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:25:38.505 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:38.506 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:25:38.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.527+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc7406c7d0 msgr2=0x7fbc7406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.527+0000 7fbc94600700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc7406c7d0 0x7fbc7406ec80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c10ed60 tx=0x7fbc78009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.527+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c102e00 msgr2=0x7fbc8c113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.527+0000 7fbc94600700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c102e00 0x7fbc8c113b00 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbc80002a50 tx=0x7fbc800046b0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.528+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 shutdown_connections 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.528+0000 7fbc94600700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbc7406c7d0 0x7fbc7406ec80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.528+0000 7fbc94600700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc8c102e00 0x7fbc8c113b00 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.528+0000 7fbc94600700 1 --2- 172.21.15.63:0/1968993298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc8c1037a0 0x7fbc8c114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.528+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 >> 172.21.15.63:0/1968993298 conn(0x7fbc8c0f89b0 msgr2=0x7fbc8c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.528+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 shutdown_connections 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.528+0000 7fbc94600700 1 -- 172.21.15.63:0/1968993298 wait complete. 2024-03-23T12:25:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:25:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.665+0000 7f3fe9400700 1 -- 172.21.15.63:0/2277766785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe4102e00 msgr2=0x7f3fe41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.665+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2277766785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe4102e00 0x7f3fe41031d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4004940 tx=0x7f3fd4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.666+0000 7f3fe9400700 1 -- 172.21.15.63:0/2277766785 shutdown_connections 2024-03-23T12:25:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.666+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2277766785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe41037a0 0x7f3fe40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.666+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2277766785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe4102e00 0x7f3fe41031d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.666+0000 7f3fe9400700 1 -- 172.21.15.63:0/2277766785 >> 172.21.15.63:0/2277766785 conn(0x7f3fe40f89b0 msgr2=0x7f3fe40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:38.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.666+0000 7f3fe9400700 1 -- 172.21.15.63:0/2277766785 shutdown_connections 2024-03-23T12:25:38.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.666+0000 7f3fe9400700 1 -- 172.21.15.63:0/2277766785 wait complete. 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe9400700 1 Processor -- start 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe9400700 1 -- start start 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe9400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe4102e00 0x7f3fe4115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe41037a0 0x7f3fe4116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe9400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe41167e0 con 0x7f3fe41037a0 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe9400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe410fbb0 con 0x7f3fe4102e00 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe41037a0 0x7f3fe4116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe1e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe41037a0 0x7f3fe4116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42708/0 (socket says 172.21.15.63:42708) 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.667+0000 7f3fe1e00700 1 -- 172.21.15.63:0/2476932196 learned_addr learned my addr 172.21.15.63:0/2476932196 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fe2800700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe4102e00 0x7f3fe4115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fe1e00700 1 -- 172.21.15.63:0/2476932196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe4102e00 msgr2=0x7f3fe4115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fe1e00700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe4102e00 0x7f3fe4115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fe1e00700 1 -- 172.21.15.63:0/2476932196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd4005040 con 0x7f3fe41037a0 2024-03-23T12:25:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fe1e00700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe41037a0 0x7f3fe4116190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000eb40 tx=0x7f3fd000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fdaa00700 1 -- 172.21.15.63:0/2476932196 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fd0007cc0 con 0x7f3fe41037a0 2024-03-23T12:25:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe410fe30 con 0x7f3fe41037a0 2024-03-23T12:25:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fdaa00700 1 -- 172.21.15.63:0/2476932196 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fd000f040 con 0x7f3fe41037a0 2024-03-23T12:25:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.668+0000 7f3fdaa00700 1 -- 172.21.15.63:0/2476932196 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fd000c8f0 con 0x7f3fe41037a0 2024-03-23T12:25:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.669+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe41103f0 con 0x7f3fe41037a0 2024-03-23T12:25:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.670+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe40fe550 con 0x7f3fe41037a0 2024-03-23T12:25:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.671+0000 7f3fdaa00700 1 -- 172.21.15.63:0/2476932196 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3fd00090d0 con 0x7f3fe41037a0 2024-03-23T12:25:38.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.671+0000 7f3fdaa00700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc806c7d0 0x7f3fc806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:38.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.671+0000 7f3fdaa00700 1 -- 172.21.15.63:0/2476932196 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3fd008b490 con 0x7f3fe41037a0 2024-03-23T12:25:38.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.672+0000 7f3fe2800700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc806c7d0 0x7f3fc806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:38.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.672+0000 7f3fe2800700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc806c7d0 0x7f3fc806ec80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40045a0 tx=0x7f3fd4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:38.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.676+0000 7f3fdaa00700 1 -- 172.21.15.63:0/2476932196 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3fd00591d0 con 0x7f3fe41037a0 2024-03-23T12:25:38.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.917+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fe4109e30 con 0x7f3fc806c7d0 2024-03-23T12:25:38.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.919+0000 7f3fdaa00700 1 -- 172.21.15.63:0/2476932196 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3fe4109e30 con 0x7f3fc806c7d0 2024-03-23T12:25:38.897 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:25:38.897 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:25:38.897 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:25:38.898 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:25:38.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc806c7d0 msgr2=0x7f3fc806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc806c7d0 0x7f3fc806ec80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40045a0 tx=0x7f3fd4009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:38.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe41037a0 msgr2=0x7f3fe4116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:38.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe41037a0 0x7f3fe4116190 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000eb40 tx=0x7f3fd000b040 comp rx=0 tx=0).stop 2024-03-23T12:25:38.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 shutdown_connections 2024-03-23T12:25:38.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fc806c7d0 0x7f3fc806ec80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe4102e00 0x7f3fe4115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 --2- 172.21.15.63:0/2476932196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fe41037a0 0x7f3fe4116190 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:38.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 >> 172.21.15.63:0/2476932196 conn(0x7f3fe40f89b0 msgr2=0x7f3fe40f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:38.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 shutdown_connections 2024-03-23T12:25:38.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:38.921+0000 7f3fe9400700 1 -- 172.21.15.63:0/2476932196 wait complete. 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 -- 172.21.15.63:0/1702435843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fe830 msgr2=0x7f3d04106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1702435843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fe830 0x7f3d04106f00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0004940 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 -- 172.21.15.63:0/1702435843 shutdown_connections 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1702435843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fe830 0x7f3d04106f00 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1702435843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d040fde90 0x7f3d040fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 -- 172.21.15.63:0/1702435843 >> 172.21.15.63:0/1702435843 conn(0x7f3d04072bc0 msgr2=0x7f3d04072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 -- 172.21.15.63:0/1702435843 shutdown_connections 2024-03-23T12:25:39.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.058+0000 7f3d08c00700 1 -- 172.21.15.63:0/1702435843 wait complete. 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.059+0000 7f3d08c00700 1 Processor -- start 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.059+0000 7f3d08c00700 1 -- start start 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d08c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fde90 0x7f3d04115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d08c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d040fe830 0x7f3d041161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d08c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d04116800 con 0x7f3d040fde90 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d08c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0410fbd0 con 0x7f3d040fe830 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fde90 0x7f3d04115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fde90 0x7f3d04115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42730/0 (socket says 172.21.15.63:42730) 2024-03-23T12:25:39.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01e00700 1 -- 172.21.15.63:0/1047598629 learned_addr learned my addr 172.21.15.63:0/1047598629 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01e00700 1 -- 172.21.15.63:0/1047598629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d040fe830 msgr2=0x7f3d041161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01400700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d040fe830 0x7f3d041161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01e00700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d040fe830 0x7f3d041161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01e00700 1 -- 172.21.15.63:0/1047598629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0005040 con 0x7f3d040fde90 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.060+0000 7f3d01400700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d040fe830 0x7f3d041161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.061+0000 7f3d01e00700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fde90 0x7f3d04115c70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400eb40 tx=0x7f3cf400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.061+0000 7f3cfa000700 1 -- 172.21.15.63:0/1047598629 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf4007cc0 con 0x7f3d040fde90 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.061+0000 7f3cfa000700 1 -- 172.21.15.63:0/1047598629 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cf400f040 con 0x7f3d040fde90 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.061+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0410fe50 con 0x7f3d040fde90 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.061+0000 7f3cfa000700 1 -- 172.21.15.63:0/1047598629 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf4009570 con 0x7f3d040fde90 2024-03-23T12:25:39.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.061+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d04110390 con 0x7f3d040fde90 2024-03-23T12:25:39.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.062+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d04106a20 con 0x7f3d040fde90 2024-03-23T12:25:39.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.063+0000 7f3cfa000700 1 -- 172.21.15.63:0/1047598629 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3cf40096d0 con 0x7f3d040fde90 2024-03-23T12:25:39.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.064+0000 7f3cfa000700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3ce806c7d0 0x7f3ce806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:25:39.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.064+0000 7f3cfa000700 1 -- 172.21.15.63:0/1047598629 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3cf408b6f0 con 0x7f3d040fde90 2024-03-23T12:25:39.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.064+0000 7f3d01400700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3ce806c7d0 0x7f3ce806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:25:39.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.065+0000 7f3d01400700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3ce806c7d0 0x7f3ce806ec80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0000c00 tx=0x7f3cf0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:25:39.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.068+0000 7f3cfa000700 1 -- 172.21.15.63:0/1047598629 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3cf4059430 con 0x7f3d040fde90 2024-03-23T12:25:39.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:38 smithi063 ceph-mon[141124]: pgmap v1036: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:39.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:38 smithi063 ceph-mon[141124]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:39.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:38 smithi063 ceph-mon[141124]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:39.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:38 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4062492801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:25:39.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:38 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1968993298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:25:39.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:38 smithi111 ceph-mon[106899]: pgmap v1036: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:39.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:38 smithi111 ceph-mon[106899]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:39.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:38 smithi111 ceph-mon[106899]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:39.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:38 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4062492801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:25:39.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:38 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1968993298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:25:39.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.369+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3d0404bed0 con 0x7f3d040fde90 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.369+0000 7f3cfa000700 1 -- 172.21.15.63:0/1047598629 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f3cf4019e10 con 0x7f3d040fde90 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:25:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:25:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:25:39.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3ce806c7d0 msgr2=0x7f3ce806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:39.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3ce806c7d0 0x7f3ce806ec80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0000c00 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-03-23T12:25:39.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fde90 msgr2=0x7f3d04115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:25:39.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fde90 0x7f3d04115c70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400eb40 tx=0x7f3cf400b040 comp rx=0 tx=0).stop 2024-03-23T12:25:39.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 shutdown_connections 2024-03-23T12:25:39.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3ce806c7d0 0x7f3ce806ec80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:39.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d040fde90 0x7f3d04115c70 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:39.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 --2- 172.21.15.63:0/1047598629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d040fe830 0x7f3d041161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:25:39.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 >> 172.21.15.63:0/1047598629 conn(0x7f3d04072bc0 msgr2=0x7f3d040fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:25:39.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 shutdown_connections 2024-03-23T12:25:39.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:25:39.372+0000 7f3d08c00700 1 -- 172.21.15.63:0/1047598629 wait complete. 2024-03-23T12:25:40.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:39 smithi063 ceph-mon[141124]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:40.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:39 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1047598629' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:25:40.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:39 smithi111 ceph-mon[106899]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:25:40.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:39 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1047598629' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:25:41.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:40 smithi063 ceph-mon[141124]: pgmap v1037: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:41.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:41.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:40 smithi111 ceph-mon[106899]: pgmap v1037: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:41.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:43.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:42 smithi063 ceph-mon[141124]: pgmap v1038: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:43.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:42 smithi111 ceph-mon[106899]: pgmap v1038: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:45.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:44 smithi063 ceph-mon[141124]: pgmap v1039: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:45.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:44 smithi111 ceph-mon[106899]: pgmap v1039: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:47.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:46 smithi063 ceph-mon[141124]: pgmap v1040: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:47.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:46 smithi111 ceph-mon[106899]: pgmap v1040: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:49.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:48 smithi063 ceph-mon[141124]: pgmap v1041: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:49.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:48 smithi111 ceph-mon[106899]: pgmap v1041: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:51.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:50 smithi063 ceph-mon[141124]: pgmap v1042: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:51.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:50 smithi111 ceph-mon[106899]: pgmap v1042: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:52 smithi063 ceph-mon[141124]: pgmap v1043: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:53.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:52 smithi111 ceph-mon[106899]: pgmap v1043: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:55.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:54 smithi063 ceph-mon[141124]: pgmap v1044: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:55.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:54 smithi111 ceph-mon[106899]: pgmap v1044: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:56 smithi063 ceph-mon[141124]: pgmap v1045: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:57.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:57.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:56 smithi111 ceph-mon[106899]: pgmap v1045: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:57.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:25:59.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:25:58 smithi063 ceph-mon[141124]: pgmap v1046: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:25:59.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:25:58 smithi111 ceph-mon[106899]: pgmap v1046: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:01.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:00 smithi063 ceph-mon[141124]: pgmap v1047: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:01.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:00 smithi111 ceph-mon[106899]: pgmap v1047: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:03.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:02 smithi063 ceph-mon[141124]: pgmap v1048: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:03.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:02 smithi111 ceph-mon[106899]: pgmap v1048: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:05.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:04 smithi063 ceph-mon[141124]: pgmap v1049: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:05.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:04 smithi111 ceph-mon[106899]: pgmap v1049: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:07.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:06 smithi063 ceph-mon[141124]: pgmap v1050: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:07.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:06 smithi111 ceph-mon[106899]: pgmap v1050: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:09.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:08 smithi063 ceph-mon[141124]: pgmap v1051: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:09.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:08 smithi111 ceph-mon[106899]: pgmap v1051: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:09.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 -- 172.21.15.63:0/1270291712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8102e00 msgr2=0x7f0db81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:09.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 --2- 172.21.15.63:0/1270291712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8102e00 0x7f0db81031d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0da8004940 tx=0x7f0da8009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:09.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 -- 172.21.15.63:0/1270291712 shutdown_connections 2024-03-23T12:26:09.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 --2- 172.21.15.63:0/1270291712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db81037a0 0x7f0db80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 --2- 172.21.15.63:0/1270291712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8102e00 0x7f0db81031d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 -- 172.21.15.63:0/1270291712 >> 172.21.15.63:0/1270291712 conn(0x7f0db80f89b0 msgr2=0x7f0db80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:09.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 -- 172.21.15.63:0/1270291712 shutdown_connections 2024-03-23T12:26:09.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.526+0000 7f0dbec00700 1 -- 172.21.15.63:0/1270291712 wait complete. 2024-03-23T12:26:09.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.527+0000 7f0dbec00700 1 Processor -- start 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.527+0000 7f0dbec00700 1 -- start start 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0dbec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db8102e00 0x7f0db8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0dbec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db81037a0 0x7f0db8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0dbec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8116880 con 0x7f0db81037a0 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0dbec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db810fc50 con 0x7f0db8102e00 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0db7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db81037a0 0x7f0db8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0db7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db8102e00 0x7f0db8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0db7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db81037a0 0x7f0db8116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58086/0 (socket says 172.21.15.63:58086) 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.528+0000 7f0db7400700 1 -- 172.21.15.63:0/3755875560 learned_addr learned my addr 172.21.15.63:0/3755875560 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0db7400700 1 -- 172.21.15.63:0/3755875560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db8102e00 msgr2=0x7f0db8115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0db7400700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db8102e00 0x7f0db8115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0db7400700 1 -- 172.21.15.63:0/3755875560 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da8005040 con 0x7f0db81037a0 2024-03-23T12:26:09.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0db7400700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db81037a0 0x7f0db8116230 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0da4002a50 tx=0x7f0da4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:09.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0dafe00700 1 -- 172.21.15.63:0/3755875560 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da4007cc0 con 0x7f0db81037a0 2024-03-23T12:26:09.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db810ff30 con 0x7f0db81037a0 2024-03-23T12:26:09.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0dafe00700 1 -- 172.21.15.63:0/3755875560 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0da400f040 con 0x7f0db81037a0 2024-03-23T12:26:09.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0dafe00700 1 -- 172.21.15.63:0/3755875560 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da400d920 con 0x7f0db81037a0 2024-03-23T12:26:09.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.529+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db81104f0 con 0x7f0db81037a0 2024-03-23T12:26:09.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.530+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db80fe550 con 0x7f0db81037a0 2024-03-23T12:26:09.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.531+0000 7f0dafe00700 1 -- 172.21.15.63:0/3755875560 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0da4009680 con 0x7f0db81037a0 2024-03-23T12:26:09.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.532+0000 7f0dafe00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d9c06c7d0 0x7f0d9c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:09.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.532+0000 7f0dafe00700 1 -- 172.21.15.63:0/3755875560 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0da408b7f0 con 0x7f0db81037a0 2024-03-23T12:26:09.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.533+0000 7f0db7e00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d9c06c7d0 0x7f0d9c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:09.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.533+0000 7f0db7e00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d9c06c7d0 0x7f0d9c06ec80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009fd0 tx=0x7f0da8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:09.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.537+0000 7f0dafe00700 1 -- 172.21.15.63:0/3755875560 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0da4059400 con 0x7f0db81037a0 2024-03-23T12:26:09.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.768+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0db8109e30 con 0x7f0d9c06c7d0 2024-03-23T12:26:09.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.769+0000 7f0dafe00700 1 -- 172.21.15.63:0/3755875560 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0db8109e30 con 0x7f0d9c06c7d0 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.771+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d9c06c7d0 msgr2=0x7f0d9c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.771+0000 7f0dbec00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d9c06c7d0 0x7f0d9c06ec80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009fd0 tx=0x7f0da8009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.771+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db81037a0 msgr2=0x7f0db8116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.771+0000 7f0dbec00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db81037a0 0x7f0db8116230 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0da4002a50 tx=0x7f0da4004600 comp rx=0 tx=0).stop 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.772+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 shutdown_connections 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.772+0000 7f0dbec00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d9c06c7d0 0x7f0d9c06ec80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.772+0000 7f0dbec00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db8102e00 0x7f0db8115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.772+0000 7f0dbec00700 1 --2- 172.21.15.63:0/3755875560 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db81037a0 0x7f0db8116230 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.772+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 >> 172.21.15.63:0/3755875560 conn(0x7f0db80f89b0 msgr2=0x7f0db80706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:09.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.772+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 shutdown_connections 2024-03-23T12:26:09.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.772+0000 7f0dbec00700 1 -- 172.21.15.63:0/3755875560 wait complete. 2024-03-23T12:26:09.764 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 -- 172.21.15.63:0/1402278738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 msgr2=0x7fdfd01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1402278738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd01031d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4004940 tx=0x7fdfc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 -- 172.21.15.63:0/1402278738 shutdown_connections 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1402278738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfd01037a0 0x7fdfd00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1402278738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd01031d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 -- 172.21.15.63:0/1402278738 >> 172.21.15.63:0/1402278738 conn(0x7fdfd00f89b0 msgr2=0x7fdfd00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 -- 172.21.15.63:0/1402278738 shutdown_connections 2024-03-23T12:26:09.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.911+0000 7fdfd8000700 1 -- 172.21.15.63:0/1402278738 wait complete. 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.912+0000 7fdfd8000700 1 Processor -- start 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd8000700 1 -- start start 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd8000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd8000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfd01037a0 0x7fdfd0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd8000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd0070d90 con 0x7fdfd0102e00 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd8000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd0070ed0 con 0x7fdfd01037a0 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:09.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58102/0 (socket says 172.21.15.63:58102) 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.913+0000 7fdfd5600700 1 -- 172.21.15.63:0/1862602156 learned_addr learned my addr 172.21.15.63:0/1862602156 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd4c00700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfd01037a0 0x7fdfd0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd5600700 1 -- 172.21.15.63:0/1862602156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfd01037a0 msgr2=0x7fdfd0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd5600700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfd01037a0 0x7fdfd0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd5600700 1 -- 172.21.15.63:0/1862602156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4005040 con 0x7fdfd0102e00 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd4c00700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfd01037a0 0x7fdfd0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd5600700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd00702c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009360 tx=0x7fdfc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfcd600700 1 -- 172.21.15.63:0/1862602156 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc400e040 con 0x7fdfd0102e00 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfcd600700 1 -- 172.21.15.63:0/1862602156 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfc4013420 con 0x7fdfd0102e00 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfcd600700 1 -- 172.21.15.63:0/1862602156 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc401c5f0 con 0x7fdfd0102e00 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd0071120 con 0x7fdfd0102e00 2024-03-23T12:26:09.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.914+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd011a200 con 0x7fdfd0102e00 2024-03-23T12:26:09.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.915+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd00fe550 con 0x7fdfd0102e00 2024-03-23T12:26:09.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.916+0000 7fdfcd600700 1 -- 172.21.15.63:0/1862602156 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fdfc4013590 con 0x7fdfd0102e00 2024-03-23T12:26:09.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.918+0000 7fdfcd600700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfb806c7d0 0x7fdfb806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:09.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.918+0000 7fdfcd600700 1 -- 172.21.15.63:0/1862602156 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdfc408cd80 con 0x7fdfd0102e00 2024-03-23T12:26:09.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.918+0000 7fdfd4c00700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfb806c7d0 0x7fdfb806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:09.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.919+0000 7fdfd4c00700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfb806c7d0 0x7fdfb806ec80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009fd0 tx=0x7fdfbc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:09.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:09.922+0000 7fdfcd600700 1 -- 172.21.15.63:0/1862602156 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdfc4056f80 con 0x7fdfd0102e00 2024-03-23T12:26:10.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.157+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfd00719b0 con 0x7fdfb806c7d0 2024-03-23T12:26:10.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.159+0000 7fdfcd600700 1 -- 172.21.15.63:0/1862602156 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdfd00719b0 con 0x7fdfb806c7d0 2024-03-23T12:26:10.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfb806c7d0 msgr2=0x7fdfb806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfb806c7d0 0x7fdfb806ec80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009fd0 tx=0x7fdfbc00b040 comp rx=0 tx=0).stop 2024-03-23T12:26:10.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 msgr2=0x7fdfd00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd00702c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009360 tx=0x7fdfc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 shutdown_connections 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fdfb806c7d0 0x7fdfb806ec80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdfd0102e00 0x7fdfd00702c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 --2- 172.21.15.63:0/1862602156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfd01037a0 0x7fdfd0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 >> 172.21.15.63:0/1862602156 conn(0x7fdfd00f89b0 msgr2=0x7fdfd00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 shutdown_connections 2024-03-23T12:26:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.161+0000 7fdfd8000700 1 -- 172.21.15.63:0/1862602156 wait complete. 2024-03-23T12:26:10.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.298+0000 7f6a2b400700 1 -- 172.21.15.63:0/2736483586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 msgr2=0x7f6a241031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.298+0000 7f6a2b400700 1 --2- 172.21.15.63:0/2736483586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a241031d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6a18004940 tx=0x7f6a18009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:10.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.299+0000 7f6a2b400700 1 -- 172.21.15.63:0/2736483586 shutdown_connections 2024-03-23T12:26:10.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.299+0000 7f6a2b400700 1 --2- 172.21.15.63:0/2736483586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a241037a0 0x7f6a240fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.299+0000 7f6a2b400700 1 --2- 172.21.15.63:0/2736483586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a241031d0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.299+0000 7f6a2b400700 1 -- 172.21.15.63:0/2736483586 >> 172.21.15.63:0/2736483586 conn(0x7f6a240f89b0 msgr2=0x7f6a240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:10.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.299+0000 7f6a2b400700 1 -- 172.21.15.63:0/2736483586 shutdown_connections 2024-03-23T12:26:10.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.299+0000 7f6a2b400700 1 -- 172.21.15.63:0/2736483586 wait complete. 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a2b400700 1 Processor -- start 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a2b400700 1 -- start start 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a2b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a24115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a2b400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a241037a0 0x7f6a241161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a2b400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24116800 con 0x7f6a24102e00 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a2b400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2410fbd0 con 0x7f6a241037a0 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a28a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a24115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a28a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a24115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58120/0 (socket says 172.21.15.63:58120) 2024-03-23T12:26:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.300+0000 7f6a28a00700 1 -- 172.21.15.63:0/3733753542 learned_addr learned my addr 172.21.15.63:0/3733753542 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a23e00700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a241037a0 0x7f6a241161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a28a00700 1 -- 172.21.15.63:0/3733753542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a241037a0 msgr2=0x7f6a241161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a28a00700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a241037a0 0x7f6a241161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a28a00700 1 -- 172.21.15.63:0/3733753542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18005040 con 0x7f6a24102e00 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a23e00700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a241037a0 0x7f6a241161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a28a00700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a24115c70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009390 tx=0x7f6a18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a20c00700 1 -- 172.21.15.63:0/3733753542 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a1800e040 con 0x7f6a24102e00 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a20c00700 1 -- 172.21.15.63:0/3733753542 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a18013420 con 0x7f6a24102e00 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2410fe50 con 0x7f6a24102e00 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a20c00700 1 -- 172.21.15.63:0/3733753542 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a1801c630 con 0x7f6a24102e00 2024-03-23T12:26:10.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.301+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a24110410 con 0x7f6a24102e00 2024-03-23T12:26:10.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.302+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a240fe550 con 0x7f6a24102e00 2024-03-23T12:26:10.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.303+0000 7f6a20c00700 1 -- 172.21.15.63:0/3733753542 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6a180139e0 con 0x7f6a24102e00 2024-03-23T12:26:10.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.304+0000 7f6a20c00700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6a0c06c7d0 0x7f6a0c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:10.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.304+0000 7f6a20c00700 1 -- 172.21.15.63:0/3733753542 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6a1808cd60 con 0x7f6a24102e00 2024-03-23T12:26:10.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.305+0000 7f6a23e00700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6a0c06c7d0 0x7f6a0c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:10.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.305+0000 7f6a23e00700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6a0c06c7d0 0x7f6a0c06ec80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009b80 tx=0x7f6a1000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:10.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.309+0000 7f6a20c00700 1 -- 172.21.15.63:0/3733753542 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a18091070 con 0x7f6a24102e00 2024-03-23T12:26:10.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.531+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a2404bf40 con 0x7f6a0c06c7d0 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.542+0000 7f6a20c00700 1 -- 172.21.15.63:0/3733753542 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f6a2404bf40 con 0x7f6a0c06c7d0 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (35m) 12m ago 48m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (32m) 12m ago 48m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (32m) 12m ago 47m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (35m) 12m ago 48m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (28m) 12m ago 42m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (28m) 12m ago 42m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (12m) 12m ago 42m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (27m) 12m ago 42m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:26:10.520 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (37m) 12m ago 50m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (36m) 12m ago 47m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (33m) 12m ago 50m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (33m) 12m ago 47m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (36m) 12m ago 48m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (36m) 12m ago 47m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (32m) 12m ago 46m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (31m) 12m ago 45m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (30m) 12m ago 45m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (30m) 12m ago 44m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (29m) 12m ago 44m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (28m) 12m ago 43m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:26:10.521 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (35m) 12m ago 48m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:26:10.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6a0c06c7d0 msgr2=0x7f6a0c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6a0c06c7d0 0x7f6a0c06ec80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009b80 tx=0x7f6a1000b040 comp rx=0 tx=0).stop 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 msgr2=0x7f6a24115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a24115c70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009390 tx=0x7f6a18009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 shutdown_connections 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f6a0c06c7d0 0x7f6a0c06ec80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a24102e00 0x7f6a24115c70 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 --2- 172.21.15.63:0/3733753542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a241037a0 0x7f6a241161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 >> 172.21.15.63:0/3733753542 conn(0x7f6a240f89b0 msgr2=0x7f6a240f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 shutdown_connections 2024-03-23T12:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.544+0000 7f6a2b400700 1 -- 172.21.15.63:0/3733753542 wait complete. 2024-03-23T12:26:10.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 -- 172.21.15.63:0/943188959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105a30 msgr2=0x7f3fec0ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 --2- 172.21.15.63:0/943188959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105a30 0x7f3fec0ff0a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8004940 tx=0x7f3fd8009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:10.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 -- 172.21.15.63:0/943188959 shutdown_connections 2024-03-23T12:26:10.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 --2- 172.21.15.63:0/943188959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105a30 0x7f3fec0ff0a0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 --2- 172.21.15.63:0/943188959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fec105090 0x7f3fec105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 -- 172.21.15.63:0/943188959 >> 172.21.15.63:0/943188959 conn(0x7f3fec0fac00 msgr2=0x7f3fec0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:10.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 -- 172.21.15.63:0/943188959 shutdown_connections 2024-03-23T12:26:10.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.680+0000 7f3ff2200700 1 -- 172.21.15.63:0/943188959 wait complete. 2024-03-23T12:26:10.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.681+0000 7f3ff2200700 1 Processor -- start 2024-03-23T12:26:10.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.681+0000 7f3ff2200700 1 -- start start 2024-03-23T12:26:10.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3ff2200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105090 0x7f3fec115ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3ff2200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fec105a30 0x7f3fec116010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3ff2200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec116660 con 0x7f3fec105090 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3ff2200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec10fc40 con 0x7f3fec105a30 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3feb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105090 0x7f3fec115ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3feb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105090 0x7f3fec115ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58146/0 (socket says 172.21.15.63:58146) 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3feb400700 1 -- 172.21.15.63:0/3650592851 learned_addr learned my addr 172.21.15.63:0/3650592851 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3feaa00700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fec105a30 0x7f3fec116010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3feb400700 1 -- 172.21.15.63:0/3650592851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fec105a30 msgr2=0x7f3fec116010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3feb400700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fec105a30 0x7f3fec116010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.682+0000 7f3feb400700 1 -- 172.21.15.63:0/3650592851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8005040 con 0x7f3fec105090 2024-03-23T12:26:10.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.683+0000 7f3feb400700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105090 0x7f3fec115ad0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc002a50 tx=0x7f3fdc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:10.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.683+0000 7f3fe3400700 1 -- 172.21.15.63:0/3650592851 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fdc007cc0 con 0x7f3fec105090 2024-03-23T12:26:10.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.683+0000 7f3fe3400700 1 -- 172.21.15.63:0/3650592851 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fdc00f040 con 0x7f3fec105090 2024-03-23T12:26:10.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.683+0000 7f3fe3400700 1 -- 172.21.15.63:0/3650592851 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fdc00da70 con 0x7f3fec105090 2024-03-23T12:26:10.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.683+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec10ff20 con 0x7f3fec105090 2024-03-23T12:26:10.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.683+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec1104e0 con 0x7f3fec105090 2024-03-23T12:26:10.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.684+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec107560 con 0x7f3fec105090 2024-03-23T12:26:10.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.685+0000 7f3fe3400700 1 -- 172.21.15.63:0/3650592851 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3fdc00dc10 con 0x7f3fec105090 2024-03-23T12:26:10.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.686+0000 7f3fe3400700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fd006c7c0 0x7f3fd006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:10.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.686+0000 7f3fe3400700 1 -- 172.21.15.63:0/3650592851 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3fdc08b180 con 0x7f3fec105090 2024-03-23T12:26:10.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.686+0000 7f3feaa00700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fd006c7c0 0x7f3fd006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:10.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.687+0000 7f3feaa00700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fd006c7c0 0x7f3fd006ec70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800b020 tx=0x7f3fd8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:10.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.690+0000 7f3fe3400700 1 -- 172.21.15.63:0/3650592851 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3fdc058e10 con 0x7f3fec105090 2024-03-23T12:26:10.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.990+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3fec10c050 con 0x7f3fec105090 2024-03-23T12:26:10.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.990+0000 7f3fe3400700 1 -- 172.21.15.63:0/3650592851 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f3fdc0589a0 con 0x7f3fec105090 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:26:10.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fd006c7c0 msgr2=0x7f3fd006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fd006c7c0 0x7f3fd006ec70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800b020 tx=0x7f3fd8009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:10.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105090 msgr2=0x7f3fec115ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:10.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105090 0x7f3fec115ad0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc002a50 tx=0x7f3fdc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 shutdown_connections 2024-03-23T12:26:10.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3fd006c7c0 0x7f3fd006ec70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fec105090 0x7f3fec115ad0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 --2- 172.21.15.63:0/3650592851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fec105a30 0x7f3fec116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:10.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 >> 172.21.15.63:0/3650592851 conn(0x7f3fec0fac00 msgr2=0x7f3fec0fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:10.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 shutdown_connections 2024-03-23T12:26:10.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:10.993+0000 7f3ff2200700 1 -- 172.21.15.63:0/3650592851 wait complete. 2024-03-23T12:26:11.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 -- 172.21.15.63:0/2288143708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e01037a0 msgr2=0x7f20e00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 --2- 172.21.15.63:0/2288143708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e01037a0 0x7f20e00fce10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f20cc004940 tx=0x7f20cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:11.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 -- 172.21.15.63:0/2288143708 shutdown_connections 2024-03-23T12:26:11.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 --2- 172.21.15.63:0/2288143708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e01037a0 0x7f20e00fce10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 --2- 172.21.15.63:0/2288143708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0102e00 0x7f20e01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 -- 172.21.15.63:0/2288143708 >> 172.21.15.63:0/2288143708 conn(0x7f20e00f89b0 msgr2=0x7f20e00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:11.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 -- 172.21.15.63:0/2288143708 shutdown_connections 2024-03-23T12:26:11.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.130+0000 7f20e8600700 1 -- 172.21.15.63:0/2288143708 wait complete. 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.131+0000 7f20e8600700 1 Processor -- start 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.131+0000 7f20e8600700 1 -- start start 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.131+0000 7f20e8600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102e00 0x7f20e0115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.131+0000 7f20e8600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e01037a0 0x7f20e01161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.131+0000 7f20e8600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0116840 con 0x7f20e0102e00 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.131+0000 7f20e8600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e010fc10 con 0x7f20e01037a0 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.131+0000 7f20e5c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102e00 0x7f20e0115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e5c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102e00 0x7f20e0115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58156/0 (socket says 172.21.15.63:58156) 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e5c00700 1 -- 172.21.15.63:0/811315455 learned_addr learned my addr 172.21.15.63:0/811315455 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:11.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e5200700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e01037a0 0x7f20e01161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e5c00700 1 -- 172.21.15.63:0/811315455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e01037a0 msgr2=0x7f20e01161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e5c00700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e01037a0 0x7f20e01161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e5c00700 1 -- 172.21.15.63:0/811315455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc005040 con 0x7f20e0102e00 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e5c00700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102e00 0x7f20e0115cb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f20d400eb40 tx=0x7f20d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20dd600700 1 -- 172.21.15.63:0/811315455 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d4007cc0 con 0x7f20e0102e00 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20dd600700 1 -- 172.21.15.63:0/811315455 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20d400f040 con 0x7f20e0102e00 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20dd600700 1 -- 172.21.15.63:0/811315455 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d4009570 con 0x7f20e0102e00 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.132+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e010fe90 con 0x7f20e0102e00 2024-03-23T12:26:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.133+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e0110400 con 0x7f20e0102e00 2024-03-23T12:26:11.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.134+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e00fe550 con 0x7f20e0102e00 2024-03-23T12:26:11.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.135+0000 7f20dd600700 1 -- 172.21.15.63:0/811315455 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f20d40096d0 con 0x7f20e0102e00 2024-03-23T12:26:11.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.136+0000 7f20dd600700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.136+0000 7f20dd600700 1 -- 172.21.15.63:0/811315455 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f20d408b7e0 con 0x7f20e0102e00 2024-03-23T12:26:11.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.136+0000 7f20e5200700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.137+0000 7f20e5200700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00b0d0 tx=0x7f20cc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:11.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.141+0000 7f20dd600700 1 -- 172.21.15.63:0/811315455 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f20d40594a0 con 0x7f20e0102e00 2024-03-23T12:26:11.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:10 smithi063 ceph-mon[141124]: pgmap v1052: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:11.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:10 smithi063 ceph-mon[141124]: from='client.35894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:11.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:10 smithi063 ceph-mon[141124]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:11.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:10 smithi111 ceph-mon[106899]: pgmap v1052: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:10 smithi111 ceph-mon[106899]: from='client.35894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:10 smithi111 ceph-mon[106899]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:11.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:11.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.399+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f20e004bed0 con 0x7f20e0102e00 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.399+0000 7f20dd600700 1 -- 172.21.15.63:0/811315455 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f20d4059030 con 0x7f20e0102e00 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:26:11.378 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:26:11.379 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 msgr2=0x7f20c806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00b0d0 tx=0x7f20cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102e00 msgr2=0x7f20e0115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102e00 0x7f20e0115cb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f20d400eb40 tx=0x7f20d400b040 comp rx=0 tx=0).stop 2024-03-23T12:26:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 shutdown_connections 2024-03-23T12:26:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102e00 0x7f20e0115cb0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 --2- 172.21.15.63:0/811315455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e01037a0 0x7f20e01161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 >> 172.21.15.63:0/811315455 conn(0x7f20e00f89b0 msgr2=0x7f20e0070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 shutdown_connections 2024-03-23T12:26:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.402+0000 7f20e8600700 1 -- 172.21.15.63:0/811315455 wait complete. 2024-03-23T12:26:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:26:11.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 -- 172.21.15.63:0/3874840189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 msgr2=0x7f5c301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 --2- 172.21.15.63:0/3874840189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c301031d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f5c24004940 tx=0x7f5c24009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:11.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 -- 172.21.15.63:0/3874840189 shutdown_connections 2024-03-23T12:26:11.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 --2- 172.21.15.63:0/3874840189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c301037a0 0x7f5c300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 --2- 172.21.15.63:0/3874840189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c30102e00 0x7f5c301031d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 -- 172.21.15.63:0/3874840189 >> 172.21.15.63:0/3874840189 conn(0x7f5c300f89b0 msgr2=0x7f5c300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:11.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 -- 172.21.15.63:0/3874840189 shutdown_connections 2024-03-23T12:26:11.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.540+0000 7f5c38200700 1 -- 172.21.15.63:0/3874840189 wait complete. 2024-03-23T12:26:11.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.541+0000 7f5c38200700 1 Processor -- start 2024-03-23T12:26:11.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.541+0000 7f5c38200700 1 -- start start 2024-03-23T12:26:11.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.541+0000 7f5c38200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c30102e00 0x7f5c30115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.541+0000 7f5c38200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c301037a0 0x7f5c30116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.541+0000 7f5c38200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c30116860 con 0x7f5c301037a0 2024-03-23T12:26:11.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.541+0000 7f5c38200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3010fc30 con 0x7f5c30102e00 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.541+0000 7f5c34e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c301037a0 0x7f5c30116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c34e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c301037a0 0x7f5c30116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47648/0 (socket says 172.21.15.63:47648) 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c34e00700 1 -- 172.21.15.63:0/3094925858 learned_addr learned my addr 172.21.15.63:0/3094925858 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c35800700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c30102e00 0x7f5c30115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c34e00700 1 -- 172.21.15.63:0/3094925858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c30102e00 msgr2=0x7f5c30115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c34e00700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c30102e00 0x7f5c30115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c34e00700 1 -- 172.21.15.63:0/3094925858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c24005040 con 0x7f5c301037a0 2024-03-23T12:26:11.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c34e00700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c301037a0 0x7f5c30116210 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c002a50 tx=0x7f5c1c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:11.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c2d600700 1 -- 172.21.15.63:0/3094925858 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c1c007cc0 con 0x7f5c301037a0 2024-03-23T12:26:11.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c2d600700 1 -- 172.21.15.63:0/3094925858 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c1c00f040 con 0x7f5c301037a0 2024-03-23T12:26:11.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c2d600700 1 -- 172.21.15.63:0/3094925858 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c1c00d990 con 0x7f5c301037a0 2024-03-23T12:26:11.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.542+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3010ff10 con 0x7f5c301037a0 2024-03-23T12:26:11.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.543+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c30110480 con 0x7f5c301037a0 2024-03-23T12:26:11.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.544+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c300fe550 con 0x7f5c301037a0 2024-03-23T12:26:11.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.545+0000 7f5c2d600700 1 -- 172.21.15.63:0/3094925858 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5c1c009710 con 0x7f5c301037a0 2024-03-23T12:26:11.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.546+0000 7f5c2d600700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.546+0000 7f5c2d600700 1 -- 172.21.15.63:0/3094925858 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5c1c08b910 con 0x7f5c301037a0 2024-03-23T12:26:11.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.546+0000 7f5c35800700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.546+0000 7f5c35800700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400b020 tx=0x7f5c24009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:11.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.550+0000 7f5c2d600700 1 -- 172.21.15.63:0/3094925858 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5c1c059520 con 0x7f5c301037a0 2024-03-23T12:26:11.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.786+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c30109e30 con 0x7f5c1806c7d0 2024-03-23T12:26:11.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.787+0000 7f5c2d600700 1 -- 172.21.15.63:0/3094925858 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5c30109e30 con 0x7f5c1806c7d0 2024-03-23T12:26:11.765 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:26:11.765 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:26:11.765 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:26:11.765 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:26:11.765 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:26:11.766 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 msgr2=0x7f5c1806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400b020 tx=0x7f5c24009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c301037a0 msgr2=0x7f5c30116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c301037a0 0x7f5c30116210 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c002a50 tx=0x7f5c1c004600 comp rx=0 tx=0).stop 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 shutdown_connections 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c1806c7d0 0x7f5c1806ec80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c30102e00 0x7f5c30115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 --2- 172.21.15.63:0/3094925858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c301037a0 0x7f5c30116210 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.789+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 >> 172.21.15.63:0/3094925858 conn(0x7f5c300f89b0 msgr2=0x7f5c300f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.790+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 shutdown_connections 2024-03-23T12:26:11.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.790+0000 7f5c38200700 1 -- 172.21.15.63:0/3094925858 wait complete. 2024-03-23T12:26:11.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.931+0000 7fe02cc00700 1 -- 172.21.15.63:0/4123298233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 msgr2=0x7fe0280fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.931+0000 7fe02cc00700 1 --2- 172.21.15.63:0/4123298233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe0280fce10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe014004940 tx=0x7fe014009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.932+0000 7fe02cc00700 1 -- 172.21.15.63:0/4123298233 shutdown_connections 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.932+0000 7fe02cc00700 1 --2- 172.21.15.63:0/4123298233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe0280fce10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.932+0000 7fe02cc00700 1 --2- 172.21.15.63:0/4123298233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe028102e00 0x7fe0281031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.932+0000 7fe02cc00700 1 -- 172.21.15.63:0/4123298233 >> 172.21.15.63:0/4123298233 conn(0x7fe0280f89b0 msgr2=0x7fe0280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.932+0000 7fe02cc00700 1 -- 172.21.15.63:0/4123298233 shutdown_connections 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.932+0000 7fe02cc00700 1 -- 172.21.15.63:0/4123298233 wait complete. 2024-03-23T12:26:11.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.932+0000 7fe02cc00700 1 Processor -- start 2024-03-23T12:26:11.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe02cc00700 1 -- start start 2024-03-23T12:26:11.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe02cc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe028102e00 0x7fe0280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe02cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe028070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe02cc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe028070d90 con 0x7fe0281037a0 2024-03-23T12:26:11.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe02cc00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe028070ed0 con 0x7fe028102e00 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe025400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe028070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe025400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe028070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47670/0 (socket says 172.21.15.63:47670) 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.933+0000 7fe025400700 1 -- 172.21.15.63:0/3881219387 learned_addr learned my addr 172.21.15.63:0/3881219387 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe025e00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe028102e00 0x7fe0280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe025400700 1 -- 172.21.15.63:0/3881219387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe028102e00 msgr2=0x7fe0280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe025400700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe028102e00 0x7fe0280702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe025400700 1 -- 172.21.15.63:0/3881219387 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe014005040 con 0x7fe0281037a0 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe025400700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe028070800 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe014009390 tx=0x7fe014009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe01e000700 1 -- 172.21.15.63:0/3881219387 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe01400e040 con 0x7fe0281037a0 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe01e000700 1 -- 172.21.15.63:0/3881219387 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe014013420 con 0x7fe0281037a0 2024-03-23T12:26:11.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe028071120 con 0x7fe0281037a0 2024-03-23T12:26:11.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.934+0000 7fe01e000700 1 -- 172.21.15.63:0/3881219387 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe01401c630 con 0x7fe0281037a0 2024-03-23T12:26:11.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.935+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02811a230 con 0x7fe0281037a0 2024-03-23T12:26:11.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.936+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0280fe550 con 0x7fe0281037a0 2024-03-23T12:26:11.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.937+0000 7fe01e000700 1 -- 172.21.15.63:0/3881219387 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe014013a00 con 0x7fe0281037a0 2024-03-23T12:26:11.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.937+0000 7fe01e000700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe00c06c7d0 0x7fe00c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:11.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.938+0000 7fe01e000700 1 -- 172.21.15.63:0/3881219387 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe01408cda0 con 0x7fe0281037a0 2024-03-23T12:26:11.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.938+0000 7fe025e00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe00c06c7d0 0x7fe00c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:11.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.938+0000 7fe025e00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe00c06c7d0 0x7fe00c06ec80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe018004990 tx=0x7fe018009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:11.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:11.942+0000 7fe01e000700 1 -- 172.21.15.63:0/3881219387 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe014056fa0 con 0x7fe0281037a0 2024-03-23T12:26:12.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:11 smithi063 ceph-mon[141124]: from='client.35902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:12.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:11 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3650592851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:26:12.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:11 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/811315455' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:26:12.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:11 smithi111 ceph-mon[106899]: from='client.35902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:12.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:11 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3650592851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:26:12.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:11 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/811315455' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:26:12.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.242+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe028109dd0 con 0x7fe0281037a0 2024-03-23T12:26:12.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.242+0000 7fe01e000700 1 -- 172.21.15.63:0/3881219387 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fe01405a5c0 con 0x7fe0281037a0 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:26:12.221 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:26:12.222 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:26:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe00c06c7d0 msgr2=0x7fe00c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe00c06c7d0 0x7fe00c06ec80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe018004990 tx=0x7fe018009ee0 comp rx=0 tx=0).stop 2024-03-23T12:26:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 msgr2=0x7fe028070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe028070800 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe014009390 tx=0x7fe014009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 shutdown_connections 2024-03-23T12:26:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe00c06c7d0 0x7fe00c06ec80 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe028102e00 0x7fe0280702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 --2- 172.21.15.63:0/3881219387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0281037a0 0x7fe028070800 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 >> 172.21.15.63:0/3881219387 conn(0x7fe0280f89b0 msgr2=0x7fe0280f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 shutdown_connections 2024-03-23T12:26:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:12.245+0000 7fe02cc00700 1 -- 172.21.15.63:0/3881219387 wait complete. 2024-03-23T12:26:13.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:12 smithi063 ceph-mon[141124]: pgmap v1053: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:13.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:12 smithi063 ceph-mon[141124]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:13.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:12 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3881219387' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:26:13.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:12 smithi111 ceph-mon[106899]: pgmap v1053: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:13.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:12 smithi111 ceph-mon[106899]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:13.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:12 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3881219387' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:26:15.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:14 smithi063 ceph-mon[141124]: pgmap v1054: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:15.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:14 smithi111 ceph-mon[106899]: pgmap v1054: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:17.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:16 smithi063 ceph-mon[141124]: pgmap v1055: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:17.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:16 smithi111 ceph-mon[106899]: pgmap v1055: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:19.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:18 smithi063 ceph-mon[141124]: pgmap v1056: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:19.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:18 smithi111 ceph-mon[106899]: pgmap v1056: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:21.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:20 smithi063 ceph-mon[141124]: pgmap v1057: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:21.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:20 smithi111 ceph-mon[106899]: pgmap v1057: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:23.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:22 smithi063 ceph-mon[141124]: pgmap v1058: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:23.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:22 smithi111 ceph-mon[106899]: pgmap v1058: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:25.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:24 smithi063 ceph-mon[141124]: pgmap v1059: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:25.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:24 smithi111 ceph-mon[106899]: pgmap v1059: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:26.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:26.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:27.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:26 smithi063 ceph-mon[141124]: pgmap v1060: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:27.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:26 smithi111 ceph-mon[106899]: pgmap v1060: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:29.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:28 smithi063 ceph-mon[141124]: pgmap v1061: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:29.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:28 smithi111 ceph-mon[106899]: pgmap v1061: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:31.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:30 smithi063 ceph-mon[141124]: pgmap v1062: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:31.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:30 smithi111 ceph-mon[106899]: pgmap v1062: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:33.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:32 smithi063 ceph-mon[141124]: pgmap v1063: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:33.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:32 smithi111 ceph-mon[106899]: pgmap v1063: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:35.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:34 smithi063 ceph-mon[141124]: pgmap v1064: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:35.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:34 smithi111 ceph-mon[106899]: pgmap v1064: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:37.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:36 smithi063 ceph-mon[141124]: pgmap v1065: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:37.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:36 smithi111 ceph-mon[106899]: pgmap v1065: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:39.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:38 smithi111 ceph-mon[106899]: pgmap v1066: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:39.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:38 smithi063 ceph-mon[141124]: pgmap v1066: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:41.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:40 smithi111 ceph-mon[106899]: pgmap v1067: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:41.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:41.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:40 smithi063 ceph-mon[141124]: pgmap v1067: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:41.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 -- 172.21.15.63:0/1041245876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 msgr2=0x7fe9c0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/1041245876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c0106f00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac004940 tx=0x7fe9ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 -- 172.21.15.63:0/1041245876 shutdown_connections 2024-03-23T12:26:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/1041245876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c0106f00 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/1041245876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c00fde90 0x7fe9c00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 -- 172.21.15.63:0/1041245876 >> 172.21.15.63:0/1041245876 conn(0x7fe9c0072bc0 msgr2=0x7fe9c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 -- 172.21.15.63:0/1041245876 shutdown_connections 2024-03-23T12:26:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.400+0000 7fe9c7e00700 1 -- 172.21.15.63:0/1041245876 wait complete. 2024-03-23T12:26:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.401+0000 7fe9c7e00700 1 Processor -- start 2024-03-23T12:26:42.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.401+0000 7fe9c7e00700 1 -- start start 2024-03-23T12:26:42.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.401+0000 7fe9c7e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c00fde90 0x7fe9c00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.401+0000 7fe9c7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.401+0000 7fe9c7e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00fc820 con 0x7fe9c00fe830 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.401+0000 7fe9c7e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00fc960 con 0x7fe9c00fde90 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c4a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c00fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53886/0 (socket says 172.21.15.63:53886) 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c4a00700 1 -- 172.21.15.63:0/3857654648 learned_addr learned my addr 172.21.15.63:0/3857654648 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c5400700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c00fde90 0x7fe9c00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c4a00700 1 -- 172.21.15.63:0/3857654648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c00fde90 msgr2=0x7fe9c00fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c4a00700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c00fde90 0x7fe9c00fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c4a00700 1 -- 172.21.15.63:0/3857654648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac005040 con 0x7fe9c00fe830 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.402+0000 7fe9c4a00700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c00fc290 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac000c00 tx=0x7fe9ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:42.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.403+0000 7fe9bd600700 1 -- 172.21.15.63:0/3857654648 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9ac021070 con 0x7fe9c00fe830 2024-03-23T12:26:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.403+0000 7fe9bd600700 1 -- 172.21.15.63:0/3857654648 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9ac013e00 con 0x7fe9c00fe830 2024-03-23T12:26:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.403+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c00fcbb0 con 0x7fe9c00fe830 2024-03-23T12:26:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.403+0000 7fe9bd600700 1 -- 172.21.15.63:0/3857654648 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9ac00e070 con 0x7fe9c00fe830 2024-03-23T12:26:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.403+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c011a130 con 0x7fe9c00fe830 2024-03-23T12:26:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.404+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0065e30 con 0x7fe9c00fe830 2024-03-23T12:26:42.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.405+0000 7fe9bd600700 1 -- 172.21.15.63:0/3857654648 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe9ac013950 con 0x7fe9c00fe830 2024-03-23T12:26:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.406+0000 7fe9bd600700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe9a806c7d0 0x7fe9a806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.406+0000 7fe9bd600700 1 -- 172.21.15.63:0/3857654648 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe9ac08c9e0 con 0x7fe9c00fe830 2024-03-23T12:26:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.406+0000 7fe9c5400700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe9a806c7d0 0x7fe9a806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.407+0000 7fe9c5400700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe9a806c7d0 0x7fe9a806ec80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009b80 tx=0x7fe9b400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:42.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.411+0000 7fe9bd600700 1 -- 172.21.15.63:0/3857654648 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe9ac056c60 con 0x7fe9c00fe830 2024-03-23T12:26:42.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.643+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9c004bed0 con 0x7fe9a806c7d0 2024-03-23T12:26:42.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.644+0000 7fe9bd600700 1 -- 172.21.15.63:0/3857654648 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe9c004bed0 con 0x7fe9a806c7d0 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe9a806c7d0 msgr2=0x7fe9a806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe9a806c7d0 0x7fe9a806ec80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009b80 tx=0x7fe9b400b040 comp rx=0 tx=0).stop 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 msgr2=0x7fe9c00fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c00fc290 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac000c00 tx=0x7fe9ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 shutdown_connections 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe9a806c7d0 0x7fe9a806ec80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c00fde90 0x7fe9c00fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 --2- 172.21.15.63:0/3857654648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9c00fe830 0x7fe9c00fc290 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 >> 172.21.15.63:0/3857654648 conn(0x7fe9c0072bc0 msgr2=0x7fe9c00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:42.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 shutdown_connections 2024-03-23T12:26:42.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.647+0000 7fe9c7e00700 1 -- 172.21.15.63:0/3857654648 wait complete. 2024-03-23T12:26:42.639 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.785+0000 7f349a800700 1 -- 172.21.15.63:0/1408058380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 msgr2=0x7f34941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.785+0000 7f349a800700 1 --2- 172.21.15.63:0/1408058380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f34941031d0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3484004940 tx=0x7f3484009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.785+0000 7f349a800700 1 -- 172.21.15.63:0/1408058380 shutdown_connections 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.786+0000 7f349a800700 1 --2- 172.21.15.63:0/1408058380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34941037a0 0x7f34940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.786+0000 7f349a800700 1 --2- 172.21.15.63:0/1408058380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f34941031d0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.786+0000 7f349a800700 1 -- 172.21.15.63:0/1408058380 >> 172.21.15.63:0/1408058380 conn(0x7f34940f89b0 msgr2=0x7f34940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.786+0000 7f349a800700 1 -- 172.21.15.63:0/1408058380 shutdown_connections 2024-03-23T12:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.786+0000 7f349a800700 1 -- 172.21.15.63:0/1408058380 wait complete. 2024-03-23T12:26:42.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.787+0000 7f349a800700 1 Processor -- start 2024-03-23T12:26:42.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.787+0000 7f349a800700 1 -- start start 2024-03-23T12:26:42.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f349a800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f3494113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:42.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f349a800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34941037a0 0x7f3494113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:42.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f349a800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34941145f0 con 0x7f3494102e00 2024-03-23T12:26:42.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f349a800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349410d9e0 con 0x7f34941037a0 2024-03-23T12:26:42.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f3493400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f3494113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:42.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f3493400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f3494113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53892/0 (socket says 172.21.15.63:53892) 2024-03-23T12:26:42.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f3493400700 1 -- 172.21.15.63:0/1336809783 learned_addr learned my addr 172.21.15.63:0/1336809783 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:42.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f3493400700 1 -- 172.21.15.63:0/1336809783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34941037a0 msgr2=0x7f3494113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:42.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f3493400700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34941037a0 0x7f3494113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:42.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.788+0000 7f3493400700 1 -- 172.21.15.63:0/1336809783 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484005040 con 0x7f3494102e00 2024-03-23T12:26:42.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.789+0000 7f3493400700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f3494113a60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3484007f50 tx=0x7f3484009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:42.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.789+0000 7f348b400700 1 -- 172.21.15.63:0/1336809783 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3484021070 con 0x7f3494102e00 2024-03-23T12:26:42.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.789+0000 7f348b400700 1 -- 172.21.15.63:0/1336809783 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3484013e00 con 0x7f3494102e00 2024-03-23T12:26:42.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.789+0000 7f348b400700 1 -- 172.21.15.63:0/1336809783 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f348400e070 con 0x7f3494102e00 2024-03-23T12:26:42.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.789+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349410dc60 con 0x7f3494102e00 2024-03-23T12:26:42.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.789+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349410e220 con 0x7f3494102e00 2024-03-23T12:26:42.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.790+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34940fe550 con 0x7f3494102e00 2024-03-23T12:26:42.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.791+0000 7f348b400700 1 -- 172.21.15.63:0/1336809783 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3484013950 con 0x7f3494102e00 2024-03-23T12:26:42.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.793+0000 7f348b400700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f347806c7d0 0x7f347806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:42.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.793+0000 7f348b400700 1 -- 172.21.15.63:0/1336809783 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f348408cd70 con 0x7f3494102e00 2024-03-23T12:26:42.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.793+0000 7f348be00700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f347806c7d0 0x7f347806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:42.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.793+0000 7f348be00700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f347806c7d0 0x7f347806ec80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f349410eba0 tx=0x7f347c00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:42.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:42.797+0000 7f348b400700 1 -- 172.21.15.63:0/1336809783 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f348405aa00 con 0x7f3494102e00 2024-03-23T12:26:43.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.030+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3494109de0 con 0x7f347806c7d0 2024-03-23T12:26:43.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.031+0000 7f348b400700 1 -- 172.21.15.63:0/1336809783 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3494109de0 con 0x7f347806c7d0 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f347806c7d0 msgr2=0x7f347806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f347806c7d0 0x7f347806ec80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f349410eba0 tx=0x7f347c00b040 comp rx=0 tx=0).stop 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 msgr2=0x7f3494113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f3494113a60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3484007f50 tx=0x7f3484009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 shutdown_connections 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f347806c7d0 0x7f347806ec80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3494102e00 0x7f3494113a60 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 --2- 172.21.15.63:0/1336809783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34941037a0 0x7f3494113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 >> 172.21.15.63:0/1336809783 conn(0x7f34940f89b0 msgr2=0x7f34940f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 shutdown_connections 2024-03-23T12:26:43.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.034+0000 7f349a800700 1 -- 172.21.15.63:0/1336809783 wait complete. 2024-03-23T12:26:43.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:42 smithi063 ceph-mon[141124]: pgmap v1068: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:43.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:42 smithi111 ceph-mon[106899]: pgmap v1068: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 -- 172.21.15.63:0/1654005102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a0102e00 msgr2=0x7f31a01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 --2- 172.21.15.63:0/1654005102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a0102e00 0x7f31a01031d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3190004940 tx=0x7f3190009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 -- 172.21.15.63:0/1654005102 shutdown_connections 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 --2- 172.21.15.63:0/1654005102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a01037a0 0x7f31a00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 --2- 172.21.15.63:0/1654005102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a0102e00 0x7f31a01031d0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 -- 172.21.15.63:0/1654005102 >> 172.21.15.63:0/1654005102 conn(0x7f31a00f89b0 msgr2=0x7f31a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 -- 172.21.15.63:0/1654005102 shutdown_connections 2024-03-23T12:26:43.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.173+0000 7f31a6c00700 1 -- 172.21.15.63:0/1654005102 wait complete. 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.174+0000 7f31a6c00700 1 Processor -- start 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.174+0000 7f31a6c00700 1 -- start start 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.174+0000 7f31a6c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a0102e00 0x7f31a0115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.174+0000 7f31a6c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a01037a0 0x7f31a0116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.174+0000 7f31a6c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0116860 con 0x7f31a01037a0 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.174+0000 7f31a6c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a010fc30 con 0x7f31a0102e00 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a01037a0 0x7f31a0116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a01037a0 0x7f31a0116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53912/0 (socket says 172.21.15.63:53912) 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319f400700 1 -- 172.21.15.63:0/3071898669 learned_addr learned my addr 172.21.15.63:0/3071898669 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:43.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319fe00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a0102e00 0x7f31a0115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319f400700 1 -- 172.21.15.63:0/3071898669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a0102e00 msgr2=0x7f31a0115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319f400700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a0102e00 0x7f31a0115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319f400700 1 -- 172.21.15.63:0/3071898669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3190005040 con 0x7f31a01037a0 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.175+0000 7f319f400700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a01037a0 0x7f31a0116210 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f318c00c890 tx=0x7f318c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.176+0000 7f3197e00700 1 -- 172.21.15.63:0/3071898669 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318c00a700 con 0x7f31a01037a0 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.176+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a010ff10 con 0x7f31a01037a0 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.176+0000 7f3197e00700 1 -- 172.21.15.63:0/3071898669 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f318c00ad10 con 0x7f31a01037a0 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.176+0000 7f3197e00700 1 -- 172.21.15.63:0/3071898669 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f318c012e70 con 0x7f31a01037a0 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.176+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a01104d0 con 0x7f31a01037a0 2024-03-23T12:26:43.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.177+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a00fe550 con 0x7f31a01037a0 2024-03-23T12:26:43.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.178+0000 7f3197e00700 1 -- 172.21.15.63:0/3071898669 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f318c00a860 con 0x7f31a01037a0 2024-03-23T12:26:43.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.179+0000 7f3197e00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f318406c7d0 0x7f318406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.179+0000 7f3197e00700 1 -- 172.21.15.63:0/3071898669 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f318c08bae0 con 0x7f31a01037a0 2024-03-23T12:26:43.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.180+0000 7f319fe00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f318406c7d0 0x7f318406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.180+0000 7f319fe00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f318406c7d0 0x7f318406ec80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f319000b840 tx=0x7f3190009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:43.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.184+0000 7f3197e00700 1 -- 172.21.15.63:0/3071898669 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f318c0597f0 con 0x7f31a01037a0 2024-03-23T12:26:43.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.406+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a0109e30 con 0x7f318406c7d0 2024-03-23T12:26:43.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.416+0000 7f3197e00700 1 -- 172.21.15.63:0/3071898669 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f31a0109e30 con 0x7f318406c7d0 2024-03-23T12:26:43.393 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:26:43.393 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (36m) 13m ago 49m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:26:43.393 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (33m) 13m ago 49m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:26:43.393 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (33m) 13m ago 48m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (35m) 13m ago 48m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (29m) 13m ago 42m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (28m) 13m ago 42m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (13m) 13m ago 42m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (28m) 13m ago 42m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (38m) 13m ago 50m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (37m) 13m ago 48m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (34m) 13m ago 50m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (33m) 13m ago 47m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (36m) 13m ago 49m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (36m) 13m ago 48m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (32m) 13m ago 47m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (32m) 13m ago 46m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (31m) 13m ago 45m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:26:43.394 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (30m) 13m ago 45m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:26:43.395 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (30m) 13m ago 44m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:26:43.395 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (29m) 13m ago 44m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:26:43.395 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (36m) 13m ago 48m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f318406c7d0 msgr2=0x7f318406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f318406c7d0 0x7f318406ec80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f319000b840 tx=0x7f3190009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a01037a0 msgr2=0x7f31a0116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a01037a0 0x7f31a0116210 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f318c00c890 tx=0x7f318c0044c0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 shutdown_connections 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f318406c7d0 0x7f318406ec80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a0102e00 0x7f31a0115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 --2- 172.21.15.63:0/3071898669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a01037a0 0x7f31a0116210 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.418+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 >> 172.21.15.63:0/3071898669 conn(0x7f31a00f89b0 msgr2=0x7f31a00f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.419+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 shutdown_connections 2024-03-23T12:26:43.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.419+0000 7f31a6c00700 1 -- 172.21.15.63:0/3071898669 wait complete. 2024-03-23T12:26:43.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.554+0000 7f931e800700 1 -- 172.21.15.63:0/2934457995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 msgr2=0x7f93180ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.554+0000 7f931e800700 1 --2- 172.21.15.63:0/2934457995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93180ff010 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f9304004940 tx=0x7f9304009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:43.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.554+0000 7f931e800700 1 -- 172.21.15.63:0/2934457995 shutdown_connections 2024-03-23T12:26:43.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.554+0000 7f931e800700 1 --2- 172.21.15.63:0/2934457995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93180ff010 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.554+0000 7f931e800700 1 --2- 172.21.15.63:0/2934457995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9318105000 0x7f93181053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.554+0000 7f931e800700 1 -- 172.21.15.63:0/2934457995 >> 172.21.15.63:0/2934457995 conn(0x7f93180fabd0 msgr2=0x7f93180fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:43.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.554+0000 7f931e800700 1 -- 172.21.15.63:0/2934457995 shutdown_connections 2024-03-23T12:26:43.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.555+0000 7f931e800700 1 -- 172.21.15.63:0/2934457995 wait complete. 2024-03-23T12:26:43.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.555+0000 7f931e800700 1 Processor -- start 2024-03-23T12:26:43.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.555+0000 7f931e800700 1 -- start start 2024-03-23T12:26:43.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f931e800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9318105000 0x7f9318115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f931e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93181161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f931e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9318116820 con 0x7f93181059a0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f931e800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931810fbf0 con 0x7f9318105000 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f9316a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93181161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f9316a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93181161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53924/0 (socket says 172.21.15.63:53924) 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f9316a00700 1 -- 172.21.15.63:0/3773869479 learned_addr learned my addr 172.21.15.63:0/3773869479 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.556+0000 7f9317400700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9318105000 0x7f9318115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f9316a00700 1 -- 172.21.15.63:0/3773869479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9318105000 msgr2=0x7f9318115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f9316a00700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9318105000 0x7f9318115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f9316a00700 1 -- 172.21.15.63:0/3773869479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9304005040 con 0x7f93181059a0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f9316a00700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93181161d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9304000c00 tx=0x7f9304009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f930f400700 1 -- 172.21.15.63:0/3773869479 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9304021070 con 0x7f93181059a0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f930f400700 1 -- 172.21.15.63:0/3773869479 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9304013de0 con 0x7f93181059a0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931810fe70 con 0x7f93181059a0 2024-03-23T12:26:43.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f930f400700 1 -- 172.21.15.63:0/3773869479 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f930400e070 con 0x7f93181059a0 2024-03-23T12:26:43.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.557+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93181103d0 con 0x7f93181059a0 2024-03-23T12:26:43.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.558+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9318102310 con 0x7f93181059a0 2024-03-23T12:26:43.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.559+0000 7f930f400700 1 -- 172.21.15.63:0/3773869479 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f930401c600 con 0x7f93181059a0 2024-03-23T12:26:43.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.560+0000 7f930f400700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f92fc06c7c0 0x7f92fc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.560+0000 7f930f400700 1 -- 172.21.15.63:0/3773869479 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f930408cc30 con 0x7f93181059a0 2024-03-23T12:26:43.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.560+0000 7f9317400700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f92fc06c7c0 0x7f92fc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.561+0000 7f9317400700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f92fc06c7c0 0x7f92fc06ec70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9308004600 tx=0x7f930800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:43.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.565+0000 7f930f400700 1 -- 172.21.15.63:0/3773869479 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f930405a8c0 con 0x7f93181059a0 2024-03-23T12:26:43.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.867+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f93181112e0 con 0x7f93181059a0 2024-03-23T12:26:43.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.868+0000 7f930f400700 1 -- 172.21.15.63:0/3773869479 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f930405a450 con 0x7f93181059a0 2024-03-23T12:26:43.845 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:26:43.845 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:26:43.845 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:26:43.846 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:26:43.847 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:26:43.847 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f92fc06c7c0 msgr2=0x7f92fc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f92fc06c7c0 0x7f92fc06ec70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9308004600 tx=0x7f930800b040 comp rx=0 tx=0).stop 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 msgr2=0x7f93181161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93181161d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9304000c00 tx=0x7f9304009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 shutdown_connections 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f92fc06c7c0 0x7f92fc06ec70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9318105000 0x7f9318115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 --2- 172.21.15.63:0/3773869479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93181059a0 0x7f93181161d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.870+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 >> 172.21.15.63:0/3773869479 conn(0x7f93180fabd0 msgr2=0x7f93181095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.871+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 shutdown_connections 2024-03-23T12:26:43.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:43.871+0000 7f931e800700 1 -- 172.21.15.63:0/3773869479 wait complete. 2024-03-23T12:26:43.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 -- 172.21.15.63:0/3606468988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c102e00 msgr2=0x7f4f1c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 --2- 172.21.15.63:0/3606468988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c102e00 0x7f4f1c1031d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c004940 tx=0x7f4f0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:43.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 -- 172.21.15.63:0/3606468988 shutdown_connections 2024-03-23T12:26:43.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 --2- 172.21.15.63:0/3606468988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f1c1037a0 0x7f4f1c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 --2- 172.21.15.63:0/3606468988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c102e00 0x7f4f1c1031d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 -- 172.21.15.63:0/3606468988 >> 172.21.15.63:0/3606468988 conn(0x7f4f1c0f89b0 msgr2=0x7f4f1c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:43.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 -- 172.21.15.63:0/3606468988 shutdown_connections 2024-03-23T12:26:43.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.009+0000 7f4f22800700 1 -- 172.21.15.63:0/3606468988 wait complete. 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.010+0000 7f4f22800700 1 Processor -- start 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.010+0000 7f4f22800700 1 -- start start 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.010+0000 7f4f22800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f1c102e00 0x7f4f1c113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.010+0000 7f4f22800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c1037a0 0x7f4f1c114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.010+0000 7f4f22800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1146b0 con 0x7f4f1c1037a0 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.010+0000 7f4f22800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c10daa0 con 0x7f4f1c102e00 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c1037a0 0x7f4f1c114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c1037a0 0x7f4f1c114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53942/0 (socket says 172.21.15.63:53942) 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13e00700 1 -- 172.21.15.63:0/1378070613 learned_addr learned my addr 172.21.15.63:0/1378070613 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f1b400700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f1c102e00 0x7f4f1c113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13e00700 1 -- 172.21.15.63:0/1378070613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f1c102e00 msgr2=0x7f4f1c113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13e00700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f1c102e00 0x7f4f1c113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13e00700 1 -- 172.21.15.63:0/1378070613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c005040 con 0x7f4f1c1037a0 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13e00700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c1037a0 0x7f4f1c114060 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f4f04002a50 tx=0x7f4f04004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.011+0000 7f4f13400700 1 -- 172.21.15.63:0/1378070613 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f04007cc0 con 0x7f4f1c1037a0 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.012+0000 7f4f13400700 1 -- 172.21.15.63:0/1378070613 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f0400f040 con 0x7f4f1c1037a0 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.012+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c10dd80 con 0x7f4f1c1037a0 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.012+0000 7f4f13400700 1 -- 172.21.15.63:0/1378070613 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0400d920 con 0x7f4f1c1037a0 2024-03-23T12:26:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.012+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c10e340 con 0x7f4f1c1037a0 2024-03-23T12:26:43.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.013+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c0fe550 con 0x7f4f1c1037a0 2024-03-23T12:26:43.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.014+0000 7f4f13400700 1 -- 172.21.15.63:0/1378070613 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4f04009680 con 0x7f4f1c1037a0 2024-03-23T12:26:43.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.015+0000 7f4f13400700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f0006c7d0 0x7f4f0006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:43.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.015+0000 7f4f13400700 1 -- 172.21.15.63:0/1378070613 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f0408b6d0 con 0x7f4f1c1037a0 2024-03-23T12:26:43.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.015+0000 7f4f1b400700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f0006c7d0 0x7f4f0006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:43.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.016+0000 7f4f1b400700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f0006c7d0 0x7f4f0006ec80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00b840 tx=0x7f4f0c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:43.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.019+0000 7f4f13400700 1 -- 172.21.15.63:0/1378070613 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f040593e0 con 0x7f4f1c1037a0 2024-03-23T12:26:44.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:43 smithi111 ceph-mon[106899]: from='client.35922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:44.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:43 smithi111 ceph-mon[106899]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:44.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:43 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3773869479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:26:44.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.276+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4f1c109dc0 con 0x7f4f1c1037a0 2024-03-23T12:26:44.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.277+0000 7f4f13400700 1 -- 172.21.15.63:0/1378070613 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f4f04058f70 con 0x7f4f1c1037a0 2024-03-23T12:26:44.254 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:26:44.254 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:26:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:26:44.256 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:44.257 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:26:44.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f0006c7d0 msgr2=0x7f4f0006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f0006c7d0 0x7f4f0006ec80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00b840 tx=0x7f4f0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:44.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c1037a0 msgr2=0x7f4f1c114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c1037a0 0x7f4f1c114060 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f4f04002a50 tx=0x7f4f04004600 comp rx=0 tx=0).stop 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 shutdown_connections 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4f0006c7d0 0x7f4f0006ec80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f1c102e00 0x7f4f1c113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 --2- 172.21.15.63:0/1378070613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f1c1037a0 0x7f4f1c114060 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 >> 172.21.15.63:0/1378070613 conn(0x7f4f1c0f89b0 msgr2=0x7f4f1c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.279+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 shutdown_connections 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.280+0000 7f4f22800700 1 -- 172.21.15.63:0/1378070613 wait complete. 2024-03-23T12:26:44.259 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:26:44.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:43 smithi063 ceph-mon[141124]: from='client.35922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:44.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:43 smithi063 ceph-mon[141124]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:44.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:43 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3773869479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.417+0000 7f3932000700 1 -- 172.21.15.63:0/424267997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c102e00 msgr2=0x7f392c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.417+0000 7f3932000700 1 --2- 172.21.15.63:0/424267997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c102e00 0x7f392c1031d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f391c004940 tx=0x7f391c009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.417+0000 7f3932000700 1 -- 172.21.15.63:0/424267997 shutdown_connections 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.417+0000 7f3932000700 1 --2- 172.21.15.63:0/424267997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f392c1037a0 0x7f392c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.417+0000 7f3932000700 1 --2- 172.21.15.63:0/424267997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c102e00 0x7f392c1031d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.417+0000 7f3932000700 1 -- 172.21.15.63:0/424267997 >> 172.21.15.63:0/424267997 conn(0x7f392c0f89b0 msgr2=0x7f392c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.418+0000 7f3932000700 1 -- 172.21.15.63:0/424267997 shutdown_connections 2024-03-23T12:26:44.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.418+0000 7f3932000700 1 -- 172.21.15.63:0/424267997 wait complete. 2024-03-23T12:26:44.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.418+0000 7f3932000700 1 Processor -- start 2024-03-23T12:26:44.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f3932000700 1 -- start start 2024-03-23T12:26:44.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f3932000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f392c102e00 0x7f392c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:44.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f3932000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c1037a0 0x7f392c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:44.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f3932000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0720b0 con 0x7f392c1037a0 2024-03-23T12:26:44.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f3932000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c072220 con 0x7f392c102e00 2024-03-23T12:26:44.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f392aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c1037a0 0x7f392c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:44.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f392b400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f392c102e00 0x7f392c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:44.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f392aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c1037a0 0x7f392c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53966/0 (socket says 172.21.15.63:53966) 2024-03-23T12:26:44.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.419+0000 7f392aa00700 1 -- 172.21.15.63:0/3916247585 learned_addr learned my addr 172.21.15.63:0/3916247585 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f392aa00700 1 -- 172.21.15.63:0/3916247585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f392c102e00 msgr2=0x7f392c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f392aa00700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f392c102e00 0x7f392c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f392aa00700 1 -- 172.21.15.63:0/3916247585 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c005040 con 0x7f392c1037a0 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f392aa00700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c1037a0 0x7f392c070800 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f391800eb40 tx=0x7f391800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f3923400700 1 -- 172.21.15.63:0/3916247585 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3918007cc0 con 0x7f392c1037a0 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f3923400700 1 -- 172.21.15.63:0/3916247585 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f391800f040 con 0x7f392c1037a0 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c070e00 con 0x7f392c1037a0 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f3923400700 1 -- 172.21.15.63:0/3916247585 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f391800c8f0 con 0x7f392c1037a0 2024-03-23T12:26:44.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.420+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c11e6d0 con 0x7f392c1037a0 2024-03-23T12:26:44.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.421+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c0fe550 con 0x7f392c1037a0 2024-03-23T12:26:44.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.422+0000 7f3923400700 1 -- 172.21.15.63:0/3916247585 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f391800cb60 con 0x7f392c1037a0 2024-03-23T12:26:44.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.423+0000 7f3923400700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f391006c7d0 0x7f391006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:44.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.423+0000 7f3923400700 1 -- 172.21.15.63:0/3916247585 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f391808b510 con 0x7f392c1037a0 2024-03-23T12:26:44.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.424+0000 7f392b400700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f391006c7d0 0x7f391006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:44.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.424+0000 7f392b400700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f391006c7d0 0x7f391006ec80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f391c00b0d0 tx=0x7f391c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:44.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.428+0000 7f3923400700 1 -- 172.21.15.63:0/3916247585 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3918059250 con 0x7f392c1037a0 2024-03-23T12:26:44.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.660+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f392c109e60 con 0x7f391006c7d0 2024-03-23T12:26:44.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.662+0000 7f3923400700 1 -- 172.21.15.63:0/3916247585 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f391c0096f0 con 0x7f391006c7d0 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:26:44.639 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:26:44.640 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:26:44.640 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:26:44.640 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:26:44.640 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:26:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f391006c7d0 msgr2=0x7f391006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f391006c7d0 0x7f391006ec80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f391c00b0d0 tx=0x7f391c009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c1037a0 msgr2=0x7f392c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c1037a0 0x7f392c070800 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f391800eb40 tx=0x7f391800b040 comp rx=0 tx=0).stop 2024-03-23T12:26:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 shutdown_connections 2024-03-23T12:26:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f391006c7d0 0x7f391006ec80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f392c102e00 0x7f392c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 --2- 172.21.15.63:0/3916247585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392c1037a0 0x7f392c070800 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 >> 172.21.15.63:0/3916247585 conn(0x7f392c0f89b0 msgr2=0x7f392c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:44.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 shutdown_connections 2024-03-23T12:26:44.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.664+0000 7f3932000700 1 -- 172.21.15.63:0/3916247585 wait complete. 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 -- 172.21.15.63:0/371104674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 msgr2=0x7fc8601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 --2- 172.21.15.63:0/371104674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc8601031d0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fc850004940 tx=0x7fc850009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 -- 172.21.15.63:0/371104674 shutdown_connections 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 --2- 172.21.15.63:0/371104674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8601037a0 0x7fc8600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 --2- 172.21.15.63:0/371104674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc8601031d0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 -- 172.21.15.63:0/371104674 >> 172.21.15.63:0/371104674 conn(0x7fc8600f89b0 msgr2=0x7fc8600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 -- 172.21.15.63:0/371104674 shutdown_connections 2024-03-23T12:26:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.804+0000 7fc865a00700 1 -- 172.21.15.63:0/371104674 wait complete. 2024-03-23T12:26:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.805+0000 7fc865a00700 1 Processor -- start 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.805+0000 7fc865a00700 1 -- start start 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc865a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc860115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc865a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8601037a0 0x7fc860116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc865a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc860116880 con 0x7fc860102e00 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc865a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86010fc50 con 0x7fc8601037a0 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc85ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc860115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc85ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc860115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53972/0 (socket says 172.21.15.63:53972) 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc85ea00700 1 -- 172.21.15.63:0/4063445344 learned_addr learned my addr 172.21.15.63:0/4063445344 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc85e000700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8601037a0 0x7fc860116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc85ea00700 1 -- 172.21.15.63:0/4063445344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8601037a0 msgr2=0x7fc860116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc85ea00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8601037a0 0x7fc860116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.806+0000 7fc85ea00700 1 -- 172.21.15.63:0/4063445344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc850005040 con 0x7fc860102e00 2024-03-23T12:26:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.807+0000 7fc85ea00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc860115cf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc850009390 tx=0x7fc850009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.807+0000 7fc856a00700 1 -- 172.21.15.63:0/4063445344 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc85000e040 con 0x7fc860102e00 2024-03-23T12:26:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.807+0000 7fc856a00700 1 -- 172.21.15.63:0/4063445344 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc850013420 con 0x7fc860102e00 2024-03-23T12:26:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.807+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86010fed0 con 0x7fc860102e00 2024-03-23T12:26:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.807+0000 7fc856a00700 1 -- 172.21.15.63:0/4063445344 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc85001c630 con 0x7fc860102e00 2024-03-23T12:26:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.807+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc860110490 con 0x7fc860102e00 2024-03-23T12:26:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.808+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8600fe550 con 0x7fc860102e00 2024-03-23T12:26:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.809+0000 7fc856a00700 1 -- 172.21.15.63:0/4063445344 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc8500139e0 con 0x7fc860102e00 2024-03-23T12:26:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.810+0000 7fc856a00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84406c7d0 0x7fc84406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:26:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.810+0000 7fc856a00700 1 -- 172.21.15.63:0/4063445344 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc85008cd60 con 0x7fc860102e00 2024-03-23T12:26:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.810+0000 7fc85e000700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84406c7d0 0x7fc84406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:26:44.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.811+0000 7fc85e000700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84406c7d0 0x7fc84406ec80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc860111590 tx=0x7fc84c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:26:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:44.814+0000 7fc856a00700 1 -- 172.21.15.63:0/4063445344 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc850091070 con 0x7fc860102e00 2024-03-23T12:26:45.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.115+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc86004bed0 con 0x7fc860102e00 2024-03-23T12:26:45.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.115+0000 7fc856a00700 1 -- 172.21.15.63:0/4063445344 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc85005aaa0 con 0x7fc860102e00 2024-03-23T12:26:45.093 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:26:45.093 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:26:45.093 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:26:45.093 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:26:45.093 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:26:45.093 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:26:45.093 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:26:45.094 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:26:45.095 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:26:45.095 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:26:45.095 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:26:45.095 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:26:45.095 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:26:45.095 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84406c7d0 msgr2=0x7fc84406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84406c7d0 0x7fc84406ec80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc860111590 tx=0x7fc84c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 msgr2=0x7fc860115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc860115cf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc850009390 tx=0x7fc850009e30 comp rx=0 tx=0).stop 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 shutdown_connections 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc84406c7d0 0x7fc84406ec80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc860102e00 0x7fc860115cf0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 --2- 172.21.15.63:0/4063445344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8601037a0 0x7fc860116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.117+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 >> 172.21.15.63:0/4063445344 conn(0x7fc8600f89b0 msgr2=0x7fc8600706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.118+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 shutdown_connections 2024-03-23T12:26:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:26:45.118+0000 7fc865a00700 1 -- 172.21.15.63:0/4063445344 wait complete. 2024-03-23T12:26:45.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:44 smithi111 ceph-mon[106899]: pgmap v1069: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:45.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:44 smithi111 ceph-mon[106899]: from='client.35930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:45.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:44 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1378070613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:26:45.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:44 smithi063 ceph-mon[141124]: pgmap v1069: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:45.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:44 smithi063 ceph-mon[141124]: from='client.35930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:45.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:44 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1378070613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:26:46.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:45 smithi063 ceph-mon[141124]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:46.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:45 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4063445344' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:26:46.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:45 smithi111 ceph-mon[106899]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:26:46.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:45 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4063445344' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:26:47.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:46 smithi063 ceph-mon[141124]: pgmap v1070: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:47.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:46 smithi111 ceph-mon[106899]: pgmap v1070: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:49.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:48 smithi063 ceph-mon[141124]: pgmap v1071: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:49.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:48 smithi111 ceph-mon[106899]: pgmap v1071: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:51.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:50 smithi063 ceph-mon[141124]: pgmap v1072: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:51.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:50 smithi111 ceph-mon[106899]: pgmap v1072: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:52 smithi063 ceph-mon[141124]: pgmap v1073: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:53.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:52 smithi111 ceph-mon[106899]: pgmap v1073: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:55.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:54 smithi063 ceph-mon[141124]: pgmap v1074: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:54 smithi111 ceph-mon[106899]: pgmap v1074: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:56.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:56.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:26:57.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:56 smithi063 ceph-mon[141124]: pgmap v1075: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:57.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:56 smithi111 ceph-mon[106899]: pgmap v1075: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:59.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:26:58 smithi063 ceph-mon[141124]: pgmap v1076: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:26:59.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:26:58 smithi111 ceph-mon[106899]: pgmap v1076: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:01.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:00 smithi063 ceph-mon[141124]: pgmap v1077: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:01.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:00 smithi111 ceph-mon[106899]: pgmap v1077: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:03.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:02 smithi063 ceph-mon[141124]: pgmap v1078: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:03.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:02 smithi111 ceph-mon[106899]: pgmap v1078: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:05.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:04 smithi063 ceph-mon[141124]: pgmap v1079: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:05.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:04 smithi111 ceph-mon[106899]: pgmap v1079: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:07.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:06 smithi063 ceph-mon[141124]: pgmap v1080: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:07.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:06 smithi111 ceph-mon[106899]: pgmap v1080: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:09.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:08 smithi063 ceph-mon[141124]: pgmap v1081: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:09.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:08 smithi111 ceph-mon[106899]: pgmap v1081: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:11.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:10 smithi063 ceph-mon[141124]: pgmap v1082: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:11.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:11.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:10 smithi111 ceph-mon[106899]: pgmap v1082: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:11.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:13.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:12 smithi063 ceph-mon[141124]: pgmap v1083: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:13.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:12 smithi111 ceph-mon[106899]: pgmap v1083: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:15.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 -- 172.21.15.63:0/3671956200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8981037c0 msgr2=0x7fc8980fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 --2- 172.21.15.63:0/3671956200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8981037c0 0x7fc8980fce30 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc880004940 tx=0x7fc880009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:15.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 -- 172.21.15.63:0/3671956200 shutdown_connections 2024-03-23T12:27:15.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 --2- 172.21.15.63:0/3671956200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8981037c0 0x7fc8980fce30 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 --2- 172.21.15.63:0/3671956200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc898102e20 0x7fc8981031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 -- 172.21.15.63:0/3671956200 >> 172.21.15.63:0/3671956200 conn(0x7fc8980f89d0 msgr2=0x7fc8980fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:15.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 -- 172.21.15.63:0/3671956200 shutdown_connections 2024-03-23T12:27:15.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.273+0000 7fc89ec00700 1 -- 172.21.15.63:0/3671956200 wait complete. 2024-03-23T12:27:15.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.274+0000 7fc89ec00700 1 Processor -- start 2024-03-23T12:27:15.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.274+0000 7fc89ec00700 1 -- start start 2024-03-23T12:27:15.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.274+0000 7fc89ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc898102e20 0x7fc8980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.274+0000 7fc89ec00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8981037c0 0x7fc898070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc89ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc898070d90 con 0x7fc898102e20 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc89ec00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc898070ed0 con 0x7fc8981037c0 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc897e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc898102e20 0x7fc8980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc897e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc898102e20 0x7fc8980702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51082/0 (socket says 172.21.15.63:51082) 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc897e00700 1 -- 172.21.15.63:0/2787031373 learned_addr learned my addr 172.21.15.63:0/2787031373 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc88fe00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8981037c0 0x7fc898070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc897e00700 1 -- 172.21.15.63:0/2787031373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8981037c0 msgr2=0x7fc898070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc897e00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8981037c0 0x7fc898070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.275+0000 7fc897e00700 1 -- 172.21.15.63:0/2787031373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc880005040 con 0x7fc898102e20 2024-03-23T12:27:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.276+0000 7fc897e00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc898102e20 0x7fc8980702c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc88800eb70 tx=0x7fc88800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:15.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.276+0000 7fc894c00700 1 -- 172.21.15.63:0/2787031373 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc888007cc0 con 0x7fc898102e20 2024-03-23T12:27:15.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.276+0000 7fc894c00700 1 -- 172.21.15.63:0/2787031373 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc88800f040 con 0x7fc898102e20 2024-03-23T12:27:15.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.276+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc898071120 con 0x7fc898102e20 2024-03-23T12:27:15.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.276+0000 7fc894c00700 1 -- 172.21.15.63:0/2787031373 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc88800c9f0 con 0x7fc898102e20 2024-03-23T12:27:15.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.276+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89811a1c0 con 0x7fc898102e20 2024-03-23T12:27:15.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.277+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8980fe570 con 0x7fc898102e20 2024-03-23T12:27:15.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.278+0000 7fc894c00700 1 -- 172.21.15.63:0/2787031373 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc88800cb50 con 0x7fc898102e20 2024-03-23T12:27:15.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.279+0000 7fc894c00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc87c06c7d0 0x7fc87c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:15.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.279+0000 7fc894c00700 1 -- 172.21.15.63:0/2787031373 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc88808b6e0 con 0x7fc898102e20 2024-03-23T12:27:15.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.279+0000 7fc88fe00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc87c06c7d0 0x7fc87c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:15.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.280+0000 7fc88fe00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc87c06c7d0 0x7fc87c06ec80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc898071c00 tx=0x7fc880009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:15.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.284+0000 7fc894c00700 1 -- 172.21.15.63:0/2787031373 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc888055860 con 0x7fc898102e20 2024-03-23T12:27:15.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:14 smithi063 ceph-mon[141124]: pgmap v1084: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:15.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:14 smithi111 ceph-mon[106899]: pgmap v1084: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:15.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.513+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8980719b0 con 0x7fc87c06c7d0 2024-03-23T12:27:15.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.514+0000 7fc894c00700 1 -- 172.21.15.63:0/2787031373 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc8980719b0 con 0x7fc87c06c7d0 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc87c06c7d0 msgr2=0x7fc87c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc87c06c7d0 0x7fc87c06ec80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc898071c00 tx=0x7fc880009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc898102e20 msgr2=0x7fc8980702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc898102e20 0x7fc8980702c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc88800eb70 tx=0x7fc88800b040 comp rx=0 tx=0).stop 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 shutdown_connections 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc87c06c7d0 0x7fc87c06ec80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc898102e20 0x7fc8980702c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 --2- 172.21.15.63:0/2787031373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8981037c0 0x7fc898070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 >> 172.21.15.63:0/2787031373 conn(0x7fc8980f89d0 msgr2=0x7fc8980f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:15.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 shutdown_connections 2024-03-23T12:27:15.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.517+0000 7fc89ec00700 1 -- 172.21.15.63:0/2787031373 wait complete. 2024-03-23T12:27:15.509 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.657+0000 7fdded400700 1 -- 172.21.15.63:0/2164218716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 msgr2=0x7fdde8070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.657+0000 7fdded400700 1 --2- 172.21.15.63:0/2164218716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde8070d10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fddd4004970 tx=0x7fddd4009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.657+0000 7fdded400700 1 -- 172.21.15.63:0/2164218716 shutdown_connections 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.657+0000 7fdded400700 1 --2- 172.21.15.63:0/2164218716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde8070d10 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.657+0000 7fdded400700 1 --2- 172.21.15.63:0/2164218716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde8071f90 0x7fdde8070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.657+0000 7fdded400700 1 -- 172.21.15.63:0/2164218716 >> 172.21.15.63:0/2164218716 conn(0x7fdde80f89b0 msgr2=0x7fdde80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.658+0000 7fdded400700 1 -- 172.21.15.63:0/2164218716 shutdown_connections 2024-03-23T12:27:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.658+0000 7fdded400700 1 -- 172.21.15.63:0/2164218716 wait complete. 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.658+0000 7fdded400700 1 Processor -- start 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.658+0000 7fdded400700 1 -- start start 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdded400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde811a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdded400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde8071f90 0x7fdde811a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdded400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde811ac40 con 0x7fdde80708a0 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdded400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8114010 con 0x7fdde8071f90 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdde6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde811a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdde6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde811a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51094/0 (socket says 172.21.15.63:51094) 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdde6800700 1 -- 172.21.15.63:0/3634595614 learned_addr learned my addr 172.21.15.63:0/3634595614 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.659+0000 7fdde5e00700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde8071f90 0x7fdde811a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdde6800700 1 -- 172.21.15.63:0/3634595614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde8071f90 msgr2=0x7fdde811a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdde6800700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde8071f90 0x7fdde811a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdde6800700 1 -- 172.21.15.63:0/3634595614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4005040 con 0x7fdde80708a0 2024-03-23T12:27:15.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdde6800700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde811a0b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fddd8002a50 tx=0x7fddd80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:15.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdddea00700 1 -- 172.21.15.63:0/3634595614 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddd8007cc0 con 0x7fdde80708a0 2024-03-23T12:27:15.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdddea00700 1 -- 172.21.15.63:0/3634595614 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddd800f040 con 0x7fdde80708a0 2024-03-23T12:27:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde81142f0 con 0x7fdde80708a0 2024-03-23T12:27:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdddea00700 1 -- 172.21.15.63:0/3634595614 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddd800d9c0 con 0x7fdde80708a0 2024-03-23T12:27:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.660+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde8114800 con 0x7fdde80708a0 2024-03-23T12:27:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.661+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde8071aa0 con 0x7fdde80708a0 2024-03-23T12:27:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.663+0000 7fdddea00700 1 -- 172.21.15.63:0/3634595614 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fddd800db20 con 0x7fdde80708a0 2024-03-23T12:27:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.664+0000 7fdddea00700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fddcc06c7d0 0x7fddcc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.664+0000 7fdddea00700 1 -- 172.21.15.63:0/3634595614 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fddd808b820 con 0x7fdde80708a0 2024-03-23T12:27:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.664+0000 7fdde5e00700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fddcc06c7d0 0x7fddcc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.665+0000 7fdde5e00700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fddcc06c7d0 0x7fddcc06ec80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009fd0 tx=0x7fddd4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.669+0000 7fdddea00700 1 -- 172.21.15.63:0/3634595614 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fddd8059430 con 0x7fdde80708a0 2024-03-23T12:27:15.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.907+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdde804bed0 con 0x7fddcc06c7d0 2024-03-23T12:27:15.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.908+0000 7fdddea00700 1 -- 172.21.15.63:0/3634595614 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdde804bed0 con 0x7fddcc06c7d0 2024-03-23T12:27:15.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fddcc06c7d0 msgr2=0x7fddcc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fddcc06c7d0 0x7fddcc06ec80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009fd0 tx=0x7fddd4009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 msgr2=0x7fdde811a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde811a0b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fddd8002a50 tx=0x7fddd80046b0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 shutdown_connections 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fddcc06c7d0 0x7fddcc06ec80 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde80708a0 0x7fdde811a0b0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.910+0000 7fdded400700 1 --2- 172.21.15.63:0/3634595614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde8071f90 0x7fdde811a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.911+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 >> 172.21.15.63:0/3634595614 conn(0x7fdde80f89b0 msgr2=0x7fdde80f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.911+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 shutdown_connections 2024-03-23T12:27:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:15.911+0000 7fdded400700 1 -- 172.21.15.63:0/3634595614 wait complete. 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 -- 172.21.15.63:0/1322396206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 msgr2=0x7f10881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 --2- 172.21.15.63:0/1322396206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f10881031d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f1078004940 tx=0x7f1078009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 -- 172.21.15.63:0/1322396206 shutdown_connections 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 --2- 172.21.15.63:0/1322396206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10881037a0 0x7f10880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 --2- 172.21.15.63:0/1322396206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f10881031d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 -- 172.21.15.63:0/1322396206 >> 172.21.15.63:0/1322396206 conn(0x7f10880f89b0 msgr2=0x7f10880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 -- 172.21.15.63:0/1322396206 shutdown_connections 2024-03-23T12:27:16.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.048+0000 7f108e000700 1 -- 172.21.15.63:0/1322396206 wait complete. 2024-03-23T12:27:16.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.049+0000 7f108e000700 1 Processor -- start 2024-03-23T12:27:16.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.049+0000 7f108e000700 1 -- start start 2024-03-23T12:27:16.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f108e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f1088115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f108e000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10881037a0 0x7f1088116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f108e000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10881168a0 con 0x7f1088102e00 2024-03-23T12:27:16.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f108e000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108810fc70 con 0x7f10881037a0 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1087400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f1088115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1087400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f1088115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51110/0 (socket says 172.21.15.63:51110) 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1087400700 1 -- 172.21.15.63:0/1702758875 learned_addr learned my addr 172.21.15.63:0/1702758875 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1086a00700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10881037a0 0x7f1088116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1087400700 1 -- 172.21.15.63:0/1702758875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10881037a0 msgr2=0x7f1088116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1087400700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10881037a0 0x7f1088116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1087400700 1 -- 172.21.15.63:0/1702758875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078005040 con 0x7f1088102e00 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.050+0000 7f1086a00700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10881037a0 0x7f1088116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.051+0000 7f1087400700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f1088115d10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1078009390 tx=0x7f1078009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.051+0000 7f107f400700 1 -- 172.21.15.63:0/1702758875 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f107800e040 con 0x7f1088102e00 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.051+0000 7f107f400700 1 -- 172.21.15.63:0/1702758875 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1078013420 con 0x7f1088102e00 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.051+0000 7f107f400700 1 -- 172.21.15.63:0/1702758875 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f107801c630 con 0x7f1088102e00 2024-03-23T12:27:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.051+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108810fef0 con 0x7f1088102e00 2024-03-23T12:27:16.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.051+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1088110460 con 0x7f1088102e00 2024-03-23T12:27:16.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.052+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10880fe550 con 0x7f1088102e00 2024-03-23T12:27:16.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.053+0000 7f107f400700 1 -- 172.21.15.63:0/1702758875 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f10780139e0 con 0x7f1088102e00 2024-03-23T12:27:16.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.054+0000 7f107f400700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f106c06c7d0 0x7f106c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.054+0000 7f107f400700 1 -- 172.21.15.63:0/1702758875 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f107808cd60 con 0x7f1088102e00 2024-03-23T12:27:16.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.054+0000 7f1086a00700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f106c06c7d0 0x7f106c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.055+0000 7f1086a00700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f106c06c7d0 0x7f106c06ec80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1074009fd0 tx=0x7f1074009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:16.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.059+0000 7f107f400700 1 -- 172.21.15.63:0/1702758875 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1078091070 con 0x7f1088102e00 2024-03-23T12:27:16.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.283+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f108804bf40 con 0x7f106c06c7d0 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.293+0000 7f107f400700 1 -- 172.21.15.63:0/1702758875 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f108804bf40 con 0x7f106c06c7d0 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (36m) 13m ago 49m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (33m) 13m ago 49m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (33m) 13m ago 48m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (36m) 13m ago 49m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (29m) 13m ago 43m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:27:16.271 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (29m) 13m ago 43m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (13m) 13m ago 43m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (28m) 13m ago 43m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (38m) 13m ago 51m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (37m) 13m ago 48m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (35m) 13m ago 51m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (34m) 13m ago 48m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (37m) 13m ago 49m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (37m) 13m ago 48m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (33m) 13m ago 47m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:27:16.272 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (32m) 13m ago 47m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:27:16.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (31m) 13m ago 46m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:27:16.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (31m) 13m ago 46m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:27:16.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (30m) 13m ago 45m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:27:16.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (30m) 13m ago 45m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:27:16.273 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (36m) 13m ago 49m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:27:16.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f106c06c7d0 msgr2=0x7f106c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f106c06c7d0 0x7f106c06ec80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1074009fd0 tx=0x7f1074009ee0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 msgr2=0x7f1088115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f1088115d10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1078009390 tx=0x7f1078009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:16.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 shutdown_connections 2024-03-23T12:27:16.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f106c06c7d0 0x7f106c06ec80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1088102e00 0x7f1088115d10 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 --2- 172.21.15.63:0/1702758875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10881037a0 0x7f1088116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.295+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 >> 172.21.15.63:0/1702758875 conn(0x7f10880f89b0 msgr2=0x7f10880704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:16.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.296+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 shutdown_connections 2024-03-23T12:27:16.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.296+0000 7f108e000700 1 -- 172.21.15.63:0/1702758875 wait complete. 2024-03-23T12:27:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 -- 172.21.15.63:0/1005647508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fe850 msgr2=0x7f5e64106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 --2- 172.21.15.63:0/1005647508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fe850 0x7f5e64106f20 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5e50004970 tx=0x7f5e50009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 -- 172.21.15.63:0/1005647508 shutdown_connections 2024-03-23T12:27:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 --2- 172.21.15.63:0/1005647508 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fe850 0x7f5e64106f20 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 --2- 172.21.15.63:0/1005647508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e640fdeb0 0x7f5e640fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 -- 172.21.15.63:0/1005647508 >> 172.21.15.63:0/1005647508 conn(0x7f5e64072bc0 msgr2=0x7f5e64072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:16.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 -- 172.21.15.63:0/1005647508 shutdown_connections 2024-03-23T12:27:16.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.431+0000 7f5e6c000700 1 -- 172.21.15.63:0/1005647508 wait complete. 2024-03-23T12:27:16.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e6c000700 1 Processor -- start 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e6c000700 1 -- start start 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e6c000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fdeb0 0x7f5e640fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e6c000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e640fe850 0x7f5e640fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e6c000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e640fc840 con 0x7f5e640fdeb0 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e6c000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e640fc980 con 0x7f5e640fe850 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e69600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fdeb0 0x7f5e640fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e69600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fdeb0 0x7f5e640fbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51130/0 (socket says 172.21.15.63:51130) 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.432+0000 7f5e69600700 1 -- 172.21.15.63:0/64760150 learned_addr learned my addr 172.21.15.63:0/64760150 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e68c00700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e640fe850 0x7f5e640fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e69600700 1 -- 172.21.15.63:0/64760150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e640fe850 msgr2=0x7f5e640fc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e69600700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e640fe850 0x7f5e640fc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e69600700 1 -- 172.21.15.63:0/64760150 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e50005040 con 0x7f5e640fdeb0 2024-03-23T12:27:16.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e69600700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fdeb0 0x7f5e640fbd70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5e58002a50 tx=0x7f5e580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e61600700 1 -- 172.21.15.63:0/64760150 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e58007cc0 con 0x7f5e640fdeb0 2024-03-23T12:27:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e61600700 1 -- 172.21.15.63:0/64760150 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e5800f040 con 0x7f5e640fdeb0 2024-03-23T12:27:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e61600700 1 -- 172.21.15.63:0/64760150 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e58009540 con 0x7f5e640fdeb0 2024-03-23T12:27:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.433+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e64119d10 con 0x7f5e640fdeb0 2024-03-23T12:27:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.434+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6411a110 con 0x7f5e640fdeb0 2024-03-23T12:27:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.435+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e64065e10 con 0x7f5e640fdeb0 2024-03-23T12:27:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.436+0000 7f5e61600700 1 -- 172.21.15.63:0/64760150 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5e580096a0 con 0x7f5e640fdeb0 2024-03-23T12:27:16.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.437+0000 7f5e61600700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4c06c7d0 0x7f5e4c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.437+0000 7f5e61600700 1 -- 172.21.15.63:0/64760150 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e5808b840 con 0x7f5e640fdeb0 2024-03-23T12:27:16.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.437+0000 7f5e68c00700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4c06c7d0 0x7f5e4c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.438+0000 7f5e68c00700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4c06c7d0 0x7f5e4c06ec80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5e640fd6b0 tx=0x7f5e50009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:16.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.442+0000 7f5e61600700 1 -- 172.21.15.63:0/64760150 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5e58055a40 con 0x7f5e640fdeb0 2024-03-23T12:27:16.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.745+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5e6405f5f0 con 0x7f5e640fdeb0 2024-03-23T12:27:16.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.745+0000 7f5e61600700 1 -- 172.21.15.63:0/64760150 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f5e58059060 con 0x7f5e640fdeb0 2024-03-23T12:27:16.723 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:27:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:27:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:27:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:27:16.724 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.747+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4c06c7d0 msgr2=0x7f5e4c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.747+0000 7f5e6c000700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4c06c7d0 0x7f5e4c06ec80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5e640fd6b0 tx=0x7f5e50009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.747+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fdeb0 msgr2=0x7f5e640fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.747+0000 7f5e6c000700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fdeb0 0x7f5e640fbd70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5e58002a50 tx=0x7f5e580046b0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.748+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 shutdown_connections 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.748+0000 7f5e6c000700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4c06c7d0 0x7f5e4c06ec80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.748+0000 7f5e6c000700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e640fdeb0 0x7f5e640fbd70 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.748+0000 7f5e6c000700 1 --2- 172.21.15.63:0/64760150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e640fe850 0x7f5e640fc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.748+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 >> 172.21.15.63:0/64760150 conn(0x7f5e64072bc0 msgr2=0x7f5e64103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:16.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.748+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 shutdown_connections 2024-03-23T12:27:16.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.748+0000 7f5e6c000700 1 -- 172.21.15.63:0/64760150 wait complete. 2024-03-23T12:27:16.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 -- 172.21.15.63:0/1853383743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 msgr2=0x7fe2200fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 --2- 172.21.15.63:0/1853383743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe2200fce10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe20c004940 tx=0x7fe20c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:16.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 -- 172.21.15.63:0/1853383743 shutdown_connections 2024-03-23T12:27:16.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 --2- 172.21.15.63:0/1853383743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe2200fce10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 --2- 172.21.15.63:0/1853383743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe220102e00 0x7fe2201031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 -- 172.21.15.63:0/1853383743 >> 172.21.15.63:0/1853383743 conn(0x7fe2200f89b0 msgr2=0x7fe2200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:16.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 -- 172.21.15.63:0/1853383743 shutdown_connections 2024-03-23T12:27:16.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.886+0000 7fe228000700 1 -- 172.21.15.63:0/1853383743 wait complete. 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.887+0000 7fe228000700 1 Processor -- start 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.887+0000 7fe228000700 1 -- start start 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.887+0000 7fe228000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe220102e00 0x7fe2200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.887+0000 7fe228000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe220070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.887+0000 7fe228000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220070d90 con 0x7fe2201037a0 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.887+0000 7fe228000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220070ed0 con 0x7fe220102e00 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.887+0000 7fe224c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe220070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe224c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe220070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51148/0 (socket says 172.21.15.63:51148) 2024-03-23T12:27:16.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe224c00700 1 -- 172.21.15.63:0/2128831480 learned_addr learned my addr 172.21.15.63:0/2128831480 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:16.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe225600700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe220102e00 0x7fe2200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe224c00700 1 -- 172.21.15.63:0/2128831480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe220102e00 msgr2=0x7fe2200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:16.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe224c00700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe220102e00 0x7fe2200702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:16.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe224c00700 1 -- 172.21.15.63:0/2128831480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c005040 con 0x7fe2201037a0 2024-03-23T12:27:16.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe224c00700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe220070800 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009fd0 tx=0x7fe20c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:16.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.888+0000 7fe21d600700 1 -- 172.21.15.63:0/2128831480 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20c00e030 con 0x7fe2201037a0 2024-03-23T12:27:16.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.889+0000 7fe21d600700 1 -- 172.21.15.63:0/2128831480 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe20c0093b0 con 0x7fe2201037a0 2024-03-23T12:27:16.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.889+0000 7fe21d600700 1 -- 172.21.15.63:0/2128831480 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20c01cc80 con 0x7fe2201037a0 2024-03-23T12:27:16.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.889+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe220071120 con 0x7fe2201037a0 2024-03-23T12:27:16.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.889+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22011a1a0 con 0x7fe2201037a0 2024-03-23T12:27:16.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.890+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2200fe550 con 0x7fe2201037a0 2024-03-23T12:27:16.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.891+0000 7fe21d600700 1 -- 172.21.15.63:0/2128831480 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe20c01c400 con 0x7fe2201037a0 2024-03-23T12:27:16.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.892+0000 7fe21d600700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe20806c7d0 0x7fe20806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:16.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.892+0000 7fe21d600700 1 -- 172.21.15.63:0/2128831480 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe20c08d1e0 con 0x7fe2201037a0 2024-03-23T12:27:16.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.892+0000 7fe225600700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe20806c7d0 0x7fe20806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:16.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.893+0000 7fe225600700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe20806c7d0 0x7fe20806ec80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe214009b80 tx=0x7fe21400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:16.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:16.896+0000 7fe21d600700 1 -- 172.21.15.63:0/2128831480 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe20c057460 con 0x7fe2201037a0 2024-03-23T12:27:17.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.154+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe22005f5f0 con 0x7fe2201037a0 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.154+0000 7fe21d600700 1 -- 172.21.15.63:0/2128831480 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fe20c05aa80 con 0x7fe2201037a0 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:27:17.133 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:27:17.134 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:17.135 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:17.136 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:17.136 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:17.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe20806c7d0 msgr2=0x7fe20806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe20806c7d0 0x7fe20806ec80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe214009b80 tx=0x7fe21400b040 comp rx=0 tx=0).stop 2024-03-23T12:27:17.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 msgr2=0x7fe220070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe220070800 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009fd0 tx=0x7fe20c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 shutdown_connections 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe20806c7d0 0x7fe20806ec80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe220102e00 0x7fe2200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 --2- 172.21.15.63:0/2128831480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2201037a0 0x7fe220070800 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 >> 172.21.15.63:0/2128831480 conn(0x7fe2200f89b0 msgr2=0x7fe2200f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 shutdown_connections 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.157+0000 7fe228000700 1 -- 172.21.15.63:0/2128831480 wait complete. 2024-03-23T12:27:17.137 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:27:17.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 -- 172.21.15.63:0/3773586120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d40708a0 msgr2=0x7fe8d4070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 --2- 172.21.15.63:0/3773586120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d40708a0 0x7fe8d4070d10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0004970 tx=0x7fe8c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:17.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 -- 172.21.15.63:0/3773586120 shutdown_connections 2024-03-23T12:27:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 --2- 172.21.15.63:0/3773586120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d40708a0 0x7fe8d4070d10 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 --2- 172.21.15.63:0/3773586120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8d4071f90 0x7fe8d4070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 -- 172.21.15.63:0/3773586120 >> 172.21.15.63:0/3773586120 conn(0x7fe8d40f89b0 msgr2=0x7fe8d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 -- 172.21.15.63:0/3773586120 shutdown_connections 2024-03-23T12:27:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.295+0000 7fe8d9600700 1 -- 172.21.15.63:0/3773586120 wait complete. 2024-03-23T12:27:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.296+0000 7fe8d9600700 1 Processor -- start 2024-03-23T12:27:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.296+0000 7fe8d9600700 1 -- start start 2024-03-23T12:27:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d9600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8d40708a0 0x7fe8d411a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d9600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d4071f90 0x7fe8d411a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d9600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d411ac40 con 0x7fe8d4071f90 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d9600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d4114010 con 0x7fe8d40708a0 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d4071f90 0x7fe8d411a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d4071f90 0x7fe8d411a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51176/0 (socket says 172.21.15.63:51176) 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d2000700 1 -- 172.21.15.63:0/436317985 learned_addr learned my addr 172.21.15.63:0/436317985 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.297+0000 7fe8d2a00700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8d40708a0 0x7fe8d411a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8d2000700 1 -- 172.21.15.63:0/436317985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8d40708a0 msgr2=0x7fe8d411a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8d2000700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8d40708a0 0x7fe8d411a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8d2000700 1 -- 172.21.15.63:0/436317985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c0005040 con 0x7fe8d4071f90 2024-03-23T12:27:17.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8d2000700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d4071f90 0x7fe8d411a5f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0005010 tx=0x7fe8c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:17.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8caa00700 1 -- 172.21.15.63:0/436317985 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c000e040 con 0x7fe8d4071f90 2024-03-23T12:27:17.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8caa00700 1 -- 172.21.15.63:0/436317985 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8c0009930 con 0x7fe8d4071f90 2024-03-23T12:27:17.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8caa00700 1 -- 172.21.15.63:0/436317985 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c001c3f0 con 0x7fe8d4071f90 2024-03-23T12:27:17.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d4114290 con 0x7fe8d4071f90 2024-03-23T12:27:17.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.298+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d4114780 con 0x7fe8d4071f90 2024-03-23T12:27:17.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.299+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d4071aa0 con 0x7fe8d4071f90 2024-03-23T12:27:17.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.301+0000 7fe8caa00700 1 -- 172.21.15.63:0/436317985 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe8c0013420 con 0x7fe8d4071f90 2024-03-23T12:27:17.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.301+0000 7fe8caa00700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8b806c7c0 0x7fe8b806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:17.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.302+0000 7fe8caa00700 1 -- 172.21.15.63:0/436317985 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe8c008ce20 con 0x7fe8d4071f90 2024-03-23T12:27:17.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.302+0000 7fe8d2a00700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8b806c7c0 0x7fe8b806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:17.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.302+0000 7fe8d2a00700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8b806c7c0 0x7fe8b806ec70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4004600 tx=0x7fe8c400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:17.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.306+0000 7fe8caa00700 1 -- 172.21.15.63:0/436317985 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe8c005ab30 con 0x7fe8d4071f90 2024-03-23T12:27:17.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:16 smithi063 ceph-mon[141124]: pgmap v1085: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:17.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:16 smithi063 ceph-mon[141124]: from='client.35950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:17.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:16 smithi063 ceph-mon[141124]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:17.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:16 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/64760150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:27:17.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:16 smithi111 ceph-mon[106899]: pgmap v1085: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:17.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:16 smithi111 ceph-mon[106899]: from='client.35950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:17.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:16 smithi111 ceph-mon[106899]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:17.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:16 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/64760150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:27:17.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.539+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8d4110450 con 0x7fe8b806c7c0 2024-03-23T12:27:17.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.540+0000 7fe8caa00700 1 -- 172.21.15.63:0/436317985 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe8d4110450 con 0x7fe8b806c7c0 2024-03-23T12:27:17.518 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:27:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:27:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:27:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:27:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:27:17.519 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8b806c7c0 msgr2=0x7fe8b806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8b806c7c0 0x7fe8b806ec70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4004600 tx=0x7fe8c400b040 comp rx=0 tx=0).stop 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d4071f90 msgr2=0x7fe8d411a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d4071f90 0x7fe8d411a5f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0005010 tx=0x7fe8c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 shutdown_connections 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8b806c7c0 0x7fe8b806ec70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8d40708a0 0x7fe8d411a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 --2- 172.21.15.63:0/436317985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8d4071f90 0x7fe8d411a5f0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 >> 172.21.15.63:0/436317985 conn(0x7fe8d40f89b0 msgr2=0x7fe8d40f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:17.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.542+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 shutdown_connections 2024-03-23T12:27:17.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.543+0000 7fe8d9600700 1 -- 172.21.15.63:0/436317985 wait complete. 2024-03-23T12:27:17.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 -- 172.21.15.63:0/285543900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe910102e00 msgr2=0x7fe9101031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 --2- 172.21.15.63:0/285543900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe910102e00 0x7fe9101031d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe900004940 tx=0x7fe900009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 -- 172.21.15.63:0/285543900 shutdown_connections 2024-03-23T12:27:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 --2- 172.21.15.63:0/285543900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9101037a0 0x7fe9100fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 --2- 172.21.15.63:0/285543900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe910102e00 0x7fe9101031d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 -- 172.21.15.63:0/285543900 >> 172.21.15.63:0/285543900 conn(0x7fe9100f89b0 msgr2=0x7fe9100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 -- 172.21.15.63:0/285543900 shutdown_connections 2024-03-23T12:27:17.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.680+0000 7fe916600700 1 -- 172.21.15.63:0/285543900 wait complete. 2024-03-23T12:27:17.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.681+0000 7fe916600700 1 Processor -- start 2024-03-23T12:27:17.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.681+0000 7fe916600700 1 -- start start 2024-03-23T12:27:17.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe916600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe910102e00 0x7fe910115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:17.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe916600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9101037a0 0x7fe910116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:17.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe916600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe910116880 con 0x7fe9101037a0 2024-03-23T12:27:17.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe916600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91010fc50 con 0x7fe910102e00 2024-03-23T12:27:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe90ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9101037a0 0x7fe910116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe90ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9101037a0 0x7fe910116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51186/0 (socket says 172.21.15.63:51186) 2024-03-23T12:27:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe90ea00700 1 -- 172.21.15.63:0/3875608906 learned_addr learned my addr 172.21.15.63:0/3875608906 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe90f400700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe910102e00 0x7fe910115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe90ea00700 1 -- 172.21.15.63:0/3875608906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe910102e00 msgr2=0x7fe910115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe90ea00700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe910102e00 0x7fe910115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.682+0000 7fe90ea00700 1 -- 172.21.15.63:0/3875608906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe900005040 con 0x7fe9101037a0 2024-03-23T12:27:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.683+0000 7fe90ea00700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9101037a0 0x7fe910116230 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc002a80 tx=0x7fe8fc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.683+0000 7fe907400700 1 -- 172.21.15.63:0/3875608906 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8fc004e80 con 0x7fe9101037a0 2024-03-23T12:27:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.683+0000 7fe907400700 1 -- 172.21.15.63:0/3875608906 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8fc00f040 con 0x7fe9101037a0 2024-03-23T12:27:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.683+0000 7fe907400700 1 -- 172.21.15.63:0/3875608906 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8fc00da00 con 0x7fe9101037a0 2024-03-23T12:27:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.683+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91010ff30 con 0x7fe9101037a0 2024-03-23T12:27:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.683+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9101104f0 con 0x7fe9101037a0 2024-03-23T12:27:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.684+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9100fe550 con 0x7fe9101037a0 2024-03-23T12:27:17.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.685+0000 7fe907400700 1 -- 172.21.15.63:0/3875608906 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe8fc007cc0 con 0x7fe9101037a0 2024-03-23T12:27:17.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.686+0000 7fe907400700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8f406c7d0 0x7fe8f406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:17.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.686+0000 7fe907400700 1 -- 172.21.15.63:0/3875608906 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe8fc08b0a0 con 0x7fe9101037a0 2024-03-23T12:27:17.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.686+0000 7fe90f400700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8f406c7d0 0x7fe8f406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:17.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.687+0000 7fe90f400700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8f406c7d0 0x7fe8f406ec80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe900009fd0 tx=0x7fe900009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:17.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.691+0000 7fe907400700 1 -- 172.21.15.63:0/3875608906 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe8fc058d30 con 0x7fe9101037a0 2024-03-23T12:27:17.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.986+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe910109de0 con 0x7fe9101037a0 2024-03-23T12:27:17.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.986+0000 7fe907400700 1 -- 172.21.15.63:0/3875608906 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fe8fc017e10 con 0x7fe9101037a0 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:27:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:27:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:27:17.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8f406c7d0 msgr2=0x7fe8f406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8f406c7d0 0x7fe8f406ec80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe900009fd0 tx=0x7fe900009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:17.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9101037a0 msgr2=0x7fe910116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:17.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9101037a0 0x7fe910116230 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc002a80 tx=0x7fe8fc004630 comp rx=0 tx=0).stop 2024-03-23T12:27:17.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 shutdown_connections 2024-03-23T12:27:17.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe8f406c7d0 0x7fe8f406ec80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe910102e00 0x7fe910115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 --2- 172.21.15.63:0/3875608906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe9101037a0 0x7fe910116230 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:17.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 >> 172.21.15.63:0/3875608906 conn(0x7fe9100f89b0 msgr2=0x7fe9100f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:17.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 shutdown_connections 2024-03-23T12:27:17.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:17.989+0000 7fe916600700 1 -- 172.21.15.63:0/3875608906 wait complete. 2024-03-23T12:27:18.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:17 smithi063 ceph-mon[141124]: from='client.35958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:18.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:17 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2128831480' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:27:18.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:17 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3875608906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:27:18.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:17 smithi111 ceph-mon[106899]: from='client.35958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:18.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:17 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2128831480' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:27:18.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:17 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3875608906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:27:19.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:18 smithi063 ceph-mon[141124]: pgmap v1086: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:19.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:18 smithi063 ceph-mon[141124]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:19.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:18 smithi111 ceph-mon[106899]: pgmap v1086: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:19.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:18 smithi111 ceph-mon[106899]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:21.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:21 smithi063 ceph-mon[141124]: pgmap v1087: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:21.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:21 smithi111 ceph-mon[106899]: pgmap v1087: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:23.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:23 smithi063 ceph-mon[141124]: pgmap v1088: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:23.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:23 smithi111 ceph-mon[106899]: pgmap v1088: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:25.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:25 smithi063 ceph-mon[141124]: pgmap v1089: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:25.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:25 smithi111 ceph-mon[106899]: pgmap v1089: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:26.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:27.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:27 smithi063 ceph-mon[141124]: pgmap v1090: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:27.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:27 smithi111 ceph-mon[106899]: pgmap v1090: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:29.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:29 smithi063 ceph-mon[141124]: pgmap v1091: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:29.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:29 smithi111 ceph-mon[106899]: pgmap v1091: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:31.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:31 smithi063 ceph-mon[141124]: pgmap v1092: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:31.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:31 smithi111 ceph-mon[106899]: pgmap v1092: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:33.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:33 smithi063 ceph-mon[141124]: pgmap v1093: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:33.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:33 smithi111 ceph-mon[106899]: pgmap v1093: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:35.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:35 smithi063 ceph-mon[141124]: pgmap v1094: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:35.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:35 smithi111 ceph-mon[106899]: pgmap v1094: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:37.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:37 smithi063 ceph-mon[141124]: pgmap v1095: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:37.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:37 smithi111 ceph-mon[106899]: pgmap v1095: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:39.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:39 smithi063 ceph-mon[141124]: pgmap v1096: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:39.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:39 smithi111 ceph-mon[106899]: pgmap v1096: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:41.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:41 smithi063 ceph-mon[141124]: pgmap v1097: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:41.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:41.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:41 smithi111 ceph-mon[106899]: pgmap v1097: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:41.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:43.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:43 smithi063 ceph-mon[141124]: pgmap v1098: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:43.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:43 smithi111 ceph-mon[106899]: pgmap v1098: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:45.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:45 smithi063 ceph-mon[141124]: pgmap v1099: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:45.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:45 smithi111 ceph-mon[106899]: pgmap v1099: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:47.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:47 smithi063 ceph-mon[141124]: pgmap v1100: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:47.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:47 smithi111 ceph-mon[106899]: pgmap v1100: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 -- 172.21.15.63:0/2169216607 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 msgr2=0x7fd9240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 --2- 172.21.15.63:0/2169216607 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd9240fce10 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fd90c004940 tx=0x7fd90c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 -- 172.21.15.63:0/2169216607 shutdown_connections 2024-03-23T12:27:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 --2- 172.21.15.63:0/2169216607 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd9240fce10 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 --2- 172.21.15.63:0/2169216607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924102e00 0x7fd9241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 -- 172.21.15.63:0/2169216607 >> 172.21.15.63:0/2169216607 conn(0x7fd9240f89b0 msgr2=0x7fd9240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 -- 172.21.15.63:0/2169216607 shutdown_connections 2024-03-23T12:27:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.145+0000 7fd929a00700 1 -- 172.21.15.63:0/2169216607 wait complete. 2024-03-23T12:27:48.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.146+0000 7fd929a00700 1 Processor -- start 2024-03-23T12:27:48.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.146+0000 7fd929a00700 1 -- start start 2024-03-23T12:27:48.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.146+0000 7fd929a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924102e00 0x7fd924113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.146+0000 7fd929a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd924114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.146+0000 7fd929a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd924114690 con 0x7fd9241037a0 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.146+0000 7fd929a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92410da80 con 0x7fd924102e00 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd91be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd924114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd91be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd924114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52316/0 (socket says 172.21.15.63:52316) 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd91be00700 1 -- 172.21.15.63:0/4099486020 learned_addr learned my addr 172.21.15.63:0/4099486020 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd922a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924102e00 0x7fd924113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd91be00700 1 -- 172.21.15.63:0/4099486020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924102e00 msgr2=0x7fd924113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd91be00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924102e00 0x7fd924113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd91be00700 1 -- 172.21.15.63:0/4099486020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c005040 con 0x7fd9241037a0 2024-03-23T12:27:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd922a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924102e00 0x7fd924113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:27:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.147+0000 7fd91be00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd924114040 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009390 tx=0x7fd90c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.148+0000 7fd91aa00700 1 -- 172.21.15.63:0/4099486020 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd90c00e040 con 0x7fd9241037a0 2024-03-23T12:27:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.148+0000 7fd91aa00700 1 -- 172.21.15.63:0/4099486020 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd90c013420 con 0x7fd9241037a0 2024-03-23T12:27:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.148+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92410dd00 con 0x7fd9241037a0 2024-03-23T12:27:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.148+0000 7fd91aa00700 1 -- 172.21.15.63:0/4099486020 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd90c01c630 con 0x7fd9241037a0 2024-03-23T12:27:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.148+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92410e2c0 con 0x7fd9241037a0 2024-03-23T12:27:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.149+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9240fe550 con 0x7fd9241037a0 2024-03-23T12:27:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.150+0000 7fd91aa00700 1 -- 172.21.15.63:0/4099486020 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fd90c013a00 con 0x7fd9241037a0 2024-03-23T12:27:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.151+0000 7fd91aa00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd90806c7d0 0x7fd90806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.151+0000 7fd91aa00700 1 -- 172.21.15.63:0/4099486020 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd90c08cc40 con 0x7fd9241037a0 2024-03-23T12:27:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.151+0000 7fd922a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd90806c7d0 0x7fd90806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.152+0000 7fd922a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd90806c7d0 0x7fd90806ec80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd914009fd0 tx=0x7fd91400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:48.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.156+0000 7fd91aa00700 1 -- 172.21.15.63:0/4099486020 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd90c05aa10 con 0x7fd9241037a0 2024-03-23T12:27:48.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.387+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd924109da0 con 0x7fd90806c7d0 2024-03-23T12:27:48.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.389+0000 7fd91aa00700 1 -- 172.21.15.63:0/4099486020 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd924109da0 con 0x7fd90806c7d0 2024-03-23T12:27:48.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd90806c7d0 msgr2=0x7fd90806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd90806c7d0 0x7fd90806ec80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd914009fd0 tx=0x7fd91400b040 comp rx=0 tx=0).stop 2024-03-23T12:27:48.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 msgr2=0x7fd924114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd924114040 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009390 tx=0x7fd90c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 shutdown_connections 2024-03-23T12:27:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fd90806c7d0 0x7fd90806ec80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924102e00 0x7fd924113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 --2- 172.21.15.63:0/4099486020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9241037a0 0x7fd924114040 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 >> 172.21.15.63:0/4099486020 conn(0x7fd9240f89b0 msgr2=0x7fd9240f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.391+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 shutdown_connections 2024-03-23T12:27:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.392+0000 7fd929a00700 1 -- 172.21.15.63:0/4099486020 wait complete. 2024-03-23T12:27:48.384 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:27:48.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 -- 172.21.15.63:0/360556008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 msgr2=0x7fbae00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 --2- 172.21.15.63:0/360556008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae00fce10 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fbac8004940 tx=0x7fbac8009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:48.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 -- 172.21.15.63:0/360556008 shutdown_connections 2024-03-23T12:27:48.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 --2- 172.21.15.63:0/360556008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae00fce10 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 --2- 172.21.15.63:0/360556008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0102e00 0x7fbae01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 -- 172.21.15.63:0/360556008 >> 172.21.15.63:0/360556008 conn(0x7fbae00f89b0 msgr2=0x7fbae00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:48.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 -- 172.21.15.63:0/360556008 shutdown_connections 2024-03-23T12:27:48.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.529+0000 7fbae6000700 1 -- 172.21.15.63:0/360556008 wait complete. 2024-03-23T12:27:48.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.530+0000 7fbae6000700 1 Processor -- start 2024-03-23T12:27:48.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.530+0000 7fbae6000700 1 -- start start 2024-03-23T12:27:48.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbae6000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0102e00 0x7fbae0115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbae6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae01161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbae6000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae01167f0 con 0x7fbae01037a0 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbae6000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae010fbc0 con 0x7fbae0102e00 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbad7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae01161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbad7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae01161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52338/0 (socket says 172.21.15.63:52338) 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbad7e00700 1 -- 172.21.15.63:0/1752708223 learned_addr learned my addr 172.21.15.63:0/1752708223 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbadf400700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0102e00 0x7fbae0115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbad7e00700 1 -- 172.21.15.63:0/1752708223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0102e00 msgr2=0x7fbae0115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbad7e00700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0102e00 0x7fbae0115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbad7e00700 1 -- 172.21.15.63:0/1752708223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8005040 con 0x7fbae01037a0 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.531+0000 7fbadf400700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0102e00 0x7fbae0115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.532+0000 7fbad7e00700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae01161a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009390 tx=0x7fbac8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.532+0000 7fbad7400700 1 -- 172.21.15.63:0/1752708223 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac800e040 con 0x7fbae01037a0 2024-03-23T12:27:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.532+0000 7fbad7400700 1 -- 172.21.15.63:0/1752708223 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbac8013420 con 0x7fbae01037a0 2024-03-23T12:27:48.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.532+0000 7fbad7400700 1 -- 172.21.15.63:0/1752708223 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbac801c630 con 0x7fbae01037a0 2024-03-23T12:27:48.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.532+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae010fe40 con 0x7fbae01037a0 2024-03-23T12:27:48.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.532+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae0110400 con 0x7fbae01037a0 2024-03-23T12:27:48.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.533+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae00fe550 con 0x7fbae01037a0 2024-03-23T12:27:48.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.534+0000 7fbad7400700 1 -- 172.21.15.63:0/1752708223 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fbac8013a00 con 0x7fbae01037a0 2024-03-23T12:27:48.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.535+0000 7fbad7400700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbac406c7d0 0x7fbac406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.536+0000 7fbad7400700 1 -- 172.21.15.63:0/1752708223 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbac808cd00 con 0x7fbae01037a0 2024-03-23T12:27:48.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.536+0000 7fbadf400700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbac406c7d0 0x7fbac406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.536+0000 7fbadf400700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbac406c7d0 0x7fbac406ec80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009b80 tx=0x7fbad000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:48.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.540+0000 7fbad7400700 1 -- 172.21.15.63:0/1752708223 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbac805aa10 con 0x7fbae01037a0 2024-03-23T12:27:48.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.773+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbae0109da0 con 0x7fbac406c7d0 2024-03-23T12:27:48.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.774+0000 7fbad7400700 1 -- 172.21.15.63:0/1752708223 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbae0109da0 con 0x7fbac406c7d0 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbac406c7d0 msgr2=0x7fbac406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbac406c7d0 0x7fbac406ec80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009b80 tx=0x7fbad000b040 comp rx=0 tx=0).stop 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 msgr2=0x7fbae01161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae01161a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009390 tx=0x7fbac8009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 shutdown_connections 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fbac406c7d0 0x7fbac406ec80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0102e00 0x7fbae0115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 --2- 172.21.15.63:0/1752708223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbae01037a0 0x7fbae01161a0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 >> 172.21.15.63:0/1752708223 conn(0x7fbae00f89b0 msgr2=0x7fbae00f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 shutdown_connections 2024-03-23T12:27:48.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.777+0000 7fbae6000700 1 -- 172.21.15.63:0/1752708223 wait complete. 2024-03-23T12:27:48.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 -- 172.21.15.63:0/906528186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93201037a0 msgr2=0x7f93200fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 --2- 172.21.15.63:0/906528186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93201037a0 0x7f93200fce10 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f930c004940 tx=0x7f930c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:48.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 -- 172.21.15.63:0/906528186 shutdown_connections 2024-03-23T12:27:48.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 --2- 172.21.15.63:0/906528186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93201037a0 0x7f93200fce10 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 --2- 172.21.15.63:0/906528186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9320102e00 0x7f93201031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 -- 172.21.15.63:0/906528186 >> 172.21.15.63:0/906528186 conn(0x7f93200f89b0 msgr2=0x7f93200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:48.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 -- 172.21.15.63:0/906528186 shutdown_connections 2024-03-23T12:27:48.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.915+0000 7f9327c00700 1 -- 172.21.15.63:0/906528186 wait complete. 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.916+0000 7f9327c00700 1 Processor -- start 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.916+0000 7f9327c00700 1 -- start start 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.916+0000 7f9327c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9320102e00 0x7f9320113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.916+0000 7f9327c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93201037a0 0x7f9320114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.916+0000 7f9327c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9320114690 con 0x7f9320102e00 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.916+0000 7f9327c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932010da80 con 0x7f93201037a0 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f9325200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9320102e00 0x7f9320113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f9325200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9320102e00 0x7f9320113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52354/0 (socket says 172.21.15.63:52354) 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f9325200700 1 -- 172.21.15.63:0/560409725 learned_addr learned my addr 172.21.15.63:0/560409725 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f931fe00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93201037a0 0x7f9320114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f9325200700 1 -- 172.21.15.63:0/560409725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93201037a0 msgr2=0x7f9320114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f9325200700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93201037a0 0x7f9320114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f9325200700 1 -- 172.21.15.63:0/560409725 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c005040 con 0x7f9320102e00 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f931fe00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93201037a0 0x7f9320114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.917+0000 7f9325200700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9320102e00 0x7f9320113b00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9314002a50 tx=0x7f93140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.918+0000 7f931cc00700 1 -- 172.21.15.63:0/560409725 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9314007cc0 con 0x7f9320102e00 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.918+0000 7f931cc00700 1 -- 172.21.15.63:0/560409725 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f931400f040 con 0x7f9320102e00 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.918+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932010dd60 con 0x7f9320102e00 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.918+0000 7f931cc00700 1 -- 172.21.15.63:0/560409725 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9314009540 con 0x7f9320102e00 2024-03-23T12:27:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.918+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932010e320 con 0x7f9320102e00 2024-03-23T12:27:48.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.919+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93200fe550 con 0x7f9320102e00 2024-03-23T12:27:48.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.920+0000 7f931cc00700 1 -- 172.21.15.63:0/560409725 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f93140096a0 con 0x7f9320102e00 2024-03-23T12:27:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.921+0000 7f931cc00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f930806c7c0 0x7f930806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.921+0000 7f931cc00700 1 -- 172.21.15.63:0/560409725 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f931408b920 con 0x7f9320102e00 2024-03-23T12:27:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.921+0000 7f931fe00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f930806c7c0 0x7f930806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.921+0000 7f931fe00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f930806c7c0 0x7f930806ec70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f930c00b3b0 tx=0x7f930c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:48.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:48.925+0000 7f931cc00700 1 -- 172.21.15.63:0/560409725 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9314059530 con 0x7f9320102e00 2024-03-23T12:27:49.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.150+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9320109da0 con 0x7f930806c7c0 2024-03-23T12:27:49.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.160+0000 7f931cc00700 1 -- 172.21.15.63:0/560409725 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f9320109da0 con 0x7f930806c7c0 2024-03-23T12:27:49.138 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (37m) 14m ago 50m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (34m) 14m ago 50m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (34m) 14m ago 49m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (36m) 14m ago 49m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (30m) 14m ago 43m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (29m) 14m ago 43m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (14m) 14m ago 43m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (29m) 14m ago 43m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (39m) 14m ago 51m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (38m) 14m ago 49m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (35m) 14m ago 51m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (35m) 14m ago 49m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:27:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (37m) 14m ago 50m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (37m) 14m ago 49m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (33m) 14m ago 48m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (33m) 14m ago 47m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (32m) 14m ago 47m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (31m) 14m ago 46m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (31m) 14m ago 46m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (30m) 14m ago 45m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:27:49.140 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (37m) 14m ago 49m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f930806c7c0 msgr2=0x7f930806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f930806c7c0 0x7f930806ec70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f930c00b3b0 tx=0x7f930c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9320102e00 msgr2=0x7f9320113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9320102e00 0x7f9320113b00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9314002a50 tx=0x7f93140046b0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 shutdown_connections 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f930806c7c0 0x7f930806ec70 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9320102e00 0x7f9320113b00 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 --2- 172.21.15.63:0/560409725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93201037a0 0x7f9320114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 >> 172.21.15.63:0/560409725 conn(0x7f93200f89b0 msgr2=0x7f93200f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:49.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 shutdown_connections 2024-03-23T12:27:49.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.163+0000 7f9327c00700 1 -- 172.21.15.63:0/560409725 wait complete. 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 -- 172.21.15.63:0/2101996057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 msgr2=0x7fc1341031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 --2- 172.21.15.63:0/2101996057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc1341031f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fc124004940 tx=0x7fc124009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 -- 172.21.15.63:0/2101996057 shutdown_connections 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 --2- 172.21.15.63:0/2101996057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1341037c0 0x7fc1340fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 --2- 172.21.15.63:0/2101996057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc1341031f0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 -- 172.21.15.63:0/2101996057 >> 172.21.15.63:0/2101996057 conn(0x7fc1340f89d0 msgr2=0x7fc1340fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 -- 172.21.15.63:0/2101996057 shutdown_connections 2024-03-23T12:27:49.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.300+0000 7fc139800700 1 -- 172.21.15.63:0/2101996057 wait complete. 2024-03-23T12:27:49.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc139800700 1 Processor -- start 2024-03-23T12:27:49.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc139800700 1 -- start start 2024-03-23T12:27:49.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc139800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc134113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:49.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc139800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1341037c0 0x7fc134114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:49.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc139800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134114690 con 0x7fc134102e20 2024-03-23T12:27:49.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc139800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13410da80 con 0x7fc1341037c0 2024-03-23T12:27:49.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc132a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc134113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:49.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.301+0000 7fc132a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc134113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52386/0 (socket says 172.21.15.63:52386) 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc132a00700 1 -- 172.21.15.63:0/587611906 learned_addr learned my addr 172.21.15.63:0/587611906 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc132000700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1341037c0 0x7fc134114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc132a00700 1 -- 172.21.15.63:0/587611906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1341037c0 msgr2=0x7fc134114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc132a00700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1341037c0 0x7fc134114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc132a00700 1 -- 172.21.15.63:0/587611906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc124005040 con 0x7fc134102e20 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc132000700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1341037c0 0x7fc134114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc132a00700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc134113b00 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc124009360 tx=0x7fc124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc12aa00700 1 -- 172.21.15.63:0/587611906 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc12400e040 con 0x7fc134102e20 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc12aa00700 1 -- 172.21.15.63:0/587611906 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc124013420 con 0x7fc134102e20 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.302+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13410dd00 con 0x7fc134102e20 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.303+0000 7fc12aa00700 1 -- 172.21.15.63:0/587611906 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc12401c5f0 con 0x7fc134102e20 2024-03-23T12:27:49.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.303+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13410e2c0 con 0x7fc134102e20 2024-03-23T12:27:49.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.304+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1340fe570 con 0x7fc134102e20 2024-03-23T12:27:49.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.305+0000 7fc12aa00700 1 -- 172.21.15.63:0/587611906 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc124013590 con 0x7fc134102e20 2024-03-23T12:27:49.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.305+0000 7fc12aa00700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc11806c7d0 0x7fc11806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:49.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.306+0000 7fc12aa00700 1 -- 172.21.15.63:0/587611906 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc12408cc80 con 0x7fc134102e20 2024-03-23T12:27:49.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.306+0000 7fc132000700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc11806c7d0 0x7fc11806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:49.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.306+0000 7fc132000700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc11806c7d0 0x7fc11806ec80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc120007fd0 tx=0x7fc120009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:49.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.310+0000 7fc12aa00700 1 -- 172.21.15.63:0/587611906 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc12405a990 con 0x7fc134102e20 2024-03-23T12:27:49.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:49 smithi063 ceph-mon[141124]: pgmap v1101: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:49.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:49 smithi111 ceph-mon[106899]: pgmap v1101: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.607+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc134109de0 con 0x7fc134102e20 2024-03-23T12:27:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.607+0000 7fc12aa00700 1 -- 172.21.15.63:0/587611906 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fc12405a520 con 0x7fc134102e20 2024-03-23T12:27:49.585 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:27:49.586 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:27:49.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:27:49.587 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc11806c7d0 msgr2=0x7fc11806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc11806c7d0 0x7fc11806ec80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc120007fd0 tx=0x7fc120009040 comp rx=0 tx=0).stop 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 msgr2=0x7fc134113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc134113b00 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc124009360 tx=0x7fc124009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 shutdown_connections 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc11806c7d0 0x7fc11806ec80 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc134102e20 0x7fc134113b00 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 --2- 172.21.15.63:0/587611906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1341037c0 0x7fc134114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 >> 172.21.15.63:0/587611906 conn(0x7fc1340f89d0 msgr2=0x7fc1340f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 shutdown_connections 2024-03-23T12:27:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.610+0000 7fc139800700 1 -- 172.21.15.63:0/587611906 wait complete. 2024-03-23T12:27:49.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 -- 172.21.15.63:0/988210550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 msgr2=0x7f95ac103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 --2- 172.21.15.63:0/988210550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac103070 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f959c004940 tx=0x7f959c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:49.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 -- 172.21.15.63:0/988210550 shutdown_connections 2024-03-23T12:27:49.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 --2- 172.21.15.63:0/988210550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95ac065630 0x7f95ac065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 --2- 172.21.15.63:0/988210550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac103070 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 -- 172.21.15.63:0/988210550 >> 172.21.15.63:0/988210550 conn(0x7f95ac071b70 msgr2=0x7f95ac071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:49.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 -- 172.21.15.63:0/988210550 shutdown_connections 2024-03-23T12:27:49.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.746+0000 7f95b0c00700 1 -- 172.21.15.63:0/988210550 wait complete. 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.747+0000 7f95b0c00700 1 Processor -- start 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.747+0000 7f95b0c00700 1 -- start start 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95b0c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95ac065630 0x7f95ac1149b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95b0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95b0c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac1154f0 con 0x7f95ac102ca0 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95b0c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac10fac0 con 0x7f95ac065630 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95aaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95aaa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac114ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52394/0 (socket says 172.21.15.63:52394) 2024-03-23T12:27:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95aaa00700 1 -- 172.21.15.63:0/1112547614 learned_addr learned my addr 172.21.15.63:0/1112547614 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:49.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95aaa00700 1 -- 172.21.15.63:0/1112547614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95ac065630 msgr2=0x7f95ac1149b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95aaa00700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95ac065630 0x7f95ac1149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95aaa00700 1 -- 172.21.15.63:0/1112547614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c005040 con 0x7f95ac102ca0 2024-03-23T12:27:49.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.748+0000 7f95aaa00700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac114ef0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9598002a50 tx=0x7f95980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:49.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.749+0000 7f95a3400700 1 -- 172.21.15.63:0/1112547614 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9598007cc0 con 0x7f95ac102ca0 2024-03-23T12:27:49.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.749+0000 7f95a3400700 1 -- 172.21.15.63:0/1112547614 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f959800f040 con 0x7f95ac102ca0 2024-03-23T12:27:49.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.749+0000 7f95a3400700 1 -- 172.21.15.63:0/1112547614 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9598009540 con 0x7f95ac102ca0 2024-03-23T12:27:49.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.749+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac10fda0 con 0x7f95ac102ca0 2024-03-23T12:27:49.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.749+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac110360 con 0x7f95ac102ca0 2024-03-23T12:27:49.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.750+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac106840 con 0x7f95ac102ca0 2024-03-23T12:27:49.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.751+0000 7f95a3400700 1 -- 172.21.15.63:0/1112547614 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f95980096a0 con 0x7f95ac102ca0 2024-03-23T12:27:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.752+0000 7f95a3400700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f959006c7d0 0x7f959006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.752+0000 7f95a3400700 1 -- 172.21.15.63:0/1112547614 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f959808b820 con 0x7f95ac102ca0 2024-03-23T12:27:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.753+0000 7f95ab400700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f959006c7d0 0x7f959006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:49.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.753+0000 7f95ab400700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f959006c7d0 0x7f959006ec80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f959c004f90 tx=0x7f959c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:49.757+0000 7f95a3400700 1 -- 172.21.15.63:0/1112547614 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9598055a20 con 0x7f95ac102ca0 2024-03-23T12:27:49.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.014+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f95ac111120 con 0x7f95ac102ca0 2024-03-23T12:27:49.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.015+0000 7f95a3400700 1 -- 172.21.15.63:0/1112547614 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f9598059040 con 0x7f95ac102ca0 2024-03-23T12:27:49.993 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:27:49.993 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:27:49.993 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:27:49.993 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:27:49.993 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:27:49.994 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:49.995 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:49.996 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:27:49.996 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:27:49.996 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:49.996 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:49.996 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:49.996 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.017+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f959006c7d0 msgr2=0x7f959006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.017+0000 7f95b0c00700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f959006c7d0 0x7f959006ec80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f959c004f90 tx=0x7f959c009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.017+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 msgr2=0x7f95ac114ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.017+0000 7f95b0c00700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac114ef0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9598002a50 tx=0x7f95980046b0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.018+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 shutdown_connections 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.018+0000 7f95b0c00700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f959006c7d0 0x7f959006ec80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.018+0000 7f95b0c00700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95ac065630 0x7f95ac1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.018+0000 7f95b0c00700 1 --2- 172.21.15.63:0/1112547614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95ac102ca0 0x7f95ac114ef0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.018+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 >> 172.21.15.63:0/1112547614 conn(0x7f95ac071b70 msgr2=0x7f95ac1039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.018+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 shutdown_connections 2024-03-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.018+0000 7f95b0c00700 1 -- 172.21.15.63:0/1112547614 wait complete. 2024-03-23T12:27:49.998 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:27:50.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 -- 172.21.15.63:0/3872398956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070c10 msgr2=0x7f0710109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 --2- 172.21.15.63:0/3872398956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070c10 0x7f0710109170 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f0700004970 tx=0x7f0700009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:50.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 -- 172.21.15.63:0/3872398956 shutdown_connections 2024-03-23T12:27:50.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 --2- 172.21.15.63:0/3872398956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070c10 0x7f0710109170 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 --2- 172.21.15.63:0/3872398956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0710070300 0x7f07100706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 -- 172.21.15.63:0/3872398956 >> 172.21.15.63:0/3872398956 conn(0x7f07100f89b0 msgr2=0x7f07100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:50.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 -- 172.21.15.63:0/3872398956 shutdown_connections 2024-03-23T12:27:50.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.158+0000 7f0718800700 1 -- 172.21.15.63:0/3872398956 wait complete. 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.159+0000 7f0718800700 1 Processor -- start 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.159+0000 7f0718800700 1 -- start start 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0718800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070300 0x7f0710115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0718800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0710070c10 0x7f0710115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0718800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0710116610 con 0x7f0710070300 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0718800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071010fbf0 con 0x7f0710070c10 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070300 0x7f0710115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070300 0x7f0710115a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52416/0 (socket says 172.21.15.63:52416) 2024-03-23T12:27:50.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715e00700 1 -- 172.21.15.63:0/1468342659 learned_addr learned my addr 172.21.15.63:0/1468342659 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:50.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715400700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0710070c10 0x7f0710115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:50.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715e00700 1 -- 172.21.15.63:0/1468342659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0710070c10 msgr2=0x7f0710115fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715e00700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0710070c10 0x7f0710115fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715e00700 1 -- 172.21.15.63:0/1468342659 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700005040 con 0x7f0710070300 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.160+0000 7f0715400700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0710070c10 0x7f0710115fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.161+0000 7f0715e00700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070300 0x7f0710115a80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f070400c8c0 tx=0x7f07040044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.161+0000 7f070a000700 1 -- 172.21.15.63:0/1468342659 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f070400a980 con 0x7f0710070300 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.161+0000 7f070a000700 1 -- 172.21.15.63:0/1468342659 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f070400ce60 con 0x7f0710070300 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.161+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071010fed0 con 0x7f0710070300 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.161+0000 7f070a000700 1 -- 172.21.15.63:0/1468342659 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07040113a0 con 0x7f0710070300 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.161+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0710110410 con 0x7f0710070300 2024-03-23T12:27:50.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.162+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0710071a40 con 0x7f0710070300 2024-03-23T12:27:50.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.163+0000 7f070a000700 1 -- 172.21.15.63:0/1468342659 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0704012400 con 0x7f0710070300 2024-03-23T12:27:50.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.164+0000 7f070a000700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f06f806c7c0 0x7f06f806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:50.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.164+0000 7f070a000700 1 -- 172.21.15.63:0/1468342659 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f070408bb30 con 0x7f0710070300 2024-03-23T12:27:50.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.164+0000 7f0715400700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f06f806c7c0 0x7f06f806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:50.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.165+0000 7f0715400700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f06f806c7c0 0x7f06f806ec70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f070000b800 tx=0x7f0700009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:50.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.168+0000 7f070a000700 1 -- 172.21.15.63:0/1468342659 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0704059840 con 0x7f0710070300 2024-03-23T12:27:50.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:50 smithi063 ceph-mon[141124]: from='client.35978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:50.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:50 smithi063 ceph-mon[141124]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:50.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:50 smithi063 ceph-mon[141124]: from='client.35986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:50.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:50 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/587611906' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:27:50.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:50 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1112547614' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:27:50.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.406+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07101110d0 con 0x7f06f806c7c0 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.407+0000 7f070a000700 1 -- 172.21.15.63:0/1468342659 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f07101110d0 con 0x7f06f806c7c0 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:27:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:27:50.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:27:50.386 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:27:50.386 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:27:50.386 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:27:50.386 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:27:50.386 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:27:50.386 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:27:50.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f06f806c7c0 msgr2=0x7f06f806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f06f806c7c0 0x7f06f806ec70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f070000b800 tx=0x7f0700009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:50.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070300 msgr2=0x7f0710115a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070300 0x7f0710115a80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f070400c8c0 tx=0x7f07040044c0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 shutdown_connections 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f06f806c7c0 0x7f06f806ec70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0710070300 0x7f0710115a80 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 --2- 172.21.15.63:0/1468342659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0710070c10 0x7f0710115fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.409+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 >> 172.21.15.63:0/1468342659 conn(0x7f07100f89b0 msgr2=0x7f0710105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.410+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 shutdown_connections 2024-03-23T12:27:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.410+0000 7f0718800700 1 -- 172.21.15.63:0/1468342659 wait complete. 2024-03-23T12:27:50.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:50 smithi111 ceph-mon[106899]: from='client.35978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:50.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:50 smithi111 ceph-mon[106899]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:50.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:50 smithi111 ceph-mon[106899]: from='client.35986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:50.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:50 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/587611906' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:27:50.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:50 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1112547614' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:27:50.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 -- 172.21.15.63:0/1132165881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fde90 msgr2=0x7fa7680fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 --2- 172.21.15.63:0/1132165881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fde90 0x7fa7680fe260 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa758004940 tx=0x7fa758009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:50.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 -- 172.21.15.63:0/1132165881 shutdown_connections 2024-03-23T12:27:50.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 --2- 172.21.15.63:0/1132165881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7680fe830 0x7fa768106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 --2- 172.21.15.63:0/1132165881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fde90 0x7fa7680fe260 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 -- 172.21.15.63:0/1132165881 >> 172.21.15.63:0/1132165881 conn(0x7fa768072bc0 msgr2=0x7fa768072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:50.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 -- 172.21.15.63:0/1132165881 shutdown_connections 2024-03-23T12:27:50.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.547+0000 7fa76d600700 1 -- 172.21.15.63:0/1132165881 wait complete. 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.548+0000 7fa76d600700 1 Processor -- start 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.548+0000 7fa76d600700 1 -- start start 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.548+0000 7fa76d600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7680fde90 0x7fa768115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.548+0000 7fa76d600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fe830 0x7fa768116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.548+0000 7fa76d600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7681168b0 con 0x7fa7680fe830 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.548+0000 7fa76d600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76810fc80 con 0x7fa7680fde90 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fe830 0x7fa768116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fe830 0x7fa768116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52438/0 (socket says 172.21.15.63:52438) 2024-03-23T12:27:50.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766000700 1 -- 172.21.15.63:0/3249576635 learned_addr learned my addr 172.21.15.63:0/3249576635 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:27:50.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766a00700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7680fde90 0x7fa768115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:50.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766000700 1 -- 172.21.15.63:0/3249576635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7680fde90 msgr2=0x7fa768115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766000700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7680fde90 0x7fa768115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766000700 1 -- 172.21.15.63:0/3249576635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa758005040 con 0x7fa7680fe830 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.549+0000 7fa766000700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fe830 0x7fa768116260 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa75400c8c0 tx=0x7fa7540044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.550+0000 7fa75ea00700 1 -- 172.21.15.63:0/3249576635 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75400a760 con 0x7fa7680fe830 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.550+0000 7fa75ea00700 1 -- 172.21.15.63:0/3249576635 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa75400ad70 con 0x7fa7680fe830 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.550+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76810ff60 con 0x7fa7680fe830 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.550+0000 7fa75ea00700 1 -- 172.21.15.63:0/3249576635 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7540113a0 con 0x7fa7680fe830 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.550+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7681104a0 con 0x7fa7680fe830 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.551+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa768106a20 con 0x7fa7680fe830 2024-03-23T12:27:50.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.552+0000 7fa75ea00700 1 -- 172.21.15.63:0/3249576635 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa754012400 con 0x7fa7680fe830 2024-03-23T12:27:50.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.553+0000 7fa75ea00700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa74c06c7d0 0x7fa74c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:27:50.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.553+0000 7fa75ea00700 1 -- 172.21.15.63:0/3249576635 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa75408ba60 con 0x7fa7680fe830 2024-03-23T12:27:50.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.553+0000 7fa766a00700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa74c06c7d0 0x7fa74c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:27:50.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.553+0000 7fa766a00700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa74c06c7d0 0x7fa74c06ec80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b3b0 tx=0x7fa758009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:27:50.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.557+0000 7fa75ea00700 1 -- 172.21.15.63:0/3249576635 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa754059770 con 0x7fa7680fe830 2024-03-23T12:27:50.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.857+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa768065e30 con 0x7fa7680fe830 2024-03-23T12:27:50.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.857+0000 7fa75ea00700 1 -- 172.21.15.63:0/3249576635 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fa754059300 con 0x7fa7680fe830 2024-03-23T12:27:50.835 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:27:50.835 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:27:50.835 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:27:50.835 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:27:50.835 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:27:50.836 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:27:50.837 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:27:50.837 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:27:50.837 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:27:50.837 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:27:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa74c06c7d0 msgr2=0x7fa74c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa74c06c7d0 0x7fa74c06ec80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b3b0 tx=0x7fa758009e30 comp rx=0 tx=0).stop 2024-03-23T12:27:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fe830 msgr2=0x7fa768116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:27:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fe830 0x7fa768116260 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa75400c8c0 tx=0x7fa7540044c0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 shutdown_connections 2024-03-23T12:27:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa74c06c7d0 0x7fa74c06ec80 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7680fde90 0x7fa768115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 --2- 172.21.15.63:0/3249576635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7680fe830 0x7fa768116260 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:27:50.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 >> 172.21.15.63:0/3249576635 conn(0x7fa768072bc0 msgr2=0x7fa7680fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:27:50.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 shutdown_connections 2024-03-23T12:27:50.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:27:50.860+0000 7fa76d600700 1 -- 172.21.15.63:0/3249576635 wait complete. 2024-03-23T12:27:51.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:51 smithi063 ceph-mon[141124]: pgmap v1102: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:51.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:51 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3249576635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:27:51.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:51 smithi111 ceph-mon[106899]: pgmap v1102: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:51.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:51 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3249576635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:27:52.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:52 smithi063 ceph-mon[141124]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:52.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:52 smithi111 ceph-mon[106899]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:27:53.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:53 smithi063 ceph-mon[141124]: pgmap v1103: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:53.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:53 smithi111 ceph-mon[106899]: pgmap v1103: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:55.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:55 smithi063 ceph-mon[141124]: pgmap v1104: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:55 smithi111 ceph-mon[106899]: pgmap v1104: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:56.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:56.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:27:57.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:57 smithi063 ceph-mon[141124]: pgmap v1105: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:57.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:57 smithi111 ceph-mon[106899]: pgmap v1105: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:59.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:27:59 smithi063 ceph-mon[141124]: pgmap v1106: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:27:59.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:27:59 smithi111 ceph-mon[106899]: pgmap v1106: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:01.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:01 smithi063 ceph-mon[141124]: pgmap v1107: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:01.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:01 smithi111 ceph-mon[106899]: pgmap v1107: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:03.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:03 smithi063 ceph-mon[141124]: pgmap v1108: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:03.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:03 smithi111 ceph-mon[106899]: pgmap v1108: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:05.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:05 smithi063 ceph-mon[141124]: pgmap v1109: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:05.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:05 smithi111 ceph-mon[106899]: pgmap v1109: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:07.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:07 smithi063 ceph-mon[141124]: pgmap v1110: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:07.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:07 smithi111 ceph-mon[106899]: pgmap v1110: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:09.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:09 smithi063 ceph-mon[141124]: pgmap v1111: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:09.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:09 smithi111 ceph-mon[106899]: pgmap v1111: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:11.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:11 smithi063 ceph-mon[141124]: pgmap v1112: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:11.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:11.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:11 smithi111 ceph-mon[106899]: pgmap v1112: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:11.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:13.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:13 smithi063 ceph-mon[141124]: pgmap v1113: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:13.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:13 smithi111 ceph-mon[106899]: pgmap v1113: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:15.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:15 smithi063 ceph-mon[141124]: pgmap v1114: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:15.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:15 smithi111 ceph-mon[106899]: pgmap v1114: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:17.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:17 smithi063 ceph-mon[141124]: pgmap v1115: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:17.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:17 smithi111 ceph-mon[106899]: pgmap v1115: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:19.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:19 smithi063 ceph-mon[141124]: pgmap v1116: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:19.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:19 smithi111 ceph-mon[106899]: pgmap v1116: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:20.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 -- 172.21.15.63:0/2902836045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d38102e00 msgr2=0x7f0d381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:20.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/2902836045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d38102e00 0x7f0d381031d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c004940 tx=0x7f0d2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:20.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 -- 172.21.15.63:0/2902836045 shutdown_connections 2024-03-23T12:28:20.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/2902836045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d381037a0 0x7f0d380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:20.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/2902836045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d38102e00 0x7f0d381031d0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:20.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 -- 172.21.15.63:0/2902836045 >> 172.21.15.63:0/2902836045 conn(0x7f0d380f89b0 msgr2=0x7f0d380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:20.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 -- 172.21.15.63:0/2902836045 shutdown_connections 2024-03-23T12:28:20.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.013+0000 7f0d3fe00700 1 -- 172.21.15.63:0/2902836045 wait complete. 2024-03-23T12:28:20.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.014+0000 7f0d3fe00700 1 Processor -- start 2024-03-23T12:28:20.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.014+0000 7f0d3fe00700 1 -- start start 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d38102e00 0x7f0d38115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d381037a0 0x7f0d38116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d38116880 con 0x7f0d381037a0 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3fe00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3810fc50 con 0x7f0d38102e00 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d381037a0 0x7f0d38116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d381037a0 0x7f0d38116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49058/0 (socket says 172.21.15.63:49058) 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3ca00700 1 -- 172.21.15.63:0/1085612591 learned_addr learned my addr 172.21.15.63:0/1085612591 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3d400700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d38102e00 0x7f0d38115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3ca00700 1 -- 172.21.15.63:0/1085612591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d38102e00 msgr2=0x7f0d38115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3ca00700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d38102e00 0x7f0d38115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.015+0000 7f0d3ca00700 1 -- 172.21.15.63:0/1085612591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c005040 con 0x7f0d381037a0 2024-03-23T12:28:20.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.016+0000 7f0d3ca00700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d381037a0 0x7f0d38116230 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0d24002a50 tx=0x7f0d24004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:20.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.016+0000 7f0d35600700 1 -- 172.21.15.63:0/1085612591 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d24007cc0 con 0x7f0d381037a0 2024-03-23T12:28:20.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.016+0000 7f0d35600700 1 -- 172.21.15.63:0/1085612591 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d2400f040 con 0x7f0d381037a0 2024-03-23T12:28:20.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.016+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3810ff30 con 0x7f0d381037a0 2024-03-23T12:28:20.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.016+0000 7f0d35600700 1 -- 172.21.15.63:0/1085612591 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d2400d920 con 0x7f0d381037a0 2024-03-23T12:28:20.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.016+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d381104f0 con 0x7f0d381037a0 2024-03-23T12:28:20.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.018+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d380fe550 con 0x7f0d381037a0 2024-03-23T12:28:20.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.018+0000 7f0d35600700 1 -- 172.21.15.63:0/1085612591 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0d24009680 con 0x7f0d381037a0 2024-03-23T12:28:20.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.019+0000 7f0d35600700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d2006c7d0 0x7f0d2006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:20.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.019+0000 7f0d35600700 1 -- 172.21.15.63:0/1085612591 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0d2408b770 con 0x7f0d381037a0 2024-03-23T12:28:20.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.019+0000 7f0d3d400700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d2006c7d0 0x7f0d2006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:20.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.020+0000 7f0d3d400700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d2006c7d0 0x7f0d2006ec80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009fd0 tx=0x7f0d2c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:21.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.024+0000 7f0d35600700 1 -- 172.21.15.63:0/1085612591 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0d24059400 con 0x7f0d381037a0 2024-03-23T12:28:21.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.259+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d38109e30 con 0x7f0d2006c7d0 2024-03-23T12:28:21.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.261+0000 7f0d35600700 1 -- 172.21.15.63:0/1085612591 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0d38109e30 con 0x7f0d2006c7d0 2024-03-23T12:28:21.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.263+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d2006c7d0 msgr2=0x7f0d2006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.263+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d2006c7d0 0x7f0d2006ec80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009fd0 tx=0x7f0d2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.263+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d381037a0 msgr2=0x7f0d38116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.263+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d381037a0 0x7f0d38116230 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0d24002a50 tx=0x7f0d24004600 comp rx=0 tx=0).stop 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.264+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 shutdown_connections 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.264+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0d2006c7d0 0x7f0d2006ec80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.264+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d38102e00 0x7f0d38115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.264+0000 7f0d3fe00700 1 --2- 172.21.15.63:0/1085612591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d381037a0 0x7f0d38116230 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.264+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 >> 172.21.15.63:0/1085612591 conn(0x7f0d380f89b0 msgr2=0x7f0d380706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.264+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 shutdown_connections 2024-03-23T12:28:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.264+0000 7f0d3fe00700 1 -- 172.21.15.63:0/1085612591 wait complete. 2024-03-23T12:28:21.255 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:28:21.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:21 smithi063 ceph-mon[141124]: pgmap v1117: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.402+0000 7fa1d6c00700 1 -- 172.21.15.63:0/1761017888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d0102e00 msgr2=0x7fa1d01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.402+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/1761017888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d0102e00 0x7fa1d01031d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0004940 tx=0x7fa1c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.402+0000 7fa1d6c00700 1 -- 172.21.15.63:0/1761017888 shutdown_connections 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.402+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/1761017888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1d01037a0 0x7fa1d00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.402+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/1761017888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d0102e00 0x7fa1d01031d0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.402+0000 7fa1d6c00700 1 -- 172.21.15.63:0/1761017888 >> 172.21.15.63:0/1761017888 conn(0x7fa1d00f89b0 msgr2=0x7fa1d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.403+0000 7fa1d6c00700 1 -- 172.21.15.63:0/1761017888 shutdown_connections 2024-03-23T12:28:21.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.403+0000 7fa1d6c00700 1 -- 172.21.15.63:0/1761017888 wait complete. 2024-03-23T12:28:21.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.403+0000 7fa1d6c00700 1 Processor -- start 2024-03-23T12:28:21.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1d6c00700 1 -- start start 2024-03-23T12:28:21.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1d6c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1d0102e00 0x7fa1d0113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:21.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1d6c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d01037a0 0x7fa1d0113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1d6c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d01145d0 con 0x7fa1d01037a0 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1d6c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d010d9c0 con 0x7fa1d0102e00 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1cf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d01037a0 0x7fa1d0113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1cf400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d01037a0 0x7fa1d0113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49070/0 (socket says 172.21.15.63:49070) 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.404+0000 7fa1cf400700 1 -- 172.21.15.63:0/161023542 learned_addr learned my addr 172.21.15.63:0/161023542 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1cfe00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1d0102e00 0x7fa1d0113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1cf400700 1 -- 172.21.15.63:0/161023542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1d0102e00 msgr2=0x7fa1d0113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1cf400700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1d0102e00 0x7fa1d0113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1cf400700 1 -- 172.21.15.63:0/161023542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c0005040 con 0x7fa1d01037a0 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1cfe00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1d0102e00 0x7fa1d0113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:28:21.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1cf400700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d01037a0 0x7fa1d0113f80 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00eb40 tx=0x7fa1bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:21.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1c7e00700 1 -- 172.21.15.63:0/161023542 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1bc007cc0 con 0x7fa1d01037a0 2024-03-23T12:28:21.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1c7e00700 1 -- 172.21.15.63:0/161023542 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1bc00f040 con 0x7fa1d01037a0 2024-03-23T12:28:21.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d010dc40 con 0x7fa1d01037a0 2024-03-23T12:28:21.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1c7e00700 1 -- 172.21.15.63:0/161023542 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1bc009570 con 0x7fa1d01037a0 2024-03-23T12:28:21.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.405+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d010e200 con 0x7fa1d01037a0 2024-03-23T12:28:21.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.407+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d00fe550 con 0x7fa1d01037a0 2024-03-23T12:28:21.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.407+0000 7fa1c7e00700 1 -- 172.21.15.63:0/161023542 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fa1bc0096d0 con 0x7fa1d01037a0 2024-03-23T12:28:21.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.409+0000 7fa1c7e00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa1b406c7d0 0x7fa1b406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:21.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.409+0000 7fa1c7e00700 1 -- 172.21.15.63:0/161023542 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa1bc08b540 con 0x7fa1d01037a0 2024-03-23T12:28:21.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.409+0000 7fa1cfe00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa1b406c7d0 0x7fa1b406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:21.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.410+0000 7fa1cfe00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa1b406c7d0 0x7fa1b406ec80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00136f0 tx=0x7fa1c0009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:21.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.413+0000 7fa1c7e00700 1 -- 172.21.15.63:0/161023542 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa1bc059280 con 0x7fa1d01037a0 2024-03-23T12:28:21.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:21 smithi111 ceph-mon[106899]: pgmap v1117: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:21.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.644+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1d0113710 con 0x7fa1b406c7d0 2024-03-23T12:28:21.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.646+0000 7fa1c7e00700 1 -- 172.21.15.63:0/161023542 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa1d0113710 con 0x7fa1b406c7d0 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa1b406c7d0 msgr2=0x7fa1b406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa1b406c7d0 0x7fa1b406ec80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00136f0 tx=0x7fa1c0009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d01037a0 msgr2=0x7fa1d0113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d01037a0 0x7fa1d0113f80 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00eb40 tx=0x7fa1bc00b040 comp rx=0 tx=0).stop 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 shutdown_connections 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fa1b406c7d0 0x7fa1b406ec80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1d0102e00 0x7fa1d0113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 --2- 172.21.15.63:0/161023542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1d01037a0 0x7fa1d0113f80 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 >> 172.21.15.63:0/161023542 conn(0x7fa1d00f89b0 msgr2=0x7fa1d00f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:21.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 shutdown_connections 2024-03-23T12:28:21.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.648+0000 7fa1d6c00700 1 -- 172.21.15.63:0/161023542 wait complete. 2024-03-23T12:28:21.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.786+0000 7f5928400700 1 -- 172.21.15.63:0/1889959175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 msgr2=0x7f59201031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.786+0000 7f5928400700 1 --2- 172.21.15.63:0/1889959175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f59201031d0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5914004940 tx=0x7f5914009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:21.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.786+0000 7f5928400700 1 -- 172.21.15.63:0/1889959175 shutdown_connections 2024-03-23T12:28:21.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.786+0000 7f5928400700 1 --2- 172.21.15.63:0/1889959175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59201037a0 0x7f59200fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.786+0000 7f5928400700 1 --2- 172.21.15.63:0/1889959175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f59201031d0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.786+0000 7f5928400700 1 -- 172.21.15.63:0/1889959175 >> 172.21.15.63:0/1889959175 conn(0x7f59200f89b0 msgr2=0x7f59200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:21.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.786+0000 7f5928400700 1 -- 172.21.15.63:0/1889959175 shutdown_connections 2024-03-23T12:28:21.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.787+0000 7f5928400700 1 -- 172.21.15.63:0/1889959175 wait complete. 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.787+0000 7f5928400700 1 Processor -- start 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.787+0000 7f5928400700 1 -- start start 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5928400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f5920115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5928400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59201037a0 0x7f5920116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5928400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920116880 con 0x7f5920102e00 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5928400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592010fc50 con 0x7f59201037a0 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5925a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f5920115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5925a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f5920115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56022/0 (socket says 172.21.15.63:56022) 2024-03-23T12:28:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5925a00700 1 -- 172.21.15.63:0/1681178364 learned_addr learned my addr 172.21.15.63:0/1681178364 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:21.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5925a00700 1 -- 172.21.15.63:0/1681178364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59201037a0 msgr2=0x7f5920116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.788+0000 7f5925a00700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59201037a0 0x7f5920116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.789+0000 7f5925a00700 1 -- 172.21.15.63:0/1681178364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914005040 con 0x7f5920102e00 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.789+0000 7f5925a00700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f5920115cf0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5914009360 tx=0x7f5914009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.789+0000 7f591d600700 1 -- 172.21.15.63:0/1681178364 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591400e040 con 0x7f5920102e00 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.789+0000 7f591d600700 1 -- 172.21.15.63:0/1681178364 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5914013420 con 0x7f5920102e00 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.789+0000 7f591d600700 1 -- 172.21.15.63:0/1681178364 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591401c5f0 con 0x7f5920102e00 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.789+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592010fed0 con 0x7f5920102e00 2024-03-23T12:28:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.789+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5920110490 con 0x7f5920102e00 2024-03-23T12:28:21.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.790+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59200fe550 con 0x7f5920102e00 2024-03-23T12:28:21.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.791+0000 7f591d600700 1 -- 172.21.15.63:0/1681178364 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5914013590 con 0x7f5920102e00 2024-03-23T12:28:21.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.792+0000 7f591d600700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f590806c7d0 0x7f590806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:21.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.792+0000 7f591d600700 1 -- 172.21.15.63:0/1681178364 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f591408cc80 con 0x7f5920102e00 2024-03-23T12:28:21.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.793+0000 7f5925000700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f590806c7d0 0x7f590806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:21.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.793+0000 7f5925000700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f590806c7d0 0x7f590806ec80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5920111590 tx=0x7f590c007cc0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:21.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:21.796+0000 7f591d600700 1 -- 172.21.15.63:0/1681178364 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f591405a990 con 0x7f5920102e00 2024-03-23T12:28:21.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.019+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5920109e30 con 0x7f590806c7d0 2024-03-23T12:28:22.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.029+0000 7f591d600700 1 -- 172.21.15.63:0/1681178364 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f5920109e30 con 0x7f590806c7d0 2024-03-23T12:28:22.006 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:28:22.006 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (37m) 14m ago 50m 14.7M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:28:22.006 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (34m) 14m ago 51m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:28:22.006 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (34m) 14m ago 49m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (37m) 14m ago 50m 74.5M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (30m) 14m ago 44m 16.6M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (30m) 14m ago 44m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (14m) 14m ago 44m 10.8M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (30m) 14m ago 44m 16.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (40m) 14m ago 52m 509M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (38m) 14m ago 49m 458M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (36m) 14m ago 52m 72.1M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (35m) 14m ago 49m 55.6M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (38m) 14m ago 50m 8052k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (38m) 14m ago 49m 8094k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (34m) 14m ago 48m 95.6M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (33m) 14m ago 48m 97.8M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (32m) 14m ago 47m 85.2M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:28:22.007 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (32m) 14m ago 47m 101M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:28:22.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (31m) 14m ago 46m 93.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:28:22.008 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (31m) 14m ago 46m 88.8M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:28:22.008 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (38m) 14m ago 50m 61.9M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:28:22.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f590806c7d0 msgr2=0x7f590806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f590806c7d0 0x7f590806ec80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5920111590 tx=0x7f590c007cc0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 msgr2=0x7f5920115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f5920115cf0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5914009360 tx=0x7f5914009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 shutdown_connections 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f590806c7d0 0x7f590806ec80 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920102e00 0x7f5920115cf0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 --2- 172.21.15.63:0/1681178364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59201037a0 0x7f5920116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.031+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 >> 172.21.15.63:0/1681178364 conn(0x7f59200f89b0 msgr2=0x7f59200706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.032+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 shutdown_connections 2024-03-23T12:28:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.032+0000 7f5928400700 1 -- 172.21.15.63:0/1681178364 wait complete. 2024-03-23T12:28:22.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 -- 172.21.15.63:0/1846513142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d01037a0 msgr2=0x7f38d00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 --2- 172.21.15.63:0/1846513142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d01037a0 0x7f38d00fce10 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f38bc004940 tx=0x7f38bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:22.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 -- 172.21.15.63:0/1846513142 shutdown_connections 2024-03-23T12:28:22.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 --2- 172.21.15.63:0/1846513142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d01037a0 0x7f38d00fce10 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 --2- 172.21.15.63:0/1846513142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38d0102e00 0x7f38d01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 -- 172.21.15.63:0/1846513142 >> 172.21.15.63:0/1846513142 conn(0x7f38d00f89b0 msgr2=0x7f38d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:22.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 -- 172.21.15.63:0/1846513142 shutdown_connections 2024-03-23T12:28:22.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.168+0000 7f38d7600700 1 -- 172.21.15.63:0/1846513142 wait complete. 2024-03-23T12:28:22.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.169+0000 7f38d7600700 1 Processor -- start 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.169+0000 7f38d7600700 1 -- start start 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.169+0000 7f38d7600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d0102e00 0x7f38d0115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.169+0000 7f38d7600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38d01037a0 0x7f38d0115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.169+0000 7f38d7600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0116630 con 0x7f38d0102e00 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.169+0000 7f38d7600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d010fc10 con 0x7f38d01037a0 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38d4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d0102e00 0x7f38d0115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38d4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d0102e00 0x7f38d0115aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56036/0 (socket says 172.21.15.63:56036) 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38d4c00700 1 -- 172.21.15.63:0/615500169 learned_addr learned my addr 172.21.15.63:0/615500169 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38cfe00700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38d01037a0 0x7f38d0115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38d4c00700 1 -- 172.21.15.63:0/615500169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38d01037a0 msgr2=0x7f38d0115fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38d4c00700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38d01037a0 0x7f38d0115fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38d4c00700 1 -- 172.21.15.63:0/615500169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc005040 con 0x7f38d0102e00 2024-03-23T12:28:22.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38cfe00700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38d01037a0 0x7f38d0115fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:28:22.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38d4c00700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d0102e00 0x7f38d0115aa0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f38c4002a50 tx=0x7f38c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:22.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.170+0000 7f38ccc00700 1 -- 172.21.15.63:0/615500169 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c4007cc0 con 0x7f38d0102e00 2024-03-23T12:28:22.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.171+0000 7f38ccc00700 1 -- 172.21.15.63:0/615500169 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38c400f040 con 0x7f38d0102e00 2024-03-23T12:28:22.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.171+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d010fef0 con 0x7f38d0102e00 2024-03-23T12:28:22.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.171+0000 7f38ccc00700 1 -- 172.21.15.63:0/615500169 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c4009540 con 0x7f38d0102e00 2024-03-23T12:28:22.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.171+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d01104b0 con 0x7f38d0102e00 2024-03-23T12:28:22.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.172+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d00fe550 con 0x7f38d0102e00 2024-03-23T12:28:22.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.173+0000 7f38ccc00700 1 -- 172.21.15.63:0/615500169 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f38c40096a0 con 0x7f38d0102e00 2024-03-23T12:28:22.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.174+0000 7f38ccc00700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f38b806c7c0 0x7f38b806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:22.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.174+0000 7f38ccc00700 1 -- 172.21.15.63:0/615500169 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f38c408b820 con 0x7f38d0102e00 2024-03-23T12:28:22.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.174+0000 7f38cfe00700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f38b806c7c0 0x7f38b806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:22.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.175+0000 7f38cfe00700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f38b806c7c0 0x7f38b806ec70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f38bc000c00 tx=0x7f38bc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:22.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.178+0000 7f38ccc00700 1 -- 172.21.15.63:0/615500169 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f38c40594b0 con 0x7f38d0102e00 2024-03-23T12:28:22.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.477+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f38d01110f0 con 0x7f38d0102e00 2024-03-23T12:28:22.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.477+0000 7f38ccc00700 1 -- 172.21.15.63:0/615500169 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f38c4059040 con 0x7f38d0102e00 2024-03-23T12:28:22.454 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:28:22.454 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:28:22.455 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:28:22.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f38b806c7c0 msgr2=0x7f38b806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f38b806c7c0 0x7f38b806ec70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f38bc000c00 tx=0x7f38bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d0102e00 msgr2=0x7f38d0115aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d0102e00 0x7f38d0115aa0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f38c4002a50 tx=0x7f38c40046b0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 shutdown_connections 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f38b806c7c0 0x7f38b806ec70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38d0102e00 0x7f38d0115aa0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 --2- 172.21.15.63:0/615500169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38d01037a0 0x7f38d0115fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 >> 172.21.15.63:0/615500169 conn(0x7f38d00f89b0 msgr2=0x7f38d00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 shutdown_connections 2024-03-23T12:28:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.480+0000 7f38d7600700 1 -- 172.21.15.63:0/615500169 wait complete. 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.616+0000 7f4906000700 1 -- 172.21.15.63:0/3192798146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49001037a0 msgr2=0x7f49000fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.616+0000 7f4906000700 1 --2- 172.21.15.63:0/3192798146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49001037a0 0x7f49000fce10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f48ec004940 tx=0x7f48ec009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.617+0000 7f4906000700 1 -- 172.21.15.63:0/3192798146 shutdown_connections 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.617+0000 7f4906000700 1 --2- 172.21.15.63:0/3192798146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49001037a0 0x7f49000fce10 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.617+0000 7f4906000700 1 --2- 172.21.15.63:0/3192798146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4900102e00 0x7f49001031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.617+0000 7f4906000700 1 -- 172.21.15.63:0/3192798146 >> 172.21.15.63:0/3192798146 conn(0x7f49000f89b0 msgr2=0x7f49000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.617+0000 7f4906000700 1 -- 172.21.15.63:0/3192798146 shutdown_connections 2024-03-23T12:28:22.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.617+0000 7f4906000700 1 -- 172.21.15.63:0/3192798146 wait complete. 2024-03-23T12:28:22.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.617+0000 7f4906000700 1 Processor -- start 2024-03-23T12:28:22.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f4906000700 1 -- start start 2024-03-23T12:28:22.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f4906000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900102e00 0x7f490010e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:22.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f4906000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49001037a0 0x7f490010b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:22.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f4906000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010ef30 con 0x7f4900102e00 2024-03-23T12:28:22.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f4906000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010f070 con 0x7f49001037a0 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f48ff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900102e00 0x7f490010e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f48ff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900102e00 0x7f490010e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56056/0 (socket says 172.21.15.63:56056) 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f48ff400700 1 -- 172.21.15.63:0/3318332057 learned_addr learned my addr 172.21.15.63:0/3318332057 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.618+0000 7f48fea00700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49001037a0 0x7f490010b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f48ff400700 1 -- 172.21.15.63:0/3318332057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49001037a0 msgr2=0x7f490010b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f48ff400700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49001037a0 0x7f490010b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f48ff400700 1 -- 172.21.15.63:0/3318332057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec005040 con 0x7f4900102e00 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f48ff400700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900102e00 0x7f490010e8e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f48f0002a50 tx=0x7f48f00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f48f7400700 1 -- 172.21.15.63:0/3318332057 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48f0007cc0 con 0x7f4900102e00 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f48f7400700 1 -- 172.21.15.63:0/3318332057 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48f000f040 con 0x7f4900102e00 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490010bf20 con 0x7f4900102e00 2024-03-23T12:28:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f48f7400700 1 -- 172.21.15.63:0/3318332057 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48f000d9c0 con 0x7f4900102e00 2024-03-23T12:28:22.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.619+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490010c4b0 con 0x7f4900102e00 2024-03-23T12:28:22.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.621+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490010c720 con 0x7f4900102e00 2024-03-23T12:28:22.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.621+0000 7f48f7400700 1 -- 172.21.15.63:0/3318332057 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f48f000db20 con 0x7f4900102e00 2024-03-23T12:28:22.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.622+0000 7f48f7400700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f48e406c7c0 0x7f48e406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:22.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.623+0000 7f48f7400700 1 -- 172.21.15.63:0/3318332057 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f48f008b760 con 0x7f4900102e00 2024-03-23T12:28:22.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.623+0000 7f48fea00700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f48e406c7c0 0x7f48e406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:22.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.623+0000 7f48fea00700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f48e406c7c0 0x7f48e406ec70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f490010d1b0 tx=0x7f48ec009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:22.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.627+0000 7f48f7400700 1 -- 172.21.15.63:0/3318332057 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f48f0055960 con 0x7f4900102e00 2024-03-23T12:28:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.883+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f490004bed0 con 0x7f4900102e00 2024-03-23T12:28:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.883+0000 7f48f7400700 1 -- 172.21.15.63:0/3318332057 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1837 (secure 0 0 0) 0x7f48f0058f80 con 0x7f4900102e00 2024-03-23T12:28:22.860 INFO:teuthology.orchestra.run.smithi063.stdout:e35 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 26 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-23T12:28:22.861 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T11:57:27.441639+0000 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:28:22.862 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:up {} 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:failed 0,1 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:28:22.863 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{-1:34422} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:28:22.864 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{-1:34454} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:28:22.865 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{-1:44393} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:28:22.865 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{-1:45001} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f48e406c7c0 msgr2=0x7f48e406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f48e406c7c0 0x7f48e406ec70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f490010d1b0 tx=0x7f48ec009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900102e00 msgr2=0x7f490010e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900102e00 0x7f490010e8e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f48f0002a50 tx=0x7f48f00046b0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 shutdown_connections 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f48e406c7c0 0x7f48e406ec70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900102e00 0x7f490010e8e0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 --2- 172.21.15.63:0/3318332057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49001037a0 0x7f490010b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 >> 172.21.15.63:0/3318332057 conn(0x7f49000f89b0 msgr2=0x7f49000f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 shutdown_connections 2024-03-23T12:28:22.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:22.886+0000 7f4906000700 1 -- 172.21.15.63:0/3318332057 wait complete. 2024-03-23T12:28:22.867 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.023+0000 7f28fde00700 1 -- 172.21.15.63:0/1898830250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 msgr2=0x7f28f8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.023+0000 7f28fde00700 1 --2- 172.21.15.63:0/1898830250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f8106f00 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f28e4004940 tx=0x7f28e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.024+0000 7f28fde00700 1 -- 172.21.15.63:0/1898830250 shutdown_connections 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.024+0000 7f28fde00700 1 --2- 172.21.15.63:0/1898830250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f8106f00 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.024+0000 7f28fde00700 1 --2- 172.21.15.63:0/1898830250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f80fde90 0x7f28f80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.024+0000 7f28fde00700 1 -- 172.21.15.63:0/1898830250 >> 172.21.15.63:0/1898830250 conn(0x7f28f8072bc0 msgr2=0x7f28f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.024+0000 7f28fde00700 1 -- 172.21.15.63:0/1898830250 shutdown_connections 2024-03-23T12:28:23.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.024+0000 7f28fde00700 1 -- 172.21.15.63:0/1898830250 wait complete. 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.025+0000 7f28fde00700 1 Processor -- start 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.025+0000 7f28fde00700 1 -- start start 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.025+0000 7f28fde00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f80fde90 0x7f28f80fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.025+0000 7f28fde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f80fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.025+0000 7f28fde00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f80fc820 con 0x7f28f80fe830 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.025+0000 7f28fde00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f80fc960 con 0x7f28f80fde90 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.025+0000 7f28f6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f80fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:23.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28f6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f80fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56076/0 (socket says 172.21.15.63:56076) 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28f6000700 1 -- 172.21.15.63:0/1744301118 learned_addr learned my addr 172.21.15.63:0/1744301118 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28f6a00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f80fde90 0x7f28f80fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28f6000700 1 -- 172.21.15.63:0/1744301118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f80fde90 msgr2=0x7f28f80fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28f6000700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f80fde90 0x7f28f80fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28f6000700 1 -- 172.21.15.63:0/1744301118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4005040 con 0x7f28f80fe830 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28f6000700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f80fc290 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f28e4000c00 tx=0x7f28e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28eea00700 1 -- 172.21.15.63:0/1744301118 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28e4021070 con 0x7f28f80fe830 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28eea00700 1 -- 172.21.15.63:0/1744301118 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28e4013e00 con 0x7f28f80fe830 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28eea00700 1 -- 172.21.15.63:0/1744301118 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28e400e070 con 0x7f28f80fe830 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.026+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f80fcbb0 con 0x7f28f80fe830 2024-03-23T12:28:23.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.027+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f811a0e0 con 0x7f28f80fe830 2024-03-23T12:28:23.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.028+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f8065e30 con 0x7f28f80fe830 2024-03-23T12:28:23.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.029+0000 7f28eea00700 1 -- 172.21.15.63:0/1744301118 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f28e4013950 con 0x7f28f80fe830 2024-03-23T12:28:23.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.029+0000 7f28eea00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28dc06c7d0 0x7f28dc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:23.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.030+0000 7f28eea00700 1 -- 172.21.15.63:0/1744301118 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f28e408cac0 con 0x7f28f80fe830 2024-03-23T12:28:23.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.030+0000 7f28f6a00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28dc06c7d0 0x7f28dc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:23.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.030+0000 7f28f6a00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28dc06c7d0 0x7f28dc06ec80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009b80 tx=0x7f28e800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:23.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.034+0000 7f28eea00700 1 -- 172.21.15.63:0/1744301118 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f28e4056cc0 con 0x7f28f80fe830 2024-03-23T12:28:23.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.269+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28f8065e30 con 0x7f28dc06c7d0 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.270+0000 7f28eea00700 1 -- 172.21.15.63:0/1744301118 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f28f8065e30 con 0x7f28dc06c7d0 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313", 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": true, 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [ 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "mon", 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "mds", 2024-03-23T12:28:23.247 INFO:teuthology.orchestra.run.smithi063.stdout: "osd", 2024-03-23T12:28:23.248 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr" 2024-03-23T12:28:23.248 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-03-23T12:28:23.248 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": "16/21 daemons upgraded", 2024-03-23T12:28:23.248 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "Currently upgrading mds daemons", 2024-03-23T12:28:23.248 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:28:23.248 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:28:23.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.272+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28dc06c7d0 msgr2=0x7f28dc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.272+0000 7f28fde00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28dc06c7d0 0x7f28dc06ec80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009b80 tx=0x7f28e800b040 comp rx=0 tx=0).stop 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.272+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 msgr2=0x7f28f80fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.272+0000 7f28fde00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f80fc290 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f28e4000c00 tx=0x7f28e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.272+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 shutdown_connections 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.272+0000 7f28fde00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f28dc06c7d0 0x7f28dc06ec80 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.272+0000 7f28fde00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f80fde90 0x7f28f80fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.273+0000 7f28fde00700 1 --2- 172.21.15.63:0/1744301118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f80fe830 0x7f28f80fc290 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.273+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 >> 172.21.15.63:0/1744301118 conn(0x7f28f8072bc0 msgr2=0x7f28f80ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.273+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 shutdown_connections 2024-03-23T12:28:23.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.273+0000 7f28fde00700 1 -- 172.21.15.63:0/1744301118 wait complete. 2024-03-23T12:28:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.409+0000 7f9ca2600700 1 -- 172.21.15.63:0/1089177746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 msgr2=0x7f9c9c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.409+0000 7f9ca2600700 1 --2- 172.21.15.63:0/1089177746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c0fe260 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c004940 tx=0x7f9c8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.410+0000 7f9ca2600700 1 -- 172.21.15.63:0/1089177746 shutdown_connections 2024-03-23T12:28:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.410+0000 7f9ca2600700 1 --2- 172.21.15.63:0/1089177746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0fe830 0x7f9c9c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.410+0000 7f9ca2600700 1 --2- 172.21.15.63:0/1089177746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c0fe260 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.410+0000 7f9ca2600700 1 -- 172.21.15.63:0/1089177746 >> 172.21.15.63:0/1089177746 conn(0x7f9c9c072bc0 msgr2=0x7f9c9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.410+0000 7f9ca2600700 1 -- 172.21.15.63:0/1089177746 shutdown_connections 2024-03-23T12:28:23.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.410+0000 7f9ca2600700 1 -- 172.21.15.63:0/1089177746 wait complete. 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.410+0000 7f9ca2600700 1 Processor -- start 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9ca2600700 1 -- start start 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9ca2600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9ca2600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0fe830 0x7f9c9c116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9ca2600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1168b0 con 0x7f9c9c0fde90 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9ca2600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c10fc80 con 0x7f9c9c0fe830 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9c9b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:23.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9c9b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56082/0 (socket says 172.21.15.63:56082) 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.411+0000 7f9c9b400700 1 -- 172.21.15.63:0/4114500189 learned_addr learned my addr 172.21.15.63:0/4114500189 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c9aa00700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0fe830 0x7f9c9c116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c9b400700 1 -- 172.21.15.63:0/4114500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0fe830 msgr2=0x7f9c9c116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c9b400700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0fe830 0x7f9c9c116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c9b400700 1 -- 172.21.15.63:0/4114500189 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c005040 con 0x7f9c9c0fde90 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c9b400700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c115d20 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c93400700 1 -- 172.21.15.63:0/4114500189 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c8c00e040 con 0x7f9c9c0fde90 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c93400700 1 -- 172.21.15.63:0/4114500189 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c8c013420 con 0x7f9c9c0fde90 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c10ff00 con 0x7f9c9c0fde90 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9c93400700 1 -- 172.21.15.63:0/4114500189 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c8c01c630 con 0x7f9c9c0fde90 2024-03-23T12:28:23.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.412+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c110440 con 0x7f9c9c0fde90 2024-03-23T12:28:23.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.414+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c106a20 con 0x7f9c9c0fde90 2024-03-23T12:28:23.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.415+0000 7f9c93400700 1 -- 172.21.15.63:0/4114500189 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f9c8c013970 con 0x7f9c9c0fde90 2024-03-23T12:28:23.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.416+0000 7f9c93400700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c8006c7d0 0x7f9c8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:23.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.416+0000 7f9c93400700 1 -- 172.21.15.63:0/4114500189 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c8c08cc70 con 0x7f9c9c0fde90 2024-03-23T12:28:23.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.416+0000 7f9c9aa00700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c8006c7d0 0x7f9c8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:23.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.417+0000 7f9c9aa00700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c8006c7d0 0x7f9c8006ec80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9c88007fd0 tx=0x7f9c88009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:23.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:23 smithi111 ceph-mon[106899]: pgmap v1118: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:23.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:23 smithi111 ceph-mon[106899]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:23.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:23 smithi111 ceph-mon[106899]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:23.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:23 smithi111 ceph-mon[106899]: from='client.36014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:23.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:23 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/615500169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:23.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:23 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3318332057' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:28:23.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.420+0000 7f9c93400700 1 -- 172.21.15.63:0/4114500189 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c8c093080 con 0x7f9c9c0fde90 2024-03-23T12:28:23.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:23 smithi063 ceph-mon[141124]: pgmap v1118: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:23.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:23 smithi063 ceph-mon[141124]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:23.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:23 smithi063 ceph-mon[141124]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:23.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:23 smithi063 ceph-mon[141124]: from='client.36014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:23.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:23 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/615500169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:23.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:23 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3318332057' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:28:23.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.720+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9c9c04bed0 con 0x7f9c9c0fde90 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.721+0000 7f9c93400700 1 -- 172.21.15.63:0/4114500189 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f9c8c05a980 con 0x7f9c9c0fde90 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout: host smithi111 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi111/config 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout:Traceback (most recent call last): 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout: "__main__", mod_spec) 2024-03-23T12:28:23.698 INFO:teuthology.orchestra.run.smithi063.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: exec(code, run_globals) 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10883, in 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 10871, in main 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 7142, in command_ceph_volume 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: File "/tmp/tmp5ecejt8s.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/run/ceph:z -v /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpk1u_2oqv:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b5fd563b95f35cd9eaf13910d3044ea4e92cbeb456d25c723e9572c4e4910313 inventory --format=json-pretty --filter-for-batch 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-23T12:28:23.699 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is degraded 2024-03-23T12:28:23.700 INFO:teuthology.orchestra.run.smithi063.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-23T12:28:23.700 INFO:teuthology.orchestra.run.smithi063.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-23T12:28:23.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c8006c7d0 msgr2=0x7f9c8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c8006c7d0 0x7f9c8006ec80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9c88007fd0 tx=0x7f9c88009040 comp rx=0 tx=0).stop 2024-03-23T12:28:23.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 msgr2=0x7f9c9c115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c115d20 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 shutdown_connections 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f9c8006c7d0 0x7f9c8006ec80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c9c0fde90 0x7f9c9c115d20 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 --2- 172.21.15.63:0/4114500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0fe830 0x7f9c9c116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 >> 172.21.15.63:0/4114500189 conn(0x7f9c9c072bc0 msgr2=0x7f9c9c0fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 shutdown_connections 2024-03-23T12:28:23.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:23.723+0000 7f9ca2600700 1 -- 172.21.15.63:0/4114500189 wait complete. 2024-03-23T12:28:24.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:24 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/4114500189' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:28:24.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:24 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/4114500189' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:28:25.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:25 smithi111 ceph-mon[106899]: pgmap v1119: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:25.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:25 smithi111 ceph-mon[106899]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:25.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:25 smithi063 ceph-mon[141124]: pgmap v1119: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:25.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:25 smithi063 ceph-mon[141124]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:26.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:26.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:27.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:27 smithi111 ceph-mon[106899]: pgmap v1120: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:27.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:27 smithi063 ceph-mon[141124]: pgmap v1120: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:29.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:29 smithi063 ceph-mon[141124]: pgmap v1121: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:29.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:29 smithi111 ceph-mon[106899]: pgmap v1121: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: pgmap v1122: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.brtzgy"}]': finished 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.mdbdrz"}]': finished 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.atkibb"}]': finished 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.ylnmzy"}]': finished 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: Standby daemon mds.cephfs.smithi063.brtzgy assigned to filesystem cephfs as rank 0 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: Standby daemon mds.cephfs.smithi063.mdbdrz assigned to filesystem cephfs as rank 1 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-23T12:28:31.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-23T12:28:31.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:replay,1=cephfs.smithi063.mdbdrz=up:replay} 2 up:standby 2024-03-23T12:28:31.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:31 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-03-23T12:28:31.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: pgmap v1122: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:31.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.brtzgy"}]: dispatch 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.brtzgy"}]': finished 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.mdbdrz"}]: dispatch 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi063.mdbdrz"}]': finished 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.atkibb"}]: dispatch 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.atkibb"}]': finished 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.ylnmzy"}]: dispatch 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.ylnmzy"}]': finished 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: Standby daemon mds.cephfs.smithi063.brtzgy assigned to filesystem cephfs as rank 0 2024-03-23T12:28:31.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: Standby daemon mds.cephfs.smithi063.mdbdrz assigned to filesystem cephfs as rank 1 2024-03-23T12:28:31.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-23T12:28:31.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-23T12:28:31.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:replay,1=cephfs.smithi063.mdbdrz=up:replay} 2 up:standby 2024-03-23T12:28:31.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:31 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: pgmap v1123: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: pgmap v1124: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all mds 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: Upgrade: Setting filesystem cephfs Joinable 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:replay,1=cephfs.smithi063.mdbdrz=up:replay} 2 up:standby 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-23T12:28:32.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-23T12:28:32.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-23T12:28:32.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-23T12:28:32.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-23T12:28:32.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-23T12:28:32.630 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-23T12:28:32.631 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:32 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:28:32.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: pgmap v1123: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:32.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: pgmap v1124: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-03-23T12:28:32.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all mds 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: Upgrade: Setting filesystem cephfs Joinable 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:replay,1=cephfs.smithi063.mdbdrz=up:replay} 2 up:standby 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-23T12:28:32.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-23T12:28:32.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-23T12:28:32.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-23T12:28:32.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:32 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all rgw 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all rbd-mirror 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all ceph-exporter 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all iscsi 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all nfs 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Setting container_image for all nvmeof 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Finalizing container_image settings 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: Upgrade: Complete! 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:33.468 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:33 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all rgw 2024-03-23T12:28:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all rbd-mirror 2024-03-23T12:28:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all ceph-exporter 2024-03-23T12:28:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all iscsi 2024-03-23T12:28:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all nfs 2024-03-23T12:28:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Setting container_image for all nvmeof 2024-03-23T12:28:33.628 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Finalizing container_image settings 2024-03-23T12:28:33.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: Upgrade: Complete! 2024-03-23T12:28:33.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:33.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:33.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:33.629 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:33 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:34.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:34 smithi063 ceph-mon[141124]: pgmap v1125: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 0 op/s 2024-03-23T12:28:34.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:34 smithi111 ceph-mon[106899]: pgmap v1125: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 0 op/s 2024-03-23T12:28:36.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:36 smithi063 ceph-mon[141124]: pgmap v1126: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 7 op/s 2024-03-23T12:28:36.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:36 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:36.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:36 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:36.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:36 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:36.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:36 smithi111 ceph-mon[106899]: pgmap v1126: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 7 op/s 2024-03-23T12:28:36.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:36 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:36.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:36 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:36.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:36 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:38.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:38 smithi063 ceph-mon[141124]: pgmap v1127: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-03-23T12:28:38.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:38 smithi111 ceph-mon[106899]: pgmap v1127: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-03-23T12:28:39.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:39 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:resolve 2024-03-23T12:28:39.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:39 smithi063 ceph-mon[141124]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:replay,1=cephfs.smithi063.mdbdrz=up:resolve} 2 up:standby 2024-03-23T12:28:39.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:39 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:resolve 2024-03-23T12:28:39.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:39 smithi111 ceph-mon[106899]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:replay,1=cephfs.smithi063.mdbdrz=up:resolve} 2 up:standby 2024-03-23T12:28:40.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:40 smithi063 ceph-mon[141124]: pgmap v1128: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8 op/s 2024-03-23T12:28:40.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:40 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:resolve 2024-03-23T12:28:40.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:40 smithi063 ceph-mon[141124]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:resolve,1=cephfs.smithi063.mdbdrz=up:resolve} 2 up:standby 2024-03-23T12:28:40.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:40 smithi111 ceph-mon[106899]: pgmap v1128: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8 op/s 2024-03-23T12:28:40.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:40 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:resolve 2024-03-23T12:28:40.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:40 smithi111 ceph-mon[106899]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:resolve,1=cephfs.smithi063.mdbdrz=up:resolve} 2 up:standby 2024-03-23T12:28:41.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: reconnect by client.24341 192.168.0.1:0/2514609317 after 0.00199998 2024-03-23T12:28:41.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: reconnect by client.14532 192.168.0.1:0/2227074864 after 0.00199998 2024-03-23T12:28:41.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: reconnect by client.14532 192.168.0.1:0/2227074864 after 0.00199998 2024-03-23T12:28:41.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: reconnect by client.24341 192.168.0.1:0/2514609317 after 0.00199998 2024-03-23T12:28:41.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:reconnect 2024-03-23T12:28:41.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:reconnect 2024-03-23T12:28:41.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:reconnect,1=cephfs.smithi063.mdbdrz=up:reconnect} 2 up:standby 2024-03-23T12:28:41.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: pgmap v1129: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 9 op/s 2024-03-23T12:28:41.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:41.627 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: reconnect by client.24341 192.168.0.1:0/2514609317 after 0.00199998 2024-03-23T12:28:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: reconnect by client.14532 192.168.0.1:0/2227074864 after 0.00199998 2024-03-23T12:28:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: reconnect by client.14532 192.168.0.1:0/2227074864 after 0.00199998 2024-03-23T12:28:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: reconnect by client.24341 192.168.0.1:0/2514609317 after 0.00199998 2024-03-23T12:28:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:reconnect 2024-03-23T12:28:41.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:reconnect 2024-03-23T12:28:41.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:reconnect,1=cephfs.smithi063.mdbdrz=up:reconnect} 2 up:standby 2024-03-23T12:28:41.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: pgmap v1129: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 9 op/s 2024-03-23T12:28:41.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:41.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:42.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:42 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:rejoin 2024-03-23T12:28:42.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:42 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:rejoin 2024-03-23T12:28:42.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:42 smithi063 ceph-mon[141124]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:rejoin,1=cephfs.smithi063.mdbdrz=up:rejoin} 2 up:standby 2024-03-23T12:28:42.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:42 smithi063 ceph-mon[141124]: daemon mds.cephfs.smithi063.mdbdrz is now active in filesystem cephfs as rank 1 2024-03-23T12:28:42.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:42 smithi063 ceph-mon[141124]: daemon mds.cephfs.smithi063.brtzgy is now active in filesystem cephfs as rank 0 2024-03-23T12:28:42.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:42 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:rejoin 2024-03-23T12:28:42.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:42 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:rejoin 2024-03-23T12:28:42.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:42 smithi111 ceph-mon[106899]: fsmap cephfs:2/2 {0=cephfs.smithi063.brtzgy=up:rejoin,1=cephfs.smithi063.mdbdrz=up:rejoin} 2 up:standby 2024-03-23T12:28:42.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:42 smithi111 ceph-mon[106899]: daemon mds.cephfs.smithi063.mdbdrz is now active in filesystem cephfs as rank 1 2024-03-23T12:28:42.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:42 smithi111 ceph-mon[106899]: daemon mds.cephfs.smithi063.brtzgy is now active in filesystem cephfs as rank 0 2024-03-23T12:28:43.621 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:43 smithi111 ceph-mon[106899]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-23T12:28:43.621 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:43 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:active 2024-03-23T12:28:43.621 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:43 smithi111 ceph-mon[106899]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:active 2024-03-23T12:28:43.621 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:43 smithi111 ceph-mon[106899]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby 2024-03-23T12:28:43.621 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:43 smithi111 ceph-mon[106899]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T12:28:43.621 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:43 smithi111 ceph-mon[106899]: pgmap v1130: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 85 B/s wr, 7 op/s 2024-03-23T12:28:43.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:43 smithi063 ceph-mon[141124]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-23T12:28:43.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:43 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] up:active 2024-03-23T12:28:43.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:43 smithi063 ceph-mon[141124]: mds.? [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] up:active 2024-03-23T12:28:43.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:43 smithi063 ceph-mon[141124]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby 2024-03-23T12:28:43.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:43 smithi063 ceph-mon[141124]: fsmap cephfs:2 {0=cephfs.smithi063.brtzgy=up:active,1=cephfs.smithi063.mdbdrz=up:active} 2 up:standby-replay 2024-03-23T12:28:43.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:43 smithi063 ceph-mon[141124]: pgmap v1130: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 85 B/s wr, 7 op/s 2024-03-23T12:28:45.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:44 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:45.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:44 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:45.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:44 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:45.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:44 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:46.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:46 smithi063 ceph-mon[141124]: pgmap v1131: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-03-23T12:28:46.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:46 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:46.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:46 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:46.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:46 smithi111 ceph-mon[106899]: pgmap v1131: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-03-23T12:28:46.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:46 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:46.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:46 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:47.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:47 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:47.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:47 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:47.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:47 smithi063 ceph-mon[141124]: pgmap v1132: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-03-23T12:28:47.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:47 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:47.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:47 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:47.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:47 smithi111 ceph-mon[106899]: pgmap v1132: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-03-23T12:28:49.876 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:49 smithi063 ceph-mon[141124]: pgmap v1133: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 8.3 KiB/s wr, 16 op/s 2024-03-23T12:28:49.896 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:49 smithi111 ceph-mon[106899]: pgmap v1133: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 8.3 KiB/s wr, 16 op/s 2024-03-23T12:28:52.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: pgmap v1134: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-03-23T12:28:52.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:52.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:52.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:28:52.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:28:52.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: pgmap v1135: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-23T12:28:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:28:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:28:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:28:52.127 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:52.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: pgmap v1134: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-03-23T12:28:52.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:52.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: pgmap v1135: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:28:52.147 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:52 smithi063 ceph-mon[141124]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-23T12:28:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:52 smithi063 ceph-mon[141124]: Cluster is now healthy 2024-03-23T12:28:53.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:52 smithi111 ceph-mon[106899]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-23T12:28:53.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:52 smithi111 ceph-mon[106899]: Cluster is now healthy 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 -- 172.21.15.63:0/2466702624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc1037a0 msgr2=0x7fcfdc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/2466702624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc1037a0 0x7fcfdc0fce10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4004940 tx=0x7fcfc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 -- 172.21.15.63:0/2466702624 shutdown_connections 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/2466702624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc1037a0 0x7fcfdc0fce10 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/2466702624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfdc102e00 0x7fcfdc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 -- 172.21.15.63:0/2466702624 >> 172.21.15.63:0/2466702624 conn(0x7fcfdc0f89b0 msgr2=0x7fcfdc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 -- 172.21.15.63:0/2466702624 shutdown_connections 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.878+0000 7fcfe2e00700 1 -- 172.21.15.63:0/2466702624 wait complete. 2024-03-23T12:28:53.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.879+0000 7fcfe2e00700 1 Processor -- start 2024-03-23T12:28:53.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.879+0000 7fcfe2e00700 1 -- start start 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfe2e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc102e00 0x7fcfdc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfe2e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfdc1037a0 0x7fcfdc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfe2e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc070d90 con 0x7fcfdc102e00 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfe2e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc070ed0 con 0x7fcfdc1037a0 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfdbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc102e00 0x7fcfdc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfdbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc102e00 0x7fcfdc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55704/0 (socket says 172.21.15.63:55704) 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfdbe00700 1 -- 172.21.15.63:0/1455170673 learned_addr learned my addr 172.21.15.63:0/1455170673 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfd3e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfdc1037a0 0x7fcfdc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfdbe00700 1 -- 172.21.15.63:0/1455170673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfdc1037a0 msgr2=0x7fcfdc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfdbe00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfdc1037a0 0x7fcfdc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.880+0000 7fcfdbe00700 1 -- 172.21.15.63:0/1455170673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc4005040 con 0x7fcfdc102e00 2024-03-23T12:28:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.881+0000 7fcfdbe00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc102e00 0x7fcfdc0702c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c830 tx=0x7fcfcc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:53.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.881+0000 7fcfd8c00700 1 -- 172.21.15.63:0/1455170673 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc0106f0 con 0x7fcfdc102e00 2024-03-23T12:28:53.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.881+0000 7fcfd8c00700 1 -- 172.21.15.63:0/1455170673 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfcc010d00 con 0x7fcfdc102e00 2024-03-23T12:28:53.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.881+0000 7fcfd8c00700 1 -- 172.21.15.63:0/1455170673 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc012e90 con 0x7fcfdc102e00 2024-03-23T12:28:53.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.881+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfdc119e00 con 0x7fcfdc102e00 2024-03-23T12:28:53.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.881+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfdc11a220 con 0x7fcfdc102e00 2024-03-23T12:28:53.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.882+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfdc0fe550 con 0x7fcfdc102e00 2024-03-23T12:28:53.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.883+0000 7fcfd8c00700 1 -- 172.21.15.63:0/1455170673 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fcfcc010850 con 0x7fcfdc102e00 2024-03-23T12:28:53.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.885+0000 7fcfd8c00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcfc006c7d0 0x7fcfc006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:53.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.885+0000 7fcfd8c00700 1 -- 172.21.15.63:0/1455170673 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcfcc08ba30 con 0x7fcfdc102e00 2024-03-23T12:28:53.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.885+0000 7fcfd3e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcfc006c7d0 0x7fcfc006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:53.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.886+0000 7fcfd3e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcfc006c7d0 0x7fcfc006ec80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc071c00 tx=0x7fcfc4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:53.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:53.890+0000 7fcfd8c00700 1 -- 172.21.15.63:0/1455170673 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcfcc055cb0 con 0x7fcfdc102e00 2024-03-23T12:28:54.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.122+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfdc0719b0 con 0x7fcfc006c7d0 2024-03-23T12:28:54.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.123+0000 7fcfd8c00700 1 -- 172.21.15.63:0/1455170673 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fcfdc0719b0 con 0x7fcfc006c7d0 2024-03-23T12:28:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.125+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcfc006c7d0 msgr2=0x7fcfc006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.125+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcfc006c7d0 0x7fcfc006ec80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc071c00 tx=0x7fcfc4009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc102e00 msgr2=0x7fcfdc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc102e00 0x7fcfdc0702c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c830 tx=0x7fcfcc004470 comp rx=0 tx=0).stop 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 shutdown_connections 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcfc006c7d0 0x7fcfc006ec80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfdc102e00 0x7fcfdc0702c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 --2- 172.21.15.63:0/1455170673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfdc1037a0 0x7fcfdc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 >> 172.21.15.63:0/1455170673 conn(0x7fcfdc0f89b0 msgr2=0x7fcfdc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 shutdown_connections 2024-03-23T12:28:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:54.126+0000 7fcfe2e00700 1 -- 172.21.15.63:0/1455170673 wait complete. 2024-03-23T12:28:54.125 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:53 smithi063 ceph-mon[141124]: pgmap v1136: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9.6 KiB/s wr, 20 op/s 2024-03-23T12:28:54.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:53 smithi111 ceph-mon[106899]: pgmap v1136: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9.6 KiB/s wr, 20 op/s 2024-03-23T12:28:54.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph orch ps' 2024-03-23T12:28:55.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:54 smithi063 ceph-mon[141124]: from='client.36034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:55.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:54 smithi111 ceph-mon[106899]: from='client.36034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:55.214 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:28:56.086 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:55 smithi063 ceph-mon[141124]: pgmap v1137: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 15 MiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-23T12:28:56.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:55 smithi111 ceph-mon[106899]: pgmap v1137: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 15 MiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 -- 172.21.15.63:0/1787120602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 msgr2=0x7f90381031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 --2- 172.21.15.63:0/1787120602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f90381031f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9028004940 tx=0x7f9028009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 -- 172.21.15.63:0/1787120602 shutdown_connections 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 --2- 172.21.15.63:0/1787120602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90381037c0 0x7f90380fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 --2- 172.21.15.63:0/1787120602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f90381031f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 -- 172.21.15.63:0/1787120602 >> 172.21.15.63:0/1787120602 conn(0x7f90380f8970 msgr2=0x7f90380fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 -- 172.21.15.63:0/1787120602 shutdown_connections 2024-03-23T12:28:57.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.040+0000 7f903f000700 1 -- 172.21.15.63:0/1787120602 wait complete. 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.041+0000 7f903f000700 1 Processor -- start 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.041+0000 7f903f000700 1 -- start start 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.041+0000 7f903f000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f9038115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.041+0000 7f903f000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90381037c0 0x7f90381162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.041+0000 7f903f000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90381168f0 con 0x7f9038102e20 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.041+0000 7f903f000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903810fcc0 con 0x7f90381037c0 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.042+0000 7f9037e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f9038115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.042+0000 7f9037e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f9038115d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55724/0 (socket says 172.21.15.63:55724) 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.042+0000 7f9037e00700 1 -- 172.21.15.63:0/1623502321 learned_addr learned my addr 172.21.15.63:0/1623502321 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.042+0000 7f9037e00700 1 -- 172.21.15.63:0/1623502321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90381037c0 msgr2=0x7f90381162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:57.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.042+0000 7f9037e00700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90381037c0 0x7f90381162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:57.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.042+0000 7f9037e00700 1 -- 172.21.15.63:0/1623502321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028005040 con 0x7f9038102e20 2024-03-23T12:28:57.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.042+0000 7f9037e00700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f9038115d60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9028009390 tx=0x7f9028009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:57.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.043+0000 7f902fe00700 1 -- 172.21.15.63:0/1623502321 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f902800e040 con 0x7f9038102e20 2024-03-23T12:28:57.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.043+0000 7f902fe00700 1 -- 172.21.15.63:0/1623502321 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9028013420 con 0x7f9038102e20 2024-03-23T12:28:57.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.043+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903810ff40 con 0x7f9038102e20 2024-03-23T12:28:57.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.043+0000 7f902fe00700 1 -- 172.21.15.63:0/1623502321 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f902801c630 con 0x7f9038102e20 2024-03-23T12:28:57.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.043+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9038110480 con 0x7f9038102e20 2024-03-23T12:28:57.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.044+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90380fe570 con 0x7f9038102e20 2024-03-23T12:28:57.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.045+0000 7f902fe00700 1 -- 172.21.15.63:0/1623502321 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f90280139e0 con 0x7f9038102e20 2024-03-23T12:28:57.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.046+0000 7f902fe00700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f901c06c7d0 0x7f901c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:28:57.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.046+0000 7f902fe00700 1 -- 172.21.15.63:0/1623502321 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f902808cd60 con 0x7f9038102e20 2024-03-23T12:28:57.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.046+0000 7f9037400700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f901c06c7d0 0x7f901c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:28:57.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.047+0000 7f9037400700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f901c06c7d0 0x7f901c06ec80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9038111600 tx=0x7f9024009ce0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:28:57.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.051+0000 7f902fe00700 1 -- 172.21.15.63:0/1623502321 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9028091070 con 0x7f9038102e20 2024-03-23T12:28:57.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:57.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:57.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:28:57.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:28:57.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.273+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9038109e50 con 0x7f901c06c7d0 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.283+0000 7f902fe00700 1 -- 172.21.15.63:0/1623502321 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f9038109e50 con 0x7f901c06c7d0 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (38m) 12s ago 51m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (35m) 12s ago 51m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (35m) 21s ago 50m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (37m) 12s ago 51m 73.9M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (31m) 12s ago 45m 93.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:28:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (31m) 12s ago 44m 117M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (15m) 21s ago 44m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (30m) 21s ago 45m 21.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (40m) 12s ago 52m 513M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (39m) 21s ago 50m 459M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (36m) 12s ago 53m 84.7M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (36m) 21s ago 50m 57.9M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (39m) 12s ago 51m 8568k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (38m) 21s ago 50m 8280k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (34m) 12s ago 49m 100M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (34m) 12s ago 48m 104M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (33m) 12s ago 48m 88.9M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:28:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (32m) 21s ago 47m 109M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:28:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (32m) 21s ago 47m 99.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:28:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (31m) 21s ago 46m 95.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:28:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (38m) 12s ago 50m 65.2M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:28:57.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.285+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f901c06c7d0 msgr2=0x7f901c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:57.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.285+0000 7f903f000700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f901c06c7d0 0x7f901c06ec80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9038111600 tx=0x7f9024009ce0 comp rx=0 tx=0).stop 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.285+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 msgr2=0x7f9038115d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.285+0000 7f903f000700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f9038115d60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9028009390 tx=0x7f9028009e30 comp rx=0 tx=0).stop 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.285+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 shutdown_connections 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.286+0000 7f903f000700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f901c06c7d0 0x7f901c06ec80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.286+0000 7f903f000700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9038102e20 0x7f9038115d60 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.286+0000 7f903f000700 1 --2- 172.21.15.63:0/1623502321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90381037c0 0x7f90381162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.286+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 >> 172.21.15.63:0/1623502321 conn(0x7f90380f8970 msgr2=0x7f90380f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.286+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 shutdown_connections 2024-03-23T12:28:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:28:57.286+0000 7f903f000700 1 -- 172.21.15.63:0/1623502321 wait complete. 2024-03-23T12:28:57.944 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:57 smithi063 ceph-mon[141124]: pgmap v1138: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 7 op/s 2024-03-23T12:28:57.970 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph orch upgrade status' 2024-03-23T12:28:58.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:57 smithi111 ceph-mon[106899]: pgmap v1138: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 7 op/s 2024-03-23T12:28:58.410 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:28:59.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:58 smithi063 ceph-mon[141124]: from='client.36038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:59.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:58 smithi111 ceph-mon[106899]: from='client.36038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:28:59.876 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:28:59 smithi063 ceph-mon[141124]: pgmap v1139: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 10 MiB/s rd, 5 op/s 2024-03-23T12:29:00.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:28:59 smithi111 ceph-mon[106899]: pgmap v1139: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 10 MiB/s rd, 5 op/s 2024-03-23T12:29:00.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 -- 172.21.15.63:0/3296519700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fe830 msgr2=0x7f263c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:00.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 --2- 172.21.15.63:0/3296519700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fe830 0x7f263c106f00 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2628004940 tx=0x7f2628009e30 comp rx=0 tx=0).stop 2024-03-23T12:29:00.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 -- 172.21.15.63:0/3296519700 shutdown_connections 2024-03-23T12:29:00.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 --2- 172.21.15.63:0/3296519700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fe830 0x7f263c106f00 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:00.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 --2- 172.21.15.63:0/3296519700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263c0fde90 0x7f263c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:00.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 -- 172.21.15.63:0/3296519700 >> 172.21.15.63:0/3296519700 conn(0x7f263c072bc0 msgr2=0x7f263c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:29:00.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 -- 172.21.15.63:0/3296519700 shutdown_connections 2024-03-23T12:29:00.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.328+0000 7f2641200700 1 -- 172.21.15.63:0/3296519700 wait complete. 2024-03-23T12:29:00.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2641200700 1 Processor -- start 2024-03-23T12:29:00.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2641200700 1 -- start start 2024-03-23T12:29:00.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2641200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fde90 0x7f263c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:00.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2641200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263c0fe830 0x7f263c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2641200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0fc820 con 0x7f263c0fde90 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2641200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0fc960 con 0x7f263c0fe830 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2639e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fde90 0x7f263c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2639e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fde90 0x7f263c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55756/0 (socket says 172.21.15.63:55756) 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.329+0000 7f2639e00700 1 -- 172.21.15.63:0/3809546466 learned_addr learned my addr 172.21.15.63:0/3809546466 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2639400700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263c0fe830 0x7f263c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2639e00700 1 -- 172.21.15.63:0/3809546466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263c0fe830 msgr2=0x7f263c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2639e00700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263c0fe830 0x7f263c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2639e00700 1 -- 172.21.15.63:0/3809546466 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2628005040 con 0x7f263c0fde90 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2639e00700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fde90 0x7f263c0fbd50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f262c002a50 tx=0x7f262c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2632000700 1 -- 172.21.15.63:0/3809546466 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f262c007cc0 con 0x7f263c0fde90 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2632000700 1 -- 172.21.15.63:0/3809546466 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f262c00f040 con 0x7f263c0fde90 2024-03-23T12:29:00.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2632000700 1 -- 172.21.15.63:0/3809546466 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f262c009540 con 0x7f263c0fde90 2024-03-23T12:29:00.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.330+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c119de0 con 0x7f263c0fde90 2024-03-23T12:29:00.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.331+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c11a180 con 0x7f263c0fde90 2024-03-23T12:29:00.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.332+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c065e30 con 0x7f263c0fde90 2024-03-23T12:29:00.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.332+0000 7f2632000700 1 -- 172.21.15.63:0/3809546466 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f262c0096a0 con 0x7f263c0fde90 2024-03-23T12:29:00.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.333+0000 7f2632000700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f262006c7d0 0x7f262006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:00.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.333+0000 7f2632000700 1 -- 172.21.15.63:0/3809546466 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f262c08b910 con 0x7f263c0fde90 2024-03-23T12:29:00.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.334+0000 7f2639400700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f262006c7d0 0x7f262006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:00.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.334+0000 7f2639400700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f262006c7d0 0x7f262006ec80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f262800b3b0 tx=0x7f2628009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:29:00.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.338+0000 7f2632000700 1 -- 172.21.15.63:0/3809546466 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f262c055a90 con 0x7f263c0fde90 2024-03-23T12:29:00.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.572+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f263c0fd440 con 0x7f262006c7d0 2024-03-23T12:29:00.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.573+0000 7f2632000700 1 -- 172.21.15.63:0/3809546466 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f263c0fd440 con 0x7f262006c7d0 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "target_image": null, 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "in_progress": false, 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "which": "", 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "services_complete": [], 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "progress": null, 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "message": "", 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "is_paused": false 2024-03-23T12:29:00.551 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f262006c7d0 msgr2=0x7f262006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f262006c7d0 0x7f262006ec80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f262800b3b0 tx=0x7f2628009e30 comp rx=0 tx=0).stop 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fde90 msgr2=0x7f263c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fde90 0x7f263c0fbd50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f262c002a50 tx=0x7f262c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 shutdown_connections 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f262006c7d0 0x7f262006ec80 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f263c0fde90 0x7f263c0fbd50 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 --2- 172.21.15.63:0/3809546466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f263c0fe830 0x7f263c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 >> 172.21.15.63:0/3809546466 conn(0x7f263c072bc0 msgr2=0x7f263c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:29:00.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 shutdown_connections 2024-03-23T12:29:00.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:00.575+0000 7f2641200700 1 -- 172.21.15.63:0/3809546466 wait complete. 2024-03-23T12:29:01.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph health detail' 2024-03-23T12:29:01.638 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:29:02.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:01 smithi063 ceph-mon[141124]: from='client.36042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:29:02.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:01 smithi063 ceph-mon[141124]: pgmap v1140: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 3 op/s 2024-03-23T12:29:02.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:01 smithi111 ceph-mon[106899]: from='client.36042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:29:02.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:01 smithi111 ceph-mon[106899]: pgmap v1140: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 3 op/s 2024-03-23T12:29:03.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.474+0000 7f6028600700 1 -- 172.21.15.63:0/698126548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 msgr2=0x7f60201053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:03.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.474+0000 7f6028600700 1 --2- 172.21.15.63:0/698126548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f60201053d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f6014004940 tx=0x7f6014009e30 comp rx=0 tx=0).stop 2024-03-23T12:29:03.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.474+0000 7f6028600700 1 -- 172.21.15.63:0/698126548 shutdown_connections 2024-03-23T12:29:03.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.474+0000 7f6028600700 1 --2- 172.21.15.63:0/698126548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60201059a0 0x7f60200ff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:03.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.474+0000 7f6028600700 1 --2- 172.21.15.63:0/698126548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f60201053d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:03.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.474+0000 7f6028600700 1 -- 172.21.15.63:0/698126548 >> 172.21.15.63:0/698126548 conn(0x7f60200fabd0 msgr2=0x7f60200fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:29:03.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.475+0000 7f6028600700 1 -- 172.21.15.63:0/698126548 shutdown_connections 2024-03-23T12:29:03.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.475+0000 7f6028600700 1 -- 172.21.15.63:0/698126548 wait complete. 2024-03-23T12:29:03.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.475+0000 7f6028600700 1 Processor -- start 2024-03-23T12:29:03.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6028600700 1 -- start start 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6028600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f6020115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6028600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60201059a0 0x7f6020116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6028600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6020116880 con 0x7f6020105000 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6028600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602010fc50 con 0x7f60201059a0 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6025c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f6020115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6025c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f6020115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56604/0 (socket says 172.21.15.63:56604) 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.476+0000 7f6025c00700 1 -- 172.21.15.63:0/3652471106 learned_addr learned my addr 172.21.15.63:0/3652471106 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f6025200700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60201059a0 0x7f6020116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f6025c00700 1 -- 172.21.15.63:0/3652471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60201059a0 msgr2=0x7f6020116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f6025c00700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60201059a0 0x7f6020116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f6025c00700 1 -- 172.21.15.63:0/3652471106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6014005040 con 0x7f6020105000 2024-03-23T12:29:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f6025c00700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f6020115cf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6014009360 tx=0x7f6014009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:29:03.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f601d600700 1 -- 172.21.15.63:0/3652471106 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f601400e040 con 0x7f6020105000 2024-03-23T12:29:03.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f601d600700 1 -- 172.21.15.63:0/3652471106 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6014013420 con 0x7f6020105000 2024-03-23T12:29:03.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602010fed0 con 0x7f6020105000 2024-03-23T12:29:03.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f601d600700 1 -- 172.21.15.63:0/3652471106 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f601401c5f0 con 0x7f6020105000 2024-03-23T12:29:03.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.477+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6020110490 con 0x7f6020105000 2024-03-23T12:29:03.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.478+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6020102310 con 0x7f6020105000 2024-03-23T12:29:03.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.479+0000 7f601d600700 1 -- 172.21.15.63:0/3652471106 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f6014013590 con 0x7f6020105000 2024-03-23T12:29:03.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.480+0000 7f601d600700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f600806c7d0 0x7f600806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:03.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.480+0000 7f601d600700 1 -- 172.21.15.63:0/3652471106 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f601408cc80 con 0x7f6020105000 2024-03-23T12:29:03.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.481+0000 7f6025200700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f600806c7d0 0x7f600806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:03.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.481+0000 7f6025200700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f600806c7d0 0x7f600806ec80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6020111590 tx=0x7f600c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:29:03.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.485+0000 7f601d600700 1 -- 172.21.15.63:0/3652471106 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f601405a990 con 0x7f6020105000 2024-03-23T12:29:03.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.793+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f602004bed0 con 0x7f6020105000 2024-03-23T12:29:03.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.793+0000 7f601d600700 1 -- 172.21.15.63:0/3652471106 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f601405a520 con 0x7f6020105000 2024-03-23T12:29:03.771 INFO:teuthology.orchestra.run.smithi063.stdout:HEALTH_OK 2024-03-23T12:29:03.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.795+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f600806c7d0 msgr2=0x7f600806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:03.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.795+0000 7f6028600700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f600806c7d0 0x7f600806ec80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6020111590 tx=0x7f600c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:29:03.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.795+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 msgr2=0x7f6020115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:03.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.795+0000 7f6028600700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f6020115cf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6014009360 tx=0x7f6014009e30 comp rx=0 tx=0).stop 2024-03-23T12:29:03.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.796+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 shutdown_connections 2024-03-23T12:29:03.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.796+0000 7f6028600700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f600806c7d0 0x7f600806ec80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:03.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.796+0000 7f6028600700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6020105000 0x7f6020115cf0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:03.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.796+0000 7f6028600700 1 --2- 172.21.15.63:0/3652471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60201059a0 0x7f6020116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:03.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.796+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 >> 172.21.15.63:0/3652471106 conn(0x7f60200fabd0 msgr2=0x7f60200fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:29:03.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.796+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 shutdown_connections 2024-03-23T12:29:03.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:03.796+0000 7f6028600700 1 -- 172.21.15.63:0/3652471106 wait complete. 2024-03-23T12:29:04.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:03 smithi063 ceph-mon[141124]: pgmap v1141: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 3 op/s 2024-03-23T12:29:04.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:03 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3652471106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:29:04.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:03 smithi111 ceph-mon[106899]: pgmap v1141: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 3 op/s 2024-03-23T12:29:04.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:03 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3652471106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-23T12:29:04.599 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph versions' 2024-03-23T12:29:05.036 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:29:06.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:05 smithi063 ceph-mon[141124]: pgmap v1142: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 4 op/s 2024-03-23T12:29:06.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:05 smithi111 ceph-mon[106899]: pgmap v1142: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 4 op/s 2024-03-23T12:29:06.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 -- 172.21.15.63:0/3947477556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 msgr2=0x7f0f9c0fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:06.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 --2- 172.21.15.63:0/3947477556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c0fc050 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c004940 tx=0x7f0f8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:29:06.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 -- 172.21.15.63:0/3947477556 shutdown_connections 2024-03-23T12:29:06.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 --2- 172.21.15.63:0/3947477556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f9c0fc590 0x7f0f9c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:06.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 --2- 172.21.15.63:0/3947477556 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c0fc050 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:06.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 -- 172.21.15.63:0/3947477556 >> 172.21.15.63:0/3947477556 conn(0x7f0f9c072bc0 msgr2=0x7f0f9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:29:06.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 -- 172.21.15.63:0/3947477556 shutdown_connections 2024-03-23T12:29:06.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.871+0000 7f0fa2400700 1 -- 172.21.15.63:0/3947477556 wait complete. 2024-03-23T12:29:06.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.872+0000 7f0fa2400700 1 Processor -- start 2024-03-23T12:29:06.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.872+0000 7f0fa2400700 1 -- start start 2024-03-23T12:29:06.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0fa2400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:06.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0fa2400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f9c0fc590 0x7f0f9c1160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:06.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0fa2400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c116740 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0fa2400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c10fb50 con 0x7f0f9c0fc590 2024-03-23T12:29:06.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0f9b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:06.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0f9b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56626/0 (socket says 172.21.15.63:56626) 2024-03-23T12:29:06.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0f9b400700 1 -- 172.21.15.63:0/627551324 learned_addr learned my addr 172.21.15.63:0/627551324 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0f93e00700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f9c0fc590 0x7f0f9c1160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0f9b400700 1 -- 172.21.15.63:0/627551324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f9c0fc590 msgr2=0x7f0f9c1160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0f9b400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f9c0fc590 0x7f0f9c1160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.873+0000 7f0f9b400700 1 -- 172.21.15.63:0/627551324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c005040 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.874+0000 7f0f9b400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c115bb0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c007f50 tx=0x7f0f8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.874+0000 7f0f93400700 1 -- 172.21.15.63:0/627551324 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8c021070 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.874+0000 7f0f93400700 1 -- 172.21.15.63:0/627551324 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f8c0093b0 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.874+0000 7f0f93400700 1 -- 172.21.15.63:0/627551324 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8c00e070 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.874+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c10fdd0 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.874+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c1102e0 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.875+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c065e30 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.876+0000 7f0f93400700 1 -- 172.21.15.63:0/627551324 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0f8c01c400 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.877+0000 7f0f93400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0f8006c7d0 0x7f0f8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:29:06.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.877+0000 7f0f93400700 1 -- 172.21.15.63:0/627551324 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0f8c08cc60 con 0x7f0f9c0fbc80 2024-03-23T12:29:06.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.877+0000 7f0f93e00700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0f8006c7d0 0x7f0f8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:29:06.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.878+0000 7f0f93e00700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0f8006c7d0 0x7f0f8006ec80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c111450 tx=0x7f0f8400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:29:06.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:06.882+0000 7f0f93400700 1 -- 172.21.15.63:0/627551324 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f8c056e60 con 0x7f0f9c0fbc80 2024-03-23T12:29:07.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.181+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0f9c065e30 con 0x7f0f9c0fbc80 2024-03-23T12:29:07.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.182+0000 7f0f93400700 1 -- 172.21.15.63:0/627551324 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f0f8c05a480 con 0x7f0f9c0fbc80 2024-03-23T12:29:07.160 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:29:07.160 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:29:07.160 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:29:07.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:29:07.160 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:29:07.161 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:29:07.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0f8006c7d0 msgr2=0x7f0f8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0f8006c7d0 0x7f0f8006ec80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c111450 tx=0x7f0f8400b040 comp rx=0 tx=0).stop 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 msgr2=0x7f0f9c115bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c115bb0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c007f50 tx=0x7f0f8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 shutdown_connections 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0f8006c7d0 0x7f0f8006ec80 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f9c0fbc80 0x7f0f9c115bb0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 --2- 172.21.15.63:0/627551324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f9c0fc590 0x7f0f9c1160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.184+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 >> 172.21.15.63:0/627551324 conn(0x7f0f9c072bc0 msgr2=0x7f0f9c105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.185+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 shutdown_connections 2024-03-23T12:29:07.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:29:07.185+0000 7f0fa2400700 1 -- 172.21.15.63:0/627551324 wait complete. 2024-03-23T12:29:07.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-23T12:29:08.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:07 smithi063 ceph-mon[141124]: pgmap v1143: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:08.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:07 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/627551324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:29:08.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:07 smithi111 ceph-mon[106899]: pgmap v1143: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:08.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:07 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/627551324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:29:08.237 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:29:10.076 INFO:teuthology.orchestra.run.smithi063.stdout:wait for servicemap items w/ changing names to refresh 2024-03-23T12:29:10.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:09 smithi063 ceph-mon[141124]: pgmap v1144: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:10.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:09 smithi111 ceph-mon[106899]: pgmap v1144: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:10.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'sleep 60' 2024-03-23T12:29:11.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:11.172 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:11.175 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:29:12.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:11 smithi063 ceph-mon[141124]: pgmap v1145: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:12.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:11 smithi111 ceph-mon[106899]: pgmap v1145: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:14.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:13 smithi063 ceph-mon[141124]: pgmap v1146: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:14.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:13 smithi111 ceph-mon[106899]: pgmap v1146: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:16.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:15 smithi063 ceph-mon[141124]: pgmap v1147: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:16.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:15 smithi111 ceph-mon[106899]: pgmap v1147: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:18.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:17 smithi063 ceph-mon[141124]: pgmap v1148: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:18.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:17 smithi111 ceph-mon[106899]: pgmap v1148: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:20.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:19 smithi063 ceph-mon[141124]: pgmap v1149: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:20.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:19 smithi111 ceph-mon[106899]: pgmap v1149: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:22.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:21 smithi111 ceph-mon[106899]: pgmap v1150: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:22.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:21 smithi063 ceph-mon[141124]: pgmap v1150: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:24.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:23 smithi111 ceph-mon[106899]: pgmap v1151: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:24.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:23 smithi063 ceph-mon[141124]: pgmap v1151: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:26.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:25 smithi111 ceph-mon[106899]: pgmap v1152: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:26.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:25 smithi063 ceph-mon[141124]: pgmap v1152: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:28.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:27 smithi111 ceph-mon[106899]: pgmap v1153: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:28.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:27 smithi063 ceph-mon[141124]: pgmap v1153: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:30.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:29 smithi111 ceph-mon[106899]: pgmap v1154: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:30.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:29 smithi063 ceph-mon[141124]: pgmap v1154: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:32.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:31 smithi063 ceph-mon[141124]: pgmap v1155: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:32.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:31 smithi111 ceph-mon[106899]: pgmap v1155: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:34.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:33 smithi063 ceph-mon[141124]: pgmap v1156: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:34.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:33 smithi111 ceph-mon[106899]: pgmap v1156: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:36.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:35 smithi063 ceph-mon[141124]: pgmap v1157: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:36.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:35 smithi111 ceph-mon[106899]: pgmap v1157: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:38.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:37 smithi063 ceph-mon[141124]: pgmap v1158: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:38.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:37 smithi111 ceph-mon[106899]: pgmap v1158: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:40.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:39 smithi063 ceph-mon[141124]: pgmap v1159: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:40.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:39 smithi111 ceph-mon[106899]: pgmap v1159: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:41.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:41.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:42.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:41 smithi063 ceph-mon[141124]: pgmap v1160: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:42.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:41 smithi111 ceph-mon[106899]: pgmap v1160: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:44.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:43 smithi063 ceph-mon[141124]: pgmap v1161: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:44.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:43 smithi111 ceph-mon[106899]: pgmap v1161: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:46.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:45 smithi063 ceph-mon[141124]: pgmap v1162: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:46.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:45 smithi111 ceph-mon[106899]: pgmap v1162: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:48.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:47 smithi063 ceph-mon[141124]: pgmap v1163: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:48.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:47 smithi111 ceph-mon[106899]: pgmap v1163: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:50.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:49 smithi063 ceph-mon[141124]: pgmap v1164: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:50.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:49 smithi111 ceph-mon[106899]: pgmap v1164: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:29:52.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:51 smithi063 ceph-mon[141124]: pgmap v1165: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:52.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:51 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:29:52.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:51 smithi111 ceph-mon[106899]: pgmap v1165: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:52.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:51 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:29:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:29:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:29:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:29:53.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:29:53.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:29:53.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:29:54.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:54 smithi063 ceph-mon[141124]: pgmap v1166: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:54.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:54 smithi111 ceph-mon[106899]: pgmap v1166: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:29:56.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:56 smithi063 ceph-mon[141124]: pgmap v1167: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:56.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:56.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:56 smithi111 ceph-mon[106899]: pgmap v1167: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:56.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:29:58.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:29:58 smithi063 ceph-mon[141124]: pgmap v1168: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:29:58.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:29:58 smithi111 ceph-mon[106899]: pgmap v1168: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:00.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:00 smithi063 ceph-mon[141124]: pgmap v1169: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:00.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:00 smithi063 ceph-mon[141124]: overall HEALTH_OK 2024-03-23T12:30:00.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:00 smithi111 ceph-mon[106899]: pgmap v1169: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:00.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:00 smithi111 ceph-mon[106899]: overall HEALTH_OK 2024-03-23T12:30:02.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:02 smithi063 ceph-mon[141124]: pgmap v1170: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:02.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:02 smithi111 ceph-mon[106899]: pgmap v1170: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:04.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:04 smithi063 ceph-mon[141124]: pgmap v1171: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:04.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:04 smithi111 ceph-mon[106899]: pgmap v1171: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:06.374 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:06 smithi063 ceph-mon[141124]: pgmap v1172: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:06.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:06 smithi111 ceph-mon[106899]: pgmap v1172: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:08.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:08 smithi063 ceph-mon[141124]: pgmap v1173: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:08.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:08 smithi111 ceph-mon[106899]: pgmap v1173: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:10.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:10 smithi063 ceph-mon[141124]: pgmap v1174: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:10.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:10 smithi111 ceph-mon[106899]: pgmap v1174: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:11.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:11.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:12.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:12 smithi111 ceph-mon[106899]: pgmap v1175: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:12.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:12 smithi063 ceph-mon[141124]: pgmap v1175: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:13.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph orch ps' 2024-03-23T12:30:13.981 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:30:14.385 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:14 smithi063 ceph-mon[141124]: pgmap v1176: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:14.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:14 smithi111 ceph-mon[106899]: pgmap v1176: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:15.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 -- 172.21.15.63:0/3163723126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 msgr2=0x7f5688070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 --2- 172.21.15.63:0/3163723126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f5688070d10 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5674004970 tx=0x7f5674009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 -- 172.21.15.63:0/3163723126 shutdown_connections 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 --2- 172.21.15.63:0/3163723126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f5688070d10 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 --2- 172.21.15.63:0/3163723126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5688071f90 0x7f5688070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 -- 172.21.15.63:0/3163723126 >> 172.21.15.63:0/3163723126 conn(0x7f56880f89b0 msgr2=0x7f56880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 -- 172.21.15.63:0/3163723126 shutdown_connections 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.910+0000 7f568d800700 1 -- 172.21.15.63:0/3163723126 wait complete. 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.911+0000 7f568d800700 1 Processor -- start 2024-03-23T12:30:15.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.911+0000 7f568d800700 1 -- start start 2024-03-23T12:30:15.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.911+0000 7f568d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f568811a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:15.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.911+0000 7f568d800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5688071f90 0x7f568811a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:15.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.911+0000 7f568d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568811ac40 con 0x7f56880708a0 2024-03-23T12:30:15.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.911+0000 7f568d800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5688114010 con 0x7f5688071f90 2024-03-23T12:30:15.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f568811a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:15.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f568811a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33200/0 (socket says 172.21.15.63:33200) 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686a00700 1 -- 172.21.15.63:0/1337942170 learned_addr learned my addr 172.21.15.63:0/1337942170 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686000700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5688071f90 0x7f568811a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686a00700 1 -- 172.21.15.63:0/1337942170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5688071f90 msgr2=0x7f568811a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686a00700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5688071f90 0x7f568811a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686a00700 1 -- 172.21.15.63:0/1337942170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5674005040 con 0x7f56880708a0 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.912+0000 7f5686a00700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f568811a0b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f567800c8c0 tx=0x7f56780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.913+0000 7f567ea00700 1 -- 172.21.15.63:0/1337942170 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f567800a760 con 0x7f56880708a0 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.913+0000 7f567ea00700 1 -- 172.21.15.63:0/1337942170 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f567800ad70 con 0x7f56880708a0 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.913+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56881142f0 con 0x7f56880708a0 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.913+0000 7f567ea00700 1 -- 172.21.15.63:0/1337942170 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56780113a0 con 0x7f56880708a0 2024-03-23T12:30:15.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.913+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5688114800 con 0x7f56880708a0 2024-03-23T12:30:15.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.914+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5688071aa0 con 0x7f56880708a0 2024-03-23T12:30:15.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.915+0000 7f567ea00700 1 -- 172.21.15.63:0/1337942170 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5678012400 con 0x7f56880708a0 2024-03-23T12:30:15.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.917+0000 7f567ea00700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f566c06c7d0 0x7f566c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:15.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.917+0000 7f567ea00700 1 -- 172.21.15.63:0/1337942170 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f567808ba60 con 0x7f56880708a0 2024-03-23T12:30:15.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.917+0000 7f5686000700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f566c06c7d0 0x7f566c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:15.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.917+0000 7f5686000700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f566c06c7d0 0x7f566c06ec80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5674009fd0 tx=0x7f5674009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:15.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:15.922+0000 7f567ea00700 1 -- 172.21.15.63:0/1337942170 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5678059770 con 0x7f56880708a0 2024-03-23T12:30:16.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.142+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 --> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f568804bed0 con 0x7f566c06c7d0 2024-03-23T12:30:16.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.153+0000 7f567ea00700 1 -- 172.21.15.63:0/1337942170 <== mgr.24659 v2:172.21.15.63:6800/3534621850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f568804bed0 con 0x7f566c06c7d0 2024-03-23T12:30:16.130 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:alertmanager.smithi063 smithi063 *:9093,9094 running (39m) 91s ago 52m 14.6M - 0.25.0 c8568f914cd2 db2f14eaa0c5 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi063 smithi063 running (36m) 91s ago 52m 6647k - 18.2.2-820-gb630d6cd 881b71eac2fe 40c6915ea7a4 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:crash.smithi111 smithi111 running (36m) 100s ago 51m 6656k - 18.2.2-820-gb630d6cd 881b71eac2fe 74eeb16a1383 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:grafana.smithi063 smithi063 *:3000 running (39m) 91s ago 52m 73.9M - 9.4.7 954c08fa6188 33e936432c0a 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.brtzgy smithi063 running (32m) 91s ago 46m 93.8M - 18.2.2-820-gb630d6cd 881b71eac2fe 6189b419070c 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi063.mdbdrz smithi063 running (32m) 91s ago 46m 117M - 18.2.2-820-gb630d6cd 881b71eac2fe e12bbb4c4d57 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.atkibb smithi111 running (16m) 100s ago 46m 16.3M - 18.2.2-820-gb630d6cd 881b71eac2fe d8327258217c 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mds.cephfs.smithi111.ylnmzy smithi111 running (31m) 100s ago 46m 21.4M - 18.2.2-820-gb630d6cd 881b71eac2fe 77f5c65ce0c5 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi063.gehkcr smithi063 *:8443,9283,8765 running (41m) 91s ago 54m 513M - 18.2.2-820-gb630d6cd 881b71eac2fe 04e7775bb6e2 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.smithi111.nnblsk smithi111 *:8443,9283,8765 running (40m) 100s ago 51m 459M - 18.2.2-820-gb630d6cd 881b71eac2fe c2dfdc38f3d1 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi063 smithi063 running (38m) 91s ago 54m 84.7M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe 48414a51fe0e 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:mon.smithi111 smithi111 running (37m) 100s ago 51m 57.9M 2048M 18.2.2-820-gb630d6cd 881b71eac2fe df03633d2cc1 2024-03-23T12:30:16.131 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi063 smithi063 *:9100 running (40m) 91s ago 52m 8568k - 1.5.0 0da6a335fe13 55f39c418a7c 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:node-exporter.smithi111 smithi111 *:9100 running (40m) 100s ago 51m 8280k - 1.5.0 0da6a335fe13 578966cd4cfc 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (36m) 91s ago 50m 100M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c0dca2307f3c 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi063 running (35m) 91s ago 50m 104M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe c3e531c0c9f7 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi063 running (34m) 91s ago 49m 88.9M 1618M 18.2.2-820-gb630d6cd 881b71eac2fe 03b76453ce48 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi111 running (34m) 100s ago 49m 109M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 060d49a70bd2 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi111 running (33m) 100s ago 48m 99.0M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe 22ff65335e1d 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi111 running (33m) 100s ago 48m 95.6M 2642M 18.2.2-820-gb630d6cd 881b71eac2fe b1ef416c095d 2024-03-23T12:30:16.132 INFO:teuthology.orchestra.run.smithi063.stdout:prometheus.smithi063 smithi063 *:9095 running (39m) 91s ago 52m 65.2M - 2.43.0 a07b618ecd1d d9ef28e676a3 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f566c06c7d0 msgr2=0x7f566c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f566c06c7d0 0x7f566c06ec80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5674009fd0 tx=0x7f5674009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 msgr2=0x7f568811a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f568811a0b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f567800c8c0 tx=0x7f56780044c0 comp rx=0 tx=0).stop 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 shutdown_connections 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f566c06c7d0 0x7f566c06ec80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56880708a0 0x7f568811a0b0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:16.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 --2- 172.21.15.63:0/1337942170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5688071f90 0x7f568811a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:16.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 >> 172.21.15.63:0/1337942170 conn(0x7f56880f89b0 msgr2=0x7f56880f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:16.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 shutdown_connections 2024-03-23T12:30:16.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:16.155+0000 7f568d800700 1 -- 172.21.15.63:0/1337942170 wait complete. 2024-03-23T12:30:16.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:16 smithi111 ceph-mon[106899]: pgmap v1177: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:16.504 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:16 smithi063 ceph-mon[141124]: pgmap v1177: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:16.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph versions' 2024-03-23T12:30:17.338 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:30:17.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:17 smithi111 ceph-mon[106899]: from='client.36054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:30:17.582 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:17 smithi063 ceph-mon[141124]: from='client.36054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-23T12:30:18.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:18 smithi111 ceph-mon[106899]: pgmap v1178: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:18.546 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:18 smithi063 ceph-mon[141124]: pgmap v1178: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:19.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.288+0000 7f0b8f000700 1 -- 172.21.15.63:0/2171571734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 msgr2=0x7f0b880fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:19.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.288+0000 7f0b8f000700 1 --2- 172.21.15.63:0/2171571734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b880fce10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f0b70004940 tx=0x7f0b70009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:19.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.288+0000 7f0b8f000700 1 -- 172.21.15.63:0/2171571734 shutdown_connections 2024-03-23T12:30:19.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.288+0000 7f0b8f000700 1 --2- 172.21.15.63:0/2171571734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b880fce10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:19.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.288+0000 7f0b8f000700 1 --2- 172.21.15.63:0/2171571734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b88102e00 0x7f0b881031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:19.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.288+0000 7f0b8f000700 1 -- 172.21.15.63:0/2171571734 >> 172.21.15.63:0/2171571734 conn(0x7f0b880f89b0 msgr2=0x7f0b880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:19.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.289+0000 7f0b8f000700 1 -- 172.21.15.63:0/2171571734 shutdown_connections 2024-03-23T12:30:19.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.289+0000 7f0b8f000700 1 -- 172.21.15.63:0/2171571734 wait complete. 2024-03-23T12:30:19.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.289+0000 7f0b8f000700 1 Processor -- start 2024-03-23T12:30:19.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b8f000700 1 -- start start 2024-03-23T12:30:19.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b8f000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b88102e00 0x7f0b88115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:19.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b8f000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b881162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:19.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b8f000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88116930 con 0x7f0b881037a0 2024-03-23T12:30:19.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b8f000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8810fd00 con 0x7f0b88102e00 2024-03-23T12:30:19.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b7fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b881162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:19.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b7fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b881162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33210/0 (socket says 172.21.15.63:33210) 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b7fe00700 1 -- 172.21.15.63:0/1013356791 learned_addr learned my addr 172.21.15.63:0/1013356791 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.290+0000 7f0b87e00700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b88102e00 0x7f0b88115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b7fe00700 1 -- 172.21.15.63:0/1013356791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b88102e00 msgr2=0x7f0b88115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b7fe00700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b88102e00 0x7f0b88115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b7fe00700 1 -- 172.21.15.63:0/1013356791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b70005040 con 0x7f0b881037a0 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b7fe00700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b881162e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009390 tx=0x7f0b70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b84c00700 1 -- 172.21.15.63:0/1013356791 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b7000e040 con 0x7f0b881037a0 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b84c00700 1 -- 172.21.15.63:0/1013356791 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b70013420 con 0x7f0b881037a0 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b84c00700 1 -- 172.21.15.63:0/1013356791 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b7001c630 con 0x7f0b881037a0 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8810ff80 con 0x7f0b881037a0 2024-03-23T12:30:19.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.291+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b881104c0 con 0x7f0b881037a0 2024-03-23T12:30:19.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.292+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b880fe550 con 0x7f0b881037a0 2024-03-23T12:30:19.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.293+0000 7f0b84c00700 1 -- 172.21.15.63:0/1013356791 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f0b70013a00 con 0x7f0b881037a0 2024-03-23T12:30:19.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.294+0000 7f0b84c00700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0b6c06c7d0 0x7f0b6c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:19.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.294+0000 7f0b84c00700 1 -- 172.21.15.63:0/1013356791 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0b7008cb20 con 0x7f0b881037a0 2024-03-23T12:30:19.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.294+0000 7f0b87e00700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0b6c06c7d0 0x7f0b6c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:19.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.295+0000 7f0b87e00700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0b6c06c7d0 0x7f0b6c06ec80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0b78004f40 tx=0x7f0b7800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:19.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.299+0000 7f0b84c00700 1 -- 172.21.15.63:0/1013356791 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0b7005a830 con 0x7f0b881037a0 2024-03-23T12:30:19.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.598+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0b8804bed0 con 0x7f0b881037a0 2024-03-23T12:30:19.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.599+0000 7f0b84c00700 1 -- 172.21.15.63:0/1013356791 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f0b7005a3c0 con 0x7f0b881037a0 2024-03-23T12:30:19.576 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-23T12:30:19.576 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": { 2024-03-23T12:30:19.576 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": { 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 2 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": { 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 6 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 4 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "overall": { 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)": 14 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-23T12:30:19.577 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0b6c06c7d0 msgr2=0x7f0b6c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0b6c06c7d0 0x7f0b6c06ec80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0b78004f40 tx=0x7f0b7800b040 comp rx=0 tx=0).stop 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 msgr2=0x7f0b881162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b881162e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009390 tx=0x7f0b70009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 shutdown_connections 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f0b6c06c7d0 0x7f0b6c06ec80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b88102e00 0x7f0b88115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 --2- 172.21.15.63:0/1013356791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b881037a0 0x7f0b881162e0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:19.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 >> 172.21.15.63:0/1013356791 conn(0x7f0b880f89b0 msgr2=0x7f0b880f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:19.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.601+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 shutdown_connections 2024-03-23T12:30:19.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:19.602+0000 7f0b8f000700 1 -- 172.21.15.63:0/1013356791 wait complete. 2024-03-23T12:30:20.204 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:20 smithi063 ceph-mon[141124]: pgmap v1179: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:20.204 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:20 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1013356791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:30:20.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-23T12:30:20.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:20 smithi111 ceph-mon[106899]: pgmap v1179: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:20.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:20 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1013356791' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:30:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:30:22.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 -- 172.21.15.63:0/293608688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 msgr2=0x7fc8b40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/293608688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b40fce10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0004940 tx=0x7fc8a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 -- 172.21.15.63:0/293608688 shutdown_connections 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/293608688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b40fce10 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/293608688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4102e00 0x7fc8b41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 -- 172.21.15.63:0/293608688 >> 172.21.15.63:0/293608688 conn(0x7fc8b40f89b0 msgr2=0x7fc8b40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 -- 172.21.15.63:0/293608688 shutdown_connections 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.485+0000 7fc8b8c00700 1 -- 172.21.15.63:0/293608688 wait complete. 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.486+0000 7fc8b8c00700 1 Processor -- start 2024-03-23T12:30:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b8c00700 1 -- start start 2024-03-23T12:30:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b8c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4102e00 0x7fc8b40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b8c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4070d90 con 0x7fc8b41037a0 2024-03-23T12:30:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b8c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4070ed0 con 0x7fc8b4102e00 2024-03-23T12:30:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34136/0 (socket says 172.21.15.63:34136) 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b1400700 1 -- 172.21.15.63:0/1684250693 learned_addr learned my addr 172.21.15.63:0/1684250693 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.487+0000 7fc8b1e00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4102e00 0x7fc8b40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8b1400700 1 -- 172.21.15.63:0/1684250693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4102e00 msgr2=0x7fc8b40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8b1400700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4102e00 0x7fc8b40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8b1400700 1 -- 172.21.15.63:0/1684250693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a0005040 con 0x7fc8b41037a0 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8b1400700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b4070800 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009380 tx=0x7fc8a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8aa000700 1 -- 172.21.15.63:0/1684250693 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a000e040 con 0x7fc8b41037a0 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8aa000700 1 -- 172.21.15.63:0/1684250693 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8a0013420 con 0x7fc8b41037a0 2024-03-23T12:30:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b4071120 con 0x7fc8b41037a0 2024-03-23T12:30:22.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8aa000700 1 -- 172.21.15.63:0/1684250693 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a001c630 con 0x7fc8b41037a0 2024-03-23T12:30:22.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.488+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b411a230 con 0x7fc8b41037a0 2024-03-23T12:30:22.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.489+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b40fe550 con 0x7fc8b41037a0 2024-03-23T12:30:22.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.490+0000 7fc8aa000700 1 -- 172.21.15.63:0/1684250693 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc8a00139b0 con 0x7fc8b41037a0 2024-03-23T12:30:22.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.491+0000 7fc8aa000700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc89806c7c0 0x7fc89806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:22.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.491+0000 7fc8aa000700 1 -- 172.21.15.63:0/1684250693 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc8a008cb90 con 0x7fc8b41037a0 2024-03-23T12:30:22.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.492+0000 7fc8b1e00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc89806c7c0 0x7fc89806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:22.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.492+0000 7fc8b1e00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc89806c7c0 0x7fc89806ec70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4007fd0 tx=0x7fc8a4009040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:22.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.496+0000 7fc8aa000700 1 -- 172.21.15.63:0/1684250693 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc8a0056e10 con 0x7fc8b41037a0 2024-03-23T12:30:22.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:22 smithi063 ceph-mon[141124]: pgmap v1180: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:22.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:22 smithi111 ceph-mon[106899]: pgmap v1180: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:22.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.798+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc8b4109da0 con 0x7fc8b41037a0 2024-03-23T12:30:22.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.798+0000 7fc8aa000700 1 -- 172.21.15.63:0/1684250693 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fc8a005a430 con 0x7fc8b41037a0 2024-03-23T12:30:22.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.800+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc89806c7c0 msgr2=0x7fc89806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.800+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc89806c7c0 0x7fc89806ec70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4007fd0 tx=0x7fc8a4009040 comp rx=0 tx=0).stop 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 msgr2=0x7fc8b4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b4070800 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009380 tx=0x7fc8a0009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 shutdown_connections 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc89806c7c0 0x7fc89806ec70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4102e00 0x7fc8b40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 --2- 172.21.15.63:0/1684250693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8b41037a0 0x7fc8b4070800 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 >> 172.21.15.63:0/1684250693 conn(0x7fc8b40f89b0 msgr2=0x7fc8b40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 shutdown_connections 2024-03-23T12:30:22.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:22.801+0000 7fc8b8c00700 1 -- 172.21.15.63:0/1684250693 wait complete. 2024-03-23T12:30:22.793 INFO:teuthology.orchestra.run.smithi063.stdout:true 2024-03-23T12:30:23.420 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:23 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1684250693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:30:23.444 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -e sha1=b630d6cd702af63dfefcf906294a092ddce1c609 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-23T12:30:23.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:23 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1684250693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:30:23.889 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:30:24.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:24 smithi063 ceph-mon[141124]: pgmap v1181: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:24.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:24 smithi111 ceph-mon[106899]: pgmap v1181: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 -- 172.21.15.63:0/779156913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 msgr2=0x7f2f381031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 --2- 172.21.15.63:0/779156913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f381031f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c004940 tx=0x7f2f2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 -- 172.21.15.63:0/779156913 shutdown_connections 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 --2- 172.21.15.63:0/779156913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f381037c0 0x7f2f380fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 --2- 172.21.15.63:0/779156913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f381031f0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 -- 172.21.15.63:0/779156913 >> 172.21.15.63:0/779156913 conn(0x7f2f380f8970 msgr2=0x7f2f380fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 -- 172.21.15.63:0/779156913 shutdown_connections 2024-03-23T12:30:25.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.783+0000 7f2f3f400700 1 -- 172.21.15.63:0/779156913 wait complete. 2024-03-23T12:30:25.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.784+0000 7f2f3f400700 1 Processor -- start 2024-03-23T12:30:25.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.784+0000 7f2f3f400700 1 -- start start 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.784+0000 7f2f3f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.784+0000 7f2f3f400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f381037c0 0x7f2f38070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.784+0000 7f2f3f400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f38070d90 con 0x7f2f38102e20 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.784+0000 7f2f3f400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f38070ed0 con 0x7f2f381037c0 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f3ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f3ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f380702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34158/0 (socket says 172.21.15.63:34158) 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f3ca00700 1 -- 172.21.15.63:0/45175563 learned_addr learned my addr 172.21.15.63:0/45175563 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f37e00700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f381037c0 0x7f2f38070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f3ca00700 1 -- 172.21.15.63:0/45175563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f381037c0 msgr2=0x7f2f38070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f3ca00700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f381037c0 0x7f2f38070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f3ca00700 1 -- 172.21.15.63:0/45175563 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c005040 con 0x7f2f38102e20 2024-03-23T12:30:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f37e00700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f381037c0 0x7f2f38070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:30:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.785+0000 7f2f3ca00700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f380702c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009390 tx=0x7f2f2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.786+0000 7f2f34c00700 1 -- 172.21.15.63:0/45175563 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2c00e040 con 0x7f2f38102e20 2024-03-23T12:30:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.786+0000 7f2f34c00700 1 -- 172.21.15.63:0/45175563 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f2c013420 con 0x7f2f38102e20 2024-03-23T12:30:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.786+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f38071120 con 0x7f2f38102e20 2024-03-23T12:30:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.786+0000 7f2f34c00700 1 -- 172.21.15.63:0/45175563 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f2c01c630 con 0x7f2f38102e20 2024-03-23T12:30:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.786+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3811a210 con 0x7f2f38102e20 2024-03-23T12:30:25.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.787+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f380fe570 con 0x7f2f38102e20 2024-03-23T12:30:25.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.788+0000 7f2f34c00700 1 -- 172.21.15.63:0/45175563 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2f2c0139e0 con 0x7f2f38102e20 2024-03-23T12:30:25.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.789+0000 7f2f34c00700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2006c8a0 0x7f2f2006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:25.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.789+0000 7f2f34c00700 1 -- 172.21.15.63:0/45175563 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f2c08cca0 con 0x7f2f38102e20 2024-03-23T12:30:25.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.789+0000 7f2f37e00700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2006c8a0 0x7f2f2006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:25.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.790+0000 7f2f37e00700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2006c8a0 0x7f2f2006ed50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009fd0 tx=0x7f2f2400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:25.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:25.793+0000 7f2f34c00700 1 -- 172.21.15.63:0/45175563 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f2c056f20 con 0x7f2f38102e20 2024-03-23T12:30:26.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.098+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f380719b0 con 0x7f2f38102e20 2024-03-23T12:30:26.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.099+0000 7f2f34c00700 1 -- 172.21.15.63:0/45175563 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f2f2c05a540 con 0x7f2f38102e20 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2006c8a0 msgr2=0x7f2f2006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2006c8a0 0x7f2f2006ed50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009fd0 tx=0x7f2f2400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 msgr2=0x7f2f380702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f380702c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009390 tx=0x7f2f2c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 shutdown_connections 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f2f2006c8a0 0x7f2f2006ed50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f38102e20 0x7f2f380702c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:26.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 --2- 172.21.15.63:0/45175563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f381037c0 0x7f2f38070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:26.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 >> 172.21.15.63:0/45175563 conn(0x7f2f380f8970 msgr2=0x7f2f380f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:26.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.101+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 shutdown_connections 2024-03-23T12:30:26.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:26.102+0000 7f2f3f400700 1 -- 172.21.15.63:0/45175563 wait complete. 2024-03-23T12:30:26.093 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)" 2024-03-23T12:30:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:26 smithi063 ceph-mon[141124]: pgmap v1182: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:26 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/45175563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:30:26.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:26 smithi111 ceph-mon[106899]: pgmap v1182: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:26.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:26.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:26 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/45175563' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-23T12:30:26.804 DEBUG:teuthology.parallel:result is None 2024-03-23T12:30:26.804 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-23T12:30:26.816 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T12:30:26.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-23T12:30:27.258 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/mon.smithi063/config 2024-03-23T12:30:28.437 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:28 smithi063 ceph-mon[141124]: pgmap v1183: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:28.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:28 smithi111 ceph-mon[106899]: pgmap v1183: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 -- 172.21.15.63:0/2243547202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 msgr2=0x7fc60c070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 --2- 172.21.15.63:0/2243547202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c070d10 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8004970 tx=0x7fc5f8009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 -- 172.21.15.63:0/2243547202 shutdown_connections 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 --2- 172.21.15.63:0/2243547202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c070d10 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 --2- 172.21.15.63:0/2243547202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c071f90 0x7fc60c070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 -- 172.21.15.63:0/2243547202 >> 172.21.15.63:0/2243547202 conn(0x7fc60c0f89b0 msgr2=0x7fc60c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 -- 172.21.15.63:0/2243547202 shutdown_connections 2024-03-23T12:30:29.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.097+0000 7fc612000700 1 -- 172.21.15.63:0/2243547202 wait complete. 2024-03-23T12:30:29.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.098+0000 7fc612000700 1 Processor -- start 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.098+0000 7fc612000700 1 -- start start 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.098+0000 7fc612000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.098+0000 7fc612000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c071f90 0x7fc60c11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.098+0000 7fc612000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c11ac60 con 0x7fc60c0708a0 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.098+0000 7fc612000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c114050 con 0x7fc60c071f90 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.099+0000 7fc60b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.099+0000 7fc60b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c11a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34174/0 (socket says 172.21.15.63:34174) 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.099+0000 7fc60b400700 1 -- 172.21.15.63:0/1870290917 learned_addr learned my addr 172.21.15.63:0/1870290917 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.099+0000 7fc60aa00700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c071f90 0x7fc60c11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.099+0000 7fc60b400700 1 -- 172.21.15.63:0/1870290917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c071f90 msgr2=0x7fc60c11a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.099+0000 7fc60b400700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c071f90 0x7fc60c11a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:29.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.099+0000 7fc60b400700 1 -- 172.21.15.63:0/1870290917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8005040 con 0x7fc60c0708a0 2024-03-23T12:30:29.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.100+0000 7fc60b400700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c11a040 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00b790 tx=0x7fc5fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:29.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.100+0000 7fc603400700 1 -- 172.21.15.63:0/1870290917 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5fc00f6f0 con 0x7fc60c0708a0 2024-03-23T12:30:29.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.100+0000 7fc603400700 1 -- 172.21.15.63:0/1870290917 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5fc00fd00 con 0x7fc60c0708a0 2024-03-23T12:30:29.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.100+0000 7fc603400700 1 -- 172.21.15.63:0/1870290917 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5fc0103a0 con 0x7fc60c0708a0 2024-03-23T12:30:29.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.100+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c114330 con 0x7fc60c0708a0 2024-03-23T12:30:29.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.100+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c114840 con 0x7fc60c0708a0 2024-03-23T12:30:29.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.101+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c071aa0 con 0x7fc60c0708a0 2024-03-23T12:30:29.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.102+0000 7fc603400700 1 -- 172.21.15.63:0/1870290917 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc5fc00f850 con 0x7fc60c0708a0 2024-03-23T12:30:29.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.103+0000 7fc603400700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc5f006c7d0 0x7fc5f006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:29.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.103+0000 7fc603400700 1 -- 172.21.15.63:0/1870290917 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc5fc08b7e0 con 0x7fc60c0708a0 2024-03-23T12:30:29.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.103+0000 7fc60aa00700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc5f006c7d0 0x7fc5f006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:29.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.104+0000 7fc60aa00700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc5f006c7d0 0x7fc5f006ec80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009fd0 tx=0x7fc5f8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:29.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.108+0000 7fc603400700 1 -- 172.21.15.63:0/1870290917 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc5fc055b20 con 0x7fc60c0708a0 2024-03-23T12:30:29.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.365+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc60c110460 con 0x7fc60c0708a0 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.365+0000 7fc603400700 1 -- 172.21.15.63:0/1870290917 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 43 v43) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fc5fc059140 con 0x7fc60c0708a0 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:e43 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 43 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-23T12:30:29.343 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-23T11:43:49.209465+0000 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-03-23T12:28:42.227161+0000 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 60 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 82 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 2 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=34422,1=34454} 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-03-23T12:30:29.344 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 1 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.brtzgy{0:34422} state up:active seq 470 join_fscid=1 addr [v2:172.21.15.63:6828/450408958,v1:172.21.15.63:6829/450408958] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.ylnmzy{0:44393} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6824/2255440447,v1:172.21.15.111:6826/2255440447] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi063.mdbdrz{1:34454} state up:active seq 465 join_fscid=1 addr [v2:172.21.15.63:6826/1595814379,v1:172.21.15.63:6827/1595814379] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.cephfs.smithi111.atkibb{1:45001} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.111:6825/755283916,v1:172.21.15.111:6827/755283916] compat {c=[1],r=[1],i=[7ff]}] 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:29.345 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:29.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.367+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc5f006c7d0 msgr2=0x7fc5f006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:29.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.367+0000 7fc612000700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc5f006c7d0 0x7fc5f006ec80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009fd0 tx=0x7fc5f8009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:29.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.367+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 msgr2=0x7fc60c11a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:29.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.367+0000 7fc612000700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c11a040 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00b790 tx=0x7fc5fc009250 comp rx=0 tx=0).stop 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.368+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 shutdown_connections 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.368+0000 7fc612000700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc5f006c7d0 0x7fc5f006ec80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.368+0000 7fc612000700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc60c0708a0 0x7fc60c11a040 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.368+0000 7fc612000700 1 --2- 172.21.15.63:0/1870290917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c071f90 0x7fc60c11a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.368+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 >> 172.21.15.63:0/1870290917 conn(0x7fc60c0f89b0 msgr2=0x7fc60c0f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.368+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 shutdown_connections 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:29.368+0000 7fc612000700 1 -- 172.21.15.63:0/1870290917 wait complete. 2024-03-23T12:30:29.347 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 43 2024-03-23T12:30:30.109 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-03-23T12:30:30.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T12:30:30.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.341+0000 7f3c25800700 1 -- 172.21.15.63:0/4109540824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 msgr2=0x7f3c200fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:30.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.341+0000 7f3c25800700 1 --2- 172.21.15.63:0/4109540824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c200fcd10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c004940 tx=0x7f3c0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:30.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.342+0000 7f3c25800700 1 -- 172.21.15.63:0/4109540824 shutdown_connections 2024-03-23T12:30:30.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.342+0000 7f3c25800700 1 --2- 172.21.15.63:0/4109540824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c200fcd10 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.342+0000 7f3c25800700 1 --2- 172.21.15.63:0/4109540824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c20102d00 0x7f3c201030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.342+0000 7f3c25800700 1 -- 172.21.15.63:0/4109540824 >> 172.21.15.63:0/4109540824 conn(0x7f3c200f8830 msgr2=0x7f3c200fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:30.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.342+0000 7f3c25800700 1 -- 172.21.15.63:0/4109540824 shutdown_connections 2024-03-23T12:30:30.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.342+0000 7f3c25800700 1 -- 172.21.15.63:0/4109540824 wait complete. 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.342+0000 7f3c25800700 1 Processor -- start 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c25800700 1 -- start start 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c25800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c20102d00 0x7f3c200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c25800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c20070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c25800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20072210 con 0x7f3c201036a0 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c25800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20070d40 con 0x7f3c20102d00 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c1e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c20070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c1e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c20070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34186/0 (socket says 172.21.15.63:34186) 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c1e000700 1 -- 172.21.15.63:0/649564808 learned_addr learned my addr 172.21.15.63:0/649564808 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.343+0000 7f3c1ea00700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c20102d00 0x7f3c200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:30.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c1e000700 1 -- 172.21.15.63:0/649564808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c20102d00 msgr2=0x7f3c200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:30.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c1e000700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c20102d00 0x7f3c200702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c1e000700 1 -- 172.21.15.63:0/649564808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c005040 con 0x7f3c201036a0 2024-03-23T12:30:30.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c1e000700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c20070800 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009390 tx=0x7f3c0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c16a00700 1 -- 172.21.15.63:0/649564808 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c0c00e040 con 0x7f3c201036a0 2024-03-23T12:30:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c16a00700 1 -- 172.21.15.63:0/649564808 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c0c004290 con 0x7f3c201036a0 2024-03-23T12:30:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c20070fc0 con 0x7f3c201036a0 2024-03-23T12:30:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c16a00700 1 -- 172.21.15.63:0/649564808 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c0c013e70 con 0x7f3c201036a0 2024-03-23T12:30:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.344+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c20119f60 con 0x7f3c201036a0 2024-03-23T12:30:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.345+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c200fe450 con 0x7f3c201036a0 2024-03-23T12:30:30.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.346+0000 7f3c16a00700 1 -- 172.21.15.63:0/649564808 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3c0c013690 con 0x7f3c201036a0 2024-03-23T12:30:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.347+0000 7f3c16a00700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0406c760 0x7f3c0406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.347+0000 7f3c16a00700 1 -- 172.21.15.63:0/649564808 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3c0c08ccc0 con 0x7f3c201036a0 2024-03-23T12:30:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.347+0000 7f3c1ea00700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0406c760 0x7f3c0406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:30.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.348+0000 7f3c1ea00700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0406c760 0x7f3c0406ec10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3c10004f40 tx=0x7f3c1000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:30.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.352+0000 7f3c16a00700 1 -- 172.21.15.63:0/649564808 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3c0c092020 con 0x7f3c201036a0 2024-03-23T12:30:30.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:30 smithi063 ceph-mon[141124]: pgmap v1184: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:30.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:30 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1870290917' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:30:30.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.570+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c20064970 con 0x7f3c201036a0 2024-03-23T12:30:30.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.572+0000 7f3c16a00700 1 -- 172.21.15.63:0/649564808 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16563 (secure 0 0 0) 0x7f3c0c056f70 con 0x7f3c201036a0 2024-03-23T12:30:30.550 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:30.550 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":86,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","created":"2024-03-23T11:35:55.761926+0000","modified":"2024-03-23T12:13:23.282611+0000","last_up_change":"2024-03-23T11:57:22.490793+0000","last_in_change":"2024-03-23T11:42:00.992881+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:40:53.432498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-23T11:43:48.146323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-23T11:43:48.722717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e4f29eb-375c-4f37-a8d8-f82b40650ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6803","nonce":2164791827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6805","nonce":2164791827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6809","nonce":2164791827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6807","nonce":2164791827}]},"public_addr":"172.21.15.63:6803/2164791827","cluster_addr":"172.21.15.63:6805/2164791827","heartbeat_back_addr":"172.21.15.63:6809/2164791827","heartbeat_front_addr":"172.21.15.63:6807/2164791827","state":["exists","up"]},{"osd":1,"uuid":"5e3ddf8b-e2c3-4188-b495-ac2743805f33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6811","nonce":3286820315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6813","nonce":3286820315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6817","nonce":3286820315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6815","nonce":3286820315}]},"public_addr":"172.21.15.63:6811/3286820315","cluster_addr":"172.21.15.63:6813/3286820315","heartbeat_back_addr":"172.21.15.63:6817/3286820315","heartbeat_front_addr":"172.21.15.63:6815/3286820315","state":["exists","up"]},{"osd":2,"uuid":"edc4babe-dff9-462c-9af7-75e30b8095af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6819","nonce":3910577428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6821","nonce":3910577428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6825","nonce":3910577428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6823","nonce":3910577428}]},"public_addr":"172.21.15.63:6819/3910577428","cluster_addr":"172.21.15.63:6821/3910577428","heartbeat_back_addr":"172.21.15.63:6825/3910577428","heartbeat_front_addr":"172.21.15.63:6823/3910577428","state":["exists","up"]},{"osd":3,"uuid":"7bc94d72-a814-447a-8112-9d50058a2759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6801","nonce":1545952189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6803","nonce":1545952189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6807","nonce":1545952189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6805","nonce":1545952189}]},"public_addr":"172.21.15.111:6801/1545952189","cluster_addr":"172.21.15.111:6803/1545952189","heartbeat_back_addr":"172.21.15.111:6807/1545952189","heartbeat_front_addr":"172.21.15.111:6805/1545952189","state":["exists","up"]},{"osd":4,"uuid":"7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6809","nonce":2121500845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6811","nonce":2121500845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6815","nonce":2121500845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6813","nonce":2121500845}]},"public_addr":"172.21.15.111:6809/2121500845","cluster_addr":"172.21.15.111:6811/2121500845","heartbeat_back_addr":"172.21.15.111:6815/2121500845","heartbeat_front_addr":"172.21.15.111:6813/2121500845","state":["exists","up"]},{"osd":5,"uuid":"2219a929-a842-4b41-ab46-096be6ef4d13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6817","nonce":983556337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6819","nonce":983556337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6823","nonce":983556337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6821","nonce":983556337}]},"public_addr":"172.21.15.111:6817/983556337","cluster_addr":"172.21.15.111:6819/983556337","heartbeat_back_addr":"172.21.15.111:6823/983556337","heartbeat_front_addr":"172.21.15.111:6821/983556337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:39:42.374999+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:17.321830+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:50.840928+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:20.211360+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:50.314996+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:42:20.376924+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6829/2516647170":"2024-03-24T11:57:27.441051+0000","172.21.15.63:0/3192464709":"2024-03-24T11:36:25.877145+0000","172.21.15.111:0/3712729872":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/2812522808":"2024-03-24T11:46:26.114316+0000","172.21.15.111:6828/3542552256":"2024-03-24T11:48:52.950910+0000","172.21.15.111:0/245431515":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3356174638":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/2483586841":"2024-03-24T11:46:26.114316+0000","172.21.15.63:0/2981669421":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6801/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6800/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/2062408886":"2024-03-24T11:52:25.664441+0000","172.21.15.63:0/2603926600":"2024-03-24T11:46:26.114316+0000","172.21.15.63:6801/2193815059":"2024-03-24T11:46:26.114316+0000","172.21.15.111:6824/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:6826/3230334326":"2024-03-24T11:57:27.441037+0000","172.21.15.63:0/2286910220":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6800/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.111:6825/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:6801/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1379382252":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3148717714":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6800/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6827/3230334326":"2024-03-24T11:57:27.441037+0000","172.21.15.63:6800/335624352":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6801/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.111:6829/3542552256":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3491519284":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1183338694":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/2975088208":"2024-03-24T11:46:26.114316+0000","172.21.15.63:0/1697976918":"2024-03-24T11:52:25.664441+0000","172.21.15.63:0/3227457751":"2024-03-24T11:37:01.693515+0000","172.21.15.111:0/3254210644":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3688530508":"2024-03-24T11:52:25.664441+0000","172.21.15.111:0/2862601802":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3670139931":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6800/2193815059":"2024-03-24T11:46:26.114316+0000","172.21.15.63:6801/335624352":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6828/2516647170":"2024-03-24T11:57:27.441051+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0406c760 msgr2=0x7f3c0406ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0406c760 0x7f3c0406ec10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3c10004f40 tx=0x7f3c1000b040 comp rx=0 tx=0).stop 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 msgr2=0x7f3c20070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c20070800 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009390 tx=0x7f3c0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 shutdown_connections 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3c0406c760 0x7f3c0406ec10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c20102d00 0x7f3c200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 --2- 172.21.15.63:0/649564808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c201036a0 0x7f3c20070800 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 >> 172.21.15.63:0/649564808 conn(0x7f3c200f8830 msgr2=0x7f3c200f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 shutdown_connections 2024-03-23T12:30:30.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.574+0000 7f3c25800700 1 -- 172.21.15.63:0/649564808 wait complete. 2024-03-23T12:30:30.570 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:40:53.432498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-23T11:43:48.146323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-23T11:43:48.722717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T12:30:30.571 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T12:30:30.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:30 smithi111 ceph-mon[106899]: pgmap v1184: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:30.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:30 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1870290917' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-23T12:30:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.789+0000 7f2664a00700 1 -- 172.21.15.63:0/930340722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660102cd0 msgr2=0x7f26601030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.789+0000 7f2664a00700 1 --2- 172.21.15.63:0/930340722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660102cd0 0x7f26601030a0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2650004940 tx=0x7f2650009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.790+0000 7f2664a00700 1 -- 172.21.15.63:0/930340722 shutdown_connections 2024-03-23T12:30:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.790+0000 7f2664a00700 1 --2- 172.21.15.63:0/930340722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2660103670 0x7f26600fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.790+0000 7f2664a00700 1 --2- 172.21.15.63:0/930340722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660102cd0 0x7f26601030a0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.790+0000 7f2664a00700 1 -- 172.21.15.63:0/930340722 >> 172.21.15.63:0/930340722 conn(0x7f26600f8820 msgr2=0x7f26600fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:30.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.790+0000 7f2664a00700 1 -- 172.21.15.63:0/930340722 shutdown_connections 2024-03-23T12:30:30.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.790+0000 7f2664a00700 1 -- 172.21.15.63:0/930340722 wait complete. 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.790+0000 7f2664a00700 1 Processor -- start 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f2664a00700 1 -- start start 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f2664a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2660102cd0 0x7f2660115920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f2664a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660103670 0x7f2660115e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f2664a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26601164b0 con 0x7f2660103670 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f2664a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266010fab0 con 0x7f2660102cd0 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f265d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660103670 0x7f2660115e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:30.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f265d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660103670 0x7f2660115e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34210/0 (socket says 172.21.15.63:34210) 2024-03-23T12:30:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f265d400700 1 -- 172.21.15.63:0/3717624631 learned_addr learned my addr 172.21.15.63:0/3717624631 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.791+0000 7f265de00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2660102cd0 0x7f2660115920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f265d400700 1 -- 172.21.15.63:0/3717624631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2660102cd0 msgr2=0x7f2660115920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f265d400700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2660102cd0 0x7f2660115920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f265d400700 1 -- 172.21.15.63:0/3717624631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2650005040 con 0x7f2660103670 2024-03-23T12:30:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f265de00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2660102cd0 0x7f2660115920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:30:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f265d400700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660103670 0x7f2660115e60 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f264c002a50 tx=0x7f264c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f2656000700 1 -- 172.21.15.63:0/3717624631 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f264c007cc0 con 0x7f2660103670 2024-03-23T12:30:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266010fd90 con 0x7f2660103670 2024-03-23T12:30:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f2656000700 1 -- 172.21.15.63:0/3717624631 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f264c00f040 con 0x7f2660103670 2024-03-23T12:30:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f2656000700 1 -- 172.21.15.63:0/3717624631 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f264c0095d0 con 0x7f2660103670 2024-03-23T12:30:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.792+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2660110350 con 0x7f2660103670 2024-03-23T12:30:30.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.793+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26600fe420 con 0x7f2660103670 2024-03-23T12:30:30.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.794+0000 7f2656000700 1 -- 172.21.15.63:0/3717624631 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f264c009730 con 0x7f2660103670 2024-03-23T12:30:30.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.795+0000 7f2656000700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f264406c7c0 0x7f264406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:30.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.795+0000 7f2656000700 1 -- 172.21.15.63:0/3717624631 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f264c08b720 con 0x7f2660103670 2024-03-23T12:30:30.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.796+0000 7f265de00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f264406c7c0 0x7f264406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:30.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.796+0000 7f265de00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f264406c7c0 0x7f264406ec70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2650000c00 tx=0x7f2650009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:30.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:30.800+0000 7f2656000700 1 -- 172.21.15.63:0/3717624631 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f264c059460 con 0x7f2660103670 2024-03-23T12:30:31.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.025+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2660109c90 con 0x7f2660103670 2024-03-23T12:30:31.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.025+0000 7f2656000700 1 -- 172.21.15.63:0/3717624631 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f264c058ff0 con 0x7f2660103670 2024-03-23T12:30:31.003 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f264406c7c0 msgr2=0x7f264406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f264406c7c0 0x7f264406ec70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2650000c00 tx=0x7f2650009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660103670 msgr2=0x7f2660115e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660103670 0x7f2660115e60 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f264c002a50 tx=0x7f264c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 shutdown_connections 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f264406c7c0 0x7f264406ec70 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2660102cd0 0x7f2660115920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 --2- 172.21.15.63:0/3717624631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2660103670 0x7f2660115e60 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 >> 172.21.15.63:0/3717624631 conn(0x7f26600f8820 msgr2=0x7f2660071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:31.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 shutdown_connections 2024-03-23T12:30:31.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.028+0000 7f2664a00700 1 -- 172.21.15.63:0/3717624631 wait complete. 2024-03-23T12:30:31.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-23T12:30:31.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 -- 172.21.15.63:0/3057252735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070c70 msgr2=0x7fb930108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 --2- 172.21.15.63:0/3057252735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070c70 0x7fb930108fe0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb91c004970 tx=0x7fb91c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:31.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 -- 172.21.15.63:0/3057252735 shutdown_connections 2024-03-23T12:30:31.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 --2- 172.21.15.63:0/3057252735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070c70 0x7fb930108fe0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 --2- 172.21.15.63:0/3057252735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb930070360 0x7fb930070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 -- 172.21.15.63:0/3057252735 >> 172.21.15.63:0/3057252735 conn(0x7fb9300f8820 msgr2=0x7fb9300fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:31.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 -- 172.21.15.63:0/3057252735 shutdown_connections 2024-03-23T12:30:31.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.246+0000 7fb937600700 1 -- 172.21.15.63:0/3057252735 wait complete. 2024-03-23T12:30:31.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.247+0000 7fb937600700 1 Processor -- start 2024-03-23T12:30:31.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.247+0000 7fb937600700 1 -- start start 2024-03-23T12:30:31.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.247+0000 7fb937600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070360 0x7fb930115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.247+0000 7fb937600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb930070c70 0x7fb930115e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.247+0000 7fb937600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9301164c0 con 0x7fb930070360 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.247+0000 7fb937600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93010fac0 con 0x7fb930070c70 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb934c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070360 0x7fb930115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb934c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070360 0x7fb930115930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34232/0 (socket says 172.21.15.63:34232) 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb934c00700 1 -- 172.21.15.63:0/1228541653 learned_addr learned my addr 172.21.15.63:0/1228541653 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb92fe00700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb930070c70 0x7fb930115e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb934c00700 1 -- 172.21.15.63:0/1228541653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb930070c70 msgr2=0x7fb930115e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb934c00700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb930070c70 0x7fb930115e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb934c00700 1 -- 172.21.15.63:0/1228541653 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c005040 con 0x7fb930070360 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb934c00700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070360 0x7fb930115930 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb92400c8c0 tx=0x7fb9240044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb92cc00700 1 -- 172.21.15.63:0/1228541653 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92400a980 con 0x7fb930070360 2024-03-23T12:30:31.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb92cc00700 1 -- 172.21.15.63:0/1228541653 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb92400ce60 con 0x7fb930070360 2024-03-23T12:30:31.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb92cc00700 1 -- 172.21.15.63:0/1228541653 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9240113a0 con 0x7fb930070360 2024-03-23T12:30:31.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.248+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93010fda0 con 0x7fb930070360 2024-03-23T12:30:31.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.249+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9301102b0 con 0x7fb930070360 2024-03-23T12:30:31.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.250+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb930071aa0 con 0x7fb930070360 2024-03-23T12:30:31.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.251+0000 7fb92cc00700 1 -- 172.21.15.63:0/1228541653 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb924012400 con 0x7fb930070360 2024-03-23T12:30:31.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.252+0000 7fb92cc00700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb91806c7c0 0x7fb91806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:31.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.252+0000 7fb92cc00700 1 -- 172.21.15.63:0/1228541653 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb92408bb30 con 0x7fb930070360 2024-03-23T12:30:31.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.252+0000 7fb92fe00700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb91806c7c0 0x7fb91806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:31.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.253+0000 7fb92fe00700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb91806c7c0 0x7fb91806ec70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009fd0 tx=0x7fb91c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:31.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.257+0000 7fb92cc00700 1 -- 172.21.15.63:0/1228541653 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb924059840 con 0x7fb930070360 2024-03-23T12:30:31.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:31 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/649564808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T12:30:31.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:31 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3717624631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T12:30:31.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.471+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fb93010be80 con 0x7fb930070360 2024-03-23T12:30:31.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.471+0000 7fb92cc00700 1 -- 172.21.15.63:0/1228541653 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb9240593d0 con 0x7fb930070360 2024-03-23T12:30:31.449 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T12:30:31.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.473+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb91806c7c0 msgr2=0x7fb91806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.473+0000 7fb937600700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb91806c7c0 0x7fb91806ec70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009fd0 tx=0x7fb91c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:31.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.473+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070360 msgr2=0x7fb930115930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.473+0000 7fb937600700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070360 0x7fb930115930 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb92400c8c0 tx=0x7fb9240044c0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.474+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 shutdown_connections 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.474+0000 7fb937600700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb91806c7c0 0x7fb91806ec70 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.474+0000 7fb937600700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb930070360 0x7fb930115930 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.474+0000 7fb937600700 1 --2- 172.21.15.63:0/1228541653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb930070c70 0x7fb930115e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.474+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 >> 172.21.15.63:0/1228541653 conn(0x7fb9300f8820 msgr2=0x7fb9300f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.474+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 shutdown_connections 2024-03-23T12:30:31.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.474+0000 7fb937600700 1 -- 172.21.15.63:0/1228541653 wait complete. 2024-03-23T12:30:31.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-23T12:30:31.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:31 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/649564808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T12:30:31.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:31 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3717624631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T12:30:31.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 -- 172.21.15.63:0/195631567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 msgr2=0x7f9794103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 --2- 172.21.15.63:0/195631567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794103090 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f9788004940 tx=0x7f9788009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:31.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 -- 172.21.15.63:0/195631567 shutdown_connections 2024-03-23T12:30:31.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 --2- 172.21.15.63:0/195631567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794103660 0x7f97940fccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 --2- 172.21.15.63:0/195631567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794103090 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 -- 172.21.15.63:0/195631567 >> 172.21.15.63:0/195631567 conn(0x7f97940f8830 msgr2=0x7f97940fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:31.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 -- 172.21.15.63:0/195631567 shutdown_connections 2024-03-23T12:30:31.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.689+0000 7f979be00700 1 -- 172.21.15.63:0/195631567 wait complete. 2024-03-23T12:30:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.690+0000 7f979be00700 1 Processor -- start 2024-03-23T12:30:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.690+0000 7f979be00700 1 -- start start 2024-03-23T12:30:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f979be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f979be00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794103660 0x7f97940702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f979be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9794070800 con 0x7f9794102cc0 2024-03-23T12:30:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f979be00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9794070970 con 0x7f9794103660 2024-03-23T12:30:31.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9799400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9799400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794071d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51710/0 (socket says 172.21.15.63:51710) 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9799400700 1 -- 172.21.15.63:0/1414120153 learned_addr learned my addr 172.21.15.63:0/1414120153 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9798a00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794103660 0x7f97940702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9799400700 1 -- 172.21.15.63:0/1414120153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794103660 msgr2=0x7f97940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9799400700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794103660 0x7f97940702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9799400700 1 -- 172.21.15.63:0/1414120153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9788005040 con 0x7f9794102cc0 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.691+0000 7f9798a00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794103660 0x7f97940702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.692+0000 7f9799400700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794071d50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f9788009390 tx=0x7f9788009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.692+0000 7f9791600700 1 -- 172.21.15.63:0/1414120153 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f978800e040 con 0x7f9794102cc0 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.692+0000 7f9791600700 1 -- 172.21.15.63:0/1414120153 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9788013420 con 0x7f9794102cc0 2024-03-23T12:30:31.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.692+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9794070bf0 con 0x7f9794102cc0 2024-03-23T12:30:31.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.692+0000 7f9791600700 1 -- 172.21.15.63:0/1414120153 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f978801c630 con 0x7f9794102cc0 2024-03-23T12:30:31.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.692+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9794071130 con 0x7f9794102cc0 2024-03-23T12:30:31.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.693+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97940fe410 con 0x7f9794102cc0 2024-03-23T12:30:31.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.694+0000 7f9791600700 1 -- 172.21.15.63:0/1414120153 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f97880139e0 con 0x7f9794102cc0 2024-03-23T12:30:31.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.695+0000 7f9791600700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f977c06c7d0 0x7f977c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:31.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.695+0000 7f9791600700 1 -- 172.21.15.63:0/1414120153 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f978808cb70 con 0x7f9794102cc0 2024-03-23T12:30:31.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.695+0000 7f9798a00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f977c06c7d0 0x7f977c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:31.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.696+0000 7f9798a00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f977c06c7d0 0x7f977c06ec80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9780009fd0 tx=0x7f9780009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:31.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.699+0000 7f9791600700 1 -- 172.21.15.63:0/1414120153 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f978805a800 con 0x7f9794102cc0 2024-03-23T12:30:31.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.915+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f9794071130 con 0x7f9794102cc0 2024-03-23T12:30:31.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.915+0000 7f9791600700 1 -- 172.21.15.63:0/1414120153 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9794071130 con 0x7f9794102cc0 2024-03-23T12:30:31.893 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f977c06c7d0 msgr2=0x7f977c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f977c06c7d0 0x7f977c06ec80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9780009fd0 tx=0x7f9780009ee0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 msgr2=0x7f9794071d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794071d50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f9788009390 tx=0x7f9788009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 shutdown_connections 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f977c06c7d0 0x7f977c06ec80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9794102cc0 0x7f9794071d50 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 --2- 172.21.15.63:0/1414120153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9794103660 0x7f97940702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 >> 172.21.15.63:0/1414120153 conn(0x7f97940f8830 msgr2=0x7f97940f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 shutdown_connections 2024-03-23T12:30:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:31.918+0000 7f979be00700 1 -- 172.21.15.63:0/1414120153 wait complete. 2024-03-23T12:30:31.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T12:30:32.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 -- 172.21.15.63:0/2577766514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 msgr2=0x7f3da40fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2577766514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da40fcd10 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c004940 tx=0x7f3d8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:32.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 -- 172.21.15.63:0/2577766514 shutdown_connections 2024-03-23T12:30:32.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2577766514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da40fcd10 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2577766514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da4102d00 0x7f3da41030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 -- 172.21.15.63:0/2577766514 >> 172.21.15.63:0/2577766514 conn(0x7f3da40f8850 msgr2=0x7f3da40fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:32.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 -- 172.21.15.63:0/2577766514 shutdown_connections 2024-03-23T12:30:32.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.130+0000 7f3da9a00700 1 -- 172.21.15.63:0/2577766514 wait complete. 2024-03-23T12:30:32.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.131+0000 7f3da9a00700 1 Processor -- start 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.131+0000 7f3da9a00700 1 -- start start 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.131+0000 7f3da9a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da4102d00 0x7f3da41159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.131+0000 7f3da9a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da4115f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.131+0000 7f3da9a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da4116580 con 0x7f3da41036a0 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.131+0000 7f3da9a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da410fb80 con 0x7f3da4102d00 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3d9be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da4115f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3d9be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da4115f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51720/0 (socket says 172.21.15.63:51720) 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3d9be00700 1 -- 172.21.15.63:0/2764399372 learned_addr learned my addr 172.21.15.63:0/2764399372 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3da2a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da4102d00 0x7f3da41159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3d9be00700 1 -- 172.21.15.63:0/2764399372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da4102d00 msgr2=0x7f3da41159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3d9be00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da4102d00 0x7f3da41159f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3d9be00700 1 -- 172.21.15.63:0/2764399372 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c005040 con 0x7f3da41036a0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.132+0000 7f3d9be00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da4115f30 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009390 tx=0x7f3d8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.133+0000 7f3d9aa00700 1 -- 172.21.15.63:0/2764399372 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d8c00e040 con 0x7f3da41036a0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.133+0000 7f3d9aa00700 1 -- 172.21.15.63:0/2764399372 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d8c013420 con 0x7f3da41036a0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.133+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da410fe00 con 0x7f3da41036a0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.133+0000 7f3d9aa00700 1 -- 172.21.15.63:0/2764399372 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d8c01c630 con 0x7f3da41036a0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.133+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da41103c0 con 0x7f3da41036a0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.134+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da40fe450 con 0x7f3da41036a0 2024-03-23T12:30:32.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.135+0000 7f3d9aa00700 1 -- 172.21.15.63:0/2764399372 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3d8c013a00 con 0x7f3da41036a0 2024-03-23T12:30:32.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.136+0000 7f3d9aa00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d8806c7c0 0x7f3d8806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:32.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.136+0000 7f3d9aa00700 1 -- 172.21.15.63:0/2764399372 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3d8c08cbf0 con 0x7f3da41036a0 2024-03-23T12:30:32.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.136+0000 7f3da2a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d8806c7c0 0x7f3d8806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:32.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.137+0000 7f3da2a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d8806c7c0 0x7f3d8806ec70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3d94004f40 tx=0x7f3d9400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:32.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.141+0000 7f3d9aa00700 1 -- 172.21.15.63:0/2764399372 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3d8c05a880 con 0x7f3da41036a0 2024-03-23T12:30:32.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.438+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3da4109ca0 con 0x7f3da41036a0 2024-03-23T12:30:32.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.439+0000 7f3d9aa00700 1 -- 172.21.15.63:0/2764399372 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4861 (secure 0 0 0) 0x7f3d8c05a410 con 0x7f3da41036a0 2024-03-23T12:30:32.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:32.417 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:42.227161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:active","state_seq":470,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:active","state_seq":465,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44393":{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_45001":{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:32.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.441+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d8806c7c0 msgr2=0x7f3d8806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.441+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d8806c7c0 0x7f3d8806ec70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3d94004f40 tx=0x7f3d9400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:32.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.441+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 msgr2=0x7f3da4115f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.441+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da4115f30 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009390 tx=0x7f3d8c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:32.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.442+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 shutdown_connections 2024-03-23T12:30:32.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.442+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f3d8806c7c0 0x7f3d8806ec70 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.442+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da4102d00 0x7f3da41159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.442+0000 7f3da9a00700 1 --2- 172.21.15.63:0/2764399372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3da41036a0 0x7f3da4115f30 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.442+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 >> 172.21.15.63:0/2764399372 conn(0x7f3da40f8850 msgr2=0x7f3da40706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:32.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.442+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 shutdown_connections 2024-03-23T12:30:32.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.442+0000 7f3da9a00700 1 -- 172.21.15.63:0/2764399372 wait complete. 2024-03-23T12:30:32.420 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 43 2024-03-23T12:30:32.434 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:32 smithi063 ceph-mon[141124]: pgmap v1185: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:32.434 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:32 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1228541653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T12:30:32.434 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:32 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1414120153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T12:30:32.436 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 20, 'max_mds': 2, 'flags': 50} 2024-03-23T12:30:32.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 -- 172.21.15.63:0/992606769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc102cd0 msgr2=0x7fc3bc1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 --2- 172.21.15.63:0/992606769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc102cd0 0x7fc3bc1030a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac004940 tx=0x7fc3ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 -- 172.21.15.63:0/992606769 shutdown_connections 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 --2- 172.21.15.63:0/992606769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3bc103670 0x7fc3bc0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 --2- 172.21.15.63:0/992606769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc102cd0 0x7fc3bc1030a0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 -- 172.21.15.63:0/992606769 >> 172.21.15.63:0/992606769 conn(0x7fc3bc0f8820 msgr2=0x7fc3bc0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 -- 172.21.15.63:0/992606769 shutdown_connections 2024-03-23T12:30:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.655+0000 7fc3c2400700 1 -- 172.21.15.63:0/992606769 wait complete. 2024-03-23T12:30:32.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.656+0000 7fc3c2400700 1 Processor -- start 2024-03-23T12:30:32.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.656+0000 7fc3c2400700 1 -- start start 2024-03-23T12:30:32.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.656+0000 7fc3c2400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3bc102cd0 0x7fc3bc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:32.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.656+0000 7fc3c2400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc103670 0x7fc3bc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:32.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.656+0000 7fc3c2400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc072210 con 0x7fc3bc103670 2024-03-23T12:30:32.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.656+0000 7fc3c2400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc070d40 con 0x7fc3bc102cd0 2024-03-23T12:30:32.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc103670 0x7fc3bc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc103670 0x7fc3bc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51738/0 (socket says 172.21.15.63:51738) 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3e00700 1 -- 172.21.15.63:0/2510700634 learned_addr learned my addr 172.21.15.63:0/2510700634 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3e00700 1 -- 172.21.15.63:0/2510700634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3bc102cd0 msgr2=0x7fc3bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3e00700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3bc102cd0 0x7fc3bc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3e00700 1 -- 172.21.15.63:0/2510700634 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac005040 con 0x7fc3bc103670 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3e00700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc103670 0x7fc3bc070800 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400c890 tx=0x7fc3a40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.657+0000 7fc3b3400700 1 -- 172.21.15.63:0/2510700634 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3a400a7d0 con 0x7fc3bc103670 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.658+0000 7fc3b3400700 1 -- 172.21.15.63:0/2510700634 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3a400ade0 con 0x7fc3bc103670 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.658+0000 7fc3b3400700 1 -- 172.21.15.63:0/2510700634 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3a40113a0 con 0x7fc3bc103670 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.658+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3bc071020 con 0x7fc3bc103670 2024-03-23T12:30:32.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.658+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc119f10 con 0x7fc3bc103670 2024-03-23T12:30:32.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.659+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3bc0fe420 con 0x7fc3bc103670 2024-03-23T12:30:32.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.660+0000 7fc3b3400700 1 -- 172.21.15.63:0/2510700634 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc3a4012400 con 0x7fc3bc103670 2024-03-23T12:30:32.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.661+0000 7fc3b3400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc3a006c7c0 0x7fc3a006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:32.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.661+0000 7fc3b3400700 1 -- 172.21.15.63:0/2510700634 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc3a408bbe0 con 0x7fc3bc103670 2024-03-23T12:30:32.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.661+0000 7fc3bb400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc3a006c7c0 0x7fc3a006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:32.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.661+0000 7fc3bb400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc3a006c7c0 0x7fc3a006ec70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac004f90 tx=0x7fc3ac009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:32.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.665+0000 7fc3b3400700 1 -- 172.21.15.63:0/2510700634 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc3a4055de0 con 0x7fc3bc103670 2024-03-23T12:30:32.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:32 smithi111 ceph-mon[106899]: pgmap v1185: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:32.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:32 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1228541653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T12:30:32.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:32 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1414120153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T12:30:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.928+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fc3bc064970 con 0x7fc3bc103670 2024-03-23T12:30:32.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.928+0000 7fc3b3400700 1 -- 172.21.15.63:0/2510700634 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v43) v1 ==== 107+0+4860 (secure 0 0 0) 0x7fc3a4059400 con 0x7fc3bc103670 2024-03-23T12:30:32.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:32.906 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:46:24.073898+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14462,"mds_1":14478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.63:6827/3230334326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3230334326},{"type":"v1","addr":"172.21.15.63:6827","nonce":3230334326}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14478":{"gid":14478,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.63:6829/2516647170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2516647170},{"type":"v1","addr":"172.21.15.63:6829","nonce":2516647170}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi111.atkibb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6827/3582848147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3582848147},{"type":"v1","addr":"172.21.15.111:6827","nonce":3582848147}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24299":{"gid":24299,"name":"cephfs.smithi111.ylnmzy","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6825/315773837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":315773837},{"type":"v1","addr":"172.21.15.111:6825","nonce":315773837}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:32.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc3a006c7c0 msgr2=0x7fc3a006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc3a006c7c0 0x7fc3a006ec70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac004f90 tx=0x7fc3ac009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc103670 msgr2=0x7fc3bc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc103670 0x7fc3bc070800 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400c890 tx=0x7fc3a40044c0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 shutdown_connections 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc3a006c7c0 0x7fc3a006ec70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3bc102cd0 0x7fc3bc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 --2- 172.21.15.63:0/2510700634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3bc103670 0x7fc3bc070800 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 >> 172.21.15.63:0/2510700634 conn(0x7fc3bc0f8820 msgr2=0x7fc3bc0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 shutdown_connections 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:32.931+0000 7fc3c2400700 1 -- 172.21.15.63:0/2510700634 wait complete. 2024-03-23T12:30:32.909 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 21 2024-03-23T12:30:32.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-03-23T12:30:33.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 -- 172.21.15.63:0/2732847835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 msgr2=0x7f05300fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 --2- 172.21.15.63:0/2732847835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f05300fcce0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f051c004940 tx=0x7f051c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:33.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 -- 172.21.15.63:0/2732847835 shutdown_connections 2024-03-23T12:30:33.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 --2- 172.21.15.63:0/2732847835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f05300fcce0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 --2- 172.21.15.63:0/2732847835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0530102cd0 0x7f05301030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 -- 172.21.15.63:0/2732847835 >> 172.21.15.63:0/2732847835 conn(0x7f05300f8820 msgr2=0x7f05300fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:33.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 -- 172.21.15.63:0/2732847835 shutdown_connections 2024-03-23T12:30:33.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.144+0000 7f0536800700 1 -- 172.21.15.63:0/2732847835 wait complete. 2024-03-23T12:30:33.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f0536800700 1 Processor -- start 2024-03-23T12:30:33.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f0536800700 1 -- start start 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f0536800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0530102cd0 0x7f0530115910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f0536800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f0530115e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f0536800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05301164a0 con 0x7f0530103670 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f0536800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053010faa0 con 0x7f0530102cd0 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f052ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f0530115e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f052ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f0530115e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51756/0 (socket says 172.21.15.63:51756) 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.145+0000 7f052ea00700 1 -- 172.21.15.63:0/3405773244 learned_addr learned my addr 172.21.15.63:0/3405773244 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f052f400700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0530102cd0 0x7f0530115910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:33.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f052ea00700 1 -- 172.21.15.63:0/3405773244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0530102cd0 msgr2=0x7f0530115910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f052ea00700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0530102cd0 0x7f0530115910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f052ea00700 1 -- 172.21.15.63:0/3405773244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c005040 con 0x7f0530103670 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f052ea00700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f0530115e50 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f051c009390 tx=0x7f051c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f0527400700 1 -- 172.21.15.63:0/3405773244 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f051c00e040 con 0x7f0530103670 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f0527400700 1 -- 172.21.15.63:0/3405773244 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f051c013420 con 0x7f0530103670 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f0527400700 1 -- 172.21.15.63:0/3405773244 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f051c01c630 con 0x7f0530103670 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053010fd20 con 0x7f0530103670 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.146+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05301102e0 con 0x7f0530103670 2024-03-23T12:30:33.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.147+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05300fe420 con 0x7f0530103670 2024-03-23T12:30:33.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.148+0000 7f0527400700 1 -- 172.21.15.63:0/3405773244 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f051c013a00 con 0x7f0530103670 2024-03-23T12:30:33.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.149+0000 7f0527400700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f051406c7c0 0x7f051406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:33.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.149+0000 7f0527400700 1 -- 172.21.15.63:0/3405773244 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f051c08cc20 con 0x7f0530103670 2024-03-23T12:30:33.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.150+0000 7f052f400700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f051406c7c0 0x7f051406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:33.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.150+0000 7f052f400700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f051406c7c0 0x7f051406ec70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0520009fd0 tx=0x7f052000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:33.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.154+0000 7f0527400700 1 -- 172.21.15.63:0/3405773244 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f051c05a8b0 con 0x7f0530103670 2024-03-23T12:30:33.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.416+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f0530109c70 con 0x7f0530103670 2024-03-23T12:30:33.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.417+0000 7f0527400700 1 -- 172.21.15.63:0/3405773244 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v43) v1 ==== 107+0+4861 (secure 0 0 0) 0x7f051c05a440 con 0x7f0530103670 2024-03-23T12:30:33.395 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:33.395 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:47:16.499928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14462,"mds_1":14478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.63:6827/3230334326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3230334326},{"type":"v1","addr":"172.21.15.63:6827","nonce":3230334326}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14478":{"gid":14478,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.63:6829/2516647170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2516647170},{"type":"v1","addr":"172.21.15.63:6829","nonce":2516647170}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi111.atkibb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6827/3582848147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3582848147},{"type":"v1","addr":"172.21.15.111:6827","nonce":3582848147}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24299":{"gid":24299,"name":"cephfs.smithi111.ylnmzy","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6825/315773837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":315773837},{"type":"v1","addr":"172.21.15.111:6825","nonce":315773837}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:33.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.419+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f051406c7c0 msgr2=0x7f051406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.419+0000 7f0536800700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f051406c7c0 0x7f051406ec70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0520009fd0 tx=0x7f052000b040 comp rx=0 tx=0).stop 2024-03-23T12:30:33.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.419+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 msgr2=0x7f0530115e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.419+0000 7f0536800700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f0530115e50 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f051c009390 tx=0x7f051c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:33.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.419+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 shutdown_connections 2024-03-23T12:30:33.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.420+0000 7f0536800700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f051406c7c0 0x7f051406ec70 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.420+0000 7f0536800700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0530102cd0 0x7f0530115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.420+0000 7f0536800700 1 --2- 172.21.15.63:0/3405773244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0530103670 0x7f0530115e50 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.420+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 >> 172.21.15.63:0/3405773244 conn(0x7f05300f8820 msgr2=0x7f05300f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:33.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.420+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 shutdown_connections 2024-03-23T12:30:33.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.420+0000 7f0536800700 1 -- 172.21.15.63:0/3405773244 wait complete. 2024-03-23T12:30:33.398 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 22 2024-03-23T12:30:33.415 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-03-23T12:30:33.475 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:33 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2764399372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T12:30:33.475 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:33 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2510700634' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 -- 172.21.15.63:0/3517120411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 msgr2=0x7f4d4c1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 --2- 172.21.15.63:0/3517120411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c1030b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c004940 tx=0x7f4d3c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 -- 172.21.15.63:0/3517120411 shutdown_connections 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 --2- 172.21.15.63:0/3517120411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c103680 0x7f4d4c0fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 --2- 172.21.15.63:0/3517120411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c1030b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 -- 172.21.15.63:0/3517120411 >> 172.21.15.63:0/3517120411 conn(0x7f4d4c0f8830 msgr2=0x7f4d4c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 -- 172.21.15.63:0/3517120411 shutdown_connections 2024-03-23T12:30:33.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.633+0000 7f4d52000700 1 -- 172.21.15.63:0/3517120411 wait complete. 2024-03-23T12:30:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.634+0000 7f4d52000700 1 Processor -- start 2024-03-23T12:30:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.634+0000 7f4d52000700 1 -- start start 2024-03-23T12:30:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d52000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d52000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c103680 0x7f4d4c115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d52000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c116510 con 0x7f4d4c102ce0 2024-03-23T12:30:33.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d52000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c10fb10 con 0x7f4d4c103680 2024-03-23T12:30:33.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d4b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:33.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d4b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c115980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51772/0 (socket says 172.21.15.63:51772) 2024-03-23T12:30:33.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d4b400700 1 -- 172.21.15.63:0/2186788728 learned_addr learned my addr 172.21.15.63:0/2186788728 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d43e00700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c103680 0x7f4d4c115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d4b400700 1 -- 172.21.15.63:0/2186788728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c103680 msgr2=0x7f4d4c115ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d4b400700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c103680 0x7f4d4c115ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d4b400700 1 -- 172.21.15.63:0/2186788728 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3c005040 con 0x7f4d4c102ce0 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.635+0000 7f4d4b400700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c115980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c008d80 tx=0x7f4d3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.636+0000 7f4d43400700 1 -- 172.21.15.63:0/2186788728 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d3c00e030 con 0x7f4d4c102ce0 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.636+0000 7f4d43400700 1 -- 172.21.15.63:0/2186788728 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d3c013de0 con 0x7f4d4c102ce0 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.636+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c10fd90 con 0x7f4d4c102ce0 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.636+0000 7f4d43400700 1 -- 172.21.15.63:0/2186788728 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d3c01cd60 con 0x7f4d4c102ce0 2024-03-23T12:30:33.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.636+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c110350 con 0x7f4d4c102ce0 2024-03-23T12:30:33.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.637+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c0fe430 con 0x7f4d4c102ce0 2024-03-23T12:30:33.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.638+0000 7f4d43400700 1 -- 172.21.15.63:0/2186788728 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4d3c01c590 con 0x7f4d4c102ce0 2024-03-23T12:30:33.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.639+0000 7f4d43400700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4d3006c7c0 0x7f4d3006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:33.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.639+0000 7f4d43400700 1 -- 172.21.15.63:0/2186788728 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4d3c08d3e0 con 0x7f4d4c102ce0 2024-03-23T12:30:33.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.639+0000 7f4d43e00700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4d3006c7c0 0x7f4d3006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:33.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.640+0000 7f4d43e00700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4d3006c7c0 0x7f4d3006ec70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4d34004600 tx=0x7f4d3400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:33.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.643+0000 7f4d43400700 1 -- 172.21.15.63:0/2186788728 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4d3c05b070 con 0x7f4d4c102ce0 2024-03-23T12:30:33.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:33 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2764399372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-23T12:30:33.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:33 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2510700634' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-23T12:30:33.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.910+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f4d4c109ca0 con 0x7f4d4c102ce0 2024-03-23T12:30:33.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.911+0000 7f4d43400700 1 -- 172.21.15.63:0/2186788728 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v43) v1 ==== 107+0+4860 (secure 0 0 0) 0x7f4d3c05ac00 con 0x7f4d4c102ce0 2024-03-23T12:30:33.888 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:33.888 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:51:34.569669+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14462,"mds_1":14478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.63:6827/3230334326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3230334326},{"type":"v1","addr":"172.21.15.63:6827","nonce":3230334326}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14478":{"gid":14478,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.63:6829/2516647170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2516647170},{"type":"v1","addr":"172.21.15.63:6829","nonce":2516647170}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi111.atkibb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6827/3582848147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3582848147},{"type":"v1","addr":"172.21.15.111:6827","nonce":3582848147}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24299":{"gid":24299,"name":"cephfs.smithi111.ylnmzy","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6825/315773837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":315773837},{"type":"v1","addr":"172.21.15.111:6825","nonce":315773837}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4d3006c7c0 msgr2=0x7f4d3006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4d3006c7c0 0x7f4d3006ec70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4d34004600 tx=0x7f4d3400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 msgr2=0x7f4d4c115980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c115980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c008d80 tx=0x7f4d3c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 shutdown_connections 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4d3006c7c0 0x7f4d3006ec70 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c102ce0 0x7f4d4c115980 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 --2- 172.21.15.63:0/2186788728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c103680 0x7f4d4c115ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:33.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.913+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 >> 172.21.15.63:0/2186788728 conn(0x7f4d4c0f8830 msgr2=0x7f4d4c0f97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:33.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.914+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 shutdown_connections 2024-03-23T12:30:33.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:33.914+0000 7f4d52000700 1 -- 172.21.15.63:0/2186788728 wait complete. 2024-03-23T12:30:33.892 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-03-23T12:30:33.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-03-23T12:30:34.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 -- 172.21.15.63:0/3478570257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720103680 msgr2=0x7f37200fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 --2- 172.21.15.63:0/3478570257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720103680 0x7f37200fccf0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f370c004940 tx=0x7f370c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 -- 172.21.15.63:0/3478570257 shutdown_connections 2024-03-23T12:30:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 --2- 172.21.15.63:0/3478570257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720103680 0x7f37200fccf0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 --2- 172.21.15.63:0/3478570257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3720102ce0 0x7f37201030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 -- 172.21.15.63:0/3478570257 >> 172.21.15.63:0/3478570257 conn(0x7f37200f8830 msgr2=0x7f37200fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 -- 172.21.15.63:0/3478570257 shutdown_connections 2024-03-23T12:30:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.125+0000 7f3728000700 1 -- 172.21.15.63:0/3478570257 wait complete. 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3728000700 1 Processor -- start 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3728000700 1 -- start start 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3728000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720102ce0 0x7f3720115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3728000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3720103680 0x7f3720115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3728000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3720116510 con 0x7f3720102ce0 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3728000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372010fb10 con 0x7f3720103680 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3725600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720102ce0 0x7f3720115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3725600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720102ce0 0x7f3720115980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51788/0 (socket says 172.21.15.63:51788) 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.126+0000 7f3725600700 1 -- 172.21.15.63:0/1230127283 learned_addr learned my addr 172.21.15.63:0/1230127283 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f3724c00700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3720103680 0x7f3720115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f3725600700 1 -- 172.21.15.63:0/1230127283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3720103680 msgr2=0x7f3720115ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f3725600700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3720103680 0x7f3720115ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f3725600700 1 -- 172.21.15.63:0/1230127283 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c005040 con 0x7f3720102ce0 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f3725600700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720102ce0 0x7f3720115980 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3714002a50 tx=0x7f37140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f371d600700 1 -- 172.21.15.63:0/1230127283 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3714007cc0 con 0x7f3720102ce0 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f371d600700 1 -- 172.21.15.63:0/1230127283 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f371400f040 con 0x7f3720102ce0 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372010fdf0 con 0x7f3720102ce0 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.127+0000 7f371d600700 1 -- 172.21.15.63:0/1230127283 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371400d9c0 con 0x7f3720102ce0 2024-03-23T12:30:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.128+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3720110380 con 0x7f3720102ce0 2024-03-23T12:30:34.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.129+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37200fe430 con 0x7f3720102ce0 2024-03-23T12:30:34.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.129+0000 7f371d600700 1 -- 172.21.15.63:0/1230127283 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f371400db20 con 0x7f3720102ce0 2024-03-23T12:30:34.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.130+0000 7f371d600700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370806c7c0 0x7f370806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:34.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.130+0000 7f371d600700 1 -- 172.21.15.63:0/1230127283 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f371408b760 con 0x7f3720102ce0 2024-03-23T12:30:34.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.131+0000 7f3724c00700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370806c7c0 0x7f370806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:34.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.131+0000 7f3724c00700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370806c7c0 0x7f370806ec70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3720110df0 tx=0x7f370c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:34.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.135+0000 7f371d600700 1 -- 172.21.15.63:0/1230127283 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f37140593f0 con 0x7f3720102ce0 2024-03-23T12:30:34.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.396+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f3720109cd0 con 0x7f3720102ce0 2024-03-23T12:30:34.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.397+0000 7f371d600700 1 -- 172.21.15.63:0/1230127283 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v43) v1 ==== 107+0+4859 (secure 0 0 0) 0x7f3714058f80 con 0x7f3720102ce0 2024-03-23T12:30:34.375 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:34.375 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:51:47.612215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14462,"mds_1":14478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.63:6827/3230334326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3230334326},{"type":"v1","addr":"172.21.15.63:6827","nonce":3230334326}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14478":{"gid":14478,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.63:6829/2516647170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2516647170},{"type":"v1","addr":"172.21.15.63:6829","nonce":2516647170}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi111.atkibb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6827/3582848147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3582848147},{"type":"v1","addr":"172.21.15.111:6827","nonce":3582848147}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24299":{"gid":24299,"name":"cephfs.smithi111.ylnmzy","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6825/315773837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":315773837},{"type":"v1","addr":"172.21.15.111:6825","nonce":315773837}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:34.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370806c7c0 msgr2=0x7f370806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370806c7c0 0x7f370806ec70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3720110df0 tx=0x7f370c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720102ce0 msgr2=0x7f3720115980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720102ce0 0x7f3720115980 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3714002a50 tx=0x7f37140046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 shutdown_connections 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f370806c7c0 0x7f370806ec70 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3720102ce0 0x7f3720115980 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 --2- 172.21.15.63:0/1230127283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3720103680 0x7f3720115ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 >> 172.21.15.63:0/1230127283 conn(0x7f37200f8830 msgr2=0x7f37200f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 shutdown_connections 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.400+0000 7f3728000700 1 -- 172.21.15.63:0/1230127283 wait complete. 2024-03-23T12:30:34.378 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-03-23T12:30:34.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-03-23T12:30:34.455 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:34 smithi063 ceph-mon[141124]: pgmap v1186: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:34.455 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:34 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3405773244' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-23T12:30:34.455 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:34 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2186788728' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-23T12:30:34.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.612+0000 7fe772000700 1 -- 172.21.15.63:0/1739876020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 msgr2=0x7fe76c1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.612+0000 7fe772000700 1 --2- 172.21.15.63:0/1739876020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c1030a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe75c004940 tx=0x7fe75c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:34.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.613+0000 7fe772000700 1 -- 172.21.15.63:0/1739876020 shutdown_connections 2024-03-23T12:30:34.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.613+0000 7fe772000700 1 --2- 172.21.15.63:0/1739876020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe76c103670 0x7fe76c0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.613+0000 7fe772000700 1 --2- 172.21.15.63:0/1739876020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c1030a0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.613+0000 7fe772000700 1 -- 172.21.15.63:0/1739876020 >> 172.21.15.63:0/1739876020 conn(0x7fe76c0f8820 msgr2=0x7fe76c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:34.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.613+0000 7fe772000700 1 -- 172.21.15.63:0/1739876020 shutdown_connections 2024-03-23T12:30:34.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.613+0000 7fe772000700 1 -- 172.21.15.63:0/1739876020 wait complete. 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.613+0000 7fe772000700 1 Processor -- start 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe772000700 1 -- start start 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe772000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe772000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe76c103670 0x7fe76c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe772000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c072210 con 0x7fe76c102cd0 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe772000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c070d40 con 0x7fe76c103670 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe76b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe76b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51804/0 (socket says 172.21.15.63:51804) 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe76b400700 1 -- 172.21.15.63:0/1966894789 learned_addr learned my addr 172.21.15.63:0/1966894789 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.614+0000 7fe763e00700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe76c103670 0x7fe76c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe76b400700 1 -- 172.21.15.63:0/1966894789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe76c103670 msgr2=0x7fe76c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe76b400700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe76c103670 0x7fe76c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe76b400700 1 -- 172.21.15.63:0/1966894789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c005040 con 0x7fe76c102cd0 2024-03-23T12:30:34.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe76b400700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c0702c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe75c007f50 tx=0x7fe75c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:34.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe763400700 1 -- 172.21.15.63:0/1966894789 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75c021070 con 0x7fe76c102cd0 2024-03-23T12:30:34.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe763400700 1 -- 172.21.15.63:0/1966894789 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe75c0093b0 con 0x7fe76c102cd0 2024-03-23T12:30:34.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c070fc0 con 0x7fe76c102cd0 2024-03-23T12:30:34.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe763400700 1 -- 172.21.15.63:0/1966894789 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75c00e070 con 0x7fe76c102cd0 2024-03-23T12:30:34.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.615+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c119e90 con 0x7fe76c102cd0 2024-03-23T12:30:34.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.616+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c0fe420 con 0x7fe76c102cd0 2024-03-23T12:30:34.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.617+0000 7fe763400700 1 -- 172.21.15.63:0/1966894789 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe75c01c400 con 0x7fe76c102cd0 2024-03-23T12:30:34.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.618+0000 7fe763400700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe75006c7c0 0x7fe75006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:34.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.618+0000 7fe763400700 1 -- 172.21.15.63:0/1966894789 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe75c08cc30 con 0x7fe76c102cd0 2024-03-23T12:30:34.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.618+0000 7fe763e00700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe75006c7c0 0x7fe75006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:34.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.619+0000 7fe763e00700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe75006c7c0 0x7fe75006ec70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0719f0 tx=0x7fe75400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:34.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.622+0000 7fe763400700 1 -- 172.21.15.63:0/1966894789 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe75c056e30 con 0x7fe76c102cd0 2024-03-23T12:30:34.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:34 smithi111 ceph-mon[106899]: pgmap v1186: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:34.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:34 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3405773244' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-23T12:30:34.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:34 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2186788728' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-23T12:30:34.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.888+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7fe76c11a020 con 0x7fe76c102cd0 2024-03-23T12:30:34.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.889+0000 7fe763400700 1 -- 172.21.15.63:0/1966894789 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v43) v1 ==== 107+0+3284 (secure 0 0 0) 0x7fe75c05a450 con 0x7fe76c102cd0 2024-03-23T12:30:34.866 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:34.866 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.173990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14462,"mds_1":14478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.63:6827/3230334326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":3230334326},{"type":"v1","addr":"172.21.15.63:6827","nonce":3230334326}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14478":{"gid":14478,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":15,"state":"up:active","state_seq":6,"addr":"172.21.15.63:6829/2516647170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2516647170},{"type":"v1","addr":"172.21.15.63:6829","nonce":2516647170}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:34.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe75006c7c0 msgr2=0x7fe75006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe75006c7c0 0x7fe75006ec70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0719f0 tx=0x7fe75400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:34.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 msgr2=0x7fe76c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:34.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c0702c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe75c007f50 tx=0x7fe75c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 shutdown_connections 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe75006c7c0 0x7fe75006ec70 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe76c102cd0 0x7fe76c0702c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 --2- 172.21.15.63:0/1966894789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe76c103670 0x7fe76c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.891+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 >> 172.21.15.63:0/1966894789 conn(0x7fe76c0f8820 msgr2=0x7fe76c0f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.892+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 shutdown_connections 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:34.892+0000 7fe772000700 1 -- 172.21.15.63:0/1966894789 wait complete. 2024-03-23T12:30:34.870 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 25 2024-03-23T12:30:34.887 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-03-23T12:30:34.887 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-03-23T12:30:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 -- 172.21.15.63:0/2542183946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c103670 msgr2=0x7f4c0c0fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2542183946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c103670 0x7f4c0c0fcce0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4004940 tx=0x7f4bf4009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 -- 172.21.15.63:0/2542183946 shutdown_connections 2024-03-23T12:30:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2542183946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c103670 0x7f4c0c0fcce0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2542183946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102cd0 0x7f4c0c1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 -- 172.21.15.63:0/2542183946 >> 172.21.15.63:0/2542183946 conn(0x7f4c0c0f8820 msgr2=0x7f4c0c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 -- 172.21.15.63:0/2542183946 shutdown_connections 2024-03-23T12:30:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.104+0000 7f4c11c00700 1 -- 172.21.15.63:0/2542183946 wait complete. 2024-03-23T12:30:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c11c00700 1 Processor -- start 2024-03-23T12:30:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c11c00700 1 -- start start 2024-03-23T12:30:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c11c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c102cd0 0x7f4c0c115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c11c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c103670 0x7f4c0c115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c11c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c1165a0 con 0x7f4c0c102cd0 2024-03-23T12:30:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c11c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c10fba0 con 0x7f4c0c103670 2024-03-23T12:30:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c0aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c102cd0 0x7f4c0c115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c0aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c102cd0 0x7f4c0c115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51824/0 (socket says 172.21.15.63:51824) 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.105+0000 7f4c0aa00700 1 -- 172.21.15.63:0/2710749536 learned_addr learned my addr 172.21.15.63:0/2710749536 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c03e00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c103670 0x7f4c0c115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c0aa00700 1 -- 172.21.15.63:0/2710749536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c103670 msgr2=0x7f4c0c115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c0aa00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c103670 0x7f4c0c115f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c0aa00700 1 -- 172.21.15.63:0/2710749536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf4005040 con 0x7f4c0c102cd0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c0aa00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c102cd0 0x7f4c0c115a10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc002a50 tx=0x7f4bfc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c02a00700 1 -- 172.21.15.63:0/2710749536 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bfc007cc0 con 0x7f4c0c102cd0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c02a00700 1 -- 172.21.15.63:0/2710749536 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bfc00f040 con 0x7f4c0c102cd0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0c10fe80 con 0x7f4c0c102cd0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.106+0000 7f4c02a00700 1 -- 172.21.15.63:0/2710749536 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bfc009540 con 0x7f4c0c102cd0 2024-03-23T12:30:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.107+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0c1103c0 con 0x7f4c0c102cd0 2024-03-23T12:30:35.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.107+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0c0fe420 con 0x7f4c0c102cd0 2024-03-23T12:30:35.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.108+0000 7f4c02a00700 1 -- 172.21.15.63:0/2710749536 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f4bfc0096a0 con 0x7f4c0c102cd0 2024-03-23T12:30:35.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.109+0000 7f4c02a00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4bf006c7c0 0x7f4bf006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:35.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.109+0000 7f4c02a00700 1 -- 172.21.15.63:0/2710749536 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4bfc08b8e0 con 0x7f4c0c102cd0 2024-03-23T12:30:35.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.110+0000 7f4c03e00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4bf006c7c0 0x7f4bf006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:35.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.110+0000 7f4c03e00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4bf006c7c0 0x7f4bf006ec70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c111240 tx=0x7f4bf4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:35.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.114+0000 7f4c02a00700 1 -- 172.21.15.63:0/2710749536 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4bfc059530 con 0x7f4c0c102cd0 2024-03-23T12:30:35.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.382+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f4c0c110ff0 con 0x7f4c0c102cd0 2024-03-23T12:30:35.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.383+0000 7f4c02a00700 1 -- 172.21.15.63:0/2710749536 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v43) v1 ==== 107+0+3269 (secure 0 0 0) 0x7f4bfc0590c0 con 0x7f4c0c102cd0 2024-03-23T12:30:35.360 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:35.360 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/4216607936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4216607936},{"type":"v1","addr":"172.21.15.111:6826","nonce":4216607936}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:35.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4bf006c7c0 msgr2=0x7f4bf006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4bf006c7c0 0x7f4bf006ec70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c111240 tx=0x7f4bf4009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:35.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c102cd0 msgr2=0x7f4c0c115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c102cd0 0x7f4c0c115a10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc002a50 tx=0x7f4bfc0046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 shutdown_connections 2024-03-23T12:30:35.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f4bf006c7c0 0x7f4bf006ec70 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c0c102cd0 0x7f4c0c115a10 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 --2- 172.21.15.63:0/2710749536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c103670 0x7f4c0c115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 >> 172.21.15.63:0/2710749536 conn(0x7f4c0c0f8820 msgr2=0x7f4c0c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:35.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.385+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 shutdown_connections 2024-03-23T12:30:35.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.386+0000 7f4c11c00700 1 -- 172.21.15.63:0/2710749536 wait complete. 2024-03-23T12:30:35.364 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 26 2024-03-23T12:30:35.381 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:35.381 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-03-23T12:30:35.381 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-03-23T12:30:35.440 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:35 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1230127283' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-23T12:30:35.441 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:35 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1966894789' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.599+0000 7f20e8400700 1 -- 172.21.15.63:0/2100568782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0103670 msgr2=0x7f20e00fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.599+0000 7f20e8400700 1 --2- 172.21.15.63:0/2100568782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0103670 0x7f20e00fcce0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f20cc004940 tx=0x7f20cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.600+0000 7f20e8400700 1 -- 172.21.15.63:0/2100568782 shutdown_connections 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.600+0000 7f20e8400700 1 --2- 172.21.15.63:0/2100568782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0103670 0x7f20e00fcce0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.600+0000 7f20e8400700 1 --2- 172.21.15.63:0/2100568782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0102cd0 0x7f20e01030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.600+0000 7f20e8400700 1 -- 172.21.15.63:0/2100568782 >> 172.21.15.63:0/2100568782 conn(0x7f20e00f8820 msgr2=0x7f20e00fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.600+0000 7f20e8400700 1 -- 172.21.15.63:0/2100568782 shutdown_connections 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.600+0000 7f20e8400700 1 -- 172.21.15.63:0/2100568782 wait complete. 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.600+0000 7f20e8400700 1 Processor -- start 2024-03-23T12:30:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e8400700 1 -- start start 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e8400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102cd0 0x7f20e00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e8400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0103670 0x7f20e0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e8400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0072210 con 0x7f20e0102cd0 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e8400700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0070d40 con 0x7f20e0103670 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102cd0 0x7f20e00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102cd0 0x7f20e00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51842/0 (socket says 172.21.15.63:51842) 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e5a00700 1 -- 172.21.15.63:0/1679990591 learned_addr learned my addr 172.21.15.63:0/1679990591 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e5000700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0103670 0x7f20e0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e5a00700 1 -- 172.21.15.63:0/1679990591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0103670 msgr2=0x7f20e0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.601+0000 7f20e5a00700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0103670 0x7f20e0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.602+0000 7f20e5a00700 1 -- 172.21.15.63:0/1679990591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc005040 con 0x7f20e0102cd0 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.602+0000 7f20e5a00700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102cd0 0x7f20e00702c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f20d400eb40 tx=0x7f20d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:35.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.602+0000 7f20dd600700 1 -- 172.21.15.63:0/1679990591 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d4007cc0 con 0x7f20e0102cd0 2024-03-23T12:30:35.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.602+0000 7f20dd600700 1 -- 172.21.15.63:0/1679990591 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20d400f040 con 0x7f20e0102cd0 2024-03-23T12:30:35.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.602+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e0070fc0 con 0x7f20e0102cd0 2024-03-23T12:30:35.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.602+0000 7f20dd600700 1 -- 172.21.15.63:0/1679990591 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d4009570 con 0x7f20e0102cd0 2024-03-23T12:30:35.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.602+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e0119e70 con 0x7f20e0102cd0 2024-03-23T12:30:35.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.604+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e00fe420 con 0x7f20e0102cd0 2024-03-23T12:30:35.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.604+0000 7f20dd600700 1 -- 172.21.15.63:0/1679990591 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f20d40096d0 con 0x7f20e0102cd0 2024-03-23T12:30:35.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.605+0000 7f20dd600700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:35.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.605+0000 7f20dd600700 1 -- 172.21.15.63:0/1679990591 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f20d408b6c0 con 0x7f20e0102cd0 2024-03-23T12:30:35.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.605+0000 7f20e5000700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:35.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.606+0000 7f20e5000700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009fd0 tx=0x7f20cc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:35.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.610+0000 7f20dd600700 1 -- 172.21.15.63:0/1679990591 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f20d4055970 con 0x7f20e0102cd0 2024-03-23T12:30:35.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:35 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1230127283' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-23T12:30:35.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:35 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1966894789' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-23T12:30:35.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.881+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f20e011a0c0 con 0x7f20e0102cd0 2024-03-23T12:30:35.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.882+0000 7f20dd600700 1 -- 172.21.15.63:0/1679990591 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v43) v1 ==== 107+0+4821 (secure 0 0 0) 0x7f20d4058f90 con 0x7f20e0102cd0 2024-03-23T12:30:35.860 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:35.860 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/4216607936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4216607936},{"type":"v1","addr":"172.21.15.111:6826","nonce":4216607936}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/856600159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":856600159},{"type":"v1","addr":"172.21.15.63:6827","nonce":856600159}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34418,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/754248720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":754248720},{"type":"v1","addr":"172.21.15.63:6829","nonce":754248720}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:35.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.884+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 msgr2=0x7f20c806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.884+0000 7f20e8400700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009fd0 tx=0x7f20cc009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:35.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.884+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102cd0 msgr2=0x7f20e00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102cd0 0x7f20e00702c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f20d400eb40 tx=0x7f20d400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 shutdown_connections 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f20c806c7c0 0x7f20c806ec70 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20e0102cd0 0x7f20e00702c0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 --2- 172.21.15.63:0/1679990591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0103670 0x7f20e0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 >> 172.21.15.63:0/1679990591 conn(0x7f20e00f8820 msgr2=0x7f20e00f9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 shutdown_connections 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:35.885+0000 7f20e8400700 1 -- 172.21.15.63:0/1679990591 wait complete. 2024-03-23T12:30:35.863 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 27 2024-03-23T12:30:35.880 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:35.880 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-03-23T12:30:35.880 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 -- 172.21.15.63:0/1034480567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 msgr2=0x7fab701030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 --2- 172.21.15.63:0/1034480567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701030a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fab60004940 tx=0x7fab60009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 -- 172.21.15.63:0/1034480567 shutdown_connections 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 --2- 172.21.15.63:0/1034480567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab70103670 0x7fab700fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 --2- 172.21.15.63:0/1034480567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701030a0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 -- 172.21.15.63:0/1034480567 >> 172.21.15.63:0/1034480567 conn(0x7fab700f8820 msgr2=0x7fab700fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 -- 172.21.15.63:0/1034480567 shutdown_connections 2024-03-23T12:30:36.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.097+0000 7fab77000700 1 -- 172.21.15.63:0/1034480567 wait complete. 2024-03-23T12:30:36.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.098+0000 7fab77000700 1 Processor -- start 2024-03-23T12:30:36.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.098+0000 7fab77000700 1 -- start start 2024-03-23T12:30:36.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab77000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab77000700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab70103670 0x7fab70113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab77000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70114350 con 0x7fab70102cd0 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab77000700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7010d950 con 0x7fab70103670 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab6fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab6fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701137c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51864/0 (socket says 172.21.15.63:51864) 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab6fe00700 1 -- 172.21.15.63:0/1657062709 learned_addr learned my addr 172.21.15.63:0/1657062709 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab6f400700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab70103670 0x7fab70113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab6fe00700 1 -- 172.21.15.63:0/1657062709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab70103670 msgr2=0x7fab70113d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.099+0000 7fab6fe00700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab70103670 0x7fab70113d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.100+0000 7fab6fe00700 1 -- 172.21.15.63:0/1657062709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60005040 con 0x7fab70102cd0 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.100+0000 7fab6fe00700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701137c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fab60009180 tx=0x7fab60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.100+0000 7fab67e00700 1 -- 172.21.15.63:0/1657062709 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6000e040 con 0x7fab70102cd0 2024-03-23T12:30:36.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.100+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7010dbd0 con 0x7fab70102cd0 2024-03-23T12:30:36.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.100+0000 7fab67e00700 1 -- 172.21.15.63:0/1657062709 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab60004290 con 0x7fab70102cd0 2024-03-23T12:30:36.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.100+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7010e190 con 0x7fab70102cd0 2024-03-23T12:30:36.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.100+0000 7fab67e00700 1 -- 172.21.15.63:0/1657062709 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab60025400 con 0x7fab70102cd0 2024-03-23T12:30:36.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.101+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab700fe420 con 0x7fab70102cd0 2024-03-23T12:30:36.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.102+0000 7fab67e00700 1 -- 172.21.15.63:0/1657062709 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fab60013790 con 0x7fab70102cd0 2024-03-23T12:30:36.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.103+0000 7fab67e00700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fab5406c7c0 0x7fab5406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:36.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.103+0000 7fab67e00700 1 -- 172.21.15.63:0/1657062709 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fab6008cd90 con 0x7fab70102cd0 2024-03-23T12:30:36.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.103+0000 7fab6f400700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fab5406c7c0 0x7fab5406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:36.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.104+0000 7fab6f400700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fab5406c7c0 0x7fab5406ec70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fab7010eb10 tx=0x7fab5c009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:36.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.108+0000 7fab67e00700 1 -- 172.21.15.63:0/1657062709 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fab6005aa20 con 0x7fab70102cd0 2024-03-23T12:30:36.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.372+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fab7010dd60 con 0x7fab70102cd0 2024-03-23T12:30:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.372+0000 7fab67e00700 1 -- 172.21.15.63:0/1657062709 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v43) v1 ==== 107+0+4045 (secure 0 0 0) 0x7fab7010dd60 con 0x7fab70102cd0 2024-03-23T12:30:36.350 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:36.350 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/4216607936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4216607936},{"type":"v1","addr":"172.21.15.111:6826","nonce":4216607936}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/856600159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":856600159},{"type":"v1","addr":"172.21.15.63:6827","nonce":856600159}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fab5406c7c0 msgr2=0x7fab5406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fab5406c7c0 0x7fab5406ec70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fab7010eb10 tx=0x7fab5c009ee0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 msgr2=0x7fab701137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701137c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fab60009180 tx=0x7fab60009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 shutdown_connections 2024-03-23T12:30:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fab5406c7c0 0x7fab5406ec70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab70102cd0 0x7fab701137c0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 --2- 172.21.15.63:0/1657062709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab70103670 0x7fab70113d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 >> 172.21.15.63:0/1657062709 conn(0x7fab700f8820 msgr2=0x7fab700f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 shutdown_connections 2024-03-23T12:30:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.375+0000 7fab77000700 1 -- 172.21.15.63:0/1657062709 wait complete. 2024-03-23T12:30:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 28 2024-03-23T12:30:36.370 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:36.371 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-03-23T12:30:36.371 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-03-23T12:30:36.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 -- 172.21.15.63:0/4154125780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c0708a0 msgr2=0x7f5f6c070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 --2- 172.21.15.63:0/4154125780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c0708a0 0x7f5f6c070d10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5f54004970 tx=0x7f5f54009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:36.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 -- 172.21.15.63:0/4154125780 shutdown_connections 2024-03-23T12:30:36.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 --2- 172.21.15.63:0/4154125780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c0708a0 0x7f5f6c070d10 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 --2- 172.21.15.63:0/4154125780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c071f90 0x7f5f6c070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 -- 172.21.15.63:0/4154125780 >> 172.21.15.63:0/4154125780 conn(0x7f5f6c0f8830 msgr2=0x7f5f6c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:36.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 -- 172.21.15.63:0/4154125780 shutdown_connections 2024-03-23T12:30:36.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.587+0000 7f5f73200700 1 -- 172.21.15.63:0/4154125780 wait complete. 2024-03-23T12:30:36.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f73200700 1 Processor -- start 2024-03-23T12:30:36.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f73200700 1 -- start start 2024-03-23T12:30:36.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f73200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c0708a0 0x7f5f6c119de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:36.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f73200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c071f90 0x7f5f6c11a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:36.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f73200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c11aa00 con 0x7f5f6c071f90 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f73200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c113f50 con 0x7f5f6c0708a0 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f63e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c071f90 0x7f5f6c11a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f63e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c071f90 0x7f5f6c11a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51892/0 (socket says 172.21.15.63:51892) 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.588+0000 7f5f63e00700 1 -- 172.21.15.63:0/887281035 learned_addr learned my addr 172.21.15.63:0/887281035 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f6be00700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c0708a0 0x7f5f6c119de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f63e00700 1 -- 172.21.15.63:0/887281035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c0708a0 msgr2=0x7f5f6c119de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f63e00700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c0708a0 0x7f5f6c119de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f63e00700 1 -- 172.21.15.63:0/887281035 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f54005040 con 0x7f5f6c071f90 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f63e00700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c071f90 0x7f5f6c11a320 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f5f54004f50 tx=0x7f5f54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f68c00700 1 -- 172.21.15.63:0/887281035 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f5400e040 con 0x7f5f6c071f90 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f68c00700 1 -- 172.21.15.63:0/887281035 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f54013420 con 0x7f5f6c071f90 2024-03-23T12:30:36.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c114170 con 0x7f5f6c071f90 2024-03-23T12:30:36.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f68c00700 1 -- 172.21.15.63:0/887281035 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f5401c610 con 0x7f5f6c071f90 2024-03-23T12:30:36.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.589+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c1146b0 con 0x7f5f6c071f90 2024-03-23T12:30:36.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.590+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c071aa0 con 0x7f5f6c071f90 2024-03-23T12:30:36.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.592+0000 7f5f68c00700 1 -- 172.21.15.63:0/887281035 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5f54013590 con 0x7f5f6c071f90 2024-03-23T12:30:36.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.592+0000 7f5f68c00700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f5006c7c0 0x7f5f5006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:36.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.592+0000 7f5f68c00700 1 -- 172.21.15.63:0/887281035 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5f5408cdb0 con 0x7f5f6c071f90 2024-03-23T12:30:36.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.593+0000 7f5f6be00700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f5006c7c0 0x7f5f5006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:36.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.593+0000 7f5f6be00700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f5006c7c0 0x7f5f5006ec70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c006fd0 tx=0x7f5f5c008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:36.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.597+0000 7f5f68c00700 1 -- 172.21.15.63:0/887281035 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5f5405a9c0 con 0x7f5f6c071f90 2024-03-23T12:30:36.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:36 smithi063 ceph-mon[141124]: pgmap v1187: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:36.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:36 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2710749536' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-23T12:30:36.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:36 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1679990591' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-23T12:30:36.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:36 smithi111 ceph-mon[106899]: pgmap v1187: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:36.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:36 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2710749536' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-23T12:30:36.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:36 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1679990591' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-23T12:30:36.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.863+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f5f6c071aa0 con 0x7f5f6c071f90 2024-03-23T12:30:36.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.864+0000 7f5f68c00700 1 -- 172.21.15.63:0/887281035 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v43) v1 ==== 107+0+4821 (secure 0 0 0) 0x7f5f5405a550 con 0x7f5f6c071f90 2024-03-23T12:30:36.842 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:36.842 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/4216607936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4216607936},{"type":"v1","addr":"172.21.15.111:6826","nonce":4216607936}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34414,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/856600159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":856600159},{"type":"v1","addr":"172.21.15.63:6827","nonce":856600159}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f5006c7c0 msgr2=0x7f5f5006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f5006c7c0 0x7f5f5006ec70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c006fd0 tx=0x7f5f5c008040 comp rx=0 tx=0).stop 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c071f90 msgr2=0x7f5f6c11a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c071f90 0x7f5f6c11a320 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f5f54004f50 tx=0x7f5f54009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 shutdown_connections 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5f5006c7c0 0x7f5f5006ec70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c0708a0 0x7f5f6c119de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 --2- 172.21.15.63:0/887281035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f6c071f90 0x7f5f6c11a320 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 >> 172.21.15.63:0/887281035 conn(0x7f5f6c0f8830 msgr2=0x7f5f6c0f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 shutdown_connections 2024-03-23T12:30:36.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:36.867+0000 7f5f73200700 1 -- 172.21.15.63:0/887281035 wait complete. 2024-03-23T12:30:36.846 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 29 2024-03-23T12:30:36.862 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:36.863 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-03-23T12:30:36.863 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-03-23T12:30:37.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 -- 172.21.15.63:0/167908002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c102cd0 msgr2=0x7f813c1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 --2- 172.21.15.63:0/167908002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c102cd0 0x7f813c1030a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8130004940 tx=0x7f8130009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 -- 172.21.15.63:0/167908002 shutdown_connections 2024-03-23T12:30:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 --2- 172.21.15.63:0/167908002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813c103670 0x7f813c0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 --2- 172.21.15.63:0/167908002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c102cd0 0x7f813c1030a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 -- 172.21.15.63:0/167908002 >> 172.21.15.63:0/167908002 conn(0x7f813c0f8820 msgr2=0x7f813c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 -- 172.21.15.63:0/167908002 shutdown_connections 2024-03-23T12:30:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.079+0000 7f8143e00700 1 -- 172.21.15.63:0/167908002 wait complete. 2024-03-23T12:30:37.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.080+0000 7f8143e00700 1 Processor -- start 2024-03-23T12:30:37.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.080+0000 7f8143e00700 1 -- start start 2024-03-23T12:30:37.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.080+0000 7f8143e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813c102cd0 0x7f813c115990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.080+0000 7f8143e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c103670 0x7f813c115ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.080+0000 7f8143e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c116520 con 0x7f813c103670 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.080+0000 7f8143e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c10fb40 con 0x7f813c102cd0 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8140a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c103670 0x7f813c115ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8140a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c103670 0x7f813c115ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51914/0 (socket says 172.21.15.63:51914) 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8140a00700 1 -- 172.21.15.63:0/3653086147 learned_addr learned my addr 172.21.15.63:0/3653086147 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8141400700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813c102cd0 0x7f813c115990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8140a00700 1 -- 172.21.15.63:0/3653086147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813c102cd0 msgr2=0x7f813c115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8140a00700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813c102cd0 0x7f813c115990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8140a00700 1 -- 172.21.15.63:0/3653086147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8130005040 con 0x7f813c103670 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8140a00700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c103670 0x7f813c115ed0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f812800c890 tx=0x7f81280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.081+0000 7f8139600700 1 -- 172.21.15.63:0/3653086147 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f812800a700 con 0x7f813c103670 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.082+0000 7f8139600700 1 -- 172.21.15.63:0/3653086147 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f812800ad10 con 0x7f813c103670 2024-03-23T12:30:37.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.082+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c10fe20 con 0x7f813c103670 2024-03-23T12:30:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.082+0000 7f8139600700 1 -- 172.21.15.63:0/3653086147 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8128012e70 con 0x7f813c103670 2024-03-23T12:30:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.082+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c1103e0 con 0x7f813c103670 2024-03-23T12:30:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.083+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c0fe420 con 0x7f813c103670 2024-03-23T12:30:37.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.084+0000 7f8139600700 1 -- 172.21.15.63:0/3653086147 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f812800a860 con 0x7f813c103670 2024-03-23T12:30:37.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.085+0000 7f8139600700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f812406c7c0 0x7f812406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.085+0000 7f8139600700 1 -- 172.21.15.63:0/3653086147 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f812808bae0 con 0x7f813c103670 2024-03-23T12:30:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.085+0000 7f8141400700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f812406c7c0 0x7f812406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:37.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.086+0000 7f8141400700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f812406c7c0 0x7f812406ec70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f813000b840 tx=0x7f8130009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:37.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.090+0000 7f8139600700 1 -- 172.21.15.63:0/3653086147 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8128055d60 con 0x7f813c103670 2024-03-23T12:30:37.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.354+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f813c109c90 con 0x7f813c103670 2024-03-23T12:30:37.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.355+0000 7f8139600700 1 -- 172.21.15.63:0/3653086147 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v43) v1 ==== 107+0+4045 (secure 0 0 0) 0x7f8128059380 con 0x7f813c103670 2024-03-23T12:30:37.333 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:37.333 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/4216607936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4216607936},{"type":"v1","addr":"172.21.15.111:6826","nonce":4216607936}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:37.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.357+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f812406c7c0 msgr2=0x7f812406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.357+0000 7f8143e00700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f812406c7c0 0x7f812406ec70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f813000b840 tx=0x7f8130009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:37.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.357+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c103670 msgr2=0x7f813c115ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.357+0000 7f8143e00700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c103670 0x7f813c115ed0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f812800c890 tx=0x7f81280044c0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.357+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 shutdown_connections 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.357+0000 7f8143e00700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f812406c7c0 0x7f812406ec70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.357+0000 7f8143e00700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813c102cd0 0x7f813c115990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.358+0000 7f8143e00700 1 --2- 172.21.15.63:0/3653086147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f813c103670 0x7f813c115ed0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.358+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 >> 172.21.15.63:0/3653086147 conn(0x7f813c0f8820 msgr2=0x7f813c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.358+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 shutdown_connections 2024-03-23T12:30:37.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.358+0000 7f8143e00700 1 -- 172.21.15.63:0/3653086147 wait complete. 2024-03-23T12:30:37.337 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 30 2024-03-23T12:30:37.354 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:37.354 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-03-23T12:30:37.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 -- 172.21.15.63:0/1944543233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 msgr2=0x7feb1c1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 --2- 172.21.15.63:0/1944543233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c1030a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7feb0c004940 tx=0x7feb0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 -- 172.21.15.63:0/1944543233 shutdown_connections 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 --2- 172.21.15.63:0/1944543233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb1c103670 0x7feb1c0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 --2- 172.21.15.63:0/1944543233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c1030a0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 -- 172.21.15.63:0/1944543233 >> 172.21.15.63:0/1944543233 conn(0x7feb1c0f8820 msgr2=0x7feb1c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 -- 172.21.15.63:0/1944543233 shutdown_connections 2024-03-23T12:30:37.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.571+0000 7feb22a00700 1 -- 172.21.15.63:0/1944543233 wait complete. 2024-03-23T12:30:37.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb22a00700 1 Processor -- start 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb22a00700 1 -- start start 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb22a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c113740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb22a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb1c103670 0x7feb1c113c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb22a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1c1142d0 con 0x7feb1c102cd0 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb22a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1c10d8d0 con 0x7feb1c103670 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb1be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb1be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c113740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51922/0 (socket says 172.21.15.63:51922) 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb1be00700 1 -- 172.21.15.63:0/676693494 learned_addr learned my addr 172.21.15.63:0/676693494 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.572+0000 7feb1b400700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb1c103670 0x7feb1c113c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb1be00700 1 -- 172.21.15.63:0/676693494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb1c103670 msgr2=0x7feb1c113c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb1be00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb1c103670 0x7feb1c113c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb1be00700 1 -- 172.21.15.63:0/676693494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c005040 con 0x7feb1c102cd0 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb1be00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c113740 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7feb0c008d80 tx=0x7feb0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb13e00700 1 -- 172.21.15.63:0/676693494 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb0c00e030 con 0x7feb1c102cd0 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb13e00700 1 -- 172.21.15.63:0/676693494 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb0c013de0 con 0x7feb1c102cd0 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb13e00700 1 -- 172.21.15.63:0/676693494 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb0c01cd00 con 0x7feb1c102cd0 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1c10db50 con 0x7feb1c102cd0 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.573+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1c10e110 con 0x7feb1c102cd0 2024-03-23T12:30:37.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.574+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1c0fe420 con 0x7feb1c102cd0 2024-03-23T12:30:37.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.575+0000 7feb13e00700 1 -- 172.21.15.63:0/676693494 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7feb0c013930 con 0x7feb1c102cd0 2024-03-23T12:30:37.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.576+0000 7feb13e00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb0006c7c0 0x7feb0006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:37.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.576+0000 7feb13e00700 1 -- 172.21.15.63:0/676693494 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb0c08d380 con 0x7feb1c102cd0 2024-03-23T12:30:37.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.576+0000 7feb1b400700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb0006c7c0 0x7feb0006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:37.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.577+0000 7feb1b400700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb0006c7c0 0x7feb0006ec70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feb1c10ea90 tx=0x7feb0800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:37.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.580+0000 7feb13e00700 1 -- 172.21.15.63:0/676693494 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb0c05b010 con 0x7feb1c102cd0 2024-03-23T12:30:37.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:37 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1657062709' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-23T12:30:37.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:37 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/887281035' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-23T12:30:37.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:37 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1657062709' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-23T12:30:37.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:37 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/887281035' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-23T12:30:37.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.840+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7feb1c109c90 con 0x7feb1c102cd0 2024-03-23T12:30:37.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.841+0000 7feb13e00700 1 -- 172.21.15.63:0/676693494 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v43) v1 ==== 107+0+4824 (secure 0 0 0) 0x7feb0c01ce60 con 0x7feb1c102cd0 2024-03-23T12:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/4216607936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4216607936},{"type":"v1","addr":"172.21.15.111:6826","nonce":4216607936}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:37.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb0006c7c0 msgr2=0x7feb0006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb0006c7c0 0x7feb0006ec70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feb1c10ea90 tx=0x7feb0800b040 comp rx=0 tx=0).stop 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 msgr2=0x7feb1c113740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c113740 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7feb0c008d80 tx=0x7feb0c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 shutdown_connections 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7feb0006c7c0 0x7feb0006ec70 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb1c102cd0 0x7feb1c113740 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 --2- 172.21.15.63:0/676693494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb1c103670 0x7feb1c113c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 >> 172.21.15.63:0/676693494 conn(0x7feb1c0f8820 msgr2=0x7feb1c0f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 shutdown_connections 2024-03-23T12:30:37.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:37.844+0000 7feb22a00700 1 -- 172.21.15.63:0/676693494 wait complete. 2024-03-23T12:30:37.823 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 31 2024-03-23T12:30:37.839 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:37.839 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-03-23T12:30:37.840 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-03-23T12:30:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 -- 172.21.15.63:0/3861051429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 msgr2=0x7fb8f40fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 --2- 172.21.15.63:0/3861051429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f40fcce0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0004940 tx=0x7fb8e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 -- 172.21.15.63:0/3861051429 shutdown_connections 2024-03-23T12:30:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 --2- 172.21.15.63:0/3861051429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f40fcce0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 --2- 172.21.15.63:0/3861051429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8f4102cd0 0x7fb8f41030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 -- 172.21.15.63:0/3861051429 >> 172.21.15.63:0/3861051429 conn(0x7fb8f40f8820 msgr2=0x7fb8f40fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 -- 172.21.15.63:0/3861051429 shutdown_connections 2024-03-23T12:30:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.056+0000 7fb8f9800700 1 -- 172.21.15.63:0/3861051429 wait complete. 2024-03-23T12:30:38.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.057+0000 7fb8f9800700 1 Processor -- start 2024-03-23T12:30:38.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.057+0000 7fb8f9800700 1 -- start start 2024-03-23T12:30:38.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f9800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8f4102cd0 0x7fb8f4113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:38.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f9800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f4113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f9800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f4114310 con 0x7fb8f4103670 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f9800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f410d910 con 0x7fb8f4102cd0 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f4113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f2000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f4113cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51946/0 (socket says 172.21.15.63:51946) 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f2000700 1 -- 172.21.15.63:0/2144728800 learned_addr learned my addr 172.21.15.63:0/2144728800 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.058+0000 7fb8f2a00700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8f4102cd0 0x7fb8f4113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8f2000700 1 -- 172.21.15.63:0/2144728800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8f4102cd0 msgr2=0x7fb8f4113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8f2000700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8f4102cd0 0x7fb8f4113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8f2000700 1 -- 172.21.15.63:0/2144728800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0005040 con 0x7fb8f4103670 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8f2000700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f4113cc0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009390 tx=0x7fb8e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8eaa00700 1 -- 172.21.15.63:0/2144728800 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e000e040 con 0x7fb8f4103670 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8eaa00700 1 -- 172.21.15.63:0/2144728800 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8e0013420 con 0x7fb8f4103670 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f410db90 con 0x7fb8f4103670 2024-03-23T12:30:38.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8eaa00700 1 -- 172.21.15.63:0/2144728800 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e001c630 con 0x7fb8f4103670 2024-03-23T12:30:38.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.059+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f410e150 con 0x7fb8f4103670 2024-03-23T12:30:38.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.060+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f40fe420 con 0x7fb8f4103670 2024-03-23T12:30:38.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.061+0000 7fb8eaa00700 1 -- 172.21.15.63:0/2144728800 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb8e0013a00 con 0x7fb8f4103670 2024-03-23T12:30:38.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.062+0000 7fb8eaa00700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8d806c7c0 0x7fb8d806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:38.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.062+0000 7fb8eaa00700 1 -- 172.21.15.63:0/2144728800 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb8e008ca60 con 0x7fb8f4103670 2024-03-23T12:30:38.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.062+0000 7fb8f2a00700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8d806c7c0 0x7fb8d806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:38.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.063+0000 7fb8f2a00700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8d806c7c0 0x7fb8d806ec70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4004f40 tx=0x7fb8e400b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.067+0000 7fb8eaa00700 1 -- 172.21.15.63:0/2144728800 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb8e005a830 con 0x7fb8f4103670 2024-03-23T12:30:38.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.332+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7fb8f4064970 con 0x7fb8f4103670 2024-03-23T12:30:38.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.332+0000 7fb8eaa00700 1 -- 172.21.15.63:0/2144728800 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v43) v1 ==== 107+0+4042 (secure 0 0 0) 0x7fb8e005a3c0 con 0x7fb8f4103670 2024-03-23T12:30:38.310 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:38.310 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:38.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8d806c7c0 msgr2=0x7fb8d806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8d806c7c0 0x7fb8d806ec70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4004f40 tx=0x7fb8e400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 msgr2=0x7fb8f4113cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f4113cc0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009390 tx=0x7fb8e0009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 shutdown_connections 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb8d806c7c0 0x7fb8d806ec70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8f4102cd0 0x7fb8f4113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 --2- 172.21.15.63:0/2144728800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8f4103670 0x7fb8f4113cc0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 >> 172.21.15.63:0/2144728800 conn(0x7fb8f40f8820 msgr2=0x7fb8f40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 shutdown_connections 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.335+0000 7fb8f9800700 1 -- 172.21.15.63:0/2144728800 wait complete. 2024-03-23T12:30:38.313 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 32 2024-03-23T12:30:38.330 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:38.330 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-03-23T12:30:38.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 -- 172.21.15.63:0/2656240134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 msgr2=0x7f671c1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 --2- 172.21.15.63:0/2656240134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c1030a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f670c004940 tx=0x7f670c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 -- 172.21.15.63:0/2656240134 shutdown_connections 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 --2- 172.21.15.63:0/2656240134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c103670 0x7f671c0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 --2- 172.21.15.63:0/2656240134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c1030a0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 -- 172.21.15.63:0/2656240134 >> 172.21.15.63:0/2656240134 conn(0x7f671c0f8820 msgr2=0x7f671c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 -- 172.21.15.63:0/2656240134 shutdown_connections 2024-03-23T12:30:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.547+0000 7f6721600700 1 -- 172.21.15.63:0/2656240134 wait complete. 2024-03-23T12:30:38.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.548+0000 7f6721600700 1 Processor -- start 2024-03-23T12:30:38.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.548+0000 7f6721600700 1 -- start start 2024-03-23T12:30:38.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.548+0000 7f6721600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:38.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.548+0000 7f6721600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c103670 0x7f671c115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.548+0000 7f6721600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1165b0 con 0x7f671c102cd0 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.548+0000 7f6721600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c10fb90 con 0x7f671c103670 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c115a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51958/0 (socket says 172.21.15.63:51958) 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671aa00700 1 -- 172.21.15.63:0/3746964475 learned_addr learned my addr 172.21.15.63:0/3746964475 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671a000700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c103670 0x7f671c115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671aa00700 1 -- 172.21.15.63:0/3746964475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c103670 msgr2=0x7f671c115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671aa00700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c103670 0x7f671c115f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671aa00700 1 -- 172.21.15.63:0/3746964475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c005040 con 0x7f671c102cd0 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671a000700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c103670 0x7f671c115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f671aa00700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c115a20 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f670c009360 tx=0x7f670c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f6712a00700 1 -- 172.21.15.63:0/3746964475 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f670c00e040 con 0x7f671c102cd0 2024-03-23T12:30:38.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f6712a00700 1 -- 172.21.15.63:0/3746964475 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f670c013420 con 0x7f671c102cd0 2024-03-23T12:30:38.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.549+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c10fe10 con 0x7f671c102cd0 2024-03-23T12:30:38.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.550+0000 7f6712a00700 1 -- 172.21.15.63:0/3746964475 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f670c01c5f0 con 0x7f671c102cd0 2024-03-23T12:30:38.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.550+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c1103d0 con 0x7f671c102cd0 2024-03-23T12:30:38.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.551+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c0fe420 con 0x7f671c102cd0 2024-03-23T12:30:38.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.552+0000 7f6712a00700 1 -- 172.21.15.63:0/3746964475 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f670c013590 con 0x7f671c102cd0 2024-03-23T12:30:38.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.552+0000 7f6712a00700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f670006c7c0 0x7f670006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:38.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.553+0000 7f6712a00700 1 -- 172.21.15.63:0/3746964475 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f670c08cc90 con 0x7f671c102cd0 2024-03-23T12:30:38.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.553+0000 7f671a000700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f670006c7c0 0x7f670006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:38.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.553+0000 7f671a000700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f670006c7c0 0x7f670006ec70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6708009fd0 tx=0x7f6708009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:38.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.557+0000 7f6712a00700 1 -- 172.21.15.63:0/3746964475 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f670c05aa50 con 0x7f671c102cd0 2024-03-23T12:30:38.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:38 smithi063 ceph-mon[141124]: pgmap v1188: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:38.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:38 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3653086147' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-23T12:30:38.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:38 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/676693494' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-23T12:30:38.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:38 smithi111 ceph-mon[106899]: pgmap v1188: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:38.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:38 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3653086147' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-23T12:30:38.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:38 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/676693494' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-23T12:30:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.819+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f671c1110c0 con 0x7f671c102cd0 2024-03-23T12:30:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.819+0000 7f6712a00700 1 -- 172.21.15.63:0/3746964475 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v43) v1 ==== 107+0+4824 (secure 0 0 0) 0x7f670c05a5e0 con 0x7f671c102cd0 2024-03-23T12:30:38.797 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:38.797 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44329,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3285026913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":3285026913},{"type":"v1","addr":"172.21.15.111:6827","nonce":3285026913}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f670006c7c0 msgr2=0x7f670006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f670006c7c0 0x7f670006ec70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6708009fd0 tx=0x7f6708009ee0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 msgr2=0x7f671c115a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c115a20 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f670c009360 tx=0x7f670c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 shutdown_connections 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f670006c7c0 0x7f670006ec70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c102cd0 0x7f671c115a20 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 --2- 172.21.15.63:0/3746964475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c103670 0x7f671c115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:38.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 >> 172.21.15.63:0/3746964475 conn(0x7f671c0f8820 msgr2=0x7f671c0f9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:38.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 shutdown_connections 2024-03-23T12:30:38.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:38.822+0000 7f6721600700 1 -- 172.21.15.63:0/3746964475 wait complete. 2024-03-23T12:30:38.801 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-03-23T12:30:38.818 DEBUG:tasks.fs:max_mds not reduced in epoch 33 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:38.818 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-03-23T12:30:38.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 -- 172.21.15.63:0/4151572380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484102cd0 msgr2=0x7fb4841030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 --2- 172.21.15.63:0/4151572380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484102cd0 0x7fb4841030a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb474004940 tx=0x7fb474009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 -- 172.21.15.63:0/4151572380 shutdown_connections 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 --2- 172.21.15.63:0/4151572380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb484103670 0x7fb4840fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 --2- 172.21.15.63:0/4151572380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484102cd0 0x7fb4841030a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 -- 172.21.15.63:0/4151572380 >> 172.21.15.63:0/4151572380 conn(0x7fb4840f8820 msgr2=0x7fb4840fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 -- 172.21.15.63:0/4151572380 shutdown_connections 2024-03-23T12:30:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.036+0000 7fb48a600700 1 -- 172.21.15.63:0/4151572380 wait complete. 2024-03-23T12:30:39.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.037+0000 7fb48a600700 1 Processor -- start 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.037+0000 7fb48a600700 1 -- start start 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.037+0000 7fb48a600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb484102cd0 0x7fb4841159a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.037+0000 7fb48a600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484103670 0x7fb484115ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.037+0000 7fb48a600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb484116530 con 0x7fb484103670 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.037+0000 7fb48a600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48410fb30 con 0x7fb484102cd0 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.037+0000 7fb47be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484103670 0x7fb484115ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484103670 0x7fb484115ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51976/0 (socket says 172.21.15.63:51976) 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47be00700 1 -- 172.21.15.63:0/1586210997 learned_addr learned my addr 172.21.15.63:0/1586210997 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb483400700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb484102cd0 0x7fb4841159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47be00700 1 -- 172.21.15.63:0/1586210997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb484102cd0 msgr2=0x7fb4841159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47be00700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb484102cd0 0x7fb4841159a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47be00700 1 -- 172.21.15.63:0/1586210997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474005040 con 0x7fb484103670 2024-03-23T12:30:39.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47be00700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484103670 0x7fb484115ee0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb46c002a50 tx=0x7fb46c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:39.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47b400700 1 -- 172.21.15.63:0/1586210997 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb46c007cc0 con 0x7fb484103670 2024-03-23T12:30:39.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47b400700 1 -- 172.21.15.63:0/1586210997 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb46c00f040 con 0x7fb484103670 2024-03-23T12:30:39.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48410fe10 con 0x7fb484103670 2024-03-23T12:30:39.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.038+0000 7fb47b400700 1 -- 172.21.15.63:0/1586210997 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb46c00d990 con 0x7fb484103670 2024-03-23T12:30:39.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.039+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4841103d0 con 0x7fb484103670 2024-03-23T12:30:39.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.040+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4840fe420 con 0x7fb484103670 2024-03-23T12:30:39.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.041+0000 7fb47b400700 1 -- 172.21.15.63:0/1586210997 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fb46c009710 con 0x7fb484103670 2024-03-23T12:30:39.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.042+0000 7fb47b400700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb46806c7c0 0x7fb46806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:39.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.042+0000 7fb47b400700 1 -- 172.21.15.63:0/1586210997 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb46c08b910 con 0x7fb484103670 2024-03-23T12:30:39.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.042+0000 7fb483400700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb46806c7c0 0x7fb46806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:39.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.043+0000 7fb483400700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb46806c7c0 0x7fb46806ec70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb47400b3b0 tx=0x7fb474009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:39.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.047+0000 7fb47b400700 1 -- 172.21.15.63:0/1586210997 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb46c059520 con 0x7fb484103670 2024-03-23T12:30:39.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.311+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7fb484109c90 con 0x7fb484103670 2024-03-23T12:30:39.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.312+0000 7fb47b400700 1 -- 172.21.15.63:0/1586210997 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v43) v1 ==== 107+0+4042 (secure 0 0 0) 0x7fb46c0590b0 con 0x7fb484103670 2024-03-23T12:30:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:39.290 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:39.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.314+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb46806c7c0 msgr2=0x7fb46806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb46806c7c0 0x7fb46806ec70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb47400b3b0 tx=0x7fb474009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484103670 msgr2=0x7fb484115ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484103670 0x7fb484115ee0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb46c002a50 tx=0x7fb46c004600 comp rx=0 tx=0).stop 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 shutdown_connections 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fb46806c7c0 0x7fb46806ec70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb484102cd0 0x7fb4841159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 --2- 172.21.15.63:0/1586210997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb484103670 0x7fb484115ee0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 >> 172.21.15.63:0/1586210997 conn(0x7fb4840f8820 msgr2=0x7fb4840f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 shutdown_connections 2024-03-23T12:30:39.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.315+0000 7fb48a600700 1 -- 172.21.15.63:0/1586210997 wait complete. 2024-03-23T12:30:39.294 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 34 2024-03-23T12:30:39.310 DEBUG:tasks.fs:max_mds not reduced in epoch 34 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:39.310 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-03-23T12:30:39.310 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 -- 172.21.15.63:0/230135147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 msgr2=0x7f79f8108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 --2- 172.21.15.63:0/230135147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8108fe0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004970 tx=0x7f79e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 -- 172.21.15.63:0/230135147 shutdown_connections 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 --2- 172.21.15.63:0/230135147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8108fe0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 --2- 172.21.15.63:0/230135147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f8070360 0x7f79f8070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 -- 172.21.15.63:0/230135147 >> 172.21.15.63:0/230135147 conn(0x7f79f80f8820 msgr2=0x7f79f80fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 -- 172.21.15.63:0/230135147 shutdown_connections 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.528+0000 7f79fee00700 1 -- 172.21.15.63:0/230135147 wait complete. 2024-03-23T12:30:39.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79fee00700 1 Processor -- start 2024-03-23T12:30:39.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79fee00700 1 -- start start 2024-03-23T12:30:39.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79fee00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f8070360 0x7f79f81159e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:39.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79fee00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8115f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:39.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79fee00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f8116560 con 0x7f79f8070c70 2024-03-23T12:30:39.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79fee00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f810fb20 con 0x7f79f8070360 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79f7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8115f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79f7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8115f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51990/0 (socket says 172.21.15.63:51990) 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.529+0000 7f79f7400700 1 -- 172.21.15.63:0/159916987 learned_addr learned my addr 172.21.15.63:0/159916987 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79f7e00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f8070360 0x7f79f81159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79f7400700 1 -- 172.21.15.63:0/159916987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f8070360 msgr2=0x7f79f81159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79f7400700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f8070360 0x7f79f81159e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79f7400700 1 -- 172.21.15.63:0/159916987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e4005040 con 0x7f79f8070c70 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79f7400700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8115f20 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004f50 tx=0x7f79e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79efe00700 1 -- 172.21.15.63:0/159916987 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e400e040 con 0x7f79f8070c70 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79efe00700 1 -- 172.21.15.63:0/159916987 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79e4013420 con 0x7f79f8070c70 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f810fda0 con 0x7f79f8070c70 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79efe00700 1 -- 172.21.15.63:0/159916987 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e401c610 con 0x7f79f8070c70 2024-03-23T12:30:39.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.530+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f81102e0 con 0x7f79f8070c70 2024-03-23T12:30:39.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.531+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f8071aa0 con 0x7f79f8070c70 2024-03-23T12:30:39.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.532+0000 7f79efe00700 1 -- 172.21.15.63:0/159916987 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f79e4013590 con 0x7f79f8070c70 2024-03-23T12:30:39.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.533+0000 7f79efe00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f79dc06c7c0 0x7f79dc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:39.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.533+0000 7f79efe00700 1 -- 172.21.15.63:0/159916987 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f79e408ccf0 con 0x7f79f8070c70 2024-03-23T12:30:39.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.534+0000 7f79f7e00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f79dc06c7c0 0x7f79dc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:39.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.534+0000 7f79f7e00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f79dc06c7c0 0x7f79dc06ec70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f79e8006fd0 tx=0x7f79e8008040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:39.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.538+0000 7f79efe00700 1 -- 172.21.15.63:0/159916987 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f79e4056e70 con 0x7f79f8070c70 2024-03-23T12:30:39.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:39 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2144728800' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-23T12:30:39.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:39 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3746964475' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-23T12:30:39.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:39 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2144728800' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-23T12:30:39.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:39 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3746964475' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-23T12:30:39.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.805+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f79f810bf00 con 0x7f79f8070c70 2024-03-23T12:30:39.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.805+0000 7f79efe00700 1 -- 172.21.15.63:0/159916987 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v43) v1 ==== 107+0+4821 (secure 0 0 0) 0x7f79e405a490 con 0x7f79f8070c70 2024-03-23T12:30:39.783 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:39.783 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T11:57:27.441639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:39.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f79dc06c7c0 msgr2=0x7f79dc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f79dc06c7c0 0x7f79dc06ec70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f79e8006fd0 tx=0x7f79e8008040 comp rx=0 tx=0).stop 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 msgr2=0x7f79f8115f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8115f20 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004f50 tx=0x7f79e4009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 shutdown_connections 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f79dc06c7c0 0x7f79dc06ec70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f8070360 0x7f79f81159e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 --2- 172.21.15.63:0/159916987 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79f8070c70 0x7f79f8115f20 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 >> 172.21.15.63:0/159916987 conn(0x7f79f80f8820 msgr2=0x7f79f80f9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 shutdown_connections 2024-03-23T12:30:39.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:39.808+0000 7f79fee00700 1 -- 172.21.15.63:0/159916987 wait complete. 2024-03-23T12:30:39.787 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-03-23T12:30:39.803 DEBUG:tasks.fs:max_mds not reduced in epoch 35 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-23T12:30:39.803 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 35 2024-03-23T12:30:39.803 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 -- 172.21.15.63:0/1765600066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 msgr2=0x7fc2481030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 --2- 172.21.15.63:0/1765600066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481030a0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc23c004940 tx=0x7fc23c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 -- 172.21.15.63:0/1765600066 shutdown_connections 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 --2- 172.21.15.63:0/1765600066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc248103670 0x7fc2480fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 --2- 172.21.15.63:0/1765600066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481030a0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 -- 172.21.15.63:0/1765600066 >> 172.21.15.63:0/1765600066 conn(0x7fc2480f8820 msgr2=0x7fc2480fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 -- 172.21.15.63:0/1765600066 shutdown_connections 2024-03-23T12:30:39.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.020+0000 7fc250800700 1 -- 172.21.15.63:0/1765600066 wait complete. 2024-03-23T12:30:39.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.021+0000 7fc250800700 1 Processor -- start 2024-03-23T12:30:39.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.021+0000 7fc250800700 1 -- start start 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc250800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc250800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc248103670 0x7fc248113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc250800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248114350 con 0x7fc248102cd0 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc250800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24810d950 con 0x7fc248103670 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc24de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc24de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481137c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52010/0 (socket says 172.21.15.63:52010) 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc24de00700 1 -- 172.21.15.63:0/485956851 learned_addr learned my addr 172.21.15.63:0/485956851 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.022+0000 7fc24d400700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc248103670 0x7fc248113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc24de00700 1 -- 172.21.15.63:0/485956851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc248103670 msgr2=0x7fc248113d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc24de00700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc248103670 0x7fc248113d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc24de00700 1 -- 172.21.15.63:0/485956851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c005040 con 0x7fc248102cd0 2024-03-23T12:30:40.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc24de00700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481137c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009390 tx=0x7fc23c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:40.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc242000700 1 -- 172.21.15.63:0/485956851 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc23c00e040 con 0x7fc248102cd0 2024-03-23T12:30:40.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc242000700 1 -- 172.21.15.63:0/485956851 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc23c013420 con 0x7fc248102cd0 2024-03-23T12:30:40.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24810dbd0 con 0x7fc248102cd0 2024-03-23T12:30:40.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc242000700 1 -- 172.21.15.63:0/485956851 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc23c01c630 con 0x7fc248102cd0 2024-03-23T12:30:40.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.023+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24810e190 con 0x7fc248102cd0 2024-03-23T12:30:40.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.024+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2480fe420 con 0x7fc248102cd0 2024-03-23T12:30:40.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.025+0000 7fc242000700 1 -- 172.21.15.63:0/485956851 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc23c0139e0 con 0x7fc248102cd0 2024-03-23T12:30:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.026+0000 7fc242000700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc23006c7c0 0x7fc23006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.026+0000 7fc242000700 1 -- 172.21.15.63:0/485956851 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc23c08cc70 con 0x7fc248102cd0 2024-03-23T12:30:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.027+0000 7fc24d400700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc23006c7c0 0x7fc23006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.027+0000 7fc24d400700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc23006c7c0 0x7fc23006ec70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc24810eb10 tx=0x7fc23800b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:40.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.031+0000 7fc242000700 1 -- 172.21.15.63:0/485956851 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc23c05aa40 con 0x7fc248102cd0 2024-03-23T12:30:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.299+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7fc248109c90 con 0x7fc248102cd0 2024-03-23T12:30:40.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.300+0000 7fc242000700 1 -- 172.21.15.63:0/485956851 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v43) v1 ==== 107+0+4843 (secure 0 0 0) 0x7fc23c05a5d0 con 0x7fc248102cd0 2024-03-23T12:30:40.277 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:40.278 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:30.838293+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc23006c7c0 msgr2=0x7fc23006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc23006c7c0 0x7fc23006ec70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc24810eb10 tx=0x7fc23800b040 comp rx=0 tx=0).stop 2024-03-23T12:30:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 msgr2=0x7fc2481137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481137c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009390 tx=0x7fc23c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 shutdown_connections 2024-03-23T12:30:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc23006c7c0 0x7fc23006ec70 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc248102cd0 0x7fc2481137c0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 --2- 172.21.15.63:0/485956851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc248103670 0x7fc248113d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.302+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 >> 172.21.15.63:0/485956851 conn(0x7fc2480f8820 msgr2=0x7fc2480f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:40.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.303+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 shutdown_connections 2024-03-23T12:30:40.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.303+0000 7fc250800700 1 -- 172.21.15.63:0/485956851 wait complete. 2024-03-23T12:30:40.281 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 36 2024-03-23T12:30:40.297 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 36 2024-03-23T12:30:40.297 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 -- 172.21.15.63:0/630570551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4102ce0 msgr2=0x7f7dd41030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 --2- 172.21.15.63:0/630570551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4102ce0 0x7f7dd41030b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8004940 tx=0x7f7dc8009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 -- 172.21.15.63:0/630570551 shutdown_connections 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 --2- 172.21.15.63:0/630570551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dd4103680 0x7f7dd4064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 --2- 172.21.15.63:0/630570551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4102ce0 0x7f7dd41030b0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 -- 172.21.15.63:0/630570551 >> 172.21.15.63:0/630570551 conn(0x7f7dd4072ad0 msgr2=0x7f7dd4072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 -- 172.21.15.63:0/630570551 shutdown_connections 2024-03-23T12:30:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.513+0000 7f7ddc200700 1 -- 172.21.15.63:0/630570551 wait complete. 2024-03-23T12:30:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.514+0000 7f7ddc200700 1 Processor -- start 2024-03-23T12:30:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.514+0000 7f7ddc200700 1 -- start start 2024-03-23T12:30:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.514+0000 7f7ddc200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dd4102ce0 0x7f7dd40fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.514+0000 7f7ddc200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4103680 0x7f7dd40fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.514+0000 7f7ddc200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40fdb90 con 0x7f7dd4103680 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.514+0000 7f7ddc200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40fc6c0 con 0x7f7dd4102ce0 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4103680 0x7f7dd40fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4103680 0x7f7dd40fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52028/0 (socket says 172.21.15.63:52028) 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd8e00700 1 -- 172.21.15.63:0/3434791252 learned_addr learned my addr 172.21.15.63:0/3434791252 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd9800700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dd4102ce0 0x7f7dd40fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd8e00700 1 -- 172.21.15.63:0/3434791252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dd4102ce0 msgr2=0x7f7dd40fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd8e00700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dd4102ce0 0x7f7dd40fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd8e00700 1 -- 172.21.15.63:0/3434791252 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc8005040 con 0x7f7dd4103680 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.515+0000 7f7dd8e00700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4103680 0x7f7dd40fc180 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0002a50 tx=0x7f7dc00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.516+0000 7f7dd1600700 1 -- 172.21.15.63:0/3434791252 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc0007cc0 con 0x7f7dd4103680 2024-03-23T12:30:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.516+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd40fc9a0 con 0x7f7dd4103680 2024-03-23T12:30:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.516+0000 7f7dd1600700 1 -- 172.21.15.63:0/3434791252 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dc000f040 con 0x7f7dd4103680 2024-03-23T12:30:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.516+0000 7f7dd1600700 1 -- 172.21.15.63:0/3434791252 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc00095d0 con 0x7f7dd4103680 2024-03-23T12:30:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.516+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd4119ec0 con 0x7f7dd4103680 2024-03-23T12:30:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.517+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd40fe430 con 0x7f7dd4103680 2024-03-23T12:30:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.518+0000 7f7dd1600700 1 -- 172.21.15.63:0/3434791252 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f7dc0009730 con 0x7f7dd4103680 2024-03-23T12:30:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.519+0000 7f7dd1600700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7dbc06c7c0 0x7f7dbc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.519+0000 7f7dd1600700 1 -- 172.21.15.63:0/3434791252 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7dc008b710 con 0x7f7dd4103680 2024-03-23T12:30:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.519+0000 7f7dd9800700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7dbc06c7c0 0x7f7dbc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.519+0000 7f7dd9800700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7dbc06c7c0 0x7f7dbc06ec70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800b020 tx=0x7f7dc8009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.523+0000 7f7dd1600700 1 -- 172.21.15.63:0/3434791252 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7dc0055910 con 0x7f7dd4103680 2024-03-23T12:30:40.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:40 smithi063 ceph-mon[141124]: pgmap v1189: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:40.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:40 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1586210997' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-23T12:30:40.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:40 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/159916987' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-23T12:30:40.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:40 smithi111 ceph-mon[106899]: pgmap v1189: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:40.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:40 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1586210997' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-23T12:30:40.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:40 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/159916987' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-23T12:30:40.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.786+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f7dd4064970 con 0x7f7dd4103680 2024-03-23T12:30:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.787+0000 7f7dd1600700 1 -- 172.21.15.63:0/3434791252 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v43) v1 ==== 107+0+4842 (secure 0 0 0) 0x7f7dc0058f30 con 0x7f7dd4103680 2024-03-23T12:30:40.764 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:40.764 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:31.839721+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7dbc06c7c0 msgr2=0x7f7dbc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7dbc06c7c0 0x7f7dbc06ec70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800b020 tx=0x7f7dc8009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4103680 msgr2=0x7f7dd40fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4103680 0x7f7dd40fc180 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0002a50 tx=0x7f7dc00046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 shutdown_connections 2024-03-23T12:30:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f7dbc06c7c0 0x7f7dbc06ec70 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dd4102ce0 0x7f7dd40fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 --2- 172.21.15.63:0/3434791252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7dd4103680 0x7f7dd40fc180 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.789+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 >> 172.21.15.63:0/3434791252 conn(0x7f7dd4072ad0 msgr2=0x7f7dd4105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.790+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 shutdown_connections 2024-03-23T12:30:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:40.790+0000 7f7ddc200700 1 -- 172.21.15.63:0/3434791252 wait complete. 2024-03-23T12:30:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 37 2024-03-23T12:30:40.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-03-23T12:30:40.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 -- 172.21.15.63:0/3941023955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c103680 msgr2=0x7fc17c0fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 --2- 172.21.15.63:0/3941023955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c103680 0x7fc17c0fccf0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fc164004940 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:40.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 -- 172.21.15.63:0/3941023955 shutdown_connections 2024-03-23T12:30:40.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 --2- 172.21.15.63:0/3941023955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c103680 0x7fc17c0fccf0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 --2- 172.21.15.63:0/3941023955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc17c102ce0 0x7fc17c1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 -- 172.21.15.63:0/3941023955 >> 172.21.15.63:0/3941023955 conn(0x7fc17c0f8830 msgr2=0x7fc17c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:40.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 -- 172.21.15.63:0/3941023955 shutdown_connections 2024-03-23T12:30:40.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.003+0000 7fc181600700 1 -- 172.21.15.63:0/3941023955 wait complete. 2024-03-23T12:30:40.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc181600700 1 Processor -- start 2024-03-23T12:30:40.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc181600700 1 -- start start 2024-03-23T12:30:40.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc181600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c102ce0 0x7fc17c1136d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc181600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc17c103680 0x7fc17c113c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc181600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c114260 con 0x7fc17c102ce0 2024-03-23T12:30:40.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc181600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c10d880 con 0x7fc17c103680 2024-03-23T12:30:40.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc17aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c102ce0 0x7fc17c1136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc17aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c102ce0 0x7fc17c1136d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52038/0 (socket says 172.21.15.63:52038) 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.004+0000 7fc17aa00700 1 -- 172.21.15.63:0/62928079 learned_addr learned my addr 172.21.15.63:0/62928079 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc173e00700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc17c103680 0x7fc17c113c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc17aa00700 1 -- 172.21.15.63:0/62928079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc17c103680 msgr2=0x7fc17c113c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc17aa00700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc17c103680 0x7fc17c113c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc17aa00700 1 -- 172.21.15.63:0/62928079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164005040 con 0x7fc17c102ce0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc173e00700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc17c103680 0x7fc17c113c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc17aa00700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c102ce0 0x7fc17c1136d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00eb40 tx=0x7fc16c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc172a00700 1 -- 172.21.15.63:0/62928079 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16c007cc0 con 0x7fc17c102ce0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc172a00700 1 -- 172.21.15.63:0/62928079 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc16c00f040 con 0x7fc17c102ce0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc172a00700 1 -- 172.21.15.63:0/62928079 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16c009570 con 0x7fc17c102ce0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c10db00 con 0x7fc17c102ce0 2024-03-23T12:30:40.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.005+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c10e0c0 con 0x7fc17c102ce0 2024-03-23T12:30:40.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.006+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c0fe430 con 0x7fc17c102ce0 2024-03-23T12:30:40.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.008+0000 7fc172a00700 1 -- 172.21.15.63:0/62928079 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc16c0096d0 con 0x7fc17c102ce0 2024-03-23T12:30:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.008+0000 7fc172a00700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc16006c7c0 0x7fc16006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.009+0000 7fc172a00700 1 -- 172.21.15.63:0/62928079 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc16c08b6f0 con 0x7fc17c102ce0 2024-03-23T12:30:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.009+0000 7fc173e00700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc16006c7c0 0x7fc16006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:40.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.009+0000 7fc173e00700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc16006c7c0 0x7fc16006ec70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc16400b0d0 tx=0x7fc164009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:40.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.013+0000 7fc172a00700 1 -- 172.21.15.63:0/62928079 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc16c059430 con 0x7fc17c102ce0 2024-03-23T12:30:41.217 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:41 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/485956851' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-23T12:30:41.217 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:41 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3434791252' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-23T12:30:41.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.281+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7fc17c10ed60 con 0x7fc17c102ce0 2024-03-23T12:30:41.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.282+0000 7fc172a00700 1 -- 172.21.15.63:0/62928079 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v43) v1 ==== 107+0+4845 (secure 0 0 0) 0x7fc16c058fc0 con 0x7fc17c102ce0 2024-03-23T12:30:41.259 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:41.259 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:38.190421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:resolve","state_seq":462,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:41.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc16006c7c0 msgr2=0x7fc16006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc16006c7c0 0x7fc16006ec70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc16400b0d0 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:41.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c102ce0 msgr2=0x7fc17c1136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c102ce0 0x7fc17c1136d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00eb40 tx=0x7fc16c00b040 comp rx=0 tx=0).stop 2024-03-23T12:30:41.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 shutdown_connections 2024-03-23T12:30:41.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc16006c7c0 0x7fc16006ec70 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c102ce0 0x7fc17c1136d0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 --2- 172.21.15.63:0/62928079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc17c103680 0x7fc17c113c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.284+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 >> 172.21.15.63:0/62928079 conn(0x7fc17c0f8830 msgr2=0x7fc17c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:41.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.285+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 shutdown_connections 2024-03-23T12:30:41.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.285+0000 7fc181600700 1 -- 172.21.15.63:0/62928079 wait complete. 2024-03-23T12:30:41.263 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 38 2024-03-23T12:30:41.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.496+0000 7f39aac00700 1 -- 172.21.15.63:0/1373442570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070300 msgr2=0x7f39a40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.496+0000 7f39aac00700 1 --2- 172.21.15.63:0/1373442570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070300 0x7f39a40706d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3994004970 tx=0x7f3994009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.496+0000 7f39aac00700 1 -- 172.21.15.63:0/1373442570 shutdown_connections 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.496+0000 7f39aac00700 1 --2- 172.21.15.63:0/1373442570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4070c10 0x7f39a4109050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.496+0000 7f39aac00700 1 --2- 172.21.15.63:0/1373442570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070300 0x7f39a40706d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.496+0000 7f39aac00700 1 -- 172.21.15.63:0/1373442570 >> 172.21.15.63:0/1373442570 conn(0x7f39a40f8830 msgr2=0x7f39a40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.496+0000 7f39aac00700 1 -- 172.21.15.63:0/1373442570 shutdown_connections 2024-03-23T12:30:41.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.497+0000 7f39aac00700 1 -- 172.21.15.63:0/1373442570 wait complete. 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.497+0000 7f39aac00700 1 Processor -- start 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.497+0000 7f39aac00700 1 -- start start 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39aac00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4070300 0x7f39a4115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39aac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070c10 0x7f39a4115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39aac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4116510 con 0x7f39a4070c10 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39aac00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a410fb10 con 0x7f39a4070300 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39a3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070c10 0x7f39a4115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39a3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070c10 0x7f39a4115ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33240/0 (socket says 172.21.15.63:33240) 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39a3400700 1 -- 172.21.15.63:0/2764122085 learned_addr learned my addr 172.21.15.63:0/2764122085 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:41.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.498+0000 7f39a3e00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4070300 0x7f39a4115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:41.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f39a3400700 1 -- 172.21.15.63:0/2764122085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4070300 msgr2=0x7f39a4115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f39a3400700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4070300 0x7f39a4115980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f39a3400700 1 -- 172.21.15.63:0/2764122085 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994005040 con 0x7f39a4070c10 2024-03-23T12:30:41.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f39a3400700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070c10 0x7f39a4115ec0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f399000c8c0 tx=0x7f39900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:41.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f399be00700 1 -- 172.21.15.63:0/2764122085 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f399000a980 con 0x7f39a4070c10 2024-03-23T12:30:41.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f399be00700 1 -- 172.21.15.63:0/2764122085 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f399000ce60 con 0x7f39a4070c10 2024-03-23T12:30:41.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f399be00700 1 -- 172.21.15.63:0/2764122085 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39900113a0 con 0x7f39a4070c10 2024-03-23T12:30:41.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a410fdf0 con 0x7f39a4070c10 2024-03-23T12:30:41.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.499+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a4110330 con 0x7f39a4070c10 2024-03-23T12:30:41.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.500+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a4071a40 con 0x7f39a4070c10 2024-03-23T12:30:41.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.501+0000 7f399be00700 1 -- 172.21.15.63:0/2764122085 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f3990012400 con 0x7f39a4070c10 2024-03-23T12:30:41.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.502+0000 7f399be00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f398806c7c0 0x7f398806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:41.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.502+0000 7f399be00700 1 -- 172.21.15.63:0/2764122085 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f399008bb30 con 0x7f39a4070c10 2024-03-23T12:30:41.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.502+0000 7f39a3e00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f398806c7c0 0x7f398806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:41.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.503+0000 7f39a3e00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f398806c7c0 0x7f398806ec70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3994009fd0 tx=0x7f3994009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:41.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.507+0000 7f399be00700 1 -- 172.21.15.63:0/2764122085 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3990059840 con 0x7f39a4070c10 2024-03-23T12:30:41.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:41.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:41 smithi063 ceph-mon[141124]: pgmap v1190: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:41 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/485956851' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-23T12:30:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:41 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3434791252' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-23T12:30:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:41.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:41 smithi111 ceph-mon[106899]: pgmap v1190: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:41.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.771+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7f39a4064970 con 0x7f39a4070c10 2024-03-23T12:30:41.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.771+0000 7f399be00700 1 -- 172.21.15.63:0/2764122085 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v43) v1 ==== 107+0+4848 (secure 0 0 0) 0x7f39900593d0 con 0x7f39a4070c10 2024-03-23T12:30:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:41.749 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:39.198229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:resolve","state_seq":467,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:resolve","state_seq":462,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f398806c7c0 msgr2=0x7f398806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f398806c7c0 0x7f398806ec70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3994009fd0 tx=0x7f3994009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070c10 msgr2=0x7f39a4115ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070c10 0x7f39a4115ec0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f399000c8c0 tx=0x7f39900044c0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 shutdown_connections 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f398806c7c0 0x7f398806ec70 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4070300 0x7f39a4115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 --2- 172.21.15.63:0/2764122085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4070c10 0x7f39a4115ec0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 >> 172.21.15.63:0/2764122085 conn(0x7f39a40f8830 msgr2=0x7f39a40f9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:41.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 shutdown_connections 2024-03-23T12:30:41.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.774+0000 7f39aac00700 1 -- 172.21.15.63:0/2764122085 wait complete. 2024-03-23T12:30:41.753 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 39 2024-03-23T12:30:41.773 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-03-23T12:30:41.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 -- 172.21.15.63:0/3675271611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c103670 msgr2=0x7f5e5c0fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 --2- 172.21.15.63:0/3675271611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c103670 0x7f5e5c0fcce0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5e44004940 tx=0x7f5e44009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 -- 172.21.15.63:0/3675271611 shutdown_connections 2024-03-23T12:30:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 --2- 172.21.15.63:0/3675271611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c103670 0x7f5e5c0fcce0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 --2- 172.21.15.63:0/3675271611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c102cd0 0x7f5e5c1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 -- 172.21.15.63:0/3675271611 >> 172.21.15.63:0/3675271611 conn(0x7f5e5c0f8820 msgr2=0x7f5e5c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 -- 172.21.15.63:0/3675271611 shutdown_connections 2024-03-23T12:30:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.990+0000 7f5e61800700 1 -- 172.21.15.63:0/3675271611 wait complete. 2024-03-23T12:30:41.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.991+0000 7f5e61800700 1 Processor -- start 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.991+0000 7f5e61800700 1 -- start start 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e61800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c102cd0 0x7f5e5c113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e61800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c103670 0x7f5e5c113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e61800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c114310 con 0x7f5e5c102cd0 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e61800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c10d910 con 0x7f5e5c103670 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e5aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c102cd0 0x7f5e5c113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e5aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c102cd0 0x7f5e5c113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33258/0 (socket says 172.21.15.63:33258) 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e5aa00700 1 -- 172.21.15.63:0/3161644147 learned_addr learned my addr 172.21.15.63:0/3161644147 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e53e00700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c103670 0x7f5e5c113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e5aa00700 1 -- 172.21.15.63:0/3161644147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c103670 msgr2=0x7f5e5c113cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e5aa00700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c103670 0x7f5e5c113cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.992+0000 7f5e5aa00700 1 -- 172.21.15.63:0/3161644147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44005040 con 0x7f5e5c102cd0 2024-03-23T12:30:41.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.993+0000 7f5e5aa00700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c102cd0 0x7f5e5c113780 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c002a50 tx=0x7f5e4c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.993+0000 7f5e52a00700 1 -- 172.21.15.63:0/3161644147 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4c007cc0 con 0x7f5e5c102cd0 2024-03-23T12:30:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.993+0000 7f5e52a00700 1 -- 172.21.15.63:0/3161644147 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e4c00f040 con 0x7f5e5c102cd0 2024-03-23T12:30:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.993+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c10dbf0 con 0x7f5e5c102cd0 2024-03-23T12:30:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.993+0000 7f5e52a00700 1 -- 172.21.15.63:0/3161644147 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4c00d9c0 con 0x7f5e5c102cd0 2024-03-23T12:30:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.993+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c10e180 con 0x7f5e5c102cd0 2024-03-23T12:30:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.994+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c0fe420 con 0x7f5e5c102cd0 2024-03-23T12:30:41.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.995+0000 7f5e52a00700 1 -- 172.21.15.63:0/3161644147 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5e4c00db20 con 0x7f5e5c102cd0 2024-03-23T12:30:41.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.996+0000 7f5e52a00700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4006c7c0 0x7f5e4006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:41.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.996+0000 7f5e52a00700 1 -- 172.21.15.63:0/3161644147 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e4c08b760 con 0x7f5e5c102cd0 2024-03-23T12:30:41.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.996+0000 7f5e53e00700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4006c7c0 0x7f5e4006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:41.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:41.997+0000 7f5e53e00700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4006c7c0 0x7f5e4006ec70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c10ebf0 tx=0x7f5e44009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:41.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.000+0000 7f5e52a00700 1 -- 172.21.15.63:0/3161644147 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5e4c0593f0 con 0x7f5e5c102cd0 2024-03-23T12:30:42.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.264+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7f5e5c109c70 con 0x7f5e5c102cd0 2024-03-23T12:30:42.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.264+0000 7f5e52a00700 1 -- 172.21.15.63:0/3161644147 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v43) v1 ==== 107+0+4852 (secure 0 0 0) 0x7f5e4c058f80 con 0x7f5e5c102cd0 2024-03-23T12:30:42.242 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:42.242 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:40.203454+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":468,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:reconnect","state_seq":463,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:42.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.266+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4006c7c0 msgr2=0x7f5e4006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4006c7c0 0x7f5e4006ec70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c10ebf0 tx=0x7f5e44009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:42.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c102cd0 msgr2=0x7f5e5c113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c102cd0 0x7f5e5c113780 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c002a50 tx=0x7f5e4c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 shutdown_connections 2024-03-23T12:30:42.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5e4006c7c0 0x7f5e4006ec70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e5c102cd0 0x7f5e5c113780 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 --2- 172.21.15.63:0/3161644147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e5c103670 0x7f5e5c113cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 >> 172.21.15.63:0/3161644147 conn(0x7f5e5c0f8820 msgr2=0x7f5e5c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:42.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 shutdown_connections 2024-03-23T12:30:42.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.267+0000 7f5e61800700 1 -- 172.21.15.63:0/3161644147 wait complete. 2024-03-23T12:30:42.245 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 40 2024-03-23T12:30:42.263 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 41 2024-03-23T12:30:42.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.479+0000 7fc174c00700 1 -- 172.21.15.63:0/1265972817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 msgr2=0x7fc170108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.479+0000 7fc174c00700 1 --2- 172.21.15.63:0/1265972817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170108fe0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc15c004970 tx=0x7fc15c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:42.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.479+0000 7fc174c00700 1 -- 172.21.15.63:0/1265972817 shutdown_connections 2024-03-23T12:30:42.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.479+0000 7fc174c00700 1 --2- 172.21.15.63:0/1265972817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170108fe0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.479+0000 7fc174c00700 1 --2- 172.21.15.63:0/1265972817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc170070360 0x7fc170070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.479+0000 7fc174c00700 1 -- 172.21.15.63:0/1265972817 >> 172.21.15.63:0/1265972817 conn(0x7fc1700f8820 msgr2=0x7fc1700fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:42.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.480+0000 7fc174c00700 1 -- 172.21.15.63:0/1265972817 shutdown_connections 2024-03-23T12:30:42.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.480+0000 7fc174c00700 1 -- 172.21.15.63:0/1265972817 wait complete. 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.480+0000 7fc174c00700 1 Processor -- start 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.480+0000 7fc174c00700 1 -- start start 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc174c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc170070360 0x7fc1701158c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc174c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170115e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc174c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170116450 con 0x7fc170070c70 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc174c00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17010fa90 con 0x7fc170070360 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc16d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170115e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc16d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170115e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33272/0 (socket says 172.21.15.63:33272) 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc16d400700 1 -- 172.21.15.63:0/615162122 learned_addr learned my addr 172.21.15.63:0/615162122 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc16de00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc170070360 0x7fc1701158c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc16d400700 1 -- 172.21.15.63:0/615162122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc170070360 msgr2=0x7fc1701158c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc16d400700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc170070360 0x7fc1701158c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.481+0000 7fc16d400700 1 -- 172.21.15.63:0/615162122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c005040 con 0x7fc170070c70 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.482+0000 7fc16d400700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170115e00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc15c004f50 tx=0x7fc15c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.482+0000 7fc166000700 1 -- 172.21.15.63:0/615162122 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc15c00e040 con 0x7fc170070c70 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.482+0000 7fc166000700 1 -- 172.21.15.63:0/615162122 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc15c013420 con 0x7fc170070c70 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.482+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17010fd10 con 0x7fc170070c70 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.482+0000 7fc166000700 1 -- 172.21.15.63:0/615162122 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc15c01c610 con 0x7fc170070c70 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.482+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc170110250 con 0x7fc170070c70 2024-03-23T12:30:42.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.483+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170071aa0 con 0x7fc170070c70 2024-03-23T12:30:42.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.484+0000 7fc166000700 1 -- 172.21.15.63:0/615162122 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fc15c013590 con 0x7fc170070c70 2024-03-23T12:30:42.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.485+0000 7fc166000700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc15406c7c0 0x7fc15406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:42.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.485+0000 7fc166000700 1 -- 172.21.15.63:0/615162122 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc15c08cca0 con 0x7fc170070c70 2024-03-23T12:30:42.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.485+0000 7fc16de00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc15406c7c0 0x7fc15406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:42.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.486+0000 7fc16de00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc15406c7c0 0x7fc15406ec70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc160004990 tx=0x7fc160009ee0 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:42.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.489+0000 7fc166000700 1 -- 172.21.15.63:0/615162122 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc15c05a930 con 0x7fc170070c70 2024-03-23T12:30:42.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:42 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/62928079' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-23T12:30:42.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:42 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2764122085' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-23T12:30:42.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:42 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/62928079' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-23T12:30:42.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:42 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2764122085' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-23T12:30:42.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.754+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 41, "format": "json"} v 0) v1 -- 0x7fc17010be80 con 0x7fc170070c70 2024-03-23T12:30:42.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.755+0000 7fc166000700 1 -- 172.21.15.63:0/615162122 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 41, "format": "json"}]=0 dumped fsmap epoch 41 v43) v1 ==== 107+0+4846 (secure 0 0 0) 0x7fc15c05a4c0 con 0x7fc170070c70 2024-03-23T12:30:42.732 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:42.732 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:41.210021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":469,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:rejoin","state_seq":464,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc15406c7c0 msgr2=0x7fc15406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc15406c7c0 0x7fc15406ec70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc160004990 tx=0x7fc160009ee0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 msgr2=0x7fc170115e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170115e00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc15c004f50 tx=0x7fc15c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 shutdown_connections 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fc15406c7c0 0x7fc15406ec70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc170070360 0x7fc1701158c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 --2- 172.21.15.63:0/615162122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc170070c70 0x7fc170115e00 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 >> 172.21.15.63:0/615162122 conn(0x7fc1700f8820 msgr2=0x7fc1700f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 shutdown_connections 2024-03-23T12:30:42.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.757+0000 7fc174c00700 1 -- 172.21.15.63:0/615162122 wait complete. 2024-03-23T12:30:42.736 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 41 2024-03-23T12:30:42.753 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 42 2024-03-23T12:30:42.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.970+0000 7f3798600700 1 -- 172.21.15.63:0/2169947754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790103670 msgr2=0x7f37900fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.970+0000 7f3798600700 1 --2- 172.21.15.63:0/2169947754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790103670 0x7f37900fcce0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f377c004940 tx=0x7f377c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:42.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.970+0000 7f3798600700 1 -- 172.21.15.63:0/2169947754 shutdown_connections 2024-03-23T12:30:42.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.970+0000 7f3798600700 1 --2- 172.21.15.63:0/2169947754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790103670 0x7f37900fcce0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.970+0000 7f3798600700 1 --2- 172.21.15.63:0/2169947754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3790102cd0 0x7f37901030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.970+0000 7f3798600700 1 -- 172.21.15.63:0/2169947754 >> 172.21.15.63:0/2169947754 conn(0x7f37900f8820 msgr2=0x7f37900fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:42.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.971+0000 7f3798600700 1 -- 172.21.15.63:0/2169947754 shutdown_connections 2024-03-23T12:30:42.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.971+0000 7f3798600700 1 -- 172.21.15.63:0/2169947754 wait complete. 2024-03-23T12:30:42.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.971+0000 7f3798600700 1 Processor -- start 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.971+0000 7f3798600700 1 -- start start 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3798600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790102cd0 0x7f3790071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3798600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3790103670 0x7f37900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3798600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790070800 con 0x7f3790102cd0 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3798600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790070970 con 0x7f3790103670 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3795c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790102cd0 0x7f3790071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3795c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790102cd0 0x7f3790071d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33284/0 (socket says 172.21.15.63:33284) 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3795c00700 1 -- 172.21.15.63:0/1161236185 learned_addr learned my addr 172.21.15.63:0/1161236185 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3795200700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3790103670 0x7f37900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:42.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3795c00700 1 -- 172.21.15.63:0/1161236185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3790103670 msgr2=0x7f37900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3795c00700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3790103670 0x7f37900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.972+0000 7f3795c00700 1 -- 172.21.15.63:0/1161236185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c005040 con 0x7f3790102cd0 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.973+0000 7f3795c00700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790102cd0 0x7f3790071d50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3784002a50 tx=0x7f37840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.973+0000 7f378d600700 1 -- 172.21.15.63:0/1161236185 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3784007cc0 con 0x7f3790102cd0 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.973+0000 7f378d600700 1 -- 172.21.15.63:0/1161236185 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f378400f040 con 0x7f3790102cd0 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.973+0000 7f378d600700 1 -- 172.21.15.63:0/1161236185 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3784009540 con 0x7f3790102cd0 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.973+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3790070c50 con 0x7f3790102cd0 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.973+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379011e460 con 0x7f3790102cd0 2024-03-23T12:30:42.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.974+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37900fe420 con 0x7f3790102cd0 2024-03-23T12:30:42.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.975+0000 7f378d600700 1 -- 172.21.15.63:0/1161236185 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f37840096a0 con 0x7f3790102cd0 2024-03-23T12:30:42.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.976+0000 7f378d600700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f377806c7c0 0x7f377806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:42.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.977+0000 7f378d600700 1 -- 172.21.15.63:0/1161236185 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f378408b820 con 0x7f3790102cd0 2024-03-23T12:30:42.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.977+0000 7f3795200700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f377806c7c0 0x7f377806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:42.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.977+0000 7f3795200700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f377806c7c0 0x7f377806ec70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f377c00b020 tx=0x7f377c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:42.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:42.981+0000 7f378d600700 1 -- 172.21.15.63:0/1161236185 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f37840594b0 con 0x7f3790102cd0 2024-03-23T12:30:43.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.245+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 42, "format": "json"} v 0) v1 -- 0x7f3790064970 con 0x7f3790102cd0 2024-03-23T12:30:43.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.246+0000 7f378d600700 1 -- 172.21.15.63:0/1161236185 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 42, "format": "json"}]=0 dumped fsmap epoch 42 v43) v1 ==== 107+0+4846 (secure 0 0 0) 0x7f3784059040 con 0x7f3790102cd0 2024-03-23T12:30:43.223 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:43.224 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44393,"name":"cephfs.smithi111.ylnmzy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6826/2255440447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2255440447},{"type":"v1","addr":"172.21.15.111:6826","nonce":2255440447}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33},{"gid":45001,"name":"cephfs.smithi111.atkibb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/755283916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6825","nonce":755283916},{"type":"v1","addr":"172.21.15.111:6827","nonce":755283916}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-23T11:43:49.209465+0000","modified":"2024-03-23T12:28:42.217101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34422,"mds_1":34454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34422":{"gid":34422,"name":"cephfs.smithi063.brtzgy","rank":0,"incarnation":36,"state":"up:active","state_seq":470,"addr":"172.21.15.63:6829/450408958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":450408958},{"type":"v1","addr":"172.21.15.63:6829","nonce":450408958}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34454":{"gid":34454,"name":"cephfs.smithi063.mdbdrz","rank":1,"incarnation":36,"state":"up:active","state_seq":465,"addr":"172.21.15.63:6827/1595814379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1595814379},{"type":"v1","addr":"172.21.15.63:6827","nonce":1595814379}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T12:30:43.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.248+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f377806c7c0 msgr2=0x7f377806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:43.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.248+0000 7f3798600700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f377806c7c0 0x7f377806ec70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f377c00b020 tx=0x7f377c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:43.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.248+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790102cd0 msgr2=0x7f3790071d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:43.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.248+0000 7f3798600700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790102cd0 0x7f3790071d50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3784002a50 tx=0x7f37840046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.249+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 shutdown_connections 2024-03-23T12:30:43.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.249+0000 7f3798600700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f377806c7c0 0x7f377806ec70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.249+0000 7f3798600700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3790102cd0 0x7f3790071d50 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.249+0000 7f3798600700 1 --2- 172.21.15.63:0/1161236185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3790103670 0x7f37900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.249+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 >> 172.21.15.63:0/1161236185 conn(0x7f37900f8820 msgr2=0x7f37900f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:43.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.249+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 shutdown_connections 2024-03-23T12:30:43.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.249+0000 7f3798600700 1 -- 172.21.15.63:0/1161236185 wait complete. 2024-03-23T12:30:43.227 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 42 2024-03-23T12:30:43.244 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-23T12:30:43.343 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-23T12:30:43.343 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T12:30:43.343 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T12:30:43.389 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T12:30:43.389 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T12:30:43.474 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T12:30:43.574 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:43 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3161644147' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-23T12:30:43.574 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:43 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/615162122' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-23T12:30:43.574 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:43 smithi063 ceph-mon[141124]: pgmap v1191: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:43.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:43 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3161644147' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-23T12:30:43.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:43 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/615162122' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-23T12:30:43.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:43 smithi111 ceph-mon[106899]: pgmap v1191: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 -- 172.21.15.63:0/1566001803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18081036a0 msgr2=0x7f18080fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 --2- 172.21.15.63:0/1566001803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18081036a0 0x7f18080fcd10 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f17f4004940 tx=0x7f17f4009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 -- 172.21.15.63:0/1566001803 shutdown_connections 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 --2- 172.21.15.63:0/1566001803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18081036a0 0x7f18080fcd10 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 --2- 172.21.15.63:0/1566001803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1808102d00 0x7f18081030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 -- 172.21.15.63:0/1566001803 >> 172.21.15.63:0/1566001803 conn(0x7f18080f8850 msgr2=0x7f18080fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 -- 172.21.15.63:0/1566001803 shutdown_connections 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.732+0000 7f180f200700 1 -- 172.21.15.63:0/1566001803 wait complete. 2024-03-23T12:30:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f180f200700 1 Processor -- start 2024-03-23T12:30:43.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f180f200700 1 -- start start 2024-03-23T12:30:43.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f180f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1808102d00 0x7f18080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:43.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f180f200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18081036a0 0x7f1808070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:43.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f180f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1808072210 con 0x7f1808102d00 2024-03-23T12:30:43.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f180f200700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1808070d40 con 0x7f18081036a0 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f1807e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1808102d00 0x7f18080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f1807e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1808102d00 0x7f18080702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33310/0 (socket says 172.21.15.63:33310) 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.733+0000 7f1807e00700 1 -- 172.21.15.63:0/864592614 learned_addr learned my addr 172.21.15.63:0/864592614 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f1807e00700 1 -- 172.21.15.63:0/864592614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18081036a0 msgr2=0x7f1808070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f1807e00700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18081036a0 0x7f1808070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f1807e00700 1 -- 172.21.15.63:0/864592614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f4005040 con 0x7f1808102d00 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f1807e00700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1808102d00 0x7f18080702c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f17f800eb40 tx=0x7f17f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f17ffe00700 1 -- 172.21.15.63:0/864592614 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f8007cc0 con 0x7f1808102d00 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f17ffe00700 1 -- 172.21.15.63:0/864592614 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17f800f040 con 0x7f1808102d00 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f17ffe00700 1 -- 172.21.15.63:0/864592614 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f8009570 con 0x7f1808102d00 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1808070fc0 con 0x7f1808102d00 2024-03-23T12:30:43.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.734+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1808119eb0 con 0x7f1808102d00 2024-03-23T12:30:43.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.735+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18080fe450 con 0x7f1808102d00 2024-03-23T12:30:43.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.736+0000 7f17ffe00700 1 -- 172.21.15.63:0/864592614 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f17f80096d0 con 0x7f1808102d00 2024-03-23T12:30:43.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.737+0000 7f17ffe00700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17ec06c7c0 0x7f17ec06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:43.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.737+0000 7f17ffe00700 1 -- 172.21.15.63:0/864592614 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f17f808b6c0 con 0x7f1808102d00 2024-03-23T12:30:43.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.737+0000 7f1807400700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17ec06c7c0 0x7f17ec06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:43.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.738+0000 7f1807400700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17ec06c7c0 0x7f17ec06ec70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f17f400b0d0 tx=0x7f17f4009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:43.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.741+0000 7f17ffe00700 1 -- 172.21.15.63:0/864592614 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f17f8055970 con 0x7f1808102d00 2024-03-23T12:30:43.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.961+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1808064970 con 0x7f1808102d00 2024-03-23T12:30:43.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.962+0000 7f17ffe00700 1 -- 172.21.15.63:0/864592614 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16563 (secure 0 0 0) 0x7f17f8058f90 con 0x7f1808102d00 2024-03-23T12:30:43.940 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:43.940 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":86,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","created":"2024-03-23T11:35:55.761926+0000","modified":"2024-03-23T12:13:23.282611+0000","last_up_change":"2024-03-23T11:57:22.490793+0000","last_in_change":"2024-03-23T11:42:00.992881+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:40:53.432498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-23T11:43:48.146323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-23T11:43:48.722717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e4f29eb-375c-4f37-a8d8-f82b40650ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6803","nonce":2164791827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6805","nonce":2164791827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6809","nonce":2164791827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6807","nonce":2164791827}]},"public_addr":"172.21.15.63:6803/2164791827","cluster_addr":"172.21.15.63:6805/2164791827","heartbeat_back_addr":"172.21.15.63:6809/2164791827","heartbeat_front_addr":"172.21.15.63:6807/2164791827","state":["exists","up"]},{"osd":1,"uuid":"5e3ddf8b-e2c3-4188-b495-ac2743805f33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6811","nonce":3286820315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6813","nonce":3286820315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6817","nonce":3286820315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6815","nonce":3286820315}]},"public_addr":"172.21.15.63:6811/3286820315","cluster_addr":"172.21.15.63:6813/3286820315","heartbeat_back_addr":"172.21.15.63:6817/3286820315","heartbeat_front_addr":"172.21.15.63:6815/3286820315","state":["exists","up"]},{"osd":2,"uuid":"edc4babe-dff9-462c-9af7-75e30b8095af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6819","nonce":3910577428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6821","nonce":3910577428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6825","nonce":3910577428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6823","nonce":3910577428}]},"public_addr":"172.21.15.63:6819/3910577428","cluster_addr":"172.21.15.63:6821/3910577428","heartbeat_back_addr":"172.21.15.63:6825/3910577428","heartbeat_front_addr":"172.21.15.63:6823/3910577428","state":["exists","up"]},{"osd":3,"uuid":"7bc94d72-a814-447a-8112-9d50058a2759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6801","nonce":1545952189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6803","nonce":1545952189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6807","nonce":1545952189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6805","nonce":1545952189}]},"public_addr":"172.21.15.111:6801/1545952189","cluster_addr":"172.21.15.111:6803/1545952189","heartbeat_back_addr":"172.21.15.111:6807/1545952189","heartbeat_front_addr":"172.21.15.111:6805/1545952189","state":["exists","up"]},{"osd":4,"uuid":"7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6809","nonce":2121500845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6811","nonce":2121500845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6815","nonce":2121500845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6813","nonce":2121500845}]},"public_addr":"172.21.15.111:6809/2121500845","cluster_addr":"172.21.15.111:6811/2121500845","heartbeat_back_addr":"172.21.15.111:6815/2121500845","heartbeat_front_addr":"172.21.15.111:6813/2121500845","state":["exists","up"]},{"osd":5,"uuid":"2219a929-a842-4b41-ab46-096be6ef4d13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6817","nonce":983556337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6819","nonce":983556337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6823","nonce":983556337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6821","nonce":983556337}]},"public_addr":"172.21.15.111:6817/983556337","cluster_addr":"172.21.15.111:6819/983556337","heartbeat_back_addr":"172.21.15.111:6823/983556337","heartbeat_front_addr":"172.21.15.111:6821/983556337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:39:42.374999+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:17.321830+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:50.840928+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:20.211360+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:50.314996+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:42:20.376924+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6829/2516647170":"2024-03-24T11:57:27.441051+0000","172.21.15.63:0/3192464709":"2024-03-24T11:36:25.877145+0000","172.21.15.111:0/3712729872":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/2812522808":"2024-03-24T11:46:26.114316+0000","172.21.15.111:6828/3542552256":"2024-03-24T11:48:52.950910+0000","172.21.15.111:0/245431515":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3356174638":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/2483586841":"2024-03-24T11:46:26.114316+0000","172.21.15.63:0/2981669421":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6801/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6800/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/2062408886":"2024-03-24T11:52:25.664441+0000","172.21.15.63:0/2603926600":"2024-03-24T11:46:26.114316+0000","172.21.15.63:6801/2193815059":"2024-03-24T11:46:26.114316+0000","172.21.15.111:6824/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:6826/3230334326":"2024-03-24T11:57:27.441037+0000","172.21.15.63:0/2286910220":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6800/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.111:6825/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:6801/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1379382252":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3148717714":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6800/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6827/3230334326":"2024-03-24T11:57:27.441037+0000","172.21.15.63:6800/335624352":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6801/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.111:6829/3542552256":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3491519284":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1183338694":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/2975088208":"2024-03-24T11:46:26.114316+0000","172.21.15.63:0/1697976918":"2024-03-24T11:52:25.664441+0000","172.21.15.63:0/3227457751":"2024-03-24T11:37:01.693515+0000","172.21.15.111:0/3254210644":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3688530508":"2024-03-24T11:52:25.664441+0000","172.21.15.111:0/2862601802":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3670139931":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6800/2193815059":"2024-03-24T11:46:26.114316+0000","172.21.15.63:6801/335624352":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6828/2516647170":"2024-03-24T11:57:27.441051+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17ec06c7c0 msgr2=0x7f17ec06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17ec06c7c0 0x7f17ec06ec70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f17f400b0d0 tx=0x7f17f4009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1808102d00 msgr2=0x7f18080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1808102d00 0x7f18080702c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f17f800eb40 tx=0x7f17f800b040 comp rx=0 tx=0).stop 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 shutdown_connections 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f17ec06c7c0 0x7f17ec06ec70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1808102d00 0x7f18080702c0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 --2- 172.21.15.63:0/864592614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18081036a0 0x7f1808070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 >> 172.21.15.63:0/864592614 conn(0x7f18080f8850 msgr2=0x7f18080f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 shutdown_connections 2024-03-23T12:30:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:43.965+0000 7f180f200700 1 -- 172.21.15.63:0/864592614 wait complete. 2024-03-23T12:30:43.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:40:53.432498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-23T11:43:48.146323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-23T11:43:48.722717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T12:30:43.961 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T12:30:44.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 -- 172.21.15.63:0/1808376263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90103670 msgr2=0x7fec900fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 --2- 172.21.15.63:0/1808376263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90103670 0x7fec900fcce0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fec7c004940 tx=0x7fec7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:44.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 -- 172.21.15.63:0/1808376263 shutdown_connections 2024-03-23T12:30:44.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 --2- 172.21.15.63:0/1808376263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90103670 0x7fec900fcce0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 --2- 172.21.15.63:0/1808376263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec90102cd0 0x7fec901030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 -- 172.21.15.63:0/1808376263 >> 172.21.15.63:0/1808376263 conn(0x7fec900f8820 msgr2=0x7fec900fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:44.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 -- 172.21.15.63:0/1808376263 shutdown_connections 2024-03-23T12:30:44.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.179+0000 7fec97800700 1 -- 172.21.15.63:0/1808376263 wait complete. 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.180+0000 7fec97800700 1 Processor -- start 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.180+0000 7fec97800700 1 -- start start 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec97800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90102cd0 0x7fec900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec97800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec90103670 0x7fec90070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec97800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90072210 con 0x7fec90102cd0 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec97800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90070d40 con 0x7fec90103670 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec94e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90102cd0 0x7fec900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec94e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90102cd0 0x7fec900702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33314/0 (socket says 172.21.15.63:33314) 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec94e00700 1 -- 172.21.15.63:0/3360227561 learned_addr learned my addr 172.21.15.63:0/3360227561 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec8fe00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec90103670 0x7fec90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:44.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec94e00700 1 -- 172.21.15.63:0/3360227561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec90103670 msgr2=0x7fec90070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec94e00700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec90103670 0x7fec90070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.181+0000 7fec94e00700 1 -- 172.21.15.63:0/3360227561 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c005040 con 0x7fec90102cd0 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.182+0000 7fec94e00700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90102cd0 0x7fec900702c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fec84002a80 tx=0x7fec840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.182+0000 7fec8cc00700 1 -- 172.21.15.63:0/3360227561 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec84007cc0 con 0x7fec90102cd0 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.182+0000 7fec8cc00700 1 -- 172.21.15.63:0/3360227561 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec8400f040 con 0x7fec90102cd0 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.182+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90071020 con 0x7fec90102cd0 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.182+0000 7fec8cc00700 1 -- 172.21.15.63:0/3360227561 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec8400da60 con 0x7fec90102cd0 2024-03-23T12:30:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.182+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90119ed0 con 0x7fec90102cd0 2024-03-23T12:30:44.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.183+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec900fe420 con 0x7fec90102cd0 2024-03-23T12:30:44.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.184+0000 7fec8cc00700 1 -- 172.21.15.63:0/3360227561 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fec8400dbc0 con 0x7fec90102cd0 2024-03-23T12:30:44.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.185+0000 7fec8cc00700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fec7806c7c0 0x7fec7806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:44.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.185+0000 7fec8cc00700 1 -- 172.21.15.63:0/3360227561 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fec8408b640 con 0x7fec90102cd0 2024-03-23T12:30:44.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.185+0000 7fec8fe00700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fec7806c7c0 0x7fec7806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:44.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.186+0000 7fec8fe00700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fec7806c7c0 0x7fec7806ec70 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fec900719f0 tx=0x7fec7c009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:44.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.189+0000 7fec8cc00700 1 -- 172.21.15.63:0/3360227561 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fec840558c0 con 0x7fec90102cd0 2024-03-23T12:30:44.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.410+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fec90064970 con 0x7fec90102cd0 2024-03-23T12:30:44.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.410+0000 7fec8cc00700 1 -- 172.21.15.63:0/3360227561 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fec84058ee0 con 0x7fec90102cd0 2024-03-23T12:30:44.388 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-03-23T12:30:44.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.412+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fec7806c7c0 msgr2=0x7fec7806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.412+0000 7fec97800700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fec7806c7c0 0x7fec7806ec70 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fec900719f0 tx=0x7fec7c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:44.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.412+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90102cd0 msgr2=0x7fec900702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.412+0000 7fec97800700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90102cd0 0x7fec900702c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fec84002a80 tx=0x7fec840046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.413+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 shutdown_connections 2024-03-23T12:30:44.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.413+0000 7fec97800700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fec7806c7c0 0x7fec7806ec70 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.413+0000 7fec97800700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec90102cd0 0x7fec900702c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.413+0000 7fec97800700 1 --2- 172.21.15.63:0/3360227561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec90103670 0x7fec90070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.413+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 >> 172.21.15.63:0/3360227561 conn(0x7fec900f8820 msgr2=0x7fec900f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:44.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.413+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 shutdown_connections 2024-03-23T12:30:44.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.413+0000 7fec97800700 1 -- 172.21.15.63:0/3360227561 wait complete. 2024-03-23T12:30:44.408 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-23T12:30:44.467 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:44 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1161236185' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 42, "format": "json"}]: dispatch 2024-03-23T12:30:44.468 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:44 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/864592614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 -- 172.21.15.63:0/2048608210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438102cd0 msgr2=0x7f24381030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 --2- 172.21.15.63:0/2048608210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438102cd0 0x7f24381030a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2428004940 tx=0x7f2428009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 -- 172.21.15.63:0/2048608210 shutdown_connections 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 --2- 172.21.15.63:0/2048608210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2438103670 0x7f24380fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 --2- 172.21.15.63:0/2048608210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438102cd0 0x7f24381030a0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 -- 172.21.15.63:0/2048608210 >> 172.21.15.63:0/2048608210 conn(0x7f24380f8820 msgr2=0x7f24380fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 -- 172.21.15.63:0/2048608210 shutdown_connections 2024-03-23T12:30:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.627+0000 7f243ee00700 1 -- 172.21.15.63:0/2048608210 wait complete. 2024-03-23T12:30:44.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.628+0000 7f243ee00700 1 Processor -- start 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.628+0000 7f243ee00700 1 -- start start 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f243ee00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2438102cd0 0x7f24381137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f243ee00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438103670 0x7f2438113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f243ee00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2438114350 con 0x7f2438103670 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f243ee00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243810d950 con 0x7f2438102cd0 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f2437400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438103670 0x7f2438113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f2437400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438103670 0x7f2438113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33330/0 (socket says 172.21.15.63:33330) 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f2437400700 1 -- 172.21.15.63:0/2795216957 learned_addr learned my addr 172.21.15.63:0/2795216957 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f2437e00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2438102cd0 0x7f24381137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f2437400700 1 -- 172.21.15.63:0/2795216957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2438102cd0 msgr2=0x7f24381137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f2437400700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2438102cd0 0x7f24381137c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.629+0000 7f2437400700 1 -- 172.21.15.63:0/2795216957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428005040 con 0x7f2438103670 2024-03-23T12:30:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.630+0000 7f2437400700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438103670 0x7f2438113d00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2424002a50 tx=0x7f2424004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.630+0000 7f242fe00700 1 -- 172.21.15.63:0/2795216957 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2424007cc0 con 0x7f2438103670 2024-03-23T12:30:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.630+0000 7f242fe00700 1 -- 172.21.15.63:0/2795216957 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f242400f040 con 0x7f2438103670 2024-03-23T12:30:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.630+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243810dc30 con 0x7f2438103670 2024-03-23T12:30:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.630+0000 7f242fe00700 1 -- 172.21.15.63:0/2795216957 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f242400d920 con 0x7f2438103670 2024-03-23T12:30:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.630+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243810e1f0 con 0x7f2438103670 2024-03-23T12:30:44.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.631+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24380fe420 con 0x7f2438103670 2024-03-23T12:30:44.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.632+0000 7f242fe00700 1 -- 172.21.15.63:0/2795216957 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f2424009680 con 0x7f2438103670 2024-03-23T12:30:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.633+0000 7f242fe00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f241c06c7c0 0x7f241c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.633+0000 7f242fe00700 1 -- 172.21.15.63:0/2795216957 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f242408b6d0 con 0x7f2438103670 2024-03-23T12:30:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.633+0000 7f2437e00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f241c06c7c0 0x7f241c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.634+0000 7f2437e00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f241c06c7c0 0x7f241c06ec70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2428009fd0 tx=0x7f2428009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.638+0000 7f242fe00700 1 -- 172.21.15.63:0/2795216957 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f24240593e0 con 0x7f2438103670 2024-03-23T12:30:44.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:44 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1161236185' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 42, "format": "json"}]: dispatch 2024-03-23T12:30:44.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:44 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/864592614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T12:30:44.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.854+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f2438109cc0 con 0x7f2438103670 2024-03-23T12:30:44.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.854+0000 7f242fe00700 1 -- 172.21.15.63:0/2795216957 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2424058f70 con 0x7f2438103670 2024-03-23T12:30:44.832 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f241c06c7c0 msgr2=0x7f241c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f241c06c7c0 0x7f241c06ec70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2428009fd0 tx=0x7f2428009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438103670 msgr2=0x7f2438113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438103670 0x7f2438113d00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2424002a50 tx=0x7f2424004600 comp rx=0 tx=0).stop 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 shutdown_connections 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f241c06c7c0 0x7f241c06ec70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2438102cd0 0x7f24381137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 --2- 172.21.15.63:0/2795216957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2438103670 0x7f2438113d00 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 >> 172.21.15.63:0/2795216957 conn(0x7f24380f8820 msgr2=0x7f24380f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 shutdown_connections 2024-03-23T12:30:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:44.857+0000 7f243ee00700 1 -- 172.21.15.63:0/2795216957 wait complete. 2024-03-23T12:30:44.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-23T12:30:45.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 -- 172.21.15.63:0/2907051214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c103670 msgr2=0x7f903c0fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 --2- 172.21.15.63:0/2907051214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c103670 0x7f903c0fcce0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f9028004940 tx=0x7f9028009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:45.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 -- 172.21.15.63:0/2907051214 shutdown_connections 2024-03-23T12:30:45.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 --2- 172.21.15.63:0/2907051214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c103670 0x7f903c0fcce0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 --2- 172.21.15.63:0/2907051214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c102cd0 0x7f903c1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 -- 172.21.15.63:0/2907051214 >> 172.21.15.63:0/2907051214 conn(0x7f903c0f8820 msgr2=0x7f903c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 -- 172.21.15.63:0/2907051214 shutdown_connections 2024-03-23T12:30:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.071+0000 7f9043a00700 1 -- 172.21.15.63:0/2907051214 wait complete. 2024-03-23T12:30:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.072+0000 7f9043a00700 1 Processor -- start 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.072+0000 7f9043a00700 1 -- start start 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9043a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c102cd0 0x7f903c10ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9043a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c103670 0x7f903c10b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9043a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c10ed90 con 0x7f903c102cd0 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9043a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c10eed0 con 0x7f903c103670 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9041000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c102cd0 0x7f903c10ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9041000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c102cd0 0x7f903c10ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33348/0 (socket says 172.21.15.63:33348) 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9041000700 1 -- 172.21.15.63:0/1890373914 learned_addr learned my addr 172.21.15.63:0/1890373914 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f903be00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c103670 0x7f903c10b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9041000700 1 -- 172.21.15.63:0/1890373914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c103670 msgr2=0x7f903c10b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9041000700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c103670 0x7f903c10b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.073+0000 7f9041000700 1 -- 172.21.15.63:0/1890373914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028005040 con 0x7f903c102cd0 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.074+0000 7f9041000700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c102cd0 0x7f903c10ae10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9030002a50 tx=0x7f90300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.074+0000 7f9038c00700 1 -- 172.21.15.63:0/1890373914 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9030007cc0 con 0x7f903c102cd0 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.074+0000 7f9038c00700 1 -- 172.21.15.63:0/1890373914 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f903000f040 con 0x7f903c102cd0 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.074+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c10b8f0 con 0x7f903c102cd0 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.074+0000 7f9038c00700 1 -- 172.21.15.63:0/1890373914 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f903000d9c0 con 0x7f903c102cd0 2024-03-23T12:30:45.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.074+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c11e450 con 0x7f903c102cd0 2024-03-23T12:30:45.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.075+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c0fe420 con 0x7f903c102cd0 2024-03-23T12:30:45.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.076+0000 7f9038c00700 1 -- 172.21.15.63:0/1890373914 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f903000db60 con 0x7f903c102cd0 2024-03-23T12:30:45.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.077+0000 7f9038c00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f902406c7c0 0x7f902406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:45.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.077+0000 7f9038c00700 1 -- 172.21.15.63:0/1890373914 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f903008b0c0 con 0x7f903c102cd0 2024-03-23T12:30:45.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.078+0000 7f903be00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f902406c7c0 0x7f902406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:45.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.078+0000 7f903be00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f902406c7c0 0x7f902406ec70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f903c10c540 tx=0x7f9028009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:45.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.082+0000 7f9038c00700 1 -- 172.21.15.63:0/1890373914 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f90300552c0 con 0x7f903c102cd0 2024-03-23T12:30:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.296+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f903c109c90 con 0x7f903c102cd0 2024-03-23T12:30:45.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.297+0000 7f9038c00700 1 -- 172.21.15.63:0/1890373914 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f90300588e0 con 0x7f903c102cd0 2024-03-23T12:30:45.275 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T12:30:45.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f902406c7c0 msgr2=0x7f902406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f902406c7c0 0x7f902406ec70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f903c10c540 tx=0x7f9028009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c102cd0 msgr2=0x7f903c10ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c102cd0 0x7f903c10ae10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9030002a50 tx=0x7f90300046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 shutdown_connections 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f902406c7c0 0x7f902406ec70 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f903c102cd0 0x7f903c10ae10 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 --2- 172.21.15.63:0/1890373914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c103670 0x7f903c10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 >> 172.21.15.63:0/1890373914 conn(0x7f903c0f8820 msgr2=0x7f903c0f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 shutdown_connections 2024-03-23T12:30:45.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.300+0000 7f9043a00700 1 -- 172.21.15.63:0/1890373914 wait complete. 2024-03-23T12:30:45.295 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 -- 172.21.15.63:0/3968449502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 msgr2=0x7fcb100706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 --2- 172.21.15.63:0/3968449502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb100706d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fcb04004970 tx=0x7fcb04009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 -- 172.21.15.63:0/3968449502 shutdown_connections 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 --2- 172.21.15.63:0/3968449502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10070c10 0x7fcb10109070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 --2- 172.21.15.63:0/3968449502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb100706d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 -- 172.21.15.63:0/3968449502 >> 172.21.15.63:0/3968449502 conn(0x7fcb100f8810 msgr2=0x7fcb100fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 -- 172.21.15.63:0/3968449502 shutdown_connections 2024-03-23T12:30:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.512+0000 7fcb18600700 1 -- 172.21.15.63:0/3968449502 wait complete. 2024-03-23T12:30:45.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.513+0000 7fcb18600700 1 Processor -- start 2024-03-23T12:30:45.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.513+0000 7fcb18600700 1 -- start start 2024-03-23T12:30:45.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.513+0000 7fcb18600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb101158f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:45.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.513+0000 7fcb18600700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10070c10 0x7fcb10115e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:45.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.513+0000 7fcb18600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb10116480 con 0x7fcb10070300 2024-03-23T12:30:45.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.513+0000 7fcb18600700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1010fa80 con 0x7fcb10070c10 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.513+0000 7fcb15c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb101158f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb101158f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33360/0 (socket says 172.21.15.63:33360) 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15c00700 1 -- 172.21.15.63:0/1086981687 learned_addr learned my addr 172.21.15.63:0/1086981687 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15200700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10070c10 0x7fcb10115e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15c00700 1 -- 172.21.15.63:0/1086981687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10070c10 msgr2=0x7fcb10115e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15c00700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10070c10 0x7fcb10115e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15c00700 1 -- 172.21.15.63:0/1086981687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04005040 con 0x7fcb10070300 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15200700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10070c10 0x7fcb10115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb15c00700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb101158f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fcb04005010 tx=0x7fcb04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb0d600700 1 -- 172.21.15.63:0/1086981687 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0400e040 con 0x7fcb10070300 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb0d600700 1 -- 172.21.15.63:0/1086981687 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb04009930 con 0x7fcb10070300 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb0d600700 1 -- 172.21.15.63:0/1086981687 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0401c3f0 con 0x7fcb10070300 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.514+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1010fd00 con 0x7fcb10070300 2024-03-23T12:30:45.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.515+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb10110240 con 0x7fcb10070300 2024-03-23T12:30:45.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.516+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb10071a40 con 0x7fcb10070300 2024-03-23T12:30:45.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.517+0000 7fcb0d600700 1 -- 172.21.15.63:0/1086981687 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fcb040136b0 con 0x7fcb10070300 2024-03-23T12:30:45.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.518+0000 7fcb0d600700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcaf806c7c0 0x7fcaf806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:45.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.518+0000 7fcb0d600700 1 -- 172.21.15.63:0/1086981687 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcb0408cd10 con 0x7fcb10070300 2024-03-23T12:30:45.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.518+0000 7fcb15200700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcaf806c7c0 0x7fcaf806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:45.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.518+0000 7fcb15200700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcaf806c7c0 0x7fcaf806ec70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcafc004600 tx=0x7fcafc00b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:45.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.522+0000 7fcb0d600700 1 -- 172.21.15.63:0/1086981687 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcb0405a9a0 con 0x7fcb10070300 2024-03-23T12:30:45.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:45 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3360227561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T12:30:45.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:45 smithi063 ceph-mon[141124]: pgmap v1192: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:45.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:45 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2795216957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T12:30:45.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:45 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3360227561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T12:30:45.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:45 smithi111 ceph-mon[106899]: pgmap v1192: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:45.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:45 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2795216957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T12:30:45.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.754+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcb1010beb0 con 0x7fcb10070300 2024-03-23T12:30:45.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.754+0000 7fcb0d600700 1 -- 172.21.15.63:0/1086981687 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2319 (secure 0 0 0) 0x7fcb0405a530 con 0x7fcb10070300 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6829/2516647170 2024-03-24T11:57:27.441051+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3192464709 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/3712729872 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2812522808 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6828/3542552256 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/245431515 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3356174638 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2483586841 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2981669421 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/213424997 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/1052733058 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2062408886 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:45.732 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2603926600 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/2193815059 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6824/1161744664 2024-03-24T11:44:05.102065+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6826/3230334326 2024-03-24T11:57:27.441037+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2286910220 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/213424997 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6825/1161744664 2024-03-24T11:44:05.102065+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/1052733058 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1379382252 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3148717714 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/1704436637 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6827/3230334326 2024-03-24T11:57:27.441037+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/335624352 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/1704436637 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6829/3542552256 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3491519284 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1183338694 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2975088208 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1697976918 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3227457751 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/3254210644 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3688530508 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/2862601802 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3670139931 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/2193815059 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/335624352 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:45.734 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6828/2516647170 2024-03-24T11:57:27.441051+0000 2024-03-23T12:30:45.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcaf806c7c0 msgr2=0x7fcaf806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcaf806c7c0 0x7fcaf806ec70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcafc004600 tx=0x7fcafc00b040 comp rx=0 tx=0).stop 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 msgr2=0x7fcb101158f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb101158f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fcb04005010 tx=0x7fcb04009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 shutdown_connections 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fcaf806c7c0 0x7fcaf806ec70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb10070300 0x7fcb101158f0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 --2- 172.21.15.63:0/1086981687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10070c10 0x7fcb10115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 >> 172.21.15.63:0/1086981687 conn(0x7fcb100f8810 msgr2=0x7fcb100f9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 shutdown_connections 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:45.757+0000 7fcb18600700 1 -- 172.21.15.63:0/1086981687 wait complete. 2024-03-23T12:30:45.736 INFO:teuthology.orchestra.run.smithi063.stderr:listed 39 entries 2024-03-23T12:30:45.752 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-23T12:30:45.752 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T12:30:45.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T12:30:46.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 -- 172.21.15.63:0/3856732995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c401036a0 msgr2=0x7f5c400fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 --2- 172.21.15.63:0/3856732995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c401036a0 0x7f5c400fcd10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f5c30004940 tx=0x7f5c30009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:46.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 -- 172.21.15.63:0/3856732995 shutdown_connections 2024-03-23T12:30:46.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 --2- 172.21.15.63:0/3856732995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c401036a0 0x7f5c400fcd10 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 --2- 172.21.15.63:0/3856732995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c40102d00 0x7f5c401030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 -- 172.21.15.63:0/3856732995 >> 172.21.15.63:0/3856732995 conn(0x7f5c400f8830 msgr2=0x7f5c400fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:46.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 -- 172.21.15.63:0/3856732995 shutdown_connections 2024-03-23T12:30:46.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 -- 172.21.15.63:0/3856732995 wait complete. 2024-03-23T12:30:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.055+0000 7f5c48800700 1 Processor -- start 2024-03-23T12:30:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c48800700 1 -- start start 2024-03-23T12:30:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c48800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c40102d00 0x7f5c40113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c48800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c401036a0 0x7f5c40113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c48800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40114300 con 0x7f5c40102d00 2024-03-23T12:30:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c48800700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4010d900 con 0x7f5c401036a0 2024-03-23T12:30:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c45e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c40102d00 0x7f5c40113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c45e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c40102d00 0x7f5c40113770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33378/0 (socket says 172.21.15.63:33378) 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c45e00700 1 -- 172.21.15.63:0/973112433 learned_addr learned my addr 172.21.15.63:0/973112433 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.056+0000 7f5c45400700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c401036a0 0x7f5c40113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c45e00700 1 -- 172.21.15.63:0/973112433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c401036a0 msgr2=0x7f5c40113cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c45e00700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c401036a0 0x7f5c40113cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c45e00700 1 -- 172.21.15.63:0/973112433 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30005040 con 0x7f5c40102d00 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c45e00700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c40102d00 0x7f5c40113770 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400eb40 tx=0x7f5c3400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c3a000700 1 -- 172.21.15.63:0/973112433 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c34007cc0 con 0x7f5c40102d00 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4010db80 con 0x7f5c40102d00 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c3a000700 1 -- 172.21.15.63:0/973112433 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c3400f040 con 0x7f5c40102d00 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c3a000700 1 -- 172.21.15.63:0/973112433 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c3400c990 con 0x7f5c40102d00 2024-03-23T12:30:46.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.057+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4010e110 con 0x7f5c40102d00 2024-03-23T12:30:46.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.058+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c400fe450 con 0x7f5c40102d00 2024-03-23T12:30:46.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.059+0000 7f5c3a000700 1 -- 172.21.15.63:0/973112433 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f5c3400caf0 con 0x7f5c40102d00 2024-03-23T12:30:46.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.060+0000 7f5c3a000700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c2806c7c0 0x7f5c2806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.060+0000 7f5c3a000700 1 -- 172.21.15.63:0/973112433 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5c3408b640 con 0x7f5c40102d00 2024-03-23T12:30:46.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.060+0000 7f5c45400700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c2806c7c0 0x7f5c2806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.061+0000 7f5c45400700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c2806c7c0 0x7f5c2806ec70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5c4010ebe0 tx=0x7f5c30009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:46.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.065+0000 7f5c3a000700 1 -- 172.21.15.63:0/973112433 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5c34059380 con 0x7f5c40102d00 2024-03-23T12:30:46.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.284+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c40109cd0 con 0x7f5c40102d00 2024-03-23T12:30:46.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.285+0000 7f5c3a000700 1 -- 172.21.15.63:0/973112433 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16563 (secure 0 0 0) 0x7f5c34058f10 con 0x7f5c40102d00 2024-03-23T12:30:46.263 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-23T12:30:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":86,"fsid":"6880ffb6-e909-11ee-95cd-87774f69a715","created":"2024-03-23T11:35:55.761926+0000","modified":"2024-03-23T12:13:23.282611+0000","last_up_change":"2024-03-23T11:57:22.490793+0000","last_in_change":"2024-03-23T11:42:00.992881+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:40:53.432498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-23T11:43:48.146323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-23T11:43:48.722717+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e4f29eb-375c-4f37-a8d8-f82b40650ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6803","nonce":2164791827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6805","nonce":2164791827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6809","nonce":2164791827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2164791827},{"type":"v1","addr":"172.21.15.63:6807","nonce":2164791827}]},"public_addr":"172.21.15.63:6803/2164791827","cluster_addr":"172.21.15.63:6805/2164791827","heartbeat_back_addr":"172.21.15.63:6809/2164791827","heartbeat_front_addr":"172.21.15.63:6807/2164791827","state":["exists","up"]},{"osd":1,"uuid":"5e3ddf8b-e2c3-4188-b495-ac2743805f33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6811","nonce":3286820315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6813","nonce":3286820315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6817","nonce":3286820315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3286820315},{"type":"v1","addr":"172.21.15.63:6815","nonce":3286820315}]},"public_addr":"172.21.15.63:6811/3286820315","cluster_addr":"172.21.15.63:6813/3286820315","heartbeat_back_addr":"172.21.15.63:6817/3286820315","heartbeat_front_addr":"172.21.15.63:6815/3286820315","state":["exists","up"]},{"osd":2,"uuid":"edc4babe-dff9-462c-9af7-75e30b8095af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6819","nonce":3910577428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6821","nonce":3910577428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6825","nonce":3910577428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3910577428},{"type":"v1","addr":"172.21.15.63:6823","nonce":3910577428}]},"public_addr":"172.21.15.63:6819/3910577428","cluster_addr":"172.21.15.63:6821/3910577428","heartbeat_back_addr":"172.21.15.63:6825/3910577428","heartbeat_front_addr":"172.21.15.63:6823/3910577428","state":["exists","up"]},{"osd":3,"uuid":"7bc94d72-a814-447a-8112-9d50058a2759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6801","nonce":1545952189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6803","nonce":1545952189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6807","nonce":1545952189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1545952189},{"type":"v1","addr":"172.21.15.111:6805","nonce":1545952189}]},"public_addr":"172.21.15.111:6801/1545952189","cluster_addr":"172.21.15.111:6803/1545952189","heartbeat_back_addr":"172.21.15.111:6807/1545952189","heartbeat_front_addr":"172.21.15.111:6805/1545952189","state":["exists","up"]},{"osd":4,"uuid":"7fe6d51e-0cf0-4a39-9a3f-d57632cbdd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6809","nonce":2121500845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6811","nonce":2121500845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6815","nonce":2121500845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2121500845},{"type":"v1","addr":"172.21.15.111:6813","nonce":2121500845}]},"public_addr":"172.21.15.111:6809/2121500845","cluster_addr":"172.21.15.111:6811/2121500845","heartbeat_back_addr":"172.21.15.111:6815/2121500845","heartbeat_front_addr":"172.21.15.111:6813/2121500845","state":["exists","up"]},{"osd":5,"uuid":"2219a929-a842-4b41-ab46-096be6ef4d13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6817","nonce":983556337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6819","nonce":983556337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6823","nonce":983556337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":983556337},{"type":"v1","addr":"172.21.15.111:6821","nonce":983556337}]},"public_addr":"172.21.15.111:6817/983556337","cluster_addr":"172.21.15.111:6819/983556337","heartbeat_back_addr":"172.21.15.111:6823/983556337","heartbeat_front_addr":"172.21.15.111:6821/983556337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:39:42.374999+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:17.321830+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:40:50.840928+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:20.211360+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:41:50.314996+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:42:20.376924+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6829/2516647170":"2024-03-24T11:57:27.441051+0000","172.21.15.63:0/3192464709":"2024-03-24T11:36:25.877145+0000","172.21.15.111:0/3712729872":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/2812522808":"2024-03-24T11:46:26.114316+0000","172.21.15.111:6828/3542552256":"2024-03-24T11:48:52.950910+0000","172.21.15.111:0/245431515":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3356174638":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/2483586841":"2024-03-24T11:46:26.114316+0000","172.21.15.63:0/2981669421":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6801/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6800/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/2062408886":"2024-03-24T11:52:25.664441+0000","172.21.15.63:0/2603926600":"2024-03-24T11:46:26.114316+0000","172.21.15.63:6801/2193815059":"2024-03-24T11:46:26.114316+0000","172.21.15.111:6824/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:6826/3230334326":"2024-03-24T11:57:27.441037+0000","172.21.15.63:0/2286910220":"2024-03-24T11:36:25.877145+0000","172.21.15.63:6800/213424997":"2024-03-24T11:38:14.792281+0000","172.21.15.111:6825/1161744664":"2024-03-24T11:44:05.102065+0000","172.21.15.63:6801/1052733058":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1379382252":"2024-03-24T11:37:01.693515+0000","172.21.15.63:0/3148717714":"2024-03-24T11:38:14.792281+0000","172.21.15.63:6800/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.63:6827/3230334326":"2024-03-24T11:57:27.441037+0000","172.21.15.63:6800/335624352":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6801/1704436637":"2024-03-24T11:37:01.693515+0000","172.21.15.111:6829/3542552256":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3491519284":"2024-03-24T11:36:25.877145+0000","172.21.15.63:0/1183338694":"2024-03-24T11:38:14.792281+0000","172.21.15.63:0/2975088208":"2024-03-24T11:46:26.114316+0000","172.21.15.63:0/1697976918":"2024-03-24T11:52:25.664441+0000","172.21.15.63:0/3227457751":"2024-03-24T11:37:01.693515+0000","172.21.15.111:0/3254210644":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3688530508":"2024-03-24T11:52:25.664441+0000","172.21.15.111:0/2862601802":"2024-03-24T11:48:52.950910+0000","172.21.15.63:0/3670139931":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6800/2193815059":"2024-03-24T11:46:26.114316+0000","172.21.15.63:6801/335624352":"2024-03-24T11:52:25.664441+0000","172.21.15.63:6828/2516647170":"2024-03-24T11:57:27.441051+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T12:30:46.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.287+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c2806c7c0 msgr2=0x7f5c2806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c2806c7c0 0x7f5c2806ec70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5c4010ebe0 tx=0x7f5c30009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c40102d00 msgr2=0x7f5c40113770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c40102d00 0x7f5c40113770 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400eb40 tx=0x7f5c3400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 shutdown_connections 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f5c2806c7c0 0x7f5c2806ec70 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c40102d00 0x7f5c40113770 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 --2- 172.21.15.63:0/973112433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c401036a0 0x7f5c40113cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 >> 172.21.15.63:0/973112433 conn(0x7f5c400f8830 msgr2=0x7f5c400f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 shutdown_connections 2024-03-23T12:30:46.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.288+0000 7f5c48800700 1 -- 172.21.15.63:0/973112433 wait complete. 2024-03-23T12:30:46.283 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:40:53.432498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-23T11:43:48.146323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-23T11:43:48.722717+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T12:30:46.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 -- 172.21.15.63:0/2031488634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac103680 msgr2=0x7f28ac0fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 --2- 172.21.15.63:0/2031488634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac103680 0x7f28ac0fccf0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2898004940 tx=0x7f2898009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 -- 172.21.15.63:0/2031488634 shutdown_connections 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 --2- 172.21.15.63:0/2031488634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac103680 0x7f28ac0fccf0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 --2- 172.21.15.63:0/2031488634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28ac102ce0 0x7f28ac1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 -- 172.21.15.63:0/2031488634 >> 172.21.15.63:0/2031488634 conn(0x7f28ac0f8830 msgr2=0x7f28ac0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 -- 172.21.15.63:0/2031488634 shutdown_connections 2024-03-23T12:30:46.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.502+0000 7f28b0e00700 1 -- 172.21.15.63:0/2031488634 wait complete. 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.503+0000 7f28b0e00700 1 Processor -- start 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.503+0000 7f28b0e00700 1 -- start start 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.504+0000 7f28b0e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac102ce0 0x7f28ac1137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.504+0000 7f28b0e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28ac103680 0x7f28ac113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.504+0000 7f28b0e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac114340 con 0x7f28ac102ce0 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.504+0000 7f28b0e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac10d940 con 0x7f28ac103680 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.504+0000 7f28a9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac102ce0 0x7f28ac1137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.504+0000 7f28a9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac102ce0 0x7f28ac1137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33398/0 (socket says 172.21.15.63:33398) 2024-03-23T12:30:46.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.504+0000 7f28a9e00700 1 -- 172.21.15.63:0/3355017925 learned_addr learned my addr 172.21.15.63:0/3355017925 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a9400700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28ac103680 0x7f28ac113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a9e00700 1 -- 172.21.15.63:0/3355017925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28ac103680 msgr2=0x7f28ac113cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a9e00700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28ac103680 0x7f28ac113cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a9e00700 1 -- 172.21.15.63:0/3355017925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2898005040 con 0x7f28ac102ce0 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a9400700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28ac103680 0x7f28ac113cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a9e00700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac102ce0 0x7f28ac1137b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f289c002a50 tx=0x7f289c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a2000700 1 -- 172.21.15.63:0/3355017925 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289c007cc0 con 0x7f28ac102ce0 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a2000700 1 -- 172.21.15.63:0/3355017925 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f289c00f040 con 0x7f28ac102ce0 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ac10dc20 con 0x7f28ac102ce0 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.505+0000 7f28a2000700 1 -- 172.21.15.63:0/3355017925 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289c009540 con 0x7f28ac102ce0 2024-03-23T12:30:46.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.506+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ac10e1e0 con 0x7f28ac102ce0 2024-03-23T12:30:46.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.506+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ac0fe430 con 0x7f28ac102ce0 2024-03-23T12:30:46.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.507+0000 7f28a2000700 1 -- 172.21.15.63:0/3355017925 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f289c0096a0 con 0x7f28ac102ce0 2024-03-23T12:30:46.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.508+0000 7f28a2000700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289006c7c0 0x7f289006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.508+0000 7f28a2000700 1 -- 172.21.15.63:0/3355017925 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f289c08b820 con 0x7f28ac102ce0 2024-03-23T12:30:46.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.509+0000 7f28a9400700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289006c7c0 0x7f289006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.509+0000 7f28a9400700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289006c7c0 0x7f289006ec70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2898009fd0 tx=0x7f2898009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:46.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.513+0000 7f28a2000700 1 -- 172.21.15.63:0/3355017925 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f289c0594b0 con 0x7f28ac102ce0 2024-03-23T12:30:46.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:46 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1890373914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T12:30:46.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:46 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1086981687' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-23T12:30:46.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:46 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1890373914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T12:30:46.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:46 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1086981687' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-23T12:30:46.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.727+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28ac109c80 con 0x7f28ac102ce0 2024-03-23T12:30:46.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.727+0000 7f28a2000700 1 -- 172.21.15.63:0/3355017925 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f289c059040 con 0x7f28ac102ce0 2024-03-23T12:30:46.705 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-03-23T12:30:46.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289006c7c0 msgr2=0x7f289006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289006c7c0 0x7f289006ec70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2898009fd0 tx=0x7f2898009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:46.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac102ce0 msgr2=0x7f28ac1137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac102ce0 0x7f28ac1137b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f289c002a50 tx=0x7f289c0046b0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 shutdown_connections 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f289006c7c0 0x7f289006ec70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac102ce0 0x7f28ac1137b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 --2- 172.21.15.63:0/3355017925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28ac103680 0x7f28ac113cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 >> 172.21.15.63:0/3355017925 conn(0x7f28ac0f8830 msgr2=0x7f28ac0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 shutdown_connections 2024-03-23T12:30:46.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.730+0000 7f28b0e00700 1 -- 172.21.15.63:0/3355017925 wait complete. 2024-03-23T12:30:46.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-23T12:30:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 -- 172.21.15.63:0/1817977321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 msgr2=0x7f8f600fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1817977321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f600fccf0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c004940 tx=0x7f8f4c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 -- 172.21.15.63:0/1817977321 shutdown_connections 2024-03-23T12:30:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1817977321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f600fccf0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1817977321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f60102ce0 0x7f8f601030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 -- 172.21.15.63:0/1817977321 >> 172.21.15.63:0/1817977321 conn(0x7f8f600f8830 msgr2=0x7f8f600fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:46.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 -- 172.21.15.63:0/1817977321 shutdown_connections 2024-03-23T12:30:46.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.946+0000 7f8f66a00700 1 -- 172.21.15.63:0/1817977321 wait complete. 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.947+0000 7f8f66a00700 1 Processor -- start 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.948+0000 7f8f66a00700 1 -- start start 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.948+0000 7f8f66a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f60102ce0 0x7f8f601137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.948+0000 7f8f66a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f60113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.948+0000 7f8f66a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60114340 con 0x7f8f60103680 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.948+0000 7f8f66a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6010d940 con 0x7f8f60102ce0 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f60113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f60113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33420/0 (socket says 172.21.15.63:33420) 2024-03-23T12:30:46.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5f400700 1 -- 172.21.15.63:0/1812124903 learned_addr learned my addr 172.21.15.63:0/1812124903 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:46.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5fe00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f60102ce0 0x7f8f601137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5f400700 1 -- 172.21.15.63:0/1812124903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f60102ce0 msgr2=0x7f8f601137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5f400700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f60102ce0 0x7f8f601137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5f400700 1 -- 172.21.15.63:0/1812124903 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c005040 con 0x7f8f60103680 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5fe00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f60102ce0 0x7f8f601137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f5f400700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f60113cf0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009390 tx=0x7f8f4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f57e00700 1 -- 172.21.15.63:0/1812124903 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f4c00e040 con 0x7f8f60103680 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6010dbc0 con 0x7f8f60103680 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.949+0000 7f8f57e00700 1 -- 172.21.15.63:0/1812124903 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f4c013420 con 0x7f8f60103680 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.950+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6010e180 con 0x7f8f60103680 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.950+0000 7f8f57e00700 1 -- 172.21.15.63:0/1812124903 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f4c0255e0 con 0x7f8f60103680 2024-03-23T12:30:46.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.951+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f600fe430 con 0x7f8f60103680 2024-03-23T12:30:46.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.952+0000 7f8f57e00700 1 -- 172.21.15.63:0/1812124903 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8f4c013a00 con 0x7f8f60103680 2024-03-23T12:30:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.952+0000 7f8f57e00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8f4406c7c0 0x7f8f4406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.953+0000 7f8f57e00700 1 -- 172.21.15.63:0/1812124903 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8f4c08cdd0 con 0x7f8f60103680 2024-03-23T12:30:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.953+0000 7f8f5fe00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8f4406c7c0 0x7f8f4406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:46.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.954+0000 7f8f5fe00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8f4406c7c0 0x7f8f4406ec70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009fd0 tx=0x7f8f5000b040 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:46.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:46.957+0000 7f8f57e00700 1 -- 172.21.15.63:0/1812124903 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8f4c05aa60 con 0x7f8f60103680 2024-03-23T12:30:47.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.171+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f8f60109c80 con 0x7f8f60103680 2024-03-23T12:30:47.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.171+0000 7f8f57e00700 1 -- 172.21.15.63:0/1812124903 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8f4c05a5f0 con 0x7f8f60103680 2024-03-23T12:30:47.149 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T12:30:47.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8f4406c7c0 msgr2=0x7f8f4406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8f4406c7c0 0x7f8f4406ec70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009fd0 tx=0x7f8f5000b040 comp rx=0 tx=0).stop 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 msgr2=0x7f8f60113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f60113cf0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009390 tx=0x7f8f4c009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 shutdown_connections 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8f4406c7c0 0x7f8f4406ec70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f60102ce0 0x7f8f601137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 --2- 172.21.15.63:0/1812124903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f60103680 0x7f8f60113cf0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 >> 172.21.15.63:0/1812124903 conn(0x7f8f600f8830 msgr2=0x7f8f600f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 shutdown_connections 2024-03-23T12:30:47.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.174+0000 7f8f66a00700 1 -- 172.21.15.63:0/1812124903 wait complete. 2024-03-23T12:30:47.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-23T12:30:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 -- 172.21.15.63:0/315653651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc102ce0 msgr2=0x7fe2cc1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/315653651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc102ce0 0x7fe2cc1030b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc004940 tx=0x7fe2bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 -- 172.21.15.63:0/315653651 shutdown_connections 2024-03-23T12:30:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/315653651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc103680 0x7fe2cc064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/315653651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc102ce0 0x7fe2cc1030b0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 -- 172.21.15.63:0/315653651 >> 172.21.15.63:0/315653651 conn(0x7fe2cc072ad0 msgr2=0x7fe2cc072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 -- 172.21.15.63:0/315653651 shutdown_connections 2024-03-23T12:30:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.389+0000 7fe2d0a00700 1 -- 172.21.15.63:0/315653651 wait complete. 2024-03-23T12:30:47.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.390+0000 7fe2d0a00700 1 Processor -- start 2024-03-23T12:30:47.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.390+0000 7fe2d0a00700 1 -- start start 2024-03-23T12:30:47.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2d0a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc102ce0 0x7fe2cc115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:47.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2d0a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc103680 0x7fe2cc115e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2d0a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc1164c0 con 0x7fe2cc103680 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2d0a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc10fac0 con 0x7fe2cc102ce0 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc103680 0x7fe2cc115e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc103680 0x7fe2cc115e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33436/0 (socket says 172.21.15.63:33436) 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c3e00700 1 -- 172.21.15.63:0/2383364014 learned_addr learned my addr 172.21.15.63:0/2383364014 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c9e00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc102ce0 0x7fe2cc115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c3e00700 1 -- 172.21.15.63:0/2383364014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc102ce0 msgr2=0x7fe2cc115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c3e00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc102ce0 0x7fe2cc115930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c3e00700 1 -- 172.21.15.63:0/2383364014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc005040 con 0x7fe2cc103680 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.391+0000 7fe2c9e00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc102ce0 0x7fe2cc115930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.392+0000 7fe2c3e00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc103680 0x7fe2cc115e70 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400eb40 tx=0x7fe2b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.392+0000 7fe2c2000700 1 -- 172.21.15.63:0/2383364014 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2b4007cc0 con 0x7fe2cc103680 2024-03-23T12:30:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.392+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc10fd40 con 0x7fe2cc103680 2024-03-23T12:30:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.392+0000 7fe2c2000700 1 -- 172.21.15.63:0/2383364014 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2b400f040 con 0x7fe2cc103680 2024-03-23T12:30:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.392+0000 7fe2c2000700 1 -- 172.21.15.63:0/2383364014 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2b4009570 con 0x7fe2cc103680 2024-03-23T12:30:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.392+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc110300 con 0x7fe2cc103680 2024-03-23T12:30:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.393+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc1061e0 con 0x7fe2cc103680 2024-03-23T12:30:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.394+0000 7fe2c2000700 1 -- 172.21.15.63:0/2383364014 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7fe2b40096d0 con 0x7fe2cc103680 2024-03-23T12:30:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.395+0000 7fe2c2000700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe2b006c7c0 0x7fe2b006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.395+0000 7fe2c2000700 1 -- 172.21.15.63:0/2383364014 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe2b408b650 con 0x7fe2cc103680 2024-03-23T12:30:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.395+0000 7fe2c9e00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe2b006c7c0 0x7fe2b006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.396+0000 7fe2c9e00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe2b006c7c0 0x7fe2b006ec70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0136f0 tx=0x7fe2bc009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:47.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.399+0000 7fe2c2000700 1 -- 172.21.15.63:0/2383364014 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe2b4059310 con 0x7fe2cc103680 2024-03-23T12:30:47.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.617+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fe2cc110f80 con 0x7fe2cc103680 2024-03-23T12:30:47.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.618+0000 7fe2c2000700 1 -- 172.21.15.63:0/2383364014 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe2b4058ea0 con 0x7fe2cc103680 2024-03-23T12:30:47.596 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-03-23T12:30:47.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.620+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe2b006c7c0 msgr2=0x7fe2b006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.620+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe2b006c7c0 0x7fe2b006ec70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0136f0 tx=0x7fe2bc009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:47.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.620+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc103680 msgr2=0x7fe2cc115e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.620+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc103680 0x7fe2cc115e70 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400eb40 tx=0x7fe2b400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:47.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.621+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 shutdown_connections 2024-03-23T12:30:47.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.621+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7fe2b006c7c0 0x7fe2b006ec70 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.621+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc102ce0 0x7fe2cc115930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.621+0000 7fe2d0a00700 1 --2- 172.21.15.63:0/2383364014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2cc103680 0x7fe2cc115e70 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.621+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 >> 172.21.15.63:0/2383364014 conn(0x7fe2cc072ad0 msgr2=0x7fe2cc0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:47.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.621+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 shutdown_connections 2024-03-23T12:30:47.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.621+0000 7fe2d0a00700 1 -- 172.21.15.63:0/2383364014 wait complete. 2024-03-23T12:30:47.613 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:47 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/973112433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T12:30:47.613 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:47 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/3355017925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T12:30:47.613 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:47 smithi063 ceph-mon[141124]: pgmap v1193: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:47.613 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:47 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1812124903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T12:30:47.616 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T12:30:47.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:47 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/973112433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-23T12:30:47.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:47 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/3355017925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-23T12:30:47.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:47 smithi111 ceph-mon[106899]: pgmap v1193: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:47.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:47 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1812124903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-23T12:30:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.834+0000 7f8c18a00700 1 -- 172.21.15.63:0/873470456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070c10 msgr2=0x7f8c14109050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.834+0000 7f8c18a00700 1 --2- 172.21.15.63:0/873470456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070c10 0x7f8c14109050 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8c00004970 tx=0x7f8c00009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.835+0000 7f8c18a00700 1 -- 172.21.15.63:0/873470456 shutdown_connections 2024-03-23T12:30:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.835+0000 7f8c18a00700 1 --2- 172.21.15.63:0/873470456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070c10 0x7f8c14109050 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.835+0000 7f8c18a00700 1 --2- 172.21.15.63:0/873470456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c14070300 0x7f8c140706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.835+0000 7f8c18a00700 1 -- 172.21.15.63:0/873470456 >> 172.21.15.63:0/873470456 conn(0x7f8c140f8830 msgr2=0x7f8c140fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.835+0000 7f8c18a00700 1 -- 172.21.15.63:0/873470456 shutdown_connections 2024-03-23T12:30:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.835+0000 7f8c18a00700 1 -- 172.21.15.63:0/873470456 wait complete. 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.835+0000 7f8c18a00700 1 Processor -- start 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c18a00700 1 -- start start 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c18a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070300 0x7f8c14115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c18a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c14070c10 0x7f8c14115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c18a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141165a0 con 0x7f8c14070300 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c18a00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1410fba0 con 0x7f8c14070c10 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c11e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070300 0x7f8c14115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c11e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070300 0x7f8c14115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33464/0 (socket says 172.21.15.63:33464) 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c11e00700 1 -- 172.21.15.63:0/1193348038 learned_addr learned my addr 172.21.15.63:0/1193348038 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-23T12:30:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.836+0000 7f8c11400700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c14070c10 0x7f8c14115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c11e00700 1 -- 172.21.15.63:0/1193348038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c14070c10 msgr2=0x7f8c14115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c11e00700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c14070c10 0x7f8c14115f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c11e00700 1 -- 172.21.15.63:0/1193348038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00005040 con 0x7f8c14070300 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c11e00700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070300 0x7f8c14115a10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400eb40 tx=0x7f8c0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c0a000700 1 -- 172.21.15.63:0/1193348038 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c04007cc0 con 0x7f8c14070300 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c0a000700 1 -- 172.21.15.63:0/1193348038 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c0400f040 con 0x7f8c14070300 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1410fe20 con 0x7f8c14070300 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c0a000700 1 -- 172.21.15.63:0/1193348038 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c04009570 con 0x7f8c14070300 2024-03-23T12:30:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.837+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c14110330 con 0x7f8c14070300 2024-03-23T12:30:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.838+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c14071a40 con 0x7f8c14070300 2024-03-23T12:30:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.839+0000 7f8c0a000700 1 -- 172.21.15.63:0/1193348038 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 39) v1 ==== 90524+0+0 (secure 0 0 0) 0x7f8c040096d0 con 0x7f8c14070300 2024-03-23T12:30:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.840+0000 7f8c0a000700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8bf806c7c0 0x7f8bf806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T12:30:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.840+0000 7f8c0a000700 1 -- 172.21.15.63:0/1193348038 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c0408b760 con 0x7f8c14070300 2024-03-23T12:30:47.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.840+0000 7f8c11400700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8bf806c7c0 0x7f8bf806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T12:30:47.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.841+0000 7f8c11400700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8bf806c7c0 0x7f8bf806ec70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009fd0 tx=0x7f8c00009e30 comp rx=0 tx=0).ready entity=mgr.24659 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T12:30:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:47.845+0000 7f8c0a000700 1 -- 172.21.15.63:0/1193348038 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8c040594a0 con 0x7f8c14070300 2024-03-23T12:30:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.082+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8c14110ff0 con 0x7f8c14070300 2024-03-23T12:30:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.083+0000 7f8c0a000700 1 -- 172.21.15.63:0/1193348038 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2319 (secure 0 0 0) 0x7f8c04059030 con 0x7f8c14070300 2024-03-23T12:30:48.060 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6829/2516647170 2024-03-24T11:57:27.441051+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3192464709 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/3712729872 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2812522808 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6828/3542552256 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/245431515 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3356174638 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2483586841 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2981669421 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/213424997 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/1052733058 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2062408886 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2603926600 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/2193815059 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:48.061 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6824/1161744664 2024-03-24T11:44:05.102065+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6826/3230334326 2024-03-24T11:57:27.441037+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2286910220 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/213424997 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6825/1161744664 2024-03-24T11:44:05.102065+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/1052733058 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1379382252 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3148717714 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/1704436637 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6827/3230334326 2024-03-24T11:57:27.441037+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/335624352 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:48.062 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/1704436637 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:6829/3542552256 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3491519284 2024-03-24T11:36:25.877145+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1183338694 2024-03-24T11:38:14.792281+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2975088208 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1697976918 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3227457751 2024-03-24T11:37:01.693515+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/3254210644 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3688530508 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.111:0/2862601802 2024-03-24T11:48:52.950910+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3670139931 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/2193815059 2024-03-24T11:46:26.114316+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/335624352 2024-03-24T11:52:25.664441+0000 2024-03-23T12:30:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6828/2516647170 2024-03-24T11:57:27.441051+0000 2024-03-23T12:30:48.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8bf806c7c0 msgr2=0x7f8bf806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8bf806c7c0 0x7f8bf806ec70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009fd0 tx=0x7f8c00009e30 comp rx=0 tx=0).stop 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070300 msgr2=0x7f8c14115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070300 0x7f8c14115a10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400eb40 tx=0x7f8c0400b040 comp rx=0 tx=0).stop 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 shutdown_connections 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:6800/3534621850,v1:172.21.15.63:6801/3534621850] conn(0x7f8bf806c7c0 0x7f8bf806ec70 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c14070300 0x7f8c14115a10 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 --2- 172.21.15.63:0/1193348038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c14070c10 0x7f8c14115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.085+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 >> 172.21.15.63:0/1193348038 conn(0x7f8c140f8830 msgr2=0x7f8c140f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.086+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 shutdown_connections 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-23T12:30:48.086+0000 7f8c18a00700 1 -- 172.21.15.63:0/1193348038 wait complete. 2024-03-23T12:30:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:listed 39 entries 2024-03-23T12:30:48.081 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi063.front.sepia.ceph.com... 2024-03-23T12:30:48.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T12:30:48.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T12:30:48.149 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T12:30:48.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:48 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/2383364014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T12:30:48.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:48 smithi063 ceph-mon[141124]: from='client.? 172.21.15.63:0/1193348038' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-23T12:30:48.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:48 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/2383364014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-23T12:30:48.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:48 smithi111 ceph-mon[106899]: from='client.? 172.21.15.63:0/1193348038' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-23T12:30:49.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:49 smithi063 ceph-mon[141124]: pgmap v1194: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:49.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:49 smithi111 ceph-mon[106899]: pgmap v1194: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:30:52.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:51 smithi063 ceph-mon[141124]: pgmap v1195: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:52.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:51 smithi111 ceph-mon[106899]: pgmap v1195: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:53.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:52 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:30:53.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:52 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:30:54.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:53 smithi063 ceph-mon[141124]: pgmap v1196: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:54.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:53 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:30:54.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:53 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:30:54.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:53 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:30:54.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:53 smithi111 ceph-mon[106899]: pgmap v1196: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:30:54.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:53 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:30:54.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:53 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:30:54.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:53 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:30:56.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:55 smithi111 ceph-mon[106899]: pgmap v1197: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:56.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:56.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:55 smithi063 ceph-mon[141124]: pgmap v1197: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:56.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:30:58.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:57 smithi111 ceph-mon[106899]: pgmap v1198: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:30:58.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:57 smithi063 ceph-mon[141124]: pgmap v1198: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:00.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:30:59 smithi111 ceph-mon[106899]: pgmap v1199: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:00.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:30:59 smithi063 ceph-mon[141124]: pgmap v1199: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:02.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:01 smithi111 ceph-mon[106899]: pgmap v1200: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:02.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:01 smithi063 ceph-mon[141124]: pgmap v1200: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:04.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:03 smithi111 ceph-mon[106899]: pgmap v1201: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:04.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:03 smithi063 ceph-mon[141124]: pgmap v1201: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:06.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:05 smithi063 ceph-mon[141124]: pgmap v1202: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:06.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:05 smithi111 ceph-mon[106899]: pgmap v1202: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:08.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:07 smithi063 ceph-mon[141124]: pgmap v1203: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:08.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:07 smithi111 ceph-mon[106899]: pgmap v1203: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:10.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:09 smithi063 ceph-mon[141124]: pgmap v1204: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:10.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:09 smithi111 ceph-mon[106899]: pgmap v1204: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:11.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:11.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:12.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:11 smithi063 ceph-mon[141124]: pgmap v1205: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:12.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:11 smithi111 ceph-mon[106899]: pgmap v1205: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:14.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:13 smithi063 ceph-mon[141124]: pgmap v1206: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:14.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:13 smithi111 ceph-mon[106899]: pgmap v1206: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:16.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:15 smithi063 ceph-mon[141124]: pgmap v1207: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:16.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:15 smithi111 ceph-mon[106899]: pgmap v1207: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:18.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:17 smithi063 ceph-mon[141124]: pgmap v1208: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:18.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:17 smithi111 ceph-mon[106899]: pgmap v1208: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:20.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:19 smithi063 ceph-mon[141124]: pgmap v1209: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:20.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:19 smithi111 ceph-mon[106899]: pgmap v1209: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:22.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:21 smithi063 ceph-mon[141124]: pgmap v1210: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:22.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:21 smithi111 ceph-mon[106899]: pgmap v1210: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:24.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:23 smithi063 ceph-mon[141124]: pgmap v1211: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:24.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:23 smithi111 ceph-mon[106899]: pgmap v1211: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:25 smithi063 ceph-mon[141124]: pgmap v1212: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:26.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:25 smithi111 ceph-mon[106899]: pgmap v1212: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:26.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:28.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:27 smithi063 ceph-mon[141124]: pgmap v1213: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:28.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:27 smithi111 ceph-mon[106899]: pgmap v1213: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:30.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:29 smithi063 ceph-mon[141124]: pgmap v1214: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:30.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:29 smithi111 ceph-mon[106899]: pgmap v1214: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:32.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:31 smithi063 ceph-mon[141124]: pgmap v1215: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:32.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:31 smithi111 ceph-mon[106899]: pgmap v1215: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:34.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:33 smithi063 ceph-mon[141124]: pgmap v1216: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:34.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:33 smithi111 ceph-mon[106899]: pgmap v1216: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:36.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:35 smithi063 ceph-mon[141124]: pgmap v1217: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:36.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:35 smithi111 ceph-mon[106899]: pgmap v1217: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:38.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:37 smithi063 ceph-mon[141124]: pgmap v1218: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:38.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:37 smithi111 ceph-mon[106899]: pgmap v1218: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:40.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:39 smithi063 ceph-mon[141124]: pgmap v1219: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:40.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:39 smithi111 ceph-mon[106899]: pgmap v1219: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:41.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:41.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:42.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:42 smithi063 ceph-mon[141124]: pgmap v1220: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:42.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:42 smithi111 ceph-mon[106899]: pgmap v1220: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:44.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:44 smithi063 ceph-mon[141124]: pgmap v1221: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:44.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:44 smithi111 ceph-mon[106899]: pgmap v1221: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:46.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:46 smithi063 ceph-mon[141124]: pgmap v1222: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:46.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:46 smithi111 ceph-mon[106899]: pgmap v1222: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:48.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:48 smithi063 ceph-mon[141124]: pgmap v1223: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:48.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:48 smithi111 ceph-mon[106899]: pgmap v1223: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:50.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:50 smithi063 ceph-mon[141124]: pgmap v1224: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:50.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:50 smithi111 ceph-mon[106899]: pgmap v1224: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:31:52.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:52 smithi063 ceph-mon[141124]: pgmap v1225: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:52.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:52 smithi111 ceph-mon[106899]: pgmap v1225: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:54.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:54 smithi063 ceph-mon[141124]: pgmap v1226: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:54.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:31:54.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:54 smithi111 ceph-mon[106899]: pgmap v1226: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:31:54.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:31:55.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:31:55.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:31:55.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:31:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:31:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:31:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:31:56.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:56 smithi063 ceph-mon[141124]: pgmap v1227: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:56.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:56 smithi111 ceph-mon[106899]: pgmap v1227: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:56.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:31:58.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:31:58 smithi063 ceph-mon[141124]: pgmap v1228: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:31:58.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:31:58 smithi111 ceph-mon[106899]: pgmap v1228: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:00.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:00 smithi111 ceph-mon[106899]: pgmap v1229: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:00.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:00 smithi063 ceph-mon[141124]: pgmap v1229: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:02.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:02 smithi111 ceph-mon[106899]: pgmap v1230: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:02.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:02 smithi063 ceph-mon[141124]: pgmap v1230: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:04.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:04 smithi111 ceph-mon[106899]: pgmap v1231: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:04.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:04 smithi063 ceph-mon[141124]: pgmap v1231: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:06.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:06 smithi063 ceph-mon[141124]: pgmap v1232: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:06.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:06 smithi111 ceph-mon[106899]: pgmap v1232: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:08.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:08 smithi063 ceph-mon[141124]: pgmap v1233: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:08.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:08 smithi111 ceph-mon[106899]: pgmap v1233: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:10.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:10 smithi063 ceph-mon[141124]: pgmap v1234: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:10.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:10 smithi111 ceph-mon[106899]: pgmap v1234: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:11.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:11.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:12.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:12 smithi063 ceph-mon[141124]: pgmap v1235: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:12.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:12 smithi111 ceph-mon[106899]: pgmap v1235: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:14.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:14 smithi063 ceph-mon[141124]: pgmap v1236: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:14 smithi111 ceph-mon[106899]: pgmap v1236: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:16.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:16 smithi063 ceph-mon[141124]: pgmap v1237: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:16.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:16 smithi111 ceph-mon[106899]: pgmap v1237: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:18.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:18 smithi063 ceph-mon[141124]: pgmap v1238: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:18.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:18 smithi111 ceph-mon[106899]: pgmap v1238: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:20.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:20 smithi063 ceph-mon[141124]: pgmap v1239: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:20.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:20 smithi111 ceph-mon[106899]: pgmap v1239: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:22.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:22 smithi063 ceph-mon[141124]: pgmap v1240: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:22.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:22 smithi111 ceph-mon[106899]: pgmap v1240: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:24.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:24 smithi063 ceph-mon[141124]: pgmap v1241: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:24.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:24 smithi111 ceph-mon[106899]: pgmap v1241: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:26.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:26 smithi063 ceph-mon[141124]: pgmap v1242: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:26.626 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:26.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:26 smithi111 ceph-mon[106899]: pgmap v1242: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:26.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:28.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:28 smithi063 ceph-mon[141124]: pgmap v1243: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:28.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:28 smithi111 ceph-mon[106899]: pgmap v1243: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:30.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:30 smithi063 ceph-mon[141124]: pgmap v1244: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:30.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:30 smithi111 ceph-mon[106899]: pgmap v1244: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:32.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:32 smithi063 ceph-mon[141124]: pgmap v1245: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:32:32.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:32 smithi111 ceph-mon[106899]: pgmap v1245: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:32:34.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:34 smithi063 ceph-mon[141124]: pgmap v1246: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:32:34.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:34 smithi111 ceph-mon[106899]: pgmap v1246: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:32:36.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:36 smithi063 ceph-mon[141124]: pgmap v1247: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:36.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:36 smithi111 ceph-mon[106899]: pgmap v1247: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:38.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:38 smithi063 ceph-mon[141124]: pgmap v1248: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:38.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:38 smithi111 ceph-mon[106899]: pgmap v1248: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:40.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:40 smithi063 ceph-mon[141124]: pgmap v1249: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T12:32:40.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:40 smithi111 ceph-mon[106899]: pgmap v1249: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T12:32:41.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:41.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:42.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:42 smithi063 ceph-mon[141124]: pgmap v1250: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:42.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:42 smithi111 ceph-mon[106899]: pgmap v1250: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:43.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:43 smithi063 ceph-mon[141124]: pgmap v1251: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:43.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:43 smithi111 ceph-mon[106899]: pgmap v1251: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:32:46.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:45 smithi111 ceph-mon[106899]: pgmap v1252: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:46.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:45 smithi063 ceph-mon[141124]: pgmap v1252: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:48.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:47 smithi111 ceph-mon[106899]: pgmap v1253: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:48.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:47 smithi063 ceph-mon[141124]: pgmap v1253: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:50.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:49 smithi063 ceph-mon[141124]: pgmap v1254: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:50.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:49 smithi111 ceph-mon[106899]: pgmap v1254: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:32:52.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:51 smithi063 ceph-mon[141124]: pgmap v1255: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:52.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:51 smithi111 ceph-mon[106899]: pgmap v1255: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:32:54.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:53 smithi063 ceph-mon[141124]: pgmap v1256: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:32:54.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:53 smithi111 ceph-mon[106899]: pgmap v1256: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:32:55.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:54 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:32:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:54 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:32:56.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:55 smithi063 ceph-mon[141124]: pgmap v1257: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:56.377 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:32:56.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:32:56.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:32:56.378 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:55 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:55 smithi111 ceph-mon[106899]: pgmap v1257: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:32:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:32:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:32:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:55 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:32:58.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:57 smithi063 ceph-mon[141124]: pgmap v1258: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:32:58.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:57 smithi111 ceph-mon[106899]: pgmap v1258: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:33:00.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:32:59 smithi063 ceph-mon[141124]: pgmap v1259: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T12:33:00.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:32:59 smithi111 ceph-mon[106899]: pgmap v1259: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-23T12:33:02.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:01 smithi063 ceph-mon[141124]: pgmap v1260: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:33:02.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:01 smithi111 ceph-mon[106899]: pgmap v1260: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-23T12:33:04.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:03 smithi063 ceph-mon[141124]: pgmap v1261: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:33:04.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:03 smithi111 ceph-mon[106899]: pgmap v1261: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-23T12:33:06.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:05 smithi063 ceph-mon[141124]: pgmap v1262: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:06.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:05 smithi111 ceph-mon[106899]: pgmap v1262: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:08.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:07 smithi063 ceph-mon[141124]: pgmap v1263: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:08.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:07 smithi111 ceph-mon[106899]: pgmap v1263: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:10.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:09 smithi063 ceph-mon[141124]: pgmap v1264: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:10.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:09 smithi111 ceph-mon[106899]: pgmap v1264: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:11.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:10 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:11.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:10 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:12.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:11 smithi063 ceph-mon[141124]: pgmap v1265: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:12.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:11 smithi111 ceph-mon[106899]: pgmap v1265: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:14.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:13 smithi063 ceph-mon[141124]: pgmap v1266: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:14.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:13 smithi111 ceph-mon[106899]: pgmap v1266: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:16.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:15 smithi063 ceph-mon[141124]: pgmap v1267: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:16.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:15 smithi111 ceph-mon[106899]: pgmap v1267: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:18.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:17 smithi063 ceph-mon[141124]: pgmap v1268: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:18.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:17 smithi111 ceph-mon[106899]: pgmap v1268: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:20.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:19 smithi063 ceph-mon[141124]: pgmap v1269: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:20.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:19 smithi111 ceph-mon[106899]: pgmap v1269: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:22.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:21 smithi063 ceph-mon[141124]: pgmap v1270: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:22.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:21 smithi111 ceph-mon[106899]: pgmap v1270: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:24.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:23 smithi063 ceph-mon[141124]: pgmap v1271: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:24.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:23 smithi111 ceph-mon[106899]: pgmap v1271: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:26.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:25 smithi063 ceph-mon[141124]: pgmap v1272: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:26.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:26.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:26 smithi111 ceph-mon[106899]: pgmap v1272: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:26.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:28.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:28 smithi063 ceph-mon[141124]: pgmap v1273: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:28.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:28 smithi111 ceph-mon[106899]: pgmap v1273: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:30.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:30 smithi063 ceph-mon[141124]: pgmap v1274: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:30.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:30 smithi111 ceph-mon[106899]: pgmap v1274: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:32.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:32 smithi063 ceph-mon[141124]: pgmap v1275: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:32.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:32 smithi111 ceph-mon[106899]: pgmap v1275: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:34.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:34 smithi063 ceph-mon[141124]: pgmap v1276: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:34.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:34 smithi111 ceph-mon[106899]: pgmap v1276: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:36.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:36 smithi063 ceph-mon[141124]: pgmap v1277: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:36.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:36 smithi111 ceph-mon[106899]: pgmap v1277: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:38.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:38 smithi063 ceph-mon[141124]: pgmap v1278: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:38.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:38 smithi111 ceph-mon[106899]: pgmap v1278: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:40.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:40 smithi063 ceph-mon[141124]: pgmap v1279: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:40.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:40 smithi111 ceph-mon[106899]: pgmap v1279: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:41.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:41.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:42.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:42 smithi063 ceph-mon[141124]: pgmap v1280: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:42.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:42 smithi111 ceph-mon[106899]: pgmap v1280: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:44.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:44 smithi063 ceph-mon[141124]: pgmap v1281: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:44.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:44 smithi111 ceph-mon[106899]: pgmap v1281: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:46.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:46 smithi063 ceph-mon[141124]: pgmap v1282: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:46.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:46 smithi111 ceph-mon[106899]: pgmap v1282: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:48.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:48 smithi063 ceph-mon[141124]: pgmap v1283: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:48.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:48 smithi111 ceph-mon[106899]: pgmap v1283: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:50.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:50 smithi063 ceph-mon[141124]: pgmap v1284: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:50.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:50 smithi111 ceph-mon[106899]: pgmap v1284: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:33:52.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:52 smithi063 ceph-mon[141124]: pgmap v1285: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:52.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:52 smithi111 ceph-mon[106899]: pgmap v1285: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:54.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:54 smithi063 ceph-mon[141124]: pgmap v1286: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:54.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:54 smithi111 ceph-mon[106899]: pgmap v1286: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:33:56.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:56 smithi063 ceph-mon[141124]: pgmap v1287: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:56.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:33:56.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:56 smithi111 ceph-mon[106899]: pgmap v1287: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:33:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:33:57.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:33:57.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:33:57.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:33:57.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:33:57.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:33:57.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:33:58.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:33:58 smithi063 ceph-mon[141124]: pgmap v1288: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:33:58.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:33:58 smithi111 ceph-mon[106899]: pgmap v1288: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:00.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:00 smithi063 ceph-mon[141124]: pgmap v1289: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:00.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:00 smithi111 ceph-mon[106899]: pgmap v1289: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:02.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:02 smithi063 ceph-mon[141124]: pgmap v1290: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:02.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:02 smithi111 ceph-mon[106899]: pgmap v1290: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:04.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:04 smithi063 ceph-mon[141124]: pgmap v1291: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:04.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:04 smithi111 ceph-mon[106899]: pgmap v1291: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:06.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:06 smithi063 ceph-mon[141124]: pgmap v1292: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:06.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:06 smithi111 ceph-mon[106899]: pgmap v1292: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:08.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:08 smithi063 ceph-mon[141124]: pgmap v1293: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:08.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:08 smithi111 ceph-mon[106899]: pgmap v1293: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:10.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:10 smithi063 ceph-mon[141124]: pgmap v1294: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:10.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:10 smithi111 ceph-mon[106899]: pgmap v1294: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:11.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:11.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:12.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:12 smithi063 ceph-mon[141124]: pgmap v1295: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:12.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:12 smithi111 ceph-mon[106899]: pgmap v1295: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:13.625 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:13 smithi063 ceph-mon[141124]: pgmap v1296: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:13.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:13 smithi111 ceph-mon[106899]: pgmap v1296: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:16.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:15 smithi063 ceph-mon[141124]: pgmap v1297: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:16.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:15 smithi111 ceph-mon[106899]: pgmap v1297: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:18.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:17 smithi063 ceph-mon[141124]: pgmap v1298: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:18.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:17 smithi111 ceph-mon[106899]: pgmap v1298: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:20.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:19 smithi063 ceph-mon[141124]: pgmap v1299: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:20.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:19 smithi111 ceph-mon[106899]: pgmap v1299: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:22.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:21 smithi063 ceph-mon[141124]: pgmap v1300: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:22.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:21 smithi111 ceph-mon[106899]: pgmap v1300: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:24.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:23 smithi063 ceph-mon[141124]: pgmap v1301: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:24.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:23 smithi111 ceph-mon[106899]: pgmap v1301: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:26.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:25 smithi063 ceph-mon[141124]: pgmap v1302: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:26.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:25 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:26.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:25 smithi111 ceph-mon[106899]: pgmap v1302: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:26.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:25 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:28.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:27 smithi063 ceph-mon[141124]: pgmap v1303: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:28.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:27 smithi111 ceph-mon[106899]: pgmap v1303: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:30.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:29 smithi063 ceph-mon[141124]: pgmap v1304: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:30.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:29 smithi111 ceph-mon[106899]: pgmap v1304: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:32.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:31 smithi063 ceph-mon[141124]: pgmap v1305: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:32.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:31 smithi111 ceph-mon[106899]: pgmap v1305: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:34.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:33 smithi063 ceph-mon[141124]: pgmap v1306: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:34.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:33 smithi111 ceph-mon[106899]: pgmap v1306: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:36.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:35 smithi063 ceph-mon[141124]: pgmap v1307: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:36.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:35 smithi111 ceph-mon[106899]: pgmap v1307: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:38.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:37 smithi063 ceph-mon[141124]: pgmap v1308: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:38.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:37 smithi111 ceph-mon[106899]: pgmap v1308: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:40.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:39 smithi063 ceph-mon[141124]: pgmap v1309: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:40.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:39 smithi111 ceph-mon[106899]: pgmap v1309: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:41.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:40 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:41.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:40 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:42.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:41 smithi063 ceph-mon[141124]: pgmap v1310: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:42.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:41 smithi111 ceph-mon[106899]: pgmap v1310: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:44.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:43 smithi063 ceph-mon[141124]: pgmap v1311: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:44.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:43 smithi111 ceph-mon[106899]: pgmap v1311: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:46.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:46 smithi063 ceph-mon[141124]: pgmap v1312: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:46.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:46 smithi111 ceph-mon[106899]: pgmap v1312: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:48.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:48 smithi063 ceph-mon[141124]: pgmap v1313: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:48.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:48 smithi111 ceph-mon[106899]: pgmap v1313: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:50.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:50 smithi063 ceph-mon[141124]: pgmap v1314: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:50.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:50 smithi111 ceph-mon[106899]: pgmap v1314: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:34:52.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:52 smithi063 ceph-mon[141124]: pgmap v1315: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:52.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:52 smithi111 ceph-mon[106899]: pgmap v1315: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:54.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:54 smithi063 ceph-mon[141124]: pgmap v1316: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:54.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:54 smithi111 ceph-mon[106899]: pgmap v1316: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:34:56.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:56 smithi063 ceph-mon[141124]: pgmap v1317: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:56.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:56 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:56 smithi111 ceph-mon[106899]: pgmap v1317: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:56.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:56 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:34:57.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:57 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:34:57.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:57 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-23T12:34:58.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:58 smithi063 ceph-mon[141124]: pgmap v1318: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:58.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:58 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:34:58.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:58 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:34:58.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:34:58 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:34:58.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:58 smithi111 ceph-mon[106899]: pgmap v1318: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:34:58.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:58 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-23T12:34:58.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:58 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-23T12:34:58.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:34:58 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' 2024-03-23T12:35:00.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:00 smithi063 ceph-mon[141124]: pgmap v1319: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:00.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:00 smithi111 ceph-mon[106899]: pgmap v1319: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:02.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:02 smithi063 ceph-mon[141124]: pgmap v1320: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:02.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:02 smithi111 ceph-mon[106899]: pgmap v1320: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:04.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:04 smithi063 ceph-mon[141124]: pgmap v1321: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:04.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:04 smithi111 ceph-mon[106899]: pgmap v1321: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:06.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:06 smithi063 ceph-mon[141124]: pgmap v1322: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:06.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:06 smithi111 ceph-mon[106899]: pgmap v1322: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:08.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:08 smithi063 ceph-mon[141124]: pgmap v1323: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:08.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:08 smithi111 ceph-mon[106899]: pgmap v1323: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:10.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:10 smithi063 ceph-mon[141124]: pgmap v1324: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:10.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:10 smithi111 ceph-mon[106899]: pgmap v1324: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:11.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:11 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:35:11.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:11 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:35:12.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:12 smithi063 ceph-mon[141124]: pgmap v1325: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:12.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:12 smithi111 ceph-mon[106899]: pgmap v1325: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:14.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:14 smithi063 ceph-mon[141124]: pgmap v1326: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:14.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:14 smithi111 ceph-mon[106899]: pgmap v1326: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:16.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:16 smithi063 ceph-mon[141124]: pgmap v1327: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:16.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:16 smithi111 ceph-mon[106899]: pgmap v1327: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:18.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:18 smithi063 ceph-mon[141124]: pgmap v1328: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:18.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:18 smithi111 ceph-mon[106899]: pgmap v1328: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:20.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:20 smithi063 ceph-mon[141124]: pgmap v1329: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:20.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:20 smithi111 ceph-mon[106899]: pgmap v1329: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:22.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:22 smithi063 ceph-mon[141124]: pgmap v1330: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:22.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:22 smithi111 ceph-mon[106899]: pgmap v1330: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:24.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:24 smithi063 ceph-mon[141124]: pgmap v1331: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:24.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:24 smithi111 ceph-mon[106899]: pgmap v1331: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:26.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:26 smithi063 ceph-mon[141124]: pgmap v1332: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:26.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:26 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:35:26.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:26 smithi111 ceph-mon[106899]: pgmap v1332: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:26.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:26 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:35:28.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:28 smithi063 ceph-mon[141124]: pgmap v1333: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:28.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:28 smithi111 ceph-mon[106899]: pgmap v1333: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:30.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:30 smithi063 ceph-mon[141124]: pgmap v1334: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:30.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:30 smithi111 ceph-mon[106899]: pgmap v1334: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:32.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:32 smithi063 ceph-mon[141124]: pgmap v1335: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:32.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:32 smithi111 ceph-mon[106899]: pgmap v1335: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:34.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:34 smithi063 ceph-mon[141124]: pgmap v1336: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:34.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:34 smithi111 ceph-mon[106899]: pgmap v1336: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:36.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:36 smithi063 ceph-mon[141124]: pgmap v1337: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:36.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:36 smithi111 ceph-mon[106899]: pgmap v1337: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:38.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:38 smithi063 ceph-mon[141124]: pgmap v1338: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:38.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:38 smithi111 ceph-mon[106899]: pgmap v1338: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:40.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:40 smithi063 ceph-mon[141124]: pgmap v1339: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:40.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:40 smithi111 ceph-mon[106899]: pgmap v1339: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-23T12:35:41.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:41 smithi063 ceph-mon[141124]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:35:41.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:41 smithi111 ceph-mon[106899]: from='mgr.24659 172.21.15.63:0/4201786560' entity='mgr.smithi063.gehkcr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-23T12:35:42.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:42 smithi063 ceph-mon[141124]: pgmap v1340: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:42.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:42 smithi111 ceph-mon[106899]: pgmap v1340: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:44.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:44 smithi063 ceph-mon[141124]: pgmap v1341: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:44.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:44 smithi111 ceph-mon[106899]: pgmap v1341: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-23T12:35:46.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:46 smithi063 ceph-mon[141124]: pgmap v1342: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:46.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:46 smithi111 ceph-mon[106899]: pgmap v1342: 65 pgs: 65 active+clean; 271 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-23T12:35:47.216 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-03-23T12:35:47.216 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/ceph_fuse.py", line 183, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-23T12:35:47.217 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-23T12:35:47.228 INFO:tasks.cephadm:Teardown begin 2024-03-23T12:35:47.228 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/cephadm.py", line 1750, in task File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/ceph_fuse.py", line 183, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-23T12:35:47.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T12:35:47.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T12:35:47.388 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-23T12:35:47.388 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-23T12:35:47.435 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-23T12:35:47.481 INFO:tasks.cephadm:Stopping all daemons... 2024-03-23T12:35:47.482 INFO:tasks.cephadm.mon.smithi063:Stopping mon.smithi063... 2024-03-23T12:35:47.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063 2024-03-23T12:35:47.877 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:47 smithi063 systemd[1]: Stopping Ceph mon.smithi063 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:35:48.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:48 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:35:48.012+0000 7fc32c800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi063 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:35:48.375 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:48 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063[141101]: 2024-03-23T12:35:48.012+0000 7fc32c800700 -1 mon.smithi063@0(leader) e3 *** Got Signal Terminated *** 2024-03-23T12:35:48.875 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Mar 23 12:35:48 smithi063 bash[207873]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi063 2024-03-23T12:35:49.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi063.service' 2024-03-23T12:35:49.386 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:35:49.386 INFO:tasks.cephadm.mon.smithi063:Stopped mon.smithi063 2024-03-23T12:35:49.386 INFO:tasks.cephadm.mon.smithi111:Stopping mon.smithi111... 2024-03-23T12:35:49.386 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi111 2024-03-23T12:35:49.798 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:49 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:35:50.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:49 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi111[106876]: 2024-03-23T12:35:49.819+0000 7f8470e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:35:50.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:49 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi111[106876]: 2024-03-23T12:35:49.819+0000 7f8470e00700 -1 mon.smithi111@1(peon) e3 *** Got Signal Terminated *** 2024-03-23T12:35:50.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 23 12:35:50 smithi111 bash[155379]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-mon-smithi111 2024-03-23T12:35:50.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@mon.smithi111.service' 2024-03-23T12:35:51.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:35:51.039 INFO:tasks.cephadm.mon.smithi111:Stopped mon.smithi111 2024-03-23T12:35:51.039 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-23T12:35:51.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.0 2024-03-23T12:35:51.375 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 12:35:51 smithi063 systemd[1]: Stopping Ceph osd.0 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:35:51.875 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 12:35:51 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[154936]: 2024-03-23T12:35:51.466+0000 7f95ab600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:35:51.875 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 12:35:51 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[154936]: 2024-03-23T12:35:51.466+0000 7f95ab600700 -1 osd.0 86 *** Got signal Terminated *** 2024-03-23T12:35:51.875 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 12:35:51 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0[154936]: 2024-03-23T12:35:51.466+0000 7f95ab600700 -1 osd.0 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T12:35:57.625 INFO:journalctl@ceph.osd.0.smithi063.stdout:Mar 23 12:35:57 smithi063 bash[208031]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-0 2024-03-23T12:35:59.500 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.0.service' 2024-03-23T12:35:59.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:35:59.590 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-23T12:35:59.590 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-23T12:35:59.590 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.1 2024-03-23T12:35:59.875 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 12:35:59 smithi063 systemd[1]: Stopping Ceph osd.1 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:36:00.375 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 12:35:59 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[161475]: 2024-03-23T12:35:59.993+0000 7fa700e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:36:00.375 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 12:35:59 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[161475]: 2024-03-23T12:35:59.993+0000 7fa700e00700 -1 osd.1 86 *** Got signal Terminated *** 2024-03-23T12:36:00.375 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 12:35:59 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1[161475]: 2024-03-23T12:35:59.993+0000 7fa700e00700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T12:36:06.125 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 23 12:36:05 smithi063 bash[208299]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-1 2024-03-23T12:36:07.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.1.service' 2024-03-23T12:36:07.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:36:07.971 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-23T12:36:07.971 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-23T12:36:07.971 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.2 2024-03-23T12:36:08.125 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 12:36:08 smithi063 systemd[1]: Stopping Ceph osd.2 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:36:08.625 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 12:36:08 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[168226]: 2024-03-23T12:36:08.386+0000 7f92ade00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:36:08.625 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 12:36:08 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[168226]: 2024-03-23T12:36:08.386+0000 7f92ade00700 -1 osd.2 86 *** Got signal Terminated *** 2024-03-23T12:36:08.625 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 12:36:08 smithi063 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2[168226]: 2024-03-23T12:36:08.386+0000 7f92ade00700 -1 osd.2 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T12:36:14.375 INFO:journalctl@ceph.osd.2.smithi063.stdout:Mar 23 12:36:14 smithi063 bash[208559]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-2 2024-03-23T12:36:16.279 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.2.service' 2024-03-23T12:36:16.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:36:16.370 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-23T12:36:16.370 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-23T12:36:16.370 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.3 2024-03-23T12:36:16.895 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 12:36:16 smithi111 systemd[1]: Stopping Ceph osd.3 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:36:17.395 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 12:36:17 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T12:36:17.006+0000 7f4b73e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:36:17.395 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 12:36:17 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T12:36:17.006+0000 7f4b73e00700 -1 osd.3 86 *** Got signal Terminated *** 2024-03-23T12:36:17.395 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 12:36:17 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T12:36:17.006+0000 7f4b73e00700 -1 osd.3 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T12:36:20.645 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 12:36:20 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T12:36:20.582+0000 7f4b6ba00700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:53.522207+0000 front 2024-03-23T12:35:53.522154+0000 (oldest deadline 2024-03-23T12:36:20.522392+0000) 2024-03-23T12:36:20.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:20 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:20.377+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:21.352 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:21.043+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:21.610 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:21.375+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:21.895 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 12:36:21 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3[119804]: 2024-03-23T12:36:21.633+0000 7f4b6ba00700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:53.522207+0000 front 2024-03-23T12:35:53.522154+0000 (oldest deadline 2024-03-23T12:36:20.522392+0000) 2024-03-23T12:36:22.388 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:22 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:22.072+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:22.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:22 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:22.411+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:23.077 INFO:journalctl@ceph.osd.3.smithi111.stdout:Mar 23 12:36:22 smithi111 bash[155526]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-3 2024-03-23T12:36:23.395 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:23 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:23.099+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:23.779 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:23 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:23.419+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:24.395 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:24 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:24.130+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:24.838 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:24 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:24.419+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:24.864 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.3.service' 2024-03-23T12:36:24.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:36:24.951 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-23T12:36:24.951 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-23T12:36:24.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.4 2024-03-23T12:36:25.145 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:25 smithi111 systemd[1]: Stopping Ceph osd.4 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:36:25.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:25 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:25.089+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:25.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:25 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:25.383+0000 7f2493000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:36:25.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:25 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:25.383+0000 7f2493000700 -1 osd.4 86 *** Got signal Terminated *** 2024-03-23T12:36:25.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:25 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:25.383+0000 7f2493000700 -1 osd.4 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T12:36:25.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:25 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:25.408+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:26.386 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:26 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:26.047+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:26.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:26 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:26.409+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:27.395 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:27 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:27.025+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:27.395 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:27 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:27.025+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:27.895 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:27 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:27.430+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:28.358 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:27 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:27.977+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:28.359 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:27 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:27.977+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:28.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:28 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:28.382+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:29.318 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:28 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:28.977+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:29.318 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:28 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:28.977+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:29.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:29 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:29.341+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:30.334 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:29 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:29.951+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:30.335 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:29 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:29.951+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:30.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:30 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:30.358+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:54.118317+0000 front 2024-03-23T12:35:54.118331+0000 (oldest deadline 2024-03-23T12:36:20.018210+0000) 2024-03-23T12:36:30.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:30 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4[126489]: 2024-03-23T12:36:30.358+0000 7f248b600700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:04.119155+0000 front 2024-03-23T12:36:04.119251+0000 (oldest deadline 2024-03-23T12:36:30.018978+0000) 2024-03-23T12:36:31.395 INFO:journalctl@ceph.osd.4.smithi111.stdout:Mar 23 12:36:31 smithi111 bash[155796]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-4 2024-03-23T12:36:31.395 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:30 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:30.978+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:31.395 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:30 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:30.978+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:32.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:31 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:31.946+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:32.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:31 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:31.946+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:33.049 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:32 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:32.961+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:33.050 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:32 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:32.961+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:33.075 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.4.service' 2024-03-23T12:36:33.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:36:33.162 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-23T12:36:33.162 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-23T12:36:33.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.5 2024-03-23T12:36:33.394 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:33 smithi111 systemd[1]: Stopping Ceph osd.5 for 6880ffb6-e909-11ee-95cd-87774f69a715... 2024-03-23T12:36:33.892 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:33 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:33.560+0000 7f3bef800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-23T12:36:33.893 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:33 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:33.560+0000 7f3bef800700 -1 osd.5 86 *** Got signal Terminated *** 2024-03-23T12:36:33.893 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:33 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:33.560+0000 7f3bef800700 -1 osd.5 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-23T12:36:33.893 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:33 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:33.916+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:34.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:33 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:33.916+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:35.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:34 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:34.880+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:35.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:34 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:34.880+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:36.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:35 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:35.832+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:36.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:35 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:35.832+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:36.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:35 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:35.832+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6822 osd.2 since back 2024-03-23T12:36:12.751022+0000 front 2024-03-23T12:36:12.751087+0000 (oldest deadline 2024-03-23T12:36:35.050897+0000) 2024-03-23T12:36:37.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:36 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:36.877+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:37.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:36 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:36.877+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:37.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:36 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:36.877+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6822 osd.2 since back 2024-03-23T12:36:12.751022+0000 front 2024-03-23T12:36:12.751087+0000 (oldest deadline 2024-03-23T12:36:35.050897+0000) 2024-03-23T12:36:38.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:37 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:37.838+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-03-23T12:35:55.149237+0000 front 2024-03-23T12:35:55.149276+0000 (oldest deadline 2024-03-23T12:36:20.448935+0000) 2024-03-23T12:36:38.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:37 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:37.838+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-03-23T12:36:02.149991+0000 front 2024-03-23T12:36:02.149974+0000 (oldest deadline 2024-03-23T12:36:26.849767+0000) 2024-03-23T12:36:38.145 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:37 smithi111 ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5[133243]: 2024-03-23T12:36:37.838+0000 7f3be6a00700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.63:6822 osd.2 since back 2024-03-23T12:36:12.751022+0000 front 2024-03-23T12:36:12.751087+0000 (oldest deadline 2024-03-23T12:36:35.050897+0000) 2024-03-23T12:36:39.645 INFO:journalctl@ceph.osd.5.smithi111.stdout:Mar 23 12:36:39 smithi111 bash[156062]: ceph-6880ffb6-e909-11ee-95cd-87774f69a715-osd-5 2024-03-23T12:36:41.418 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6880ffb6-e909-11ee-95cd-87774f69a715@osd.5.service' 2024-03-23T12:36:41.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-23T12:36:41.504 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-23T12:36:41.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-23T12:36:45.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi063.stderr:ceph-fuse[95348]: fuse finished with error 0 and tester_r 0 2024-03-23T12:37:04.086 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-23T12:37:18.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T12:37:19.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-23T12:37:19.135 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-23T12:37:19.137 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239/remote/smithi063/crash 2024-03-23T12:37:19.138 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash -- . 2024-03-23T12:37:19.206 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-23T12:37:19.206 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-03-23T12:37:19.209 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239/remote/smithi111/crash 2024-03-23T12:37:19.210 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash -- . 2024-03-23T12:37:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-23T12:37:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-03-23T12:37:19.277 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-23T12:37:19.277 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-23T12:37:19.353 INFO:teuthology.orchestra.run.smithi063.stderr:grep: /var/log/ceph/6880ffb6-e909-11ee-95cd-87774f69a715/ceph.log: No such file or directory 2024-03-23T12:37:19.359 INFO:tasks.cephadm:Compressing logs... 2024-03-23T12:37:19.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T12:37:19.363 DEBUG:teuthology.orchestra.run.smithi111:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T12:37:19.425 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-23T12:37:19.428 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-23T12:37:37.658 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-03-23T12:40:07.528 INFO:tasks.cephadm:Archiving logs... 2024-03-23T12:40:07.528 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239/remote/smithi063/log 2024-03-23T12:40:07.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T12:40:25.266 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239/remote/smithi111/log 2024-03-23T12:40:25.267 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T12:40:31.717 INFO:tasks.cephadm:Removing cluster... 2024-03-23T12:40:31.717 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 --force 2024-03-23T12:40:32.851 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6880ffb6-e909-11ee-95cd-87774f69a715 --force 2024-03-23T12:40:33.687 INFO:tasks.cephadm:Removing cephadm ... 2024-03-23T12:40:33.687 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-23T12:40:33.734 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-23T12:40:33.778 INFO:tasks.cephadm:Teardown complete 2024-03-23T12:40:33.779 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-23T12:40:33.793 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/ceph_fuse.py", line 183, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-23T12:40:33.794 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-23T12:40:33.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T12:40:33.799 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T12:40:33.937 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-23T12:40:33.949 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-23T12:40:33.949 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T12:40:33.953 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T12:40:33.991 INFO:teuthology.orchestra.run.smithi063.stderr:bash: ntpq: command not found 2024-03-23T12:40:33.996 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2024-03-23T12:40:33.998 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T12:40:33.999 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-23T12:40:33.999 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +1429us[+1429us] +/- 137ms 2024-03-23T12:40:33.999 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv02.front.sepia.ceph.com 3 7 377 33 -723us[ -741us] +/- 53ms 2024-03-23T12:40:33.999 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 94 +261us[ +248us] +/- 79ms 2024-03-23T12:40:33.999 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T12:40:34.004 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T12:40:34.004 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-23T12:40:34.004 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +1408us[+1408us] +/- 137ms 2024-03-23T12:40:34.004 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 7 377 96 -737us[ -793us] +/- 52ms 2024-03-23T12:40:34.004 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 95 +247us[ +247us] +/- 79ms 2024-03-23T12:40:34.004 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T12:40:34.005 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-23T12:40:34.017 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-23T12:40:34.017 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-23T12:40:34.028 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T12:40:34.180 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T12:40:34.312 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T12:40:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T12:40:34.384 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T12:40:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T12:40:34.455 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-23T12:40:34.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-23T12:40:34.481 INFO:teuthology.task.internal:Duration was 4619.535257 seconds 2024-03-23T12:40:34.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-23T12:40:34.493 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-23T12:40:34.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T12:40:34.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T12:40:34.606 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T12:40:34.609 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T12:40:34.924 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-23T12:40:34.924 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-03-23T12:40:34.924 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T12:40:34.983 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-03-23T12:40:34.983 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T12:40:35.042 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-23T12:40:35.042 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T12:40:35.046 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T12:40:35.157 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-23T12:40:35.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T12:40:35.161 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T12:40:35.361 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-23T12:40:35.374 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-23T12:40:35.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T12:40:35.442 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T12:40:35.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-23T12:40:35.522 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T12:40:35.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T12:40:35.589 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-03-23T12:40:35.591 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-03-23T12:40:35.646 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T12:40:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T12:40:35.688 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T12:40:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T12:40:35.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-23T12:40:35.744 INFO:teuthology.task.internal:Transferring archived files... 2024-03-23T12:40:35.745 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239/remote/smithi063 2024-03-23T12:40:35.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T12:40:35.832 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239/remote/smithi111 2024-03-23T12:40:35.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T12:40:35.916 INFO:teuthology.task.internal:Removing archive directory... 2024-03-23T12:40:35.916 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T12:40:35.920 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T12:40:35.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-23T12:40:35.979 INFO:teuthology.task.internal:Not uploading archives. 2024-03-23T12:40:35.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-23T12:40:35.991 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-23T12:40:35.992 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T12:40:36.005 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T12:40:36.046 INFO:teuthology.orchestra.run.smithi063.stdout: 524320 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 23 12:40 /home/ubuntu/cephtest 2024-03-23T12:40:36.046 INFO:teuthology.orchestra.run.smithi063.stdout: 786900 4 d--------- 2 ubuntu ubuntu 4096 Mar 23 11:44 /home/ubuntu/cephtest/mnt.0 2024-03-23T12:40:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-03-23T12:40:36.048 INFO:teuthology.orchestra.run.smithi063.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-23T12:40:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T12:40:36.049 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/ceph_fuse.py", line 183, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-23T12:40:36.050 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-23T12:40:36.063 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-23T12:40:36.075 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-23T12:40:36.076 INFO:teuthology.nuke:Checking targets against current locks 2024-03-23T12:40:36.100 DEBUG:teuthology.nuke:shortname: smithi063 2024-03-23T12:40:36.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T12:40:36.122 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:12:55.021359', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZ/qoO9C16Im0WCrliXXl/Y2P2XOEe+HAy+M748RPelNmAHfXFhkVNM7bBHuNW7JJ1GoeUNXTomtEQMb0/j7u4='} 2024-03-23T12:40:36.126 DEBUG:teuthology.nuke:shortname: smithi111 2024-03-23T12:40:36.126 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T12:40:36.146 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:12:55.022832', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5j5A6YBMOOvWGAY3htlUVPNCZy+Qx5eeo7L3ouY7ywjmsNbxbhHOsxbHaOhwejKBEuJ2x7V4KNcQrCPKQuWY0='} 2024-03-23T12:40:36.168 INFO:teuthology.orchestra.console.smithi063:Power off 2024-03-23T12:40:36.168 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T12:40:36.192 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-03-23T12:40:36.192 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:36.205 INFO:teuthology.orchestra.console.smithi111:Power off 2024-03-23T12:40:36.206 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T12:40:36.214 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-23T12:40:36.226 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-03-23T12:40:36.227 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:36.249 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-23T12:40:40.215 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:40.250 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:40.325 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-23T12:40:40.361 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-23T12:40:44.330 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:44.362 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:44.443 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-23T12:40:44.472 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-23T12:40:48.448 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:48.473 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T12:40:48.559 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-03-23T12:40:48.582 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-03-23T12:40:48.661 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-03-23T12:40:48.683 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-03-23T12:40:48.790 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4619.535257101059 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default host.a-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 host.b-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 owner: scheduled_rishabh@teuthology status: fail success: false 2024-03-23T12:40:48.790 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T12:40:48.874 INFO:teuthology.run:FAIL